From toon.knapen at fft.be Thu Mar 1 02:04:08 2007 From: toon.knapen at fft.be (Toon Knapen) Date: Thu, 01 Mar 2007 08:04:08 +0100 Subject: [Numpy-discussion] numpy.dot and ACML In-Reply-To: <20070219171120.GA5609@kotnet.org> References: <20070219171120.GA5609@kotnet.org> Message-ID: <45E67AE8.60708@fft.be> Hi, I am also looking to verify the vendor-libs being used. What does numpy.__config__.show() tell you ? toon Yves Frederix wrote: > Hi all, > > I have managed to compile numpy using pathscale and ACML on a 64 bit AMD > system. Now I wanted to verify that numpy.dot indeed uses the ACML > libs. The example for dot() > (http://www.scipy.org/Numpy_Example_List?highlight=%28example%29#head-c7a573f030ff7cbaea62baf219599b3976136bac) suggest a way of doing this: > > 1 u0050015 at lo-03-02 .../core $ python -c "import numpy; print id(numpy.dot)==id(numpy.core.multiarray.dot);" > True > > This indicates that I am not using the acml libraries. > > When running a benchmark (see attach) and comparing to a non-ACML > installation though, the strange thing is that there is a clear > speed difference, suggesting again that the acml libraries are indeed > used. > > Because this is not all that clear to me, I was wondering whether there > exists an alternative way of verifying what libraries are used. > > Many thanks, > YVES > > > ------------------------------------------------------------------------ > > ACML: > > dim x.T*y x*y.T A*x A*B A.T*x > ----------------------------------------------------------------- > 5000 0.002492 0.002417 0.002412 0.002399 0.002416 > 50000 0.020074 0.020024 0.020004 0.020003 0.020024 > 100000 0.092777 0.093690 0.100220 0.093787 0.094250 > 200000 0.184933 0.198623 0.196120 0.197089 0.197273 > 300000 0.276583 0.279177 0.280898 0.284016 0.276204 > 500000 0.476340 0.481987 0.471875 0.480868 0.481501 > 1000000.0 0.892623 0.895500 0.915173 0.894815 0.922501 > 5000000.0 4.450555 4.465748 4.467870 4.468188 4.469083 > > No ACML: > > dim x.T*y x*y.T A*x A*B A.T*x > ----------------------------------------------------------------- > 5000 0.002523 0.002428 0.002410 0.002430 0.002419 > 50000 0.024756 0.061520 0.036575 0.036399 0.036450 > 100000 0.338576 0.353074 0.169472 0.302087 0.334633 > 200000 0.670803 0.735732 0.538166 0.649335 0.744496 > 300000 1.004381 1.269259 0.482542 2.194308 0.611997 > 500000 1.110656 1.504701 1.571736 1.656021 1.491146 > 1000000.0 2.182746 2.234478 2.254645 2.439508 2.537558 > 5000000.0 10.878910 16.578266 8.265109 8.905976 17.124400 > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From faltet at carabos.com Thu Mar 1 03:14:40 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 01 Mar 2007 09:14:40 +0100 Subject: [Numpy-discussion] PyCon 2007 In-Reply-To: <45E5EE39.2040905@ee.byu.edu> References: <45E5EE39.2040905@ee.byu.edu> Message-ID: <1172736880.2703.1.camel@localhost.localdomain> El dc 28 de 02 del 2007 a les 14:03 -0700, en/na Travis Oliphant va escriure: > I took the opportunity to go to PyCon this year and met several people > there. I had a really good time although I would have liked to stay > longer. If you want to see the slides for my talk they are here: > > http://us.pycon.org/common/talkdata/PyCon2007/045/PythonTalk.pdf > Excellent presentation. Congratullations Travis! -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From williams at astro.ox.ac.uk Thu Mar 1 05:45:12 2007 From: williams at astro.ox.ac.uk (Michael Williams) Date: Thu, 1 Mar 2007 10:45:12 +0000 Subject: [Numpy-discussion] NumPy in Teaching In-Reply-To: <45E4FFA6.9010408@shrogers.com> References: <45E4FFA6.9010408@shrogers.com> Message-ID: <20070301104511.GA23386@astro.ox.ac.uk> On Tue, Feb 27, 2007 at 09:05:58PM -0700, Steven H. Rogers wrote: > I'm doing an informal survey on the use of Array Programming Languages > for teaching. If you're using NumPy in this manner I'd like to hear > from you. What subject was/is taught, academic level, results, lessons > learned, etc. If Numeric counts, I used that back in 2002 as part of an introductory programming course I wrote for the Department of Physics at Oxford. We really only used to to provide an element-wise array method. Brief introduction: http://pentangle.net/python/pyzine.php The course (aka "Handbook") and report on the course's successes and failures: http://pentangle.net/python/ -- Mike From steve at shrogers.com Thu Mar 1 08:53:59 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Thu, 01 Mar 2007 06:53:59 -0700 Subject: [Numpy-discussion] Buffer PEP in NumPy SVN In-Reply-To: <45E5EA4D.1010303@ee.byu.edu> References: <45E5EA4D.1010303@ee.byu.edu> Message-ID: <45E6DAF7.3090007@shrogers.com> Travis Oliphant wrote: > I just wanted to point people to the online version of the PEP. I'm > still looking for comments and suggestions. The current version is here: > > http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt > > -Travis > Hi Travis: I'd like to help, but am not really prepared to comment at this time. It does look good at first glance. I need to review it in greater detail along with the current buffer code. Maybe this weekend. Do you have any kind of target dates that we need to keep in mind for this project? # Steve From steve at shrogers.com Thu Mar 1 09:09:01 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Thu, 01 Mar 2007 07:09:01 -0700 Subject: [Numpy-discussion] NumPy in Teaching In-Reply-To: <20070301104511.GA23386@astro.ox.ac.uk> References: <45E4FFA6.9010408@shrogers.com> <20070301104511.GA23386@astro.ox.ac.uk> Message-ID: <45E6DE7D.9020307@shrogers.com> Thanks Mike: Michael Williams wrote: > On Tue, Feb 27, 2007 at 09:05:58PM -0700, Steven H. Rogers wrote: > >> I'm doing an informal survey on the use of Array Programming Languages >> for teaching. If you're using NumPy in this manner I'd like to hear >> from you. What subject was/is taught, academic level, results, lessons >> learned, etc. >> > > If Numeric counts, I used that back in 2002 as part of an introductory > programming course I wrote for the Department of Physics at Oxford. We > really only used to to provide an element-wise array method. > Yes, Numeric and Numarray certainly count. The comments I've received about Matlab and IDL are also welcome. > Brief introduction: http://pentangle.net/python/pyzine.php > > The course (aka "Handbook") and report on the course's successes and > failures: http://pentangle.net/python/ > > -- Mike > ________ # Steve From aisaac at american.edu Thu Mar 1 10:14:42 2007 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 1 Mar 2007 10:14:42 -0500 Subject: [Numpy-discussion] [SciPy-user] NumPy in Teaching In-Reply-To: <45E4FFA6.9010408@shrogers.com> References: <45E4FFA6.9010408@shrogers.com> Message-ID: On Tue, 27 Feb 2007, "Steven H. Rogers" apparently wrote: > I'm doing an informal survey on the use of Array > Programming Languages for teaching. If you're using NumPy > in this manner I'd like to hear from you. What subject > was/is taught, academic level, results, lessons learned, > etc. I'm using NumPy in a PhD level microeconomics course. The students are using numpy primarily for the matrix class. In the past I have used GAUSS. Programming is a fairly small component of the course. Although most of the students have no programming background, they are finding Python and NumPy surprisingly easy to use. Cheers, Alan Isaac From mpmusu at cc.usu.edu Thu Mar 1 11:03:48 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 09:03:48 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists Message-ID: <45E6F964.10509@cc.usu.edu> I've been using Numpy arrays for some work recently. Just for fun, I compared some "representative" code using Numpy arrays and an object comprised of nested lists to represent my arrays. To my surprise, the array of nested lists outperformed Numpy in this particular application (in my actual code by 10%, results below are more dramatic). Can anyone shed some insight here? The functions that I use in reality are much more complicated than those listed below, but they are nonetheless representative of the type of thing that I'm doing. ##imports import numpy as NP from numpy.random import randint #numpy array code array1 = NP.zeros((50,50), int) def random1(): c = array1(randint(10), randint(10)) t=timeit.Timer("random1()", "from __main__ import random1") >>> t.timeit(10000) 0.1085283185432786 >>> t.timeit(10000) 0.10784806448862128 >>> t.timeit(10000) 0.1095533091495895 #python 2d array based on nested lists array2 = [] for aa in xrange(50): array2.append([]) for bb in xrange(50): array2[aa].append([]) array2[aa][bb] = 0 def random2(): c = array2[randint(50)][randint(50)] >>> t=timeit.Timer("random2()", "from __main__ import random2") >>> t.timeit(10000) 0.076737965300061717 >>> t.timeit(10000) 0.072883564810638291 >>> t.timeit(10000) 0.07668181291194287 Thanks, -Mark From perry at stsci.edu Thu Mar 1 11:03:34 2007 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 1 Mar 2007 11:03:34 -0500 Subject: [Numpy-discussion] NumPy in Teaching In-Reply-To: <200703010032.l210WfhI005995@glup.physics.ucf.edu> References: <200703010032.l210WfhI005995@glup.physics.ucf.edu> Message-ID: On Feb 28, 2007, at 7:32 PM, Joe Harrington wrote: > Hi Steve, > > I have taught Astronomical Data Analysis twice at Cornell using IDL, > and I will be teaching it next Fall at UCF using NumPy. Though I've > been active here in the recent past, I'm actually not a regular NumPy > user myself yet (I used Numeric experimentally for about 6 months in > 1997), so I'm a bit nervous. There isn't the kind of documentation > and how-to support for Numpy that there is for IDL, though our web > site is a start in that direction. One thought I've had in making the > transition easier is to put up a syntax and function concordance, > similar to that available for MATLAB. I thought this existed. Maybe > Perry can point me to it. Just adding a column to the MATLAB one > would be fine. I made one for IDL, but I don't recall one for matlab. If anyone has done one, I sure would like to incorporate it into the tutorial I'm revising if possible. Perry y From ivilata at carabos.com Thu Mar 1 11:06:43 2007 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Thu, 1 Mar 2007 17:06:43 +0100 Subject: [Numpy-discussion] [ANN] PyTables 2.0 alpha2 Message-ID: <20070301160643.GK7777@tardis.terramar.selidor.net> Hi all, I'm posting this message to announce the availability of the *second alpha release of PyTables 2.0*, the new and shiny major version of PyTables. This release settles the file format used in this major version, removing the need to use pickled objects in order to store system attributes, so we expect that no more changes will happen to the on-disk format for future 2.0 releases. The storage and handling of group filters has also been streamlined. The new release also allows running the complete test suite from within Python, enables new tests and fixes some problems with test data installation, among other fixes. We expect to have the documentation revised and the API definitely settled very soon in order to release the first beta version. The official announcement follows. Enjoy data! :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" ---- =========================== Announcing PyTables 2.0a2 =========================== This is the second *alpha* version of PyTables 2.0. This release, although being fairly stable regarding its operativity, is tagged as alpha because the API can still change a bit (but hopefully not a great deal), so it is meant basically for developers and people who want to get a taste of the new exciting features in this major version. You can download a source package of the version 2.0a2 with generated PDF and HTML docs from http://www.pytables.org/download/preliminary/ You can also get the latest sources from the Subversion repository at http://pytables.org/svn/pytables/trunk/ If you are afraid of Subversion (you shouldn't), you can always download the latest, daily updated, packed sources from http://www.pytables.org/download/snapshot/ Please have in mind that some sections in the manual can be obsolete (specially the "Optimization tips" chapter). The reference chapter should be fairly up-to-date though. You may also want to have an in-deep read of the ``RELEASE-NOTES.txt`` file where you will find an entire section devoted to how to migrate your existing PyTables 1.x apps to the 2.0 version. You can find an HTML version of this document at http://www.pytables.org/moin/ReleaseNotes/Release_2.0a2 Changes more in depth ===================== Improvements: - NumPy is finally at the core! That means that PyTables no longer needs numarray in order to operate, although it continues to be supported (as well as Numeric). This also means that you should be able to run PyTables in scenarios combining Python 2.5 and 64-bit platforms (these are a source of problems with numarray/Numeric because they don't support this combination yet). - Most of the operations in PyTables have experimented noticeable speed-ups (sometimes up to 2x, like in regular Python table selections). This is a consequence of both using NumPy internally and a considerable effort in terms of refactorization and optimization of the new code. - Numexpr has been integrated in all in-kernel selections. So, now it is possible to perform complex selections like:: result = [ row['var3'] for row in table.where('(var2 < 20) | (var1 == "sas")') ] or:: complex_cond = '((%s <= col5) & (col2 <= %s)) ' \ '| (sqrt(col1 + 3.1*col2 + col3*col4) > 3)' result = [ row['var3'] for row in table.where(complex_cond % (inf, sup)) ] and run them at full C-speed (or perhaps more, due to the cache-tuned computing kernel of Numexpr). - Now, it is possible to get fields of the ``Row`` iterator by specifiying their position, or even ranges of positions (extended slicing is supported). For example, you can do:: result = [ row[4] for row in table # fetch field #4 if row[1] < 20 ] result = [ row[:] for row in table # fetch all fields if row['var2'] < 20 ] result = [ row[1::2] for row in # fetch odd fields table.iterrows(2, 3000, 3) ] in addition to the classical:: result = [row['var3'] for row in table.where('var2 < 20')] - ``Row`` has received a new method called ``fetch_all_fields()`` in order to easily retrieve all the fields of a row in situations like:: [row.fetch_all_fields() for row in table.where('column1 < 0.3')] The difference between ``row[:]`` and ``row.fetch_all_fields()`` is that the former will return all the fields as a tuple, while the latter will return the fields in a NumPy void type and should be faster. Choose whatever fits better to your needs. - Now, all data that is read from disk is converted, if necessary, to the native byteorder of the hosting machine (before, this only happened with ``Table`` objects). This should help to accelerate apps that have to do computations with data generated in platforms with a byteorder different than the user machine. - All the leaf constructors have received a new parameter called ``byteorder`` that lets the user specify the byteorder of their data *on disk*. This effectively allows to create datasets in other byteorders than the native platform. - Native HDF5 datasets with ``H5T_ARRAY`` datatypes are fully supported for reading now. - The test suites for the different packages are installed now, so you don't need a copy of the PyTables sources to run the tests. Besides, you can run the test suite from the Python console by using:: >>> tables.tests() Bug fixes: - As mentioned above, the fact that NumPy is at the core makes that certain bizarre interactions between numarray and NumPy scalars don't affect the behaviour of table selections anymore. Fixes http://www.pytables.org/trac/ticket/29. - Did I mention that PyTables 2.0 can be safely used in 64-bit platforms in combination with Python 2.5? ;) Deprecated features: - Not many, really. Please see ``RELEASE-NOTES.txt`` file. Backward-incompatible changes: - Many. Please see ``RELEASE-NOTES.txt`` file. Important note for Windows users ================================ In order to keep PyTables happy, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win-net.ZIP Please remember that, from PyTables 2.0 on, Python 2.3 (and lesser) is not supported anymore. What is PyTables? ================= **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and NumPy package (but numarray and Numeric are also supported) for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables Pro sports indexing capabilities as well, allowing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC (and PowerPC64) and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About NumPy: http://numpy.scipy.org/ To know more about the company behind the development of PyTables, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 309 bytes Desc: Digital signature URL: From perry at stsci.edu Thu Mar 1 11:09:24 2007 From: perry at stsci.edu (Perry Greenfield) Date: Thu, 1 Mar 2007 11:09:24 -0500 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E6F964.10509@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> Message-ID: <6471D6D8-C071-400A-A36E-112936C598C9@stsci.edu> On Mar 1, 2007, at 11:03 AM, Mark P. Miller wrote: > I've been using Numpy arrays for some work recently. Just for fun, I > compared some "representative" code using Numpy arrays and an object > comprised of nested lists to represent my arrays. To my surprise, the > array of nested lists outperformed Numpy in this particular > application > (in my actual code by 10%, results below are more dramatic). > > Can anyone shed some insight here? The functions that I use in > reality > are much more complicated than those listed below, but they are > nonetheless representative of the type of thing that I'm doing. I'm guessing it has to do with the number of elements you are using. You have to understand that there is a fair amount of overhead in setting up an array operation (e.g., for ufuncs). Typically (I'm going from old results here so I may be off significantly) it isn't until you have arrays of around 1000 elements that as much times is spent doing the operation as it is in setting up for it. So for very large arrays (> 10,000 elements) the overhead is insignificant. For small arrays (e.g., 50 elements, it's all overhead. In that size range, lists will usually be much faster. Perry Greenfield From mpmusu at cc.usu.edu Thu Mar 1 11:41:05 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 09:41:05 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <6471D6D8-C071-400A-A36E-112936C598C9@stsci.edu> References: <45E6F964.10509@cc.usu.edu> <6471D6D8-C071-400A-A36E-112936C598C9@stsci.edu> Message-ID: <45E70221.4070903@cc.usu.edu> Interesting... I also tried the following and got similar results (using a 1,000 x 1,000 arrays). The time required to initialize the nested list array was much higher (but nonetheless small in the context of the overall time that my programs will run). But array element access is always faster when using the nested list approach. Any other thoughts? (more code below) array1 = NP.zeros((10000,10000), int) array2 = [] for aa in xrange(10000): array2.append([]) for bb in xrange(10000): array2[aa].append([]) array2[aa][bb] = 0 >>> t=timeit.Timer("random1()", "from __main__ import random1") >>> t.timeit(10000) 0.51156278242069675 >>> t.timeit(10000) 0.11799264990395386 >>> t.timeit(10000) 0.11274142383990693 >>> t.timeit(10000) 0.11590411630504605 >>> t=timeit.Timer("random2()", "from __main__ import random2") >>> t.timeit(10000) 0.24217882440370886 >>> t.timeit(10000) 0.077239146316060214 >>> t.timeit(10000) 0.07531906988197079 >>> t.timeit(10000) 0.075705711200498627 -Mark Perry Greenfield wrote: > On Mar 1, 2007, at 11:03 AM, Mark P. Miller wrote: > >> I've been using Numpy arrays for some work recently. Just for fun, I >> compared some "representative" code using Numpy arrays and an object >> comprised of nested lists to represent my arrays. To my surprise, the >> array of nested lists outperformed Numpy in this particular >> application >> (in my actual code by 10%, results below are more dramatic). >> >> Can anyone shed some insight here? The functions that I use in >> reality >> are much more complicated than those listed below, but they are >> nonetheless representative of the type of thing that I'm doing. > > I'm guessing it has to do with the number of elements you are using. > You have to understand that there is a fair amount of overhead in > setting up an array operation (e.g., for ufuncs). Typically (I'm > going from old results here so I may be off significantly) it isn't > until you have arrays of around 1000 elements that as much times is > spent doing the operation as it is in setting up for it. So for very > large arrays (> 10,000 elements) the overhead is insignificant. For > small arrays (e.g., 50 elements, it's all overhead. In that size > range, lists will usually be much faster. > > Perry Greenfield > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From strawman at astraw.com Thu Mar 1 11:57:46 2007 From: strawman at astraw.com (Andrew Straw) Date: Thu, 01 Mar 2007 08:57:46 -0800 Subject: [Numpy-discussion] [SciPy-user] NumPy in Teaching In-Reply-To: References: <200703010032.l210WfhI005995@glup.physics.ucf.edu> Message-ID: <45E7060A.9040105@astraw.com> Perry Greenfield wrote: > On Feb 28, 2007, at 7:32 PM, Joe Harrington wrote: > > >> Hi Steve, >> >> I have taught Astronomical Data Analysis twice at Cornell using IDL, >> and I will be teaching it next Fall at UCF using NumPy. Though I've >> been active here in the recent past, I'm actually not a regular NumPy >> user myself yet (I used Numeric experimentally for about 6 months in >> 1997), so I'm a bit nervous. There isn't the kind of documentation >> and how-to support for Numpy that there is for IDL, though our web >> site is a start in that direction. One thought I've had in making the >> transition easier is to put up a syntax and function concordance, >> similar to that available for MATLAB. I thought this existed. Maybe >> Perry can point me to it. Just adding a column to the MATLAB one >> would be fine. >> > > I made one for IDL, but I don't recall one for matlab. If anyone has > done one, I sure would like to incorporate it into the tutorial I'm > revising if possible. > > I believe the reference is to this: http://scipy.org/NumPy_for_Matlab_Users From oliphant at ee.byu.edu Thu Mar 1 12:08:40 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 01 Mar 2007 10:08:40 -0700 Subject: [Numpy-discussion] Buffer PEP in NumPy SVN In-Reply-To: <45E6DAF7.3090007@shrogers.com> References: <45E5EA4D.1010303@ee.byu.edu> <45E6DAF7.3090007@shrogers.com> Message-ID: <45E70898.5080909@ee.byu.edu> Steven H. Rogers wrote: >Travis Oliphant wrote: > > >>I just wanted to point people to the online version of the PEP. I'm >>still looking for comments and suggestions. The current version is here: >> >>http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt >> >>-Travis >> >> >> >Hi Travis: > >I'd like to help, but am not really prepared to comment at this time. >It does look good at first glance. I need to review it in greater >detail along with the current buffer code. Maybe this weekend. Do you >have any kind of target dates that we need to keep in mind for this project? > > > Accepted PEPs need to be in by April. So, I'm thinking March 15 for this PEP. Thanks for your willingness to help. -teo From oliphant at ee.byu.edu Thu Mar 1 12:17:33 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 01 Mar 2007 10:17:33 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E6F964.10509@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> Message-ID: <45E70AAD.8050609@ee.byu.edu> Mark P. Miller wrote: >I've been using Numpy arrays for some work recently. Just for fun, I >compared some "representative" code using Numpy arrays and an object >comprised of nested lists to represent my arrays. To my surprise, the >array of nested lists outperformed Numpy in this particular application >(in my actual code by 10%, results below are more dramatic). > >Can anyone shed some insight here? The functions that I use in reality >are much more complicated than those listed below, but they are >nonetheless representative of the type of thing that I'm doing. > > NumPy arrays are also not optimized generic containers like lists are. In order to support the fancy indexing operations, simple indexing is a bit slower for NumPy arrays. You don't see the difference until you do nested loops like this. NumPy arrays are not designed to be used inside of nested loops. There are some tricks, however, you can apply in order to speed up usage of NumPy arrays inside of nested loops. 1) Use the a.item and a.itemset methods to get and set single elements in the array. 2) Use the optional arguments to ufuncs to by-pass two of the slowest portions of the ufunc set-up code that is usually not needed inside of a loop. Also, selecting by an array of integers is not a particularly fast thing to do (it is in C, but it still has to loop over all the integers and pull out the results. Using getitem twice ([]) as you are doing is going to be slower then using the cross-product and using getitem one time ([]) -Travis From oliphant at ee.byu.edu Thu Mar 1 12:25:45 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 01 Mar 2007 10:25:45 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E6F964.10509@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> Message-ID: <45E70C99.4070201@ee.byu.edu> Mark P. Miller wrote: >I've been using Numpy arrays for some work recently. Just for fun, I >compared some "representative" code using Numpy arrays and an object >comprised of nested lists to represent my arrays. To my surprise, the >array of nested lists outperformed Numpy in this particular application >(in my actual code by 10%, results below are more dramatic). > >Can anyone shed some insight here? The functions that I use in reality >are much more complicated than those listed below, but they are >nonetheless representative of the type of thing that I'm doing. > > >##imports >import numpy as NP >from numpy.random import randint > >#numpy array code >array1 = NP.zeros((50,50), int) > >def random1(): > c = array1(randint(10), randint(10)) > > Is this a bug? You can't "call" an array. Did you mean, array1[randint(10), randint(10)]? Getting single indices like this is a bit slower for NumPy then for lists because of all the possibilities that must be distinguished for array indexing. List indexing is a very simple thing. Thus, lists can be seen as collections that are optimized for simple indexing. If all you are doing is simple indexing, then lists are going to be faster. You can try using array1.item(randint(10), randint(10)), like this: getitem = array1.item def random1(): getitem(randint(50), randint(50)) Also, you might try using array2 = array1.tolist() instead of creating one like you do. I get comparable speeds in this case. From charlesr.harris at gmail.com Thu Mar 1 12:26:34 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 10:26:34 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E6F964.10509@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> Message-ID: On 3/1/07, Mark P. Miller wrote: > > I've been using Numpy arrays for some work recently. Just for fun, I > compared some "representative" code using Numpy arrays and an object > comprised of nested lists to represent my arrays. To my surprise, the > array of nested lists outperformed Numpy in this particular application > (in my actual code by 10%, results below are more dramatic). > > Can anyone shed some insight here? The functions that I use in reality > are much more complicated than those listed below, but they are > nonetheless representative of the type of thing that I'm doing. > > > ##imports > import numpy as NP > from numpy.random import randint > > #numpy array code > array1 = NP.zeros((50,50), int) > > def random1(): > c = array1(randint(10), randint(10)) > > t=timeit.Timer("random1()", "from __main__ import random1") > >>> t.timeit(10000) > 0.1085283185432786 > >>> t.timeit(10000) > 0.10784806448862128 > >>> t.timeit(10000) > 0.1095533091495895 > > > #python 2d array based on nested lists > array2 = [] > for aa in xrange(50): > array2.append([]) > for bb in xrange(50): > array2[aa].append([]) > array2[aa][bb] = 0 > > def random2(): > c = array2[randint(50)][randint(50)] > > >>> t=timeit.Timer("random2()", "from __main__ import random2") > >>> t.timeit(10000) > 0.076737965300061717 > >>> t.timeit(10000) > 0.072883564810638291 > >>> t.timeit(10000) > 0.07668181291194287 I'm going to guess that it is the indexing. Numpy tends to be slow when using explicit indexes and they should be avoided when possible. Vectorization also helps in this case. In [19]: def random1() : ....: c = array1[randint(10), randint(10)] ....: In [20]: def random2() : ....: i = randint(10, size=10000) ....: j = randint(10, size=10000) ....: c = array1[i,j] ....: In [21]: t=timeit.Timer("random1()", "from __main__ import random1") In [22]: t.timeit(10000) Out[22]: 0.032405853271484375 In [28]: t=timeit.Timer("random2()", "from __main__ import random2") In [29]: t.timeit(1) Out[29]: 0.0022358894348144531 Speeds things up about 15x. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpmusu at cc.usu.edu Thu Mar 1 12:38:47 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 10:38:47 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E70C99.4070201@ee.byu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> Message-ID: <45E70FA7.5040305@cc.usu.edu> >> ##imports >> import numpy as NP >>from numpy.random import randint >> #numpy array code >> array1 = NP.zeros((50,50), int) >> >> def random1(): >> c = array1(randint(10), randint(10)) >> >> > Is this a bug? You can't "call" an array. Did you mean, > array1[randint(10), randint(10)]? Good catch...I accidentally cut and pasted a line containing a typo. I actually used c = array1[randint(50), randint(50)] when doing my comparisons. > > Getting single indices like this is a bit slower for NumPy then for > lists because of all the possibilities that must be distinguished for > array indexing. List indexing is a very simple thing. Thus, lists can > be seen as collections that are optimized for simple indexing. If all > you are doing is simple indexing, then lists are going to be faster. > > You can try using array1.item(randint(10), randint(10)), like this: > > getitem = array1.item > def random1(): > getitem(randint(50), randint(50)) > > Also, you might try using > > array2 = array1.tolist() > > instead of creating one like you do. > > I get comparable speeds in this case. Excellent...I'll check on this. It appears that the simplest approach is not the best in this case. Thanks for pointing this out. -Mark > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From mpmusu at cc.usu.edu Thu Mar 1 13:27:44 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 11:27:44 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E70FA7.5040305@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> Message-ID: <45E71B20.503@cc.usu.edu> Travis: Can you clarify this for me. The book doesn't have much detail here and seems to differ from your notation (which gives me errors). >> Getting single indices like this is a bit slower for NumPy then for >> lists because of all the possibilities that must be distinguished for >> array indexing. List indexing is a very simple thing. Thus, lists can >> be seen as collections that are optimized for simple indexing. If all >> you are doing is simple indexing, then lists are going to be faster. >> >> You can try using array1.item(randint(10), randint(10)), like this: >> >> getitem = array1.item >> def random1(): >> getitem(randint(50), randint(50)) >> > > >>> array1=numpy.zeros((10,10),int) >>> array1.item(0,0) Traceback (most recent call last): File "", line 1, in ? array1.item(0,0) TypeError: function takes exactly 0 arguments (2 given) >>> getitem = array1.item >>> getitem(0,0) Traceback (most recent call last): File "", line 1, in ? getitem(0,0) TypeError: function takes exactly 0 arguments (2 given) I also assume that .itemset should do the converse? However, the syntax escapes me. python 2.4.3 numpy 0.9.9.2706 WinXP Thanks, -Mark > > >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Thu Mar 1 13:25:56 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Mar 2007 12:25:56 -0600 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E71B20.503@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> Message-ID: <45E71AB4.9000103@gmail.com> Mark P. Miller wrote: > Travis: Can you clarify this for me. The book doesn't have much detail > here and seems to differ from your notation (which gives me errors). > numpy 0.9.9.2706 This is the problem. You will need to upgrade to 1.0.1. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From rudolf.sykora at gmail.com Thu Mar 1 13:33:48 2007 From: rudolf.sykora at gmail.com (Rudolf Sykora) Date: Thu, 1 Mar 2007 19:33:48 +0100 Subject: [Numpy-discussion] building an array using smaller arrays Message-ID: Hello, since noone has reacted to my last e-mail yet (for several days), I feel the need to ask again (since I still do not know a good answer). Please help me. >> Hello everybody, >> I wonder how I could most easily accomplish the following: >> >>Say I have sth like: >> a = array( [1, 2] ) >> and I want to use this array to build another array in the following sence: >> b = array( [[1, 2, 3, a], [5, a, 6, 7], [0, 2-a, 3, 4]]) # this doesn't work >> >> I would like to obtain >> b = array( [[1, 2, 3, 1, 2], [5 ,1 ,2 ,6 ,7], [0, 1, 0, 3, 4]] ) >> I know a rather complicated way but believe there must be an easy one. >> Thank you very much. >> Ruda I would need some sort of flattening operator... The solution I know is very ugly: b = array(( concatenate(([1, 2, 3], a)), concatenate(([5], a, [6, 7])), concatenate(([0], 2-a, [3, 4])) )) Isn't there something simpler? Thank you for your advice! Ruda -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpmusu at cc.usu.edu Thu Mar 1 14:03:56 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 12:03:56 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E71AB4.9000103@gmail.com> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> Message-ID: <45E7239C.60509@cc.usu.edu> Sorry to pester, but is this the intended behavior of itemset? >>> array1=numpy.zeros((10,10),int) >>> array1 array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) >>> array1.itemset(5,5,9) >>> array1 array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 9], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) -Mark (now using 1.0.1) Robert Kern wrote: > Mark P. Miller wrote: >> Travis: Can you clarify this for me. The book doesn't have much detail >> here and seems to differ from your notation (which gives me errors). > >> numpy 0.9.9.2706 > > This is the problem. You will need to upgrade to 1.0.1. > From faltet at carabos.com Thu Mar 1 14:22:44 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 01 Mar 2007 20:22:44 +0100 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E7239C.60509@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> Message-ID: <1172776964.2540.5.camel@localhost.localdomain> El dj 01 de 03 del 2007 a les 12:03 -0700, en/na Mark P. Miller va escriure: > Sorry to pester, but is this the intended behavior of itemset? > > >>> array1=numpy.zeros((10,10),int) > >>> array1 > array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) > >>> array1.itemset(5,5,9) try with >>> array1.itemset((5,5),9) HTH, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From Chris.Barker at noaa.gov Thu Mar 1 14:35:14 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 01 Mar 2007 11:35:14 -0800 Subject: [Numpy-discussion] mac binary distributions In-Reply-To: <97e4e62e0702281317r6086008dja9a5b98d6a1a5826@mail.gmail.com> References: <97e4e62e0702281317r6086008dja9a5b98d6a1a5826@mail.gmail.com> Message-ID: <45E72AF2.7060801@noaa.gov> There are binaries of 1.0.1 here: http://pythonmac.org/packages/py24-fat/index.html and here: http://pythonmac.org/packages/py25-fat/index.html -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mpmusu at cc.usu.edu Thu Mar 1 14:47:26 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 12:47:26 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <1172776964.2540.5.camel@localhost.localdomain> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> Message-ID: <45E72DCE.9020909@cc.usu.edu> > try with > >>>> array1.itemset((5,5),9) Yep...tried that. But I don't get it! >>> import numpy >>> array1 = numpy.zeros((10,10),int) >>> array1 array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) >>> array1.itemset((5,5),9) >>> array1 array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 9], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) >>> array1.item(5,5) 0 >>> array1.item(5,9) 9 Sorry for being obtuse. -Mark From faltet at carabos.com Thu Mar 1 14:53:35 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 01 Mar 2007 20:53:35 +0100 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E72DCE.9020909@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> Message-ID: <1172778815.2540.9.camel@localhost.localdomain> El dj 01 de 03 del 2007 a les 12:47 -0700, en/na Mark P. Miller va escriure: > > try with > > > >>>> array1.itemset((5,5),9) > > > Yep...tried that. But I don't get it! > > >>> import numpy > >>> array1 = numpy.zeros((10,10),int) > >>> array1 > array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) > >>> array1.itemset((5,5),9) > >>> array1 > array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 9], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) > >>> array1.item(5,5) > 0 > >>> array1.item(5,9) > 9 > Ops, this seems a bug with your numpy version: In [46]:array1 = numpy.zeros((10,10),int) In [47]:array1.itemset((5,5),9) In [48]:array1 Out[48]: array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 9, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) In [49]:array1.item(5,5) Out[49]:9 In [50]:numpy.__version__ Out[50]:'1.0.2.dev3546' So, this seems solved in SVN. -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From mpmusu at cc.usu.edu Thu Mar 1 15:09:07 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 13:09:07 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <1172778815.2540.9.camel@localhost.localdomain> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> Message-ID: <45E732E3.2000707@cc.usu.edu> >> > > Ops, this seems a bug with your numpy version: > > In [46]:array1 = numpy.zeros((10,10),int) > In [47]:array1.itemset((5,5),9) > In [48]:array1 > Out[48]: > array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 9, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) > In [49]:array1.item(5,5) > Out[49]:9 > In [50]:numpy.__version__ > Out[50]:'1.0.2.dev3546' > > So, this seems solved in SVN. > OK...thanks for the reassurance. Is there any indication of when the next release will be made? I'm not savvy enough with these things to work with the svn releases. -Mark From faltet at carabos.com Thu Mar 1 15:10:03 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 01 Mar 2007 21:10:03 +0100 Subject: [Numpy-discussion] In-place fancy selection Message-ID: <1172779803.2540.17.camel@localhost.localdomain> Hi, I don't think there is a solution for this, but perhaps anybody may offer some idea. Given: In [79]:a=numpy.arange(9,-1,-1) In [80]:b=numpy.arange(10) In [81]:numpy.random.shuffle(b) In [82]:b Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) In [83]:a=a[b] In [84]:a Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) is there a way to make the step 83 without having to keep 3 arrays in-memory at the same time? This is, some way of doing fancy indexing, but changing the elements *inplace*. The idea is to keep memory requeriments as low as possible when a and b are large arrays. Thanks! -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From charlesr.harris at gmail.com Thu Mar 1 15:26:09 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 13:26:09 -0700 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: <1172779803.2540.17.camel@localhost.localdomain> References: <1172779803.2540.17.camel@localhost.localdomain> Message-ID: On 3/1/07, Francesc Altet wrote: > > Hi, > > I don't think there is a solution for this, but perhaps anybody may > offer some idea. Given: > > In [79]:a=numpy.arange(9,-1,-1) > In [80]:b=numpy.arange(10) > In [81]:numpy.random.shuffle(b) > In [82]:b > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > In [83]:a=a[b] > In [84]:a > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > is there a way to make the step 83 without having to keep 3 arrays > in-memory at the same time? This is, some way of doing fancy indexing, > but changing the elements *inplace*. The idea is to keep memory > requeriments as low as possible when a and b are large arrays. > > Thanks! I think that would be tough because of overlap between the two sides. The permutation could be factored into cycles which would mostly avoid that, but that is more theoretical than practical here. What is it you are trying to do? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Mar 1 15:31:13 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 13:31:13 -0700 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: References: <1172779803.2540.17.camel@localhost.localdomain> Message-ID: On 3/1/07, Charles R Harris wrote: > > > > On 3/1/07, Francesc Altet wrote: > > > > Hi, > > > > I don't think there is a solution for this, but perhaps anybody may > > offer some idea. Given: > > > > In [79]:a=numpy.arange(9,-1,-1) > > In [80]:b=numpy.arange(10) > > In [81]:numpy.random.shuffle(b) > > In [82]:b > > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > > In [83]:a=a[b] > > In [84]:a > > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > > > is there a way to make the step 83 without having to keep 3 arrays > > in-memory at the same time? This is, some way of doing fancy indexing, > > but changing the elements *inplace*. The idea is to keep memory > > requeriments as low as possible when a and b are large arrays. > > > > Thanks! > > > I think that would be tough because of overlap between the two sides. The > permutation could be factored into cycles which would mostly avoid that, but > that is more theoretical than practical here. What is it you are trying to > do? > > Chuck > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Mar 1 15:40:58 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 13:40:58 -0700 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: <1172779803.2540.17.camel@localhost.localdomain> References: <1172779803.2540.17.camel@localhost.localdomain> Message-ID: On 3/1/07, Francesc Altet wrote: > > Hi, > > I don't think there is a solution for this, but perhaps anybody may > offer some idea. Given: > > In [79]:a=numpy.arange(9,-1,-1) > In [80]:b=numpy.arange(10) > In [81]:numpy.random.shuffle(b) > In [82]:b > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > In [83]:a=a[b] > In [84]:a > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > is there a way to make the step 83 without having to keep 3 arrays > in-memory at the same time? This is, some way of doing fancy indexing, > but changing the elements *inplace*. The idea is to keep memory > requeriments as low as possible when a and b are large arrays. > > Thanks! You can also put the arrays together and implement it as an inplace sort, which will save space at the price of n*log(n) operations. The idea is to sort on the shuffled array while carrying the corresponding elements of the other array along in the exchanges, which I think you can now do using fields and the order keyword in the sort. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Thu Mar 1 15:44:19 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 01 Mar 2007 21:44:19 +0100 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: References: <1172779803.2540.17.camel@localhost.localdomain> Message-ID: <1172781859.2540.30.camel@localhost.localdomain> El dj 01 de 03 del 2007 a les 13:26 -0700, en/na Charles R Harris va escriure: > > > On 3/1/07, Francesc Altet wrote: > Hi, > > I don't think there is a solution for this, but perhaps > anybody may > offer some idea. Given: > > In [79]:a=numpy.arange(9,-1,-1) > In [80]:b=numpy.arange(10) > In [81]:numpy.random.shuffle(b) > In [82]:b > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > In [83]:a=a[b] > In [84]:a > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > is there a way to make the step 83 without having to keep 3 > arrays > in-memory at the same time? This is, some way of doing fancy > indexing, > but changing the elements *inplace*. The idea is to keep > memory > requeriments as low as possible when a and b are large arrays. > > Thanks! > > I think that would be tough because of overlap between the two sides. > The permutation could be factored into cycles which would mostly avoid > that, but that is more theoretical than practical here. What is it you > are trying to do? Yeah, the problem is the overlap. Well, what I'm trying to do is, given two arrays on-disk (say, block and block_idx), sort one of them, and then, re-order the other following with the same order than the first one. My best approach until now is: block = tmp_sorted[nslice] # read block from disk sblock_idx = block.argsort() block.sort() # do things with block... del block # get rid of block block_idx = tmp_indices[nslice] # read bock_idx from disk indices[nslice] = block_idx[sblock_idx] but the last line will take 3 times the memory that takes block_idx alone. My goal would be that the algorithm above would take only 2 times the memory of block_idx, but I don't think this is going to be possible. Thanks! -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From charlesr.harris at gmail.com Thu Mar 1 15:58:32 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 13:58:32 -0700 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: <1172781859.2540.30.camel@localhost.localdomain> References: <1172779803.2540.17.camel@localhost.localdomain> <1172781859.2540.30.camel@localhost.localdomain> Message-ID: On 3/1/07, Francesc Altet wrote: > > El dj 01 de 03 del 2007 a les 13:26 -0700, en/na Charles R Harris va > escriure: > > > > > > On 3/1/07, Francesc Altet wrote: > > Hi, > > > > I don't think there is a solution for this, but perhaps > > anybody may > > offer some idea. Given: > > > > In [79]:a=numpy.arange(9,-1,-1) > > In [80]:b=numpy.arange(10) > > In [81]:numpy.random.shuffle(b) > > In [82]:b > > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > > In [83]:a=a[b] > > In [84]:a > > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > > > is there a way to make the step 83 without having to keep 3 > > arrays > > in-memory at the same time? This is, some way of doing fancy > > indexing, > > but changing the elements *inplace*. The idea is to keep > > memory > > requeriments as low as possible when a and b are large arrays. > > > > Thanks! > > > > I think that would be tough because of overlap between the two sides. > > The permutation could be factored into cycles which would mostly avoid > > that, but that is more theoretical than practical here. What is it you > > are trying to do? > > Yeah, the problem is the overlap. Well, what I'm trying to do is, given > two arrays on-disk (say, block and block_idx), sort one of them, and > then, re-order the other following with the same order than the first > one. My best approach until now is: > > block = tmp_sorted[nslice] # read block from disk > sblock_idx = block.argsort() > block.sort() > # do things with block... > del block # get rid of block > block_idx = tmp_indices[nslice] # read bock_idx from disk > indices[nslice] = block_idx[sblock_idx] > > but the last line will take 3 times the memory that takes block_idx > alone. My goal would be that the algorithm above would take only 2 times > the memory of block_idx, but I don't think this is going to be possible. Argsort sorta does what you want, it is an indirect sort but the other array is a list of integers and at present, it doesn't exchange the key array. If you want a special case sort it wouldn't be too difficult to modify it to do what you want. If you want to operate on small pieces disk to disk, an indirect version of merge sort might be the way to go. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Thu Mar 1 15:59:58 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 01 Mar 2007 21:59:58 +0100 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: References: <1172779803.2540.17.camel@localhost.localdomain> Message-ID: <1172782798.2540.38.camel@localhost.localdomain> El dj 01 de 03 del 2007 a les 13:40 -0700, en/na Charles R Harris va escriure: > > > On 3/1/07, Francesc Altet wrote: > Hi, > > I don't think there is a solution for this, but perhaps > anybody may > offer some idea. Given: > > In [79]:a=numpy.arange(9,-1,-1) > In [80]:b=numpy.arange(10) > In [81]:numpy.random.shuffle(b) > In [82]:b > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > In [83]:a=a[b] > In [84]:a > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > is there a way to make the step 83 without having to keep 3 > arrays > in-memory at the same time? This is, some way of doing fancy > indexing, > but changing the elements *inplace*. The idea is to keep > memory > requeriments as low as possible when a and b are large arrays. > > Thanks! > > You can also put the arrays together and implement it as an inplace > sort, which will save space at the price of n*log(n) operations. The > idea is to sort on the shuffled array while carrying the corresponding > elements of the other array along in the exchanges, which I think you > can now do using fields and the order keyword in the sort. Nice idea! I think your approach is going to work: In [18]:a=numpy.arange(9.,-1,-1) In [19]:b=numpy.arange(10) In [20]:numpy.random.shuffle(b) In [21]:c=numpy.rec.fromarrays([a,b], dtype='i4,i4') In [22]:c Out[22]: recarray([(9, 1), (8, 6), (7, 9), (6, 5), (5, 3), (4, 4), (3, 7), (2, 8), (1, 0), (0, 2)], dtype=[('f0', ' -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From peridot.faceted at gmail.com Thu Mar 1 16:11:40 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Mar 2007 16:11:40 -0500 Subject: [Numpy-discussion] In-place fancy selection In-Reply-To: <1172779803.2540.17.camel@localhost.localdomain> References: <1172779803.2540.17.camel@localhost.localdomain> Message-ID: On 01/03/07, Francesc Altet wrote: > Hi, > > I don't think there is a solution for this, but perhaps anybody may > offer some idea. Given: > > In [79]:a=numpy.arange(9,-1,-1) > In [80]:b=numpy.arange(10) > In [81]:numpy.random.shuffle(b) > In [82]:b > Out[82]:array([2, 6, 3, 5, 4, 9, 0, 8, 7, 1]) > In [83]:a=a[b] > In [84]:a > Out[84]:array([7, 3, 6, 4, 5, 0, 9, 1, 2, 8]) > > is there a way to make the step 83 without having to keep 3 arrays > in-memory at the same time? This is, some way of doing fancy indexing, > but changing the elements *inplace*. The idea is to keep memory > requeriments as low as possible when a and b are large arrays. > > Thanks! In general, this is impossible: a=arange(3) b=a[[0,1,0,1,0]] b is bigger than a, so you can't hope to do this in place. >From the example you gave, though, it's likely that what you want is to apply a permutation (this turns up frequently with argsorts). That's possible in principle, though even in C I think you need to either maintain an array of auxiliary data or destroy the indexing array. The problem is that every element you move, you need to clear space for it. You can chase this around one cycle in the permutation with no difficulty, but as you move from cycle to cycle you need to keep track of which elements of the array have already been shuffled and which haven't. You can do this with bitfields, or you can adjust the indexing array as you go. Since either of these effectively require O(n) extra storage, they don't have much appeal - except that you might want to permute just one axis of an array, in which case the extra storage. could be much smaller than the array itself. I've attached an implementation of this (in Python, but it might be useful anyway since the row copies are done by numpy). Anne -------------- next part -------------- A non-text attachment was scrubbed... Name: permute.py Type: text/x-python Size: 759 bytes Desc: not available URL: From cookedm at physics.mcmaster.ca Thu Mar 1 16:14:32 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 1 Mar 2007 16:14:32 -0500 Subject: [Numpy-discussion] building an array using smaller arrays In-Reply-To: References: Message-ID: On Mar 1, 2007, at 13:33 , Rudolf Sykora wrote: > Hello, > > since noone has reacted to my last e-mail yet (for several days), I > feel the need to ask again (since I still do not know a good answer). > Please help me. > > >> Hello everybody, > >> I wonder how I could most easily accomplish the following: > >> > >>Say I have sth like: > >> a = array( [1, 2] ) > >> and I want to use this array to build another array in the > following sence: > >> b = array( [[1, 2, 3, a], [5, a, 6, 7], [0, 2-a, 3, 4]]) # this > doesn't work > >> > >> I would like to obtain > >> b = array( [[1, 2, 3, 1, 2], [5 ,1 ,2 ,6 ,7], [0, 1, 0, 3, 4]] ) > > >> I know a rather complicated way but believe there must be an > easy one. > >> Thank you very much. > > >> Ruda > > I would need some sort of flattening operator... > The solution I know is very ugly: > > b = array(( concatenate(([1, 2, 3], a)), concatenate(([5], a, [6, > 7])), concatenate(([0], 2-a, [3, 4])) )) Define a helper function def row(*args): res = [] for a in args: a = asarray(a) if len(a.shape) == 0: res.append(a) elif len(a.shape) == 1: res += a.tolist() else: raise ValueError("arguments to row must be row-like") return array(res) then b = array([ row(1,2,3,a), row(5,a,6,7), row(0,2-a,3,4) ]) -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part URL: From Chris.Barker at noaa.gov Thu Mar 1 16:49:10 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 01 Mar 2007 13:49:10 -0800 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <1172778815.2540.9.camel@localhost.localdomain> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> Message-ID: <45E74A56.5080007@noaa.gov> Francesc Altet wrote: > Ops, this seems a bug with your numpy version: yup, it's a bug here too: >>> numpy.__version__ '1.0.1' this is the dmg for python2.5 on pythonmac.org/packages -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Jim.Vickroy at noaa.gov Thu Mar 1 17:05:45 2007 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Thu, 01 Mar 2007 15:05:45 -0700 Subject: [Numpy-discussion] how to combine (stack) two masked arrays? Message-ID: <45E74E39.5050305@noaa.gov> Hello all, I am stuck in my attempts to use masked arrays to combine the contents of two arrays (please see the masked-arrays.py attachment) such that missing values in one array are provided by the second array. There are no ambiguities in the combination operation because each of the two arrays only has values where the other array does not. Referring to masked-arrays.py, I want to apply some function/operator to the *clipped_mask* and *rescaled* arrays to obtain the following result: [[0 0 0 0 0] [0 1220 4095 1220 0] [1220 4095 4095 4095 1220] [0 1220 4095 1220 0] [0 0 0 0 0]] I am a new user of numpy so I would appreciate feedback on awkward areas in the code and/or the suitability of the approach. Thanks, -- jv P.S. Apologies if this is in the discussion archives; I was not able to find a similar discussion there nor in my copy of numpybook.pdf. -------------- next part -------------- A non-text attachment was scrubbed... Name: masked-arrays.py Type: application/x-python Size: 891 bytes Desc: not available URL: From oliphant at ee.byu.edu Thu Mar 1 17:10:04 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 01 Mar 2007 15:10:04 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E732E3.2000707@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E732E3.2000707@cc.usu.edu> Message-ID: <45E74F3C.4010501@ee.byu.edu> Mark P. Miller wrote: >>Ops, this seems a bug with your numpy version: >> >>In [46]:array1 = numpy.zeros((10,10),int) >>In [47]:array1.itemset((5,5),9) >>In [48]:array1 >>Out[48]: >>array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 9, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) >>In [49]:array1.item(5,5) >>Out[49]:9 >>In [50]:numpy.__version__ >>Out[50]:'1.0.2.dev3546' >> >>So, this seems solved in SVN. >> >> >> > >OK...thanks for the reassurance. > >Is there any indication of when the next release will be made? I'm not >savvy enough with these things to work with the svn releases. > > Any day now. I'm working on revisions to a paper right now and don't have time, but perhaps next week. -Travis From mpmusu at cc.usu.edu Thu Mar 1 17:21:17 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 15:21:17 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E74A56.5080007@noaa.gov> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> Message-ID: <45E751DD.8010002@cc.usu.edu> One more note (this perhaps may need a separate topic): I've been using the Enthought python edition that contains python 2.4.3 and numpy 0.9.9.2706. After Robert Kern pointed out that I should try numpy 1.0.1, I went ahead and installed it (downloaded a few hours ago: "numpy-1.0.1.win32-py2.4.exe"). Now however, I'm seeing perhaps a more serious problem: The test program that I'm working with went from taking ~80 seconds to run to taking over 10 minutes to run. I've rolled back to my old numpy version and confirmed that the old version was much faster. I also tried re-installing 1.0.1, and once again am seeing much slower runtimes. My code contains numpy arrays where items are accessed on an element-by-element basis. I also make use of some of numpy's random number generators. Just thought I'd point this out, -Mark Christopher Barker wrote: > Francesc Altet wrote: >> Ops, this seems a bug with your numpy version: > > yup, it's a bug here too: > > >>> numpy.__version__ > '1.0.1' > > this is the dmg for python2.5 on pythonmac.org/packages > > > From robert.kern at gmail.com Thu Mar 1 17:20:11 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Mar 2007 16:20:11 -0600 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E751DD.8010002@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> Message-ID: <45E7519B.2070002@gmail.com> Mark P. Miller wrote: > Now however, I'm seeing perhaps a more serious problem: The test > program that I'm working with went from taking ~80 seconds to run to > taking over 10 minutes to run. I've rolled back to my old numpy version > and confirmed that the old version was much faster. I also tried > re-installing 1.0.1, and once again am seeing much slower runtimes. Give us a small, self-contained benchmark, and we'll see what we can do. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Thu Mar 1 17:28:21 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 1 Mar 2007 17:28:21 -0500 Subject: [Numpy-discussion] how to combine (stack) two masked arrays? In-Reply-To: <45E74E39.5050305@noaa.gov> References: <45E74E39.5050305@noaa.gov> Message-ID: <200703011728.21425.pgmdevlist@gmail.com> Hello, The simplest is: rescaled[rescaled.mask]=clipped_mask[rescaled.mask] But for this particular kind of operation, I don't think that you need masked array at all: this could do the trick >>>rescaled = clipped.copy() >>>inside = numpy.logical_and(clipped>=123, clipped<=354) >>>outside = numpy.logical_not(inside) >>>newrescaled[inside] = 5*clipped[inside] >>>newrescaled[outside] = clipped[outside] A side note: when using numpy.core.ma, it's more efficient to deal w/ the _data and _mask parts separately than at once. You may want to try an alternative implementation of maskedarray, available in the sandbox of the scipy SVN. I'd be very happy to have your feedback Let me know if this helps P. From charlesr.harris at gmail.com Thu Mar 1 17:38:37 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 15:38:37 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E751DD.8010002@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> Message-ID: On 3/1/07, Mark P. Miller wrote: > > One more note (this perhaps may need a separate topic): > > I've been using the Enthought python edition that contains python 2.4.3 > and numpy 0.9.9.2706. After Robert Kern pointed out that I should try > numpy 1.0.1, I went ahead and installed it (downloaded a few hours ago: > "numpy-1.0.1.win32-py2.4.exe"). > > Now however, I'm seeing perhaps a more serious problem: The test > program that I'm working with went from taking ~80 seconds to run to > taking over 10 minutes to run. I've rolled back to my old numpy version > and confirmed that the old version was much faster. I also tried > re-installing 1.0.1, and once again am seeing much slower runtimes. > > My code contains numpy arrays where items are accessed on an > element-by-element basis. I also make use of some of numpy's random > number generators. > > Just thought I'd point this out, > > -Mark Seems odd. Some example code would help us see what is going. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Mar 1 17:39:06 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 01 Mar 2007 15:39:06 -0700 Subject: [Numpy-discussion] Scalar coercion Message-ID: <45E7560A.2020206@ee.byu.edu> A ticket was posted that emphasizes that the current behavior of NumPy with regards to scalar coercion is different than numarray's behavior. If we were pre 1.0, I would probably change the behavior to be in-line with numarray. But, now I think it needs some discussion because we are changing the behavior of a released version of NumPy and we need some more conservatism in how changes happen. If we can classify the current behavior as a bug then we can change it. Otherwise, I'm concerned. The behavior has to do with a mixed scalar/array computation. NumPy does not let scalars (equivalently 0-d arrays) determine the output type of the array operation, *unless* the scalar is of a fundamentally different kind (i.e. the array is an integer-type but the scalar is a floating-point type). In this case, the current behavior is to coerce the array to the smallest-type in that general category of scalars. The reason for this behavior is to make sure that array([1,2,3,4],int8)*10 returns an int8 (instead of an int32 because of how the 10 is interpreted by Python). The current behavior, however, also means that array([1,2,3,4],int8)*10.0 will return a float32 array. I think numarray would return a float64 array in this case (i.e. the type of the scalar would be honored when the coercion was between two different kinds of arrays). So the question is: Do we keep the current behavior or change the behavior to be more consistent with numarray. My current preference is to change the behavior so it is more consistent with numarray (even though it's actually not going to be trivial to do that). -Travis From robert.kern at gmail.com Thu Mar 1 17:48:28 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Mar 2007 16:48:28 -0600 Subject: [Numpy-discussion] Scalar coercion In-Reply-To: <45E7560A.2020206@ee.byu.edu> References: <45E7560A.2020206@ee.byu.edu> Message-ID: <45E7583C.4090007@gmail.com> Travis Oliphant wrote: > So the question is: Do we keep the current behavior or change the > behavior to be more consistent with numarray. My current preference is > to change the behavior so it is more consistent with numarray (even > though it's actually not going to be trivial to do that). The current behavior is very obscure and almost never the behavior that one desires. Indeed, the first time it came up, you seemed to think it was a bug earlier in the thread before (presumably) you went into the code and rediscovered the justification. I suspect that most people who have coded with this behavior in mind have coded to *avoid* the behavior and that their code will still work even after changing numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Jim.Vickroy at noaa.gov Thu Mar 1 17:53:34 2007 From: Jim.Vickroy at noaa.gov (Jim Vickroy) Date: Thu, 01 Mar 2007 15:53:34 -0700 Subject: [Numpy-discussion] how to combine (stack) two masked arrays? In-Reply-To: <200703011728.21425.pgmdevlist@gmail.com> References: <45E74E39.5050305@noaa.gov> <200703011728.21425.pgmdevlist@gmail.com> Message-ID: <45E7596E.7070701@noaa.gov> Thanks for the quick response! Please see my in-line comments below. Pierre GM wrote: > Hello, > The simplest is: > rescaled[rescaled.mask]=clipped_mask[rescaled.mask] > > But for this particular kind of operation, I don't think that you need masked > array at all: this could do the trick > >>>> rescaled = clipped.copy() >>>> inside = numpy.logical_and(clipped>=123, clipped<=354) >>>> outside = numpy.logical_not(inside) >>>> newrescaled[inside] = 5*clipped[inside] >>>> newrescaled[outside] = clipped[outside] >>>> > > Great; that is slick (replaced *newrescaled* with *rescaled* in the above code)! I approached numpy with a preconceived notion of how to perform this task (i.e., masked arrays) and did not look at the broader set of package capabilities > A side note: when using numpy.core.ma, it's more efficient to deal w/ the > _data and _mask parts separately than at once. > Thanks for the tip. > > You may want to try an alternative implementation of maskedarray, available in > the sandbox of the scipy SVN. I'd be very happy to have your feedback > > While browsing the archived discussions, I did see mention of this, but decided to avoid it because I was a newbie. > Let me know if this helps > P. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mpmusu at cc.usu.edu Thu Mar 1 18:16:18 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Thu, 01 Mar 2007 16:16:18 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E7519B.2070002@gmail.com> References: <45E6F964.10509@cc.usu.edu> <45E70C99.4070201@ee.byu.edu> <45E70FA7.5040305@cc.usu.edu> <45E71B20.503@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> <45E7519B.2070002@gmail.com> Message-ID: <45E75EC2.7090905@cc.usu.edu> OK...here goes. This code is going to look goofy, so please bear in mind that it is only an abstraction of what my real code does (which happens to provide interesting and meaning insights!). I've attached saved versions of my interactive python sessions that document the phenomenon. Again, this code mainly serves to illustrate what I previously saw. Please note that I DO NOT a) redefine arrays, or b) re-import functions in my real code. 1) Fresh install of Enthought python edition and reboot. Python 2.4.3 - Enthought Edition 1.0.0 (#69, Aug 2 2006, 12:09:59) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.1.3 ==== No Subprocess ==== >>> import numpy >>> numpy.__version__ '0.9.9.2706' >>> def numpytest(): array1=numpy.zeros((100,100),int) from numpy.random import normal for aa in xrange(25,50): for bb in xrange(25,50): x1=int(normal(0,1)) x2=int(normal(0,1)) array1[aa,bb]=array1[x1,x2] >>> import timeit >>> t=timeit.Timer("numpytest()", "from __main__ import numpytest") >>> t.timeit(100) 1.110151150495426 >>> t.timeit(100) 1.1139172716085568 >>> t.timeit(100) 1.1248401682336748 >>> 2) Install Numpy 1.0.1 from "numpy-1.0.1.win32-py2.4.exe" (downloaded today). Reboot. Python 2.4.3 - Enthought Edition 1.0.0 (#69, Aug 2 2006, 12:09:59) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.1.3 ==== No Subprocess ==== >>> import numpy >>> numpy.__version__ '1.0.1' >>> def numpytest(): array1=numpy.zeros((100,100),int) from numpy.random import normal for aa in xrange(25,50): for bb in xrange(25,50): x1=int(normal(0,1)) x2=int(normal(0,1)) array1[aa,bb]=array1[x1,x2] >>> import timeit >>> t=timeit.Timer("numpytest()", "from __main__ import numpytest") >>> t.timeit(100) 17.143569084897678 >>> t.timeit(100) 17.183305140737161 >>> t.timeit(100) 17.283449432818969 >>> Hope this helps or is informative somehow. -Mark Robert Kern wrote: > Mark P. Miller wrote: > >> Now however, I'm seeing perhaps a more serious problem: The test >> program that I'm working with went from taking ~80 seconds to run to >> taking over 10 minutes to run. I've rolled back to my old numpy version >> and confirmed that the old version was much faster. I also tried >> re-installing 1.0.1, and once again am seeing much slower runtimes. > > Give us a small, self-contained benchmark, and we'll see what we can do. > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy 0.9.9.2706.py URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy 1.0.1.py URL: From oliphant at ee.byu.edu Thu Mar 1 18:23:05 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 01 Mar 2007 16:23:05 -0700 Subject: [Numpy-discussion] Scalar coercion In-Reply-To: <45E7583C.4090007@gmail.com> References: <45E7560A.2020206@ee.byu.edu> <45E7583C.4090007@gmail.com> Message-ID: <45E76059.2040605@ee.byu.edu> Robert Kern wrote: >Travis Oliphant wrote: > > > >>So the question is: Do we keep the current behavior or change the >>behavior to be more consistent with numarray. My current preference is >>to change the behavior so it is more consistent with numarray (even >>though it's actually not going to be trivial to do that). >> >> > >The current behavior is very obscure and almost never the behavior that one >desires. Indeed, the first time it came up, you seemed to think it was a bug >earlier in the thread before (presumably) you went into the code and >rediscovered the justification. > > What I discovered is that there wasn't really a justification at all, except ease in coding using the current ufunc selection mechanism. It's just that the code that makes the mixed scalar / array coercion do the right thing in the case where both are the same type does not handle the mixed type correctly. There are really two places in the code where this is handled. One is in the ufunc code where the "right" 1-d loop is determined and the second is in the call PyArray_ConvertToCommonType. The mechanisms used for determining the types of the outputs are coded differently in each case. So, I'm going to try and change this for the 1.0.2 release. It might take me a little bit, so if anybody sees a way to do it, feel free to chime in. I'm probably missing something obvious. -Travis From charlesr.harris at gmail.com Thu Mar 1 19:39:49 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 17:39:49 -0700 Subject: [Numpy-discussion] Scalar coercion In-Reply-To: <45E7560A.2020206@ee.byu.edu> References: <45E7560A.2020206@ee.byu.edu> Message-ID: On 3/1/07, Travis Oliphant wrote: > > > A ticket was posted that emphasizes that the current behavior of NumPy > with regards to scalar coercion is different than numarray's behavior. > If we were pre 1.0, I would probably change the behavior to be in-line > with numarray. But, now I think it needs some discussion because we are > changing the behavior of a released version of NumPy and we need some > more conservatism in how changes happen. > > If we can classify the current behavior as a bug then we can change it. > Otherwise, I'm concerned. > > The behavior has to do with a mixed scalar/array computation. NumPy > does not let scalars (equivalently 0-d arrays) determine the output type > of the array operation, *unless* the scalar is of a fundamentally > different kind (i.e. the array is an integer-type but the scalar is a > floating-point type). In this case, the current behavior is to coerce > the array to the smallest-type in that general category of scalars. > > The reason for this behavior is to make sure that > > array([1,2,3,4],int8)*10 returns an int8 (instead of an int32 because > of how the 10 is interpreted by Python). > > The current behavior, however, also means that > > array([1,2,3,4],int8)*10.0 will return a float32 array. > > > I think numarray would return a float64 array in this case (i.e. the > type of the scalar would be honored when the coercion was between two > different kinds of arrays). I feel that the default types should be integer, float64, cfloat64, with the other types for designer apps with special needs. However array([1,2,3,4],int8)*float32(10.0) Should coerce to float32, not float64, because whoever wrote it took the trouble to be specific about the type for a reason. So I agree with the change you propose and I doubt it will be noticed except by those who find numpy behaving as they expect. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at shrogers.com Thu Mar 1 20:47:44 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Thu, 01 Mar 2007 18:47:44 -0700 Subject: [Numpy-discussion] [SciPy-user] NumPy in Teaching In-Reply-To: References: <45E4FFA6.9010408@shrogers.com> Message-ID: <45E78240.4060503@shrogers.com> Alan G Isaac wrote: > On Tue, 27 Feb 2007, "Steven H. Rogers" apparently wrote: > >> I'm doing an informal survey on the use of Array >> Programming Languages for teaching. If you're using NumPy >> in this manner I'd like to hear from you. What subject >> was/is taught, academic level, results, lessons learned, >> etc. >> > > I'm using NumPy in a PhD level microeconomics course. The > students are using numpy primarily for the matrix class. In > the past I have used GAUSS. Programming is a fairly small > component of the course. Although most of the students have > no programming background, they are finding Python and NumPy > surprisingly easy to use. > > Cheers, > Alan Isaac > Thanks Alan. May I ask the institution where you're teaching? # Steve From charlesr.harris at gmail.com Thu Mar 1 21:40:06 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 19:40:06 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E75EC2.7090905@cc.usu.edu> References: <45E6F964.10509@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> <45E7519B.2070002@gmail.com> <45E75EC2.7090905@cc.usu.edu> Message-ID: On 3/1/07, Mark P. Miller wrote: > > OK...here goes. This code is going to look goofy, so please bear in > mind that it is only an abstraction of what my real code does (which > happens to provide interesting and meaning insights!). > > I've attached saved versions of my interactive python sessions that > document the phenomenon. Again, this code mainly serves to illustrate > what I previously saw. Please note that I DO NOT a) redefine arrays, or > b) re-import functions in my real code. > > 1) Fresh install of Enthought python edition and reboot. > > Python 2.4.3 - Enthought Edition 1.0.0 (#69, Aug 2 2006, 12:09:59) [MSC > v.1310 32 bit (Intel)] on win32 > Type "copyright", "credits" or "license()" for more information. > > **************************************************************** > Personal firewall software may warn about the connection IDLE > makes to its subprocess using this computer's internal loopback > interface. This connection is not visible on any external > interface and no data is sent to or received from the Internet. > **************************************************************** In [46]: def test1() : ....: x = normal(0,1,1000) ....: In [47]: def test2() : ....: for i in range(1000) : ....: x = normal(0,1) In [50]: t = timeit.Timer('test1()', "from __main__ import test1, normal") In [51]: t.timeit(100) Out[51]: 0.022681951522827148 In [52]: t = timeit.Timer('test2()', "from __main__ import test2, normal") In [53]: t.timeit(100) Out[53]: 4.3481810092926025 Looks like function call overhead has gone way up or the cost of returning a float vs an array has gone way up. The loop overhead is about .01 and not significant. So something is definitely wrong here. Time to go look in trac ;) I won't comment on the code itself. Tell us what you want to do and I bet we can speed it up. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Mar 1 22:05:37 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Mar 2007 21:05:37 -0600 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: References: <45E6F964.10509@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> <45E7519B.2070002@gmail.com> <45E75EC2.7090905@cc.usu.edu> Message-ID: <45E79481.80307@gmail.com> Charles R Harris wrote: > Looks like function call overhead has gone way up or the cost of > returning a float vs an array has gone way up. The loop overhead is > about .01 and not significant. So something is definitely wrong here. > Time to go look in trac ;) It might have been when Travis put in the broadcasting capabilities to the nonuniform distribution generators. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Mar 1 22:48:12 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Mar 2007 20:48:12 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: <45E79481.80307@gmail.com> References: <45E6F964.10509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> <45E7519B.2070002@gmail.com> <45E75EC2.7090905@cc.usu.edu> <45E79481.80307@gmail.com> Message-ID: On 3/1/07, Robert Kern wrote: > > Charles R Harris wrote: > > > Looks like function call overhead has gone way up or the cost of > > returning a float vs an array has gone way up. The loop overhead is > > about .01 and not significant. So something is definitely wrong here. > > Time to go look in trac ;) > > It might have been when Travis put in the broadcasting capabilities to the > nonuniform distribution generators. Could be, the time is about right: changeset 2892, 7/25/06. Hmmm, there's a lot of code there. Must say nothing jumps out at me 8) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at ar.media.kyoto-u.ac.jp Thu Mar 1 22:56:33 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 02 Mar 2007 12:56:33 +0900 Subject: [Numpy-discussion] [SciPy-user] NumPy in Teaching In-Reply-To: <45E7060A.9040105@astraw.com> References: <200703010032.l210WfhI005995@glup.physics.ucf.edu> <45E7060A.9040105@astraw.com> Message-ID: <45E7A071.5020303@ar.media.kyoto-u.ac.jp> Andrew Straw wrote: > Perry Greenfield wrote: >> On Feb 28, 2007, at 7:32 PM, Joe Harrington wrote: >> >> >>> Hi Steve, >>> >>> I have taught Astronomical Data Analysis twice at Cornell using IDL, >>> and I will be teaching it next Fall at UCF using NumPy. Though I've >>> been active here in the recent past, I'm actually not a regular NumPy >>> user myself yet (I used Numeric experimentally for about 6 months in >>> 1997), so I'm a bit nervous. There isn't the kind of documentation >>> and how-to support for Numpy that there is for IDL, though our web >>> site is a start in that direction. One thought I've had in making the >>> transition easier is to put up a syntax and function concordance, >>> similar to that available for MATLAB. I thought this existed. Maybe >>> Perry can point me to it. Just adding a column to the MATLAB one >>> would be fine. >>> >> I made one for IDL, but I don't recall one for matlab. If anyone has >> done one, I sure would like to incorporate it into the tutorial I'm >> revising if possible. >> >> > I believe the reference is to this: http://scipy.org/NumPy_for_Matlab_Users Another reference which was really useful for me was this one http://37mm.no/matlab-python-xref.html I actually found it more useful than the wiki (at the time I made the step towards numpy for all my numerical needs, that is around one year ago). Note that this site also have reference wrt IDL and R, the latter having been useful for me when I thought about ditching matlab for R David From david at ar.media.kyoto-u.ac.jp Fri Mar 2 00:02:30 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 02 Mar 2007 14:02:30 +0900 Subject: [Numpy-discussion] [ANN] garnumpy 0.1, a gar system to build a self contained scipy environment (unix only) Message-ID: <45E7AFE6.9010602@ar.media.kyoto-u.ac.jp> Hi, I am not sure whether this is really useful to other, but anyway: I release the first version of garnumpy, a system derived from Nick Moffit's gar system, to build a set of packages from sources. The idea is to build numpy + scipy automatically with its dependencies, like a complete LAPACK, umfpack, etc... Right now, support for ATLAS (3.7.28), NETLIB BLAS/LAPACK only is there. It can also build a FFT library (fftw3 supported for now), and a UMFPACK system. This can be useful: - if you are on distributions without a full blas/lapack or umfpack packaged on your system - with a CPU not well supported by last release of ATLAS (Core2Duo, which got a lot of improvements in the 3.7 series) - if you do not want (or like me cannot) install numpy system-wide It has been successfully used on Fedora Core and Ubuntu with gcc+g77 or gcc+gfortran (by successfully, I mean all scipy tests passed). I don't really know if this is easier to use than building scipy/numpy by hand (it is for me, at least:) ). http://www.ar.media.kyoto-u.ac.jp/members/david/archives/garnumpy/garnumpy-0.1.tgz Here is the Readme: What is garnumpy ? Garnumpy is a system derived from the GAR system by Nick Moffitt to build a self contained scipy environment. It supports optional compilation of most scipy dependencies, including fftw3, ATLAS or NETLIB Blas and Lapack, UMFPACK. What for ? If you want to install a scipy environment, with special libraries, or libraries not packaged on your OS (full lapack using ATLAS, etc...). If you want a self contained directory where everything is installed, for easy removing later (everything in in one directory) As it installs everything in a special directory, you can use garnumpy without special rights (eg non root), and without the risk of destroying anything on your system. How to use ? Short story: cd platform/scipy; make install will install numpy, scipy and all the dependencies (by default, it will build fftw3, and NETLIB BLAS/LAPACK), if you have a "standard" GNU userland. Then, in a shell: source startgarnumpy.sh will set the necessary env variables to use scipy in the current shell. Longer story: before the above, you should: - set main_prefix and GARCHIVEROOT in gar.conf.mk to some values - make garchive will download all the sources in one step (useful if you plan on trying different build options) - if you change main_prefix, you should change accordingly startgarnumpy.sh Other variable to adjust in gar.conf.mk: - BLASLAPACK: set to the wanted BLAS/LAPACK set. By default, netlib, but atlas (for using ATLAS BLAS/LAPACK) and system (using already installed BLAS/LAPACK) are also supported. - SCIPYSANDPKG: a list of packages in scipy sandbox to install - TESTNELIB: set to 1 to test the compiled NETLIB LAPACK library - SOURCEFORGEDL: set to a proper sourceforge mirror (redirection often fail). Variable to adjust in gar.cc.mk (everything related to the build tools, mostly compiler, compiler options and link options should be set here) - CC, F77 and CXX: C, Fortran and C++ compilers You can use the two following working templates: - gar.cc.mk.g77: GNU build system with g77 for F77 code. - gar.cc.mk.gfortran: GNU build system with gfortran for all Fortran code. Supported softwares: - numpy and scipy (in platform) - ipython (in tools) - matplolib (in gui) Dependencises: UBUNTU (6.10): You can build a fully functional scipy + matplotlib by installing the following: sudo apt-get install python python-dev gcc g77 python-gtk2-dev patch swig g++ Fedora Core (6): You can build a fully functional numpy + scipy by installing the following: sudo yum install python python-devel gcc compat-gcc-34-g77 gcc-c++ swig Cheers, David From ivilata at carabos.com Fri Mar 2 06:40:32 2007 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Fri, 2 Mar 2007 12:40:32 +0100 Subject: [Numpy-discussion] [ANN] PyTables 2.0 alpha2 In-Reply-To: <20070301160643.GK7777@tardis.terramar.selidor.net> References: <20070301160643.GK7777@tardis.terramar.selidor.net> Message-ID: <20070302114032.GA5969@rampella.terramar.selidor.net> Hi everybody once again, We have done a new micro-release of the second alpha of PyTables 2.0, PyTables 2.0a2a. This fixes a missing import (thanks to Antonio Valentino and Steven H. Rogers for the information) and missing images in the HTML version of the manual in the 2.0a2 version released yesterday. We hope that the next release will be a beta one, and we encourage you to test it. Thank you! As usual, the released files are available at http://www.pytables.org/download/preliminary/ For more information on PyTables, visit http://www.pytables.org/ Cheers, :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 309 bytes Desc: Digital signature URL: From mpmusu at cc.usu.edu Fri Mar 2 09:50:00 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Fri, 02 Mar 2007 07:50:00 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: References: <45E6F964.10509@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> <45E7519B.2070002@gmail.com> <45E75EC2.7090905@cc.usu.edu> Message-ID: <45E83998.5020609@cc.usu.edu> > > I won't comment on the code itself. Appreciate it :). Again, I'm just pointing out an example! I would guess that I'm not the only person using numpy in this sort of unsophisticated manner! Tell us what you want to do and I > bet we can speed it up. > > Chuck I'll probably do that in the future. Just need some more time AND a good way to represent my code in a simplified manner so that it you all get the gist of my overall project goals. For the most part, it's appearing that vectorization approaches won't really do the trick for me. -Mark From david.huard at gmail.com Fri Mar 2 14:07:26 2007 From: david.huard at gmail.com (David Huard) Date: Fri, 2 Mar 2007 14:07:26 -0500 Subject: [Numpy-discussion] Unifying numpy, scipy, and matplotlib docstring formats In-Reply-To: <20070227095043.GB26292@crater.logilab.fr> References: <45D4ED02.1050908@noaa.gov> <45D50245.1080102@noaa.gov> <1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com> <1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com> <20070227095043.GB26292@crater.logilab.fr> Message-ID: <91cf711d0703021107u3dfb1bebxb27820e223c2add2@mail.gmail.com> To get back to the original subject of the thread, there has been a commit today on the docutils plugins branch that allows folks to write their own parser/writer and somehow include it in docutils, without having to rebuild docutils. In other words, NumPy could define a plugin, and if docutils is installed, link the plugin to the docutils installation. This would allow third party package that use docutils to process numpy docstrings using the plugin. I don't know how far we can disgress from the reST syntax using those plugins, but it's probably something worth looking at. Check out: svn://svn.berlios.de/docutils/branches/plugins and the info about the plugin hookup is in docs/howto/extensions.txt David 2007/2/27, Alexandre Fayolle : > > On Sun, Feb 25, 2007 at 06:44:37PM +0200, Jouni K. Sepp??nen wrote: > > "Barry Wark" writes: > > > > > Yes, I agree. I wasn't coming at so much from the goal of making Pylab > > > a Matlab clone (as you point out, that's silly, and misses much of the > > > advantage of Python), but rather from the goal of making interactive > > > use as efficient as possible. When I fire up ipython -pylab to do some > > > quick exploration, it's nice not to have to type N.blah or pylab.plot > > > > IMHO the greatest strength of Matlab in interactive use is the matrix > > input format. For one thing, it is easier to type something like > > > > [0 1 0; 1 0 0; 0 0 1] > > > > than > > > > array([[0,1,0],[1,0,0],[0,0,1]]) > > > A very nice shortcut in my opinion is the one supported by the HP > scientific calculators (HP28, HP48 in my days), which would look like: > > array([[0,1,0], 1, 0, 0, 0, 0, 1]) > > I'm not quite sure how this could be generalized nicely for arbitrary > shapes, but for arrays of rank 2 (which are a very common case where > people are actually typing the values) it feels nice (especially on a > french keyboard where the square brackets are awkward to type in) > > -- > Alexandre Fayolle LOGILAB, Paris (France) > Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations > D?veloppement logiciel sur mesure: http://www.logilab.fr/services > Informatique scientifique: http://www.logilab.fr/science > Reprise et maintenance de sites CPS: http://www.migration-cms.com/ > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (GNU/Linux) > > iQEVAwUBReP+816T+PKoJ87eAQITUQf/bVo3QH6hZV/kM71v89WzMODMeZb8IxXs > BwoSSuL5YZGETKiR9wMmAcqVceJ6P0sW/5slWlJjXAONDfvcdTFkFaV1tyC2gWbP > tTJu4kLe8BX7y/UsCM1y2U90GGI/LbYVAjMa+4HdWdsMoBTRN4tyeEuq/3Upg/rN > GWW7+X+1poO1ZtjWDOpVS1DcZeioiYtsXvU0zAKe+briPM1QnIc1K2zfkwUBhA7F > L7osi3uq+muucKE/sEpBQ9SotLbfJ+yFIZOYBSDiFPCSJi9bE6KvKVm+EfXt23KY > 09dOd3rx+H471Pv90uzAVULLDUXucU/tKLBxVHhzMu6rzdb297QmfA== > =1RL1 > -----END PGP SIGNATURE----- > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Fri Mar 2 14:35:44 2007 From: aisaac at american.edu (Alan Isaac) Date: Fri, 02 Mar 2007 14:35:44 -0500 Subject: [Numpy-discussion] Unifying numpy, scipy, and matplotlib docstring formats In-Reply-To: <91cf711d0703021107u3dfb1bebxb27820e223c2add2@mail.gmail.com> References: <45D4ED02.1050908@noaa.gov> <45D50245.1080102@noaa.gov> <1e2af89e0702160148u6f3b4ce4re11423f4ca9a35ec@mail.gmail.com> <1e2af89e0702180346g24f324a7je1f535f5fd6e0b96@mail.gmail.com> <20070227095043.GB26292@crater.logilab.fr> <91cf711d0703021107u3dfb1bebxb27820e223c2add2@mail.gmail.com> Message-ID: <45E87C90.3030203@american.edu> David Huard wrote: > I don't know how far we can disgress from the reST syntax using > those plugins, but it's probably something worth looking at. Digression => maintenance burden. Cheers, Alan Isaac From Chris.Barker at noaa.gov Fri Mar 2 16:10:53 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 02 Mar 2007 13:10:53 -0800 Subject: [Numpy-discussion] Should array_equal work with non-numeric types? Message-ID: <45E892DD.5080303@noaa.gov> Hi all, Should array_equal work with non-numeric types? >>> a array(['a', 'b', 'c', 'd', 'e', 'f', 'g'], dtype='|S1') >>> b = a.copy() >>> a == b array([True, True, True, True, True, True, True], dtype=bool) >>> numpy.array_equal(a,b) Traceback (most recent call last): File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy/core/numeric.py", line 652, in array_equal return logical_and.reduce(equal(a1,a2).ravel()) AttributeError: 'NotImplementedType' object has no attribute 'ravel' I can use: >>> numpy.all(a==b) True But I was wondering if array_equal should work. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Fri Mar 2 16:43:38 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 3 Mar 2007 06:43:38 +0900 Subject: [Numpy-discussion] [SciPy-user] NumPy in Teaching In-Reply-To: <45E7060A.9040105@astraw.com> References: <200703010032.l210WfhI005995@glup.physics.ucf.edu> <45E7060A.9040105@astraw.com> Message-ID: On 3/2/07, Andrew Straw wrote: > Perry Greenfield wrote: > > On Feb 28, 2007, at 7:32 PM, Joe Harrington wrote: > > > > > > > > > I believe the reference is to this: http://scipy.org/NumPy_for_Matlab_Users That page also has this link at the bottom: http://37mm.no/matlab-python-xref.html That page contains a Numpy command cross reference for Matlab IDL and R. --bb From pgmdevlist at gmail.com Fri Mar 2 16:58:37 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 2 Mar 2007 16:58:37 -0500 Subject: [Numpy-discussion] Should array_equal work with non-numeric types? In-Reply-To: <45E892DD.5080303@noaa.gov> References: <45E892DD.5080303@noaa.gov> Message-ID: <200703021658.37364.pgmdevlist@gmail.com> On Friday 02 March 2007 16:10:53 Christopher Barker wrote: > Hi all, > > Should array_equal work with non-numeric types? I'm all for that. I was regularly running into that problem when comparing object arrays or chararrays, that's why I modified assert_equal in the "maskedarray" version of testutils (scipy SVN). >>>from maskedarray.testutils import assert_equal >>>a = N.array(['a','b','c']) >>>assert_equal(a,a.copy()) >>>assert_equal(a,a[::-1]) AssertionError: Items are not equal: item=0 ACTUAL: 'a' DESIRED: 'c' From schaffer at optonline.net Sat Mar 3 09:25:03 2007 From: schaffer at optonline.net (Les Schaffer) Date: Sat, 03 Mar 2007 09:25:03 -0500 Subject: [Numpy-discussion] NumPy in Teaching In-Reply-To: <200703010032.l210WfhI005995@glup.physics.ucf.edu> References: <200703010032.l210WfhI005995@glup.physics.ucf.edu> Message-ID: <45E9853F.4050506@optonline.net> Joe Harrington wrote: > My syllabi (there are undergrad and grad versions) are at: > > Cornell courses (undergrad only): > http://physics.ucf.edu/~jh/ast/ast234-2003/ > http://physics.ucf.edu/~jh/ast/ast234-2004/ > > File not found .... Les Schaffer From jdh2358 at gmail.com Sat Mar 3 13:22:19 2007 From: jdh2358 at gmail.com (John Hunter) Date: Sat, 3 Mar 2007 12:22:19 -0600 Subject: [Numpy-discussion] pretty print array Message-ID: <88e473830703031022g3d57582di822100b19c87fe16@mail.gmail.com> I have a numpy array of floats, and would like an easy way of specifying the format string when printing the array, eg print x.pprint('%1.3f') would do the normal repr of the array but using my format string for the individual elements. Is there and easy way to get something like this currently? Thanks, JDH From steve at shrogers.com Sat Mar 3 14:17:23 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 03 Mar 2007 12:17:23 -0700 Subject: [Numpy-discussion] Buffer PEP in NumPy SVN In-Reply-To: <45E5EA4D.1010303@ee.byu.edu> References: <45E5EA4D.1010303@ee.byu.edu> Message-ID: <45E9C9C3.3060600@shrogers.com> Travis Oliphant wrote: > I just wanted to point people to the online version of the PEP. I'm > still looking for comments and suggestions. The current version is here: > > http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt > > -Travis > Hi Travis: Have you contacted PIL developers to see if they will support this? Your proposal looks like it would take a manageable amount of work to implement and shouldn't cause any long term maintenance issues. Have you had any response from the Python core developers? # Steve From Glen.Mabey at swri.org Sat Mar 3 14:46:50 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Sat, 3 Mar 2007 13:46:50 -0600 Subject: [Numpy-discussion] [numpy] argmin in multidimensional arrays Message-ID: <20070303194650.GA9171@swri16wm.electro.swri.edu> On Thu, 06 Jul 2006 04:59:22 -0600, Travis wrote: > Subject: numpy : argmin in multidimensional arrays > > TG wrote: > > Hi there. > > > > I am working with multi-dimensional arrays and I need to get > > coordinates of the min value in it. > > > > using myarray.argmin() returns the index in the flatten array, which is > > a first step, but I wonder if it is possible to get the coordinates > > directly as an array, rather than calculating them myself by using this > > flat index and the shape of the array. > > > > well, in fact i'm not sure to understand how argmin(myarray) works, > > when myarray is multidimensional. > > > By default, the argmin method flattens the array and returns the flat > index. You can get the corresponding element using > > myarray.flat[index] > > Alternatively, you can use the function unravel_index > > unravel_index(flat_index, myarray.shape) > > to return an N-dimensional index. Does anyone else find this behavior counter-intuitive? It seems to me that one of the great design features of numpy is the n-dim generality it provides, and argmin is one function in this breaks down, IMHO. Glen Mabey From robert.kern at gmail.com Sat Mar 3 16:09:35 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 03 Mar 2007 15:09:35 -0600 Subject: [Numpy-discussion] [numpy] argmin in multidimensional arrays In-Reply-To: <20070303194650.GA9171@swri16wm.electro.swri.edu> References: <20070303194650.GA9171@swri16wm.electro.swri.edu> Message-ID: <45E9E40F.8030203@gmail.com> Glen W. Mabey wrote: > Does anyone else find this behavior counter-intuitive? > > It seems to me that one of the great design features of numpy is the > n-dim generality it provides, and argmin is one function in this breaks > down, IMHO. Not at all. It consistently applies the simple rule: if the method operates over an axis, it takes an axis= keyword argument. The default for the axis= argument is None, which means that it operates over the flattened array. Other axes need to be specified explicitly. See .sum(), .mean(), .var(), .repeat(), .min(), etc. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From steve at shrogers.com Sat Mar 3 16:51:19 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 03 Mar 2007 14:51:19 -0700 Subject: [Numpy-discussion] Buffer PEP in NumPy SVN In-Reply-To: <45E9C9C3.3060600@shrogers.com> References: <45E5EA4D.1010303@ee.byu.edu> <45E9C9C3.3060600@shrogers.com> Message-ID: <45E9EDD7.2090304@shrogers.com> Steven H. Rogers wrote: > Travis Oliphant wrote: > >> I just wanted to point people to the online version of the PEP. I'm >> still looking for comments and suggestions. The current version is here: >> >> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt >> >> -Travis >> >> > Hi Travis: > > Have you contacted PIL developers to see if they will support this? > Your proposal looks like it would take a manageable amount of work to > implement and shouldn't cause any long term maintenance issues. Have > you had any response from the Python core developers? > > # Steve > Disregard this question. I found a thread on this PEP in python-dev. # Steve From torgil.svensson at gmail.com Sat Mar 3 17:11:53 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Sat, 3 Mar 2007 23:11:53 +0100 Subject: [Numpy-discussion] Buffer PEP in NumPy SVN In-Reply-To: <45E5EA4D.1010303@ee.byu.edu> References: <45E5EA4D.1010303@ee.byu.edu> Message-ID: Impressive work and an very intresting approach. I have yet to figure out pros/cons or the variety of options this will give when writing applications/modules. Big plus for giving ctypes a way out of their crappy struct definitions. A few quick questions: Is the concept here that items always are contiguous in memory (=why the strides cant be defined in the format-spec) ? What will PyObject_SizeFromFormat(''(5,5)5t") return ? What are the total memory size of format:'5t' shape:ndim=2,shape=(5,5),strides=NULL ? Huw would the "getbufferproc" function signature look inside the "X{}"-format spec ? //Torgil On 2/28/07, Travis Oliphant wrote: > > I just wanted to point people to the online version of the PEP. I'm > still looking for comments and suggestions. The current version is here: > > http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From Glen.Mabey at swri.org Sat Mar 3 17:32:24 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Sat, 3 Mar 2007 16:32:24 -0600 Subject: [Numpy-discussion] [numpy] argmin in multidimensional arrays In-Reply-To: <45E9E40F.8030203@gmail.com> References: <20070303194650.GA9171@swri16wm.electro.swri.edu> <45E9E40F.8030203@gmail.com> Message-ID: <20070303223224.GA13668@swri16wm.electro.swri.edu> On Sat, Mar 03, 2007 at 03:09:35PM -0600, Robert Kern wrote: > Glen W. Mabey wrote: > > Does anyone else find this behavior counter-intuitive? > > > > It seems to me that one of the great design features of numpy is the > > n-dim generality it provides, and argmin is one function in this breaks > > down, IMHO. > > Not at all. It consistently applies the simple rule: if the method operates over > an axis, it takes an axis= keyword argument. The default for the axis= argument > is None, which means that it operates over the flattened array. Other axes need > to be specified explicitly. > > See .sum(), .mean(), .var(), .repeat(), .min(), etc. Okay, I see the reasoning. I am glad that functions like sum, max, and min return a single value, unless axis is specified. However, it seems to me that the arg* functions are inherently different. After all, it is an index that is sought instead of a value, and when this result cannot be directly applied to the original array, then at first glance it appears to be ... less than intuitive. Thanks for your reply. Glen From hayes.tyler at gmail.com Sat Mar 3 18:05:01 2007 From: hayes.tyler at gmail.com (Tyler Hayes) Date: Sat, 3 Mar 2007 18:05:01 -0500 Subject: [Numpy-discussion] First Numerical Python code...comments? Message-ID: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> Hello All: Well, I recently made the switch to Python for scientific computing (using NumPy and Matplotlib/PyLab) and got my first program to work. It is a simple 1D finite element code for a tightly stretched wire and is a textbook example. To solve it, I also created a symmetric, banded matrix solver module from a Fortran90 routine using f2py (thank you Pearu for your help the other day to get f2py working). Anyways, my background is in Fortran and I was got the program to run, but I can't help but wonder if I did not use the most efficient method for calcualting the matrix values, etc. I would appreciate your comments on my code that I have attached below. My concern is that I used some "for" loops as I would have using Fortran, but perhaps there is a more intelligent (i.e., efficient) way using native NumPy matirx routines? I would love to hear what you think. Also, while I'm at it. I have a simple question about importing NumPy, SciPy, and PyLab. If I import PyLab, it says I also import NumPy. Do I ever need to call in SciPy as well? Or is this the same thing? Cheers, t. The "code" #!/usr/bin/env python # # This is the example code from Thompson for a tight wire problem # # Coded by Tyler Hayes March 1, 2007 # ################################################################# # Load the pylab module for plotting (it also imports NumPy) from pylab import * # Load the sparse matrix solver symgauss created with f2py import symgauss as G """ This is the example program from Thompson which will plot the displacement of a tight wire with constant loading. DEPENDS: This code requires the symgauss module to solve the sparse symmetric matrix. """ # Initialize data #---------------------------------------------------------------- # Parameters tens = 20000.0 numel = 8 numnp = numel + 1 IB = 2 BLAST = 1.0e6 # Boundary conditions npbc = zeros(numnp,float) npbc[0] = 1.0 npbc[-1] = 1.0 # Displacement U = [0.0]*numnp # Element X-coordinates xord = zeros(numnp,float) xord[1] = 2.0 xord[2] = 4.0 xord[3] = 5.0 xord[4] = 6.0 xord[5] = 7.0 xord[6] = 8.0 xord[7] = 10.0 xord[8] = 12.0 # Force data vector F = -1.0*ones(numnp,float) F[1] = -2.0 F[2] = -1.5 F[-2]= -2.0 F[-3]= -1.5 # Initialize SK --- the sparse symmetrix stiffness matrix "K" SK = zeros((numnp,IB),float) # Main routine #---------------------------------------------------------------- # Fill in SK with proper values for i in range(numel): RL = xord[i+1] - xord[i] RK = tens/RL SK[i][0] = SK[i][0] + RK SK[i][1] = SK[i][1] - RK IP1 = i+1 SK[IP1][0] = SK[IP1][0] + RK # Set boundary conditions for i in range(numnp): if npbc[i] == 1: SK[i][0] = SK[i][0]*BLAST F[i] = U[i]*SK[i][0] # Call Symmetric Sparse matrix solver U = G.symgauss(SK,F,numnp,IB) # Visualize the data #---------------------------------------------------------------- # Plot data using matplotlib plot(xord,U) title('Deflection of a tighly stretched wire') xlabel('Distance along wire (normalized)') ylabel('Deflection of wire (normalized)') grid(True) show() -- Tyler Joseph Hayes 600 Talbot St. -- Apt. 812 London, Ontario N6A 5L9 Tel : 519.435.0967 Fax : 519.661.3198 Cell : 416.655.7897 email: thayes at uwo.ca GPG Key ID# 0x3AE05130 From faltet at carabos.com Sun Mar 4 01:53:21 2007 From: faltet at carabos.com (Francesc Altet) Date: Sun, 04 Mar 2007 07:53:21 +0100 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> Message-ID: <1172991201.2620.26.camel@localhost.localdomain> Hi Tyler, I'll try to give you some advices on your style but I won't comment on whether you can substitute some loops by 'more intelligent' operations (other people more savvy than me in these issues can respond better). El ds 03 de 03 del 2007 a les 18:05 -0500, en/na Tyler Hayes va escriure: > Hello All: > > Well, I recently made the switch to Python for scientific computing > (using NumPy and Matplotlib/PyLab) and got my first program to work. > It is a simple 1D finite element code for a tightly stretched wire and > is a textbook example. > > To solve it, I also created a symmetric, banded matrix solver module > from a Fortran90 routine using f2py (thank you Pearu for your help the > other day to get f2py working). > > Anyways, my background is in Fortran and I was got the program to run, > but I can't help but wonder if I did not use the most efficient method > for calcualting the matrix values, etc. > > I would appreciate your comments on my code that I have attached > below. My concern is that I used some "for" loops as I would have > using Fortran, but perhaps there is a more intelligent (i.e., > efficient) way using native NumPy matirx routines? > > I would love to hear what you think. > > Also, while I'm at it. I have a simple question about importing NumPy, > SciPy, and PyLab. If I import PyLab, it says I also import NumPy. Do I > ever need to call in SciPy as well? Or is this the same thing? > > Cheers, > > t. > > The "code" > > #!/usr/bin/env python > # > # This is the example code from Thompson for a tight wire problem > # > # Coded by Tyler Hayes March 1, 2007 > # > ################################################################# > > # Load the pylab module for plotting (it also imports NumPy) > from pylab import * > # Load the sparse matrix solver symgauss created with f2py > import symgauss as G > """ > This is the example program from Thompson which will plot the > displacement of a tight wire with constant loading. > > DEPENDS: > This code requires the symgauss module to solve the sparse > symmetric matrix. > """ > > # Initialize data > #---------------------------------------------------------------- > > # Parameters > tens = 20000.0 > numel = 8 > numnp = numel + 1 > IB = 2 > BLAST = 1.0e6 > > # Boundary conditions > npbc = zeros(numnp,float) > npbc[0] = 1.0 > npbc[-1] = 1.0 > > # Displacement > U = [0.0]*numnp > > # Element X-coordinates > xord = zeros(numnp,float) > xord[1] = 2.0 > xord[2] = 4.0 > xord[3] = 5.0 > xord[4] = 6.0 > xord[5] = 7.0 > xord[6] = 8.0 > xord[7] = 10.0 > xord[8] = 12.0 Probably you know that you can achieve the same with: xord = array([0., 2., 4., 5., 6., 7., 8., 10., 12.]) [you don't need to specify 'float' because you have already specified the decimal point in values] > > # Force data vector > F = -1.0*ones(numnp,float) > F[1] = -2.0 > F[2] = -1.5 > F[-2]= -2.0 > F[-3]= -1.5 you can achieve the same using fancy indexing: F = -1.0*ones(numnp,float) F[[1,2,-2,-3]] = -2., -1.5, -2., -1.5 which is handy for saving a few lines in initialization > > # Initialize SK --- the sparse symmetrix stiffness matrix "K" > SK = zeros((numnp,IB),float) > > > # Main routine > #---------------------------------------------------------------- > > # Fill in SK with proper values > for i in range(numel): Always try to use xrange(numel) instead of range(numel). range() will return you a python list, and for large values of numel these can take precious amounts of memory. On his part, xrange() returns an iterator, and it never takes more than a few bytes. > RL = xord[i+1] - xord[i] > RK = tens/RL > SK[i][0] = SK[i][0] + RK a possible shortcut would be: SK[i][0] += RK which generally takes less time to type, reads better and allows for in-place operations (more memory efficient) when the left operand is an array. > SK[i][1] = SK[i][1] - RK > IP1 = i+1 > SK[IP1][0] = SK[IP1][0] + RK the same here: SK[i][1] -= RK IP1 = i+1 SK[IP1][0] += RK > > # Set boundary conditions > for i in range(numnp): for i in xrange(numnp): > if npbc[i] == 1: This if clause is comparing a float with a value. In general, try to avoid using strict equality comparations between floats mainly because they use an inexact representation. Instead, use the numpy.allclose() function better: if allclose(npbc[i], 1.): see the allclose function docs for more details. > SK[i][0] = SK[i][0]*BLAST SK[i][0] *= BLAST > F[i] = U[i]*SK[i][0] > > # Call Symmetric Sparse matrix solver > U = G.symgauss(SK,F,numnp,IB) > > > # Visualize the data > #---------------------------------------------------------------- > > # Plot data using matplotlib > plot(xord,U) > title('Deflection of a tighly stretched wire') > xlabel('Distance along wire (normalized)') > ylabel('Deflection of wire (normalized)') > grid(True) > show() > > HTH, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From tim.leslie at gmail.com Sun Mar 4 04:15:58 2007 From: tim.leslie at gmail.com (Tim Leslie) Date: Sun, 4 Mar 2007 20:15:58 +1100 Subject: [Numpy-discussion] Adding a delete flag to make_svn_version_py() in distutils Message-ID: Hi All, I've got a small patch which I thought I'd run past the list before checking in. While working on the NIPY project (http://neuroimaging.scipy.org/) we spend a lot of time working from a local source install, rather than a version installed in site-packages. The mechanism which lets you do things like numpy.__version__ and neuroimaging.__version__ and have it show the svn version e.g. In [3]: neuroimaging.__version__ Out[3]: '0.1.2.dev1570' requires the file __svn_version__.py to be in the tree. Currently this file is deleted automatically. The patch adds a flag which makes deleting this file optional (with delete=True being default). This makes the svn version magic work for us and shouldn't break any existing code for anyone, Is there any reason why this shouldn't go in? I'm happy to make the checkin once it gets the OK from someone. Cheers, Tim Leslie Index: numpy/distutils/misc_util.py =================================================================== --- numpy/distutils/misc_util.py (revision 3567) +++ numpy/distutils/misc_util.py (working copy) @@ -1304,7 +1304,7 @@ return version - def make_svn_version_py(self): + def make_svn_version_py(self, delete=True): """ Generate package __svn_version__.py file from SVN revision number, it will be removed after python exits but will be available when sdist, etc commands are executed. @@ -1326,10 +1326,12 @@ import atexit def rm_file(f=target,p=self.info): - try: os.remove(f); p('removed '+f) - except OSError: pass - try: os.remove(f+'c'); p('removed '+f+'c') - except OSError: pass + if delete: + try: os.remove(f); p('removed '+f) + except OSError: pass + try: os.remove(f+'c'); p('removed '+f+'c') + except OSError: pass + atexit.register(rm_file) return target From pearu at cens.ioc.ee Sun Mar 4 05:37:52 2007 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Sun, 4 Mar 2007 12:37:52 +0200 (EET) Subject: [Numpy-discussion] Adding a delete flag to make_svn_version_py() in distutils In-Reply-To: References: Message-ID: <2003.82.131.69.129.1173004672.squirrel@cens.ioc.ee> Hi, The patch looks ok to me. Pearu > Hi All, > > I've got a small patch which I thought I'd run past the list before > checking in. While working on the NIPY project > (http://neuroimaging.scipy.org/) we spend a lot of time working from a > local source install, rather than a version installed in > site-packages. The mechanism which lets you do things like > numpy.__version__ and neuroimaging.__version__ and have it show the > svn version e.g. > > In [3]: neuroimaging.__version__ > Out[3]: '0.1.2.dev1570' > > requires the file __svn_version__.py to be in the tree. Currently this > file is deleted automatically. The patch adds a flag which makes > deleting this file optional (with delete=True being default). This > makes the svn version magic work for us and shouldn't break any > existing code for anyone, Is there any reason why this shouldn't go > in? I'm happy to make the checkin once it gets the OK from someone. > > Cheers, > > Tim Leslie > > Index: numpy/distutils/misc_util.py > =================================================================== > --- numpy/distutils/misc_util.py (revision 3567) > +++ numpy/distutils/misc_util.py (working copy) > @@ -1304,7 +1304,7 @@ > > return version > > - def make_svn_version_py(self): > + def make_svn_version_py(self, delete=True): > """ Generate package __svn_version__.py file from SVN revision > number, > it will be removed after python exits but will be available > when sdist, etc commands are executed. > @@ -1326,10 +1326,12 @@ > > import atexit > def rm_file(f=target,p=self.info): > - try: os.remove(f); p('removed '+f) > - except OSError: pass > - try: os.remove(f+'c'); p('removed '+f+'c') > - except OSError: pass > + if delete: > + try: os.remove(f); p('removed '+f) > + except OSError: pass > + try: os.remove(f+'c'); p('removed '+f+'c') > + except OSError: pass > + > atexit.register(rm_file) > > return target > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From faltet at carabos.com Sun Mar 4 08:18:22 2007 From: faltet at carabos.com (Francesc Altet) Date: Sun, 04 Mar 2007 14:18:22 +0100 Subject: [Numpy-discussion] Recarray sort() slowness Message-ID: <1173014302.2620.39.camel@localhost.localdomain> Hi, I've finally implemented Chuck's suggestion of sorting of a recarray of two fields (the first being the actual array to be sorted and the other being the array to be reordered following the resulting order for the first one). Indeed, this approach saves me an amount of memory equivalent to the size of the second column, which is really nice. However, I'm afraid that I wouldn't be able to use this approach as it is 25x slower (see the attached benchmark; beware! only runs on a Linux kernel 2.6!) than regular argsorting of the first field and then doing a fancy indexing over the second. If the slowdown would be 2x I still can have a chance to use it, but 25x is a no go. I'm curious why the recarray.sort(order='fieldN') is so slow, and I'm wondering if this can be speed-up in some way or another. Thanks, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth -------------- next part -------------- A non-text attachment was scrubbed... Name: sort-ra.py Type: text/x-python Size: 2030 bytes Desc: not available URL: From torgil.svensson at gmail.com Sun Mar 4 09:17:39 2007 From: torgil.svensson at gmail.com (Torgil Svensson) Date: Sun, 4 Mar 2007 15:17:39 +0100 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> Message-ID: > # Fill in SK with proper values > for i in range(numel): > RL = xord[i+1] - xord[i] > RK = tens/RL > SK[i][0] = SK[i][0] + RK > SK[i][1] = SK[i][1] - RK > IP1 = i+1 > SK[IP1][0] = SK[IP1][0] + RK To get advantage of numpy you'll need to calculate with arrays instead of individual numbers: RL=xord[1:]-xord[:-1] RK=tens/RL SK[:-1,0]+=RK SK[:-1,1]-=RK SK[1:,0]+=RK hope I got that right... but you'll get the point anyway //Torgil On 3/4/07, Tyler Hayes wrote: > > Hello All: > > Well, I recently made the switch to Python for scientific computing > (using NumPy and Matplotlib/PyLab) and got my first program to work. > It is a simple 1D finite element code for a tightly stretched wire and > is a textbook example. > > To solve it, I also created a symmetric, banded matrix solver module > from a Fortran90 routine using f2py (thank you Pearu for your help the > other day to get f2py working). > > Anyways, my background is in Fortran and I was got the program to run, > but I can't help but wonder if I did not use the most efficient method > for calcualting the matrix values, etc. > > I would appreciate your comments on my code that I have attached > below. My concern is that I used some "for" loops as I would have > using Fortran, but perhaps there is a more intelligent (i.e., > efficient) way using native NumPy matirx routines? > > I would love to hear what you think. > > Also, while I'm at it. I have a simple question about importing NumPy, > SciPy, and PyLab. If I import PyLab, it says I also import NumPy. Do I > ever need to call in SciPy as well? Or is this the same thing? > > Cheers, > > t. > > The "code" > > #!/usr/bin/env python > # > # This is the example code from Thompson for a tight wire problem > # > # Coded by Tyler Hayes March 1, 2007 > # > ################################################################# > > # Load the pylab module for plotting (it also imports NumPy) > from pylab import * > # Load the sparse matrix solver symgauss created with f2py > import symgauss as G > """ > This is the example program from Thompson which will plot the > displacement of a tight wire with constant loading. > > DEPENDS: > This code requires the symgauss module to solve the sparse > symmetric matrix. > """ > > # Initialize data > #---------------------------------------------------------------- > > # Parameters > tens = 20000.0 > numel = 8 > numnp = numel + 1 > IB = 2 > BLAST = 1.0e6 > > # Boundary conditions > npbc = zeros(numnp,float) > npbc[0] = 1.0 > npbc[-1] = 1.0 > > # Displacement > U = [0.0]*numnp > > # Element X-coordinates > xord = zeros(numnp,float) > xord[1] = 2.0 > xord[2] = 4.0 > xord[3] = 5.0 > xord[4] = 6.0 > xord[5] = 7.0 > xord[6] = 8.0 > xord[7] = 10.0 > xord[8] = 12.0 > > # Force data vector > F = -1.0*ones(numnp,float) > F[1] = -2.0 > F[2] = -1.5 > F[-2]= -2.0 > F[-3]= -1.5 > > # Initialize SK --- the sparse symmetrix stiffness matrix "K" > SK = zeros((numnp,IB),float) > > > # Main routine > #---------------------------------------------------------------- > > # Fill in SK with proper values > for i in range(numel): > RL = xord[i+1] - xord[i] > RK = tens/RL > SK[i][0] = SK[i][0] + RK > SK[i][1] = SK[i][1] - RK > IP1 = i+1 > SK[IP1][0] = SK[IP1][0] + RK > > # Set boundary conditions > for i in range(numnp): > if npbc[i] == 1: > SK[i][0] = SK[i][0]*BLAST > F[i] = U[i]*SK[i][0] > > # Call Symmetric Sparse matrix solver > U = G.symgauss(SK,F,numnp,IB) > > > # Visualize the data > #---------------------------------------------------------------- > > # Plot data using matplotlib > plot(xord,U) > title('Deflection of a tighly stretched wire') > xlabel('Distance along wire (normalized)') > ylabel('Deflection of wire (normalized)') > grid(True) > show() > > > -- > Tyler Joseph Hayes > 600 Talbot St. -- Apt. 812 > London, Ontario > N6A 5L9 > > Tel : 519.435.0967 > Fax : 519.661.3198 > Cell : 416.655.7897 > email: thayes at uwo.ca > > GPG Key ID# 0x3AE05130 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 4 10:37:45 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 4 Mar 2007 08:37:45 -0700 Subject: [Numpy-discussion] Recarray sort() slowness In-Reply-To: <1173014302.2620.39.camel@localhost.localdomain> References: <1173014302.2620.39.camel@localhost.localdomain> Message-ID: On 3/4/07, Francesc Altet wrote: > > Hi, > > I've finally implemented Chuck's suggestion of sorting of a recarray of > two fields (the first being the actual array to be sorted and the other > being the array to be reordered following the resulting order for the > first one). Indeed, this approach saves me an amount of memory > equivalent to the size of the second column, which is really nice. > > However, I'm afraid that I wouldn't be able to use this approach as it > is 25x slower (see the attached benchmark; beware! only runs on a Linux > kernel 2.6!) than regular argsorting of the first field and then doing a > fancy indexing over the second. If the slowdown would be 2x I still can > have a chance to use it, but 25x is a no go. > > I'm curious why the recarray.sort(order='fieldN') is so slow, and I'm > wondering if this can be speed-up in some way or another. I suspect there are several reasons. 1) It defines a new type with the comparison done on all fields 2) exchanges are done by copying the specified number of bytes I think Travis was clever to define a new type, it made things easy and very general, but it wasn't aimed at speed. There might be some optimizations possible in there that Travis could speak to. It would be pretty easy to modify argsort itself to do what you want in a type specific way using a key vector and a vector to be sorted by the keys. I expect it would be about 1/2 as fast as the normal argsort. Hmmm, something like keysort(...). Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 4 10:54:42 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 4 Mar 2007 08:54:42 -0700 Subject: [Numpy-discussion] Recarray sort() slowness In-Reply-To: References: <1173014302.2620.39.camel@localhost.localdomain> Message-ID: On 3/4/07, Charles R Harris wrote: > > > > On 3/4/07, Francesc Altet wrote: > > > > Hi, > > > > I've finally implemented Chuck's suggestion of sorting of a recarray of > > two fields (the first being the actual array to be sorted and the other > > being the array to be reordered following the resulting order for the > > first one). Indeed, this approach saves me an amount of memory > > equivalent to the size of the second column, which is really nice. > > > > However, I'm afraid that I wouldn't be able to use this approach as it > > is 25x slower (see the attached benchmark; beware! only runs on a Linux > > kernel 2.6!) than regular argsorting of the first field and then doing a > > fancy indexing over the second. If the slowdown would be 2x I still can > > have a chance to use it, but 25x is a no go. > > > > I'm curious why the recarray.sort(order='fieldN') is so slow, and I'm > > wondering if this can be speed-up in some way or another. > > > I suspect there are several reasons. > > 1) It defines a new type with the comparison done on all fields > 2) exchanges are done by copying the specified number of bytes > > I think Travis was clever to define a new type, it made things easy and > very general, but it wasn't aimed at speed. There might be some > optimizations possible in there that Travis could speak to. > > It would be pretty easy to modify argsort itself to do what you want in a > type specific way using a key vector and a vector to be sorted by the keys. > I expect it would be about 1/2 as fast as the normal argsort. Hmmm, > something like keysort(...). > To expand a bit, argsort sorts a vector of indices on the keys. IIRC, it doesn't exchange the keys (a tradeoff between exchange overhead and cache locality), so that would have to be changed, and instead of passing in an array of indices you would pass in the array you want to sort. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 4 10:59:09 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 4 Mar 2007 08:59:09 -0700 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: <1172991201.2620.26.camel@localhost.localdomain> References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> <1172991201.2620.26.camel@localhost.localdomain> Message-ID: On 3/3/07, Francesc Altet wrote: > > > # Fill in SK with proper values > > for i in range(numel): > > Always try to use xrange(numel) instead of range(numel). range() will > return you a python list, and for large values of numel these can take > precious amounts of memory. On his part, xrange() returns an iterator, > and it never takes more than a few bytes. I believe that is no longer the case, range is now iterable and I read somewhere that xrange is deprecated. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sun Mar 4 11:25:20 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 4 Mar 2007 11:25:20 -0500 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com><1172991201.2620.26.camel@localhost.localdomain> Message-ID: On Sun, 4 Mar 2007, Charles R Harris apparently wrote: > range is now iterable and I read > somewhere that xrange is deprecated. There has been a rumor that range will effectively become xrange in Python 300 http://www.python.org/doc/essays/ppt/accu2006/Py3kACCU.ppt but range has always been iterable---did you mean to say "iterator"? but xrange objects to not support a next() method either---and as of Python 2.5 xrange is not deprecated (unless the documentation somehow got out of sync). http://docs.python.org/lib/built-in-funcs.html Cheers, Alan Isaac From rudolf.sykora at gmail.com Sun Mar 4 11:31:10 2007 From: rudolf.sykora at gmail.com (Rudolf Sykora) Date: Sun, 4 Mar 2007 17:31:10 +0100 Subject: [Numpy-discussion] building an array using smaller arrays In-Reply-To: References: Message-ID: Thank you for your reply very much indeed. :) I guessed it could somehow so..., though I thought it could be somwhat simpler (not saying that your solution is too complicated :)) Ruda On 01/03/07, David M. Cooke wrote: > > On Mar 1, 2007, at 13:33 , Rudolf Sykora wrote: > > > Hello, > > > > since noone has reacted to my last e-mail yet (for several days), I > > feel the need to ask again (since I still do not know a good answer). > > Please help me. > > > > >> Hello everybody, > > >> I wonder how I could most easily accomplish the following: > > >> > > >>Say I have sth like: > > >> a = array( [1, 2] ) > > >> and I want to use this array to build another array in the > > following sence: > > >> b = array( [[1, 2, 3, a], [5, a, 6, 7], [0, 2-a, 3, 4]]) # this > > doesn't work > > >> > > >> I would like to obtain > > >> b = array( [[1, 2, 3, 1, 2], [5 ,1 ,2 ,6 ,7], [0, 1, 0, 3, 4]] ) > > > > >> I know a rather complicated way but believe there must be an > > easy one. > > >> Thank you very much. > > > > >> Ruda > > > > I would need some sort of flattening operator... > > The solution I know is very ugly: > > > > b = array(( concatenate(([1, 2, 3], a)), concatenate(([5], a, [6, > > 7])), concatenate(([0], 2-a, [3, 4])) )) > > Define a helper function > > def row(*args): > res = [] > for a in args: > a = asarray(a) > if len(a.shape) == 0: > res.append(a) > elif len(a.shape) == 1: > res += a.tolist() > else: > raise ValueError("arguments to row must be row-like") > return array(res) > > then > > b = array([ row(1,2,3,a), row(5,a,6,7), row(0,2-a,3,4) ]) > > -- > |>|\/|< > /------------------------------------------------------------------\ > |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ > |cookedm at physics.mcmaster.ca > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 4 12:46:55 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 4 Mar 2007 10:46:55 -0700 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> <1172991201.2620.26.camel@localhost.localdomain> Message-ID: On 3/4/07, Alan G Isaac wrote: > > On Sun, 4 Mar 2007, Charles R Harris apparently wrote: > > range is now iterable and I read > > somewhere that xrange is deprecated. > > There has been a rumor that range will effectively become > xrange in Python 300 > http://www.python.org/doc/essays/ppt/accu2006/Py3kACCU.ppt > but range has always been iterable---did you mean to say > "iterator"? but xrange objects to not support a next() > method either---and as of Python 2.5 xrange is not > deprecated (unless the documentation somehow got out of > sync). > http://docs.python.org/lib/built-in-funcs.html My impression was that the traditional range returned a list and then the iteration proceeded over the list, and that has changed, just as it is no longer necessary to use the xreadlines method for files to iterate over the lines. I don't recall where I read this stuff. Hmm, but maybe you are right, that this is planned for python 3.0 but maybe not yet in current versions of python. From a ppt presentation by Guido: dict.keys(), range(), zip() won't return lists killing dict.iterkeys(), xrange(), itertools.izip() Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at shrogers.com Sun Mar 4 14:08:50 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sun, 04 Mar 2007 12:08:50 -0700 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> <1172991201.2620.26.camel@localhost.localdomain> Message-ID: <45EB1942.9080406@shrogers.com> Charles R Harris wrote: > > > On 3/4/07, *Alan G Isaac* > wrote: > > On Sun, 4 Mar 2007, Charles R Harris apparently wrote: > > range is now iterable and I read > > somewhere that xrange is deprecated. > > There has been a rumor that range will effectively become > xrange in Python 300 > http://www.python.org/doc/essays/ppt/accu2006/Py3kACCU.ppt > but range has always been iterable---did you mean to say > "iterator"? but xrange objects to not support a next() > method either---and as of Python 2.5 xrange is not > deprecated (unless the documentation somehow got out of > sync). > http://docs.python.org/lib/built-in-funcs.html > > > > My impression was that the traditional range returned a list and then > the iteration proceeded over the list, and that has changed, just as it > is no longer necessary to use the xreadlines method for files to iterate > over the lines. I don't recall where I read this stuff. Hmm, but maybe > you are right, that this is planned for python 3.0 but maybe not yet in > current versions of python. From a ppt presentation by Guido: > > dict.keys(), range(), zip() won't return lists > killing dict.iterkeys(), xrange(), itertools.izip() > > Chuck > At PyCon, Guido said that range would have xrange behavior and the old range would be deprecated in Python 3.0. This may be back-ported to 2.6. # Steve From aisaac at american.edu Sun Mar 4 14:29:50 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 4 Mar 2007 14:29:50 -0500 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: <45EB1942.9080406@shrogers.com> References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> <1172991201.2620.26.camel@localhost.localdomain> <45EB1942.9080406@shrogers.com> Message-ID: > Charles R Harris wrote: >> that this is planned for python 3.0 but maybe not yet in >> current versions of python. From a ppt presentation by >> Guido: >> dict.keys(), range(), zip() won't return lists >> killing dict.iterkeys(), xrange(), itertools.izip() On Sun, 04 Mar 2007, "Steven H. Rogers" apparently wrote: > At PyCon, Guido said that range would have xrange behavior and the old > range would be deprecated in Python 3.0. This may be back-ported to 2.6. Did he discuss whether the change in ``zip`` is still planned? Unlike the change in range, I find the proposed change in ``zip`` not to be innocuous. (I.e., I'm sure some of my code will fail if my zip object is empty after iteration.) Cheers, Alan From faltet at carabos.com Sun Mar 4 14:30:03 2007 From: faltet at carabos.com (Francesc Altet) Date: Sun, 04 Mar 2007 20:30:03 +0100 Subject: [Numpy-discussion] Recarray sort() slowness In-Reply-To: References: <1173014302.2620.39.camel@localhost.localdomain> Message-ID: <1173036603.3282.10.camel@localhost.localdomain> El dg 04 de 03 del 2007 a les 08:54 -0700, en/na Charles R Harris va escriure: > > > On 3/4/07, Charles R Harris wrote: > > > On 3/4/07, Francesc Altet wrote: > Hi, > > I've finally implemented Chuck's suggestion of sorting > of a recarray of > two fields (the first being the actual array to be > sorted and the other > being the array to be reordered following the > resulting order for the > first one). Indeed, this approach saves me an amount > of memory > equivalent to the size of the second column, which is > really nice. > > However, I'm afraid that I wouldn't be able to use > this approach as it > is 25x slower (see the attached benchmark; beware! > only runs on a Linux > kernel 2.6!) than regular argsorting of the first > field and then doing a > fancy indexing over the second. If the slowdown would > be 2x I still can > have a chance to use it, but 25x is a no go. > > I'm curious why the recarray.sort(order='fieldN') is > so slow, and I'm > wondering if this can be speed-up in some way or > another. > > I suspect there are several reasons. > > 1) It defines a new type with the comparison done on all > fields > 2) exchanges are done by copying the specified number of bytes > > I think Travis was clever to define a new type, it made things > easy and very general, but it wasn't aimed at speed. There > might be some optimizations possible in there that Travis > could speak to. > > It would be pretty easy to modify argsort itself to do what > you want in a type specific way using a key vector and a > vector to be sorted by the keys. I expect it would be about > 1/2 as fast as the normal argsort. Hmmm, something like > keysort(...). > > To expand a bit, argsort sorts a vector of indices on the keys. IIRC, > it doesn't exchange the keys (a tradeoff between exchange overhead and > cache locality), so that would have to be changed, and instead of > passing in an array of indices you would pass in the array you want to > sort. Yes, I think I got the point. Something like: numpy.keysort(vector1, vector2) that would sort vector1 in-place and also reorder vector2 but following vector1 order at the same time. Of course, different dtypes for vector1 and vector2 should be supported (although the shape should be the same). It would be a nice thing to have in NumPy indeed (and would fit perfectly to my needs :). Cheers, > -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From hayes.tyler at gmail.com Sun Mar 4 14:49:56 2007 From: hayes.tyler at gmail.com (Tyler J Hayes) Date: Sun, 4 Mar 2007 19:49:56 +0000 (UTC) Subject: [Numpy-discussion] First Numerical Python code...comments? References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> Message-ID: Thanks for all of your suggestions. It's weird, I *know* about the array methods mentioned as well as the indexing tricks (it's one of the first things you read about), and yet I still *think* in Fortran. I have to get out of that frame of mind. Well, perfect practice, makes perfect, so on to more coding.... Cheers, t. From steve at shrogers.com Sun Mar 4 15:33:12 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sun, 04 Mar 2007 13:33:12 -0700 Subject: [Numpy-discussion] First Numerical Python code...comments? In-Reply-To: References: <92bcafc50703031505x7d0ac32fm3892ae1fe6e975a3@mail.gmail.com> <1172991201.2620.26.camel@localhost.localdomain> <45EB1942.9080406@shrogers.com> Message-ID: <45EB2D08.5030908@shrogers.com> Alan G Isaac wrote: >> Charles R Harris wrote: >>> that this is planned for python 3.0 but maybe not yet in >>> current versions of python. From a ppt presentation by >>> Guido: >>> dict.keys(), range(), zip() won't return lists >>> killing dict.iterkeys(), xrange(), itertools.izip() > > On Sun, 04 Mar 2007, "Steven H. Rogers" apparently wrote: >> At PyCon, Guido said that range would have xrange behavior and the old >> range would be deprecated in Python 3.0. This may be back-ported to 2.6. > > Did he discuss whether the change in ``zip`` is still > planned? Unlike the change in range, I find the proposed > change in ``zip`` not to be innocuous. (I.e., I'm sure some > of my code will fail if my zip object is empty after > iteration.) > Sorry, I don't recall him mentioning zip, but this (http://www.python.org/dev/peps/pep-3099/) implies that zip behavior will be unchanged. # Steve From jh at physics.ucf.edu Sun Mar 4 21:25:29 2007 From: jh at physics.ucf.edu (Joe Harrington) Date: Sun, 4 Mar 2007 21:25:29 -0500 Subject: [Numpy-discussion] NumPy in Teaching Message-ID: <200703050225.l252PTdh020416@glup.physics.ucf.edu> I wrote: > Cornell courses (undergrad only): > http://physics.ucf.edu/~jh/ast/ast234-2003/ > http://physics.ucf.edu/~jh/ast/ast234-2004/ I should have written: Cornell courses (undergrad only): http://physics.ucf.edu/~jh/ast/astro234-2003/ http://physics.ucf.edu/~jh/ast/astro234-2004/ --jh-- From steffen.loeck at gmx.de Mon Mar 5 02:01:54 2007 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Mon, 5 Mar 2007 07:01:54 +0000 Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms In-Reply-To: References: Message-ID: <200703050701.54955.steffen.loeck@gmx.de> Fernando Perez wrote: > I recently got a report of a bug triggered only on 64-bit hardware, > and on a machine (in case it's relevant) that runs python 2.5. This > is with current numpy SVN which I just rebuilt a moment ago to > triple-check: > > In [3]: a = numpy.array([[1.0,2],[3,4]]) > > In [4]: numpy.linalg.qr(a) > ** On entry to DGEQRF parameter number 2 had an illegal value > sage[~]> # dumped back at system prompt Has there been any progress in solving this problem? I get the same error message and have no idea how to solve it. Regards, Steffen From lfriedri at imtek.de Mon Mar 5 02:24:05 2007 From: lfriedri at imtek.de (Lars Friedrich) Date: Mon, 05 Mar 2007 08:24:05 +0100 Subject: [Numpy-discussion] inconsistent mgrid results In-Reply-To: References: Message-ID: <1173079445.3812.9.camel@localhost> Hello Andrew, Hello all, like Andrew, I had some strange experience with mgrid. Adrew writes: Am Dienstag, den 27.02.2007, 19:43 -0600 schrieb Andrew Corrigan: > I'm confused about the following: > > >>> print mgrid[2.45:2.6:0.05, 0:5:1] > [[[ 2.45 2.45 2.45 2.45 2.45] > [ 2.5 2.5 2.5 2.5 2.5 ]] > > [[ 0. 1. 2. 3. 4. ] > [ 0. 1. 2. 3. 4. ]]] > >>> print mgrid[2.45:2.6:0.05] > [ 2.45 2.5 2.55] > > In the first case in the first dimension I get 2.45, 2.5. In the > second case in the first dimension I get 2.45, 2.5, 2.55 In both > cases > I'm using 2.45:2.6:0.05 to specify the grid in the first dimension. I think this is because for the one-dimensional case numpy.nd_grid relies on numpy.arange. This is basically a good idea, but the more-dimensional case behaves different, like Andrew states. My problem is the following: >>>mgrid[0.1:0.2:1, 0.2:0.3:1] gives array([], shape=(2, 0, 0), dtype=float64) What I wanted to create was: array([[[ 0.1]], [[ 0.2]]]) which I finally got with >>>mgrid[0.1:1.2:1, 0.2:1.3:1] Since this behaviour is different from arange, I think it is not very intentional. But maybe there is a good reason for this behaviour? I am using numpy, version 1.0.1. Maybe the behaviour was already changed in more recent versions? Thank you for any comment Lars Friedrich -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-K?hler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfriedri at imtek.de From yves.frederix at gmail.com Mon Mar 5 05:10:07 2007 From: yves.frederix at gmail.com (Yves Frederix) Date: Mon, 5 Mar 2007 11:10:07 +0100 Subject: [Numpy-discussion] numpy.dot and ACML In-Reply-To: <45E67AE8.60708@fft.be> References: <20070219171120.GA5609@kotnet.org> <45E67AE8.60708@fft.be> Message-ID: <20070305101007.GA1122@kotnet.org> Hi, > I am also looking to verify the vendor-libs being used. > > What does numpy.__config__.show() tell you ? > In the case of the ACML compilation, I get: 0 u0050015 at lo-03-02 .../u0050015 $ python -c "import numpy; numpy.show_config()" atlas_threads_info: libraries = ['lapack', 'acml', 'python2.4'] library_dirs = ['/apps/prod/python/2.4.3/lib'] language = f77 blas_opt_info: libraries = ['acml', 'python2.4'] library_dirs = ['/apps/prod/python/2.4.3/lib'] define_macros = [('NO_ATLAS_INFO', 2)] language = c atlas_blas_threads_info: libraries = ['acml', 'python2.4'] library_dirs = ['/apps/prod/python/2.4.3/lib'] language = c lapack_opt_info: libraries = ['lapack', 'acml', 'python2.4'] library_dirs = ['/apps/prod/python/2.4.3/lib'] define_macros = [('NO_ATLAS_INFO', 2)] language = f77 lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE In the case of a more standard installation (blas/lapack/atlas) I guess this might give some extra information, but as I understand these are just the paths to the libraries that were used at compile time. If show_config() in your case does not show your lapack dir, I am afraid you are not using them as they were not found during compilation. In order to use ACML I had to use a custom site.cfg (which basically says what show_config() now reports), so I wanted some kind of extra verification that the installation is fine. My confusion actually comes from the fact that importing _dotblas in my installation results in undefined references. Yet the timings of numpy.dot seem ok, so 'some' optimized library is used, it is just not clear which one. 0 u0050015 at lo-03-02 .../core $ pwd /data/home/u0050015/inst/lib/python2.4/site-packages/numpy/core 0 u0050015 at lo-03-02 .../core $ python -c "import _dotblas" Traceback (most recent call last): File "", line 1, in ? ImportError: /apps/prod/python/2.4.3/lib/libacml.so: undefined symbol: vcosf Regards, YVES From giorgio.luciano at chimica.unige.it Mon Mar 5 05:50:42 2007 From: giorgio.luciano at chimica.unige.it (Giorgio Luciano) Date: Mon, 05 Mar 2007 11:50:42 +0100 Subject: [Numpy-discussion] new repository for free software and forum community started Message-ID: <45EBF602.6010902@chimica.unige.it> Dear All I'm happy and proud to announce that a repository for free chemometric software (in particular Python oriented) has started. at www.chemometrics.it Some time ago I've sent an email to the ICS asking if anybody knows the esistence of a python repository for chemometrics, I had a positive feedback from the community and with the help of other chemometricians I've decided to gather free software available for doing common chemometric tasks. In the site you will also find a forum where to discuss -> theoretical aspects of chemometrics -> job opportunities and cooperation with other chemometricians -> software request/news I've tried to link to existing sites that already give a huge contribute to chemometrics (KVL,Chemometrics.se just to cite two of them) but I will be very glad if you would like to contribute with your own links I hope this initiave could help to spread chemometrics and the use of free routine/software for chemometrics but that it also will become a place of discussion for anybody interested. Obviously we need your help and enthusiasm. If you would like to upload software links, routines etc, just register for free and do it, it's just so easy and you are very encouraged to do it. The initiave is just at its beginning so feel free to report any technical problem Any kind of feedback is appreciated Since it's a no profit personal initiave (I bought the domain and the rent the hosting) finally let me say that any mirroring or hosting will be greatly appreciated. I hope that this initiative will be useful ;) Best Regards Giorgio Luciano -- -======================- Dr Giorgio Luciano Ph.D. Di.C.T.F.A. Dipartimento di Chimica e Tecnologie Farmaceutiche e Alimentari Via Brigata Salerno (ponte) - 16147 Genova (GE) - Italy email luciano at dictfa.unige.it -======================- From nwagner at iam.uni-stuttgart.de Mon Mar 5 07:08:43 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 05 Mar 2007 13:08:43 +0100 Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms In-Reply-To: <200703050701.54955.steffen.loeck@gmx.de> References: <200703050701.54955.steffen.loeck@gmx.de> Message-ID: <45EC084B.6050003@iam.uni-stuttgart.de> Steffen Loeck wrote: > Fernando Perez wrote: > >> I recently got a report of a bug triggered only on 64-bit hardware, >> and on a machine (in case it's relevant) that runs python 2.5. This >> is with current numpy SVN which I just rebuilt a moment ago to >> triple-check: >> >> In [3]: a = numpy.array([[1.0,2],[3,4]]) >> >> In [4]: numpy.linalg.qr(a) >> ** On entry to DGEQRF parameter number 2 had an illegal value >> sage[~]> # dumped back at system prompt >> > > Has there been any progress in solving this problem? I get the same error > message and have no idea how to solve it. > > Regards, Steffen > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > Confirmed with Python 2.4.1 (#1, Oct 13 2006, 16:51:58) [GCC 4.0.2 20050901 (prerelease) (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.__version__ '1.0.2.dev3571' >>> a = numpy.array([[1.0,2],[3,4]]) >>> numpy.linalg.qr(a) ** On entry to DGEQRF parameter number 2 had an illegal value Nils From steve at shrogers.com Mon Mar 5 08:01:42 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Mon, 05 Mar 2007 06:01:42 -0700 Subject: [Numpy-discussion] Buffer PEP in NumPy SVN In-Reply-To: <45E5EA4D.1010303@ee.byu.edu> References: <45E5EA4D.1010303@ee.byu.edu> Message-ID: <45EC14B6.2090902@shrogers.com> Travis Oliphant wrote: > I just wanted to point people to the online version of the PEP. I'm > still looking for comments and suggestions. The current version is here: > > http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/pep_buffer.txt > Adding some use cases to the rationale might help convince the Python developers that this change is worth the development and maintenance burden. What about Python database adapters or data acquisition software? As the developers are very concerned with keeping their maintenance load manageable, a reference implementation would be persuasive, and they may want a commitment to help update affected standard library modules. # Steve From ogdude at googlemail.com Mon Mar 5 08:13:02 2007 From: ogdude at googlemail.com (David Koch) Date: Mon, 5 Mar 2007 14:13:02 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing Message-ID: Hello, I am trying to translate some Matlab code to NumPy. I started reading the NumPy book and, yeah it's a very long read :-/ One thing I am completely confused about are the concpets of "basic" vs. "advanced" indexing. Are there some good examples out there where for the same piece of code - Matlab and NumPy are compared? I feel looking at those maybe more helpful to get me started than an in-depth study of the book. I already read "Numpy for Matlab users" and at a glance it doesn't seem to contain equivalents to all the common indexing operations you'd do in Matlab. Some concrete problems/questions I have right now: - Am I correct in assuming that all arrays have to be initialized to their final number of elements in NumPy (using empty/zero for instance)? - Given a matrix R, is there an equvialent to the Matlab operation R(:,j) = [] (which removes column j and "shrinks" the matrix? - For (basic ?) indexing of an ndarray - is there any reason to prefer A[i,j] over A[i][j]? - Is it "pythonic" to initialize vectors to 2 dimensions so that vec.shape== (len(vec), 1) instead of vec.shape == (len(vec),)? Alright, Thanks for now, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Mon Mar 5 09:24:15 2007 From: sturla at molden.no (Sturla Molden) Date: Mon, 05 Mar 2007 15:24:15 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: Message-ID: <45EC280F.5070103@molden.no> On 3/5/2007 2:13 PM, David Koch wrote: > - Am I correct in assuming that all arrays have to be initialized to > their final number of elements in NumPy (using empty/zero for instance)? You can also create an array from a Python list, data in a file, another array or a memory mapping. In these cases you don't need to know the size of the array in advance. But once created you cannot append elements to a NumPy array as you can with a Python list. NumPy arrays never call realloc due to the complication from view arrays. > - Given a matrix R, is there an equvialent to the Matlab operation > R(:,j) = [] (which removes column j and "shrinks" the matrix? Matlab does not have view arrays and no regard for memory consumption or fragmentation, so it can make a copy behind the scenes. In NumPy you have to be more explicit about what is done. def remove_column(a,j): idx = arange(0,a.shape[1]) idx = idx[idx != j] return a[:,idx] > - For (basic ?) indexing of an ndarray - is there any reason to prefer > A[i,j] over A[i][j]? The latter involves two function calls and creation of an intermediate object (a view array). > - Is it "pythonic" to initialize vectors to 2 dimensions so that > vec.shape == (len(vec), 1) instead of vec.shape == (len(vec),)? I have no idea. S.M. From aisaac at american.edu Mon Mar 5 09:35:06 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 5 Mar 2007 09:35:06 -0500 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45EC280F.5070103@molden.no> References: <45EC280F.5070103@molden.no> Message-ID: On 3/5/2007 2:13 PM, David Koch wrote: > - Given a matrix R, is there an equvialent to the Matlab > operation R(:,j) = [] (which removes column j and > "shrinks" the matrix? >>> help(numpy.delete) Help on function delete in module numpy.lib.function_base: delete(arr, obj, axis=None) Return a new array with sub-arrays along an axis deleted. Return a new array with the sub-arrays (i.e. rows or columns) deleted along the given axis as specified by obj obj may be a slice_object (s_[3:5:2]) or an integer or an array of integers indicated which sub-arrays to remove. If axis is None, then ravel the array first. Example: >>> arr = [[3,4,5], [1,2,3], [6,7,8]] >>> delete(arr, 1, 1) array([[3,5], [1,3], [6,8]) >>> delete(arr, 1, 0) array([[3,4,5], [6,7,8]]) hth, Alan Isaac From chanley at stsci.edu Mon Mar 5 10:23:33 2007 From: chanley at stsci.edu (Christopher Hanley) Date: Mon, 05 Mar 2007 10:23:33 -0500 Subject: [Numpy-discussion] Scalar coercion In-Reply-To: <45E76059.2040605@ee.byu.edu> References: <45E7560A.2020206@ee.byu.edu> <45E7583C.4090007@gmail.com> <45E76059.2040605@ee.byu.edu> Message-ID: <45EC35F5.50604@stsci.edu> Hello Everyone, Another behavior we might consider changing for 1.0.2 that I believe is somewhat related in theme is the default type used in computations like the mean() method. This is best illustrated with the following example: sparty> python Python 2.5 (r25:51908, Sep 21 2006, 13:33:15) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-56)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as n >>> n.__version__ '1.0.2.dev3568' >>> a = n.ones((1000,1000),dtype=n.float32)*132.00005 >>> print a [[ 132.00004578 132.00004578 132.00004578 ..., 132.00004578 132.00004578 132.00004578] [ 132.00004578 132.00004578 132.00004578 ..., 132.00004578 132.00004578 132.00004578] [ 132.00004578 132.00004578 132.00004578 ..., 132.00004578 132.00004578 132.00004578] ..., [ 132.00004578 132.00004578 132.00004578 ..., 132.00004578 132.00004578 132.00004578] [ 132.00004578 132.00004578 132.00004578 ..., 132.00004578 132.00004578 132.00004578] [ 132.00004578 132.00004578 132.00004578 ..., 132.00004578 132.00004578 132.00004578]] >>> a.min() 132.000045776 >>> a.max() 132.000045776 >>> a.mean() 133.96639999999999 >>> Having the mean be greater than the max is a tad odd. The calculation of the mean is occurring with a single precision accumulator. I do understand that I can force a double precision calculation with the following command: >>> a.mean(dtype=n.float64) 132.00004577636719 >>> I realize that one reason for not doing all calculations as double precision is performance. However, my users would rather have the correct answer by default than quickly arriving at the wrong one. In my opinion we should swap the default behavior. All calculations should be done in double precision. If you need the performance you can then go back and start setting data types. Not having to worry about overflow would also be consistent with numarray's behavior. Thank you for considering my opinion, Chris From charlesr.harris at gmail.com Mon Mar 5 12:04:31 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Mar 2007 10:04:31 -0700 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: Message-ID: On 3/5/07, David Koch wrote: > > Hello, > > I am trying to translate some Matlab code to NumPy. I started reading the > NumPy book and, yeah it's a very long read :-/ One thing I am completely > confused about are the concpets of "basic" vs. "advanced" indexing. Are > there some good examples out there where for the same piece of code - Matlab > and NumPy are compared? I feel looking at those maybe more helpful to get me > started than an in-depth study of the book. I already read "Numpy for Matlab > users" and at a glance it doesn't seem to contain equivalents to all the > common indexing operations you'd do in Matlab. > > Some concrete problems/questions I have right now: > > - Am I correct in assuming that all arrays have to be initialized to their > final number of elements in NumPy (using empty/zero for instance)? Usually arrays are created from the data with their final dimensions set as part of the creation. You don't need to define an array and then fill in the values and it is best to avoid this where possible. Subarrays are best treated as views that require not copying of data, something that is missing from Matlab. - Given a matrix R, is there an equvialent to the Matlab operation R(:,j) = > [] (which removes column j and "shrinks" the matrix? Look at delete. It is best to avoid such constructions when possible because they involve a data copy. - For (basic ?) indexing of an ndarray - is there any reason to prefer > A[i,j] over A[i][j]? > - Is it "pythonic" to initialize vectors to 2 dimensions so that vec.shape== (len(vec), 1) instead of > vec.shape == (len(vec),)? There are two schools here, those who use matrices so that everything is 2d, and those who use arrays. One dimensional arrays will work fine for such things as matrix multiplication, so there is really no need for the second dimension. In any case, a faster way to implement your example is vec.reshape(-1,1) where the -1 fills in the dimension to use all of the array. matrix(vec) will return an 1xn matrix Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Mar 5 13:04:03 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 05 Mar 2007 10:04:03 -0800 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: Message-ID: <45EC5B93.2030108@noaa.gov> David Koch wrote: > - Is it "pythonic" to initialize vectors to 2 dimensions so that > vec.shape == (len(vec), 1) instead of vec.shape == (len(vec),)? It depends what it means -- and this is not a question of Pythonic -- maybe Numpythonic? Numpy is an n-d array package -- NOT a matrix package (if you really want matrices, see the numpy matrix package). thus, a Vector can be represented as a (n,) shaped array a Matrix can be represented by a (n,m) shaped array etc... If what you want is a matrix that happens to have one column, you want a (n,1) array, if you want one row, you want a (1,n) array. In linear algebra terms, these are row and column vectors. In other math, a vector is (n,) in shape. Other than linear algebra, a reason to use 2-d "vectors" is for array broadcasting: >>> import numpy as N >>> x = N.arange(10).reshape(1,-1) # a "row" vector >>> y = N.arange(5).reshape(-1,1) # a "column" vector >>> x.shape (1, 10) >>> y.shape (5, 1) >>> z = x * y >>> z array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18], [ 0, 3, 6, 9, 12, 15, 18, 21, 24, 27], [ 0, 4, 8, 12, 16, 20, 24, 28, 32, 36]]) >>> ## note that you didn't' really need to reshape x, as a (n,) array is interpreted as a row vector for broadcasting purposes, but I like to do it for clarities sake. In MATLAB, there is no such thing as a vector, so you only have the last two options -- not the first. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ogdude at googlemail.com Mon Mar 5 13:13:35 2007 From: ogdude at googlemail.com (David Koch) Date: Mon, 5 Mar 2007 19:13:35 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45EC5B93.2030108@noaa.gov> References: <45EC5B93.2030108@noaa.gov> Message-ID: Thank you everybody for your replies. Completely off-topic: I just read your sig Christoper: Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R Do you work with tsunami early-warning systems? I once had to give a presentation at Uni about the type of buoys that are deployed to detect tsunamis. Interesting stuff. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sameerslists at gmail.com Mon Mar 5 19:26:21 2007 From: sameerslists at gmail.com (Sameer DCosta) Date: Mon, 5 Mar 2007 18:26:21 -0600 Subject: [Numpy-discussion] Empty record arrays Message-ID: <8fb8cc060703051626u394322f6j1acb23c42a1e1e8a@mail.gmail.com> Hi, I am trying to create record arrays using the numpy.rec.fromrecords. This function works just fine, however I am having a little trouble using it to create empty record arrays. If the array I start off with is empty and I have provided a dtype (or the names and the formats), then I expect rec.fromrecords to return an empty record array. However I just get an error. I am using numpy version 1.0, but a quick glance at numpy/core/records.py in the svn repository seems to indicate that I will have this problem even with the svn version. Any thoughts? Thanks in advance. Sameer # ********* code snippet *********** from numpy import * recs = [( 'Bill' , 31, 260.0), ( 'Fred' , 15, 145.0)] dt = [("name", "S30"), ("age", "i2"), ("weight", "f4")] # Using populated array r = rec.fromrecords(recs, dtype=dt) print r.name try: # Using empty array gives an error r = rec.fromrecords([], dtype=dt) print r except IndexError: print "Index Error using empty array" # workaround r = array([], dtype=dt).view(recarray) print r.name From Chris.Barker at noaa.gov Mon Mar 5 19:54:12 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 05 Mar 2007 16:54:12 -0800 Subject: [Numpy-discussion] Empty record arrays In-Reply-To: <8fb8cc060703051626u394322f6j1acb23c42a1e1e8a@mail.gmail.com> References: <8fb8cc060703051626u394322f6j1acb23c42a1e1e8a@mail.gmail.com> Message-ID: <45ECBBB4.9060302@noaa.gov> empty() works too, but why would you want an empty array? >>> import numpy as N >>> dt = [("name", "S30"), ("age", "i2"), ("weight", "f4")] >>> a = N.empty((0,), dtype=dt) >>> a array([], dtype=[('name', '|S30'), ('age', '>i2'), ('weight', '>f4')]) -Chris Sameer DCosta wrote: > Hi, > > I am trying to create record arrays using the numpy.rec.fromrecords. > This function works just fine, however I am having a little trouble > using it to create empty record arrays. If the array I start off with > is empty and I have provided a dtype (or the names and the formats), > then I expect rec.fromrecords to return an empty record array. However > I just get an error. > > I am using numpy version 1.0, but a quick glance at > numpy/core/records.py in the svn repository seems to indicate that I > will have this problem even with the svn version. Any thoughts? Thanks > in advance. > > Sameer > > > # ********* code snippet *********** > from numpy import * > recs = [( 'Bill' , 31, 260.0), ( 'Fred' , 15, 145.0)] > dt = [("name", "S30"), ("age", "i2"), ("weight", "f4")] > > # Using populated array > r = rec.fromrecords(recs, dtype=dt) > print r.name > > try: > # Using empty array gives an error > r = rec.fromrecords([], dtype=dt) > print r > except IndexError: > print "Index Error using empty array" > > # workaround > r = array([], dtype=dt).view(recarray) > print r.name > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Mon Mar 5 19:55:43 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 05 Mar 2007 18:55:43 -0600 Subject: [Numpy-discussion] Empty record arrays In-Reply-To: <45ECBBB4.9060302@noaa.gov> References: <8fb8cc060703051626u394322f6j1acb23c42a1e1e8a@mail.gmail.com> <45ECBBB4.9060302@noaa.gov> Message-ID: <45ECBC0F.3010207@gmail.com> Christopher Barker wrote: > empty() works too, but why would you want an empty array? Consistently handling edge-case inputs to library functions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sameerslists at gmail.com Mon Mar 5 20:44:00 2007 From: sameerslists at gmail.com (Sameer DCosta) Date: Mon, 5 Mar 2007 19:44:00 -0600 Subject: [Numpy-discussion] Empty record arrays In-Reply-To: <45ECBBB4.9060302@noaa.gov> References: <8fb8cc060703051626u394322f6j1acb23c42a1e1e8a@mail.gmail.com> <45ECBBB4.9060302@noaa.gov> Message-ID: <8fb8cc060703051744w51f30fb0i3616e87821c0bb48@mail.gmail.com> On 3/5/07, Christopher Barker wrote: > empty() works too, but why would you want an empty array? I wanted to use rec.fromrecords so that the interface is consistent. For example, suppose I have a list comprehension that generates the variable "recs". Sometimes the list comprehension returns an empty list. It seems natural to except rec.fromrecords to generate an empty record array in that case. From david at ar.media.kyoto-u.ac.jp Mon Mar 5 22:38:03 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Tue, 06 Mar 2007 12:38:03 +0900 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: Message-ID: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> David Koch wrote: > Hello, > > I am trying to translate some Matlab code to NumPy. I started reading > the NumPy book and, yeah it's a very long read :-/ One thing I am > completely confused about are the concpets of "basic" vs. "advanced" > indexing. Are there some good examples out there where for the same > piece of code - Matlab and NumPy are compared? I feel looking at those > maybe more helpful to get me started than an in-depth study of the > book. I already read "Numpy for Matlab users" and at a glance it > doesn't seem to contain equivalents to all the common indexing > operations you'd do in Matlab. I have been there too ("long time" user of matlab, numpy convert for one year now), and two links which help me a lot to have a direct mapping between matlab and python at the beginning are: http://www.scipy.org/NumPy_for_Matlab_Users http://www.37mm.no/matlab-python-xref.html Generally, except if you are doing a lot of linear algebra, and have to write a lot things like dot(a, dot(b, dot(c, d))), my experience is that you should avoid thinking the "matlab" way, and use nd array (that is vector is rank 1, matrices are rank 2, etc...). It means that sometimes, you will have to be more specific about some things, but make some other things easier. For me (signal processing), once I switched to numpy way of thinking, I found it easier to use nd arrays. David From hayes.tyler at gmail.com Mon Mar 5 23:13:21 2007 From: hayes.tyler at gmail.com (Tyler Hayes) Date: Mon, 5 Mar 2007 23:13:21 -0500 Subject: [Numpy-discussion] F2PY clarification of "so" file Message-ID: <92bcafc50703052013r7e561908p96d60a6c01d87367@mail.gmail.com> Hello All: Well, I recently managed to get some small FEM code to work using NumPy and some .so modules I created from Fortran90 routines via f2py. Anyways, my Prof was impressed with f2py but had a simple question that I'm afriad I didn't precisely know the answer to. The question is this: Is the .so file a Fortran90 executed code (or C) that runs at their speeds? Or is it a Python code that is translated from Fortran90? The concern my Prof had was that for a simple 12 -- 50 element model speed isn't a concern. However, for an FEM model with tens of thousands of nodes, speed is crucial. Any clarification is greatly appreciated. Cheers, t. -- Tyler Joseph Hayes GPG Key ID# 0x3AE05130 From robert.kern at gmail.com Mon Mar 5 23:22:22 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 05 Mar 2007 22:22:22 -0600 Subject: [Numpy-discussion] F2PY clarification of "so" file In-Reply-To: <92bcafc50703052013r7e561908p96d60a6c01d87367@mail.gmail.com> References: <92bcafc50703052013r7e561908p96d60a6c01d87367@mail.gmail.com> Message-ID: <45ECEC7E.8050900@gmail.com> Tyler Hayes wrote: > Hello All: > > Well, I recently managed to get some small FEM code to work using > NumPy and some .so modules I created from Fortran90 routines via f2py. > > Anyways, my Prof was impressed with f2py but had a simple question > that I'm afriad I didn't precisely know the answer to. > > The question is this: > > Is the .so file a Fortran90 executed code (or C) that runs at their > speeds? Yes. > Or is it a Python code that is translated from Fortran90? No. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From hayes.tyler at gmail.com Tue Mar 6 09:08:39 2007 From: hayes.tyler at gmail.com (Tyler J Hayes) Date: Tue, 6 Mar 2007 14:08:39 +0000 (UTC) Subject: [Numpy-discussion] F2PY clarification of "so" file References: <92bcafc50703052013r7e561908p96d60a6c01d87367@mail.gmail.com> <45ECEC7E.8050900@gmail.com> Message-ID: Robert Kern gmail.com> writes: > Yes. > > No. > That was easy.... Thanks Robert, t. From mpmusu at cc.usu.edu Tue Mar 6 10:14:53 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Tue, 06 Mar 2007 08:14:53 -0700 Subject: [Numpy-discussion] when and where to use numpy arrays vs nested lists In-Reply-To: References: <45E6F964.10509@cc.usu.edu> <45E71AB4.9000103@gmail.com> <45E7239C.60509@cc.usu.edu> <1172776964.2540.5.camel@localhost.localdomain> <45E72DCE.9020909@cc.usu.edu> <1172778815.2540.9.camel@localhost.localdomain> <45E74A56.5080007@noaa.gov> <45E751DD.8010002@cc.usu.edu> <45E7519B.2070002@gmail.com> <45E75EC2.7090905@cc.usu.edu> Message-ID: <45ED856D.5090504@cc.usu.edu> Charles R Harris wrote: > I won't comment on the code itself. Tell us what you want to do and I > bet we can speed it up. > Here's a bit of sample code. It's actually very useful for my purposes. Are there good ways to express these ideas in numpy as opposed to using all of the nested loops that I currently employ? ######### import random import numpy uniformRND = random.random #faster than numpy's normalRND = numpy.random.normal #faster than Python's def pick(a,b,d1,d2,sd): # this function picks two random values with mean = 0 and # sd = sd. Random values are added (in one form or another) # to a and b. All other code ensures that the random values # do not generate "index out of range" errors after new1 and # new2 are returned (needed when sd is set to large values) new1 = 2*d1 + 1 while new1 < -d1 or new1 > 2*d1: new1=int(round(normalRND(0,sd))) new2 = 2*d2 + 1 while new2 < -d2 or new2 > 2*d2: new2=int(round(normalRND(0,sd))) tmp1 = a + new1 if 0 <= tmp1 < d1: new1 = tmp1 elif tmp1 < 0: new1 = -(a + new1) else: new1 = a - new1 tmp2 = b + new2 if 0 <= tmp2 < d2: new2 = tmp2 elif tmp2 < 0: new2 = -(b + new2) else: new2 = b - new2 return new1, new2 def main(): changerate = 0.25 sd = 2 d1=50 d2=100 d3=10 ncycles = 10 array1 = numpy.zeros((d1, d2, d3, 2), int) array2 = numpy.zeros((d1, d2, d3, 2), int) for zz in xrange(ncycles): for aa in xrange(d1): for bb in xrange(d2): item1a, item1b = pick(aa,bb,d1,d2,sd) item2a, item2b = pick(aa,bb,d1,d2,sd) for cc in xrange(d3): element1 = array1[item1a,item1b,cc, int(uniformRND() < 0.5)] element2 = array1[item2a,item2b,cc, int(uniformRND() < 0.5)] if uniformRND() < changerate: element1 = int(uniformRND()*100) if uniformRND() < changerate: element2 = int(uniformRND()*100) array2[aa,bb,cc,0]=element1 array2[aa,bb,cc,1]=element2 array1=array2.copy() if __name__ == '__main__': main() From haase at msg.ucsf.edu Tue Mar 6 23:49:00 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 6 Mar 2007 20:49:00 -0800 Subject: [Numpy-discussion] how to round to int (I mean 5.2 to 5 not 5.0) Message-ID: Hi, why does numpy.round(a) return a float ? I need something that I can use as indices for another array. Do I have to (implicitly) create a temporary array and use: N.round(a).astype(N.int) ? Or is there a simple, clean and easy way to just round [1.1 4.8] into [1 5] Thanks, Sebastian From robert.kern at gmail.com Tue Mar 6 23:53:54 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 06 Mar 2007 22:53:54 -0600 Subject: [Numpy-discussion] how to round to int (I mean 5.2 to 5 not 5.0) In-Reply-To: References: Message-ID: <45EE4562.5030206@gmail.com> Sebastian Haase wrote: > Hi, > why does > numpy.round(a) > return a float ? Partly because that's what the underlying C standard library function does and what the Python function round() does. The reason they do that is because the range of integers supported by the double precision floating point type is larger than the range of integers supported by C ints. In [17]: numpy.round(1e16) Out[17]: 1e+16 In [18]: numpy.round(1e16).astype(int) Out[18]: -2147483648 > I need something that I can use as indices for another array. Do I > have to (implicitly) create a temporary array and use: > N.round(a).astype(N.int) ? Yup. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aisaac at american.edu Wed Mar 7 00:02:59 2007 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 7 Mar 2007 00:02:59 -0500 Subject: [Numpy-discussion] how to round to int (I mean 5.2 to 5 not 5.0) In-Reply-To: References: Message-ID: On Tue, 6 Mar 2007, Sebastian Haase apparently wrote: > why does > numpy.round(a) > return a float ? Because it works with a copy of a. >>> help(N.round) Help on function round_ in module numpy.core.fromnumeric: round_(a, decimals=0, out=None) Returns reference to result. Copies a and rounds to 'decimals' places. Keyword arguments: decimals -- number of decimal places to round to (default 0). out -- existing array to use for output (default copy of a). Returns: Reference to out, where None specifies a copy of the original array a. >>> a = N.random.random((10,))*100 >>> a array([ 45.01971148, 8.32961759, 39.75272544, 79.76986159, 23.66331127, 24.25584246, 38.17354106, 16.57977389, 50.63676986, 83.15113716]) >>> b = N.empty((10,),dtype='int') >>> N.round(a,out=b) array([45, 8, 40, 80, 24, 24, 38, 17, 51, 83]) >>> b array([45, 8, 40, 80, 24, 24, 38, 17, 51, 83]) Cheers, Alan Isaac From haase at msg.ucsf.edu Wed Mar 7 00:26:34 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue, 6 Mar 2007 21:26:34 -0800 Subject: [Numpy-discussion] how to round to int (I mean 5.2 to 5 not 5.0) In-Reply-To: References: Message-ID: On 3/6/07, Alan G Isaac wrote: > On Tue, 6 Mar 2007, Sebastian Haase apparently wrote: > > why does > > numpy.round(a) > > return a float ? > > Because it works with a copy of a. > > >>> help(N.round) > Help on function round_ in module numpy.core.fromnumeric: > round_(a, decimals=0, out=None) > Returns reference to result. Copies a and rounds to 'decimals' places. > > Keyword arguments: > decimals -- number of decimal places to round to (default 0). > out -- existing array to use for output (default copy of a). > > Returns: > Reference to out, where None specifies a copy of the original array a. > >>> a = N.random.random((10,))*100 > >>> a > array([ 45.01971148, 8.32961759, 39.75272544, 79.76986159, > 23.66331127, 24.25584246, 38.17354106, 16.57977389, > 50.63676986, 83.15113716]) > >>> b = N.empty((10,),dtype='int') > >>> N.round(a,out=b) > array([45, 8, 40, 80, 24, 24, 38, 17, 51, 83]) > >>> b > array([45, 8, 40, 80, 24, 24, 38, 17, 51, 83]) > > > Cheers, > Alan Isaac > Would it be a useful / possible enhancement to add an additional dtype argument to round !? So that if out is not given, but dtype is given (e.g. as N.int) the "copy-array" is created with that dtype ? Just a thought -- otherwise I might add it to my personal collection of "useful" functions. Thanks, Sebastian Haase From lars.bittrich at googlemail.com Wed Mar 7 04:57:10 2007 From: lars.bittrich at googlemail.com (Lars Bittrich) Date: Wed, 7 Mar 2007 10:57:10 +0100 Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms In-Reply-To: <200703050701.54955.steffen.loeck@gmx.de> References: <200703050701.54955.steffen.loeck@gmx.de> Message-ID: <200703071057.10495.lars.bittrich@googlemail.com> On Monday 05 March 2007 08:01, Steffen Loeck wrote: > Has there been any progress in solving this problem? I get the same error > message and have no idea how to solve it. I do not understand those code parts very well but I think the values passed to the lapack routines must be integer and not long integer on 64bit architecture. A few tests with the attached patch worked well. Best Regards, Lars -------------- next part -------------- A non-text attachment was scrubbed... Name: lapack_litemodule.patch Type: text/x-diff Size: 3473 bytes Desc: not available URL: From ellisonbg.net at gmail.com Wed Mar 7 14:16:30 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 7 Mar 2007 12:16:30 -0700 Subject: [Numpy-discussion] Bug in numpy.distutils Message-ID: <6ce0ac130703071116w788af4d0v5cdb408014c9f9e7@mail.gmail.com> Hi, I am building numpy on a bunch of different systems right now and for the most part I am always successful. Today though, I found a wierd problem. Here is the traceback from doing python setup.py (below): This is on an intel 10.4 box with no fortran compiler installed. Incidently, after googling around, I did find that someone else has seen this before - looks like they didn't report it. http://sage.math.washington.edu:9002/sage_trac/query?status=new&status=assigned&status=reopened&format=rss&order=id I have a hack that covers this up, but I figured it would be better to really understand what is going on. Any ideas? Brian running install running build running config_fc running build_src building py_modules sources creating build creating build/src.macosx-10.3-i386-2.5 creating build/src.macosx-10.3-i386-2.5/numpy creating build/src.macosx-10.3-i386-2.5/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src.macosx-10.3-i386-2.5/numpy/core Generating build/src.macosx-10.3-i386-2.5/numpy/core/config.h customize NAGFCompiler 'linker_exe' customize AbsoftFCompiler 'linker_exe' customize IbmFCompiler 'linker_exe' Could not locate executable g77 Could not locate executable f77 Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize Gnu95FCompiler customize G95FCompiler 'linker_exe' customize GnuFCompiler customize Gnu95FCompiler customize NAGFCompiler Traceback (most recent call last): File "setup.py", line 89, in setup_package() File "setup.py", line 82, in setup_package configuration=configuration ) File "/Users/foo/build/numpy-20070205/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/foo/build/numpy-20070205/numpy/distutils/command/install.py", line 16, in run r = old_install.run(self) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/install.py", line 506, in run self.run_command('build') File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/foo/build/numpy-20070205/numpy/distutils/command/build_src.py", line 87, in run self.build_sources() File "/Users/foo/build/numpy-20070205/numpy/distutils/command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/Users/foo/build/numpy-20070205/numpy/distutils/command/build_src.py", line 212, in build_extension_sources sources = self.generate_sources(sources, ext) File "/Users/foo/build/numpy-20070205/numpy/distutils/command/build_src.py", line 270, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 46, in generate_config_h library_dirs = default_lib_dirs) File "/Users/foo/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/config.py", line 278, in try_run self._check_compiler() File "/Users/foo/build/numpy-20070205/numpy/distutils/command/config.py", line 39, in _check_compiler self.fcompiler.customize(self.distribution) File "/Users/foo/build/numpy-20070205/numpy/distutils/fcompiler/__init__.py", line 345, in customize linker_exe_flags = self.__get_flags(self.get_flags_linker_exe,'LDFLAGS') File "/Users/foo/build/numpy-20070205/numpy/distutils/fcompiler/__init__.py", line 527, in __get_flags var = command() File "/Users/foo/build/numpy-20070205/numpy/distutils/fcompiler/__init__.py", line 204, in get_flags_linker_exe if self.executables['linker_exe']: KeyError: 'linker_exe' From oliphant at ee.byu.edu Wed Mar 7 14:12:47 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 07 Mar 2007 12:12:47 -0700 Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms In-Reply-To: <200703050701.54955.steffen.loeck@gmx.de> References: <200703050701.54955.steffen.loeck@gmx.de> Message-ID: <45EF0EAF.3070502@ee.byu.edu> Steffen Loeck wrote: >Fernando Perez wrote: > > >>I recently got a report of a bug triggered only on 64-bit hardware, >>and on a machine (in case it's relevant) that runs python 2.5. This >>is with current numpy SVN which I just rebuilt a moment ago to >>triple-check: >> >>In [3]: a = numpy.array([[1.0,2],[3,4]]) >> >>In [4]: numpy.linalg.qr(a) >> ** On entry to DGEQRF parameter number 2 had an illegal value >>sage[~]> # dumped back at system prompt >> >> > >Has there been any progress in solving this problem? I get the same error >message and have no idea how to solve it. > > I just added a patch from Lars Bittrich that probably solves it. Try it now. -Travis From dmahler at gmail.com Wed Mar 7 16:34:20 2007 From: dmahler at gmail.com (Daniel Mahler) Date: Wed, 7 Mar 2007 21:34:20 +0000 Subject: [Numpy-discussion] in place random generation Message-ID: Is there an efficient way to fill an existing array with random numbers without allocating a new array? thanks Daniel From robert.kern at gmail.com Wed Mar 7 16:44:31 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 07 Mar 2007 15:44:31 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: Message-ID: <45EF323F.7060208@gmail.com> Daniel Mahler wrote: > Is there an efficient way to fill an existing array with random > numbers without allocating a new array? No, sorry. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at ieee.org Wed Mar 7 16:51:23 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 7 Mar 2007 14:51:23 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45EF323F.7060208@gmail.com> References: <45EF323F.7060208@gmail.com> Message-ID: On 3/7/07, Robert Kern wrote: > > Daniel Mahler wrote: > > Is there an efficient way to fill an existing array with random > > numbers without allocating a new array? > > No, sorry. There is however an only moderately inefficient way if you are primarily concerned with keeping your total memory usage down for some reason. In that case, you can fill your array in chunks; for example getting 1000 random numbers at a time from random.random and successively copying them into your array. It's probably not worth the trouble unless you have a really big array though. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkg at cs.nyu.edu Wed Mar 7 16:54:01 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Wed, 07 Mar 2007 16:54:01 -0500 Subject: [Numpy-discussion] overlapping rows Message-ID: <45EF3479.3080606@cs.nyu.edu> I would like to twiddle with the strides of a matrix such that the rows overlap each other. I've gotten this far: In [1]: import numpy as N In [2]: mat = N.arange(12).reshape(3,4) In [3]: mat Out[3]: array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) In [4]: mat.strides Out[4]: (16, 4) In [5]: omat = mat[...] In [6]: omat.strides = (8,4) In [7]: omat Out[7]: array([[0, 1, 2, 3], [2, 3, 4, 5], [4, 5, 6, 7]]) So far so good, but I'd like to have "omat"'s first dimension grow to size 6 so that all of the data is used. In other words, I'd like for omat to look like: array([[0, 1, 2, 3], [2, 3, 4, 5], [4, 5, 6, 7], [6, 7, 8, 9], [8, 9, 10, 11]]) However, when I try to resize the first dimension to 6 I get the following error: In [8]: omat.shape = (6,4) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/mkg/ ValueError: total size of new array must be unchanged Is there any workaround to this? It seems like something I should be able to do, since I'm only growing omat into memory that's owned by omat.base. Any help would be much appreciated, -- Matt From tim.hochberg at ieee.org Wed Mar 7 17:19:52 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 7 Mar 2007 15:19:52 -0700 Subject: [Numpy-discussion] overlapping rows In-Reply-To: <45EF3479.3080606@cs.nyu.edu> References: <45EF3479.3080606@cs.nyu.edu> Message-ID: On 3/7/07, Matthew Koichi Grimes wrote: > > I would like to twiddle with the strides of a matrix such that the rows > overlap each other. I've gotten this far: > > In [1]: import numpy as N > > In [2]: mat = N.arange(12).reshape(3,4) > > In [3]: mat > Out[3]: > array([[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]]) > > In [4]: mat.strides > Out[4]: (16, 4) > > In [5]: omat = mat[...] > > In [6]: omat.strides = (8,4) > > In [7]: omat > Out[7]: > array([[0, 1, 2, 3], > [2, 3, 4, 5], > [4, 5, 6, 7]]) > > So far so good, but I'd like to have "omat"'s first dimension grow to > size 6 so that all of the data is used. In other words, I'd like for > omat to look like: > > array([[0, 1, 2, 3], > [2, 3, 4, 5], > [4, 5, 6, 7], > [6, 7, 8, 9], > [8, 9, 10, 11]]) > > However, when I try to resize the first dimension to 6 I get the > following error: > > In [8]: omat.shape = (6,4) > > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most > recent call last) > > /home/mkg/ > > ValueError: total size of new array must be unchanged > > > Is there any workaround to this? It seems like something I should be > able to do, since I'm only growing omat into memory that's owned > by omat.base. > > Any help would be much appreciated, I think you're on thin ice here, but: >>> a = np.arange(12) >>> b = np.ndarray([6,4], int, a, strides=[8,4]) >>> b array([[ 0, 1, 2, 3], [ 2, 3, 4, 5], [ 4, 5, 6, 7], [ 6, 7, 8, 9], [ 8, 9, 10, 11], [ 10, 11, 458762, 19464224]]) (See the docs for ndarray). Hmm. Perhaps you meant shape [5,4[? >>> c = np.ndarray([5,4], int, a, strides=[8,4]) >>> c array([[ 0, 1, 2, 3], [ 2, 3, 4, 5], [ 4, 5, 6, 7], [ 6, 7, 8, 9], [ 8, 9, 10, 11]]) That seems more plausible. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmahler at gmail.com Wed Mar 7 19:33:20 2007 From: dmahler at gmail.com (Daniel Mahler) Date: Thu, 8 Mar 2007 00:33:20 +0000 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> Message-ID: My problem is not space, but time. I am creating a small array over and over, and this is turning out to be a bottleneck. My experiments suggest that problem is the allocation, not the random number generation. Allocating all the arrays as one n+1 dim and grabbing rows from it is faster than allocating the small arrays individually. I am iterating too many times to allocate everything at once though. I can just do a nested loop where create manageably large arrays in the outer and grab the rows in the inner, but I wanted something cleaner. Besides, I thought avoiding allocation altogether would be even faster. cheers Daniel On 3/7/07, Timothy Hochberg wrote: > On 3/7/07, Robert Kern wrote: > > > > Daniel Mahler wrote: > > > Is there an efficient way to fill an existing array with random > > > numbers without allocating a new array? > > > > No, sorry. > > > There is however an only moderately inefficient way if you are primarily > concerned with keeping your total memory usage down for some reason. In that > case, you can fill your array in chunks; for example getting 1000 random > numbers at a time from random.random and successively copying them into your > array. It's probably not worth the trouble unless you have a really big > array though. > > > -- > > //=][=\\ > > tim.hochberg at ieee.org > From peridot.faceted at gmail.com Wed Mar 7 19:54:18 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 7 Mar 2007 19:54:18 -0500 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> Message-ID: On 07/03/07, Daniel Mahler wrote: > My problem is not space, but time. > I am creating a small array over and over, > and this is turning out to be a bottleneck. > My experiments suggest that problem is the allocation, > not the random number generation. > Allocating all the arrays as one n+1 dim and grabbing rows from it > is faster than allocating the small arrays individually. > I am iterating too many times to allocate everything at once though. > I can just do a nested loop > where create manageably large arrays in the outer > and grab the rows in the inner, > but I wanted something cleaner. > Besides, I thought avoiding allocation altogether would be even faster. This doesn't make much practical difference, but allocation is definitely not the only difference between the two situations you describe - if the random number generator function does some significant setup or teardown, you are changing that as well. In fact, modern malloc()s are extremely fast on average (vastly faster than filling any significant amount of memory), so I would hesitate to blame malloc(). Creating numpy arrays, though, could easily be taking the extra time, though depending on how you extract the rows you may be creating just as many array objects. None of this answers your real question, of how to speed up your program. I will observe, though, that there are many places in numpy where a compromise between a simple for loop over all elements and a single operation on giant arrays is faster than either extreme; this is what numexpr is for, for example. Anne M. Archibald From matthew.brett at gmail.com Wed Mar 7 20:07:38 2007 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 7 Mar 2007 17:07:38 -0800 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> Message-ID: <1e2af89e0703071707x615aa787g9883340d2a53d7cd@mail.gmail.com> > > My problem is not space, but time. > > I am creating a small array over and over, > > and this is turning out to be a bottleneck. How about making one large random number array and taking small views? Matthew From charlesr.harris at gmail.com Wed Mar 7 22:54:16 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 7 Mar 2007 20:54:16 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> Message-ID: On 3/7/07, Daniel Mahler wrote: > > My problem is not space, but time. > I am creating a small array over and over, > and this is turning out to be a bottleneck. > My experiments suggest that problem is the allocation, > not the random number generation. > Allocating all the arrays as one n+1 dim and grabbing rows from it > is faster than allocating the small arrays individually. > I am iterating too many times to allocate everything at once though. > I can just do a nested loop > where create manageably large arrays in the outer > and grab the rows in the inner, > but I wanted something cleaner. > Besides, I thought avoiding allocation altogether would be even faster. The slow down is probably related to this from a previous thread: In [46]: def test1() : ....: x = normal(0,1,1000) ....: In [47]: def test2() : ....: for i in range(1000) : ....: x = normal(0,1) In [50]: t = timeit.Timer('test1()', "from __main__ import test1, normal") In [51]: t.timeit(100) Out[51]: 0.022681951522827148 In [52]: t = timeit.Timer('test2()', "from __main__ import test2, normal") In [53]: t.timeit(100) Out[53]: 4.3481810092926025 Robert thought this might relate to Travis' changes adding broadcasting to the random number generator. It does seem certain that generating small arrays of random numbers has a very high overhead. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From steffen.loeck at gmx.de Thu Mar 8 02:46:01 2007 From: steffen.loeck at gmx.de (Steffen Loeck) Date: Thu, 8 Mar 2007 07:46:01 +0000 Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms In-Reply-To: <200703071057.10495.lars.bittrich@googlemail.com> References: <200703050701.54955.steffen.loeck@gmx.de> <200703071057.10495.lars.bittrich@googlemail.com> Message-ID: <200703080746.01679.steffen.loeck@gmx.de> On Wednesday 07 March 2007 09:57, Lars Bittrich wrote: > I do not understand those code parts very well but I think the values > passed to the lapack routines must be integer and not long integer on 64bit > architecture. A few tests with the attached patch worked well. It works fine for me now. Thanks a lot, Steffen From cookedm at physics.mcmaster.ca Thu Mar 8 03:27:39 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 8 Mar 2007 03:27:39 -0500 Subject: [Numpy-discussion] numpy.linalg.qr bug on 64-bit platforms In-Reply-To: <200703071057.10495.lars.bittrich@googlemail.com> References: <200703050701.54955.steffen.loeck@gmx.de> <200703071057.10495.lars.bittrich@googlemail.com> Message-ID: <24DCCF8F-19D3-4C32-85B0-4463D93A632D@physics.mcmaster.ca> On Mar 7, 2007, at 04:57 , Lars Bittrich wrote: > On Monday 05 March 2007 08:01, Steffen Loeck wrote: >> Has there been any progress in solving this problem? I get the >> same error >> message and have no idea how to solve it. > > I do not understand those code parts very well but I think the > values passed > to the lapack routines must be integer and not long integer on 64bit > architecture. A few tests with the attached patch worked well. Yeah, this problem is kind of ugly; we should have a test for the integer size in Fortran compilers. However, I don't know how to do that. We've been pretty lucky so far: INTEGER seems to be a C int on everything. I know this is true for GNU Fortran (g77, gfortran) on Linux, on both 32-bit and 64-bit platforms. But I don't think there's any guarantee that it would be true in general. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part URL: From dmahler at gmail.com Thu Mar 8 10:24:16 2007 From: dmahler at gmail.com (Daniel Mahler) Date: Thu, 8 Mar 2007 15:24:16 +0000 Subject: [Numpy-discussion] in place random generation In-Reply-To: <1e2af89e0703071707x615aa787g9883340d2a53d7cd@mail.gmail.com> References: <45EF323F.7060208@gmail.com> <1e2af89e0703071707x615aa787g9883340d2a53d7cd@mail.gmail.com> Message-ID: On 3/8/07, Matthew Brett wrote: > > > My problem is not space, but time. > > > I am creating a small array over and over, > > > and this is turning out to be a bottleneck. > > How about making one large random number array and taking small views? > How is that different from: ++++++++++++++++++++++++++++++++++++++++++++++++++ Allocating all the arrays as one n+1 dim and grabbing rows from it is faster than allocating the small arrays individually. I am iterating too many times to allocate everything at once though. I can just do a nested loop where create manageably large arrays in the outer and grab the rows in the inner, but I wanted something cleaner. +++++++++++++++++++++++++++++++++++++++++++++++++++ later in the post > Matthew Daniel > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From dmahler at gmail.com Thu Mar 8 10:37:03 2007 From: dmahler at gmail.com (Daniel Mahler) Date: Thu, 8 Mar 2007 15:37:03 +0000 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> Message-ID: On 3/8/07, Charles R Harris wrote: > The slow down is probably related to this from a previous thread: > > In [46]: def test1() : > ....: x = normal(0,1,1000) > ....: > > In [47]: def test2() : > ....: for i in range(1000) : > ....: x = normal(0,1) > > In [50]: t = timeit.Timer('test1()', "from __main__ import test1, normal") > > In [51]: t.timeit(100) > Out[51]: 0.022681951522827148 > > In [52]: t = timeit.Timer('test2()', "from __main__ import test2, normal") > > In [53]: t.timeit(100) > Out[53]: 4.3481810092926025 Yes, that is the kind of test that prompted my post in the first place. > Robert thought this might relate to Travis' changes adding broadcasting to > the random number generator. It does seem certain that generating small > arrays of random numbers has a very high overhead. Does that mean someone is working on fixing this? Also what does 'adding broadcasting to the number generator' mean? The random generators currently do not take arrays as args, so I am not sure what it would mean with the current interface. Something like normal(0,1,target=A) though would be what I was asking about in originally, but it seems like this is the cause of my problem rather that the solution :(. thanks D > Chuck > From robert.kern at gmail.com Thu Mar 8 12:48:06 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 08 Mar 2007 11:48:06 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> Message-ID: <45F04C56.7060009@gmail.com> Daniel Mahler wrote: > On 3/8/07, Charles R Harris wrote: >> Robert thought this might relate to Travis' changes adding broadcasting to >> the random number generator. It does seem certain that generating small >> arrays of random numbers has a very high overhead. > > Does that mean someone is working on fixing this? It's not on the top of my list, no. > Also what does 'adding broadcasting to the number generator' mean? normal([[0.0], [0.5]], [1.0, 2.0, 3.0]) That gives you a (2, 3) array of random numbers drawn from 6 different normal distributions: [[(mean=0, stdev=1), (mean=0, stdev=2), (mean=0, stdev=3)], [(mean=0.5, stdev=1), (mean=0.5, stdev=2), (mean=0.5, stdev=3)]] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Mar 8 13:46:14 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 8 Mar 2007 11:46:14 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F04C56.7060009@gmail.com> References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> Message-ID: On 3/8/07, Robert Kern wrote: > > Daniel Mahler wrote: > > On 3/8/07, Charles R Harris wrote: > > >> Robert thought this might relate to Travis' changes adding broadcasting > to > >> the random number generator. It does seem certain that generating small > >> arrays of random numbers has a very high overhead. > > > > Does that mean someone is working on fixing this? > > It's not on the top of my list, no. > > > Also what does 'adding broadcasting to the number generator' mean? > > normal([[0.0], [0.5]], [1.0, 2.0, 3.0]) > > That gives you a (2, 3) array of random numbers drawn from 6 different > normal > distributions: [[(mean=0, stdev=1), (mean=0, stdev=2), (mean=0, stdev=3)], > [(mean=0.5, stdev=1), (mean=0.5, stdev=2), (mean=0.5, > stdev=3)]] For normals this seems overkill as the same result can be achieved by an offset and scale, i.e., if r is an array of random numbers with mean 0 and sigma 1, then myrandomarray = (r*mysigma + mymean) easily achieves the same result. Other distributions don't have such happy properties, unfortunately, and will have high overhead regardless. For instance, Poisson distributions require a computation of new internal parameters for each value of the mean and doing this on an item by item basis over a whole array is a terrible idea. Hmm, I am not convinced that broadcasting is going to buy you much except overhead. Perhaps this problem should be approached on a case by case basis rather than by some global scheme. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Mar 8 17:36:43 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 8 Mar 2007 17:36:43 -0500 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> Message-ID: On 08/03/07, Charles R Harris wrote: > For normals this seems overkill as the same result can be achieved by an > offset and scale, i.e., if r is an array of random numbers with mean 0 and > sigma 1, then > > myrandomarray = (r*mysigma + mymean) > > easily achieves the same result. Other distributions don't have such happy > properties, unfortunately, and will have high overhead regardless. For > instance, Poisson distributions require a computation of new internal > parameters for each value of the mean and doing this on an item by item > basis over a whole array is a terrible idea. Hmm, I am not convinced that > broadcasting is going to buy you much except overhead. Perhaps this problem > should be approached on a case by case basis rather than by some global > scheme. Whether it's efficient or not in terms of CPU time, it's extremely handy to be able to do photons = numpy.random.poisson(0.1*cos(omega*numpy.arange(1000000)+0.9) Of course I'd prefer if it ran faster, but many of numpy's operations are notably slower than the same operation on (say) python lists (try the Programming Language Shootout sometime...); their real advantage is that they allow programs to be written quickly and clearly. Anne From ceball at gmail.com Thu Mar 8 20:01:41 2007 From: ceball at gmail.com (Chris Ball) Date: Fri, 9 Mar 2007 09:01:41 +0800 Subject: [Numpy-discussion] 0/0 is 0 or nan? Message-ID: <4e2a3a120703081701v6531c4dp272e06d8593946b@mail.gmail.com> Hi, I noticed some behavior that seems inconsistent to me: >>> from numpy import divide, seterr >>> seterr(divide='ignore') {'over': 'raise', 'divide': 'raise', 'invalid': 'raise', 'under': 'raise'} >>> seterr() {'over': 'raise', 'divide': 'ignore', 'invalid': 'raise', 'under': 'raise'} >>> divide(0,0) 0 >>> divide(0.0,0.0) Traceback (most recent call last): File "", line 1, in ? FloatingPointError: invalid value encountered in divide Please could someone tell me whether this behavior comes from numpy or elsewhere? Thanks, Chris From robert.kern at gmail.com Thu Mar 8 21:45:27 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 08 Mar 2007 20:45:27 -0600 Subject: [Numpy-discussion] 0/0 is 0 or nan? In-Reply-To: <4e2a3a120703081701v6531c4dp272e06d8593946b@mail.gmail.com> References: <4e2a3a120703081701v6531c4dp272e06d8593946b@mail.gmail.com> Message-ID: <45F0CA47.9090802@gmail.com> Chris Ball wrote: > Hi, > > I noticed some behavior that seems inconsistent to me: > >>>> from numpy import divide, seterr >>>> seterr(divide='ignore') > {'over': 'raise', 'divide': 'raise', 'invalid': 'raise', 'under': 'raise'} >>>> seterr() > {'over': 'raise', 'divide': 'ignore', 'invalid': 'raise', 'under': 'raise'} >>>> divide(0,0) > 0 >>>> divide(0.0,0.0) > Traceback (most recent call last): > File "", line 1, in ? > FloatingPointError: invalid value encountered in divide > > > Please could someone tell me whether this behavior comes from numpy or > elsewhere? seterr() would normally only control the signal handlers for IEEE-754 floating point exceptions (note: these are not Python exception objects, but signals that are thrown at the hardware/OS level, I believe). We have extended this functionality to work with numpy arrays and numpy scalars of non-floating point types. However, I believe that divide() on two Python ints goes through a different path than for numpy arrays and numpy scalars. Rather, they are recognized as objects that implement .__div__(), and that is called instead. Builtin Python ints defer the computation to C directly without the extra error-handling that we've implemented in numpy. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From berthold.hoellmann at gl-group.com Fri Mar 9 03:37:53 2007 From: berthold.hoellmann at gl-group.com (=?iso-8859-15?Q?Berthold_H=F6llmann?=) Date: Fri, 09 Mar 2007 09:37:53 +0100 Subject: [Numpy-discussion] Conservative order in .argsort Message-ID: In our process of switching from Numeric to numpy we had a problem with a function using argsort. The function expected a conservative order of the argsort result but we got: >>> import numpy >>> numpy.argsort(numpy.ones(16, dtype=numpy.dtype('float64'))) array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]) >>> numpy.argsort(numpy.ones(17, dtype=numpy.dtype('float64'))) array([ 0, 14, 13, 12, 11, 10, 9, 15, 8, 6, 5, 4, 3, 2, 1, 7, 16]) >>> numpy.ones(17, dtype=numpy.dtype('float64')).argsort() array([ 0, 14, 13, 12, 11, 10, 9, 15, 8, 6, 5, 4, 3, 2, 1, 7, 16]) >>> numpy.ones(17, dtype=numpy.dtype('float64')).argsort(kind='heap') array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 0]) >>> numpy.ones(17, dtype=numpy.dtype('float64')).argsort(kind='merge') array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]) >>> numpy.ones(100, dtype=numpy.dtype('float64')).argsort(kind='merge') array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]) My question is, is "kind='merge'" sorting garanteed to be conservative? Kind regards Berthold H?llmann -- Germanischer Lloyd AG CAE Development Vorsetzen 35 20459 Hamburg Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellmann at gl-group.com Internet: http://www.gl-group.com This e-mail and any attachment thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as total or partial copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc. Germanischer Lloyd AG, 31393 AG HH, Hamburg, Vorstand: Dr. Hermann J. Klein, Rainer Sch?ndube, Vorsitzender des Aufsichtsrats: Dr. Wolfgang Peiner From gregory.lielens at fft.be Fri Mar 9 03:38:05 2007 From: gregory.lielens at fft.be (Gregory Lielens) Date: Fri, 09 Mar 2007 09:38:05 +0100 Subject: [Numpy-discussion] In need of Bessel functions - cephes/amos bindings for new numpy? In-Reply-To: References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> Message-ID: <45F11CED.1050709@fft.be> Hi, we are looking for Bessel/Hankel ufuncs, and naturally found the cephes module developed by Travis. Had a quick look, and it seems the latest version I found (cepehs-1.3) is quite old, targeted to python 1.5 and numerics. Questions and discussions about it on newsgroup seems to have died off around 2003 too :( . Is there a newer version targeted at the current Numpy distribution? Travis being the developper, I quite expected to find it easily, but so far have not succeeded.... Best regards, Greg. From arnd.baecker at web.de Fri Mar 9 03:53:57 2007 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri, 9 Mar 2007 09:53:57 +0100 (CET) Subject: [Numpy-discussion] In need of Bessel functions - cephes/amos bindings for new numpy? In-Reply-To: <45F11CED.1050709@fft.be> References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> <45F11CED.1050709@fft.be> Message-ID: On Fri, 9 Mar 2007, Gregory Lielens wrote: > Hi, > we are looking for Bessel/Hankel ufuncs, and naturally found the cephes > module developed by Travis. Had a quick look, and it seems the latest > version I found (cepehs-1.3) is quite old, targeted to python 1.5 and > numerics. Questions and discussions about it on newsgroup seems to have > died off around 2003 too :( . > Is there a newer version targeted at the current Numpy distribution? > Travis being the developper, I quite expected to find it easily, but so > far have not succeeded.... have a look at scipy: scipy.special.* should have all what you need (and more ;-) Best, Arnd From nwagner at iam.uni-stuttgart.de Fri Mar 9 04:03:35 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 09 Mar 2007 10:03:35 +0100 Subject: [Numpy-discussion] In need of Bessel functions - cephes/amos bindings for new numpy? In-Reply-To: References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> <45F11CED.1050709@fft.be> Message-ID: <45F122E7.5010804@iam.uni-stuttgart.de> Arnd Baecker wrote: > On Fri, 9 Mar 2007, Gregory Lielens wrote: > > >> Hi, >> we are looking for Bessel/Hankel ufuncs, and naturally found the cephes >> module developed by Travis. Had a quick look, and it seems the latest >> version I found (cepehs-1.3) is quite old, targeted to python 1.5 and >> numerics. Questions and discussions about it on newsgroup seems to have >> died off around 2003 too :( . >> Is there a newer version targeted at the current Numpy distribution? >> Travis being the developper, I quite expected to find it easily, but so >> far have not succeeded.... >> > > have a look at scipy: scipy.special.* > should have all what you need (and more ;-) > > Best, Arnd > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > http://projects.scipy.org/scipy/scipy/ticket/15 How about that ? Nils From robert.kern at gmail.com Fri Mar 9 04:13:42 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 09 Mar 2007 03:13:42 -0600 Subject: [Numpy-discussion] Conservative order in .argsort In-Reply-To: References: Message-ID: <45F12546.8090206@gmail.com> Berthold H?llmann wrote: > My question is, is "kind='merge'" sorting garanteed to be conservative? Yes, such a sort is called "stable" (at least in US computer science) and is documented in the docstring. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From berthold.hoellmann at gl-group.com Fri Mar 9 05:16:23 2007 From: berthold.hoellmann at gl-group.com (=?iso-8859-15?Q?Berthold_H=F6llmann?=) Date: Fri, 09 Mar 2007 11:16:23 +0100 Subject: [Numpy-discussion] Conservative order in .argsort In-Reply-To: <45F12546.8090206@gmail.com> (Robert Kern's message of "Fri, 09 Mar 2007 03:13:42 -0600") References: <45F12546.8090206@gmail.com> Message-ID: Robert Kern writes: > Berthold H?llmann wrote: > >> My question is, is "kind='merge'" sorting garanteed to be conservative? > > Yes, such a sort is called "stable" (at least in US computer science) and is > documented in the docstring. I looked in the numpy book, but this information is missing there. Thanks for your help. Kind regards Berthold H?llmann -- Germanischer Lloyd AG CAE Development Vorsetzen 35 20459 Hamburg Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellmann at gl-group.com Internet: http://www.gl-group.com This e-mail and any attachment thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as total or partial copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc. Germanischer Lloyd AG, 31393 AG HH, Hamburg, Vorstand: Dr. Hermann J. Klein, Rainer Sch?ndube, Vorsitzender des Aufsichtsrats: Dr. Wolfgang Peiner From jbednar at inf.ed.ac.uk Fri Mar 9 06:03:42 2007 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Fri, 9 Mar 2007 11:03:42 +0000 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 6, Issue 18 In-Reply-To: References: Message-ID: <17905.16142.504870.41788@lodestar.inf.ed.ac.uk> | From: Robert Kern | Subject: Re: [Numpy-discussion] in place random generation | | Daniel Mahler wrote: | > On 3/8/07, Charles R Harris wrote: | | >> Robert thought this might relate to Travis' changes adding | >> broadcasting to the random number generator. It does seem | >> certain that generating small arrays of random numbers has a | >> very high overhead. | > | > Does that mean someone is working on fixing this? | | It's not on the top of my list, no. I just wanted to put in a vote saying that generating a large quantity of small arrays of random numbers is quite important in my field, and is something that is definitely slowing us down right now. We often simulate neural networks whose many, many small weight matrices need to be initialized with random numbers, and we are seeing quite slow startup times (on the order of minutes, even though reloading a pickled snapshot of the same simulation once it has been initialized takes only a few seconds). The quality of these particular random numbers doesn't matter very much for us, so we are looking for some cheaper way to fill a bunch of small matrices with at least passably random values. But it would of course be better if the regular high-quality random number support in Numpy were speedy under these conditions... Jim From haase at msg.ucsf.edu Fri Mar 9 09:58:32 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 9 Mar 2007 06:58:32 -0800 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 6, Issue 18 In-Reply-To: <17905.16142.504870.41788@lodestar.inf.ed.ac.uk> References: <17905.16142.504870.41788@lodestar.inf.ed.ac.uk> Message-ID: On 3/9/07, James A. Bednar wrote: > | From: Robert Kern > | Subject: Re: [Numpy-discussion] in place random generation > | > | Daniel Mahler wrote: > | > On 3/8/07, Charles R Harris wrote: > | > | >> Robert thought this might relate to Travis' changes adding > | >> broadcasting to the random number generator. It does seem > | >> certain that generating small arrays of random numbers has a > | >> very high overhead. > | > > | > Does that mean someone is working on fixing this? > | > | It's not on the top of my list, no. > > I just wanted to put in a vote saying that generating a large quantity > of small arrays of random numbers is quite important in my field, and > is something that is definitely slowing us down right now. > > We often simulate neural networks whose many, many small weight > matrices need to be initialized with random numbers, and we are seeing > quite slow startup times (on the order of minutes, even though > reloading a pickled snapshot of the same simulation once it has been > initialized takes only a few seconds). > > The quality of these particular random numbers doesn't matter very > much for us, so we are looking for some cheaper way to fill a bunch of > small matrices with at least passably random values. But it would of > course be better if the regular high-quality random number support in > Numpy were speedy under these conditions... > > Jim > Hey Jim, Could you not create all the many arrays to use "one large chunck" of contiguous memory ? like: 1) create a large 1D array 2) create all small arrays in a for loop using numpy.ndarray(buffer=largeArray[offset], shape=..., dtype=...) --- you increment offset appropriately during the loop 3) then you can reset all small arrays to new random numbers with one call to resetting the large array ((they all have the same statistics (mean,stddev, type), right ? Maybe this helps, Sebastian Haase From faltet at carabos.com Fri Mar 9 11:11:28 2007 From: faltet at carabos.com (faltet at carabos.com) Date: Fri, 9 Mar 2007 16:11:28 +0000 Subject: [Numpy-discussion] [PATCH] a fix for compiling numexpr with MSVC Toolkit 2003 Message-ID: <20070309161128.GB16960@carabos.com> Hi, I've done a patch for allowing compiling the last version of numexpr with the MSVC Toolkit 2003 compiler on Windows platforms. You can fetch it from: http://www.pytables.org/trac/changeset/2514/trunk BTW, I understand now why Tim Hochberg was so worried about the time that it takes to compile numexpr on Win platforms. On my Pentium4 @ 2 GHz, and using the MSVC Toolkit 2003, compiling numexpr takes 20 minutes aprox. (!). With the same machine and using GCC under Linux it takes no more than 1 minute. Mmmm, I think it's time to look at the MINGW compiler (GCC based). Cheers, Francesc From mpmusu at cc.usu.edu Fri Mar 9 11:35:25 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Fri, 09 Mar 2007 09:35:25 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> Message-ID: <45F18CCD.3070002@cc.usu.edu> This discussion has much in common with a previous thread that I started ("When and where to use Numpy..."). I fully admit to being a naive numpy user, but it seems to me that it would be helpful if the documentation provided some explicit statements to inform potential users about the best types of situations where numpy will be useful. And it would be even better if it could be pointed out where just using pure python will be advantageous! As an aside, are the random number generators from scipy.random the same as those for numpy.random? If not, will those of us who need to just use a few random numbers here and there throughout our code (we don't need arrays of random numbers or broadcasting abilities) benefit more from using those libraries? Anne Archibald wrote: > On 08/03/07, Charles R Harris wrote: > >> For normals this seems overkill as the same result can be achieved by an >> offset and scale, i.e., if r is an array of random numbers with mean 0 and >> sigma 1, then >> >> myrandomarray = (r*mysigma + mymean) >> >> easily achieves the same result. Other distributions don't have such happy >> properties, unfortunately, and will have high overhead regardless. For >> instance, Poisson distributions require a computation of new internal >> parameters for each value of the mean and doing this on an item by item >> basis over a whole array is a terrible idea. Hmm, I am not convinced that >> broadcasting is going to buy you much except overhead. Perhaps this problem >> should be approached on a case by case basis rather than by some global >> scheme. > > Whether it's efficient or not in terms of CPU time, it's extremely > handy to be able to do > photons = numpy.random.poisson(0.1*cos(omega*numpy.arange(1000000)+0.9) > > Of course I'd prefer if it ran faster, but many of numpy's operations > are notably slower than the same operation on (say) python lists (try > the Programming Language Shootout sometime...); their real advantage > is that they allow programs to be written quickly and clearly. > > Anne > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From Chris.Barker at noaa.gov Fri Mar 9 12:13:49 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 09 Mar 2007 09:13:49 -0800 Subject: [Numpy-discussion] 0/0 is 0 or nan? In-Reply-To: <45F0CA47.9090802@gmail.com> References: <4e2a3a120703081701v6531c4dp272e06d8593946b@mail.gmail.com> <45F0CA47.9090802@gmail.com> Message-ID: <45F195CD.3080302@noaa.gov> Robert Kern wrote: > However, I believe that divide() on two Python ints goes through a different > path than for numpy arrays and numpy scalars. Rather, they are recognized as > objects that implement .__div__(), and that is called instead. Then wouldn't this be different: >>> a = N.array(0, dtype=N.int) >>> b = N.array(0, dtype=N.int) >>> N.divide(a,b) 0 or to really make sure we're not using Python integers: >>> a = N.array(0, dtype=N.int16) >>> b = N.array(0, dtype=N.int16) >>> N.divide(a,b) 0 for completeness: >>> a = N.array(0, dtype=N.float) >>> b = N.array(0, dtype=N.float) >>> N.divide(a,b) Warning: invalid value encountered in divide nan -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Fri Mar 9 12:40:16 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 9 Mar 2007 09:40:16 -0800 Subject: [Numpy-discussion] Which dtype are supported by numexpr ? Message-ID: Hi ! This is really only one question: Which dtypes are supported by numexpr ? We are very interested in numexpr ! Where is the latest / most-up-to-date documentation ? Thanks, Sebastian Haase From faltet at carabos.com Fri Mar 9 12:56:28 2007 From: faltet at carabos.com (Francesc Altet) Date: Fri, 9 Mar 2007 18:56:28 +0100 Subject: [Numpy-discussion] Which dtype are supported by numexpr ? In-Reply-To: References: Message-ID: <200703091856.30141.faltet@carabos.com> A Divendres 09 Mar? 2007 18:40, Sebastian Haase escrigu?: > Hi ! > This is really only one question: > > Which dtypes are supported by numexpr ? Well, numexpr does support any dtype that is homogeneous, except 'uint64'. This is because internally all the unsigned types are upcasted to the smallest *signed* integer that can fit the info for it. As it happens that 'uint64' doesn't have a standard signed type above that is able to keep its info: this is why it is unsupported. Besides, there is the limitation that Win32 doesn't have such a 'uint64', and computations in numpy or python are normally done by converting them to python long integers (correct me if I'm wrong here), whose arithmetic is very slow compared with the types supported by the compiler. So, the best approach is to avoid 'uint64' types in general. > We are very interested in numexpr ! > Where is the latest / most-up-to-date documentation ? At the moment, I think documentation is reduced at some docstrings. If you want more, you will have to look at the sources. HTH, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From robert.kern at gmail.com Fri Mar 9 13:01:18 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 09 Mar 2007 12:01:18 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F18CCD.3070002@cc.usu.edu> References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> Message-ID: <45F1A0EE.60306@gmail.com> Mark P. Miller wrote: > As an aside, are the random number generators from scipy.random the same > as those for numpy.random? If not, will those of us who need to just > use a few random numbers here and there throughout our code (we don't > need arrays of random numbers or broadcasting abilities) benefit more > from using those libraries? scipy.random is not a package. scipy/__init__.py does a "from numpy import *" and thus pulls in numpy.random. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Mar 9 13:03:33 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 09 Mar 2007 12:03:33 -0600 Subject: [Numpy-discussion] 0/0 is 0 or nan? In-Reply-To: <45F195CD.3080302@noaa.gov> References: <4e2a3a120703081701v6531c4dp272e06d8593946b@mail.gmail.com> <45F0CA47.9090802@gmail.com> <45F195CD.3080302@noaa.gov> Message-ID: <45F1A175.5020505@gmail.com> Christopher Barker wrote: > Robert Kern wrote: >> However, I believe that divide() on two Python ints goes through a different >> path than for numpy arrays and numpy scalars. Rather, they are recognized as >> objects that implement .__div__(), and that is called instead. > > Then wouldn't this be different: > > >>> a = N.array(0, dtype=N.int) > >>> b = N.array(0, dtype=N.int) > >>> N.divide(a,b) > 0 What version of numpy are you using? In [13]: from numpy import * In [14]: a = array(0, dtype=int) In [15]: b = array(0, dtype=int) In [16]: divide(a, b) Warning: divide by zero encountered in divide Out[16]: 0 In [17]: geterr() Out[17]: {'divide': 'print', 'invalid': 'print', 'over': 'print', 'under': 'ignore'} In [18]: import numpy In [19]: numpy.__version__ Out[19]: '1.0.2.dev3569' -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Fri Mar 9 13:14:57 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 9 Mar 2007 13:14:57 -0500 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F1A0EE.60306@gmail.com> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> Message-ID: On 09/03/07, Robert Kern wrote: > Mark P. Miller wrote: > > As an aside, are the random number generators from scipy.random the same > > as those for numpy.random? If not, will those of us who need to just > > use a few random numbers here and there throughout our code (we don't > > need arrays of random numbers or broadcasting abilities) benefit more > > from using those libraries? > > scipy.random is not a package. scipy/__init__.py does a "from numpy import *" > and thus pulls in numpy.random. Why does it do this? It makes it a bit confusing (for example, the post above), but presumably it was done for a compelling reason... Thanks, Anne M. Archibald From mpmusu at cc.usu.edu Fri Mar 9 13:19:37 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Fri, 09 Mar 2007 11:19:37 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F1A0EE.60306@gmail.com> References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> Message-ID: <45F1A539.9050304@cc.usu.edu> Robert Kern wrote: > scipy.random is not a package. scipy/__init__.py does a "from numpy import *" > and thus pulls in numpy.random. > Got it...and one more question: What about using something like from numpy.random import mtrand And then using mtrand.seed and mtrand.normal in code? Would this by any chance get around some of the code that seems to be slowing down the random number generators when one can't efficiently use arrays of random numbers? It seems useful, but I've been hesitant to re-install numpy 1.0.1 again to try it. -Mark From robert.kern at gmail.com Fri Mar 9 14:07:41 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 09 Mar 2007 13:07:41 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F1A539.9050304@cc.usu.edu> References: <45EF323F.7060208@gmail.com> <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1A539.9050304@cc.usu.edu> Message-ID: <45F1B07D.2010800@gmail.com> Mark P. Miller wrote: > Robert Kern wrote: > > scipy.random is not a package. scipy/__init__.py does a "from numpy > import *" > > and thus pulls in numpy.random. > > Got it...and one more question: > > What about using something like > from numpy.random import mtrand > > And then using mtrand.seed and mtrand.normal in code? > > Would this by any chance get around some of the code that seems to be > slowing down the random number generators when one can't efficiently use > arrays of random numbers? No, all of the symbols exposed by numpy.random are just aliases to those in mtrand, too. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Fri Mar 9 14:10:11 2007 From: faltet at carabos.com (Francesc Altet) Date: Fri, 9 Mar 2007 20:10:11 +0100 Subject: [Numpy-discussion] ANN: PyTables 2.0 beta1 released Message-ID: <200703092010.13076.faltet@carabos.com> ========================= Announcing PyTables 2.0b1 ========================= The PyTables development team is very happy to announce the public availability of the first *beta* version of PyTables 2.0. Starting with this release, both the API and the file format have entered in the stage of freezing (i.e. nothing will be modified unless something is clearly *wrong*), so people can use it in order to begin with the migration of their existing PyTables 1.x scripts as well as start enjoying the new exciting features in this major version ;) You can download a source package of the version 2.0b1 with generated PDF and HTML docs from http://www.pytables.org/download/preliminary/ [Starting with the beta phase, and in order to make the life easier for Windows beta-testers, we are delivering binary versions for Win32.] You can also get the latest sources from the Subversion repository at http://pytables.org/svn/pytables/trunk/ If you are afraid of Subversion (you shouldn't), you can always download the latest, daily updated, packed sources from http://www.pytables.org/download/snapshot/ Please have in mind that some sections in the manual can be obsolete (specially the "Optimization tips" chapter). The tutorials chapter is in the process of being updated as well. The reference chapter should be fairly up-to-date though. You may also want to have an in-deep read of the ``RELEASE-NOTES.txt`` file where you will find an entire section devoted to how to migrate your existing PyTables 1.x apps to the 2.0 version. You can find an HTML version of this document at http://www.pytables.org/moin/ReleaseNotes/Release_2.0b1 Changes more in depth ===================== Improvements: - NumPy is finally at the core! That means that PyTables no longer needs numarray in order to operate, although it continues to be supported (as well as Numeric). This also means that you should be able to run PyTables in scenarios combining Python 2.5 and 64-bit platforms (these are a source of problems with numarray/Numeric because they don't support this combination yet). - Most of the operations in PyTables have experimented noticeable speed-ups (sometimes up to 2x, like in regular Python table selections). This is a consequence of both using NumPy internally and a considerable effort in terms of refactorization and optimization of the new code. - Combined conditions are finally supported for in-kernel selections. So, now it is possible to perform complex selections like:: result = [ row['var3'] for row in table.where('(var2 < 20) | (var1 == "sas")') ] or:: complex_cond = '((%s <= col5) & (col2 <= %s)) ' \ '| (sqrt(col1 + 3.1*col2 + col3*col4) > 3)' result = [ row['var3'] for row in table.where(complex_cond % (inf, sup)) ] and run them at full C-speed (or perhaps more, due to the cache-tuned computing kernel of numexpr, which has been integrated into PyTables). - Now, it is possible to get fields of the ``Row`` iterator by specifying their position, or even ranges of positions (extended slicing is supported). For example, you can do:: result = [ row[4] for row in table # fetch field #4 if row[1] < 20 ] result = [ row[:] for row in table # fetch all fields if row['var2'] < 20 ] result = [ row[1::2] for row in # fetch odd fields table.iterrows(2, 3000, 3) ] in addition to the classical:: result = [row['var3'] for row in table.where('var2 < 20')] - ``Row`` has received a new method called ``fetch_all_fields()`` in order to easily retrieve all the fields of a row in situations like:: [row.fetch_all_fields() for row in table.where('column1 < 0.3')] The difference between ``row[:]`` and ``row.fetch_all_fields()`` is that the former will return all the fields as a tuple, while the latter will return the fields in a NumPy void type and should be faster. Choose whatever fits better to your needs. - Now, all data that is read from disk is converted, if necessary, to the native byteorder of the hosting machine (before, this only happened with ``Table`` objects). This should help to accelerate apps that have to do computations with data generated in platforms with a byteorder different than the user machine. - The modification of values in ``*Array`` objects (through __setitem__) now doesn't make a copy of the value in the case that the shape of the value passed is the same as the slice to be overwritten. This results in considerable memory savings when you are modifying disk objects with big array values. - All the leaf constructors have received a new parameter called ``byteorder`` that lets the user specify the byteorder of their data *on disk*. This effectively allows to create datasets in other byteorders than the native platform. - Native HDF5 datasets with ``H5T_ARRAY`` datatypes are fully supported for reading now. - The test suites for the different packages are installed now, so you don't need a copy of the PyTables sources to run the tests. Besides, you can run the test suite from the Python console by using:: >>> tables.tests() Bug fixes: - As mentioned above, the fact that NumPy is at the core makes that certain bizarre interactions between numarray and NumPy scalars don't affect the behaviour of table selections anymore. Fixes http://www.pytables.org/trac/ticket/29. - Did I mention that PyTables 2.0 can be safely used in 64-bit platforms in combination with Python 2.5? ;) Deprecated features: - Not many, really. Please see ``RELEASE-NOTES.txt`` file. Backward-incompatible changes: - Many. Please see ``RELEASE-NOTES.txt`` file. Important note for Windows users ================================ In order to keep PyTables happy, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at ftp://ftp.hdfgroup.org/HDF5/current/bin/windows/5-165-win-net.ZIP Please remember that, from PyTables 2.0 on, Python 2.3 (and lesser) is not supported anymore. What is PyTables? ================= **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and NumPy package (but numarray and Numeric are also supported) for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables Pro sports indexing capabilities as well, allowing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC (and PowerPC64) and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About NumPy: http://numpy.scipy.org/ To know more about the company behind the development of PyTables, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From robert.kern at gmail.com Fri Mar 9 14:12:05 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 09 Mar 2007 13:12:05 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> Message-ID: <45F1B185.9070108@gmail.com> Anne Archibald wrote: > On 09/03/07, Robert Kern wrote: >> Mark P. Miller wrote: >>> As an aside, are the random number generators from scipy.random the same >>> as those for numpy.random? If not, will those of us who need to just >>> use a few random numbers here and there throughout our code (we don't >>> need arrays of random numbers or broadcasting abilities) benefit more >>> from using those libraries? >> scipy.random is not a package. scipy/__init__.py does a "from numpy import *" >> and thus pulls in numpy.random. > > Why does it do this? It makes it a bit confusing (for example, the > post above), but presumably it was done for a compelling reason... Back when scipy was based on Numeric, scipy provided a slew of new, basic array functions (e.g. linspace(), et al.) that have now been folded into numpy. It also overrode the math ufuncs with faster variants that would yield nans and infs rather than throwing a ZeroDivisionError. The idea was to be able to use "scipy" as a replacement for "Numeric". I wish we didn't do it today, either, but we are stuck with a lot of legacy code. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From matthew.brett at gmail.com Fri Mar 9 14:12:17 2007 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 9 Mar 2007 11:12:17 -0800 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45EF323F.7060208@gmail.com> <1e2af89e0703071707x615aa787g9883340d2a53d7cd@mail.gmail.com> Message-ID: <1e2af89e0703091112r456d47bai3f48ecd69ac75167@mail.gmail.com> Oh dear, sorry, I should have read your email more carefully. Matthew On 3/8/07, Daniel Mahler wrote: > On 3/8/07, Matthew Brett wrote: > > > > My problem is not space, but time. > > > > I am creating a small array over and over, > > > > and this is turning out to be a bottleneck. > > > > How about making one large random number array and taking small views? > > > > How is that different from: > > ++++++++++++++++++++++++++++++++++++++++++++++++++ > Allocating all the arrays as one n+1 dim and grabbing rows from it > is faster than allocating the small arrays individually. > I am iterating too many times to allocate everything at once though. > I can just do a nested loop > where create manageably large arrays in the outer > and grab the rows in the inner, > but I wanted something cleaner. > +++++++++++++++++++++++++++++++++++++++++++++++++++ > > later in the post > > > Matthew > > Daniel > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From lechtlr at yahoo.com Fri Mar 9 14:41:23 2007 From: lechtlr at yahoo.com (lechtlr) Date: Fri, 9 Mar 2007 11:41:23 -0800 (PST) Subject: [Numpy-discussion] Variable String Format Message-ID: <701717.73526.qm@web57905.mail.re3.yahoo.com> I would very much appreciate, if someone can give suggestions to implement a loop to generate a string in the following format. Variables with assigned values are given in A: A = {a1:2.0, a2:4.0, ,an:5.0} I want to transform what is in the dictionary into a string in the following format: X = ?a1:2.0, a2:4.0, ,an:5.0? I want to implement a loop to generate X for a given A with varying n. Is there a way to do this in python..? Thanks --------------------------------- The fish are biting. Get more visitors on your site using Yahoo! Search Marketing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Mar 9 14:46:45 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 09 Mar 2007 11:46:45 -0800 Subject: [Numpy-discussion] 0/0 is 0 or nan? In-Reply-To: <45F1A175.5020505@gmail.com> References: <4e2a3a120703081701v6531c4dp272e06d8593946b@mail.gmail.com> <45F0CA47.9090802@gmail.com> <45F195CD.3080302@noaa.gov> <45F1A175.5020505@gmail.com> Message-ID: <45F1B9A5.1030503@noaa.gov> Robert Kern wrote: > What version of numpy are you using? >>> from numpy import * >>> print __version__ 1.0.1 >>> a = array(0, dtype=int) >>> b = array(0, dtype=int) >>> divide(a, b) Warning: divide by zero encountered in divide 0 >>> geterr() {'over': 'print', 'divide': 'print', 'invalid': 'print', 'under': 'ignore'} >>> divide(0.0, 0.0) Warning: invalid value encountered in divide nan I had reset error reporting to match the OP before. It looks like I get the same as you with the defaults. There is no NaN for integers, so I guess 0 is as good as anything -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Fri Mar 9 14:46:23 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 10 Mar 2007 04:46:23 +0900 Subject: [Numpy-discussion] Variable String Format In-Reply-To: <701717.73526.qm@web57905.mail.re3.yahoo.com> References: <701717.73526.qm@web57905.mail.re3.yahoo.com> Message-ID: str(A)[1:-1] ? --bb On 3/10/07, lechtlr wrote: > > I would very much appreciate, if someone can give suggestions to > implement a loop to generate a string in the following format. > Variables with assigned values are given in A: > A = {a1:2.0, a2:4.0,???,an:5.0} > I want to transform what is in the dictionary into a string in the > following format: > X = 'a1:2.0, a2:4.0, ??,an:5.0' > I want to implement a loop to generate X for a given A with varying n. Is > there a way to do this in python..? > > Thanks > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Mar 9 15:01:21 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 10 Mar 2007 05:01:21 +0900 Subject: [Numpy-discussion] Putting some random back into the top-level? Message-ID: Has enough time passed with no top level random function that we can now put one back in? If I recall, the former top level rand() was basically removed because it didn't adhere to the "shapes are always tuples" convention. Has enough time passed now that we can put something like it back in the top level, in tuple-taking form? I think this is a function people use pretty frequently when writing quick tests. And numpy.random.random_sample seems a rather long and not so obvious name for something that is used relatively frequently. --bb From jbednar at inf.ed.ac.uk Fri Mar 9 15:04:56 2007 From: jbednar at inf.ed.ac.uk (James A. Bednar) Date: Fri, 9 Mar 2007 20:04:56 +0000 Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 6, Issue 20 In-Reply-To: References: Message-ID: <17905.48616.676069.986674@lodestar.inf.ed.ac.uk> | Date: Fri, 9 Mar 2007 06:58:32 -0800 | From: "Sebastian Haase" | Subject: Re: [Numpy-discussion] Numpy-discussion Digest, Vol 6, Issue 18 | To: "Discussion of Numerical Python" | | On 3/9/07, James A. Bednar wrote: | > | From: Robert Kern | > | Subject: Re: [Numpy-discussion] in place random generation | > | | > | Daniel Mahler wrote: | > | > On 3/8/07, Charles R Harris wrote: | > | | > | >> Robert thought this might relate to Travis' changes adding | > | >> broadcasting to the random number generator. It does seem | > | >> certain that generating small arrays of random numbers has a | > | >> very high overhead. | > | > | > | > Does that mean someone is working on fixing this? | > | | > | It's not on the top of my list, no. | > | > I just wanted to put in a vote saying that generating a large quantity | > of small arrays of random numbers is quite important in my field, and | > is something that is definitely slowing us down right now. | > | > We often simulate neural networks whose many, many small weight | > matrices need to be initialized with random numbers, and we are seeing | > quite slow startup times (on the order of minutes, even though | > reloading a pickled snapshot of the same simulation once it has been | > initialized takes only a few seconds). | > | > The quality of these particular random numbers doesn't matter very | > much for us, so we are looking for some cheaper way to fill a bunch of | > small matrices with at least passably random values. But it would of | > course be better if the regular high-quality random number support in | > Numpy were speedy under these conditions... | | Hey Jim, | | Could you not create all the many arrays to use "one large chunck" of | contiguous memory ? | like: 1) create a large 1D array | 2) create all small arrays in a for loop using | numpy.ndarray(buffer=largeArray[offset], shape=..., dtype=...) --- | you increment offset appropriately during the loop | 3) then you can reset all small arrays to new random numbers with one | call to resetting the large array ((they all have the same statistics | (mean,stddev, type), right ? In principle, I *think* we could make that work. But we maintain a large object-oriented toolkit for computational neuroscience (see topographica.org), and we try to let each object take care of its own business as much as possible, so that people can later swap things out with their own customized versions. That's hard to do if we set up global dependencies like this, and the results are quite difficult to maintain. Of course, we can and do put in optimizations for certain special cases, but I suspect that it will be simpler in this case just to add some fast-and-dirty but general way to fill small arrays with random values. Still, it would be much simpler for us if the basic numpy small random array support had less overhead... Jim From kwgoodman at gmail.com Fri Mar 9 15:08:32 2007 From: kwgoodman at gmail.com (Keith Goodman) Date: Fri, 9 Mar 2007 12:08:32 -0800 Subject: [Numpy-discussion] Putting some random back into the top-level? In-Reply-To: References: Message-ID: On 3/9/07, Bill Baxter wrote: > Has enough time passed with no top level random function that we can > now put one back in? > If I recall, the former top level rand() was basically removed because > it didn't adhere to the "shapes are always tuples" convention. Top-level rand functions are still around for those who use numpy.matlib as the top level. From pgmdevlist at gmail.com Fri Mar 9 15:17:08 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 9 Mar 2007 15:17:08 -0500 Subject: [Numpy-discussion] Variable String Format In-Reply-To: <701717.73526.qm@web57905.mail.re3.yahoo.com> References: <701717.73526.qm@web57905.mail.re3.yahoo.com> Message-ID: <200703091517.08356.pgmdevlist@gmail.com> On Friday 09 March 2007 14:41:23 lechtlr wrote: > I would very much appreciate, if someone can give suggestions to > implement a loop to generate a string in the following format. Variables > with assigned values are given in A: > A = {a1:2.0, a2:4.0, ,an:5.0} > I want to transform what is in the dictionary into a string in the > following format: X = ?a1:2.0, a2:4.0, ,an:5.0? > I want to implement a loop to generate X for a given A with varying n. > Is there a way to do this in python..? As a starter: ','.join(["%s:%s" % (k,v) for (k,v) in A.iteritems()]) From aisaac at american.edu Fri Mar 9 15:24:02 2007 From: aisaac at american.edu (Alan Isaac) Date: Fri, 09 Mar 2007 15:24:02 -0500 Subject: [Numpy-discussion] Variable String Format In-Reply-To: <701717.73526.qm@web57905.mail.re3.yahoo.com> References: <701717.73526.qm@web57905.mail.re3.yahoo.com> Message-ID: <45F1C262.1000608@american.edu> lechtlr wrote: > I would very much appreciate, if someone can give suggestions to implement a loop to generate a string in the following format. > Variables with assigned values are given in A: > A = {a1:2.0, a2:4.0,???,an:5.0} > I want to transform what is in the dictionary into a string in the following format: > X = ?a1:2.0, a2:4.0, ??,an:5.0? > I want to implement a loop to generate X for a given A with varying n. Is there a way to do this in python..? Is this good enough? >>> A = dict(a1=2.0, a2=4.0, a3=3.0, a4=4.0, a5=5.0) >>> A {'a1': 2.0, 'a3': 3.0, 'a2': 4.0, 'a5': 5.0, 'a4': 4.0} >>> str(A) "{'a1': 2.0, 'a3': 3.0, 'a2': 4.0, 'a5': 5.0, 'a4': 4.0}" If not, just strip out what you do not like. Cheers, Alan Isaac From cookedm at physics.mcmaster.ca Fri Mar 9 17:19:43 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri, 9 Mar 2007 17:19:43 -0500 Subject: [Numpy-discussion] [PATCH] a fix for compiling numexpr with MSVC Toolkit 2003 In-Reply-To: <20070309161128.GB16960@carabos.com> References: <20070309161128.GB16960@carabos.com> Message-ID: <20070309171943.43b39757@arbutus.physics.mcmaster.ca> On Fri, 9 Mar 2007 16:11:28 +0000 faltet at carabos.com wrote: > Hi, > > I've done a patch for allowing compiling the last version of numexpr with > the MSVC Toolkit 2003 compiler on Windows platforms. You can fetch it > from: > > http://www.pytables.org/trac/changeset/2514/trunk Checked in. It didn't match up; you've got about 150 more lines in your version than scipy's version. > BTW, I understand now why Tim Hochberg was so worried about the time > that it takes to compile numexpr on Win platforms. On my Pentium4 @ 2 > GHz, and using the MSVC Toolkit 2003, compiling numexpr takes 20 minutes > aprox. (!). With the same machine and using GCC under Linux it takes no > more than 1 minute. Mmmm, I think it's time to look at the MINGW > compiler (GCC based). Wow! I wonder if lowering the optimisation level would help. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From jturner at gemini.edu Fri Mar 9 18:44:34 2007 From: jturner at gemini.edu (James Turner) Date: Fri, 09 Mar 2007 20:44:34 -0300 Subject: [Numpy-discussion] nd_image.affine_transform edge effects Message-ID: <45F1F162.2030102@gemini.edu> Last year I wrote a program that uses the affine_transform() function in numarray to resample and co-add datacubes with WCS offsets in 3D. This function makes it relatively easy to handle N-D offsets and rotations with a smooth interpolant, which is exactly what I wanted. However, I am finding that the results contain edge effects which translate into artefacts in the middle of the final mosaic. As a test, I tried resampling only a single input cube onto the original output grid. In the result, it looks like there is a reflection around the boundary of the input data. Counting from the top, for example, the first and second rows from the input image more-or-less map to the second and third rows in the output, but the top row in the output looks the same as the second row of the input or the third row of the output. I can send a couple of 12k JPEGs illustrating this problem if someone wants to see what I mean. I have tried altering the affine_transform() parameters, such as order or cval, but without managing to improve this behaviour. Does anyone know whether this is a known problem? Given the reflection-like behaviour, I'm wondering whether it is a local property of the interpolant symmetry, an issue with calculating array indices or something like that. These are just quick ideas off the top of my head. I'll be grateful if someone (especially the developers) can comment on what can be done about this, since the artefacts will limit my program's usefulness for science-quality work and it is otherwise ready for use. Thanks a lot! James. From ceball at gmail.com Fri Mar 9 23:00:36 2007 From: ceball at gmail.com (Chris Ball) Date: Sat, 10 Mar 2007 12:00:36 +0800 Subject: [Numpy-discussion] 0/0 is 0 or nan? Message-ID: <4e2a3a120703092000g1f9cf149x3fd056daf9c7bd2@mail.gmail.com> Thank you both for your replies - the difference is clear to me now. Chris From sgwoodjr at gmail.com Sat Mar 10 02:14:24 2007 From: sgwoodjr at gmail.com (Simon Wood) Date: Sat, 10 Mar 2007 02:14:24 -0500 Subject: [Numpy-discussion] Python vector and matrix formatter similar to Matlab Message-ID: To all, I came across an old thread in the archives in which Alexander Schmolck gave an example of a Matlab like matrix formatter he authored for Python. Is this formatter still available some where? Or has anyone else done anything similar? -Simon P.S. Here is a short example of the output: >>>* a *array([[-9.90000000e+01, -9.72817182e+01, 0.00000000e+00, -7.99144631e+01], [-4.54018500e+01, 4.84131591e+01, 3.03428793e+02, 9.96633158e+02], [2.88095799e+03, 8.00308393e+03, 2.19264658e+04, 5.97741417e+04]]) >>>* m = matrix(a) *>>>* m *matrix(''' 1.0E+04 * -0.00990 -0.00973 0.00000 -0.00799 -0.00454 0.00484 0.03034 0.09966 0.28810 0.80031 2.19265 5.97741 ''') >>>* m.show('long') *matrix(''' 1.0E+04 * Columns 0 through 3 -0.009900000000000 -0.009728171817154 0.000000000000000 -0.004540184996686 0.004841315910258 0.030342879349274 0.288095798704173 0.800308392757538 2.192646579480672 Columns 3 through 4 -0.007991446307681 0.099663315842846 5.977414171519782 ''') -------------- next part -------------- An HTML attachment was scrubbed... URL: From jks at iki.fi Sat Mar 10 08:15:22 2007 From: jks at iki.fi (=?iso-8859-1?Q?Jouni_K=2E_Sepp=E4nen?=) Date: Sat, 10 Mar 2007 15:15:22 +0200 Subject: [Numpy-discussion] setmember1d: docstring vs. code Message-ID: Hi, According to an earlier posting on this list, the elements of the first argument to setmember1d are assumed to be unique: http://thread.gmane.org/gmane.comp.python.numeric.general/13251/focus=13307 The docstring for setmember1d does not state this: | Definition: numpy.setmember1d(ar1, ar2) | Docstring: | Return an array of shape of ar1 containing 1 where the elements of | ar1 are in ar2 and 0 otherwise. | | See also: ediff1d, unique1d, intersect1d, intersect1d_nu, setxor1d, | union1d, setdiff1d The numpy book has a slightly different description, which claims that setmember1d(tocheck, set) is "[e]quivalent to array([x in set for x in tocheck])." This is not true when tocheck has repeated elements: In [29]:tocheck = [1,3,2,3] In [30]:set = [2] In [31]:numpy.setmember1d(tocheck, set) Out[31]:array([False, True, True, False], dtype=bool) In [32]:array([x in set for x in tocheck]) Out[32]:array([False, False, True, False], dtype=bool) I think that either the docstring (and the book) should be corrected to mention the assumption, or the code should be made to work in the arbitrary case. I would prefer the latter choice (but perhaps the current code has some advantages). -- Jouni K. Sepp?nen http://www.iki.fi/jks From a.schmolck at gmx.net Sat Mar 10 11:57:30 2007 From: a.schmolck at gmx.net (Alexander Schmolck) Date: 10 Mar 2007 16:57:30 +0000 Subject: [Numpy-discussion] Python vector and matrix formatter similar to Matlab In-Reply-To: References: Message-ID: "Simon Wood" writes: > To all, > I came across an old thread in the archives in which Alexander Schmolck gave > an example of a Matlab like matrix formatter he authored for Python. Is this > formatter still available some where? Yup. I've still got it as part of a matrix class I wrote many years ago (because I found the one in Numeric mostly useless and arrays syntactically to awkward for linear algebra). Numpy sort of broke my design (which was flawed anyway and partly rather incompetently implemented) and for various reasons I ended up mostly coding numeric stuff in matlab for some time anyway, but as I thankfully seem to be moving mostly back to python again, I do intend to salvage, port and clean up at least the formatting code (possibly more, there were some other things I liked about it). With a bit of work this could either go into numpy/scipy if there's interest or maybe make a useful addition to ipython (as the main benefit is that it's really a much more handy way to display matrices (or indeed arrays) for interactive work). How urgently are you interested in this? 'as From mpmusu at cc.usu.edu Sat Mar 10 12:13:09 2007 From: mpmusu at cc.usu.edu (Mark P. Miller) Date: Sat, 10 Mar 2007 10:13:09 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F1B185.9070108@gmail.com> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> Message-ID: <45F2E725.9050901@cc.usu.edu> Robert: Just a thought on this topic: Would it be possible for the Scipy folks to add a new module based solely off your old mtrand code (pre-broadcast)? I have to say that the mtrand code from numpy 0.9.8 has some excellent advantages over the core python random number generators. This would allow Scipy to have a pure random number generator, whereas numpy could keep its fancier (but slower) array-based random number generators. Robert Kern wrote: > Anne Archibald wrote: >> On 09/03/07, Robert Kern wrote: >>> Mark P. Miller wrote: >>>> As an aside, are the random number generators from scipy.random the same >>>> as those for numpy.random? If not, will those of us who need to just >>>> use a few random numbers here and there throughout our code (we don't >>>> need arrays of random numbers or broadcasting abilities) benefit more >>>> from using those libraries? >>> scipy.random is not a package. scipy/__init__.py does a "from numpy import *" >>> and thus pulls in numpy.random. >> Why does it do this? It makes it a bit confusing (for example, the >> post above), but presumably it was done for a compelling reason... > > Back when scipy was based on Numeric, scipy provided a slew of new, basic array > functions (e.g. linspace(), et al.) that have now been folded into numpy. It > also overrode the math ufuncs with faster variants that would yield nans and > infs rather than throwing a ZeroDivisionError. The idea was to be able to use > "scipy" as a replacement for "Numeric". > > I wish we didn't do it today, either, but we are stuck with a lot of legacy code. > From sgwoodjr at gmail.com Sat Mar 10 12:12:01 2007 From: sgwoodjr at gmail.com (Simon Wood) Date: Sat, 10 Mar 2007 12:12:01 -0500 Subject: [Numpy-discussion] Python vector and matrix formatter similar to Matlab In-Reply-To: References: Message-ID: On 10 Mar 2007 16:57:30 +0000, Alexander Schmolck wrote: > > "Simon Wood" writes: > > > To all, > > I came across an old thread in the archives in which Alexander Schmolck > gave > > an example of a Matlab like matrix formatter he authored for Python. Is > this > > formatter still available some where? > > Yup. I've still got it as part of a matrix class I wrote many years ago > (because I found the one in Numeric mostly useless and arrays > syntactically to > awkward for linear algebra). Numpy sort of broke my design (which was > flawed > anyway and partly rather incompetently implemented) and for various > reasons I > ended up mostly coding numeric stuff in matlab for some time anyway, but > as I > thankfully seem to be moving mostly back to python again, I do intend to > salvage, port and clean up at least the formatting code (possibly more, > there > were some other things I liked about it). > > With a bit of work this could either go into numpy/scipy if there's > interest > or maybe make a useful addition to ipython (as the main benefit is that > it's > really a much more handy way to display matrices (or indeed arrays) for > interactive work). > > How urgently are you interested in this? > > 'as > I wouldn't mind getting my hands on the formatting code sooner rather than later if you can dig it up. I'm in the process of extending some underlying C++ matrix and vector classes into Python and would like a good way (i.e. Matlab way) to display the vectors and matrices. I currently convert my underlying data to an array using the 'asarryay' method and then simply use the 'repr' for arrays in displaying my data. I assume I will have to do some work in getting my array/matrix data to work with your formatter as well. -Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgwoodjr at gmail.com Sat Mar 10 12:12:01 2007 From: sgwoodjr at gmail.com (Simon Wood) Date: Sat, 10 Mar 2007 12:12:01 -0500 Subject: [Numpy-discussion] Python vector and matrix formatter similar to Matlab In-Reply-To: References: Message-ID: On 10 Mar 2007 16:57:30 +0000, Alexander Schmolck wrote: > > "Simon Wood" writes: > > > To all, > > I came across an old thread in the archives in which Alexander Schmolck > gave > > an example of a Matlab like matrix formatter he authored for Python. Is > this > > formatter still available some where? > > Yup. I've still got it as part of a matrix class I wrote many years ago > (because I found the one in Numeric mostly useless and arrays > syntactically to > awkward for linear algebra). Numpy sort of broke my design (which was > flawed > anyway and partly rather incompetently implemented) and for various > reasons I > ended up mostly coding numeric stuff in matlab for some time anyway, but > as I > thankfully seem to be moving mostly back to python again, I do intend to > salvage, port and clean up at least the formatting code (possibly more, > there > were some other things I liked about it). > > With a bit of work this could either go into numpy/scipy if there's > interest > or maybe make a useful addition to ipython (as the main benefit is that > it's > really a much more handy way to display matrices (or indeed arrays) for > interactive work). > > How urgently are you interested in this? > > 'as > I wouldn't mind getting my hands on the formatting code sooner rather than later if you can dig it up. I'm in the process of extending some underlying C++ matrix and vector classes into Python and would like a good way (i.e. Matlab way) to display the vectors and matrices. I currently convert my underlying data to an array using the 'asarryay' method and then simply use the 'repr' for arrays in displaying my data. I assume I will have to do some work in getting my array/matrix data to work with your formatter as well. -Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Sat Mar 10 13:18:30 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Sat, 10 Mar 2007 11:18:30 -0700 Subject: [Numpy-discussion] [PATCH] a fix for compiling numexpr with MSVC Toolkit 2003 In-Reply-To: <20070309171943.43b39757@arbutus.physics.mcmaster.ca> References: <20070309161128.GB16960@carabos.com> <20070309171943.43b39757@arbutus.physics.mcmaster.ca> Message-ID: On 3/9/07, David M. Cooke wrote: > > On Fri, 9 Mar 2007 16:11:28 +0000 > faltet at carabos.com wrote: > > > Hi, > > > > I've done a patch for allowing compiling the last version of numexpr > with > > the MSVC Toolkit 2003 compiler on Windows platforms. You can fetch it > > from: > > > > http://www.pytables.org/trac/changeset/2514/trunk > > Checked in. It didn't match up; you've got about 150 more lines in your > version than scipy's version. > > > BTW, I understand now why Tim Hochberg was so worried about the time > > that it takes to compile numexpr on Win platforms. On my Pentium4 @ 2 > > GHz, and using the MSVC Toolkit 2003, compiling numexpr takes 20 minutes > > aprox. (!). With the same machine and using GCC under Linux it takes no > > more than 1 minute. Mmmm, I think it's time to look at the MINGW > > compiler (GCC based). > > Wow! I wonder if lowering the optimisation level would help. Yes. In my local copy, I use -O1, not -O2 and it compiles quite fast. It's been quite a while since I measured it, but as I recall, dropping the optimization didn't result in a noticeable performance decrease. In fact, I have a vague recollection that it may have run slight faster?! I guess it might be a good idea to just use -O1 under variations of VCC. Just to clarify my position though, compile time isn't my concern with the switch statement. My worry there is that as the switch statement grows, performance will take a hit as the switch overflows the code cache. Or something like that. I'm unsure of the details, I just know that the python dev crowd is always worring about that in the main interpreter loop. -tim -- > |>|\/|< > > /--------------------------------------------------------------------------\ > |David M. Cooke > http://arbutus.physics.mcmaster.ca/dmc/ > |cookedm at physics.mcmaster.ca > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 10 14:26:59 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 10 Mar 2007 13:26:59 -0600 Subject: [Numpy-discussion] setmember1d: docstring vs. code In-Reply-To: References: Message-ID: <45F30683.2080106@gmail.com> Jouni K. Sepp?nen wrote: > I think that either the docstring (and the book) should be corrected > to mention the assumption, or the code should be made to work in the > arbitrary case. This is the current docstring: In [2]: setmember1d? Type: function Base Class: Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy-1.0.2.dev3569-py2.5-macosx-10.3-fat.egg/numpy/lib/arraysetops.py Definition: setmember1d(ar1, ar2) Docstring: Return a boolean array of shape of ar1 containing True where the elements of ar1 are in ar2 and False otherwise. Use unique1d() to generate arrays with only unique elements to use as inputs to this function. :Parameters: - `ar1` : array - `ar2` : array :Returns: - `mask` : bool array The values ar1[mask] are in ar2. :See also: numpy.lib.arraysetops has a number of other functions for performing set operations on arrays. > I would prefer the latter choice (but perhaps the > current code has some advantages). Well, it has the advantage of existing. If you have an implementation that is just as efficient, but works for general arrays, I'd love to see it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Mar 10 14:29:42 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 10 Mar 2007 13:29:42 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F2E725.9050901@cc.usu.edu> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> Message-ID: <45F30726.9010406@gmail.com> Mark P. Miller wrote: > Robert: Just a thought on this topic: > > Would it be possible for the Scipy folks to add a new module based > solely off your old mtrand code (pre-broadcast)? I have to say that the > mtrand code from numpy 0.9.8 has some excellent advantages over the core > python random number generators. I won't. That effort would be better spent adding appropriate shortcuts to the current code such that requests for scalars will take a faster path. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From steve at shrogers.com Sat Mar 10 15:30:02 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 10 Mar 2007 13:30:02 -0700 Subject: [Numpy-discussion] Teaching Array Languages Message-ID: <45F3154A.5070808@shrogers.com> Thanks to all who responded to my question about teaching array programming. I've compiled a brief summary of the responses. NumPy ===== * Subject - Physics/Astronomy 3 - Biotechnology 1 - Engineering 2 - Microeconomics 1 * Level - College/University 7 J = * Subject - Math 1 - Engineering 1 - Technical English * Level - College/University 3 - Mathematica =========== * Subject - Math 1 * Level - College/University 1 Matlab ====== * Subject - Engineering 1 IDL === * Subject - Physics/Astronomy 2 * Level - College/University 2 Discussion ========== While this informal survey is neither precise nor comprehensive, I think it is interesting. I queried the NumPy/SciPy and J mailing lists because those are the ones that I follow. A more rigorous and broadly distributed survey may be in order. Regards, Steve From haase at msg.ucsf.edu Sat Mar 10 16:04:49 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 10 Mar 2007 13:04:49 -0800 Subject: [Numpy-discussion] email notification from the TRAC systems Message-ID: Hi, Myself and a friend of mine were surprised that we did not get emailed when someone commented on a bug reports we made on the numpy or scipy TRAC bug tracker system. (I started at some point adding an explicit CC entry for myself :-( ) I just googled for this, an I found that in a trac.ini file there should be this option: [notification] always_notify_owner = true Why is this (or is it !?) not used in numpy/scipy TRAC system ? I think - since one has to sign up using an email address - that it's naturally to expect that one would get comments copied by email. -Sebastian Haase From haase at msg.ucsf.edu Sat Mar 10 16:23:10 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 10 Mar 2007 13:23:10 -0800 Subject: [Numpy-discussion] Teaching Array Languages In-Reply-To: <45F3154A.5070808@shrogers.com> References: <45F3154A.5070808@shrogers.com> Message-ID: Sorry for being so dense - what do the numbers mean ? S.H. On 3/10/07, Steven H. Rogers wrote: > Thanks to all who responded to my question about teaching array > programming. I've compiled a brief summary of the responses. > > NumPy > ===== > * Subject > - Physics/Astronomy 3 > - Biotechnology 1 > - Engineering 2 > - Microeconomics 1 > * Level > - College/University 7 > > J > = > * Subject > - Math 1 > - Engineering 1 > - Technical English > * Level > - College/University 3 > - > > Mathematica > =========== > * Subject > - Math 1 > * Level > - College/University 1 > > Matlab > ====== > * Subject > - Engineering 1 > > IDL > === > * Subject > - Physics/Astronomy 2 > * Level > - College/University 2 > > Discussion > ========== > While this informal survey is neither precise nor comprehensive, I think > it is interesting. I queried the NumPy/SciPy and J mailing lists > because those are the ones that I follow. A more rigorous and broadly > distributed survey may be in order. > > Regards, > Steve > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From steve at shrogers.com Sat Mar 10 16:43:49 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 10 Mar 2007 14:43:49 -0700 Subject: [Numpy-discussion] Teaching Array Languages In-Reply-To: References: <45F3154A.5070808@shrogers.com> Message-ID: <45F32695.1060103@shrogers.com> Sebastian Haase wrote: > Sorry for being so dense - what do the numbers mean ? > S.H. > Sorry for being unclear. The numbers are the number of responses for the category. There were 7 total responses for NumPy, all at the college/university level, and 3 respondents cited physics/astronomy as the subject area, etc. > > On 3/10/07, Steven H. Rogers wrote: >> Thanks to all who responded to my question about teaching array >> programming. I've compiled a brief summary of the responses. >> >> NumPy >> ===== >> * Subject >> - Physics/Astronomy 3 >> - Biotechnology 1 >> - Engineering 2 >> - Microeconomics 1 >> * Level >> - College/University 7 >> >> J >> = >> * Subject >> - Math 1 >> - Engineering 1 >> - Technical English >> * Level >> - College/University 3 >> - >> >> Mathematica >> =========== >> * Subject >> - Math 1 >> * Level >> - College/University 1 >> >> Matlab >> ====== >> * Subject >> - Engineering 1 >> >> IDL >> === >> * Subject >> - Physics/Astronomy 2 >> * Level >> - College/University 2 >> >> Discussion >> ========== >> While this informal survey is neither precise nor comprehensive, I think >> it is interesting. I queried the NumPy/SciPy and J mailing lists >> because those are the ones that I follow. A more rigorous and broadly >> distributed survey may be in order. >> >> Regards, >> Steve >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Sat Mar 10 17:16:35 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 10 Mar 2007 15:16:35 -0700 Subject: [Numpy-discussion] setmember1d: docstring vs. code In-Reply-To: <45F30683.2080106@gmail.com> References: <45F30683.2080106@gmail.com> Message-ID: On 3/10/07, Robert Kern wrote: > > Jouni K. Sepp?nen wrote: > > > I think that either the docstring (and the book) should be corrected > > to mention the assumption, or the code should be made to work in the > > arbitrary case. > > This is the current docstring: > > In [2]: setmember1d? > Type: function > Base Class: > Namespace: Interactive > File: > /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/numpy- > 1.0.2.dev3569-py2.5-macosx-10.3-fat.egg/numpy/lib/arraysetops.py > Definition: setmember1d(ar1, ar2) > Docstring: > Return a boolean array of shape of ar1 containing True where the > elements > of ar1 are in ar2 and False otherwise. > > Use unique1d() to generate arrays with only unique elements to use as > inputs > to this function. > > :Parameters: > - `ar1` : array > - `ar2` : array > > :Returns: > - `mask` : bool array > The values ar1[mask] are in ar2. > > :See also: > numpy.lib.arraysetops has a number of other functions for performing > set > operations on arrays. > > > I would prefer the latter choice (but perhaps the > > current code has some advantages). > > Well, it has the advantage of existing. If you have an implementation that > is > just as efficient, but works for general arrays, I'd love to see it. Well, the version I posted awhile back works for general arrays. How efficient it is compared to the current version I haven't investigated. For reference, here it is again. def ismember(ar1, ar2) : a = sort(ar2) il = a.searchsorted(ar1, side='left') ir = a.searchsorted(ar1, side='right') return ir != il Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dbecker at alum.dartmouth.org Sun Mar 11 01:02:13 2007 From: dbecker at alum.dartmouth.org (Dan Becker) Date: Sun, 11 Mar 2007 06:02:13 +0000 (UTC) Subject: [Numpy-discussion] A surprising result from benchmarking Message-ID: Hi Everyone, I'm new to numpy, and I'm finding it hard to predict what is fast in python/numpy and what is slow. The following seems puzzling: I am doing the same thing an ugly way and a cleaner way. But the ugly map/lambda/filter expression is 15x faster than using numpy's internals. Can anyone explain why? For now, this makes me nervous about incorporating basic numpy functionality into real programs. ---Code starts here--- import scipy import time import psyco from numpy import matrix print("New run") myMat=scipy.randn(500,500) t1=time.time() highEnough=myMat>0.6 greaterPerLine=[sum(x) for x in highEnough] elapsed1=time.time()-t1 print("method 1 took %f seconds"%elapsed1) t2=time.time() greaterPerLine2=map(lambda(x):len(filter(lambda(y):y>0.6,x)),myMat) elapsed2=time.time()-t2 print("method 2 took %f seconds"%elapsed2) ---Output starts here--- New run method 1 took 3.566760 seconds method 2 took 0.232356 seconds --- Thanks so much! Dan From dbecker at alum.dartmouth.org Sun Mar 11 01:32:10 2007 From: dbecker at alum.dartmouth.org (Dan Becker) Date: Sun, 11 Mar 2007 06:32:10 +0000 (UTC) Subject: [Numpy-discussion] A surprising result from benchmarking References: Message-ID: As soon as I posted that I realized it's due to the type conversions from True to 1. For some reason, this --- myMat=scipy.randn(500,500) t1=time.time() highEnough=(myMat>0.6)+0 greaterPerLine=[sum(x) for x in highEnough] elapsed1=time.time()-t1 print("method 1 took %f seconds"%elapsed1) --- remedies that to some extent. It is only 20% slower than the map. Still, there must be some way for me to make the clean way faster than greaterPerLine2=map(lambda(x):len(filter(lambda(y):y>0.6,x)),myMat) I appreciate any advice on how to do that. Thanks again, Dan From bioinformed at gmail.com Sun Mar 11 01:43:56 2007 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Sun, 11 Mar 2007 01:43:56 -0500 Subject: [Numpy-discussion] A surprising result from benchmarking In-Reply-To: References: Message-ID: <2e1434c10703102243j3ab4282am13c3b972a4299a11@mail.gmail.com> The inefficiency comes in the generic iteration and construction of int objects needed by the builtin sum function. Using the native numarray sum method on each row is much much faster, summing over the axis directly even faster still: t1=time.time() highEnough=myMat>0.6 greaterPerLine=[x.sum() for x in highEnough] elapsed1=time.time()-t1 print("method 1a took %f seconds"%elapsed1) t1=time.time() highEnough=myMat>0.6 greaterPerLine=highEnough.sum(axis=1) elapsed1=time.time()-t1 print("method 1b took %f seconds"%elapsed1) method 1 took 1.503523 seconds method 2 took 0.163641 seconds method 1a took 0.006665 seconds method 1b took 0.004070 seconds -Kevin On 3/11/07, Dan Becker wrote: > > As soon as I posted that I realized it's due to the type conversions from > True > to 1. For some reason, this > > --- > myMat=scipy.randn(500,500) > t1=time.time() > highEnough=(myMat>0.6)+0 > greaterPerLine=[sum(x) for x in highEnough] > elapsed1=time.time()-t1 > print("method 1 took %f seconds"%elapsed1) > --- > > remedies that to some extent. It is only 20% slower than the map. Still, > there > must be some way for me to make the clean way faster than > > greaterPerLine2=map(lambda(x):len(filter(lambda(y):y>0.6,x)),myMat) > > I appreciate any advice on how to do that. > > Thanks again, > Dan > > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at stevesimmons.com Sun Mar 11 04:10:45 2007 From: mail at stevesimmons.com (Stephen Simmons) Date: Sun, 11 Mar 2007 19:10:45 +1100 Subject: [Numpy-discussion] Feedback pls on proposed changes to bincount() In-Reply-To: <2e1434c10703102243j3ab4282am13c3b972a4299a11@mail.gmail.com> References: <2e1434c10703102243j3ab4282am13c3b972a4299a11@mail.gmail.com> Message-ID: <45F3B985.50600@stevesimmons.com> Hi, I'd like to propose some minor modifications to the function bincount(arr, weights=None), so would like some feedback from other uses of bincount() before I write this up as a proper patch, . Background: bincount() has two forms: - bincount(x) returns an integer array ians of length max(x)+1 where ians[n] is the number of times n appears in x. - bincount(x, weights) returns a double array dans of length max(x)+1 where dans[n] is the sum of elements in the weight vector weights[i] at the positions where x[i]==n In both cases, all elements of x must be non-negative. Proposed changes: (1) Remove the restriction that elements of x must be non-negative. Currently bincount() starts by finding max(x) and min(x). If the min value is negative, an exception is raised. This change proposes dropping the initial search for min(x), and instead testing for non-negativity while summing values in the return arrays ians or dans. Any indexes where where x is negative will be silently ignored. This will allow selective bincounts where values to ignore are flagged with a negative bin number. (2) Allow an optional argument for maximum bin number. Currently bincount(x) returns an array whose length is dependent on max(x). It is sometimes preferable to specify the exact size of the returned array, so this change would add a new optional argument, max_bin, which is one less than the size of the returned array. Under this change, bincount() starts by finding max(x) only if max_bin is not specified. Then the returned array ians or dans is created with length max_bin+1, and any indexes that would overflow the output array are silently ignored. (3) Allow an optional output array, y. Currently bincount() creates a new output array each time. Sometimes it is preferable to add results to an existing output array, for example, when the input array is only available in smaller chunks, or for a progressive update strategy to avoid fp precision problems when adding lots of small weights to larger subtotals. Thus we can add an extra optional argument y that bypasses the creation of an output array. With these three change, the function signature of bincount() would become: bincount(x, weights=None, y=None, max_bin=None) Anyway, that's the general idea. I'd be grateful for any feedback before I code this up as a patch to _compiled_base.c. Cheers Stephen From robert.kern at gmail.com Sun Mar 11 22:23:34 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 11 Mar 2007 21:23:34 -0500 Subject: [Numpy-discussion] email notification from the TRAC systems In-Reply-To: References: Message-ID: <45F4B9A6.2000900@gmail.com> Sebastian Haase wrote: > Hi, > Myself and a friend of mine were surprised that we did not get emailed > when someone commented on a bug reports we made on the numpy or scipy > TRAC bug tracker system. > (I started at some point adding an explicit CC entry for myself :-( ) > > I just googled for this, an I found that in a > trac.ini file there should be this option: > > [notification] > always_notify_owner = true > > Why is this (or is it !?) not used in numpy/scipy TRAC system ? It is. Something else is not working. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From steveire at gmail.com Fri Mar 2 11:00:41 2007 From: steveire at gmail.com (Stephen Kelly) Date: Fri, 2 Mar 2007 16:00:41 +0000 Subject: [Numpy-discussion] arrayfns in numpy? Message-ID: <18fbbe5a0703020800h7503ddealfb6682929ff4d4d2@mail.gmail.com> Hi, I'm working on a project that requires interpolation, and I found this post (http://mail.python.org/pipermail/python-list/2000-August/050462.html ) which works fine on my linux box (which seems to have Numeric installed), but does not work on my windows platform because the arrayfns module does not seem to be available on the latest numpy. Is there some other way I should be doing interpolation using numpy, or is the omission of arrayfns an oversight? Kind regards, Stephen. From fsenkel at lynx.neu.edu Fri Mar 2 12:43:49 2007 From: fsenkel at lynx.neu.edu (monkeyboy) Date: Fri, 02 Mar 2007 09:43:49 -0800 Subject: [Numpy-discussion] NumPy in Teaching In-Reply-To: <45E4FFA6.9010408@shrogers.com> References: <45E4FFA6.9010408@shrogers.com> Message-ID: <1172857429.932944.5460@30g2000cwc.googlegroups.com> Although I don't teach, I did use NumPy for coding in a grad level CFD course. I found much nicer to use than MATLAB. As the codes get more complex you can introduce code profiling and inline C or FORTRAN. The Python syntax is so clean, you spend more time on the algorithm, less on the syntax. On Feb 27, 11:05 pm, "Steven H. Rogers" wrote: > I'm doing an informal survey on the use of Array Programming Languages > for teaching. If you're using NumPy in this manner I'd like to hear > from you. What subject was/is taught, academic level, results, lessons > learned, etc. > > Regards, > Steve > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From znevingerek at gmail.com Fri Mar 2 14:56:21 2007 From: znevingerek at gmail.com (nevin) Date: Fri, 02 Mar 2007 19:56:21 -0000 Subject: [Numpy-discussion] problem with installing scipy Message-ID: <1172865381.096764.136950@j27g2000cwj.googlegroups.com> I am trying to use scipy optimize module but I am having problem when I try to import it: >>> import numpy >>> import scipy >>> from scipy import optimize Traceback (most recent call last): File "", line 1, in ? File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/optimize/__init__.py", line 7, in ? from optimize import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/optimize/optimize.py", line 25, in ? import linesearch File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy/optimize/linesearch.py", line 3, in ? import minpack2 ImportError: Inappropriate file type for dynamic loading How can I fix this problem? My system is Mac OSX Tiger- Pentium. Thanks. From Johanne.Duhaime at ircm.qc.ca Tue Mar 6 11:35:28 2007 From: Johanne.Duhaime at ircm.qc.ca (Duhaime Johanne) Date: Tue, 6 Mar 2007 11:35:28 -0500 Subject: [Numpy-discussion] compiling numarrray on Cygwin Message-ID: <96C071542ED58D49BC08210D3456D5802EC011@pandore.ircm.priv> I have problem to install numarray on cygwin. I have seen emails on that topic but I finally do not have a solution. Below is the error message for numarray-1.5.2 on python 2.4.3. I am using numarray instead of numpy because I need the module to be compatible with a software : MAT (Model-based Analysis of Tiling-array). I have seen some solutions but I do not know exactly how to do since I am not familiar with python and cygwin. Someone suggests to disable the package that contains fetestexcept and fetestexcept. I do not know if this is important for my software but I am willing to try. The problem is that I do not know how to disable this package... The same person mention that it would put a "solution" in the csv repository. If I am right there is no csv repository for numarray. Should I look to the python csv. In fact I did that, but have not seen any comment related to my problem. As you can see my questions are very basic. Sorry about that. But I would really appreciate help to solve that problem. Thank you in advance for any help. -----------------Compile: Building many extension but then with 'numarray.libnumarray' I have an error. (...) building 'numarray._ufuncComplex32' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -IInclude/nu array -I/usr/include/python2.4 -c Src/_ufuncComplex32module.c -o build/temp.cyg in-1.5.24-i686-2.4/Src/_ufuncComplex32module.o gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.24-i686-2.4/Src/ ufuncComplex32module.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cyg in-1.5.24-i686-2.4/numarray/_ufuncComplex32.dll -lm -L/lib -lm -lc -lgcc -L/lib mingw -lmingwex building 'numarray._ufuncComplex64' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -IInclude/nu array -I/usr/include/python2.4 -c Src/_ufuncComplex64module.c -o build/temp.cyg in-1.5.24-i686-2.4/Src/_ufuncComplex64module.o gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.24-i686-2.4/Src/ ufuncComplex64module.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cyg in-1.5.24-i686-2.4/numarray/_ufuncComplex64.dll -lm -L/lib -lm -lc -lgcc -L/lib mingw -lmingwex building 'numarray.libnumarray' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -IInclude/num array -I/usr/include/python2.4 -c Src/libnumarraymodule.c -o build/temp.cygwin-1 .5.24-i686-2.4/Src/libnumarraymodule.o gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.24-i686-2.4/Src/l ibnumarraymodule.o -L/usr/lib/python2.4/config -lpython2.4 -o build/lib.cygwin-1 .5.24-i686-2.4/numarray/libnumarray.dll -lm -L/lib -lm -lc -lgcc -L/lib/mingw -l mingwex /lib/mingw/libmingwex.a(feclearexcept.o):feclearexcept.c:(.text+0x21): undefined reference to `___cpu_features' /lib/mingw/libmingwex.a(fetestexcept.o):fetestexcept.c:(.text+0x7): undefined re ference to `___cpu_features' collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 duhaimj at ircm.qc.ca From roberto at dealmeida.net Thu Mar 8 13:49:13 2007 From: roberto at dealmeida.net (Roberto De Almeida) Date: Thu, 8 Mar 2007 15:49:13 -0300 Subject: [Numpy-discussion] Problem compiling numpy with Python 2.5 on Powerbook Message-ID: <10c662fe0703081049o490aa22fsa8d1229ef2f038f3@mail.gmail.com> Hi, guys. I'm trying to install numpy from subversion on my Powerbook 12", from a fresh install (Mac OS X 10.4.8). I downloaded the official universal Python 2.5 from python.org, and installed GCC (powerpc-apple-darwin8-gcc-4.0.0) from XCode. Installation fails with the message: gcc: installation problem, cannot exec 'i686-apple-darwin8-gcc-4.0.0': No such file or directory (The full log is attached.) Numpy 1.0.1 also fails. Any hints? Could this be because I'm using the universal binary for Python? Thanks, --Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: inst.log Type: application/octet-stream Size: 510186 bytes Desc: not available URL: From brown at northwestern.edu Fri Mar 9 09:32:38 2007 From: brown at northwestern.edu (Eric Brown) Date: Fri, 9 Mar 2007 08:32:38 -0600 Subject: [Numpy-discussion] Vectorized ufuncs with OS X frameworks Message-ID: Hi All, I have a set of large arrays to which I have to do exp functions over and over again. I am wondering if there is any benefit to teaching numpy how to use the vForce frameworks to do this function. Does anyone have experience hacking the source to get numpy to use system libraries to do vectorized versions of sqrt, exp, log, addition, etc.? I am also interested in doing this for other operating systems. Best regards, Eric From beliavsky at aol.com Sat Mar 10 18:56:55 2007 From: beliavsky at aol.com (Beliavsky) Date: Sat, 10 Mar 2007 15:56:55 -0800 Subject: [Numpy-discussion] Teaching Array Languages In-Reply-To: <45F3154A.5070808@shrogers.com> References: <45F3154A.5070808@shrogers.com> Message-ID: <1173571015.437737.174570@30g2000cwc.googlegroups.com> On Mar 10, 3:30 pm, "Steven H. Rogers" wrote: > Thanks to all who responded to my question about teaching array > programming. I've compiled a brief summary of the responses. Fortran 90 and later versions of the language can be used to teach "array programming", since the language allows operations on whole arrays and array sections. You could survey people on comp.lang.fortran comp-fortran-90 http://www.jiscmail.ac.uk/lists/comp-fortran-90.html gg95 Google Group http://groups.google.com/group/gg95 From beliavsky at aol.com Sat Mar 10 18:59:59 2007 From: beliavsky at aol.com (Beliavsky) Date: Sat, 10 Mar 2007 15:59:59 -0800 Subject: [Numpy-discussion] Teaching Array Languages In-Reply-To: <45F3154A.5070808@shrogers.com> References: <45F3154A.5070808@shrogers.com> Message-ID: <1173571199.085750.327160@s48g2000cws.googlegroups.com> On Mar 10, 3:30 pm, "Steven H. Rogers" wrote: > Thanks to all who responded to my question about teaching array > programming. I've compiled a brief summary of the responses. Fortran 90 and 95 can and are being used to teach "array programming", since the language allows operations on whole arrays and array sections. You could survey people on comp.lang.fortran comp-fortran-90 http://www.jiscmail.ac.uk/lists/comp-fortran-90.html gg95 Google Group http://groups.google.com/group/gg95 From shyu at yahoo.com Sun Mar 11 12:10:45 2007 From: shyu at yahoo.com (Haw-Jye Shyu) Date: Sun, 11 Mar 2007 09:10:45 -0700 (PDT) Subject: [Numpy-discussion] No module named Numeric for numpy-1.0.1.win32-py2.5.exe on Python 2.5 Message-ID: <298178.28569.qm@web52101.mail.yahoo.com> Hi folks: How are you? I am a newbie to the Numpy. Today, I have downloaded numpy-1.0.1.win32-py2.5.exe from the web site of http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 I double clicked on the numpy-1.0.1.win32-py2.5.exe and the installation seems to run smoothly, seeing the installation winidows saying "Click the finish button to exit the setup wizard". And yet, when I type in import Numric in the window of Python Shell, I got this follow message:ImportError: No module named Numeric. It said I did not install the Numpy. Following is a screen shot from my Python Shell. Please help. Your help is highly appreciated. Sincerely, Howard Shyu ================================================================================================== Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.2 >>> import Numeric Traceback (most recent call last): File "", line 1, in import Numeric ImportError: No module named Numeric >>> ____________________________________________________________________________________ It's here! Your new message! Get new email alerts with the free Yahoo! Toolbar. http://tools.search.yahoo.com/toolbar/features/mail/ From shyu at yahoo.com Sun Mar 11 12:20:57 2007 From: shyu at yahoo.com (Haw-Jye Shyu) Date: Sun, 11 Mar 2007 09:20:57 -0700 (PDT) Subject: [Numpy-discussion] Error in running the numpy\setup.py numpy-1.0.1.win32-py2.5.exe with python 2.5 Message-ID: <777255.18407.qm@web52108.mail.yahoo.com> Hi Folks: On the second try, I use the Python Shell to open a file named C:\Python25\Lib\site-package\numpy\setup.py and run it on the Pythono Shell. I got the following message: Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.2 >>> ================================ RESTART ================================ >>> ======================================================================== Starting interactive session ------------------------------------------------------------------------ Tasks: i - Show python/platform/machine information ie - Show environment information c - Show C compilers information c - Set C compiler (current:None) f - Show Fortran compilers information f - Set Fortran compiler (current:None) e - Edit proposed sys.argv[1:]. Task aliases: 0 - Configure 1 - Build 2 - Install 2 - Install with prefix. 3 - Inplace build 4 - Source distribution 5 - Binary distribution Proposed sys.argv = ['C:\\Python25\\Lib\\site-packages\\numpy\\setup.py'] Choose a task (^D to quit, Enter to continue with setup): 2 Tasks: i - Show python/platform/machine information ie - Show environment information c - Show C compilers information c - Set C compiler (current:None) f - Show Fortran compilers information f - Set Fortran compiler (current:None) e - Edit proposed sys.argv[1:]. Task aliases: 0 - Configure 1 - Build 2 - Install 2 - Install with prefix. 3 - Inplace build 4 - Source distribution 5 - Binary distribution Proposed sys.argv = ['C:\\Python25\\Lib\\site-packages\\numpy\\setup.py', 'install'] Choose a task (^D to quit, Enter to continue with setup): ------------------------------------------------------------------------ Warning: Assuming default configuration (distutils\command/{setup_command,setup}.py was not found) Appending numpy.distutils.command configuration to numpy.distutils Ignoring attempt to set 'name' (from 'numpy.distutils' to 'numpy.distutils.command') Warning: Assuming default configuration (distutils\fcompiler/{setup_fcompiler,setup}.py was not found) Appending numpy.distutils.fcompiler configuration to numpy.distutils Ignoring attempt to set 'name' (from 'numpy.distutils' to 'numpy.distutils.fcompiler') non-existing path in 'distutils': 'site.cfg' Appending numpy.distutils configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.distutils') Appending numpy.testing configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.testing') Warning: Assuming default configuration (f2py\lib\parser/{setup_parser,setup}.py was not found) Appending numpy.f2py.lib.parser configuration to numpy.f2py.lib Ignoring attempt to set 'name' (from 'numpy.f2py.lib' to 'numpy.f2py.lib.parser') Appending numpy.f2py.lib configuration to numpy.f2py Ignoring attempt to set 'name' (from 'numpy.f2py' to 'numpy.f2py.lib') F2PY Version 2_3473 Appending numpy.f2py configuration to numpy Ignoring attempt to set 'name' (from 'numpy' to 'numpy.f2py') Ignoring attempt to set 'version' (from None to '2_3473') Traceback (most recent call last): File "C:\Python25\Lib\site-packages\numpy\setup.py", line 29, in setup(configuration=configuration) File "C:\Python25\Lib\site-packages\numpy\distutils\core.py", line 112, in setup return setup(**attr) File "C:\Python25\Lib\site-packages\numpy\distutils\core.py", line 144, in setup config = configuration() File "C:\Python25\Lib\site-packages\numpy\setup.py", line 9, in configuration config.add_subpackage('core') File "C:\Python25\Lib\site-packages\numpy\distutils\misc_util.py", line 765, in add_subpackage caller_level = 2) File "C:\Python25\Lib\site-packages\numpy\distutils\misc_util.py", line 748, in get_subpackage caller_level = caller_level + 1) File "C:\Python25\Lib\site-packages\numpy\distutils\misc_util.py", line 695, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "C:\Python25\Lib\site-packages\numpy\core\setup.py", line 33, in configuration open(generate_umath_py,'U'),generate_umath_py, IOError: [Errno 2] No such file or directory: 'core\\code_generators\\generate_umath.py' >>> ____________________________________________________________________________________ No need to miss a message. Get email on-the-go with Yahoo! Mail for Mobile. Get started. http://mobile.yahoo.com/mail From oliphant at ee.byu.edu Mon Mar 12 12:33:13 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 12 Mar 2007 11:33:13 -0500 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F2E725.9050901@cc.usu.edu> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> Message-ID: <45F580C9.70206@ee.byu.edu> Mark P. Miller wrote: >Robert: Just a thought on this topic: > >Would it be possible for the Scipy folks to add a new module based >solely off your old mtrand code (pre-broadcast)? I have to say that the >mtrand code from numpy 0.9.8 has some excellent advantages over the core >python random number generators. > > I'm not convinced that the broadcasting is causing the slow-downs. Currently, the code has two path-ways. One gets called when the inputs are scalars which is equivalent to the old code and the second gets called when broadcasting is desired. Their should be only a very small speed hit because of the check. So, it should be possible to do both without great differences in speed. Perhaps the speed hit is due to something else (like creating 0-d arrays from scalars, or something similar). So, in other words, I think the problem is fixable, but we need someone who can track down where the slowness is coming from. -Travis From hetland at tamu.edu Mon Mar 12 15:29:31 2007 From: hetland at tamu.edu (Rob Hetland) Date: Mon, 12 Mar 2007 14:29:31 -0500 Subject: [Numpy-discussion] Putting some random back into the top-level? In-Reply-To: References: Message-ID: I, for one, would also like this. Perhaps it should not be called 'rand', however, as that conflicts with the pylab rand. (numpy load and pylab load also conflict -- probably the only reason I ever use import pylab as pl in my scripts). 'random' is already taken by the whole package... What does this leave that is still sensible? -r On Mar 9, 2007, at 2:01 PM, Bill Baxter wrote: > Has enough time passed with no top level random function that we can > now put one back in? > If I recall, the former top level rand() was basically removed because > it didn't adhere to the "shapes are always tuples" convention. > > Has enough time passed now that we can put something like it back in > the top level, in tuple-taking form? > > I think this is a function people use pretty frequently when writing > quick tests. > And numpy.random.random_sample seems a rather long and not so obvious > name for something that is used relatively frequently. > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion ---- Rob Hetland, Associate Professor Dept. of Oceanography, Texas A&M University http://pong.tamu.edu/~rob phone: 979-458-0096, fax: 979-845-6331 From Glen.Mabey at swri.org Mon Mar 12 16:20:55 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Mon, 12 Mar 2007 15:20:55 -0500 Subject: [Numpy-discussion] cannot pickle large numpy objects when memory resources are already stressed Message-ID: <20070312202055.GE9245@bams.ccf.swri.edu> Hello, After running a simulation that took 6 days to complete, my script proceeded to attempt to write the results out to a file, pickled. The operation failed even though there was 1G of RAM free (4G machine). I've since reconsidered using the pickle format for storing data sets that include large numpy arrays. However, somehow I assumed that one would be able to pickle anything that you already had in memory, but I see now that this was a rash assumption. Ought there to be a way to do this, or should I forget about being able to bundle large numpy arrays and other objects in a single pickle? Thanks, Glen (these commands performed on a different machine with 1G RAM) In [10]: za = numpy.zeros( (100000000,), dtype=numpy.float32 ) In [11]: import cPickle In [12]: zfile = file( '/tmp/zfile', 'w' ) In [13]: cPickle.dump( za, zfile ) --------------------------------------------------------------------------- exceptions.MemoryError Traceback (most recent call last) From jstrunk at enthought.com Wed Mar 14 00:01:49 2007 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 13 Mar 2007 23:01:49 -0500 Subject: [Numpy-discussion] network outage Message-ID: <200703132301.49106.jstrunk@enthought.com> Good evening, Earlier this evening we had a network outage due to a network equipment malfunction. This outage prevented access to the Enthought and SciPy servers from about 8:45-10pm CDT. I have fixed the problem and everything should be back to normal. I apologize for the inconvenience. Jeff Strunk IT Administrator Enthought, Inc. From ceball at gmail.com Tue Mar 13 05:10:04 2007 From: ceball at gmail.com (Christopher Ball) Date: Tue, 13 Mar 2007 17:10:04 +0800 Subject: [Numpy-discussion] 0/0 is 0 or nan? In-Reply-To: <4e2a3a120703092000g1f9cf149x3fd056daf9c7bd2@mail.gmail.com> References: <4e2a3a120703092000g1f9cf149x3fd056daf9c7bd2@mail.gmail.com> Message-ID: <45F66A6C.8050602@gmail.com> > Thank you both for your replies - the difference is clear to me now. Actually, sorry, I'm still confused! If I want to be able to have a/0 be inf (for a!=0), then why does that stop 0/0 from being nan? In Python, a/0 gives a divide-by-zero error for any a, but in numpy, I can ignore divide-by-zero errors. If I do this, why can't I still have 0/0 be nan? seterr(divide='ignore') a=array([0],dtype=float_) b=array([0],dtype=float_) divide(a,b) gives nan, so why doesn't seterr(divide='ignore') a=array([0],dtype=int_) b=array([0],dtype=int_) divide(a,b) also give nan, rather than 0 as it gives now? Is this because inf is not something to which an int can be set (as Christopher Barker wrote), and it matches Python's behavior? I'm not at all knowledgeable in this area - I'm not requesting any changes, just some more explanation. Thanks again, Chris From robert.kern at gmail.com Wed Mar 14 00:24:42 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 13 Mar 2007 23:24:42 -0500 Subject: [Numpy-discussion] No module named Numeric for numpy-1.0.1.win32-py2.5.exe on Python 2.5 In-Reply-To: <298178.28569.qm@web52101.mail.yahoo.com> References: <298178.28569.qm@web52101.mail.yahoo.com> Message-ID: <45F7790A.1040601@gmail.com> Haw-Jye Shyu wrote: > Hi folks: > > How are you? > I am a newbie to the Numpy. Today, I have downloaded numpy-1.0.1.win32-py2.5.exe > from the web site of > http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > I double clicked on the numpy-1.0.1.win32-py2.5.exe and the installation seems to run smoothly, > seeing the installation winidows saying "Click the finish button to exit the setup wizard". > > And yet, when I type in import Numric in the window of Python Shell, I got this follow > message:ImportError: No module named Numeric. It said I did not install the Numpy. > Following is a screen shot from my Python Shell. Please help. Your help is highly appreciated. The current package "numpy" is the replacement for "Numeric". Unfortunately, "NumPy" was a sort of nickname for "Numeric". There is more detail on that history here: http://www.scipy.org/History_of_SciPy You can find up-to-date documentation on numpy here: http://www.scipy.org/Documentation -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Mar 14 00:25:43 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 13 Mar 2007 23:25:43 -0500 Subject: [Numpy-discussion] Error in running the numpy\setup.py numpy-1.0.1.win32-py2.5.exe with python 2.5 In-Reply-To: <777255.18407.qm@web52108.mail.yahoo.com> References: <777255.18407.qm@web52108.mail.yahoo.com> Message-ID: <45F77947.7060000@gmail.com> Haw-Jye Shyu wrote: > Hi Folks: > > On the second try, I use the Python Shell to open a file named > C:\Python25\Lib\site-package\numpy\setup.py > > and run it on the Pythono Shell. Don't. It's not useful once installed. It's just for building from source. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Mar 14 00:30:17 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 13 Mar 2007 23:30:17 -0500 Subject: [Numpy-discussion] 0/0 is 0 or nan? In-Reply-To: <45F66A6C.8050602@gmail.com> References: <4e2a3a120703092000g1f9cf149x3fd056daf9c7bd2@mail.gmail.com> <45F66A6C.8050602@gmail.com> Message-ID: <45F77A59.60505@gmail.com> Christopher Ball wrote: > In Python, a/0 gives a divide-by-zero error for any a, but in numpy, I > can ignore divide-by-zero errors. If I do this, why can't I still have > 0/0 be nan? > > seterr(divide='ignore') > a=array([0],dtype=float_) > b=array([0],dtype=float_) > divide(a,b) > > gives nan, so why doesn't > > seterr(divide='ignore') > a=array([0],dtype=int_) > b=array([0],dtype=int_) > divide(a,b) > > also give nan, rather than 0 as it gives now? Is this because inf is not > something to which an int can be set (as Christopher Barker wrote), and > it matches Python's behavior? Pretty much. ints operated with ints will give ints, not floats. nan and inf are floating point concepts. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From mail at stevesimmons.com Wed Mar 14 00:52:07 2007 From: mail at stevesimmons.com (Stephen Simmons) Date: Wed, 14 Mar 2007 15:52:07 +1100 Subject: [Numpy-discussion] 3-10x speedup in bincount() In-Reply-To: <45F3B985.50600@stevesimmons.com> References: <2e1434c10703102243j3ab4282am13c3b972a4299a11@mail.gmail.com> <45F3B985.50600@stevesimmons.com> Message-ID: <45F77F77.5030406@stevesimmons.com> Well, there were no responses to my earlier email proposing changes to numpy.bincount() to make it faster and more flexible. Does this mean noone is using bincount? :-) Anyway I've now made the proposed modifications and got substantial speedups of 3-10. Details are in this extract from my C code. For the time being I will test this as a standalone extension module. When stable and tests are written, I'll submit as a patch to the numpy's _compiled_base.c. Cheers, Stephen /************************************************ * Faster versions of bincount and casting strings to integers * * Author: Stephen Simmons, mail at stevesimmons.com * Date: 11 March 2007 * * This module contains C code for functions I am using to accelerate * SQL-like aggregate functions for a column-oriented database based on numpy. * * subtotal's bincount is typically 3-10 times faster than numpy's bincount, * and more flexible * - takes bins as they are, without promoting their type to int32 * - takes weights as they are, without promoting their type to double * - ignores negative bins, rather than reporting an error * - allows optional int32/double output array to be passed in * - specify maximum bin number to use. Larger bins numbers are ignored * - only scans for max bin number if neither max_bin nor out array specified * * atoi is 30-60 times faster than casting a string array to an integer type, * and may optionally use a translation table. The translation table is * a convenient way to map sparse integer strings to adjacent bins before * using bincount. * * Typical usage * ------------- # Set up arrays 5,000,000 elts long. s1 has strings filled with '1' and '2'. # w are weights >>> s1 = numpy.ndarray(shape=5000000, dtype='S1'); s1[:]='2'; s1[1::2]='1' >>> w = numpy.arange(5000000,dtype='f4') # Using standard numpy functions, string conversion is slow >>> i = s1.astype('i1') -> 5.95 sec (!) >>> numpy.bincount(i) -> 113 ms >>> numpy.bincount(i, w) -> 272 ms >>> numpy.bincount(s1.astype('i1'), w) -> 6.12 sec # Using the faster functions here: >>> i = subtotal.atoi(s1) -> 90 ms (60x faster) >>> subtotal.bincount(i, max_bin=2) -> 31 ms (3.6x faster) >>> subtotal.bincount(i, w, max_bin=2) -> 51 ms (5.3x faster) # In both bases, output is array([2, 1, 2, ..., 1, 2, 1], dtype=int8) array([ 0, 2500000, 2500000]) array([ 0.00000000e+00, 6.25000000e+12, 6.24999750e+12]) # As an example of using a translate table, run bincount from atoi applying # a translate table for counting odd vs even strings. This does the whole lot # in 158ms, a speedup of 38x from 6.12 s above. >>> t = numpy.arange(256, dtype='i1') % 2 >>> subtotal.bincount(subtotal.atoi(s1, t), w, max_bin=t.max()) -> 158 ms array([ 6.24999750e+12, 6.25000000e+12]) * * These timings are based on numpy-1.0.1 Windows binary distribution * for Python 2.5, compared with building this code using standard distutils * without any particular compiler optimisations: C:\mnt\dev\subtotal> python setup.py build -cmingw32 * Processor is a 1.83GHz Pentium-M ****************************************************/ << rest of C code omitted >> Stephen Simmons wrote: > Hi, > > I'd like to propose some minor modifications to the function > bincount(arr, weights=None), so would like some feedback from other > uses of bincount() before I write this up as a proper patch, . > > Background: > bincount() has two forms: > - bincount(x) returns an integer array ians of length max(x)+1 where > ians[n] is the number of times n appears in x. > - bincount(x, weights) returns a double array dans of length max(x)+1 > where dans[n] is the sum of elements in the weight vector weights[i] > at the positions where x[i]==n > In both cases, all elements of x must be non-negative. > > Proposed changes: > (1) Remove the restriction that elements of x must be non-negative. > Currently bincount() starts by finding max(x) and min(x). If the min > value is negative, an exception is raised. This change proposes > dropping the initial search for min(x), and instead testing for > non-negativity while summing values in the return arrays ians or dans. > Any indexes where where x is negative will be silently ignored. This > will allow selective bincounts where values to ignore are flagged with > a negative bin number. > > (2) Allow an optional argument for maximum bin number. > Currently bincount(x) returns an array whose length is dependent on > max(x). It is sometimes preferable to specify the exact size of the > returned array, so this change would add a new optional argument, > max_bin, which is one less than the size of the returned array. Under > this change, bincount() starts by finding max(x) only if max_bin is > not specified. Then the returned array ians or dans is created with > length max_bin+1, and any indexes that would overflow the output array > are silently ignored. > > (3) Allow an optional output array, y. > Currently bincount() creates a new output array each time. Sometimes > it is preferable to add results to an existing output array, for > example, when the input array is only available in smaller chunks, or > for a progressive update strategy to avoid fp precision problems when > adding lots of small weights to larger subtotals. Thus we can add an > extra optional argument y that bypasses the creation of an output array. > > With these three change, the function signature of bincount() would > become: > bincount(x, weights=None, y=None, max_bin=None) > > > Anyway, that's the general idea. I'd be grateful for any feedback > before I code this up as a patch to _compiled_base.c. > > Cheers > > Stephen > > From lbolla at gmail.com Tue Mar 13 12:09:40 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Tue, 13 Mar 2007 17:09:40 +0100 Subject: [Numpy-discussion] efficient norm of a vector Message-ID: <80c99e790703130909l2cb7bd02r11bf80ebccf443d5@mail.gmail.com> Hi all, just a quick (and easy?) question. what is the best (fastest) way to implement the euclidean norm of a vector, i.e. the function: import scipy as S def norm(x): """normalize a vector.""" return S.sqrt(S.sum(S.absolute(x)**2)) ? thanks in advance, Lorenzo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists.steve at arachnedesign.net Wed Mar 14 01:13:24 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 14 Mar 2007 01:13:24 -0400 Subject: [Numpy-discussion] Problem compiling numpy with Python 2.5 on Powerbook In-Reply-To: <10c662fe0703081049o490aa22fsa8d1229ef2f038f3@mail.gmail.com> References: <10c662fe0703081049o490aa22fsa8d1229ef2f038f3@mail.gmail.com> Message-ID: <870C81DE-4220-430B-827D-A3B2FBE099ED@arachnedesign.net> Hi, > Hi, guys. > > I'm trying to install numpy from subversion on my Powerbook 12", from > a fresh install (Mac OS X 10.4.8). I downloaded the official universal > Python 2.5 from python.org, and installed GCC > (powerpc-apple-darwin8-gcc-4.0.0) from XCode. > > Installation fails with the message: > > gcc: installation problem, cannot exec > 'i686-apple-darwin8-gcc-4.0.0': No such file or directory I'm not sure what the problem is exactly, but is it weird that there's something to do w/ 'i686' when you're running on a powerbook being that the pbook is PowerPC? I noticed you also have this in your log: C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/ MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 (it's the i386 that looks weird to me) I don't really know if that's a problem, though -- how are you trying to install it, are you just doing a "python setup.py build" and "python setup.py install"? FWIW, my version of XCode gcc is actually 4.0.1 -- perhaps there's a newer version of XCode for you to install? I am on an intel mac, though ... -steve From wbaxter at gmail.com Wed Mar 14 01:49:00 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 14 Mar 2007 14:49:00 +0900 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib Message-ID: Howdy Folks, I was missing the good ole days of using Matlab back at the Uni when I could debug my code, stop at breakpoints and plot various data without fear of blocking the interpreter process. Using "ipython -pylab" is what has been suggested to me in the past, but the problem is I don't do my debugging from ipython. I have a very nice IDE that works very well, and it has a lovely interactive debugging prompt that I can use to probe my code when stopped at a breakpoint. It's great except I can't really use matplotlib for debugging there because it causes things to freeze up. So I've come up with a decent (though not perfect) solution for quickie interactive plots which is to run matplotlib in a separate process. I call the result it 'ezplot'. The first alpha version of this is now available at the Cheeseshop. (I made an egg too, so if you have setuptools you can do "easy_install ezplot".) The basic usage is like so: In [1]: import ezplot In [2]: p = ezplot.Plotter() In [3]: p.plot([1,2,3],[1,4,9],marker='o') Connecting to server... waiting... connected to plotserver 0.1.0a1 on http://localhost:8397 Out[3]: True In [4]: from numpy import * In [5]: x = linspace(-5,5,20) In [13]: p.clf() Out[13]: True In [14]: p.plot(x, x*x*log(x*x+0.01)) (Imagine lovely plots popping up on your screen as these commands are typed.) The only return values you get back are True (success...probably) or False (failure...for sure). So no fancy plot object manipulation is possible. But you can do basic plots no problem. The nice part is that this (unlike ipython's built-in -pylab threading mojo) should work just as well from wherever you're using python. Whether it's ipython (no -pylab) or Idle, or a plain MS-DOS console, or WingIDE's debug probe, or SPE, or a PyCrust shell or whatever. It doesn't matter because all the client is doing is packing up data and shipping over a socket. All the GUI plotting mojo happens in a completely separate process. There are plenty of ways this could be made better, but for me, for now, this probably does pretty much all I need, so it's back to Real Work. But if anyone is interested in making improvements to this, let me know. Here's a short list of things that could be improved: * Right now I assume use of the wxAGG backend for matplotlib. Don't know how much work it would be to support other back ends (or how to go about it, really). wxAGG is what I always use. * Returning more error/exception info from the server would be nice * Returning full fledged proxy plot objects would be nice too, but I suspect that's a huge effort * SOAP may be better for this than xmlrpclib but I just couldn't get it to work (SOAPpy + Twisted). * A little more safety would be nice. Anyone know how to make a Twisted xmlrpc server not accept connections from anywhere except localhost? * There's a little glitch in that the spawned plot server dies with the parent that created it. Maybe there's a flag to subprocess.Popen to fix that? * Sometimes when you click on "Exit Server", if there are plot windows open it hangs while shutting down. Only tested on Win32 but there's nothing much platform specific in there. Give it a try and let me know what you think! --bb From wbaxter at gmail.com Wed Mar 14 01:54:57 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 14 Mar 2007 14:54:57 +0900 Subject: [Numpy-discussion] efficient norm of a vector In-Reply-To: <80c99e790703130909l2cb7bd02r11bf80ebccf443d5@mail.gmail.com> References: <80c99e790703130909l2cb7bd02r11bf80ebccf443d5@mail.gmail.com> Message-ID: There is numpy.linalg.norm. Here's what it does: def norm(x, ord=None): x = asarray(x) nd = len(x.shape) if ord is None: # check the default case first and handle it immediately return sqrt(add.reduce((x.conj() * x).ravel().real)) if nd == 1: if ord == Inf: return abs(x).max() elif ord == -Inf: return abs(x).min() elif ord == 1: return abs(x).sum() # special case for speedup elif ord == 2: return sqrt(((x.conj()*x).real).sum()) # special case for speedup else: return ((abs(x)**ord).sum())**(1.0/ord) elif nd == 2: if ord == 2: return svd(x, compute_uv=0).max() elif ord == -2: return svd(x, compute_uv=0).min() elif ord == 1: return abs(x).sum(axis=0).max() elif ord == Inf: return abs(x).sum(axis=1).max() elif ord == -1: return abs(x).sum(axis=0).min() elif ord == -Inf: return abs(x).sum(axis=1).min() elif ord in ['fro','f']: return sqrt(add.reduce((x.conj() * x).real.ravel())) else: raise ValueError, "Invalid norm order for matrices." else: raise ValueError, "Improper number of dimensions to norm." --bb On 3/14/07, lorenzo bolla wrote: > Hi all, > just a quick (and easy?) question. > what is the best (fastest) way to implement the euclidean norm of a vector, > i.e. the function: > > import scipy as S > def norm(x): > """normalize a vector.""" > return S.sqrt(S.sum(S.absolute(x)**2)) > > ? > > thanks in advance, > Lorenzo. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Wed Mar 14 03:07:56 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Wed, 14 Mar 2007 08:07:56 +0100 Subject: [Numpy-discussion] arrayfns in numpy? In-Reply-To: <18fbbe5a0703020800h7503ddealfb6682929ff4d4d2@mail.gmail.com> References: <18fbbe5a0703020800h7503ddealfb6682929ff4d4d2@mail.gmail.com> Message-ID: On the scipy user list, this exac question appeared last month, so yu can check the answers on the archive :) Matthieu 2007/3/2, Stephen Kelly : > > Hi, > > I'm working on a project that requires interpolation, and I found this > post (http://mail.python.org/pipermail/python-list/2000-August/050462.html > ) which works fine on my linux box (which seems to have Numeric > installed), but does not work on my windows platform because the > arrayfns module does not seem to be available on the latest numpy. > > Is there some other way I should be doing interpolation using numpy, > or is the omission of arrayfns an oversight? > > Kind regards, > > Stephen. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lbolla at gmail.com Wed Mar 14 04:21:58 2007 From: lbolla at gmail.com (lorenzo bolla) Date: Wed, 14 Mar 2007 09:21:58 +0100 Subject: [Numpy-discussion] efficient norm of a vector In-Reply-To: References: <80c99e790703130909l2cb7bd02r11bf80ebccf443d5@mail.gmail.com> Message-ID: <80c99e790703140121s68b1c2c3mb7dddcf882d79266@mail.gmail.com> thanks. I hadn't seen it. anyway, from very rough benchmarks I did, the quickest and easiest way of computing the euclidean norm of a 1D array is: n = sqrt(dot(x,x.conj())) much faster than: n = sqrt(sum(abs(x)**2)) and much much faster than: n = scipy.linalg.norm(x) regards, lorenzo. On 3/14/07, Bill Baxter wrote: > > There is numpy.linalg.norm. > > Here's what it does: > > def norm(x, ord=None): > x = asarray(x) > nd = len(x.shape) > if ord is None: # check the default case first and handle it > immediately > return sqrt(add.reduce((x.conj() * x).ravel().real)) > if nd == 1: > if ord == Inf: > return abs(x).max() > elif ord == -Inf: > return abs(x).min() > elif ord == 1: > return abs(x).sum() # special case for speedup > elif ord == 2: > return sqrt(((x.conj()*x).real).sum()) # special case for > speedup > else: > return ((abs(x)**ord).sum())**(1.0/ord) > elif nd == 2: > if ord == 2: > return svd(x, compute_uv=0).max() > elif ord == -2: > return svd(x, compute_uv=0).min() > elif ord == 1: > return abs(x).sum(axis=0).max() > elif ord == Inf: > return abs(x).sum(axis=1).max() > elif ord == -1: > return abs(x).sum(axis=0).min() > elif ord == -Inf: > return abs(x).sum(axis=1).min() > elif ord in ['fro','f']: > return sqrt(add.reduce((x.conj() * x).real.ravel())) > else: > raise ValueError, "Invalid norm order for matrices." > else: > raise ValueError, "Improper number of dimensions to norm." > > > > --bb > > > > On 3/14/07, lorenzo bolla wrote: > > Hi all, > > just a quick (and easy?) question. > > what is the best (fastest) way to implement the euclidean norm of a > vector, > > i.e. the function: > > > > import scipy as S > > def norm(x): > > """normalize a vector.""" > > return S.sqrt(S.sum(S.absolute(x)**2)) > > > > ? > > > > thanks in advance, > > Lorenzo. > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Tue Mar 13 04:55:40 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 13 Mar 2007 10:55:40 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <45F1F162.2030102@gemini.edu> References: <45F1F162.2030102@gemini.edu> Message-ID: <20070313085540.GJ15580@mentat.za.net> Hi James On Fri, Mar 09, 2007 at 08:44:34PM -0300, James Turner wrote: > Last year I wrote a program that uses the affine_transform() > function in numarray to resample and co-add datacubes with WCS > offsets in 3D. This function makes it relatively easy to handle > N-D offsets and rotations with a smooth interpolant, which is > exactly what I wanted. However, I am finding that the results > contain edge effects which translate into artefacts in the > middle of the final mosaic. Is this related to http://projects.scipy.org/scipy/scipy/ticket/213 in any way? Code snippets to illustrate the problem would be welcome. Regards St?fan From faltet at carabos.com Wed Mar 14 08:02:10 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 14 Mar 2007 13:02:10 +0100 Subject: [Numpy-discussion] Which dtype are supported by numexpr ? In-Reply-To: <200703091856.30141.faltet@carabos.com> References: <200703091856.30141.faltet@carabos.com> Message-ID: <200703141302.13084.faltet@carabos.com> A Divendres 09 Mar? 2007 18:56, Francesc Altet escrigu?: > A Divendres 09 Mar? 2007 18:40, Sebastian Haase escrigu?: > > Which dtypes are supported by numexpr ? > > Well, numexpr does support any dtype that is homogeneous, except 'uint64'. > This is because internally all the unsigned types are upcasted to the > smallest *signed* integer that can fit the info for it. As it happens > that 'uint64' doesn't have a standard signed type above that is able to > keep its info: this is why it is unsupported. > > Besides, there is the limitation that Win32 doesn't have such a 'uint64', > and computations in numpy or python are normally done by converting them to > python long integers (correct me if I'm wrong here), whose arithmetic is > very slow compared with the types supported by the compiler. So, the best > approach is to avoid 'uint64' types in general. The info above is somewhat inexact. I was talking about the enhanced numexpr version included in PyTables 2.0 (see [1]). The original version of numexpr (see [2]) doesn't have support for int64 on 32-bit platforms and also neither does for strings. Sorry for the confusion. [1] http://www.pytables.org/trac/browser/trunk/tables/numexpr [2] http://projects.scipy.org/scipy/scipy/browser/trunk/Lib/sandbox/numexpr > > We are very interested in numexpr ! > > Where is the latest / most-up-to-date documentation ? > > At the moment, I think documentation is reduced at some docstrings. If you > want more, you will have to look at the sources. Ups, I spoke too fast. David Cooke kindly added this page: http://www.scipy.org/SciPyPackages/NumExpr Actually, numexpr doesn't need many more info that this as it is pretty straighforward to use already. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Mar 14 08:27:52 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 14 Mar 2007 13:27:52 +0100 Subject: [Numpy-discussion] [PATCH] a fix for compiling numexpr with MSVC Toolkit 2003 In-Reply-To: References: <20070309161128.GB16960@carabos.com> <20070309171943.43b39757@arbutus.physics.mcmaster.ca> Message-ID: <200703141327.53913.faltet@carabos.com> A Dissabte 10 Mar? 2007 19:18, Timothy Hochberg escrigu?: > On 3/9/07, David M. Cooke wrote: > > On Fri, 9 Mar 2007 16:11:28 +0000 > > > > faltet at carabos.com wrote: > > > Hi, > > > > > > I've done a patch for allowing compiling the last version of numexpr > > > > with > > > > > the MSVC Toolkit 2003 compiler on Windows platforms. You can fetch it > > > from: > > > > > > http://www.pytables.org/trac/changeset/2514/trunk > > > > Checked in. It didn't match up; you've got about 150 more lines in your > > version than scipy's version. > > > > > BTW, I understand now why Tim Hochberg was so worried about the time > > > that it takes to compile numexpr on Win platforms. On my Pentium4 @ 2 > > > GHz, and using the MSVC Toolkit 2003, compiling numexpr takes 20 > > > minutes aprox. (!). With the same machine and using GCC under Linux it > > > takes no more than 1 minute. Mmmm, I think it's time to look at the > > > MINGW compiler (GCC based). > > > > Wow! I wonder if lowering the optimisation level would help. > > Yes. In my local copy, I use -O1, not -O2 and it compiles quite fast. It's > been quite a while since I measured it, but as I recall, dropping the > optimization didn't result in a noticeable performance decrease. In fact, I > have a vague recollection that it may have run slight faster?! I guess it > might be a good idea to just use -O1 under variations of VCC. Nice. I'll try with -O1 and will see about performance. > Just to clarify my position though, compile time isn't my concern with the > switch statement. My worry there is that as the switch statement grows, > performance will take a hit as the switch overflows the code cache. Or > something like that. I'm unsure of the details, I just know that the python > dev crowd is always worring about that in the main interpreter loop. That's right. However, we've significantly enhanced the switch statement for the PyTables version of numexpr and haven't experienced any noticeable slowdown yet, even using machines with small cache sizes (64+64 KB for primary cache and 64 KB of secondary cache on my pretty old AMD Duron machine). But I'll try to do more stringent benchmarks and see again. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Mar 14 08:47:36 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 14 Mar 2007 13:47:36 +0100 Subject: [Numpy-discussion] Vectorized ufuncs with OS X frameworks In-Reply-To: References: Message-ID: <200703141347.37985.faltet@carabos.com> Hi Eric, A Divendres 09 Mar? 2007 15:32, Eric Brown escrigu?: > Hi All, > > I have a set of large arrays to which I have to do exp functions > over and over again. I am wondering if there is any benefit to > teaching numpy how to use the vForce frameworks to do this function. > > Does anyone have experience hacking the source to get numpy to use > system libraries to do vectorized versions of sqrt, exp, log, > addition, etc.? I am also interested in doing this for other > operating systems. Maybe it is not what you want, but perhaps it is worth for you to have a look at numexpr [1] which has a kernel for vectorizing the sort of computations you are interested in (sqrt, exp, log, pow, trigonometric, addition...). It does the vectorization by extracting blocks of the numpy arrays and passing them to the computing kernel (which are macros in C for achieving maximum performance). Also, it should be possible to adapt the kernel of numexpr for VForce, although I'm not sure about how vForce is going to achieve more efficiency than numexpr, as IMO the cost for these kind of computations is mostly dominated by fetch/store memory cells rather than doing actual computations (although I might be wrong). [1] http://projects.scipy.org/scipy/scipy/browser/trunk/Lib/sandbox/numexpr/ Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From david.huard at gmail.com Wed Mar 14 09:35:55 2007 From: david.huard at gmail.com (David Huard) Date: Wed, 14 Mar 2007 09:35:55 -0400 Subject: [Numpy-discussion] 3-10x speedup in bincount() In-Reply-To: <45F77F77.5030406@stevesimmons.com> References: <2e1434c10703102243j3ab4282am13c3b972a4299a11@mail.gmail.com> <45F3B985.50600@stevesimmons.com> <45F77F77.5030406@stevesimmons.com> Message-ID: <91cf711d0703140635k69053bc9hb44d591ea527d8b6@mail.gmail.com> Hi Stephen, I'd de glad to test your bincount function for histogramming purposes. David 2007/3/14, Stephen Simmons : > > Well, there were no responses to my earlier email proposing changes to > numpy.bincount() to make it faster and more flexible. Does this mean > noone is using bincount? :-) > > Anyway I've now made the proposed modifications and got substantial > speedups of 3-10. Details are in this extract from my C code. For the > time being I will test this as a standalone extension module. When > stable and tests are written, I'll submit as a patch to the numpy's > _compiled_base.c. > > Cheers, > > Stephen > > > /************************************************ > * Faster versions of bincount and casting strings to integers > * > * Author: Stephen Simmons, mail at stevesimmons.com > * Date: 11 March 2007 > * > * This module contains C code for functions I am using to accelerate > * SQL-like aggregate functions for a column-oriented database based on > numpy. > * > * subtotal's bincount is typically 3-10 times faster than numpy's > bincount, > * and more flexible > * - takes bins as they are, without promoting their type to int32 > * - takes weights as they are, without promoting their type to double > * - ignores negative bins, rather than reporting an error > * - allows optional int32/double output array to be passed in > * - specify maximum bin number to use. Larger bins numbers are ignored > * - only scans for max bin number if neither max_bin nor out array > specified > * > * atoi is 30-60 times faster than casting a string array to an integer > type, > * and may optionally use a translation table. The translation table is > * a convenient way to map sparse integer strings to adjacent bins before > * using bincount. > * > * Typical usage > * ------------- > > # Set up arrays 5,000,000 elts long. s1 has strings filled with '1' and > '2'. > # w are weights > >>> s1 = numpy.ndarray(shape=5000000, dtype='S1'); s1[:]='2'; s1[1::2]='1' > >>> w = numpy.arange(5000000,dtype='f4') > > # Using standard numpy functions, string conversion is slow > >>> i = s1.astype('i1') -> 5.95 sec (!) > >>> numpy.bincount(i) -> 113 ms > >>> numpy.bincount(i, w) -> 272 ms > >>> numpy.bincount(s1.astype('i1'), w) -> 6.12 sec > > # Using the faster functions here: > >>> i = subtotal.atoi(s1) -> 90 ms (60x faster) > >>> subtotal.bincount(i, max_bin=2) -> 31 ms (3.6x faster) > >>> subtotal.bincount(i, w, max_bin=2) -> 51 ms (5.3x faster) > > # In both bases, output is > array([2, 1, 2, ..., 1, 2, 1], dtype=int8) > array([ 0, 2500000, 2500000]) > array([ 0.00000000e+00, 6.25000000e+12, 6.24999750e+12]) > > # As an example of using a translate table, run bincount from atoi > applying > # a translate table for counting odd vs even strings. This does the > whole lot > # in 158ms, a speedup of 38x from 6.12 s above. > >>> t = numpy.arange(256, dtype='i1') % 2 > >>> subtotal.bincount(subtotal.atoi(s1, t), w, max_bin=t.max()) -> > 158 ms > array([ 6.24999750e+12, 6.25000000e+12]) > > * > * These timings are based on numpy-1.0.1 Windows binary distribution > * for Python 2.5, compared with building this code using standard > distutils > * without any particular compiler optimisations: > C:\mnt\dev\subtotal> python setup.py build -cmingw32 > * Processor is a 1.83GHz Pentium-M > ****************************************************/ > > << rest of C code omitted >> > > > Stephen Simmons wrote: > > Hi, > > > > I'd like to propose some minor modifications to the function > > bincount(arr, weights=None), so would like some feedback from other > > uses of bincount() before I write this up as a proper patch, . > > > > Background: > > bincount() has two forms: > > - bincount(x) returns an integer array ians of length max(x)+1 where > > ians[n] is the number of times n appears in x. > > - bincount(x, weights) returns a double array dans of length max(x)+1 > > where dans[n] is the sum of elements in the weight vector weights[i] > > at the positions where x[i]==n > > In both cases, all elements of x must be non-negative. > > > > Proposed changes: > > (1) Remove the restriction that elements of x must be non-negative. > > Currently bincount() starts by finding max(x) and min(x). If the min > > value is negative, an exception is raised. This change proposes > > dropping the initial search for min(x), and instead testing for > > non-negativity while summing values in the return arrays ians or dans. > > Any indexes where where x is negative will be silently ignored. This > > will allow selective bincounts where values to ignore are flagged with > > a negative bin number. > > > > (2) Allow an optional argument for maximum bin number. > > Currently bincount(x) returns an array whose length is dependent on > > max(x). It is sometimes preferable to specify the exact size of the > > returned array, so this change would add a new optional argument, > > max_bin, which is one less than the size of the returned array. Under > > this change, bincount() starts by finding max(x) only if max_bin is > > not specified. Then the returned array ians or dans is created with > > length max_bin+1, and any indexes that would overflow the output array > > are silently ignored. > > > > (3) Allow an optional output array, y. > > Currently bincount() creates a new output array each time. Sometimes > > it is preferable to add results to an existing output array, for > > example, when the input array is only available in smaller chunks, or > > for a progressive update strategy to avoid fp precision problems when > > adding lots of small weights to larger subtotals. Thus we can add an > > extra optional argument y that bypasses the creation of an output array. > > > > With these three change, the function signature of bincount() would > > become: > > bincount(x, weights=None, y=None, max_bin=None) > > > > > > Anyway, that's the general idea. I'd be grateful for any feedback > > before I code this up as a patch to _compiled_base.c. > > > > Cheers > > > > Stephen > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogdude at googlemail.com Wed Mar 14 09:43:18 2007 From: ogdude at googlemail.com (David Koch) Date: Wed, 14 Mar 2007 14:43:18 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> Message-ID: Hi, so one thing I came across now is the following, very simple: Matlab: A = [] while .... A = [A some_scalar_value] end In Python, I tried: A = empty((0,0)) while .... A = concatenate((A, array([someScalarValue])), 1) end which returns an error since the shape of the empty A does not match the vector I want to concatenate with. Any way to get around this without having to use some obscure "if exist A" clause. I am still stuck in my Matlab ways - hoping to get rid of them though. /Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Wed Mar 14 09:46:38 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 14 Mar 2007 14:46:38 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> Message-ID: <45F7FCBE.5060904@ntc.zcu.cz> David Koch wrote: > Hi, > > so one thing I came across now is the following, very simple: > > Matlab: > A = [] > while .... > A = [A some_scalar_value] > end > > > In Python, I tried: > > A = empty((0,0)) > while .... > A = concatenate((A, array([someScalarValue])), 1) > end > > which returns an error since the shape of the empty A does not match the > vector I want to concatenate with. Any way to get around this without > having > to use some obscure "if exist A" clause. I am still stuck in my Matlab ways > - hoping to get rid of them though. a = empty( (0,) ) ... I would bet that using a regular Python list would be faster in this case than growing the numpy array. a = [] while ... a.append( scalar ) a = array( a ) r. From svetosch at gmx.net Wed Mar 14 10:06:17 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Wed, 14 Mar 2007 15:06:17 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> Message-ID: <45F80159.20704@gmx.net> David Koch schrieb: > > In Python, I tried: > > A = empty((0,0)) > while .... > A = concatenate((A, array([someScalarValue])), 1) > end > > which returns an error since the shape of the empty A does not match the > vector I want to concatenate with. Any way to get around this without > having to use some obscure "if exist A" clause. I am still stuck in my > Matlab ways - hoping to get rid of them though. If you want a 1d-array in the end you could try empty(0) to start with, and then do hstack((A, your_scalar)) or something like that. hth, sven From ogdude at googlemail.com Wed Mar 14 10:24:35 2007 From: ogdude at googlemail.com (David Koch) Date: Wed, 14 Mar 2007 15:24:35 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45F80159.20704@gmx.net> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F80159.20704@gmx.net> Message-ID: On 3/14/07, Sven Schreiber wrote: > > > > If you want a 1d-array in the end you could try empty(0) to start with, > and then do hstack((A, your_scalar)) or something like that. Yeah, that works - odd, I thought concatenate((a,b),0) == hstack((a,b)) Thanks /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Wed Mar 14 11:08:31 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 14 Mar 2007 08:08:31 -0700 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F80159.20704@gmx.net> Message-ID: On 3/14/07, David Koch wrote: > > On 3/14/07, Sven Schreiber wrote: > > > > > > > > If you want a 1d-array in the end you could try empty(0) to start with, > > and then do hstack((A, your_scalar)) or something like that. > > Depending on what your generating routine looks like, you might also try "a = fromiter(...)" where you package up your scalar source as an iterator. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Wed Mar 14 11:11:43 2007 From: sturla at molden.no (Sturla Molden) Date: Wed, 14 Mar 2007 16:11:43 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45F7FCBE.5060904@ntc.zcu.cz> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> Message-ID: <45F810AF.3060900@molden.no> On 3/14/2007 2:46 PM, Robert Cimrman wrote: > a = [] > while ... > a.append( scalar ) > a = array( a ) While it may help, growing Python lists is also an O(N) process. One can reduce the amount of allocations by preallocating an ndarray of a certain size (e.g. 1024 scalars), filling it up, and storing it in a linked list. Finally, the stored arrays are retrieved as a single contiguous array. Example code below (cf. class scalar_list). Sturla Molden import numpy class ndarray_list: """ a single linked list of numpy ndarrays.""" class node: def __init__(self, data): self.next = None self.data = data def __init__(self): self.head = None self.tail = None self.len = 0 def append(self, data): tmp = self.node(data) if self.tail == None: self.tail = tmp self.head = tmp self.len = len(data) else: self.tail.next = tmp self.tail = tmp self.len += len(data) def length(self): return self.len def flatten(self, dtype=float): tmp = numpy.empty(self.len, dtype=dtype) cur = self.head idx0 = 0 while cur: tmp[idx0:idx0+len(cur.data)] = cur.data idx0 += len(cur.data) cur = cur.next return tmp class scalar_list: """ a single linked list of numpy scalars.""" def __init__(self, size=1024, dtype=float): self.cur = 0 self.size = size self.dtype = dtype self.arr = numpy.empty(size,dtype) self.arrlist = ndarray_list() def append(self, scalar): cur = self.cur self.arr[cur] = scalar self.cur += 1 if self.cur == self.size: self.arrlist.append(self.arr) self.arr = numpy.empty(self.size,self.dtype) self.cur = 0 def array(self): if self.cur: self.arrlist.append(self.arr[:self.cur]) retval = self.arrlist.flatten(self.dtype) self.cur = 0 self.arr = numpy.empty(self.size,self.dtype) self.arrlist = ndarray_list() self.arrlist.append(retval.copy()) return retval From miguel.oliveira.jr at gmail.com Wed Mar 14 12:03:04 2007 From: miguel.oliveira.jr at gmail.com (Miguel Oliveira, Jr.) Date: Wed, 14 Mar 2007 16:03:04 +0000 Subject: [Numpy-discussion] arrayrange Message-ID: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> Hello, I've got a few codes that use "arrayrange" within numpy. It happens that the new version of numpy apparently doesn't recognize "arrayrange"... I've tried to change it to "arange", but it doesn't work... So, for example, the code below used to create a sine wave file, but it's not working anymore... Does anyone have any clue of what may be a solution for this? Thanks!!! Miguel # Basic sound generation # Generate a sine wave and stick it in an AIFC file # Assumes mono, 16-bit sound # SRate, duration, and frequency are variables from numpy import arange, sin, pi from array import array import aifc print "----------------" SRate = 44100.0 # sampling rate of the output file durSecs = 1.0 #its duration in seconds durSamps = int(SRate * durSecs) # resulting number of samples freq = 330 #frequency of the sine wave in Hz (cycles per second) maxShort = 2** 15 # the largest possible signed 16-bit integer # Make a floating point sinewave array using two handy functions from NumPy: t = arange(0, durSecs, 1.0/SRate) sineWave = sin(2*pi*freq*t) # print "max value of sine is: " + str( max(sineWave)) # Now write it into a 16-bit integer array, adjusting the values to "fill" that format sineWaveArray = array('h', [0]*len(sineWave)) for i in range(0, len(sineWave)): sineWaveArray[i] = int(sineWave[i] * maxShort) # Finally, write it to an AIFF file filename = str(freq) + ".aiff" out = aifc.open(filename, "wb") out.aiff() #out.setparams((1, 2, 44100, 100, 'NONE', 'not compressed')) out.setparams((1, 2, int(SRate), int(durSamps), 'NONE', 'not compressed')) out.writeframes(sineWaveArray.tostring()) # this is the crucial step: the .tostring() method forces this into the string format that AIFC requires out.close() print "Successfully wrote a sine wave of frequency " + str(freq) + " to file: " + filename From oliphant at ee.byu.edu Wed Mar 14 12:46:46 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 14 Mar 2007 09:46:46 -0700 Subject: [Numpy-discussion] cannot pickle large numpy objects when memory resources are already stressed In-Reply-To: <20070312202055.GE9245@bams.ccf.swri.edu> References: <20070312202055.GE9245@bams.ccf.swri.edu> Message-ID: <45F826F6.9040408@ee.byu.edu> Glen W. Mabey wrote: >Hello, > >After running a simulation that took 6 days to complete, my script >proceeded to attempt to write the results out to a file, pickled. > >The operation failed even though there was 1G of RAM free (4G machine). >I've since reconsidered using the pickle format for storing data sets >that include large numpy arrays. However, somehow I assumed that one >would be able to pickle anything that you already had in memory, but I >see now that this was a rash assumption. > >Ought there to be a way to do this, or should I forget about being able >to bundle large numpy arrays and other objects in a single pickle? > >Thanks, >Glen > > >(these commands performed on a different machine with 1G RAM) > >In [10]: za = numpy.zeros( (100000000,), dtype=numpy.float32 ) > >In [11]: import cPickle > >In [12]: zfile = file( '/tmp/zfile', 'w' ) > >In [13]: cPickle.dump( za, zfile ) >--------------------------------------------------------------------------- > > The pickle operation requires making a string from the data in memory before it is written out. This is a limitation of the pickle format, as far as I've been able to figure out. Perhaps when the new bytes type is added to Python we will have a way to view a memory area as a bytes object and be able to make a pickle without creating that extra copy in memory. -Travis From haase at msg.ucsf.edu Wed Mar 14 12:54:34 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Mar 2007 09:54:34 -0700 Subject: [Numpy-discussion] problem with installing scipy In-Reply-To: <1172865381.096764.136950@j27g2000cwj.googlegroups.com> References: <1172865381.096764.136950@j27g2000cwj.googlegroups.com> Message-ID: On 3/2/07, nevin wrote: > I am trying to use scipy optimize module but I am having problem when > I try to import it: > > >>> import numpy > >>> import scipy > >>> from scipy import optimize > Traceback (most recent call last): > File "", line 1, in ? > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/scipy/optimize/__init__.py", line 7, in ? > from optimize import * > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/scipy/optimize/optimize.py", line 25, in ? > import linesearch > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/scipy/optimize/linesearch.py", line 3, in ? > import minpack2 > ImportError: Inappropriate file type for dynamic loading > > How can I fix this problem? My system is Mac OSX Tiger- Pentium. > Thanks. Hi Nevin, I got the same error message -- your scipy package is for non-Intel (PPC) ! You either have to recompile scipy yourself -- which requires a working fortran compiler -- or hope that someone knows where to get a pre-compiled version for Intel. -Sebastian From Glen.Mabey at swri.org Wed Mar 14 12:54:50 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Wed, 14 Mar 2007 11:54:50 -0500 Subject: [Numpy-discussion] cannot pickle large numpy objects when memory resources are already stressed In-Reply-To: <45F826F6.9040408@ee.byu.edu> References: <20070312202055.GE9245@bams.ccf.swri.edu> <45F826F6.9040408@ee.byu.edu> Message-ID: <20070314165450.GF9245@bams.ccf.swri.edu> On Wed, Mar 14, 2007 at 09:46:46AM -0700, Travis Oliphant wrote: > Perhaps when the new bytes type is added to Python we will have a way to > view a memory area as a bytes object and be able to make a pickle > without creating that extra copy in memory. Perhaps this is an aspect that could be mentioned in the PEP as motivation? It seems kind of minor though; there certainly are more compelling justifications for the PEP. Thanks, Glen From haase at msg.ucsf.edu Wed Mar 14 13:02:33 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Mar 2007 10:02:33 -0700 Subject: [Numpy-discussion] compiling numarrray on Cygwin In-Reply-To: <96C071542ED58D49BC08210D3456D5802EC011@pandore.ircm.priv> References: <96C071542ED58D49BC08210D3456D5802EC011@pandore.ircm.priv> Message-ID: Hi Duhaime, are you having a *very* good reason that you want to compile numarray instead of the newer and better numpy. Essentially everyone here has switched either from numarray or from numeric to numpy. You won't get much help with numarray and problems might not ever get fixed. However, people here are very helpful - once you go with the new numpy - to help converting any old code you might have. Essentially numarray and numpy (mostly, 98% ?) Python-code compatible anyway !!! Do the the switch and you will get help. Regards, Sebastian On 3/6/07, Duhaime Johanne wrote: > > I have problem to install numarray on cygwin. I have seen emails on that > topic but I finally do not have a solution. > > Below is the error message for numarray-1.5.2 on python 2.4.3. I am > using numarray instead of numpy because I need the module to be > compatible with a software : MAT (Model-based Analysis of Tiling-array). > > > I have seen some solutions but I do not know exactly how to do since I > am not familiar with python and cygwin. Someone suggests to disable the > package that contains fetestexcept and fetestexcept. I do not know if > this is important for my software but I am willing to try. The problem > is that I do not know how to disable this package... > > The same person mention that it would put a "solution" in the csv > repository. If I am right there is no csv repository for numarray. > Should I look to the python csv. In fact I did that, but have not seen > any comment related to my problem. > > As you can see my questions are very basic. Sorry about that. But I > would really appreciate help to solve that problem. Thank you in > advance for any help. > > > -----------------Compile: > > Building many extension but then with 'numarray.libnumarray' I have an > error. > (...) > > building 'numarray._ufuncComplex32' extension > gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall > -Wstrict-prototypes -IInclude/nu > array -I/usr/include/python2.4 -c Src/_ufuncComplex32module.c -o > build/temp.cyg > in-1.5.24-i686-2.4/Src/_ufuncComplex32module.o > gcc -shared -Wl,--enable-auto-image-base > build/temp.cygwin-1.5.24-i686-2.4/Src/ > ufuncComplex32module.o -L/usr/lib/python2.4/config -lpython2.4 > -o build/lib.cyg > in-1.5.24-i686-2.4/numarray/_ufuncComplex32.dll -lm -L/lib -lm > -lc -lgcc -L/lib > mingw -lmingwex > building 'numarray._ufuncComplex64' extension > gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall > -Wstrict-prototypes -IInclude/nu > array -I/usr/include/python2.4 -c Src/_ufuncComplex64module.c -o > build/temp.cyg > in-1.5.24-i686-2.4/Src/_ufuncComplex64module.o > gcc -shared -Wl,--enable-auto-image-base > build/temp.cygwin-1.5.24-i686-2.4/Src/ > ufuncComplex64module.o -L/usr/lib/python2.4/config -lpython2.4 > -o build/lib.cyg > in-1.5.24-i686-2.4/numarray/_ufuncComplex64.dll -lm -L/lib -lm > -lc -lgcc -L/lib > mingw -lmingwex > > building 'numarray.libnumarray' extension > > gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall > -Wstrict-prototypes -IInclude/num > > array -I/usr/include/python2.4 -c Src/libnumarraymodule.c -o > build/temp.cygwin-1 > > .5.24-i686-2.4/Src/libnumarraymodule.o > > gcc -shared -Wl,--enable-auto-image-base > build/temp.cygwin-1.5.24-i686-2.4/Src/l > > ibnumarraymodule.o -L/usr/lib/python2.4/config -lpython2.4 -o > build/lib.cygwin-1 > > .5.24-i686-2.4/numarray/libnumarray.dll -lm -L/lib -lm -lc -lgcc > -L/lib/mingw -l > > mingwex > > > /lib/mingw/libmingwex.a(feclearexcept.o):feclearexcept.c:(.text+0x21): > undefined > > reference to `___cpu_features' > > > /lib/mingw/libmingwex.a(fetestexcept.o):fetestexcept.c:(.text+0x7): > undefined re > > ference to `___cpu_features' > > collect2: ld returned 1 exit status > > error: command 'gcc' failed with exit status 1 > > > > > duhaimj at ircm.qc.ca > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From vinjvinj at gmail.com Tue Mar 13 19:56:30 2007 From: vinjvinj at gmail.com (vinjvinj) Date: Tue, 13 Mar 2007 23:56:30 -0000 Subject: [Numpy-discussion] Migrating from numeric to numpy Message-ID: <1173830190.657659.158030@o5g2000hsb.googlegroups.com> So far my migration seems to be going well. I have one problem: I've been using the scipy_base.insert and scipy_base.extract functions and the behavior in numpy is not the same. a = [0, 0, 0, 0] mask = [0, 0, 0, 1] c = [10] numpy.insert(a, mask, c) would change a so that a = [0, 0, 0, 10] This is not what I'm seeing in numpy. Thanks, VJ From vinjvinj at gmail.com Tue Mar 13 09:31:44 2007 From: vinjvinj at gmail.com (vinjvinj) Date: Tue, 13 Mar 2007 13:31:44 -0000 Subject: [Numpy-discussion] Questions regarding migrating from Numeric to numpy Message-ID: <1173792704.642436.183090@s48g2000cws.googlegroups.com> I'm in the process of migrating from Numeric to numpy. In some of my code I have the following: a = zeros(num_elements, PyObject) b = zeros(num_elements, PyObject) a is an array of python string objects and b is an array holding mx.DateTime objects. What do I have to do to migrate this over to numpy? PS: I tried to register with the scipy mailing list but got no email confirmation. I tried with two different email addresses. Thanks, VJ From haase at msg.ucsf.edu Wed Mar 14 13:08:15 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Mar 2007 10:08:15 -0700 Subject: [Numpy-discussion] Putting some random back into the top-level? In-Reply-To: References: Message-ID: Hi, Please remind me what's wrong with pylab's rand and randn ! I just learned about their existence recently and thought they seem quite handy and should go directly into (the top-level of) numpy. Functions that have the same name and do the same thing don't conflict either ;-) -Sebastian On 3/12/07, Rob Hetland wrote: > > I, for one, would also like this. Perhaps it should not be called > 'rand', however, as that conflicts with the pylab rand. (numpy load > and pylab load also conflict -- probably the only reason I ever use > import pylab as pl in my scripts). 'random' is already taken by the > whole package... What does this leave that is still sensible? > > -r > > On Mar 9, 2007, at 2:01 PM, Bill Baxter wrote: > > > Has enough time passed with no top level random function that we can > > now put one back in? > > If I recall, the former top level rand() was basically removed because > > it didn't adhere to the "shapes are always tuples" convention. > > > > Has enough time passed now that we can put something like it back in > > the top level, in tuple-taking form? > > > > I think this is a function people use pretty frequently when writing > > quick tests. > > And numpy.random.random_sample seems a rather long and not so obvious > > name for something that is used relatively frequently. > > > > --bb From oliphant at ee.byu.edu Wed Mar 14 13:08:26 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 14 Mar 2007 10:08:26 -0700 Subject: [Numpy-discussion] Migrating from numeric to numpy In-Reply-To: <1173830190.657659.158030@o5g2000hsb.googlegroups.com> References: <1173830190.657659.158030@o5g2000hsb.googlegroups.com> Message-ID: <45F82C0A.1030809@ee.byu.edu> vinjvinj wrote: >So far my migration seems to be going well. I have one problem: > >I've been using the scipy_base.insert and scipy_base.extract functions >and the behavior in numpy is not the same. > >a = [0, 0, 0, 0] >mask = [0, 0, 0, 1] >c = [10] > >numpy.insert(a, mask, c) > >would change a so that > >a = [0, 0, 0, 10] > >This is not what I'm seeing in numpy. > > scipy_base.insert --> numpy.place -Travis From oliphant at ee.byu.edu Wed Mar 14 13:09:10 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 14 Mar 2007 10:09:10 -0700 Subject: [Numpy-discussion] Questions regarding migrating from Numeric to numpy In-Reply-To: <1173792704.642436.183090@s48g2000cws.googlegroups.com> References: <1173792704.642436.183090@s48g2000cws.googlegroups.com> Message-ID: <45F82C36.5000705@ee.byu.edu> vinjvinj wrote: >I'm in the process of migrating from Numeric to numpy. In some of my >code I have the following: > >a = zeros(num_elements, PyObject) >b = zeros(num_elements, PyObject) > > PyObject --> object -Travis From tim.hochberg at ieee.org Wed Mar 14 13:09:12 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 14 Mar 2007 10:09:12 -0700 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45F810AF.3060900@molden.no> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> Message-ID: On 3/14/07, Sturla Molden wrote: > > On 3/14/2007 2:46 PM, Robert Cimrman wrote: > > > a = [] > > while ... > > a.append( scalar ) > > a = array( a ) > > > While it may help, growing Python lists is also an O(N) process. This may just be a terminology problem, but just to be clear, appending to a Python list is amortized (on average) constant time (O(1)), not O(N). One can reduce the amount of allocations by preallocating an ndarray of > a certain size (e.g. 1024 scalars), filling it up, and storing it in a > linked list. Finally, the stored arrays are retrieved as a single > contiguous array. Example code below (cf. class scalar_list). > > > Sturla Molden > > > > > import numpy > > class ndarray_list: > > """ a single linked list of numpy ndarrays.""" > > class node: > def __init__(self, data): > self.next = None > self.data = data > > def __init__(self): > self.head = None > self.tail = None > self.len = 0 > > def append(self, data): > tmp = self.node(data) > if self.tail == None: > self.tail = tmp > self.head = tmp > self.len = len(data) > else: > self.tail.next = tmp > self.tail = tmp > self.len += len(data) > > def length(self): return self.len > > def flatten(self, dtype=float): > tmp = numpy.empty(self.len, dtype=dtype) > cur = self.head > idx0 = 0 > while cur: > tmp[idx0:idx0+len(cur.data)] = cur.data > idx0 += len(cur.data) > cur = cur.next > return tmp > > > class scalar_list: > > """ a single linked list of numpy scalars.""" > > def __init__(self, size=1024, dtype=float): > self.cur = 0 > self.size = size > self.dtype = dtype > self.arr = numpy.empty(size,dtype) > self.arrlist = ndarray_list() > > def append(self, scalar): > cur = self.cur > self.arr[cur] = scalar > self.cur += 1 > if self.cur == self.size: > self.arrlist.append(self.arr) > self.arr = numpy.empty(self.size,self.dtype) > self.cur = 0 > > def array(self): > if self.cur: self.arrlist.append(self.arr[:self.cur]) > retval = self.arrlist.flatten(self.dtype) > self.cur = 0 > self.arr = numpy.empty(self.size,self.dtype) > self.arrlist = ndarray_list() > self.arrlist.append(retval.copy()) > return retval > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Wed Mar 14 13:17:59 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 14 Mar 2007 18:17:59 +0100 Subject: [Numpy-discussion] cannot pickle large numpy objects when memory resources are already stressed In-Reply-To: <45F826F6.9040408@ee.byu.edu> References: <20070312202055.GE9245@bams.ccf.swri.edu> <45F826F6.9040408@ee.byu.edu> Message-ID: <1173892679.2543.12.camel@localhost.localdomain> El dc 14 de 03 del 2007 a les 09:46 -0700, en/na Travis Oliphant va escriure: > Glen W. Mabey wrote: > > >Hello, > > > >After running a simulation that took 6 days to complete, my script > >proceeded to attempt to write the results out to a file, pickled. > > > >The operation failed even though there was 1G of RAM free (4G machine). > >I've since reconsidered using the pickle format for storing data sets > >that include large numpy arrays. However, somehow I assumed that one > >would be able to pickle anything that you already had in memory, but I > >see now that this was a rash assumption. > > > >Ought there to be a way to do this, or should I forget about being able > >to bundle large numpy arrays and other objects in a single pickle? If you can afford using another package for doing I/O perhaps PyTables can save your day. It is optimized for saving a retrieving very large amounts of data with ease. In particular, it can save your in-memory arrays without a need to do another copy in memory (provided the array is contiguous). It also allows compressing the data in a transparent way, without a need of using additional memory. Furthermore, a recent optimization introduced in the 2.0 branch a week ago also allows to *update* an array on disk without doing copies neither. HTH, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From haase at msg.ucsf.edu Wed Mar 14 13:20:27 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Mar 2007 10:20:27 -0700 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: Hey Bill, what are you using to communicate with the server ? May I recommend looking at Pyro ! (Python remote objects) It would allow you to get your proxy objects. And also handles exception super clean and easy. I have used it for many years ! It's very stable ! (If you run into problems, take a look at the "one-way" calls to ensure that functions, that would block, won't wait for the function to return) Just a thought -- -Sebastian Haase On 3/13/07, Bill Baxter wrote: > Howdy Folks, > > I was missing the good ole days of using Matlab back at the Uni when I > could debug my code, stop at breakpoints and plot various data without > fear of blocking the interpreter process. > > Using "ipython -pylab" is what has been suggested to me in the past, > but the problem is I don't do my debugging from ipython. I have a > very nice IDE that works very well, and it has a lovely interactive > debugging prompt that I can use to probe my code when stopped at a > breakpoint. It's great except I can't really use matplotlib for > debugging there because it causes things to freeze up. > > So I've come up with a decent (though not perfect) solution for > quickie interactive plots which is to run matplotlib in a separate > process. I call the result it 'ezplot'. The first alpha version of > this is now available at the Cheeseshop. (I made an egg too, so if > you have setuptools you can do "easy_install ezplot".) > > The basic usage is like so: > > In [1]: import ezplot > In [2]: p = ezplot.Plotter() > In [3]: p.plot([1,2,3],[1,4,9],marker='o') > Connecting to server... waiting... > connected to plotserver 0.1.0a1 on http://localhost:8397 > Out[3]: True > In [4]: from numpy import * > In [5]: x = linspace(-5,5,20) > In [13]: p.clf() > Out[13]: True > In [14]: p.plot(x, x*x*log(x*x+0.01)) > > (Imagine lovely plots popping up on your screen as these commands are typed.) > > The only return values you get back are True (success...probably) or > False (failure...for sure). So no fancy plot object manipulation is > possible. But you can do basic plots no problem. > > The nice part is that this (unlike ipython's built-in -pylab threading > mojo) should work just as well from wherever you're using python. > Whether it's ipython (no -pylab) or Idle, or a plain MS-DOS console, > or WingIDE's debug probe, or SPE, or a PyCrust shell or whatever. It > doesn't matter because all the client is doing is packing up data and > shipping over a socket. All the GUI plotting mojo happens in a > completely separate process. > > There are plenty of ways this could be made better, but for me, for > now, this probably does pretty much all I need, so it's back to Real > Work. But if anyone is interested in making improvements to this, let > me know. > > Here's a short list of things that could be improved: > * Right now I assume use of the wxAGG backend for matplotlib. Don't > know how much work it would be to support other back ends (or how to > go about it, really). wxAGG is what I always use. > * Returning more error/exception info from the server would be nice > * Returning full fledged proxy plot objects would be nice too, but I > suspect that's a huge effort > * SOAP may be better for this than xmlrpclib but I just couldn't get > it to work (SOAPpy + Twisted). > * A little more safety would be nice. Anyone know how to make a > Twisted xmlrpc server not accept connections from anywhere except > localhost? > * There's a little glitch in that the spawned plot server dies with > the parent that created it. Maybe there's a flag to subprocess.Popen > to fix that? > * Sometimes when you click on "Exit Server", if there are plot windows > open it hangs while shutting down. > > > Only tested on Win32 but there's nothing much platform specific in there. > > Give it a try and let me know what you think! > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From tim.hochberg at ieee.org Wed Mar 14 13:21:34 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Wed, 14 Mar 2007 10:21:34 -0700 Subject: [Numpy-discussion] Putting some random back into the top-level? In-Reply-To: References: Message-ID: On 3/14/07, Sebastian Haase wrote: > > Hi, > Please remind me what's wrong with pylab's > rand and randn ! > I just learned about their existence recently and thought > they seem quite handy and should go directly into (the top-level of) > numpy. > Functions that have the same name and do the same thing don't conflict > either ;-) I don't know what the problem, if any, is with rand and randn, but I can tell you what the problem with stuffing stuff in the main namespace is: it's allready much too crowded, which makes it difficult to find functions when you need them. Have you tried dir(numpy) recently? If all of the random number functions live in one package, it's easy to scan them, see what's there and decide what is appropriate, if they are smeared out across multiple packages. You might say that we should put these functions in both the top level package and in the relevant subpackage, but then it's difficult to find what packages exist since the top level package is too crowded (a situation that already exists today to a certain extent). Note that it's pretty easy for a downstream user to create a customized package that pulls in all of the functions that they feel are deserving: a file with just "from numpy import *; from numpy.random import randn" comes to mind. It's considerably more difficult for such a user to trim the main namespace in a sensible way. Thus, it's my opinion that we should be moving stuff out of the main package, not putting stuff in. Regards, -Sebastian > > > > On 3/12/07, Rob Hetland wrote: > > > > I, for one, would also like this. Perhaps it should not be called > > 'rand', however, as that conflicts with the pylab rand. (numpy load > > and pylab load also conflict -- probably the only reason I ever use > > import pylab as pl in my scripts). 'random' is already taken by the > > whole package... What does this leave that is still sensible? > > > > -r > > > > On Mar 9, 2007, at 2:01 PM, Bill Baxter wrote: > > > > > Has enough time passed with no top level random function that we can > > > now put one back in? > > > If I recall, the former top level rand() was basically removed because > > > it didn't adhere to the "shapes are always tuples" convention. > > > > > > Has enough time passed now that we can put something like it back in > > > the top level, in tuple-taking form? > > > > > > I think this is a function people use pretty frequently when writing > > > quick tests. > > > And numpy.random.random_sample seems a rather long and not so obvious > > > name for something that is used relatively frequently. > > > > > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From lxander.m at gmail.com Wed Mar 14 13:31:54 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Wed, 14 Mar 2007 13:31:54 -0400 Subject: [Numpy-discussion] Multiplying Each Column of a Matrix by a Vector (Element-Wise) Message-ID: <525f23e80703141031r473a5c43mc80a3bff4f450572@mail.gmail.com> Is there a clever way to multiply each column of a matrix by a vector *element-wise*? That is, the equivalent of (from some 1D v and 2D m): r = numpy.empty_like(m) for i in range(m.shape[-1]): r[...,i] = v*m[...,i] Thanks, Alex From robert.kern at gmail.com Wed Mar 14 13:31:57 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 14 Mar 2007 12:31:57 -0500 Subject: [Numpy-discussion] arrayrange In-Reply-To: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> References: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> Message-ID: <45F8318D.9030209@gmail.com> Miguel Oliveira, Jr. wrote: > Hello, > > I've got a few codes that use "arrayrange" within numpy. It happens > that the new version of numpy apparently doesn't recognize > "arrayrange"... I've tried to change it to "arange", but it doesn't > work... So, for example, the code below used to create a sine wave > file, but it's not working anymore... Does anyone have any clue of > what may be a solution for this? arange() is the appropriate replacement for arrayrange(); arrayrange() was just an alias to arange(). What did you expect it to do that it isn't doing? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Mar 14 13:48:48 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 14 Mar 2007 12:48:48 -0500 Subject: [Numpy-discussion] Multiplying Each Column of a Matrix by a Vector (Element-Wise) In-Reply-To: <525f23e80703141031r473a5c43mc80a3bff4f450572@mail.gmail.com> References: <525f23e80703141031r473a5c43mc80a3bff4f450572@mail.gmail.com> Message-ID: <45F83580.3000707@gmail.com> Alexander Michael wrote: > Is there a clever way to multiply each column of a matrix by a vector > *element-wise*? That is, the equivalent of (from some 1D v and 2D m): > > r = numpy.empty_like(m) > for i in range(m.shape[-1]): > r[...,i] = v*m[...,i] numpy.multiply(m, v[:,numpy.newaxis]) If m is actually just a shape-(n,m) array rather than a numpy.matrix object, then you can spell that like so: m * v[:,numpy.newaxis] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From haase at msg.ucsf.edu Wed Mar 14 13:50:54 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed, 14 Mar 2007 10:50:54 -0700 Subject: [Numpy-discussion] Putting some random back into the top-level? In-Reply-To: References: Message-ID: On 3/14/07, Timothy Hochberg wrote: > > > On 3/14/07, Sebastian Haase wrote: > > Hi, > > Please remind me what's wrong with pylab's > > rand and randn ! > > I just learned about their existence recently and thought > > they seem quite handy and should go directly into (the top-level of) > numpy. > > Functions that have the same name and do the same thing don't conflict > > either ;-) > > I don't know what the problem, if any, is with rand and randn, but I can > tell you what the problem with stuffing stuff in the main namespace is: it's > allready much too crowded, which makes it difficult to find functions when > you need them. Have you tried dir(numpy) recently? Hey Tim, yes, I have done this many times -- just to scare myself .... ;-) As I see it most of them are "historical problems" -- and we will likely be stuck with them forever -- since the 1.0 commitment apparently doesn't even allow to make numpy.resize and array.resize to fill in the same way [[ one adds 0s, the other repeats the array ]] . (Especially I'm thinking of hanning and hamming and other things I understand even less ...) The only argument here, was that one or two [ :-) ] random functions [ how about rand() and randn() ?] would be nice to have "as a shortcut".... Yes, I have some modules myself, containing a bunch of home-made things, that I call "useful". I understand the argument here was to get the "best possible" "common ground". I don't have (very) strong feelings about this. -Sebastian From eike.welk at gmx.net Wed Mar 14 13:55:42 2007 From: eike.welk at gmx.net (Eike Welk) Date: Wed, 14 Mar 2007 18:55:42 +0100 Subject: [Numpy-discussion] arrayrange In-Reply-To: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> References: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> Message-ID: <200703141855.42724.eike.welk@gmx.net> I would use something like this: t = linspace(0, durSecs, durSecs*SRate) Do you know the 'Numpy Example List' http://www.scipy.org/Numpy_Example_List Regards Eike. PS: Ah, you did subscribe. From corzneffect at gmail.com Wed Mar 14 14:36:49 2007 From: corzneffect at gmail.com (Cory Davis) Date: Wed, 14 Mar 2007 18:36:49 +0000 Subject: [Numpy-discussion] import numpy segmentation fault Message-ID: <4948f97c0703141136j522fd32kbe202874ff19e3cf@mail.gmail.com> Hi there, I have just installed numpy-1.0.1 from source, which seemed to go fine. However when I try to "import numpy" I get a segmentation fault. A have a 64 bit machine running RedHat Enterprise Linux and Python 2.34 Any clues greatly appreciated. Cheers, Cory. From gael.varoquaux at normalesup.org Wed Mar 14 14:46:05 2007 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 14 Mar 2007 19:46:05 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <45F810AF.3060900@molden.no> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> Message-ID: <20070314184605.GD18520@clipper.ens.fr> On Wed, Mar 14, 2007 at 04:11:43PM +0100, Sturla Molden wrote: > On 3/14/2007 2:46 PM, Robert Cimrman wrote: > > a = [] > > while ... > > a.append( scalar ) > > a = array( a ) > While it may help, growing Python lists is also an O(N) process. > One can reduce the amount of allocations by preallocating an ndarray of > a certain size (e.g. 1024 scalars), filling it up, and storing it in a > linked list. Finally, the stored arrays are retrieved as a single > contiguous array. Example code below (cf. class scalar_list). > Sturla Molden I definitely second this comment. Using arrays when you are trying to append a lot of data is using the wrong data format. And the code is so much more readable with lists. If you want to do something even nicer you might want to use a generator, possibly one created with the "yield" keyword. -- Ga?l From zpincus at stanford.edu Wed Mar 14 14:49:10 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed, 14 Mar 2007 13:49:10 -0500 Subject: [Numpy-discussion] import numpy segmentation fault In-Reply-To: <4948f97c0703141136j522fd32kbe202874ff19e3cf@mail.gmail.com> References: <4948f97c0703141136j522fd32kbe202874ff19e3cf@mail.gmail.com> Message-ID: <0F58E0ED-B2C4-4CF6-8EE3-4A25BEC73854@stanford.edu> If I recall correctly, there's a bug in numpy 1.0.1 on Linux-x86-64 that causes this segfault. This is fixed in the latest SVN version of numpy, so if you can grab that, it should work. I can't find the trac ticket, but I ran into this some weeks ago. Zach On Mar 14, 2007, at 1:36 PM, Cory Davis wrote: > Hi there, > > I have just installed numpy-1.0.1 from source, which seemed to go > fine. However when I try to "import numpy" I get a segmentation > fault. > > A have a 64 bit machine running RedHat Enterprise Linux and Python > 2.34 > > Any clues greatly appreciated. > > Cheers, > Cory. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cookedm at physics.mcmaster.ca Wed Mar 14 16:10:09 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed, 14 Mar 2007 16:10:09 -0400 Subject: [Numpy-discussion] Which dtype are supported by numexpr ? In-Reply-To: <200703141302.13084.faltet@carabos.com> References: <200703091856.30141.faltet@carabos.com> <200703141302.13084.faltet@carabos.com> Message-ID: <20070314161009.7816899c@arbutus.physics.mcmaster.ca> On Wed, 14 Mar 2007 13:02:10 +0100 Francesc Altet wrote: > The info above is somewhat inexact. I was talking about the enhanced > numexpr version included in PyTables 2.0 (see [1]). The original version of > numexpr (see [2]) doesn't have support for int64 on 32-bit platforms and > also neither does for strings. Sorry for the confusion. What are you doing with strings in numexpr? Ivan Vilata i Balaguer submitted a patch to add them, but I rejected it as I didn't have any use-cases for strings that needed the speed of numexpr, and were worth adding the complexity. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faltet at carabos.com Wed Mar 14 17:05:51 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 14 Mar 2007 22:05:51 +0100 Subject: [Numpy-discussion] numexpr efficency depends on the size of the computing kernel Message-ID: <1173906351.2543.71.camel@localhost.localdomain> Hi, Now that I'm commanding my old AMD Duron machine, I've made some benchmarks just to prove that the numexpr computing is not influenced by the size of the CPU cache, but I failed miserably (and Tim was right: there is a dependency of the numexpr efficency on CPU cache size). Provided that the pytables instance of the computing kernel of numexpr is quite larger (it supports more datatypes) than the original, comparing the performance of both versions can be a good way to check the influence of CPU cache on the computing efficency. The attached benchmark is a small modification of the timing.py that comes with the numexpr package (the modification was needed to allow the numexpr version of pytables to run all the cases). Basically, the expressions tested operations with arrays of 1 million of elements, with a mix of contiguous and strided arrays (no unaligned arrays are present here). See the code in benchmark for the details. The speed-ups of numexpr over plain numpy on a AMD Duron machine (64 + 64 KB L1 cache, 64 KB L2 cache) are: For the original numexpr package: 2.14, 2.21, 2.21 (these represent averages for 3 complete runs) For the modified pytables version (enlarged computing kernel): 1.32, 1.34, 1.37 So, with a CPU with a very small cache, the original numexpr kernel is 1.6x faster than the pytables one. However, using a AMD Opteron which has a much bigger L2 cache (64 + 64 KB L1 cache, 1 MB L2 cache), the speed-ups are quite similar: For the original numexpr package: 3.10, 3.35, 3.35 For the modified pytables version (enlarged computing kernel): 3.37, 3.50, 3.45 So, there is effectively a dependency on the CPU cache size. It would be nice to run the benchmark with other CPUs with a L2 cache in the range between 64 KB and 1 MB so as to find the point where the performance starts to be similar (this should be a good guess on the size of the computing kernel). Meanwhile, the lesson learned is that Tim worries were correct: one should be very careful on adding more opcodes (at least, until CPUs with a very small L2 cache are in use). With this, perhaps we will have to reduce the opcodes in the numexpr version for pytables to a bare minimum :-/ Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth -------------- next part -------------- Expression: b*c+d*e numpy: 0.284756803513 Skipping weave timing numexpr: 0.267185997963 Speed-up of numexpr over numpy: 1.06576244894 Expression: 2*a+3*b numpy: 0.228031897545 Skipping weave timing numexpr: 0.190967607498 Speed-up of numexpr over numpy: 1.19408679059 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.875679397583 Skipping weave timing numexpr: 0.729962491989 Speed-up of numexpr over numpy: 1.19962245621 Expression: 2*a + arctan2(a, b) numpy: 0.530754685402 Skipping weave timing numexpr: 0.440991616249 Speed-up of numexpr over numpy: 1.20354824411 Expression: a**2 + (b+1)**-2.5 numpy: 0.830808615685 Skipping weave timing numexpr: 0.408902907372 Speed-up of numexpr over numpy: 2.03179923817 Expression: (a+1)**50 numpy: 0.486846494675 Skipping weave timing numexpr: 0.394672584534 Speed-up of numexpr over numpy: 1.23354525689 Expression: sqrt(a**2 + b**2) numpy: 0.387914180756 Skipping weave timing numexpr: 0.292760682106 Speed-up of numexpr over numpy: 1.3250214406 Average = 1.32191226793 Expression: b*c+d*e numpy: 0.279518294334 Skipping weave timing numexpr: 0.225658392906 Speed-up of numexpr over numpy: 1.23867891965 Expression: 2*a+3*b numpy: 0.227924203873 Skipping weave timing numexpr: 0.190263104439 Speed-up of numexpr over numpy: 1.19794221031 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.865833806992 Skipping weave timing numexpr: 0.736699199677 Speed-up of numexpr over numpy: 1.17528810588 Expression: 2*a + arctan2(a, b) numpy: 0.536459088326 Skipping weave timing numexpr: 0.465694189072 Speed-up of numexpr over numpy: 1.15195572742 Expression: a**2 + (b+1)**-2.5 numpy: 0.803207492828 Skipping weave timing numexpr: 0.402952003479 Speed-up of numexpr over numpy: 1.99330810095 Expression: (a+1)**50 numpy: 0.506087398529 Skipping weave timing numexpr: 0.390724515915 Speed-up of numexpr over numpy: 1.29525376043 Expression: sqrt(a**2 + b**2) numpy: 0.390014004707 Skipping weave timing numexpr: 0.292934322357 Speed-up of numexpr over numpy: 1.33140426007 Average = 1.34054729781 Expression: b*c+d*e numpy: 0.282696795464 Skipping weave timing numexpr: 0.227395987511 Speed-up of numexpr over numpy: 1.2431916612 Expression: 2*a+3*b numpy: 0.247914505005 Skipping weave timing numexpr: 0.206929206848 Speed-up of numexpr over numpy: 1.19806434665 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.87483150959 Skipping weave timing numexpr: 0.722416090965 Speed-up of numexpr over numpy: 1.21098009932 Expression: 2*a + arctan2(a, b) numpy: 0.546046590805 Skipping weave timing numexpr: 0.440475416183 Speed-up of numexpr over numpy: 1.23967552046 Expression: a**2 + (b+1)**-2.5 numpy: 0.841809201241 Skipping weave timing numexpr: 0.40777721405 Speed-up of numexpr over numpy: 2.06438509126 Expression: (a+1)**50 numpy: 0.484260010719 Skipping weave timing numexpr: 0.37349460125 Speed-up of numexpr over numpy: 1.29656495462 Expression: sqrt(a**2 + b**2) numpy: 0.428371477127 Skipping weave timing numexpr: 0.316362810135 Speed-up of numexpr over numpy: 1.35405130883 Average = 1.37241614033 Averages: 1.32, 1.34, 1.37 -------------- next part -------------- Expression: b*c+d*e numpy: 0.290255403519 Skipping weave timing numexpr: 0.190418314934 Speed-up of numexpr over numpy: 1.52430402306 Expression: 2*a+3*b numpy: 0.226468586922 Skipping weave timing numexpr: 0.127545499802 Speed-up of numexpr over numpy: 1.77559057179 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.87546172142 Skipping weave timing numexpr: 0.621131896973 Speed-up of numexpr over numpy: 1.4094618642 Expression: 2*a + arctan2(a, b) numpy: 0.528830099106 Skipping weave timing numexpr: 0.346895003319 Speed-up of numexpr over numpy: 1.52446732886 Expression: a**2 + (b+1)**-2.5 numpy: 0.792816495895 Skipping weave timing numexpr: 0.218543100357 Speed-up of numexpr over numpy: 3.62773519091 Expression: (a+1)**50 numpy: 0.482146501541 Skipping weave timing numexpr: 0.186633110046 Speed-up of numexpr over numpy: 2.58339209705 Expression: sqrt(a**2 + b**2) numpy: 0.388063216209 Skipping weave timing numexpr: 0.151627588272 Speed-up of numexpr over numpy: 2.55931800164 Average = 2.14346701107 Expression: b*c+d*e numpy: 0.283156108856 Skipping weave timing numexpr: 0.181364917755 Speed-up of numexpr over numpy: 1.56125072236 Expression: 2*a+3*b numpy: 0.226498603821 Skipping weave timing numexpr: 0.124421000481 Speed-up of numexpr over numpy: 1.8204210137 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.868006300926 Skipping weave timing numexpr: 0.623650097847 Speed-up of numexpr over numpy: 1.39181618655 Expression: 2*a + arctan2(a, b) numpy: 0.517928004265 Skipping weave timing numexpr: 0.348434090614 Speed-up of numexpr over numpy: 1.48644469131 Expression: a**2 + (b+1)**-2.5 numpy: 0.799534797668 Skipping weave timing numexpr: 0.216258502007 Speed-up of numexpr over numpy: 3.69712538582 Expression: (a+1)**50 numpy: 0.487076807022 Skipping weave timing numexpr: 0.164514088631 Speed-up of numexpr over numpy: 2.96069966455 Expression: sqrt(a**2 + b**2) numpy: 0.387224507332 Skipping weave timing numexpr: 0.153417181969 Speed-up of numexpr over numpy: 2.52399700192 Average = 2.20596495232 Expression: b*c+d*e numpy: 0.278421878815 Skipping weave timing numexpr: 0.18240711689 Speed-up of numexpr over numpy: 1.52637618291 Expression: 2*a+3*b numpy: 0.234265589714 Skipping weave timing numexpr: 0.124828195572 Speed-up of numexpr over numpy: 1.87670412635 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.852713894844 Skipping weave timing numexpr: 0.606571722031 Speed-up of numexpr over numpy: 1.40579236366 Expression: 2*a + arctan2(a, b) numpy: 0.5161703825 Skipping weave timing numexpr: 0.348170495033 Speed-up of numexpr over numpy: 1.48252189621 Expression: a**2 + (b+1)**-2.5 numpy: 0.794040799141 Skipping weave timing numexpr: 0.215844082832 Speed-up of numexpr over numpy: 3.67877028975 Expression: (a+1)**50 numpy: 0.481977200508 Skipping weave timing numexpr: 0.164862012863 Speed-up of numexpr over numpy: 2.92351883941 Expression: sqrt(a**2 + b**2) numpy: 0.386767506599 Skipping weave timing numexpr: 0.14988219738 Speed-up of numexpr over numpy: 2.58047662338 Average = 2.21059433167 Averages: 2.14, 2.21, 2.21 -------------- next part -------------- A non-text attachment was scrubbed... Name: numexpr-timing.py Type: text/x-python Size: 3269 bytes Desc: not available URL: -------------- next part -------------- Expression: b*c+d*e numpy: 0.0430510997772 Skipping weave timing numexpr: 0.0235065937042 Speed-up of numexpr over numpy: 1.83144781923 Expression: 2*a+3*b numpy: 0.0429566144943 Skipping weave timing numexpr: 0.0219662904739 Speed-up of numexpr over numpy: 1.95556981026 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.286458492279 Skipping weave timing numexpr: 0.250001215935 Speed-up of numexpr over numpy: 1.14582839611 Expression: 2*a + arctan2(a, b) numpy: 0.139817690849 Skipping weave timing numexpr: 0.121367192268 Speed-up of numexpr over numpy: 1.15202212588 Expression: a**2 + (b+1)**-2.5 numpy: 0.369387292862 Skipping weave timing numexpr: 0.0481228113174 Speed-up of numexpr over numpy: 7.67592920591 Expression: (a+1)**50 numpy: 0.283995580673 Skipping weave timing numexpr: 0.0360183000565 Speed-up of numexpr over numpy: 7.88475803211 Expression: sqrt(a**2 + b**2) numpy: 0.0699777126312 Skipping weave timing numexpr: 0.03638920784 Speed-up of numexpr over numpy: 1.9230347893 Average = 3.36694145411 Expression: b*c+d*e numpy: 0.0497439146042 Skipping weave timing numexpr: 0.0267603874207 Speed-up of numexpr over numpy: 1.85886376838 Expression: 2*a+3*b numpy: 0.0438626050949 Skipping weave timing numexpr: 0.0191017150879 Speed-up of numexpr over numpy: 2.29626527739 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.277396702766 Skipping weave timing numexpr: 0.269183421135 Speed-up of numexpr over numpy: 1.03051184058 Expression: 2*a + arctan2(a, b) numpy: 0.159837794304 Skipping weave timing numexpr: 0.137581419945 Speed-up of numexpr over numpy: 1.16176875023 Expression: a**2 + (b+1)**-2.5 numpy: 0.375256705284 Skipping weave timing numexpr: 0.0533778905869 Speed-up of numexpr over numpy: 7.03018986248 Expression: (a+1)**50 numpy: 0.317774915695 Skipping weave timing numexpr: 0.0351259946823 Speed-up of numexpr over numpy: 9.04671650068 Expression: sqrt(a**2 + b**2) numpy: 0.0805351018906 Skipping weave timing numexpr: 0.039293885231 Speed-up of numexpr over numpy: 2.04955812888 Average = 3.49626773266 Expression: b*c+d*e numpy: 0.0495269060135 Skipping weave timing numexpr: 0.0265894889832 Speed-up of numexpr over numpy: 1.86264978785 Expression: 2*a+3*b numpy: 0.0449105024338 Skipping weave timing numexpr: 0.0221442937851 Speed-up of numexpr over numpy: 2.02808465556 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.312991595268 Skipping weave timing numexpr: 0.283522415161 Speed-up of numexpr over numpy: 1.10393950718 Expression: 2*a + arctan2(a, b) numpy: 0.159363889694 Skipping weave timing numexpr: 0.13733689785 Speed-up of numexpr over numpy: 1.16038655444 Expression: a**2 + (b+1)**-2.5 numpy: 0.368414521217 Skipping weave timing numexpr: 0.0534101009369 Speed-up of numexpr over numpy: 6.89784356807 Expression: (a+1)**50 numpy: 0.312214398384 Skipping weave timing numexpr: 0.0343459129333 Speed-up of numexpr over numpy: 9.09029260599 Expression: sqrt(a**2 + b**2) numpy: 0.077935218811 Skipping weave timing numexpr: 0.0383999109268 Speed-up of numexpr over numpy: 2.02956769768 Average = 3.45325205383 Averages: 3.37, 3.50, 3.45 -------------- next part -------------- Expression: b*c+d*e numpy: 0.0426661014557 Skipping weave timing numexpr: 0.0238104820251 Speed-up of numexpr over numpy: 1.79190414586 Expression: 2*a+3*b numpy: 0.0391938924789 Skipping weave timing numexpr: 0.0195196151733 Speed-up of numexpr over numpy: 2.00792342118 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.274058103561 Skipping weave timing numexpr: 0.248371481895 Speed-up of numexpr over numpy: 1.10342017316 Expression: 2*a + arctan2(a, b) numpy: 0.139664411545 Skipping weave timing numexpr: 0.121141600609 Speed-up of numexpr over numpy: 1.15290214792 Expression: a**2 + (b+1)**-2.5 numpy: 0.331180119514 Skipping weave timing numexpr: 0.0499030828476 Speed-up of numexpr over numpy: 6.63646613829 Expression: (a+1)**50 numpy: 0.282083797455 Skipping weave timing numexpr: 0.0398853063583 Speed-up of numexpr over numpy: 7.07237384416 Expression: sqrt(a**2 + b**2) numpy: 0.0711817026138 Skipping weave timing numexpr: 0.0363766908646 Speed-up of numexpr over numpy: 1.95679433511 Average = 3.10311202938 Expression: b*c+d*e numpy: 0.0431445121765 Skipping weave timing numexpr: 0.0230684041977 Speed-up of numexpr over numpy: 1.87028594639 Expression: 2*a+3*b numpy: 0.0386809110641 Skipping weave timing numexpr: 0.0188805103302 Speed-up of numexpr over numpy: 2.04872169172 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.275234413147 Skipping weave timing numexpr: 0.247427392006 Speed-up of numexpr over numpy: 1.11238457034 Expression: 2*a + arctan2(a, b) numpy: 0.138790893555 Skipping weave timing numexpr: 0.120497584343 Speed-up of numexpr over numpy: 1.15181473813 Expression: a**2 + (b+1)**-2.5 numpy: 0.330480790138 Skipping weave timing numexpr: 0.0492552995682 Speed-up of numexpr over numpy: 6.70954786664 Expression: (a+1)**50 numpy: 0.282364106178 Skipping weave timing numexpr: 0.0327146053314 Speed-up of numexpr over numpy: 8.63113289363 Expression: sqrt(a**2 + b**2) numpy: 0.0695419073105 Skipping weave timing numexpr: 0.0363955020905 Speed-up of numexpr over numpy: 1.91072806573 Average = 3.34780225322 Expression: b*c+d*e numpy: 0.04261469841 Skipping weave timing numexpr: 0.0229945898056 Speed-up of numexpr over numpy: 1.85324890639 Expression: 2*a+3*b numpy: 0.0387926101685 Skipping weave timing numexpr: 0.0188351154327 Speed-up of numexpr over numpy: 2.05958972256 Expression: 2*a + (cos(3)+5)*sinh(cos(b)) numpy: 0.275676703453 Skipping weave timing numexpr: 0.24797129631 Speed-up of numexpr over numpy: 1.11172828289 Expression: 2*a + arctan2(a, b) numpy: 0.139141917229 Skipping weave timing numexpr: 0.121482086182 Speed-up of numexpr over numpy: 1.14536983684 Expression: a**2 + (b+1)**-2.5 numpy: 0.330592417717 Skipping weave timing numexpr: 0.04945499897 Speed-up of numexpr over numpy: 6.68471185122 Expression: (a+1)**50 numpy: 0.281901097298 Skipping weave timing numexpr: 0.0324407100677 Speed-up of numexpr over numpy: 8.68973264484 Expression: sqrt(a**2 + b**2) numpy: 0.0694071054459 Skipping weave timing numexpr: 0.0360074043274 Speed-up of numexpr over numpy: 1.92757869506 Average = 3.35313713426 Averages: 3.10, 3.35, 3.35 From faltet at carabos.com Wed Mar 14 17:28:32 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 14 Mar 2007 22:28:32 +0100 Subject: [Numpy-discussion] Which dtype are supported by numexpr ? In-Reply-To: <20070314161009.7816899c@arbutus.physics.mcmaster.ca> References: <200703091856.30141.faltet@carabos.com> <200703141302.13084.faltet@carabos.com> <20070314161009.7816899c@arbutus.physics.mcmaster.ca> Message-ID: <1173907712.2543.91.camel@localhost.localdomain> El dc 14 de 03 del 2007 a les 16:10 -0400, en/na David M. Cooke va escriure: > On Wed, 14 Mar 2007 13:02:10 +0100 > Francesc Altet wrote: > > > The info above is somewhat inexact. I was talking about the enhanced > > numexpr version included in PyTables 2.0 (see [1]). The original version of > > numexpr (see [2]) doesn't have support for int64 on 32-bit platforms and > > also neither does for strings. Sorry for the confusion. > > What are you doing with strings in numexpr? Ivan Vilata i Balaguer submitted > a patch to add them, but I rejected it as I didn't have any use-cases for > strings that needed the speed of numexpr, and were worth adding the > complexity. Well, basically we need them because we use an (modified) instance of numexpr internally in the new PyTables 2.0 in order to allow selections of data on tables (kind of recarrays, but on-disk). For example, if we have a table with 3 columns (col1, col2 and col3) with three different datatypes (say string, int64 and float64 repectively), the user will be able to do things like: expr = '(((col1 == "item1") & (col2 < 3)) | (col3 > 24.1e12))' selected_rows = [ row.now for row in table.where(expr) ] Of course, using numexpr for doing this kind of stuff is twofold: - The expresion interpreter comes integrated - The computations are done much faster than using plain numpy However, as I've shown on a recent message: http://thread.gmane.org/gmane.comp.python.numeric.general/14130 adding complexity to the numexpr kernel is hurting its performance (most specially in CPUs with reduced L2 caches), so perhaps is not a good idea to add this kind of stuff (I mean, string support) to the generic numexpr. In that sense, we will also see what we can do for reducing the kernel computing in the pytables version. So, unfortunately enough, it seems that we will have to end maintaining our own version of numexpr for pytables purposes :-/ Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From subscriber100 at rjs.org Wed Mar 14 18:02:38 2007 From: subscriber100 at rjs.org (Ray S) Date: Wed, 14 Mar 2007 14:02:38 -0800 Subject: [Numpy-discussion] zoom FFT with numpy? Message-ID: <6.2.3.4.2.20070314132537.05ccf838@blue-cove.com> We'd like to do what most call a "zoom FFT"; we only are interested in the frequencies of say, 6kHZ to 9kHz with a given N, and so the computations from DC to 6kHz are wasted CPU time. Can this be done without additional numpy pre-filtering computations? If explicit filtering is needed to "baseband" the data, is it worth it? It sounds like we need to generate cosine data once for each band and N, then multiple with the data for each FFT. Has anyone coded this up before? I couldn't find any references other than http://www.dsprelated.com/showmessage/38917/1.php and http://www.numerix-dsp.com/siglib/examples/test_zft.c (which uses Numerix). Ray From wbaxter at gmail.com Wed Mar 14 18:47:22 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 15 Mar 2007 07:47:22 +0900 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: Thanks, Sebastian. I'll take a look at Pyro. Hadn't heard of it. I'm using just xmlrpclib with pickle right now. --bb On 3/15/07, Sebastian Haase wrote: > Hey Bill, > what are you using to communicate with the server ? > May I recommend looking at Pyro ! > (Python remote objects) > It would allow you to get your proxy objects. > And also handles exception super clean and easy. > I have used it for many years ! It's very stable ! > > (If you run into problems, take a look at the "one-way" calls to > ensure that functions, that would block, won't wait for the function > to return) > > Just a thought -- > -Sebastian Haase > > > On 3/13/07, Bill Baxter wrote: > > Howdy Folks, > > > > I was missing the good ole days of using Matlab back at the Uni when I > > could debug my code, stop at breakpoints and plot various data without > > fear of blocking the interpreter process. > > > > Using "ipython -pylab" is what has been suggested to me in the past, > > but the problem is I don't do my debugging from ipython. I have a > > very nice IDE that works very well, and it has a lovely interactive > > debugging prompt that I can use to probe my code when stopped at a > > breakpoint. It's great except I can't really use matplotlib for > > debugging there because it causes things to freeze up. > > > > So I've come up with a decent (though not perfect) solution for > > quickie interactive plots which is to run matplotlib in a separate > > process. I call the result it 'ezplot'. The first alpha version of > > this is now available at the Cheeseshop. (I made an egg too, so if > > you have setuptools you can do "easy_install ezplot".) > > > > The basic usage is like so: > > > > In [1]: import ezplot > > In [2]: p = ezplot.Plotter() > > In [3]: p.plot([1,2,3],[1,4,9],marker='o') > > Connecting to server... waiting... > > connected to plotserver 0.1.0a1 on http://localhost:8397 > > Out[3]: True > > In [4]: from numpy import * > > In [5]: x = linspace(-5,5,20) > > In [13]: p.clf() > > Out[13]: True > > In [14]: p.plot(x, x*x*log(x*x+0.01)) > > > > (Imagine lovely plots popping up on your screen as these commands are typed.) > > > > The only return values you get back are True (success...probably) or > > False (failure...for sure). So no fancy plot object manipulation is > > possible. But you can do basic plots no problem. > > > > The nice part is that this (unlike ipython's built-in -pylab threading > > mojo) should work just as well from wherever you're using python. > > Whether it's ipython (no -pylab) or Idle, or a plain MS-DOS console, > > or WingIDE's debug probe, or SPE, or a PyCrust shell or whatever. It > > doesn't matter because all the client is doing is packing up data and > > shipping over a socket. All the GUI plotting mojo happens in a > > completely separate process. > > > > There are plenty of ways this could be made better, but for me, for > > now, this probably does pretty much all I need, so it's back to Real > > Work. But if anyone is interested in making improvements to this, let > > me know. > > > > Here's a short list of things that could be improved: > > * Right now I assume use of the wxAGG backend for matplotlib. Don't > > know how much work it would be to support other back ends (or how to > > go about it, really). wxAGG is what I always use. > > * Returning more error/exception info from the server would be nice > > * Returning full fledged proxy plot objects would be nice too, but I > > suspect that's a huge effort > > * SOAP may be better for this than xmlrpclib but I just couldn't get > > it to work (SOAPpy + Twisted). > > * A little more safety would be nice. Anyone know how to make a > > Twisted xmlrpc server not accept connections from anywhere except > > localhost? > > * There's a little glitch in that the spawned plot server dies with > > the parent that created it. Maybe there's a flag to subprocess.Popen > > to fix that? > > * Sometimes when you click on "Exit Server", if there are plot windows > > open it hangs while shutting down. > > > > > > Only tested on Win32 but there's nothing much platform specific in there. > > > > Give it a try and let me know what you think! > > > > --bb > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From dmahler at gmail.com Wed Mar 14 21:59:35 2007 From: dmahler at gmail.com (Daniel Mahler) Date: Thu, 15 Mar 2007 01:59:35 +0000 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F580C9.70206@ee.byu.edu> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> <45F580C9.70206@ee.byu.edu> Message-ID: On 3/12/07, Travis Oliphant wrote: > I'm not convinced that the broadcasting is causing the slow-downs. > Currently, the code has two path-ways. One gets called when the inputs > are scalars which is equivalent to the old code and the second gets > called when broadcasting is desired. Their should be only a very small > speed hit because of the check. So, it should be possible to do both > without great differences in speed. > > Perhaps the speed hit is due to something else (like creating 0-d arrays > from scalars, or something similar). So, in other words, I think the > problem is fixable, but we need someone who can track down where the > slowness is coming from. > > -Travis The problem is very easy to reproduce. What would it take to do the tracking? I'd like to see this fixed. Presumably the python profiler is no use since the problem is inside C code, which is not my expertise. D From wbaxter at gmail.com Wed Mar 14 22:26:41 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 15 Mar 2007 11:26:41 +0900 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: On 3/15/07, Bill Baxter wrote: > Thanks, Sebastian. I'll take a look at Pyro. Hadn't heard of it. > I'm using just xmlrpclib with pickle right now. I took a look at Pyro -- it looks nice. The only thing I couldn't find, though, is how decouple the wx GUI on the server side from the Pyro remote call handler. Both wx and Pyro want to run a main loop. With the XML-RPC, I could use twisted and its wxreactor class. That does all the necessary magic under the hood to run both loops. Basically all you have to do to make it work is: class MainApp(wx.App, twisted.web.xmlrpc.XMLRPC): ... twisted.internet.wxreactor.install() app = MainApp() twisted.internet.reactor.registerWxApp(app) twisted.internet.reactor.run() And then you're good to go. reactor.run() takes care of both main loops somehow. Do you know of any good examples showing how to do that sort of thing with Pyro? It must be possible. I mean it's the exact same sort of thing you'd need if you're writing a simple GUI internet chat program. My googling has turned up nothing, though. --bb From charlesr.harris at gmail.com Wed Mar 14 23:24:07 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 14 Mar 2007 21:24:07 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> <45F580C9.70206@ee.byu.edu> Message-ID: On 3/14/07, Daniel Mahler wrote: > > On 3/12/07, Travis Oliphant wrote: > > > I'm not convinced that the broadcasting is causing the slow-downs. > > Currently, the code has two path-ways. One gets called when the inputs > > are scalars which is equivalent to the old code and the second gets > > called when broadcasting is desired. Their should be only a very small > > speed hit because of the check. So, it should be possible to do both > > without great differences in speed. > > > > Perhaps the speed hit is due to something else (like creating 0-d arrays > > from scalars, or something similar). So, in other words, I think the > > problem is fixable, but we need someone who can track down where the > > slowness is coming from. > > > > -Travis > > The problem is very easy to reproduce. > What would it take to do the tracking? > I'd like to see this fixed. > Presumably the python profiler is no use since the problem is inside C > code, > which is not my expertise. Well, it is not creation of scalars per se that is slow. In [6]: def normal1() : ...: a = normal(0,1,1000) ...: for i in xrange(1000) : ...: b = a[i] In [7]: def normal2() : ...: for i in xrange(1000) : ...: b = normal(0,1) In [18]: t1=Timer("normal1()","from __main__ import normal1") In [19]: t2=Timer("normal2()","from __main__ import normal2") In [22]: t1.timeit(100) Out[22]: 0.086430072784423828 In [23]: t2.timeit(100) Out[23]: 4.276231050491333 Looks to me like the call overhead is dominate and that the fast path is not nearly as fast as it could be. Either it takes a long time to decide which path to take or something is getting reinitialized that needn't. Could be that normals should be special cased but I can't tell till I look at the code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Mar 14 23:40:10 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 14 Mar 2007 21:40:10 -0600 Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: <6.2.3.4.2.20070314132537.05ccf838@blue-cove.com> References: <6.2.3.4.2.20070314132537.05ccf838@blue-cove.com> Message-ID: On 3/14/07, Ray S wrote: > > We'd like to do what most call a "zoom FFT"; we only are interested > in the frequencies of say, 6kHZ to 9kHz with a given N, and so the > computations from DC to 6kHz are wasted CPU time. > Can this be done without additional numpy pre-filtering computations? > > If explicit filtering is needed to "baseband" the data, is it worth > it? It sounds like we need to generate cosine data once for each band > and N, then multiple with the data for each FFT. > > Has anyone coded this up before? I couldn't find any references other > than http://www.dsprelated.com/showmessage/38917/1.php > and http://www.numerix-dsp.com/siglib/examples/test_zft.c (which uses > Numerix). Sounds like you want to save cpu cycles. How much you can save will depend on the ratio of the bandwidth to the nyquist. Generally, it is easiest to just fft, but if the band is very narrow and time matters, you might try: 1) explicit multipy by the transform matrix but only use a few rows. 2) frequency shift, {low pass}, and downsample 3) just downsample and fix up the aliasing Much depends on the length of the data run and what the signal to noise is. Google indicates that the zoom fft is 2 without the low pass, which is fine if you don't need optimum s/n in the result. Should be easy to cook up your own routine. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From subscriber100 at rjs.org Thu Mar 15 01:24:25 2007 From: subscriber100 at rjs.org (Ray Schumacher) Date: Wed, 14 Mar 2007 21:24:25 -0800 Subject: [Numpy-discussion] zoom FFT with numpy? Message-ID: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> On 3/14/07, "Charles R Harris" wrote: > Sounds like you want to save cpu cycles. > How much you can save will depend > on the ratio of the bandwidth to the nyquist. The desired band is rather narrow, as the goal is to determine the f of a peak that always occurs in a narrow band of about 1kHz around 7kHz >2) frequency shift, {low pass}, and downsample > Much depends on the length of the data run and what the signal to noise is. data is stable for 2-4k samps, and s/n is low, ~5-10 > Google indicates that the zoom fft is 2 without the low pass, which is fine if you don't need optimum s/n in the result. What I had been doing is a 2048 N full real_FFT with a Hann window, and further analyzing the side lobe/bin energy (via linear interp) to try to more precisely determine the f within the peak's bin. (How legitimately valuable that is I'm not sure... IANAM) It's awfully hard to beat the FFTPACK library's full FFT by hand-rolling anything else, I'd guess. I was hoping for some all-numpy procedure that I could use weave with; that might stand a chance of being faster. Thanks, Ray From oliphant at ee.byu.edu Thu Mar 15 00:57:46 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 14 Mar 2007 22:57:46 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: References: <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> <45F580C9.70206@ee.byu.edu> Message-ID: <45F8D24A.9060201@ee.byu.edu> Charles R Harris wrote: > > > On 3/14/07, *Daniel Mahler* > wrote: > > On 3/12/07, Travis Oliphant > wrote: > > > I'm not convinced that the broadcasting is causing the slow-downs. > > Currently, the code has two path-ways. One gets called when the > inputs > > are scalars which is equivalent to the old code and the second gets > > called when broadcasting is desired. Their should be only a > very small > > speed hit because of the check. So, it should be possible to > do both > > without great differences in speed. > > > > Perhaps the speed hit is due to something else (like creating > 0-d arrays > > from scalars, or something similar). So, in other words, I > think the > > problem is fixable, but we need someone who can track down where > the > > slowness is coming from. > > > > -Travis > > The problem is very easy to reproduce. > What would it take to do the tracking? > I'd like to see this fixed. > Presumably the python profiler is no use since the problem is > inside C code, > which is not my expertise. > > > Well, it is not creation of scalars per se that is slow. I think the problem is in two places (with 2 being the largest slow-down). 1) Conversion of scalars to 0-d arrays 2) Using ufuncs to check for illegal parameters. What used to be a simple Python object comparison is now a ufunc. While this works for arrays of parameters it is *a lot* slower for scalars. The solution as I see it is to branch earlier down the scalar / array paths so that scalars are never converted to 0-d arrays at all. -Travis From peridot.faceted at gmail.com Thu Mar 15 00:50:44 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 15 Mar 2007 00:50:44 -0400 Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> References: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> Message-ID: On 15/03/07, Ray Schumacher wrote: > The desired band is rather narrow, as the goal is to determine the f of a > peak that always occurs in a narrow band of about 1kHz around 7kHz > > >2) frequency shift, {low pass}, and downsample By this I would take it to mean, multiply by a complex sinusoid at 7 kHz, optionally low-pass filter, dispose of every high frequency in the result by downsampling (boxcar averaging will be an ultra-simple way to include a crummy low-pass filter) then FFT. (Or some equivalent purely-real operation, doesn't much matter.) Basically implement a heterodyne receiver in software. > > Much depends on the length of the data run and what the signal to noise is. > > data is stable for 2-4k samps, and s/n is low, ~5-10 It seems like this would benefit from some filtering... you could try numpy.convolve to apply a FIR filter, but scipy.signal has a whole heap of tools for efficiently designing and applying FIR and IIR filters. If you've got a good filter design arrangement, I suppose you could just band-pass filter the data then downsample and FFT; the aliasing would drop the frequency you want into the band-pass in a predictable way. Saves generating and multiplying the complex sinusoids. > What I had been doing is a 2048 N full real_FFT with a Hann window, and > further analyzing the side lobe/bin energy (via linear interp) to try to > more precisely determine the f within the peak's bin. (How legitimately > valuable that is I'm not sure... IANAM) This can be a good idea; if what you're observing really is an isolated sinusoidal signal, you can get frequency accuracy limited only by your signal-to-noise ratio. Off the top of my head I don't remember how you do the magic, but Ransom et al. 2002 ("Fourier Techniques for Very Long Astrophysical Time Series Analysis", published in ApJ but more accessible at http://lanl.arxiv.org/abs/astro-ph/0204349 ; hi Scott!) describe the process without undue pain. They also discuss the issue of computing only part of an FFT and claim that it's rarely worth the trouble. > It's awfully hard to beat the FFTPACK library's full FFT by hand-rolling > anything else, I'd guess. I was hoping for some all-numpy procedure that I > could use weave with; that might stand a chance of being faster. Well, mixing it down then FFTing should be pretty fast. Particularly if the problem is that you have to do it a zillion times, rather than that your FFT is huge, so that you can keep the LO array around. And scipy.signal should help. Anne M. Archibald From charlesr.harris at gmail.com Thu Mar 15 00:59:17 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 14 Mar 2007 22:59:17 -0600 Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> References: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> Message-ID: On 3/14/07, Ray Schumacher wrote: > > On 3/14/07, "Charles R Harris" wrote: > > > Sounds like you want to save cpu cycles. > > How much you can save will depend > > on the ratio of the bandwidth to the nyquist. > > The desired band is rather narrow, as the goal is to determine the f of a > peak that always occurs in a narrow band of about 1kHz around 7kHz > > >2) frequency shift, {low pass}, and downsample > > > Much depends on the length of the data run and what the signal to noise > is. > > data is stable for 2-4k samps, and s/n is low, ~5-10 > > > Google indicates that the zoom fft is 2 without the low pass, which is > fine > if you don't need optimum s/n in the result. > > What I had been doing is a 2048 N full real_FFT with a Hann window, and > further analyzing the side lobe/bin energy (via linear interp) to try to > more precisely determine the f within the peak's bin. (How legitimately > valuable that is I'm not sure... IANAM) That's usually fine. You might want to zero fill to get more samples through the band. It would help if you gave the sample frequency in Hz too. Anyway, unless time is important, I would just zero fill by a factor of 4-8 and transform. You can get the same effect with a chirp-z transform, but again this is depends on how much programming work you want to do. If you just have a couple of lines in the band that you want to locate you could also try maximum entropy methods. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From strawman at astraw.com Thu Mar 15 04:07:20 2007 From: strawman at astraw.com (Andrew Straw) Date: Thu, 15 Mar 2007 01:07:20 -0700 Subject: [Numpy-discussion] [Matplotlib-users] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: <45F8FEB8.3070508@astraw.com> Bill, very cool. Also, thanks for showing me how Twisted can be used like Pyro, more-or-less, I think. (If I understand your code from my 1 minute perusal.) On Mac OS X, there's one issue I don't have time to follow any further: sys.executable points to /Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python whereas /Library/Frameworks/Python.framework/Versions/Current/bin/python is the file actually on my path. For some reason, when I run the latter ezplot is found, when the former, it is not. Thus, your auto-spawning of a plotserver instance fails on my installation. Other than that, the example you gave works as advertised and looks great. (Ohh, those anti-aliased lines look better and better the more I suffer through my colleagues' aliased plots...) Bill Baxter wrote: > Howdy Folks, > > I was missing the good ole days of using Matlab back at the Uni when I > could debug my code, stop at breakpoints and plot various data without > fear of blocking the interpreter process. > > Using "ipython -pylab" is what has been suggested to me in the past, > but the problem is I don't do my debugging from ipython. I have a > very nice IDE that works very well, and it has a lovely interactive > debugging prompt that I can use to probe my code when stopped at a > breakpoint. It's great except I can't really use matplotlib for > debugging there because it causes things to freeze up. > > So I've come up with a decent (though not perfect) solution for > quickie interactive plots which is to run matplotlib in a separate > process. I call the result it 'ezplot'. The first alpha version of > this is now available at the Cheeseshop. (I made an egg too, so if > you have setuptools you can do "easy_install ezplot".) > > The basic usage is like so: > > In [1]: import ezplot > In [2]: p = ezplot.Plotter() > In [3]: p.plot([1,2,3],[1,4,9],marker='o') > Connecting to server... waiting... > connected to plotserver 0.1.0a1 on http://localhost:8397 > Out[3]: True > In [4]: from numpy import * > In [5]: x = linspace(-5,5,20) > In [13]: p.clf() > Out[13]: True > In [14]: p.plot(x, x*x*log(x*x+0.01)) > > (Imagine lovely plots popping up on your screen as these commands are typed.) > > The only return values you get back are True (success...probably) or > False (failure...for sure). So no fancy plot object manipulation is > possible. But you can do basic plots no problem. > > The nice part is that this (unlike ipython's built-in -pylab threading > mojo) should work just as well from wherever you're using python. > Whether it's ipython (no -pylab) or Idle, or a plain MS-DOS console, > or WingIDE's debug probe, or SPE, or a PyCrust shell or whatever. It > doesn't matter because all the client is doing is packing up data and > shipping over a socket. All the GUI plotting mojo happens in a > completely separate process. > > There are plenty of ways this could be made better, but for me, for > now, this probably does pretty much all I need, so it's back to Real > Work. But if anyone is interested in making improvements to this, let > me know. > > Here's a short list of things that could be improved: > * Right now I assume use of the wxAGG backend for matplotlib. Don't > know how much work it would be to support other back ends (or how to > go about it, really). wxAGG is what I always use. > * Returning more error/exception info from the server would be nice > * Returning full fledged proxy plot objects would be nice too, but I > suspect that's a huge effort > * SOAP may be better for this than xmlrpclib but I just couldn't get > it to work (SOAPpy + Twisted). > * A little more safety would be nice. Anyone know how to make a > Twisted xmlrpc server not accept connections from anywhere except > localhost? > * There's a little glitch in that the spawned plot server dies with > the parent that created it. Maybe there's a flag to subprocess.Popen > to fix that? > * Sometimes when you click on "Exit Server", if there are plot windows > open it hangs while shutting down. > > > Only tested on Win32 but there's nothing much platform specific in there. > > Give it a try and let me know what you think! > > --bb > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Matplotlib-users mailing list > Matplotlib-users at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > From strawman at astraw.com Thu Mar 15 04:20:40 2007 From: strawman at astraw.com (Andrew Straw) Date: Thu, 15 Mar 2007 01:20:40 -0700 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: <45F901D8.5010803@astraw.com> Bill Baxter wrote: > On 3/15/07, Bill Baxter wrote: > >> Thanks, Sebastian. I'll take a look at Pyro. Hadn't heard of it. >> I'm using just xmlrpclib with pickle right now. >> > > I took a look at Pyro -- it looks nice. > The only thing I couldn't find, though, is how decouple the wx GUI on > the server side from the Pyro remote call handler. Both wx and Pyro > want to run a main loop. > > With the XML-RPC, I could use twisted and its wxreactor class. That > does all the necessary magic under the hood to run both loops. > Basically all you have to do to make it work is: > > class MainApp(wx.App, twisted.web.xmlrpc.XMLRPC): > ... > > twisted.internet.wxreactor.install() > app = MainApp() > twisted.internet.reactor.registerWxApp(app) > twisted.internet.reactor.run() > > And then you're good to go. reactor.run() takes care of both main > loops somehow. > > Do you know of any good examples showing how to do that sort of thing > with Pyro? It must be possible. I mean it's the exact same sort of > thing you'd need if you're writing a simple GUI internet chat program. > My googling has turned up nothing, though. It is possible to do this with Pyro. I think Pyro can "auto-background" (my terminology not theirs) if you allow it to do threading. Otherwise, you could handle requests by 1) putting Pyro in a thread you manage or 2) a GUI timer calling Pyro's daemon.handleRequests(...) fairly often. But I hear nothing but good things about Twisted, and your code works, so I say find some bigger fish to fry (especially since I can see potential issues with each of the above options...). From miguel.oliveira.jr at gmail.com Thu Mar 15 05:05:40 2007 From: miguel.oliveira.jr at gmail.com (Miguel Oliveira, Jr.) Date: Thu, 15 Mar 2007 09:05:40 +0000 Subject: [Numpy-discussion] arrayrange In-Reply-To: <200703141855.42724.eike.welk@gmx.net> References: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> <200703141855.42724.eike.welk@gmx.net> Message-ID: <45E22AA7-E97C-4779-935F-DE171C578BC3@gmail.com> Hi Eike! Yes, I did subscribe! Thanks once again for your reply... I'm sorry but I don't know exactly what you mean... Do you think I should replace t = arange(0, durSecs, 1.0/SRate) by t = linspace(0, durSecs, durSecs*SRate)? That won't work... Maybe I am missing something?... The resulting file should be am aiff sine wave... Best, Miguel On 14 Mar 2007, at 17:55, Eike Welk wrote: > I would use something like this: > t = linspace(0, durSecs, durSecs*SRate) > > Do you know the 'Numpy Example List' > http://www.scipy.org/Numpy_Example_List > > Regards Eike. > > PS: Ah, you did subscribe. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From miguel.oliveira.jr at gmail.com Thu Mar 15 05:08:45 2007 From: miguel.oliveira.jr at gmail.com (Miguel Oliveira, Jr.) Date: Thu, 15 Mar 2007 09:08:45 +0000 Subject: [Numpy-discussion] arrayrange In-Reply-To: <45F8318D.9030209@gmail.com> References: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> <45F8318D.9030209@gmail.com> Message-ID: <1E779C59-8259-4124-BA45-E44B602DB5F2@gmail.com> Hi Robert, I'm expecting that the code generates a sine wave sound file in aiff format... Apparently the code worked before when arrayrange() was used... Thanks! M. On 14 Mar 2007, at 17:31, Robert Kern wrote: > Miguel Oliveira, Jr. wrote: >> Hello, >> >> I've got a few codes that use "arrayrange" within numpy. It happens >> that the new version of numpy apparently doesn't recognize >> "arrayrange"... I've tried to change it to "arange", but it doesn't >> work... So, for example, the code below used to create a sine wave >> file, but it's not working anymore... Does anyone have any clue of >> what may be a solution for this? > > arange() is the appropriate replacement for arrayrange(); arrayrange > () was just > an alias to arange(). What did you expect it to do that it isn't > doing? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a > harmless enigma > that is made terrible by our own mad attempt to interpret it as > though it had > an underlying truth." > -- Umberto Eco > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From nadavh at visionsense.com Thu Mar 15 05:53:49 2007 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 15 Mar 2007 11:53:49 +0200 Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: <6.2.3.4.2.20070314132537.05ccf838@blue-cove.com> References: <6.2.3.4.2.20070314132537.05ccf838@blue-cove.com> Message-ID: <1173952429.8746.3.camel@nadav.envision.co.il> A long time ago I translated a free code of chirp z transform (zoom fft) into python. Attached here the source and two translations (probably one of the is right) Nadav. On Wed, 2007-03-14 at 14:02 -0800, Ray S wrote: > We'd like to do what most call a "zoom FFT"; we only are interested > in the frequencies of say, 6kHZ to 9kHz with a given N, and so the > computations from DC to 6kHz are wasted CPU time. > Can this be done without additional numpy pre-filtering computations? > > If explicit filtering is needed to "baseband" the data, is it worth > it? It sounds like we need to generate cosine data once for each band > and N, then multiple with the data for each FFT. > > Has anyone coded this up before? I couldn't find any references other > than http://www.dsprelated.com/showmessage/38917/1.php > and http://www.numerix-dsp.com/siglib/examples/test_zft.c (which uses > Numerix). > > Ray > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: czt2.py Type: text/x-python Size: 2564 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: czt1.py Type: text/x-python Size: 3051 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: czt.m Type: text/x-objcsrc Size: 2545 bytes Desc: not available URL: From bblais at bryant.edu Thu Mar 15 06:01:34 2007 From: bblais at bryant.edu (Brian Blais) Date: Thu, 15 Mar 2007 06:01:34 -0400 Subject: [Numpy-discussion] correct way to specify type in array definition Message-ID: <45F9197E.9070001@bryant.edu> Hello, Can someone tell me what the preferred way to specify the type of an array? I want it to be a float array, no matter what is given (say, integers). I can do: a=numpy.array([1,2,3],numpy.dtype('float')) or a=numpy.array([1,2,3],type(1.0)) or perhaps many others. Is there a way that is recommended? thanks, Brian Blais -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From fullung at gmail.com Thu Mar 15 06:11:15 2007 From: fullung at gmail.com (Albert Strasheim) Date: Thu, 15 Mar 2007 12:11:15 +0200 Subject: [Numpy-discussion] Possible bug in PyArray_RemoveSmallest? Message-ID: <20070315101115.GA15365@dogbert.sdsl.sun.ac.za> Hello all I was poking around in the NumPy internals and I came across the following code in PyArray_RemoveSmallest in arrayobject.c: intp sumstrides[NPY_MAXDIMS]; ... for (i=0; ind; i++) { sumstrides[i] = 0; for (j=0; jnumiter; j++) { sumstrides[i] = multi->iters[j]->strides[i]; } } This might be a red herring, but from the name of the variable (sumstrides) and the code (iterating over a bunch of strides) I'm guessing the author might have intended to write: sumstrides[i] += multi->iters[j]->strides[i]; Cheers, Albert From faltet at carabos.com Thu Mar 15 06:23:36 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 15 Mar 2007 11:23:36 +0100 Subject: [Numpy-discussion] correct way to specify type in array definition In-Reply-To: <45F9197E.9070001@bryant.edu> References: <45F9197E.9070001@bryant.edu> Message-ID: <1173954217.2606.2.camel@localhost.localdomain> El dj 15 de 03 del 2007 a les 06:01 -0400, en/na Brian Blais va escriure: > Hello, > > Can someone tell me what the preferred way to specify the type of an array? I want > it to be a float array, no matter what is given (say, integers). I can do: > > a=numpy.array([1,2,3],numpy.dtype('float')) > > or > > a=numpy.array([1,2,3],type(1.0)) > > or perhaps many others. Is there a way that is recommended? Well, this depends on your preferences, I guess, but I like to be explicit, so I normally use: a=numpy.array([1,2,3], numpy.float64) but, if you are a bit lazy to type, the next is just fine as well: a=numpy.array([1,2,3], 'f8') Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From ogdude at googlemail.com Thu Mar 15 07:26:10 2007 From: ogdude at googlemail.com (David Koch) Date: Thu, 15 Mar 2007 12:26:10 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <20070314184605.GD18520@clipper.ens.fr> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> Message-ID: On 3/14/07, Gael Varoquaux wrote: > > > I definitely second this comment. Using arrays when you are trying to > append a lot of data is using the wrong data format. And the code is so > much more readable with lists. Thank you, I will consider it, Next thing, I have A_Idx = array([1, 0]) and B is a matrix. How would I select the four elements in B indicated by "meshgrid(A_Idx)", that ist: B[1,1], B[1,0], B[0,1], B[0,0] In Matlab you would simply use B(A_idx, A_idx) whereas in NumPy B[A_Idx, A_idx] returns B[1,1] and B[0,0] only. I read that A.take(A_idx).take(A_idx, axis = 1) should work but it doesn't. Thank you, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogdude at googlemail.com Thu Mar 15 09:49:53 2007 From: ogdude at googlemail.com (David Koch) Date: Thu, 15 Mar 2007 14:49:53 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> Message-ID: On 3/15/07, David Koch wrote: > > ... NumPy equiv for Matlab B(A_idx, A_Idx) Ok, I did like this: A_Idx = array([1, 0]) B = random.randn(3,3) rowInd = kron(ones((1, len(A_Idx)), int), A_Idx[:, newaxis]) colInd = kron(ones((len(A_Idx), 1), int), A_Idx) B[rowInd, colInd] - returns the equivalent of Matlabs B(A_Idx, A_Idx) ... that cannot possibly be the easiest way :-/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Thu Mar 15 11:07:29 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 16 Mar 2007 00:07:29 +0900 Subject: [Numpy-discussion] Broadcasting for hstack, vstack etc Message-ID: I just had a need to append a column of 1's to an array, and given how big numpy is on broadcasting I thought this might work: column_stack((m1,m2, 1)) But it doesn't. Is there any reason why that couldn't or shouldn't be made to work? --bb From robert.kern at gmail.com Thu Mar 15 12:52:41 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 15 Mar 2007 11:52:41 -0500 Subject: [Numpy-discussion] arrayrange In-Reply-To: <1E779C59-8259-4124-BA45-E44B602DB5F2@gmail.com> References: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> <45F8318D.9030209@gmail.com> <1E779C59-8259-4124-BA45-E44B602DB5F2@gmail.com> Message-ID: <45F979D9.6030601@gmail.com> Miguel Oliveira, Jr. wrote: > Hi Robert, > > I'm expecting that the code generates a sine wave sound file in aiff > format... Apparently the code worked before when arrayrange() was > used... Then the problem lies somewhere else; a number of other things changed between Numeric and numpy. arrayrange() *was* arange(). If you think the problem is still with arange(), then you need to show us what you think arange() *by itself* should be producing and what it is actually producing. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Mar 15 12:53:06 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 15 Mar 2007 11:53:06 -0500 Subject: [Numpy-discussion] arrayrange In-Reply-To: <45E22AA7-E97C-4779-935F-DE171C578BC3@gmail.com> References: <0F8FF6EE-4C47-4440-88CF-C21C7C99940E@gmail.com> <200703141855.42724.eike.welk@gmx.net> <45E22AA7-E97C-4779-935F-DE171C578BC3@gmail.com> Message-ID: <45F979F2.1050908@gmail.com> Miguel Oliveira, Jr. wrote: > Hi Eike! > > Yes, I did subscribe! Thanks once again for your reply... I'm sorry > but I don't know exactly what you mean... Do you think I should replace > t = arange(0, durSecs, 1.0/SRate) > by > t = linspace(0, durSecs, durSecs*SRate)? > > That won't work... *How* does it not work? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From roberto at dealmeida.net Thu Mar 15 13:00:58 2007 From: roberto at dealmeida.net (Roberto De Almeida) Date: Thu, 15 Mar 2007 14:00:58 -0300 Subject: [Numpy-discussion] Problem compiling numpy with Python 2.5 on Powerbook In-Reply-To: <870C81DE-4220-430B-827D-A3B2FBE099ED@arachnedesign.net> References: <10c662fe0703081049o490aa22fsa8d1229ef2f038f3@mail.gmail.com> <870C81DE-4220-430B-827D-A3B2FBE099ED@arachnedesign.net> Message-ID: <10c662fe0703151000u626a5e58ief26b7b980fb0fe7@mail.gmail.com> Hi, Steve. On 3/14/07, Steve Lianoglou wrote: > I'm not sure what the problem is exactly, but is it weird that > there's something to do w/ 'i686' when you're running on a powerbook > being that the pbook is PowerPC? I managed to compile numpy by first compiling Python 2.5 as a ppc build only -- the bug occurs when I try to compile numpy using the universal Python build from python.org. I think it's trying to create a universal build of numpy, that's why you see the gcc -arch ppc -arch i386 line on the log, ie, both "ppc" and "i386" architectures. IIRC I don't have the universal SDK installed, so that could be the problem. Thanks, --Rob From robert.kern at gmail.com Thu Mar 15 13:16:03 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 15 Mar 2007 12:16:03 -0500 Subject: [Numpy-discussion] Problem compiling numpy with Python 2.5 on Powerbook In-Reply-To: <10c662fe0703151000u626a5e58ief26b7b980fb0fe7@mail.gmail.com> References: <10c662fe0703081049o490aa22fsa8d1229ef2f038f3@mail.gmail.com> <870C81DE-4220-430B-827D-A3B2FBE099ED@arachnedesign.net> <10c662fe0703151000u626a5e58ief26b7b980fb0fe7@mail.gmail.com> Message-ID: <45F97F53.4090003@gmail.com> Roberto De Almeida wrote: > Hi, Steve. > > On 3/14/07, Steve Lianoglou wrote: >> I'm not sure what the problem is exactly, but is it weird that >> there's something to do w/ 'i686' when you're running on a powerbook >> being that the pbook is PowerPC? > > I managed to compile numpy by first compiling Python 2.5 as a ppc > build only -- the bug occurs when I try to compile numpy using the > universal Python build from python.org. I think it's trying to create > a universal build of numpy, that's why you see the > > gcc -arch ppc -arch i386 > > line on the log, ie, both "ppc" and "i386" architectures. IIRC I don't > have the universal SDK installed, so that could be the problem. That almost certainly is the problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From subscriber100 at rjs.org Thu Mar 15 14:30:40 2007 From: subscriber100 at rjs.org (Ray S) Date: Thu, 15 Mar 2007 10:30:40 -0800 Subject: [Numpy-discussion] zoom FFT with numpy? (Nadav Horesh) Message-ID: <6.2.3.4.2.20070315100720.05cab340@blue-cove.com> Hi Nadev, >A long time ago I translated a free code of chirp z transform (zoom fft) >into python. Thanks, I'll try it out. I did, however read before on the differences: From Numerix http://www.numerix-dsp.com/zoomfft.html: "One common question is : Is the zoom FFT the same as the chirp z-transform. The answer is : Absolutely not. The FFT calculates the FFT at N equally spaced points around the unit circle in the z-plane, the chirp z-transform modifies the locations of these points along a contour that can lie anywhere on the z-plane. In contrast, the zoom-FFT uses digital down conversion techniques to localise the standard FFT to a narrow band of frequencies that are centered on a higher frequency. The chirp z-transform is often used to analyze signals such as speech, that have certain frequency domain charactgeristics. The zoom-FFT is used to reduce the sample rate required when analysing narrowband signals - E.G. in HF communications." http://www-gatago.com/comp/dsp/34830442.html I just saw was good reading too. It will be interesting, and the code is appreciated! Also, czt.c might be particularly fast if compiled with the Intel FFT lib and weave.blitz(). Again, the goal is increased f resolution within a known small band for the ~same CPU cycles... Ray From focke at slac.stanford.edu Thu Mar 15 14:58:40 2007 From: focke at slac.stanford.edu (Warren Focke) Date: Thu, 15 Mar 2007 11:58:40 -0700 (PDT) Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: References: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> Message-ID: On Wed, 14 Mar 2007, Charles R Harris wrote: > On 3/14/07, Ray Schumacher wrote: > > > > What I had been doing is a 2048 N full real_FFT with a Hann window, and > > further analyzing the side lobe/bin energy (via linear interp) to try to > > more precisely determine the f within the peak's bin. (How legitimately > > valuable that is I'm not sure... IANAM) > > > That's usually fine. You might want to zero fill to get more samples through > the band. It would help if you gave the sample frequency in Hz too. Anyway, > unless time is important, I would just zero fill by a factor of 4-8 and > transform. You can get the same effect with a chirp-z transform, but again > this is depends on how much programming work you want to do. If you just > have a couple of lines in the band that you want to locate you could also > try maximum entropy methods. Or do 1) explicit multipy by the transform matrix but only use a few rows. but evaluate the transform matrix on a frequency grid finer than the 1/(2048*tsamp) that is actually independent. Then fit sin(f)/f to the power spectrum. Either of these should give better interpolation than linear. I've seen this done (and pass peer review) to determine pulsar frequencies. I also remain unconvinced that interpolation provides a better result, but that can be determined by analyzing fake data with a known frequency. If you're trying to determine the significance of the result, the fit should somehow take into account the fact that the interpolated data points are not real degrees of freedom. But your power estimates are already not independent since you've applied a Hann window. Probably should also fit to the line response of a Hann window rather than sin(f)/f. Plus something (flat?) for the noise. You could determine the real # of degrees of freedom by simulating the procedure many times (with noise) and fitting a chisquare function to the distribution of fit chisquare values that you see. This would also give an empirical estimate of how well you were determining the frequency, which is probably better than mucking about with degrees of freedom, anyway. w From charlesr.harris at gmail.com Thu Mar 15 15:09:56 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 15 Mar 2007 13:09:56 -0600 Subject: [Numpy-discussion] zoom FFT with numpy? (Nadav Horesh) In-Reply-To: <6.2.3.4.2.20070315100720.05cab340@blue-cove.com> References: <6.2.3.4.2.20070315100720.05cab340@blue-cove.com> Message-ID: On 3/15/07, Ray S wrote: > > Hi Nadev, > >A long time ago I translated a free code of chirp z transform (zoom > fft) > >into python. > > Thanks, I'll try it out. > > I did, however read before on the differences: > From Numerix http://www.numerix-dsp.com/zoomfft.html: > "One common question is : Is the zoom FFT the same as the chirp > z-transform. > > The answer is : Absolutely not. The FFT calculates the FFT at N > equally spaced points around the unit circle in the z-plane, the > chirp z-transform modifies the locations of these points along a > contour that can lie anywhere on the z-plane. In contrast, the > zoom-FFT uses digital down conversion techniques to localise the > standard FFT to a narrow band of frequencies that are centered on a > higher frequency. The points in the chirp z-transform can be densely spaced in the band you are interested in, so it is probably closer to what you want to do. The same effect can be obtained by zero filling, but that is probably not as efficient unless you down sample first. Essentially, you will end up with sinc interpolated points in the spectrum and you can just search for the max. BTW, did the s/n you gave refer to the signal or to it's transform? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Mar 15 15:29:51 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 15 Mar 2007 13:29:51 -0600 Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: References: <5.2.1.1.2.20070314205228.02816b50@pop-server.san.rr.com> Message-ID: On 3/15/07, Warren Focke wrote: > > > > On Wed, 14 Mar 2007, Charles R Harris wrote: > > > On 3/14/07, Ray Schumacher wrote: > > > > > > What I had been doing is a 2048 N full real_FFT with a Hann window, > and > > > further analyzing the side lobe/bin energy (via linear interp) to try > to > > > more precisely determine the f within the peak's bin. (How > legitimately > > > valuable that is I'm not sure... IANAM) > > > > > > That's usually fine. You might want to zero fill to get more samples > through > > the band. It would help if you gave the sample frequency in Hz too. > Anyway, > > unless time is important, I would just zero fill by a factor of 4-8 and > > transform. You can get the same effect with a chirp-z transform, but > again > > this is depends on how much programming work you want to do. If you just > > have a couple of lines in the band that you want to locate you could > also > > try maximum entropy methods. > > Or do > > 1) explicit multipy by the transform matrix but only use a few rows. > > but evaluate the transform matrix on a frequency grid finer than the > 1/(2048*tsamp) that is actually independent. Then fit sin(f)/f to the > power spectrum. You can actually zero fill by a factor of two, then build optimum least squares interpolators for bandlimited signals using a reasonable number of sample around each frequency interval. The result can be fitted to single precision accuracy with a 9'th degree polynomial and an ordinary zero solver used on the derivative. Works well, I did this some 20 years ago as part of a package for fourier spectroscopy, but it is probably more work than warranted for the present case. Either of these should give better interpolation than > linear. I've seen this done (and pass peer review) to determine pulsar > frequencies. I also remain unconvinced that interpolation provides a > better result, but that can be determined by analyzing fake data with a > known frequency. > > If you're trying to determine the significance of the result, the fit > should somehow take into account the fact that the interpolated data > points are not real degrees of freedom. But your power estimates are > already not independent since you've applied a Hann window. Probably > should also fit to the line response of a Hann window rather than > sin(f)/f. Sinc interpolation will work fine for the windowed spectrum as it contains the same range of frequencies as the original. Where you can gain something is explicitly interpolating the unwindowed spectrum with the Hann, or stronger, window. Because the window functions fall off much faster than the sinc you don't need to use so many points in the convolution. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Thu Mar 15 17:18:15 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 15 Mar 2007 17:18:15 -0400 Subject: [Numpy-discussion] correct way to specify type in array definition In-Reply-To: <1173954217.2606.2.camel@localhost.localdomain> References: <45F9197E.9070001@bryant.edu> <1173954217.2606.2.camel@localhost.localdomain> Message-ID: <20070315211815.GA18324@arbutus.physics.mcmaster.ca> On Thu, Mar 15, 2007 at 11:23:36AM +0100, Francesc Altet wrote: > El dj 15 de 03 del 2007 a les 06:01 -0400, en/na Brian Blais va > escriure: > > Hello, > > > > Can someone tell me what the preferred way to specify the type of an array? I want > > it to be a float array, no matter what is given (say, integers). I can do: > > > > a=numpy.array([1,2,3],numpy.dtype('float')) > > > > or > > > > a=numpy.array([1,2,3],type(1.0)) > > > > or perhaps many others. Is there a way that is recommended? > > Well, this depends on your preferences, I guess, but I like to be > explicit, so I normally use: > > a=numpy.array([1,2,3], numpy.float64) > > but, if you are a bit lazy to type, the next is just fine as well: > > a=numpy.array([1,2,3], 'f8') > I just do a = numpy.array([1,2,3], dtype=float) The Python types int, float, and bool translate to numpy.int_, numpy.double, and numpy.bool (i.e., the C equivalents of the Pythonn types; note that int_ is a C long). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From jturner at gemini.edu Thu Mar 15 18:01:55 2007 From: jturner at gemini.edu (James Turner) Date: Thu, 15 Mar 2007 18:01:55 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: References: Message-ID: <45F9C253.7080202@gemini.edu> Hi Stefan, Thanks for the suggestions! > Is this related to > > http://projects.scipy.org/scipy/scipy/ticket/213 > > in any way? As far as I can see, the problems look different, but thanks for the example of how to document this. I did confirm that your example exhibits the same behaviour under numarray, in case that is useful. > Code snippets to illustrate the problem would be welcome. OK. I have had a go at producing a code snippet. I apologize that this is based on numarray rather than numpy, since I'm using STScI Python, but I think it should be very easy to convert if you have numpy instead. What I am doing is to transform overlapping input images onto a common, larger grid and co-adding them. Although I'm using affine_ transform on 3D data from FITS images, the issue can be illustrated using a simple 1D translation of a single 2D test array. The input values are just [4., 3., 2., 1.] in each row. With a translation of -0.1, the values should therefore be something like [X, 3.1, 2.1, 1.1, X, X], where the Xs represent points outside the original data range. What I actually get, however, is roughly [X, 3.1, 2.1, 1.0, 1.9, X]. The 5th value of 1.9 contaminates the co-added data in the final output array. Now I'm looking at this element-by-element, I suppose the bad value of 1.9 is just a result of extrapolating in order to preserve the original number of data points, isn't it? Sorry I wasn't clear on that in my original post -- but surely a blank value (as specified by cval) would be better? I suppose I could work around this by blanking out the extrapolated column after doing the affine_transform. I could calculate which is the column to blank out based on the sense of the offset and the input array dimensions. It seems pretty messy and inefficient though. Another idea is to split the translation into integer and fractional parts, keep the input and output array dimensions the same initially and then copy the output into a larger array with integer offsets. That is messy to keep track of though. Maybe a parameter could instead be added to affine_transform that tells it to shrink the number of elements instead of extrapolating? I'd be a bit out of my depth trying to implement that though, even if the authors agree... (maybe in a few months though). Can anyone comment on whether this problem should be considered a bug, or whether it's intended behaviour that I should work around? The code snippet follows below. Thanks for your patience with someone who isn't accustomed to posting questions like this routinely :-). James. ----- import numarray as N import numarray.nd_image as ndi # Create a 2D test pattern: I = N.zeros((2,4),N.Float32) I[:,:] = N.arange(4.0, 0.0, -1.0) # Transformation parameters for a simple translation in 1D: trmatrix = N.array([[1,0],[0,1]]) troffset = (0.0, -0.1) # Apply the offset to the test pattern: I_off1 = ndi.affine_transform(I, trmatrix, troffset, order=3, mode='constant', cval=-1.0, output_shape=(2,6)) I_off2 = ndi.affine_transform(I, trmatrix, troffset, order=3, mode='constant', cval=-1.0, output_shape=(2,6), prefilter=False) # Compare the data before and after interpolation: print I print I_off1 print I_off2 From jturner at gemini.edu Thu Mar 15 18:45:29 2007 From: jturner at gemini.edu (James Turner) Date: Thu, 15 Mar 2007 18:45:29 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <45F9C253.7080202@gemini.edu> References: <45F9C253.7080202@gemini.edu> Message-ID: <45F9CC89.8020509@gemini.edu> Sorry I accidentally broke this thread in 2 (see thread of March 9). I tried manually adding the right reference in the mail header to continue the same thread, but obviously got it wrong (I think because I replied to the digest email instead of starting a new one). Not sure whether there is a better way to reply to messages in a digest (I'm using Thunderbird)? Hope it works this time... James. From robert.kern at gmail.com Thu Mar 15 18:51:04 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 15 Mar 2007 17:51:04 -0500 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <45F9CC89.8020509@gemini.edu> References: <45F9C253.7080202@gemini.edu> <45F9CC89.8020509@gemini.edu> Message-ID: <45F9CDD8.2010302@gmail.com> James Turner wrote: > Sorry I accidentally broke this thread in 2 (see thread of March 9). > > I tried manually adding the right reference in the mail header to > continue the same thread, but obviously got it wrong (I think because > I replied to the digest email instead of starting a new one). Not > sure whether there is a better way to reply to messages in a digest > (I'm using Thunderbird)? Hope it works this time... I see it correctly threaded in Thunderbird. Don't worry about it, regardless. We're smart people; we'll catch on. :-) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From subscriber100 at rjs.org Thu Mar 15 20:49:33 2007 From: subscriber100 at rjs.org (Ray S) Date: Thu, 15 Mar 2007 16:49:33 -0800 Subject: [Numpy-discussion] zoom FFT with numpy? (Nadav Horesh) Message-ID: <6.2.3.4.2.20070315164603.05ca1af0@blue-cove.com> "Charles R Harris" wrote: >BTW, did the s/n you gave refer to the signal or to it's transform? That is the time domain s/n, approximately... Thanks to all for the replies, some digestion is needed. Ray From david at ar.media.kyoto-u.ac.jp Fri Mar 16 00:13:46 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 16 Mar 2007 13:13:46 +0900 Subject: [Numpy-discussion] numpy.distutils, windows dll vs lib Message-ID: <45FA197A.3020808@ar.media.kyoto-u.ac.jp> Hi, recently, I got some problems detecting a dynamic library (dll) with numpy.distutils. Basically, I have a package which uses a class derived from system_info from numpy.distutils to detect a dll I use through ctypes. If only the dll is present, my numpy.distutils.system_info derived class does not find the library; if the .lib is present too, then it is detected. Why is that ? Can I modify my class do detecting the dll is enough ? I don't know how windows dynamic linking and dynamically loaded libraries work, and I am kind of confused by this (I thought .lib was .a, and .dll was .so, and that symbols were not exported by default on windows contrary to Unix, but the difference seems more subtle than this....). cheers, David From robert.kern at gmail.com Fri Mar 16 00:40:36 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 15 Mar 2007 23:40:36 -0500 Subject: [Numpy-discussion] numpy.distutils, windows dll vs lib In-Reply-To: <45FA197A.3020808@ar.media.kyoto-u.ac.jp> References: <45FA197A.3020808@ar.media.kyoto-u.ac.jp> Message-ID: <45FA1FC4.5000008@gmail.com> David Cournapeau wrote: > Hi, > > recently, I got some problems detecting a dynamic library (dll) with > numpy.distutils. Basically, I have a package which uses a class derived > from system_info from numpy.distutils to detect a dll I use through ctypes. > If only the dll is present, my numpy.distutils.system_info derived > class does not find the library; if the .lib is present too, then it is > detected. Why is that ? Can I modify my class do detecting the dll is > enough ? I don't know how windows dynamic linking and dynamically loaded > libraries work, and I am kind of confused by this (I thought .lib was > .a, and .dll was .so, and that symbols were not exported by default on > windows contrary to Unix, but the difference seems more subtle than > this....). Generally, you don't use .dll files to link against. With MSVC, you need a .lib file corresponding to your target .dll file which has the symbols. mingw and cygwin use similar .a files. This information is coded in the system_info.library_extensions() method, which you can override if you need something different. Of course, since you don't mention what methods on system_info that you are using, I can't quite be sure this will satisfy your needs. Note that so_ext here is obtained from the distutils configuration for your platform and is the file extension for extension modules, thus '.pyd' on Windows, not '.dll'. def library_extensions(self): static_exts = ['.a'] if sys.platform == 'win32': static_exts.append('.lib') # .lib is used by MSVC if self.search_static_first: exts = static_exts + [so_ext] else: exts = [so_ext] + static_exts if sys.platform == 'cygwin': exts.append('.dll.a') if sys.platform == 'darwin': exts.append('.dylib') return exts -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at ar.media.kyoto-u.ac.jp Fri Mar 16 00:51:48 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 16 Mar 2007 13:51:48 +0900 Subject: [Numpy-discussion] numpy.distutils, windows dll vs lib In-Reply-To: <45FA1FC4.5000008@gmail.com> References: <45FA197A.3020808@ar.media.kyoto-u.ac.jp> <45FA1FC4.5000008@gmail.com> Message-ID: <45FA2264.50904@ar.media.kyoto-u.ac.jp> Robert Kern wrote: > David Cournapeau wrote: >> Hi, >> >> recently, I got some problems detecting a dynamic library (dll) with >> numpy.distutils. Basically, I have a package which uses a class derived >> from system_info from numpy.distutils to detect a dll I use through ctypes. >> If only the dll is present, my numpy.distutils.system_info derived >> class does not find the library; if the .lib is present too, then it is >> detected. Why is that ? Can I modify my class do detecting the dll is >> enough ? I don't know how windows dynamic linking and dynamically loaded >> libraries work, and I am kind of confused by this (I thought .lib was >> .a, and .dll was .so, and that symbols were not exported by default on >> windows contrary to Unix, but the difference seems more subtle than >> this....). > > Generally, you don't use .dll files to link against. With MSVC, you need a .lib > file corresponding to your target .dll file which has the symbols. mingw and > cygwin use similar .a files. This information is coded in the > system_info.library_extensions() method, which you can override if you need > something different. Of course, since you don't mention what methods on > system_info that you are using, I can't quite be sure this will satisfy your needs. Sorry for the lack of details; I did this a long time ago, and the problem popped up for some people my package with windows (which I don't at all) just recently. I don't use the library for linking: I only need to be able to load it dynamically through ctypes. What I did is simply overriding the calc_info method, in which I try to detect both library and header files. For the library, I do the following: # Look for the shared library sndfile_libs = self.get_libs('sndfile_libs', self.libname) lib_dirs = self.get_lib_dirs() for i in lib_dirs: tmp = self.check_libs(i, sndfile_libs) if tmp is not None: info = tmp break else: return When I look at the system_info.check_libs code, it looks like it is trying to look for any extension, and the first found is returned... But this is not what I get, and I am not sure to see what I am doing wrong. cheers, David From haase at msg.ucsf.edu Fri Mar 16 00:57:44 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 15 Mar 2007 21:57:44 -0700 Subject: [Numpy-discussion] rant against from numpy import * / from pylab import * Message-ID: Hi! I use the wxPython PyShell. I like especially the feature that when typing a module and then the dot "." I get a popup list of all available functions (names) inside that module. Secondly, I think it really makes code clearer when one can see where a function comes from. I have a default import numpy as N executed before my shell even starts. In fact I have a bunch of my "standard" modules imported as . This - I think - is a good compromise to the commonly used "extra typing" and "unreadable" argument. a = sin(b) * arange(10,50, .1) * cos(d) vs. a = N.sin(b) * N.arange(10,50, .1) * N.cos(d) I would like to hear some comments by others. On a different note: I just started using pylab, so I did added an automatic "from matplotlib import pylab as P" -- but now P contains everything that I already have in N. It makes it really hard to *find* (as in *see* n the popup-list) the pylab-only functions. -- what can I do about this ? Thanks, Sebastian From charlesr.harris at gmail.com Fri Mar 16 01:19:41 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 15 Mar 2007 23:19:41 -0600 Subject: [Numpy-discussion] zoom FFT with numpy? (Nadav Horesh) In-Reply-To: <6.2.3.4.2.20070315164603.05ca1af0@blue-cove.com> References: <6.2.3.4.2.20070315164603.05ca1af0@blue-cove.com> Message-ID: On 3/15/07, Ray S wrote: > > "Charles R Harris" wrote: > > >BTW, did the s/n you gave refer to the signal or to it's transform? > > That is the time domain s/n, approximately... Ah, good. The s/n for lines spectra in the transform will go up as the sqrt of the number of sample points so you will end up with a s/n around 160. Chuck. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Mar 16 01:25:32 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 16 Mar 2007 00:25:32 -0500 Subject: [Numpy-discussion] numpy.distutils, windows dll vs lib In-Reply-To: <45FA2264.50904@ar.media.kyoto-u.ac.jp> References: <45FA197A.3020808@ar.media.kyoto-u.ac.jp> <45FA1FC4.5000008@gmail.com> <45FA2264.50904@ar.media.kyoto-u.ac.jp> Message-ID: <45FA2A4C.8070807@gmail.com> David Cournapeau wrote: > I don't use the library for linking: I only need to be able to load it > dynamically through ctypes. What I did is simply overriding the > calc_info method, in which I try to detect both library and header > files. For the library, I do the following: > > # Look for the shared library > sndfile_libs = self.get_libs('sndfile_libs', self.libname) > lib_dirs = self.get_lib_dirs() > for i in lib_dirs: > tmp = self.check_libs(i, sndfile_libs) > if tmp is not None: > info = tmp > break > else: > return > > When I look at the system_info.check_libs code, it looks like it is > trying to look for any extension, and the first found is returned... But > this is not what I get, and I am not sure to see what I am doing wrong. Well, since the first line of that method is this: exts = self.library_extensions() I conclude that if you were to override that method to return only ['.dll'], then your code will work. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fullung at gmail.com Fri Mar 16 01:53:02 2007 From: fullung at gmail.com (Albert Strasheim) Date: Fri, 16 Mar 2007 07:53:02 +0200 Subject: [Numpy-discussion] numpy.distutils, windows dll vs lib References: <45FA197A.3020808@ar.media.kyoto-u.ac.jp><45FA1FC4.5000008@gmail.com> <45FA2264.50904@ar.media.kyoto-u.ac.jp> Message-ID: <003901c7678f$5bea8970$0100a8c0@sun.ac.za> Hello all ----- Original Message ----- From: "David Cournapeau" To: "Discussion of Numerical Python" Sent: Friday, March 16, 2007 6:51 AM Subject: Re: [Numpy-discussion] numpy.distutils, windows dll vs lib > Robert Kern wrote: >> David Cournapeau wrote: >> Generally, you don't use .dll files to link against. With MSVC, you need >> a .lib >> file corresponding to your target .dll file which has the symbols. mingw >> and >> cygwin use similar .a files. This information is coded in the >> system_info.library_extensions() method, which you can override if you >> need >> something different. Of course, since you don't mention what methods on >> system_info that you are using, I can't quite be sure this will satisfy >> your needs. > Sorry for the lack of details; I did this a long time ago, and the > problem popped up for some people my package with windows (which I don't > at all) just recently. > > I don't use the library for linking: I only need to be able to load it > dynamically through ctypes. What I did is simply overriding the > calc_info method, in which I try to detect both library and header > files. For the library, I do the following: You might try numpy.ctypeslib.load_library. Cheers, Albert From stefan at sun.ac.za Fri Mar 16 02:48:38 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 16 Mar 2007 08:48:38 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <45F9C253.7080202@gemini.edu> References: <20070313085540.GJ15580@mentat.za.net> <45F9C253.7080202@gemini.edu> Message-ID: <20070316064838.GD6168@mentat.za.net> Hi James On Thu, Mar 15, 2007 at 06:01:55PM -0400, James Turner wrote: > >Code snippets to illustrate the problem would be welcome. > > OK. I have had a go at producing a code snippet. I apologize that > this is based on numarray rather than numpy, since I'm using STScI > Python, but I think it should be very easy to convert if you have > numpy instead. > > What I am doing is to transform overlapping input images onto a > common, larger grid and co-adding them. Although I'm using affine_ > transform on 3D data from FITS images, the issue can be illustrated > using a simple 1D translation of a single 2D test array. The input > values are just [4., 3., 2., 1.] in each row. With a translation of > -0.1, the values should therefore be something like > [X, 3.1, 2.1, 1.1, X, X], where the Xs represent points outside the > original data range. What I actually get, however, is roughly > [X, 3.1, 2.1, 1.0, 1.9, X]. The 5th value of 1.9 contaminates the > co-added data in the final output array. Now I'm looking at this > element-by-element, I suppose the bad value of 1.9 is just a result > of extrapolating in order to preserve the original number of data > points, isn't it? Sorry I wasn't clear on that in my original post > -- but surely a blank value (as specified by cval) would be better? I'd like to confirm that you see the same results when running your script: [[ 4. 3. 2. 1.] [ 4. 3. 2. 1.]] [[-1. 3.12520003 2.11439991 1.01719999 1.87479997 -1. ] [-1. 3.12520003 2.11439991 1.01719999 1.87479997 -1. ]] [[-1. 3.0996666 2.0999999 1.34300005 1.90033329 -1. ] [-1. 3.0996666 2.0999999 1.34300005 1.90033329 -1. ]] Cheers St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: jturner.py Type: text/x-python Size: 686 bytes Desc: not available URL: From efiring at hawaii.edu Fri Mar 16 03:12:55 2007 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 15 Mar 2007 21:12:55 -1000 Subject: [Numpy-discussion] rant against from numpy import * / from pylab import * In-Reply-To: References: Message-ID: <45FA4377.6010201@hawaii.edu> Sebastian Haase wrote: > Hi! > I use the wxPython PyShell. > I like especially the feature that when typing a module and then the > dot "." I get a popup list of all available functions (names) inside > that module. > > Secondly, I think it really makes code clearer when one can see where > a function comes from. > > I have a default > import numpy as N > executed before my shell even starts. > In fact I have a bunch of my "standard" modules imported as single capital letter>. > > This - I think - is a good compromise to the commonly used "extra > typing" and "unreadable" argument. > > a = sin(b) * arange(10,50, .1) * cos(d) > vs. > a = N.sin(b) * N.arange(10,50, .1) * N.cos(d) I generally do the latter, but really, all those "N." bits are still visual noise when it comes to reading the code--that is, seeing the algorithm rather than where the functions come from. I don't think there is anything wrong with explicitly importing commonly-used names, especially things like sin and cos. > > I would like to hear some comments by others. > > > On a different note: I just started using pylab, so I did added an > automatic "from matplotlib import pylab as P" -- but now P contains > everything that I already have in N. It makes it really hard to > *find* (as in *see* n the popup-list) the pylab-only functions. -- > what can I do about this ? A quick and dirty solution would be to comment out most of the imports in pylab.py; they are not needed for the pylab functions and are there only to give people lots of functionality in a single namespace. I am cross-posting this to matplotlib-users because it involves mpl, and an alternative solution would be for us to add an rcParam entry to allow one to turn off all of the namespace consolidation. A danger is that if someone is using "from pylab import *" in a script, then whether it would run would depend on the matplotlibrc file. To get around that, another possibility would be to break pylab.py into two parts, with pylab.py continuing to do the namespace consolidation and importing the second part, which would contain the actual pylab functions. Then if you don't want the namespace consolidation, you could simply import the second part instead of pylab. There may be devils in the details, but it seems to me that this last alternative--splitting pylab.py--might make a number of people happier while having no adverse effects on everyone else. Eric > > > Thanks, > Sebastian > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Fri Mar 16 03:13:07 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 16 Mar 2007 09:13:07 +0200 Subject: [Numpy-discussion] zoom FFT with numpy? In-Reply-To: <1173952429.8746.3.camel@nadav.envision.co.il> References: <6.2.3.4.2.20070314132537.05ccf838@blue-cove.com> <1173952429.8746.3.camel@nadav.envision.co.il> Message-ID: <20070316071307.GF6168@mentat.za.net> Here is another implementation of the Chirp-Z transform in Python, that handles complex numbers. Regards St?fan On Thu, Mar 15, 2007 at 11:53:49AM +0200, Nadav Horesh wrote: > A long time ago I translated a free code of chirp z transform (zoom fft) into > python. > Attached here the source and two translations (probably one of the is right) > > Nadav. > > On Wed, 2007-03-14 at 14:02 -0800, Ray S wrote: > > We'd like to do what most call a "zoom FFT"; we only are interested > in the frequencies of say, 6kHZ to 9kHz with a given N, and so the > computations from DC to 6kHz are wasted CPU time. > Can this be done without additional numpy pre-filtering computations? -------------- next part -------------- A non-text attachment was scrubbed... Name: chirpz.py Type: text/x-python Size: 1135 bytes Desc: not available URL: From david at ar.media.kyoto-u.ac.jp Fri Mar 16 06:43:19 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 16 Mar 2007 19:43:19 +0900 Subject: [Numpy-discussion] numpy.distutils, windows dll vs lib In-Reply-To: <003901c7678f$5bea8970$0100a8c0@sun.ac.za> References: <45FA197A.3020808@ar.media.kyoto-u.ac.jp><45FA1FC4.5000008@gmail.com> <45FA2264.50904@ar.media.kyoto-u.ac.jp> <003901c7678f$5bea8970$0100a8c0@sun.ac.za> Message-ID: <45FA74C7.1080702@ar.media.kyoto-u.ac.jp> Albert Strasheim wrote: > > > You might try numpy.ctypeslib.load_library. I tried this some time ago, and I couldn't make it work as I wanted (cannot remember what exactly, this was like 6 months ago), and I found more reliable to detect the library when installing the package, and hardcoding the path at configuration time. And I need the header file anyway, now, so I need to do use numpy.distutils anyway. cheers, David P.S: Alber, I saw on your page that you had some problems with my package pyem. Could it be possible to have more details, so that I can try to solve them ? From jturner at gemini.edu Fri Mar 16 11:49:56 2007 From: jturner at gemini.edu (James Turner) Date: Fri, 16 Mar 2007 11:49:56 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070316064838.GD6168@mentat.za.net> References: <20070316064838.GD6168@mentat.za.net> Message-ID: <45FABCA4.40608@gemini.edu> Hi Stefan, > I'd like to confirm that you see the same results when running your > script: > > [[ 4. 3. 2. 1.] > [ 4. 3. 2. 1.]] > [[-1. 3.12520003 2.11439991 1.01719999 1.87479997 -1. ] > [-1. 3.12520003 2.11439991 1.01719999 1.87479997 -1. ]] > [[-1. 3.0996666 2.0999999 1.34300005 1.90033329 -1. ] > [-1. 3.0996666 2.0999999 1.34300005 1.90033329 -1. ]] Yes, I get exactly the same numbers with numarray on a PC running Linux. I just rounded the numbers off in my explanatory text to simplify the discussion; sorry if that was confusing. Thanks! James. From stefan at sun.ac.za Fri Mar 16 18:15:28 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 17 Mar 2007 00:15:28 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <45FABCA4.40608@gemini.edu> References: <20070316064838.GD6168@mentat.za.net> <45FABCA4.40608@gemini.edu> Message-ID: <20070316221528.GO6168@mentat.za.net> On Fri, Mar 16, 2007 at 11:49:56AM -0400, James Turner wrote: > Hi Stefan, > > > I'd like to confirm that you see the same results when running your > > script: > > > > [[ 4. 3. 2. 1.] > > [ 4. 3. 2. 1.]] > > [[-1. 3.12520003 2.11439991 1.01719999 1.87479997 -1. ] > > [-1. 3.12520003 2.11439991 1.01719999 1.87479997 -1. ]] > > [[-1. 3.0996666 2.0999999 1.34300005 1.90033329 -1. ] > > [-1. 3.0996666 2.0999999 1.34300005 1.90033329 -1. ]] > > Yes, I get exactly the same numbers with numarray on a PC running Linux. > I just rounded the numbers off in my explanatory text to simplify the > discussion; sorry if that was confusing. Not at all, just wanted to make sure. I am starting to form an idea of what is happening here. Check out the following result: In [25]: import numpy as N In [26]: x = N.array([[4,3,8,1],[4,3,8,1.]]) In [27]: ndi.geometric_transform(x,shift,output_shape=(2,6),prefilter=False,order=0,cval=-1) Out[27]: array([[-1., 3., 8., 1., 8., -1.], [-1., 3., 8., 1., 8., -1.]]) Looks like the spline fit is done on mirrored input data, instead of padding the input with the cval. Regards St?fan From corzneffect at gmail.com Sat Mar 17 09:34:03 2007 From: corzneffect at gmail.com (Cory Davis) Date: Sat, 17 Mar 2007 13:34:03 +0000 Subject: [Numpy-discussion] import numpy segmentation fault In-Reply-To: <0F58E0ED-B2C4-4CF6-8EE3-4A25BEC73854@stanford.edu> References: <4948f97c0703141136j522fd32kbe202874ff19e3cf@mail.gmail.com> <0F58E0ED-B2C4-4CF6-8EE3-4A25BEC73854@stanford.edu> Message-ID: <4948f97c0703170634q6b11f803u7ed58d9b5ac1b1f5@mail.gmail.com> Thanks Zach, yes, the SVN version works, Cheers, Cory. On 3/14/07, Zachary Pincus wrote: > If I recall correctly, there's a bug in numpy 1.0.1 on Linux-x86-64 > that causes this segfault. This is fixed in the latest SVN version of > numpy, so if you can grab that, it should work. > > I can't find the trac ticket, but I ran into this some weeks ago. > > Zach > > > > On Mar 14, 2007, at 1:36 PM, Cory Davis wrote: > > > Hi there, > > > > I have just installed numpy-1.0.1 from source, which seemed to go > > fine. However when I try to "import numpy" I get a segmentation > > fault. > > > > A have a 64 bit machine running RedHat Enterprise Linux and Python > > 2.34 > > > > Any clues greatly appreciated. > > > > Cheers, > > Cory. > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Sat Mar 17 11:12:25 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 17 Mar 2007 17:12:25 +0200 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> Message-ID: <20070317151224.GA7104@mentat.za.net> On Thu, Mar 15, 2007 at 12:26:10PM +0100, David Koch wrote: > Next thing, I have > > A_Idx = array([1, 0]) > > and B is a matrix. How would I select the four elements in B indicated by > "meshgrid(A_Idx)", that ist: B[1,1], B[1,0], B[0,1], B[0,0] > In Matlab you would simply use B(A_idx, A_idx) whereas in NumPy B[A_Idx, A_idx] > returns B[1,1] and B[0,0] only. > > I read that A.take(A_idx).take(A_idx, axis = 1) should work but it > doesn't. In [12]: B[N.ix_(A_idx,A_idx)] Out[12]: array([[4, 3], [2, 1]]) Cheers St?fan From mkg at cs.nyu.edu Sat Mar 17 18:53:03 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Sat, 17 Mar 2007 18:53:03 -0400 Subject: [Numpy-discussion] scalar recordarrays Message-ID: <45FC714F.1040605@cs.nyu.edu> I've found that if I have a record array of shape [] (i.e. a scalar recarray), I can't set its fields. >>> recarr recarray((0.0, 0.0, 0.0), dtype=[('x', '>> recarr.x[...] = 1.0 TypeError: object does not support item assignment In the above, recarr.x returned a float "0.0", then attempted the assignment "0.0 = 1.0", which raised the TypeError. This is in contrast with the behavior of scalar ndarrays, where the above works as expected: >>> arr array(0.0) >>> arr[...] = 1.0 >>> arr array(1.0) I this a bug, or is there some reason for this behavior? The obvious workaround is reshape a scalar to a size-1 array before setting .x: >>> if recarr.ndim == 0: ....: recarr.reshape([1]).x[...] = 1.0 else: recarr.x[...] = 1.0 ....: else: ....: recarr.x[...] = 1.0 But this is cumbersome and easy to forget to do. -- Matt From granett at IfA.Hawaii.Edu Sat Mar 17 20:46:35 2007 From: granett at IfA.Hawaii.Edu (Ben Granett) Date: Sat, 17 Mar 2007 14:46:35 -1000 (HST) Subject: [Numpy-discussion] histogramdd shapes Message-ID: Hi, There seems to be a problem with histogramdd (numpy 1.0.1). Depending on the number of bins in each axis, it may not produce an array with the dimensions oriented correctly. The bug is in the axis swapping code at the end of the routine. To reproduce, you can run, >>> x = random.randn(100,3) >>> H, edg = histogramdd(x, bins=(7,5,6)) >>> print H.shape (5, 7, 6) A fix is to replace this: # Shape into a proper matrix hist = hist.reshape(sort(nbin)) for i,j in enumerate(ni): hist = hist.swapaxes(i,j) if (hist.shape == nbin).all(): break with this: for i in arange(nbin.size): j = ni[i] hist = hist.swapaxes(i,j) ni[i],ni[j] = ni[j],ni[i] That is, elements of ni need to be swapped too. I hope this wasn't a known bug, but I couldn't find any mention of it. Thanks, Ben From corzneffect at gmail.com Sun Mar 18 14:20:36 2007 From: corzneffect at gmail.com (Cory Davis) Date: Sun, 18 Mar 2007 18:20:36 +0000 Subject: [Numpy-discussion] Is the select function broken in numpy? Message-ID: <4948f97c0703181120n46f8b847q64ca77935e621c5d@mail.gmail.com> I have a few day old svn update of numpy where the following code produces an error, yet not in numpy-0.9.8 from numpy import * print select([array([1]),array([0])],[array([2]),array([3])]) File "/tmp/tmpY-7FJN/lib64/python/numpy/lib/function_base.py", line 426, in select File "/tmp/tmpY-7FJN/lib64/python/numpy/core/fromnumeric.py", line 151, in choose TypeError: array cannot be safely cast to required type [ Any clues? From corzneffect at gmail.com Sun Mar 18 14:37:34 2007 From: corzneffect at gmail.com (Cory Davis) Date: Sun, 18 Mar 2007 18:37:34 +0000 Subject: [Numpy-discussion] Is the select function broken in numpy? In-Reply-To: <4948f97c0703181120n46f8b847q64ca77935e621c5d@mail.gmail.com> References: <4948f97c0703181120n46f8b847q64ca77935e621c5d@mail.gmail.com> Message-ID: <4948f97c0703181137n58f18ba7p71dfc08f2aa3e001@mail.gmail.com> I have solved the problem by changing line 426 in function_base.py from return choose(S.astype(int), tuple(choicelist)) to return choose(S, tuple(choicelist)) On 3/18/07, Cory Davis wrote: > I have a few day old svn update of numpy where the following code > produces an error, yet not in numpy-0.9.8 > > from numpy import * > > print select([array([1]),array([0])],[array([2]),array([3])]) > > File "/tmp/tmpY-7FJN/lib64/python/numpy/lib/function_base.py", line > 426, in select > File "/tmp/tmpY-7FJN/lib64/python/numpy/core/fromnumeric.py", line > 151, in choose > TypeError: array cannot be safely cast to required type > [ > Any clues? > From corzneffect at gmail.com Sun Mar 18 14:38:27 2007 From: corzneffect at gmail.com (Cory Davis) Date: Sun, 18 Mar 2007 18:38:27 +0000 Subject: [Numpy-discussion] Is the select function broken in numpy? In-Reply-To: <4948f97c0703181137n58f18ba7p71dfc08f2aa3e001@mail.gmail.com> References: <4948f97c0703181120n46f8b847q64ca77935e621c5d@mail.gmail.com> <4948f97c0703181137n58f18ba7p71dfc08f2aa3e001@mail.gmail.com> Message-ID: <4948f97c0703181138mb056caey8bdb6b7009bc8af2@mail.gmail.com> no sorry, that is the wrong way around I changed it from return choose(S, tuple(choicelist)) to return choose(S.astype(int), tuple(choicelist)) On 3/18/07, Cory Davis wrote: > I have solved the problem by changing line 426 in function_base.py from > return choose(S.astype(int), tuple(choicelist)) > to > return choose(S, tuple(choicelist)) > > > On 3/18/07, Cory Davis wrote: > > I have a few day old svn update of numpy where the following code > > produces an error, yet not in numpy-0.9.8 > > > > from numpy import * > > > > print select([array([1]),array([0])],[array([2]),array([3])]) > > > > File "/tmp/tmpY-7FJN/lib64/python/numpy/lib/function_base.py", line > > 426, in select > > File "/tmp/tmpY-7FJN/lib64/python/numpy/core/fromnumeric.py", line > > 151, in choose > > TypeError: array cannot be safely cast to required type > > [ > > Any clues? > > > From wbaxter at gmail.com Sun Mar 18 21:42:26 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 19 Mar 2007 10:42:26 +0900 Subject: [Numpy-discussion] Little module to get numpy examples Message-ID: I wrote a little python module to go fetch the Numpy examples from the scipy wiki page, parse them, and print out entries. Is there a good place on the wiki for this? It didn't really seem right in the cookbook, and it doesn't quite fit with documentation. Here's an example of use: ----------------------------------------- In [1]: import numpy In [2]: from NumpyExamples import examples In [3]: examples(numpy.hstack) --- hstack --- >>> from numpy import * >>> a =array([[1],[2]]) # 2x1 array >>> b = array([[3,4],[5,6]]) # 2x2 array >>> hstack((a,b,a)) # only the 2nd dimension of the arrays is allowed to be different array([[1, 3, 4, 1], [2, 5, 6, 2]]) See also: column_stack, concatenate, dstack, vstack, hsplit ----------------------------------------- --bb -------------- next part -------------- A non-text attachment was scrubbed... Name: NumpyExamples.py Type: text/x-python Size: 8883 bytes Desc: not available URL: From faltet at carabos.com Mon Mar 19 05:00:14 2007 From: faltet at carabos.com (Francesc Altet) Date: Mon, 19 Mar 2007 10:00:14 +0100 Subject: [Numpy-discussion] scalar recordarrays In-Reply-To: <45FC714F.1040605@cs.nyu.edu> References: <45FC714F.1040605@cs.nyu.edu> Message-ID: <1174294814.2551.27.camel@localhost.localdomain> El ds 17 de 03 del 2007 a les 18:53 -0400, en/na Matthew Koichi Grimes va escriure: > I've found that if I have a record array of shape [] (i.e. a scalar > recarray), I can't set its fields. > > >>> recarr > recarray((0.0, 0.0, 0.0), > dtype=[('x', ' >>> recarr.x[...] = 1.0 > TypeError: object does not support item assignment > > In the above, recarr.x returned a float "0.0", then attempted the > assignment "0.0 = 1.0", which raised the TypeError. This is in contrast > with the behavior of scalar ndarrays, where the above works as expected: > > >>> arr > array(0.0) > >>> arr[...] = 1.0 > >>> arr > array(1.0) You are having a couple of confusions here. The first is in terms of nomenclature: in your examples, 'recarr' and 'arr' are *rank-0* ndarrays and not scalar objects. Although both rank-0 and scalar objects do contain the same information, they behave differently: a rank-0 array behaves more similar to an array and the scalar more like a python scalar. The other confusion is that, while 'recarr' is a rank-0 array, recarr.x is actually a true scalar, and you can assign to it in the regular scalar way: In [67]:recarr.x Out[67]:1.0 In [68]:recarr.x = 2 In [69]:recarr.x Out[69]:2.0 Having said this, I think there is a real inconsistency here that I think it is worth to note. Let's see the next recarray behaviour: In [70]:recarr2=numpy.rec.array([(1.0, 0, 3)], dtype) In [71]:recarr2.x Out[71]:array([ 1.]) In [72]:recarr2.x[...] = 0 In [73]:recarr2.x Out[73]:array([ 0.]) i.e. 'recarr2' is a rank-1 array, and 'recarr2.x' is a rank-1 array (and thus, assigning works as regular array assigns). In the same way, with a rank-0 'recarr', 'recarr.x' should return a rank-0 array (for consistency), but it doesn't: In [74]:recarr=numpy.rec.array((1.0, 0, 3), dtype) In [75]:recarr.x Out[75]:1.0 In [76]:type(recarr.x) Out[76]: While I find this inconsistent, I'm not sure whether it should be be fixed or not because the boundaries between rank-0 and scalar objects are always a bit fuzzy. Travis can probably bring more light here. Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From oliphant at ee.byu.edu Mon Mar 19 13:21:35 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 19 Mar 2007 10:21:35 -0700 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45F2E725.9050901@cc.usu.edu> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> Message-ID: <45FEC69F.2040805@ee.byu.edu> Mark P. Miller wrote: >Robert: Just a thought on this topic: > >Would it be possible for the Scipy folks to add a new module based >solely off your old mtrand code (pre-broadcast)? I have to say that the >mtrand code from numpy 0.9.8 has some excellent advantages over the core >python random number generators. > >This would allow Scipy to have a pure random number generator, whereas >numpy could keep its fancier (but slower) array-based random number >generators. > > I've just added a faster path through the random-number generators for scalar parameters to the SVN code tree. It would be great if those who use this could check to see if 1) it is correct 2) it is indeed faster for scalar parameters Thanks, -Travis From robert.kern at gmail.com Mon Mar 19 13:34:50 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 19 Mar 2007 12:34:50 -0500 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45FEC69F.2040805@ee.byu.edu> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> <45FEC69F.2040805@ee.byu.edu> Message-ID: <45FEC9BA.2030006@gmail.com> Travis Oliphant wrote: > I've just added a faster path through the random-number generators for > scalar parameters to the SVN code tree. > > It would be great if those who use this could check to see if > > 1) it is correct > 2) it is indeed faster for scalar parameters It's faster, certainly. I'll check for correctness later. Good work! In [13]: import timeit In [14]: tstd = timeit.Timer("random.normalvariate(0, 1)", "import random") In [15]: tscalar = timeit.Timer("random.normal(0, 1)", "from numpy import random") In [16]: tarray = timeit.Timer("random.normal(array(0), array(1))", "from numpy import array, random") In [17]: tstd.repeat() Out[17]: [2.700355052947998, 2.6895289421081543, 2.7158269882202148] In [18]: tscalar.repeat() Out[18]: [0.60753297805786133, 0.60142016410827637, 0.59989500045776367] In [19]: tarray.repeat() Out[19]: [2.0689051151275635, 2.0820560455322266, 2.0542140007019043] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jstrunk at enthought.com Mon Mar 19 13:54:51 2007 From: jstrunk at enthought.com (Jeff Strunk) Date: Mon, 19 Mar 2007 12:54:51 -0500 Subject: [Numpy-discussion] New Trac feature: TracReSTMacro Message-ID: <200703191254.52001.jstrunk@enthought.com> Good afternoon, By request, I have installed the TracReSTMacro on the numpy, scipy, and scikits tracs. This plugin allows you to display ReST formatted text directly from svn. For example, http://projects.scipy.org/neuroimaging/ni/wiki/ReadMe in its entirety is: [[ReST(/ni/trunk/README)]] Thank you, Jeff Strunk IT Administrator Enthought, Inc. From charlesr.harris at gmail.com Mon Mar 19 14:34:41 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 19 Mar 2007 12:34:41 -0600 Subject: [Numpy-discussion] in place random generation In-Reply-To: <45FEC69F.2040805@ee.byu.edu> References: <45F04C56.7060009@gmail.com> <45F18CCD.3070002@cc.usu.edu> <45F1A0EE.60306@gmail.com> <45F1B185.9070108@gmail.com> <45F2E725.9050901@cc.usu.edu> <45FEC69F.2040805@ee.byu.edu> Message-ID: On 3/19/07, Travis Oliphant wrote: > > Mark P. Miller wrote: > > >Robert: Just a thought on this topic: > > > >Would it be possible for the Scipy folks to add a new module based > >solely off your old mtrand code (pre-broadcast)? I have to say that the > >mtrand code from numpy 0.9.8 has some excellent advantages over the core > >python random number generators. > > > >This would allow Scipy to have a pure random number generator, whereas > >numpy could keep its fancier (but slower) array-based random number > >generators. > > > > > > I've just added a faster path through the random-number generators for > scalar parameters to the SVN code tree. > > It would be great if those who use this could check to see if > > 1) it is correct > 2) it is indeed faster for scalar parameters It looks much better from the speed standpoint. Using the dirty benchmark I used in an earlier post I get In [5]: t1.timeit(100) Out[5]: 0.088869810104370117 In [6]: t2.timeit(100) Out[6]: 0.12822699546813965 whereas before I got In [22]: t1.timeit(100) Out[22]: 0.086430072784423828 In [23]: t2.timeit(100) Out[23]: 4.276231050491333 i.e. about a 33x speedup. Good job! There is probably a bit more to be gained but I'm not sure it's worth the effort. Here is a quick test of normal: In [9]: a = numpy.random.normal(2,3,10000) In [10]: a.std() Out[10]: 2.96173131656 In [11]: a.mean() Out[11]: 2.0158598547 So that's within the expected error. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jturner at gemini.edu Mon Mar 19 15:05:52 2007 From: jturner at gemini.edu (James Turner) Date: Mon, 19 Mar 2007 15:05:52 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070316221528.GO6168@mentat.za.net> References: <20070316221528.GO6168@mentat.za.net> Message-ID: <45FEDF10.3010707@gemini.edu> Thanks, Stefan. > In [25]: import numpy as N > In [26]: x = N.array([[4,3,8,1],[4,3,8,1.]]) > In [27]: > ndi.geometric_transform(x,shift,output_shape=(2,6),prefilter=False,order=0,cval=-1) > Out[27]: > array([[-1., 3., 8., 1., 8., -1.], > [-1., 3., 8., 1., 8., -1.]]) Your example seems to be missing a definition for "shift", but I see that shift = lambda (y, x): (y, x-xoff) reproduces your result for any 0. < xoff <= 0.5. > Looks like the spline fit is done on mirrored input data, instead of > padding the input with the cval. Yes, that makes sense. I suppose the idea is to *avoid* creating edge effects due to an artificial sharp cut-off(?). That is fine for calculating the interpolant, but it seems like a mistake to include the mirrored values in the output result, doesn't it? Do people agree that points outside the bounds of the original data should be blank, such that the correct result in the above case would be [-1., 3., 8., 1., -1., -1.]? If so, I can go ahead and file a bug report (should it be a scipy-ticket, an email to STScI or both?). I would offer to have a go at changing the nd_image code, but I think I might be well out of my depth at this point... Cheers, James. From mkg at cs.nyu.edu Mon Mar 19 15:59:12 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Mon, 19 Mar 2007 15:59:12 -0400 Subject: [Numpy-discussion] scalar recordarrays In-Reply-To: References: Message-ID: <45FEEB90.2030904@cs.nyu.edu> Francesc Altet wrote: > with a > rank-0 'recarr', 'recarr.x' should return a rank-0 array (for > consistency), but it doesn't: > > In [74]:recarr=numpy.rec.array((1.0, 0, 3), dtype) > In [75]:recarr.x > Out[75]:1.0 > In [76]:type(recarr.x) > Out[76]: > > While I find this inconsistent, I'm not sure whether it should be be > fixed or not because the boundaries between rank-0 and scalar objects > are always a bit fuzzy. Travis can probably bring more light here. > Here's my case for "yes, it's a problem that should be fixed (please?)": In my particular project, this inconsistency becomes a pervasive problem when writing vectorized functions that are supposed to accept and return tensors of various ranks. To accomodate the inconsistency, one has to take what would have been a simple statement of the form: output.x[...] = blah and replace it with a bunch of boolean switches: if output.ndim == 0: output = output.reshape([1]) outputWas0dim = True else: outputWas0dim = False output.x[...] = blah if outputWas0dim: output = output.reshape([]) return output -- Matt From faltet at carabos.com Mon Mar 19 16:08:51 2007 From: faltet at carabos.com (Francesc Altet) Date: Mon, 19 Mar 2007 21:08:51 +0100 Subject: [Numpy-discussion] scalar recordarrays In-Reply-To: <45FEEB90.2030904@cs.nyu.edu> References: <45FEEB90.2030904@cs.nyu.edu> Message-ID: <1174334931.2551.39.camel@localhost.localdomain> El dl 19 de 03 del 2007 a les 15:59 -0400, en/na Matthew Koichi Grimes va escriure: > Francesc Altet wrote: > > with a > > rank-0 'recarr', 'recarr.x' should return a rank-0 array (for > > consistency), but it doesn't: > > > > In [74]:recarr=numpy.rec.array((1.0, 0, 3), dtype) > > In [75]:recarr.x > > Out[75]:1.0 > > In [76]:type(recarr.x) > > Out[76]: > > > > While I find this inconsistent, I'm not sure whether it should be be > > fixed or not because the boundaries between rank-0 and scalar objects > > are always a bit fuzzy. Travis can probably bring more light here. > > > Here's my case for "yes, it's a problem that should be fixed (please?)": > In my particular project, this inconsistency becomes a pervasive problem > when writing vectorized functions that are supposed to accept and return > tensors of various ranks. To accomodate the inconsistency, one has to > take what would have been a simple statement of the form: > > > output.x[...] = blah > > > and replace it with a bunch of boolean switches: > > > if output.ndim == 0: > output = output.reshape([1]) > outputWas0dim = True > else: > outputWas0dim = False > > output.x[...] = blah > > if outputWas0dim: > output = output.reshape([]) > > return output > Yeah, I agree. In fact, one of the main reasons to keep rank-0 arrays around is to provide generality enough to address to these kind of problems in an elegant way. Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From ckkart at hoc.net Mon Mar 19 21:42:59 2007 From: ckkart at hoc.net (Christian K.) Date: Tue, 20 Mar 2007 01:42:59 +0000 (UTC) Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib References: Message-ID: Hi Bill, I just tried ezplot and encountered some problems: In [1]: import ezplot In [2]: p = ezplot.Plotter() In [3]: p.plot([1,2,3],[1,4,9],marker='o') At this point a window pops up for a second, closes again and plot does not return. I'm running python 2.4.4 on kubuntu linux with wxPython 2.6.3.3, twisted 2.4. Any ideas how to solve that? Christian From wbaxter at gmail.com Mon Mar 19 23:58:13 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 20 Mar 2007 12:58:13 +0900 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: On 3/20/07, Christian K. wrote: > Hi Bill, > > I just tried ezplot and encountered some problems: > > In [1]: import ezplot > In [2]: p = ezplot.Plotter() > In [3]: p.plot([1,2,3],[1,4,9],marker='o') > > At this point a window pops up for a second, closes again and plot does not > return. > > I'm running python 2.4.4 on kubuntu linux with wxPython 2.6.3.3, twisted 2.4. > > Any ideas how to solve that? Can you see if you can run the plotserver directly? python plotserver.py If one is not running then that p.plot command tries to start one up using the process module. But apparently that doesn't work right on MacOS, so maybe it's got problems on linux too. If you start the plotserver up by itself first there should be a little (non-plot) window that pops up for it. Once it's up you can try the lines you typed in above again (try that from a clean python session just to be sure). --bb From ckkart at hoc.net Tue Mar 20 02:03:16 2007 From: ckkart at hoc.net (Christian) Date: Tue, 20 Mar 2007 15:03:16 +0900 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: Bill Baxter wrote: > On 3/20/07, Christian K. wrote: >> Hi Bill, >> >> I just tried ezplot and encountered some problems: >> >> In [1]: import ezplot >> In [2]: p = ezplot.Plotter() >> In [3]: p.plot([1,2,3],[1,4,9],marker='o') >> >> At this point a window pops up for a second, closes again and plot does not >> return. >> >> I'm running python 2.4.4 on kubuntu linux with wxPython 2.6.3.3, twisted 2.4. >> >> Any ideas how to solve that? > > Can you see if you can run the plotserver directly? > python plotserver.py I found out that my version of matplotlib (0.87.7) does not know all methods that ezplot is expecting to be known. Thus init_pylab_methods raises an Exception which was not visible because loggiong to the wx.TextCtrl does not work - nothing is ever printed there. I removed the unknown methods and now it works. Very nice work! Once more I am really impressed by python (and good python programmers). Christian From wbaxter at gmail.com Tue Mar 20 03:29:40 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 20 Mar 2007 16:29:40 +0900 Subject: [Numpy-discussion] Nonblocking Plots with Matplotlib In-Reply-To: References: Message-ID: On 3/20/07, Christian wrote: > Bill Baxter wrote: > I found out that my version of matplotlib (0.87.7) does not know all methods > that ezplot is expecting to be known. Aha. Thanks for diagnosing the problem. Should be fixed in the latest ezplot I just uploaded to pypi. ("easy_install --upgrade ezplot" if you have easy_install). > Thus init_pylab_methods raises an > Exception which was not visible because loggiong to the wx.TextCtrl does not > work - nothing is ever printed there. > I removed the unknown methods and now it works. Hmm. Yeh my error reporting is quite weak. > Very nice work! Thanks. > Once more I am really impressed by python (and good python programmers). Yeh, me too. I've never used setuptools before, or distributed any python code but it's really easy to whip up a dist (at least when the code is 100% python). Twisted and matplotlib are great too. Wouldn't have been able to do this in a month without those two, let alone a day. After using it for a few days I've been really happy with the results myself. Better error messages would be nice, but I'm finding it very nice to be able to pop up plots whenever and wherever I want while debugging. And they stay up even after the debugee goes away, which is also pretty nice. --bb From ogdude at googlemail.com Tue Mar 20 06:48:09 2007 From: ogdude at googlemail.com (David Koch) Date: Tue, 20 Mar 2007 11:48:09 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <20070317151224.GA7104@mentat.za.net> References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> <20070317151224.GA7104@mentat.za.net> Message-ID: Hi, naive question - how do I get an overview over everything to do with "sparse functionality" in SciPy 0.5.2 ... I can't find any documentation anywhere. /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From svetosch at gmx.net Tue Mar 20 07:06:48 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 20 Mar 2007 12:06:48 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> <20070317151224.GA7104@mentat.za.net> Message-ID: <45FFC048.9060807@gmx.net> David Koch schrieb: > Hi, > > naive question - how do I get an overview over everything to do with > "sparse functionality" in SciPy 0.5.2 ... I can't find any documentation > anywhere. > First of all I would recommend to start a new and properly named thread for that.... good luck, sven From ogdude at googlemail.com Tue Mar 20 07:15:23 2007 From: ogdude at googlemail.com (David Koch) Date: Tue, 20 Mar 2007 12:15:23 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> <20070317151224.GA7104@mentat.za.net> Message-ID: I will consider it Sven, I thought it was a good idea to collect everything which had to do with Matlab -> Python in one thread. Anyway, Specifically, I was looking for an equivalent to Matlab's "sprand" which allows one to create sparse normally distributed matrices. The function also accepts a "density" parameter. And by the way - whenever I do a .__doc__ all newline characters are printed as "\n" in the python console ... how do I change that? /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From lxander.m at gmail.com Tue Mar 20 08:27:17 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Tue, 20 Mar 2007 08:27:17 -0400 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: References: <45ECE21B.2080806@ar.media.kyoto-u.ac.jp> <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> <20070317151224.GA7104@mentat.za.net> Message-ID: <525f23e80703200527u2da02515o17d9b2202f6e5755@mail.gmail.com> On 3/20/07, David Koch wrote: > And by the way - whenever I do a .__doc__ all newline characters > are printed as "\n" in the python console ... how do I change that? The easiest way to access the doc strings is to type help() in the python interpreter, or ? in ipython (). The reason you are seeing the escaped newlines is that by directly accessing the __doc__ attribute, you are getting the string which the interpreter is kind enough to display in repr form, but you are not explicitly printing it (in str form). Compare: >>> print repr('hello\nworld') 'hello\nworld' >>> print str('hello\nworld') hello world So that if you said print .__doc__, you would not see the escaped newlines. From ogdude at googlemail.com Tue Mar 20 12:06:44 2007 From: ogdude at googlemail.com (David Koch) Date: Tue, 20 Mar 2007 17:06:44 +0100 Subject: [Numpy-discussion] Adapting algorithm to accept Scipy sparse matrix Message-ID: Hi, I implemented an algorithm in NumPy which assumes that the input is of type ndarray, so elementwise multiplication is done as dot(x,y), equation solving using linalg.solve etc. I now want to modify the whole thing to accept scipy.sparse matrices (which for instance has linsolve.spsolve instead of linalg.solve) and I am wondering what's the best way to go about it. I don't want to check sparse.issparse all the time or even create a separate sub-class for each data type ... what to do, Ahh! Help! /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Tue Mar 20 12:54:28 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue, 20 Mar 2007 12:54:28 -0400 Subject: [Numpy-discussion] New Trac feature: TracReSTMacro In-Reply-To: <200703191254.52001.jstrunk@enthought.com> References: <200703191254.52001.jstrunk@enthought.com> Message-ID: <20070320165427.GA29522@arbutus.physics.mcmaster.ca> On Mon, Mar 19, 2007 at 12:54:51PM -0500, Jeff Strunk wrote: > Good afternoon, > > By request, I have installed the TracReSTMacro on the numpy, scipy, and > scikits tracs. This plugin allows you to display ReST formatted text directly > from svn. > > For example, http://projects.scipy.org/neuroimaging/ni/wiki/ReadMe in its > entirety is: > [[ReST(/ni/trunk/README)]] Hmm, I'm getting an Internal Server Error on http://projects.scipy.org/scipy/numpy/wiki/NumPyCAPI which has the content [[ReST(/numpy/trunk/doc/CAPI.txt)]] -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From shane.holloway at ieee.org Tue Mar 20 12:58:08 2007 From: shane.holloway at ieee.org (Shane Holloway) Date: Tue, 20 Mar 2007 10:58:08 -0600 Subject: [Numpy-discussion] Array of Callables Message-ID: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> To the vector-processing masters of numpy! I'm wanting to optimize calling a list (or array) of callable objects. Consider the following: vCallables = numpy.array([]) vParam1 = numpy.array([]) vParam2 = numpy.array([]) vParam3 = numpy.array([]) vResults = numpy.array([None for e in vCallables]) Is it possible to have numpy call them in a way something like:: for i in xrange(vCallables.shape[0]): vResults[i] = vCallables[i] ( vParam1[i], vParam2[i], vParam3[i] ) With the core of that loop implemented in C? Thanks for reading and pondering! -Shane Holloway From robert.kern at gmail.com Tue Mar 20 13:10:43 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Mar 2007 12:10:43 -0500 Subject: [Numpy-discussion] Array of Callables In-Reply-To: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> Message-ID: <46001593.2040206@gmail.com> Shane Holloway wrote: > To the vector-processing masters of numpy! > > I'm wanting to optimize calling a list (or array) of callable > objects. Consider the following: > > vCallables = numpy.array([ classes, builtin functions>]) > vParam1 = numpy.array([]) > vParam2 = numpy.array([]) > vParam3 = numpy.array([]) > vResults = numpy.array([None for e in vCallables]) > > Is it possible to have numpy call them in a way something like:: > > for i in xrange(vCallables.shape[0]): > vResults[i] = vCallables[i] ( vParam1[i], vParam2[i], vParam3[i] ) > > > With the core of that loop implemented in C? There's nothing sitting around, no. It's easy enough to write with Pyrex (and easier without bothering with numpy); however, unless if all of your functions are builtin, you won't see much, if any, speed gains. Python's function call overhead will dominate the savings of a C for loop over a Python for loop. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jstrunk at enthought.com Tue Mar 20 13:57:45 2007 From: jstrunk at enthought.com (Jeff Strunk) Date: Tue, 20 Mar 2007 12:57:45 -0500 Subject: [Numpy-discussion] New Trac feature: TracReSTMacro In-Reply-To: <20070320165427.GA29522@arbutus.physics.mcmaster.ca> References: <200703191254.52001.jstrunk@enthought.com> <20070320165427.GA29522@arbutus.physics.mcmaster.ca> Message-ID: <200703201257.45215.jstrunk@enthought.com> On Tuesday 20 March 2007 11:54 am, David M. Cooke wrote: > On Mon, Mar 19, 2007 at 12:54:51PM -0500, Jeff Strunk wrote: > > Good afternoon, > > > > By request, I have installed the TracReSTMacro on the numpy, scipy, and > > scikits tracs. This plugin allows you to display ReST formatted text > > directly from svn. > > > > For example, http://projects.scipy.org/neuroimaging/ni/wiki/ReadMe in its > > entirety is: > > [[ReST(/ni/trunk/README)]] > > Hmm, I'm getting an Internal Server Error on > http://projects.scipy.org/scipy/numpy/wiki/NumPyCAPI > which has the content > [[ReST(/numpy/trunk/doc/CAPI.txt)]] The content was: [[ReST(/numpy/trunk/numpy/doc/CAPI.txt)]] It should have been: [[ReST(/trunk/numpy/doc/CAPI.txt)]] I fixed it in the database, and the page works. Thanks, Jeff From cookedm at physics.mcmaster.ca Tue Mar 20 14:02:57 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue, 20 Mar 2007 14:02:57 -0400 Subject: [Numpy-discussion] New Trac feature: TracReSTMacro In-Reply-To: <200703201257.45215.jstrunk@enthought.com> References: <200703191254.52001.jstrunk@enthought.com> <20070320165427.GA29522@arbutus.physics.mcmaster.ca> <200703201257.45215.jstrunk@enthought.com> Message-ID: <20070320180257.GA29925@arbutus.physics.mcmaster.ca> On Tue, Mar 20, 2007 at 12:57:45PM -0500, Jeff Strunk wrote: > On Tuesday 20 March 2007 11:54 am, David M. Cooke wrote: > > On Mon, Mar 19, 2007 at 12:54:51PM -0500, Jeff Strunk wrote: > > > Good afternoon, > > > > > > By request, I have installed the TracReSTMacro on the numpy, scipy, and > > > scikits tracs. This plugin allows you to display ReST formatted text > > > directly from svn. > > > > > > For example, http://projects.scipy.org/neuroimaging/ni/wiki/ReadMe in its > > > entirety is: > > > [[ReST(/ni/trunk/README)]] > > > > Hmm, I'm getting an Internal Server Error on > > http://projects.scipy.org/scipy/numpy/wiki/NumPyCAPI > > which has the content > > [[ReST(/numpy/trunk/doc/CAPI.txt)]] > > The content was: > [[ReST(/numpy/trunk/numpy/doc/CAPI.txt)]] > > It should have been: > [[ReST(/trunk/numpy/doc/CAPI.txt)]] > > I fixed it in the database, and the page works. > > Thanks, > Jeff Ahh, ok. I was trying to puzzle it out from your neuroimaging example. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From charlesr.harris at gmail.com Tue Mar 20 15:29:01 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Mar 2007 13:29:01 -0600 Subject: [Numpy-discussion] PyChecker Message-ID: All, I've run pychecker on some of my programs that import numpy and it issues many warnings about numpy. Many (most?) of the warnings look bogus, but some also look valid. I will try to work through some of these, but I want to suggest that we run pychecker, and maybe pylint, over the python code to look for errors. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Tue Mar 20 15:36:01 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 20 Mar 2007 15:36:01 -0400 Subject: [Numpy-discussion] PyChecker In-Reply-To: References: Message-ID: <200703201536.01902.pgmdevlist@gmail.com> On Tuesday 20 March 2007 15:29:01 Charles R Harris wrote: > but I want > to suggest that we run pychecker, and maybe pylint, over the python code to > look for errors. I'm using Pydev w/ Eclipse that supports pylint. I usually don't have any warning from pylint about numpy. Could you be more specific ? From pgmdevlist at gmail.com Tue Mar 20 17:06:08 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 20 Mar 2007 17:06:08 -0400 Subject: [Numpy-discussion] distutils for a Pyrex module Message-ID: <200703201706.08227.pgmdevlist@gmail.com> All, I'm trying to write a numpy.distutils setup.py for a pyrex module that involves both external C and fortran sources, and where the fortran sources need to be linked w/ blas and lapack. Here's what I have so far: ################################## def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration, get_mathlibs config = Configuration('cloess',parent_package,top_path) from numpy.distutils.system_info import get_info, dict_append libs = [] generate_libraries = [] f_sources = ['loessf.f', 'linpack_lite.f'] c_sources = ['loess.c', 'loessc.c', 'misc.c', 'predict.c',] # blas_info = get_info('blas_opt') lapack_info = get_info('lapack_opt') build_info = {} dict_append(build_info, **blas_info) dict_append(build_info, **lapack_info) # config.add_extension('cloess', sources=[join('src', x) for x in (f_sources+c_sources)], depends = [join('src','*.h'), join('src','*.pyx'), join('src','*.pxd') ], **build_info ) config.add_data_dir(join('.','examples')) config.add_data_files('pyloess.py') return config ############################ But that doesn't do what I want: during the setup, the .f files are processed by f2py. I don't need that, I just want to have the Fortran files compiled along the C sources. What's the M.O., then ? FYI, here are the commands I'm using to build the library w/o distutils: pyrexc cloess.pyx || exit gcc -g -c -fPIC loess.c loessc.c misc.c predict.c loessf.f linpack_lite.f gcc -g -c -fPIC cloess.c -I/usr/include/python2.4 \ -I/usr/lib64/python2.4/site-packages/numpy/core/include gcc -g -shared loessf.o linpack_lite.o loess.o loessc.o misc.o predict.o cloess.o -llapack -lblas -o cloess.so Any help will be welcome. Thanks a lot in advance (PS: initially posted on the pyrex mailing list and moved here on Robert Kern's request). From robert.kern at gmail.com Tue Mar 20 17:46:28 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Mar 2007 16:46:28 -0500 Subject: [Numpy-discussion] distutils for a Pyrex module In-Reply-To: <200703201706.08227.pgmdevlist@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> Message-ID: <46005634.6000504@gmail.com> Pierre GM wrote: > All, > I'm trying to write a numpy.distutils setup.py for a pyrex module that > involves both external C and fortran sources, and where the fortran sources > need to be linked w/ blas and lapack. Here's what I have so far: > > ################################## > def configuration(parent_package='',top_path=None): > from numpy.distutils.misc_util import Configuration, get_mathlibs > config = Configuration('cloess',parent_package,top_path) > from numpy.distutils.system_info import get_info, dict_append > > libs = [] > generate_libraries = [] > f_sources = ['loessf.f', 'linpack_lite.f'] > c_sources = ['loess.c', 'loessc.c', 'misc.c', 'predict.c',] > # > blas_info = get_info('blas_opt') > lapack_info = get_info('lapack_opt') > build_info = {} > dict_append(build_info, **blas_info) > dict_append(build_info, **lapack_info) > # > config.add_extension('cloess', > sources=[join('src', x) for x in > (f_sources+c_sources)], > depends = [join('src','*.h'), > join('src','*.pyx'), > join('src','*.pxd') > ], > **build_info > ) > config.add_data_dir(join('.','examples')) > config.add_data_files('pyloess.py') This line *should* be unnecessary. .py files should be picked up automatically. > return config > ############################ > > But that doesn't do what I want: during the setup, the .f files are processed > by f2py. I don't need that, I just want to have the Fortran files compiled > along the C sources. What's the M.O., then ? The first file in the sources list should be the one that actually implements the module, i.e. the C file generated by Pyrex. FORTRAN files specified after the first one won't be processed by f2py. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Tue Mar 20 18:06:03 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 20 Mar 2007 18:06:03 -0400 Subject: [Numpy-discussion] distutils for a Pyrex module In-Reply-To: <46005634.6000504@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> <46005634.6000504@gmail.com> Message-ID: <200703201806.03340.pgmdevlist@gmail.com> On Tuesday 20 March 2007 17:46:28 Robert Kern wrote: > The first file in the sources list should be the one that actually > implements the module, i.e. the C file generated by Pyrex. FORTRAN files > specified after the first one won't be processed by f2py. Mmh. I had to get rid of the '*.pyx' and '*.pxd' lines as numpy.distutils complained of a multiple definition of initcloess. However, that still doesn't work: I switched the order of f_sources and c_sources to >>>sources=[join('src', x) for x in (c_sources+f_sources)], where c_sources now have 'cloess.c' as first element (that's the output of pyrex), and f2py still processes the fortran sources. At the end, the module contains only the f2py-processed fortran subroutines, which are useless on their own. I'm obviously missing something. Any other idea ? Thanks a lot in advance From robert.kern at gmail.com Tue Mar 20 18:13:09 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Mar 2007 17:13:09 -0500 Subject: [Numpy-discussion] distutils for a Pyrex module In-Reply-To: <200703201806.03340.pgmdevlist@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> <46005634.6000504@gmail.com> <200703201806.03340.pgmdevlist@gmail.com> Message-ID: <46005C75.70903@gmail.com> Pierre GM wrote: > On Tuesday 20 March 2007 17:46:28 Robert Kern wrote: > >> The first file in the sources list should be the one that actually >> implements the module, i.e. the C file generated by Pyrex. FORTRAN files >> specified after the first one won't be processed by f2py. > > Mmh. I had to get rid of the '*.pyx' and '*.pxd' lines as numpy.distutils > complained of a multiple definition of initcloess. > However, that still doesn't work: I switched the order of f_sources and > c_sources to >>>> sources=[join('src', x) for x in (c_sources+f_sources)], > where c_sources now have 'cloess.c' as first element (that's the output of > pyrex), and f2py still processes the fortran sources. At the end, the module > contains only the f2py-processed fortran subroutines, which are useless on > their own. Hmm. Okay, put the FORTRAN files into a library instead. config.add_library('floess', sources=f_sources, ) dict_append(build_info, libraries=['floess']) config.add_extension('cloess', sources=c_sources, depends=..., **build_info, ) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pearu at cens.ioc.ee Tue Mar 20 18:13:26 2007 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Wed, 21 Mar 2007 00:13:26 +0200 (EET) Subject: [Numpy-discussion] distutils for a Pyrex module In-Reply-To: <200703201806.03340.pgmdevlist@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> <46005634.6000504@gmail.com> <200703201806.03340.pgmdevlist@gmail.com> Message-ID: <56570.82.131.69.129.1174428806.squirrel@cens.ioc.ee> > On Tuesday 20 March 2007 17:46:28 Robert Kern wrote: > >> The first file in the sources list should be the one that actually >> implements the module, i.e. the C file generated by Pyrex. FORTRAN files >> specified after the first one won't be processed by f2py. > > Mmh. I had to get rid of the '*.pyx' and '*.pxd' lines as numpy.distutils > complained of a multiple definition of initcloess. > However, that still doesn't work: I switched the order of f_sources and > c_sources to >>>>sources=[join('src', x) for x in (c_sources+f_sources)], > where c_sources now have 'cloess.c' as first element (that's the output of > pyrex), and f2py still processes the fortran sources. At the end, the > module > contains only the f2py-processed fortran subroutines, which are useless on > their own. > I'm obviously missing something. Any other idea ? In case you only need to compile fortran sources for linking with an extension module then specify fortran sources under libraries. For example, config.add_library('somename', sources = f_sources) config.add_extension('extname', sources = c_courses, libraries = ['somename']) HTH, Pearu From pgmdevlist at gmail.com Tue Mar 20 18:45:35 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 20 Mar 2007 18:45:35 -0400 Subject: [Numpy-discussion] distutils for a Pyrex module In-Reply-To: <46005C75.70903@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> <200703201806.03340.pgmdevlist@gmail.com> <46005C75.70903@gmail.com> Message-ID: <200703201845.36736.pgmdevlist@gmail.com> On Tuesday 20 March 2007 18:13:09 Robert Kern wrote: > Pierre GM wrote: > Hmm. Okay, put the FORTRAN files into a library instead. Robert, Pearu, great ! Creating a library did the trick. I eventually come with that: ######################## def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration, get_mathlibs config = Configuration('cloess',parent_package,top_path) from numpy.distutils.system_info import get_info, dict_append # f_sources = ('loessf.f', 'linpack_lite.f') config.add_library('floess', sources = [join('src',x) for x in f_sources]) # blas_info = get_info('blas_opt') lapack_info = get_info('lapack_opt') # c_sources = ['cloess.c', 'loess.c', 'loessc.c', 'misc.c', 'predict.c',] # build_info = {} dict_append(build_info, **blas_info) dict_append(build_info, **lapack_info) dict_append(build_info, libraries=['floess']) # config.add_extension('_loess', sources=[join('src', x) for x in c_sources], **build_info ) config.add_data_dir(join('.','examples')) config.add_data_files('pyloess.py') return config ######################### However, I still don't get a usable module: I have: ImportError: dynamic module does not define init function (init_loess). It looks like src/loess.c gets compiled before /src/cloess.c... From robert.kern at gmail.com Tue Mar 20 18:57:18 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Mar 2007 17:57:18 -0500 Subject: [Numpy-discussion] distutils for a Pyrex module In-Reply-To: <200703201845.36736.pgmdevlist@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> <200703201806.03340.pgmdevlist@gmail.com> <46005C75.70903@gmail.com> <200703201845.36736.pgmdevlist@gmail.com> Message-ID: <460066CE.7060402@gmail.com> Pierre GM wrote: > On Tuesday 20 March 2007 18:13:09 Robert Kern wrote: >> Pierre GM wrote: >> Hmm. Okay, put the FORTRAN files into a library instead. > > Robert, Pearu, great ! Creating a library did the trick. I eventually come > with that: > ######################## > def configuration(parent_package='',top_path=None): > from numpy.distutils.misc_util import Configuration, get_mathlibs > config = Configuration('cloess',parent_package,top_path) > from numpy.distutils.system_info import get_info, dict_append > # > f_sources = ('loessf.f', 'linpack_lite.f') > config.add_library('floess', > sources = [join('src',x) for x in f_sources]) > # > blas_info = get_info('blas_opt') > lapack_info = get_info('lapack_opt') > # > c_sources = ['cloess.c', 'loess.c', 'loessc.c', 'misc.c', 'predict.c',] > # > build_info = {} > dict_append(build_info, **blas_info) > dict_append(build_info, **lapack_info) > dict_append(build_info, libraries=['floess']) > # > config.add_extension('_loess', > sources=[join('src', x) for x in c_sources], > **build_info > ) > config.add_data_dir(join('.','examples')) > config.add_data_files('pyloess.py') > return config > ######################### > However, I still don't get a usable module: I have: > ImportError: dynamic module does not define init function (init_loess). It > looks like src/loess.c gets compiled before /src/cloess.c... Is there an init_loess function in cloess.c? If cloess.c was created by Pyrex from cloess.pyx, then Pyrex will make an initcloess function. The module name needs to be consistent throughout. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Tue Mar 20 22:48:46 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Mar 2007 20:48:46 -0600 Subject: [Numpy-discussion] PyChecker In-Reply-To: <200703201536.01902.pgmdevlist@gmail.com> References: <200703201536.01902.pgmdevlist@gmail.com> Message-ID: On 3/20/07, Pierre GM wrote: > > On Tuesday 20 March 2007 15:29:01 Charles R Harris wrote: > > but I want > > to suggest that we run pychecker, and maybe pylint, over the python code > to > > look for errors. > > I'm using Pydev w/ Eclipse that supports pylint. I usually don't have any > warning from pylint about numpy. Could you be more specific ? > ___ For instance, make a file warn.py containing the line: from numpy import * Then $[charris at fedora scratch]$ pychecker warn among the warning returned are ... /usr/lib/python2.4/site-packages/numpy/lib/type_check.py:30: Local variable (typecodes) shadows global defined on line 0 in file ... /usr/lib/python2.4/site-packages/numpy/lib/utils.py:287: (object) shadows builtin 34 errors suppressed, use -#/--limit to increase the number of errors displayed Neither is an error, but it is bad style to shadow builtins and unexpected errors can result. The second warning refers to this bit of code: def info(object=None,maxwidth=76,output=sys.stdout,toplevel='numpy'): """Get help information for a function, class, or module. Example: >>> from numpy import * >>> info(polyval) # doctest: +SKIP polyval(p, x) Evaluate the polymnomial p at x. Description: If p is of length N, this function returns the value: p[0]*(x**N-1) + p[1]*(x**N-2) + ... + p[N-2]*x + p[N-1] """ global _namedict, _dictlist import pydoc if hasattr(object,'_ppimport_importer') or \ hasattr(object, '_ppimport_module'): object = object._ppimport_module elif hasattr(object, '_ppimport_attr'): object = object._ppimport_attr Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Wed Mar 21 01:59:55 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Wed, 21 Mar 2007 14:59:55 +0900 Subject: [Numpy-discussion] Little module to get numpy examples In-Reply-To: References: Message-ID: On 3/19/07, Bill Baxter wrote: > I wrote a little python module to go fetch the Numpy examples from the > scipy wiki page, parse them, and print out entries. > > Is there a good place on the wiki for this? > It didn't really seem right in the cookbook, and it doesn't quite fit > with documentation. Since no-one responded to this, I went ahead and added various links. If you don't like em nuke em. --bb From alexandre.fayolle at logilab.fr Wed Mar 21 04:07:05 2007 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Wed, 21 Mar 2007 09:07:05 +0100 Subject: [Numpy-discussion] PyChecker In-Reply-To: References: <200703201536.01902.pgmdevlist@gmail.com> Message-ID: <20070321080705.GB5134@crater.logilab.fr> On Tue, Mar 20, 2007 at 08:48:46PM -0600, Charles R Harris wrote: > On 3/20/07, Pierre GM wrote: > > > >On Tuesday 20 March 2007 15:29:01 Charles R Harris wrote: > >> but I want > >> to suggest that we run pychecker, and maybe pylint, over the python code > >to > >> look for errors. > > > >I'm using Pydev w/ Eclipse that supports pylint. I usually don't have any > >warning from pylint about numpy. Could you be more specific ? > >___ > > > For instance, make a file warn.py containing the line: > > from numpy import * > > > Neither is an error, but it is bad style to shadow builtins and unexpected > errors can result. The second warning refers to this bit of code: If you go that way, it is bad style to import * in production code, precisely because it can lead to identifier shadowing. You should try import numpy as n Remember the zen of Python: Namespaces are one honking great idea -- let's do more of those! -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: Digital signature URL: From acorriga at gmu.edu Wed Mar 21 01:02:26 2007 From: acorriga at gmu.edu (Andrew Corrigan) Date: Wed, 21 Mar 2007 05:02:26 +0000 (UTC) Subject: [Numpy-discussion] Array of Callables References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> <46001593.2040206@gmail.com> Message-ID: Robert Kern gmail.com> writes: > > Shane Holloway wrote: > > To the vector-processing masters of numpy! > > > > I'm wanting to optimize calling a list (or array) of callable > > objects. Consider the following: > > > > vCallables = numpy.array([ > classes, builtin functions>]) > > vParam1 = numpy.array([]) > > vParam2 = numpy.array([]) > > vParam3 = numpy.array([]) > > vResults = numpy.array([None for e in vCallables]) > > > > Is it possible to have numpy call them in a way something like:: > > > > for i in xrange(vCallables.shape[0]): > > vResults[i] = vCallables[i] ( vParam1[i], vParam2[i], vParam3[i] ) > > > > > > With the core of that loop implemented in C? > > There's nothing sitting around, no. It's easy enough to write with Pyrex (and > easier without bothering with numpy); however, unless if all of your functions > are builtin, you won't see much, if any, speed gains. Python's function call > overhead will dominate the savings of a C for loop over a Python for loop. > This is a feature I've been wanting for a long time, so I'm really glad that Shane brought this up. While I was hoping for a gain in speed, that isn't the only reason that I would like to see this added. In fact, the most compelling reason for me is that it's a very natural way to express certain operations. I really hope that support for this feature can be added to NumPy. - Andrew From peridot.faceted at gmail.com Wed Mar 21 08:58:10 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 21 Mar 2007 08:58:10 -0400 Subject: [Numpy-discussion] Array of Callables In-Reply-To: References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> <46001593.2040206@gmail.com> Message-ID: On 21/03/07, Andrew Corrigan wrote: > This is a feature I've been wanting for a long time, so I'm really glad that > Shane brought this up. > > While I was hoping for a gain in speed, that isn't the only reason that I would > like to see this added. In fact, the most compelling reason for me is that it's > a very natural way to express certain operations. > > I really hope that support for this feature can be added to NumPy. Vectorizing apply is what you're looking for, by the sound of it: In [13]: a = array([lambda x: x**2, lambda x: x**3]) In [14]: b = arange(5) In [15]: va = vectorize(lambda f, x: f(x)) In [16]: va(a[:,newaxis],b[newaxis,:]) Out[16]: array([[ 0, 1, 4, 9, 16], [ 0, 1, 8, 27, 64]]) Once in a while it would also be nice to vectorize methods, either over self or not over self, but the same trick (vectorize an anonymous function wrapper) should work fine. Varadic functions do give you headaches; I don't think even frompyfunc will allow you to vectorize only some of the arguments of a function and leave the others unchanged. Anne From acorriga at gmu.edu Wed Mar 21 09:52:16 2007 From: acorriga at gmu.edu (Andrew Corrigan) Date: Wed, 21 Mar 2007 13:52:16 +0000 (UTC) Subject: [Numpy-discussion] Array of Callables References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> <46001593.2040206@gmail.com> Message-ID: Anne Archibald gmail.com> writes: > Vectorizing apply is what you're looking for, by the sound of it: > In [13]: a = array([lambda x: x**2, lambda x: x**3]) > > In [14]: b = arange(5) > > In [15]: va = vectorize(lambda f, x: f(x)) > > In [16]: va(a[:,newaxis],b[newaxis,:]) > Out[16]: > array([[ 0, 1, 4, 9, 16], > [ 0, 1, 8, 27, 64]]) Thanks for pointing that out. Technically that works, but it doesn't really express this operation as concisely and as naturally as I'd like to be able to. What I really want is to be able to write: >>> a = array([lambda x: x**2, lambda x: x**3]) >>> b = arange(5) >>> a(b) and get: array([[ 0, 1, 4, 9, 16], [ 0, 1, 8, 27, 64]]) instead of the present error message: Traceback (most recent call last): File "", line 1, in ? TypeError: 'numpy.ndarray' object is not callable From ogdude at googlemail.com Wed Mar 21 10:47:55 2007 From: ogdude at googlemail.com (David Koch) Date: Wed, 21 Mar 2007 15:47:55 +0100 Subject: [Numpy-discussion] Adapting algorithm to accept Scipy sparse matrix In-Reply-To: References: Message-ID: Ok, I will bump this once ... The "worst" problem I encountered is that sparse matrices do not seem to support the kind of indexing I need. At least I get "NotImplementedError: sequence indexing not yet fully supported" and " supports slices only of a single row" errors all the time. Any advice on how to manage the transition ndarray -> sparse.matrix? Maybe I should wait until sparse matrix support in NumPy has matured? Thank you, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From mclay at l2sg.com Wed Mar 21 10:53:36 2007 From: mclay at l2sg.com (Michael McLay) Date: Wed, 21 Mar 2007 10:53:36 -0400 Subject: [Numpy-discussion] Array of Callables In-Reply-To: References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> Message-ID: <200703211053.36290.mclay@l2sg.com> On Wednesday 21 March 2007 09:52, Andrew Corrigan wrote: > Anne Archibald gmail.com> writes: > > Vectorizing apply is what you're looking for, by the sound of it: > > In [13]: a = array([lambda x: x**2, lambda x: x**3]) > > > > In [14]: b = arange(5) > > > > In [15]: va = vectorize(lambda f, x: f(x)) > > > > In [16]: va(a[:,newaxis],b[newaxis,:]) > > Out[16]: > > array([[ 0, 1, 4, 9, 16], > > [ 0, 1, 8, 27, 64]]) > > Thanks for pointing that out. Technically that works, but it doesn't > really express this operation as concisely and as naturally as I'd like to > be able to. > > What I really want is to be able to write: > >>> a = array([lambda x: x**2, lambda x: x**3]) > >>> b = arange(5) > >>> a(b) > > and get: > array([[ 0, 1, 4, 9, 16], > [ 0, 1, 8, 27, 64]]) > > instead of the present error message: > Traceback (most recent call last): > File "", line 1, in ? > TypeError: 'numpy.ndarray' object is not callable You could create a class with a __call__ method like the following. from numpy import arange, array,vectorize,newaxis class VA: def __init__(self,arg): self.a = array(arg) def __call__(self,b): va = vectorize(lambda f, x: f(x)) return va(self.a[:,newaxis],b[newaxis,:]) a = VA([lambda x: x**2, lambda x: x**3]) b = arange(5) print a(b) From lists.steve at arachnedesign.net Wed Mar 21 11:04:33 2007 From: lists.steve at arachnedesign.net (Steve Lianoglou) Date: Wed, 21 Mar 2007 11:04:33 -0400 Subject: [Numpy-discussion] Adapting algorithm to accept Scipy sparse matrix In-Reply-To: References: Message-ID: <3C07E2F3-6FEC-4784-935C-3E0B8EF5D8D0@arachnedesign.net> Hi David, > The "worst" problem I encountered is that sparse matrices do not > seem to support the kind of indexing I need. At least I get > "NotImplementedError: sequence indexing not yet fully supported" > and " supports slices only of a single row" > errors all the time. I agree .. this is kind of sucky. There was a (short) thread (I think) a few weeks ago that had mentioned something about doing more work on the sparse matrix stuff, but I can't seem to find it. The day when we'll have a unified interface to normal/dense matrices and sparse matrices will be a great day indeed ... well, ok, at least a pretty darn good day ... -steve From ogdude at googlemail.com Wed Mar 21 11:47:58 2007 From: ogdude at googlemail.com (David Koch) Date: Wed, 21 Mar 2007 16:47:58 +0100 Subject: [Numpy-discussion] Matlab -> NumPy translation and indexing In-Reply-To: <525f23e80703200527u2da02515o17d9b2202f6e5755@mail.gmail.com> References: <45F7FCBE.5060904@ntc.zcu.cz> <45F810AF.3060900@molden.no> <20070314184605.GD18520@clipper.ens.fr> <20070317151224.GA7104@mentat.za.net> <525f23e80703200527u2da02515o17d9b2202f6e5755@mail.gmail.com> Message-ID: Ah! So much ado about nothing. What I was looking for was in fact: B[A_idx][:,A_idx] ... it's even explained in the the NumPy for Matlab Users doc on scipy.org /Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From shane.holloway at ieee.org Wed Mar 21 12:01:03 2007 From: shane.holloway at ieee.org (Shane Holloway) Date: Wed, 21 Mar 2007 10:01:03 -0600 Subject: [Numpy-discussion] Array of Callables In-Reply-To: References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> <46001593.2040206@gmail.com> Message-ID: <45C50E80-9179-4D4A-A782-58A98E3F75D6@ieee.org> On Mar 21, 2007, at 6:58 AM, Anne Archibald wrote: > Vectorizing apply is what you're looking for, by the sound of it: > In [13]: a = array([lambda x: x**2, lambda x: x**3]) > > In [14]: b = arange(5) > > In [15]: va = vectorize(lambda f, x: f(x)) > > In [16]: va(a[:,newaxis],b[newaxis,:]) > Out[16]: > array([[ 0, 1, 4, 9, 16], > [ 0, 1, 8, 27, 64]]) > > Once in a while it would also be nice to vectorize methods, either > over self or not over self, but the same trick (vectorize an anonymous > function wrapper) should work fine. Varadic functions do give you > headaches; I don't think even frompyfunc will allow you to vectorize > only some of the arguments of a function and leave the others > unchanged. > > Anne Thanks for the info. I tried this, and found that it benchmarks about half of just iterating over the methods in the list and calling them. I think the reason is that two python frames are actually being run -- one for the vectorized apply, and one for whatever I'm calling. So what I'm thinking is that I could accomplish this if I implemented the apply call as a C-level ufunc. I'll try hacking about with this idea. Thanks Anne! From acorriga at gmu.edu Wed Mar 21 14:10:35 2007 From: acorriga at gmu.edu (Andrew Corrigan) Date: Wed, 21 Mar 2007 18:10:35 +0000 (UTC) Subject: [Numpy-discussion] Array of Callables References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> <200703211053.36290.mclay@l2sg.com> Message-ID: Good point! I think I will, Thanks a lot. From peridot.faceted at gmail.com Wed Mar 21 14:32:21 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 21 Mar 2007 14:32:21 -0400 Subject: [Numpy-discussion] Array of Callables In-Reply-To: References: <371590A8-5563-45B6-82EE-9A1BC024CF79@ieee.org> <46001593.2040206@gmail.com> Message-ID: On 21/03/07, Andrew Corrigan wrote: > Thanks for pointing that out. Technically that works, but it doesn't really > express this operation as concisely and as naturally as I'd like to be able to. > > What I really want is to be able to write: > > >>> a = array([lambda x: x**2, lambda x: x**3]) > >>> b = arange(5) > >>> a(b) > > and get: > array([[ 0, 1, 4, 9, 16], > [ 0, 1, 8, 27, 64]]) Well, this only makes sense for an array of callables. So you could hope for a "callable" dtype to be put into numpy, or you could write your own array class, derived from the standard one, that only holds callables and that implements a __call__ method. Anne From openopt at ukr.net Mon Mar 19 08:57:42 2007 From: openopt at ukr.net (dmitrey) Date: Mon, 19 Mar 2007 14:57:42 +0200 Subject: [Numpy-discussion] basearray & dimarray : you have dead URL; + some questions Message-ID: <45FE88C6.3020108@ukr.net> Hallo! Excuse my bad English. In the web page http://www.scipy.org/BaseArray/Application in the section Project details and tentative schedule you refer to the dead link with name http://numeric.scipy.org/array_interface.html which leads to http://numpy.scipy.org//array_interface.html which is dead (with singular '/' before array_interface.html - too) I'm using Mozilla browser if it's important. Now I'm writing some code and want to know, what will dotwise and matrix operations look like? Is there any discussion where these and related questions are concerned? So, will they be MATLAB/Octave/omatrix/SciLab/etc look-like: dot is present => dotwise operation dot is absend => matrix operation or you'll continue numpy array operations, which seems to be very unclear for newbies? As for me I'm sure that in long-term period 1st approach is much more better, because users will not have to dig in numpy documentation for to chose appropriate operator, and translating code from those languages will be much more easier. Of course, many people can say that 2nd way simplifies translating code from already-written py-files, but it will be done only once, while migrating MATLAB, Octave, etc users will continue for years or dozens of years or even more. Maybe you'll organize a voting or separate discussion thread or something else to gather users opinions? And 2nd question is - is it so important to have own class? Maybe something from c/c++ boost (www.boost.org), or collaboration with Octave developers will be more fast & require less workers and money? And it will be possible to fork any time when something will be going wrong. As far as I understood from my discussions in octave mailing lists with David Bateman, they still have many problems with for example sparse matrices. Even Mathworks still have some troubles, and sparse witn dimension > 2 isn't implemented at all. So collaborating could be much more productive. And, if you still want to create 100% your own code, what are tasks for students participating in GSoC? I want to estimate the complexity of jobs (am I able to finish them successfully till the deadlines or no for any reasons). Thank you in advance for your answer, Dmitrey. From mike at biglan.org Tue Mar 20 13:44:58 2007 From: mike at biglan.org (Mike Biglan) Date: Tue, 20 Mar 2007 10:44:58 -0700 Subject: [Numpy-discussion] Grouping & Collapsing multi-dimensional data Message-ID: <1174412698.965769.66660@b75g2000hsg.googlegroups.com> I might be using the wrong terminology but I'm trying to take a 2d array where each row has a department object and then 36 floats after it, eg: [dept1, 3,6,7...] With SQL or R i know how to collapse a simple 2d data structure like this. For example in SQL: select dept, stddev(field1)... from tbl_x group by dept But I want to end up with either a collapsed table grouped on dept where each element is some summary statistic, or better yet where each value is a dictionary of name=summary statistic and value = value of that statistic. Just to sort and perform this by group has been problematic because it gives me an error when sorting by an object if that object has __cmp__ method. I can do an index-based sort after using a field in the object -- but then must figure out the groups of rows manually. There must be easier methods to collapse on a dimension, grouping by one or more elements and applying arbitrary functions. Can this be done with numpy? Should i work with scipy? thanks - mike biglan ps: my example here is 2d but i'm hoping that this functionality would work for any-d From miquel.poch at gmail.com Wed Mar 21 04:23:51 2007 From: miquel.poch at gmail.com (Miquel Poch) Date: Wed, 21 Mar 2007 09:23:51 +0100 Subject: [Numpy-discussion] Translation of Matlab function find() In-Reply-To: <8b0882fc0703201120v22c09460o3c10471b39606852@mail.gmail.com> References: <8b0882fc0703201120v22c09460o3c10471b39606852@mail.gmail.com> Message-ID: <8b0882fc0703210123j333ea3d1ve66788285c9c55e@mail.gmail.com> Hi, I'm trying to translate some Matlab functions. I don't know exactly how make and equivalent for function find(). This give us the index of an array where a condition it's true. I've found some options like this one: (a>0).nonzero(), where a is the array and (a>0) the condition. But the problem appear with this function return. It's something like this: >>> from numpy import * >>> a=array([0,1,0,1,2,0,1,2,0,0]) >>> print a [0 1 0 1 2 0 1 2 0 0] >>> b=(a>0).nonzero() >>> print b (array([1, 3, 4, 6, 7]),) >>> print (a>0).nonzero() (array([1, 3, 4, 6, 7]),) I can't acces to b data, doing something like b[x]. Why b is not an array? I think another possible is use where() function, also avilable in numpy library. It's better or worse? Thanks in advance, Miquel -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at romelfanger.com Wed Mar 21 11:42:44 2007 From: fred at romelfanger.com (Fred Romelfanger) Date: Wed, 21 Mar 2007 11:42:44 -0400 Subject: [Numpy-discussion] problems building exported version of numpy from svn Message-ID: <46015274.9050103@romelfanger.com> When I use co to check the code out of svn, running setup.py and building the code works fine, but when I export the code or create a source distribution that includes numpy the .svn directories get stripped. svn export http://svn.scipy.org/svn/numpy/trunk numpy I then get the following error when I run "python setup.py build" AssertionError: hmm, why I am not inside SVN tree??? How do I export a copy of numpy that I can build outside of an svn tree? We were using numpy 1.0.1, but it doesn't build cleanly on RHEL v3 with the old version of ATLAS that comes with the system software. I was hoping a newer version of numpy from svn might correct that problem. Thanks -- fred. From robert.kern at gmail.com Wed Mar 21 15:02:55 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Mar 2007 14:02:55 -0500 Subject: [Numpy-discussion] basearray & dimarray : you have dead URL; + some questions In-Reply-To: <45FE88C6.3020108@ukr.net> References: <45FE88C6.3020108@ukr.net> Message-ID: <4601815F.2060409@gmail.com> dmitrey wrote: > Hallo! > Excuse my bad English. > > In the web page > http://www.scipy.org/BaseArray/Application > in the section Note that this was an application for last year's GSoC. It's not an idea for this year's GSoC. > Now I'm writing some code and want to know, what will dotwise and matrix > operations look like? I believe the scope of that project was not to provide a full matrix or even array computation package; we already have one in numpy. Rather, it was to put a multi-dimensional array object into the Python standard library such that it could be used by multiple packages for data interchange rather than an object that would be used for computation itself. The current proposal has moved to specifying a protocol that any object can provide. http://svn.scipy.org/svn/numpy/trunk/numpy/doc/pep_buffer.txt Consequently, the rest of your questions don't have answers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Mar 21 15:08:56 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Mar 2007 14:08:56 -0500 Subject: [Numpy-discussion] Translation of Matlab function find() In-Reply-To: <8b0882fc0703210123j333ea3d1ve66788285c9c55e@mail.gmail.com> References: <8b0882fc0703201120v22c09460o3c10471b39606852@mail.gmail.com> <8b0882fc0703210123j333ea3d1ve66788285c9c55e@mail.gmail.com> Message-ID: <460182C8.60002@gmail.com> Miquel Poch wrote: > Hi, > > I'm trying to translate some Matlab functions. I don't know exactly how > make and equivalent for function find(). This give us the index of an > array where a condition it's true. > > I've found some options like this one: (a>0).nonzero(), where a is the > array and (a>0) the condition. But the problem appear with this function > return. It's something like this: > >>>> from numpy import * >>>> a=array([0,1,0,1,2,0,1,2,0,0]) >>>> print a > [0 1 0 1 2 0 1 2 0 0] >>>> b=(a>0).nonzero() >>>> print b > (array([1, 3, 4, 6, 7]),) >>>> print (a>0).nonzero() > (array([1, 3, 4, 6, 7]),) > > I can't acces to b data, doing something like b[x]. Why b is not an array? Because when `a` is a multi-dimensional array, the result of (a>0).nonzero() must be a tuple. For consistency's sake, the result of .nonzero() (and where(a>0) for that matter) is always a tuple even if it just has a single element. Thus, you probably want b = (a>0).nonzero()[0] > I think another possible is use where() function, also avilable in numpy > library. It's better or worse? More or less the same. It's possibly more straightforward to read. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Wed Mar 21 15:11:23 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Mar 2007 14:11:23 -0500 Subject: [Numpy-discussion] problems building exported version of numpy from svn In-Reply-To: <46015274.9050103@romelfanger.com> References: <46015274.9050103@romelfanger.com> Message-ID: <4601835B.3060500@gmail.com> Fred Romelfanger wrote: > When I use co to check the code out of svn, running setup.py and building > the code works fine, but when I export the code or create a source > distribution that includes numpy the .svn directories get stripped. > > svn export http://svn.scipy.org/svn/numpy/trunk numpy > > I then get the following error when I run "python setup.py build" > > AssertionError: hmm, why I am not inside SVN tree??? > > How do I export a copy of numpy that I can build outside of an svn tree? We grab some of the versioning information from the .svn/ metadata. If you want a tarball without the .svn/ data for some reason, you should do an SVN checkout, run "python setup.py sdist" and then the source tarball will be in the dist/ directory with the appropriate versioning information already there. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From faltet at carabos.com Wed Mar 21 15:34:13 2007 From: faltet at carabos.com (Francesc Altet) Date: Wed, 21 Mar 2007 20:34:13 +0100 Subject: [Numpy-discussion] Translation of Matlab function find() In-Reply-To: <8b0882fc0703210123j333ea3d1ve66788285c9c55e@mail.gmail.com> References: <8b0882fc0703201120v22c09460o3c10471b39606852@mail.gmail.com> <8b0882fc0703210123j333ea3d1ve66788285c9c55e@mail.gmail.com> Message-ID: <1174505653.2598.50.camel@localhost.localdomain> El dc 21 de 03 del 2007 a les 09:23 +0100, en/na Miquel Poch va escriure: > Hi, > > I'm trying to translate some Matlab functions. I don't know exactly > how make and equivalent for function find(). This give us the index of > an array where a condition it's true. > > I've found some options like this one: (a>0).nonzero(), where a is the > array and (a>0) the condition. But the problem appear with this > function return. It's something like this: > > >>> from numpy import * > >>> a=array([0,1,0,1,2,0,1,2,0,0]) > >>> print a > [0 1 0 1 2 0 1 2 0 0] > >>> b=(a>0).nonzero() > >>> print b > (array([1, 3, 4, 6, 7]),) > >>> print (a>0).nonzero() > (array([1, 3, 4, 6, 7]),) > > I can't acces to b data, doing something like b[x]. Why b is not an > array? This is in order to allow generality. Think about this: >>> b=numpy.array([0,1,0,1,2,0,1,2,0,0]).reshape(2,5) >>> numpy.where(b>0) (array([0, 0, 0, 1, 1]), array([1, 3, 4, 1, 2])) The result are the coordinates of the elements that are greater than zero, where in the first result array are the 1st dimension coords and in the second are the second dimension ones. If you want to group the indices by element, rather than dimension, use: >>> numpy.transpose(numpy.where(b>0)) array([[0, 1], [0, 3], [0, 4], [1, 1], [1, 2]]) Extrapolating this, for a 1-dimensional a array, we have: >>> a=numpy.array([0,1,0,1,2,0,1,2,0,0]) >>> numpy.where(a>0) (array([1, 3, 4, 6, 7]),) if this is not what you want, just index the result: >>> c = numpy.where(a>0)[0] >>> c array([1, 3, 4, 6, 7]) and you are done. > I think another possible is use where() function, also avilable in > numpy library. It's better or worse? Well, where() does more things than .nonzero() (which is a particular case of where()); do a help(numpy.where) for more info. However, they perform similar: >>> t1=Timer('numpy.where(a>0)', 'import numpy; a=numpy.arange(10)') >>> t2=Timer('(a>0).nonzero()', 'import numpy; a=numpy.arange(10)') >>> t1.repeat(3, 10000) [0.27704501152038574, 0.23136401176452637, 0.23428702354431152] >>> t2.repeat(3, 10000) [0.26156210899353027, 0.21894097328186035, 0.21954011917114258] so, use whatever you prefer. Salutacions, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From ogdude at googlemail.com Wed Mar 21 15:47:15 2007 From: ogdude at googlemail.com (David Koch) Date: Wed, 21 Mar 2007 20:47:15 +0100 Subject: [Numpy-discussion] Adapting algorithm to accept Scipy sparse matrix In-Reply-To: <3C07E2F3-6FEC-4784-935C-3E0B8EF5D8D0@arachnedesign.net> References: <3C07E2F3-6FEC-4784-935C-3E0B8EF5D8D0@arachnedesign.net> Message-ID: Alright, may all the trickery rest until that day. One thing I need to do however is patch a column of "ones" onto a sparse matrix of format n * d with n >> d. I tried "concatenate" and it didn't work so I did like this: def spInsCol(X): "insert doc string" n, d = shape(X) X = X.tocsc() newX = sparse.lil_matrix((d + 1,n)) newX[0,:] = ones(n) for i in range(1, d + 1): newX[i,:] = X[:,i - 1].toarray().flatten() return newX.transpose() It basically, blows each column in the old matrix to dense format and assigns it to a row in a new matrix which already contains an extra row of ones. In the end I transpose. Is there a simpler and more efficient solution? Thank you, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at enthought.com Wed Mar 21 15:59:06 2007 From: eric at enthought.com (eric jones) Date: Wed, 21 Mar 2007 14:59:06 -0500 Subject: [Numpy-discussion] Little module to get numpy examples In-Reply-To: References: Message-ID: <46018E8A.3050507@enthought.com> Just looked at this... Now that is just cool. I'd say it should be part of Numpy. eric Bill Baxter wrote: > On 3/19/07, Bill Baxter wrote: > >> I wrote a little python module to go fetch the Numpy examples from the >> scipy wiki page, parse them, and print out entries. >> >> Is there a good place on the wiki for this? >> It didn't really seem right in the cookbook, and it doesn't quite fit >> with documentation. >> > > Since no-one responded to this, I went ahead and added various links. > If you don't like em nuke em. > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From stefan at sun.ac.za Wed Mar 21 17:32:46 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 21 Mar 2007 23:32:46 +0200 Subject: [Numpy-discussion] Little module to get numpy examples In-Reply-To: <46018E8A.3050507@enthought.com> References: <46018E8A.3050507@enthought.com> Message-ID: <20070321213246.GB6085@mentat.za.net> On Wed, Mar 21, 2007 at 02:59:06PM -0500, eric jones wrote: > Just looked at this... Now that is just cool. > > I'd say it should be part of Numpy. Very useful! A file cache would be handy, and can be implemented using the checksum of the page from http://www.scipy.org/Numpy_Example_List?action=info&general=1 Is the goal to eventually merge these docs into numpy, or to keep them separate it can easily be updated by anyone? I'm somewhat apprehensive of having code in numpy that displays uncensored information directly from the wiki. Cheers St?fan From wbaxter at gmail.com Wed Mar 21 18:21:30 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Thu, 22 Mar 2007 07:21:30 +0900 Subject: [Numpy-discussion] Little module to get numpy examples In-Reply-To: <20070321213246.GB6085@mentat.za.net> References: <46018E8A.3050507@enthought.com> <20070321213246.GB6085@mentat.za.net> Message-ID: On 3/22/07, Stefan van der Walt wrote: > On Wed, Mar 21, 2007 at 02:59:06PM -0500, eric jones wrote: > > Just looked at this... Now that is just cool. > > > > I'd say it should be part of Numpy. > > Very useful! A file cache would be handy, and can be implemented > using the checksum of the page from > > http://www.scipy.org/Numpy_Example_List?action=info&general=1 Is there any way to get that checksum without html formatting? Like the 'action=raw' that gets just the text of the page? > Is the goal to eventually merge these docs into numpy, or to keep them > separate it can easily be updated by anyone? I'm somewhat > apprehensive of having code in numpy that displays uncensored > information directly from the wiki. I agree that a built-in command that's part of numpy that grabs raw info off the web would be odd. It think it's better to just leave it as a useful add-on tool. --bb From bryan at cole.uklinux.net Wed Mar 21 18:31:00 2007 From: bryan at cole.uklinux.net (Bryan Cole) Date: Wed, 21 Mar 2007 22:31:00 +0000 Subject: [Numpy-discussion] memory leak w/ numpy & Numeric together Message-ID: <1174516260.6343.13.camel@pc1.cole.uklinux.net> I'm not sure where best to post this, but I get a memory leak when using code with both numpy and FFT(from Numeric) together: e.g. >>> import numpy >>> import FFT >>> def test(): ... while 1: ... data=numpy.random.random(2048) ... newdata = FFT.real_fft(data) >>> test() and memory consumption goes through the roof. I'm using numpy-1.0.1 and Numeric-24.2 (on FC6). This has now forced me to go 100% numpy... BC From oliphant at ee.byu.edu Wed Mar 21 18:46:09 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 21 Mar 2007 15:46:09 -0700 Subject: [Numpy-discussion] memory leak w/ numpy & Numeric together In-Reply-To: <1174516260.6343.13.camel@pc1.cole.uklinux.net> References: <1174516260.6343.13.camel@pc1.cole.uklinux.net> Message-ID: <4601B5B1.3020008@ee.byu.edu> Bryan Cole wrote: >I'm not sure where best to post this, but I get a memory leak when using >code with both numpy and FFT(from Numeric) together: > >e.g. > > > >>>>import numpy >>>>import FFT >>>>def test(): >>>> >>>> >... while 1: >... data=numpy.random.random(2048) >... newdata = FFT.real_fft(data) > > >>>>test() >>>> >>>> > >and memory consumption goes through the roof. I'm using numpy-1.0.1 and >Numeric-24.2 (on FC6). > > > There may be a memory leak in the way Numeric uses the array interface. >This has now forced me to go 100% numpy... > > Not a bad thing :-) -Travis From pgmdevlist at gmail.com Wed Mar 21 19:09:42 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 21 Mar 2007 19:09:42 -0400 Subject: [Numpy-discussion] loess (was Re: distutils for a Pyrex module) In-Reply-To: <460066CE.7060402@gmail.com> References: <200703201706.08227.pgmdevlist@gmail.com> <200703201845.36736.pgmdevlist@gmail.com> <460066CE.7060402@gmail.com> Message-ID: <200703211909.44111.pgmdevlist@gmail.com> On Tuesday 20 March 2007 18:57:18 Robert Kern wrote: > Is there an init_loess function in cloess.c? If cloess.c was created by > Pyrex from cloess.pyx, then Pyrex will make an initcloess function. The > module name needs to be consistent throughout. Ahah, that was the problem. Thanks a lot Robert, that did it ! I just moved the loess package out of its temporary sandbox. The routines are now available in the Scipy sandbox, in the package pyloess, along with lowess and STL. Everything works but for a couple of details (two arrays of parameters have to be set globally instead of element-wise). As usual, the tests directory contains tests and examples. And as usual again, I'd be happy to get some kind of feedback. Thanks again for everything P. From mkg at cs.nyu.edu Wed Mar 21 23:42:34 2007 From: mkg at cs.nyu.edu (Matthew Koichi Grimes) Date: Wed, 21 Mar 2007 23:42:34 -0400 Subject: [Numpy-discussion] scalar recordarrays In-Reply-To: References: Message-ID: <4601FB2A.2020703@cs.nyu.edu> If there are no objections, I'll file this ticket in the trac site: Title: Return type inconsistency in recarray Description: The sub-arrays of rank-0 recarrays are returned as scalars rather than rank-0 ndarrays. Example: >>> import numpy as N >>> dt = N.dtype([('x','f8'),('y','f8')]) >>> rarr = N.zeros((), dtype = dt).view(N.recarray) >>> rarr recarray((0.0, 0.0), dtype=[('x', '>> # oddly, rarr.x is not an array >>> rarr.x 0.0 >>> # This makes it impossible to fill values in the usual manner >>> rarr.x[...] = 2.0 TypeError: object does not support item assignment >>> # A workaround is to reshape to non-zero rank >>> rarr.shape = [1] >>> # Now rarr.x returns a ndarray of the same rank as rarr >>> rarr.x array([ 0.]) >>> # Value-setting now works as expected. >>> rarr.x[...] = 2.0 >>> rarr recarray([(2.0, 0.0)], dtype=[('x', ' -- Matt Francesc Altet wrote: > Yeah, I agree. In fact, one of the main reasons to keep rank-0 arrays > around is to provide generality enough to address to these kind of > problems in an elegant way. > > Cheers, > From jturner at gemini.edu Thu Mar 22 14:34:54 2007 From: jturner at gemini.edu (James Turner) Date: Thu, 22 Mar 2007 14:34:54 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <45FEDF10.3010707@gemini.edu> References: <45FEDF10.3010707@gemini.edu> Message-ID: <4602CC4E.3060102@gemini.edu> The people at STScI put me in touch with Peter Verveer, the author of nd_image. Unfortunately Peter is currently unable to maintain the code (either in numarray or scipy), but he did send me some comments on the problem discussed in this thread. Here's what he said: James. ----- Hi James, Yes, it could be that you see some mirroring. Let me first explain what the mode options do: If you try to interpolate a value that falls outside of the boundaries, then that is done by either setting it constant (mode='constant') or by mapping to a position inside the boundaries, i.e. by mirroring, and then interpolating. So the mode argument really deals with extrapolating. Problem is when you are mapping a value that is inside the boundaries, but very close. Interpolation is done by splines which require that a window is placed around the point you are interpolating, and the result is calculated from the values inside that window. Thus, if you are close to the boundary, part of that window will fall outside the boundaries, and the algorithm must choose how to set the values outside the boundary. Ideally that would be done in the same way as above, i.e. you should have the choice if that is done by a constant value, or by mirroring etc. Unfortunately, the spline algorithms that I use (references for the algorithm are in the manual) have an intrinsic mirroring assumption build in. Thus for interpolating values inside the boundaries, a mirror boundary is the only option. I did not figure out a way to modify the interpolation algorithms. Therefore, if you choose a mode different from mirroring, then you end up with small artifacts at the boundaries. Presumably these artifacts will get bigger if the order of spline interpolation you select is larger. So, its not really a bug, its a undesired feature... Cheers, Peter From peridot.faceted at gmail.com Thu Mar 22 14:41:52 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 22 Mar 2007 14:41:52 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <4602CC4E.3060102@gemini.edu> References: <45FEDF10.3010707@gemini.edu> <4602CC4E.3060102@gemini.edu> Message-ID: On 22/03/07, James Turner wrote: > So, its not really a bug, its a undesired feature... It is curable, though painful - you can pad the image out, given an estimate of the size of the window. Yes, this sucks. Anne. From zpincus at stanford.edu Thu Mar 22 15:07:55 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu, 22 Mar 2007 12:07:55 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <4602CC4E.3060102@gemini.edu> References: <45FEDF10.3010707@gemini.edu> <4602CC4E.3060102@gemini.edu> Message-ID: Hi James, Would it be possible to ask Peter if he knows anything that could help us resolve scipy ticket 213 ( http://projects.scipy.org/scipy/ scipy/ticket/213 )? The basic issue is that ndimage.spline_filter seems to introduce nasty ringing artifacts, which make all of the geometric transforms (affine_transform included) perform very badly. Examples of these artifacts can be seen in the attachments to this ticket: http://projects.scipy.org/scipy/scipy/attachment/ticket/213/ rotate_artifacts.jpg http://projects.scipy.org/scipy/scipy/attachment/ticket/213/spline% 20error.png If Peter has any suggestions at all as to what's happened, that would be very helpful. Hopefully this won't be too much of an imposition, but these problems with the spline prefiltering are really hampering ndimage, so hopefully it will be worth it. Thanks, Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine On Mar 22, 2007, at 11:34 AM, James Turner wrote: > The people at STScI put me in touch with Peter Verveer, the author of > nd_image. Unfortunately Peter is currently unable to maintain the code > (either in numarray or scipy), but he did send me some comments on the > problem discussed in this thread. Here's what he said: > > James. > > ----- > > Hi James, > > Yes, it could be that you see some mirroring. Let me first explain > what the mode > options do: > > If you try to interpolate a value that falls outside of the > boundaries, then > that is done by either setting it constant (mode='constant') or by > mapping to a > position inside the boundaries, i.e. by mirroring, and then > interpolating. So > the mode argument really deals with extrapolating. > > Problem is when you are mapping a value that is inside the > boundaries, but very > close. Interpolation is done by splines which require that a window > is placed > around the point you are interpolating, and the result is > calculated from the > values inside that window. Thus, if you are close to the boundary, > part of that > window will fall outside the boundaries, and the algorithm must > choose how to > set the values outside the boundary. Ideally that would be done in > the same way > as above, i.e. you should have the choice if that is done by a > constant value, > or by mirroring etc. > > Unfortunately, the spline algorithms that I use (references for > the algorithm > are in the manual) have an intrinsic mirroring assumption build in. > Thus for > interpolating values inside the boundaries, a mirror boundary is > the only > option. I did not figure out a way to modify the interpolation > algorithms. > Therefore, if you choose a mode different from mirroring, then you > end up with > small artifacts at the boundaries. Presumably these artifacts will > get bigger if > the order of spline interpolation you select is larger. > > So, its not really a bug, its a undesired feature... > > Cheers, Peter > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Thu Mar 22 16:25:11 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 22 Mar 2007 14:25:11 -0600 Subject: [Numpy-discussion] Contiguous flags not quite right yet. Message-ID: Example. In [18]:a = array([1,2,3]) In [19]:a.flags Out[19]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [20]:a.shape = (1,3) In [21]:a.flags Out[21]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [22]:a.shape = (3,1) In [23]:a.flags Out[23]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False All three shapes are both C_CONTIGUOUS and F_CONTIGUOUS. I think ignoring all 1's in the shape would give the right results for otherwise contiguous arrays because in those positions the index can only take the value 0. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Mar 22 18:19:36 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 22 Mar 2007 15:19:36 -0700 Subject: [Numpy-discussion] Contiguous flags not quite right yet. In-Reply-To: References: Message-ID: <460300F8.3010204@ee.byu.edu> Charles R Harris wrote: > All three shapes are both C_CONTIGUOUS and F_CONTIGUOUS. I think > ignoring all 1's in the shape would give the right results for > otherwise contiguous arrays because in those positions the index can > only take the value 0. > I've thought about this before too. But, every time I've tried to do something like that with the flags, I get many errors in numpy and scipy because of the way contiguousness is actually used. Somehow, I've never gotten the right check to allow additional striding arrays to work correctly. The _IsContiguous and _IsFortranContiguous functions are what should be changed if anything is done. -Travis From stefan at sun.ac.za Thu Mar 22 18:35:19 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Mar 2007 00:35:19 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: References: <45FEDF10.3010707@gemini.edu> <4602CC4E.3060102@gemini.edu> Message-ID: <20070322223518.GA6085@mentat.za.net> On Thu, Mar 22, 2007 at 02:41:52PM -0400, Anne Archibald wrote: > On 22/03/07, James Turner wrote: > > > So, its not really a bug, its a undesired feature... > > It is curable, though painful - you can pad the image out, given an > estimate of the size of the window. Yes, this sucks. I would rather opt for changing the spline fitting algorithm than for padding with zeros. St?fan From oliphant at ee.byu.edu Thu Mar 22 19:33:53 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 22 Mar 2007 16:33:53 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070322223518.GA6085@mentat.za.net> References: <45FEDF10.3010707@gemini.edu> <4602CC4E.3060102@gemini.edu> <20070322223518.GA6085@mentat.za.net> Message-ID: <46031261.4020607@ee.byu.edu> Stefan van der Walt wrote: >On Thu, Mar 22, 2007 at 02:41:52PM -0400, Anne Archibald wrote: > > >>On 22/03/07, James Turner wrote: >> >> >> >>>So, its not really a bug, its a undesired feature... >>> >>> >>It is curable, though painful - you can pad the image out, given an >>estimate of the size of the window. Yes, this sucks. >> >> > >I would rather opt for changing the spline fitting algorithm than for >padding with zeros. > > From what I understand, the splines used in ndimage have the implicit mirror-symmetric boundary condition which also allows them to be computed rapidly. There may be ways to adapt other boundary conditions and maintain rapid evaluation, but it is not trivial as far as I know. Standard spline-fitting allows multiple boundary conditions because matrix inversion is used. I think the spline-fitting done in ndimage relies on equal-spacing and mirror-symmetry to allow simple IIR filters to be used to compute the spline coefficients very rapidly. This assumption would have to be re-visited for any changes to be made. -Travis From bblais at bryant.edu Thu Mar 22 20:13:22 2007 From: bblais at bryant.edu (Brian Blais) Date: Thu, 22 Mar 2007 20:13:22 -0400 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D Message-ID: <46031BA2.4020405@bryant.edu> Hello, I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns (one for each of the original 1D arrays). I thought this would work: In [47]:a=arange(0,10,1) In [48]:a Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [49]:b=arange(-10,0,1) In [51]:b Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) In [54]:concatenate((a,b)) Out[54]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) In [55]:concatenate((a,b),axis=1) Out[55]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) but it never expands the dimensions. Do I have to do this... In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) Out[65]: array([[ 0, -10], [ 1, -9], [ 2, -8], [ 3, -7], [ 4, -6], [ 5, -5], [ 6, -4], [ 7, -3], [ 8, -2], [ 9, -1]]) ? I thought there would be an easier way. Did I overlook something? thanks, Brian Blais -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From wbaxter at gmail.com Thu Mar 22 20:17:53 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 23 Mar 2007 09:17:53 +0900 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: <46031BA2.4020405@bryant.edu> References: <46031BA2.4020405@bryant.edu> Message-ID: Try column_stack, and also try the "See also" parts of the Numpy Examples List. very handy for finding things like this. http://www.scipy.org/Numpy_Example_List --bb On 3/23/07, Brian Blais wrote: > Hello, > > I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns > (one for each of the original 1D arrays). I thought this would work: > > > In [47]:a=arange(0,10,1) > > In [48]:a > Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > In [49]:b=arange(-10,0,1) > > In [51]:b > Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) > > In [54]:concatenate((a,b)) > Out[54]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > -7, -6, -5, -4, -3, -2, -1]) > > In [55]:concatenate((a,b),axis=1) > Out[55]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > -7, -6, -5, -4, -3, -2, -1]) > > > but it never expands the dimensions. Do I have to do this... > > In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > Out[65]: > array([[ 0, -10], > [ 1, -9], > [ 2, -8], > [ 3, -7], > [ 4, -6], > [ 5, -5], > [ 6, -4], > [ 7, -3], > [ 8, -2], > [ 9, -1]]) > > > ? > > I thought there would be an easier way. Did I overlook something? > > thanks, > > Brian Blais > > -- > ----------------- > > bblais at bryant.edu > http://web.bryant.edu/~bblais > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Thu Mar 22 20:20:05 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Mar 2007 02:20:05 +0200 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: <46031BA2.4020405@bryant.edu> References: <46031BA2.4020405@bryant.edu> Message-ID: <20070323002005.GC6085@mentat.za.net> On Thu, Mar 22, 2007 at 08:13:22PM -0400, Brian Blais wrote: > Hello, > > I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns > (one for each of the original 1D arrays). I thought this would work: > > > In [47]:a=arange(0,10,1) > > In [48]:a > Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > In [49]:b=arange(-10,0,1) > > In [51]:b > Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) > > In [54]:concatenate((a,b)) > Out[54]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > -7, -6, -5, -4, -3, -2, -1]) > > In [55]:concatenate((a,b),axis=1) > Out[55]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > -7, -6, -5, -4, -3, -2, -1]) > > > but it never expands the dimensions. Do I have to do this... > > In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > Out[65]: > array([[ 0, -10], > [ 1, -9], > [ 2, -8], > [ 3, -7], > [ 4, -6], > [ 5, -5], > [ 6, -4], > [ 7, -3], > [ 8, -2], > [ 9, -1]]) > > > ? > > I thought there would be an easier way. Did I overlook something? How about N.vstack((a,b)).T Cheers St?fan From jturner at gemini.edu Thu Mar 22 21:34:58 2007 From: jturner at gemini.edu (James Turner) Date: Thu, 22 Mar 2007 21:34:58 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: References: Message-ID: <46032EC2.3080204@gemini.edu> Hi Zachary, OK - I sent Peter the URL for your post... Cheers, James. From stefan at sun.ac.za Thu Mar 22 21:47:44 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Mar 2007 03:47:44 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <46031261.4020607@ee.byu.edu> References: <45FEDF10.3010707@gemini.edu> <4602CC4E.3060102@gemini.edu> <20070322223518.GA6085@mentat.za.net> <46031261.4020607@ee.byu.edu> Message-ID: <20070323014744.GE6085@mentat.za.net> On Thu, Mar 22, 2007 at 04:33:53PM -0700, Travis Oliphant wrote: > >I would rather opt for changing the spline fitting algorithm than for > >padding with zeros. > > > > > From what I understand, the splines used in ndimage have the implicit > mirror-symmetric boundary condition which also allows them to be > computed rapidly. There may be ways to adapt other boundary conditions > and maintain rapid evaluation, but it is not trivial as far as I know. > Standard spline-fitting allows multiple boundary conditions because > matrix inversion is used. I think the spline-fitting done in ndimage > relies on equal-spacing and mirror-symmetry to allow simple IIR filters > to be used to compute the spline coefficients very rapidly. Thanks, Travis. I wasn't aware of these restrictions. Would it be possible to call fitpack to do the spline fitting? I noticed that it doesn't exhibit the same mirror-property: In [24]: z = scipy.interpolate.splrep([0,1,2,3,4],[0,4,3,2,1]) In [25]: scipy.interpolate.splev([0,1,2,3,4,5],z) Out[25]: array([ -1.32724622e-16, 4.00000000e+00, 3.00000000e+00, 2.00000000e+00, 1.00000000e+00, -1.25000000e+00]) Regards St?fan From jturner at gemini.edu Thu Mar 22 23:13:19 2007 From: jturner at gemini.edu (James Turner) Date: Thu, 22 Mar 2007 23:13:19 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: References: Message-ID: <460345CF.3030009@gemini.edu> By the way, ringing at sharp edges is an intrinsic feature of higher- order spline interpolation, right? I believe this kind of interpolant is really intended for smooth (band-limited) data. I'm not sure why the pre-filtering makes a difference though; I don't yet understand well enough what the pre-filter actually does. I'm not sure what people normally do in computer graphics, since I'm working more with natural band-limited images, but in the case of Stefan's "rotate_artifacts" example, wouldn't it be appropriate to use the 1st- or 0th-order spline instead of the 2nd order? If your real-life data are smooth enough, however, then in theory the ringing with higher orders should go away. Sorry if I'm stating the obvious and missing the real point! I just wanted to make sure this hasn't been overlooked... Likewise, sorry I didn't mention this before if it does turn out to be relevant. Let me know if you want references to explain what I said above. James. From ceball at gmail.com Fri Mar 23 00:33:19 2007 From: ceball at gmail.com (Christopher Ball) Date: Fri, 23 Mar 2007 12:33:19 +0800 Subject: [Numpy-discussion] numpy types for xml pickling Message-ID: <4603588F.9060104@gmail.com> Hi, Gnosis Utils (http://www.gnosis.cx/download/Gnosis_Utils.More/) contains several modules for XML processing, one of which (xml.pickle) serializes objects to and from XML and has an API compatible with Python's pickle [http://freshmeat.net/projects/gnosisxml/]. The xml.pickle module needs to be updated to work with numpy rather than Numeric, and the author would like some code that creates a data structure with one instance of all the numpy types to help with testing. Is there any easy way to do that, or an easy way to get a list of all the types? I have the numpy book, but I couldn't see such a list. I've also tried writing some code to go through the numpy module and find all types, but I wonder if there isn't some better way? Thanks, Chris From haase at msg.ucsf.edu Fri Mar 23 01:02:36 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 22 Mar 2007 22:02:36 -0700 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: <20070323002005.GC6085@mentat.za.net> References: <46031BA2.4020405@bryant.edu> <20070323002005.GC6085@mentat.za.net> Message-ID: On 3/22/07, Stefan van der Walt wrote: > On Thu, Mar 22, 2007 at 08:13:22PM -0400, Brian Blais wrote: > > Hello, > > > > I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns > > (one for each of the original 1D arrays). I thought this would work: > > > > > > In [47]:a=arange(0,10,1) > > > > In [48]:a > > Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > > > In [49]:b=arange(-10,0,1) > > > > In [51]:b > > Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) > > > > In [54]:concatenate((a,b)) > > Out[54]: > > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > > -7, -6, -5, -4, -3, -2, -1]) > > > > In [55]:concatenate((a,b),axis=1) > > Out[55]: > > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > > -7, -6, -5, -4, -3, -2, -1]) > > > > > > but it never expands the dimensions. Do I have to do this... > > > > In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > > Out[65]: > > array([[ 0, -10], > > [ 1, -9], > > [ 2, -8], > > [ 3, -7], > > [ 4, -6], > > [ 5, -5], > > [ 6, -4], > > [ 7, -3], > > [ 8, -2], > > [ 9, -1]]) > > > > > > ? > > > > I thought there would be an easier way. Did I overlook something? > > How about > > N.vstack((a,b)).T > Also mentioned here should be the use of newaxis. As in a[:,newaxis] However I never got a "good feel" for how to use it, so I can't complete the code you would need. -Sebastian Haase From zpincus at stanford.edu Fri Mar 23 02:20:37 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu, 22 Mar 2007 23:20:37 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <460345CF.3030009@gemini.edu> References: <460345CF.3030009@gemini.edu> Message-ID: <04EA582C-BBBC-4D77-A32D-A3B27646E5D2@stanford.edu> Hello all, > By the way, ringing at sharp edges is an intrinsic feature of higher- > order spline interpolation, right? I believe this kind of interpolant > is really intended for smooth (band-limited) data. I'm not sure why > the pre-filtering makes a difference though; I don't yet understand > well enough what the pre-filter actually does. > > I'm not sure what people normally do in computer graphics, since I'm > working more with natural band-limited images, but in the case of > Stefan's "rotate_artifacts" example, wouldn't it be appropriate to > use the 1st- or 0th-order spline instead of the 2nd order? If your > real-life data are smooth enough, however, then in theory the > ringing with higher orders should go away. James is indeed correct. This whole thing is my mistake based on a mis-interpretation of terminology. I've more carefully re-read the paper on which the ndimage spline code is based ('Splines: A Perfect Fit for Signal/Image Processing' by Michael Unser; it's on citeseer). I now (hopefully) understand that the spline "pre-filter", while explicitly analogous to a traditional anti-aliasing pre-filter, is actually computing the spline coefficients via a filtering-type operation. While a traditional anti-aliasing filter should blur an image (as a band-limiting step), the fact that the anti-aliasing pre- filter does not is of no concern since the filtered output isn't a band-limited set of pixels, but a set of coefficients for a band- limited spline. The actual transform operators then use these coefficients to (properly) compute pixel values at different locations. I just assumed that the "pre-filtering" was broken (even on natural images with smooth variations) because images pre-filtered with the spline filter didn't look like traditional pre-filtered images ought. IMO, ticket 213 should be closed as PEBCAK (I'll do that forthwith); further I'm sorry to have caused Peter to be further bothered about this non-issue. Zach On Mar 22, 2007, at 8:13 PM, James Turner wrote: > By the way, ringing at sharp edges is an intrinsic feature of higher- > order spline interpolation, right? I believe this kind of interpolant > is really intended for smooth (band-limited) data. I'm not sure why > the pre-filtering makes a difference though; I don't yet understand > well enough what the pre-filter actually does. > > I'm not sure what people normally do in computer graphics, since I'm > working more with natural band-limited images, but in the case of > Stefan's "rotate_artifacts" example, wouldn't it be appropriate to > use the 1st- or 0th-order spline instead of the 2nd order? If your > real-life data are smooth enough, however, then in theory the > ringing with higher orders should go away. > > Sorry if I'm stating the obvious and missing the real point! I just > wanted to make sure this hasn't been overlooked... Likewise, sorry I > didn't mention this before if it does turn out to be relevant. Let > me know if you want references to explain what I said above. > > James. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From curiousjan at gmail.com Fri Mar 23 02:32:48 2007 From: curiousjan at gmail.com (Jan Strube) Date: Thu, 22 Mar 2007 23:32:48 -0700 Subject: [Numpy-discussion] assignment to array.dtype Message-ID: Hi list, maybe this is a really stupid idea, and I don't want to advertise this, but what actually happens when I reassign the dtype of an object? Does it return the same as array.view? say I have the following code In [64]: b Out[64]: array([[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11], [12, 13, 14]]) In [69]: t = N.dtype(zip(('a', 'b', 'c'), (N.int32, N.int32, N.int32))) In [70]: b.view(t) Out[70]: array([[(0, 1, 2)], [(3, 4, 5)], [(6, 7, 8)], [(9, 10, 11)], [(12, 13, 14)]], dtype=[('a', ' In [75]: b.dtype=t In [76]: b Out[76]: array([[(0, 1, 2)], [(3, 4, 5)], [(6, 7, 8)], [(9, 10, 11)], [(12, 13, 14)]], dtype=[('a', ' From efiring at hawaii.edu Fri Mar 23 02:38:01 2007 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 22 Mar 2007 20:38:01 -1000 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: References: <46031BA2.4020405@bryant.edu> <20070323002005.GC6085@mentat.za.net> Message-ID: <460375C9.8020203@hawaii.edu> Sebastian Haase wrote: > On 3/22/07, Stefan van der Walt wrote: >> On Thu, Mar 22, 2007 at 08:13:22PM -0400, Brian Blais wrote: >>> Hello, >>> >>> I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns >>> (one for each of the original 1D arrays). I thought this would work: >>> >>> >>> In [47]:a=arange(0,10,1) >>> >>> In [48]:a >>> Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> >>> In [49]:b=arange(-10,0,1) >>> >>> In [51]:b >>> Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) >>> >>> In [54]:concatenate((a,b)) >>> Out[54]: >>> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, >>> -7, -6, -5, -4, -3, -2, -1]) >>> >>> In [55]:concatenate((a,b),axis=1) >>> Out[55]: >>> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, >>> -7, -6, -5, -4, -3, -2, -1]) >>> >>> >>> but it never expands the dimensions. Do I have to do this... >>> >>> In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) >>> Out[65]: >>> array([[ 0, -10], >>> [ 1, -9], >>> [ 2, -8], >>> [ 3, -7], >>> [ 4, -6], >>> [ 5, -5], >>> [ 6, -4], >>> [ 7, -3], >>> [ 8, -2], >>> [ 9, -1]]) >>> >>> >>> ? >>> >>> I thought there would be an easier way. Did I overlook something? >> How about >> >> N.vstack((a,b)).T >> > Also mentioned here should be the use of > newaxis. > As in > a[:,newaxis] > > However I never got a "good feel" for how to use it, so I can't > complete the code you would need. n [9]:c = N.concatenate((a[:,N.newaxis], b[:,N.newaxis]), axis=1) In [10]:c Out[10]: array([[ 0, -10], [ 1, -9], [ 2, -8], [ 3, -7], [ 4, -6], [ 5, -5], [ 6, -4], [ 7, -3], [ 8, -2], [ 9, -1]]) > > -Sebastian Haase > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From wbaxter at gmail.com Fri Mar 23 02:53:00 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 23 Mar 2007 15:53:00 +0900 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: <460375C9.8020203@hawaii.edu> References: <46031BA2.4020405@bryant.edu> <20070323002005.GC6085@mentat.za.net> <460375C9.8020203@hawaii.edu> Message-ID: On 3/23/07, Eric Firing wrote: > Sebastian Haase wrote: > > On 3/22/07, Stefan van der Walt wrote: > >> On Thu, Mar 22, 2007 at 08:13:22PM -0400, Brian Blais wrote: > >>> Hello, > >>> > >>> I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns > >>> (one for each of the original 1D arrays). I thought this would work: > >>> > >>> > >>> In [47]:a=arange(0,10,1) > >>> > >>> In [48]:a > >>> Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > >>> > >>> In [49]:b=arange(-10,0,1) > >>> > >>> In [51]:b > >>> Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) > >>> > >>> In [54]:concatenate((a,b)) > >>> Out[54]: > >>> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > >>> -7, -6, -5, -4, -3, -2, -1]) > >>> > >>> In [55]:concatenate((a,b),axis=1) > >>> Out[55]: > >>> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > >>> -7, -6, -5, -4, -3, -2, -1]) > >>> > >>> > >>> but it never expands the dimensions. Do I have to do this... > >>> > >>> In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > >>> Out[65]: > >>> array([[ 0, -10], > >>> [ 1, -9], > >>> [ 2, -8], > >>> [ 3, -7], > >>> [ 4, -6], > >>> [ 5, -5], > >>> [ 6, -4], > >>> [ 7, -3], > >>> [ 8, -2], > >>> [ 9, -1]]) > >>> > >>> > >>> ? > >>> > >>> I thought there would be an easier way. Did I overlook something? > >> How about > >> > >> N.vstack((a,b)).T > >> > > Also mentioned here should be the use of > > newaxis. > > As in > > a[:,newaxis] > > > > However I never got a "good feel" for how to use it, so I can't > > complete the code you would need. > > n [9]:c = N.concatenate((a[:,N.newaxis], b[:,N.newaxis]), axis=1) > > In [10]:c > Out[10]: > array([[ 0, -10], > [ 1, -9], > [ 2, -8], > [ 3, -7], > [ 4, -6], > [ 5, -5], > [ 6, -4], > [ 7, -3], > [ 8, -2], > [ 9, -1]]) > Then of course, there's r_ and c_: c = numpy.c_[a,b] c = numpy.r_[a[None],b[None]].T --bb From jturner at gemini.edu Fri Mar 23 04:28:21 2007 From: jturner at gemini.edu (James Turner) Date: Fri, 23 Mar 2007 04:28:21 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <04EA582C-BBBC-4D77-A32D-A3B27646E5D2@stanford.edu> References: <04EA582C-BBBC-4D77-A32D-A3B27646E5D2@stanford.edu> Message-ID: <46038FA5.10405@gemini.edu> Hi Zach, > further I'm sorry to have caused Peter to be further bothered about > this non-issue. Don't worry -- I have let him know that we've probably figured it out. I hope Stefan agrees. > I now (hopefully) understand that the spline "pre-filter", while > explicitly analogous to a traditional anti-aliasing pre-filter, is > actually computing the spline coefficients via a filtering-type > operation. Ah, sounds like it's just calculating the weights for the specific sample grids. That makes sense. So, if you want a smooth result in the rotate_artifacts example, you probably first want to apply a convolution as an anti-aliasing step using one of the nd_image filter functions (eg. convolve or gaussian_filter), then interpolate as before with the higher-order spline. That ought to get rid of the artefacts if nd_image is indeed doing the right thing. I think you figured out that much already. Although not strictly band limited, I think a Gaussian with a sigma of around 1 pixel should reduce the aliasing to just a couple of percent (I did that calculation before in connection with my astronomy stuff). I just tried adding the following line to the code snippet from ticket 213, before the line beginning "I_rot_1 =" and it looks to me like the artefacts have gone (plays havoc with the colours though!). It's hard to tell the difference between the 1st and 2nd order results, however, even zooming right in. I = ndi.gaussian_filter(I, 1.0, order=0, mode='constant', cval=0.0) Hope I'm making sense; it's 4:20am here. Cheers, James. From stefan at sun.ac.za Fri Mar 23 06:04:03 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Mar 2007 12:04:03 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <04EA582C-BBBC-4D77-A32D-A3B27646E5D2@stanford.edu> References: <460345CF.3030009@gemini.edu> <04EA582C-BBBC-4D77-A32D-A3B27646E5D2@stanford.edu> Message-ID: <20070323100403.GI6085@mentat.za.net> On Thu, Mar 22, 2007 at 11:20:37PM -0700, Zachary Pincus wrote: > The actual transform operators then use these coefficients to > (properly) compute pixel values at different locations. I just > assumed that the "pre-filtering" was broken (even on natural images > with smooth variations) because images pre-filtered with the spline > filter didn't look like traditional pre-filtered images ought. > > IMO, ticket 213 should be closed as PEBCAK (I'll do that forthwith); > further I'm sorry to have caused Peter to be further bothered about > this non-issue. The ticket is more concerned with the usability of ndimage. Currently, the principle of least surprise is violated. If you use the default arguments of ndimage.rotate (except for axes, which should still be fixed to be (1,0) by default) and rotate Lena by 30 degrees, you get something with green splotches on (attached to the ticket). IMO, we should either change the default parameters or update the documentation before closing the ticket. Cheers St?fan From jturner at gemini.edu Fri Mar 23 07:36:44 2007 From: jturner at gemini.edu (James Turner) Date: Fri, 23 Mar 2007 07:36:44 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <46031261.4020607@ee.byu.edu> References: <46031261.4020607@ee.byu.edu> Message-ID: <4603BBCC.3050008@gemini.edu> Thanks for the explanation, Travis. It now looks like there are 2 distinct issues getting mixed up here, however. First, there is the issue of the mirror symmetry of the spline algorithm affecting the data towards the edges, as described by Peter and Travis (this is probably also what Anne is referring to). It is certainly useful to know about this, but I think this effect *within* the data boundaries must be relatively subtle compared with what I'm noticing. Second, there is a reflection 1 pixel *outside* the bounds of the original data, instead of the blank value I would expect. I think the code is just calculating one too many rows. I talked to Peter about this a bit more and he now agrees that it might be a real bug. I'll attach the most relevant part of his email below, along with my question (quoted). This doesn't solve the problem of fixing the code, of course, but I think it's useful to agree what the behaviour should be. Cheers, James. ----- > Actually, just going back to your last email, it looks like the mirrored > values in the output are *outside* the bounds of the input data (by 1 > pixel), not just close enough for the interpolation window to overlap the > edge. In the example from my last email, the value of 1.899 in the 5th > column corresponds to a data point beyond the final 1.0 of the input data. You are right, that values is actually interpolating from a point just outside the boundary of the input. Probably that should have been -1, so that you could call a bug. Why it calculates that value (and 2 for order=0) I am not sure... From lxander.m at gmail.com Fri Mar 23 09:33:23 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 23 Mar 2007 09:33:23 -0400 Subject: [Numpy-discussion] Array of Arrays Message-ID: <525f23e80703230633o112d0374j24e4103b6e176ecf@mail.gmail.com> How can I do something like the following? a = empty((5,7), dtype=<4 element array of floats>) c = a[:,-1] # last column of 4 element arrays a[0,0] = 2.0 print a[0,0] [2. 2. 2. 2.] a[1,0] = 3.0 a[0,1] = a[0,0] * a[1,0] print a[0,1] [6. 6. 6. 6.] etc. As always, thanks for the help! Alex From david.huard at gmail.com Fri Mar 23 10:18:04 2007 From: david.huard at gmail.com (David Huard) Date: Fri, 23 Mar 2007 10:18:04 -0400 Subject: [Numpy-discussion] histogramdd shapes In-Reply-To: References: Message-ID: <91cf711d0703230718r431ff848i98249fd40a6b0f16@mail.gmail.com> Thanks Ben, I submitted a patch with your fix. Ticket 189 David 2007/3/17, Ben Granett : > > Hi, > There seems to be a problem with histogramdd (numpy 1.0.1). Depending on > the > number of bins in each axis, it may not produce an array with the > dimensions > oriented correctly. The bug is in the axis swapping code at the end of > the > routine. > > To reproduce, you can run, > >>> x = random.randn(100,3) > >>> H, edg = histogramdd(x, bins=(7,5,6)) > >>> print H.shape > (5, 7, 6) > > > > A fix is to replace this: > # Shape into a proper matrix > hist = hist.reshape(sort(nbin)) > for i,j in enumerate(ni): > hist = hist.swapaxes(i,j) > if (hist.shape == nbin).all(): > break > > > with this: > for i in arange(nbin.size): > j = ni[i] > hist = hist.swapaxes(i,j) > ni[i],ni[j] = ni[j],ni[i] > > > That is, elements of ni need to be swapped too. > > I hope this wasn't a known bug, but I couldn't find any mention of > it. Thanks, > Ben > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Fri Mar 23 10:20:51 2007 From: nadavh at visionsense.com (Nadav Horesh) Date: Fri, 23 Mar 2007 16:20:51 +0200 Subject: [Numpy-discussion] Array of Arrays Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8F1FE@exchange2k.envision.co.il> How about a = empty((5,7,4)) c = a[...,-1] . . . Nadav -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Alexander Michael Sent: Fri 23-Mar-07 15:33 To: Discussion of Numerical Python Cc: Subject: [Numpy-discussion] Array of Arrays How can I do something like the following? a = empty((5,7), dtype=<4 element array of floats>) c = a[:,-1] # last column of 4 element arrays a[0,0] = 2.0 print a[0,0] [2. 2. 2. 2.] a[1,0] = 3.0 a[0,1] = a[0,0] * a[1,0] print a[0,1] [6. 6. 6. 6.] etc. As always, thanks for the help! Alex _______________________________________________ Numpy-discussion mailing list Numpy-discussion at scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 2727 bytes Desc: not available URL: From ogdude at googlemail.com Fri Mar 23 10:28:20 2007 From: ogdude at googlemail.com (David Koch) Date: Fri, 23 Mar 2007 15:28:20 +0100 Subject: [Numpy-discussion] Adapting algorithm to accept Scipy sparse matrix In-Reply-To: References: <3C07E2F3-6FEC-4784-935C-3E0B8EF5D8D0@arachnedesign.net> Message-ID: Hi, Ok, I got it to work now but - damn, it's ugly. I thought I'd have to watch the differences between ndarray and matrix type but it turns out sparseMatrix is yet again different from matrix in several respects when it comes to certain operations. Is this intended or something that will be mended as sparse stuff becomes more mature? Concrete examples for matrix/vector multiplication: testMat = asmatrix(around(10 * random.randn(3,2))) testSp = sparse.lil_matrix(testMat) testMat * array([1, 2]) # matrix([[-24., -26., -26.]]) testMat * array([[1, 2]]) # ValueError: matrices are not aligned testMat * array([[1, 2]]).transpose() # matrix([[-24.], # [-26.], # [-26.]]) ... and "dot" has the same effect as "*". Basically mat * rank1 array = row matrix and mat * rank2 array = column matrix For sparse matrices on the other hand the same operations: testSp * array([1, 2]) # array([-24., -26., -26.]) testSp * array([[1, 2]]) # array([-24., -26., -26.]) testSp * array([[1, 2]]).transpose() # array([-24., -26., -26.]) ... and "dot" != "*" (don't know what the former does exactly). So unlike with regular matrices, "*" operations return a (dense) rank1 array regardless of the shape/rank of the vector. I find this confusing and it makes for very ugly code, since a lot of ndim, shape checking is involved if a routine is supposed to accept array AND matrix as input. Any comments or hints? /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Fri Mar 23 10:48:14 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 23 Mar 2007 07:48:14 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070323014744.GE6085@mentat.za.net> References: <45FEDF10.3010707@gemini.edu> <4602CC4E.3060102@gemini.edu> <20070322223518.GA6085@mentat.za.net> <46031261.4020607@ee.byu.edu> <20070323014744.GE6085@mentat.za.net> Message-ID: <4603E8AE.4020701@ee.byu.edu> Stefan van der Walt wrote: >On Thu, Mar 22, 2007 at 04:33:53PM -0700, Travis Oliphant wrote: > > >>>I would rather opt for changing the spline fitting algorithm than for >>>padding with zeros. >>> >>> >>> >>> >> From what I understand, the splines used in ndimage have the implicit >>mirror-symmetric boundary condition which also allows them to be >>computed rapidly. There may be ways to adapt other boundary conditions >>and maintain rapid evaluation, but it is not trivial as far as I know. >>Standard spline-fitting allows multiple boundary conditions because >>matrix inversion is used. I think the spline-fitting done in ndimage >>relies on equal-spacing and mirror-symmetry to allow simple IIR filters >>to be used to compute the spline coefficients very rapidly. >> >> > >Thanks, Travis. I wasn't aware of these restrictions. > >Would it be possible to call fitpack to do the spline fitting? I >noticed that it doesn't exhibit the same mirror-property: > > It doesn't. It uses standard spline-fitting techniques (which are also slower). So, yes you could call fitpack but for a large multi-dimensional array it's going to take longer. -Travis From oliphant at ee.byu.edu Fri Mar 23 10:56:43 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 23 Mar 2007 07:56:43 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <460345CF.3030009@gemini.edu> References: <460345CF.3030009@gemini.edu> Message-ID: <4603EAAB.30308@ee.byu.edu> James Turner wrote: >By the way, ringing at sharp edges is an intrinsic feature of higher- >order spline interpolation, right? I believe this kind of interpolant >is really intended for smooth (band-limited) data. I'm not sure why >the pre-filtering makes a difference though; I don't yet understand >well enough what the pre-filter actually does. > > Yes, ringing at edges is an intrinsic feature of higher-order spline interpolation. Eventually, the spline interpolant becomes a band-limited sinc-interpolator which will assume that edges are really points sampled from a sinc. So, if you re-sample at different points you get the "ringing" effect. But, you shouldn't see a lot of ringing below order 7. The pre-filter obtains the spline-interpolation coefficients. The spline assumes the continuous function represented by the samples at x_i is f(x,y) = sum(c_ij beta^o(x-x_i) beta^o(y-y_i)) The "pre-filter" is computing the coefficients c_ij. You then evaluate at any point you like using the continuous function implied by the spline. The function beta^o is a spline function and depends on the order of the spline. >I'm not sure what people normally do in computer graphics, since I'm >working more with natural band-limited images, but in the case of >Stefan's "rotate_artifacts" example, wouldn't it be appropriate to >use the 1st- or 0th-order spline instead of the 2nd order? If your >real-life data are smooth enough, however, then in theory the >ringing with higher orders should go away. > > Yes, that is true. But, you really shouldn't see much ringing with a 3rd order spline, though. I studied these splines for a while, but my memory can fail me. You can look at the papers of M. Unser for much more information. Here is a link to a good review paper. http://bigwww.epfl.ch/publications/unser0001.pdf -Travis From rpyle at post.harvard.edu Fri Mar 23 11:09:03 2007 From: rpyle at post.harvard.edu (Robert Pyle) Date: Fri, 23 Mar 2007 11:09:03 -0400 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: <46031BA2.4020405@bryant.edu> References: <46031BA2.4020405@bryant.edu> Message-ID: <48E6CA1B-7379-4AAF-80C9-E7781EC24298@post.harvard.edu> On Mar 22, 2007, at 8:13 PM, Brian Blais wrote: > Hello, > > I'd like to concatenate a couple of 1D arrays to make it a 2D > array, with two columns > (one for each of the original 1D arrays). I thought this would work: > > > In [47]:a=arange(0,10,1) > > In [48]:a > Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > In [49]:b=arange(-10,0,1) > > In [51]:b > Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) > > In [54]:concatenate((a,b)) > Out[54]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, > -8, > -7, -6, -5, -4, -3, -2, -1]) > > In [55]:concatenate((a,b),axis=1) > Out[55]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, > -8, > -7, -6, -5, -4, -3, -2, -1]) > > > but it never expands the dimensions. Do I have to do this... > > In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > Out[65]: > array([[ 0, -10], > [ 1, -9], > [ 2, -8], > [ 3, -7], > [ 4, -6], > [ 5, -5], > [ 6, -4], > [ 7, -3], > [ 8, -2], > [ 9, -1]]) > > > ? > > I thought there would be an easier way. Did I overlook something? What's wrong with zip? Or did *I* miss the point? (I'm just getting the hang of numpy.) ~ $ python Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * >>> a=arange(0,10,1) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> b=arange(-10,0,1) >>> b array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) >>> array(zip(a,b)) array([[ 0, -10], [ 1, -9], [ 2, -8], [ 3, -7], [ 4, -6], [ 5, -5], [ 6, -4], [ 7, -3], [ 8, -2], [ 9, -1]]) >>> Bob Pyle From lxander.m at gmail.com Fri Mar 23 11:22:09 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 23 Mar 2007 11:22:09 -0400 Subject: [Numpy-discussion] Array of Arrays In-Reply-To: <07C6A61102C94148B8104D42DE95F7E8C8F1FE@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E8C8F1FE@exchange2k.envision.co.il> Message-ID: <525f23e80703230822g40093b38g4cbd6940499736b9@mail.gmail.com> On 3/23/07, Nadav Horesh wrote: > How about > > a = empty((5,7,4)) > c = a[...,-1] Solely because I want to use the array with code that assumes it is working with two-dimensional arrays but yet only performs operations on the "outer" two-dimensional array that would be consistent with an "inner" array type (i.e. scalar assignment, element-wise multiplication, etc.). I own all the code, so perhaps I can replace a[mask,-1] with a[mask,-1,...] and such. Hmm. Not bad reminder, thanks. From seb.haase at gmx.net Fri Mar 23 11:24:40 2007 From: seb.haase at gmx.net (Sebastian Haase) Date: Fri, 23 Mar 2007 08:24:40 -0700 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: References: <46031BA2.4020405@bryant.edu> <20070323002005.GC6085@mentat.za.net> <460375C9.8020203@hawaii.edu> Message-ID: On 3/22/07, Bill Baxter wrote: > On 3/23/07, Eric Firing wrote: > > Sebastian Haase wrote: > > > On 3/22/07, Stefan van der Walt wrote: > > >> On Thu, Mar 22, 2007 at 08:13:22PM -0400, Brian Blais wrote: > > >>> Hello, > > >>> > > >>> I'd like to concatenate a couple of 1D arrays to make it a 2D array, with two columns > > >>> (one for each of the original 1D arrays). I thought this would work: > > >>> > > >>> > > >>> In [47]:a=arange(0,10,1) > > >>> > > >>> In [48]:a > > >>> Out[48]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > >>> > > >>> In [49]:b=arange(-10,0,1) > > >>> > > >>> In [51]:b > > >>> Out[51]:array([-10, -9, -8, -7, -6, -5, -4, -3, -2, -1]) > > >>> > > >>> In [54]:concatenate((a,b)) > > >>> Out[54]: > > >>> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > > >>> -7, -6, -5, -4, -3, -2, -1]) > > >>> > > >>> In [55]:concatenate((a,b),axis=1) > > >>> Out[55]: > > >>> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -10, -9, -8, > > >>> -7, -6, -5, -4, -3, -2, -1]) > > >>> > > >>> > > >>> but it never expands the dimensions. Do I have to do this... > > >>> > > >>> In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > > >>> Out[65]: > > >>> array([[ 0, -10], > > >>> [ 1, -9], > > >>> [ 2, -8], > > >>> [ 3, -7], > > >>> [ 4, -6], > > >>> [ 5, -5], > > >>> [ 6, -4], > > >>> [ 7, -3], > > >>> [ 8, -2], > > >>> [ 9, -1]]) > > >>> > > >>> > > >>> ? > > >>> > > >>> I thought there would be an easier way. Did I overlook something? > > >> How about > > >> > > >> N.vstack((a,b)).T > > >> > > > Also mentioned here should be the use of > > > newaxis. > > > As in > > > a[:,newaxis] > > > > > > However I never got a "good feel" for how to use it, so I can't > > > complete the code you would need. > > > > n [9]:c = N.concatenate((a[:,N.newaxis], b[:,N.newaxis]), axis=1) > > > > In [10]:c > > Out[10]: > > array([[ 0, -10], > > [ 1, -9], > > [ 2, -8], > > [ 3, -7], > > [ 4, -6], > > [ 5, -5], > > [ 6, -4], > > [ 7, -3], > > [ 8, -2], > > [ 9, -1]]) > > > > Then of course, there's r_ and c_: > > c = numpy.c_[a,b] > > c = numpy.r_[a[None],b[None]].T > > --bb So, None is the same as newaxis - right? But what is a[None] vs. a[:,N.newaxis] ? -Sebastian From zpincus at stanford.edu Fri Mar 23 12:20:09 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri, 23 Mar 2007 09:20:09 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070323100403.GI6085@mentat.za.net> References: <460345CF.3030009@gemini.edu> <04EA582C-BBBC-4D77-A32D-A3B27646E5D2@stanford.edu> <20070323100403.GI6085@mentat.za.net> Message-ID: Hello all, On Mar 23, 2007, at 3:04 AM, Stefan van der Walt wrote: > On Thu, Mar 22, 2007 at 11:20:37PM -0700, Zachary Pincus wrote: >> The actual transform operators then use these coefficients to >> (properly) compute pixel values at different locations. I just >> assumed that the "pre-filtering" was broken (even on natural images >> with smooth variations) because images pre-filtered with the spline >> filter didn't look like traditional pre-filtered images ought. >> >> IMO, ticket 213 should be closed as PEBCAK (I'll do that forthwith); >> further I'm sorry to have caused Peter to be further bothered about >> this non-issue. > > The ticket is more concerned with the usability of ndimage. > Currently, the principle of least surprise is violated. If you use > the default arguments of ndimage.rotate (except for axes, which should > still be fixed to be (1,0) by default) and rotate Lena by 30 degrees, > you get something with green splotches on (attached to the ticket). > > IMO, we should either change the default parameters or update the > documentation before closing the ticket. Hmm, this is worrisome. There really shouldn't be ringing on continuous-tone images like Lena -- right? (And at no step in an image like that should gaussian filtering be necessary if you're doing spline interpolation -- also right?) I assumed the problem was a non-issue after some testing with a few natural images didn't introduce any artifacts like those, but now with the Lena example I'm not sure. This is frustrating, to be sure, mostly because of my limited knowledge on the subject -- just enough to be dangerous. Zach From stefan at sun.ac.za Fri Mar 23 12:23:53 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 23 Mar 2007 18:23:53 +0200 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: <48E6CA1B-7379-4AAF-80C9-E7781EC24298@post.harvard.edu> References: <46031BA2.4020405@bryant.edu> <48E6CA1B-7379-4AAF-80C9-E7781EC24298@post.harvard.edu> Message-ID: <20070323162353.GR6085@mentat.za.net> On Fri, Mar 23, 2007 at 11:09:03AM -0400, Robert Pyle wrote: > > In [65]:concatenate((a.reshape(10,1),b.reshape(10,1)),axis=1) > > Out[65]: > > array([[ 0, -10], > > [ 1, -9], > > [ 2, -8], > > [ 3, -7], > > [ 4, -6], > > [ 5, -5], > > [ 6, -4], > > [ 7, -3], > > [ 8, -2], > > [ 9, -1]]) > > > > > > ? > > > > I thought there would be an easier way. Did I overlook something? > > What's wrong with zip? Or did *I* miss the point? (I'm just getting > the hang of numpy.) If you use 'zip' you don't make use of numpy's fast array mechanisms. I attach some code you can run as a benchmark. From my ipython session: In [1]: run vsbench.py In [2]: timeit using_vstack(x,y) 1000 loops, best of 3: 997 ?s per loop In [3]: timeit using_zip(x,y) 10 loops, best of 3: 503 ms per loop In [4]: timeit using_custom_iteration(x,y) 1000 loops, best of 3: 1.64 ms per loop Cheers St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: vsbench.py Type: text/x-python Size: 492 bytes Desc: not available URL: From oliphant at ee.byu.edu Fri Mar 23 12:46:36 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 23 Mar 2007 09:46:36 -0700 Subject: [Numpy-discussion] assignment to array.dtype In-Reply-To: References: Message-ID: <4604046C.4020205@ee.byu.edu> Jan Strube wrote: > Hi list, > maybe this is a really stupid idea, and I don't want to advertise > this, but what actually happens when I reassign the dtype of an object? > Does it return the same as array.view? Yes, it is the same. In fact, IIRC it uses very similar code. -Travis From zpincus at stanford.edu Fri Mar 23 12:28:15 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri, 23 Mar 2007 09:28:15 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <4603EAAB.30308@ee.byu.edu> References: <460345CF.3030009@gemini.edu> <4603EAAB.30308@ee.byu.edu> Message-ID: <280D5248-0596-4162-A68A-E7D164D982BF@stanford.edu> Hello again, On Mar 23, 2007, at 7:56 AM, Travis Oliphant wrote: >> I'm not sure what people normally do in computer graphics, since I'm >> working more with natural band-limited images, but in the case of >> Stefan's "rotate_artifacts" example, wouldn't it be appropriate to >> use the 1st- or 0th-order spline instead of the 2nd order? If your >> real-life data are smooth enough, however, then in theory the >> ringing with higher orders should go away. >> >> > Yes, that is true. But, you really shouldn't see much ringing with a > 3rd order spline, though. There are a couple of different examples of ringing that we've seen (specifically, the first and third attachments to scipy ticket 213: http://projects.scipy.org/scipy/scipy/ticket/213 The first was on Stefan's artificial data which had sharp edges, and got very nasty ringing artifacts even with 3rd order splines. From your recollection, is this expected behavior based on splines and the nature of Stefan's image, or more likely to be a bug? The second example is rotated version of Lena -- an image with natural and much smoother edges -- where 3rd order spline interpolation still introduced strong ringing (or something nasty). This looks more like a bug, but (as per this discussion) my insight is clearly limited in this case. Zach From wbaxter at gmail.com Fri Mar 23 14:57:09 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 24 Mar 2007 03:57:09 +0900 Subject: [Numpy-discussion] concatenating 1-D arrays to 2D In-Reply-To: References: <46031BA2.4020405@bryant.edu> <20070323002005.GC6085@mentat.za.net> <460375C9.8020203@hawaii.edu> Message-ID: On 3/24/07, Sebastian Haase wrote: > > > > Then of course, there's r_ and c_: > > > > c = numpy.c_[a,b] > > > > c = numpy.r_[a[None],b[None]].T > > > > --bb > So, > None is the same as newaxis - right? Yes, newaxis is None. None is newaxis. Same thing. I just don't see much advantage in spelling it numpy.newaxis, since it's pretty common and not likely to ever change. > But what is a[None] vs. a[:,N.newaxis] ? a[None] is the same as a[None,:]. It prepends the new axis, so a shape of (5,) becomes (1,5), a "row vector" a[:,None] puts the new axis after the first axis, so shape of (5,) becomes (5,1) a "column vector" a[None,:,None] puts a new axis both before and after, so (5,) becomes (1,5,1). If 'a' is higher dimensional, the same kind of thing goes. Wherever None/newaxis appears in the index, insert a new axis there in the result. Say A is shape (2,3,4), then A[:,None,:,None] is shape (2,1,3,1,4). (Same as A[:,None,:,None,:] since unspecified trailing indices are always taken to be ':') --bb From berthold at xn--hllmanns-n4a.de Fri Mar 23 15:26:15 2007 From: berthold at xn--hllmanns-n4a.de (=?utf-8?q?Berthold_H=C3=B6llmann?=) Date: Fri, 23 Mar 2007 20:26:15 +0100 Subject: [Numpy-discussion] Array of Arrays In-Reply-To: <525f23e80703230633o112d0374j24e4103b6e176ecf@mail.gmail.com> (Alexander Michael's message of "Fri, 23 Mar 2007 09:33:23 -0400") References: <525f23e80703230633o112d0374j24e4103b6e176ecf@mail.gmail.com> Message-ID: "Alexander Michael" writes: > How can I do something like the following? > > a = empty((5,7), dtype=<4 element array of floats>) > > c = a[:,-1] # last column of 4 element arrays > > a[0,0] = 2.0 > print a[0,0] > [2. 2. 2. 2.] > > a[1,0] = 3.0 > a[0,1] = a[0,0] * a[1,0] > > print a[0,1] > [6. 6. 6. 6.] The nearest I could come with is:: .>>> import numpy .>>> a = numpy.zeros((5,7), numpy.dtype('object')) .>>> a[:,:]=None .>>> class O: .... def __init__(self, val): .... self.val = val .... def __mul__(self, other): .... return self.__class__(self.val*other.val) .... def __str__(self): .... return "%s" % (numpy.ones(4, numpy.dtype('float')) * self.val) .... .>>> a[0,0] = O(2.0) .>>> print a[0,0] [ 2. 2. 2. 2.] .>>> a[1,0] = O(3.0) .>>> a[0,1] = a[0,0] * a[1,0] .>>> print a[0,1] [ 6. 6. 6. 6.] Regards -- From haase at msg.ucsf.edu Fri Mar 23 17:03:25 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 23 Mar 2007 14:03:25 -0700 Subject: [Numpy-discussion] swig svn commits Message-ID: Hi, this is regarding the svn commit by wfspotz. Author: wfspotz at sandia.gov Date: 2007-03-23 13:04:37 -0500 (Fri, 23 Mar 2007) New Revision: 3593 Modified: trunk/numpy/doc/swig/numpy.i Log: Added typecheck typemaps to INPLACE_ARRAY typemap suites Hi wfspotz, I was just wondering if you consider checking for "native byte order" as part of your inplace-typemap. I found that to be a problem in my SWIG type maps that I hi-jacked / boiled down from the older numpy swig files. (They are mostly for 3D image data, only for a small number of types) -Sebastian From haase at msg.ucsf.edu Fri Mar 23 17:18:25 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri, 23 Mar 2007 14:18:25 -0700 Subject: [Numpy-discussion] Array of Arrays In-Reply-To: <525f23e80703230822g40093b38g4cbd6940499736b9@mail.gmail.com> References: <07C6A61102C94148B8104D42DE95F7E8C8F1FE@exchange2k.envision.co.il> <525f23e80703230822g40093b38g4cbd6940499736b9@mail.gmail.com> Message-ID: On 3/23/07, Alexander Michael wrote: > On 3/23/07, Nadav Horesh wrote: > > How about > > > > a = empty((5,7,4)) > > c = a[...,-1] > > Solely because I want to use the array with code that assumes it is > working with two-dimensional arrays but yet only performs operations > on the "outer" two-dimensional array that would be consistent with an > "inner" array type (i.e. scalar assignment, element-wise > multiplication, etc.). I own all the code, so perhaps I can replace > a[mask,-1] with a[mask,-1,...] and such. Hmm. Not bad reminder, > thanks. Hold on -- aren't the "..." at the *end* always implicit: I you have a.shape = (6,5,4,3) a[3,2] is the same as a[3,2,:,:] is the same as a[3,2,...] only if you wanted a[...,3,2] you would have to change your code !? Am I confused !? -Sebastian From lxander.m at gmail.com Fri Mar 23 17:26:43 2007 From: lxander.m at gmail.com (Alexander Michael) Date: Fri, 23 Mar 2007 17:26:43 -0400 Subject: [Numpy-discussion] Array of Arrays In-Reply-To: References: <07C6A61102C94148B8104D42DE95F7E8C8F1FE@exchange2k.envision.co.il> <525f23e80703230822g40093b38g4cbd6940499736b9@mail.gmail.com> Message-ID: <525f23e80703231426x79901536wac099621959540d0@mail.gmail.com> On 3/23/07, Sebastian Haase wrote: > Hold on -- aren't the "..." at the *end* always implicit: > I you have > a.shape = (6,5,4,3) > a[3,2] is the same as a[3,2,:,:] is the same as a[3,2,...] > > only if you wanted a[...,3,2] you would have to change your code !? > Am I confused !? Ha! I knew numpy had some trick up its sleeve! I missed that part from the numpy book (still digesting it all (the recycled paper doesn't go down so well ;))), but it says it right there: "If the number of objects in the selection tuple is less than N, then ':' is assumed for any remaining dimensions." I think that is exactly what I am looking for-- thanks for making me aware of it! Alex From charlesr.harris at gmail.com Fri Mar 23 22:22:09 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 23 Mar 2007 20:22:09 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray Message-ID: Anyone, What is the easiest way to detect in python/C if an object is a subclass of ndarray? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sat Mar 24 00:09:50 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 24 Mar 2007 13:09:50 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() Message-ID: I mentioned in another thread Travis started on the scipy list that I would find it useful if there were a function like dot() that could multiply more than just two things. Here's a sample implementation called 'mdot'. mdot(a,b,c,d) ==> dot(dot(dot(a,b),c),d) mdot(a,(b,c),d) ==> dot(dot(a,dot(b,c),d) mdot(a,(b,(c,d))) ==> dot(a,dot(b,dot(c,d)) --- def mdot(*args): """Multiply all the arguments using matrix product rules. The output is equivalent to multiplying the arguments one by one from left to right using dot(). Precedence can be controlled by creating tuples of arguments, for instance mdot(a,((b,c),d)) multiplies a (a*((b*c)*d)). Note that this means the output of dot(a,b) and mdot(a,b) will differ if a or b is a pure tuple of numbers. """ if len(args)==1: return args[0] elif len(args)==2: return _mdot_r(args[0],args[1]) else: return _mdot_r(args[:-1],args[-1]) def _mdot_r(a,b): """Recursive helper for mdot""" if type(a)==types.TupleType: if len(a)>1: a = mdot(*a) else: a = a[0] if type(b)==types.TupleType: if len(b)>1: b = mdot(*b) else: b = b[0] return numpy.dot(a,b) --- It does about twice as many function calls as just using dot which I'm guessing is mostly why its a good bit slower than just doing the nested dot calls. Maybe one of you smart folks can figure out a way to bring it closer to dot's speed. --bb From peridot.faceted at gmail.com Sat Mar 24 00:35:07 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 24 Mar 2007 00:35:07 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: On 24/03/07, Bill Baxter wrote: > I mentioned in another thread Travis started on the scipy list that I > would find it useful if there were a function like dot() that could > multiply more than just two things. > > Here's a sample implementation called 'mdot'. > > mdot(a,b,c,d) ==> dot(dot(dot(a,b),c),d) > mdot(a,(b,c),d) ==> dot(dot(a,dot(b,c),d) > mdot(a,(b,(c,d))) ==> dot(a,dot(b,dot(c,d)) > > --- > def mdot(*args): > """Multiply all the arguments using matrix product rules. > The output is equivalent to multiplying the arguments one by one > from left to right using dot(). > Precedence can be controlled by creating tuples of arguments, > for instance mdot(a,((b,c),d)) multiplies a (a*((b*c)*d)). > Note that this means the output of dot(a,b) and mdot(a,b) will differ if > a or b is a pure tuple of numbers. > """ > if len(args)==1: > return args[0] > elif len(args)==2: > return _mdot_r(args[0],args[1]) > else: > return _mdot_r(args[:-1],args[-1]) > > def _mdot_r(a,b): > """Recursive helper for mdot""" > if type(a)==types.TupleType: > if len(a)>1: > a = mdot(*a) > else: > a = a[0] > if type(b)==types.TupleType: > if len(b)>1: > b = mdot(*b) > else: > b = b[0] > return numpy.dot(a,b) You can do better: In [1]: from numpy import * In [2]: a = array([[0,-1],[1,0]]) In [3]: reduce(dot,(a,a,a,a)) Out[3]: array([[1, 0], [0, 1]]) In [4]: def mdot(*args): ...: return reduce(dot,args) ...: In [5]: mdot(a,a,a,a) Out[5]: array([[1, 0], [0, 1]]) Not every two-line Python function has to come pre-written --Tim Peters Anne P.S. reduce isn't even a numpy thing, it's one of python's much-neglected lispy functions. From peridot.faceted at gmail.com Sat Mar 24 00:37:24 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 24 Mar 2007 00:37:24 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 23/03/07, Charles R Harris wrote: > Anyone, > > What is the easiest way to detect in python/C if an object is a subclass of > ndarray? Um, how about isinstance or issubclass? (if you want strictness you can look at whether x.__class__ is zeros(1).__class__) Anne From charlesr.harris at gmail.com Sat Mar 24 01:39:35 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 23 Mar 2007 23:39:35 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 3/23/07, Anne Archibald wrote: > > On 23/03/07, Charles R Harris wrote: > > Anyone, > > > > What is the easiest way to detect in python/C if an object is a subclass > of > > ndarray? > > Um, how about isinstance or issubclass? (if you want strictness you > can look at whether x.__class__ is zeros(1).__class__) What I am trying to do is figure out the easiest way to fix up matrix multiplies so they work properly. For instance, the following gives the wrong result: In [15]: I = matrix(eye(2)) In [16]: I*ones(2) Out[16]: matrix([[ 1., 1.]]) where the output should be a column vector. In order to do that, I need to reshape the rh multiplicand before __mul__ calls the dot function. There is also this problem In [17]: I*[[1],[1]] --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/charris/ TypeError: can't multiply sequence by non-int A similar multiplication on the left works just fine. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jturner at gemini.edu Sat Mar 24 01:41:21 2007 From: jturner at gemini.edu (James Turner) Date: Sat, 24 Mar 2007 01:41:21 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: References: Message-ID: <4604BA01.5040606@gemini.edu> Hi Zach, > Hmm, this is worrisome. There really shouldn't be ringing on > continuous-tone images like Lena -- right? (And at no step in an > image like that should gaussian filtering be necessary if you're > doing spline interpolation -- also right?) That's hard to say. Just because it's mainly a continuous-tone image doesn't necessarily mean it is well sampled everywhere. This depends both on the subject and the camera optics. Unlike the data I usually work with, I think everyday digital photographs (probably a photo scan in the case of Lena) do not generally have the detector sampling frequency matched to the optical resolution of the image. If that's true, the presence of aliasing in interpolated images depends on the structure of the subject and whether the scene has edges or high- frequency patterns in it. Stefan's rotated Lena example is indeed a bit bizarre on zooming in! However, the artefacts are clearly localized to distinct edges, so I suspect this is indeed some kind of aliasing. Moreover, it looks like Lena has been decimated (reduced in size) prior to the rotation. That is definitely a good way to get artefacts, unless an anti-aliasing filter is applied before shrinking the image. My impression is that this image is probably somewhat undersampled (to understand exactly what that means, read up on the Sampling Theorem). I can't say that the green blobs are *not* caused by a flaw in the algorithm. In particular, I am not used to working with colour images of this kind, so I don't have a good feeling for what aliasing looks like in a case like this. However, I definitely would not rule out intrinsic aliasing effects as the cause of this problem without investigating it further. One experiment might be to blur the original Lena with a Gaussian whose sigma is 1 pixel of the shrunken image before actually shrinking her, then do the rotation. > The first was on Stefan's artificial data which had sharp edges, and > got very nasty ringing artifacts even with 3rd order splines. From > your recollection, is this expected behavior based on splines and the > nature of Stefan's image, or more likely to be a bug? Your question was aimed at Travis, so I don't want to discourage him from answering it :-), but looking at this in more detail, I do think the amplitude of the artefacts here is greater than I might expect due to ringing with a quadratic b-spline kernel, which I think has minima with amplitudes <10% of the central peak. There has to be SOME oscillation, but in Stefan's "rotate_artifacts" example it seems to be at the level of ~100%. Also, it is not present on one of the inner edges for some reason. So I do wonder if the algorithm in nd_image is making this worse than it needs to be. These thoughts seem consistent with Travis's comments. Is it possible to transform the same data using the fitpack routines that Stefan mentioned in post 026641 and compare the results? I just tried doing a similar rotation in PyRAF on a monochrome image with a bicubic spline, and see considerably smaller artefacts (just a compact overshoot of probably a few % at the edge). Cheers, James. From wbaxter at gmail.com Sat Mar 24 03:25:47 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 24 Mar 2007 16:25:47 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: On 3/24/07, Anne Archibald wrote: > On 24/03/07, Bill Baxter wrote: > > I mentioned in another thread Travis started on the scipy list that I > > would find it useful if there were a function like dot() that could > > multiply more than just two things. > > > > Here's a sample implementation called 'mdot'. > > > > mdot(a,b,c,d) ==> dot(dot(dot(a,b),c),d) > > mdot(a,(b,c),d) ==> dot(dot(a,dot(b,c),d) > > mdot(a,(b,(c,d))) ==> dot(a,dot(b,dot(c,d)) > > > > --- > > def mdot(*args): > > """Multiply all the arguments using matrix product rules. > > The output is equivalent to multiplying the arguments one by one > > from left to right using dot(). > > Precedence can be controlled by creating tuples of arguments, > > for instance mdot(a,((b,c),d)) multiplies a (a*((b*c)*d)). > > Note that this means the output of dot(a,b) and mdot(a,b) will differ if > > a or b is a pure tuple of numbers. > > """ > > if len(args)==1: > > return args[0] > > elif len(args)==2: > > return _mdot_r(args[0],args[1]) > > else: > > return _mdot_r(args[:-1],args[-1]) > > > > def _mdot_r(a,b): > > """Recursive helper for mdot""" > > if type(a)==types.TupleType: > > if len(a)>1: > > a = mdot(*a) > > else: > > a = a[0] > > if type(b)==types.TupleType: > > if len(b)>1: > > b = mdot(*b) > > else: > > b = b[0] > > return numpy.dot(a,b) > > You can do better: > In [1]: from numpy import * > > In [2]: a = array([[0,-1],[1,0]]) > > In [3]: reduce(dot,(a,a,a,a)) > Out[3]: > array([[1, 0], > [0, 1]]) > > In [4]: def mdot(*args): > ...: return reduce(dot,args) > ...: > > In [5]: mdot(a,a,a,a) > Out[5]: > array([[1, 0], > [0, 1]]) > Nice, but how does that fare on things like mdot(a,(b,c),d) ? I'm pretty sure it doesn't handle it. I think an mdot that can only multiply things left to right comes up short compared to an infix operator that can easily use parentheses to control order. --bb From wbaxter at gmail.com Sat Mar 24 04:54:49 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 24 Mar 2007 17:54:49 +0900 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 3/24/07, Charles R Harris wrote: > > > On 3/23/07, Anne Archibald wrote: > > On 23/03/07, Charles R Harris wrote: > > > Anyone, > > > > > > What is the easiest way to detect in python/C if an object is a subclass > of > > > ndarray? > > > > Um, how about isinstance or issubclass? (if you want strictness you > > can look at whether x.__class__ is zeros(1).__class__) > > What I am trying to do is figure out the easiest way to fix up matrix > multiplies so they work properly. For instance, the following gives the > wrong result: > > In [15]: I = matrix(eye(2)) > > In [16]: I*ones(2) > Out[16]: matrix([[ 1., 1.]]) > > where the output should be a column vector. In order to do that, I need to > reshape the rh multiplicand before __mul__ calls the dot function. There is > also this problem I have a little 'ascol()' function I use for that kind of situation. In your case I'd say I * ascol(ones(2)) --------------------- def ascol(v): """Turns 1-d inputs into a column vectors. For all other inputs acts like atleast_2d. (Q: Should this also transpose any (1xN)'s to be columns? The current thinking is if you have a 1-d then you haven't really decided whether it's a row or col, and this method is asserting that it should be a column. But if it's already a 2-d row, then it's probably a row for a reason, and you should transpose it explicitly if you want a column.) """ arr = numpy.array(v,copy=False,subok=True) if arr.ndim<2: return numpy.transpose(numpy.atleast_2d(arr)) else: return arr --------------------- --bb From peridot.faceted at gmail.com Sat Mar 24 05:29:33 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 24 Mar 2007 05:29:33 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: On 24/03/07, Bill Baxter wrote: > Nice, but how does that fare on things like mdot(a,(b,c),d) ? I'm > pretty sure it doesn't handle it. > I think an mdot that can only multiply things left to right comes up > short compared to an infix operator that can easily use parentheses to > control order. Well, since exact dotting is associative, the parenthesization doesn't matter there; if you're worried about roundoff or efficiency, you're going to have to be more explicit. Unfortunately your approach requires tuples to be treated differently from arrays. Most functions in numpy will happily treat a tuple of length n, a list of length n, and an array of length n the same way. You could do this, and for your own code maybe it's worth it, but I think it would be confusing in the library. Anne From openopt at ukr.net Sat Mar 24 05:37:07 2007 From: openopt at ukr.net (dmitrey) Date: Sat, 24 Mar 2007 11:37:07 +0200 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: <4604F143.1010004@ukr.net> but how about the things like a = dot(array([8]), ones([1000,1000], array([15])))? it will be much faster if we will dot 8 x 15 at first, and than the result to the big array. D. Anne Archibald wrote: > On 24/03/07, Bill Baxter wrote: > > >> Nice, but how does that fare on things like mdot(a,(b,c),d) ? I'm >> pretty sure it doesn't handle it. >> I think an mdot that can only multiply things left to right comes up >> short compared to an infix operator that can easily use parentheses to >> control order. >> > > Well, since exact dotting is associative, the parenthesization doesn't > matter there; if you're worried about roundoff or efficiency, you're > going to have to be more explicit. Unfortunately your approach > requires tuples to be treated differently from arrays. Most functions > in numpy will happily treat a tuple of length n, a list of length n, > and an array of length n the same way. You could do this, and for your > own code maybe it's worth it, but I think it would be confusing in the > library. > > Anne > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > From steve at shrogers.com Sat Mar 24 09:05:43 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 24 Mar 2007 07:05:43 -0600 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: <46052227.70500@shrogers.com> Anne Archibald wrote: > > P.S. reduce isn't even a numpy thing, it's one of python's > much-neglected lispy functions. > It looks like reduce(), map(), and filter() are going away for Python 3.0 since GvR believes that they are redundant and list comprehensions and generator expressions are more readable alternatives. lambda was on the block as well, but will be retained. # Steve From aisaac at american.edu Sat Mar 24 09:44:21 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 09:44:21 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <46052227.70500@shrogers.com> References: <46052227.70500@shrogers.com> Message-ID: On Sat, 24 Mar 2007, "Steven H. Rogers" apparently wrote: > It looks like reduce(), map(), and filter() are going away for Python > 3.0 since GvR believes that they are redundant and list comprehensions > and generator expressions are more readable alternatives. lambda was on > the block as well, but will be retained. I assume you are referring to As I understand it, **if** reduce, map, and filter "go away", they will go away as built-ins and become available in a ``functional`` module. That said, I think losing reduce as a built-in would be sad. It is math friendly. Cheers, Alan Isaac From aisaac at american.edu Sat Mar 24 09:44:23 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 09:44:23 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On Fri, 23 Mar 2007, Charles R Harris apparently wrote: > the following gives the wrong result: > In [15]: I = matrix(eye(2)) > In [16]: I*ones(2) > Out[16]: matrix([[ 1., 1.]]) > where the output should be a column vector. Why should this output a column? I would prefer an exception. Add the axis if you want it: I*ones(2)[:,None] works fine. Matrix multiplication should only work for 2d arrays. Making a larger point: I find the mixing of matrices and arrays when using ``*`` to be highly suspect and far too implicit. This is going to bite people. Finally, however else this moves forward, I think making ``*`` work differently than ``dot`` is a bad idea. Cheers, Alan Isaac From steve at shrogers.com Sat Mar 24 10:54:35 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 24 Mar 2007 08:54:35 -0600 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46052227.70500@shrogers.com> Message-ID: <46053BAB.5090201@shrogers.com> Alan G Isaac wrote: > On Sat, 24 Mar 2007, "Steven H. Rogers" apparently wrote: >> It looks like reduce(), map(), and filter() are going away for Python >> 3.0 since GvR believes that they are redundant and list comprehensions >> and generator expressions are more readable alternatives. lambda was on >> the block as well, but will be retained. > > I assume you are referring to > > > > As I understand it, **if** reduce, map, and filter "go > away", they will go away as built-ins and become available > in a ``functional`` module. > > That said, I think losing reduce as a built-in would be sad. > It is math friendly. > Putting these functions in a 'functional' module (maybe the math module would be better?) has been suggested, but I haven't found anything that indicates that the matter is settled. Sending expressions of interest and use cases could make the final decision more rather than less favorable to the NumPy/SciPy community. There's also a plan to restructure the standard library and possibly reduce the number of library packages included in the standard distribution for Python 3.0. # Steve From charlesr.harris at gmail.com Sat Mar 24 11:33:30 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 09:33:30 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 3/24/07, Alan G Isaac wrote: > > On Fri, 23 Mar 2007, Charles R Harris apparently wrote: > > the following gives the wrong result: > > In [15]: I = matrix(eye(2)) > > In [16]: I*ones(2) > > Out[16]: matrix([[ 1., 1.]]) > > where the output should be a column vector. > > Why should this output a column? > I would prefer an exception. > Add the axis if you want it: > I*ones(2)[:,None] > works fine. Because it is mathematically correct. You can't multiply a vector by a 2x2 matrix and get a 1x2 matrix as the result. Sure, there are work arounds, but if matrix multiplication is going to work when mixed with arrays, it should work correctly. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wfspotz at sandia.gov Sat Mar 24 12:04:41 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Sat, 24 Mar 2007 10:04:41 -0600 Subject: [Numpy-discussion] swig svn commits In-Reply-To: References: Message-ID: <8561377F-2E52-4BFC-A487-15024612D3DD@sandia.gov> No, I don't consider native byte order. What was your solution? The typecheck typemap is so that swig can perform isolated type checking when it is creating dispatch code for overloaded functions. The typechecks I added for INPLACE arrays require that the argument be a numpy array and that PyArray_EquivTypenums() return true for the provided and requested types. On Mar 23, 2007, at 3:03 PM, Sebastian Haase wrote: > Hi, > this is regarding the svn commit by wfspotz. > > Author: wfspotz at sandia.gov > Date: 2007-03-23 13:04:37 -0500 (Fri, 23 Mar 2007) > New Revision: 3593 > > Modified: > trunk/numpy/doc/swig/numpy.i > Log: > Added typecheck typemaps to INPLACE_ARRAY typemap suites > > Hi wfspotz, > I was just wondering if you consider checking for "native byte order" > as part of your inplace-typemap. > I found that to be a problem in my SWIG type maps > that I hi-jacked / boiled down from the older numpy swig files. > (They are mostly for 3D image data, only for a small number of types) > > -Sebastian > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From aisaac at american.edu Sat Mar 24 12:22:36 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 12:22:36 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: >> On Fri, 23 Mar 2007, Charles R Harris apparently wrote: >>> the following gives the wrong result: >>> In [15]: I = matrix(eye(2)) >>> In [16]: I*ones(2) >>> Out[16]: matrix([[ 1., 1.]]) >>> where the output should be a column vector. > On 3/24/07, Alan G Isaac wrote: >> Why should this output a column? >> I would prefer an exception. >> Add the axis if you want it: >> I*ones(2)[:,None] >> works fine. On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > Because it is mathematically correct. You can't multiply > a vector by a 2x2 matrix and get a 1x2 matrix as the > result. I do not understand: what do you mean mathematically correct? What is the space on which this operator is being defined? In my opinion, it is mathematically meaningless. Forcing meaning into it is a bad idea. Just because there is a sloppy convention to call n?1 matrices and 1?n matrices "vectors" does not mean they are not 2d. They are. That is *why* matrix multiplication is defined for them. As I suggested, the right thing to do when someone attemps matrix multiplication with an object that is not 2d is to raise an exception. The wrong thing to do is to pretend than an object without an orientation has one: that is inherently not explicit and will surely bite someone sooner or later, probably sooner. Cheers, Alan Isaac From charlesr.harris at gmail.com Sat Mar 24 13:19:21 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 11:19:21 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 3/24/07, Alan G Isaac wrote: > > >> On Fri, 23 Mar 2007, Charles R Harris apparently wrote: > >>> the following gives the wrong result: > >>> In [15]: I = matrix(eye(2)) > >>> In [16]: I*ones(2) > >>> Out[16]: matrix([[ 1., 1.]]) > >>> where the output should be a column vector. > > > > On 3/24/07, Alan G Isaac wrote: > >> Why should this output a column? > >> I would prefer an exception. > >> Add the axis if you want it: > >> I*ones(2)[:,None] > >> works fine. > > > On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > > Because it is mathematically correct. You can't multiply > > a vector by a 2x2 matrix and get a 1x2 matrix as the > > result. > > > I do not understand: > what do you mean mathematically correct? > What is the space on which this operator is being defined? > > In my opinion, it is mathematically meaningless. > Forcing meaning into it is a bad idea. > > Just because there is a sloppy convention to call n?1 > matrices and 1?n matrices "vectors" does not mean they are > not 2d. They are. That is *why* matrix multiplication is > defined for them. > > As I suggested, the right thing to do when someone attemps > matrix multiplication with an object that is not 2d is to > raise an exception. The wrong thing to do is to pretend > than an object without an orientation has one: that is > inherently not explicit and will surely bite someone sooner > or later, probably sooner. I think it is reasonable to raise an exception in this case, but that is not how numpy currently works, so it is a larger policy decision that I can't make on my own. For the case under consideration it is possible to define 1-D to 2-D rules that are consistent and work, so I am thinking of adding those to the __mul__ operator of the matrix class. However, as you say, such rules are more likely to let some errors slip by. In essence, it is rather like type conversion in mixed expressions: convenient, but open to error. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sat Mar 24 13:31:56 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 13:31:56 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > I think it is reasonable to raise an exception in this > case, but that is not how numpy currently works, so it is > a larger policy decision that I can't make on my own. For > the case under consideration it is possible to define 1-D > to 2-D rules that are consistent and work, so I am > thinking of adding those to the __mul__ operator of the > matrix class. However, as you say, such rules are more > likely to let some errors slip by. In essence, it is > rather like type conversion in mixed expressions: > convenient, but open to error. Your proposed change is also not how numpy currently works. If any change is introduced, I hope it will be to raise an exception. Thanks! Alan From charlesr.harris at gmail.com Sat Mar 24 13:36:33 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 11:36:33 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 3/24/07, Alan G Isaac wrote: > > On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > > I think it is reasonable to raise an exception in this > > case, but that is not how numpy currently works, so it is > > a larger policy decision that I can't make on my own. For > > the case under consideration it is possible to define 1-D > > to 2-D rules that are consistent and work, so I am > > thinking of adding those to the __mul__ operator of the > > matrix class. However, as you say, such rules are more > > likely to let some errors slip by. In essence, it is > > rather like type conversion in mixed expressions: > > convenient, but open to error. > > > Your proposed change is also not how numpy currently works. True, numpy currently returns an incorrect result. We can either fix that, or we can disallow 1-D vectors in matrix computations. But I hesitate to do the latter without discussion and agreement on the list. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Sat Mar 24 14:39:28 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 24 Mar 2007 11:39:28 -0700 Subject: [Numpy-discussion] swig svn commits In-Reply-To: <8561377F-2E52-4BFC-A487-15024612D3DD@sandia.gov> References: <8561377F-2E52-4BFC-A487-15024612D3DD@sandia.gov> Message-ID: On 3/24/07, Bill Spotz wrote: > No, I don't consider native byte order. What was your solution? > I think there is only one solution: If somebody requests INPLACE handling but provides data of wrong byteorder, I have to throw an exception -- in SWIG terms, the typecheck returns False (if I remember right). For me this is just a "last-line of defence" - meaning that I have most of my functions wrapped by another level of python "convenience" functions, and those take care of type and byte-order issues beforehand as needed. -Sebastian > The typecheck typemap is so that swig can perform isolated type > checking when it is creating dispatch code for overloaded functions. > The typechecks I added for INPLACE arrays require that the argument > be a numpy array and that PyArray_EquivTypenums() return true for the > provided and requested types. > > On Mar 23, 2007, at 3:03 PM, Sebastian Haase wrote: > > > Hi, > > this is regarding the svn commit by wfspotz. > > > > Author: wfspotz at sandia.gov > > Date: 2007-03-23 13:04:37 -0500 (Fri, 23 Mar 2007) > > New Revision: 3593 > > > > Modified: > > trunk/numpy/doc/swig/numpy.i > > Log: > > Added typecheck typemaps to INPLACE_ARRAY typemap suites > > > > Hi wfspotz, > > I was just wondering if you consider checking for "native byte order" > > as part of your inplace-typemap. > > I found that to be a problem in my SWIG type maps > > that I hi-jacked / boiled down from the older numpy swig files. > > (They are mostly for 3D image data, only for a small number of types) > > > > -Sebastian > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-5451 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From cjw at sympatico.ca Sat Mar 24 14:06:16 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat, 24 Mar 2007 14:06:16 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: Charles R Harris wrote: > > > On 3/24/07, *Alan G Isaac* > wrote: > > On Fri, 23 Mar 2007, Charles R Harris apparently wrote: > > the following gives the wrong result: > > In [15]: I = matrix(eye(2)) > > In [16]: I*ones(2) > > Out[16]: matrix([[ 1., 1.]]) > > where the output should be a column vector. > > Why should this output a column? > I would prefer an exception. > Add the axis if you want it: > I*ones(2)[:,None] > works fine. > > > Because it is mathematically correct. You can't multiply a vector by a > 2x2 matrix and get a 1x2 matrix as the result. Sure, there are work > arounds, but if matrix multiplication is going to work when mixed with > arrays, it should work correctly. > > Chuck > It depends on the convention you use when working with matrices. Suppose you adopt the notion, for matrices, a vector is always represented by a matrix. This a row vector would have the shape (1, n) and the column vector would have (n, 1). If A were a (3, 4) matrix and b were a 4 element column vector, then the product of A by b, using matrix arithmetic, would give a 3 element column vector. Colin W. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cjw at sympatico.ca Sat Mar 24 14:41:40 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat, 24 Mar 2007 14:41:40 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: Bill Baxter wrote: > On 3/24/07, Anne Archibald wrote: >> On 24/03/07, Bill Baxter wrote: >>> I mentioned in another thread Travis started on the scipy list that I >>> would find it useful if there were a function like dot() that could >>> multiply more than just two things. >>> >>> Here's a sample implementation called 'mdot'. >>> >>> mdot(a,b,c,d) ==> dot(dot(dot(a,b),c),d) >>> mdot(a,(b,c),d) ==> dot(dot(a,dot(b,c),d) >>> mdot(a,(b,(c,d))) ==> dot(a,dot(b,dot(c,d)) >>> >>> --- >>> def mdot(*args): >>> """Multiply all the arguments using matrix product rules. >>> The output is equivalent to multiplying the arguments one by one >>> from left to right using dot(). >>> Precedence can be controlled by creating tuples of arguments, >>> for instance mdot(a,((b,c),d)) multiplies a (a*((b*c)*d)). >>> Note that this means the output of dot(a,b) and mdot(a,b) will differ if >>> a or b is a pure tuple of numbers. >>> """ >>> if len(args)==1: >>> return args[0] >>> elif len(args)==2: >>> return _mdot_r(args[0],args[1]) >>> else: >>> return _mdot_r(args[:-1],args[-1]) >>> >>> def _mdot_r(a,b): >>> """Recursive helper for mdot""" >>> if type(a)==types.TupleType: >>> if len(a)>1: >>> a = mdot(*a) >>> else: >>> a = a[0] >>> if type(b)==types.TupleType: >>> if len(b)>1: >>> b = mdot(*b) >>> else: >>> b = b[0] >>> return numpy.dot(a,b) >> You can do better: >> In [1]: from numpy import * >> >> In [2]: a = array([[0,-1],[1,0]]) >> >> In [3]: reduce(dot,(a,a,a,a)) >> Out[3]: >> array([[1, 0], >> [0, 1]]) >> >> In [4]: def mdot(*args): >> ...: return reduce(dot,args) >> ...: >> >> In [5]: mdot(a,a,a,a) >> Out[5]: >> array([[1, 0], >> [0, 1]]) >> > > Nice, but how does that fare on things like mdot(a,(b,c),d) ? I'm > pretty sure it doesn't handle it. > I think an mdot that can only multiply things left to right comes up > short compared to an infix operator that can easily use parentheses to > control order. +1 for infix Colin W. > > --bb From wbaxter at gmail.com Sat Mar 24 14:52:27 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 03:52:27 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <46052227.70500@shrogers.com> References: <46052227.70500@shrogers.com> Message-ID: On 3/24/07, Steven H. Rogers wrote: > Anne Archibald wrote: > > > > P.S. reduce isn't even a numpy thing, it's one of python's > > much-neglected lispy functions. > > > > It looks like reduce(), map(), and filter() are going away for Python > 3.0 since GvR believes that they are redundant and list comprehensions > and generator expressions are more readable alternatives. lambda was on > the block as well, but will be retained. Are you sure reduce is on the chopping block? The PEP for generator expressions doesn't say anything about reduce becoming redundant. http://www.python.org/dev/peps/pep-0289/ They give the example of reduce(lambda s, a: s + a.myattr, data, 0) becoming sum(a[3] for a in data) but that does nothing to address how functions like sum themselves should be written. For a general binary operator, the equivalent of "sum" can be conveniently expressed as a reduce operation. As far as I can tell Python has no "prod" function. So if one just changes the example to reduce(lambda s, a: s * a.myattr, data, 1) How does one write that in a simplified way using generator expressions without calling on reduce? --bb From perry at stsci.edu Sat Mar 24 14:59:43 2007 From: perry at stsci.edu (Perry Greenfield) Date: Sat, 24 Mar 2007 14:59:43 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46052227.70500@shrogers.com> Message-ID: <84955DD3-D36D-4022-9F0B-256B4B6BFA7C@stsci.edu> On Mar 24, 2007, at 2:52 PM, Bill Baxter wrote: > On 3/24/07, Steven H. Rogers wrote: >> Anne Archibald wrote: >>> >>> P.S. reduce isn't even a numpy thing, it's one of python's >>> much-neglected lispy functions. >>> >> >> It looks like reduce(), map(), and filter() are going away for Python >> 3.0 since GvR believes that they are redundant and list >> comprehensions >> and generator expressions are more readable alternatives. lambda >> was on >> the block as well, but will be retained. > > Are you sure reduce is on the chopping block? The PEP for generator > expressions doesn't say anything about reduce becoming redundant. While it isn't in his slides for Pycon this year, I could have sworn he said he was getting rid of reduce (but not lambda, nor map IIRC but not sure about the latter) Perry From wbaxter at gmail.com Sat Mar 24 15:03:11 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 04:03:11 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: On 3/24/07, Anne Archibald wrote: > On 24/03/07, Bill Baxter wrote: > > > Nice, but how does that fare on things like mdot(a,(b,c),d) ? I'm > > pretty sure it doesn't handle it. > > I think an mdot that can only multiply things left to right comes up > > short compared to an infix operator that can easily use parentheses to > > control order. > > Well, since exact dotting is associative, the parenthesization doesn't > matter there; if you're worried about roundoff or efficiency, you're > going to have to be more explicit. Yes performance was the thing I was thinking about. You often know a little something about the sizes of your inputs that can help make things more efficient. Like BigMatrix1 * BigMatrix2 * LittleVector Is much faster to calculate as BigMatrix1 * (BigMatrix2 * LittleVector) than (BigMatrix1 * BigMatrix2) * LittleVector > Unfortunately your approach requires tuples to be treated differently from arrays. Yeh, that was a conscious decision (which I noted in the docstring). You are right that it is different from what Numpy normally does, which is one reason I didn't say "lets replace dot with this function". > Most functions > in numpy will happily treat a tuple of length n, a list of length n, > and an array of length n the same way. Yep, it's nice in general. But in practice I very rarely do math on tuples. The only thing I can think of is I used to do prod(arr.shape) before I realized that arr.size give the same thing. I'm positive I've never used dot on tuples. > You could do this, and for your > own code maybe it's worth it, but I think it would be confusing in the > library. Could be. Doesn't seem so confusing to me as long as it's documented clearly in the docstring, but YMMV. --bb From charlesr.harris at gmail.com Sat Mar 24 15:15:57 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 13:15:57 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On 3/24/07, Colin J. Williams wrote: > > Charles R Harris wrote: > > > > > > On 3/24/07, *Alan G Isaac* > > wrote: > > > > On Fri, 23 Mar 2007, Charles R Harris apparently wrote: > > > the following gives the wrong result: > > > In [15]: I = matrix(eye(2)) > > > In [16]: I*ones(2) > > > Out[16]: matrix([[ 1., 1.]]) > > > where the output should be a column vector. > > > > Why should this output a column? > > I would prefer an exception. > > Add the axis if you want it: > > I*ones(2)[:,None] > > works fine. > > > > > > Because it is mathematically correct. You can't multiply a vector by a > > 2x2 matrix and get a 1x2 matrix as the result. Sure, there are work > > arounds, but if matrix multiplication is going to work when mixed with > > arrays, it should work correctly. > > > > Chuck > > > It depends on the convention you use when working with matrices. > > Suppose you adopt the notion, for matrices, a vector is always > represented by a matrix. This a row vector would have the shape (1, n) > and the column vector would have (n, 1). > > If A were a (3, 4) matrix and b were a 4 element column vector, then > the product of A by b, using matrix arithmetic, would give a 3 element > column vector. Yes, that is what I am thinking. Given that there are only the two possibilities, row or column, choose the only one that is compatible with the multiplying matrix. The result will not always be a column vector, for instance, mat([[1]])*ones(3) will be a 1x3 row vector. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sat Mar 24 15:24:49 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 04:24:49 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <84955DD3-D36D-4022-9F0B-256B4B6BFA7C@stsci.edu> References: <46052227.70500@shrogers.com> <84955DD3-D36D-4022-9F0B-256B4B6BFA7C@stsci.edu> Message-ID: On 3/25/07, Perry Greenfield wrote: > > On Mar 24, 2007, at 2:52 PM, Bill Baxter wrote: > > > On 3/24/07, Steven H. Rogers wrote: > >> Anne Archibald wrote: > >>> > >>> P.S. reduce isn't even a numpy thing, it's one of python's > >>> much-neglected lispy functions. > >>> > >> > >> It looks like reduce(), map(), and filter() are going away for Python > >> 3.0 since GvR believes that they are redundant and list > >> comprehensions > >> and generator expressions are more readable alternatives. lambda > >> was on > >> the block as well, but will be retained. > > > > Are you sure reduce is on the chopping block? The PEP for generator > > expressions doesn't say anything about reduce becoming redundant. > > While it isn't in his slides for Pycon this year, I could have sworn > he said he was getting rid of reduce (but not lambda, nor map IIRC > but not sure about the latter) I could definitely see map and filter getting killed off, because their functionality is almost 100% covered by list comprehensions. map(function, s1,s2,s3...) ~== [function(x) for x in izip(s1,s2,s3...) ] filter(function, sequence) ~== [x for x in sequence if function(x)] but reduce? Makes no sense. --bb From steve at shrogers.com Sat Mar 24 15:25:45 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sat, 24 Mar 2007 13:25:45 -0600 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <84955DD3-D36D-4022-9F0B-256B4B6BFA7C@stsci.edu> References: <46052227.70500@shrogers.com> <84955DD3-D36D-4022-9F0B-256B4B6BFA7C@stsci.edu> Message-ID: <46057B39.4090907@shrogers.com> Perry Greenfield wrote: > On Mar 24, 2007, at 2:52 PM, Bill Baxter wrote: > >> On 3/24/07, Steven H. Rogers wrote: >>> Anne Archibald wrote: >>>> P.S. reduce isn't even a numpy thing, it's one of python's >>>> much-neglected lispy functions. >>>> >>> It looks like reduce(), map(), and filter() are going away for Python >>> 3.0 since GvR believes that they are redundant and list >>> comprehensions >>> and generator expressions are more readable alternatives. lambda >>> was on >>> the block as well, but will be retained. >> Are you sure reduce is on the chopping block? The PEP for generator >> expressions doesn't say anything about reduce becoming redundant. > > While it isn't in his slides for Pycon this year, I could have sworn > he said he was getting rid of reduce (but not lambda, nor map IIRC > but not sure about the latter) > The generator expression PEP doesn't say this, but the Python 3000 planning PEP (http://www.python.org/dev/peps/pep-3100/) has map() and filter() on the 'to-be-removed' list with a parenthetic comment that they can stay. Removal of reduce() is annotated as 'done'. My recollection of GvR's comments at PyCon was that the status of map() and filter() is still undecided. # Steve From aisaac at american.edu Sat Mar 24 15:36:15 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 15:36:15 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46052227.70500@shrogers.com> Message-ID: On Sun, 25 Mar 2007, Bill Baxter apparently wrote: > So if one just > changes the example to > reduce(lambda s, a: s * a.myattr, data, 1) > How does one write that in a simplified way using generator > expressions without calling on reduce? Eliminating the expressiveness of ``reduce`` has in my opinion never been publically justified. E.g., :: #evaluate polynomial (coefs=[an,...,a0]) at x # using Horner's rule def horner(coefs, x): return reduce(lambda a1,a0: a1*x+a0, coefs) I suspect that ``reduce`` remains in danger only because the math community has been too quiet. Cheers, Alan Isaac From aisaac at american.edu Sat Mar 24 15:36:18 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 15:36:18 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > Yes, that is what I am thinking. Given that there are only the two > possibilities, row or column, choose the only one that is compatible with > the multiplying matrix. The result will not always be a column vector, for > instance, mat([[1]])*ones(3) will be a 1x3 row vector. Ack! The simple rule `post multiply means its a column vector` would be horrible enough: A*ones(n)*B becomes utterly obscure. Now even that simple rule is to be violated?? Down this path lies madness. Please, just raise an exception. Cheers, Alan Isaac From wbaxter at gmail.com Sat Mar 24 15:35:51 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 04:35:51 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <46057B39.4090907@shrogers.com> References: <46052227.70500@shrogers.com> <84955DD3-D36D-4022-9F0B-256B4B6BFA7C@stsci.edu> <46057B39.4090907@shrogers.com> Message-ID: On 3/25/07, Steven H. Rogers wrote: > The generator expression PEP doesn't say this, but the Python 3000 > planning PEP (http://www.python.org/dev/peps/pep-3100/) has map() and > filter() on the 'to-be-removed' list with a parenthetic comment that > they can stay. Removal of reduce() is annotated as 'done'. My > recollection of GvR's comments at PyCon was that the status of map() and > filter() is still undecided. Well, I'll be. That pep points to guido's blog which has this comment: """ So now reduce(). This is actually the one I've always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it's better to write out the accumulation loop explicitly. There aren't a whole lot of associative operators. (Those are operators X for which (a X b) X c equals a X (b X c).) I think it's just about limited to +, *, &, |, ^, and shortcut and/or. We already have sum(); I'd happily trade reduce() for product(), so that takes care of the two most common uses. The bitwise operators are rather specialized and we can put fast versions in a library module if there's demand; for shortcut booleans I have the following proposal. """ So basically reduce() is going away because Guido has trouble understanding it and because he can only imagine 7 possible ways under the sun to combine inputs associatively. Wow. --bb From oliphant at ee.byu.edu Sat Mar 24 15:46:36 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 24 Mar 2007 13:46:36 -0600 Subject: [Numpy-discussion] New Operators in Python Message-ID: <4605801C.6030300@ee.byu.edu> Every so often the idea of new operators comes up because of the need to do both "matrix-multiplication" and element-by-element multiplication. I think this is one area where the current Python approach is not as nice because we have a limited set of operators to work with. One thing I wonder is if we are being vocal enough with the Python 3000 crowd to try and get additional operators into the language itself. What if we could get a few new operators into the language to help us. If we don't ask for it, it certainly won't happen. My experience is that the difficulty of using the '*' operator for both matrix multiplication and element-by-element multiplication depending on the class of the object is not especially robust. It makes it harder to write generic code, and we still haven't gotten everything completely right. It is somewhat workable as it stands, but I think it would be nicer if we could have some "meta" operator that allowed an alternative definition of major operators. Something like @* for example (just picking a character that is already used for decorators). I wonder if we should propose such a thing for Python 3000. -Travis From oliphant at ee.byu.edu Sat Mar 24 16:02:59 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 24 Mar 2007 14:02:59 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: <460583F3.9030002@ee.byu.edu> Alan G Isaac wrote: > On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > >> Yes, that is what I am thinking. Given that there are only the two >> possibilities, row or column, choose the only one that is compatible with >> the multiplying matrix. The result will not always be a column vector, for >> instance, mat([[1]])*ones(3) will be a 1x3 row vector. >> > > > > Ack! The simple rule `post multiply means its a column vector` > would be horrible enough: A*ones(n)*B becomes utterly obscure. > Now even that simple rule is to be violated?? > > Down this path lies madness. > Please, just raise an exception. > My opinion is that a 1-d array in matrix-multiplication should always be interpreted as a row vector. Is this not what is currently done? If not, then it is a bug in my mind. -Travis From zpincus at stanford.edu Sat Mar 24 15:56:05 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Sat, 24 Mar 2007 12:56:05 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <4604BA01.5040606@gemini.edu> References: <4604BA01.5040606@gemini.edu> Message-ID: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> Hello folks, >> Hmm, this is worrisome. There really shouldn't be ringing on >> continuous-tone images like Lena -- right? (And at no step in an >> image like that should gaussian filtering be necessary if you're >> doing spline interpolation -- also right?) > > That's hard to say. Just because it's mainly a continuous-tone image > doesn't necessarily mean it is well sampled everywhere. This depends > both on the subject and the camera optics. Unlike the data I usually > work with, I think everyday digital photographs (probably a photo > scan in the case of Lena) do not generally have the detector sampling > frequency matched to the optical resolution of the image. If that's > true, the presence of aliasing in interpolated images depends on the > structure of the subject and whether the scene has edges or high- > frequency patterns in it. Based on my reading of the two excellent Unser papers (both the one that ndimage's spline code is based on, and the one that Travis linked to), it seems like a major point of using splines for interpolation is *better* behavior in the case of non-band-limited data than the traditional 'anti-aliasing [e.g. lowpass] filter' + 'resampling' + 'reconstruction [e.g. lowpass] filter' procedure. That is, based on my (very limited) understanding (I understand the basics of the sampling theorem and can follow the Unser paper's update thereof, but not much more than that), in the spline case the spline fitting step *replaces* the anti-aliasing filter in the above procedure as the method for dealing with non-band-limited data. And the claim is that it in fact works better in many cases. So it seems that something is wrong if the spline interpolation tools in ndimage only work properly in the band-limited case, or require lowpass filtering. > Stefan's rotated Lena example is indeed a bit bizarre on zooming in! > However, the artefacts are clearly localized to distinct edges, so I > suspect this is indeed some kind of aliasing. Moreover, it looks like > Lena has been decimated (reduced in size) prior to the rotation. That > is definitely a good way to get artefacts, unless an anti-aliasing > filter is applied before shrinking the image. My impression is that > this image is probably somewhat undersampled (to understand exactly > what that means, read up on the Sampling Theorem). Agreed, it looks like aliasing. Nevertheless, any resampling procedure is supposed to deal with this internally, right? Either by lowpass filtering (traditional case), or by spline fitting (spline case as described by Unser and understood by me) -- it shouldn't be letting aliasing bubble through, correct? >> The first was on Stefan's artificial data which had sharp edges, and >> got very nasty ringing artifacts even with 3rd order splines. From >> your recollection, is this expected behavior based on splines and the >> nature of Stefan's image, or more likely to be a bug? > > Your question was aimed at Travis, so I don't want to discourage him > from answering it :-), but looking at this in more detail, I do think > the amplitude of the artefacts here is greater than I might expect due > to ringing with a quadratic b-spline kernel, which I think has minima > with amplitudes <10% of the central peak. There has to be SOME > oscillation, but in Stefan's "rotate_artifacts" example it seems to be > at the level of ~100%. Also, it is not present on one of the inner > edges for some reason. So I do wonder if the algorithm in nd_image is > making this worse than it needs to be. Good observation! Here are cubic spline fits to a step and a delta function, which both have quite small amounts of ringing compared to what's observed -- at least on the color images. Maybe 10% ringing in each color channel adds up perceptually more than it does mathematically? import numpy, Gnuplot, scipy.interpolate # step x = numpy.arange(-10, 10) y = numpy.where(x > 0, 1, 0) tck = scipy.interpolate.splrep(x, y, s=0, k=3) nx = numpy.linspace(-10, 9, 200, True) ny = scipy.interpolate.splev(nx, tck) d = Gnuplot.Data(numpy.transpose([x, y]), with='points') nd = Gnuplot.Data(numpy.transpose([nx,ny]), with='lines') g = Gnuplot.Gnuplot() g.plot(d, nd) ny.min() # -0.107490563074 -- largest ringing is ~10% of step size # delta x = numpy.arange(-10, 10) y = numpy.where(x == 0, 1, 0) tck = scipy.interpolate.splrep(x, y, s=0, k=3) nx = numpy.linspace(-10, 9, 200, True) ny = scipy.interpolate.splev(nx, tck) d = Gnuplot.Data(numpy.transpose([x, y]), with='points') nd = Gnuplot.Data(numpy.transpose([nx,ny]), with='lines') g = Gnuplot.Gnuplot() g.plot(d, nd) ny.min() # -0.136449610107 -- largest ringing is ~13% of impulse size Now let's use ndimage to rotate a step function by 45%, or zoom in on it: import numpy, scipy.ndimage o = numpy.ones((100, 50), dtype=float) z = numpy.zeros((100, 50), dtype=float) a = numpy.concatenate([o, z], axis=1) b = scipy.ndimage.rotate(a, 45, order=3) b.max() # 1.08832325055 b.min() # -0.0883232505527 c = scipy.ndimage.zoom(a, 5, order=3) c.min() # -0.107600148039 c.max() # 1.10760014804 This (very simple) example looks exactly right. (Also, when I manually inspect the images in ImageJ -- which can load 32-bit floating point TIFFs, which is great -- they look correct). So what's going on with Lena and the other color/otherwise more complex images? Zach From aisaac at american.edu Sat Mar 24 16:06:04 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 16:06:04 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: <460583F3.9030002@ee.byu.edu> References: <460583F3.9030002@ee.byu.edu> Message-ID: On Sat, 24 Mar 2007, Travis Oliphant apparently wrote: > My opinion is that a 1-d array in matrix-multiplication > should always be interpreted as a row vector. Is this not > what is currently done? If not, then it is a bug in my > mind. >>> N.__version__ '1.0' >>> I matrix([[ 1., 0.], [ 0., 1.]]) >>> I*N.ones(2) matrix([[ 1., 1.]]) If N.ones(2) were treated as a row vector, matrix multiplication is not possible. So the question is what should happen. I would like an exception raised. The current behavior is lmost certainly not desirable, although it has the virute of matching ``dot``. >>> N.dot(I,N.ones(2)) matrix([[ 1., 1.]]) Cheers, Alan Isaac From charlesr.harris at gmail.com Sat Mar 24 16:05:59 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 14:05:59 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: <460583F3.9030002@ee.byu.edu> References: <460583F3.9030002@ee.byu.edu> Message-ID: On 3/24/07, Travis Oliphant wrote: > > Alan G Isaac wrote: > > On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > > > >> Yes, that is what I am thinking. Given that there are only the two > >> possibilities, row or column, choose the only one that is compatible > with > >> the multiplying matrix. The result will not always be a column vector, > for > >> instance, mat([[1]])*ones(3) will be a 1x3 row vector. > >> > > > > > > > > Ack! The simple rule `post multiply means its a column vector` > > would be horrible enough: A*ones(n)*B becomes utterly obscure. > > Now even that simple rule is to be violated?? > > > > Down this path lies madness. > > Please, just raise an exception. > > > > My opinion is that a 1-d array in matrix-multiplication should always be > interpreted as a row vector. Is this not what is currently done? If > not, then it is a bug in my mind. Fair enough. But in that case mat(eye(2))*ones(2) should raise an error instead of returning a row vector. In [1]: mat(eye(2))*ones(2) Out[1]: matrix([[ 1., 1.]]) The problem is that the return object is created as a row matrix and then filled as if the rhs was a column vector in PyArray_MatrixProduct. So this should be made a bug? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Sat Mar 24 16:31:14 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 24 Mar 2007 14:31:14 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: <460583F3.9030002@ee.byu.edu> Message-ID: <46058A92.6050607@ee.byu.edu> Charles R Harris wrote: > > > On 3/24/07, *Travis Oliphant* > wrote: > > Alan G Isaac wrote: > > On Sat, 24 Mar 2007, Charles R Harris apparently wrote: > > > >> Yes, that is what I am thinking. Given that there are only the two > >> possibilities, row or column, choose the only one that is > compatible with > >> the multiplying matrix. The result will not always be a column > vector, for > >> instance, mat([[1]])*ones(3) will be a 1x3 row vector. > >> > > > > > > > > Ack! The simple rule `post multiply means its a column vector` > > would be horrible enough: A*ones(n)*B becomes utterly obscure. > > Now even that simple rule is to be violated?? > > > > Down this path lies madness. > > Please, just raise an exception. > > > > My opinion is that a 1-d array in matrix-multiplication should > always be > interpreted as a row vector. Is this not what is currently done? If > not, then it is a bug in my mind. > > > Fair enough. But in that case mat(eye(2))*ones(2) should raise an > error instead of returning a row vector. > > In [1]: mat(eye(2))*ones(2) > Out[1]: matrix([[ 1., 1.]]) > > The problem is that the return object is created as a row matrix and > then filled as if the rhs was a column vector in > PyArray_MatrixProduct. So this should be made a bug? > Yes, as you describe, dot allows 2x2 times a 1-d array (and interprets the 1-d array as a column vector). This has always been the behavior of dot and won't change. But, that doesn't mean we have to allow this behavior for matrices. In other words we should probably check against it for matrix-multiplication. The dot operator allows mixing n-d arrays, but I'm persuaded that the matrix multiplication should only work for 2-d arrays and if it is given a 1-d array, it seems that it should have a specific interpretation of that 1-d array as a 2-d array. -Travis > Chuck > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From oliphant at ee.byu.edu Sat Mar 24 16:33:04 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 24 Mar 2007 14:33:04 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: <460583F3.9030002@ee.byu.edu> Message-ID: <46058B00.8000105@ee.byu.edu> Alan G Isaac wrote: > On Sat, 24 Mar 2007, Travis Oliphant apparently wrote: > >> My opinion is that a 1-d array in matrix-multiplication >> should always be interpreted as a row vector. Is this not >> what is currently done? If not, then it is a bug in my >> mind. >> > > >>>> N.__version__ >>>> > '1.0' > >>>> I >>>> > matrix([[ 1., 0.], > [ 0., 1.]]) > >>>> I*N.ones(2) >>>> > matrix([[ 1., 1.]]) > > If N.ones(2) were treated as a row vector, > matrix multiplication is not possible. > So the question is what should happen. > I would like an exception raised. > The current behavior is lmost certainly not desirable, > although it has the virute of matching ``dot``. > > I think that's why it is the current behavior --- by default rather than by design. But, dot has different use cases. I'd be fine with an error raised on matrix multiplication (as long as dot is not changed). In other words, I'd like to see 1-d arrays always interpreted the same way (as row vectors) when used in matrix multiplication. -Travis From charlesr.harris at gmail.com Sat Mar 24 16:29:35 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 14:29:35 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: <46058B00.8000105@ee.byu.edu> References: <460583F3.9030002@ee.byu.edu> <46058B00.8000105@ee.byu.edu> Message-ID: On 3/24/07, Travis Oliphant wrote: > > Alan G Isaac wrote: > > On Sat, 24 Mar 2007, Travis Oliphant apparently wrote: > > > >> My opinion is that a 1-d array in matrix-multiplication > >> should always be interpreted as a row vector. Is this not > >> what is currently done? If not, then it is a bug in my > >> mind. > >> > > > > > >>>> N.__version__ > >>>> > > '1.0' > > > >>>> I > >>>> > > matrix([[ 1., 0.], > > [ 0., 1.]]) > > > >>>> I*N.ones(2) > >>>> > > matrix([[ 1., 1.]]) > > > > If N.ones(2) were treated as a row vector, > > matrix multiplication is not possible. > > So the question is what should happen. > > I would like an exception raised. > > The current behavior is lmost certainly not desirable, > > although it has the virute of matching ``dot``. > > > > > I think that's why it is the current behavior --- by default rather than > by design. But, dot has different use cases. > > I'd be fine with an error raised on matrix multiplication (as long as > dot is not changed). In other words, I'd like to see 1-d arrays always > interpreted the same way (as row vectors) when used in matrix > multiplication. Sounds good to me, it keeps things simple. The behavious 1-D -> row vector when promoted to a matrix should be documented as official behaviour. I filed ticket #473 on this yesterday. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sat Mar 24 16:34:04 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 16:34:04 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: <46058B00.8000105@ee.byu.edu> References: <460583F3.9030002@ee.byu.edu><46058B00.8000105@ee.byu.edu> Message-ID: On Sat, 24 Mar 2007, Travis Oliphant apparently wrote: > I'd be fine with an error raised on matrix multiplication > (as long as dot is not changed). In other words, I'd > like to see 1-d arrays always interpreted the same way (as > row vectors) when used in matrix multiplication. My preference would be for matrix multiplication with 1d arrays to always raise an error. Anything else is too "implicit". As a second choice, having them always interpreted as a specific kind of vector (either row or colum, as long as its *always* the same) is equivalent to me. But shouldn't there be a clear justification of such implicit conversion? Is it really so useful? Cheers, Alan From peridot.faceted at gmail.com Sat Mar 24 16:40:30 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 24 Mar 2007 16:40:30 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: <460583F3.9030002@ee.byu.edu> References: <460583F3.9030002@ee.byu.edu> Message-ID: On 24/03/07, Travis Oliphant wrote: > My opinion is that a 1-d array in matrix-multiplication should always be > interpreted as a row vector. Is this not what is currently done? If > not, then it is a bug in my mind. An alternative approach, in line with the usual usage, is that a vector should be interpreted as a row vector or a column vector as necessary to make the expression work out. Of course this can be ambiguous if there's no matrices in the expression, but that's not relevant here. Certainly the *result* of such an operation should be a matrix, not a vector, of the shape that would naturally result. So mat(eye(2))*ones(2) should realize that the operation doesn't make sense if it's a row vector, convert it to a column vector, do the multiplication, and return a column vector. (Which is not what it does now, it seems.) Note that taking a vector and left-multiplying it by a matrix is a very natural operation that won't work any more if you treat all vectors as if they were row vectors. Are there instances where this makes matrix multiplication non-associative? Where it is surprising? I've never used numpy's matrices, so this is more here to make sure we consider all the alternatives. Anne From matthieu.brucher at gmail.com Sat Mar 24 16:41:45 2007 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Sat, 24 Mar 2007 21:41:45 +0100 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <4605801C.6030300@ee.byu.edu> References: <4605801C.6030300@ee.byu.edu> Message-ID: Hi, I followed the discussion on the scipy ML, and I would advocate it as well. I miss the dichotomy that is present in Matlab, and to have a similar degree of freedom, it would be good to have it in the upcoming major release of Python. Matthieu 2007/3/24, Travis Oliphant : > > Every so often the idea of new operators comes up because of the need to > do both "matrix-multiplication" and element-by-element multiplication. > > I think this is one area where the current Python approach is not as > nice because we have a limited set of operators to work with. > > One thing I wonder is if we are being vocal enough with the Python 3000 > crowd to try and get additional operators into the language itself. > > What if we could get a few new operators into the language to help us. > If we don't ask for it, it certainly won't happen. > My experience is that the difficulty of using the '*' operator for both > matrix multiplication and element-by-element multiplication depending on > the class of the object is not especially robust. It makes it harder to > write generic code, and we still haven't gotten everything completely > right. > > It is somewhat workable as it stands, but I think it would be nicer if > we could have some "meta" operator that allowed an alternative > definition of major operators. Something like @* for example (just > picking a character that is already used for decorators). > > I wonder if we should propose such a thing for Python 3000. > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 24 16:48:17 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 14:48:17 -0600 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <4605801C.6030300@ee.byu.edu> References: <4605801C.6030300@ee.byu.edu> Message-ID: On 3/24/07, Travis Oliphant wrote: > > Every so often the idea of new operators comes up because of the need to > do both "matrix-multiplication" and element-by-element multiplication. > > I think this is one area where the current Python approach is not as > nice because we have a limited set of operators to work with. > > One thing I wonder is if we are being vocal enough with the Python 3000 > crowd to try and get additional operators into the language itself. > > What if we could get a few new operators into the language to help us. > If we don't ask for it, it certainly won't happen. > My experience is that the difficulty of using the '*' operator for both > matrix multiplication and element-by-element multiplication depending on > the class of the object is not especially robust. It makes it harder to > write generic code, and we still haven't gotten everything completely > right. > > It is somewhat workable as it stands, but I think it would be nicer if > we could have some "meta" operator that allowed an alternative > definition of major operators. Something like @* for example (just > picking a character that is already used for decorators). Yes indeed, this is an old complaint. Just having an infix operator would be an improvement: A dot B dot C Not that I am suggesting dot in this regard ;) In particular, it wouldn't parse without spaces. What about division? Matlab has both / and \ for left and right matrix division and something like that could call solve instead of inverse, leading to some efficiencies. We also have both dot and tensordot, which raises the problem of interpretation when ndim > 2. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 24 16:49:00 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 24 Mar 2007 15:49:00 -0500 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: Message-ID: <46058EBC.90804@gmail.com> Bill Baxter wrote: > On 3/24/07, Anne Archibald wrote: >> You could do this, and for your >> own code maybe it's worth it, but I think it would be confusing in the >> library. > > Could be. Doesn't seem so confusing to me as long as it's documented > clearly in the docstring, but YMMV. I don't know. Given our previous history with convenience functions with different calling semantics (anyone remember rand()?), I think it probably will confuse some people. I'd really like to see it on a cookbook page, though. I'd use it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wbaxter at gmail.com Sat Mar 24 16:51:41 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 05:51:41 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46052227.70500@shrogers.com> Message-ID: On 3/25/07, Alan G Isaac wrote: > On Sun, 25 Mar 2007, Bill Baxter apparently wrote: > > So if one just > > changes the example to > > reduce(lambda s, a: s * a.myattr, data, 1) > > How does one write that in a simplified way using generator > > expressions without calling on reduce? > > > Eliminating the expressiveness of ``reduce`` has in my > opinion never been publically justified. Though the post on Guido's blog got lots of people jumping on the witch-burning bandwagon for getting rid of reduce(). I think I saw "good riddance" more than once. On the other hand slightly fewer people people said "hey I use that!" > > E.g., :: > > #evaluate polynomial (coefs=[an,...,a0]) at x > # using Horner's rule > def horner(coefs, x): > return reduce(lambda a1,a0: a1*x+a0, coefs) > > I suspect that ``reduce`` remains in danger only because the > math community has been too quiet. To be fair, the alternative is def horner2(coefs, x): ret = 0 for a in coefs: ret = ret*x + a return ret Which, though more typing, isn't really any harder or easier to read in my mind. But there is a difference in timings: In [107]: t1 = timeit.Timer('horner([4,3,2,1],2)','from __main__ import horner') In [108]: t2 = timeit.Timer('horner2([4,3,2,1],2)','from __main__ import horner2') In [111]: t1.timeit(), t2.timeit() Out[111]: (2.463477341393947, 1.1736731903077064) The explicit loop runs about twice as fast in this case. I think it's fine for filter()/reduce()/map() to be taken out of builtins and moved to a standard module, but it's not clear that that's what they're going to do. That py3K web page just says "remove reduce()... done". --bb From robert.kern at gmail.com Sat Mar 24 16:57:37 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 24 Mar 2007 15:57:37 -0500 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46052227.70500@shrogers.com> Message-ID: <460590C1.7050507@gmail.com> Bill Baxter wrote: > I think it's fine for filter()/reduce()/map() to be taken out of > builtins and moved to a standard module, but it's not clear that > that's what they're going to do. That py3K web page just says "remove > reduce()... done". http://svn.python.org/projects/python/branches/p3yk/Lib/functools.py http://svn.python.org/projects/python/branches/p3yk/Modules/_functoolsmodule.c -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stefan at sun.ac.za Sat Mar 24 16:58:54 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 24 Mar 2007 22:58:54 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <4604BA01.5040606@gemini.edu> References: <4604BA01.5040606@gemini.edu> Message-ID: <20070324205854.GZ6085@mentat.za.net> On Sat, Mar 24, 2007 at 01:41:21AM -0400, James Turner wrote: > That's hard to say. Just because it's mainly a continuous-tone image > doesn't necessarily mean it is well sampled everywhere. This depends > both on the subject and the camera optics. Unlike the data I usually > work with, I think everyday digital photographs (probably a photo > scan in the case of Lena) do not generally have the detector sampling > frequency matched to the optical resolution of the image. If that's > true, the presence of aliasing in interpolated images depends on the > structure of the subject and whether the scene has edges or high- > frequency patterns in it. Agreed, but the aliasing effects isn't not the problem here, as it should be visible in the input image as well. I'd expect a third-order spline interpolation to be more smooth than a first-order interpolant, but in the resulting images this isn't the case. See http://mentat.za.net/results/lena_small.png http://mentat.za.net/results/img_rot_30_1.png (1st order spline) http://mentat.za.net/results/img_rot_30_3.png (3rd order spline) > Lena has been decimated (reduced in size) prior to the rotation. That > is definitely a good way to get artefacts, unless an anti-aliasing > filter is applied before shrinking the image. My impression is that > this image is probably somewhat undersampled (to understand exactly > what that means, read up on the Sampling Theorem). The artefacts arn't visible in the source image (url above). The image definately is a scaled down version of the original Lena -- very interesting, btw, see http://www.cs.cmu.edu/~chuck/lennapg/lenna.shtml > investigating it further. One experiment might be to blur the original > Lena with a Gaussian whose sigma is 1 pixel of the shrunken image > before actually shrinking her, then do the rotation. A rotation should take place without significant shifts in colour. This almost looks like a value overflow problem. > So I do wonder if the algorithm in nd_image is making this worse > than it needs to be. That is my suspicion, too. > compare the results? I just tried doing a similar rotation in PyRAF on > a monochrome image with a bicubic spline, and see considerably smaller > artefacts (just a compact overshoot of probably a few % at the > edge). Could you apply the PyRAF rotation on the Lena given above and post the result? I always thought we could simply revert to using bilinear and bicubic polygon interpolation (instead of spline interpolation), but now I read on wikipedia: """ In the mathematical subfield of numerical analysis, spline interpolation is a form of interpolation where the interpolant is a special type of piecewise polynomial called a spline. Spline interpolation is preferred over polynomial interpolation because the interpolation error can be made small even when using low degree polynomials for the spline. Thus, spline interpolation avoids the problem of Runge's phenomenon which occurs when using high degree polynomials. """ http://en.wikipedia.org/wiki/Spline_interpolation also take a look at http://en.wikipedia.org/wiki/Runge%27s_phenomenon So much for side-stepping Runge's phenomenon :) Cheers St?fan From haase at msg.ucsf.edu Sat Mar 24 16:59:09 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 24 Mar 2007 13:59:09 -0700 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <4605801C.6030300@ee.byu.edu> Message-ID: On 3/24/07, Charles R Harris wrote: > > > On 3/24/07, Travis Oliphant wrote: > > Every so often the idea of new operators comes up because of the need to > > do both "matrix-multiplication" and element-by-element multiplication. > > > > I think this is one area where the current Python approach is not as > > nice because we have a limited set of operators to work with. > > > > One thing I wonder is if we are being vocal enough with the Python 3000 > > crowd to try and get additional operators into the language itself. > > > > What if we could get a few new operators into the language to help us. > > If we don't ask for it, it certainly won't happen. > > My experience is that the difficulty of using the '*' operator for both > > matrix multiplication and element-by-element multiplication depending on > > the class of the object is not especially robust. It makes it harder to > > write generic code, and we still haven't gotten everything completely > > right. > > > > It is somewhat workable as it stands, but I think it would be nicer if > > we could have some "meta" operator that allowed an alternative > > definition of major operators. Something like @* for example (just > > picking a character that is already used for decorators). > > Yes indeed, this is an old complaint. Just having an infix operator would be > an improvement: > > A dot B dot C > > Not that I am suggesting dot in this regard ;) In particular, it wouldn't > parse without spaces. What about division? Matlab has both / and \ for left > and right matrix division and something like that could call solve instead > of inverse, leading to some efficiencies. We also have both dot and > tensordot, which raises the problem of interpretation when ndim > 2. > > Chuck > I understand the convenience of more infix operators. And I sometimes think one should just be able to define new ones at will .... On the other hand, I'm now playing the devil's advocate: A "math specific" language like Matlab has obviously an overwhelming need for a second set of matrix/array operators. However, a language -- as broadly used as -- Python might be just better off having a simple, concise and limited set of infix operators. I assume that this is the official argument. I got especially "worried" when being remember of the "\" right-to-left division operator. (As I said, it very useful to have in Matlab, and I wish sometimes we could add things like this). It is just important to keep the "con" - argument clearly in mind. I hope this helps the discussion. - Sebastian Haase From peridot.faceted at gmail.com Sat Mar 24 17:14:04 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sat, 24 Mar 2007 17:14:04 -0400 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <4605801C.6030300@ee.byu.edu> Message-ID: On 24/03/07, Charles R Harris wrote: > Yes indeed, this is an old complaint. Just having an infix operator would be > an improvement: > > A dot B dot C > > Not that I am suggesting dot in this regard ;) In particular, it wouldn't > parse without spaces. What about division? Matlab has both / and \ for left > and right matrix division and something like that could call solve instead > of inverse, leading to some efficiencies. We also have both dot and > tensordot, which raises the problem of interpretation when ndim > 2. There's always the Haskell approach: *every* function can be an infix operator, just surround the name with backticks; functions with line-noise names (like + * - /) are infix by default. Not a good idea for python, I think. Anne. From wbaxter at gmail.com Sat Mar 24 17:35:15 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 06:35:15 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <460590C1.7050507@gmail.com> References: <46052227.70500@shrogers.com> <460590C1.7050507@gmail.com> Message-ID: On 3/25/07, Robert Kern wrote: > Bill Baxter wrote: > > > I think it's fine for filter()/reduce()/map() to be taken out of > > builtins and moved to a standard module, but it's not clear that > > that's what they're going to do. That py3K web page just says "remove > > reduce()... done". > > http://svn.python.org/projects/python/branches/p3yk/Lib/functools.py > http://svn.python.org/projects/python/branches/p3yk/Modules/_functoolsmodule.c Great! That's a relief. --bb From wbaxter at gmail.com Sat Mar 24 17:39:40 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sun, 25 Mar 2007 06:39:40 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: <46058EBC.90804@gmail.com> References: <46058EBC.90804@gmail.com> Message-ID: On 3/25/07, Robert Kern wrote: > Bill Baxter wrote: > I don't know. Given our previous history with convenience functions with > different calling semantics (anyone remember rand()?), I think it probably will > confuse some people. > > I'd really like to see it on a cookbook page, though. I'd use it. Done. http://www.scipy.org/Cookbook/MultiDot --bb From zpincus at stanford.edu Sat Mar 24 18:25:38 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Sat, 24 Mar 2007 15:25:38 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070324205854.GZ6085@mentat.za.net> References: <4604BA01.5040606@gemini.edu> <20070324205854.GZ6085@mentat.za.net> Message-ID: Hello all, Mystery solved, I think. I downloaded St?fan's Lena images and tried rotating them myself. As far as I can tell, the artifacts are caused by numerical overflow, as Lena is an unsigned 8-bit image. If Lena is converted to floating-point before the rotation is applied, and then the intensity range is clipped to [0,255] and converted back to uint8 before saving, everything looks fine. So, the "problem" was indeed the ringing that spline interpolation introduces. Despite the fact that this ringing was on the order of a few percent (as shown earlier), that few percent made a big difference when the it caused intensity values to ring over the top or under the bottom of the numerical type's range. Thus the bizarre wrap-around artifacts only in certain edge regions. When using floating-point images throughout, the spline interpolation looks great even on the step-function images, up to order 4 and 5 where the ringing gets (slightly) noticeable. So, is this a bug? Well, I still think so. Given that small ringing is going to happen on all but the very smoothest images, and given that ndimage is going to be used on non-floating-point types, it would be good if there were some explicit internal clipping to the data type's range. Otherwise, the ndimage resampling tools are unfit for use on non-floating-point data that resides near the edges of the range of the data type. Though I'm not quite sure how one would structure the calculations so that it would be possible to tell when over/underflow happened... it might not be possible. In which case, either the tools should use floating-point math at some of the steps internally (as few as possible) before clipping and converting to the required data type, or explicit warnings should be added to the documentation. Zach On Mar 24, 2007, at 1:58 PM, Stefan van der Walt wrote: > On Sat, Mar 24, 2007 at 01:41:21AM -0400, James Turner wrote: >> That's hard to say. Just because it's mainly a continuous-tone image >> doesn't necessarily mean it is well sampled everywhere. This depends >> both on the subject and the camera optics. Unlike the data I usually >> work with, I think everyday digital photographs (probably a photo >> scan in the case of Lena) do not generally have the detector sampling >> frequency matched to the optical resolution of the image. If that's >> true, the presence of aliasing in interpolated images depends on the >> structure of the subject and whether the scene has edges or high- >> frequency patterns in it. > > Agreed, but the aliasing effects isn't not the problem here, as it > should be visible in the input image as well. I'd expect a > third-order spline interpolation to be more smooth than a first-order > interpolant, but in the resulting images this isn't the case. > > See > > http://mentat.za.net/results/lena_small.png > http://mentat.za.net/results/img_rot_30_1.png (1st order spline) > http://mentat.za.net/results/img_rot_30_3.png (3rd order spline) > >> Lena has been decimated (reduced in size) prior to the rotation. That >> is definitely a good way to get artefacts, unless an anti-aliasing >> filter is applied before shrinking the image. My impression is that >> this image is probably somewhat undersampled (to understand exactly >> what that means, read up on the Sampling Theorem). > > The artefacts arn't visible in the source image (url above). The > image definately is a scaled down version of the original Lena -- very > interesting, btw, see > > http://www.cs.cmu.edu/~chuck/lennapg/lenna.shtml > >> investigating it further. One experiment might be to blur the >> original >> Lena with a Gaussian whose sigma is 1 pixel of the shrunken image >> before actually shrinking her, then do the rotation. > > A rotation should take place without significant shifts in colour. > This almost looks like a value overflow problem. > >> So I do wonder if the algorithm in nd_image is making this worse >> than it needs to be. > > That is my suspicion, too. > >> compare the results? I just tried doing a similar rotation in >> PyRAF on >> a monochrome image with a bicubic spline, and see considerably >> smaller >> artefacts (just a compact overshoot of probably a few % at the >> edge). > > Could you apply the PyRAF rotation on the Lena given above and post > the result? > > I always thought we could simply revert to using bilinear and bicubic > polygon interpolation (instead of spline interpolation), but now I > read on wikipedia: > > """ > In the mathematical subfield of numerical analysis, spline > interpolation is a form of interpolation where the interpolant is a > special type of piecewise polynomial called a spline. Spline > interpolation is preferred over polynomial interpolation because the > interpolation error can be made small even when using low degree > polynomials for the spline. Thus, spline interpolation avoids the > problem of Runge's phenomenon which occurs when using high degree > polynomials. > """ > > http://en.wikipedia.org/wiki/Spline_interpolation > > also take a look at > > http://en.wikipedia.org/wiki/Runge%27s_phenomenon > > So much for side-stepping Runge's phenomenon :) > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From charlesr.harris at gmail.com Sat Mar 24 18:29:51 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 16:29:51 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: <46058B00.8000105@ee.byu.edu> References: <460583F3.9030002@ee.byu.edu> <46058B00.8000105@ee.byu.edu> Message-ID: On 3/24/07, Travis Oliphant wrote: > > Alan G Isaac wrote: > > On Sat, 24 Mar 2007, Travis Oliphant apparently wrote: > > > >> My opinion is that a 1-d array in matrix-multiplication > >> should always be interpreted as a row vector. Is this not > >> what is currently done? If not, then it is a bug in my > >> mind. > >> > > > > > >>>> N.__version__ > >>>> > > '1.0' > > > >>>> I > >>>> > > matrix([[ 1., 0.], > > [ 0., 1.]]) > > > >>>> I*N.ones(2) > >>>> > > matrix([[ 1., 1.]]) > > > > If N.ones(2) were treated as a row vector, > > matrix multiplication is not possible. > > So the question is what should happen. > > I would like an exception raised. > > The current behavior is lmost certainly not desirable, > > although it has the virute of matching ``dot``. > > > > > I think that's why it is the current behavior --- by default rather than > by design. But, dot has different use cases. > > I'd be fine with an error raised on matrix multiplication (as long as > dot is not changed). In other words, I'd like to see 1-d arrays always > interpreted the same way (as row vectors) when used in matrix > multiplication. The relevant bit of code is def __mul__(self, other): if isinstance(other, N.ndarray) or N.isscalar(other) or \ not hasattr(other, '__rmul__'): return N.dot(self, other) else: return NotImplemented How about I just replace N.dot(self, other) by N.dot(self, asmatrix(other)) or maybe something like try : rhs = asmatrix(other) val = dot(self, rhs) except : val = NotImplemented return val It is not clear to me why the check for __rmul__ is included in the original code, as I believe it causes problems when the rhs is a list, i.e. mat(eye(2))*[1,1] currently fails. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sat Mar 24 19:10:12 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sat, 24 Mar 2007 19:10:12 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: <460583F3.9030002@ee.byu.edu> Message-ID: On Sat, 24 Mar 2007, Anne Archibald apparently wrote: > Note that taking a vector and left-multiplying it by > a matrix is a very natural operation that won't work any > more if you treat all vectors as if they were row vectors. Can you be more specific on this "naturalness"? What is the cost of being explicit (in actual practice)? I am inclined to claim that this seems "natural" to us only when we abuse language and forget that our column vectors and row vectors are in fact matrices with two dimensions/axes. In what situation does a need or major convenience from being so implicit arise? Thank you, Alan Isaac From stefan at sun.ac.za Sat Mar 24 19:09:47 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 25 Mar 2007 01:09:47 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: References: <4604BA01.5040606@gemini.edu> <20070324205854.GZ6085@mentat.za.net> Message-ID: <20070324230947.GA6085@mentat.za.net> On Sat, Mar 24, 2007 at 03:25:38PM -0700, Zachary Pincus wrote: > If Lena is converted to floating-point before the rotation is > applied, and then the intensity range is clipped to [0,255] and > converted back to uint8 before saving, everything looks fine. Thanks, Zachary! I can confirm that. > So, is this a bug? Well, I still think so. Given that small ringing > is going to happen on all but the very smoothest images, and given > that ndimage is going to be used on non-floating-point types, it > would be good if there were some explicit internal clipping to the > data type's range. Otherwise, the ndimage resampling tools are unfit > for use on non-floating-point data that resides near the edges of the > range of the data type. I agree. > Though I'm not quite sure how one would structure the calculations so > that it would be possible to tell when over/underflow happened... it > might not be possible. In which case, either the tools should use > floating-point math at some of the steps internally (as few as > possible) before clipping and converting to the required data type, > or explicit warnings should be added to the documentation. I think the spline interpolation already uses floating point math? Looks like we are seeing a type conversion without range checking: In [47]: x = N.array([1.2,200,255,255.6,270]) In [48]: x.astype(N.uint8) Out[48]: array([ 1, 200, 255, 255, 14], dtype=uint8) I'll have to go and take a look at the code, but this shouldn't be hard to fix -- clipping is fairly fast (don't we have a fast clipping method by David C now?), even for extremely large images. Regards St?fan From charlesr.harris at gmail.com Sat Mar 24 19:59:55 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 17:59:55 -0600 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: <460583F3.9030002@ee.byu.edu> Message-ID: On 3/24/07, Alan G Isaac wrote: > > On Sat, 24 Mar 2007, Anne Archibald apparently wrote: > > Note that taking a vector and left-multiplying it by > > a matrix is a very natural operation that won't work any > > more if you treat all vectors as if they were row vectors. > > Can you be more specific on this "naturalness"? > What is the cost of being explicit (in actual practice)? > > I am inclined to claim that this seems "natural" to us only > when we abuse language and forget that our column vectors > and row vectors are in fact matrices with two dimensions/axes. > > In what situation does a need or major convenience > from being so implicit arise? I committed a change that converts nonscalars to matrices before calling dot. The following now raises an alignment error: mat(eye(2))*ones(2) On the other hand mat(eye(2))*[[1],[1]] now returns the expected (by me) result. Scream loudly if any of this bothers you. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jturner at gemini.edu Sat Mar 24 20:31:43 2007 From: jturner at gemini.edu (James Turner) Date: Sat, 24 Mar 2007 20:31:43 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> References: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> Message-ID: <4605C2EF.6000301@gemini.edu> Hi Zach, > Based on my reading of the two excellent Unser papers (both the one > that ndimage's spline code is based on, and the one that Travis > linked to), it seems like a major point of using splines for > interpolation is *better* behavior in the case of non-band-limited > data than the traditional 'anti-aliasing [e.g. lowpass] filter' + > 'resampling' + 'reconstruction [e.g. lowpass] filter' procedure. It's certainly true that intermediate-order spline interpolants will cause less ringing than an "ideal" sinc function. So their behaviour is better for non-band-limited data than applying simplistic formulae derived from the Sampling Theorem. This fact would help you out if you don't use a low-pass filter. However, I wouldn't go as far as to say that splines *replace* some form of low-pass filtering. I haven't read Unser's papers in much detail, though (at least not recently) and my applications are different from yours; it may depend exactly what you're trying to do. > So it seems that something is wrong if the spline interpolation > tools in ndimage only work properly in the band-limited case, or > require lowpass filtering. It depends what you mean by working properly -- but in this case it does look like something is wrong and that you figured it out in your next post :-). James. From charlesr.harris at gmail.com Sat Mar 24 20:50:42 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 18:50:42 -0600 Subject: [Numpy-discussion] Is a string a scalar type? Message-ID: In [10]: isscalar('hello world') Out[10]: True Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Mar 24 20:54:28 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 24 Mar 2007 19:54:28 -0500 Subject: [Numpy-discussion] Is a string a scalar type? In-Reply-To: References: Message-ID: <4605C844.1050004@gmail.com> Charles R Harris wrote: > In [10]: isscalar('hello world') > Out[10]: True As far as numpy is concerned, they are. >>> from numpy import * >>> array('hello world') array('hello world', dtype='|S11') -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sat Mar 24 20:56:33 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 24 Mar 2007 19:56:33 -0500 Subject: [Numpy-discussion] Is a string a scalar type? In-Reply-To: References: Message-ID: <4605C8C1.1090703@gmail.com> Charles R Harris wrote: > In [10]: isscalar('hello world') > Out[10]: True I would say that, intrinsically, yes, strings are constructed as sequences of other things. However, essentially every use case I have for *testing* whether or not something is a sequence (or inversely, a scalar), I want strings to answer as "not a sequence". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wfspotz at sandia.gov Sat Mar 24 21:15:34 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Sat, 24 Mar 2007 19:15:34 -0600 Subject: [Numpy-discussion] swig svn commits In-Reply-To: References: <8561377F-2E52-4BFC-A487-15024612D3DD@sandia.gov> Message-ID: <38A9B5A2-52A7-4FC8-AE50-7154FD53027E@sandia.gov> OK. So if I understand this correctly, PyArray_EquivTypenums() can't do native byte order checking. But I'm assuming PyArray_EquivTypes() could. Unfortunately, the numpy.i typemaps are based on the enumerated typenums, not PyArray_Descr objects. There are a couple of other places where the more sophisticated PyArray_Descr approach would be beneficial, such as for bools and complex types (as well as a bunch of others, I'm assuming), so I'm open to it, but it would take some work. On Mar 24, 2007, at 12:39 PM, Sebastian Haase wrote: > On 3/24/07, Bill Spotz wrote: >> No, I don't consider native byte order. What was your solution? >> > I think there is only one solution: > If somebody requests INPLACE handling but provides data of > wrong byteorder, I have to throw an exception -- in SWIG terms, the > typecheck returns False (if I remember right). > > For me this is just a "last-line of defence" - meaning that I have > most of my functions wrapped by another level of python "convenience" > functions, and those take care of type and byte-order issues > beforehand as needed. > > -Sebastian > > >> The typecheck typemap is so that swig can perform isolated type >> checking when it is creating dispatch code for overloaded functions. >> The typechecks I added for INPLACE arrays require that the argument >> be a numpy array and that PyArray_EquivTypenums() return true for the >> provided and requested types. >> >> On Mar 23, 2007, at 3:03 PM, Sebastian Haase wrote: >> >>> Hi, >>> this is regarding the svn commit by wfspotz. >>> >>> Author: wfspotz at sandia.gov >>> Date: 2007-03-23 13:04:37 -0500 (Fri, 23 Mar 2007) >>> New Revision: 3593 >>> >>> Modified: >>> trunk/numpy/doc/swig/numpy.i >>> Log: >>> Added typecheck typemaps to INPLACE_ARRAY typemap suites >>> >>> Hi wfspotz, >>> I was just wondering if you consider checking for "native byte >>> order" >>> as part of your inplace-typemap. >>> I found that to be a problem in my SWIG type maps >>> that I hi-jacked / boiled down from the older numpy swig files. >>> (They are mostly for 3D image data, only for a small number of >>> types) >>> >>> -Sebastian >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at scipy.org >>> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> >> ** Bill Spotz ** >> ** Sandia National Laboratories Voice: (505)845-0170 ** >> ** P.O. Box 5800 Fax: (505)284-5451 ** >> ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** >> >> >> >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at scipy.org >> http://projects.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From jturner at gemini.edu Sat Mar 24 21:16:38 2007 From: jturner at gemini.edu (James Turner) Date: Sat, 24 Mar 2007 21:16:38 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070324205854.GZ6085@mentat.za.net> References: <20070324205854.GZ6085@mentat.za.net> Message-ID: <4605CD76.6070108@gemini.edu> Hi St?fan, > Agreed, but the aliasing effects isn't not the problem here, as it > should be visible in the input image as well. It's a bit academic now that Zach seems to have found the answer, but I don't think this is true. Aliasing is *present* in the input image, but is simply manifested as blockiness (due to the larger pixel size relative to the scene). It is only once you interpolate the data in a more general way that the aliasing turns into other artefacts. The reduced image contains too few sample points to determine the frequency content of the underlying scene uniquely for interpolating, but the values still *have* to be meaningful until you resample in some other way, because all you did was average over contiguous areas. > The image definately is a scaled down version of the original Lena > -- very interesting, btw, see Cool. I wondered where the picture came from (I suppose I could have looked that up in wikipedia too). The full frame is also nice :-). > A rotation should take place without significant shifts in colour. > This almost looks like a value overflow problem. Sounds like this idea was correct. I was also just starting to form some suspicion about the integers getting maxed out... > Could you apply the PyRAF rotation on the Lena given above and post > the result? I'd be happy to, but is the problem solved now? I can't handle colour images directly in IRAF/PyRAF, but I could split the 3 channels into separate images, rotate each of them and put them back together. Do I need to use something like PIL to read your PNG into numarray? I usually use FITS files... Let me know if you still want to try this. > I always thought we could simply revert to using bilinear and > bicubic polygon interpolation (instead of spline interpolation) Interesting. In my brief look at your paper and Web page (so far), I had missed that your polygon interpolation can be bicubic, rather than just linear like Drizzle. I'll have to figure out what this means and whether it might be useful for what I'm doing. Cheers, James. From charlesr.harris at gmail.com Sat Mar 24 21:23:48 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 19:23:48 -0600 Subject: [Numpy-discussion] Is a string a scalar type? In-Reply-To: <4605C8C1.1090703@gmail.com> References: <4605C8C1.1090703@gmail.com> Message-ID: On 3/24/07, Robert Kern wrote: > > Charles R Harris wrote: > > In [10]: isscalar('hello world') > > Out[10]: True > > I would say that, intrinsically, yes, strings are constructed as sequences > of > other things. However, essentially every use case I have for *testing* > whether > or not something is a sequence (or inversely, a scalar), I want strings to > answer as "not a sequence". I see 'is_string_like', 'iscomplex', 'iscomplexobj', 'isfinite', 'isfortran', 'ishold', 'isinf', 'isinteractive', 'isnan', 'isneginf', 'isposinf', 'ispower2', 'isreal', 'isrealobj', 'isscalar', 'issctype', 'issubclass_', 'issubdtype', 'issubsctype', But apparently none of them test for a scalar number. That is a strange functionality to be missing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Mar 24 21:27:31 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Mar 2007 19:27:31 -0600 Subject: [Numpy-discussion] Is a string a scalar type? In-Reply-To: <4605C844.1050004@gmail.com> References: <4605C844.1050004@gmail.com> Message-ID: On 3/24/07, Robert Kern wrote: > > Charles R Harris wrote: > > In [10]: isscalar('hello world') > > Out[10]: True > > As far as numpy is concerned, they are. > > >>> from numpy import * > >>> array('hello world') > array('hello world', > dtype='|S11') Dunno, by that reasoning objects could also be considered scalars as they can be elements of object arrays. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Sun Mar 25 03:41:44 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Sun, 25 Mar 2007 09:41:44 +0200 Subject: [Numpy-discussion] defmatrix.py Message-ID: Hi, Several tests didn't pass due to the recent changes in defmatrix.py. Nils ====================================================================== ERROR: check_matmat (scipy.sparse.tests.test_sparse.test_csc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", line 142, in check_matmat assert_array_almost_equal((a*bsp).todense(), a*b) File "/usr/local/lib64/python2.5/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_rmatvec (scipy.sparse.tests.test_sparse.test_csc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", line 107, in check_rmatvec assert_array_almost_equal(row*M, row*M.todense()) File "/usr/local/lib64/python2.5/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_matmat (scipy.sparse.tests.test_sparse.test_csr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib64/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", line 142, in check_matmat assert_array_almost_equal((a*bsp).todense(), a*b) File "/usr/local/lib64/python2.5/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_rmatvec (scipy.sparse.tests.test_sparse.test_csr) ... From curiousjan at gmail.com Sun Mar 25 05:03:17 2007 From: curiousjan at gmail.com (Jan Strube) Date: Sun, 25 Mar 2007 02:03:17 -0700 Subject: [Numpy-discussion] array multiplication with different dtypes Message-ID: I'm having a difficult time understanding the following behavior: import numpy as N # create a new array 4 rows, 3 columns x = N.random.random((4, 3)) # elementwise multiplication x*x newtype = N.dtype([('x', N.float64), ('y', N.float64), ('z', N.float64)]) # interpret the array as an array of cartesian coordinates x.dtype = newtype x*x --> TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and ' numpy.ndarray' N.__version__ '1.0.2.dev3498' So just by assigning names to the columns, I can't multiply the arrays any more? Numpy itself still claims that x is an ndarray. I hope this is a bug, because it would be really convenient if this still worked after assigning names to columns. What I really wanted to do was N.dot(x, x.T) to get the length of the cartesian vectors, but (I think) this fails for the same reason. Please tell me this is a bug ;-) Cheers, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From curiousjan at gmail.com Sun Mar 25 07:09:11 2007 From: curiousjan at gmail.com (Jan Strube) Date: Sun, 25 Mar 2007 04:09:11 -0700 Subject: [Numpy-discussion] dtype confusion Message-ID: There seems to be a fundamental lack of understanding on my behalf when it comes to dtypes and record arrays. Please consider the following snippet: import numpy as N newtype = N.dtype([('x', N.float64), ('y', N.float64), ('z', N.float64)]) a = N.random.random((100,3)) a.dtype=newtype b = N.column_stack([a['x'].ravel(), a['y'].ravel(), a['z'].ravel()]) b.dtype = newtype --> ValueError: new type not compatible with array. I don't understand two things about this: i) the shape of a changes from (100,3) to (100,1) after assigning the dtype. ii) the shape of b is obviously (100,3), so why can't I assign the new dtype? Could someone please point out the flaws in this approach? Thanks much, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sun Mar 25 08:25:11 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sun, 25 Mar 2007 14:25:11 +0200 Subject: [Numpy-discussion] dtype confusion In-Reply-To: References: Message-ID: <20070325122511.GL6085@mentat.za.net> On Sun, Mar 25, 2007 at 04:09:11AM -0700, Jan Strube wrote: > There seems to be a fundamental lack of understanding on my behalf when it > comes to dtypes and record arrays. > Please consider the following snippet: > > import numpy as N > newtype = N.dtype([('x', N.float64 ), ('y', N.float64), ('z', N.float64)]) > a = N.random.random((100,3)) > a.dtype=newtype > b = N.column_stack([a['x'].ravel(), a['y'].ravel(), a['z'].ravel()]) > b.dtype = newtype > --> ValueError: new type not compatible with array. > > I don't understand two things about this: > i) the shape of a changes from (100,3) to (100,1) after assigning > the dtype. Every group of three floats now become one element in the new array, i.e. float64 -> (float64,float64,float64) > ii) the shape of b is obviously (100,3), so why can't I assign the > new dtype? The array is no longer a contiguous block of memory, so the new dtype can't be applied: In [23]: b.flags Out[23]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False The following does a copy of the array to contiguous memory: In [24]: N.ascontiguousarray(b).dtype = newtype If you want to move back to the original view you can do b.view(N.float64) Regards St?fan From cjw at sympatico.ca Sun Mar 25 08:48:30 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 08:48:30 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: Alan G Isaac wrote: > On Sat, 24 Mar 2007, Charles R Harris apparently wrote: >> Yes, that is what I am thinking. Given that there are only the two >> possibilities, row or column, choose the only one that is compatible with >> the multiplying matrix. The result will not always be a column vector, for >> instance, mat([[1]])*ones(3) will be a 1x3 row vector. > > > > Ack! The simple rule `post multiply means its a column vector` > would be horrible enough: A*ones(n)*B becomes utterly obscure. > Now even that simple rule is to be violated?? It depends whether ones delivers an instance of the Matrix/vector class or a simple array. I assume that, in the above A and B represent matrices. Colin W. > > Down this path lies madness. > Please, just raise an exception. > > Cheers, > Alan Isaac From cjw at sympatico.ca Sun Mar 25 09:56:09 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 09:56:09 -0400 Subject: [Numpy-discussion] Detect subclass of ndarray In-Reply-To: References: Message-ID: Colin J. Williams wrote: > Alan G Isaac wrote: >> On Sat, 24 Mar 2007, Charles R Harris apparently wrote: >>> Yes, that is what I am thinking. Given that there are only the two >>> possibilities, row or column, choose the only one that is compatible with >>> the multiplying matrix. The result will not always be a column vector, for >>> instance, mat([[1]])*ones(3) will be a 1x3 row vector. >> >> >> Ack! The simple rule `post multiply means its a column vector` >> would be horrible enough: A*ones(n)*B becomes utterly obscure. >> Now even that simple rule is to be violated?? > > It depends whether ones delivers an instance of the Matrix/vector class > or a simple array. > > I assume that, in the above A and B represent matrices. > > Colin W. Postscript: I hadn't read the later postings when I posted the above. PyMatrix used the convention mentioned in an earlier posting. Simply a vector is considered as a single row matrix or a single column matrix. This same approach can largely be used with numpy's mat: *** Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32. *** >>> import numpy as _n >>> _n.ones(3) array([ 1., 1., 1.]) >>> a= _n.ones(3) >>> a.T array([ 1., 1., 1.]) >>> _n.mat(a) matrix([[ 1., 1., 1.]]) >>> _n.mat(a).T matrix([[ 1.], [ 1.], [ 1.]]) >>> b= _n.mat(a).T >>> a * b matrix([[ 3.]]) # Something has gone wrong here - it looks as though there is normalization under the counter. >>> In any event, the problem posed by Alan Isaac can be handled with this approach: A * mat(ones(3)).t * B can produce the desired result. I haven't tested it. Colin W. >> Down this path lies madness. >> Please, just raise an exception. >> >> Cheers, >> Alan Isaac From cjw at sympatico.ca Sun Mar 25 10:02:00 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 10:02:00 -0400 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46058EBC.90804@gmail.com> Message-ID: Bill Baxter wrote: > On 3/25/07, Robert Kern wrote: >> Bill Baxter wrote: > >> I don't know. Given our previous history with convenience functions with >> different calling semantics (anyone remember rand()?), I think it probably will >> confuse some people. >> >> I'd really like to see it on a cookbook page, though. I'd use it. > > Done. > http://www.scipy.org/Cookbook/MultiDot > > --bb I wasn't able to connect to this link but I gather that the proposal was to used dot(A, B, C) to represent the product of the 3 arrays. if A, B and C were matrices then this could more clearly be written as A * B * C Colin W. From cjw at sympatico.ca Sun Mar 25 10:12:54 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 10:12:54 -0400 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <4605801C.6030300@ee.byu.edu> Message-ID: Charles R Harris wrote: > > > On 3/24/07, *Travis Oliphant* > wrote: > > Every so often the idea of new operators comes up because of the need to > do both "matrix-multiplication" and element-by-element multiplication. > > I think this is one area where the current Python approach is not as > nice because we have a limited set of operators to work with. > > One thing I wonder is if we are being vocal enough with the Python 3000 > crowd to try and get additional operators into the language itself. > > What if we could get a few new operators into the language to help us. > If we don't ask for it, it certainly won't happen. > My experience is that the difficulty of using the '*' operator for both > matrix multiplication and element-by-element multiplication > depending on > the class of the object is not especially robust. It makes it harder to > write generic code, and we still haven't gotten everything completely > right. > > It is somewhat workable as it stands, but I think it would be nicer if > we could have some "meta" operator that allowed an alternative > definition of major operators. Something like @* for example (just > picking a character that is already used for decorators). > > > Yes indeed, this is an old complaint. Just having an infix operator > would be an improvement: > > A dot B dot C > > Not that I am suggesting dot in this regard ;) In particular, it > wouldn't parse without spaces. What about division? Matlab has both / > and \ for left and right matrix division and something like that could > call solve instead of inverse, leading to some efficiencies. Yes, thanks to a suggestion from Alan Isaac, this was implemented in PyMatrix (based on numarray and not yet converted to numpy). / served for one and // for the other. Regarding an additional multiply operator, I don't see the need for it. A matrix and and array are similar dut different animals. Colin W. We also > have both dot and tensordot, which raises the problem of interpretation > when ndim > 2. > > Chuck > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From davidnovakovic at gmail.com Sun Mar 25 11:08:41 2007 From: davidnovakovic at gmail.com (dpn) Date: Mon, 26 Mar 2007 01:08:41 +1000 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <4605801C.6030300@ee.byu.edu> Message-ID: <59d13e7d0703250808l54d78234g8c2a3aeafc40af1e@mail.gmail.com> With the possible inclusion of generic functions in py3k I dont really see the point of adding more operators. (While i do miss mat1 x mat2 from PDL). mat3 = mat1.mm(mat2) or the like seems to be sufficient. I find matrix multiplication annoying in the case of SVD reconstruction: final = matrixmultiply(matrixmultiply(u, s), v) A minor gripe though. I'm more inclined to err on the side of clean python rather than more operators, especially since aforementioned GF's will help with many cases dpn On 3/26/07, Colin J. Williams wrote: > Charles R Harris wrote: > > > > > > On 3/24/07, *Travis Oliphant* > > wrote: > > > > Every so often the idea of new operators comes up because of the need to > > do both "matrix-multiplication" and element-by-element multiplication. > > > > I think this is one area where the current Python approach is not as > > nice because we have a limited set of operators to work with. > > > > One thing I wonder is if we are being vocal enough with the Python 3000 > > crowd to try and get additional operators into the language itself. > > > > What if we could get a few new operators into the language to help us. > > If we don't ask for it, it certainly won't happen. > > My experience is that the difficulty of using the '*' operator for both > > matrix multiplication and element-by-element multiplication > > depending on > > the class of the object is not especially robust. It makes it harder to > > write generic code, and we still haven't gotten everything completely > > right. > > > > It is somewhat workable as it stands, but I think it would be nicer if > > we could have some "meta" operator that allowed an alternative > > definition of major operators. Something like @* for example (just > > picking a character that is already used for decorators). > > > > > > Yes indeed, this is an old complaint. Just having an infix operator > > would be an improvement: > > > > A dot B dot C > > > > Not that I am suggesting dot in this regard ;) In particular, it > > wouldn't parse without spaces. What about division? Matlab has both / > > and \ for left and right matrix division and something like that could > > call solve instead of inverse, leading to some efficiencies. > > Yes, thanks to a suggestion from Alan Isaac, this was implemented in > PyMatrix (based on numarray and not yet converted to numpy). / served > for one and // for the other. > > Regarding an additional multiply operator, I don't see the need for it. > A matrix and and array are similar dut different animals. > > Colin W. > > We also > > have both dot and tensordot, which raises the problem of interpretation > > when ndim > 2. > > > > Chuck > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From pjssilva at ime.usp.br Sun Mar 25 11:18:13 2007 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Sun, 25 Mar 2007 12:18:13 -0300 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <59d13e7d0703250808l54d78234g8c2a3aeafc40af1e@mail.gmail.com> References: <4605801C.6030300@ee.byu.edu> <59d13e7d0703250808l54d78234g8c2a3aeafc40af1e@mail.gmail.com> Message-ID: <1174835893.6310.7.camel@localhost.localdomain> Em Seg, 2007-03-26 ?s 01:08 +1000, dpn escreveu: > With the possible inclusion of generic functions in py3k I dont really > see the point of adding more operators. (While i do miss mat1 x mat2 > from PDL). > > mat3 = mat1.mm(mat2) or the like seems to be sufficient. > > I find matrix multiplication annoying in the case of SVD reconstruction: > > final = matrixmultiply(matrixmultiply(u, s), v) > Matrix multiplication is just too common in numerical linear algebra, one of the main areas for numpy/scipy. Even though I can get used with using dot (or matrixmultiply) to do it, I can easily see the benefit of having a special operator here. This will be beneficial for mathematicians that use numpy/scipy to prototype some ideas or to newcomers from Matlab. Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call practice) From aisaac at american.edu Sun Mar 25 11:33:02 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 11:33:02 -0400 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <1174835893.6310.7.camel@localhost.localdomain> References: <4605801C.6030300@ee.byu.edu><59d13e7d0703250808l54d78234g8c2a3aeafc40af1e@mail.gmail.com><1174835893.6310.7.camel@localhost.localdomain> Message-ID: On Sun, 25 Mar 2007, Paulo Jose da Silva e Silva apparently wrote: > Even though I can get used with using dot (or > matrixmultiply) to do it, I can easily see the benefit of > having a special operator here. Some newer languages are taking advantage of the expressiveness of Unicode for math purposes. I believe somone on this list asked whether this might provide a way forward. Can it (taking a long view)? For example, it would be natural to use as operators the Unicode multiplication sign U+00D7 (?) and the Unicode dot operator U+22C5 (?). Cheers, Alan Isaac From oliphant at ee.byu.edu Sun Mar 25 12:08:27 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun, 25 Mar 2007 10:08:27 -0600 Subject: [Numpy-discussion] array multiplication with different dtypes In-Reply-To: References: Message-ID: <46069E7B.4000600@ee.byu.edu> Jan Strube wrote: > I'm having a difficult time understanding the following behavior: > > import numpy as N > # create a new array 4 rows, 3 columns > x = N.random.random((4, 3)) > # elementwise multiplication > x*x > > newtype = N.dtype([('x', N.float64), ('y', N.float64), ('z', N.float64)]) > > # interpret the array as an array of cartesian coordinates > x.dtype = newtype > > x*x > > --> TypeError: unsupported operand type(s) for *: ' numpy.ndarray' and > 'numpy.ndarray' > > N.__version__ > '1.0.2.dev3498' > > So just by assigning names to the columns, I can't multiply the arrays > any more? No, it's not a bug. It's a "missing feature"." You have created a "record-array". All of the ufuncs are undefined for arrays with fields (what is sometimes called a variable item-size array). They are undefined for two reasons 1) It's not clear how to define them. It is ambiguous in general. For this specific case, it is probably clear what you want, but what do you want for the general case data-type with fields defined. This has never been thought out clearly. 2) It's not trivial to implement. Basically, handling the general case would require some alterations to the main ufunc code loops in order to pass information about the size of the array element that is presently not available. Perhaps some day we will be able to add element-by-element ufuncs for record arrays that will basically recurse through the fields, but that will require some coding effort that is not on my radar for the next while. What you can do, is maintain a view of the data as a 4x3 array of floats and do the multiplication with that array. The same memory can then also be viewed as a length 4 1-d array of coordinates if you like. You can also be more explicit about what you want to do with each column by writing y = x.copy() y['x'] = x['x']*x['x'] y['y'] = x['y']*x['y'] y['z'] = x['z']*x['z'] > > Please tell me this is a bug ;-) Sorry, it's not that easy. -Travis From aisaac at american.edu Sun Mar 25 13:07:25 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 13:07:25 -0400 Subject: [Numpy-discussion] matrix indexing question Message-ID: One thing keeps bugging me when I use numpy.matrix. All this is fine:: >>> x=N.mat('1 1;1 0') >>> x matrix([[1, 1], [1, 0]]) >>> x[1,:] matrix([[1, 0]]) But it seems to me that I should be able to extract a matrix row as an array. So this :: >>> x[1] matrix([[1, 0]]) feels wrong. (Similarly when iterating across rows.) Of course I realize that I can just :: >>> x.A[1] array([1, 0]) but since the above keeps feeling wrong I felt I should raise this as a possible design issue, better discussed early than latter. Cheers, Alan Isaac From daniel.wheeler at nist.gov Sun Mar 25 13:35:38 2007 From: daniel.wheeler at nist.gov (Daniel Wheeler) Date: Sun, 25 Mar 2007 17:35:38 +0000 Subject: [Numpy-discussion] header file location upon installation Message-ID: Hi, Should the header files in .../lib/python2.4/site-packages/python/numpy/core/include/ be copied to .../include/python2.4/numpy/ upon installation of numpy? As far as I can tell this is not happening. Just wondering what the default behavior should be. Thanks -- Daniel Wheeler -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Sun Mar 25 13:48:44 2007 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Sun, 25 Mar 2007 14:48:44 -0300 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: <1174844924.6310.27.camel@localhost.localdomain> Em Dom, 2007-03-25 ?s 13:07 -0400, Alan G Isaac escreveu: > So this :: > > >>> x[1] > matrix([[1, 0]]) > > feels wrong. (Similarly when iterating across rows.) > Of course I realize that I can just :: > > >>> x.A[1] > array([1, 0]) > > but since the above keeps feeling wrong I felt I should > raise this as a possible design issue, better discussed > early than latter. I think the point here is that if you are using matrices, then all you "should" want are matrices, just like in MATLAB: >> A = [1 2; 3 4] A = 1 2 3 4 >> b = A(1, :) b = 1 2 >> size(b) ans = 1 2 >> b = A(:, 1) b = 1 3 >> size(b) ans = 2 1 >> b = 1 b = 1 >> size(b) ans = 1 1 You see, rows, columnes, and even numbers, are treated as matrices. Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call practice) From aisaac at american.edu Sun Mar 25 14:39:03 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 14:39:03 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <1174844924.6310.27.camel@localhost.localdomain> References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: > Em Dom, 2007-03-25 ?s 13:07 -0400, Alan G Isaac escreveu: >> >>> x[1] >> matrix([[1, 0]]) >> feels wrong. (Similarly when iterating across rows.) On Sun, 25 Mar 2007, Paulo Jose da Silva e Silva apparently wrote: > I think the point here is that if you are using matrices, > then all you "should" want are matrices, just like in > MATLAB: > >> b = A(1, :) > b = > 1 2 Yes, that is the idea behind this, which I am also accustomed to from GAUSS. But note again that the Matlab equivalent :: >>> x=N.mat('1 2;3 4') >>> x[0,:] matrix([[1, 2]]) does provide this behavior. The question I am raising is a design question and is I think really not addressed by the rule of thumb you offer. Specifically, that rule of thumb if it is indeed the justification of :: >>> x[1] matrix([[3, 4]]) finds itself in basic conflict with the idea that I ought to be able to iterate over the objects in an iterable container. I mean really, does this not "feel" wrong? :: >>> for item in x: print item.__repr__() ... matrix([[1, 2]]) matrix([[3, 4]]) Cheers, Alan Isaac From chanley at stsci.edu Sun Mar 25 14:42:04 2007 From: chanley at stsci.edu (Christopher Hanley) Date: Sun, 25 Mar 2007 14:42:04 -0400 Subject: [Numpy-discussion] subversion site down Message-ID: <4606C27C.9030802@stsci.edu> Hi, It appears that the subversion server is down for numpy. Chris From jh at physics.ucf.edu Sun Mar 25 14:46:57 2007 From: jh at physics.ucf.edu (Joe Harrington) Date: Sun, 25 Mar 2007 14:46:57 -0400 (EDT) Subject: [Numpy-discussion] New Operators in Python Message-ID: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> I strongly agree with the need for different infix operators for matrix and array multiplication. In IDL, I have used both in the same code. IDL uses * for array and # for matrix (hey, # even *looks* like a matrix, very intuitive). It uses ## for right-multiply (why people don't just M-t the variable names I don't know, though maybe people who do more linear algebra than I can say). There's an important justification that hasn't come up yet, and that is advocacy. Languages are good or not good depending on how readable and writeable they are for various purposes. LISP is a technically very powerful language. It was object-oriented a zillion years before C++. It had and has powerful proponents within CS, such as Stallman. But, whenever CS people try to push LISP outside their ivory tower, it gets trashed. The LISP dialect Guile was written as the GNU project's extension language and nobody used it. Now projects like GIMP have two extension languages, Guile and Python. Why? LISP is great for analyzing the flow and structure of computer code, but LISP math does not look like real math, which makes it very difficult and counterintuitive to use for math or math-heavy topics like graphics. It has little syntactic sugar to make common math tasks either quick to write or easy to understand or verify, until you've retrained your brain. So, even though there are tons of technical people exposed to LISP, nobody outside CS adopts it who doesn't have to. On the other hand, Python, IDL, and Matlab are attractive to us mainly because their syntaxes allow us to see the math, understand it on inspection, and verify its correctness. The math we write in these languages looks as much like the math we do on paper as ASCII will allow. (Obviously, we also choose them because we don't like writing loops and declaring variables.) So, whenever I hear someone suggest using a functional syntax for a concept that exists notationally, I cringe. We're alienating a class of users each time we do that. Those are people who will never come to Python. There are extremes to which this argument cannot go - a prime will never be a derivative because quotes are more important - but I think that matrix multiplication is a no-brainer here. We should let the Python developers know we want it now and then follow up with a syntax and implementation. --jh-- Prof. Joseph Harrington Department of Physics MAP 420 University of Central Florida Orlando, FL 32816-2385 (407) 823-3416 voice (407) 823-5112 fax (407) 823-2325 physics office jh at physics.ucf.edu (direct) jh at alum.mit.edu (permanent forwarding address, will work forever) From curiousjan at gmail.com Sun Mar 25 16:11:22 2007 From: curiousjan at gmail.com (Jan Strube) Date: Sun, 25 Mar 2007 13:11:22 -0700 Subject: [Numpy-discussion] dtype confusion Message-ID: Thanks Stefan and Travis for their explanations. First a request: Could both explanations be added to the manual, please? Thanks. So the problem I was having was that I thought this difference in behavior would be caused by two different types: recarray and ndarray. I feel that there is still some underlying behavior that I don't understand, but for now the puzzling thing was that I thought that the new dtype simply assigns names and aliases varname[:,0] to varname['x'], varname[:,1] to varname['y'], and so on. I didn't expect it to fundamentally change the array in that way, I would have expected varname.view(recarray) to do that. Would it make sense for me then to create something like a "namedarray" subclass of ndarray that doesn't change the dtype but simply let's you provide names and assign them to the columns? Maybe in my case it's easier, because all columns of the record array have the same type I guess I'll re-read the dtypes and recarray parts of the manual now... Thanks, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 25 16:23:23 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 25 Mar 2007 14:23:23 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: Message-ID: On 3/25/07, Nils Wagner wrote: > > Hi, > > Several tests didn't pass due to the recent changes > in defmatrix.py. > > Nils > > ====================================================================== > ERROR: check_matmat > (scipy.sparse.tests.test_sparse.test_csc) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/usr/local/lib64/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", > line 142, in check_matmat > assert_array_almost_equal((a*bsp).todense(), a*b) > File > "/usr/local/lib64/python2.5/site-packages/numpy/core/defmatrix.py", > line 162, in __mul__ > return N.dot(self, other) > ValueError: objects are not aligned > > ====================================================================== > ERROR: check_rmatvec > (scipy.sparse.tests.test_sparse.test_csc) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/usr/local/lib64/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", > line 107, in check_rmatvec > assert_array_almost_equal(row*M, row*M.todense()) > File > "/usr/local/lib64/python2.5/site-packages/numpy/core/defmatrix.py", > line 162, in __mul__ > return N.dot(self, other) > ValueError: objects are not aligned > > ====================================================================== > ERROR: check_matmat > (scipy.sparse.tests.test_sparse.test_csr) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/usr/local/lib64/python2.5/site-packages/scipy/sparse/tests/test_sparse.py", > line 142, in check_matmat > assert_array_almost_equal((a*bsp).todense(), a*b) > File > "/usr/local/lib64/python2.5/site-packages/numpy/core/defmatrix.py", > line 162, in __mul__ > return N.dot(self, other) > ValueError: objects are not aligned > > ====================================================================== > ERROR: check_rmatvec > (scipy.sparse.tests.test_sparse.test_csr) > > ... Looks to me like the __rmul__ operator for sparse matrices needs to be called in these cases which have in common the form matrix * sparse Right now conversion to asmatrix is called on the rhs. Hmmm, I didn't want to forgo constructs like matrix * [[1],[2]] I wonder what asmatrix(sparse) is doing as it doesn't seem to raise an error. Also, how does one call spmatrix, it doesn't seem to be a standalone function, nor do I see how it gets into the test class as the test class doesn't subclass anything? SVN seems to be down, so there is nothing I can do at the moment. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Sun Mar 25 16:28:29 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 26 Mar 2007 05:28:29 +0900 Subject: [Numpy-discussion] Simple multi-arg wrapper for dot() In-Reply-To: References: <46058EBC.90804@gmail.com> Message-ID: On 3/25/07, Colin J. Williams wrote: > Bill Baxter wrote: > > On 3/25/07, Robert Kern wrote: > >> Bill Baxter wrote: > > > >> I don't know. Given our previous history with convenience functions with > >> different calling semantics (anyone remember rand()?), I think it probably will > >> confuse some people. > >> > >> I'd really like to see it on a cookbook page, though. I'd use it. > > > > Done. > > http://www.scipy.org/Cookbook/MultiDot > > > > --bb > I wasn't able to connect to this link Scipy.org seems to be down now. > but I gather that the proposal was > to used dot(A, B, C) to represent the product of the 3 arrays. More or less, but the idea of modifying dot itself was quickly dropped. > if A, B and C were matrices then this could more clearly be written as > A * B * C Yep, if they were, that would work great. But If you're not sure if they are matrices or not you need to either use dot to make sure you get the matrix multiply behavior, or convert to matrices. Actually I think converting just the first one to a matrix suffices, so you could write it mat(A) * B * C But that won't preserve the array type of the inputs. --bb From wbaxter at gmail.com Sun Mar 25 16:44:45 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 26 Mar 2007 05:44:45 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: On 3/26/07, Alan G Isaac wrote: > > Em Dom, 2007-03-25 ?s 13:07 -0400, Alan G Isaac escreveu: > >> >>> x[1] > >> matrix([[1, 0]]) > >> feels wrong. (Similarly when iterating across rows.) > > > On Sun, 25 Mar 2007, Paulo Jose da Silva e Silva apparently wrote: > > I think the point here is that if you are using matrices, > > then all you "should" want are matrices, just like in > > MATLAB: > > >> b = A(1, :) > > b = > > 1 2 > > > Yes, that is the idea behind this, which I am also > accustomed to from GAUSS. But note again that the Matlab > equivalent :: > > >>> x=N.mat('1 2;3 4') > >>> x[0,:] > matrix([[1, 2]]) > > does provide this behavior. The question I am raising > is a design question and is I think really not addressed > by the rule of thumb you offer. Specifically, that rule > of thumb if it is indeed the justification of :: > > >>> x[1] > matrix([[3, 4]]) > > finds itself in basic conflict with the idea that I ought to > be able to iterate over the objects in an iterable container. > > I mean really, does this not "feel" wrong? :: > > >>> for item in x: print item.__repr__() > ... > matrix([[1, 2]]) > matrix([[3, 4]]) This may sound silly, but I really think seeing all those brackets is what makes it feel wrong. Matlab's output doesn't put it in your face that your 4 is really a matrix([[4]]), even though that's what it is to Matlab. But I don't see a good way to change that behavior. The other thing I find problematic about matrices is the inability to go higher than 2d. To me that means that it's impossible to go "pure matrix" in my code because I'll have to switch back to arrays any time I want more than 2d (or use a mixed solution like a list of matrices). Matlab allows allows >2D. --bb From robert.kern at gmail.com Sun Mar 25 17:44:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 25 Mar 2007 16:44:30 -0500 Subject: [Numpy-discussion] header file location upon installation In-Reply-To: References: Message-ID: <4606ED3E.2000602@gmail.com> Daniel Wheeler wrote: > Hi, > > Should the header files in > > .../lib/python2.4/site-packages/python/numpy/core/include/ > > be copied to > > .../include/python2.4/numpy/ > > upon installation of numpy? No. For reasons that we've discussed several times here, the only reliable place to keep headers is in the package itself. Some users don't have write access to sys.prefix and some users use eggs and/or have multiple versions of numpy installed at any one time. For building extensions, either use numpy.distutils, which will take care of everything for you, or use numpy.get_include() to get the directory with headers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cjw at sympatico.ca Sun Mar 25 19:16:13 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 19:16:13 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: Alan G Isaac wrote: > One thing keeps bugging me when I use numpy.matrix. > > All this is fine:: > > >>> x=N.mat('1 1;1 0') > >>> x > matrix([[1, 1], > [1, 0]]) > >>> x[1,:] > matrix([[1, 0]]) > > But it seems to me that I should be able > to extract a matrix row as an array. This can easily be done: *** Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32. *** >>> import numpy as _n >>> A= _n.mat([[1, 2], [3, 4]]) >>> A[1] matrix([[3, 4]]) >>> A[1].getA1() array([3, 4]) An array and a matrix are different animals. Conversion from one to the other should be spelled out. As you have done below. Colin W. > So this :: > > >>> x[1] > matrix([[1, 0]]) > > feels wrong. (Similarly when iterating across rows.) > Of course I realize that I can just :: > > >>> x.A[1] > array([1, 0]) > > but since the above keeps feeling wrong I felt I should > raise this as a possible design issue, better discussed > early than latter. > > Cheers, > Alan Isaac From steve at shrogers.com Sun Mar 25 19:21:04 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sun, 25 Mar 2007 17:21:04 -0600 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> Message-ID: <460703E0.1000902@shrogers.com> Joe Harrington wrote: > > On the other hand, Python, IDL, and Matlab are attractive to us mainly > because their syntaxes allow us to see the math, understand it on > inspection, and verify its correctness. The math we write in these > languages looks as much like the math we do on paper as ASCII will > allow. (Obviously, we also choose them because we don't like writing > loops and declaring variables.) > > So, whenever I hear someone suggest using a functional syntax for a > concept that exists notationally, I cringe. We're alienating a class > of users each time we do that. Those are people who will never come > to Python. There are extremes to which this argument cannot go - a > prime will never be a derivative because quotes are more important - > but I think that matrix multiplication is a no-brainer here. We > should let the Python developers know we want it now and then follow > up with a syntax and implementation. > > Sun's Fortress programming language (http://research.sun.com/projects/plrg/faq/index.html) will include Unicode operators and two dimensional equations. Something like this should be possible for NumPy. The question is how much can and should be pushed into the Python core language. # Steve From cjw at sympatico.ca Sun Mar 25 19:20:21 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 19:20:21 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Alan G Isaac wrote: >> Em Dom, 2007-03-25 ? s 13:07 -0400, Alan G Isaac escreveu: >>> >>> x[1] >>> matrix([[1, 0]]) >>> feels wrong. (Similarly when iterating across rows.) > > > On Sun, 25 Mar 2007, Paulo Jose da Silva e Silva apparently wrote: >> I think the point here is that if you are using matrices, >> then all you "should" want are matrices, just like in >> MATLAB: >> >> b = A(1, :) >> b = >> 1 2 > > > Yes, that is the idea behind this, which I am also > accustomed to from GAUSS. But note again that the Matlab > equivalent :: > > >>> x=N.mat('1 2;3 4') > >>> x[0,:] > matrix([[1, 2]]) > > does provide this behavior. The question I am raising > is a design question and is I think really not addressed > by the rule of thumb you offer. Specifically, that rule > of thumb if it is indeed the justification of :: > > >>> x[1] > matrix([[3, 4]]) > > finds itself in basic conflict with the idea that I ought to > be able to iterate over the objects in an iterable container. > > I mean really, does this not "feel" wrong? :: > > >>> for item in x: print item.__repr__() > ... > matrix([[1, 2]]) > matrix([[3, 4]]) > > Cheers, > Alan Isaac > > Perhaps this would be clearer with: >>> for rowVector in x: print item.__repr__() ... matrix([[1, 2]]) matrix([[3, 4]]) Colin W. From cjw at sympatico.ca Sun Mar 25 19:34:38 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun, 25 Mar 2007 19:34:38 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Bill Baxter wrote: > On 3/26/07, Alan G Isaac wrote: >>> Em Dom, 2007-03-25 ?s 13:07 -0400, Alan G Isaac escreveu: >>>> >>> x[1] >>>> matrix([[1, 0]]) >>>> feels wrong. (Similarly when iterating across rows.) >> >> On Sun, 25 Mar 2007, Paulo Jose da Silva e Silva apparently wrote: >>> I think the point here is that if you are using matrices, >>> then all you "should" want are matrices, just like in >>> MATLAB: >>> >> b = A(1, :) >>> b = >>> 1 2 >> >> Yes, that is the idea behind this, which I am also >> accustomed to from GAUSS. But note again that the Matlab >> equivalent :: >> >> >>> x=N.mat('1 2;3 4') >> >>> x[0,:] >> matrix([[1, 2]]) >> >> does provide this behavior. The question I am raising >> is a design question and is I think really not addressed >> by the rule of thumb you offer. Specifically, that rule >> of thumb if it is indeed the justification of :: >> >> >>> x[1] >> matrix([[3, 4]]) >> >> finds itself in basic conflict with the idea that I ought to >> be able to iterate over the objects in an iterable container. >> >> I mean really, does this not "feel" wrong? :: >> >> >>> for item in x: print item.__repr__() >> ... >> matrix([[1, 2]]) >> matrix([[3, 4]]) > > This may sound silly, but I really think seeing all those brackets is > what makes it feel wrong. Matlab's output doesn't put it in your > face that your 4 is really a matrix([[4]]), even though that's what it > is to Matlab. But I don't see a good way to change that behavior. > > The other thing I find problematic about matrices is the inability to > go higher than 2d. To me that means that it's impossible to go "pure > matrix" in my code because I'll have to switch back to arrays any time > I want more than 2d (or use a mixed solution like a list of matrices). > Matlab allows allows >2D. > > --bb "pure matrix" seems to me an area of exploration, does it have any application in numerical computation at this time? A list of matrices seems to be a logical structure. PyMatrix deals with lists in building a larger matrix from sub-matrices. Suppose that we have matrices A (3, 4), B (3, 6), C (4, 2) and D (4, 8). Then E= M([[A, B], [C, D]]) gives E (7, 10). Colin W. From wbaxter at gmail.com Sun Mar 25 19:47:31 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 26 Mar 2007 08:47:31 +0900 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <460703E0.1000902@shrogers.com> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <460703E0.1000902@shrogers.com> Message-ID: On 3/26/07, Steven H. Rogers wrote: > Joe Harrington wrote: > > > > On the other hand, Python, IDL, and Matlab are attractive to us mainly > > because their syntaxes allow us to see the math, understand it on > > inspection, and verify its correctness. The math we write in these > > languages looks as much like the math we do on paper as ASCII will > > allow. (Obviously, we also choose them because we don't like writing > > loops and declaring variables.) > > > > So, whenever I hear someone suggest using a functional syntax for a > > concept that exists notationally, I cringe. We're alienating a class > > of users each time we do that. Those are people who will never come > > to Python. There are extremes to which this argument cannot go - a > > prime will never be a derivative because quotes are more important - > > but I think that matrix multiplication is a no-brainer here. We > > should let the Python developers know we want it now and then follow > > up with a syntax and implementation. > > > > > > Sun's Fortress programming language > (http://research.sun.com/projects/plrg/faq/index.html) will include > Unicode operators and two dimensional equations. Something like this > should be possible for NumPy. The question is how much can and should > be pushed into the Python core language. Until we get to the point that it's actually harder to find a non-Unicode console/editor than a Unicode one, I think the idea of using Unicode symbols as part of the syntax of a general purpose language is a bad one. I'm looking forward to see what becomes of Fortress, but it's basically still an experiment at this point. --bb From wbaxter at gmail.com Sun Mar 25 20:10:34 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 26 Mar 2007 09:10:34 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: On 3/26/07, Colin J. Williams wrote: > Bill Baxter wrote: > > This may sound silly, but I really think seeing all those brackets is > > what makes it feel wrong. Matlab's output doesn't put it in your > > face that your 4 is really a matrix([[4]]), even though that's what it > > is to Matlab. But I don't see a good way to change that behavior. > > > > The other thing I find problematic about matrices is the inability to > > go higher than 2d. To me that means that it's impossible to go "pure > > matrix" in my code because I'll have to switch back to arrays any time > > I want more than 2d (or use a mixed solution like a list of matrices). > > Matlab allows allows >2D. > > > > --bb > "pure matrix" seems to me an area of exploration, does it have any > application in numerical computation at this time? I'm not sure what you thought I meant, but all I meant by going "pure matrix" was having my Numpy code use the 'matrix' type exclusively instead of some mix of 'matrix' and the base 'ndarray' type. Things become messy when you mix and match them because you don't know any more if an expression like A[1] is going to give you a 1-D thing or a 2-D thing, and you can't be sure what A * B will do without always coercing A and B. > A list of matrices seems to be a logical structure. Yes, and it's the only option if you want to make a list of matrices of different shapes, but I frequently have a need for things like a list of per-point transformation matrices. Each column from each of those matrices can be thought of as a vector. Sometimes its convenient to consider all the X basis vectors together, for instance, which is a simple and efficient M[:,:,0] slice if I have all the data in a 3-D array, but it's a slow list comprehension plus matrix constructor if I have the matrices in a list -- something like matrix([m[:,0] for m in M]) but that line is probably incorrect. > PyMatrix deals with > lists in building a larger matrix from sub-matrices. > > Suppose that we have matrices A (3, 4), B (3, 6), C (4, 2) and D (4, 8). > > Then E= M([[A, B], [C, D]]) gives E (7, 10). Numpy generally tries to treat all lists and tuples as array literals. That's not likely to change. --bb From charlesr.harris at gmail.com Sun Mar 25 20:21:12 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 25 Mar 2007 18:21:12 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: Message-ID: On 3/25/07, Nils Wagner wrote: > > Hi, BTW, I can't import scipy.sparse, I get the following error: ImportError: cannot import name densetocsr What am I doing wrong? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sun Mar 25 20:32:01 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 25 Mar 2007 18:32:01 -0600 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <460703E0.1000902@shrogers.com> Message-ID: On 3/25/07, Bill Baxter wrote: > > On 3/26/07, Steven H. Rogers wrote: > > Joe Harrington wrote: > > > > > > On the other hand, Python, IDL, and Matlab are attractive to us mainly > > > because their syntaxes allow us to see the math, understand it on > > > inspection, and verify its correctness. The math we write in these > > > languages looks as much like the math we do on paper as ASCII will > > > allow. (Obviously, we also choose them because we don't like writing > > > loops and declaring variables.) > > > > > > So, whenever I hear someone suggest using a functional syntax for a > > > concept that exists notationally, I cringe. We're alienating a class > > > of users each time we do that. Those are people who will never come > > > to Python. There are extremes to which this argument cannot go - a > > > prime will never be a derivative because quotes are more important - > > > but I think that matrix multiplication is a no-brainer here. We > > > should let the Python developers know we want it now and then follow > > > up with a syntax and implementation. > > > > > > > > > > Sun's Fortress programming language > > (http://research.sun.com/projects/plrg/faq/index.html) will include > > Unicode operators and two dimensional equations. Something like this > > should be possible for NumPy. The question is how much can and should > > be pushed into the Python core language. > > Until we get to the point that it's actually harder to find a > non-Unicode console/editor than a Unicode one, I think the idea of > using Unicode symbols as part of the syntax of a general purpose > language is a bad one. > > I'm looking forward to see what becomes of Fortress, but it's > basically still an experiment at this point. What might work better is simply some sort of sign that causes a function to be parsed as infix, x @dot y for instance, although Python already uses @ for other things. I don't know what symbols are left unused at this point, maybe ! , $, and ~. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Sun Mar 25 20:48:37 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 20:48:37 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: > Alan G Isaac wrote: >> So this :: >> >>> x[1] >> matrix([[1, 0]]) >> feels wrong. (Similarly when iterating across rows.) >> Of course I realize that I can just :: >> >>> x.A[1] >> array([1, 0]) On Sun, 25 Mar 2007, "Colin J. Williams" apparently wrote: > An array and a matrix are different animals. Conversion > from one to the other should be spelled out. But you are just begging the question here. The question is: when I iterate across matrix rows, why am I iterating across matrices and not arrays. This seems quite out of whack with general Python practice. You cannot just say "conversion should be explicit" because that assumes (incorrectly actually) that the rows are matrices. The "conversion should be explicit" argument actually cuts in the opposite direction of what you appear to believe. Cheers, Alan Isaac From aisaac at american.edu Sun Mar 25 21:00:13 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 21:00:13 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: > On 3/26/07, Alan G Isaac wrote: >> finds itself in basic conflict with the idea that I ought >> to be able to iterate over the objects in an iterable >> container. >> I mean really, does this not "feel" wrong? :: >> >>> for item in x: print item.__repr__() >> ... >> matrix([[1, 2]]) >> matrix([[3, 4]]) On Mon, 26 Mar 2007, Bill Baxter apparently wrote: > This may sound silly, but I really think seeing all those > brackets is what makes it feel wrong. I appreciate the agreement that it feels wrong, but I dispute the analysis of this symptom. What makes it "feel wrong", I contend, is that experience with Python make this a **surprising** behavior. And that is precisely why I suggest that this may point to a design issue. Cheers, Alan Isaac From jstrunk at enthought.com Sun Mar 25 21:07:46 2007 From: jstrunk at enthought.com (Jeff Strunk) Date: Sun, 25 Mar 2007 20:07:46 -0500 Subject: [Numpy-discussion] subversion site down In-Reply-To: <4606C27C.9030802@stsci.edu> References: <4606C27C.9030802@stsci.edu> Message-ID: <200703252007.46392.jstrunk@enthought.com> Thank you for letting me know. I restarted the server at 5:30pm central. -Jeff On Sunday 25 March 2007 1:42 pm, Christopher Hanley wrote: > Hi, > > It appears that the subversion server is down for numpy. > > Chris From steve at shrogers.com Sun Mar 25 21:20:02 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Sun, 25 Mar 2007 19:20:02 -0600 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <460703E0.1000902@shrogers.com> Message-ID: <46071FC2.9030503@shrogers.com> Bill Baxter wrote: > > Until we get to the point that it's actually harder to find a > non-Unicode console/editor than a Unicode one, I think the idea of > using Unicode symbols as part of the syntax of a general purpose > language is a bad one. Given that most editors lack good Unicode support, it can be used as an alternate syntax, but not the only syntax. For the foreseeable future, both syntaxes would have to be supported. All that is needed is *one* reasonable editor with sufficient support for a Unicode Python syntax, and others will follow. Since Idle is often used by Python beginners, it would be a good candidate. > > I'm looking forward to see what becomes of Fortress, but it's > basically still an experiment at this point. > I think Fortress is more than an experiment, though it is still a long way from being useful for real work. # Steve From wbaxter at gmail.com Sun Mar 25 21:31:04 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 26 Mar 2007 10:31:04 +0900 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <460703E0.1000902@shrogers.com> Message-ID: On 3/26/07, Charles R Harris wrote: > > What might work better is simply some sort of sign that causes a function to > be parsed as infix, x @dot y for instance, although Python already uses @ > for other things. I don't know what symbols are left unused at this point, > maybe ! , $, and ~. I'm not really an expert on compilers, but from what I understand, the biggest problem with adding new operators is defining precedence. Without it, the interpreter won't know what to do with something like A @dot B @plus C. Currently the interpreter only has to look at what the tokens are to build a parse tree out of something like A*B+C. It doesn't matter what the types of A B and C are, that's always parsed as (A*B)+C. If you add a @dot operator where do you define its precedence? On the class that defines the operator? What if two classes define the same operator with different precedences? It gets to be a big ambiguous mess. So if you're going to add new operators they pretty much need to have globally predetermined precedence to allow the parser to remain sane. I think it's ML or Haskell that lets you define any operator you want out of the core operator tokens (eg "*+*") but IIRC the first char determines the precedence. So *+* has the same precedence as "*". Requiring the new operator to include an existing operator char seems like a reasonable strategy since it gives a simple rule-based mechanism for determining precedence that is easy for both the parser and for users. --bb From wbaxter at gmail.com Sun Mar 25 21:49:40 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Mon, 26 Mar 2007 10:49:40 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: On 3/26/07, Alan G Isaac wrote: > > On 3/26/07, Alan G Isaac wrote: > >> finds itself in basic conflict with the idea that I ought > >> to be able to iterate over the objects in an iterable > >> container. > > >> I mean really, does this not "feel" wrong? :: > > >> >>> for item in x: print item.__repr__() > >> ... > >> matrix([[1, 2]]) > >> matrix([[3, 4]]) > > > On Mon, 26 Mar 2007, Bill Baxter apparently wrote: > > This may sound silly, but I really think seeing all those > > brackets is what makes it feel wrong. > > > I appreciate the agreement that it feels wrong, but > I dispute the analysis of this symptom. What makes it "feel > wrong", I contend, is that experience with Python make this > a **surprising** behavior. And that is precisely why > I suggest that this may point to a design issue. So you're saying this is what you'd find more pythonic? >>> X[1] matrix([2,3]) >>> X[:,1] matrix([[3, 4]]) Just trying to make it clear what you're proposing. Probably about half the bugs I get from mixing and matching matrix and array are things like row = A[i] ... z = row[2] Which works for an array but not for a matrix. I think Matlab makes it more bearable by having a single value index like X[i] be equivalent to X.flat()[i]. So X[2] is the same for row or col vec in Matlab. Now that I think about it, that's probably the main reason I feel more comfortable with array than matrix in Numpy. If I have a vector, I should only need one index to get the ith component. --bb From aisaac at american.edu Sun Mar 25 22:26:24 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 22:26:24 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: >> On 3/26/07, Alan G Isaac wrote: >>> finds itself in basic conflict with the idea that >>> I ought to be able to iterate over the objects in an >>> iterable container. I mean really, does this not "feel" >>> wrong? :: >>> for item in x: print item.__repr__() >>> ... >>> matrix([[1, 2]]) >>> matrix([[3, 4]]) On Mon, 26 Mar 2007, Bill Baxter apparently wrote: > So you're saying this is what you'd find more pythonic? >>>> X[1] > matrix([2,3]) >>>> X[:,1] > matrix([[3, > 4]]) > Just trying to make it clear what you're proposing. No; that is not possible, since a matrix is inherently 2d. I just want to get the constituent arrays when I iterate over the matrix object or use regular Python indexing, but a matrix when I use matrix/array indexing. That is :: >>> X[1] array([2,3]) >>> X[1,:] matrix([[3, 4]]) That behavior seems completely natural and unsurprising. > Probably about half the bugs I get from mixing and matching matrix and > array are things like > row = A[i] > ... > z = row[2] > Which works for an array but not for a matrix. Exactly! That is the evidence of a "bad surprise" in the current behavior. Iterating over a Python iterable should provide access to the contained objects. Cheers, Alan Isaac From aisaac at american.edu Sun Mar 25 22:30:32 2007 From: aisaac at american.edu (Alan G Isaac) Date: Sun, 25 Mar 2007 22:30:32 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Oooops, they should match of course. :: >>> X[1] array([3,4]) >>> X[1,:] matrix([[3, 4]]) But again the point is: indexing for submatrices should produce matrices. Normal Python indexing should access the constituent arrays. Cheers, Alan Isaac From openopt at ukr.net Mon Mar 26 02:42:14 2007 From: openopt at ukr.net (dmitrey) Date: Mon, 26 Mar 2007 09:42:14 +0300 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <460703E0.1000902@shrogers.com> Message-ID: <46076B46.1040607@ukr.net> Hallo! 1)Isn't it possible to add .A & .M attributes to the array & matrix instances? I would find them very useful for some cases more convenient than asarray() or mat(). Let x.A just do nothing if x is array and x.M do nothing if x i matrix. 2)And if y=flat(x), what about y.R and y.C for to making y row-ndarray & column-ndarray (for to further convenient error-safe multiplication to 2-dim arrays)? + maybe the same thing for ndarray & matrix instances? 3)What about new dotwise & matrix operators - after reading the discussion I would better wait for unicode instead of implementing 2-symbol operators, for to make it once & forever. WBR, D. Bill Baxter wrote: > On 3/26/07, Charles R Harris wrote: > >> What might work better is simply some sort of sign that causes a function to >> be parsed as infix, x @dot y for instance, although Python already uses @ >> for other things. I don't know what symbols are left unused at this point, >> maybe ! , $, and ~. >> > > I'm not really an expert on compilers, but from what I understand, the > biggest problem with adding new operators is defining precedence. > Without it, the interpreter won't know what to do with something like > A @dot B @plus C. Currently the interpreter only has to look at what > the tokens are to build a parse tree out of something like A*B+C. It > doesn't matter what the types of A B and C are, that's always parsed > as (A*B)+C. If you add a @dot operator where do you define its > precedence? On the class that defines the operator? What if two > classes define the same operator with different precedences? It gets > to be a big ambiguous mess. > > So if you're going to add new operators they pretty much need to have > globally predetermined precedence to allow the parser to remain sane. > I think it's ML or Haskell that lets you define any operator you want > out of the core operator tokens (eg "*+*") but IIRC the first char > determines the precedence. So *+* has the same precedence as "*". > Requiring the new operator to include an existing operator char seems > like a reasonable strategy since it gives a simple rule-based > mechanism for determining precedence that is easy for both the parser > and for users. > > --bb > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > From curiousjan at gmail.com Mon Mar 26 04:05:32 2007 From: curiousjan at gmail.com (Jan Strube) Date: Mon, 26 Mar 2007 01:05:32 -0700 Subject: [Numpy-discussion] dtype confusion Message-ID: I'm afraid I'm not quite done with this, yet. There's still more I am confused about, even after reading the manual (again). There is something fundamentally weird about record arrays, that doesn't seem to click with me. Please have a look at this piece of code: import numpy as N newtype = N.dtype([('x', N.float64), ('y', N.float64), ('z', N.float64)]) x = N.zeros((10, 3)) x.dtype = newtype print x x = N.zeros((10, 3), dtype=newtype) print x I think I understand what's going on here: x = N.zeros((10, 1), dtype=newtype) print x So can I summarize that in the first case a (10,3) array is reinterpreted ( numpy.view) as a record array, while in the second case I am actually requesting a (10,3) array of the dtype, which itself has 3 elements? And the record array should not be viewed as a column-wise array with named columns (although it can be accessed that way), but rather as an array that groups the columns in each row into a record. Because after turning it into a record array, x is still c-contiguous and not fortran-contiguous... Maybe I'm actually starting to understand this a bit. Anyways, thanks for listening ;-) Cheers, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbastian at free.fr Mon Mar 26 04:08:52 2007 From: rbastian at free.fr (=?iso-8859-1?q?Ren=E9=20Bastian?=) Date: Mon, 26 Mar 2007 10:08:52 +0200 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <46076B46.1040607@ukr.net> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <46076B46.1040607@ukr.net> Message-ID: <07032610085200.00754@rbastian> Hello, I am interest both in numarray type multiplication and matrix type multiplication. But I am not shure that I can buy an Unicode keyboard. May be it would be possible to implement a class with user "definissable" (?) signs. My choice : a * b -> numarray type multi a !* b -> matrix -- Ren? Bastian http://www.musiques-rb.org http://pythoneon.musiques-rb.org From jturner at gemini.edu Mon Mar 26 04:16:51 2007 From: jturner at gemini.edu (James Turner) Date: Mon, 26 Mar 2007 04:16:51 -0400 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> References: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> Message-ID: <46078173.7030908@gemini.edu> PS... (sorry for all the posts, for anyone who isn't interested...) > Agreed, it looks like aliasing. Nevertheless, any resampling > procedure is supposed to deal with this internally, right? Either by > lowpass filtering (traditional case), or by spline fitting (spline > case as described by Unser and understood by me) -- it shouldn't be > letting aliasing bubble through, correct? In the general case, I don't think it is appropriate for the resampling procedure to use low-pass filtering internally to avoid artefacts, except perhaps when downsampling. It probably makes sense for computer graphics work, but there are cases where the input data are band limited to begin with and any degradation in resolution is unacceptable. Where needed, I think low-pass filtering should either be the responsibility of the main program or an option. It's not even possible for the resampling procedure to prevent artefacts in every case, since the aliasing in a badly undersampled image cannot be removed post factum (this is for undersampled photos rather than artificial graphics, which I think are fundamentally different because everything is defined on the grid, although I haven't sat down and proved it mathematically). I'm also not sure how the procedure could decide on the level of smoothing needed for a given dataset without external information. Of course intermediate-order splines will probably keep everyone happy, being reasonably robust against ringing effects without causing much smoothing or interpolation error :-). By the way, I think you and Stefan might be interested in a medical imaging paper by Lehmann et al. (1999), which gives a very nice overview of the properties of different interpolation kernels: http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/42/17698/00816070.pdf?arnumber=816070 For what it's worth, I'd agree with both of you that the numeric overflow should be documented if not fixed. It sounds like Stefan has figured out a solution for it though. If you make sense of the code in "ni_interpolation.c", Stefan, I'd be very interested in how to make it calculate one less value at the edges :-). Cheers, James. From zpincus at stanford.edu Mon Mar 26 04:39:29 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon, 26 Mar 2007 01:39:29 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <46078173.7030908@gemini.edu> References: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> <46078173.7030908@gemini.edu> Message-ID: <7A3C3EB8-66D2-4AB7-874D-C96D06C227AF@stanford.edu> Thanks for the information and the paper link, James. I certainly appreciate the perspective, and now see why the anti-aliasing and reconstruction filtering might best be left to clients of a resampling procedure. Hopefully at least some of the kinks in the spline interpolation (to date: obligate mirror boundary conditions, extra edge values, integer overflow) can be smoothed out. I can't guarantee that I'll have the time or expertise to deal with ni_interpolation.c, but I'll try to give it a shot some time here. Zach On Mar 26, 2007, at 1:16 AM, James Turner wrote: > PS... (sorry for all the posts, for anyone who isn't interested...) > >> Agreed, it looks like aliasing. Nevertheless, any resampling >> procedure is supposed to deal with this internally, right? Either by >> lowpass filtering (traditional case), or by spline fitting (spline >> case as described by Unser and understood by me) -- it shouldn't be >> letting aliasing bubble through, correct? > > In the general case, I don't think it is appropriate for the > resampling > procedure to use low-pass filtering internally to avoid artefacts, > except perhaps when downsampling. It probably makes sense for computer > graphics work, but there are cases where the input data are band > limited to begin with and any degradation in resolution is > unacceptable. > Where needed, I think low-pass filtering should either be the > responsibility of the main program or an option. It's not even > possible > for the resampling procedure to prevent artefacts in every case, since > the aliasing in a badly undersampled image cannot be removed post > factum (this is for undersampled photos rather than artificial > graphics, > which I think are fundamentally different because everything is > defined > on the grid, although I haven't sat down and proved it > mathematically). > I'm also not sure how the procedure could decide on the level of > smoothing needed for a given dataset without external information. > > Of course intermediate-order splines will probably keep everyone > happy, > being reasonably robust against ringing effects without causing much > smoothing or interpolation error :-). > > By the way, I think you and Stefan might be interested in a medical > imaging paper by Lehmann et al. (1999), which gives a very nice > overview > of the properties of different interpolation kernels: > > http://ieeexplore.ieee.org/Xplore/login.jsp?url=/ > iel5/42/17698/00816070.pdf?arnumber=816070 > > For what it's worth, I'd agree with both of you that the numeric > overflow should be documented if not fixed. It sounds like Stefan has > figured out a solution for it though. If you make sense of the code in > "ni_interpolation.c", Stefan, I'd be very interested in how to make it > calculate one less value at the edges :-). > > Cheers, > > James. > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From openopt at ukr.net Mon Mar 26 05:42:35 2007 From: openopt at ukr.net (dmitrey) Date: Mon, 26 Mar 2007 12:42:35 +0300 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <07032610085200.00754@rbastian> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <46076B46.1040607@ukr.net> <07032610085200.00754@rbastian> Message-ID: <4607958B.8050306@ukr.net> The unicode keyboards sailing everywhere is just a matter of time And python 2-symbol operators soon will look obsolete, this will increase migrating from python to Sun fortress etc. I took a look at their unicode syntax for math formulas http://research.sun.com/projects/plrg/faq/NAS-CG.pdf it looks (vs Python) (or even MATLAB) like Pentium 4 vs Intel 386 processors. It just allows copy-paste from articles of many formulas, including "ro", 'beta' and other non-ascii symbols Also, implementing unicode will allow separate operators for (for example) MATLAB cross() equivalent (vector multiplication of vectors). WBR, D. Ren? Bastian wrote: > Hello, > > I am interest both in numarray type multiplication and matrix type > multiplication. > > But I am not shure that I can buy an Unicode keyboard. > > May be it would be possible to implement a class with > user "definissable" (?) signs. > > My choice : > > a * b -> numarray type multi > a !* b -> matrix > > > From zpincus at stanford.edu Mon Mar 26 05:56:06 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon, 26 Mar 2007 02:56:06 -0700 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <4607958B.8050306@ukr.net> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <46076B46.1040607@ukr.net> <07032610085200.00754@rbastian> <4607958B.8050306@ukr.net> Message-ID: <3E41CE22-B213-438C-AC5C-01BAEC5037E1@stanford.edu> Hi folks, Sorry to rain on this parade, but unicode variable names and/or other syntactic elements have already been rejected for Python 3: http://www.python.org/dev/peps/pep-3099/ > Python 3000 source code won't use non-ASCII Unicode characters for > anything except string literals or comments. To tell the truth, in my code (for an n=1 example), I use elementwise (or broadcasted) multiplication about an order of magnitude more than matrix multiplication. Now, there's plenty of linear algebra in my work, but it's usually boxed off enough from the rest that converting everything to matrices, or using the 'dot' function (etc.), is just fine. Personally, I even prefer the current way that numpy does things -- I've always *really* disliked matlab's special syntax for elementwise multiplication. Now, clearly there are cases where this is handy, but at least in looking over my code, I find that those cases are quite rare, really. So, in large part, I really don't feel a strong need for new operators in numpy. (And in the rest of python too! The @ decorator pie-syntax is quite enough for line-noise characters, in my personal opinion. And I think that python-dev pretty well agrees too, based on the raging flame wars about adding even that.) Now, this is of course just my single opinion, but folks should recall that even C++, which rarely met a feature that it didn't like, drew the line at adding extra syntactic operators to the language that existed only to be overridden/implemented by users. (Which is precisely what's being proposed here.) Anyhow, feel free to disagree with me -- I'm no expert here. I'm only mentioning this as a public service to make it clear that most of what's being proposed in this thread is, for better or worse, 100% dead-in-the-water for Python 3, and the rest will have a fairly significant uphill battle. Zach On Mar 26, 2007, at 2:42 AM, dmitrey wrote: > The unicode keyboards sailing everywhere is just a matter of time > And python 2-symbol operators soon will look obsolete, this will > increase migrating from python to Sun fortress etc. I took a look at > their unicode syntax for math formulas > http://research.sun.com/projects/plrg/faq/NAS-CG.pdf > it looks (vs Python) (or even MATLAB) like Pentium 4 vs Intel 386 > processors. > It just allows copy-paste from articles of many formulas, including > "ro", 'beta' and other non-ascii symbols > Also, implementing unicode will allow separate operators for (for > example) MATLAB cross() equivalent (vector multiplication of vectors). > WBR, D. > > Ren? Bastian wrote: >> Hello, >> >> I am interest both in numarray type multiplication and matrix type >> multiplication. >> >> But I am not shure that I can buy an Unicode keyboard. >> >> May be it would be possible to implement a class with >> user "definissable" (?) signs. >> >> My choice : >> >> a * b -> numarray type multi >> a !* b -> matrix >> >> >> > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From t.zito at biologie.hu-berlin.de Mon Mar 26 06:08:47 2007 From: t.zito at biologie.hu-berlin.de (Tiziano Zito) Date: Mon, 26 Mar 2007 12:08:47 +0200 Subject: [Numpy-discussion] Modular toolkit for Data Processing 2.1 released! Message-ID: <20070326100847.GB5038@lotka.biologie.hu-berlin.de> MDP version 2.1 and symeig 1.2 have been released! What's new in version 2.1? -------------------------- - Fully compatible with NumpPy 1.0, the first stable release of the descendant of the Numeric python extension module - symeig project resumed and updated - For increased speed, scipy and symeig are automatically used if available - New nodes: Independent Slow Feature Analysis and quadratic forms analysis algorithms - General improvements, several bug fixes, and code cleanups What is it? ----------- Modular toolkit for Data Processing (MDP) is a data processing framework written in Python. From the user's perspective, MDP consists of a collection of trainable supervised and unsupervised algorithms that can be combined into data processing flows. The base of readily available algorithms includes Principal Component Analysis, two flavors of Independent Component Analysis, Slow Feature Analysis, Independent Slow Feature Analysis, and many more. From the developer's perspective, MDP is a framework to make the implementation of new algorithms easier. MDP takes care of tedious tasks like numerical type and dimensionality checking, leaving the developer free to concentrate on the implementation of the training and execution phases. The new elements then seamlessly integrate with the rest of the library. MDP has been written in the context of theoretical research in neuroscience, but it has been designed to be helpful in any context where trainable data processing algorithms are used. Its simplicity on the user side together with the reusability of the implemented nodes make it also a valid educational tool. As its user base is increasing, MDP is becoming a common repository of user-supplied, freely available, Python-implemented data processing algorithms. The optional symeig module contains a Python wrapper for the LAPACK functions to solve the standard and generalized eigenvalue problems for symmetric (hermitian) positive definite matrices. Those specialized algorithms give an important speed-up with respect to the generic LAPACK eigenvalue problem solver used by NumPy. Resources --------- Download: http://sourceforge.net/project/showfiles.php?group_id=116959 Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://sourceforge.net/mail/?group_id=116959 -- Tiziano Zito Institute for Theoretical Biology Humboldt-Universitaet zu Berlin Invalidenstrasse, 43 10115 Berlin, Germany Pietro Berkes Gatsby Computational Neuroscience Unit Alexandra House, 17 Queen Square London WC1N 3AR, United Kingdom From berkes at gatsby.ucl.ac.uk Mon Mar 26 06:55:39 2007 From: berkes at gatsby.ucl.ac.uk (Pietro Berkes) Date: Mon, 26 Mar 2007 11:55:39 +0100 (BST) Subject: [Numpy-discussion] random state not portable? Message-ID: Dear Numpy devels, First of all, thank you for the great job done so far! We've been updating our MDP module, and had a chance to appreciate it. We found an issue in the random module: the random state contains an array of int32 or int64 depending on the architecture, making it not portable. For example, try the following on two machines with different architecures: Python 2.4.4 (#2, Jan 13 2007, 17:50:26) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import platform >>> import pickle >>> >>> platform.architecture() ('32bit', '') >>> >>> state = numpy.random.get_state() >>> pickle.dump(state, file('/tmp/random_state.pic','w')) >>> numpy.random.permutation(numpy.arange(10)) array([8, 6, 3, 1, 0, 5, 7, 4, 2, 9]) >>> Python 2.5 (r25:51908, Mar 19 2007, 13:41:07) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import platform >>> import pickle >>> >>> platform.architecture() ('64bit', 'ELF') >>> >>> state = pickle.load(file('/tmp/random_state.pic','r')) >>> numpy.random.set_state(state) >>> numpy.random.permutation(numpy.arange(10)) array([3, 9, 4, 8, 1, 6, 2, 0, 5, 7]) >>> Is there any known workaround? Thanks! -- Tiziano Zito Institute for Theoretical Biology Humboldt-Universitaet zu Berlin Invalidenstrasse, 43 10115 Berlin, Germany Pietro Berkes Gatsby Computational Neuroscience Unit Alexandra House, 17 Queen Square London WC1N 3AR, United Kingdom From cjw at sympatico.ca Mon Mar 26 07:39:40 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon, 26 Mar 2007 07:39:40 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Bill Baxter wrote: > On 3/26/07, Colin J. Williams wrote: >> Bill Baxter wrote: >>> This may sound silly, but I really think seeing all those brackets is >>> what makes it feel wrong. Matlab's output doesn't put it in your >>> face that your 4 is really a matrix([[4]]), even though that's what it >>> is to Matlab. But I don't see a good way to change that behavior. >>> >>> The other thing I find problematic about matrices is the inability to >>> go higher than 2d. To me that means that it's impossible to go "pure >>> matrix" in my code because I'll have to switch back to arrays any time >>> I want more than 2d (or use a mixed solution like a list of matrices). >>> Matlab allows allows >2D. >>> >>> --bb >> "pure matrix" seems to me an area of exploration, does it have any >> application in numerical computation at this time? > > I'm not sure what you thought I meant, but all I meant by going "pure > matrix" was having my Numpy code use the 'matrix' type exclusively > instead of some mix of 'matrix' and the base 'ndarray' type. It was a term I had not come across before but I assumed that you were referring to something like this link - beyond my comprehension. http://72.14.203.104/search?q=cache:Yu9gbUQEfWkJ:math.ca/Events/winter05/abs/pdf/ma-df.pdf+pure+matrix&hl=en&ct=clnk&cd=4&gl=ca&lr=lang_en Things > become messy when you mix and match them because you don't know any > more if an expression like A[1] is going to give you a 1-D thing or a > 2-D thing, and you can't be sure what A * B will do without always > coercing A and B. Yes, to my mind it's best to consider the multi-dimensional array and the matrix to be two distinct data types. In most cases, it's best that conversions between the two should be explicit. > >> A list of matrices seems to be a logical structure. > > Yes, and it's the only option if you want to make a list of matrices > of different shapes, but I frequently have a need for things like a > list of per-point transformation matrices. Each column from each of > those matrices can be thought of as a vector. Sometimes its > convenient to consider all the X basis vectors together, for instance, > which is a simple and efficient M[:,:,0] slice if I have all the data > in a 3-D array, but it's a slow list comprehension plus matrix > constructor if I have the matrices in a list -- something like > matrix([m[:,0] for m in M]) > but that line is probably incorrect. Logically, this makes sense, where M is a list of matrices. My guess is that it would be a little faster to build one larger matrix and then slice it as needed. > >> PyMatrix deals with >> lists in building a larger matrix from sub-matrices. >> >> Suppose that we have matrices A (3, 4), B (3, 6), C (4, 2) and D (4, 8). >> >> Then E= M([[A, B], [C, D]]) gives E (7, 10). > > Numpy generally tries to treat all lists and tuples as array literals. > That's not likely to change. That need no be a problem is there is clarity of thinking about the essential difference between the matrix data type (even if is is built as a sub-type of the array) and the multi-dimensional array. > > --bb Colin W. From cjw at sympatico.ca Mon Mar 26 07:53:09 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon, 26 Mar 2007 07:53:09 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Alan G Isaac wrote: >>> On 3/26/07, Alan G Isaac wrote: >>>> finds itself in basic conflict with the idea that >>>> I ought to be able to iterate over the objects in an >>>> iterable container. I mean really, does this not "feel" >>>> wrong? :: > >>> for item in x: print item.__repr__() > >>> ... > >>> matrix([[1, 2]]) > >>> matrix([[3, 4]]) > > > On Mon, 26 Mar 2007, Bill Baxter apparently wrote: >> So you're saying this is what you'd find more pythonic? >>>>> X[1] >> matrix([2,3]) >>>>> X[:,1] >> matrix([[3, >> 4]]) >> Just trying to make it clear what you're proposing. > > > No; that is not possible, since a matrix is inherently 2d. > I just want to get the constituent arrays when I iterate > over the matrix object or use regular Python indexing, but > a matrix when I use matrix/array indexing. That is :: > > >>> X[1] > array([2,3]) > >>> X[1,:] > matrix([[3, 4]]) > > That behavior seems completely natural and unsurprising. Perhaps things would be clearer if we thought of the constituent groups of data in a matrix as being themselves matrices. X[1] could represent the second row of a matrix. A row of a matrix is a row vector, a special case of a matrix. To get an array, I suggest that an explicit conversion X[1].A is a clearer way to handle things. Similarly, X[2, 3] is best returned as a value which is of a Python type. Colin W. > > >> Probably about half the bugs I get from mixing and matching matrix and >> array are things like >> row = A[i] >> ... >> z = row[2] >> Which works for an array but not for a matrix. > > > Exactly! > That is the evidence of a "bad surprise" in the current > behavior. Iterating over a Python iterable should provide > access to the contained objects. > > Cheers, > Alan Isaac From svetosch at gmx.net Mon Mar 26 08:07:36 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 26 Mar 2007 14:07:36 +0200 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: <4607B788.1070606@gmx.net> Alan G Isaac schrieb: > Oooops, they should match of course. :: > >>> X[1] > array([3,4]) > >>> X[1,:] > matrix([[3, 4]]) > > But again the point is: > indexing for submatrices should produce matrices. > Normal Python indexing should access the constituent arrays. > I think this is a tricky business. There's also the rule "indexing reduces the rank, slicing preserves it". Numpy-matrices form an exception to this rule, always being 2d, but the exception is consistently enforced. Now it seems you want to have an exception from the exception, correct? Isn't this why the .A1 method for numpy-matrices was introduced even after 1.0rc? -sven (matrix fan) From nwagner at iam.uni-stuttgart.de Mon Mar 26 08:09:59 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Mon, 26 Mar 2007 14:09:59 +0200 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: Message-ID: <4607B817.3050803@iam.uni-stuttgart.de> Charles R Harris wrote: > > > On 3/25/07, *Nils Wagner* > wrote: > > Hi, > > > > > BTW, I can't import scipy.sparse, I get the following error: > > ImportError: cannot import name densetocsr > > What am I doing wrong? > > Chuck > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > This works for me. Python 2.4.1 (#1, Oct 13 2006, 16:51:58) [GCC 4.0.2 20050901 (prerelease) (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from scipy import sparse >>> import scipy.sparse >>> import scipy >>> scipy.__version__ '0.5.3.dev2869' but scipy.test(1) results in ... ====================================================================== ERROR: check_matmat (scipy.sparse.tests.test_sparse.test_csc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 142, in check_matmat assert_array_almost_equal((a*bsp).todense(), a*b) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_rmatvec (scipy.sparse.tests.test_sparse.test_csc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 107, in check_rmatvec assert_array_almost_equal(row*M, row*M.todense()) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_matmat (scipy.sparse.tests.test_sparse.test_csr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 142, in check_matmat assert_array_almost_equal((a*bsp).todense(), a*b) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_rmatvec (scipy.sparse.tests.test_sparse.test_csr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 107, in check_rmatvec assert_array_almost_equal(row*M, row*M.todense()) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_matmat (scipy.sparse.tests.test_sparse.test_dok) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 142, in check_matmat assert_array_almost_equal((a*bsp).todense(), a*b) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: Does the matrix's mean(,axis=0) method work? ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 54, in check_mean assert_array_equal(self.dat.mean(axis=0), self.datsp.mean(axis=0)) File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 423, in mean mean = self.sum(0) File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 402, in sum return o * self File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_rmatvec (scipy.sparse.tests.test_sparse.test_dok) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 107, in check_rmatvec assert_array_almost_equal(row*M, row*M.todense()) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: Does the matrix's sum(,axis=0) method work? ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 46, in check_sum assert_array_equal(self.dat.sum(axis=0), self.datsp.sum(axis=0)) File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 402, in sum return o * self File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_matmat (scipy.sparse.tests.test_sparse.test_lil) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 142, in check_matmat assert_array_almost_equal((a*bsp).todense(), a*b) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: Does the matrix's mean(,axis=0) method work? ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 54, in check_mean assert_array_equal(self.dat.mean(axis=0), self.datsp.mean(axis=0)) File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 423, in mean mean = self.sum(0) File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 402, in sum return o * self File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: check_rmatvec (scipy.sparse.tests.test_sparse.test_lil) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 107, in check_rmatvec assert_array_almost_equal(row*M, row*M.todense()) File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== ERROR: Does the matrix's sum(,axis=0) method work? ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/sparse/tests/test_sparse.py", line 46, in check_sum assert_array_equal(self.dat.sum(axis=0), self.datsp.sum(axis=0)) File "/usr/lib64/python2.4/site-packages/scipy/sparse/sparse.py", line 402, in sum return o * self File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 162, in __mul__ return N.dot(self, other) ValueError: objects are not aligned ====================================================================== FAIL: test_explicit (scipy.tests.test_odr.test_odr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/odr/tests/test_odr.py", line 49, in test_explicit np.array([ 1.2646548050648876e+03, -5.4018409956678255e+01, File "/usr/lib64/python2.4/site-packages/numpy/testing/utils.py", line 230, in assert_array_almost_equal header='Arrays are not almost equal') File "/usr/lib64/python2.4/site-packages/numpy/testing/utils.py", line 215, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 1.26462971e+03, -5.42545890e+01, -8.64250389e-02]) y: array([ 1.26465481e+03, -5.40184100e+01, -8.78497122e-02]) ====================================================================== FAIL: test_multi (scipy.tests.test_odr.test_odr) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/scipy/odr/tests/test_odr.py", line 190, in test_multi np.array([ 4.3799880305938963, 2.4333057577497703, 8.0028845899503978, File "/usr/lib64/python2.4/site-packages/numpy/testing/utils.py", line 230, in assert_array_almost_equal header='Arrays are not almost equal') File "/usr/lib64/python2.4/site-packages/numpy/testing/utils.py", line 215, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 4.31272063, 2.44289312, 7.76215871, 0.55995622, 0.46423343]) y: array([ 4.37998803, 2.43330576, 8.00288459, 0.51011472, 0.51739023]) ---------------------------------------------------------------------- Ran 1629 tests in 3.773s FAILED (failures=2, errors=12) From svetosch at gmx.net Mon Mar 26 08:10:17 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 26 Mar 2007 14:10:17 +0200 Subject: [Numpy-discussion] Modular toolkit for Data Processing 2.1 released! In-Reply-To: <20070326100847.GB5038@lotka.biologie.hu-berlin.de> References: <20070326100847.GB5038@lotka.biologie.hu-berlin.de> Message-ID: <4607B829.7010508@gmx.net> Tiziano Zito schrieb: > The optional symeig module contains a Python wrapper for the LAPACK > functions to solve the standard and generalized eigenvalue problems > for symmetric (hermitian) positive definite matrices. Those > specialized algorithms give an important speed-up with respect to the > generic LAPACK eigenvalue problem solver used by NumPy. > This sounds very good. Is symeig completely independent from that other package, in the sense that it only depends on numpy? thanks, Sven From cjw at sympatico.ca Mon Mar 26 08:10:35 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon, 26 Mar 2007 08:10:35 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: Alan G Isaac wrote: >> Alan G Isaac wrote: >>> So this :: >>> >>> x[1] >>> matrix([[1, 0]]) >>> feels wrong. (Similarly when iterating across rows.) >>> Of course I realize that I can just :: >>> >>> x.A[1] >>> array([1, 0]) > > > On Sun, 25 Mar 2007, "Colin J. Williams" apparently wrote: >> An array and a matrix are different animals. Conversion >> from one to the other should be spelled out. > > > But you are just begging the question here. > The question is: when I iterate across matrix rows, > why am I iterating across matrices and not arrays. > This seems quite out of whack with general Python practice. > > You cannot just say "conversion should be explicit" > because that assumes (incorrectly actually) that > the rows are matrices. The "conversion should be explicit" > argument actually cuts in the opposite direction of what > you appear to believe. Alan, Yes, this is where we appear to differ. I believe that vectors are best represented as matrices, with a shape of (1, n) or (m, 1). The choice of these determines whether we have a column or a row vectors. Thus any (m, n) matrix can be considered as either a collection of column vectors or a collection of row vectors. If the end result is required as an array or a list, this can be done explicitly with X[1].A or A[1].tolist(). Here, A is a property of the M (matrix) class. > > Cheers, > Alan Isaac A long time ago, you proposed that PyMatrix should provide for matrix division in two way, as is done in MatLab. This was implemented, but PyMatrix has not yet been ported to numpy - perhaps this summer. Regards, Colin W. From steve at shrogers.com Mon Mar 26 08:11:29 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Mon, 26 Mar 2007 06:11:29 -0600 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <3E41CE22-B213-438C-AC5C-01BAEC5037E1@stanford.edu> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <46076B46.1040607@ukr.net> <07032610085200.00754@rbastian> <4607958B.8050306@ukr.net> <3E41CE22-B213-438C-AC5C-01BAEC5037E1@stanford.edu> Message-ID: <4607B871.4030600@shrogers.com> Zachary Pincus wrote: > > Anyhow, feel free to disagree with me -- I'm no expert here. I'm only > mentioning this as a public service to make it clear that most of > what's being proposed in this thread is, for better or worse, 100% > dead-in-the-water for Python 3, and the rest will have a fairly > significant uphill battle. > Don't disagree about the uphill battle part, but I believe that you're overly pessimistic about Unicode. Unicode variables and operators have only been rejected for Python 3.0, though the degree of Unicode support is still under discussion. If the core developers are convinced that enough people would use it, and it is implemented in a maintainable way, it is certainly possible for 3.1 or 3.2. # Steve From aisaac at american.edu Mon Mar 26 09:17:15 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 26 Mar 2007 09:17:15 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: > Perhaps things would be clearer if we thought of the > constituent groups of data in a matrix as being themselves > matrices. This "thinking of" is what you have suggested before. You need to explain why it is not begging the question. Cheers, Alan Isaac From ogdude at googlemail.com Mon Mar 26 09:26:47 2007 From: ogdude at googlemail.com (David Koch) Date: Mon, 26 Mar 2007 14:26:47 +0100 Subject: [Numpy-discussion] Adapting algorithm to accept Scipy sparse matrix In-Reply-To: References: <3C07E2F3-6FEC-4784-935C-3E0B8EF5D8D0@arachnedesign.net> Message-ID: Hi again, I want to select/access several columns from a sparse csc_matrix. The only way I could think of is the following enormously inefficient algorithm which basically initalizes a new lil_matrix (for assigments) and loops over all the specified columns and does sparse -> dense -> sparse. All this, to overcome the inability of using "multi-column" slices in csc_matrices. def spSelCol(X, A): "insert doc string" n = size(X,0) d = size(A) X = X.tocsc() newX = sparse.lil_matrix((d,n)) for i in range(0, d): # sparse -> dense -> sparse: not good! newX[i,:] = X[:,A[i]].toarray().flatten() return newX.transpose() Is there any way the operation can be made more efficient or should I look elsewhere (CVXOPT Python toolbox ...) Thanks, /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Mon Mar 26 09:31:06 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 26 Mar 2007 09:31:06 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <4607B788.1070606@gmx.net> References: <1174844924.6310.27.camel@localhost.localdomain><4607B788.1070606@gmx.net> Message-ID: > Alan G Isaac schrieb: >> >>> X[1] >> array([3,4]) >> >>> X[1,:] >> matrix([[3, 4]]) >> But again the point is: >> indexing for submatrices should produce matrices. >> Normal Python indexing should access the constituent arrays. On Mon, 26 Mar 2007, Sven Schreiber apparently wrote: > I think this is a tricky business. > There's also the rule "indexing reduces the rank, slicing preserves it". > Numpy-matrices form an exception to this rule, always being 2d, but the > exception is consistently enforced. > Now it seems you want to have an exception from the exception, correct? What I want: the best design. I did not claim that the current design is flawed, only to suspect it. Why I wrote: current behavior feels wrong -> suspect design flaw. What feels wrong: iterating over a container does not give access to the contained objects. This is not Pythonic. *Symptom* of the underlying problem: for matrix M, M[0] returns a matrix. Would the change I suggest mean that the behavior of the matrix class deviates less from the array class: yes. Does this mean the matrix class behavior would be less "consistent"? I have tried to explain why the answer is "no". hth, Alan Isaac From svetosch at gmx.net Mon Mar 26 09:48:08 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Mon, 26 Mar 2007 15:48:08 +0200 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain><4607B788.1070606@gmx.net> Message-ID: <4607CF18.8020300@gmx.net> Alan G Isaac schrieb: > > What feels wrong: iterating over a container does not give > access to the contained objects. This is not Pythonic. If you iterate over the rows of the matrix, it feels natural to me to get the row vectors -- and as you know a 1d-array does not contain the information of "row-ness", so you need to get a 2d thing to properly return those "contained objects". So in my view the current behavior actually does exactly what you're calling for. (But I admit I'm a 2d fan so much so that I didn't even know that using a single index is possible with a matrix. I thought one always had to be explicit about both dimensions... So thanks for pointing this out. -- BTW, I'm aware that sticking to numpy-matrices exclusively is not possible. For example, eigenvalues are returned in a 1d-array even if you pass a matrix, and it's intended behavior.) cheers, sven From aisaac at american.edu Mon Mar 26 09:54:26 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 26 Mar 2007 09:54:26 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <4607CF18.8020300@gmx.net> References: <1174844924.6310.27.camel@localhost.localdomain><4607B788.1070606@gmx.net><4607CF18.8020300@gmx.net> Message-ID: > Alan G Isaac schrieb: >> What feels wrong: iterating over a container does not give >> access to the contained objects. This is not Pythonic. On Mon, 26 Mar 2007, Sven Schreiber apparently wrote: > If you iterate over the rows of the matrix, it feels > natural to me to get the row vectors Natural in what way? Again, I am raising the question of what would be expected from someone familiar with Python. Abstractly, what do you expect to get when you iterate over a container? Seriously. > But I admit I'm a 2d fan so much so that I didn't even > know that using a single index is possible with a matrix. Exactly. When you want submatrices, you expect to index for them. EXACTLY!! Cheers, Alan Isaac From ogdude at googlemail.com Mon Mar 26 11:11:20 2007 From: ogdude at googlemail.com (David Koch) Date: Mon, 26 Mar 2007 16:11:20 +0100 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy Message-ID: Hi, I ran some tests on the very same matrices in Matlab/Numpy and it seems that for sparse matrix multipilcation to be faster than dense multiplication - the degree of sparsity has to be much higher in Numpy than in Matlab. Is there anything I can tune in the underlying routines? I need good performance at about 0.02 +- 0.01 nnz elements. Thanks, /David Matrix size: 10k * 1k, times are in seconds: sparsity: 0.01 nonzero elements Matlab dense 3.9, sparse 0.09 Numpy dense 2.9, sparse 15.24 sparsity: increase sparsity to 0.001 nnz Matlab dense 3.9, sparse 0.002 Numpy dense 2.9, sparse 0.27 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Mon Mar 26 11:17:34 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 26 Mar 2007 17:17:34 +0200 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: References: Message-ID: <4607E40E.5000300@ntc.zcu.cz> David Koch wrote: > Hi, > > I ran some tests on the very same matrices in Matlab/Numpy and it seems > that > for sparse matrix multipilcation to be faster than dense multiplication - > the degree of sparsity has to be much higher in Numpy than in Matlab. Is > there anything I can tune in the underlying routines? I need good > performance at about 0.02 +- 0.01 nnz elements. Hi David, Could you be more specific on which type of the sparse matrix storage did you use? They are not equivalent in performance w.r.t. different aspects, e.g. LIL, DOK matrices a good for populating an empty sparse matrix, while CSR, CSC should be used for computations. r. From haase at msg.ucsf.edu Mon Mar 26 11:27:16 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon, 26 Mar 2007 08:27:16 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <4607B788.1070606@gmx.net> <4607CF18.8020300@gmx.net> Message-ID: On 3/26/07, Alan G Isaac wrote: > > Alan G Isaac schrieb: > >> What feels wrong: iterating over a container does not give > >> access to the contained objects. This is not Pythonic. > > On Mon, 26 Mar 2007, Sven Schreiber apparently wrote: > > If you iterate over the rows of the matrix, it feels > > natural to me to get the row vectors > > Natural in what way? > Again, I am raising the question of what > would be expected from someone familiar with Python. > Abstractly, what do you expect to get when you iterate > over a container? Seriously. > > > > But I admit I'm a 2d fan so much so that I didn't even > > know that using a single index is possible with a matrix. > > Exactly. When you want submatrices, you expect to index for > them. EXACTLY!! > If may chime in... I think Sven's argument in on the side saying, A "matrix" is an object that you expect a certain (mathematical !) behavior from. If some object behaves intuitively right -- that's ultimately pythonic ! The clash is, NOT to see a matrix "just as another container". Instead a matrix is a mathematical object , that has rows and columns. It is used in a field (lin-alg) where every vector is either a row or a column vector -- apparently that's big thing ;-) The whole reason to add a special matrix class to numpy in the first place, is to provide a better degree of convenience to lin-alg related applications. I would argue that it was just not consistently considered, that this should also come with "a column of a matrix is something else than a row -- (1,n) vs. (n,1) and not (n,) more notes/points: a) I have never heard about the m.A1 - what is it ? b) I don't think that if m[1] would return a (rank 2) matrix, that m[1].A could return a (rank 1) array ... c) I'm curious if there is a unique way to extend the matrix class into 3D or ND. -Sebastian From daniel.wheeler at nist.gov Mon Mar 26 11:33:38 2007 From: daniel.wheeler at nist.gov (Daniel Wheeler) Date: Mon, 26 Mar 2007 15:33:38 +0000 Subject: [Numpy-discussion] header file location upon installation In-Reply-To: <4606ED3E.2000602@gmail.com> References: <4606ED3E.2000602@gmail.com> Message-ID: <38A66034-974D-4FA8-B5F1-FC374A214842@nist.gov> On Mar 25, 2007, at 9:44 PM, Robert Kern wrote: > For building extensions, either use numpy.distutils, which will > take care of > everything for you, or use numpy.get_include() to get the directory > with headers. get_include() does the trick. Thanks for your help. -- Daniel Wheeler -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogdude at googlemail.com Mon Mar 26 11:50:26 2007 From: ogdude at googlemail.com (David Koch) Date: Mon, 26 Mar 2007 16:50:26 +0100 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: <4607E40E.5000300@ntc.zcu.cz> References: <4607E40E.5000300@ntc.zcu.cz> Message-ID: On 3/26/07, Robert Cimrman wrote: > > Could you be more specific on which type of the sparse matrix storage > did you use? Hi Robert, I used csc_matrix. /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Mon Mar 26 11:53:33 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 26 Mar 2007 11:53:33 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <4607B788.1070606@gmx.net><4607CF18.8020300@gmx.net> Message-ID: On Mon, 26 Mar 2007, Sebastian Haase apparently wrote: > A "matrix" is an object that you expect a certain > (mathematical !) behavior from. If some object behaves > intuitively right -- that's ultimately pythonic! The problem is, as I am not the only one to point out, this particular behavior is NOT intuitively right. > The clash is, NOT to see a matrix "just as another > container". But be serious, no object is "just another container". Again, this just begs the question. The question is a design question. E.g., what is the principle of least surprise? > more notes/points: > a) I have never heard about the m.A1 - what is it ? It returns a 1d array holding the raveled matrix. > b) I don't think that if m[1] would return a (rank 2) > matrix, that m[1].A could return a (rank 1) array ... It does not, of course. (But both should, I believe.) > c) I'm curious if there is a unique way to extend the > matrix class into 3D or ND. Is that not what an array is for?? Cheers, Alan Isaac From cimrman3 at ntc.zcu.cz Mon Mar 26 11:59:09 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon, 26 Mar 2007 17:59:09 +0200 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: References: <4607E40E.5000300@ntc.zcu.cz> Message-ID: <4607EDCD.5020001@ntc.zcu.cz> David Koch wrote: > On 3/26/07, Robert Cimrman wrote: >> >> Could you be more specific on which type of the sparse matrix storage >> did you use? > > > > Hi Robert, > > I used csc_matrix. OK, good. Would you mind measuring csc * csr, csc * csc, csr * csc and csr * csr? I am curious how this will compare. r. ps: this thread might be more appropriate for scipy-user or scipy-dev... From cjw at sympatico.ca Mon Mar 26 12:05:18 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon, 26 Mar 2007 12:05:18 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Alan G Isaac wrote: > On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: >> Perhaps things would be clearer if we thought of the >> constituent groups of data in a matrix as being themselves >> matrices. > > This "thinking of" is what you have suggested before. > You need to explain why it is not begging the question. > > Cheers, > Alan Isaac Perhaps it would now help if you redefined the question. In an earlier posting, you appeared anxious that the matrix and the array behave in the same way. Since they are different animals, I see sameness of behaviour as being lower on the list of desirables than fitting the standard ideas of matrix algebra. Suppose that a is a row vector, b a column vector and A a conforming matrix then: a * A A * b and b.T * A are all acceptable operations. One would expect the iteration over A to return row vectors, represented by (1, n) matrices. Colin W. From daniel.egloff at zkb.ch Fri Mar 23 10:13:04 2007 From: daniel.egloff at zkb.ch (daniel.egloff at zkb.ch) Date: Fri, 23 Mar 2007 15:13:04 +0100 Subject: [Numpy-discussion] fastest way to do multiplication with diagonal matrices from left or right Message-ID: Dear list what is the fastet way to multiply with a diagonal matrix from left or right and without to build a square matrix from the diagonal. Here it what I am looking for: import numpy as N def diagmult(X, Y): """ Matrix multiplication X*Y where either X or Y is a diagonal matrix. """ if X.ndim == 1 and Y.ndim == 2: R = Y.copy() for i, d in enumerate(X): R[i,:] *= d return R elif X.ndim == 2 and Y.ndim == 1: R = X.copy() for i, d in enumerate(Y): R[:,i] *= d return R elif X.ndim == 1 and Y.ndim == 1: return X*Y else raise ValueError('diagmult dimension mismatch X.ndim = %d, Y.ndim = %d' % (X.ndim, Y.ndim)) Freundliche Gr?sse Daniel Egloff Z?rcher Kantonalbank Leiter(in) Financial Computing, ZEF Josefstrasse 222, 8005 Z?rich Telefon 044 292 45 33, Fax 044 292 45 95 Briefadresse: Postfach, 8010 Z?rich, http://www.zkb.ch ___________________________________________________________________ Disclaimer: Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimmt. Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden und anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopien zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verboten. This message is intended only for the named recipient and may contain confidential or privileged information. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Fri Mar 23 16:38:14 2007 From: pav at iki.fi (Pauli Virtanen) Date: Fri, 23 Mar 2007 22:38:14 +0200 Subject: [Numpy-discussion] Array of Arrays In-Reply-To: <525f23e80703230822g40093b38g4cbd6940499736b9@mail.gmail.com> References: <07C6A61102C94148B8104D42DE95F7E8C8F1FE@exchange2k.envision.co.il> <525f23e80703230822g40093b38g4cbd6940499736b9@mail.gmail.com> Message-ID: <46043AB6.4080009@iki.fi> Alexander Michael kirjoitti: > On 3/23/07, Nadav Horesh wrote: > >> How about >> >> a = empty((5,7,4)) >> c = a[...,-1] >> > > Solely because I want to use the array with code that assumes it is > working with two-dimensional arrays but yet only performs operations > on the "outer" two-dimensional array that would be consistent with an > "inner" array type (i.e. scalar assignment, element-wise > multiplication, etc.). I own all the code, so perhaps I can replace > a[mask,-1] with a[mask,-1,...] and such. Hmm. Not bad reminder, > thanks. > In some cases the 2d code works without modifications also for 3d-arrays, because in numpy a[1,2] is the same as a[1,2,...] for a 3d-array. For example, the examples in your first post run as intended: >>> from numpy import * >>> a = empty((5,7,4), dtype=float64) >>> c = a[:,-1] # shape (5,4) array, i.e. last column of 4 element arrays >>> a[0,0] = 2.0 >>> print a[0,0] [ 2. 2. 2. 2.] >>> a[1,0] = 3.0 >>> a[0,1] = a[0,0] * a[1,0] >>> print a[0,1] [ 6. 6. 6. 6.] -- Pauli Virtanen From zpincus at stanford.edu Sat Mar 24 15:49:00 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Sat, 24 Mar 2007 12:49:00 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <4604BA01.5040606@gemini.edu> References: <4604BA01.5040606@gemini.edu> Message-ID: <18BB7A69-CCCA-4810-B6C3-734065E9A652@stanford.edu> Hello folks, >> Hmm, this is worrisome. There really shouldn't be ringing on >> continuous-tone images like Lena -- right? (And at no step in an >> image like that should gaussian filtering be necessary if you're >> doing spline interpolation -- also right?) > > That's hard to say. Just because it's mainly a continuous-tone image > doesn't necessarily mean it is well sampled everywhere. This depends > both on the subject and the camera optics. Unlike the data I usually > work with, I think everyday digital photographs (probably a photo > scan in the case of Lena) do not generally have the detector sampling > frequency matched to the optical resolution of the image. If that's > true, the presence of aliasing in interpolated images depends on the > structure of the subject and whether the scene has edges or high- > frequency patterns in it. Based on my reading of the two excellent Unser papers (both the one that ndimage's spline code is based on, and the one that Travis linked to), it seems like a major point of using splines for interpolation is *better* behavior in the case of non-band-limited data than the traditional 'anti-aliasing [e.g. lowpass] filter' + 'resampling' + 'reconstruction [e.g. lowpass] filter' procedure. That is, based on my (very limited) understanding (I understand the basics of the sampling theorem and can follow the Unser paper's update thereof, but not much more than that), in the spline case the spline fitting step *replaces* the anti-aliasing filter in the above procedure as the method for dealing with non-band-limited data. And the claim is that it in fact works better in many cases. So it seems that something is wrong if the spline interpolation tools in ndimage only work properly in the band-limited case, or require lowpass filtering. > Stefan's rotated Lena example is indeed a bit bizarre on zooming in! > However, the artefacts are clearly localized to distinct edges, so I > suspect this is indeed some kind of aliasing. Moreover, it looks like > Lena has been decimated (reduced in size) prior to the rotation. That > is definitely a good way to get artefacts, unless an anti-aliasing > filter is applied before shrinking the image. My impression is that > this image is probably somewhat undersampled (to understand exactly > what that means, read up on the Sampling Theorem). Agreed, it looks like aliasing. Nevertheless, any resampling procedure is supposed to deal with this internally, right? Either by lowpass filtering (traditional case), or by spline fitting (spline case as described by Unser and understood by me) -- it shouldn't be letting aliasing bubble through, correct? >> The first was on Stefan's artificial data which had sharp edges, and >> got very nasty ringing artifacts even with 3rd order splines. From >> your recollection, is this expected behavior based on splines and the >> nature of Stefan's image, or more likely to be a bug? > > Your question was aimed at Travis, so I don't want to discourage him > from answering it :-), but looking at this in more detail, I do think > the amplitude of the artefacts here is greater than I might expect due > to ringing with a quadratic b-spline kernel, which I think has minima > with amplitudes <10% of the central peak. There has to be SOME > oscillation, but in Stefan's "rotate_artifacts" example it seems to be > at the level of ~100%. Also, it is not present on one of the inner > edges for some reason. So I do wonder if the algorithm in nd_image is > making this worse than it needs to be. Good observation! Here are cubic spline fits to a step and a delta function, which both have quite small amounts of ringing compared to what's observed -- at least on the color images. Maybe 10% ringing in each color channel adds up perceptually more than it does mathematically? import numpy, Gnuplot, scipy.interpolate # step x = numpy.arange(-10, 10) y = numpy.where(x > 0, 1, 0) tck = scipy.interpolate.splrep(x, y, s=0, k=3) nx = numpy.linspace(-10, 9, 200, True) ny = scipy.interpolate.splev(nx, tck) d = Gnuplot.Data(numpy.transpose([x, y]), with='points') nd = Gnuplot.Data(numpy.transpose([nx,ny]), with='lines') g = Gnuplot.Gnuplot() g.plot(d, nd) ny.min() # -0.107490563074 -- largest ringing is ~10% of step size # delta x = numpy.arange(-10, 10) y = numpy.where(x == 0, 1, 0) tck = scipy.interpolate.splrep(x, y, s=0, k=3) nx = numpy.linspace(-10, 9, 200, True) ny = scipy.interpolate.splev(nx, tck) d = Gnuplot.Data(numpy.transpose([x, y]), with='points') nd = Gnuplot.Data(numpy.transpose([nx,ny]), with='lines') g = Gnuplot.Gnuplot() g.plot(d, nd) ny.min() # -0.136449610107 -- largest ringing is ~13% of impulse size (sample plots are attached for reference). Now let's use ndimage to rotate a step function by 45%, or zoom in on it: import numpy, scipy.ndimage o = numpy.ones((100, 50), dtype=float) z = numpy.zeros((100, 50), dtype=float) a = numpy.concatenate([o, z], axis=1) b = scipy.ndimage.rotate(a, 45, order=3) b.max() # 1.08832325055 b.min() # -0.0883232505527 c = scipy.ndimage.zoom(a, 5, order=3) c.min() # -0.107600148039 c.max() # 1.10760014804 This (very simple) example looks exactly right. (Also, when I manually inspect the images in ImageJ -- which can load 32-bit floating point TIFFs, which is great -- they look correct). So what's going on with Lena and the other color/otherwise more complex images? Zach -------------- next part -------------- A non-text attachment was scrubbed... Name: delta.pdf Type: application/pdf Size: 15861 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: step.pdf Type: application/pdf Size: 16996 bytes Desc: not available URL: -------------- next part -------------- From beliavsky at aol.com Sun Mar 25 13:51:33 2007 From: beliavsky at aol.com (Beliavsky) Date: Sun, 25 Mar 2007 10:51:33 -0700 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: References: <4605801C.6030300@ee.byu.edu> Message-ID: <1174845093.568423.109320@e65g2000hsc.googlegroups.com> On Mar 24, 3:48 pm, "Charles R Harris" wrote: > > It is somewhat workable as it stands, but I think it would be nicer if > > we could have some "meta" operator that allowed an alternative > > definition of major operators. Something like @* for example (just > > picking a character that is already used for decorators). > > Yes indeed, this is an old complaint. Just having an infix operator would be > an improvement: > > A dot B dot C In Fortran 90 and later versions, A*B means element-wise multiplication, and there exist intrinsic functions such as matmul(A,B) and dot_product(A,B). One can define operators so that A .x. B .x. C is the same as matmul(A,matmul(B,C)) although I am unsure of the order of operations implied by the former syntax. The Matran http://www.cs.umd.edu/~stewart/matran/Matran.html package does this. I think Fortran's infix operators are a good solution but don't know if it can be carried over to Python. From berkes at gatsby.ucl.ac.uk Mon Mar 26 12:55:47 2007 From: berkes at gatsby.ucl.ac.uk (Pietro Berkes) Date: Mon, 26 Mar 2007 17:55:47 +0100 (BST) Subject: [Numpy-discussion] fastest way to do multiplication with diagonal matrices from left or right In-Reply-To: References: Message-ID: This is a small function I use to speed up multiplication with diagonal matrices. I don't know if it's the *fastest* way to do it, but it's pretty fast. def mult_diag(d, mtx, left=True): """Multiply a full matrix by a diagonal matrix. This function should always be faster than dot. Input: d -- 1D (N,) array (contains the diagonal elements) mtx -- 2D (N,N) array Output: mult_diag(d, mts, left=True) == dot(diag(d), mtx) mult_diag(d, mts, left=False) == dot(mtx, diag(d)) """ if left: return (d*mtx.T).T else: return d*mtx On Fri, 23 Mar 2007, daniel.egloff at zkb.ch wrote: > > > Dear list > > what is the fastet way to multiply with a diagonal matrix from left or > right and without to build a square matrix from the diagonal. > Here it what I am looking for: > > import numpy as N > > def diagmult(X, Y): > """ > Matrix multiplication X*Y where either X or Y is a diagonal matrix. > """ > if X.ndim == 1 and Y.ndim == 2: > R = Y.copy() > for i, d in enumerate(X): > R[i,:] *= d > return R > elif X.ndim == 2 and Y.ndim == 1: > R = X.copy() > for i, d in enumerate(Y): > R[:,i] *= d > return R > elif X.ndim == 1 and Y.ndim == 1: > return X*Y > else > raise ValueError('diagmult dimension mismatch X.ndim = %d, Y.ndim = > %d' % (X.ndim, Y.ndim)) > > Freundliche Gr?sse > Daniel Egloff > Z?rcher Kantonalbank > Leiter(in) Financial Computing, ZEF > > Josefstrasse 222, 8005 Z?rich > Telefon 044 292 45 33, Fax 044 292 45 95 > Briefadresse: Postfach, 8010 Z?rich, http://www.zkb.ch > ___________________________________________________________________ > > Disclaimer: > > > Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger bestimmt. > > Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird, > bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden und > anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige Kopien > zu vernichten bzw. zu loeschen. Der Gebrauch der Information ist verboten. > > > This message is intended only for the named recipient and may contain > confidential or privileged information. > > If you have received it in error, please advise the sender by return e-mail > and delete this message and any attachments. Any unauthorised use or > dissemination of this information is strictly prohibited. From aisaac at american.edu Mon Mar 26 12:57:15 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 26 Mar 2007 12:57:15 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: > One would expect the iteration over A to return row > vectors, represented by (1, n) matrices. This is again simple assertion. **Why** would "one" expect this? Some people clearly do not. One person commented that this unexpected behavior was a source of error in their code. Another person commented that they did not even guess that such a thing would be possible. Experience with Python should lead to the ability to anticipate the outcome. Apparently this is not the case. That suggests a design problem. What about **Python** would lead us to expect this behavior?? In *contrast*, everyone agrees that for a matrix M, we should get a matrix from M[0,:]. This is expected and desirable. Cheers, Alan Isaac From robert.kern at gmail.com Mon Mar 26 13:00:55 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 26 Mar 2007 12:00:55 -0500 Subject: [Numpy-discussion] fastest way to do multiplication with diagonal matrices from left or right In-Reply-To: References: Message-ID: <4607FC47.3020306@gmail.com> daniel.egloff at zkb.ch wrote: > Dear list > > what is the fastet way to multiply with a diagonal matrix from left or > right and without to build a square matrix from the diagonal. Use broadcasting to do your work for you. from numpy import array, newaxis diags = array([...]) mymatrix = array([[...]]) # From the right: mymatrix * diags # From the left: diags[:,newaxis] * mymatrix -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ogdude at googlemail.com Mon Mar 26 13:02:11 2007 From: ogdude at googlemail.com (David Koch) Date: Mon, 26 Mar 2007 18:02:11 +0100 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: <4607EDCD.5020001@ntc.zcu.cz> References: <4607E40E.5000300@ntc.zcu.cz> <4607EDCD.5020001@ntc.zcu.cz> Message-ID: Ok, I did and the results are: csc * csc: 372.601957083 csc * csc: 3.90811300278 csr * csc: 15.3202679157 csr * csr: 3.84498214722 Mhm, quite insightful. Note, that in an operation X.transpose() * X, where X is csc_matrix, then X.tranpose() is automatically cast to csr_matrix. A re-cast to csc make the whole operation faster. It's still about 1000 times slower than Matlab but 4 times faster than before. Note, that .transpose already switches the matrix On 3/26/07, Robert Cimrman wrote: > > David Koch wrote: > > On 3/26/07, Robert Cimrman wrote: > >> > >> Could you be more specific on which type of the sparse matrix storage > >> did you use? > > > > > > > > Hi Robert, > > > > I used csc_matrix. > > OK, good. Would you mind measuring csc * csr, csc * csc, csr * csc and > csr * csr? I am curious how this will compare. > > r. > > ps: this thread might be more appropriate for scipy-user or scipy-dev... > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tan2tan2 at gmail.com Mon Mar 26 13:18:04 2007 From: tan2tan2 at gmail.com (tan2) Date: Mon, 26 Mar 2007 10:18:04 -0700 Subject: [Numpy-discussion] New Operators in Python In-Reply-To: <07032610085200.00754@rbastian> References: <20070325184657.ADE32BA10C2@phaser.physics.ucf.edu> <46076B46.1040607@ukr.net> <07032610085200.00754@rbastian> Message-ID: On 3/26/07, Ren? Bastian wrote: > > May be it would be possible to implement a class with > user "definissable" (?) signs. > > Yes, it is possible and is done. See this recipe to define an Infix operator class either: x |op| y or: x <> y http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122 Eh Tan -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Mar 26 15:30:29 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 13:30:29 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: <4607B817.3050803@iam.uni-stuttgart.de> References: <4607B817.3050803@iam.uni-stuttgart.de> Message-ID: On 3/26/07, Nils Wagner wrote: > > Charles R Harris wrote: > > > > > > On 3/25/07, *Nils Wagner* > > wrote: > > > > Hi, > > > > Ok, things should be working now. In order to get rid of the scipy problems I just needed to rm the scipy package in site-packages and then reinstall. I still need to figure out a real fix down in the dot routine because In [1]: dot(mat(eye(2)),ones(2)) Out[1]: matrix([[ 1., 1.]]) Which is incorrect. But as long as you use * for matrices I think things are working correctly, or at least correctly by numpy convention. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Mar 26 15:51:36 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 13:51:36 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: On 3/26/07, Alan G Isaac wrote: > > On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: > > One would expect the iteration over A to return row > > vectors, represented by (1, n) matrices. > > This is again simple assertion. > **Why** would "one" expect this? > Some people clearly do not. Well, and what *should* they expect. I think it is expected because the for iterates over rows and rows of matrices are 1xn. Matrices and arrays, as has been stated, are different animals. Probably it would have been best to stick with arrays and I suspect that matrices appeared because of the dearth of Python operators, in particular to make matrix multiplication simpler. On the other hand, certain errors slip by when one is implementing matrix algebra with arrays, but they can be avoided by never using 1-d vectors. So all this mess results from catering to the matrix community. Matlab has the opposite problem, multidimensional arrays were tacked on later and they don't operate properly with everything. Chuck One person commented that this unexpected behavior was > a source of error in their code. > > Another person commented that they did not even guess that > such a thing would be possible. > > Experience with Python should lead to the ability to > anticipate the outcome. Apparently this is not the case. > That suggests a design problem. > > What about **Python** would lead us to expect this behavior?? > > In *contrast*, everyone agrees that for a matrix M, > we should get a matrix from M[0,:]. > This is expected and desirable. > > Cheers, > Alan Isaac > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 26 16:03:48 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 13:03:48 -0700 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: <4607B817.3050803@iam.uni-stuttgart.de> Message-ID: <46082724.7010208@ee.byu.edu> Charles R Harris wrote: > > > On 3/26/07, *Nils Wagner* > wrote: > > Charles R Harris wrote: > > > > > > On 3/25/07, *Nils Wagner* > > >> wrote: > > > > Hi, > > > > > > > > Ok, things should be working now. In order to get rid of the scipy > problems I just needed to rm the scipy package in site-packages and > then reinstall. > > I still need to figure out a real fix down in the dot routine because No, don't fix the dot routine. Remember the dot routine should accept mixed 2-d arrays and 1-d arrays because it is more general than matrix multiplication. -Travis From aisaac at american.edu Mon Mar 26 16:18:04 2007 From: aisaac at american.edu (Alan G Isaac) Date: Mon, 26 Mar 2007 16:18:04 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: >> On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: >>> One would expect the iteration over A to return row >>> vectors, represented by (1, n) matrices. > On 3/26/07, Alan G Isaac wrote: >> This is again simple assertion. >> **Why** would "one" expect this? >> Some people clearly do not. On Mon, 26 Mar 2007, Charles R Harris apparently wrote: > Well, and what should they expect. Since matrices are an iterable Python object, we *expect* to iterate over the contained objects. (Arrays.) I am not sure why this is not evident to all, but it is surely the sticking point in this discussion. A matrix is not a container of matrices. That it acts like one is surprsing. Surprises are bad unless they have a clear justification. Perhaps a clear justification exists, but it has not been offered in this discussion. Cheers, Alan Isaac From charlesr.harris at gmail.com Mon Mar 26 16:22:13 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 14:22:13 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: <46082724.7010208@ee.byu.edu> References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> Message-ID: On 3/26/07, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > > > On 3/26/07, *Nils Wagner* > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > On 3/25/07, *Nils Wagner* > > > > > >> wrote: > > > > > > Hi, > > > > > > > > > > > > > > Ok, things should be working now. In order to get rid of the scipy > > problems I just needed to rm the scipy package in site-packages and > > then reinstall. > > > > I still need to figure out a real fix down in the dot routine because > > > No, don't fix the dot routine. Remember the dot routine should accept > mixed 2-d arrays and 1-d arrays because it is more general than matrix > multiplication. Sure, but it can't return a row vector when multiplying a 1-d vector. It just ain't mathematically possible no matter how the 1-d vector is interpreted. It needs to return either a column vector or a 1-d vector. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From zpincus at stanford.edu Mon Mar 26 16:29:26 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon, 26 Mar 2007 13:29:26 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: <7C1BD563-0C6E-4B83-9EAC-720F8B6E5F43@stanford.edu> > Since matrices are an iterable Python object, > we *expect* to iterate over the contained objects. > (Arrays.) I am not sure why this is not evident to all, > but it is surely the sticking point in this discussion. > > A matrix is not a container of matrices. > That it acts like one is surprsing. > Surprises are bad unless they have a clear justification. > Perhaps a clear justification exists, > but it has not been offered in this discussion. I think that a clear justification has been offered several times on the list recently, though maybe not all in this thread. Matrices in numpy seem to exist as a construct to facilitate linear algebra. One such property is that row- and column-vector slices of matrices are (1xN) or (Nx1) matrices, because otherwise common linear algebra things -- like how you multiply a row-vector or a column vector by a matrix, and whether and when it needs to be transposed -- do not translate properly from "linear algebra notation" like in textbooks and papers. Once the matrix class is committed to providing row- and column- vector slices as other matrices, it makes no sense to have iteration over the matrix provide a different data type than slicing. Basically, my rule of thumb has been to *only* use matrices when I'm copying linear algebra operations out of textbooks and papers, and I want the notations to align. Doing other, non-linear-algebra operations with matrices -- like iterating over their elements -- is not really worth it. There's a meta question, of course: should things be changed to make it "worth it" to do "pythonic" tasks with matrices? Should there be special elementwise vs. matrix-operation operators? Should numpy work a lot more like matlab? That discussion is on-going in another thread, I think. Zach From oliphant at ee.byu.edu Mon Mar 26 16:30:17 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 13:30:17 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: <46082D59.5000509@ee.byu.edu> Alan G Isaac wrote: >>>On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: >>> >>> >>>>One would expect the iteration over A to return row >>>>vectors, represented by (1, n) matrices. >>>> >>>> > > > > >>On 3/26/07, Alan G Isaac wrote: >> >> >>>This is again simple assertion. >>>**Why** would "one" expect this? >>>Some people clearly do not. >>> >>> > > >On Mon, 26 Mar 2007, Charles R Harris apparently wrote: > > >>Well, and what should they expect. >> >> > > >Since matrices are an iterable Python object, >we *expect* to iterate over the contained objects. >(Arrays.) I am not sure why this is not evident to all, >but it is surely the sticking point in this discussion. > >A matrix is not a container of matrices. >That it acts like one is surprsing. >Surprises are bad unless they have a clear justification. >Perhaps a clear justification exists, >but it has not been offered in this discussion. > > It actually has been offered. You just don't accept it. Matrices are containers of matrices. If M is an (mxn) matrix then M[0] is a (1xn) matrix. Viewing this 1xn matrix as a 1-d array loses it's row-vectorness. This seems perfectly logical and acceptable to me. I'm waiting for a better explanation as to why this is not acceptable. Arguments that rest on what is and what isn't "Pythonic" are not very persuasive as this is very often in the eye of the beholder. -Travis From charlesr.harris at gmail.com Mon Mar 26 16:31:59 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 14:31:59 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: On 3/26/07, Alan G Isaac wrote: > > >> On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: > >>> One would expect the iteration over A to return row > >>> vectors, represented by (1, n) matrices. > > > > On 3/26/07, Alan G Isaac wrote: > >> This is again simple assertion. > >> **Why** would "one" expect this? > >> Some people clearly do not. > > > On Mon, 26 Mar 2007, Charles R Harris apparently wrote: > > Well, and what should they expect. > > > Since matrices are an iterable Python object, > we *expect* to iterate over the contained objects. > (Arrays.) I am not sure why this is not evident to all, > but it is surely the sticking point in this discussion. > > Matrices don't contain arrays. Indeed, in linear algebra they are elements of the tensor product of a vector space with its dual, so one would expect either row or column vectors depending on the index position. But this discussion is not really relevant at this point. What happens is a convention and the numpy convention is out there. It isn't going to change now, it would break too much code. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 26 16:38:04 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 13:38:04 -0700 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> Message-ID: <46082F2C.6060901@ee.byu.edu> Charles R Harris wrote: > > > On 3/26/07, *Travis Oliphant* > wrote: > > Charles R Harris wrote: > > > > > > > On 3/26/07, *Nils Wagner* > > >> wrote: > > > > Charles R Harris wrote: > > > > > > > > > On 3/25/07, *Nils Wagner* > > > > > > > > >>> wrote: > > > > > > Hi, > > > > > > > > > > > > > > Ok, things should be working now. In order to get rid of the scipy > > problems I just needed to rm the scipy package in site-packages and > > then reinstall. > > > > I still need to figure out a real fix down in the dot routine > because > > > No, don't fix the dot routine. Remember the dot routine should accept > mixed 2-d arrays and 1-d arrays because it is more general than matrix > multiplication. > > > Sure, but it can't return a row vector when multiplying a 1-d vector. > It just ain't mathematically possible no matter how the 1-d vector is > interpreted. It needs to return either a column vector or a 1-d vector. Ah.. I see what your point is. How to fix that. The problem is choosing the Python type of the output array. It's being selected as matrix right now because of the precedence of the matrix subtype, but it should be chosen as array in this instance. Perhaps we should make dot always choose the array as the output type?? -Travis From oliphant at ee.byu.edu Mon Mar 26 16:52:28 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 13:52:28 -0700 Subject: [Numpy-discussion] Fixed scalar coercion model in NumPy Message-ID: <4608328C.4030000@ee.byu.edu> I've finally made the changes to fix the scalar coercion model problems in NumPy 1.0.1 Now, scalar coercion rules only apply when involved types are of the same basic "kind". Thus, array([1,2,3],int8)*10 returns an int8 array but array([1,2,3],int8)*10.0 returns a float64 array. If you want a float32 array you must use array([1,2,3],int8)*float32(10.0) This is actually a behavioral change which is why I asked earlier if we should change it. However, the previous behavior was not documented anywhere and any previous discussion on this point should have been interpreted as the behavior now in SVN. It's also a rare use-case and so should not create too many issues. In running tests with NumPy and SciPy there appear to be three tests in ndimage breaking now. I really do want to get 1.0.2 out the door soon. What still needs to be fixed before then? -Travis From perry at stsci.edu Mon Mar 26 17:15:13 2007 From: perry at stsci.edu (Perry Greenfield) Date: Mon, 26 Mar 2007 17:15:13 -0400 Subject: [Numpy-discussion] Fixed scalar coercion model in NumPy In-Reply-To: <4608328C.4030000@ee.byu.edu> References: <4608328C.4030000@ee.byu.edu> Message-ID: Great! On Mar 26, 2007, at 4:52 PM, Travis Oliphant wrote: > I've finally made the changes to fix the scalar coercion model > problems > in NumPy 1.0.1 > > Now, scalar coercion rules only apply when involved types are of the > same basic "kind". [...] From oliphant at ee.byu.edu Mon Mar 26 17:30:25 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 14:30:25 -0700 Subject: [Numpy-discussion] Fixed scalar coercion model in NumPy In-Reply-To: References: <4608328C.4030000@ee.byu.edu> Message-ID: <46083B71.1070705@ee.byu.edu> Perry Greenfield wrote: >Great! > >On Mar 26, 2007, at 4:52 PM, Travis Oliphant wrote: > > > >>I've finally made the changes to fix the scalar coercion model >>problems >>in NumPy 1.0.1 >> >>Now, scalar coercion rules only apply when involved types are of the >>same basic "kind". >> >> Actually, the rule is better stated. The scalar coercion rules apply when involved types are of the same basic "kind" or if the scalar is of "lesser" kind than the array. Thus, array([1,2,3],float32)*10 produces a float32 array array([1,2,3],float32)*10.0 produces a float32 array array([1,2,3],float32)*10.0j produces a complex128 array -Travis From charlesr.harris at gmail.com Mon Mar 26 17:43:36 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 15:43:36 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: <46082F2C.6060901@ee.byu.edu> References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> Message-ID: On 3/26/07, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > > > On 3/26/07, *Travis Oliphant* > > wrote: > > > > Charles R Harris wrote: > > > > > > > > > > > On 3/26/07, *Nils Wagner* > > > > > >> wrote: > > > > > > Charles R Harris wrote: > > > > > > > > > > > > On 3/25/07, *Nils Wagner* > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Ok, things should be working now. In order to get rid of the scipy > > > problems I just needed to rm the scipy package in site-packages > and > > > then reinstall. > > > > > > I still need to figure out a real fix down in the dot routine > > because > > > > > > No, don't fix the dot routine. Remember the dot routine should > accept > > mixed 2-d arrays and 1-d arrays because it is more general than > matrix > > multiplication. > > > > > > Sure, but it can't return a row vector when multiplying a 1-d vector. > > It just ain't mathematically possible no matter how the 1-d vector is > > interpreted. It needs to return either a column vector or a 1-d vector. > > Ah.. I see what your point is. > > How to fix that. The problem is choosing the Python type of the output > array. It's being selected as matrix right now because of the > precedence of the matrix subtype, but it should be chosen as array in > this instance. Perhaps we should make dot always choose the array as > the output type?? I think that might be the simplest thing, dot overrides subtypes. BTW, here is another ambiguity In [6]: dot(array([[1]]),ones(2)) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/charris/ ValueError: matrices are not aligned Note that in this case dot acts like the rhs is always a column vector although it returns a 1-d vector. I don't know that this is a bad thing, but perhaps we should extend this behaviour to matrices, which would be different from the now current 1-d is always a *row* vector, i.e. In [8]: mat([[1]])*ones(2) Out[8]: matrix([[ 1., 1.]]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 26 18:02:02 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 15:02:02 -0700 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> Message-ID: <460842DA.3070000@ee.byu.edu> > I think that might be the simplest thing, dot overrides subtypes. BTW, > here is another ambiguity > > In [6]: dot(array([[1]]),ones(2)) > --------------------------------------------------------------------------- > exceptions.ValueError Traceback (most > recent call last) > > /home/charris/ > > ValueError: matrices are not aligned > > Note that in this case dot acts like the rhs is always a column vector > although it returns a 1-d vector. I don't know that this is a bad > thing, but perhaps we should extend this behaviour to matrices, which > would be different from the now current 1-d is always a *row* vector, i.e. The rule 1-d is always a *row* vector only applies when converting to a matrix. In this case, the dot operator does not "convert to a matrix" but uses rules for operating with mixed 2-d and 1-d arrays inherited from Numeric. I'm very hesitant to change those rules. -Travis From stefan at sun.ac.za Mon Mar 26 18:14:15 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 27 Mar 2007 00:14:15 +0200 Subject: [Numpy-discussion] Fixed scalar coercion model in NumPy In-Reply-To: <4608328C.4030000@ee.byu.edu> References: <4608328C.4030000@ee.byu.edu> Message-ID: <20070326221414.GF13460@mentat.za.net> On Mon, Mar 26, 2007 at 01:52:28PM -0700, Travis Oliphant wrote: > I really do want to get 1.0.2 out the door soon. What still needs to be > fixed before then? The code in ticket 469 still causes a memory error, so that might be worth fixing. Cheers St?fan From charlesr.harris at gmail.com Mon Mar 26 18:30:49 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 16:30:49 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: <460842DA.3070000@ee.byu.edu> References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> <460842DA.3070000@ee.byu.edu> Message-ID: On 3/26/07, Travis Oliphant wrote: > > > > I think that might be the simplest thing, dot overrides subtypes. BTW, > > here is another ambiguity > > > > In [6]: dot(array([[1]]),ones(2)) > > > --------------------------------------------------------------------------- > > exceptions.ValueError Traceback (most > > recent call last) > > > > /home/charris/ > > > > ValueError: matrices are not aligned > > > > Note that in this case dot acts like the rhs is always a column vector > > although it returns a 1-d vector. I don't know that this is a bad > > thing, but perhaps we should extend this behaviour to matrices, which > > would be different from the now current 1-d is always a *row* vector, > i.e. > > > The rule 1-d is always a *row* vector only applies when converting to a > matrix. > > In this case, the dot operator does not "convert to a matrix" but uses > rules for operating with mixed 2-d and 1-d arrays inherited from Numeric. > > I'm very hesitant to change those rules. I wasn't suggesting that, just noticing that the rule was 1-d vector on right is treated as a column vector by dot, which is why an exception was raised in the posted case. If it is traditional for matrix routines always treat is as a row vector, so be it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 26 18:36:59 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 26 Mar 2007 15:36:59 -0700 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> <460842DA.3070000@ee.byu.edu> Message-ID: <46084B0B.10807@ee.byu.edu> Charles R Harris wrote: > > > The rule 1-d is always a *row* vector only applies when converting > to a > matrix. > > In this case, the dot operator does not "convert to a matrix" but > uses > rules for operating with mixed 2-d and 1-d arrays inherited from > Numeric. > > I'm very hesitant to change those rules. > > > I wasn't suggesting that, just noticing that the rule was 1-d vector > on right is treated as a column vector by dot, which is why an > exception was raised in the posted case. If it is traditional for > matrix routines always treat is as a row vector, so be it. O.K. So, the problem is that when I defined matrix multiplication, I mistakenly believed that dot always interpreted 1-d arrays as row vectors which it is now clear it doesn't. I think this led to the issues. So, making dot always return arrays seems like the needed fix. -Travis From charlesr.harris at gmail.com Mon Mar 26 19:07:52 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Mar 2007 17:07:52 -0600 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: <46084B0B.10807@ee.byu.edu> References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> <460842DA.3070000@ee.byu.edu> <46084B0B.10807@ee.byu.edu> Message-ID: On 3/26/07, Travis Oliphant wrote: > > Charles R Harris wrote: > > > > > > > The rule 1-d is always a *row* vector only applies when converting > > to a > > matrix. > > > > In this case, the dot operator does not "convert to a matrix" but > > uses > > rules for operating with mixed 2-d and 1-d arrays inherited from > > Numeric. > > > > I'm very hesitant to change those rules. > > > > > > I wasn't suggesting that, just noticing that the rule was 1-d vector > > on right is treated as a column vector by dot, which is why an > > exception was raised in the posted case. If it is traditional for > > matrix routines always treat is as a row vector, so be it. > > O.K. So, the problem is that when I defined matrix multiplication, I > mistakenly believed that dot always interpreted 1-d arrays as row > vectors which it is now clear it doesn't. > > I think this led to the issues. > > So, making dot always return arrays seems like the needed fix. Sounds good. Do you want to leave the matrix * operator as it now is? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.bedo at nicta.com.au Mon Mar 26 20:20:41 2007 From: justin.bedo at nicta.com.au (Justin Bedo) Date: Tue, 27 Mar 2007 10:20:41 +1000 Subject: [Numpy-discussion] ATLAS problems Message-ID: <8c1504fc0703261720l93cf8cfmf9c8aa1c1fef1a17@mail.gmail.com> Hi All, I'm trying to compile NumPy using the ATLAS libraries but am having some difficulty with it. I've managed to build numpy such that lapack_lite.so is linked as follows: linux-gate.so.1 => (0xffffe000) libatlas.so => /usr/local/atlas/lib/libatlas.so (0xb79e9000) libptcblas.so => /usr/local/atlas/lib/libptcblas.so (0xb79c7000) libptf77blas.so => /usr/local/atlas/lib/libptf77blas.so (0xb79ac000) libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb788f000) libgfortran.so.1 => /usr/lib/libgfortran.so.1 (0xb7817000) libm.so.6 => /lib/libm.so.6 (0xb77f1000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb77e5000) libc.so.6 => /lib/libc.so.6 (0xb76b7000) libpthread.so.0 => /lib/libpthread.so.0 (0xb769f000) libdl.so.2 => /lib/libdl.so.2 (0xb769a000) libutil.so.1 => /lib/libutil.so.1 (0xb7696000) /lib/ld-linux.so.2 (0x80000000) However despite this, the performance when multiplying matrices is extremely slow, and only a single thread is used even though lapack_lite.so is linked with the pt atlas libraries. Does anybody know how to fix this problem? Thanks Justin From aisaac at american.edu Tue Mar 27 01:05:02 2007 From: aisaac at american.edu (Alan Isaac) Date: Tue, 27 Mar 2007 00:05:02 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: On Mon, 26 Mar 2007, Charles R Harris wrote: > What happens is a convention Certainly true. > It isn't going to change now, it would break too much > code. That is a different kind of argument. It might be true. May I see a use case where the desired return when iterating through a matrix is rows as matrices? That has never been what I wanted. Thank you, Alan Isaac From aisaac at american.edu Tue Mar 27 01:05:06 2007 From: aisaac at american.edu (Alan Isaac) Date: Tue, 27 Mar 2007 00:05:06 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <46082D59.5000509@ee.byu.edu> References: <46082D59.5000509@ee.byu.edu> Message-ID: On Mon, 26 Mar 2007, Travis Oliphant wrote: > It actually has been offered. You just don't accept it. > Matrices are containers of matrices. > If M is an (mxn) matrix then M[0] is a (1xn) matrix. > Viewing this 1xn matrix as a 1-d array loses it's row-vectorness. > This seems perfectly logical and acceptable to me. I'm waiting for a > better explanation as to why this is not acceptable. > Arguments that rest on what is and what isn't "Pythonic" are not very > persuasive as this is very often in the eye of the beholder. Again, I only raised a *question* about whether there might be a design problem here. My goal was only to have this explored, and I've tried to explain why. The evidence I offer: - it is surprising (at least to some) that iterating over a matrix yields matrices - I believe it is unexpected (prior to instruction) and that there is a natural more expected behavior - if that is right, deviation from the expected should have a good justification - this behavior has tripped up at least a couple people and I expect that to happen to many others over time (because I believe the behavior is unexpected) - when I desire to iterate over a matrix I always want the arrays. (Of course there is a way to have them; that's not the point). I'm interested to see a use case where the rows are desired as matrices As you indicate, none of this constitutes an "argument". And since nobody appears to agree, I should shut up. This will be my last post on this subject. Cheers, Alan Isaac From wbaxter at gmail.com Tue Mar 27 00:21:23 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 27 Mar 2007 13:21:23 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: On 3/27/07, Alan Isaac wrote: > May I see a use case where the desired > return when iterating through a matrix > is rows as matrices? That has never > been what I wanted. If you use a row vector convention it make plenty of sense. AllMyPoints = mat(rand(100,2)) # 100 two-d points for pt in AllMyPoints: xformedPt = pt * TransformMatrix # do something to transformed point --bb From aisaac at american.edu Tue Mar 27 01:54:55 2007 From: aisaac at american.edu (Alan Isaac) Date: Tue, 27 Mar 2007 00:54:55 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: > On 3/27/07, Alan Isaac wrote: >> May I see a use case where the desired >> return when iterating through a matrix >> is rows as matrices? That has never >> been what I wanted. On Tue, 27 Mar 2007, Bill Baxter wrote: > AllMyPoints = mat(rand(100,2)) # 100 two-d points > for pt in AllMyPoints: > xformedPt = pt * TransformMatrix > # do something to transformed point This seems artificial to me for several reasons, but here is one reason: AllxformedPts = AllMyPoints * TransformMatrix Note that I am no longer disputing the convention, just trying to understand its usefulness. Cheers, Alan Isaac From wbaxter at gmail.com Tue Mar 27 02:14:04 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Tue, 27 Mar 2007 15:14:04 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: On 3/27/07, Alan Isaac wrote: > > On 3/27/07, Alan Isaac wrote: > >> May I see a use case where the desired > >> return when iterating through a matrix > >> is rows as matrices? That has never > >> been what I wanted. > > > On Tue, 27 Mar 2007, Bill Baxter wrote: > > AllMyPoints = mat(rand(100,2)) # 100 two-d points > > for pt in AllMyPoints: > > xformedPt = pt * TransformMatrix > > # do something to transformed point > > > This seems artificial to me for several reasons, > but here is one reason: > AllxformedPts = AllMyPoints * TransformMatrix Yeh, I was afraid you'd say that. :-) But maybe I've got a lot of points, and I don't feel like making a copy of the whole set. Or maybe it's not a linear transform, but instead xformedPt = someComplicatedNonLinearThing(pt) I do stuff like the above quite frequently in my code, although with arrays rather than matrices. :-) For instance in finite elements there's assembling the global stiffness matrix step where for each node (point) in your mesh you set some entries in the big matrix K. Something like for i,pt in enumerate(points): K[shape_fn_indices(i)] = stiffness_fn(pt) That's cartoon code, but I think you get the idea. I don't think there's any good way to make that into one vectorized expression. The indices of K that get set depend on the connectivity of your mesh. > Note that I am no longer disputing the convention, > just trying to understand its usefulness. --bb From stefan at sun.ac.za Tue Mar 27 02:45:01 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Tue, 27 Mar 2007 08:45:01 +0200 Subject: [Numpy-discussion] two previously unresolved issues Message-ID: <20070327064501.GH13460@mentat.za.net> Hi, I just went through my mail archive and found these two minor outstanding issues. Thought I'd ask for comments before the new release: """ From: "Charles R Harris" Subject: Re: [Numpy-discussion] Assign NaN, get zero On 11/11/06, Lisandro Dalcin wrote: > > On 11/11/06, Stefan van der Walt wrote: > > NaN (or inf) is a floating point number, so seeing a zero in integer > > representation seems correct: > > > > In [2]: int(N.nan) > > Out[2]: 0L > > > > Just to learn myself: Why int(N.nan) should be 0? Is it C behavior? In [1]: int32(0)/int32(0) Warning: divide by zero encountered in long_scalars Out[1]: 0 In [2]: float32(0)/float32(0) Out[2]: nan In [3]: int(nan) Out[3]: 0L I think it was just a default for numpy. Hmmm, numpy now warns on integer division by zero, didn't used to. Looks like a warning should also be raised when casting nan to integer. It is probably a small bug not to. I also suspect int(nan) should return a normal python zero, not 0L. """ """ From: "Bill Baxter" To: numpy-discussion at scipy.org Subject: [Numpy-discussion] linalg.lstsq for complex Is this code from linalg.lstsq for the complex case correct? lapack_routine = lapack_lite.zgelsd lwork = 1 rwork = zeros((lwork,), real_t) work = zeros((lwork,),t) results = lapack_routine(m, n, n_rhs, a, m, bstar, ldb, s, rcond, 0, work, -1, rwork, iwork, 0) lwork = int(abs(work[0])) rwork = zeros((lwork,),real_t) a_real = zeros((m,n),real_t) bstar_real = zeros((ldb,n_rhs,),real_t) results = lapack_lite.dgelsd(m, n, n_rhs, a_real, m, bstar_real, ldb, s, rcond, 0, rwork, -1, iwork, 0) lrwork = int(rwork[0]) work = zeros((lwork,), t) rwork = zeros((lrwork,), real_t) results = lapack_routine(m, n, n_rhs, a, m, bstar, ldb, s, rcond, The middle call to dgelsd looks unnecessary to me. At the very least, allocating astar_real and bstar_real shouldn't be necessary since they aren't referenced anywhere else in the lstsq function. The lapack documentation for zgelsd also doesn't mention any need to call dgelsd to compute the size of the work array. """ Cheers St?fan From svetosch at gmx.net Tue Mar 27 08:00:51 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Tue, 27 Mar 2007 14:00:51 +0200 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: <46084B0B.10807@ee.byu.edu> References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> <460842DA.3070000@ee.byu.edu> <46084B0B.10807@ee.byu.edu> Message-ID: <46090773.3060309@gmx.net> Travis Oliphant schrieb: > O.K. So, the problem is that when I defined matrix multiplication, I > mistakenly believed that dot always interpreted 1-d arrays as row > vectors which it is now clear it doesn't. > > I think this led to the issues. > > So, making dot always return arrays seems like the needed fix. > Does that mean that... a) dot(a,b) will return an array even if a and b are both numpy-matrices? and/or b) dot(c,d) will return an array if c or d is a numpy-matrix? Thanks for any clarifications, Sven From aisaac at american.edu Tue Mar 27 08:27:16 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 27 Mar 2007 08:27:16 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: On Tue, 27 Mar 2007, Bill Baxter apparently wrote: > xformedPt = someComplicatedNonLinearThing(pt) > I do stuff like the above quite frequently in my code, > although with arrays rather than matrices. Exactly: that was one other thing I found artificial. Surely the points will then be wanted as arrays. So my view is that we still do not have a use case for wanting matrices yielded when iterating across rows of a matrix. Alan Isaac PS In the "large number of points" case, the thing to do would be to extract modest sized submatrices. From simon at arrowtheory.com Tue Mar 27 11:20:13 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 27 Mar 2007 08:20:13 -0700 Subject: [Numpy-discussion] ATLAS problems In-Reply-To: <8c1504fc0703261720l93cf8cfmf9c8aa1c1fef1a17@mail.gmail.com> References: <8c1504fc0703261720l93cf8cfmf9c8aa1c1fef1a17@mail.gmail.com> Message-ID: <20070327082013.87641e08.simon@arrowtheory.com> The last time I had this problem i ended up running python in gdb and setting breakpoints at the appropriate dgemm calls. That way i could see exactly what numpy was using to do dgemm. Simon. On Tue, 27 Mar 2007 10:20:41 +1000 "Justin Bedo" wrote: > Hi All, > > I'm trying to compile NumPy using the ATLAS libraries but am having > some difficulty with it. I've managed to build numpy such that > lapack_lite.so is linked as follows: > > linux-gate.so.1 => (0xffffe000) > libatlas.so => /usr/local/atlas/lib/libatlas.so (0xb79e9000) > libptcblas.so => /usr/local/atlas/lib/libptcblas.so (0xb79c7000) > libptf77blas.so => /usr/local/atlas/lib/libptf77blas.so (0xb79ac000) > libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb788f000) > libgfortran.so.1 => /usr/lib/libgfortran.so.1 (0xb7817000) > libm.so.6 => /lib/libm.so.6 (0xb77f1000) > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb77e5000) > libc.so.6 => /lib/libc.so.6 (0xb76b7000) > libpthread.so.0 => /lib/libpthread.so.0 (0xb769f000) > libdl.so.2 => /lib/libdl.so.2 (0xb769a000) > libutil.so.1 => /lib/libutil.so.1 (0xb7696000) > /lib/ld-linux.so.2 (0x80000000) > > However despite this, the performance when multiplying matrices is > extremely slow, and only a single thread is used even though > lapack_lite.so is linked with the pt atlas libraries. Does anybody > know how to fix this problem? > > Thanks > Justin > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From zpincus at stanford.edu Tue Mar 27 13:16:47 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue, 27 Mar 2007 10:16:47 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: Message-ID: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> > Exactly: that was one other thing I found artificial. > Surely the points will then be wanted as arrays. > > So my view is that we still do not have a use case > for wanting matrices yielded when iterating across > rows of a matrix. It's pretty clear from my perspective: 1-D slices of matrices *must* be matrices. I gave an intuitive make-it-fit-with-known-notation explanation, and Charles gave a theoretically-grounded explanation. There was no objection to this from any quarter. Given that 1-D slices of matrices must be matrices for deep mathematical reasons as well as notational compatibility with the rest of linear algebra -- e.g. that m[0] yields a matrix if m is a matrix-- it almost certainly would violate the principle of least surprise for iteration over m (intuitively understood to be choosing m [0] then m[1] and so forth) to yield anything other than a matrix. This can't possibly be what you're asking for, right? You aren't suggesting that m[0] and list(iter(m))[0] should be different types? There are many clear and definite use cases for m[0] being a matrix, by the way, in addition to the theoretical arguments -- these aren't hard to come by. Whether or nor there are actual good use-cases for iterating over a matrix, if you believe that m[0] should be a matrix, it's madness to suggest that list(iter(m))[0] should be otherwise. My opinion? Don't iterate over matrices. Use matrices for linear algebra. There's no "iteration" construct in linear algebra. The behavior you find so puzzling is a necessary by-product of the fundamental behavior of the matrix class -- which has been explained and which you offered no resistance to. Zach From cimrman3 at ntc.zcu.cz Tue Mar 27 13:27:32 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 27 Mar 2007 19:27:32 +0200 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: References: <4607E40E.5000300@ntc.zcu.cz> <4607EDCD.5020001@ntc.zcu.cz> Message-ID: <46095404.4020106@ntc.zcu.cz> David Koch wrote: > Ok, > > I did and the results are: > csc * csc: 372.601957083 > csc * csc: 3.90811300278 a typo here? which one is csr? > csr * csc: 15.3202679157 > csr * csr: 3.84498214722 > > Mhm, quite insightful. Note, that in an operation X.transpose() * X, > where X > is csc_matrix, then X.tranpose() is automatically cast to csr_matrix. A > re-cast to csc make the whole operation faster. It's still about 1000 times > slower than Matlab but 4 times faster than before. ok. now which version of scipy (scipy.__version__) do you use (you may have posted it, but I missed it)? Not so long ago, there was an effort by Nathan Bell and others reimplementing sparsetools + scipy.sparse to get better usability and performance. My (almost latest) version is 0.5.3.dev2860. r. From ogdude at googlemail.com Tue Mar 27 13:41:03 2007 From: ogdude at googlemail.com (David Koch) Date: Tue, 27 Mar 2007 19:41:03 +0200 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: <46095404.4020106@ntc.zcu.cz> References: <4607E40E.5000300@ntc.zcu.cz> <4607EDCD.5020001@ntc.zcu.cz> <46095404.4020106@ntc.zcu.cz> Message-ID: On 3/27/07, Robert Cimrman wrote: > > > ok. now which version of scipy (scipy.__version__) do you use (you may > have posted it, but I missed it)? Not so long ago, there was an effort > by Nathan Bell and others reimplementing sparsetools + scipy.sparse to > get better usability and performance. My (almost latest) version is > 0.5.3.dev2860. My version is 0.5.2, the latest official version available for download, right? Thanks, /David -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Mar 27 13:47:56 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 27 Mar 2007 10:47:56 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> Message-ID: <460958CC.4030806@noaa.gov> Zachary Pincus wrote: > rest of linear algebra -- e.g. that m[0] yields a matrix if m is a > matrix-- it almost certainly would violate the principle of least > surprise for iteration over m (intuitively understood to be choosing m > [0] then m[1] and so forth) to yield anything other than a matrix. I don't think the OP was suggesting that. Rather, I think the suggestion was that for a mXn matrix, M: M[i].shape == (n,) M[i:i+1].shape == (1,n) that is, indexing (or iterating returns a vector, and slicing returns a matrix). This is, indeed exactly how numpy arrays behave! The problem with this is: numpy matrices were created specifically to support linear algebra calculations. For linear algebra, the distinction between row vectors and column vectors is critical. By definition, a row vector has shape: (1,n), and a column vector has shape (m,1). In this case, perhaps the OP is thinking that a shape (n,) array could be considered a row vector, but in that case: M[1,:].shape == (n,) M[:,1].shape == (m,) which of these is the row and which the column? This is why matrices index this way instead: M[1,:].shape == (1, n) M[:,1].shape == (m, 1) now we know exactly what is a row and what is a column. By the way, I think with the way numpy works: M[i] == M[i,:] by definition, so those couldn't yield different shaped results. Is that right? I think we got a bit sidetracked by the example given. If I have a bunch of points I want to store, I'm going to use an (m,2) *array*, not a matrix, then then A[i] will yield a (2,) array, which makes sense for (2-d) points. In fact, I do this a LOT. If I happen to need to do some linear algebra on that array of points, I'll convert to a matrix, do the linear algebra, then convert back to an a array (or just use the linear algebra functions on the array). I hope this helps -Chris > This can't possibly be what you're asking for, right? You aren't > suggesting that m[0] and list(iter(m))[0] should be different types? > > There are many clear and definite use cases for m[0] being a matrix, > by the way, in addition to the theoretical arguments -- these aren't > hard to come by. Whether or nor there are actual good use-cases for > iterating over a matrix, if you believe that m[0] should be a matrix, > it's madness to suggest that list(iter(m))[0] should be otherwise. > > My opinion? Don't iterate over matrices. Use matrices for linear > algebra. There's no "iteration" construct in linear algebra. The > behavior you find so puzzling is a necessary by-product of the > fundamental behavior of the matrix class -- which has been explained > and which you offered no resistance to. > > Zach > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Tue Mar 27 13:57:57 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 27 Mar 2007 13:57:57 -0400 Subject: [Numpy-discussion] missing use case In-Reply-To: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> Message-ID: Hi Zachary, I think your response highlights very well the apparent design flaw. Here is your response to my request for a use case where iteration over a matrix should yield matrices: do not iterate! Does some part of you not find that just a little bizarre?! As for offering as a use case that with a matrix M you are allowed to use M[0] instead of M[0,:] when you want the first row as a matrix, I really cannot take that trivial convenience seriously as the basis of a fundamental design decision. However there is no sympathy for my view on this, so I am not debating it any more. Instead I have asked a simple question: what use case is this design decision supporting? I am interested in this so that I can see into the decision better. Nowith Chris proposes that "M[i] == M[i,:] by definition". If so, that is an design-based answer to my question. I agree that M[i,:] should return a matrix. But my understanding was different: I thought M[i] relied on standard Python idexing while M[i,:] was numpy indexing. Cheers, Alan Isaac From cjw at sympatico.ca Tue Mar 27 14:08:12 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue, 27 Mar 2007 14:08:12 -0400 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> <460842DA.3070000@ee.byu.edu> Message-ID: Charles R Harris wrote: > > > On 3/26/07, *Travis Oliphant* > wrote: > > > > I think that might be the simplest thing, dot overrides subtypes. > BTW, > > here is another ambiguity > > > > In [6]: dot(array([[1]]),ones(2)) > > > --------------------------------------------------------------------------- > > > exceptions.ValueError Traceback (most > > recent call last) > > > > /home/charris/ > > > > ValueError: matrices are not aligned > > > > Note that in this case dot acts like the rhs is always a column > vector > > although it returns a 1-d vector. I don't know that this is a bad > > thing, but perhaps we should extend this behaviour to matrices, > which > > would be different from the now current 1-d is always a *row* > vector, i.e. > > > The rule 1-d is always a *row* vector only applies when converting to a > matrix. > > In this case, the dot operator does not "convert to a matrix" but uses > rules for operating with mixed 2-d and 1-d arrays inherited from > Numeric. > > I'm very hesitant to change those rules. > > > I wasn't suggesting that, just noticing that the rule was 1-d vector on > right is treated as a column vector by dot, which is why an exception > was raised in the posted case. If it is traditional for matrix routines > always treat is as a row vector, so be it. My recollection is that text books treat the column vector, represented by a lower case letter, bold or underlined, as the default. If b (dressed as described before) is a column vector, then b' represents a row vector. For numpy, it makes sense to consider b as a row vector, since the underlying array uses the C convention where each row is stored contiguously. Colin W. > > Chuck > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cjw at sympatico.ca Tue Mar 27 14:10:57 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue, 27 Mar 2007 14:10:57 -0400 Subject: [Numpy-discussion] defmatrix.py In-Reply-To: References: <4607B817.3050803@iam.uni-stuttgart.de> <46082724.7010208@ee.byu.edu> <46082F2C.6060901@ee.byu.edu> <460842DA.3070000@ee.byu.edu> <46084B0B.10807@ee.byu.edu> Message-ID: Charles R Harris wrote: > > > On 3/26/07, *Travis Oliphant* > wrote: > > Charles R Harris wrote: > > > > > > > The rule 1-d is always a *row* vector only applies when > converting > > to a > > matrix. > > > > In this case, the dot operator does not "convert to a matrix" > but > > uses > > rules for operating with mixed 2-d and 1-d arrays inherited from > > Numeric. > > > > I'm very hesitant to change those rules. > > > > > > I wasn't suggesting that, just noticing that the rule was 1-d vector > > on right is treated as a column vector by dot, which is why an > > exception was raised in the posted case. If it is traditional for > > matrix routines always treat is as a row vector, so be it. > > O.K . So, the problem is that when I defined matrix multiplication, I > mistakenly believed that dot always interpreted 1-d arrays as row > vectors which it is now clear it doesn't. > > I think this led to the issues. > > So, making dot always return arrays seems like the needed fix. > > > Sounds good. Do you want to leave the matrix * operator as it now is? I hope so. numarray used the row convention. Colin W. > > Chuck > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From cimrman3 at ntc.zcu.cz Tue Mar 27 14:16:25 2007 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Tue, 27 Mar 2007 20:16:25 +0200 Subject: [Numpy-discussion] Tuning sparse stuff in NumPy In-Reply-To: References: <4607E40E.5000300@ntc.zcu.cz> <4607EDCD.5020001@ntc.zcu.cz> <46095404.4020106@ntc.zcu.cz> Message-ID: <46095F79.5060306@ntc.zcu.cz> David Koch wrote: > On 3/27/07, Robert Cimrman wrote: >> >> >> ok. now which version of scipy (scipy.__version__) do you use (you may >> have posted it, but I missed it)? Not so long ago, there was an effort >> by Nathan Bell and others reimplementing sparsetools + scipy.sparse to >> get better usability and performance. My (almost latest) version is >> 0.5.3.dev2860. >> > My version is 0.5.2, the latest official version available for download, > right? I see. Now while it is desirable and fine in majority of cases to install stable version of software, with numpy and scipy it is different :-), as the development is very fast. You should try the 'bleeding edge' versions via SVN, as many issues you see could have been already fixed/improved. Thanks to the extensive unit testing, the latest version usually (read: 99.9%) works better than the stable one. r. From dahl.joachim at gmail.com Tue Mar 27 14:23:17 2007 From: dahl.joachim at gmail.com (Joachim Dahl) Date: Tue, 27 Mar 2007 20:23:17 +0200 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <460958CC.4030806@noaa.gov> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> Message-ID: <47347f490703271123u71a6fabdue32fc9ce2e0df565@mail.gmail.com> I would like to hear your opinion on developing an explicit sparse/dense 2D matrix class with indexing similar to Matlab, and without significant differences between sparse and dense matrices from the user's perspective... I know that it's not one of Numpy/Scipy's goals to clone Matlab, but I think I represent a potentially large scientific Python user base, who would find Python matrices that feel a bit more like Matlab/Octave etc. extremely useful. I have a great respect for all the work in Numpy and Scipy, but at the same time I feel that numerical linear algebra in Python would benefit from a more dedicated matrix library, and I think that Numpy (or another single package) should provide that in a homogeneous way - without ties to how Numpy array works. - Joachim On 3/27/07, Christopher Barker wrote: > > Zachary Pincus wrote: > > rest of linear algebra -- e.g. that m[0] yields a matrix if m is a > > matrix-- it almost certainly would violate the principle of least > > surprise for iteration over m (intuitively understood to be choosing m > > [0] then m[1] and so forth) to yield anything other than a matrix. > > I don't think the OP was suggesting that. Rather, I think the suggestion > was that for a mXn matrix, M: > > M[i].shape == (n,) > > M[i:i+1].shape == (1,n) > > that is, indexing (or iterating returns a vector, and slicing returns a > matrix). This is, indeed exactly how numpy arrays behave! > > The problem with this is: > > numpy matrices were created specifically to support linear algebra > calculations. For linear algebra, the distinction between row vectors > and column vectors is critical. By definition, a row vector has shape: > (1,n), and a column vector has shape (m,1). > > In this case, perhaps the OP is thinking that a shape (n,) array could > be considered a row vector, but in that case: > > M[1,:].shape == (n,) > M[:,1].shape == (m,) > > which of these is the row and which the column? This is why matrices > index this way instead: > > M[1,:].shape == (1, n) > M[:,1].shape == (m, 1) > > now we know exactly what is a row and what is a column. > > By the way, I think with the way numpy works: M[i] == M[i,:] by > definition, so those couldn't yield different shaped results. Is that > right? > > I think we got a bit sidetracked by the example given. If I have a bunch > of points I want to store, I'm going to use an (m,2) *array*, not a > matrix, then then A[i] will yield a (2,) array, which makes sense for > (2-d) points. In fact, I do this a LOT. > > If I happen to need to do some linear algebra on that array of points, > I'll convert to a matrix, do the linear algebra, then convert back to an > a array (or just use the linear algebra functions on the array). > > I hope this helps > > -Chris > > > > > > > > > > This can't possibly be what you're asking for, right? You aren't > > suggesting that m[0] and list(iter(m))[0] should be different types? > > > > There are many clear and definite use cases for m[0] being a matrix, > > by the way, in addition to the theoretical arguments -- these aren't > > hard to come by. Whether or nor there are actual good use-cases for > > iterating over a matrix, if you believe that m[0] should be a matrix, > > it's madness to suggest that list(iter(m))[0] should be otherwise. > > > > My opinion? Don't iterate over matrices. Use matrices for linear > > algebra. There's no "iteration" construct in linear algebra. The > > behavior you find so puzzling is a necessary by-product of the > > fundamental behavior of the matrix class -- which has been explained > > and which you offered no resistance to. > > > > Zach > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Tue Mar 27 14:39:22 2007 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue, 27 Mar 2007 14:39:22 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <1174844924.6310.27.camel@localhost.localdomain> Message-ID: Alan G Isaac wrote: > On Mon, 26 Mar 2007, "Colin J. Williams" apparently wrote: >> One would expect the iteration over A to return row >> vectors, represented by (1, n) matrices. > > This is again simple assertion. > **Why** would "one" expect this? > Some people clearly do not. > > One person commented that this unexpected behavior was > a source of error in their code. > > Another person commented that they did not even guess that > such a thing would be possible. > > Experience with Python should lead to the ability to > anticipate the outcome. Apparently this is not the case. > That suggests a design problem. > > What about **Python** would lead us to expect this behavior?? > > In *contrast*, everyone agrees that for a matrix M, > we should get a matrix from M[0,:]. > This is expected and desirable. Perhaps our differences lies in two things: 1. the fact that the text books typically take the column vector as the default. For a Python version, based on C it makes more sense to treat the rows as vectors, as data is stored contiguously by row. 2. the convention has been proposed that the vector is more conveniently implemented as a matrix, where one dimension is one. The vector could be treated as a subclass of the matrix but this adds complexity with little clear benefit. PyMatrix has matrix methods isVector, isCVector and isRVector. I can see some merit in conforming to text book usage and would be glad to consider changes when I complete the port to numpy, in a few months. Colin W. > > Cheers, > Alan Isaac From ludwigbrinckmann at gmail.com Tue Mar 27 13:50:46 2007 From: ludwigbrinckmann at gmail.com (Ludwig) Date: Tue, 27 Mar 2007 17:50:46 +0000 (UTC) Subject: [Numpy-discussion] Complex 'where' expression Message-ID: A bit new to numpy, trying to move over some IDL code. I have a complex array selection expression, where I need an array of indexes as a result, so I can use this as a selection expression to apply changes to a different array. I have two images, image1 and image2, plus an array sigma, the result of a previous calculation. All have the same shape. quality is a scalar threshold. I need the indexes where image1 and image2 are not 0 and the sigma value at that point is lower than my threshold. I then take these indices and store some value against this in a different array. In pseudo code: indexes = where(image1 > 0 and image2 > 0 and sigma < quality) result[indexes] = i # scalar When I run this numpy tells me that the that the truth value of the array comparison is ambiguous -- but I do not want to compare the whole arrays here, but the value for every point. How do I do this Regards Ludwig From robert.kern at gmail.com Tue Mar 27 14:59:11 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 27 Mar 2007 13:59:11 -0500 Subject: [Numpy-discussion] Complex 'where' expression In-Reply-To: References: Message-ID: <4609697F.7060002@gmail.com> Ludwig wrote: > In pseudo code: > > indexes = where(image1 > 0 and image2 > 0 and sigma < quality) > result[indexes] = i # scalar > > > When I run this numpy tells me that the that the truth value of the array > comparison is ambiguous -- but I do not want to compare the whole arrays here, > but the value for every point. The Python keywords "and", "or", and "not" try to evaluate the truth value of the objects. We cannot override them to operate elementwise. Instead, the operators &, |, and ~, respectively, are the elementwise operators on arrays of booleans. Thus, you want this: indexes = where((image1>0) & (image2>0) & (sigma0) & (image2>0) & (sigma References: <46082D59.5000509@ee.byu.edu> Message-ID: <46096C68.8010601@ee.byu.edu> Alan Isaac wrote: > On Mon, 26 Mar 2007, Travis Oliphant wrote: > >> It actually has been offered. You just don't accept it. >> Matrices are containers of matrices. >> If M is an (mxn) matrix then M[0] is a (1xn) matrix. >> Viewing this 1xn matrix as a 1-d array loses it's row-vectorness. >> This seems perfectly logical and acceptable to me. I'm waiting for a >> better explanation as to why this is not acceptable. >> Arguments that rest on what is and what isn't "Pythonic" are not very >> persuasive as this is very often in the eye of the beholder. >> > > Again, I only raised a *question* about whether > there might be a design problem here. Thanks for listing your points. I can see that this is an issue where reasonable people will disagree because there are multiple ways of looking at it. The idea that matrix selection would return matrices really comes from wanting to keep matrices more persistent through operations. M[0] could be seen either as a 1xn matrix or a n-length array. I agree that both concepts are possible. Seeing it as a 1xn matrix allows matrices to remain persistent more often. So, the arguments for the current approach and the arguments against it to me seem on the same level, so I don't see a reason to change the current behavior and see a lot of strong reasons not to change the behavior (we are in a 1.0 release and could not change anything until at least 1.1 anyway). With that said: One of my goals for the next year or two is to create a matrix class in C and incorporate CVXOPT matrices and it's coupled sparse matrix. We can re-visit this question in that process. I would like for there to be a sparse matrix implementation in NumPy (without the solver which will remain in SciPy). But, the sparse matrix and the matrix need to have the same behaviors and should be able to interoperate with each other. So, if you would like to help with that project all input is welcome. Best regards, -Travis > My goal > was only to have this explored, and I've tried > to explain why. > > The evidence I offer: > - it is surprising (at least to some) that iterating > over a matrix yields matrices > - I believe it is unexpected (prior to instruction) and that > there is a natural more expected behavior > - if that is right, deviation from the expected should have > a good justification > - this behavior has tripped up at least a couple people and > I expect that to happen to many others over time (because > I believe the behavior is unexpected) > - when I desire to iterate over a matrix I always want the > arrays. (Of course there is a way to have them; that's > not the point). I'm interested to see a use case where > the rows are desired as matrices > > As you indicate, none of this constitutes an "argument". > And since nobody appears to agree, I should shut up. > This will be my last post on this subject. > > Cheers, > Alan Isaac > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > From wfspotz at sandia.gov Tue Mar 27 15:28:25 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Tue, 27 Mar 2007 13:28:25 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <46096C68.8010601@ee.byu.edu> References: <46082D59.5000509@ee.byu.edu> <46096C68.8010601@ee.byu.edu> Message-ID: <23120BD9-9860-4030-8DA6-D99202B9DC26@sandia.gov> It seems to me that using shapes, (m,1) versus (1,n), to determine whether a vector is column- or row-oriented is a hack (or at least feels like one). If we know we have a vector, then we want to use a single index to obtain a scalar, and that extra "0," or ",0" shouldn't be necessary. It also seems to me that there is an object-oriented design solution to this, namely defining column_vector and row_vector classes that inherit from matrix, that accept a single index, but behave as matrices consistent with their type. I'm sure there are implications to this I haven't thought through yet, but at least it gives us logical indexing AND persistence of matrices through operations. On Mar 27, 2007, at 1:11 PM, Travis Oliphant wrote: > Thanks for listing your points. I can see that this is an issue > where > reasonable people will disagree because there are multiple ways of > looking at it. > > The idea that matrix selection would return matrices really comes from > wanting to keep matrices more persistent through operations. > > M[0] could be seen either as a 1xn matrix or a n-length array. I > agree that both concepts are possible. Seeing it as a 1xn matrix > allows > matrices to remain persistent more often. > > So, the arguments for the current approach and the arguments > against it > to me seem on the same level, so I don't see a reason to change the > current behavior and see a lot of strong reasons not to change the > behavior (we are in a 1.0 release and could not change anything > until at > least 1.1 anyway). > > With that said: > > One of my goals for the next year or two is to create a matrix > class in > C and incorporate CVXOPT matrices and it's coupled sparse > matrix. We > can re-visit this question in that process. I would like for there to > be a sparse matrix implementation in NumPy (without the solver which > will remain in SciPy). > > But, the sparse matrix and the matrix need to have the same behaviors > and should be able to interoperate with each other. > > So, if you would like to help with that project all input is welcome. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From oliphant at ee.byu.edu Tue Mar 27 15:42:23 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 27 Mar 2007 13:42:23 -0600 Subject: [Numpy-discussion] Buffer interface PEP Message-ID: <4609739F.8080309@ee.byu.edu> Hi all, I'm having a good discussion with Carl Banks and Greg Ewing over on python-dev about the buffer interface. We are converging on a pretty good design, IMHO. If anybody wants to participate in the discussion, please read the PEP (there are a few things that still need to change) and add your two cents over on python-dev (you can read it through the gmane gateway and participate without signing up). The PEP is stored in numpy/doc/pep_buffer.txt on the SVN tree for NumPy Best regards, -Travis From zpincus at stanford.edu Tue Mar 27 15:45:23 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue, 27 Mar 2007 12:45:23 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <460958CC.4030806@noaa.gov> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> Message-ID: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Hello all, I suspect my previous email did not contain the full chain of my reasoning, because I thought that some parts were basically obvious. My point was merely that given some pretty basic fundamental tenants of numpy, Alan's suggestions quickly lead to *serious issues* far worse than the original problem. Thanks Chris for making this more explicit; here I will do so further. Let A be an array of arbitrary shape, and M be m-by-n matrix. (1) A[i] equals by definition A[i,...] This is a pretty fundamental part of numpy, right? That property (and its relatives) along with the broadcasting behavior, are in large part what make numpy so very expressive compared to matlab, etc. I can write code in numpy to do multidimensional operations over arbitrary-shaped and arbitrary-dimensioned data that look *exactly* like the corresponding 1D operations. It's amazing, really, how numpy's "implicit indexing" (this property), broadcasting (sort of the inverse of this, really, where shape tuples are extended as necessary with implicit 1's, instead of slices being extended with :'s as necessary), and fancy indexing, allow complex operations to be written compactly, clearly, and in a fully-general manner. I thought that point (1) was obvious, as it's a pretty fundamental part of numpy (and Numeric and Numarray before that). It's in absolutely no way a "trivial convenience". Of course, broadcasting and ufuncs, etc, contribute a lot more to expressiveness than this property, but it's part of the family. (2) Thus, M[i] equals by definition M[i,:]. By the mathematical definition of matrices and for pragmatic reasons, M[i,:] is another matrix. Here the trouble starts. Should matrices, not being arbitrary in their dimension, even have the ability to be indexed as M[i]? Or should they be completely different beasts than arrays? Which violates the "principle of least surprise" more? Now, I think we all more or less agree that M[i,:] has to be a matrix, because otherwise the matrix class is pretty worthless for linear algebra, and doesn't operate according to the mathematical definition of a matrix. Indeed, this is one of the basic properties of the numpy matrix -- that, and that the multiplication operator is defined as matrix-multiplication. Remove that and there's almost no point in *having* a separate matrix class. Want a use-case? Do the singular value decomposition, and get the product of the second-smallest singular vectors as Vt[:,-2] * U [-2,:]. (Things vaguely akin to this come up a lot.) You can copy the operation from any linear algebra text if the row- and column-vectors are treated as above. Otherwise you have to wonder whether that's supposed to be an outer or inner product, and use the corresponding operator -- and at that point, why was I using matrix classes anyway? (3) If M[i] = M[i,:], and M[i,:] is a matrix, then the iteration behavior of matrices *has* to yield other matrices, unless Alan is willing to suggest that list(iter(m))[i] should have a different type than M[i] -- a clear absurdity. This is the point that I was trying to make previously, having mistakenly assumed that point (1) was clear to all, and that point (2) had been clearly established. So, if we trace the reasoning of Alan's suggestion, coupled with these above properties, we get just that: (a) Iteration over M should yield arrays. (b) Ergo M[i] should be an array. (c) Ergo M[i,:] should be an array. (d) Ergo the matrix class should be identical to the array class except for overloaded multiplication, and the only way to get a 'row' or 'column' vector from a matrix that operates as a proper row or column vector should is M[i:i+1,:]. Whicy is a lot of typing to get the 'i-th' vector from the matrix. I don't think these changes are worth it -- it basically discards the utility of the matrix class for linear algebra in order to make the matrix class more useful for general purpose data storage (a role already filled by the array type). Here is another set of changes which would make matrices fully consistent with their linear-algebra roots: (a) M[i,:] should be a matrix. (b) M[i] is an error. (c) Iteration over M is an error. That's kind of lame though, right? Because then matrices are completely inconsistent with their numpy roots. So we are left with the current behavior: (a) M[i,:] is a matrix. (b) M[i] is a matrix. (c) Iteration over M yields matrices. My view is that, fundamentally, they are linear algebra constructs. However, I also think it would be far more harmful to remove basic behavior common to the rest of numpy (e.g. that M[i] = M[i,:]), and behavior that comes along with that (iterability). Hence my suggestion: treat matrices like linear algebra constructs -- don't iterate over them (unless you know what you're doing). Don't index them like M[i] (unless you know what you're doing). Maybe it's "just a little bizarre" to suggest this, but I think the other options above are much more bizarre. Zach From aisaac at american.edu Tue Mar 27 16:03:25 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 27 Mar 2007 16:03:25 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: Hi Zach, The use case I requested was for iteration over a matrix where it is desirable that matrices are yielded. That is not what you offered. The context for this request is my own experience: whenever I have needed to iterate over matrices, I have always wanted the arrays. So I am simply interested in seeing an example of the opposite desire. Cheers, Alan Isaac From robert.kern at gmail.com Tue Mar 27 16:21:15 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 27 Mar 2007 15:21:15 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: <46097CBB.4090303@gmail.com> Alan G Isaac wrote: > Hi Zach, > > The use case I requested was for iteration over a > matrix where it is desirable that matrices are yielded. > That is not what you offered. > > The context for this request is my own experience: > whenever I have needed to iterate over matrices, > I have always wanted the arrays. So I am simply > interested in seeing an example of the opposite desire. Gram-Schmidt orthogonalization. ortho = [mat[0] / sqrt(mat[0] * mat[0].T)] for rowv in mat[1:]: for base in ortho: rowv = rowv - base * float(rowv * base.T) rowv = rowv / sqrt(rowv * rowv.T) ortho.append(rowv) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aisaac at american.edu Tue Mar 27 16:37:36 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 27 Mar 2007 16:37:36 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <46097CBB.4090303@gmail.com> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><46097CBB.4090303@gmail.com> Message-ID: On Tue, 27 Mar 2007, Robert Kern apparently wrote: > Gram-Schmidt orthogonalization I take it from context that you consider it desirable to end up with a list of matrices? I guess I would find it more natural to work with the arrays, but perhaps that starts just being taste. Thank you, Alan Isaac From zpincus at stanford.edu Tue Mar 27 16:44:09 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue, 27 Mar 2007 13:44:09 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> > The use case I requested was for iteration over a > matrix where it is desirable that matrices are yielded. > That is not what you offered. No, I offered a thorough discussion of why the design of numpy, as I imperfectly understand it, make the trade-offs to achieve your desired goal unpalatable. > The context for this request is my own experience: > whenever I have needed to iterate over matrices, > I have always wanted the arrays. So I am simply > interested in seeing an example of the opposite desire. I'm glad that Robert provided such a nice example. Nevertheless, I think that even absent any example it should be pretty clear at this point that if you want iteration over matrices to return arrays, either: (1) you need to be willing to accept that list(iter(M))[i] != M[i] or (2) M[i], which equals M[i,:], should be an array. (or 3, that M[i] and iteration over M is not defined) Clearly, (1) is fundamentally ridiculous. Clearly use-cases (as well as the mathematical definition of matrices) abound to illustrate why (2) is no good. Option (3) helps nobody. So which is it? You have to choose one! It is my assertion that if you are iterating over matrices and hoping to get arrays, you are misusing the matrix class. Almost by definition if you expect M[i,:] or M[i] (or equivalently, the output of an iterator over M) to be an array, then M is being used outside of a linear algebra context -- and thus, in my view, misused. In those cases, the matrix should be cast to an array, which has the desired behavior. Robert nicely illustrated a linear-algebraic context for matrix iteration. Now, Bill offers up a different suggestion: indexing M yields neither a matrix nor an array, but a class that operates more or less like an array, except insofar as it interacts with other matrix objects, or other objects of similar classes. I'm interested in hearing more about this, what trade-offs or other compromises it might involve. Zach > > Cheers, > Alan Isaac > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From fperez.net at gmail.com Tue Mar 27 16:51:32 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 27 Mar 2007 14:51:32 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: On 3/27/07, Alan G Isaac wrote: > Hi Zach, > > The use case I requested was for iteration over a > matrix where it is desirable that matrices are yielded. I'm personally not really partial to any side of this discussion, given how I don't use matrices at all (I'm content with a simple mental model of arrays, and use dot() as necessary). But it's probably worth mentioning, as a data point, that the python language has a well established precedent for 'iteration over a FOO where a FOO is yielded': In [1]: s='abc' In [2]: type(s) Out[2]: In [3]: map(type,s) Out[3]: [, , ] I know strings aren't matrices, so take this as you will in terms of being plus, neutral or minus regarding your discussion. I'm just offering the data, not interpreting it :) Cheers, f From dahl.joachim at gmail.com Tue Mar 27 16:53:16 2007 From: dahl.joachim at gmail.com (Joachim Dahl) Date: Tue, 27 Mar 2007 22:53:16 +0200 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: <47347f490703271353h5b378d10hdb959a1fcb35df7b@mail.gmail.com> On 3/27/07, Zachary Pincus wrote: > > > So we are left with the current behavior: > (a) M[i,:] is a matrix. > (b) M[i] is a matrix. > (c) Iteration over M yields matrices. > > I find it useful if M[i] indexes the matrix interpreted as a column-vector (vec(M), or M(:) in Matlab notation), e.g., you could pick out the diagonal as M[::n+1], or if N is an index-set of certain elements in M, then you could access those elements as M[N]. Then I would also say that iterating over a matrix should just return the elements of vec(M) one by one. In general, I think the Matlab notation works well: * M[I,J] where I and J and index-sets returns a len(I) x len(J) matrix * M[I] returns a len(I) vector (a column or row vector depending on orientation of I) The index-sets could be a single integer, a list, or a slice. But I can see there are good arguments for using the NumPy conventions as well... -------------- next part -------------- An HTML attachment was scrubbed... URL: From zpincus at stanford.edu Tue Mar 27 17:21:09 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue, 27 Mar 2007 14:21:09 -0700 Subject: [Numpy-discussion] Buffer interface PEP In-Reply-To: <4609739F.8080309@ee.byu.edu> References: <4609739F.8080309@ee.byu.edu> Message-ID: Hi, I have a specific question and then a general question, and some minor issues for clarification. Specifically, regarding the arguments to getbufferproc: > 166 format > 167 address of a format-string (following extended struct > 168 syntax) indicating what is in each element of > 169 of memory. The number of elements is len / itemsize, > 170 where itemsize is the number of bytes implied by the format. > 171 NULL if not needed in which case format is "B" for > 172 unsigned bytes. The memory for this string must not > 173 be freed by the consumer --- it is managed by the exporter. Is this saying that either NULL or a pointer to "B" can be supplied by getbufferproc to indicate to the caller that the array is unsigned bytes? If so, is there a specific reason to put the (minor) complexity of handling this case in the caller's hands, instead of dealing with it internally to getbufferproc? In either case, the wording is a bit unclear, I think. The general question is that there are several other instances where getbufferproc is allowed to return ambiguous information which must be handled on the client side. For example, C-contiguous data can be indicated either by a NULL strides pointer or a pointer to a properly- constructed strides array. Clients that can't handle C-contiguous data (contrived example, I know there is a function to deal with that) would then need to check both for NULL *and* inside the strides array if not null, before properly deciding that the data isn't usable them. Similarly, the suboffsets can be either all negative or NULL to indicate the same thing. Might it be more appropriate to specify only one canonical behavior in these cases? Otherwise clients which don't do all the checks on the data might not properly interoperate with providers which format these values in the alternate manner. Also, some typos, and places additional clarification could help: > 253 PYBUF_STRIDES (strides and isptr) Should 'isptr' be 'suboffsets'? > 75 of a larger array can be described without copying the data. T Dangling 'T'. > 279 Get the buffer and optional information variables about the > buffer. > 280 Return an object-specific view object (which may be simply a > 281 borrowed reference to the object itself). This phrasing (and similar phrasing elsewhere) is somewhat opaque to me. What's an "object-specific view object"? > 287 Call this function to tell obj that you are done with your "view" Similarly, the 'view' concept and terminology should be defined more clearly in the preamble. > 333 The struct string-syntax is missing some characters to fully > 334 implement data-format descriptions already available elsewhere (in > 335 ctypes and NumPy for example). Here are the proposed additions: Is the following table just the additions? If so, it might be good to show the full spec, and flag the specific additions. If not, then the additions should be flagged. > 341 't' bit (number before states how many bits) vs. > 372 According to the struct-module, a number can preceed a character > 373 code to specify how many of that type there are. The I'm confused -- could this be phrased more clearly? Does '5t' refer to a field 5-bits wide, or 5-one bit fields? Is 'ttttt' allowed? If so, is it equivalent to or different from '5t'? > 378 Functions should be added to ctypes to create a ctypes object from > 379 a struct description, and add long-double, and ucs-2 to ctypes. Very cool. In general, the logic of the 'locking mechanism' should be described at a high level at some point. It's described in nitty-gritty details, but at least I would have appreciated a bit more of a discussion about the general how and why -- this would be helpful to clients trying to use the locking mechanism properly. Thanks to Travis and everyone else involved for your work on this cogent and sorely-needed PEP. Zach On Mar 27, 2007, at 12:42 PM, Travis Oliphant wrote: > > Hi all, > > I'm having a good discussion with Carl Banks and Greg Ewing over on > python-dev about the buffer interface. We are converging on a pretty > good design, IMHO. If anybody wants to participate in the > discussion, > please read the PEP (there are a few things that still need to change) > and add your two cents over on python-dev (you can read it through the > gmane gateway and participate without signing up). > > The PEP is stored in numpy/doc/pep_buffer.txt on the SVN tree for > NumPy > > Best regards, > > -Travis > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From pgmdevlist at gmail.com Tue Mar 27 17:28:46 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Mar 2007 17:28:46 -0400 Subject: [Numpy-discussion] A unique function... Message-ID: <200703271728.46416.pgmdevlist@gmail.com> All, I could swear that I ran once into a numpy (or scipy) function that output the unique values of a 1d ndarray along with the number of occurences of each element. If I'm not completely mistaken, it 's a private function initially in Fortran. Does this ring a bell to anyone ? Where could I find this function ? From zpincus at stanford.edu Tue Mar 27 17:34:36 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue, 27 Mar 2007 14:34:36 -0700 Subject: [Numpy-discussion] A unique function... In-Reply-To: <200703271728.46416.pgmdevlist@gmail.com> References: <200703271728.46416.pgmdevlist@gmail.com> Message-ID: Hello, There's unique and unique1d, but these don't output the number of occurences. There's also bincount, which outputs the number of each element, but includes zeros for non-present elements and so could be problematic for certain data. Zach On Mar 27, 2007, at 2:28 PM, Pierre GM wrote: > All, > I could swear that I ran once into a numpy (or scipy) function that > output the > unique values of a 1d ndarray along with the number of occurences > of each > element. If I'm not completely mistaken, it 's a private function > initially > in Fortran. > Does this ring a bell to anyone ? Where could I find this function ? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From robert.kern at gmail.com Tue Mar 27 17:34:52 2007 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 27 Mar 2007 16:34:52 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><46097CBB.4090303@gmail.com> Message-ID: <46098DFC.8030601@gmail.com> Alan G Isaac wrote: > On Tue, 27 Mar 2007, Robert Kern apparently wrote: >> Gram-Schmidt orthogonalization > > I take it from context that you consider it desirable > to end up with a list of matrices? Honestly, I don't care. You asked about iteration, and I gave you an example where it was important that the iterates were matrices (such that .T worked correctly). Please stop moving the goal posts. > I guess I would find it more natural to work with the > arrays, but perhaps that starts just being taste. Personally, I find working with arrays from start to finish the most natural. If you really want iterates to be arrays, just use .A. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Glen.Mabey at swri.org Tue Mar 27 17:42:18 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 27 Mar 2007 16:42:18 -0500 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap Message-ID: <20070327214218.GH28420@bams.ccf.swri.edu> I've been using numpy.memmap to access large arrays very nicely, but now I desire to instantiate a masked array that uses memmap objects. Using a memmap instance for the data works just fine and as expected. However, using a memmap for the mask value seems to load the entire object into memory, even if it is already of type bool. How hard would it be to change that behavior? Also, it seems that median() is not implemented on masked arrays? Is there some method for a masked array like ravel() but that only includes the non-masked values? Thank you, Glen Mabey From pgmdevlist at gmail.com Tue Mar 27 17:54:05 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Mar 2007 17:54:05 -0400 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap In-Reply-To: <20070327214218.GH28420@bams.ccf.swri.edu> References: <20070327214218.GH28420@bams.ccf.swri.edu> Message-ID: <200703271754.05179.pgmdevlist@gmail.com> On Tuesday 27 March 2007 17:42:18 Glen W. Mabey wrote: > However, using a memmap for the mask value seems to load the entire > object into memory, even if it is already of type bool. > > How hard would it be to change that behavior? Oh, I have no idea about that... > Also, it seems that median() is not implemented on masked arrays? Great minds meet, I just uploaded some very basic stats functions for the new implementation of masked array in the scipy SVN. > Is there some method for a masked array like ravel() but that only > includes the non-masked values? Yep, compressed(). Note that it will flatten your array first, so if you work with nd arrays, that's not the best solution. If you work with 2d arrays, you can simply loop on the other dimension, such as (n,p) = data.shape med = numpy.empty((n,),dtype=data.dtype) for i in range(p): med[i] = numpy.median(data[:,i].compressed()) That works well if p < n/3. From Glen.Mabey at swri.org Tue Mar 27 18:04:41 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 27 Mar 2007 17:04:41 -0500 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap In-Reply-To: <200703271754.05179.pgmdevlist@gmail.com> References: <20070327214218.GH28420@bams.ccf.swri.edu> <200703271754.05179.pgmdevlist@gmail.com> Message-ID: <20070327220441.GI28420@bams.ccf.swri.edu> On Tue, Mar 27, 2007 at 05:54:05PM -0400, Pierre GM wrote: > On Tuesday 27 March 2007 17:42:18 Glen W. Mabey wrote: > > However, using a memmap for the mask value seems to load the entire > > object into memory, even if it is already of type bool. > > > > How hard would it be to change that behavior? > > Oh, I have no idea about that... Anyone else care to comment on this? > > Also, it seems that median() is not implemented on masked arrays? > > Great minds meet, I just uploaded some very basic stats functions for the new > implementation of masked array in the scipy SVN. Whoa. You didn't mean numpy SVN, I take it. So, will numpy.median not ever take into account the mask? And, how recently did you upload them? I updated scipy today and I still get: In [41]:scipy.stats.median( ma_data[100:,1] ) Out[41]:1989792384.0 In [42]:numpy.median( ma_data[100:,1].filled() ) Out[42]:1989792384.0 In [43]:ma_data[100:,1] Out[43]: array(data = [ 8829714. -1000000. 7007859. ..., 3921109.5 3815157. 4447688. ], mask = [False True False ..., False False False], fill_value=-1000000.0) > > Is there some method for a masked array like ravel() but that only > > includes the non-masked values? > > Yep, compressed(). Perfect. Thank you. -- Glen W. Mabey "Happiness in marriage and parenthood can exceed a thousand times any other happiness." -- James Esdras Faust From pgmdevlist at gmail.com Tue Mar 27 18:08:56 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Mar 2007 18:08:56 -0400 Subject: [Numpy-discussion] A unique function... In-Reply-To: References: <200703271728.46416.pgmdevlist@gmail.com> Message-ID: <200703271808.56095.pgmdevlist@gmail.com> Zach, > There's unique and unique1d, but these don't output the number of > occurences. > There's also bincount, which outputs the number of each element, but > includes zeros for non-present elements and so could be problematic > for certain data. Well, I found it anyway. As I was pretty sure it was a private function, I grep on 'def _' | sed '/__/' and find it in scipy.stats.morestats: it's the find_repeat function. Note that I could probably built one with a combo unique1d/setmember1d. Thanks nevertheless. From aisaac at american.edu Tue Mar 27 19:18:56 2007 From: aisaac at american.edu (Alan Isaac) Date: Tue, 27 Mar 2007 18:18:56 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: On Tue, 27 Mar 2007, Fernando Perez wrote: > it's probably worth mentioning, as a data point, that the > python language has a well established precedent for > 'iteration over a FOO where a FOO is yielded': > In [1]: s='abc' Yes. And I recall being bitten by this long ago. Now it feels natural enough, I confess. But then, on reflection, it is an oddity. ;-) Cheers, Alan Isaac From pgmdevlist at gmail.com Tue Mar 27 18:35:51 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Mar 2007 18:35:51 -0400 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap In-Reply-To: <20070327220441.GI28420@bams.ccf.swri.edu> References: <20070327214218.GH28420@bams.ccf.swri.edu> <200703271754.05179.pgmdevlist@gmail.com> <20070327220441.GI28420@bams.ccf.swri.edu> Message-ID: <200703271835.51576.pgmdevlist@gmail.com> > Whoa. You didn't mean numpy SVN, I take it. So, will numpy.median not > ever take into account the mask? Mmh, that should be part of numpy.core.ma instead: let's keep the masked arrays a bit on the side. > And, how recently did you upload them? I updated scipy today and I > still get: Ah, sorry, I should have been clearer: scipy SVN, sandbox/maskedarray/mstats.py is the file I was mentioning. That's more of a place holder than an actual library at this point, I'll update it regularly when the need arises. I can't start messing with scipy.stats to take masked array into accounts , as I'm far too biased towards maskedarray vs numpy.core.ma. I'm not familiar with memmap, but the fact that numpy.core.ma masked arrays are not ndarrays may be a hint. From Glen.Mabey at swri.org Tue Mar 27 18:47:24 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 27 Mar 2007 17:47:24 -0500 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap In-Reply-To: <200703271835.51576.pgmdevlist@gmail.com> References: <20070327214218.GH28420@bams.ccf.swri.edu> <200703271754.05179.pgmdevlist@gmail.com> <20070327220441.GI28420@bams.ccf.swri.edu> <200703271835.51576.pgmdevlist@gmail.com> Message-ID: <20070327224724.GJ28420@bams.ccf.swri.edu> On Tue, Mar 27, 2007 at 06:35:51PM -0400, Pierre GM wrote: > > > Whoa. You didn't mean numpy SVN, I take it. So, will numpy.median not > > ever take into account the mask? > > Mmh, that should be part of numpy.core.ma instead: let's keep the masked > arrays a bit on the side. > > > And, how recently did you upload them? I updated scipy today and I > > still get: > > Ah, sorry, I should have been clearer: scipy SVN, > sandbox/maskedarray/mstats.py is the file I was mentioning. That's more of a > place holder than an actual library at this point, I'll update it regularly > when the need arises. > I can't start messing with scipy.stats to take masked array into accounts , as > I'm far too biased towards maskedarray vs numpy.core.ma. Uh, forgive me but I'm gleaning from your comments that numpy.ma is not the same as a maskedarray in scipy? Okay, yup, that's clarified in sandbox/maskedarray/README . Is there something deficient about numpy.core.ma? Thanks, Glen From Glen.Mabey at swri.org Tue Mar 27 18:51:55 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Tue, 27 Mar 2007 17:51:55 -0500 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap In-Reply-To: <20070327224724.GJ28420@bams.ccf.swri.edu> References: <20070327214218.GH28420@bams.ccf.swri.edu> <200703271754.05179.pgmdevlist@gmail.com> <20070327220441.GI28420@bams.ccf.swri.edu> <200703271835.51576.pgmdevlist@gmail.com> <20070327224724.GJ28420@bams.ccf.swri.edu> Message-ID: <20070327225155.GK28420@bams.ccf.swri.edu> On Tue, Mar 27, 2007 at 05:47:24PM -0500, Glen W. Mabey wrote: > Is there something deficient about numpy.core.ma? Okay, I just found sandbox/maskedarray/CHANGELOG . Glen From pgmdevlist at gmail.com Tue Mar 27 18:56:04 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Mar 2007 18:56:04 -0400 Subject: [Numpy-discussion] masked array with mask an instance of numpy.memmap In-Reply-To: <20070327224724.GJ28420@bams.ccf.swri.edu> References: <20070327214218.GH28420@bams.ccf.swri.edu> <200703271835.51576.pgmdevlist@gmail.com> <20070327224724.GJ28420@bams.ccf.swri.edu> Message-ID: <200703271856.04868.pgmdevlist@gmail.com> > Is there something deficient about numpy.core.ma? numpy.core.ma objects are not ndarrays. Therefore, their mask disappear when used as the input of something like numpy.array(data, subok=True), which is quickly problematic. Moreover, they are not especially subclassing-friendly, which was my biggest bone, and the main reason why I started rewriting the implementation. From oliphant at ee.byu.edu Tue Mar 27 19:45:45 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue, 27 Mar 2007 17:45:45 -0600 Subject: [Numpy-discussion] Buffer interface PEP In-Reply-To: References: <4609739F.8080309@ee.byu.edu> Message-ID: <4609ACA9.8020205@ee.byu.edu> Zachary Pincus wrote: > Hi, > > > Is this saying that either NULL or a pointer to "B" can be supplied > by getbufferproc to indicate to the caller that the array is unsigned > bytes? If so, is there a specific reason to put the (minor) > complexity of handling this case in the caller's hands, instead of > dealing with it internally to getbufferproc? In either case, the > wording is a bit unclear, I think. > Yes, the wording could be more clear. I'm trying to make it easy for exporters to change to the new buffer interface. The main idea I really want to see is that if the caller just passes NULL instead of an address then it means they are assuming the data will be "unsigned bytes" It is up to the exporter to either allow this or raise an error. The exporter should always be explicit if an argument for returning the format is provided (I may have thought differently a few days ago). > The general question is that there are several other instances where > getbufferproc is allowed to return ambiguous information which must > be handled on the client side. For example, C-contiguous data can be > indicated either by a NULL strides pointer or a pointer to a properly- > constructed strides array. Here. I'm trying to be easy on the exporter and the consumer. If the data is contiguous, then neither the exporter nor will likely care about the strides. Allowing this to be NULL is like the current array protocol convention which allows this to be None. > Clients that can't handle C-contiguous > data (contrived example, I know there is a function to deal with > that) would then need to check both for NULL *and* inside the strides > array if not null, before properly deciding that the data isn't > usable them. Not really. A client that cannot deal with strides will simply not pass an address to a stride array to the buffer protocol (that argument will be NULL). If the exporter cannot provide memory without stride information, then an error will be raised. > Similarly, the suboffsets can be either all negative or > NULL to indicate the same thing. > I think it's much easier to check if suboffsets is NULL rather than checking all the entries to see if they are -1 for the very common case (i.e. the NumPy case) of no dereferencing. Also, if you can't deal with suboffsets you would just not provide an address for them. > Might it be more appropriate to specify only one canonical behavior > in these cases? Otherwise clients which don't do all the checks on > the data might not properly interoperate with providers which format > these values in the alternate manner. > It's important to also be easy to use. I don't think clients should be required to ask for strides and suboffsets if they can't handle them. > > Also, some typos, and places additional clarification could help: > > >> 253 PYBUF_STRIDES (strides and isptr) >> > Should 'isptr' be 'suboffsets'? > Yes, but I think we are going to take out the multiple locks. > >> 75 of a larger array can be described without copying the data. T >> > Dangling 'T'. > Thanks, > >> 279 Get the buffer and optional information variables about the >> buffer. >> 280 Return an object-specific view object (which may be simply a >> 281 borrowed reference to the object itself). >> > This phrasing (and similar phrasing elsewhere) is somewhat opaque to > me. What's an "object-specific view object"? > At the moment it's the buffer provider. It is not defined because it could be a different thing for each exporter. We are still discussing this particular point and may drop it. > >> 333 The struct string-syntax is missing some characters to fully >> 334 implement data-format descriptions already available elsewhere (in >> 335 ctypes and NumPy for example). Here are the proposed additions: >> > Is the following table just the additions? If so, it might be good to > show the full spec, and flag the specific additions. If not, then the > additions should be flagged. > Yes, these are just the additions. I don't want to do the full spec, it is already available elsewhere in the Python docs. > >> 341 't' bit (number before states how many bits) >> > vs. > >> 372 According to the struct-module, a number can preceed a character >> 373 code to specify how many of that type there are. The >> > I'm confused -- could this be phrased more clearly? Does '5t' refer > to a field 5-bits wide, or 5-one bit fields? Is 'ttttt' allowed? If > so, is it equivalent to or different from '5t'? > Yes, 'ttttt' is equivalent to '5t' and the difference between one field 5-bits wide or 5-one bit fields is a confusion based on thinking there are fields at all. Both of those are equivalent. If you want "fields" then you have to define names. > >> 378 Functions should be added to ctypes to create a ctypes object from >> 379 a struct description, and add long-double, and ucs-2 to ctypes. >> > Very cool. > > In general, the logic of the 'locking mechanism' should be described > at a high level at some point. It's described in nitty-gritty > details, but at least I would have appreciated a bit more of a > discussion about the general how and why -- this would be helpful to > clients trying to use the locking mechanism properly. > The point of locking is so that the exporter knows when it can reallocate its buffer. Right now, reference counting is the only way to do that. But reference counting is not specific enough. Perhaps the reference is because of an object that is using the same memory but perhaps the reference is just another name pointing to exactly the same object. In the case of NumPy, NumPy needs to know when the resize method can be safely applied. Currently, it is ambiguous and un-clear when a NumPy array can re-allocate its own buffer. Also, in the past exposing the array object in Python's memory and then later re-allocating it led to problems. I'll try and address this more clearly. Thanks for your feedback, -Travis From tom.denniston at alum.dartmouth.org Tue Mar 27 19:59:41 2007 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Tue, 27 Mar 2007 18:59:41 -0500 Subject: [Numpy-discussion] Complex 'where' expression In-Reply-To: References: Message-ID: You need to use numpy.logical_and. The and and or operators do not work elementwise on arrays to the best of my understanding. I think it is a limitation on the way logical operators are applied in python or maybe it is an intentional numpy limitation. I'm sure others on the list could explain better. Also where is the wrong function. You want compress: In [8]: image2 = numpy.array([-1,1,1,1]) In [9]: image1 = numpy.array([1,-1,2,3]) In [10]: numpy.arange(len(image2)).compress(numpy.logical_and(image1 > 0, image2 > 0)) Out[10]: array([2, 3]) Others might have a better way but this at least works. --Tom On 3/27/07, Ludwig wrote: > A bit new to numpy, trying to move over some IDL code. > > I have a complex array selection expression, where I need an array of indexes as > a result, so I can use this as a selection expression to apply changes to a > different array. > > I have two images, image1 and image2, plus an array sigma, the result of a > previous calculation. All have the same shape. quality is a scalar threshold. > > I need the indexes where image1 and image2 are not 0 and the sigma value at that > point is lower than my threshold. I then take these indices and store some value > against this in a different array. > > In pseudo code: > > indexes = where(image1 > 0 and image2 > 0 and sigma < quality) > result[indexes] = i # scalar > > > When I run this numpy tells me that the that the truth value of the array > comparison is ambiguous -- but I do not want to compare the whole arrays here, > but the value for every point. > > How do I do this > > Regards > > Ludwig > > > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From bhendrix at enthought.com Tue Mar 27 20:08:04 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Tue, 27 Mar 2007 19:08:04 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray Message-ID: <4609B1E4.3060601@enthought.com> We have a class which is a subclass of ndarray which defines __array_finalize__ to add an attribute. The class looks something like this: class UnitArray(ndarray): # ... def __new__(cls, data, dtype=None, copy=True, units=None): # ... arr = array(data, dtype=dtype, copy=copy) res = ndarray.__new__(cls, arr.shape, arr.dtype,buffer=arr) res.units = units return res def __array_finalize__(self, obj): if (isinstance(obj, UnitArray)): if hasattr(obj, 'units'): self.units = obj.units This works for all the ufuncs I've tried, but the result of concatenate is missing the units attribute. Is this a bug with concatenate, or expected behavior? The real class can be found here: https://svn.enthought.com/svn/enthought/trunk/src/lib/enthought/numerical_modeling/units/unit_array.py Bryce From pgmdevlist at gmail.com Tue Mar 27 20:24:15 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Mar 2007 20:24:15 -0400 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <4609B1E4.3060601@enthought.com> References: <4609B1E4.3060601@enthought.com> Message-ID: <200703272024.15720.pgmdevlist@gmail.com> On Tuesday 27 March 2007 20:08:04 Bryce Hendrix wrote: > We have a class which is a subclass of ndarray which defines > __array_finalize__ to add an attribute. The class looks something like > this: Ahah, we ran into this problem a few months ago: You should not initialize your "units" attribute in the __new__ method, as it's not thread-safe first, but more important as you will lose it if you try to get a view of your new object (because in that case, __new__ is not called). Instead, try something like that: def __array_finalize__(self, obj): self.units = getattr(obj, 'units', yourdefaultunit) where your defaultunit is the value you would use in __new__. True, yourdefaultunit may change, and even be lost if there's a break between __new__ and __array_finalize__, but that should work. Lemme know. Note that things are always tricky w/ concatenate: when you concatenate your UnitArray, you should have a UnitArray, OK. But if you mix UnitArray and ndarray ? What should take precedence ? From aisaac at american.edu Tue Mar 27 21:43:59 2007 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 27 Mar 2007 21:43:59 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: >> On Tue, 27 Mar 2007, Robert Kern apparently wrote: >>> Gram-Schmidt orthogonalization > Alan G Isaac wrote: >> I take it from context that you consider it desirable >> to end up with a list of matrices? Robert wrote: > Honestly, I don't care. You asked about iteration, and > I gave you an example > where it was important that the iterates were matrices > (such that .T worked correctly). > Please stop moving the goal posts. It is really not my intent to be irritating or to move the posts. But I had two reasons to ask. First, your claim is false that this was important in your example. (To see this, replace M[1:] with M.A[1:].) So actually your code would work the same if iteration over matrices were producing arrays (after replacing M[0] with M[0,:] to match that case). Second, when I spoke of *desirability*, the output is relevant. Try nump.mat(ortho) to see what I mean. If iteration were to produce arrays, the outcome of implementing the algorithm (e.g., using ``dot``) would be I suggest more "desirable", in that numpy.mat(ortho) would work as hoped/expected. In this sense, the example perhaps speaks against your intent. You offer code that would work just fine if iteration yielded arrays. Apologies in advance if this again seems tangential to my original request. To me it does not. > If you really want iterates to be arrays, just use .A. I have mentioned that several times. It is orthogonal to the design issue I raised. (I suggested that under the current behavior you do not usually end up with what you want when iterating over matrices.) But again, I recognize that I am alone in that view. I am just trying to understand why. Cheers, Alan Isaac From charlesr.harris at gmail.com Tue Mar 27 23:01:04 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 27 Mar 2007 21:01:04 -0600 Subject: [Numpy-discussion] Further matrix oddities: no inner product Message-ID: In mathematics, and inner product is a sesquilinear form on pairs of vectors, so at the least it should return a scalar. In numpy inner is a sum over the last indices. OK, so we have In [10]: inner(ones(2),ones(2)) Out[10]: 2.0 This doesn't work as an inner product for column vectors, which would be the usual textbook convention, but that's alright, it's not a 'real' inner product. But what happens when matrices are involved? In [11]: inner(mat(ones(2)),mat(ones(2))) Out[11]: array([[ 2.]]) Hmm, we get an array, not a scalar. Maybe we can cheat In [12]: mat(ones(2))*mat(ones(2)).T Out[12]: matrix([[ 2.]]) What about vdot (conjugate of the mathematical convention, i.e., the Dirac convention) In [17]: vdot(mat(ones(2)),mat(ones(2))) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/charris/ ValueError: vectors have different lengths In [18]: vdot(mat(ones(2)),mat(ones(2)).T) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/charris/ ValueError: vectors have different lengths Nope, vdot doesn't work for row and column vectors. So there is *no* builtin inner product that works for matrices. I wonder if we should have one, and if so, what it should be called. I think that vdot should probably be modified to do the job. There is also the question of whether or not v.T * v should be a scalar when v is a column vector. I believe that construction is commonly used in matrix algebra as an alias for the inner product, although strictly speaking it uses the mapping between a vector space and its dual that the inner product provides. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From zpincus at stanford.edu Wed Mar 28 00:35:20 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue, 27 Mar 2007 21:35:20 -0700 Subject: [Numpy-discussion] Buffer interface PEP In-Reply-To: <4609ACA9.8020205@ee.byu.edu> References: <4609739F.8080309@ee.byu.edu> <4609ACA9.8020205@ee.byu.edu> Message-ID: <0FD6122B-8661-49C9-9693-E5DA2D4460E9@stanford.edu> >> Is this saying that either NULL or a pointer to "B" can be supplied >> by getbufferproc to indicate to the caller that the array is unsigned >> bytes? If so, is there a specific reason to put the (minor) >> complexity of handling this case in the caller's hands, instead of >> dealing with it internally to getbufferproc? In either case, the >> wording is a bit unclear, I think. >> > > Yes, the wording could be more clear. I'm trying to make it easy for > exporters to change > to the new buffer interface. > > The main idea I really want to see is that if the caller just passes > NULL instead of an address then it means they are assuming the data > will > be "unsigned bytes" It is up to the exporter to either allow this or > raise an error. > > The exporter should always be explicit if an argument for returning > the > format is provided (I may have thought differently a few days ago). Understood -- I'm for the exporters being as explicit as possible if the argument is provided. >> The general question is that there are several other instances where >> getbufferproc is allowed to return ambiguous information which must >> be handled on the client side. For example, C-contiguous data can be >> indicated either by a NULL strides pointer or a pointer to a >> properly- >> constructed strides array. > > Here. I'm trying to be easy on the exporter and the consumer. If the > data is contiguous, then neither the exporter nor will likely care > about > the strides. Allowing this to be NULL is like the current array > protocol convention which allows this to be None. See below. My comments here aren't suggesting that NULL should be disallowed. I'm basically wondering whether it is a good idea to allow NULL and something else to represent the same information. (E.g. as above, an exporter could choose to show C-contiguous data with a NULL returned to the client, or with a trivial strides array). Otherwise two different exporters exporting identical data could provide different representations, which the clients would need to be able to handle. I'm not sure that this is a recipe for perfect interoperability. >> Clients that can't handle C-contiguous >> data (contrived example, I know there is a function to deal with >> that) would then need to check both for NULL *and* inside the strides >> array if not null, before properly deciding that the data isn't >> usable them. > Not really. A client that cannot deal with strides will simply not > pass > an address to a stride array to the buffer protocol (that argument > will > be NULL). If the exporter cannot provide memory without stride > information, then an error will be raised. This doesn't really address my question, which I obscured with a poorly-chosen example. The PEP says (or at least that's how I read it) that if the client *does* provide an address for the stride array, then for un-strided arrays, the exporter may either choose to fill on NULL at that address, or provide a strides array. Might it be easier for clients if the PEP required that NULL be returned if the array is C-contiguous? Or at least strongly suggested that? (I understand that there might be cases where an naive exporter "thinks" it is dealing with a strided array but it really is contiguous, and the exporter shouldn't be required to do that detection.) The use-case isn't too strong here, but I think it's clear in the suboffsets case (see below). >> Similarly, the suboffsets can be either all negative or >> NULL to indicate the same thing. > I think it's much easier to check if suboffsets is NULL rather than > checking all the entries to see if they are -1 for the very common > case > (i.e. the NumPy case) of no dereferencing. Also, if you can't deal > with suboffsets you would just not provide an address for them. My point exactly! As written, the PEP allows an exporter to either return NULL, or an array of all negative numbers (in the case that the client requested that information), forcing a fully -conforming client to make *both* checks in order to decide what to do. Especially in this case, it would make sense to require a NULL be returned in the case of no suboffsets. This makes things easier for both clients that can deal with both suboffsets or non-offsets (e.g. they can branch on NULL, not on NULL or all-are-negative), and also for clients that can *only* deal with suboffsets. Now, in these two cases, the use-case is pretty narrow, I agree. Basically it makes things easier for savvy clients that can deal with different data types, by not forcing them to make two checks (strides == NULL or strides array is trivial; suboffsets == NULL or suboffsets are all negative) when one would do. Again, this PEP allows the same information can be passed in two very different ways, when it really doesn't seem like that ambiguity makes life that much easier for exporters. Maybe I'm wrong about this last point, though. Then there comes the trade-off -- should savvy clients bear the complexity of checking two different things? (Simple clients needn't check anything -- they just pass in NULL.) Or should the complexity be pushed to the savvy exporter to do those checks? (Simple exporters just return NULL in those variables.) I guess the question comes down to which side of the API to make the simplest, given that it appears to me that the complexity has to live somewhere. As a separate suggestion, I think a few sentences in the PEP about the protocol design, and what parts are explicitly added to make it easy for simple clients and exporters, would be helpful. Something like: "Clients that cannot deal with strided or suboffset-ed arrays should put NULL values in the corresponding getbufferproc call parameters. Then exporters will provide data in that format (either because the data are already in that format, or because the exporter chooses to convert it on behalf of the client), or the exporter will set an exception. This simplifies matters greatly for simple clients. Likewise, simple exporters which only provide C-contiguous data, or data with no suboffsets can simply return NULL if those values are requested." >> Might it be more appropriate to specify only one canonical behavior >> in these cases? Otherwise clients which don't do all the checks on >> the data might not properly interoperate with providers which format >> these values in the alternate manner. >> > It's important to also be easy to use. I don't think clients > should be > required to ask for strides and suboffsets if they can't handle them. Again, that wasn't my suggestion. My suggestion was merely that if clients ask for that information, that it come in a canonical form, so that NULL values are meaningful, as opposed to that client potentially needing to check two different things before deciding which code-path to embark upon. As above, this might or might not have the effect of adding extra complexity to the exporters. If not, good; if so, then it's worth specifically deciding which side that complexity ought to live upon. >>> 279 Get the buffer and optional information variables about the >>> buffer. >>> 280 Return an object-specific view object (which may be simply a >>> 281 borrowed reference to the object itself). >>> >> This phrasing (and similar phrasing elsewhere) is somewhat opaque to >> me. What's an "object-specific view object"? >> > At the moment it's the buffer provider. It is not defined because it > could be a different thing for each exporter. We are still > discussing > this particular point and may drop it. Fair enough. Definitely worth a clear explanation if its not dropped though. >>> 333 The struct string-syntax is missing some characters to fully >>> 334 implement data-format descriptions already available >>> elsewhere (in >>> 335 ctypes and NumPy for example). Here are the proposed additions: >>> >> Is the following table just the additions? If so, it might be good to >> show the full spec, and flag the specific additions. If not, then the >> additions should be flagged. > > Yes, these are just the additions. I don't want to do the full > spec, it > is already available elsewhere in the Python docs. Would be useful to link to the full spec from the PEP, in that case. > >> >>> 341 't' bit (number before states how many bits) >>> >> vs. >> >>> 372 According to the struct-module, a number can preceed a character >>> 373 code to specify how many of that type there are. The >>> >> I'm confused -- could this be phrased more clearly? Does '5t' refer >> to a field 5-bits wide, or 5-one bit fields? Is 'ttttt' allowed? If >> so, is it equivalent to or different from '5t'? >> > Yes, 'ttttt' is equivalent to '5t' and the difference between one > field > 5-bits wide or 5-one bit fields is a confusion based on thinking there > are fields at all. Both of those are equivalent. If you want > "fields" > then you have to define names. In that case, line 341 should be clarified. Right now, that line sort of makes it seem like the struct module should somehow unpack a '5t' into a single python object of some type, analogous to how the other entities are unpacked into single objects (which may themselves be composite, e.g. the list-of-lists and nested cases). Lines 372-3 make it clear that, say, '5g' would be unpacked into five python floats, but the fact that the 'bit' definition in line 341 explicitly mentions the number before, while no other definitions do so, almost make it appear that '5t' is supposed to be treated as a single atomic object in the same way that 'g' alone would be. Since this isn't the case, I would suggest dropping the parenthetical in line 341 as redundant and potentially misleading. >> In general, the logic of the 'locking mechanism' should be described >> at a high level at some point. It's described in nitty-gritty >> details, but at least I would have appreciated a bit more of a >> discussion about the general how and why -- this would be helpful to >> clients trying to use the locking mechanism properly. >> > > The point of locking is so that the exporter knows when it can > reallocate its buffer. Right now, reference counting is the only > way to > do that. But reference counting is not specific enough. Perhaps the > reference is because of an object that is using the same memory but > perhaps the reference is just another name pointing to exactly the > same > object. > > In the case of NumPy, NumPy needs to know when the resize method > can be > safely applied. Currently, it is ambiguous and un-clear when a NumPy > array can re-allocate its own buffer. Also, in the past exposing the > array object in Python's memory and then later re-allocating it led to > problems. > > I'll try and address this more clearly. That makes sense. What I think is needed is a high-ish level introduction to the "moving parts" of locking -- essentially covering what a simple client or exporter needs to know in order to use the interface. I felt that the discussion was a bit too low-level, leaving folks in danger of missing the forest for the trees as it were. Zach From giorgio.luciano at chimica.unige.it Wed Mar 28 10:58:42 2007 From: giorgio.luciano at chimica.unige.it (Giorgio Luciano) Date: Wed, 28 Mar 2007 16:58:42 +0200 Subject: [Numpy-discussion] matlab, idle, interactivity and teaching Message-ID: <460A82A2.90602@chimica.unige.it> Hello to all, I've thread that apperead some time ago on this list about matlab and teaching. I've discovered python recently and translated part of the routine I use in python (www.chemometrics.it). Some of my collegue asked me if I could show them how to use python. For matlab user I guess the first problem is to setup everything, but I just fixed it preparing a directory with all the package I need and a matplotlibrc file for interactive mode + a shortcut for idle -n use. The second problem is that people now wants some bells and whistles of matlab that I have to admit sometime can be very helpful for saving time. The bells and whistles are about the workspace. It's difficult to cut and paste from gnumeric/excel (I generally use txt file but it's no so immediate) and also there is no "visual" workspace. I cannot succeed also in saving workspace (I know there is a function so iosave.mat but I didn't manage easily hot to use it) For overpass this problems I've tried to use QME-DEV which is in early stage of development (alpha) but promise well. What people like of python/matplot/scipy -its free ;) -they like a lot the plotting style and capabilities (they find the png and svg file very clear and accurate) -they like IDLE as editor (ehy it's has the same color of matlab ;) ! ) So my question is . Do you have a similar experience ? How do you help people in moving the first step ? do you use (and also does it exist) a more friendly environment than IDLE except from QME-DEV. I know that this question may look silly, but in my opinion also how much is user friendly a software is very important for getting new users. Cheers to all Giorgio From stefan at sun.ac.za Wed Mar 28 11:14:59 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 28 Mar 2007 17:14:59 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <46078173.7030908@gemini.edu> References: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> <46078173.7030908@gemini.edu> Message-ID: <20070328151459.GQ18196@mentat.za.net> Hi, I notice now that we've been having this discussion on the wrong list -- oops! We're nearly done, though. On Mon, Mar 26, 2007 at 04:16:51AM -0400, James Turner wrote: > For what it's worth, I'd agree with both of you that the numeric > overflow should be documented if not fixed. It sounds like Stefan has > figured out a solution for it though. If you make sense of the code in > "ni_interpolation.c", Stefan, I'd be very interested in how to make it > calculate one less value at the edges :-). I fixed the overflow issue in SVN. I'd appreciate it if you could test and let me know whether everything works as expected. The output values are now simply clipped to the bounds of the numeric type, i.e. UInt8 to [0,255] etc. As for the values at the edges, I'm still working on it. Since we have a fundamental problem with the spline filter approach at the borders, we may also want to look at subdivision interpolation schemes, like the one described in this article: http://citeseer.ist.psu.edu/devilliers03dubucdeslauriers.html Regards St?fan From jmack at wm7d.net Wed Mar 28 12:11:05 2007 From: jmack at wm7d.net (Joseph Mack NA3T) Date: Wed, 28 Mar 2007 09:11:05 -0700 (PDT) Subject: [Numpy-discussion] Installing python/Numeric/ScientificPython Message-ID: Dear List, I'm not a python programmer, but am familiar with installing programs on Linux. I'm installing some python packages for a user. The app calls Numeric/ScientificPython. I have a separate install in ~ so as not to overwrite the standard install while I'm getting it working. The app calls my local python. zzz at amber:~/src/py_proj> head -1 ncf_info.py #!/home/zzz/usr/bin/python The architecture is ia64, Altix, 2.6.16, gcc-3.4.6. With the out-of-the-box (standard options in ../configure) python install, I get the error zzz at amber:~/src/py_proj> ./ncf_info.py area/01.ncf Traceback (most recent call last): File "./ncf_info.py", line 2, in ? from Numeric import * File "/home/zzz/usr//lib/python2.4/site-packages/Numeric/Numeric.py", line 91, in ? import multiarray ImportError: /home/zzz/usr/lib/python2.4/site-packages/Numeric/multiarray.so: undefined symbol: _Py_RefTotal I assume then that all the libraries have been found, but there is an undefined variable. I can't find *Py_RefTotal* anywhere in Numeric or in the app, so I can't find where its being called from (to turn it off if possible). However the Python build has several defines, including -DPy_REF_DEBUG and -DPy_TRACE_REFS (Misc/SpecialBuilds.txt). I can't find anyway of turning these on with ./configure, so have manually added the $DEFS to the Makefile. Now I don't get the error anymore but I get a long list of alloc/free debugging with the application run. I asked the local zpug people for advise and got the following build script zzz at amber:~/src/Python-2.4.4/objdir> more build.sh #!/bin/bash PREFIX=/home/zzz/usr/ OPTS="-Wall -O2 -D_GNU_SOURCE -fPIC" OPTS_C="--with-threads --with-pymalloc --prefix=$PREFIX --enable-shared --enable-unicode=ucs4" BLDLIBRARY="-Xlinker -rpath -Xlinker $PREFIX/lib -L. -lpython2.4" env OPT="$OPTS" ../configure $OPTS_C make OPT="$OPTS" BLDLIBRARY="$BLDLIBRARY" This doesn't seem very different to the setup I had with the standard configure, and it gives the same undefined symbol error. I don't know that the app needs to know the the RefTotal, so I'd be happy to have it switched off. Alternately I don't understand the various defines in SpecialBuilds.txt and how they interact. Can I define _Py_RefTotal without getting the debugging output? Thanks Joe -- Joseph Mack NA3T EME(B,D), FM05lw North Carolina jmack (at) wm7d (dot) net - azimuthal equidistant map generator at http://www.wm7d.net/azproj.shtml Homepage http://www.austintek.com/ It's GNU/Linux! From robert.kern at gmail.com Wed Mar 28 12:18:46 2007 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 28 Mar 2007 11:18:46 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu> Message-ID: <460A9566.80907@gmail.com> Alan G Isaac wrote: >>> On Tue, 27 Mar 2007, Robert Kern apparently wrote: >>>> Gram-Schmidt orthogonalization > >> Alan G Isaac wrote: >>> I take it from context that you consider it desirable >>> to end up with a list of matrices? > > Robert wrote: >> Honestly, I don't care. You asked about iteration, and >> I gave you an example >> where it was important that the iterates were matrices >> (such that .T worked correctly). >> Please stop moving the goal posts. > > It is really not my intent to be irritating or to move the posts. > But I had two reasons to ask. > > First, your claim is false that this was important in your example. > (To see this, replace M[1:] with M.A[1:].) > So actually your code would work the same if iteration over > matrices were producing arrays > (after replacing M[0] with M[0,:] to match that case). You're right, it does, but only because the first place it is used, it gets operated with the matrix row-vector base before it does anything else, gets coerced into a matrix, and then gets assigned back to rowv. I don't tolerate that kind of implicitness in my code. > Second, when I spoke of *desirability*, the output is relevant. > Try nump.mat(ortho) to see what I mean. If I wanted something other than a list of matrix row-vectors (which I might), then I'd post-process. Keeping the clarity of the main algorithm is also important. > If iteration were to produce arrays, > the outcome of implementing the algorithm > (e.g., using ``dot``) > would be I suggest more "desirable", > in that numpy.mat(ortho) would work as hoped/expected. Ditto for not using matrix objects at all. > In this sense, the example perhaps speaks against your intent. > You offer code that would work just fine if iteration > yielded arrays. > > Apologies in advance if this again seems tangential to my > original request. To me it does not. > >> If you really want iterates to be arrays, just use .A. > > I have mentioned that several times. > It is orthogonal to the design issue I raised. > (I suggested that under the current behavior you do not > usually end up with what you want when iterating over matrices.) > But again, I recognize that I am alone in that view. > I am just trying to understand why. People have been giving you reasons, over and over again. You are simply refusing to listen to them. You have a use case for arrays being the iterates. You are presuming that the only argument that can beat that is another use case for matrix objects being the iterates. This is not true; there are other principles at work. If we were to make matrix objects behave in different ways for different methods to satisfy what one person thinks is the most convenient way to use each method, we would have a patchwork object with no overall principle that people can use to understand what is going on. They would simply have to remember which behavior someone thought was the most convenient for each thing. Remember the lessons of rand() and randn(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bhendrix at enthought.com Wed Mar 28 12:42:52 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Wed, 28 Mar 2007 11:42:52 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <200703272024.15720.pgmdevlist@gmail.com> References: <4609B1E4.3060601@enthought.com> <200703272024.15720.pgmdevlist@gmail.com> Message-ID: <460A9B0C.5060105@enthought.com> Thanks Pierre, works like a charm. One question though, how is defining a class attribute in __new__ any more thread-safe? Bryce Pierre GM wrote: > On Tuesday 27 March 2007 20:08:04 Bryce Hendrix wrote: > >> We have a class which is a subclass of ndarray which defines >> __array_finalize__ to add an attribute. The class looks something like >> this: >> > > Ahah, we ran into this problem a few months ago: > You should not initialize your "units" attribute in the __new__ method, as > it's not thread-safe first, but more important as you will lose it if you try > to get a view of your new object (because in that case, __new__ is not > called). > Instead, try something like that: > > def __array_finalize__(self, obj): > self.units = getattr(obj, 'units', yourdefaultunit) > > where your defaultunit is the value you would use in __new__. True, > yourdefaultunit may change, and even be lost if there's a break between > __new__ and __array_finalize__, but that should work. Lemme know. > > Note that things are always tricky w/ concatenate: when you concatenate your > UnitArray, you should have a UnitArray, OK. But if you mix UnitArray and > ndarray ? What should take precedence ? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Wed Mar 28 13:25:57 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 28 Mar 2007 19:25:57 +0200 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070328151459.GQ18196@mentat.za.net> References: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> <46078173.7030908@gemini.edu> <20070328151459.GQ18196@mentat.za.net> Message-ID: <20070328172557.GU18196@mentat.za.net> On Wed, Mar 28, 2007 at 05:14:59PM +0200, Stefan van der Walt wrote: > As for the values at the edges, I'm still working on it. OK, that was a one-line patch. Please test to see if there are any subtle conditions on the border that I may have missed. I know of one already, but I'd be glad if you can find any others :) Cheers St?fan From pgmdevlist at gmail.com Wed Mar 28 13:25:53 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Wed, 28 Mar 2007 13:25:53 -0400 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <460A9B0C.5060105@enthought.com> References: <4609B1E4.3060601@enthought.com> <200703272024.15720.pgmdevlist@gmail.com> <460A9B0C.5060105@enthought.com> Message-ID: <200703281325.53941.pgmdevlist@gmail.com> On Wednesday 28 March 2007 12:42:52 Bryce Hendrix wrote: > Thanks Pierre, works like a charm. One question though, how is defining > a class attribute in __new__ any more thread-safe? It's not, of course, and that's why it shouldn't be used. However, it's quite convenient and easier to use, and do you really need thread-safe objects ? In your example, if "yourdefaultunit" takes some simple value, you could use that value directly instead of the class attributes, which could be slightly messier to read but definitely thread-safe. The second aspect about initialization is that when a ndarray is viewed as one of its subclasses, the actual memory space has already been allocated, so there's no call to __new__. Instead, you have to rely on __array_finalize__ to initialize the attributes specific to your subclass. From brad.malone at gmail.com Wed Mar 28 13:54:54 2007 From: brad.malone at gmail.com (Brad Malone) Date: Wed, 28 Mar 2007 10:54:54 -0700 Subject: [Numpy-discussion] Help with linear_least_squares Message-ID: Hi, I'm running into an error with linear least squares that I'm not sure what to do about. For reference, I'm trying to fit a curve corresponding to the example at the bottom of this wikipedia page: http://en.wikipedia.org/wiki/Linear_least_squares My code looks like this: from numpy import * import LinearAlgebra A=array([0,1,2,1,4,1,-1,1]) A.shape=(4,2) b=array([3,3,4,2]) b.shape= (4,1) sol=LinearAlgebra.linear_least_squares(A,b) And I'm getting the following error: File "", line 1, in ? File "/usr/lib/python2.4/site-packages/Numeric/LinearAlgebra.py", line 410, in linear_least_squares t =_commonType(a, b) File "/usr/lib/python2.4/site-packages/Numeric/LinearAlgebra.py", line 30, in _commonType t = a.typecode() AttributeError: 'numpy.ndarray' object has no attribute 'typecode' -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Wed Mar 28 14:04:45 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 28 Mar 2007 11:04:45 -0700 Subject: [Numpy-discussion] Help with linear_least_squares In-Reply-To: References: Message-ID: <460AAE3D.6060107@ee.byu.edu> Brad Malone wrote: > Hi, I'm running into an error with linear least squares that I'm not > sure what to do about. For reference, I'm trying to fit a curve > corresponding to the example at the bottom of this wikipedia page: > http://en.wikipedia.org/wiki/Linear_least_squares > You are mixing numpy arrays with a package from Numeric (which numpy replaces) You should instead use the linear algebra routines from numpy. Thus, eliminate import LinearAlgebrea Then, as the last line sol = linalg.lstsq(A,b) You also don't need to reshape the b array. It will work fine as a 1-d array. So, code is from numpy import * A = reshape([0,1,2,1,4,1,-1,1],(4,2)) b = array([3,3,4,2]) sol,resids,rank, s = linalg.lstsq(A,b) sol is your desired solution vector -Travis From Chris.Barker at noaa.gov Wed Mar 28 14:08:22 2007 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 28 Mar 2007 11:08:22 -0700 Subject: [Numpy-discussion] Help with linear_least_squares In-Reply-To: References: Message-ID: <460AAF16.4040407@noaa.gov> Brad Malone wrote: > Hi, I'm running into an error with linear least squares that I'm not > sure what to do about. For reference, I'm trying to fit a curve > corresponding to the example at the bottom of this wikipedia page: > http://en.wikipedia.org/wiki/Linear_least_squares > > My code looks like this: > > from numpy import * > import LinearAlgebra > A=array([0,1,2,1,4,1,-1,1]) > A.shape=(4,2) > b=array([3,3,4,2]) > b.shape= (4,1) > sol=LinearAlgebra.linear_least_squares(A,b) Your using the old Numeric LinearAlgebra module with the new numpy;s arrays. Try this: >>> from numpy import linalg >>> sol=linalg.lstsq(A,b) >>> sol (array([[ 0.33898305], [ 2.57627119]]), array([ 0.30508475]), 2, array([ 4.72879916, 1.62433326])) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Wed Mar 28 17:47:15 2007 From: aisaac at american.edu (Alan Isaac) Date: Wed, 28 Mar 2007 16:47:15 -0500 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> Message-ID: On Tue, 27 Mar 2007, Zachary Pincus wrote: > M[i], which equals M[i,:] Of course this equality must break. That was stated at the outset. As I said before, this may be impossible or undesirable. But, as I said before, it seems prima facie natural for M[i] to be ordinary Python indexing while M[i,:] is numpy indexing, each with its own natural interpretation. In any case, you may be spending too much energy on this. I am not a developer, and no developer has expressed a desire to make such a change. The current behavior is currently safe (if odd). Cheers, Alan Isaac From zpincus at stanford.edu Wed Mar 28 17:00:31 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed, 28 Mar 2007 14:00:31 -0700 Subject: [Numpy-discussion] nd_image.affine_transform edge effects In-Reply-To: <20070328172557.GU18196@mentat.za.net> References: <1F5FEF4C-264F-4FD1-8E92-77811C200B9A@stanford.edu> <46078173.7030908@gemini.edu> <20070328151459.GQ18196@mentat.za.net> <20070328172557.GU18196@mentat.za.net> Message-ID: Very cool -- Thanks for digging into the code and making these fixes, St?fan! The ndimage C code is non-trivial for sure. I'll test things out in the next couple of days. Thanks again, Zach On Mar 28, 2007, at 10:25 AM, Stefan van der Walt wrote: > On Wed, Mar 28, 2007 at 05:14:59PM +0200, Stefan van der Walt wrote: >> As for the values at the edges, I'm still working on it. > > OK, that was a one-line patch. Please test to see if there are any > subtle conditions on the border that I may have missed. I know of one > already, but I'd be glad if you can find any others :) > > Cheers > St?fan > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From aisaac at american.edu Wed Mar 28 18:25:57 2007 From: aisaac at american.edu (Alan Isaac) Date: Wed, 28 Mar 2007 17:25:57 -0500 Subject: [Numpy-discussion] matrix indexing question (final post) In-Reply-To: <460A9566.80907@gmail.com> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><460A9566.80907@gmail.com> Message-ID: On Wed, 28 Mar 2007, Robert Kern wrote: > People have been giving you reasons, over and over again. > You are simply refusing to listen to them. Exploring whether the reasoning is adequate is not the same as refusing to listen. I do not presume my view is correct. > You have a use case for arrays being the iterates. You >are presuming that the only argument that can beat that is >another use case for matrix objects being the iterates. >This is not true; there are other principles at work. Put slightly differently: given the surprising passion of the attacks at the suggestion that perhaps iteration over a matrix might more consistently yield arrays, I presumed there must be *many* instances in which it was obviously desirable that such iteration should yield matrices. So I asked to see some. In the context of this discussion, I found the (lack of) responses very interesting. Even in your thoughtful response it proved irrelevant rather than important for iteration over matrices to yield matrices. I understand that some people claim that a general principle of consistency is involved. I have not been able to understand this particular design decision as a matter of consistency, and I have tried to say why. However I am just a user (and supporter) of numpy, and as indicated in other posts, I make no pretense of deep insight into the design decisions. In this case, I simply wanted to open a discussion of a design decision, not win an "argument". Anyway, I understand that I am being perceived as bull-headed here, so I'll let this go. Thanks for your attempt to help me see the virtues of the current design. Cheers, Alan Isaac From zpincus at stanford.edu Wed Mar 28 17:39:35 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed, 28 Mar 2007 14:39:35 -0700 Subject: [Numpy-discussion] matrix indexing question (final post) In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><460A9566.80907@gmail.com> Message-ID: <2E0AE389-F17F-4683-836D-DA3163FFDD11@stanford.edu> It's because the property that A[i] == A[i,...] is much more important to most numpy users than the results of a particular (mis) use of the matrix class. This has been explained in many different contexts over many different email messages by many different people. You're not looking at the big picture, which has been fairly explicitly spelled out by myself and others. I'm not even sure why I'm replying at this point. Zach On Mar 28, 2007, at 3:25 PM, Alan Isaac wrote: > On Wed, 28 Mar 2007, Robert Kern wrote: >> People have been giving you reasons, over and over again. >> You are simply refusing to listen to them. > > Exploring whether the reasoning is adequate is not the same > as refusing to listen. I do not presume my view is correct. > >> You have a use case for arrays being the iterates. You >> are presuming that the only argument that can beat that is >> another use case for matrix objects being the iterates. >> This is not true; there are other principles at work. > > Put slightly differently: > given the surprising passion of the attacks at the > suggestion that perhaps iteration over a matrix might more > consistently yield arrays, I presumed there must be *many* > instances in which it was obviously desirable that such > iteration should yield matrices. So I asked to see some. > > In the context of this discussion, I found the (lack of) > responses very interesting. Even in your thoughtful > response it proved irrelevant rather than important for > iteration over matrices to yield matrices. > > I understand that some people claim that a general principle > of consistency is involved. I have not been able to > understand this particular design decision as a matter of > consistency, and I have tried to say why. However I am just > a user (and supporter) of numpy, and as indicated in other > posts, I make no pretense of deep insight into the design > decisions. In this case, I simply wanted to open > a discussion of a design decision, not win an "argument". > > Anyway, I understand that I am being perceived as > bull-headed here, so I'll let this go. Thanks for your > attempt to help me see the virtues of the current design. > > Cheers, > Alan Isaac > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Wed Mar 28 17:46:25 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Wed, 28 Mar 2007 23:46:25 +0200 Subject: [Numpy-discussion] matrix indexing question (final post) In-Reply-To: References: <460A9566.80907@gmail.com> Message-ID: <20070328214625.GW18196@mentat.za.net> On Wed, Mar 28, 2007 at 05:25:57PM -0500, Alan Isaac wrote: > given the surprising passion of the attacks at the > suggestion that perhaps iteration over a matrix might more > consistently yield arrays, I presumed there must be *many* > instances in which it was obviously desirable that such > iteration should yield matrices. So I asked to see some. Matrices strike me as a bit of an anomaly. I would expect an N-dimensional container to contain (N-1)-dimensional objects. But matrices cannot have dimensions less than 2, (and in numpy, can't have dimensions higher than 2). So, to me these operations don't make much sense. Is a matrix then more of a collection of matrices than a container of matrices? I'm glad we have the ndarray concept in numpy where these concepts consistently make sense. Cheers St?fan From aisaac at american.edu Wed Mar 28 20:05:00 2007 From: aisaac at american.edu (Alan Isaac) Date: Wed, 28 Mar 2007 19:05:00 -0500 Subject: [Numpy-discussion] matrix indexing question (final post) In-Reply-To: <20070328214625.GW18196@mentat.za.net> References: <460A9566.80907@gmail.com><20070328214625.GW18196@mentat.za.net> Message-ID: On Wed, 28 Mar 2007, Stefan van der Walt wrote: > Matrices strike me as a bit of an anomaly. I would expect > an N-dimensional container to contain (N-1)-dimensional > objects. Yes indeed. Cheers, Alan Isaac From stefan at sun.ac.za Wed Mar 28 19:49:54 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 29 Mar 2007 01:49:54 +0200 Subject: [Numpy-discussion] matrix indexing question (final post) In-Reply-To: References: <20070328214625.GW18196@mentat.za.net> Message-ID: <20070328234954.GA18196@mentat.za.net> On Wed, Mar 28, 2007 at 07:05:00PM -0500, Alan Isaac wrote: > On Wed, 28 Mar 2007, Stefan van der Walt wrote: > > Matrices strike me as a bit of an anomaly. I would expect > > an N-dimensional container to contain (N-1)-dimensional > > objects. > > Yes indeed. Doesn't seem to be the way the matrix world works though: octave:2> x = zeros(3,3,5); octave:3> size(x) ans = 3 3 5 octave:4> size(x(:,:,1)) ans = 3 3 octave:5> size(x(:,1,1)) ans = 3 1 Cheers St?fan From ceball at gmail.com Wed Mar 28 21:59:18 2007 From: ceball at gmail.com (Christopher Ball) Date: Thu, 29 Mar 2007 09:59:18 +0800 Subject: [Numpy-discussion] Windows binary release? Message-ID: <460B1D76.9050906@gmail.com> Hi, I wonder if anyone can tell me when the next release is planned? Or specifically, any idea when a new Windows binary installer will be released? I'm interested because we'd like to release a new version of our neural map-modeling software (which depends on numpy) soon, but the subtract.reduce bug (http://projects.scipy.org/scipy/numpy/ticket/413, now fixed) that we assume is in the current binary means we could not release a binary Windows installation ourselves. Thanks, Chris (Our software is at topographica.org) From peridot.faceted at gmail.com Thu Mar 29 01:16:39 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 29 Mar 2007 01:16:39 -0400 Subject: [Numpy-discussion] vectorized methods Message-ID: Hi, What is the current idiom for vectorizing instance methods? I don't need vectorization over self. For functions: from numpy import * @vectorize def f(x): if x>0: return 1 else: return 2 print f(array([-1,0,1])) does the right thing. But for instance methods: class useless: def __init__(self,v): self.v = v @vectorize def f(self, x): if x>self.v: return 1 else: return 0 u = useless(0) print u.f(array([-1,0,1])) raises an exception: ValueError: mismatch between python function inputs and received arguments frompyfunc is possibly more powerful (though it doesn't seem to serve the purpose either) but the fact that it always returns object arrays is thoroughly unappealing. I can create an inner function and vectorize it on every call to f, but that's not especially efficient or syntactically appealing. Surely there must be a right way to do this? Thanks, Anne M. Archibald From oliphant at ee.byu.edu Thu Mar 29 01:49:26 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 28 Mar 2007 23:49:26 -0600 Subject: [Numpy-discussion] Windows binary release? In-Reply-To: <460B1D76.9050906@gmail.com> References: <460B1D76.9050906@gmail.com> Message-ID: <460B5366.1000300@ee.byu.edu> Christopher Ball wrote: > Hi, > > I wonder if anyone can tell me when the next release is planned? Or > specifically, any idea when a new Windows binary installer will be > released? > I've been meaning to soon, but I've been stressed with too many things on my plate. If anybody would like to take on the role of release manager, I would welcome the help. -Travis From brad.malone at gmail.com Thu Mar 29 02:58:15 2007 From: brad.malone at gmail.com (Brad Malone) Date: Wed, 28 Mar 2007 23:58:15 -0700 Subject: [Numpy-discussion] Best way to run python parallel Message-ID: Hi, I use python for some fairly heavy scientific computations (at least to be running on a single processor) and would like to use it in parallel. I've seen some stuff online about Parallel Python and mpiPy, but I don't know much about them. Is a python-specific program needed to run python in parallel or are the others (e.g., mpi/poe) just more difficult to work with? And which one would you recommend? Thanks for your time, I appreciate it. Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandus at gmail.com Thu Mar 29 04:52:01 2007 From: mandus at gmail.com (Mandus) Date: Thu, 29 Mar 2007 10:52:01 +0200 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: References: Message-ID: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> On 3/29/07, Brad Malone wrote: > > Hi, I use python for some fairly heavy scientific computations (at least > to be running on a single processor) and would like to use it in parallel. > I've seen some stuff online about Parallel Python and mpiPy, but I don't > know much about them. Is a python-specific program needed to run python in > parallel or are the others ( e.g., mpi/poe) just more difficult to work > with? And which one would you recommend? There are, speaking generally, two ways to achive this. One is running a modified python linked with mpi, the other is using a python module linked with mpi, which can be used directly in a regular python. In both cases you will run your application using mpirun, just as with a C/C++ based parallel program. Both these approaches have pros and cons, it depends on your needs. As you mention, there exist several python mpi wrappers.There are mpi4py at http://www.cimec.org.ar/python/mpi4py.html (also at http://mpi4py.scipy.org/, but that page seems to be hacked today). If I started today with Python and MPI, I probably have checked out mpi4py first as it seems to be most uptodate - but I haven't tested it myself... I have used pypar ( http://datamining.anu.edu.au/~ole/pypar/) with success in the past, but I'm not sure it's updated for numpy. There is also PyMPI ( http://pympi.sourceforge.net), I don't think this is very actively developed anymore. There are a few more options, I can dig them out if you need more. You have to watch out for one thing: Some of these implementations can communicate NumPy arrays directly using low-level C API (basically send/recv of the data-pointer in a NumPy array), which gives very efficient communication. Some of them can not do this, and will use pickle to send a NumPy array - which yields bad performance. Pypar is in the first category (with NumPy == Numeric, if it's not updated recently...), PyMPI was in the second category last time I checked. I'm not sure about mpi4py, but I think it is in the first category - the good one. But watch out for this! I hope this gives you some pointers at where you can proceed! -- Mandus The only dr. Mandus around. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at shrogers.com Thu Mar 29 08:11:41 2007 From: steve at shrogers.com (Steven H. Rogers) Date: Thu, 29 Mar 2007 06:11:41 -0600 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> References: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> Message-ID: <460BACFD.1030503@shrogers.com> Mandus wrote: > > > On 3/29/07, *Brad Malone* > wrote: > > Hi, I use python for some fairly heavy scientific computations (at > least to be running on a single processor) and would like to use it > in parallel. I've seen some stuff online about Parallel Python and > mpiPy, but I don't know much about them. Is a python-specific > program needed to run python in parallel or are the others ( e.g., > mpi/poe) just more difficult to work with? And which one would you > recommend? > > > > There are, speaking generally, two ways to achive this. One is running a > modified python linked with mpi, the other is using a python module > linked with mpi, which can be used directly in a regular python. In both > cases you will run your application using mpirun, just as with a C/C++ > based parallel program. Both these approaches have pros and cons, it > depends on your needs. > Another option coming 'Real Soon Now` is `IPython1 `. It's the enhanced Python interactive shell with the added capability for distributed interactive computing. # Steve From svetosch at gmx.net Thu Mar 29 09:42:48 2007 From: svetosch at gmx.net (Sven Schreiber) Date: Thu, 29 Mar 2007 15:42:48 +0200 Subject: [Numpy-discussion] Further matrix oddities: no inner product In-Reply-To: References: Message-ID: <460BC258.2050308@gmx.net> Charles R Harris schrieb: > > Nope, vdot doesn't work for row and column vectors. So there is *no* > builtin inner product that works for matrices. I wonder if we should > have one, and if so, what it should be called. I think that vdot should > probably be modified to do the job. There is also the question of > whether or not v.T * v should be a scalar when v is a column vector. I > believe that construction is commonly used in matrix algebra as an alias > for the inner product, although strictly speaking it uses the mapping > between a vector space and its dual that the inner product provides. > As a matrix-using user and w/o too much thinking, I would suggest to treat inner() as a reduce-like method returning a scalar (I believe such in the context of other functions a similar issue was discussed here some time ago), and leave '*'-style multiplication alone (no special casing there -- actually due to numpy's broadcasting capabilities, it shouldn't be a problem to get a 1,1-matrix in place of a scalar, right?). Thanks for caring about matrices, Sven From giorgio.luciano at chimica.unige.it Thu Mar 29 09:54:14 2007 From: giorgio.luciano at chimica.unige.it (Giorgio Luciano) Date: Thu, 29 Mar 2007 15:54:14 +0200 Subject: [Numpy-discussion] Very odd problem of installation of matplotlib/numpy on XP Message-ID: <460BC506.9040802@chimica.unige.it> I've installed in my machine in the following order python 2.5 numpy 1.01 matplot lib 0.87 scipy 0.52 wxPython 2.8 with no problem I've also installed the same packages at home and in another two computer and everything went fine. The I was asked to install this "configuaration" in some classroom machines and also on another computer and I continue getting this error The import of the numpy version of the _transforms module, _ns_transforms, failed. This is is either because numpy was unavailable when matplotlib was compiled, because a dependency of _ns_transforms could not be satisfied, or because the build flag for this module was turned off in setup.py. If it appears that _ns_transforms was not built, make sure you have a working copy of numpy and then re-install matplotlib. Otherwise, the following traceback gives more details: Traceback (most recent call last): File "", line 1, in from pylab import * File "C:\Python25\Lib\site-packages\pylab.py", line 1, in from matplotlib.pylab import * File "C:\Python25\Lib\site-packages\matplotlib\pylab.py", line 201, in from axes import Axes, PolarAxes File "C:\Python25\Lib\site-packages\matplotlib\axes.py", line 14, in from artist import Artist, setp File "C:\Python25\Lib\site-packages\matplotlib\artist.py", line 4, in from transforms import identity_transform File "C:\Python25\Lib\site-packages\matplotlib\transforms.py", line 223, in from _transforms import Value, Point, Interval, Bbox, Affine File "C:\Python25\Lib\site-packages\matplotlib\_transforms.py", line 17, in from matplotlib._ns_transforms import * ImportError: DLL load failed: Impossibile trovare il modulo specificato but I can assure that If I check numpy installation before installing matplot lib it seems everything fine. All computer have Windows XP home edition 2002 SP2 the only difference is in the RAM quantity. (more than 256 in the computer where everything works) but it seems so strange to me that it is the ram (I've also installed in another computer , old one, and everything works) Any IDEA ???? Thanks in advance for any suggestion Giorgio From fperez.net at gmail.com Thu Mar 29 11:44:22 2007 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 29 Mar 2007 09:44:22 -0600 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: <460BACFD.1030503@shrogers.com> References: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> <460BACFD.1030503@shrogers.com> Message-ID: On 3/29/07, Steven H. Rogers wrote: > Another option coming 'Real Soon Now` is `IPython1 > `. It's the enhanced Python > interactive shell with the added capability for distributed interactive > computing. This is moving slower than we'd all like due to the many constraints on our time, but it is being actively developed. I'm planning on releasing ipython 0.7.4 in the next week or two, and Brian is hard at work cleaning up the 'saw' branch where all recent development has taken place. Once 0.7.4 (of trunk) is out, we'll bring all that code into saw to begin full integration. On April 28, we're having a coding sprint in Boulder, Colorado, to work on this with other interested parties from the Colorado Front Range area: http://ipython.scipy.org/moin/Developer_Zone/Sprint I hope that sprint will both serve as a motivating deadline for me, and as a good opportunity to move forward and recruit new participants. In summary, I hope that by May things may have stabilized at least to the point that: 1. The saw branch is the only one actively developed, and will be usable for day to day production work, albeit with the caveats of a not-yet-set-in-stone API. 2. We'll have some simple examples/demos on how to use it included in the distribution. 3. That branch will contain a 'normal' ipython as well, though I don't expect that one to replace trunk yet. HTH. Regards, f From bhendrix at enthought.com Thu Mar 29 12:04:51 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 29 Mar 2007 11:04:51 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <200703281325.53941.pgmdevlist@gmail.com> References: <4609B1E4.3060601@enthought.com> <200703272024.15720.pgmdevlist@gmail.com> <460A9B0C.5060105@enthought.com> <200703281325.53941.pgmdevlist@gmail.com> Message-ID: <460BE3A3.2000305@enthought.com> I spoke too soon, this code fails with the example you gave: def test_concatenate(self): unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters) # create another instance with different units. This instance # is never used, but calls __new__ with different units unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet) new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) self.assertEqual(new_unit_ary.units, meters) Any other ideas? Bryce Pierre GM wrote: > On Wednesday 28 March 2007 12:42:52 Bryce Hendrix wrote: > >> Thanks Pierre, works like a charm. One question though, how is defining >> a class attribute in __new__ any more thread-safe? >> > > It's not, of course, and that's why it shouldn't be used. However, it's quite > convenient and easier to use, and do you really need thread-safe objects ? > In your example, if "yourdefaultunit" takes some simple value, you could use > that value directly instead of the class attributes, which could be slightly > messier to read but definitely thread-safe. > > The second aspect about initialization is that when a ndarray is viewed as one > of its subclasses, the actual memory space has already been allocated, so > there's no call to __new__. Instead, you have to rely on __array_finalize__ > to initialize the attributes specific to your subclass. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Mar 29 12:21:28 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Mar 2007 10:21:28 -0600 Subject: [Numpy-discussion] Windows binary release? In-Reply-To: <460B5366.1000300@ee.byu.edu> References: <460B1D76.9050906@gmail.com> <460B5366.1000300@ee.byu.edu> Message-ID: On 3/28/07, Travis Oliphant wrote: > > Christopher Ball wrote: > > Hi, > > > > I wonder if anyone can tell me when the next release is planned? Or > > specifically, any idea when a new Windows binary installer will be > > released? > > > > I've been meaning to soon, but I've been stressed with too many things > on my plate. If anybody would like to take on the role of release > manager, I would welcome the help. Is there any reason we shouldn't just release the development branch at this point? I'm sure there are more bugs to be fixed and more documentation to write, but we could wait forever trying to get everything just right. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Mar 29 12:42:34 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 29 Mar 2007 09:42:34 -0700 Subject: [Numpy-discussion] Windows binary release? In-Reply-To: References: <460B1D76.9050906@gmail.com> <460B5366.1000300@ee.byu.edu> Message-ID: <460BEC7A.4030204@ee.byu.edu> Charles R Harris wrote: > > > On 3/28/07, *Travis Oliphant* > wrote: > > Christopher Ball wrote: > > Hi, > > > > I wonder if anyone can tell me when the next release is planned? Or > > specifically, any idea when a new Windows binary installer will be > > released? > > > > I've been meaning to soon, but I've been stressed with too many things > on my plate. If anybody would like to take on the role of release > manager, I would welcome the help. > > > Is there any reason we shouldn't just release the development branch > at this point? I'm sure there are more bugs to be fixed and more > documentation to write, but we could wait forever trying to get > everything just right. I've been going through the tickets and trying to resolve the most serious ones. There is still a memory error in one instance, but I think that can wait. Does anybody else know of a show-stopper bug? Let's shoot for a release date of next Monday (April 2, 2007). -Travis From ellisonbg.net at gmail.com Thu Mar 29 12:49:01 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 29 Mar 2007 10:49:01 -0600 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: References: Message-ID: <6ce0ac130703290949j16ac1d44r866602675d3faca1@mail.gmail.com> A few replies have already given you some idea of the options. A comments to supplement those made so far: 1. mpi4py. I have used this quite a bit and it is incredible. It seems to build just about anywhere, and it has very complete coverage of the mpi spec - even the mpi-2 stuff. In fact, the coverage of the MPI spec is so good that the test suite for mpi4py has uncovered _many_ bugs in the various underlying MPI implementations. mpi4py also handles numpy arrays well. Lisandro Dalcin (author of mpi4py) has written a paper that has performance benchmarks. Amazingly, mpi4py has almost the same performance as low-level C code when numpy arrays are passed around. If you need to use MPI, you will be very happy that you used mpi4py. 2. IPython1. As Fernando mentioned, we have been doing substantial work to enable interactive parallel computing with IPython. The currently stable version of this work is documented here: http://ipython.scipy.org/moin/Parallel_Computing People are using this today in production codes. We are, however, busy working on a new release, called "saw" - it should be out in the next 2 weeks and has many new features and improvements over the current release. There is a talk that I gave at PyCon about IPython1 here: http://ipython.scipy.org/moin/About/Presentations The most important question you for though, is which of these tools should you use? Here is my take. If you have a problem that can be trivially parallelized (meaning the different processors don't have to communicate much), then IPython1 is going to be your best bet. It will allow you to do this type of stuff quickly and easily - and in a fully interactive way. The new version of IPython1 (saw) that is coming out soon even has a nice load balanced task farming system that makes this type of stuff _super_ easy. If, however, you need to send lots of data between the processors, you should absolutely use MPI.. I highly recommend using OpenMPI for your MPi implementation and using mpi4py on top of that. The best part is that IPython1 has been designed to play well with mpi4py (and the other python MPI bindings). Thus, you can write regular MPI code and then run/debug/control it interactively using IPython1. Please let us know if you have other questions or need help getting going with IPython1 or mpi4py. Cheers, Brian On 3/29/07, Brad Malone wrote: > Hi, I use python for some fairly heavy scientific computations (at least to > be running on a single processor) and would like to use it in parallel. > I've seen some stuff online about Parallel Python and mpiPy, but I don't > know much about them. Is a python-specific program needed to run python in > parallel or are the others ( e.g., mpi/poe) just more difficult to work > with? And which one would you recommend? > > Thanks for your time, I appreciate it. > > Brad > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From faltet at carabos.com Thu Mar 29 13:02:49 2007 From: faltet at carabos.com (Francesc Altet) Date: Thu, 29 Mar 2007 19:02:49 +0200 Subject: [Numpy-discussion] Windows binary release? In-Reply-To: <460BEC7A.4030204@ee.byu.edu> References: <460B1D76.9050906@gmail.com> <460B5366.1000300@ee.byu.edu> <460BEC7A.4030204@ee.byu.edu> Message-ID: <1175187769.2820.8.camel@localhost.localdomain> El dj 29 de 03 del 2007 a les 09:42 -0700, en/na Travis Oliphant va escriure: > Charles R Harris wrote: > > > > > > > On 3/28/07, *Travis Oliphant* > > wrote: > > > > Christopher Ball wrote: > > > Hi, > > > > > > I wonder if anyone can tell me when the next release is planned? Or > > > specifically, any idea when a new Windows binary installer will be > > > released? > > > > > > > I've been meaning to soon, but I've been stressed with too many things > > on my plate. If anybody would like to take on the role of release > > manager, I would welcome the help. > > > > > > Is there any reason we shouldn't just release the development branch > > at this point? I'm sure there are more bugs to be fixed and more > > documentation to write, but we could wait forever trying to get > > everything just right. > > > I've been going through the tickets and trying to resolve the most > serious ones. There is still a memory error in one instance, but I > think that can wait. Does anybody else know of a show-stopper bug? Well, the next are not exactly a show-stopper, but they are a bit uncomfortable to live with: #483 Junk in the representation of string fields of recarrays #479 MemoryError creating recarrays from medium sized python objects Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth From pgmdevlist at gmail.com Thu Mar 29 13:36:58 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 29 Mar 2007 13:36:58 -0400 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <460BE3A3.2000305@enthought.com> References: <4609B1E4.3060601@enthought.com> <200703281325.53941.pgmdevlist@gmail.com> <460BE3A3.2000305@enthought.com> Message-ID: <200703291336.59493.pgmdevlist@gmail.com> On Thursday 29 March 2007 12:04:51 Bryce Hendrix wrote: > I spoke too soon, this code fails with the example you gave: mmh, I tried to use the class you linked to last time: the only modifications I gave are listed below class UnitArray(numpy.ndarray): __array_priority__ = 10.0 default_unit = "meters" def __array_finalize__(self, obj): self.units = getattr(obj,'units',UnitArray.default_unit) then I used that for testing: meters = "meters" feet = "feet" unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters) unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet) new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) print new_unit_ary.units And it seems to work. Could you be a bit more specific ? From bhendrix at enthought.com Thu Mar 29 13:47:48 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 29 Mar 2007 12:47:48 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <200703291336.59493.pgmdevlist@gmail.com> References: <4609B1E4.3060601@enthought.com> <200703281325.53941.pgmdevlist@gmail.com> <460BE3A3.2000305@enthought.com> <200703291336.59493.pgmdevlist@gmail.com> Message-ID: <460BFBC4.6020605@enthought.com> doh! I followed the example on the Wiki which does not define the class attribute in the class scope, but in __new__. Adding the declaration to the class scope seems to work. Thanks, Bryce Pierre GM wrote: > On Thursday 29 March 2007 12:04:51 Bryce Hendrix wrote: > >> I spoke too soon, this code fails with the example you gave: >> > > mmh, I tried to use the class you linked to last time: the only modifications > I gave are listed below > > class UnitArray(numpy.ndarray): > __array_priority__ = 10.0 > default_unit = "meters" > > def __array_finalize__(self, obj): > self.units = getattr(obj,'units',UnitArray.default_unit) > > then I used that for testing: > meters = "meters" > feet = "feet" > unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) > unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters) > > unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet) > > new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) > print new_unit_ary.units > > And it seems to work. Could you be a bit more specific ? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Mar 29 14:09:07 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 29 Mar 2007 14:09:07 -0400 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <460BFBC4.6020605@enthought.com> References: <4609B1E4.3060601@enthought.com> <200703291336.59493.pgmdevlist@gmail.com> <460BFBC4.6020605@enthought.com> Message-ID: <200703291409.07407.pgmdevlist@gmail.com> On Thursday 29 March 2007 13:47:48 Bryce Hendrix wrote: > doh! I followed the example on the Wiki which does not define the class > attribute in the class scope, but in __new__. Adding the declaration to > the class scope seems to work. Yeah, sorry about that, I really should update this wiki page. I gonna try later this afternoon/evening. Glad it works, though. However, that only part of your problem: With concatenate, what unit should take precedence ? In your example, concatenating unit_ary_1 and unit_ary_2 should give a unit_array w/ meters as units. But what if you concatenate unit_1 and unit_3 ? Meters or feet ? The idea would be to write a custom concatenate function, that would convert the units of each argument to the unit of the first one (for example), and call numpy.concatenate. Or something like that. From bhendrix at enthought.com Thu Mar 29 14:09:38 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 29 Mar 2007 13:09:38 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <200703291336.59493.pgmdevlist@gmail.com> References: <4609B1E4.3060601@enthought.com> <200703281325.53941.pgmdevlist@gmail.com> <460BE3A3.2000305@enthought.com> <200703291336.59493.pgmdevlist@gmail.com> Message-ID: <460C00E2.4050304@enthought.com> I really should widen my tests before proclaiming success... If you change the default units to "feet", the result of concatenating two UnitArrays instances with "meters" units is a UnitArray with "feet". Bryce Pierre GM wrote: > On Thursday 29 March 2007 12:04:51 Bryce Hendrix wrote: > >> I spoke too soon, this code fails with the example you gave: >> > > mmh, I tried to use the class you linked to last time: the only modifications > I gave are listed below > > class UnitArray(numpy.ndarray): > __array_priority__ = 10.0 > default_unit = "meters" > > def __array_finalize__(self, obj): > self.units = getattr(obj,'units',UnitArray.default_unit) > > then I used that for testing: > meters = "meters" > feet = "feet" > unit_ary_1 = UnitArray(numpy.array((1,2,3)), units=meters) > unit_ary_2 = UnitArray(numpy.array((1,2,3)), units=meters) > > unit_ary_3 = UnitArray(numpy.array((1,2,3)), units=feet) > > new_unit_ary = numpy.concatenate([unit_ary_1, unit_ary_2]) > print new_unit_ary.units > > And it seems to work. Could you be a bit more specific ? > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhendrix at enthought.com Thu Mar 29 14:12:49 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 29 Mar 2007 13:12:49 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <200703291409.07407.pgmdevlist@gmail.com> References: <4609B1E4.3060601@enthought.com> <200703291336.59493.pgmdevlist@gmail.com> <460BFBC4.6020605@enthought.com> <200703291409.07407.pgmdevlist@gmail.com> Message-ID: <460C01A1.5010905@enthought.com> Yeah, I think we may need to write some custom wrapper functions for the functions we use which don't call __new__. I don't really see another way around it since by the time __array_finalize__ gets called there are no units and the problems with mixed units. bryce Pierre GM wrote: > On Thursday 29 March 2007 13:47:48 Bryce Hendrix wrote: > >> doh! I followed the example on the Wiki which does not define the class >> attribute in the class scope, but in __new__. Adding the declaration to >> the class scope seems to work. >> > > Yeah, sorry about that, I really should update this wiki page. I gonna try > later this afternoon/evening. > Glad it works, though. > > However, that only part of your problem: > With concatenate, what unit should take precedence ? In your example, > concatenating unit_ary_1 and unit_ary_2 should give a unit_array w/ meters as > units. But what if you concatenate unit_1 and unit_3 ? Meters or feet ? > > The idea would be to write a custom concatenate function, that would convert > the units of each argument to the unit of the first one (for example), and > call numpy.concatenate. Or something like that. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Mar 29 14:37:40 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 29 Mar 2007 14:37:40 -0400 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <460C00E2.4050304@enthought.com> References: <4609B1E4.3060601@enthought.com> <200703291336.59493.pgmdevlist@gmail.com> <460C00E2.4050304@enthought.com> Message-ID: <200703291437.41267.pgmdevlist@gmail.com> On Thursday 29 March 2007 14:09:38 Bryce Hendrix wrote: > I really should widen my tests before proclaiming success... If you > change the default units to "feet", the result of concatenating two > UnitArrays instances with "meters" units is a UnitArray with "feet". Not a surprise at all, and I should have thought about it: concatenate creates a plain ndarray first and ends up calling Unit_array.__array_finalize__. As the caller has no unit yet (it's a plain ndarray), the result gets the Unit_array.default_unit. That's why you should really write a custom concatenate function that checks the units of its inputs, and sets the unit of the output accordingly. From bhendrix at enthought.com Thu Mar 29 14:40:35 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 29 Mar 2007 13:40:35 -0500 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <200703291437.41267.pgmdevlist@gmail.com> References: <4609B1E4.3060601@enthought.com> <200703291336.59493.pgmdevlist@gmail.com> <460C00E2.4050304@enthought.com> <200703291437.41267.pgmdevlist@gmail.com> Message-ID: <460C0823.9050407@enthought.com> Yup, we've decided to write custom concatenate & where methods. Thanks for all the help. Bryce Pierre GM wrote: > On Thursday 29 March 2007 14:09:38 Bryce Hendrix wrote: > >> I really should widen my tests before proclaiming success... If you >> change the default units to "feet", the result of concatenating two >> UnitArrays instances with "meters" units is a UnitArray with "feet". >> > > Not a surprise at all, and I should have thought about it: > > concatenate creates a plain ndarray first and ends up calling > Unit_array.__array_finalize__. As the caller has no unit yet > (it's a plain ndarray), the result gets the Unit_array.default_unit. > > That's why you should really write a custom concatenate function that checks > the units of its inputs, and sets the unit of the output accordingly. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgmdevlist at gmail.com Thu Mar 29 14:48:34 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Thu, 29 Mar 2007 14:48:34 -0400 Subject: [Numpy-discussion] use of concatenate with subclass of ndarray In-Reply-To: <460C0823.9050407@enthought.com> References: <4609B1E4.3060601@enthought.com> <200703291437.41267.pgmdevlist@gmail.com> <460C0823.9050407@enthought.com> Message-ID: <200703291448.34957.pgmdevlist@gmail.com> On Thursday 29 March 2007 14:40:35 Bryce Hendrix wrote: > Yup, we've decided to write custom concatenate & where methods. Thanks > for all the help. Don't mention it, my pleasure. From oliphant.travis at ieee.org Thu Mar 29 15:20:55 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 29 Mar 2007 12:20:55 -0700 Subject: [Numpy-discussion] Should 0-d arrays with fields defined return a 0-d array or a scalar Message-ID: <460C1197.5030404@ieee.org> Hi all, Ticket #474 discusses the problem that getting a field from a 0-d array automatically produces a scalar (which then cannot be set). This produces the problem that recarrays code must often special-case the 0-d possibility. Thus, rarr.x[...] = blah doesn't work for 0-d arrays because rarr.x is a scalar. It makes some sense to make field selection for 0-d arrays return 0-d arrays as consistent with the changes that were made prior to the 1.0 release to allow persistence of 0-d arrays. However, changing field selection to return 0-d arrays does change behavior. A 0-d array is not a scalar (the 0-d array is not hashable for example, and the 0-d string array does not inherit from the Python string). Thus, just making the change, may not be advised. It is easy to account for and fix any errors that might arise. But, we are in a major release, I need some advice as to whether or not this is a "bug-fix" or a feature enhancement that must wait for 1.1? Any stake holders in the current behavior of arrays with records? -Travis From sberub at gmail.com Thu Mar 29 15:43:03 2007 From: sberub at gmail.com (Simon Berube) Date: Thu, 29 Mar 2007 19:43:03 -0000 Subject: [Numpy-discussion] 2D Arrays column operations Message-ID: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> Hi, I am relatively new to Python/NumPy switching over from Matlab and while porting some of my matlab code for practice I ran into the following problem. Assume we have a 2D Matrix such that a = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) If I want the second row I can simply enough take c = a[1] However, I would like to do a similar operation on the columns of the 2D Array. In matlab I could simply do c = a(:,2) to get the values array([2,5,8]) In numPy this seems to not be a valid operation. I understand that since numPy arrays are more akin to C pointer this cannot be done as easily so my question is; what is the best way to go around obtaining the column information I want the "numPy" way? I could simply take a transpose and get the information this way but this seems wasteful. From oliphant at ee.byu.edu Thu Mar 29 15:46:05 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 29 Mar 2007 12:46:05 -0700 Subject: [Numpy-discussion] 2D Arrays column operations In-Reply-To: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> References: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> Message-ID: <460C177D.60200@ee.byu.edu> Simon Berube wrote: >Hi, > I am relatively new to Python/NumPy switching over from Matlab and >while porting some of my matlab code for practice I ran into the >following problem. > >Assume we have a 2D Matrix such that >a = array([[1, 2, 3], > [4, 5, 6], > [7, 8, 9]]) > >If I want the second row I can simply enough take > >c = a[1] > >However, I would like to do a similar operation on the columns of the >2D Array. In matlab I could simply do > >c = a(:,2) to get the values array([2,5,8]) > > c = a[:,2] -Travis From oliphant at ee.byu.edu Thu Mar 29 15:46:31 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu, 29 Mar 2007 12:46:31 -0700 Subject: [Numpy-discussion] 2D Arrays column operations In-Reply-To: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> References: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> Message-ID: <460C1797.1040702@ee.byu.edu> Simon Berube wrote: >c = a(:,2) to get the values array([2,5,8]) > > Actually (0-based indexing) c = a[:,1] -Travis From stefan at sun.ac.za Thu Mar 29 15:51:28 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu, 29 Mar 2007 21:51:28 +0200 Subject: [Numpy-discussion] 2D Arrays column operations In-Reply-To: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> References: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> Message-ID: <20070329195128.GK18196@mentat.za.net> On Thu, Mar 29, 2007 at 07:43:03PM -0000, Simon Berube wrote: > Hi, > I am relatively new to Python/NumPy switching over from Matlab and > while porting some of my matlab code for practice I ran into the > following problem. > > Assume we have a 2D Matrix such that > a = array([[1, 2, 3], > [4, 5, 6], > [7, 8, 9]]) > > If I want the second row I can simply enough take > > c = a[1] > > However, I would like to do a similar operation on the columns of the > 2D Array. In matlab I could simply do > > c = a(:,2) to get the values array([2,5,8]) > > In numPy this seems to not be a valid operation. I understand that Not? In [2]: a = array([[1, 2, 3], ...: [4, 5, 6], ...: [7, 8, 9]]) In [3]: a[:,1] Out[3]: array([2, 5, 8]) Cheers St?fan From sberub at gmail.com Thu Mar 29 15:51:46 2007 From: sberub at gmail.com (Simon Berube) Date: Thu, 29 Mar 2007 19:51:46 -0000 Subject: [Numpy-discussion] 2D Arrays column operations In-Reply-To: <460C177D.60200@ee.byu.edu> References: <1175197383.526114.211370@e65g2000hsc.googlegroups.com> <460C177D.60200@ee.byu.edu> Message-ID: <1175197906.576359.155420@y66g2000hsf.googlegroups.com> Awww, this is quite right. I kept using the a[0][:] notation and I assume I am simply pulling out single arrays from the array "list". Thank you very much for the prompt reply. (And sorry for wasting your time :P) On Mar 29, 3:46 pm, Travis Oliphant wrote: > Simon Berube wrote: > >Hi, > > I am relatively new to Python/NumPy switching over from Matlab and > >while porting some of my matlab code for practice I ran into the > >following problem. > > >Assume we have a 2D Matrix such that > >a = array([[1, 2, 3], > > [4, 5, 6], > > [7, 8, 9]]) > > >If I want the second row I can simply enough take > > >c = a[1] > > >However, I would like to do a similar operation on the columns of the > >2D Array. In matlab I could simply do > > >c = a(:,2) to get the values array([2,5,8]) > > c = a[:,2] > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From chanley at stsci.edu Thu Mar 29 16:18:28 2007 From: chanley at stsci.edu (Christopher Hanley) Date: Thu, 29 Mar 2007 16:18:28 -0400 Subject: [Numpy-discussion] Should 0-d arrays with fields defined return a 0-d array or a scalar In-Reply-To: <460C1197.5030404@ieee.org> References: <460C1197.5030404@ieee.org> Message-ID: <460C1F14.7090506@stsci.edu> Hi Travis, We will need a little time to inspect our code to see if this is going to be a problem for us. We can't think of any major problems with this change off the top of our heads. Chris Travis Oliphant wrote: > Hi all, > > Ticket #474 discusses the problem that getting a field from a 0-d array > automatically produces a scalar (which then cannot be set). > This produces the problem that recarrays code must often special-case > the 0-d possibility. > > Thus, > > rarr.x[...] = blah > > doesn't work for 0-d arrays because rarr.x is a scalar. > > It makes some sense to make field selection for 0-d arrays return 0-d > arrays as consistent with the changes that were made prior to the 1.0 > release to allow persistence of 0-d arrays. > > However, changing field selection to return 0-d arrays does change > behavior. A 0-d array is not a scalar (the 0-d array is not hashable > for example, and the 0-d string array does not inherit from the Python > string). Thus, just making the change, may not be advised. > > It is easy to account for and fix any errors that might arise. But, we > are in a major release, I need some advice as to whether or not this is > a "bug-fix" or a feature enhancement that must wait for 1.1? > > Any stake holders in the current behavior of arrays with records? > > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338 From lou_boog2000 at yahoo.com Thu Mar 29 16:32:17 2007 From: lou_boog2000 at yahoo.com (Lou Pecora) Date: Thu, 29 Mar 2007 13:32:17 -0700 (PDT) Subject: [Numpy-discussion] 2D Arrays column operations In-Reply-To: <1175197906.576359.155420@y66g2000hsf.googlegroups.com> Message-ID: <448926.65560.qm@web34415.mail.mud.yahoo.com> Others can correct me, but I believe the ndarrys of numpy are *not* stored like C arrays, but rather as a contiguous memory chunk (there are some exceptions, but ignore for the moment). Then along with the data is a structure that tells numpy how to "address" the array's data memory using indices (+ other array info). In particular, the structure tells numpy the dimension and the strides for each dimension so it can skip along and pick out the right components when you write c=a[:,1]. I'm sure Travis will correct me if I'm off, but I think this is basically how numpy arrays operate. --- Simon Berube wrote: > Awww, this is quite right. I kept using the a[0][:] > notation and I > assume I am simply pulling out single arrays from > the array "list". > > Thank you very much for the prompt reply. (And sorry > for wasting your > time :P) -- Lou Pecora, my views are my own. --------------- "I knew I was going to take the wrong train, so I left early." --Yogi Berra ____________________________________________________________________________________ Be a PS3 game guru. Get your game face on with the latest PS3 news and previews at Yahoo! Games. http://videogames.yahoo.com/platform?platform=120121 From peter.skomoroch at gmail.com Thu Mar 29 16:37:02 2007 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Thu, 29 Mar 2007 20:37:02 +0000 (UTC) Subject: [Numpy-discussion] Best way to run python parallel References: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> <460BACFD.1030503@shrogers.com> Message-ID: If you want to use PyMPI or PyPar, I'm writing a series of tutorials on how to get them running on Amazon EC2, http://www.datawrangling.com/on-demand-mpi-cluster-with-python-and-ec2-part-1-of-3.html I'm using PyMPI on a 20 node EC2 cluster and everything seems groovy, but I'm relatively new to MPI, so I have probably overlooked some easier solutions. Any feedback on the writeups from Python gurus would be appreciated. -Pete From aisaac at american.edu Thu Mar 29 17:24:42 2007 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 29 Mar 2007 17:24:42 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> Message-ID: On Tue, 27 Mar 2007, Zachary Pincus apparently wrote: > Now, Bill offers up a different suggestion: indexing > M yields neither a matrix nor an array, but a class that > operates more or less like an array, except insofar as it > interacts with other matrix objects, or other objects of > similar classes. I'm interested in hearing more about > this, what trade-offs or other compromises it might > involve. So would you agree that in this case M[0] and M[0,:] would have to be different things? Or would you then want even such numpy indexing not to produce matrices?? Cheers, Alan Isaac From wfspotz at sandia.gov Thu Mar 29 18:13:44 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 29 Mar 2007 16:13:44 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> Message-ID: What I envisioned was that M[i,:] would return a row_vector and M [:,j] would return a column_vector, because this would be symmetric behavior. M[i], by convention, would behave the same as M[i,:]. But then I personally don't distinguish between "python indexing" and "numpy indexing". In both cases, __getitem__() (or __setitem__()) is called. For multiple indexes, the index object is a tuple. In any case, the behavior of "numpy indexing" as I have proposed it would return an object that inherits from matrix, thus would BE a matrix, although it would behave like an array. On Mar 29, 2007, at 3:24 PM, Alan G Isaac wrote: > On Tue, 27 Mar 2007, Zachary Pincus apparently wrote: >> Now, Bill offers up a different suggestion: indexing >> M yields neither a matrix nor an array, but a class that >> operates more or less like an array, except insofar as it >> interacts with other matrix objects, or other objects of >> similar classes. I'm interested in hearing more about >> this, what trade-offs or other compromises it might >> involve. > > So would you agree that in this case M[0] and M[0,:] would > have to be different things? Or would you then want even > such numpy indexing not to produce matrices?? > > Cheers, > Alan Isaac ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From Glen.Mabey at swri.org Thu Mar 29 18:25:17 2007 From: Glen.Mabey at swri.org (Glen W. Mabey) Date: Thu, 29 Mar 2007 17:25:17 -0500 Subject: [Numpy-discussion] .data doesn't account for .transpose()? Message-ID: <20070329222517.GB2164@bams.ccf.swri.edu> Hello, I imagine that perhaps this issue I'm seeing is only an issue because I don't thoroughly understand the buffer issues associated with numpy arrays, but here it is anyway: In [16]:a1 = numpy.zero numpy.zeros numpy.zeros_like In [16]:a1 = numpy.zeros( (2,2) ) In [17]:a1[0,:] = 1 In [18]:a1 Out[18]: array([[ 1., 1.], [ 0., 0.]]) In [19]:str(a1.data) Out[19]:'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' In [20]:a2 = a1.transpose() In [21]:str(a2.data) Out[21]:'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' That is, when getting the .data from an array, if it was C_CONTIGUOUS but was .transposed(), the .data does not reflect this operation, right? So, would that imply that a .copy() should be done first on any array that you want to access .data on? Thanks, Glen From robert.kern at gmail.com Thu Mar 29 18:32:00 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Mar 2007 17:32:00 -0500 Subject: [Numpy-discussion] .data doesn't account for .transpose()? In-Reply-To: <20070329222517.GB2164@bams.ccf.swri.edu> References: <20070329222517.GB2164@bams.ccf.swri.edu> Message-ID: <460C3E60.6020801@gmail.com> Glen W. Mabey wrote: > Hello, > > I imagine that perhaps this issue I'm seeing is only an issue because I > don't thoroughly understand the buffer issues associated with numpy > arrays, but here it is anyway: > > In [16]:a1 = numpy.zero > numpy.zeros numpy.zeros_like > > In [16]:a1 = numpy.zeros( (2,2) ) > > In [17]:a1[0,:] = 1 > > In [18]:a1 > Out[18]: > array([[ 1., 1.], > [ 0., 0.]]) > > In [19]:str(a1.data) > Out[19]:'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > > In [20]:a2 = a1.transpose() > > In [21]:str(a2.data) > Out[21]:'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > > That is, when getting the .data from an array, if it was C_CONTIGUOUS > but was .transposed(), the .data does not reflect this operation, right? Correct, it just gives you the data as-is in memory, not as interpreted via the strides. Transposition and slicing are implemented by changing the strides and not changing any data. > So, would that imply that a .copy() should be done first on any array > that you want to access .data on? asarray(a, order='C') should be sufficient, I think. That shouldn't copy if it is already correctly ordered. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Thu Mar 29 18:32:42 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Mar 2007 17:32:42 -0500 Subject: [Numpy-discussion] .data doesn't account for .transpose()? In-Reply-To: <20070329222517.GB2164@bams.ccf.swri.edu> References: <20070329222517.GB2164@bams.ccf.swri.edu> Message-ID: <460C3E8A.9090505@gmail.com> Glen W. Mabey wrote: > So, would that imply that a .copy() should be done first on any array > that you want to access .data on? Or even ascontiguousarray(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From efiring at hawaii.edu Thu Mar 29 18:37:36 2007 From: efiring at hawaii.edu (Eric Firing) Date: Thu, 29 Mar 2007 12:37:36 -1000 Subject: [Numpy-discussion] GC support Message-ID: <460C3FB0.2080504@hawaii.edu> Travis and others, In the course of trying to understand memory leaks in matplotlib I have been trying to understand a bit about the garbage collector. If I understand correctly, any container that can can hold references to other containers could lead to a reference cycle; if the container supports the gc mechanism, then the gc can at least find the cycles. If the containers do not have __del__ methods, then the gc can also break the cycles and reclaim the memory. (This also seems to imply that __del__ methods should be avoided if at all possible, and I don't understand the implications and applications of this.) I notice that numpy.ndarray does not support the gc, correct? And since an ndarray can hold other containers, it could lead to uncollectable cycles, correct? Did you decide not to include gc support because it is not actually needed or useful? If so, what am I missing? I don't think the lack of gc support in numpy has anything to do with the present leak problem in mpl, so I am asking about numpy partly out of curiosity, and partly in the hope that your answer will help me understand exactly when one really needs to worry about gc support in extension code--mpl has quite a bit of extension code, and relies on much outside extension code as well. The one little bit of extension code I wrote for numpy, the wrapper part of the contour routine, does not support the gc--and if this is a mistake, I want to know about it. (I am beginning to suspect that it should support the gc, although it is not part of our most basic problem at the moment.) Googling has not turned up much information beyond the standard python docs about the gc, extension code, and memory leaks in python. Thanks for whatever insight and advice you can provide. Eric From haase at msg.ucsf.edu Thu Mar 29 18:43:44 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu, 29 Mar 2007 15:43:44 -0700 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: References: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> <460BACFD.1030503@shrogers.com> Message-ID: Hi, What is the general feeling towards BSP on this list !? I remeber Konrad Hinsen advertising it on the SciPy workshop '03 . It is supposed to be much simpler to use than MPI, yet still powerful and flexible enough for most all applications. It is part of Konrad's ScientificPython ( != SciPy ) Some links are here: http://www.bsp-worldwide.org/ http://en.wikipedia.org/wiki/Bulk_Synchronous_Parallel Evaluating Scientific Python/BSP on selected parallel computers http://ove.nipen.no/diplom/ http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ - Sebastian Haase On 3/29/07, Peter Skomoroch wrote: > > > If you want to use PyMPI or PyPar, I'm writing a series of tutorials on how to > get them running on Amazon EC2, > > http://www.datawrangling.com/on-demand-mpi-cluster-with-python-and-ec2-part-1-of-3.html > > > I'm using PyMPI on a 20 node EC2 cluster and everything seems groovy, but I'm > relatively new to MPI, so I have probably overlooked some easier solutions. > > Any feedback on the writeups from Python gurus would be appreciated. > > -Pete From peridot.faceted at gmail.com Thu Mar 29 18:48:25 2007 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 29 Mar 2007 18:48:25 -0400 Subject: [Numpy-discussion] .data doesn't account for .transpose()? In-Reply-To: <460C3E8A.9090505@gmail.com> References: <20070329222517.GB2164@bams.ccf.swri.edu> <460C3E8A.9090505@gmail.com> Message-ID: On 29/03/07, Robert Kern wrote: > Glen W. Mabey wrote: > > > So, would that imply that a .copy() should be done first on any array > > that you want to access .data on? > > Or even ascontiguousarray(). I'd like to point out that the numpy usage of the word "contiguous" is a bit misleading: while naively one would expect it to mean that the data were contiguous in memory, what it actually means is that they are contiguous and the indexing is C-ordered: In [7]: a = arange(5) In [8]: a.flags["CONTIGUOUS"] Out[8]: True In [9]: a[::-1].flags["CONTIGUOUS"] Out[9]: False In [10]: eye(2).flags["CONTIGUOUS"] Out[10]: True In [11]: transpose(eye(2)).flags["CONTIGUOUS"] Out[11]: False There's no help for it now, I suppose. Anne From charlesr.harris at gmail.com Thu Mar 29 19:07:44 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Mar 2007 17:07:44 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> Message-ID: On 3/29/07, Bill Spotz wrote: > > What I envisioned was that M[i,:] would return a row_vector and M > [:,j] would return a column_vector, because this would be symmetric > behavior. M[i], by convention, would behave the same as M[i,:]. > > But then I personally don't distinguish between "python indexing" and > "numpy indexing". In both cases, __getitem__() (or __setitem__()) is > called. For multiple indexes, the index object is a tuple. > > In any case, the behavior of "numpy indexing" as I have proposed it > would return an object that inherits from matrix, thus would BE a > matrix, although it would behave like an array. I'm thinking that a basic problem is not having row and column types distinct from matrices. Column types would represent elements of a vector space and row types elements of the dual, they would both be 1-dimensional. One could go full bore with tensors and index signatures, upper and lower, but I think plain old matrices with the two vector types would solve most ambiguities. Thus if v were a column vector, then v.T*v would be a scalar where the product in this case is shorthand for v.T(v). Likewise, v*v.Twould be a matrix (in tensor form the indices would be upper lower respectively, but ignore that). The default translation of a normal 1-D vector would preferably be a column vector in this case, opposite to current usage, but that is not really crucial. Note that nx1 and 1xn matrices are not quite the same thing as column and row vectors, as the latter would normally be considered linear maps and v.T*v would return a scalar matrix, quite a different thing from a scalar. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg.net at gmail.com Thu Mar 29 19:10:55 2007 From: ellisonbg.net at gmail.com (Brian Granger) Date: Thu, 29 Mar 2007 17:10:55 -0600 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: References: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> <460BACFD.1030503@shrogers.com> Message-ID: <6ce0ac130703291610o4aa107f4u263495cad3e44086@mail.gmail.com> We looked at the BSP model at various points in implementing the parallel IPython stuff. While I wouldn't say that IPython uses a BSP model, there are some similarities. But in the broader realm of scientific computing, BSP has never really caught on like MPI has - in spite of having some nice ideas like being able to predict the performance of a parallel BSP code. The main difficulty is that BSP is a much more limited model that MPI - which is why you can predict the performance of BSP using codes. The main limitation is that communication and computation cannot overlap. For some codes that is fine and there might be benefits for using BSP over MPI. For general parallel codes and algorithms however, you do want to overlap communications and computation. In fact, if you don't you can severely limit the scalability of your code. Brian On 3/29/07, Sebastian Haase wrote: > Hi, > What is the general feeling towards BSP on this list !? > I remeber Konrad Hinsen advertising it on the SciPy workshop '03 . > It is supposed to be much simpler to use than MPI, yet still powerful > and flexible enough for most all applications. > It is part of Konrad's ScientificPython ( != SciPy ) > > Some links are here: > http://www.bsp-worldwide.org/ > http://en.wikipedia.org/wiki/Bulk_Synchronous_Parallel > > Evaluating Scientific Python/BSP on selected parallel computers > http://ove.nipen.no/diplom/ > > http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ > > - Sebastian Haase > > > > On 3/29/07, Peter Skomoroch wrote: > > > > > > If you want to use PyMPI or PyPar, I'm writing a series of tutorials on how to > > get them running on Amazon EC2, > > > > http://www.datawrangling.com/on-demand-mpi-cluster-with-python-and-ec2-part-1-of-3.html > > > > > > I'm using PyMPI on a 20 node EC2 cluster and everything seems groovy, but I'm > > relatively new to MPI, so I have probably overlooked some easier solutions. > > > > Any feedback on the writeups from Python gurus would be appreciated. > > > > -Pete > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From charlesr.harris at gmail.com Thu Mar 29 19:22:45 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Mar 2007 17:22:45 -0600 Subject: [Numpy-discussion] .data doesn't account for .transpose()? In-Reply-To: References: <20070329222517.GB2164@bams.ccf.swri.edu> <460C3E8A.9090505@gmail.com> Message-ID: On 3/29/07, Anne Archibald wrote: > > On 29/03/07, Robert Kern wrote: > > Glen W. Mabey wrote: > > > > > So, would that imply that a .copy() should be done first on any array > > > that you want to access .data on? > > > > Or even ascontiguousarray(). > > I'd like to point out that the numpy usage of the word "contiguous" is > a bit misleading: while naively one would expect it to mean that the > data were contiguous in memory, what it actually means is that they > are contiguous and the indexing is C-ordered: > > In [7]: a = arange(5) > > In [8]: a.flags["CONTIGUOUS"] > Out[8]: True > > In [9]: a[::-1].flags["CONTIGUOUS"] > Out[9]: False > > In [10]: eye(2).flags["CONTIGUOUS"] > Out[10]: True > > In [11]: transpose(eye(2)).flags["CONTIGUOUS"] > Out[11]: False > > There's no help for it now, I suppose. I think the preferred names are C_CONTIGUOUS and F_CONTIGUOUS, for instance: In [2]:eye(2).flags['C_CONTIGUOUS'] Out[2]:True In [3]:eye(2).T.flags['F_CONTIGUOUS'] Out[3]:True However, that may only be in svn at the moment. C_CONTIGUOUS is an alias for CONTIGUOUS and F_CONTIGUOUS is an alias for F. I think the new names are clearer than before. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Thu Mar 29 19:29:39 2007 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu, 29 Mar 2007 19:29:39 -0400 Subject: [Numpy-discussion] .data doesn't account for .transpose()? In-Reply-To: References: <20070329222517.GB2164@bams.ccf.swri.edu> <460C3E8A.9090505@gmail.com> Message-ID: <460C4BE3.3080807@physics.mcmaster.ca> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Charles R Harris wrote: > On 3/29/07, Anne Archibald wrote: >> > > I think the preferred names are C_CONTIGUOUS and F_CONTIGUOUS, for > instance: > > In [2]:eye(2).flags['C_CONTIGUOUS'] > Out[2]:True > > In [3]:eye(2).T.flags['F_CONTIGUOUS'] > Out[3]:True > > However, that may only be in svn at the moment. C_CONTIGUOUS is an alias > for > CONTIGUOUS and > F_CONTIGUOUS is an alias for F. I think the new names are clearer than > before. FWIW, you can use attributes on flags too: In [1]: eye(2).flags.c_contiguous Out[1]: True In [2]: eye(2).T.flags.f_contiguous Out[2]: True - -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGDEvj+kNzddXW8YwRAi9oAKDQCLoZjAPSSMscVkvVsxpiHgU4LwCgzhjD PQ4QdFd4urTaJND85u4ONbI= =ZflB -----END PGP SIGNATURE----- From aisaac at american.edu Thu Mar 29 20:48:43 2007 From: aisaac at american.edu (Alan G Isaac) Date: Thu, 29 Mar 2007 20:48:43 -0400 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> Message-ID: On Thu, 29 Mar 2007, Bill Spotz apparently wrote: > What I envisioned was that M[i,:] would return > a row_vector and M[:,j] would return a column_vector, > because this would be symmetric behavior. M[i], by > convention, would behave the same as M[i,:]. Can you please be explicit about the value added by that convention, as you see it? Thank you, Alan Isaac PS I assume your "vector" class would always accept a single index, for both row and column vectors? What is the return type of v[i]? From wfspotz at sandia.gov Thu Mar 29 21:43:39 2007 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 29 Mar 2007 19:43:39 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu><460958CC.4030806@noaa.gov><9A597818-A755-4190-8186-88590788CA7A@stanford.edu><9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> Message-ID: <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> On Mar 29, 2007, at 6:48 PM, Alan G Isaac wrote: > On Thu, 29 Mar 2007, Bill Spotz apparently wrote: >> What I envisioned was that M[i,:] would return >> a row_vector and M[:,j] would return a column_vector, >> because this would be symmetric behavior. M[i], by >> convention, would behave the same as M[i,:]. > > Can you please be explicit about the value added by that > convention, as you see it? I assume by "convention" you are referring to the convention that M [i] is M[i,:] is a row_vector. I see it as a design decision: Does M[i] mean something? No: Then raise an exception. If users want to iterate over M, this forces them to use the double index notation to specify what they are iterating over. I would find this acceptable. Yes: What should M[i] represent/return? Array representing row vector: Array representing column vector: I would argue against linear algebra objects returning raw arrays because their meaning/interface can be ambiguous, as this thread demonstrates row_vector: Reasonable choice, analogous to conventional mathematical notation column_vector: Reasonable choice, but less analogous to conventional mathematical notation Of all the choices, raising an exception and returning a row_vector are (in my opinion, anyway) the best. Unless someone has a better argument. > Thank you, > Alan Isaac > > PS I assume your "vector" class would always accept a single index, > for both row and column vectors? What is the return type of v[i]? My best guess at a best design would be that it would return a scalar. And yes, vectors would always accept a single index. But because they are matrices, they would accept two indexes as well. > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-5451 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From tim.hochberg at ieee.org Thu Mar 29 23:44:17 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 29 Mar 2007 20:44:17 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: """matrix.py The discussion about matrix indexing has been interminible and for the most part pretty pointless IMO. However, it does point out one thing: the interaction between the matrix and array classes is still pretty klunky despite a fair amount of effort trying to make them interoperate. Here's one possible alternative approach to enabling matrix operations within the context of numpy arrays. I started with a few requirements: 1. Matrices should be proper subclasses of arrays in the sense that one should be able to use matrices wherever one uses arrays with no change in the result. 2. Indexing into matrices should produce row or column vectors where appropriate. 3. There should be some syntax for matrix multiplication. I know that there are other operations defined on the matrix class, but I don't consider them worth the headache of having two class hierarchies. This is in part inspired by some comments Bill Spotz, but after listening to his later comments, I don't think that this is really what he had in mind. So, please don't blame him for any deficiencies you find herein. The usual caveats apply: this is just a sketch, it's probably buggy, use at your own risk, etc. However, you should be able to save this mail as "matrix.py" and execute it to run all of the examples below. I enforce point #1 by not overriding any ndarray methods except for __array_finalize__ and __getitem__ and they are overridden in a way that shouldn't effect code unless it is specifically testing for types. With that summarily taken care of, let's move onto point #2: indexing. First let's create a matrix. Scratch that, let's create an array of matrices. One things that this approach does is let you create arrays of matrices and vectors in a natural way. The matrix below has a shape of (3,2,2), which correspons to an array of 3 2x2 matrices. >>> m = matrix([ [[1, 0],[0,1]], [[2, 0],[0,2]], [[3, 0],[0,3]] ]) If we index on the first axis, we are selecting a single matrix, so we would expect an ndmatrix back. >>> m[0] ndmatrix([[1, 0], [0, 1]]) On the other hand, if we index along the second axis, we are indexing on the matrices columns and thus expect to get an array of row matrices. >>> m[:,0] ndrow([[1, 0], [2, 0], [3, 0]]) Finally if we index on the last index, we get an array of column matrices. >>> m[...,0] ndcolumn([[1, 0], [2, 0], [3, 0]]) Indexing of row and column matrices works similarly. It's not shown here and thus will probably turn out to be buggy. For matrix multiplication, I chose to abuse call. Perhaps someday we'll be able to convinve the powers that be to free up another operator, but for the time being this is better than "*" since that forces matrix and array into separate camps and arguably more readable than most of the new syntax proposals that I've seen. Here we multiply our set of three matrices 'm', with a single 2x2 matrix 'n': >>> n = matrix([[0, 1], [3, 0]]) >>> (m)(n) ndmatrix([[[0, 1], [3, 0]], [[0, 2], [6, 0]], [[0, 3], [9, 0]]]) Things behave similarly when multiplying a matrix with a row vector or a row vector with a column vector. >>> c = column([3, 5]) >>> (m)(c) ndcolumn([[ 3, 5], [ 6, 10], [ 9, 15]]) >>> (n)(c) ndcolumn([5, 9]) >>> r = row([[2, 1], [3, 4]]) >>> (r)(c) array([11, 29]) >>> (r[0])(c) 11 Note, however that you can't (for instance) multiply column vector with a row vector: >>> (c)(r) Traceback (most recent call last): ... TypeError: Cannot matrix multiply columns with anything Finally, you'd like to be able to transpose a matrix or vector. Transposing a vector swaps the type from ndrow and ndcolumn while transposing a matrix transposes the last two axes of the array it is embedded in. Note that I use ".t" for this instead of ".T" to avoid interfering with the existing ".T". (Frankly, if it were up to me, I'd just deprecate .T). >>> r ndrow([[2, 1], [3, 4]]) >>> r.t ndcolumn([[2, 1], [3, 4]]) >>> n.t ndmatrix([[0, 3], [1, 0]]) That's about it. Enjoy (or not) as you will. """ from numpy import * _ULT = 1 # Utlimate AKA last _PEN = 2 # Penultimate AKA second to last # I'm sure there's a better way to do this and this is probably buggy... def _reduce(key, ndim): if isinstance(key, tuple): m = len(key) if m == 0: return 0 elif m == 1: key = key[0] else: if Ellipsis in key: key = (None,)*(ndim-m) + key if len(key) == ndim: return isinstance(key[-1], int) * _ULT + isinstance(key[-2], int) * _PEN if len(key) == ndim-1: return isinstance(key[-1], int) * _PEN if isinstance(key, int): if ndim == 1: return _ULT if ndim == 2: return _PEN return 0 class ndmatrix(ndarray): def __array_finalize__(self, obj): if obj.ndim < 2: raise ValueError("matrices must have dimension of at least 2") def __getitem__(self, key): rmask = _reduce(key, self.ndim) value = ndarray.__getitem__(self, key) if isinstance(value, ndmatrix): if rmask == _ULT | _PEN: return value.view(ndarray) if rmask == _ULT: return value.view(ndcolumn) if rmask == _PEN: return value.view(ndrow) return value def __call__(self, other): if isinstance(other, ndcolumn): return sum(self * other[...,newaxis,:], -1).view(ndcolumn) if isinstance(other, ndmatrix): return sum(self[...,newaxis] * other[...,newaxis,:], -2).view(ndmatrix) else: raise TypeError("Can only matrix multiply matrices by matrices or vectors") def transpose_vector(self): return self.swapaxes(-1,-2) t = property(transpose_vector) class ndcolumn(ndarray): def __array_finalize__(self, obj): if obj.ndim < 1: raise ValueError("vectors must have dimension of at least 1") def __getitem__(self, key): rmask = _reduce(key, self.ndim) value = ndarray.__getitem__(self, key) if isinstance(value, ndmatrix): if rmask == _ULT: return value.view(ndarray) return value def __call__(self, other): raise TypeError("Cannot matrix multiply columns with anything") def transpose_vector(self): return self.view(ndrow) t = property(transpose_vector) class ndrow(ndarray): def __array_finalize__(self, obj): if obj.ndim < 1: raise ValueError("vectors must have dimension of at least 1") def __getitem__(self, key): rmask = _reduce(key, self.ndim) value = ndarray.__getitem__(self, key) if isinstance(value, ndmatrix): if rmask == _ULT: return value.view(ndarray) return value def __call__(self, other): if isinstance(other, ndcolumn): return sum(self * other, -1).view(ndarray) if isinstance(othe, ndmatrix): raise notimplemented else: raise TypeError("Can only matrix multiply rows with matrices or columns") def transpose_vector(self): return self.view(ndcolumn) t = property(transpose_vector) def matrix(*args, **kwargs): m = array(*args, **kwargs) return m.view(ndmatrix) def column(*args, **kwargs): v = array(*args, **kwargs) return v.view(ndcolumn) def row(*args, **kwargs): v = array(*args, **kwargs) return v.view(ndrow) if __name__ == "__main__": import doctest, matrix doctest.testmod(matrix) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Mar 30 00:10:36 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 30 Mar 2007 13:10:36 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: On 3/30/07, Timothy Hochberg wrote: > Note, however that you can't (for instance) multiply column vector with > a row vector: > > >>> (c)(r) > Traceback (most recent call last): > ... > TypeError: Cannot matrix multiply columns with anything > That should be allowed. (N,1)*(1,M) is just an (N,M) matrix with entries C[i,j] = A[i,0]*B[0,] I kind of like the idea of using call for multiply, though. If it doesn't turn out to have any major down sides it could be a good way to give ndarray a concise syntax for "dot". --bb From tim.hochberg at ieee.org Fri Mar 30 00:19:35 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 29 Mar 2007 21:19:35 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: On 3/29/07, Bill Baxter wrote: > > On 3/30/07, Timothy Hochberg wrote: > > Note, however that you can't (for instance) multiply column vector with > > a row vector: > > > > >>> (c)(r) > > Traceback (most recent call last): > > ... > > TypeError: Cannot matrix multiply columns with anything > > > > That should be allowed. (N,1)*(1,M) is just an (N,M) matrix with > entries C[i,j] = A[i,0]*B[0,] I thought about that a little, and while I agree that it could be allowed, I'm not sure that it should be allowed. It's a trade off between a bit of what I would guess is little used functionality with some enhanced error checking (I would guess that usually row*column signals a mistake). However, I don't care much one way or the other; it's not hard to allow. I kind of like the idea of using call for multiply, though. If it > doesn't turn out to have any major down sides it could be a good way > to give ndarray a concise syntax for "dot". We'll see how it goes down this time. I've proposed using call before, since I've thought the matrix situation was kind of silly for what seems like ages now, but it always sinks without a ripple. -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at ieee.org Fri Mar 30 01:10:50 2007 From: tim.hochberg at ieee.org (Timothy Hochberg) Date: Thu, 29 Mar 2007 22:10:50 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: On 3/29/07, Timothy Hochberg wrote: > > > > On 3/29/07, Bill Baxter wrote: > > > > On 3/30/07, Timothy Hochberg wrote: > > > Note, however that you can't (for instance) multiply column vector > > with > > > a row vector: > > > > > > >>> (c)(r) > > > Traceback (most recent call last): > > > ... > > > TypeError: Cannot matrix multiply columns with anything > > > > > > > That should be allowed. (N,1)*(1,M) is just an (N,M) matrix with > > entries C[i,j] = A[i,0]*B[0,] > > > I thought about that a little, and while I agree that it could be allowed, > I'm not sure that it should be allowed. It's a trade off between a bit of > what I would guess is little used functionality with some enhanced error > checking (I would guess that usually row*column > Make that column*row. Bah. I'm off to bed. signals a mistake). However, I don't care much one way or the other; it's > not hard to allow. > > I kind of like the idea of using call for multiply, though. If it > > doesn't turn out to have any major down sides it could be a good way > > to give ndarray a concise syntax for "dot". > > > We'll see how it goes down this time. I've proposed using call before, > since I've thought the matrix situation was kind of silly for what seems > like ages now, but it always sinks without a ripple. > > > -- > > //=][=\\ > > tim.hochberg at ieee.org -- //=][=\\ tim.hochberg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Fri Mar 30 01:21:07 2007 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu, 29 Mar 2007 23:21:07 -0600 Subject: [Numpy-discussion] I've just commited a fast-clip function Message-ID: <460C9E43.1070002@ieee.org> Hey folks, I've just committed a revision of ticket #425 to speed up clipping in the scalar case. I also altered the PyArray_Conjugate function (called by the conjugate method) to use the ufunc for complex data. These were some relatively largish changes to the source code (all behind the scences and no interface changes) --- enough to make me want to see some more testing. I would appreciate it, if people could test out the new clip function and conjugate method to make sure they are working well. All tests pass, but there are some things we are not testing for. I need to still add the clip tests from ticket #425 --- unless somebody beats me to it. We need to test the output argument of both of those functions as well as test for unaligned, byteswapped, etc. inputs. Thanks much, -Travis From charlesr.harris at gmail.com Fri Mar 30 01:16:26 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Mar 2007 23:16:26 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: On 3/29/07, Timothy Hochberg wrote: > > > > On 3/29/07, Bill Baxter wrote: > > > > On 3/30/07, Timothy Hochberg wrote: > > > Note, however that you can't (for instance) multiply column vector > > with > > > a row vector: > > > > > > >>> (c)(r) > > > Traceback (most recent call last): > > > ... > > > TypeError: Cannot matrix multiply columns with anything > > > > > > > That should be allowed. (N,1)*(1,M) is just an (N,M) matrix with > > entries C[i,j] = A[i,0]*B[0,] > > > I thought about that a little, and while I agree that it could be allowed, > I'm not sure that it should be allowed. It's a trade off between a bit of > what I would guess is little used functionality with some enhanced error > checking (I would guess that usually row*column signals a mistake). However, > I don't care much one way or the other; it's not hard to allow. > It's really a sort of tensor product, so use outer(.,.). In my mind, row and column vectors are *not* matrices, they only have a single dimension. On the other hand (r)(c) is really the application of the dual vector r (a functional) to the vector c, i.e., r is a map from vectors into the reals (complex). However, I think overloading the multiplication in this case is reasonable. I kind of like the idea of using call for multiply, though. If it > > doesn't turn out to have any major down sides it could be a good way > > to give ndarray a concise syntax for "dot". > > Hmmm, have to try it a bit to see how it looks. Overall, I like this approach. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From zpincus at stanford.edu Fri Mar 30 01:31:49 2007 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu, 29 Mar 2007 22:31:49 -0700 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: Looks promising! > On 3/29/07, Bill Baxter wrote: On 3/30/07, > Timothy Hochberg wrote: > > Note, however that you can't (for instance) multiply column > vector with > > a row vector: > > > > >>> (c)(r) > > Traceback (most recent call last): > > ... > > TypeError: Cannot matrix multiply columns with anything > > > > That should be allowed. (N,1)*(1,M) is just an (N,M) matrix with > entries C[i,j] = A[i,0]*B[0,] > > I thought about that a little, and while I agree that it could be > allowed, I'm not sure that it should be allowed. It's a trade off > between a bit of what I would guess is little used functionality > with some enhanced error checking (I would guess that usually > row*column signals a mistake). However, I don't care much one way > or the other; it's not hard to allow. I'm also for allowing it; from the perspective of one writing code that "looks like" typical (if potentially technically incorrect) notation, being able to write direct analogs to a'b (to get a scalar) or ab' (to get an MxN matrix) and have everything "work" would be quite useful. Especially in various reconstruction tasks (e.g. using svd to approximate a given matrix with a lower-rank one), the "outer product" of a row and a column vector comes up often enough that I would be loathe to have it raise an error. > I kind of like the idea of using call for multiply, though. If it > doesn't turn out to have any major down sides it could be a good way > to give ndarray a concise syntax for "dot". > > We'll see how it goes down this time. I've proposed using call > before, since I've thought the matrix situation was kind of silly > for what seems like ages now, but it always sinks without a ripple. The call syntax is nice, if a bit opaque to someone looking at the code for the first time. On the other hand, it's explicit in a way that overloaded multiplication just isn't. I like it (for what little that's worth). Zach > > > -- > > //=][=\\ > > tim.hochberg at ieee.org > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From wbaxter at gmail.com Fri Mar 30 01:46:32 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Fri, 30 Mar 2007 14:46:32 +0900 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: On 3/30/07, Timothy Hochberg wrote: > > > On 3/29/07, Bill Baxter wrote: > > On 3/30/07, Timothy Hochberg wrote: > > > Note, however that you can't (for instance) multiply column vector with > > > a row vector: > > > > > > >>> (c)(r) > > > Traceback (most recent call last): > > > ... > > > TypeError: Cannot matrix multiply columns with anything > > > > > > > That should be allowed. (N,1)*(1,M) is just an (N,M) matrix with > > entries C[i,j] = A[i,0]*B[0,] > > I thought about that a little, and while I agree that it could be allowed, > I'm not sure that it should be allowed. It's a trade off between a bit of > what I would guess is little used functionality with some enhanced error > checking (I would guess that usually row*column signals a mistake). However, > I don't care much one way or the other; it's not hard to allow. > It's useful for many things. You can use it in the computation of the least squares best fits between sets of points. The sum of p_i p_^t comes up in that context. (Which is also the covariance matrix of the points) The derivative of a unit vector (useful for mass spring dynamics calcs, among other things, I'm sure) is given by something like c * (I - x x^t). Householder reflections are useful for a lot of things such as implementing QR factorization. They're given by : H = I - 2 x x^t / x^t x http://www.cs.ut.ee/~toomas_l/linalg/lin2/node6.html I'm positive I've seen col * row come up in other contexts too, though I can't think of any other particulars right now. --bb From david at ar.media.kyoto-u.ac.jp Fri Mar 30 02:58:16 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 30 Mar 2007 15:58:16 +0900 Subject: [Numpy-discussion] I've just commited a fast-clip function In-Reply-To: <460C9E43.1070002@ieee.org> References: <460C9E43.1070002@ieee.org> Message-ID: <460CB508.8090308@ar.media.kyoto-u.ac.jp> Travis Oliphant wrote: > Hey folks, > > I've just committed a revision of ticket #425 to speed up clipping in > the scalar case. I also altered the PyArray_Conjugate function (called > by the conjugate method) to use the ufunc for complex data. > > These were some relatively largish changes to the source code (all > behind the scences and no interface changes) --- enough to make me want > to see some more testing. > > I would appreciate it, if people could test out the new clip function > and conjugate method to make sure they are working well. All tests > pass, but there are some things we are not testing for. I need to still > add the clip tests from ticket #425 --- unless somebody beats me to it. Hi Travis, Would the test I included in the patch be OK, once converted to be usable after your modifications ? They were covering many corner cases (including some which crashed the old clip). cheers, David From giorgio.luciano at chimica.unige.it Fri Mar 30 03:14:37 2007 From: giorgio.luciano at chimica.unige.it (Giorgio Luciano) Date: Fri, 30 Mar 2007 09:14:37 +0200 Subject: [Numpy-discussion] odd installation problem of numpy/matplotlib Message-ID: <460CB8DD.2000605@chimica.unige.it> Sorry if the message will arrive in duplicate I had some problem with posting in the mailing list I've installed in my machine in the following order python 2.5 numpy 1.01 matplot lib 0.87 scipy 0.52 wxPython 2.8 with no problem I've also installed the same packages at home and in another two computer and everything went fine. The I was asked to install this "configuaration" in some classroom machines and also on another computer and I continue getting this error The import of the numpy version of the _transforms module, _ns_transforms, failed. This is is either because numpy was unavailable when matplotlib was compiled, because a dependency of _ns_transforms could not be satisfied, or because the build flag for this module was turned off in setup.py. If it appears that _ns_transforms was not built, make sure you have a working copy of numpy and then re-install matplotlib. Otherwise, the following traceback gives more details: Traceback (most recent call last): File "", line 1, in from pylab import * File "C:\Python25\Lib\site-packages\pylab.py", line 1, in from matplotlib.pylab import * File "C:\Python25\Lib\site-packages\matplotlib\pylab.py", line 201, in from axes import Axes, PolarAxes File "C:\Python25\Lib\site-packages\matplotlib\axes.py", line 14, in from artist import Artist, setp File "C:\Python25\Lib\site-packages\matplotlib\artist.py", line 4, in from transforms import identity_transform File "C:\Python25\Lib\site-packages\matplotlib\transforms.py", line 223, in from _transforms import Value, Point, Interval, Bbox, Affine File "C:\Python25\Lib\site-packages\matplotlib\_transforms.py", line 17, in from matplotlib._ns_transforms import * ImportError: DLL load failed: Impossibile trovare il modulo specificato but I can assure that If I check numpy installation before installing matplot lib it seems everything fine. All computer have Windows XP home edition 2002 SP2 the only difference is in the RAM quantity. (more than 256 in the computer where everything works) but it seems so strange to me that it is the ram (I've also installed in another computer , old one, and everything works) Any IDEA ???? From nwagner at iam.uni-stuttgart.de Fri Mar 30 05:20:30 2007 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Fri, 30 Mar 2007 11:20:30 +0200 Subject: [Numpy-discussion] Ticket 418 Message-ID: <460CD65E.50704@iam.uni-stuttgart.de> Hi all, Is someone able to reproduce the segfault described at http://projects.scipy.org/scipy/numpy/ticket/418 with a recent svn version ? I am using >>> numpy.__version__ '1.0.2.dev3616' >>> scipy.__version__ '0.5.3.dev2892' Nils From charlesr.harris at gmail.com Fri Mar 30 06:25:44 2007 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 30 Mar 2007 04:25:44 -0600 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <7094E618-059F-4116-B3AA-5BF0582BD4A4@stanford.edu> <460958CC.4030806@noaa.gov> <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: On 3/29/07, Timothy Hochberg wrote: > > """matrix.py > > The discussion about matrix indexing has been interminible and for > the most part pretty pointless IMO. However, it does point out one > thing: the interaction between the matrix and array classes is still > pretty klunky despite a fair amount of effort trying to make them > interoperate. > > Here's one possible alternative approach to enabling matrix operations > within the context of numpy arrays. I started with a few requirements: > 1. Matrices should be proper subclasses of arrays in the sense > that one should be able to use matrices wherever one uses > arrays with no change in the result. > 2. Indexing into matrices should produce row or column vectors > where appropriate. > 3. There should be some syntax for matrix multiplication. I > know that there are other operations defined on the matrix > class, but I don't consider them worth the headache of > having two class hierarchies. > > Note, however that you can't (for instance) multiply column vector with > a row vector: Well, let's carry this to extremes. Strictly speaking, from the functional point of view, (r)(c) == (c)(r), the dual of the dual is the original vector space, for finite dimensional spaces anyway. However, custom reserves the latter form for the tensor product, and so that is probably a good thing to keep. However, to really make the transpose operate correctly it should also conjugate. This could just be a flag in the row vector, no changes in the data needed, but vdot would be called instead of dot when computing (c.t)(c). A similar thing could be done for matrices if vdot were extended to deal with matrices -- currently it only works for scalars and vectors. Now, products like (c.t)(M.t) would need to conjugate/transpose both, but this could be computed as ((M)(c)).t. So on and so forth. A lot of these types of operations are in BLAS, IIRC, so in some sense this is just a mapping to BLAS. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gary.pajer at gmail.com Fri Mar 30 06:54:15 2007 From: gary.pajer at gmail.com (Gary Pajer) Date: Fri, 30 Mar 2007 06:54:15 -0400 Subject: [Numpy-discussion] Ticket 418 In-Reply-To: <460CD65E.50704@iam.uni-stuttgart.de> References: <460CD65E.50704@iam.uni-stuttgart.de> Message-ID: <88fe22a0703300354j125b5822racfeab027b713254@mail.gmail.com> On 3/30/07, Nils Wagner wrote: > Hi all, > > Is someone able to reproduce the segfault described at > > http://projects.scipy.org/scipy/numpy/ticket/418 > > with a recent svn version ? > > I am using > >>> numpy.__version__ > '1.0.2.dev3616' > >>> scipy.__version__ > '0.5.3.dev2892' My version is not shiny-new, bu gap at gap-laptop:~/downloads$ python sina3.py Optimization terminated successfully. Current function value: 1.761814 Iterations: 8 Function evaluations: 42 Gradient evaluations: 14 Segmentation fault In [3]: scipy.__version__ Out[3]: '0.5.3.dev2844' In [4]: numpy.__version__ Out[4]: '1.0.2.dev3573' > > > Nils > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > From mandus at gmail.com Fri Mar 30 06:59:20 2007 From: mandus at gmail.com (Mandus) Date: Fri, 30 Mar 2007 12:59:20 +0200 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: <6ce0ac130703291610o4aa107f4u263495cad3e44086@mail.gmail.com> References: <6fdd76590703290152q130c2946n62aee64263ed9d5c@mail.gmail.com> <460BACFD.1030503@shrogers.com> <6ce0ac130703291610o4aa107f4u263495cad3e44086@mail.gmail.com> Message-ID: <6fdd76590703300359v3942d9fr84d1f4e6b9149ebc@mail.gmail.com> We have also done some work with BSP in the past (actually together with Konrad). It's a great model, and quite comfortable to work with. Also, with Konrads implementations it was very efficient at sending Numeric arrays around. But the main problem with BSP is that it is not very much used in the community, as Brian points out. This mean that if you get access to some parallel computer, it is highly unlikely that BSP is available to you. We also had a hard time building the stuff on our own computers, as the libraries was already quite old back when we did this stuff. About the BSP model itself, as Brian points out, its usablility may depend on the kind of problems you need to solve. If you need to solve PDEs (which is what we did with it) and these PDEs gives sparse linear systems using for instance finite element mehthods and linearization - then BSP can be quite efficient. For some problems, we actually found that python & BSP performed on par with C & MPI, both in terms of total runtime and parallel efficiency. No doubt, that's not a general conclusion, but it certainly shows the possibilites. On 3/30/07, Brian Granger wrote: > > We looked at the BSP model at various points in implementing the > parallel IPython stuff. While I wouldn't say that IPython uses a BSP > model, there are some similarities. But in the broader realm of > scientific computing, BSP has never really caught on like MPI has - in > spite of having some nice ideas like being able to predict the > performance of a parallel BSP code. > > The main difficulty is that BSP is a much more limited model that MPI > - which is why you can predict the performance of BSP using codes. > The main limitation is that communication and computation cannot > overlap. For some codes that is fine and there might be benefits for > using BSP over MPI. For general parallel codes and algorithms > however, you do want to overlap communications and computation. In > fact, if you don't you can severely limit the scalability of your > code. > > Brian > > > > > > On 3/29/07, Sebastian Haase wrote: > > Hi, > > What is the general feeling towards BSP on this list !? > > I remeber Konrad Hinsen advertising it on the SciPy workshop '03 . > > It is supposed to be much simpler to use than MPI, yet still powerful > > and flexible enough for most all applications. > > It is part of Konrad's ScientificPython ( != SciPy ) > > > > Some links are here: > > http://www.bsp-worldwide.org/ > > http://en.wikipedia.org/wiki/Bulk_Synchronous_Parallel > > > > Evaluating Scientific Python/BSP on selected parallel computers > > http://ove.nipen.no/diplom/ > > > > http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ > > > > - Sebastian Haase > > > > > > > > On 3/29/07, Peter Skomoroch wrote: > > > > > > > > > If you want to use PyMPI or PyPar, I'm writing a series of tutorials > on how to > > > get them running on Amazon EC2, > > > > > > > http://www.datawrangling.com/on-demand-mpi-cluster-with-python-and-ec2-part-1-of-3.html > > > > > > > > > I'm using PyMPI on a 20 node EC2 cluster and everything seems groovy, > but I'm > > > relatively new to MPI, so I have probably overlooked some easier > solutions. > > > > > > Any feedback on the writeups from Python gurus would be appreciated. > > > > > > -Pete > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- Mandus The only dr. Mandus around. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Fri Mar 30 09:02:48 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 30 Mar 2007 15:02:48 +0200 Subject: [Numpy-discussion] I've just commited a fast-clip function In-Reply-To: <460C9E43.1070002@ieee.org> References: <460C9E43.1070002@ieee.org> Message-ID: <20070330130248.GS18196@mentat.za.net> Hi Travis On Thu, Mar 29, 2007 at 11:21:07PM -0600, Travis Oliphant wrote: > I've just committed a revision of ticket #425 to speed up clipping in > the scalar case. I also altered the PyArray_Conjugate function (called > by the conjugate method) to use the ufunc for complex data. > > These were some relatively largish changes to the source code (all > behind the scences and no interface changes) --- enough to make me want > to see some more testing. > > I would appreciate it, if people could test out the new clip function > and conjugate method to make sure they are working well. All tests > pass, but there are some things we are not testing for. I need to still > add the clip tests from ticket #425 --- unless somebody beats me to > it. While writing some tests, I saw that the comparitor operators are now less than happy: import numpy as N x = (N.random.random(2000)*1024).astype(N.float32) x.clip(0.,0.,x) N.all(x <= 0.) N.all(x >= 0.) causes errors like these: ==26879== Invalid read of size 4 ==26879== at 0x46219C3: PyArray_IterNew (arrayobject.c:8791) ==26879== by 0x467B7B6: construct_loop (ufuncobject.c:1222) ==26879== by 0x467C872: PyUFunc_GenericFunction (ufuncobject.c:1751) ==26879== by 0x467D9AA: ufunc_generic_call (ufuncobject.c:3064) ==26879== by 0x805B22C: (within /usr/bin/python2.5) ==26879== by 0x805E7DE: _PyObject_CallFunction_SizeT (in /usr/bin/python2.5) ==26879== by 0x461F49F: PyArray_GenericBinaryFunction (arrayobject.c:3362) ==26879== by 0x464F2B8: array_richcompare (arrayobject.c:4639) ==26879== by 0x808634F: (within /usr/bin/python2.5) ==26879== by 0x8087E64: PyObject_RichCompare (in /usr/bin/python2.5) ==26879== Invalid read of size 4 ==26879== at 0x463B0D2: PyArray_FromArray (arrayobject.c:7947) ==26879== by 0x4631345: PyArray_FromAny (arrayobject.c:8368) ==26879== by 0x467B302: construct_loop (ufuncobject.c:1147) ==26879== by 0x467C872: PyUFunc_GenericFunction (ufuncobject.c:1751) ==26879== by 0x467D9AA: ufunc_generic_call (ufuncobject.c:3064) ==26879== by 0x805B22C: (within /usr/bin/python2.5) ==26879== by 0x805E7DE: _PyObject_CallFunction_SizeT (in /usr/bin/python2.5) ==26879== by 0x461F49F: PyArray_GenericBinaryFunction (arrayobject.c:3362) ==26879== by 0x464F2B8: array_richcompare (arrayobject.c:4639) ==26879== by 0x808634F: (within /usr/bin/python2.5) Cheers St?fan From stefan at sun.ac.za Fri Mar 30 09:11:38 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 30 Mar 2007 15:11:38 +0200 Subject: [Numpy-discussion] I've just commited a fast-clip function In-Reply-To: <460C9E43.1070002@ieee.org> References: <460C9E43.1070002@ieee.org> Message-ID: <20070330131138.GT18196@mentat.za.net> Hi Travis, On Thu, Mar 29, 2007 at 11:21:07PM -0600, Travis Oliphant wrote: > I would appreciate it, if people could test out the new clip function > and conjugate method to make sure they are working well. All tests > pass, but there are some things we are not testing for. I need to still > add the clip tests from ticket #425 --- unless somebody beats me to > it. Passing negative values to clip for an N.uint8 array causes problems: In [95]: z.clip(0,100).max() Out[95]: 100 In [96]: z = (N.random.random(1000)*128).astype(N.uint8) In [97]: z.clip(0,100).max() Out[97]: 100 In [98]: z = (N.random.random(1000)*128).astype(N.uint8) In [99]: z.clip(-10,100).max() Out[99]: 246 Cheers St?fan From chanley at stsci.edu Fri Mar 30 10:09:05 2007 From: chanley at stsci.edu (Christopher Hanley) Date: Fri, 30 Mar 2007 10:09:05 -0400 Subject: [Numpy-discussion] Should 0-d arrays with fields defined return a 0-d array or a scalar In-Reply-To: <460C1F14.7090506@stsci.edu> References: <460C1197.5030404@ieee.org> <460C1F14.7090506@stsci.edu> Message-ID: <460D1A01.40303@stsci.edu> Hi Travis, This change should not have any impact on our code. We are not opposed to making the change as part of the 1.0.2 release. Chris Christopher Hanley wrote: > Hi Travis, > > We will need a little time to inspect our code to see if this is going > to be a problem for us. We can't think of any major problems with this > change off the top of our heads. > > Chris > > From stefan at sun.ac.za Fri Mar 30 11:22:33 2007 From: stefan at sun.ac.za (Stefan van der Walt) Date: Fri, 30 Mar 2007 17:22:33 +0200 Subject: [Numpy-discussion] I've just commited a fast-clip function In-Reply-To: <460C9E43.1070002@ieee.org> References: <460C9E43.1070002@ieee.org> Message-ID: <20070330152233.GU18196@mentat.za.net> On Thu, Mar 29, 2007 at 11:21:07PM -0600, Travis Oliphant wrote: > I would appreciate it, if people could test out the new clip function > and conjugate method to make sure they are working well. All tests > pass, but there are some things we are not testing for. I need to still > add the clip tests from ticket #425 --- unless somebody beats me to it. Record arrays also cause problems, i.e. import numpy as N rec = N.array([(1.0, 2.0, 3.0), (5.0, 4.0, 3.0)], dtype=[('x', ' References: <460CD65E.50704@iam.uni-stuttgart.de> <88fe22a0703300354j125b5822racfeab027b713254@mail.gmail.com> Message-ID: On 3/30/07, Gary Pajer wrote: > On 3/30/07, Nils Wagner wrote: > > Hi all, > > > > Is someone able to reproduce the segfault described at > > > > http://projects.scipy.org/scipy/numpy/ticket/418 > > > > with a recent svn version ? > > > > I am using > > >>> numpy.__version__ > > '1.0.2.dev3616' > > >>> scipy.__version__ > > '0.5.3.dev2892' > > My version is not shiny-new, bu > > gap at gap-laptop:~/downloads$ python sina3.py > Optimization terminated successfully. > Current function value: 1.761814 > Iterations: 8 > Function evaluations: 42 > Gradient evaluations: 14 > Segmentation fault > > > In [3]: scipy.__version__ > Out[3]: '0.5.3.dev2844' > > In [4]: numpy.__version__ > Out[4]: '1.0.2.dev3573' I get the exact same crash as Gary, with: In [3]: N.__version__ Out[3]: '1.0.2.dev3607' In [5]: S.__version__ Out[5]: '0.5.3.dev2878' I'm running a 32-bit Ubuntu Edgy install on a Pentium 4. Cheers, f From markbak at gmail.com Fri Mar 30 08:56:13 2007 From: markbak at gmail.com (Mark Bakker) Date: Fri, 30 Mar 2007 14:56:13 +0200 Subject: [Numpy-discussion] converting scalar to array with dimension 1 Message-ID: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> Hello list - I have a function that normally accepts an array as input, but sometimes a scalar. I figured the easiest way to make sure the input is an array, is to make it an array. But if I make a float an array, it has 0 dimension, and I can still not do array manipulation on it. >>> a = 3 >>> a = array(a) >>> shape(a) () >>> a[0] Traceback (most recent call last): File "", line 1, in ? a[0] IndexError: 0-d arrays can't be indexed What would be the best (and easiest, this is for an intro class I am teaching) way to convert a to an array (recall, most of the time a is already an array). Thanks for your help, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From markbak at gmail.com Fri Mar 30 10:41:22 2007 From: markbak at gmail.com (mark) Date: Fri, 30 Mar 2007 14:41:22 -0000 Subject: [Numpy-discussion] Should 0-d arrays with fields defined return a 0-d array or a scalar In-Reply-To: <460C1197.5030404@ieee.org> References: <460C1197.5030404@ieee.org> Message-ID: <1175265682.092682.194960@e65g2000hsc.googlegroups.com> Does this mean, we could do something like this? a = 3 a = array(a) a[ a<4 ] = 5 If so, that would be great! Mark On Mar 29, 9:20 pm, Travis Oliphant wrote: > Hi all, > > Ticket #474 discusses the problem that getting a field from a 0-d array > automatically produces a scalar (which then cannot be set). > This produces the problem that recarrays code must often special-case > the 0-d possibility. > > Thus, > > rarr.x[...] = blah > > doesn't work for 0-d arrays because rarr.x is a scalar. > > It makes some sense to make field selection for 0-d arrays return 0-d > arrays as consistent with the changes that were made prior to the 1.0 > release to allow persistence of 0-d arrays. > > However, changing field selection to return 0-d arrays does change > behavior. A 0-d array is not a scalar (the 0-d array is not hashable > for example, and the 0-d string array does not inherit from the Python > string). Thus, just making the change, may not be advised. > > It is easy to account for and fix any errors that might arise. But, we > are in a major release, I need some advice as to whether or not this is > a "bug-fix" or a feature enhancement that must wait for 1.1? > > Any stake holders in the current behavior of arrays with records? > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion From markbak at gmail.com Fri Mar 30 10:45:29 2007 From: markbak at gmail.com (mark) Date: Fri, 30 Mar 2007 14:45:29 -0000 Subject: [Numpy-discussion] isarray in numpy? Message-ID: <1175265929.047308.325880@o5g2000hsb.googlegroups.com> Is there a way to check whether something is an array? It seems that isarray(a) is not there. Thanks and sorry for the newbie question, Mark From oliphant at ee.byu.edu Fri Mar 30 13:22:52 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 30 Mar 2007 10:22:52 -0700 Subject: [Numpy-discussion] isarray in numpy? In-Reply-To: <1175265929.047308.325880@o5g2000hsb.googlegroups.com> References: <1175265929.047308.325880@o5g2000hsb.googlegroups.com> Message-ID: <460D476C.6080102@ee.byu.edu> mark wrote: >Is there a way to check whether something is an array? >It seems that > > isinstance(a, numpy.ndarray) This will return True if a is an array or a sub-class. -Travis From oliphant at ee.byu.edu Fri Mar 30 13:26:08 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 30 Mar 2007 10:26:08 -0700 Subject: [Numpy-discussion] Should 0-d arrays with fields defined return a 0-d array or a scalar In-Reply-To: <1175265682.092682.194960@e65g2000hsc.googlegroups.com> References: <460C1197.5030404@ieee.org> <1175265682.092682.194960@e65g2000hsc.googlegroups.com> Message-ID: <460D4830.9040001@ee.byu.edu> mark wrote: >Does this mean, we could do something like this? > >a = 3 >a = array(a) >a[ a<4 ] = 5 > > No. That would be a separate change. -Travis From efiring at hawaii.edu Fri Mar 30 13:33:57 2007 From: efiring at hawaii.edu (Eric Firing) Date: Fri, 30 Mar 2007 07:33:57 -1000 Subject: [Numpy-discussion] isarray in numpy? In-Reply-To: <460D476C.6080102@ee.byu.edu> References: <1175265929.047308.325880@o5g2000hsb.googlegroups.com> <460D476C.6080102@ee.byu.edu> Message-ID: <460D4A05.7070300@hawaii.edu> Travis Oliphant wrote: > mark wrote: > >> Is there a way to check whether something is an array? >> It seems that >> >> > isinstance(a, numpy.ndarray) > > This will return True if a is an array or a sub-class. Watch out if you use numpy.ma; or use Pierre G-M's maskedarray instead (assuming you want a masked array to be identified as an array): In [1]:import numpy as N In [2]:xx = N.ma.array([1,2,3]) In [3]:isinstance(xx, N.ndarray) Out[3]:False In [4]:import maskedarray as MA In [5]:yy = MA.array([1,2,3]) In [6]:isinstance(yy, N.ndarray) Out[6]:True Eric > > -Travis > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion From oliphant at ee.byu.edu Fri Mar 30 14:26:53 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 30 Mar 2007 11:26:53 -0700 Subject: [Numpy-discussion] I've just commited a fast-clip function In-Reply-To: <460CB508.8090308@ar.media.kyoto-u.ac.jp> References: <460C9E43.1070002@ieee.org> <460CB508.8090308@ar.media.kyoto-u.ac.jp> Message-ID: <460D566D.1070806@ee.byu.edu> David Cournapeau wrote: >Travis Oliphant wrote: > > >>Hey folks, >> >>I've just committed a revision of ticket #425 to speed up clipping in >>the scalar case. I also altered the PyArray_Conjugate function (called >>by the conjugate method) to use the ufunc for complex data. >> >>These were some relatively largish changes to the source code (all >>behind the scences and no interface changes) --- enough to make me want >>to see some more testing. >> >>I would appreciate it, if people could test out the new clip function >>and conjugate method to make sure they are working well. All tests >>pass, but there are some things we are not testing for. I need to still >>add the clip tests from ticket #425 --- unless somebody beats me to it. >> >> >Hi Travis, > > Would the test I included in the patch be OK, once converted to be >usable after your modifications ? They were covering many corner cases > > Yes, the test functions should be fine (except there is no "fastclip" function). -Travis From wbaxter at gmail.com Fri Mar 30 14:57:38 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 31 Mar 2007 03:57:38 +0900 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> Message-ID: atleast_1d will do the trick In [11]: a = 3 In [12]: a = atleast_1d(a) In [13]: shape(a) Out[13]: (1,) In [14]: a.shape # also works ;-) Out[14]: (1,) In [15]: a[0] Out[15]: 3 --bb On 3/30/07, Mark Bakker wrote: > Hello list - > > I have a function that normally accepts an array as input, but sometimes a > scalar. > I figured the easiest way to make sure the input is an array, is to make it > an array. > But if I make a float an array, it has 0 dimension, and I can still not do > array manipulation on it. > > >>> a = 3 > >>> a = array(a) > >>> shape(a) > () > >>> a[0] > Traceback (most recent call last): > File "", line 1, in ? > a[0] > IndexError: 0-d arrays can't be indexed > > What would be the best (and easiest, this is for an intro class I am > teaching) way > to convert a to an array (recall, most of the time a is already an array). > > Thanks for your help, Mark > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Fri Mar 30 15:27:50 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri, 30 Mar 2007 12:27:50 -0700 Subject: [Numpy-discussion] I've just commited a fast-clip function In-Reply-To: <20070330152233.GU18196@mentat.za.net> References: <460C9E43.1070002@ieee.org> <20070330152233.GU18196@mentat.za.net> Message-ID: <460D64B6.8050002@ee.byu.edu> Stefan van der Walt wrote: >On Thu, Mar 29, 2007 at 11:21:07PM -0600, Travis Oliphant wrote: > > > >Record arrays also cause problems, i.e. > > I think I've fixed these errors (reference counting problems), now. If we can get the tests added, then we can just run numpy.test() Thanks for your help. -Travis From aisaac at american.edu Fri Mar 30 15:29:14 2007 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 30 Mar 2007 15:29:14 -0400 Subject: [Numpy-discussion] organizational question Message-ID: Is either NumPy or SciPy substantially supported by an identifiable and actual non-profit organization? I ask because I think both fit under http://www.mellon.org/grant_programs/programs/copy_of_research item 4. Here is the announcement: http://matc.mellon.org/ Note that universities are among the nominees: http://matc.mellon.org/2007_nominations Cheers, Alan Isaac From pgmdevlist at gmail.com Fri Mar 30 15:50:38 2007 From: pgmdevlist at gmail.com (P GM) Date: Fri, 30 Mar 2007 15:50:38 -0400 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> Message-ID: <777651ce0703301250y1a969543h25bb7440d9970b2b@mail.gmail.com> Actually, there's even faster than that: a = 3 a = array(a, ndmin=1) atleast_1d is nothing but a wrapper function, that works best when used with several inputs. When using only one array as inputs, the trick above should be more appropriate. On 3/30/07, Bill Baxter wrote: > > atleast_1d will do the trick > > In [11]: a = 3 > In [12]: a = atleast_1d(a) > In [13]: shape(a) > Out[13]: (1,) > In [14]: a.shape # also works ;-) > Out[14]: (1,) > In [15]: a[0] > Out[15]: 3 > > --bb > > On 3/30/07, Mark Bakker wrote: > > Hello list - > > > > I have a function that normally accepts an array as input, but sometimes > a > > scalar. > > I figured the easiest way to make sure the input is an array, is to make > it > > an array. > > But if I make a float an array, it has 0 dimension, and I can still not > do > > array manipulation on it. > > > > >>> a = 3 > > >>> a = array(a) > > >>> shape(a) > > () > > >>> a[0] > > Traceback (most recent call last): > > File "", line 1, in ? > > a[0] > > IndexError: 0-d arrays can't be indexed > > > > What would be the best (and easiest, this is for an intro class I am > > teaching) way > > to convert a to an array (recall, most of the time a is already an > array). > > > > Thanks for your help, Mark > > > > > > > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at scipy.org > > http://projects.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wbaxter at gmail.com Fri Mar 30 16:04:04 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 31 Mar 2007 05:04:04 +0900 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: <777651ce0703301250y1a969543h25bb7440d9970b2b@mail.gmail.com> References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> <777651ce0703301250y1a969543h25bb7440d9970b2b@mail.gmail.com> Message-ID: On 3/31/07, P GM wrote: > Actually, there's even faster than that: > > a = 3 > a = array(a, ndmin=1) > > > atleast_1d is nothing but a wrapper function, that works best when used with > several inputs. When using only one array as inputs, the trick above should > be more appropriate. I think you'll want to add the copy=False arg if you go that route, or else you'll end up with something that's much slower than atleast_1d for any array that gets passed in. :-) a = array(a, copy=0,ndmin=1) Anyway, sounds like premature optimization to me. --bb From pgmdevlist at gmail.com Fri Mar 30 16:14:59 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 30 Mar 2007 16:14:59 -0400 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> <777651ce0703301250y1a969543h25bb7440d9970b2b@mail.gmail.com> Message-ID: <200703301614.59661.pgmdevlist@gmail.com> > I think you'll want to add the copy=False arg if you go that route, or > else you'll end up with something that's much slower than atleast_1d > for any array that gets passed in. :-) Yep indeed. We can also add the subok=True flag. > a = array(a, copy=0,ndmin=1) > > Anyway, sounds like premature optimization to me. Ah, prematurity depends on the context, doesn't it ? Isn't there some famous quote about two-liners ? Here, we have a function that does little more but calling array(x,subok=True, copy=False, ndmin=1) in a loop. Is skipping the loop for some very specific applications really premature ? For example, I tend to use more and more the subok=True flag instead of 'asanyarray': is there any 'official' recommendation about this ? From robert.kern at gmail.com Fri Mar 30 16:26:26 2007 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 30 Mar 2007 15:26:26 -0500 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: <200703301614.59661.pgmdevlist@gmail.com> References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> <777651ce0703301250y1a969543h25bb7440d9970b2b@mail.gmail.com> <200703301614.59661.pgmdevlist@gmail.com> Message-ID: <460D7272.70803@gmail.com> Pierre GM wrote: >Bill Baxter wrote: >> a = array(a, copy=0,ndmin=1) >> >> Anyway, sounds like premature optimization to me. > > Ah, prematurity depends on the context, doesn't it ? Isn't there some famous > quote about two-liners ? Here, we have a function that does little more but > calling array(x,subok=True, copy=False, ndmin=1) in a loop. Is skipping the > loop for some very specific applications really premature ? For example, I > tend to use more and more the subok=True flag instead of 'asanyarray': is > there any 'official' recommendation about this ? Well, Guido once mentioned a reasonable principle about API design: if your function has an argument which is almost always going to be specified as a literal (e.g. copy=False rather than copy=should_i_copy), then make separate functions for each variant. Note that the atleast_*() functions used to be not in Numeric. scipy had them in scipy_base and used them so extensively (because they are incredibly useful for writing generic and robust code) that they were added to numpy along with most of the other stuff in scipy_base. True, not every two-liner should be in the core, but very-frequently-used two-liners that state the authors intent clearer can have a good case made for them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Fri Mar 30 16:42:33 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 30 Mar 2007 16:42:33 -0400 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: <460D7272.70803@gmail.com> References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> <200703301614.59661.pgmdevlist@gmail.com> <460D7272.70803@gmail.com> Message-ID: <200703301642.33518.pgmdevlist@gmail.com> On Friday 30 March 2007 16:26:26 Robert Kern wrote: > True, not every > two-liner should be in the core, but very-frequently-used two-liners that > state the authors intent clearer can have a good case made for them. Fair enough, I'll keep that in mind. Thanks again! From wbaxter at gmail.com Fri Mar 30 17:43:42 2007 From: wbaxter at gmail.com (Bill Baxter) Date: Sat, 31 Mar 2007 06:43:42 +0900 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: <200703301614.59661.pgmdevlist@gmail.com> References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> <777651ce0703301250y1a969543h25bb7440d9970b2b@mail.gmail.com> <200703301614.59661.pgmdevlist@gmail.com> Message-ID: On 3/31/07, Pierre GM wrote: > > > I think you'll want to add the copy=False arg if you go that route, or > > else you'll end up with something that's much slower than atleast_1d > > for any array that gets passed in. :-) > > Yep indeed. We can also add the subok=True flag. > > > a = array(a, copy=0,ndmin=1) > > > > Anyway, sounds like premature optimization to me. > > Ah, prematurity depends on the context, doesn't it ? Isn't there some famous > quote about two-liners ? Here, we have a function that does little more but > calling array(x,subok=True, copy=False, ndmin=1) in a loop. Is skipping the > loop for some very specific applications really premature ? Perhaps not. But the OP was definitely not talking about some specific application. I find asarray_1d et al to be clearer to read, and easier to type than the array call with all the flags. So barring some specific performance critical situation, I go with asarray_1d. Actually I didn't realize that it had a loop in it, so thanks for pointing that out. I thought it was just and alias for array with some args. > For example, I > tend to use more and more the subok=True flag instead of 'asanyarray': is > there any 'official' recommendation about this ? Actually, I tend to use subok=False more and more. Matrix, despite being a subclass of ndarray, is too incompatible with ndarray to really mix and match most of the time. So it seems safest just to force everything to be a bog-stock ndarray. I made some convenience functions that provide the right args to array for my own use. --bb From pgmdevlist at gmail.com Fri Mar 30 18:18:29 2007 From: pgmdevlist at gmail.com (Pierre GM) Date: Fri, 30 Mar 2007 18:18:29 -0400 Subject: [Numpy-discussion] converting scalar to array with dimension 1 In-Reply-To: References: <6946b9500703300556u5edd8e61ybfc9f8e22e116cc1@mail.gmail.com> <200703301614.59661.pgmdevlist@gmail.com> Message-ID: <200703301818.29618.pgmdevlist@gmail.com> On Friday 30 March 2007 17:43:42 Bill Baxter wrote: > Actually I > didn't realize that it had a loop in it, so thanks for pointing that > out. I thought it was just and alias for array with some args. I just realized that myself, going directly in the sources: that's how I found that the ndmin argument was available in the Python interface. > Actually, I tend to use subok=False more and more. Matrix, despite > being a subclass of ndarray, is too incompatible with ndarray to > really mix and match most of the time. So it seems safest just to > force everything to be a bog-stock ndarray. I made some convenience > functions that provide the right args to array for my own use. Yes, Matrix objects are tricky beasts... I use quite regularly masked arrays and time series (as ndarray subclasses), and don't want to lose the extra information w/ subok=False. But it's all matter of personal goals, I agree. From dalcinl at gmail.com Fri Mar 30 19:51:52 2007 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 30 Mar 2007 20:51:52 -0300 Subject: [Numpy-discussion] Best way to run python parallel In-Reply-To: References: Message-ID: On 3/29/07, Brad Malone wrote: > Hi, I use python for some fairly heavy scientific computations (at least to > be running on a single processor) and would like to use it in parallel. > I've seen some stuff online about Parallel Python and mpiPy, but I don't > know much about them. Is a python-specific program needed to run python in > parallel or are the others ( e.g., mpi/poe) just more difficult to work > with? And which one would you recommend? Brad, I am the author of mpi4py. I really suggest you try this for several reasons: - As Brian Granger said, it build almost anywhere (I only use linux, but other guys use it on MAC and Win). - It is just a python module, you don't need to rebuild a python interpreter (really true unless you want to use old MPICH1 implementation). However, parallel debuggin can be really hard. I am waiting for Brian's work on IPython1. - You can communicate numpy arrays very efficiently (for moderately sized arrays, the performace is almost the same that the one in C on gigabit ethernet clusters). In the current state, you have to manually specify the MPI datatypes. This could be automated in the future, once the Travis's buffer interface gets in on core Python. Additionally, you can communicate almost any object supporting pickling, this is not so efficient, but really transparent is you want to communicate a small, nested object like a dict. - And finally, one thing that is really important to me (and in fact, what motivated my to develop mpi4py). Its API is really similar to the C++ MPI bindings, so you can read any MPI book/tutorial/code snipet, and translate it to working Python code. A final note. Just in case you need to do parallel linear algebra, and solve linear/nonlinear systems of ecuations, I've also developed petsc4py, a port to PETSc libraries. You can find mpi4py (and also petsc4py) at PyPI for download (or just try setuptools 'easy_install', this will download, build and install mpi4py for you, just be sure of having 'mpicc' MPI compiler wrappers on your PATH). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From mjanikas at esri.com Fri Mar 30 19:55:04 2007 From: mjanikas at esri.com (Mark Janikas) Date: Fri, 30 Mar 2007 16:55:04 -0700 Subject: [Numpy-discussion] Dynamic module not initialized properly Message-ID: <627102C921CD9745B070C3B10CB8199B010EC023@hardwire.esri.com> Hello all, I am having an issue importing numpy on subsequent (I.e. not on first load) attempts in our software. The majority of the code is written in C, C++ and I am a python developer and do not have direct access to a lot of it. This is a bit of a difficult question to ask all of you because I cant provide you a direct example. All I can do is point to a numpy thread that discusses the issue: http://groups.google.com/group/Numpy-discussion/browse_thread/thread/321 77a82deab05ae/d8eecaf494ba5ad5?lnk=st&q=dynamic+module+not+initialized+p roperly+numpy&rnum=1&hl=en#d8eecaf494ba5ad5 ERROR: exceptions.SystemError: dynamic module not initialized properly What is really odd about my specific issue is that if I don't change anything in the source code.... Then the error doesn't pop up. Furthermore, the error doesn't show on some attempts even after I make a change!!!! Not sure whether there is anything I can do from the scripting side (some alternative form of reload?)... or if I have to forward it along to the C developers. You have my appreciation ahead of time. Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 909-793-2853 (2563) mjanikas at esri.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Sat Mar 31 13:16:34 2007 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Sat, 31 Mar 2007 14:16:34 -0300 Subject: [Numpy-discussion] matrix indexing question In-Reply-To: References: <9A597818-A755-4190-8186-88590788CA7A@stanford.edu> <9D0CB59C-4C7F-4C76-A05E-7DCC86DFAE64@stanford.edu> <3DEF41F4-8D15-4E62-B244-4142480C27C9@sandia.gov> Message-ID: <1175361394.5694.0.camel@localhost.localdomain> > > I'm positive I've seen col * row come up in other contexts too, though > I can't think of any other particulars right now. > Add to that quasi-Newton updates for quadratic models in optimization, like the BFGS. Paulo From haase at msg.ucsf.edu Sat Mar 31 17:39:23 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 31 Mar 2007 14:39:23 -0700 Subject: [Numpy-discussion] ticket #450 Message-ID: Hi! Could someone comment on ticket #450 [Numpy-tickets] [NumPy] #450: Make a.min() not copy data It seems odd that calculating the min of an array causes a memory error. The demonstrated data array is a 3D cube of uint8 and about 1GB in size. even creation of an extra (2d) array when reducing the 1st dimension, would only need about 1MB -- that can not really explain the memory error. Thanks, Sebastian From haase at msg.ucsf.edu Sat Mar 31 17:59:40 2007 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Sat, 31 Mar 2007 14:59:40 -0700 Subject: [Numpy-discussion] comments to ticket #454 !? : Importing numpy prevents decrementing references for local variables Message-ID: Another rather old ticket is this one: http://projects.scipy.org/scipy/numpy/ticket/454> Any comments on this !? Thanks, -Sebastian On 2/16/07, NumPy wrote: > #454: Importing numpy prevents decrementing references for local variables > ------------------------+--------------------------------------------------- > Reporter: goddard | Owner: somebody > Type: defect | Status: new > Priority: normal | Milestone: > Component: numpy.core | Version: 1.0.1 > Severity: major | Keywords: > ------------------------+--------------------------------------------------- > If the first import of numpy is within a function it prevents that > function's > local variables and all locals in higher call frames that led to the > import > from being deleted. This can prevent freeing up large numpy arrays. > > The problem is that numpy __init__.py imports numpy _import_tools.py > and creates a PackageLoader object that saves the parent frame > > self.parent_frame = frame = sys._getframe(1) > > Saving the parent frame results in saving the entire call stack even after > all those functions have returned. > > I've attached Python code illustrating the problem. From oliphant at ee.byu.edu Sat Mar 31 23:54:49 2007 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sat, 31 Mar 2007 21:54:49 -0600 Subject: [Numpy-discussion] Release of NumPy 1.0.2 for Monday Message-ID: <460F2D09.40506@ee.byu.edu> I'm going to be tagging the tree for the NumPy 1.0.2 release tomorrow evening in preparation for the release on Monday. I've closed several bugs. Are there any show-stoppers remaining to be fixed? -Travis