From zpincus at stanford.edu Wed Mar 1 01:07:02 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed Mar 1 01:07:02 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? Message-ID: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> Hi folks, I'm using numpy 0.95 and came across an odd error with numpy.dot and degenerate 2D floating-point arrays. See below for an illustration. In [1]: import numpy In [2]: numpy.version.version Out[2]: '0.9.5' In [3]: numpy.dot([[5]],[[1,1,1]]) # 2D int case OK Out[3]: array([[5, 5, 5]]) In [4]: numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? Out[4]: array([[ 5.00000000e+000, 4.46601253e-316, 5.42111867e-306]]) In [5]: numpy.dot([5.],[1,1,1]) # 1D float case OK Out[5]: array([ 5., 5., 5.]) In [6]: numpy.dot([[[5.]]],[[[1,1,1]]]) # 3D float case OK Out[6]: array([[[[ 5., 5., 5.]]]]) Zach From Mithrandir42 at web.de Wed Mar 1 03:41:02 2006 From: Mithrandir42 at web.de (Niklas Volbers) Date: Wed Mar 1 03:41:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array Message-ID: <1307204529@web.de> [[Oops, I accidentally sent my post to scipy-users, but it was intended for numpy-discussion, so here's another attempt]] Hey Michael, take a look at my attempt of such a Table implementation! The newest release 0.5.2 of my plotting project http://developer.berlios.de/projects/sloppyplot contains a Table class (in Sloppy.Base.dataset) which wraps a heterogeneous numpy array. The class should be fairly self-documenting, at least I hope so. Don't get confused by the 'undolist' stuff, this is my private undo implementation which could be easily removed from the code. If you want a similar implementation using a list of 1-dimensional arrays, then download the previous release 0.5.1 (which uses Numeric). The reason I switched over to the heterogeneous approach was that it is easier to provide similar wrappers for 2-d table like data (using a 2d heterogeneous array) and for 2-d matrix like data (using a 2d homogeneous array). Using a list of arrays gives you some problems when you would like to access the rows, because then you are in charge of creating a new 1-d array that represents the row, while with the heterogeneous array you can access both the columns and the rows quite naturally. By the way, I had first planned to subclass ndarray, but I did not know how to resize the array and still keep the array as such persistent. This is why I wrapped the array into a class called 'Dataset' which you can consider constant. If you need some more help on this, feel free to ask, Niklas Volbers. Travis Oliphant schrieb am 01.03.06 08:16:15: > > Michael Sorich wrote: > > > Hi, > > > > I am looking for a table like array. Something like a 'data frame' > > object to those familiar with the statistical languages R and Splus. > > This is mainly to hold and manipulate 2D spreadsheet like data, which > > tends to be of relatively small size (compared to what many people > > seem to use numpy for), heterogenous, have column and row names, and > > often contains missing data. > > You could subclass the ndarray to produce one of these fairly easily, I > think. The missing data item could be handled by a mask stored along > with the array (or even in the array itself). Or you could use a masked > array as your core object (though I'm not sure how it handles the > arbitrary (i.e. record-like) data-types yet). > > Alternatively, and probably the easiest way to get started, you could > just create your own table-like class and use simple 1-d arrays or 1-d > masked arrays for each of the columns --- This has always been a way to > store record-like tables. > > It really depends what you want the data-frames to be able to do and > what you want them to "look-like." > > > A RecArray seems potentially useful, as it allows different fields to > > have different data types and holds the name of the field. However it > > doesn't seem easy to manipulate the data. Or perhaps I am simply > > having difficulty finding documentation on there features. > > Adding a new column/field means basically creating a new array with a > new data-type and copying data over into the already-defined fields. > Data-types always have a fixed number of bytes per item. What those > bytes represent can be quite arbitrary but it's always fixed. So, it > is always "more work" to insert a new column. You could make that > seamless in your table class so the user doesn't see it though. > > You'll want to thoroughly understand the dtype object including it's > attributes and methods. Particularly the fields attribute of the dtype > object. > > > eg > > adding a new column/field (and to a lesser extent a new row/record) to > > the recarray > > Adding a new row or record is actually similar because once an array is > created it is usually resized by creating another array and copying the > old array into it in the right places. > > > Changing the field/column names > > make a new table by selecting a subset of fields/columns. (you can > > select a single field/column, but not multiple). > > Right. So far you can't select multiple columns. It would be possible > to add this feature with a little-bit of effort if there were a strong > demand for it, but it would be much easier to do it in your subclass > and/or container class. > > How many people would like to see x['f1','f2','f5'] return a new array > with a new data-type descriptor constructed from the provided fields? > > > It would also be nice for the table to be able to deal easily with > > masked data (I have not tried this with recarray yet) and perhaps also > > to be able to give the rows/records unique ids that could be used to > > select the rows/records (in addition to the row/record index), in the > > same way that the fieldnames can select the fields. > > Adding fieldnames to the "rows" is definitely something that a subclass > would be needed for. I'm not sure how you would even propose to select > using row names. Would you also use getitem semantics? > > > Can anyone comment on this issue? Particularly whether code exists for > > this purpose, and if not ideas about how best to go about developing > > such a Table like array (this would need to be limited to python > > programing as my ability to program in c is very limited). > > I don't know of code that already exists for this, but I don't think it > would be too hard to construct your own data-frame object. > > I would probably start with an implementation that just used standard > arrays of a particular type to represent the internal columns and then > handle the indexing using your own over-riding of the __getitem__ and > __setitem__ special methods. This would be the easiest to get working, > I think. > > -Travis ______________________________________________________________ Verschicken Sie romantische, coole und witzige Bilder per SMS! Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193 From pjssilva at ime.usp.br Wed Mar 1 05:10:09 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Wed Mar 1 05:10:09 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> Message-ID: <1141218124.8183.0.camel@localhost.localdomain> Em Qua, 2006-03-01 ?s 01:05 -0800, Zachary Pincus escreveu: > In [4]: numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? > Out[4]: array([[ 5.00000000e+000, 4.46601253e-316, > 5.42111867e-306]]) It seems fixed in the repository In [1]:import numpy In [2]:numpy.dot([[5]],[[1,1,1]]) # 2D int case OK Out[2]:array([[5, 5, 5]]) In [3]:numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? Out[3]:array([[ 5., 5., 5.]]) In [4]:numpy.version.version Out[4]:'0.9.6.2179' Best, Paulo From pebarrett at gmail.com Wed Mar 1 06:45:02 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Wed Mar 1 06:45:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array In-Reply-To: <44054A19.6040202@ieee.org> References: <16761e100602282240y5bcf869fme9dd2f42771066c4@mail.gmail.com> <44054A19.6040202@ieee.org> Message-ID: <40e64fa20603010644x129d3f63ka07ab0b061469fd4@mail.gmail.com> On 3/1/06, Travis Oliphant wrote: > > > How many people would like to see x['f1','f2','f5'] return a new array > with a new data-type descriptor constructed from the provided fields? > > +1 I'm surprised that it's not already available. -- Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at utoronto.ca Wed Mar 1 09:03:10 2006 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Wed Mar 1 09:03:10 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <1141218124.8183.0.camel@localhost.localdomain> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> Message-ID: <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> Is there a test for this in the test suite.? It looks really bad for numpy if ppl cant do matrix multiplication properly on released versions. :( J. On 3/1/06, Paulo Jose da Silva e Silva wrote: > Em Qua, 2006-03-01 ?s 01:05 -0800, Zachary Pincus escreveu: > > > > In [4]: numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? > > Out[4]: array([[ 5.00000000e+000, 4.46601253e-316, > > 5.42111867e-306]]) > > It seems fixed in the repository > > In [1]:import numpy > > In [2]:numpy.dot([[5]],[[1,1,1]]) # 2D int case OK > Out[2]:array([[5, 5, 5]]) > > In [3]:numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? > Out[3]:array([[ 5., 5., 5.]]) > > In [4]:numpy.version.version > Out[4]:'0.9.6.2179' > > Best, > > Paulo > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Mar 1 09:33:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 1 09:33:05 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> Message-ID: <4405DA9B.9000307@ieee.org> Jonathan Taylor wrote: >Is there a test for this in the test suite.? It looks really bad for >numpy if ppl cant do matrix multiplication properly on released >versions. :( > > There are tests for matrix multiplication, just not one for this particular path through the code which is scalar multiplication by a row vector. (This is the optimized _dotblas.c code by the way which was just changed in 0.9.5 to try and make it faster -- obviously some bugs crept in at the same time). But, these bugs were quickly fixed when they were made apparent. That's why we are still not at 1.0 yet. Feel free to contribute more tests. -Travis From bsouthey at gmail.com Wed Mar 1 10:12:06 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Wed Mar 1 10:12:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array In-Reply-To: <1307204529@web.de> References: <1307204529@web.de> Message-ID: Hi, Thanks for your link, Niklas, as it looks interesting. The ability to create some sort of data array object would be very useful for me. The record array was not sufficiently flexible when I looked especially for missing values. The approach that I used was a class that defined, amongst other things, a masked array to be able to handle missing values and a dictionary to recode alphanumeric values into numeric values. The downside is that there is overhead in converting this to use in other functions and in some cases writing new functions. Bruce On 3/1/06, Niklas Volbers wrote: > [[Oops, I accidentally sent my post to scipy-users, but it was intended for numpy-discussion, so here's another attempt]] > > > Hey Michael, > > take a look at my attempt of such a Table implementation! > > The newest release 0.5.2 of my plotting project > > http://developer.berlios.de/projects/sloppyplot > > contains a Table class (in Sloppy.Base.dataset) which wraps a heterogeneous numpy array. The class should be fairly self-documenting, at least I hope so. Don't get confused by the 'undolist' stuff, this is my private undo implementation which could be easily removed from the code. > > If you want a similar implementation using a list of 1-dimensional arrays, then download the previous release 0.5.1 (which uses Numeric). > > The reason I switched over to the heterogeneous approach was that it is easier to provide similar wrappers for 2-d table like data (using a 2d heterogeneous array) and for 2-d matrix like data (using a 2d homogeneous array). Using a list of arrays gives you some problems when you would like to access the rows, because then you are in charge of creating a new 1-d array that represents the row, while with the heterogeneous array you can access both the columns and the rows quite naturally. > > By the way, I had first planned to subclass ndarray, but I did not know how to resize the array and still keep the array as such persistent. This is why I wrapped the array into a class called 'Dataset' which you can consider constant. > > If you need some more help on this, feel free to ask, > > Niklas Volbers. > > > > > > Travis Oliphant schrieb am 01.03.06 08:16:15: > > > > Michael Sorich wrote: > > > > > Hi, > > > > > > I am looking for a table like array. Something like a 'data frame' > > > object to those familiar with the statistical languages R and Splus. > > > This is mainly to hold and manipulate 2D spreadsheet like data, which > > > tends to be of relatively small size (compared to what many people > > > seem to use numpy for), heterogenous, have column and row names, and > > > often contains missing data. > > > > You could subclass the ndarray to produce one of these fairly easily, I > > think. The missing data item could be handled by a mask stored along > > with the array (or even in the array itself). Or you could use a masked > > array as your core object (though I'm not sure how it handles the > > arbitrary (i.e. record-like) data-types yet). > > > > Alternatively, and probably the easiest way to get started, you could > > just create your own table-like class and use simple 1-d arrays or 1-d > > masked arrays for each of the columns --- This has always been a way to > > store record-like tables. > > > > It really depends what you want the data-frames to be able to do and > > what you want them to "look-like." > > > > > A RecArray seems potentially useful, as it allows different fields to > > > have different data types and holds the name of the field. However it > > > doesn't seem easy to manipulate the data. Or perhaps I am simply > > > having difficulty finding documentation on there features. > > > > Adding a new column/field means basically creating a new array with a > > new data-type and copying data over into the already-defined fields. > > Data-types always have a fixed number of bytes per item. What those > > bytes represent can be quite arbitrary but it's always fixed. So, it > > is always "more work" to insert a new column. You could make that > > seamless in your table class so the user doesn't see it though. > > > > You'll want to thoroughly understand the dtype object including it's > > attributes and methods. Particularly the fields attribute of the dtype > > object. > > > > > eg > > > adding a new column/field (and to a lesser extent a new row/record) to > > > the recarray > > > > Adding a new row or record is actually similar because once an array is > > created it is usually resized by creating another array and copying the > > old array into it in the right places. > > > > > Changing the field/column names > > > make a new table by selecting a subset of fields/columns. (you can > > > select a single field/column, but not multiple). > > > > Right. So far you can't select multiple columns. It would be possible > > to add this feature with a little-bit of effort if there were a strong > > demand for it, but it would be much easier to do it in your subclass > > and/or container class. > > > > How many people would like to see x['f1','f2','f5'] return a new array > > with a new data-type descriptor constructed from the provided fields? > > > > > It would also be nice for the table to be able to deal easily with > > > masked data (I have not tried this with recarray yet) and perhaps also > > > to be able to give the rows/records unique ids that could be used to > > > select the rows/records (in addition to the row/record index), in the > > > same way that the fieldnames can select the fields. > > > > Adding fieldnames to the "rows" is definitely something that a subclass > > would be needed for. I'm not sure how you would even propose to select > > using row names. Would you also use getitem semantics? > > > > > Can anyone comment on this issue? Particularly whether code exists for > > > this purpose, and if not ideas about how best to go about developing > > > such a Table like array (this would need to be limited to python > > > programing as my ability to program in c is very limited). > > > > I don't know of code that already exists for this, but I don't think it > > would be too hard to construct your own data-frame object. > > > > I would probably start with an implementation that just used standard > > arrays of a particular type to represent the internal columns and then > > handle the indexing using your own over-riding of the __getitem__ and > > __setitem__ special methods. This would be the easiest to get working, > > I think. > > > > -Travis > > ______________________________________________________________ > Verschicken Sie romantische, coole und witzige Bilder per SMS! > Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193 > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Wed Mar 1 11:03:05 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 1 11:03:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array In-Reply-To: <40e64fa20603010644x129d3f63ka07ab0b061469fd4@mail.gmail.com> References: <16761e100602282240y5bcf869fme9dd2f42771066c4@mail.gmail.com> <44054A19.6040202@ieee.org> <40e64fa20603010644x129d3f63ka07ab0b061469fd4@mail.gmail.com> Message-ID: <4405EFE6.3020201@sympatico.ca> Paul Barrett wrote: > > On 3/1/06, *Travis Oliphant* > wrote: > > > How many people would like to see x['f1','f2','f5'] return a new > array > with a new data-type descriptor constructed from the provided fields? > > > +1 > > I'm surprised that it's not already available. > > -- Paul I would like some clarification as to just what is proposed. The expression looks like a subscription but the purpose is closer to that of a function or method. Is this a new method for something which can be done easily in the standard way? -1 Colin W From ndarray at mac.com Wed Mar 1 11:29:10 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 1 11:29:10 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <4405DA9B.9000307@ieee.org> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> <4405DA9B.9000307@ieee.org> Message-ID: On 3/1/06, Travis Oliphant wrote: > ... > Feel free to contribute more tests. There are some things that we can do to help users contribute tests. Before I make specific suggestions, let me just express an opinion that developers are less likely to write effective tests than users because they tend to overlook in testing exactly the same cases that they get wrong in coding. Here are the things that we can do: 1. Documentation. Tell users how to submit bug reports that can be immediately converted into unit tests. 2. Improve NumpyTest: add facilities to generate test data with various shapes and datatypes and filled with wide range of values: extremely large/small numbers, IEEE special values, etc. Add routines to automatically test universal functions on all datatypes. Automate consistency checks - when appropriate check that computations on different datatypes produce similar results. 3. Add support for doctests. This is probably the simplest type of test that a user can produce - just cut and paste from python session. Doctests can be placed in separate files and don't need to clutter real documentation. Python 2.4 contains code integrating doctests and unit tests and that code has 95% of what is needed. If we have to support 2.3, it may be appropriate to distribute a backport with numpy. The remaining 5% include support for ipython sessions, maybe better output checking for printed arrays (such as scan through to a closing paren rather than an empty line etc.). Any volunteers? I guess the first step would be to put up a wiki page. Since this is really a developers/advanced users project it belongs to the "hard hat" area. From cookedm at physics.mcmaster.ca Wed Mar 1 11:42:07 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Mar 1 11:42:07 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> <4405DA9B.9000307@ieee.org> Message-ID: <20060301194050.GA31179@arbutus.physics.mcmaster.ca> On Wed, Mar 01, 2006 at 02:28:28PM -0500, Sasha wrote: > On 3/1/06, Travis Oliphant wrote: > > ... > > Feel free to contribute more tests. > 3. Add support for doctests. This is probably the simplest type of > test that a user can produce - just cut and paste from python session. > Doctests can be placed in separate files and don't need to clutter > real documentation. Python 2.4 contains code integrating doctests and > unit tests and that code has 95% of what is needed. If we have to > support 2.3, it may be appropriate to distribute a backport with > numpy. The remaining 5% include support for ipython sessions, maybe > better output checking for printed arrays (such as scan through to a > closing paren rather than an empty line etc.). Already there. See the tests in numpy/lib/tests/test_polynomial.py -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From skip at pobox.com Wed Mar 1 16:03:04 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed Mar 1 16:03:04 2006 Subject: [Numpy-discussion] fold distutils back into Python distro? Message-ID: <17414.13849.523893.552309@montanaro.dyndns.org> As I understand it, numpy (or maybe it's scipy) has its own fork of distutils. What would it take to fold it back into the main distribution? Skip From oliphant at ee.byu.edu Wed Mar 1 16:14:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 1 16:14:03 2006 Subject: [Numpy-discussion] fold distutils back into Python distro? In-Reply-To: <17414.13849.523893.552309@montanaro.dyndns.org> References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: <440638AE.6020504@ee.byu.edu> skip at pobox.com wrote: >As I understand it, numpy (or maybe it's scipy) has its own fork of >distutils. What would it take to fold it back into the main distribution? > > > I guess a PEP that nobody has been willing to write. I suspect that not all of the numpy extensions are going to be seen as necessary by the Python people, and I don't know of anybody that's been willing to try and wrestle with that legwork. And just for clarification. As far as I understand it, it's not technically a fork of distutils. It is based on distutils but just extends it in many ways (i.e. new subclasses that extend from the Distutils classes). Pearu is the expert on it, however. -Travis From wbaxter at gmail.com Wed Mar 1 16:22:02 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 1 16:22:02 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function Message-ID: The NumPy for Matlab Users's wiki is currently pleading to have someone fill in "*the idiom for supporting both matrices and arrays in a function". *Does anyone know what this is? My guess is it depends first off on whether your goal is - to treat everything as a matrix, but also accept 2D arrays and do the conversion to matrix internally (asmatrix()) - to treat everything as a general array, but also accept matrices as a special case of 2D array, converting to arrays I guess it's likely that you'd also like to make it so the type returned is the same as the type input. Is there some slick way to do that? It doesn't seem to me that the latter case should really be specific to matrix/array since matrix is just one particular subclass of array. Is this what those UFuncs are for? Is there some way to write functions that will accept any object that supports the "array protocal"? This comment about the idiom for matrix/array functions is the last major wart remaining on that wiki page, so I'd like to wipe it out. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 1 16:23:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 1 16:23:02 2006 Subject: [Numpy-discussion] Re: fold distutils back into Python distro? In-Reply-To: <17414.13849.523893.552309@montanaro.dyndns.org> References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > As I understand it, numpy (or maybe it's scipy) has its own fork of > distutils. What would it take to fold it back into the main distribution? numpy.distutils (nee scipy_distutils) is not really a fork. It's just a set of extensions to distutils. One of the primary features is handling extensions with FORTRAN code. If you can convince Guido that standard distutils ought to handle FORTRAN, then we might have a deal. Some of the other features might be rolled into the standard library, but probably not on the 2.5 time frame. The useful ones might be the system_info stuff (which you have recently become far too familiar with :-)), the build_src capabilities, colored output, and the Configuration object. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant at ee.byu.edu Wed Mar 1 16:46:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 1 16:46:02 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: References: Message-ID: <4406400E.6080600@ee.byu.edu> Bill Baxter wrote: > The NumPy for Matlab Users's wiki is currently pleading to have > someone fill in "/*the idiom for supporting both matrices and arrays > in a function". > */Does anyone know what this is? I'm not sure what they want, exactly. Since a matrix is a subclass of an array you can treat them the same for the most part. The * and ** operators are the two that act differently on arrays and matrices so if you want to be sure then you use the functions dot and umath.multiply instead of the infix notation. If you want to convert arbitrary objects to "some-kind of array" then asanyarray(...) allows sub-classes of the array to pass to your function. To be perfectly careful, however, you will need to use explicity functions to perform any operation you may need. The other approach is to store the __array_wrap__ method of the input object (if there is any), convert everything to arrays using asarray() and then wrap the final result --- this is what ufuncs do internally. Again, I'm not sure what they mean? -Travis From robert.kern at gmail.com Wed Mar 1 16:52:32 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 1 16:52:32 2006 Subject: [Numpy-discussion] Re: The idiom for supporting matrices and arrays in a function In-Reply-To: References: Message-ID: Bill Baxter wrote: > The NumPy for Matlab Users's wiki is currently pleading to have someone > fill in "/*the idiom for supporting both matrices and arrays in a > function". > * /Does anyone know what this is? The subclassing functionality is rather new, so I don't know if the proper idioms have really been discovered. I would suggest converting to arrays using asarray() right at the beginning of the function. If you want to spit out matrices/what-have-you out again, then you will need to do some more work. I would suggest looking at the procedure that ufuncs do with __array_finalize__ and __array_priority__ and creating a pure Python reference implementation that others could use. It's possible that you would be able to turn it into a decorator. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant at ee.byu.edu Wed Mar 1 17:10:38 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 1 17:10:38 2006 Subject: [Numpy-discussion] Re: The idiom for supporting matrices and arrays in a function In-Reply-To: References: Message-ID: <440645D0.80701@ee.byu.edu> Robert Kern wrote: >Bill Baxter wrote: > > >>The NumPy for Matlab Users's wiki is currently pleading to have someone >>fill in "/*the idiom for supporting both matrices and arrays in a >>function". >>* /Does anyone know what this is? >> >> > >The subclassing functionality is rather new, so I don't know if the proper >idioms have really been discovered. I would suggest converting to arrays using >asarray() right at the beginning of the function. If you want to spit out >matrices/what-have-you out again, then you will need to do some more work. I >would suggest looking at the procedure that ufuncs do with __array_finalize__ >and __array_priority__ and creating a pure Python reference implementation that >others could use. It's possible that you would be able to turn it into a decorator. > > This is kind of emerging as the right strategy (although it's the __array_wrap__ and __array_priority__ methods that are actually relevant here. But, it is all rather new so we can forgive Robert this time ;-) ) The __array_priority__ is important if you've got "competing" wrappers (i.e. a 2-input, 1-output function) and don't know which __array_wrap__ to use for the output. In matlab you have one basic type of object. In NumPy you have a basic ndarray object that can be sub-classed which gives rise to all kinds of possibilities. I think we are still figuring out what the best thing to do is. Ufuncs convert everything to base-class arrays (what asarray does) and then use the __array_wrap__ of the input with the highest __array_priority__ to wrap all of the outputs (except that now output arrays passed in use their own __array_wrap__). So, I suppose following their example is a reasonable approach. Look in numpy/linlag/linlag.py for examples of using the asarray/__array_wrap__ strategy... The other approach is to let sub-classes through the array conversion (asanyarray), but this approach could rise to later errors if the sub-class redefines an operation you didn't expect it to, so is probably not safe unless all of your operations are functions that can handle any array subclass. And yes, I think a decorator could be written that would manage this. I'll leave that for the motivated... -Travis From skip at pobox.com Wed Mar 1 17:14:18 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed Mar 1 17:14:18 2006 Subject: [Numpy-discussion] fold distutils back into Python distro? In-Reply-To: <440638AE.6020504@ee.byu.edu> References: <17414.13849.523893.552309@montanaro.dyndns.org> <440638AE.6020504@ee.byu.edu> Message-ID: <17414.18081.540554.463915@montanaro.dyndns.org> Travis> And just for clarification. As far as I understand it, it's not Travis> technically a fork of distutils. It is based on distutils but Travis> just extends it in many ways (i.e. new subclasses that extend Travis> from the Distutils classes). Ah, okay. That's different. I was thinking it was a fork. Skip From skip at pobox.com Wed Mar 1 17:16:19 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed Mar 1 17:16:19 2006 Subject: [Numpy-discussion] Re: fold distutils back into Python distro? In-Reply-To: References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: <17414.18223.900432.86652@montanaro.dyndns.org> Robert> numpy.distutils (nee scipy_distutils) is not really a fork. It's Robert> just a set of extensions to distutils. One of the primary Robert> features is handling extensions with FORTRAN code. If you can Robert> convince Guido that standard distutils ought to handle FORTRAN, Robert> then we might have a deal. I see no reason why that wouldn't be possible in principle. After all, the bdist_* stuff keeps growing to handle a number of different package formats. Robert> Some of the other features might be rolled into the standard Robert> library, but probably not on the 2.5 time frame. The useful ones Robert> might be the system_info stuff (which you have recently become Robert> far too familiar with :-)), the build_src capabilities, colored Robert> output, and the Configuration object. Any idea if patches for each could be created fairly easily for each of these? Skip From haase at msg.ucsf.edu Wed Mar 1 18:14:02 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Mar 1 18:14:02 2006 Subject: [Numpy-discussion] numarray: need Float32 abs from array of type na.Complex64 or na.Complex32 Message-ID: <200603011813.40773.haase@msg.ucsf.edu> Hi, I just was hunting a very strange bug: I got weird results .... anyway. I found that I was using this line of code: if img.type() in (na.Complex64, na.Complex32): img = abs(na.asarray(img, na.Float32)) Now I don't understand how this code worked at all ? But I only saw those "weird results" after looking at the "dimmest corners" in my image and otherwise all images looked O.K. The goal here was to get a single precision absolute for either a complex64 or complex32 valued image array WITHOUT creating a temporary array. My thinking was that: img = na.abs( na.asarray(img, na.Complex32) ) would create a complete temporary version of img starts out being Complex64 Is this really the case ? Thanks for any hints, Sebastian Haase From gerard.vermeulen at grenoble.cnrs.fr Thu Mar 2 00:30:02 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Thu Mar 2 00:30:02 2006 Subject: [Numpy-discussion] Re: fold distutils back into Python distro? In-Reply-To: References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: <20060302092859.70abb8ce.gerard.vermeulen@grenoble.cnrs.fr> On Wed, 01 Mar 2006 18:21:41 -0600 Robert Kern wrote: > Some of the other features might be rolled into the standard library, but > probably not on the 2.5 time frame. The useful ones might be the system_info > stuff (which you have recently become far too familiar with :-)), the build_src > capabilities, colored output, and the Configuration object. IMO, the colored output should be an optional feature. My emacs buffers do not understand it and the only way to disable it is modifying the source as far as I remember. Gerard From hurak at control.felk.cvut.cz Thu Mar 2 03:23:03 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Thu Mar 2 03:23:03 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. Message-ID: Hello, Can anybody recommend the book: Python Scripting for Computational Science Series: Texts in Computational Science and Engineering, Vol. 3 Langtangen, Hans Petter 2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover ISBN: 3-540-29415-5 I am just a beginner in Python programming (but I master C, Matlab). I do research in applied numerical computing (oriented at control design). Is this a book for me? It is hard to guess just from the table contents. I especially doubt if the information on Numpy package is relevant at all with the wild development witnessed in the past recent months. However, peeling off these "numerical" parts, isn't buying some comprehensive Python book a better buy? Thanks for your tips, Zdenek Hurak From vinicius.lobosco at gmail.com Thu Mar 2 03:47:01 2006 From: vinicius.lobosco at gmail.com (Vinicius Lobosco) Date: Thu Mar 2 03:47:01 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. In-Reply-To: References: Message-ID: <200603021246.08706.vinicius.lobosco@gmail.com> Hi Zdenek! I have this book, and I like this a lot as the other book of Langtangen about Diffpack. I think he is very good in collecting in one place most of the aspects one would like to use within this subject. So if you think about implementing your software as a web service, to implement a GUI or to glue you C, C++ or Fortran software with Python or something else, this is the right book. And it is very well written. On the other hand, if you're just looking for information regarding NumPy (from a Matlab user perspective), it would be better to borrow it in the library and copy the only chapter about it. There is however, no book alternative. The best material is available on-line, to my knowledge. Second, you're also right that some stuffs are not updated anymore. Best regards, Vinicius On Thursday 02 March 2006 12.21, Zden?k Hur?k wrote: > Hello, > > Can anybody recommend the book: > > Python Scripting for Computational Science > Series: Texts in Computational Science and Engineering, Vol. 3 > Langtangen, Hans Petter > 2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover > ISBN: 3-540-29415-5 > > I am just a beginner in Python programming (but I master C, Matlab). I do > research in applied numerical computing (oriented at control design). Is > this a book for me? It is hard to guess just from the table contents. I > especially doubt if the information on Numpy package is relevant at all > with the wild development witnessed in the past recent months. However, > peeling off these "numerical" parts, isn't buying some comprehensive Python > book a better buy? > > Thanks for your tips, > Zdenek Hurak ------------------------ Vinicius Lobosco, PhD Process Intelligence www.paperplat.com +46 8 612 7803 +46 73 925 8476 (cell phone) Bj?rnn?sv?gen 21 SE-113 47 Stockholm, Sweden From cjw at sympatico.ca Thu Mar 2 05:39:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 2 05:39:01 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: <4406400E.6080600@ee.byu.edu> References: <4406400E.6080600@ee.byu.edu> Message-ID: <4406F52E.8000603@sympatico.ca> Travis Oliphant wrote: > Bill Baxter wrote: > >> The NumPy for Matlab Users's wiki is currently pleading to have >> someone fill in "/*the idiom for supporting both matrices and arrays >> in a function". */Does anyone know what this is? > > > I'm not sure what they want, exactly. > > Since a matrix is a subclass of an array you can treat them the same > for the most part. The * and ** operators are the two that act > differently on arrays and matrices so if you want to be sure then you > use the functions dot and umath.multiply instead of the infix notation. > > If you want to convert arbitrary objects to "some-kind of array" then > > asanyarray(...) How does asanyyarray() differ from asarray()? Colin W. > > allows sub-classes of the array to pass to your function. To be > perfectly careful, however, you will need to use explicity functions > to perform any operation you may need. > > The other approach is to store the __array_wrap__ method of the input > object (if there is any), convert everything to arrays using asarray() > and then wrap the final result --- this is what ufuncs do internally. > > Again, I'm not sure what they mean? > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From bsouthey at gmail.com Thu Mar 2 05:50:03 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Mar 2 05:50:03 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. In-Reply-To: References: Message-ID: Hi, I have the first edition which is very comprehensive on Python, numarray and Numeric - note that by NumPy he is means only Numeric and numarray and not the new numpy. If you have the basic of Python and numerical computing then it is a great book as it cover many different topics. For example, there is considerable detail on using Fortran and C/C++ with Numeric and numarray including examples. For the majority of the material in the book is still valid for the new Numpy. He does provide a number of advanced topics in Python, numarray and Numeric that are hard to find elsewhere. If you are going to extensive numerical work then it is really worth it. Bruce On 3/2/06, Zden?k Hur?k wrote: > Hello, > > Can anybody recommend the book: > > Python Scripting for Computational Science > Series: Texts in Computational Science and Engineering, Vol. 3 > Langtangen, Hans Petter > 2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover > ISBN: 3-540-29415-5 > > I am just a beginner in Python programming (but I master C, Matlab). I do > research in applied numerical computing (oriented at control design). Is > this a book for me? It is hard to guess just from the table contents. I > especially doubt if the information on Numpy package is relevant at all > with the wild development witnessed in the past recent months. However, > peeling off these "numerical" parts, isn't buying some comprehensive Python > book a better buy? > > Thanks for your tips, > Zdenek Hurak > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aisaac at american.edu Thu Mar 2 06:59:00 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu Mar 2 06:59:00 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: <4406F52E.8000603@sympatico.ca> References: <4406400E.6080600@ee.byu.edu><4406F52E.8000603@sympatico.ca> Message-ID: On Thu, 02 Mar 2006, "Colin J. Williams" apparently wrote: > How does asanyyarray() differ from asarray()? It leaves alone subclasses of ndarray. See example below. Cheers, Alan Isaac Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as N >>> y=N.mat([[1,2],[3,4]]) >>> z1=N.asarray(y) >>> z2=N.asanyarray(y) >>> z1 array([[1, 2], [3, 4]]) >>> z2 matrix([[1, 2], [3, 4]]) >>> From jmiller at stsci.edu Thu Mar 2 07:14:03 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Mar 2 07:14:03 2006 Subject: [Numpy-discussion] numarray: need Float32 abs from array of type na.Complex64 or na.Complex32 In-Reply-To: <200603011813.40773.haase@msg.ucsf.edu> References: <200603011813.40773.haase@msg.ucsf.edu> Message-ID: <44070B9C.4040805@stsci.edu> Sebastian Haase wrote: > Hi, > I just was hunting a very strange bug: I got weird results .... anyway. > > I found that I was using this line of code: > if img.type() in (na.Complex64, na.Complex32): > img = abs(na.asarray(img, na.Float32)) > > Now I don't understand how this code worked at all ? But I only saw those > "weird results" after looking at the "dimmest corners" in my image and > otherwise all images looked O.K. > > The goal here was to get a single precision absolute for either a complex64 or > complex32 valued image array WITHOUT creating a temporary array. > This idea sounds inconsistent to me. If the array is Complex64 and you pass it into asarray(,Float32), you're going to get a temporary. AND you're also truncating the imaginary part as you downcast to Float32... you're not getting a complex magnitude. Since abs() transforms from complex to real, you're going to get a temporary unless you get a little fancy, maybe like this: na.abs(img, img.real) # stores the abs() into the real component of the original array. img.real is a view not a copy. img.imag = 0 # optional step which makes the complex img array a real valued array with complex storage. img = img.real # just forget that img is using complex storage. Note that img is now discontiguous since no copy was made and there are still interleaved 0-valued imaginary components. So, I think there are two points to avoiding a copy here: (1) use the optional ufunc output parameter (2) store the abs() result into the .real view of the original complex array. > My thinking was that: > img = na.abs( na.asarray(img, na.Complex32) ) > would create a complete temporary version of img starts out being Complex64 > Is this really the case ? > Yes. From stefan at sun.ac.za Thu Mar 2 08:14:04 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu Mar 2 08:14:04 2006 Subject: [Numpy-discussion] setting package path In-Reply-To: References: <20060228221937.GE10590@alpha> Message-ID: <20060302161325.GC12227@alpha> On Tue, Feb 28, 2006 at 11:34:43PM -0600, Pearu Peterson wrote: > >which calculates "somepath". Which is wrong: the docstring, the code > >or my interpretation? > > You have to read also following code: > > d1 = > os.path.join(d,'build','lib.%s-%s'%(get_platform(),sys.version[:3])) > if not os.path.isdir(d1): > d1 = os.path.dirname(d) # <- here we get "somepath/.." > sys.path.insert(0,d1) Pearu, Thanks for pointing this out. I was trying to figure out why my module's tests didn't run when I did, for example: $ python test_geom.py But now I realise it is because typical package layout is "somepath/some_dir/tests/test_file.py", not "somepath/some_dir/test_file.py". For example, we have "numpy/core/tests/test_umath.py". As a temporary workaround, set_local_path("../../..") works for me. Regards St?fan From jcollins_boulder at earthlink.net Thu Mar 2 09:01:08 2006 From: jcollins_boulder at earthlink.net (Jeffery D. Collins) Date: Thu Mar 2 09:01:08 2006 Subject: [Numpy-discussion] Numpy licenses Message-ID: <440724A4.4010804@earthlink.net> The SF site claims the following for Numeric/Numpy: License : OSI-Approved Open Source , GNU General Public License (GPL) Can this be changed to reflect LICENSE.TXT? Numpy (and the older Numeric-*) has never been GPL'd, correct? Also, a previous thread mentioned that the licensing terms for f2py will be changed from LGPL. In the svn trunk, some of its documentation still refers to LGPL, particularly f2py2e.py. Will this be changed before the next release? Thanks! Jeff From mfmorss at aep.com Thu Mar 2 09:03:02 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Thu Mar 2 09:03:02 2006 Subject: [Numpy-discussion] problems installing numpy on AIX Message-ID: I returned to this task. After some minor tweaks discussed here earlier, the build appears to succeed. There are two issues. Issue 1 is, to me, of unknown significance; it does not prevent the build. Issue 2 is that numpy can't be imported. There are problems with newdocs, type_check and numerictypes. The symptoms are shown below. Issue 1. creating build/temp.aix-5.2-2.4/build/src/numpy/core/src compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/app/sandbox/s625662/installed/include/python2.4 -c' cc_r: build/src/numpy/core/src/umathmodule.c "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. Issue 2. $ python Python 2.4.2 (#2, Feb 22 2006, 08:38:08) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy import core -> failed: import random -> failed: 'module' object has no attribute 'dtype' import lib -> failed: import linalg -> failed: import dft -> failed: Traceback (most recent call last): File "", line 1, in ? File "/mydirectory/lib/python2.4/site-packages/numpy/__init__.py", line 45, in ? import add_newdocs File "/mydirectory/lib/python2.4/site-packages/numpy/add_newdocs.py", line 2, in ? from lib import add_newdoc File "/mydirectory/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in ? from type_check import * File "/mydirectory/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in ? import numpy.core.numeric as _nx File "/mydirectory/lib/python2.4/site-packages/numpy/core/__init__.py", line 7, in ? import numerictypes as nt File "/mydirectory/lib/python2.4/site-packages/numpy/core/numerictypes.py", line 371, in ? _unicodesize = array('u','U').itemsize MemoryError >>> Mark F. Morss Principal Analyst, Market Risk American Electric Power From pfdubois at gmail.com Thu Mar 2 09:07:04 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Thu Mar 2 09:07:04 2006 Subject: [Numpy-discussion] Numpy licenses In-Reply-To: <440724A4.4010804@earthlink.net> References: <440724A4.4010804@earthlink.net> Message-ID: Numeric and numpy are not related from a licensing point of view. Numeric was released under an LLNL-written open source license. Travis wrote numpy and he can put any license he wants on that. On 02 Mar 2006 09:03:12 -0800, Jeffery D. Collins < jcollins_boulder at earthlink.net> wrote: > > The SF site claims the following for Numeric/Numpy: > > License : OSI-Approved Open Source > , GNU > General Public License (GPL) > > > > Can this be changed to reflect LICENSE.TXT? Numpy (and the older > Numeric-*) has never been GPL'd, correct? > > Also, a previous thread mentioned that the licensing terms for f2py will > be changed from LGPL. In the svn trunk, some of its documentation still > refers to LGPL, particularly f2py2e.py. Will this be changed before > the next release? > > Thanks! > > Jeff > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfmorss at aep.com Thu Mar 2 09:20:01 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Thu Mar 2 09:20:01 2006 Subject: [Numpy-discussion] problems installing numpy on AIX In-Reply-To: Message-ID: Minor correction. Mark F. Morss Principal Analyst, Market Risk 614-583-6757 Audinet 220-6757 mfmorss at aep.com Sent by: numpy-discussion- To admin at lists.sourc numpy-discussion eforge.net cc 03/02/2006 11:59 AM Subject [Numpy-discussion] problems installing numpy on AIX I returned to this task. After some minor tweaks discussed here earlier, the build appears to succeed. There are two issues. Issue 1 is, to me, of unknown significance; it does not prevent the build. Issue 2 is that numpy can't be imported. There are problems with newdocs, type_check and numerictypes. The symptoms are shown below. Issue 1. creating build/temp.aix-5.2-2.4/build/src/numpy/core/src compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/mydirectory/include/python2.4 -c' cc_r: build/src/numpy/core/src/umathmodule.c "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. Issue 2. $ python Python 2.4.2 (#2, Feb 22 2006, 08:38:08) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy import core -> failed: import random -> failed: 'module' object has no attribute 'dtype' import lib -> failed: import linalg -> failed: import dft -> failed: Traceback (most recent call last): File "", line 1, in ? File "/mydirectory/lib/python2.4/site-packages/numpy/__init__.py", line 45, in ? import add_newdocs File "/mydirectory/lib/python2.4/site-packages/numpy/add_newdocs.py", line 2, in ? from lib import add_newdoc File "/mydirectory/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in ? from type_check import * File "/mydirectory/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in ? import numpy.core.numeric as _nx File "/mydirectory/lib/python2.4/site-packages/numpy/core/__init__.py", line 7, in ? import numerictypes as nt File "/mydirectory/lib/python2.4/site-packages/numpy/core/numerictypes.py", line 371, in ? _unicodesize = array('u','U').itemsize MemoryError >>> Mark F. Morss Principal Analyst, Market Risk American Electric Power ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From robert.kern at gmail.com Thu Mar 2 09:26:10 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 2 09:26:10 2006 Subject: [Numpy-discussion] Re: Numpy licenses In-Reply-To: <440724A4.4010804@earthlink.net> References: <440724A4.4010804@earthlink.net> Message-ID: Jeffery D. Collins wrote: > The SF site claims the following for Numeric/Numpy: > > License : OSI-Approved Open Source > , GNU > General Public License (GPL) > > > > Can this be changed to reflect LICENSE.TXT? Numpy (and the older > Numeric-*) has never been GPL'd, correct? It's possible that there used to be some code checked into the repository that was GPLed. It was certainly never a part of Numeric, though. What worries me more is that someone thought Numeric et al. belong to the Multi-User Dungeons topic. I would like to stress, though, that Sourceforge is *not* the place for information on the new NumPy. The only remaining use for it is the mailing list and downloads, and I hope that we can start using PyPI instead for placing our distributions. > Also, a previous thread mentioned that the licensing terms for f2py will > be changed from LGPL. In the svn trunk, some of its documentation still > refers to LGPL, particularly f2py2e.py. Will this be changed before > the next release? Done. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From cookedm at physics.mcmaster.ca Thu Mar 2 10:13:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Mar 2 10:13:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <440530AC.2010000@enthought.com> (eric jones's message of "Tue, 28 Feb 2006 23:27:08 -0600") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: eric jones writes: > Travis Oliphant wrote: > >> Weave can still help with the "auto-compilation" of the specific >> library for your type. Ultimately such code will be faster than >> NumPy can every be. > > Yes. weave.blitz() can be used to do the equivalent of this lazy > evaluation for you in many cases without much effort. For example: > > import weave > from scipy import arange > > a = arange(1e7) > b = arange(1e7) > c=2.0*a+3.0*b > > # or with weave > weave.blitz("c=2.0*a+3.0*b") > > As Paul D. mentioned.what Tim outlined is essentially template > expressions in C++. blitz++ (http://www.oonumerics.org/blitz/) is a > C++ template expressions library for array operations, and weave.blitz > translates a Numeric expression into C++ blitz code. For the example > above on large arrays, you get about a factor of 4 speed up on large > arrays. (Notice, the first time you run the example it will be much > slower because of compile time. Use timings from subsequent runs.) > > C:\temp>weave_time.py > Expression: c=2.0*a+3.0*b > Numeric: 0.678311899322 > Weave: 0.162177084984 > Speed-up: 4.18253848494 > > All this to say, I think weave basically accomplishes what Tim wants > with a different mechanism (letting C++ compilers do the optimization > instead of writing this optimization at the python level). It does > require a compiler on client machines in its current form (even that > can be fixed...), but I think it might prove faster than > re-implementing a numeric expression compiler at the python level > (though that sounds fun as well). I couldn't leave it at that :-), so I wrote my bytecode idea up. You can grab it at http://arbutus.mcmaster.ca/dmc/software/numexpr-0.1.tar.gz Here are the performance numbers (note I use 1e6 elements instead of 1e7) cookedm at arbutus$ py -c 'import numexpr.timing; numexpr.timing.compare()' Expression: b*c+d*e numpy: 0.0934900999069 Weave: 0.0459051132202 Speed-up of weave over numpy: 2.03659447388 numexpr: 0.0467489004135 Speed-up of numexpr over numpy: 1.99983527056 Expression: 2*a+3*b numpy: 0.0784019947052 Weave: 0.0292909860611 Speed-up of weave over numpy: 2.67665945222 numexpr: 0.0323888063431 Speed-up of numexpr over numpy: 2.42065094572 Wow. On par with weave.blitz, and no C++ compiler!!! :-) You use it like this: from numexpr import numexpr func = numexpr("2*a+3*b") a = arange(1e6) b = arange(1e6) c = func(a, b) Alternatively, you can use it like weave.blitz, like this: from numexpr import evaluate a = arange(1e6) b = arange(1e6) c = evaluate("2*a+3*b") How it works ============ Well, first of all, it only works with the basic operations (+, -, *, and /), and only on real constants and arrays of doubles. (These restrictions are mainly for lack of time, of course.) Given an expression, it first compiles it to a program written in a small bytecode. Here's what it looks like: In [1]: from numexpr import numexpr In [2]: numexpr("2*a+3*b", precompiled=True) # precompiled=True just returns the program before it's made into a # bytecode object, so you can what it looks like Out[2]: [('mul_c', Register(0), Register(1, a), Constant(0, 2)), ('mul_c', Temporary(3), Register(2, b), Constant(1, 3)), ('add', Register(0), Register(0), Temporary(3))] In [3]: c = numexpr("2*a+3*b") In [4]: c.program Out[4]: '\x08\x00\x01\x00\x08\x03\x02\x01\x02\x00\x00\x03' In [5]: c.constants Out[5]: array([ 2., 3.]) In [6]: c.n_temps Out[6]: 1 In [7]: type(c) Out[7]: The bytecode program is a string of 4-character groups, where the first character of each group is the opcode, the second is the "register" to store in, and the third and fourth are the register numbers to use as arguments. Inputs and outputs are assigned to registers (so in the example above, the output is Register(0), and the two inputs are Register(1) and Register(2)), and temporaries are in remaining registers. The group of registers is actually an array of double*. The secret behind the speed is that the bytecode program is run blocked: it's run repeatedly, operating on 128 elements of each input array at at time (then 8 elements then 1 element for the leftover). This does a tradeoff between cache misses (the arguments for each time through the program will most likely be able to fit in the cache each time), and the overhead of the branching from evaluating the program. It's like doing this in numpy: c[0:128] = 2*a[0:128] + 3*b[0:128] c[128:256] = 2*a[128:256] + 3*b[128:256] etc. I could see this getting a nice speedup from using a CPU's vector instructions. For instance, it should be pretty easy to use liboil for the intermediate vector evaluations. If people think it's useful enough, I can check it into scipy's sandbox. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Thu Mar 2 11:41:00 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 2 11:41:00 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <440530AC.2010000@enthought.com> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <44074A05.30700@noaa.gov> eric jones wrote: > All this to say, I think weave basically accomplishes what Tim wants > with a different mechanism (letting C++ compilers do the optimization > instead of writing this optimization at the python level). It does > require a compiler on client machines in its current form (even that can > be fixed...) Perhaps by teaching Psyco about nd-arrays? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Mar 2 13:13:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 2 13:13:04 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <44075FA9.5040104@ee.byu.edu> David M. Cooke wrote: >I couldn't leave it at that :-), so I wrote my bytecode idea up. > >You can grab it at http://arbutus.mcmaster.ca/dmc/software/numexpr-0.1.tar.gz > > Awesome. Please do check it in to the sandbox. -Travis From schofield at ftw.at Thu Mar 2 13:25:01 2006 From: schofield at ftw.at (Ed Schofield) Date: Thu Mar 2 13:25:01 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <3FAADB1F-9C8C-4E92-91AA-244D15E32C3B@ftw.at> On 02/03/2006, at 7:12 PM, David M. Cooke wrote: > I couldn't leave it at that :-), so I wrote my bytecode idea up. > > You can grab it at http://arbutus.mcmaster.ca/dmc/software/ > numexpr-0.1.tar.gz I'd had a brief look at the code, and I think it's great! I particularly like the power available with the "numexpr.evaluate (string_expression)" syntax. I'd fully support your adding it to the scipy sandbox. I'm sure we could fill it out with more operators and data types quite fast. Great work! -- Ed From tim.hochberg at cox.net Thu Mar 2 15:21:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 2 15:21:04 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <44074A05.30700@noaa.gov> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <44074A05.30700@noaa.gov> Message-ID: <44077DA3.70801@cox.net> Christopher Barker wrote: > eric jones wrote: > >> All this to say, I think weave basically accomplishes what Tim wants >> with a different mechanism (letting C++ compilers do the optimization >> instead of writing this optimization at the python level). It does >> require a compiler on client machines in its current form (even that >> can be fixed...) > > > Perhaps by teaching Psyco about nd-arrays? Psyco is unlikely to ever be much good for floating point stuff. This is because it only knows about longs/pointers and arrays of longs/pointers. The floating point support it has at present involves breaking a double into two long sized chunks. Then whenever you need to manipulate the float you have to find the pieces and put them back together[1]. All of this means that float support is much poorer then integer support in Psyco. In theory this could be fixed, but since Armin's no longer actively working on Psyco, just maintaining it, I don't see this changing. Perhaps when Psyco like technology gets incorporated in PyPy it will include better support for floating point. Regards, -tim [1] This has all gotten rather fuzzy too me now, it's been a while since I looked at it and my understanding was at best fragile anyway. From tim.hochberg at cox.net Thu Mar 2 15:49:11 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 2 15:49:11 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <4407843A.5010300@cox.net> David M. Cooke wrote: >eric jones writes: > > > >>Travis Oliphant wrote: >> >> >> >>>Weave can still help with the "auto-compilation" of the specific >>>library for your type. Ultimately such code will be faster than >>>NumPy can every be. >>> >>> >>Yes. weave.blitz() can be used to do the equivalent of this lazy >>evaluation for you in many cases without much effort. For example: >> >>import weave >>from scipy import arange >> >>a = arange(1e7) >>b = arange(1e7) >>c=2.0*a+3.0*b >> >># or with weave >>weave.blitz("c=2.0*a+3.0*b") >> >>As Paul D. mentioned.what Tim outlined is essentially template >>expressions in C++. blitz++ (http://www.oonumerics.org/blitz/) is a >>C++ template expressions library for array operations, and weave.blitz >>translates a Numeric expression into C++ blitz code. For the example >>above on large arrays, you get about a factor of 4 speed up on large >>arrays. (Notice, the first time you run the example it will be much >>slower because of compile time. Use timings from subsequent runs.) >> >>C:\temp>weave_time.py >>Expression: c=2.0*a+3.0*b >>Numeric: 0.678311899322 >>Weave: 0.162177084984 >>Speed-up: 4.18253848494 >> >>All this to say, I think weave basically accomplishes what Tim wants >>with a different mechanism (letting C++ compilers do the optimization >>instead of writing this optimization at the python level). It does >>require a compiler on client machines in its current form (even that >>can be fixed...), but I think it might prove faster than >>re-implementing a numeric expression compiler at the python level >>(though that sounds fun as well). >> >> > >I couldn't leave it at that :-), so I wrote my bytecode idea up. > >You can grab it at http://arbutus.mcmaster.ca/dmc/software/numexpr-0.1.tar.gz > >Here are the performance numbers (note I use 1e6 elements instead of >1e7) > >cookedm at arbutus$ py -c 'import numexpr.timing; numexpr.timing.compare()' >Expression: b*c+d*e >numpy: 0.0934900999069 >Weave: 0.0459051132202 >Speed-up of weave over numpy: 2.03659447388 >numexpr: 0.0467489004135 >Speed-up of numexpr over numpy: 1.99983527056 > >Expression: 2*a+3*b >numpy: 0.0784019947052 >Weave: 0.0292909860611 >Speed-up of weave over numpy: 2.67665945222 >numexpr: 0.0323888063431 >Speed-up of numexpr over numpy: 2.42065094572 > >Wow. On par with weave.blitz, and no C++ compiler!!! :-) > > That's awesome! I was also tempted by this, but I never got beyond prototyping some stuff in Python. >You use it like this: > >from numexpr import numexpr > >func = numexpr("2*a+3*b") > >a = arange(1e6) >b = arange(1e6) >c = func(a, b) > > Does this just uses the order that variable are initially used in the expression to determine the input order? I'm not sure I like that. It is very convenient for simple expressions though. >Alternatively, you can use it like weave.blitz, like this: > >from numexpr import evaluate > >a = arange(1e6) >b = arange(1e6) >c = evaluate("2*a+3*b") > > That's pretty sweet. And I see that you cache the expressions, so it should be pretty fast if you need to loop. [snip details] >If people think it's useful enough, I can check it into scipy's sandbox. > > Definitely check it in. It won't compile here with VC7, but I'll see if I can figure out why. This is probably thinking two far ahead, but an interesting thing to try would be adding conditional expressions: c = evaluate("(2*a + b) if (a > b) else (2*b + a)") If that could be made to work, and work fast, it would save both memory and time in those cases where you have to vary the computation based on the value. At present I end up computing the full arrays for each case and then choosing which result to use based on a mask, so it takes three times as much space as doing it element by element. -tim From oliphant at ee.byu.edu Thu Mar 2 16:05:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 2 16:05:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <4407881A.6090108@ee.byu.edu> David M. Cooke wrote: >I couldn't leave it at that :-), so I wrote my bytecode idea up. > > > I like what you've done. It makes me wonder what could be done with just a Python-level solution (that might be easier to generalize to more data-types and operators). You've definitely given us all some great stuff to play with. -Travis From cookedm at physics.mcmaster.ca Thu Mar 2 16:20:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Mar 2 16:20:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4407843A.5010300@cox.net> (Tim Hochberg's message of "Thu, 02 Mar 2006 16:48:10 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> Message-ID: Tim Hochberg writes: > David M. Cooke wrote: > >>Wow. On par with weave.blitz, and no C++ compiler!!! :-) >> >> > That's awesome! I was also tempted by this, but I never got beyond > prototyping some stuff in Python. > >>You use it like this: >> >>from numexpr import numexpr >> >>func = numexpr("2*a+3*b") >> >>a = arange(1e6) >>b = arange(1e6) >>c = func(a, b) >> >> > Does this just uses the order that variable are initially used in the > expression to determine the input order? I'm not sure I like that. It > is very convenient for simple expressions though. By default, it does lexical order (whatever .sort() on the list of variable names gives). You can specify the order with func = numexpr("2*a+3*b", input_order=('a', 'b')) >>Alternatively, you can use it like weave.blitz, like this: >> >>from numexpr import evaluate >> >>a = arange(1e6) >>b = arange(1e6) >>c = evaluate("2*a+3*b") >> >> > That's pretty sweet. And I see that you cache the expressions, so it > should be pretty fast if you need to loop. > > [snip details] > >>If people think it's useful enough, I can check it into scipy's sandbox. >> >> > Definitely check it in. It won't compile here with VC7, but I'll see > if I can figure out why. It's in. Check the setup.py -- I left some gcc'isms in there (-funroll-loops). That should be conditional on the compiler. > This is probably thinking two far ahead, but an interesting thing to > try would be adding conditional expressions: > > c = evaluate("(2*a + b) if (a > b) else (2*b + a)") > > If that could be made to work, and work fast, it would save both > memory and time in those cases where you have to vary the computation > based on the value. At present I end up computing the full arrays for > each case and then choosing which result to use based on a mask, so it > takes three times as much space as doing it element by element. It's on my list of things to think about, along with implementing reductions (sum, in particular). It'd probably look more like c = evaluate("where(a > b, 2*a+b, 2*b+a)") because of the vector nature of the code. Doing the "if" elementwise would mean the bytecode program would have to be run for each element, instead of on a vector of some fixed length. I wrote up a bit on my blog at http://isobaric.blogspot.com/ (finally, something to blog about :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant.travis at ieee.org Thu Mar 2 17:05:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 2 17:05:19 2006 Subject: [Numpy-discussion] Re: problems installing numpy on AIX In-Reply-To: References: Message-ID: mfmorss at aep.com wrote: > > I returned to this task. > > After some minor tweaks discussed here earlier, the build appears to > succeed. There are two issues. Issue 1 is, to me, of unknown > significance; it does not prevent the build. These errors seem to indicate that while the build finished it did not result in useful code. Could you post the output of the build log? It may contain warnings that are relevant. I don't understand why you are getting the errors you are getting. They seem unusual. Make sure you are running from outside the directory you built in (i.e. don't run in the numpy source directory). > > Issue 2 is that numpy can't be imported. There are problems with newdocs, > type_check and numerictypes. > > The symptoms are shown below. > > > Issue 1. > > creating build/temp.aix-5.2-2.4/build/src/numpy/core/src > compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include > -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include > -I/mydirectory/include/python2.4 -c' > cc_r: build/src/numpy/core/src/umathmodule.c > "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) > Function argument assignment between types "long double*" and "double*" is > not allowed. This is a strange as it seems your math library has the wrong definition for modfl. Can you verify what section of code this is balking on? Thanks for the attempts, -Travis From wbaxter at gmail.com Thu Mar 2 17:23:23 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu Mar 2 17:23:23 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: References: <4406400E.6080600@ee.byu.edu> <4406F52E.8000603@sympatico.ca> Message-ID: Thanks for all the info folks. This all sounds a little more complicated than is warranted for the "Numpy for Matlab Users" wiki page, though. So I think I'll just comment on the presence of asarray, asanyarray, and asmatrix for type conversions. --bb On 3/3/06, Alan G Isaac wrote: > > On Thu, 02 Mar 2006, "Colin J. Williams" apparently wrote: > > How does asanyyarray() differ from asarray()? > > It leaves alone subclasses of ndarray. See example below. > Cheers, > Alan Isaac > > Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit > (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as N > >>> y=N.mat([[1,2],[3,4]]) > >>> z1=N.asarray(y) > >>> z2=N.asanyarray(y) > >>> z1 > array([[1, 2], > [3, 4]]) > >>> z2 > matrix([[1, 2], > [3, 4]]) > >>> > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Mar 2 20:18:03 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Mar 2 20:18:03 2006 Subject: [Numpy-discussion] Matlab page on scipy wiki In-Reply-To: <20060210122242.GA21950@sun.ac.za> References: <20060210122242.GA21950@sun.ac.za> Message-ID: <4407C359.3030601@msg.ucsf.edu> Hi, I noted on http://www.scipy.org/Wiki/NumPy_for_Matlab_Users matlab a(1:5,:) numpy a[0:4] or a[0:4,:] a[0:4] or a[0:4,:] the first five rows of a I think this is wrong! in numpy it would be: a[:5] (or a[0:5] ) for the first five elements To the best of my knowledge (I have never used Matlab myself!) this is one of the biggest points of confusion for Matlab users !! WHY DOES a[4:6] NOT INCLUDE THE ELEMENTS 4,5 *AND* 6 ??? The only explanation I have is that its a) like a C/C++-for-loop (for(i=4;i<6;i++) --> note the '<' (not '<=') b) it then also always is true that: "last" minus first is equal to number-of-elements (example: 6-4 = 2) c) it turns out from experience that this convention WILL save you lots of '-1' and '+1' in your code (actually almost all of them) [if I see a "+1" in a Matlab person's numpy code I can almost always guess that he/she made a mistake !] Maybe this paragraph could be added to the wiki ... Thanks for this wiki page - I think I looks great - Sebastian Haase From jh at oobleck.astro.cornell.edu Thu Mar 2 20:24:02 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Thu Mar 2 20:24:02 2006 Subject: [Numpy-discussion] Numpy licenses In-Reply-To: <20060302170802.597DD88A09@sc8-sf-spam1.sourceforge.net> (numpy-discussion-request@lists.sourceforge.net) References: <20060302170802.597DD88A09@sc8-sf-spam1.sourceforge.net> Message-ID: <200603030423.k234NgCD004844@oobleck.astro.cornell.edu> > Travis wrote numpy and he can put any license he wants on that. Really, he didn't take any lines of code from its predecessors? My understanding was that much borrowing occurred. This is good, and is the strength of OSS, but it does require attention to licensing for derived works. Travis, if I'm mistaken, please correct me. --jh-- From wbaxter at gmail.com Thu Mar 2 20:48:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu Mar 2 20:48:01 2006 Subject: [Numpy-discussion] Matlab page on scipy wiki In-Reply-To: <4407C359.3030601@msg.ucsf.edu> References: <20060210122242.GA21950@sun.ac.za> <4407C359.3030601@msg.ucsf.edu> Message-ID: Ah, right you are. Fixed. Thanks. I guess the python version could also be written a[:5]. --bb On 3/3/06, Sebastian Haase wrote: > > Hi, > I noted on > http://www.scipy.org/Wiki/NumPy_for_Matlab_Users > > matlab a(1:5,:) > numpy a[0:4] or a[0:4,:] a[0:4] or a[0:4,:] > the first five rows of a > > I think this is wrong! > > in numpy it would be: a[:5] (or a[0:5] ) for the first five elements > > To the best of my knowledge (I have never used Matlab myself!) this is > one of the biggest points of confusion for Matlab users !! > WHY DOES a[4:6] NOT INCLUDE THE ELEMENTS 4,5 *AND* 6 ??? > > The only explanation I have is that its > a) like a C/C++-for-loop (for(i=4;i<6;i++) --> note the '<' (not '<=') > b) it then also always is true that: "last" minus first is equal to > number-of-elements (example: 6-4 = 2) > c) it turns out from experience that this convention WILL save you lots > of '-1' and '+1' in your code (actually almost all of them) > [if I see a "+1" in a Matlab person's numpy code I can almost always > guess that he/she made a mistake !] > > Maybe this paragraph could be added to the wiki ... > > Thanks for this wiki page - I think I looks great > > - Sebastian Haase > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vidar+list at 37mm.no Fri Mar 3 00:29:06 2006 From: vidar+list at 37mm.no (Vidar Gundersen) Date: Fri Mar 3 00:29:06 2006 Subject: [Numpy-discussion] Matlab page on scipy wiki In-Reply-To: <4407C359.3030601@msg.ucsf.edu> (Sebastian Haase's message of "Thu, 02 Mar 2006 20:17:29 -0800") References: <20060210122242.GA21950@sun.ac.za> <4407C359.3030601@msg.ucsf.edu> Message-ID: ===== Original message from Sebastian Haase | 3 Mar 2006: > I noted on > http://www.scipy.org/Wiki/NumPy_for_Matlab_Users there is one other difference, which i think is not so easily seen, and not noted on the above mentioned page: a(:) in Matlab, will flatten the array by following each column, whereas a.flatten() in NumPy will walk through the matrix like we read comics, putting each row after each other: In [1]: a = arange(9).reshape(3,-1) In [2]: a Out[2]: array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) In [3]: a.flatten() Out[3]: array([0, 1, 2, 3, 4, 5, 6, 7, 8]) In [4]: a.flatten(1) # same output as a(:) in Matlab Out[4]: array([0, 3, 6, 1, 4, 7, 2, 5, 8]) the user have to be aware of this when reshaping matrices also. I've updated my similar Matlab-Python reference to cover NumPy: http://37mm.no/mpy/matlab-numpy.html (example plot graphics is not yet included in the HTML version.) See other command equivalents, like IDL or R commands in NumPy, here: http://37mm.no/matlab-python-xref.html Vidar___ From gerard.vermeulen at grenoble.cnrs.fr Fri Mar 3 00:45:01 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Fri Mar 3 00:45:01 2006 Subject: [Numpy-discussion] polynomial subtraction buglet in 0.9.6.2193 Message-ID: <20060303094406.1d8b2a5e.gerard.vermeulen@grenoble.cnrs.fr> I found the following buglet in the polynomial subtraction of numpy-0.9.6.2193: diff -u polynomial.py.dtype polynomial.py --- polynomial.py.dtype 2006-03-03 09:34:40.000000000 +0100 +++ polynomial.py 2006-03-03 09:36:29.000000000 +0100 @@ -265,10 +265,10 @@ if diff == 0: return a1 - a2 elif diff > 0: - zr = NX.zeros(diff, a1) + zr = NX.zeros(diff, a1.dtype) val = NX.concatenate((zr, a1)) - a2 else: - zr = NX.zeros(abs(diff), a2) + zr = NX.zeros(abs(diff), a2.dtype) val = a1 - NX.concatenate((zr, a2)) if truepoly: val = poly1d(val) Gerard From oliphant.travis at ieee.org Fri Mar 3 08:28:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 3 08:28:02 2006 Subject: [Numpy-discussion] Numpy licenses In-Reply-To: <200603030423.k234NgCD004844@oobleck.astro.cornell.edu> References: <20060302170802.597DD88A09@sc8-sf-spam1.sourceforge.net> <200603030423.k234NgCD004844@oobleck.astro.cornell.edu> Message-ID: <44086E3F.9020302@ieee.org> Joe Harrington wrote: >> Travis wrote numpy and he can put any license he wants on that. >> > > Joe is basically right. I started from the Numeric code and then changed most of it at least a little bit :-), but there are still recognizable chunks from the old Numeric code base. Then, I borrowed mostly ideas from Numarray except for the _sortmodule.c which I pretty-much grabbed whole-cloth. So, I was always under the impression that I was modifying already produced code. The license should continue to be similar to what it was. -Travis From tim.hochberg at cox.net Fri Mar 3 09:24:10 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 09:24:10 2006 Subject: [Numpy-discussion] Truth values of arrays: Message-ID: <44087B9E.5020903@cox.net> I'm not a fan of the following behaviour, which I just noticed: >>> a = arange(1); b = arange(1) >>> cmp(a,b) 0 >>> not (a == b) False My feeling is that only shape-() arrays should be usable as truth values. Allowing size-1 arrays to be compared as well is just adding a weird, not particularly useful, corner case that's going to bite someone eventually. -tim From mfmorss at aep.com Fri Mar 3 09:26:05 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Fri Mar 3 09:26:05 2006 Subject: [Numpy-discussion] Re: problems installing numpy on AIX In-Reply-To: <440870AF.5000308@ieee.org> Message-ID: Well, thank you Travis, that was most exceedingly helpful! With that, numpy installs, and python imports it without complaint. Next: SciPy. Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To mfmorss at aep.com 03/03/2006 11:37 cc AM Subject Re: [Numpy-discussion] Re: problems installing numpy on AIX mfmorss at aep.com wrote: > (See attached file: Numpy_install.txt) > > Try changing the line 371 in numerictypes.py to _unicodesize = array('u','U1').itemsize and see what that produces... -Travis From tim.hochberg at cox.net Fri Mar 3 09:39:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 09:39:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4407881A.6090108@ee.byu.edu> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407881A.6090108@ee.byu.edu> Message-ID: <44087F21.1000203@cox.net> Travis Oliphant wrote: > David M. Cooke wrote: > >> I couldn't leave it at that :-), so I wrote my bytecode idea up. >> >> >> > I like what you've done. It makes me wonder what could be done with > just a Python-level solution (that might be easier to generalize to > more data-types and operators). I tried hand coding stuff at the pure Python level to see how close I could come to David's results. I performed the operations in chunks (larger chunks than he did so that Python overhead didn't kill me) For the second case he presents (2*a+3*b), I managed to get a 70% speedup. While not negligible, it's a far cry from the 180% speedup I saw with David's code for the same result. So while doing things at the Python level would certainly be easier, the payoff isn't likely to be nearly as big. > You've definitely given us all some great stuff to play with. Indeed. From oliphant at ee.byu.edu Fri Mar 3 09:54:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 3 09:54:05 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <44087F21.1000203@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407881A.6090108@ee.byu.edu> <44087F21.1000203@cox.net> Message-ID: <44088298.7020606@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> David M. Cooke wrote: >> >>> I couldn't leave it at that :-), so I wrote my bytecode idea up. >>> >>> >>> >> I like what you've done. It makes me wonder what could be done with >> just a Python-level solution (that might be easier to generalize to >> more data-types and operators). > > > I tried hand coding stuff at the pure Python level to see how close I > could come to David's results. I performed the operations in chunks > (larger chunks than he did so that Python overhead didn't kill me) For > the second case he presents (2*a+3*b), I managed to get a 70% speedup. > While not negligible, it's a far cry from the 180% speedup I saw with > David's code for the same result. So while doing things at the Python > level would certainly be easier, the payoff isn't likely to be nearly > as big. That's good information. Still, it might be useful for more general-operators and data-types to have something that does essentially this in Python only. Then, if you have a simpler expression with a few select data-types you could have the big-speed up with something more specific. -Travis From tim.hochberg at cox.net Fri Mar 3 10:20:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 10:20:01 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> Message-ID: <44088897.6070802@cox.net> David M. Cooke wrote: >Tim Hochberg writes: > > > >>David M. Cooke wrote: >> >> > >It's in. Check the setup.py -- I left some gcc'isms in there >(-funroll-loops). That should be conditional on the compiler. > > I does whine about that, but it ignores it, so I did to ;). The misplaced declaration of dims I mentioned earlier was the real culprit. >>This is probably thinking two far ahead, but an interesting thing to >>try would be adding conditional expressions: >> >>c = evaluate("(2*a + b) if (a > b) else (2*b + a)") >> >>If that could be made to work, and work fast, it would save both >>memory and time in those cases where you have to vary the computation >>based on the value. At present I end up computing the full arrays for >>each case and then choosing which result to use based on a mask, so it >>takes three times as much space as doing it element by element. >> >> > >It's on my list of things to think about, along with implementing >reductions (sum, in particular). It'd probably look more like > >c = evaluate("where(a > b, 2*a+b, 2*b+a)") > >because of the vector nature of the code. Doing the "if" elementwise >would mean the bytecode program would have to be run for each element, >instead of on a vector of some fixed length. > > That makes sense. One thought I had with respect to the various numpy functions (sin, cos, pow, etc) was to just have the bytecodes: call_unary_function, function_id, store_in, source call_binary_function, function_id, store_in, source1, source2 call_trinary_function, function_id, store_in, source1, source2, source3 Then just store pointers to the functions in relevant tables. In it's most straightforward form, you'd need 6 character chunks of bytecode instead of four. However, if that turns out to slow everything else down I think it could be packed down to 4 again. The function_ids could probably be packed into the opcode (as long as we stay below 200 or so functions, which is probably safe), the other way to pack things down is to require that one of the sources for trinary functions is always a certain register (say register-0). That requires a bit more cleverness at the compiler level, but is probably feasible. OT, stupid subversion question: where do I find the sandbox or whereever this got checked in? -tim From ndarray at mac.com Fri Mar 3 10:36:01 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 3 10:36:01 2006 Subject: [Numpy-discussion] Truth values of arrays: In-Reply-To: <44087B9E.5020903@cox.net> References: <44087B9E.5020903@cox.net> Message-ID: On 3/3/06, Tim Hochberg wrote: > ... > My feeling is that only shape-() arrays should be usable as truth > values. Allowing size-1 arrays to be compared as well is just adding a > weird, not particularly useful, corner case that's going to bite someone > eventually. I agree, but size-1 arrays are also special with respect to broadcasting where they behave much like scalars: >>> x = arange(5) >>> x + [1] array([1, 2, 3, 4, 5]) From tim.hochberg at cox.net Fri Mar 3 10:36:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 10:36:06 2006 Subject: [Numpy-discussion] Truth values of arrays: In-Reply-To: References: <44087B9E.5020903@cox.net> Message-ID: <44088C83.3090703@cox.net> Sasha wrote: >On 3/3/06, Tim Hochberg wrote: > > >> ... >>My feeling is that only shape-() arrays should be usable as truth >>values. Allowing size-1 arrays to be compared as well is just adding a >>weird, not particularly useful, corner case that's going to bite someone >>eventually. >> >> > >I agree, but size-1 arrays are also special with respect to >broadcasting where they behave much like scalars: > > > >>>>x = arange(5) >>>>x + [1] >>>> >>>> >array([1, 2, 3, 4, 5]) > > True, but there's a good reason for that in that there's really not a good way to say "broadcast on this dimension", so being of dimension 1 has to stand in for that. There are plenty of good ways to spell compare the first (and only) element of this array however. The obvious being (a[0] References: <20060210122242.GA21950@sun.ac.za> <4407C359.3030601@msg.ucsf.edu> Message-ID: <44089706.2020700@noaa.gov> Bill Baxter wrote: > Ah, right you are. Fixed. Thanks. > I guess the python version could also be written a[:5]. I've just added that -- it's a very useful idiom. by the way, to prevent [:5] from being interpreted as a link by the Wiki software, I needed to wrap it in ``, to make it look like code. I think that's a good idea for all the code in that table. Maybe we can go in a bit at a time and add the ``. I did a few lines near the one I was working on anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tim.hochberg at cox.net Fri Mar 3 12:13:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 12:13:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> Message-ID: <4408A308.3010003@cox.net> FYI, In my local copy of numexpr, I added a few unary functions just to see how it would be. Pretty easy reall. I think binary functions will also be easy. Trinary functions (e.g., where) will be a bit harder unless we move to 5 byte opcodes. At present I'm not doing anything fancy like looking up functions in a table, I'm just giving each it's own opcodes. If the code gets big enough that it starts falling out of the cache, we might consider that, but adding opcodes is simpler. I also fixed a couple places where constant folding seemed to be not completely working. Once I figure out where the sandbox is I'll upload the changes.... Expression: 2*a+(cos(3)+5)*sin(b) numpy: 0.224821311088 numexpr: 0.112924547673 Speed-up of numexpr over numpy: 1.99089848683 -tim From robert.kern at gmail.com Fri Mar 3 12:24:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 3 12:24:02 2006 Subject: [Numpy-discussion] Re: Numpy and PEP 343 In-Reply-To: <44088897.6070802@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: Tim Hochberg wrote: > OT, stupid subversion question: where do I find the sandbox or > whereever this got checked in? http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/numexpr/ Our sandbox is actually scipy.sandbox. Uncomment out the appropriate line in Lib/sandbox/setup.py to build. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From cookedm at physics.mcmaster.ca Fri Mar 3 12:59:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 3 12:59:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <44088897.6070802@cox.net> (Tim Hochberg's message of "Fri, 03 Mar 2006 11:19:03 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: Tim Hochberg writes: > That makes sense. One thought I had with respect to the various numpy > functions (sin, cos, pow, etc) was to just have the bytecodes: > > call_unary_function, function_id, store_in, source > call_binary_function, function_id, store_in, source1, source2 > call_trinary_function, function_id, store_in, source1, source2, source3 > > Then just store pointers to the functions in relevant tables. In it's > most straightforward form, you'd need 6 character chunks of bytecode > instead of four. However, if that turns out to slow everything else > down I think it could be packed down to 4 again. The function_ids > could probably be packed into the opcode (as long as we stay below 200 > or so functions, which is probably safe), the other way to pack things > down is to require that one of the sources for trinary functions is > always a certain register (say register-0). That requires a bit more > cleverness at the compiler level, but is probably feasible. That's along the lines I'm thinking of. It seems to me that if evaluating the function requires a function call (and not an inlined machine instruction like the basic ops), then we may as well dispatch like this (plus, it's easier :). This could also allow for user extensions. Binary and trinary (how many of those do we have??) could maybe be handled by storing the extra arguments in a separate array. I'm going to look at adding more smarts to the compiler, too. Got a couple books on them :-) Different data types could be handled by separate input arrays, and a conversion opcode ('int2float', say). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From aisaac at american.edu Fri Mar 3 13:02:01 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri Mar 3 13:02:01 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: References: <4406400E.6080600@ee.byu.edu> <4406F52E.8000603@sympatico.ca> Message-ID: On Fri, 3 Mar 2006, Bill Baxter apparently wrote: > I'll just comment on the presence of asarray, asanyarray, > and asmatrix for type conversions. By the way, I believe asmatrix is currently not documented in Guide to NumPy. Cheers, Alan Isaac From tim.hochberg at cox.net Fri Mar 3 13:36:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 13:36:01 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: <4408B679.8080608@cox.net> David M. Cooke wrote: >Tim Hochberg writes: > > > >>That makes sense. One thought I had with respect to the various numpy >>functions (sin, cos, pow, etc) was to just have the bytecodes: >> >>call_unary_function, function_id, store_in, source >>call_binary_function, function_id, store_in, source1, source2 >>call_trinary_function, function_id, store_in, source1, source2, source3 >> >>Then just store pointers to the functions in relevant tables. In it's >>most straightforward form, you'd need 6 character chunks of bytecode >>instead of four. However, if that turns out to slow everything else >>down I think it could be packed down to 4 again. The function_ids >>could probably be packed into the opcode (as long as we stay below 200 >>or so functions, which is probably safe), the other way to pack things >>down is to require that one of the sources for trinary functions is >>always a certain register (say register-0). That requires a bit more >>cleverness at the compiler level, but is probably feasible. >> >> > >That's along the lines I'm thinking of. It seems to me that if >evaluating the function requires a function call (and not an inlined >machine instruction like the basic ops), then we may as well dispatch >like this (plus, it's easier :). This could also allow for user >extensions. > Hmm. If we are going to allow user extensions, we should think about this now. I was thinking of just putting all of the functions in a big table and doing a lookup based on bytecode. I'm not sure how this should work if we are allowing user extensions though. >Binary and trinary (how many of those do we have??) > I'm sure there are more that I'm not thinking of, but the main function that I am concerned with is arctan2. For trinary, the main one is which. > could >maybe be handled by storing the extra arguments in a separate array. > > If there really are only a few functions, we could give them their own opcodes. That means binary function still fit in 4 characters without a problem. That is: OP_ATAN2 R0, R1, R2 would compute: R0[:] = arctan2(R1, R2) Trinary could be defined as using their input as one of the arguments. This would sometimes require an extra copy and some cleverness in the compiler, but I think it would be reasonable. Thus OP_WHICH R0, R1, R2 would compute: R0[:] = which(R0, R1, R2) since the first argument of 'which' is often thrown away anyway, for instance 'which (a < 1, b, c)', the extra copy could often be avoided if we're smart. This might be a bad idea if there turns out to be several more trinary functions that I'm not thinking of or if there are quaternary functions we want to support. (I can't think of any though). >I'm going to look at adding more smarts to the compiler, too. Got a >couple books on them :-) > > Do you have any recomendations? >Different data types could be handled by separate input arrays, and a >conversion opcode ('int2float', say). > > That'd be cool. At some point I want to handle complex, but I'm going to wait till the main issues with floats shake themselves out. -tim From tim.hochberg at cox.net Fri Mar 3 13:41:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 13:41:05 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4408B679.8080608@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408B679.8080608@cox.net> Message-ID: <4408B7C7.2040304@cox.net> Tim Hochberg wrote: >> > If there really are only a few functions, we could give them their own > opcodes. That means binary function still fit in 4 characters without > a problem. That is: > > OP_ATAN2 R0, R1, R2 > > would compute: > R0[:] = arctan2(R1, R2) > > Trinary could be defined as using their input as one of the arguments. > This would sometimes require an extra copy and some cleverness in the > compiler, but I think it would be reasonable. Thus > > OP_WHICH R0, R1, R2 > > would compute: > R0[:] = which(R0, R1, R2) > > since the first argument of 'which' is often thrown away anyway, for > instance 'which (a < 1, b, c)', the extra copy could often be avoided > if we're smart. This might be a bad idea if there turns out to be > several more trinary functions that I'm not thinking of or if there > are quaternary functions we want to support. (I can't think of any > though). > Then again, it might be better to just do a simple, general solution to this case and pass in the arguments in an array as you say. The above might be pointless overengineering. -tim From tim.hochberg at cox.net Fri Mar 3 14:36:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 14:36:04 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: <4408C4B5.4090503@cox.net> David M. Cooke wrote: >Tim Hochberg writes: > > > >>That makes sense. One thought I had with respect to the various numpy >>functions (sin, cos, pow, etc) was to just have the bytecodes: >> >>call_unary_function, function_id, store_in, source >>call_binary_function, function_id, store_in, source1, source2 >>call_trinary_function, function_id, store_in, source1, source2, source3 >> >>Then just store pointers to the functions in relevant tables. In it's >>most straightforward form, you'd need 6 character chunks of bytecode >>instead of four. However, if that turns out to slow everything else >>down I think it could be packed down to 4 again. The function_ids >>could probably be packed into the opcode (as long as we stay below 200 >>or so functions, which is probably safe), the other way to pack things >>down is to require that one of the sources for trinary functions is >>always a certain register (say register-0). That requires a bit more >>cleverness at the compiler level, but is probably feasible. >> >> > >That's along the lines I'm thinking of. It seems to me that if >evaluating the function requires a function call (and not an inlined >machine instruction like the basic ops), then we may as well dispatch >like this (plus, it's easier :). This could also allow for user >extensions. Binary and trinary (how many of those do we have??) could >maybe be handled by storing the extra arguments in a separate array. > > We may want to reconsider this at least partially. I tried implementing a few 1-argument functions two way. First as a table lookup and second using a dedicated opcode. The first gave me a speedup of almost 60%, but the latter gave me a speedup of 100%. The difference suprised me, but I suspect it's do to the fact that the x86 supports some functions directly, so the function call gets optimized away for sin and cos just as it does for +-*/. That implies that some functions should get there own opcodes, while others are not worthy. This little quote from wikipedia lists the functions we would want to give there own opcodes to: x86 (since the 80486DX processor) assembly language includes a stack based floating point unit which can perform addition, subtraction, negation, multiplication, division, remainder, square roots, integer truncation, fraction truncation, and scale by power of two. The operations also include conversion instructions which can load or store a value from memory in any of the following formats: Binary coded decimal, 32-bit integer, 64-bit integer, 32-bit floating point, 64-bit floating point or 80-bit floating point (upon loading, the value is converted to the currently floating point mode). The x86 also includes a number of transcendental functions including sine, cosine, tangent, arctangent, exponentiation with the base 2 and logarithms to bases 2, 10, or e . So, that's my new proposal: some functions get there own opcodes (sin, cos, ln, log10, etc), while others get shunted to table lookup (not sure what's in that list yet, but I'm sure there's lots). -tim >I'm going to look at adding more smarts to the compiler, too. Got a >couple books on them :-) > >Different data types could be handled by separate input arrays, and a >conversion opcode ('int2float', say). > > > From oliphant at ee.byu.edu Fri Mar 3 14:42:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 3 14:42:05 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4408C4B5.4090503@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: <4408C616.5040202@ee.byu.edu> Tim Hochberg wrote: > We may want to reconsider this at least partially. I tried > implementing a few 1-argument functions two way. First as a table > lookup and second using a dedicated opcode. The first gave me a > speedup of almost 60%, but the latter gave me a speedup of 100%. The > difference suprised me, but I suspect it's do to the fact that the x86 > supports some functions directly, so the function call gets optimized > away for sin and cos just as it does for +-*/. That implies that some > functions should get there own opcodes, while others are not worthy. > This little quote from wikipedia lists the functions we would want to > give there own opcodes to: > > x86 (since the 80486DX processor) assembly language includes a stack > based floating point unit which can perform addition, subtraction, > negation, multiplication, division, remainder, square roots, integer > truncation, fraction truncation, and scale by power of two. The > operations also include conversion instructions which can load or > store a value from memory in any of the following formats: Binary > coded decimal, 32-bit integer, 64-bit integer, 32-bit floating > point, 64-bit floating point or 80-bit floating point (upon loading, > the value is converted to the currently floating point mode). The > x86 also includes a number of transcendental functions including > sine, cosine, tangent, arctangent, exponentiation with the base 2 > and logarithms to bases 2, 10, or e > > . > > > > So, that's my new proposal: some functions get there own opcodes (sin, > cos, ln, log10, etc), while others get shunted to table lookup (not > sure what's in that list yet, but I'm sure there's lots). > For the same reason, I think these same functions should get their own ufunc loops instead of using the default loop with function pointers. Thanks for providing this link. It's a useful list. -Travis From cookedm at physics.mcmaster.ca Fri Mar 3 14:45:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 3 14:45:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4408C4B5.4090503@cox.net> (Tim Hochberg's message of "Fri, 03 Mar 2006 15:35:33 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: Tim Hochberg writes: >> > We may want to reconsider this at least partially. I tried > implementing a few 1-argument functions two way. First as a table > lookup and second using a dedicated opcode. The first gave me a > speedup of almost 60%, but the latter gave me a speedup of 100%. The > difference suprised me, but I suspect it's do to the fact that the x86 > supports some functions directly, so the function call gets optimized > away for sin and cos just as it does for +-*/. That implies that some > functions should get there own opcodes, while others are not worthy. Yeah, I was going to stare at the assembly output from gcc before deciding. Note that some functions that are supported directly by the CPU may still be a function call because they aren't good enough. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Fri Mar 3 15:08:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 3 15:08:06 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: (David M. Cooke's message of "Fri, 03 Mar 2006 17:44:28 -0500") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: cookedm at physics.mcmaster.ca (David M. Cooke) writes: > Tim Hochberg writes: >>> >> We may want to reconsider this at least partially. I tried >> implementing a few 1-argument functions two way. First as a table >> lookup and second using a dedicated opcode. The first gave me a >> speedup of almost 60%, but the latter gave me a speedup of 100%. The >> difference suprised me, but I suspect it's do to the fact that the x86 >> supports some functions directly, so the function call gets optimized >> away for sin and cos just as it does for +-*/. That implies that some >> functions should get there own opcodes, while others are not worthy. > > Yeah, I was going to stare at the assembly output from gcc before > deciding. > > Note that some functions that are supported directly by the CPU may > still be a function call because they aren't good enough. ... and a quick test shows me that with gcc 4 under linux, only sqrt is inlined (and it does a test for NaN (I think) and calls the C library sqrt on that). On PowerPC under OS X, even sqrt is a function call. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Fri Mar 3 15:45:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 3 15:45:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: <4408D4FA.6050205@ee.byu.edu> David M. Cooke wrote: >cookedm at physics.mcmaster.ca (David M. Cooke) writes: > > >>Yeah, I was going to stare at the assembly output from gcc before >>deciding. >> >>Note that some functions that are supported directly by the CPU may >>still be a function call because they aren't good enough. >> >> > >... and a quick test shows me that with gcc 4 under linux, only sqrt >is inlined (and it does a test for NaN (I think) and calls the C >library sqrt on that). On PowerPC under OS X, even sqrt is a function >call. > > There were some posts earlier showing that numarray had faster sin(x) functions on an INTEL platform (with a particular compiler that I'm not sure was specified). I'm pretty sure the difference was the function call versus the inline sin computation that the compiler used. I bet it depends on the compiler. But, you bring up a good point that the on-chip provided functions might not be "math-lib" worthy according to different compilers. It would be nice if there was still some way to use them though letting the user decide if they were "good enough" -Travis From zpincus at stanford.edu Fri Mar 3 17:29:16 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri Mar 3 17:29:16 2006 Subject: [Numpy-discussion] numpy.linalg prevents use of scipy.linalg? Message-ID: Hi folks, I can't seem to use any of the functions in scipy.linalg because numpy defines its own linalg which shadows that of scipy! Specifically, scipy.linalg defines 'norm' (in linalg/basic.py), and numpy doesn't. (This among other differences, I assume.) In [1]: import scipy In [2]: scipy.linalg.norm AttributeError: 'module' object has no attribute 'norm' In [3]: scipy.linalg.__file__ Out[3]: '[...]/python2.4/site-packages/numpy/linalg/__init__.pyc' ^^^^^^^ Now, look what happens when we import scipy.linalg directly: In [1]: import scipy.linalg In [2]: scipy.linalg.norm Out[2]: In [3]: scipy.linalg.__file__ Out[3]: '[...]/python2.4/site-packages/scipy/linalg/__init__.pyc' ^^^^^^^ Something needs to be done to fix this -- but what? Scipy historically imports * from numpy. Scipy historically has a linalg module. The new thing is that numpy has a linalg module, too, which is loaded by default into the numpy namespace. (Compared to Numeric's LinearAlgebra module.) The only thing I can think of is to fold all of scipy.linalg into numpy.linalg, and remove the former. This provides for backwards compatibility for everyone, but perhaps a bit of work. Zach From robert.kern at gmail.com Fri Mar 3 17:49:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 3 17:49:01 2006 Subject: [Numpy-discussion] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: References: Message-ID: Zachary Pincus wrote: > Hi folks, > > I can't seem to use any of the functions in scipy.linalg because > numpy defines its own linalg which shadows that of scipy! I believe I've already fixed this. http://projects.scipy.org/scipy/scipy/changeset/1617 What version of scipy are you using? Before reporting a bug, please make sure it exists in the current SVN revision. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From schofield at ftw.at Sat Mar 4 05:15:04 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat Mar 4 05:15:04 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: <44096620.8080505@gmail.com> References: <44096620.8080505@gmail.com> Message-ID: <440992C9.6090801@ftw.at> Robert Kern wrote: > Zachary Pincus wrote: > >>> What version of scipy are you using? Before reporting a bug, please >>> make sure it >>> exists in the current SVN revision. >>> >> Sorry I didn't include the version (0.4.6, by the way) in my first >> email. My oversight. >> >> As to your latter statement, are you sure about that? You're only >> interested in getting bug reports from people who rebuild scipy from >> the svn every night, or who have tracked down the problem to the line >> so they know where in the svn to see if it's fixed? >> > > No, I don't expect everyone to rebuild scipy every day. However, when you think > you've found a bug, it is standard procedure for you to build the most recent > version to test it and see if the bug is still there. So yes, I do expect people > to build from SVN when they're about to report a bug, but no more often. > I don't expect this. I think a bug report against the latest released version is better than no bug report. If we think we've seen and fixed the bug in SVN, it takes us less time to reply "Hmmm, check this with the latest SVN" than it would take Zachary or another would-be bug reporter to check out and build the whole tree from SVN, just on the off-chance we've fixed it since then. This lowers the barrier to entry for helpers. Reading bug reports about the current released version can also be a helpful reminder to us of which bugs are still outstanding in the most recent release. -- Ed From tim.hochberg at cox.net Sat Mar 4 06:59:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 4 06:59:04 2006 Subject: [Fwd: Re: [Numpy-discussion] Numpy and PEP 343] Message-ID: <4409AB20.7060201@cox.net> Check this out: Expression: where(0.1*a > arctan2(a, b), 2*a, arctan2(a,b)) numpy: 0.706635284652 Skipping weave timing numexpr: 0.19127785794 Speed-up of numexpr over numpy: 3.69428689898 270% speed up! Pretty cool. Unfortunately, my implementation of two and three argument functions is still buggy if one of the arguments is a constant. I'm not sure what the best tactic is to deal with that. My strategy for dealing with opcodes that require more than three arguments (just 'where' at present) was to just use two opcode blocks. The second opcode block had the opcode OP_NOOP (which does nothing), but the first opcode steals arguments from it. It's pretty simple and more or less infinitely expandable. I'm off to bed now, perhaps the answer to the constant dilemma will be clear in the morning. [next morning] OK, I know how to fix it, I just need to scrape together a few minutes to do it. -tim From tim.hochberg at cox.net Sat Mar 4 07:00:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 4 07:00:03 2006 Subject: [Fwd: [Fwd: Re: [Numpy-discussion] Numpy and PEP 343]] Message-ID: <4409AB3A.8090908@cox.net> Check this out: Expression: where(0.1*a > arctan2(a, b), 2*a, arctan2(a,b)) numpy: 0.706635284652 Skipping weave timing numexpr: 0.19127785794 Speed-up of numexpr over numpy: 3.69428689898 270% speed up! Pretty cool. Unfortunately, my implementation of two and three argument functions is still buggy if one of the arguments is a constant. I'm not sure what the best tactic is to deal with that. My strategy for dealing with opcodes that require more than three arguments (just 'where' at present) was to just use two opcode blocks. The second opcode block had the opcode OP_NOOP (which does nothing), but the first opcode steals arguments from it. It's pretty simple and more or less infinitely expandable. I'm off to bed now, perhaps the answer to the constant dilemma will be clear in the morning. [next morning] OK, I know how to fix it, I just need to scrape together a few minutes to do it. -tim From oliphant.travis at ieee.org Sat Mar 4 09:08:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Mar 4 09:08:04 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: <440992C9.6090801@ftw.at> References: <44096620.8080505@gmail.com> <440992C9.6090801@ftw.at> Message-ID: <4409C944.1020806@ieee.org> Ed Schofield wrote: >> No, I don't expect everyone to rebuild scipy every day. However, when you think >> you've found a bug, it is standard procedure for you to build the most recent >> version to test it and see if the bug is still there. So yes, I do expect people >> to build from SVN when they're about to report a bug, but no more often. >> >> > > I don't expect this. I think a bug report against the latest released > version is better than no bug report. If we think we've seen and fixed > the bug in SVN, it takes us less time to reply "Hmmm, check this with > the latest SVN" than it would take Zachary or another would-be bug > reporter to check out and build the whole tree from SVN, just on the > off-chance we've fixed it since then. This lowers the barrier to entry > for helpers. Reading bug reports about the current released version can > also be a helpful reminder to us of which bugs are still outstanding in > the most recent release. > > I don't want potential bug reports not to get filed, either. This is another advantage of the ticket system on the trac pages. The bug-reporter could just check that page to see if the problem has been fixed. Alternatively, the bug-reporter can do a search through the newsgroup to see if the problem has been talked about and reported as fixed. I think requiring all bug-reporters to build from SVN is a bit of a barrier that we don't want to push at this stage. Especially, when there may be many little bugs lurking from the transition. -Travis From robert.kern at gmail.com Sat Mar 4 11:45:13 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat Mar 4 11:45:13 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: <4409C944.1020806@ieee.org> References: <44096620.8080505@gmail.com> <440992C9.6090801@ftw.at> <4409C944.1020806@ieee.org> Message-ID: Travis Oliphant wrote: > I don't want potential bug reports not to get filed, either. This is > another advantage of the ticket system on the trac pages. The > bug-reporter could just check that page to see if the problem has been > fixed. Alternatively, the bug-reporter can do a search through the > newsgroup to see if the problem has been talked about and reported as > fixed. > > I think requiring all bug-reporters to build from SVN is a bit of a > barrier that we don't want to push at this stage. Especially, when > there may be many little bugs lurking from the transition. I worded myself poorly. I don't want people to forgo entering bugs in the tracker if they can't find the time to rebuild from SVN. But I do want them to try! -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From tim.hochberg at cox.net Sun Mar 5 16:30:53 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 5 16:30:53 2006 Subject: [Numpy-discussion] numexpr In-Reply-To: <44095626.50604@ieee.org> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> <4408D4FA.6050205@ee.byu.edu> <44092A05.5080905@cox.net> <44095626.50604@ieee.org> Message-ID: <440A6583.4040609@cox.net> I just commited a bunch of changes to numexpr in the sandbox (BTW, thanks to Robert Kern for pointing me to the right location for that). At this point, there's a fairly complete framework for evaluationg numeric expressions (as long as they're not complex, and have unit stride and are all the same size etc, OK maybe not so complete, but potentially very useful nonetheless). A couple basic operations (** and %) need to be filled in as well as a bunch of functions. Still most basic operaions work. A bunch of 2- and 3-arg functions work as well as 'where'. I'll wait to hear from David Cooke to see how he likes the current function framework -- if he likes it, I'll go ahead and fill in the other common functions. It would also be nice to support sum and product as David mentioned, but I haven't though about that at all yet. At some point I'd like to add support for complex numbers as well, but I'd like to let the floating point stuff settle a bit first. -tim From pfpntdfhcz at montgomery.com Sun Mar 5 16:47:40 2006 From: pfpntdfhcz at montgomery.com (pfpntdfhcz) Date: Sun Mar 5 16:47:40 2006 Subject: [Numpy-discussion] Fw: numpy-discussion Message-ID: <000f01c64058$c8ec8000$1a041253@malpikrz008dft> ----- Original Message ----- From: Bass Aileen To: sgnptwk at southtrust.com Sent: Saturday, March 4, 2006 8:21 AM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: wucpxgkjjc.gif Type: image/gif Size: 9405 bytes Desc: not available URL: From tim.hochberg at cox.net Sun Mar 5 16:51:47 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 5 16:51:47 2006 Subject: [Numpy-discussion] numexpr thoughts Message-ID: <440AF97C.5040106@cox.net> 1. David already mentioned opcodes to copy from an integer array into a float register. Another idea would be to have an opcode to copy from a strided array into a register. This would save a copy for noncontiguous matrices and opens up the door to doing broadcasting. I think we'd have to tweak the main interpreter loop a bit, but I think it's doable. Depending how crunched we feel for opcode space, the casting array and this array could overlap (OP_CAST_AND_COPY for instance). 2. Something I noticed while writing tests is that the rearangement of operations for 'a/const' to '(1/const)*a' changes the results slightly (try const=3). I don't think I care -- I just thought I'd point it out. 3. It may simplify things to use copy_c to eliminate a bunch of the extra bytecodes for operating on functions of more than one argument. I need to check the timings on this -- I don't know how much of a speed hit using copy_c will cause. 4. At some point I plan to add complex operations. My original thought was to just duplicate the current, floating-point interpreter for complex numbers. However, if we have opcodes for casting, it might instead make sense to have one interpreter that works with both. This would be more efficient for mixed expressions since we wouldnt' have to case everything complex right away. It looks like there's enough opcode space, although I worry a bit whether making that main switch loop huge is going to slow things down. Probably not, but it's something to keep an eye on. 5. Same thoughts as 4., but for integer operations. Potentially the interpreter could work with longs, doubles and cdoubles, downcasting as appropriate on the way out. It's looking like numexpr could come pretty close to evaluating most numpy expressions if we put enough work into it. None of the required changes look like they should slow down the original, simple, fast case. However we should keep an eye on that as we go along. It's all very cool; I'd like to thank David for providing such a fun toy. -tim From ndarray at mac.com Sun Mar 5 17:02:05 2006 From: ndarray at mac.com (Sasha) Date: Sun Mar 5 17:02:05 2006 Subject: [Numpy-discussion] Tests fail in svn r2195 Message-ID: Reproduced on Linux with python 2.4 and on MacOS X with python 2.3. ====================================================================== ERROR: Check creation from tuples ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 151, in check_tuple h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check creation from tuples ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 151, in check_tuple h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check reading the top fields of a nested array ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 244, in check_access_top_fields h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check reading the nested fields of a nested array (1st level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 262, in check_nested1_acessors h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check access nested descriptors of a nested array (1st level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 310, in check_nested1_descriptor h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check reading the nested fields of a nested array (2nd level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 292, in check_nested2_acessors h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check access nested descriptors of a nested array (2nd level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 318, in check_nested2_descriptor h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: check_access_fields (numpy.core.tests.test_numerictypes.test_read_values_plain_single) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 210, in check_access_fields h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ---------------------------------------------------------------------- Ran 337 tests in 1.131s FAILED (errors=8) From charlesr.harris at gmail.com Sun Mar 5 17:03:34 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun Mar 5 17:03:34 2006 Subject: [Numpy-discussion] Thoughts on an ndarray super-class In-Reply-To: <43FD5914.4060506@ieee.org> References: <43FD5914.4060506@ieee.org> Message-ID: On 2/22/06, Travis Oliphant wrote: > > The bigndarray class is going to disappear (probably in the next release > of NumPy). It was a stop-gap measure as the future of 64-bit fixes in > Python was unclear. Python 2.5 will have removed the 64-bit limitations > that led to the bigndarray and so it will be removed. > > I have been thinking, however, of replacing it with a super-class that > does not define the dimensions or strides. I use arrays a lot as buffers to pass binary data between C programs. At the moment I use the buffer interface, but if the superclass was just as convenient that would do. Mostly I just want simple. Are you thinking of keeping the dtype? Chuck From ndarray at mac.com Mon Mar 6 08:49:01 2006 From: ndarray at mac.com (Sasha) Date: Mon Mar 6 08:49:01 2006 Subject: [Numpy-discussion] Code cleanup patches Message-ID: I would like to thank timl for the code cleanup patches. I've commited the ma.py patch to svn and the other patches seem to be acceptable as well. I assume you used some kind of automated code checking tool to spot the irregularities that you've fixed. It would be nice if you could describe your procedure and the tool that you used somewere on the developers' wiki. I've created a stub at http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines . I assume that for the most part numpy can simply follow general python guidelines, but as more people start contributing code it maybe useful to document some guidelines. From mpi at osc.kiku.dk Mon Mar 6 13:04:02 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Mon Mar 6 13:04:02 2006 Subject: [Numpy-discussion] Is sum() slow ? In-Reply-To: References: Message-ID: Hi, Here are some timings that puzzle me a bit. Sum the two rows of a 2xn matrix, where n is some large number python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x.sum(0)" 10000 loops, best of 3: 36.2 usec per loop python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x[0] + x[1]" 100000 loops, best of 3: 5.35 usec per loop The two calculations are completely equivalent (at least from the numerical results point of view). But using the built-in sum() function is app. 6 times slower. Just for the reference - using Numeric gives python -m timeit -s "from Numeric import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "sum(x)" 100000 loops, best of 3: 7.08 usec per loop Any suggestions to why this might be - or am I doing something wrong here? Thanks // Mads PS. Thanks to everybody that responded to my previous posting on problems and questions regarding rint() and related coding issues. Your replies were very helpful! From cookedm at physics.mcmaster.ca Mon Mar 6 15:00:04 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 6 15:00:04 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440AF97C.5040106@cox.net> (Tim Hochberg's message of "Sun, 05 Mar 2006 07:45:16 -0700") References: <440AF97C.5040106@cox.net> Message-ID: Tim Hochberg writes: > 1. David already mentioned opcodes to copy from an integer array into > a float register. Another idea would be to have an opcode to copy from > a strided array into a register. This would save a copy for > noncontiguous matrices and opens up the door to doing broadcasting. I > think we'd have to tweak the main interpreter loop a bit, but I think > it's doable. Depending how crunched we feel for opcode space, the > casting array and this array could overlap (OP_CAST_AND_COPY for > instance). Hadn't thought about doing strided arrays like that; it should work. > 2. Something I noticed while writing tests is that the rearangement of > operations for 'a/const' to '(1/const)*a' changes the results > slightly (try const=3). I don't think I care -- I just thought I'd > point it out. I don't care either :-) Although it may be worthwile throwing in a compiler option to keep a/const. > 3. It may simplify things to use copy_c to eliminate a bunch of the > extra bytecodes for operating on functions of more than one argument. > I need to check the timings on this -- I don't know how much of a > speed hit using copy_c will cause. I'm implementing another solution: I'm getting rid of individual constant operators, like 'add_c'. Constants will be stored in the temps array, as 128 repetitions of the constant. That way 'add' (for instance) can operate on both vector + vector and vector + scalar: it would take the register numbers for the two vectors for the first one, and for the second case, the scalar argument would be the register number for the repeated constant. Timings on my iBook show this is actually slightly faster. I didn't expect that :-) > 4. At some point I plan to add complex operations. My original thought > was to just duplicate the current, floating-point interpreter for > complex numbers. However, if we have opcodes for casting, it might > instead make sense to have one interpreter that works with both. This > would be more efficient for mixed expressions since we wouldnt' have > to case everything complex right away. It looks like there's enough > opcode space, although I worry a bit whether making that main switch > loop huge is going to slow things down. Probably not, but it's > something to keep an eye on. The way I see it, there should be two machines: one for single precision, and one for double precision (maybe one for longdoubles, but I'm not worrying about that now). That's simple enough. Each machine should work with floats, and the complex form of that float. Potentially, cast opcodes could be used to convert single to double, for instance. I'm still thinking about how to fit that. > 5. Same thoughts as 4., but for integer operations. Potentially the > interpreter could work with longs, doubles and cdoubles, downcasting > as appropriate on the way out. I suppose this could be considered a "reduction" operation: instead of collapsing the array, we're collapsing the precision. > It's looking like numexpr could come pretty close to evaluating most > numpy expressions if we put enough work into it. None of the required > changes look like they should slow down the original, simple, fast > case. However we should keep an eye on that as we go along. It's all > very cool; I'd like to thank David for providing such a fun toy. thanks Right now, here's my thoughts on where to go: 1. I'm rewriting the compiler to be easier to play with. Top things on the list are better register allocation, and common subexpression elimination. 2. Replace individual constants with "constant arrays": repeations of the constant in the vector registers. 3. Reduction. I figure this could be done at the end of the program in each loop: sum or multiply the output register. Downcasting the output could be done here too. 4. Add complex numbers. If it doesn't look really messy, we could add them to the current machine. Otherwise, we could make a machine that works on complex numbers only. 5. Currently, we use a big switch statement. There are ways (taken from Forth) that are better: indirect and direct threading. Unfortunately, it looks the easy way to do these uses GCC's capability to take the address of local labels. I'll add that if I can refactor the machine enough so that both variants can be produced. Have a look at http://www.complang.tuwien.ac.at/anton/vmgen/ which is the virtual machine generator used for gforth (but applicable to other things). I may use this. 6. Replace register numbers in the program with actual pointers to the correct register. That should remove a layer of indirection. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Mon Mar 6 15:00:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 6 15:00:06 2006 Subject: [Numpy-discussion] numexpr In-Reply-To: <440A6583.4040609@cox.net> (Tim Hochberg's message of "Sat, 04 Mar 2006 21:13:55 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> <4408D4FA.6050205@ee.byu.edu> <44092A05.5080905@cox.net> <44095626.50604@ieee.org> <440A6583.4040609@cox.net> Message-ID: Tim Hochberg writes: > I just commited a bunch of changes to numexpr in the sandbox (BTW, > thanks to Robert Kern for pointing me to the right location for that). > > At this point, there's a fairly complete framework for evaluationg > numeric expressions (as long as they're not complex, and have unit > stride and are all the same size etc, OK maybe not so complete, but > potentially very useful nonetheless). A couple basic operations (** > and %) need to be filled in as well as a bunch of functions. Still > most basic operaions work. A bunch of 2- and 3-arg functions work as > well > as 'where'. > > I'll wait to hear from David Cooke to see how he likes the current > function framework -- if he likes it, I'll go ahead and fill in the > other common functions. It would also be nice to support sum and > product as David mentioned, but I haven't though about that at all > yet. Looks good. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From robert.kern at gmail.com Mon Mar 6 15:32:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 6 15:32:02 2006 Subject: [Numpy-discussion] Re: numexpr thoughts In-Reply-To: References: <440AF97C.5040106@cox.net> Message-ID: David M. Cooke wrote: > 5. Currently, we use a big switch statement. There are ways (taken > from Forth) that are better: indirect and direct threading. > Unfortunately, it looks the easy way to do these uses GCC's > capability to take the address of local labels. I'll add that if I > can refactor the machine enough so that both variants can be > produced. Have a look at > http://www.complang.tuwien.ac.at/anton/vmgen/ > which is the virtual machine generator used for gforth (but > applicable to other things). I may use this. Hmmm. If LLVM weren't so huge and such a pain to install, I might recommend looking at using it. It could make a fun experiment, though. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Mon Mar 6 15:39:05 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 6 15:39:05 2006 Subject: [Numpy-discussion] Re: numexpr thoughts In-Reply-To: (Robert Kern's message of "Mon, 06 Mar 2006 17:30:36 -0600") References: <440AF97C.5040106@cox.net> Message-ID: Robert Kern writes: > David M. Cooke wrote: > >> 5. Currently, we use a big switch statement. There are ways (taken >> from Forth) that are better: indirect and direct threading. >> Unfortunately, it looks the easy way to do these uses GCC's >> capability to take the address of local labels. I'll add that if I >> can refactor the machine enough so that both variants can be >> produced. Have a look at >> http://www.complang.tuwien.ac.at/anton/vmgen/ >> which is the virtual machine generator used for gforth (but >> applicable to other things). I may use this. > > Hmmm. If LLVM weren't so huge and such a pain to install, I might recommend > looking at using it. It could make a fun experiment, though. Yeah, I had a look at that. PyPy is using it, so things could be stolen from that. Fortunately, our virtual machine can be simpler than most, because we don't have conditionals or jumps :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From tim.hochberg at cox.net Mon Mar 6 19:24:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 6 19:24:01 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: References: <440AF97C.5040106@cox.net> Message-ID: <440CFCA1.70007@cox.net> David M. Cooke wrote: >1. David already mentioned opcodes to copy from an integer array into >> a float register. Another idea would be to have an opcode to copy from >> a strided array into a register. This would save a copy for >> noncontiguous matrices and opens up the door to doing broadcasting. I >> think we'd have to tweak the main interpreter loop a bit, but I think >> it's doable. Depending how crunched we feel for opcode space, the >> casting array and this array could overlap (OP_CAST_AND_COPY for >> instance). > > > >Hadn't thought about doing strided arrays like that; it should work. > > This opens another interesting can of worms: two stage compilation. If we aren't copying the arrays on the way in, it might make sense to have an abstract opcode OP_COPY_ANY, that copied data from any allowable source. This would be produced during the compilation stage. When numexpr was called, before the loop was executed, OP_COPY_ANY would be replaced depending on the input value. For example: OP_COPY_ANY destreg input would translate to: OP_COPY_INT16ARR destreg input # input is an int array OP_COPY_FLT32SCL destreg input # input is an FP scalar etc. The data would then be copied from the array into a register with appropriate striding and casting. If 'input' was already a contiguous double array, then it could translate into simply setting a pointer as is done now (OP_COPY_SIMPLE perhaps). [SNIP] >Right now, here's my thoughts on where to go: > >1. I'm rewriting the compiler to be easier to play with. Top things > on the list are better register allocation, and common > subexpression elimination. > > Cool. >2. Replace individual constants with "constant arrays": repeations of > the constant in the vector registers. > > I'm already using this trick to a certain extent to reduce the number op func_xy opcodes. I copy the constants into arrays using OP_COPY_C. This sounds like essentially the same thing as what you are doing. >3. Reduction. I figure this could be done at the end of the program in > each loop: sum or multiply the output register. Downcasting the > output could be done here too. > > Do you mean that sum() and friends could only occur as the outermost function. That is: "sum(a+3*b)" would work, but: "where(a, sum(a+3*b), c)" would not? Or am I misunderstanding you here? I don't think I like that limitation if that's the case. I don' think it should be necessary either. >4. Add complex numbers. If it doesn't look really messy, we could add > them to the current machine. Otherwise, we could make a machine that > works on complex numbers only. > > It doesn't seem like it should be bad at all. The slickest thing would be to use to adjacent float buffers for a complex buffer. That would make the compiler a bit more complex, but it would keep the intrepeter simpler as there would only be a single buffer type. All that needs to be supported are the basic operations (+,-,*,/,// and **); comparisons don't work for complex numbers anyway and all of the functions can go through the function pointers since they're slow anyway. The one exception is where, which would be mixed complex and float and should be fast. The other question is integers. There would be some advantages to supporting mixed integer and floating point operations. This adds a larger crop of operators though; The basic ones as above, plus comparisons, plus bitwise operators. >5. Currently, we use a big switch statement. There are ways (taken > from Forth) that are better: indirect and direct threading. > Unfortunately, it looks the easy way to do these uses GCC's > capability to take the address of local labels. I'll add that if I > can refactor the machine enough so that both variants can be > produced. Have a look at > http://www.complang.tuwien.ac.at/anton/vmgen/ > which is the virtual machine generator used for gforth (but > applicable to other things). I may use this. > > Better as in lower overhead? Or better as in simpler to write? The current code is competitive with weave already which would seem to indicate that we're already dominated by the overhead of the FP math, not the big switch statement. I'd only rewrite things if it's going to be simpler to work with. >6. Replace register numbers in the program with actual pointers to the > correct register. That should remove a layer of indirection. > > This would be done at function calling time as well? So this is more two-stage compilation stuff? -tim From mpi at osc.kiku.dk Mon Mar 6 23:16:02 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Mon Mar 6 23:16:02 2006 Subject: [Numpy-discussion] Is sum() slow? Message-ID: Hi, Sorry if this is a double post - unsure if it made it the first time: Here are some timings that puzzle me a bit. Sum the two rows of a 2xn matrix, where n is some large number python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x.sum(0)" 10000 loops, best of 3: 36.2 usec per loop python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x[0] + x[1]" 100000 loops, best of 3: 5.35 usec per loop The two calculations are completely equivalent (at least from the numerical results point of view). But using the built-in sum() function is app. 6 times slower. Just for the reference - using Numeric gives python -m timeit -s "from Numeric import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "sum(x)" 100000 loops, best of 3: 7.08 usec per loop Any suggestions to why this might be - or am I doing something wrong here? Thanks // Mads PS. Thanks to everybody that responded to my previous posting on problems and questions regarding rint() and related coding issues. Your replies were very helpful! From tim.hochberg at cox.net Tue Mar 7 10:35:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 7 10:35:02 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440CFCA1.70007@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> Message-ID: <440DD209.5060900@cox.net> Tim Hochberg wrote: >> 3. Reduction. I figure this could be done at the end of the program in >> each loop: sum or multiply the output register. Downcasting the >> output could be done here too. >> >> > Do you mean that sum() and friends could only occur as the outermost > function. That is: > "sum(a+3*b)" > would work, but: > "where(a, sum(a+3*b), c)" > would not? Or am I misunderstanding you here? I don't think I like > that limitation if that's the case. I don' think it should be > necessary either. OK, I thought about this some more and I think I was mostly wrong. I suspect that reduction does need to happen as the last step. Still it would be nice for "where(a, sum(a+3*b), c)" to work. This could be done by doing the following transformation: a = evaluate("where(a, sum(a+3*b), c)") => temp=evaluate("sum(a+3*b)"); a = evaluate("where(a, temp, c)") I suspect that this this would be fairly easy to do automagically as long as it was at the Python level. That is, numexpr would return a python object that would call the lower level interpreter.numexpr appropriately. This would have some other advantages as well -- it would make it easy to deal with keyword arguments for one. It would also make it easy to do the bytecode rewriting if we decide to go that route. It does add more call overhead, but if that turns out to be we can move stuff into C later. I'm still not excited about summing over the whole output buffer though. That ends up allocating and scanning through a whole extra buffer which may result in a signifigant speed and memory hit for large arrays. Since if we're only doing this on the way out, there should be no problem just allocating a single double (or complex) to do the sum in. On the way in, this could be set to zero or one based on what the last opcode is (sum or product). Then the SUM opcode could simply do something like: BTW, the cleanup of the interpreter looks pretty slick. I'm going to look at timings for using COPY_C versus using add directly and see about reducing the number of opcodes. If this works out OK, the number comparison opcodes could be reduced a lot. Sorry about munging the line endings. -tim From oliphant at ee.byu.edu Tue Mar 7 10:38:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 7 10:38:03 2006 Subject: [Numpy-discussion] Is sum() slow? In-Reply-To: References: Message-ID: <440DD2FA.4060301@ee.byu.edu> Mads Ipsen wrote: >Hi, > >Sorry if this is a double post - unsure if it made it the first time: > >Here are some timings that puzzle me a bit. Sum the two rows of a 2xn >matrix, where n is some large number > >python -m timeit -s "from numpy import array,sum,reshape; x = >array([1.5]*1000); x = reshape(x,(2,500))" "x.sum(0)" >10000 loops, best of 3: 36.2 usec per loop > >python -m timeit -s "from numpy import array,sum,reshape; x = >array([1.5]*1000); x = reshape(x,(2,500))" "x[0] + x[1]" >100000 loops, best of 3: 5.35 usec per loop > > This is probably reasonable. There is overhead in the looping construct (basically what happens is that the first element is copied into the output and then a function called --- which in this case has a loop of size 1 to compute the sum). This is then repeated 500 times. So, you have 500 C-function pointer calls in the first case. In the second case you have basically a single call to the same function where the 500-element loop is done. I'm a little surprised that Numeric is so much faster for this case as you show later. The sum code is actually add.reduce... which uses a generic reduction concept in ufuncs. It has overhead over what you might do using some less general approach. If anyone can figure out how to make the NOBUFFER secion in GenericReduce faster in ufuncobject.c it will be greatly welcomed. Speed improvements are always welcome. -Travis From cookedm at physics.mcmaster.ca Tue Mar 7 10:53:05 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 7 10:53:05 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440DD209.5060900@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> Message-ID: <20060307185127.GA31063@arbutus.physics.mcmaster.ca> On Tue, Mar 07, 2006 at 11:33:45AM -0700, Tim Hochberg wrote: > Tim Hochberg wrote: > > >>3. Reduction. I figure this could be done at the end of the program in > >> each loop: sum or multiply the output register. Downcasting the > >> output could be done here too. > >> > > I'm still not excited about summing over the whole output buffer though. > That ends up allocating and scanning through a whole extra buffer which > may result in a signifigant speed and memory hit for large arrays. Since > if we're only doing this on the way out, there should be no problem just > allocating a single double (or complex) to do the sum in. On the way > in, this could be set to zero or one based on what the last opcode is > (sum or product). Then the SUM opcode could simply do something like: No, no, we'd just sum over the 128 element output vector (mem[0]), and add the result to cumulative sum. That vector should already be in cache, as the last op would put it there. > BTW, the cleanup of the interpreter looks pretty slick. Not finished yet :-) Look for a checkin today (if I have time). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From tim.hochberg at cox.net Tue Mar 7 11:09:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 7 11:09:02 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <20060307185127.GA31063@arbutus.physics.mcmaster.ca> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> Message-ID: <440DDA10.4020904@cox.net> David M. Cooke wrote: >On Tue, Mar 07, 2006 at 11:33:45AM -0700, Tim Hochberg wrote: > > >>Tim Hochberg wrote: >> >> >> >>>>3. Reduction. I figure this could be done at the end of the program in >>>> each loop: sum or multiply the output register. Downcasting the >>>> output could be done here too. >>>> >>>> >>>> >>I'm still not excited about summing over the whole output buffer though. >>That ends up allocating and scanning through a whole extra buffer which >>may result in a signifigant speed and memory hit for large arrays. Since >>if we're only doing this on the way out, there should be no problem just >>allocating a single double (or complex) to do the sum in. On the way >>in, this could be set to zero or one based on what the last opcode is >>(sum or product). Then the SUM opcode could simply do something like: >> >> > >No, no, we'd just sum over the 128 element output vector (mem[0]), and >add the result to cumulative sum. That vector should already be in >cache, as the last op would put it there. > > Ah! Ok then. That's what I was thinking of too. For some reason I thought you were proposing building the whole result vector then summing it. Here's another wrinkle: how do we deal with: >>> a = reshape(arange(9), (3,3)) >>> sum(a) array([ 9, 12, 15]) Just forbid it? For the time being at least? > > >>BTW, the cleanup of the interpreter looks pretty slick. >> >> > >Not finished yet :-) Look for a checkin today (if I have time). > > They didn't seem to have any speed advantage, so I ripped out all the compare with constant opcodes amd used COPY_C instead. I'm probably going to rip out OP_WHERE_XXC and OP_WHERE_XCX depending on the timings there. Should I also kill OP_ADD_C and friends as well while I'm at it? -tim From cookedm at physics.mcmaster.ca Tue Mar 7 12:36:01 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 7 12:36:01 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440DDA10.4020904@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> <440DDA10.4020904@cox.net> Message-ID: <20060307203412.GA31277@arbutus.physics.mcmaster.ca> On Tue, Mar 07, 2006 at 12:08:00PM -0700, Tim Hochberg wrote: > David M. Cooke wrote: > > >On Tue, Mar 07, 2006 at 11:33:45AM -0700, Tim Hochberg wrote: > > > > > >>Tim Hochberg wrote: > >> > >> > >> > >>>>3. Reduction. I figure this could be done at the end of the program in > >>>>each loop: sum or multiply the output register. Downcasting the > >>>>output could be done here too. > >>>> > >>>> > >>>> > >>I'm still not excited about summing over the whole output buffer though. > >>That ends up allocating and scanning through a whole extra buffer which > >>may result in a signifigant speed and memory hit for large arrays. Since > >>if we're only doing this on the way out, there should be no problem just > >>allocating a single double (or complex) to do the sum in. On the way > >>in, this could be set to zero or one based on what the last opcode is > >>(sum or product). Then the SUM opcode could simply do something like: > >> > >> > > > >No, no, we'd just sum over the 128 element output vector (mem[0]), and > >add the result to cumulative sum. That vector should already be in > >cache, as the last op would put it there. > > > > > > Ah! Ok then. That's what I was thinking of too. For some reason I > thought you were proposing building the whole result vector then summing it. > > Here's another wrinkle: how do we deal with: > > >>> a = reshape(arange(9), (3,3)) > >>> sum(a) > array([ 9, 12, 15]) > > > Just forbid it? For the time being at least? Well, I think for now, sum(a) = sum(a.ravel()). Doing differently will mean working with dimensions, which is something else to figure out. > They didn't seem to have any speed advantage, so I ripped out all the > compare with constant opcodes amd used COPY_C instead. I'm probably > going to rip out OP_WHERE_XXC and OP_WHERE_XCX depending on the timings > there. Should I also kill OP_ADD_C and friends as well while I'm at it? Don't bother; the version I'm going to check in already has them gone. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cjw at sympatico.ca Tue Mar 7 14:17:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 7 14:17:03 2006 Subject: [Numpy-discussion] numpy doc strings - a tool to list problem cases Message-ID: <440E0643.40904@sympatico.ca> Some of the docstrings have lines which are too long. Some names have no associated docstring. The effects of this are that, for the long lines, the PyDoc generated documentation is difficult to read on a browser and, for the missing cases, the generated docs are incomplete. The question of how long is too long is a matter of personal choice. I used 78 characters in the little script below that lists problem cases. # to check numpy docs, for long and missing lines import numpy lst= dir(numpy) lnMax= 78 # maximum number characters per line print 'Report of missing docstrings and docstrings with lines that are too long' print for nm in lst: txt= eval('numpy.' + nm + '.__doc__') if txt is None: print nm, 'No docstring' continue lns= txt.split('\n') for lnNo in range(len(lns)): ln= lns[lnNo] if len(ln) > lnMax: print nm, 'Line#:', lnNo, 'Line Length:', len(ln) z= 1 z= 1 Colin W. From cookedm at physics.mcmaster.ca Tue Mar 7 14:59:04 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 7 14:59:04 2006 Subject: [Numpy-discussion] Power optimization branch merged to Numpy trunk Message-ID: Since Tim and I are now fiddling around with numexpr, I've merged what we've got with the power optimization to the Numpy trunk. This includes: * x**2 is _as_fast_ as x*x now * optimization of x**n for x as an array and n as a Python scalar. * x**2, x**(-1), x**0 are optimized to use the new square, reciprocal, and ones_like ufuncs * complex integer powers should be faster Also, PyArray_FromAny has been speeded up when converting Python scalars. A side effect of this change is that the array protocol won't be looked for on subclasses of Python scalars (int, long, float, complex). I don't expect that to be a problem. Some things we hadn't got around to: - integer arrays to a power (currently, these use pow()) - more optimization of integer powers enjoy the speed :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From fred at ucar.edu Tue Mar 7 16:48:03 2006 From: fred at ucar.edu (Fred Clare) Date: Tue Mar 7 16:48:03 2006 Subject: [Numpy-discussion] NumPy/Numeric/PyArray_Check Message-ID: <1d36eb09073891383f5397ac3b191c09@ucar.edu> I have been a long-time Numeric user and have just started to use numpy-0.9.5 on a Mac running OS X Version 10.3.9. I installed NumPy in the same Python that has Numeric 24.2 installed. In using Numeric in my Python codes I have used only "import Numeric" for the import. After having installed "numpy" I replaced "import Numeric" with "import numpy as Numeric". All seems to go O.K. until I call a function in a module that uses Numeric (and not numpy) to create an array in my Python script. When I pass this Numeric array to a function in an extension module (that was compiled using the arrayobject.h from the NumPy distribution) and do a "PyArray_Check" on the Numeric array, it returns false. Is here a way around this? Based on the statement in the "Guide to NumPy" that Numeric and NumPy can be used simultaneously I was hoping that this would work. Fred Clare From hgamboa at gmail.com Wed Mar 8 04:25:03 2006 From: hgamboa at gmail.com (Hugo Gamboa) Date: Wed Mar 8 04:25:03 2006 Subject: [Numpy-discussion] Power optimization branch merged to Numpy trunk In-Reply-To: References: Message-ID: <86522b1a0603080424l7a696a8enfae90393eeed7829@mail.gmail.com> Since you are mentioning some improvements (I have confirmed them!) I would like to ask what code are you using for benchmarking numpy. I wanted to know what code was faster: sum(|x|)/N or sqrt(sum(x**2))/sqrt(N) I wrote a timeit test and discovered that using the function dot(x,x) to compute sum(x**2) gives the best result and is one order of magnitude faster than sum(|x|). I found that sum and absolute are approx. 5 times slower than dot. I also noticed that the std function is slightly slower than a python implementation code. These are the results for some operations in a randn vector of size 1000: 48.9+-22.6us: np.dot(v,v) 628.8+-87.3us: np.sum(np.absolute(v)) 518.3+-65.9us: np.sum(v*v) 118.0+-1.4us: v*v 234.4+-56.5us: np.absolute(v) 234.3+-56.2us: v.sum() 1175.2+-58.6us: v.std() 969.6+-77.4us: vv=np.mean(v)-v;(np.sqrt(np.dot(vv,vv)/(len(v)-1))) The code for timming the numpy operations: ################# import timeit import numpy as np p= """ import numpy as np vsize=1000 v=np.randn(vsize) """ vs=[ 'np.dot(v,v)', 'np.sum(np.absolute(v))', 'np.sum(v*v)', 'v*v', 'np.absolute(v)', 'v.sum()', 'v.std()', 'vv=np.mean(v)-v;(np.sqrt(np.dot(vv,vv)/(len(v)-1)))'] ntests=1000 tries=5 for s in vs: t = timeit.Timer(s,p) r=(np.array(t.repeat(tries,ntests))/ntests) * 10.0e6 print ("%10.1f+-%3.1fus: %s " % (np.mean(r), np.std(r) ,s)) ################# If some one have a complete benchmarking set of functions I would like to use them. Thanks. Hugo Gamboa -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikhil.padmanabhan at gmail.com Wed Mar 8 08:14:07 2006 From: nikhil.padmanabhan at gmail.com (Nikhil Padmanabhan) Date: Wed Mar 8 08:14:07 2006 Subject: [Numpy-discussion] asin, acos, etc... Message-ID: Hi, Is there support for the inverse trigonometric functions in numpy? I haven't been able to find any references in the documentation, but I might have not been looking hard enough. I am using v0.9.4. Thanks in advance, -- Nikhil ------------------------------------ Nikhil Padmanabhan nikhil.padmanabhan at gmail.com From ryanlists at gmail.com Wed Mar 8 08:17:08 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed Mar 8 08:17:08 2006 Subject: [Numpy-discussion] asin, acos, etc... In-Reply-To: References: Message-ID: They all begin with arc In [4]: numpy.arc numpy.arccos numpy.arcsin numpy.arctan numpy.arctanh numpy.arccosh numpy.arcsinh numpy.arctan2 In [4]: numpy.arccos(0) Out[4]: 1.5707963267948966 On 3/8/06, Nikhil Padmanabhan wrote: > Hi, > > Is there support for the inverse trigonometric functions in numpy? I > haven't been able to find any references in the documentation, but I > might have not been looking hard enough. > > I am using v0.9.4. > > Thanks in advance, > -- Nikhil > ------------------------------------ > Nikhil Padmanabhan > nikhil.padmanabhan at gmail.com > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From nikhil.padmanabhan at gmail.com Wed Mar 8 08:19:04 2006 From: nikhil.padmanabhan at gmail.com (Nikhil Padmanabhan) Date: Wed Mar 8 08:19:04 2006 Subject: [Numpy-discussion] asin, acos, etc... In-Reply-To: References: Message-ID: <3B1D8207-9332-4904-A67B-7573104FB2EE@gmail.com> Thanks! -- Nikhil ------------------------------------ Nikhil Padmanabhan npadmana at princeton.edu http://wwwphy.princeton.edu/~npadmana nikhil@(609) 258-4355 On Mar 8, 2006, at 11:16 AM, Ryan Krauss wrote: > They all begin with arc > > In [4]: numpy.arc > numpy.arccos numpy.arcsin numpy.arctan numpy.arctanh > numpy.arccosh numpy.arcsinh numpy.arctan2 > > In [4]: numpy.arccos(0) > Out[4]: 1.5707963267948966 > > > On 3/8/06, Nikhil Padmanabhan wrote: >> Hi, >> >> Is there support for the inverse trigonometric functions in numpy? I >> haven't been able to find any references in the documentation, but I >> might have not been looking hard enough. >> >> I am using v0.9.4. >> >> Thanks in advance, >> -- Nikhil >> ------------------------------------ >> Nikhil Padmanabhan >> nikhil.padmanabhan at gmail.com >> >> >> >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by xPML, a groundbreaking scripting >> language >> that extends applications into web and mobile media. Attend the >> live webcast >> and join the prime developer group breaking into this new coding >> territory! >> http://sel.as-us.falkag.net/sel? >> cmd=lnk&kid=110944&bid=241720&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> From svetosch at gmx.net Wed Mar 8 09:00:08 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 8 09:00:08 2006 Subject: [Numpy-discussion] Matrix times scalar is wacky In-Reply-To: <43FEE539.9010606@ieee.org> References: <43FEE539.9010606@ieee.org> Message-ID: <440F0D78.4050406@gmx.net> Travis Oliphant schrieb: > Bill Baxter wrote: > >> Multiplying a matrix times a scalar seems to return junk for some reason: >> >> >>> A = numpy.asmatrix(numpy.rand(1,2)) >> >>> A >> matrix([[ 0.30604211, 0.98475225]]) >> >>> A * 0.2 >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) >> >>> 0.2 * A >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) >> >>> numpy.__version__ >> '0.9.5' >> > This should be fixed in SVN. > > I have just been bitten by this bug, so I would like to ask when to expect the next release. And/or are there any workarounds? Thanks, Sven From oliphant at ee.byu.edu Wed Mar 8 14:54:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 8 14:54:06 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F0D78.4050406@gmx.net> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> Message-ID: <440F606B.8010604@ee.byu.edu> I think it's time for a new release of NumPy. There have been several important bug-fixes and speed improvements recently. Please log any outstanding issues to the ticket system on http://projects.scipy.org/scipy/numpy/timeline I'd like to put another release out over the weekend. -Travis From svetosch at gmx.net Wed Mar 8 15:16:01 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 8 15:16:01 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F606B.8010604@ee.byu.edu> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> <440F606B.8010604@ee.byu.edu> Message-ID: <440F656E.1010206@gmx.net> Travis Oliphant schrieb: > I think it's time for a new release of NumPy. There have been several > important bug-fixes and speed improvements recently. > > Please log any outstanding issues to the ticket system on > http://projects.scipy.org/scipy/numpy/timeline > Don't know if it qualifies as an outstanding issue, but may I ask about the status of making matrix decompositions (and ideally some other functions as well, like lstsq for example) preserve matrix types? Thanks much, Sven From wbaxter at gmail.com Wed Mar 8 15:40:03 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 8 15:40:03 2006 Subject: [Numpy-discussion] Matrix times scalar is wacky In-Reply-To: <440F0D78.4050406@gmx.net> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> Message-ID: Workarounds I know of are: asmatrix(scalar * m.A) or I noticed the other day that scalar division works ok, so if you know scalar is != 0, m / (1.0/scalar) where m is a numpy.matrix, of course. I find myself repeatedly getting bitten by this, since I'm writing code that does lots of lerping between vectors. This is pretty dang ugly to see all over the place: vlerp = asmatrix((1-t) * v1.A + t * v2.A) when it should just be vlerp = (1-t)*v1 + t*v2 yeh yeh, I should write a function... --bb On 3/9/06, Sven Schreiber wrote: > > Travis Oliphant schrieb: > > Bill Baxter wrote: > > > >> Multiplying a matrix times a scalar seems to return junk for some > reason: > >> > >> >>> A = numpy.asmatrix(numpy.rand(1,2)) > >> >>> A > >> matrix([[ 0.30604211, 0.98475225]]) > >> >>> A * 0.2 > >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) > >> >>> 0.2 * A > >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) > >> >>> numpy.__version__ > >> '0.9.5' > >> > > This should be fixed in SVN. > > > > > I have just been bitten by this bug, so I would like to ask when to > expect the next release. And/or are there any workarounds? > > Thanks, > Sven > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Wed Mar 8 15:45:16 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 8 15:45:16 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F656E.1010206@gmx.net> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> <440F606B.8010604@ee.byu.edu> <440F656E.1010206@gmx.net> Message-ID: <440F6C65.80005@ee.byu.edu> Sven Schreiber wrote: >Travis Oliphant schrieb: > > >>I think it's time for a new release of NumPy. There have been several >>important bug-fixes and speed improvements recently. >> >>Please log any outstanding issues to the ticket system on >>http://projects.scipy.org/scipy/numpy/timeline >> >> >> > >Don't know if it qualifies as an outstanding issue, but may I ask about >the status of making matrix decompositions (and ideally some other >functions as well, like lstsq for example) preserve matrix types? > > > We should be doing better in this regard. The __array_wrap__ is being applied (correctly now) in all the linalg functions, so that matrices are returned if they are used. All the ufuncs already preserve arguments. But, we should also peruse the code for asarray calls, save the __array_wrap__ method if available and wrap back the results so that types can be preserved. Someone spoke of writing a decorator to do that automatically which would indeed be nice. If you have examples of functions that are not being preserved, please report them. -Travis From Fernando.Perez at colorado.edu Wed Mar 8 16:17:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 8 16:17:02 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F6C65.80005@ee.byu.edu> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> <440F606B.8010604@ee.byu.edu> <440F656E.1010206@gmx.net> <440F6C65.80005@ee.byu.edu> Message-ID: <440F73E3.3090503@colorado.edu> Travis Oliphant wrote: > Someone spoke of writing a decorator to do that automatically which > would indeed be nice. As long as it's used in the 'old-fashioned' way (i.e., as func = decorator(func) and not @decorator func so that numpy remains 2.3-compatible. Just a heads-up so that 2.4-isms don't appear unintentionally. Cheers, f From tim.hochberg at cox.net Wed Mar 8 17:12:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 8 17:12:05 2006 Subject: [Numpy-discussion] numexpr In-Reply-To: <20060307185127.GA31063@arbutus.physics.mcmaster.ca> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> Message-ID: <440F80A3.8070006@cox.net> I made some more changes to numexp including adding the optimiztions to pow that we earlier added to numpy. It would be nice to have some way to select the level of optimization. That way, we could work on very aggressive optimization without worrying about messing someone up later who would can't tolerate the moderate accuracy loss. The only idea I have thus far is to create a new Expression each time, instead of just using the precreated 'E'. This would allow us to pass in a context to Expression and this could be passed onto the Nodes that were subsequently created from it so that it would be available where it's needed. That may not make any sense to anyone but David, but I invite you look numexpr/expression.py and numexpr/compiler.py. Then it'll be clear as,..., well clear as something. -tim From dimgmdrp at eaton.com Wed Mar 8 17:15:13 2006 From: dimgmdrp at eaton.com (dimgmdrp) Date: Wed Mar 8 17:15:13 2006 Subject: [Numpy-discussion] Fw: numpy-discussion Message-ID: <000701c64316$b5631970$94097a57@winzix> ----- Original Message ----- From: Burr Colin To: uemyevc at flowserve.com Sent: Monday, March 6, 2006 9:30 AM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kljrnwrziyaflo.gif Type: image/gif Size: 9582 bytes Desc: not available URL: From alexander.belopolsky at gmail.com Wed Mar 8 19:31:02 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed Mar 8 19:31:02 2006 Subject: [Numpy-discussion] Is sum() slow? In-Reply-To: <440DD2FA.4060301@ee.byu.edu> References: <440DD2FA.4060301@ee.byu.edu> Message-ID: On 3/7/06, Travis Oliphant wrote: > ... > I'm a little surprised that Numeric is so much faster for this case as > you show later. > Here is another puzzle: > python -m timeit -s "from Numeric import zeros,sum; x = zeros((2,500),'f')" "sum(x,0)" 100000 loops, best of 3: 6.05 usec per loop > python -m timeit -s "from Numeric import zeros,sum; x = zeros((500,2),'f')" "sum(x,1)" 10000 loops, best of 3: 26.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((2,500),'f')" "x.sum(0)" 10000 loops, best of 3: 22.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((500,2),'f')" "x.sum(1)" 10000 loops, best of 3: 23.6 usec per loop Numpy and Numeric perform very similarly when reducing along the second axis. > If anyone can figure out how to make the NOBUFFER secion in > GenericReduce faster in ufuncobject.c it will be greatly welcomed. Here is my $.02: BTW, I've tried to take loop->... redirections out of the loop: no effect with gcc -O3 . From ndarray at mac.com Wed Mar 8 19:33:02 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 8 19:33:02 2006 Subject: [Numpy-discussion] Is sum() slow? In-Reply-To: <440DD2FA.4060301@ee.byu.edu> References: <440DD2FA.4060301@ee.byu.edu> Message-ID: On 3/7/06, Travis Oliphant wrote: > ... > I'm a little surprised that Numeric is so much faster for this case as > you show later. > Here is another puzzle: > python -m timeit -s "from Numeric import zeros,sum; x = zeros((2,500),'f')" "sum(x,0)" 100000 loops, best of 3: 6.05 usec per loop > python -m timeit -s "from Numeric import zeros,sum; x = zeros((500,2),'f')" "sum(x,1)" 10000 loops, best of 3: 26.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((2,500),'f')" "x.sum(0)" 10000 loops, best of 3: 22.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((500,2),'f')" "x.sum(1)" 10000 loops, best of 3: 23.6 usec per loop Numpy and Numeric perform very similarly when reducing along the second axis. > If anyone can figure out how to make the NOBUFFER secion in > GenericReduce faster in ufuncobject.c it will be greatly welcomed. Here is my $.02: BTW, I've tried to take loop->... redirections out of the loop: no effect with gcc -O3 . From tim.hochberg at cox.net Wed Mar 8 20:50:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 8 20:50:03 2006 Subject: [Numpy-discussion] numexpr: optimizing pow In-Reply-To: <440F80A3.8070006@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> <440F80A3.8070006@cox.net> Message-ID: <440FB3C7.7070704@cox.net> I just checked in some changes that do aggressive optimization on the pow operator in numexpr. Now all integral and half integral powers between [-50 and 50] are computed using multiples and sqrt. (Empirically 50 seemed to be the closest round number to the breakeven point.) I mention this primarily because I think it's cool. But also, it's the kind of optimization that I don't think would be feasible in numpy itself short of defining a whole pile of special cases, either separate ufuncs or separate loops within a single ufunc, one for each case that needed optimizing. Otherwise the bookkeeping overhead would overwhelm the savings of replacing pow with multiplies. Now all of the bookkeeping is done in Python, which makes it easy; and done once ahead of time and translated into bytecode, which makes it fast. The actual code that does the optimization is included below for those of you interested enough to care, but not interested enough to check it out of the sandbox. It could be made simpler, but I jump through some hoops to avoid unnecessary mulitplies. For instance, starting 'r' as 'OpNode('ones_like', [a])' would simplify things signifigantly, but at the cost of adding an extra multiply in most cases. That brings up an interesting idea. If 'mul' were made smarter, so that it recognized OpNode('ones_like', [a]) and ConstantNode(1), then not only would that speed some 'mul' cases up, it would simplify the code for 'pow' as well. I'll have to look into that tomorrow. Regards, -tim if True: # Aggressive RANGE = 50 # Approximate break even point with pow(x,y) # Optimize all integral and half integral powers in [-RANGE, RANGE] # Note: for complex numbers RANGE would be larger. if (int(2*x) == 2*x) and (-RANGE <= abs(x) <= RANGE): n = int(abs(x)) ishalfpower = int(abs(2*x)) % 2 r = None p = a mask = 1 while True: if (n & mask): if r is None: r = p else: r = OpNode('mul', [r,p]) mask <<= 1 if mask > n: break p = OpNode('mul', [p,p]) if ishalfpower: sqrta = OpNode('sqrt', [a]) if r is None: r = sqrta else: r = OpNode('mul', [r, sqrta]) if r is None: r = OpNode('ones_like', [a]) if x < 0: r = OpNode('div', [ConstantNode(1), r]) return r From alodiaswint at varingen.no Thu Mar 9 05:46:02 2006 From: alodiaswint at varingen.no (Alodia Swint) Date: Thu Mar 9 05:46:02 2006 Subject: [Numpy-discussion] Re: PTaramcy news Message-ID: <000001c6437f$a0565d80$781fa8c0@iyw43> sD qVxiaaugsrka $ a 69,95 (10 sH t Cd abIets) iVzauIeieuym $ n 105,45 (3 DY 0 t ep abIets) fCiilalImits $ 9 u 9,95 (1 cD 0 tab XL Iets) MO And many oth rY er http://cig56.hospotte.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Thu Mar 9 05:58:00 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 9 05:58:00 2006 Subject: [Numpy-discussion] ndarray - reshaping a sub-class Message-ID: <44103429.4070502@sympatico.ca> Is this a bug which should be reported to the Tracker? Colin W. http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3038233 From ndarray at mac.com Thu Mar 9 06:27:12 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 9 06:27:12 2006 Subject: [Numpy-discussion] ndarray - reshaping a sub-class In-Reply-To: <44103429.4070502@sympatico.ca> References: <44103429.4070502@sympatico.ca> Message-ID: I would say yes, report it. It is trivial to fix: just change asarray to asanyarray in reshape definition in oldnumeric.py, but I am not sure this is desirable. Oldnumeric is a backward compatibility module and probably matches old Numeric behavior in this respect. On 3/9/06, Colin J. Williams wrote: > Is this a bug which should be reported to the Tracker? > > Colin W. > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3038233 > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Thu Mar 9 10:46:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 9 10:46:04 2006 Subject: [Numpy-discussion] Archives Message-ID: <441077E1.7070007@sympatico.ca> The SourceForge archive for Numpy seemed to fizzle late in February but the Active State archive appears current. Colin W. "http://sourceforge.net/mailarchive/forum.php?forum_id=4890&max_rows=25&style=flat&viewmonth=200602&viewday=25" From efiring at hawaii.edu Thu Mar 9 10:50:02 2006 From: efiring at hawaii.edu (Eric Firing) Date: Thu Mar 9 10:50:02 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: <441078AB.5090003@hawaii.edu> Travis et al., Jeff Whitaker found that the imaged_masked.py example (with the colorbar line uncommented) in matplotlib 0.87 triggered a numpy bug--the script works normally with Numeric and numarray. He committed a workaround to mpl svn, but I suspect you may want to track it down and squash it in numpy before the next release. It is not clear to me whether it is really in ma, or merely revealed by ma. The attached script triggers the bug. Thanks. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_ma_bug.py Type: text/x-python Size: 144 bytes Desc: not available URL: From dav at alum.mit.edu Thu Mar 9 11:40:02 2006 From: dav at alum.mit.edu (Dav Clark) Date: Thu Mar 9 11:40:02 2006 Subject: [Numpy-discussion] numpy with mod_python... Message-ID: <200603091139.11872.dav@alum.mit.edu> So, This seems like an obvious thing, but there are so many levels of things involved that I'd really appreciate some guidance. 'import numpy' works fine most places, but it strangely fails when done from ../site-packages/numpy. That, I don't care so much, but I found that problem in persuing why 'import numpy' fails when called from a script within mod_python. In this case, the cwd is '/'. The precise error (in both cases) is: --- /usr/lib/python2.4/site-packages/numpy/core/__init__.py 20 21 __all__ = ['char','rec','memmap','ma'] ---> 22 __all__ += numeric.__all__ 23 __all__ += oldnumeric.__all__ 24 __all__ += defmatrix.__all__ NameError: name 'numeric' is not defined --- The obvious solution / hack to get this working in mod_python is to simply os.chdir() before running an import. Except that doesn't work. So, ideas about how to hack mod_python or numpy - or even better how to _fix_ numpy - would be greatly appreciated! Thanks, -- Dav Clark Holistic Neuroscientist 917-544-8408 ))<>(( From robert.kern at gmail.com Thu Mar 9 13:47:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 9 13:47:02 2006 Subject: [Numpy-discussion] Re: numpy with mod_python... In-Reply-To: <200603091139.11872.dav@alum.mit.edu> References: <200603091139.11872.dav@alum.mit.edu> Message-ID: Dav Clark wrote: > So, > > This seems like an obvious thing, but there are so many levels of things > involved that I'd really appreciate some guidance. > > 'import numpy' works fine most places, but it strangely fails when done > from ../site-packages/numpy. > > That, I don't care so much, but I found that problem in persuing why 'import > numpy' fails when called from a script within mod_python. In this case, the > cwd is '/'. > > The precise error (in both cases) is: > > > --- > /usr/lib/python2.4/site-packages/numpy/core/__init__.py > 20 > 21 __all__ = ['char','rec','memmap','ma'] > ---> 22 __all__ += numeric.__all__ > 23 __all__ += oldnumeric.__all__ > 24 __all__ += defmatrix.__all__ > > NameError: name 'numeric' is not defined > --- > > The obvious solution / hack to get this working in mod_python is to simply > os.chdir() before running an import. Except that doesn't work. > > So, ideas about how to hack mod_python or numpy - or even better how to _fix_ > numpy - would be greatly appreciated! Before you import numpy, what is your sys.path? Where is numpy/? What files are in numpy/core/? What version of numpy are you using? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zpincus at stanford.edu Thu Mar 9 15:55:04 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu Mar 9 15:55:04 2006 Subject: [Numpy-discussion] argmax() return type can't index python lists Message-ID: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> Hi folks, I'm using a fresh build of numpy and python 2.5, both checked out from SVN yesterday. It seems that numpy.argmax() and ndarray.argmax() are now returning 'int32scalar' types instead of 'int' types. Despite the fact that the former seems to inherit from the latter, 'int32scalar' types do not appear to be usable as indices into python lists. Anyhow, I'm not sure if this is a python 2.5 regression, or a problem in all versions of python that's exposed by this behavior, or a numpy problem. Any thoughts? Zach Test case: In [1]: import numpy In [2]: l = [1,2,3] In [3]: l[numpy.argmax(l)] : list indices must be integers In [4]: type(numpy.argmax(l)) Out[4]: In [5]: print numpy.version.version 0.9.6.2208 In [6]: import sys; print sys.version 2.5a0 (trunk:42924M, Mar 8 2006, 19:29:24) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] From dbrown at ucar.edu Thu Mar 9 15:57:04 2006 From: dbrown at ucar.edu (David Ian Brown) Date: Thu Mar 9 15:57:04 2006 Subject: [Numpy-discussion] numpy C API bugs? Message-ID: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> Hi Travis and NumPy developers, I have recently been engaged in updating the PyNIO module (http://www.pyngl.ucar.edu/Nio.html) to work with the NumPy C API. It was originally coded using the Numeric's C API. In the process I have encountered what appear to me to be a couple of bugs in the interface. They were relatively minor and not difficult to work around, but you would probably like to know of them if you don't already. In the NumPy version of arrayobject.h the macros PyArray_GETITEM and PyArray_SETITEM are defined in way that does not work. Here is the GETITEM macro: #define PyArray_GETITEM(obj,itemptr) \ ((char *)itemptr, \ (PyArrayObject *)obj); When I try to use this macro in my code I get a compile error, because 'getitem' is not a member of NumPy's PyArray_Descr struct. Instead it is further nested within the struct member 'PyArray_ArrFuncs *f'. So, unless I misunderstand, the macro needs to access the 'getitem' function as ((PyArrayObject *)(obj))->descr->f->getitem(... The other issue with 'getitem', apparently the actual method, not just this macro, is that the 2 arguments are ordered oppositely from what is stated in "Guide To NumPy", as well as from the way this method was implemented in Numeric. Notice in the macro (and in NumPy's implementation), 'itemptr' is first and the PyArrayObject pointer is second. The documentation and Numeric has it the other way around. I hope this is useful. -dave From dbrown at ucar.edu Thu Mar 9 16:59:01 2006 From: dbrown at ucar.edu (David Ian Brown) Date: Thu Mar 9 16:59:01 2006 Subject: [Numpy-discussion] numpy C API bugs? In-Reply-To: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> References: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> Message-ID: <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> On Mar 9, 2006, at 4:56 PM, David Ian Brown wrote: > > Hi Travis and NumPy developers, > > I have recently been engaged in updating the PyNIO module > (http://www.pyngl.ucar.edu/Nio.html) > to work with the NumPy C API. It was originally coded using the > Numeric's C API. > > In the process I have encountered what appear to me to be a couple of > bugs in the interface. They were relatively minor and not difficult > to work around, but you would probably like to know of them if you > don't already. > > In the NumPy version of arrayobject.h the macros PyArray_GETITEM and > PyArray_SETITEM > are defined in way that does not work. Here is the GETITEM macro: > > #define PyArray_GETITEM(obj,itemptr) \ > ((char *)itemptr, \ > (PyArrayObject *)obj); > > When I try to use this macro in my code I get a compile error, because > 'getitem' is not a member of NumPy's > PyArray_Descr struct. Instead it is further nested within the struct > member 'PyArray_ArrFuncs *f'. > So, unless I misunderstand, the macro needs to access the 'getitem' > function as > > ((PyArrayObject *)(obj))->descr->f->getitem(... > > The other issue with 'getitem', apparently the actual method, not just > this macro, is that the 2 arguments are > ordered oppositely from what is stated in "Guide To NumPy", as well > as from the way this method was > implemented in Numeric. Notice in the macro (and in NumPy's > implementation), 'itemptr' is first and > the PyArrayObject pointer is second. The documentation and Numeric has > it the other way around. Correction: actually "Guide to NumPy" has the 'getitem' function documented with the arguments ordered as implemented in NumPy (but still opposite from Numeric). Only the macro is documented opposite from its implementation. > > I hope this is useful. > -dave > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ndarray at mac.com Thu Mar 9 17:33:01 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 9 17:33:01 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: <441078AB.5090003@hawaii.edu> References: <4404A71B.10600@cox.net> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <441078AB.5090003@hawaii.edu> Message-ID: I would say it is an ma bug, but don't know how to fix it properly without changing numpy scalar arithmetic to call an enhanced version of __array__ and pass context in. The core problem can be demonstrated by the following session: >>> from numpy import * >>> x = ma.array([1],mask=[1]) >>> int_(1)*x Traceback (most recent call last): File "", line 1, in ? File ".../numpy/core/ma.py", line 622, in __array__ raise MAError, \ numpy.core.ma.MAError: Cannot automatically convert masked array to numeric because data is masked in one or more locations. Note that x*int_(1) works as expected. This is so because python dispatches multiplication to int.__mul__ rather than ma.array.__mul__ if int_(1) is the first multiplicand. I've fixed a similar problem for array*ma.array case and array(1)*x works in the current version of numpy. I will not have time to work on this before the weekend, so if someone is motivated enought to fix this bug before the upcoming release, please take a look at http://projects.scipy.org/scipy/numpy/wiki/MaskedArray ("Ufuncs and Masked Arrays"). It should be straightforward to generalize that approach for array scalars. On 3/9/06, Eric Firing wrote: > Travis et al., > > Jeff Whitaker found that the imaged_masked.py example (with the colorbar > line uncommented) in matplotlib 0.87 triggered a numpy bug--the script > works normally with Numeric and numarray. He committed a workaround to > mpl svn, but I suspect you may want to track it down and squash it in > numpy before the next release. It is not clear to me whether it is > really in ma, or merely revealed by ma. The attached script triggers > the bug. > > Thanks. > > Eric > > > > From oliphant.travis at ieee.org Thu Mar 9 21:08:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 9 21:08:00 2006 Subject: [Numpy-discussion] argmax() return type can't index python lists In-Reply-To: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> References: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> Message-ID: <4411096A.50203@ieee.org> Zachary Pincus wrote: > Hi folks, > > I'm using a fresh build of numpy and python 2.5, both checked out from > SVN yesterday. It seems that numpy.argmax() and ndarray.argmax() are > now returning 'int32scalar' types instead of 'int' types. This has always been the case for numpy so it's not new. > Despite the fact that the former seems to inherit from the latter, > 'int32scalar' types do not appear to be usable as indices into python > lists. This is new and is a Python 2.5 problem it would appear. > > Anyhow, I'm not sure if this is a python 2.5 regression, or a problem > in all versions of python that's exposed by this behavior, or a numpy > problem. > Note that the __index__ PEP I wrote was just accepted and checked-in so in fact *all* numpy scalar types will be acceptable as indices in Python2.5 (but we have to write the code for that still in numpy). So, it's possible that something in Python 2.5 changed is causing it not to work. This very-well could be a bug in the new implementation of __index__. -Travis From oliphant.travis at ieee.org Thu Mar 9 21:29:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 9 21:29:02 2006 Subject: [Numpy-discussion] numpy C API bugs? In-Reply-To: <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> References: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> Message-ID: <44110E6F.4080703@ieee.org> David Ian Brown wrote: >> >> I have recently been engaged in updating the PyNIO module >> (http://www.pyngl.ucar.edu/Nio.html) >> to work with the NumPy C API. It was originally coded using the >> Numeric's C API. >> >> In the process I have encountered what appear to me to be a couple >> of bugs in the interface. They were relatively minor and not >> difficult to work around, but you would probably like to know of them >> if you don't already. >> >> In the NumPy version of arrayobject.h the macros PyArray_GETITEM and >> PyArray_SETITEM >> are defined in way that does not work. Here is the GETITEM macro: >> >> #define PyArray_GETITEM(obj,itemptr) \ >> ((char *)itemptr, \ >> (PyArrayObject *)obj); >> Thank you very much for catching this. The introduction of 'f' was more recent then the writing of these macros and so this error crept in. I'm definitely glad to fix it. > Correction: actually "Guide to NumPy" has the 'getitem' function > documented with the arguments ordered > as implemented in NumPy (but still opposite from Numeric). Only the > macro is documented opposite from > its implementation. I think further clarification is needed. First of all Numeric did not have macros. They just had function pointers to getitem. Second, Numpy's function pointers to getitem and setitem are almost exactly the same. The only difference is the *addition* of another argument to both functions to allow the array to be passed in (so that the size of the array and whether or not it's well-behaved can be known). So, the order is *not* different in NumPy and Numeric, there is just another argument needed. The new macros introduced do have a different order (and I don't think it's documented oppositely --- it's perhaps not documented well at all, however :-) and so may be confusing). It is recommended to use the macros. They are PyArray_GETITEM(array, itemptr) Return a Python object corresponding to the item in the array at the location itemptr. PyArray_SETITEM(array, itemptr, obj) Set object into the array at the memory pointed to by itemptr (an attempt is made to convert to the right type if that is possible). Thanks for the report. -Travis From zpincus at stanford.edu Thu Mar 9 21:34:02 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu Mar 9 21:34:02 2006 Subject: [Numpy-discussion] argmax() return type can't index python lists In-Reply-To: <4411096A.50203@ieee.org> References: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> <4411096A.50203@ieee.org> Message-ID: <9F90F4AA-70DF-48AE-B295-1AEDE1BB172C@stanford.edu> Thanks for your reply, Travis. > Note that the __index__ PEP I wrote was just accepted and checked- > in so in fact *all* numpy scalar types will be acceptable as > indices in Python2.5 (but we have to write the code for that still > in numpy). So, it's possible that something in Python 2.5 changed > is causing it not to work. > > This very-well could be a bug in the new implementation of __index__. The problem is that I can't reproduce this bug outside of numpy. If I go to the python people, they'll most likely say it's a numpy problem, unless I can show some other test case. Right now, all of the following classes work as indices in 2.5. The only thing that doesn't work is a raw numpy.int32 value. I'd be happy to file a bug report on python if I could figure out a test case. Zach Cases which work fine: class int_sublcass(int): pass class empty(object): pass class int_1(int, empty): pass class int_2(empty, int): pass class int_numpy_1(numpy.signedinteger, int): def __init__(self, i): int.__init__(self, i) class int_numpy_2(int, numpy.signedinteger): def __init__(self, i): int.__init__(self, i) l = [1,2,3] l[int_sublcass(1)] l[int_1(1)] l[int_2(1)] l[int_numpy_1(1)] l[int_numpy_2(1)] From oliphant.travis at ieee.org Fri Mar 10 02:32:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 10 02:32:04 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: References: <4404A71B.10600@cox.net> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <441078AB.5090003@hawaii.edu> Message-ID: <4411557B.4020006@ieee.org> Sasha wrote: > I would say it is an ma bug, but don't know how to fix it properly > without changing numpy scalar arithmetic to call an enhanced version > of __array__ and pass context in. > I think it was an array scalar bug. I've fixed it by always calling the ufunc (which handles conversions to and from other objects better than the array scalars were doing in the generic arithmetic code). The result is shorter code and symmetric handling of the * vs. * case. I also changed the error to a warning for backward compatibility. Calling __array__() on a masked array should succeed and we will assume the user knows what they are doing. -Travis From pfdubois at gmail.com Fri Mar 10 07:07:12 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Fri Mar 10 07:07:12 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: <4411557B.4020006@ieee.org> References: <4404A71B.10600@cox.net> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <441078AB.5090003@hawaii.edu> <4411557B.4020006@ieee.org> Message-ID: It was a design decision: if an automatic conversion of a masked array to a numeric one is attempted, and there actually are masked values, it is logical to either: a. raise an exception, because there is no equivalent numeric array, or; b. fill the masked value with the fill value (b) suffers from a problem: the default fill value might not be appropriate for the use that is about to occur. My original conclusion was that it was better to choose (a) to force the user to explicitly do the conversion with x.fill(value) and pass THAT to whatever was going to consume it. I don't think I've changed my mind. On 10 Mar 2006 02:32:19 -0800, Travis Oliphant wrote: > > Sasha wrote: > > I would say it is an ma bug, but don't know how to fix it properly > > without changing numpy scalar arithmetic to call an enhanced version > > of __array__ and pass context in. > > > I think it was an array scalar bug. I've fixed it by always calling the > ufunc (which handles conversions to and from other objects better than > the array scalars were doing in the generic arithmetic code). > > The result is shorter code and symmetric handling of the > * vs. * case. > > I also changed the error to a warning for backward compatibility. > Calling __array__() on a masked array should succeed and we will assume > the user knows what they are doing. > > -Travis > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Mar 10 10:20:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Mar 10 10:20:03 2006 Subject: [Numpy-discussion] Key 'i' disappeared in numpy.typeNA Message-ID: <200603101919.40714.faltet@carabos.com> Hi, I'm in a hurry, so not able to track this right down, but I have just checked the numpy SVN version of numpy and discovered that the 'i' entry for typeNA has disappeared: >>> numpy.__version__ '0.9.6.2214' >>> numpy.typeNA['i'] Traceback (most recent call last): File "", line 1, in ? KeyError: 'i' >>> numpy.typeNA['l'] 'Int32' Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From dbrown at ucar.edu Fri Mar 10 11:52:07 2006 From: dbrown at ucar.edu (David Ian Brown) Date: Fri Mar 10 11:52:07 2006 Subject: [Numpy-discussion] numpy C API bugs? In-Reply-To: <44110E6F.4080703@ieee.org> References: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> <44110E6F.4080703@ieee.org> Message-ID: <10dbcbe9aebaf94303d3194a1f97dde4@ucar.edu> Hi Travis, Yes, now I see that I just wasn't looking closely enough at the the docs, that Numeric's getitem did have only one argument, and the docs have the arguments in the correct order both for the function and the macro. I was just confused by the fact that the macro has the arguments in the opposite order of the function. So the only problem was the missing 'f' in the macro. -dave On Mar 9, 2006, at 10:28 PM, Travis Oliphant wrote: > David Ian Brown wrote: >>> >>> I have recently been engaged in updating the PyNIO module >>> (http://www.pyngl.ucar.edu/Nio.html) >>> to work with the NumPy C API. It was originally coded using the >>> Numeric's C API. >>> >>> In the process I have encountered what appear to me to be a couple >>> of bugs in the interface. They were relatively minor and not >>> difficult to work around, but you would probably like to know of >>> them if you don't already. >>> >>> In the NumPy version of arrayobject.h the macros PyArray_GETITEM and >>> PyArray_SETITEM >>> are defined in way that does not work. Here is the GETITEM macro: >>> >>> #define PyArray_GETITEM(obj,itemptr) \ >>> ((char *)itemptr, \ >>> (PyArrayObject *)obj); >>> > > Thank you very much for catching this. The introduction of 'f' was > more recent then the writing of these macros and so this error crept > in. I'm definitely glad to fix it. > >> Correction: actually "Guide to NumPy" has the 'getitem' function >> documented with the arguments ordered >> as implemented in NumPy (but still opposite from Numeric). Only the >> macro is documented opposite from >> its implementation. > > I think further clarification is needed. > First of all Numeric did not have macros. They just had function > pointers to getitem. Second, Numpy's function pointers to getitem and > setitem are almost exactly the same. The only difference is the > *addition* of another argument to both functions to allow the array to > be passed in (so that the size of the array and whether or not it's > well-behaved can be known). > So, the order is *not* different in NumPy and Numeric, there is just > another argument needed. > The new macros introduced do have a different order (and I don't think > it's documented oppositely --- it's perhaps not documented well at > all, however :-) and so may be confusing). > > It is recommended to use the macros. They are > > PyArray_GETITEM(array, itemptr) > > Return a Python object corresponding to the item in the array at the > location itemptr. > > PyArray_SETITEM(array, itemptr, obj) > > Set object into the array at the memory pointed to by itemptr (an > attempt is made to convert to the right type if that is possible). > > Thanks for the report. > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From dav at alum.mit.edu Fri Mar 10 12:22:03 2006 From: dav at alum.mit.edu (Dav Clark) Date: Fri Mar 10 12:22:03 2006 Subject: [Numpy-discussion] Further info on mod_python / numpy crashing Message-ID: <4411DFB4.8000103@alum.mit.edu> First - I realized I failed to include version numbers. I'm using numpy 0.9.5, and mod_python 3.1.4 and apache 2.0.54. Standard (I think) for red hat FC4 on an intel box. So, I've spent some time tracing through mod_python... the bad code is here, from the import_module def in mod_python/apache.py: parts = module_name.split('.') for i in range(len(parts)): f, p, d = imp.find_module(parts[i], path) try: mname = ".".join(parts[:i+1]) module = imp.load_module(mname, f, p, d) finally: if f: f.close() if hasattr(module, "__path__"): path = module.__path__ From a mod_python interpreter - all calls to import get replaced with import_module (to prevent reloads). The setting of the 'path' parameter in this way is (I think) invoking the same bug that causes the crash when imoporting numpy from ../site-packages/numpy. Can anyone even replicate this? Should I cross-post this to mod_python? Cheers, Dav From dav at alum.mit.edu Fri Mar 10 14:58:03 2006 From: dav at alum.mit.edu (Dav Clark) Date: Fri Mar 10 14:58:03 2006 Subject: [Numpy-discussion] solution for mod_python / numpy interaction Message-ID: <44120475.7020706@alum.mit.edu> So mod_python 3.2(.8) included a major overhaul of the import mechanism. I'm still not sure what the problem is, and I think an 'import numpy' from site-packages/numpy _shouldn't_ cause an error (though it does). But I can at least do my work. DC From mark at mitre.org Fri Mar 10 17:44:05 2006 From: mark at mitre.org (Mark Heslep) Date: Fri Mar 10 17:44:05 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. In-Reply-To: References: Message-ID: <44122B58.1010707@mitre.org> +1 Chapter 8 Advanced Py is one of the best Python howto's Ive seen, and the other chapters covering the Numpy API and especially howto use are excellent. Bruce Southey wrote: >Hi, >I have the first edition which is very comprehensive on Python, >numarray and Numeric - note that by NumPy he is means only Numeric and >numarray and not the new numpy. If you have the basic of Python and >numerical computing then it is a great book as it cover many different >topics. For example, there is considerable detail on using Fortran and >C/C++ with Numeric and numarray including examples. > >For the majority of the material in the book is still valid for the >new Numpy. He does provide a number of advanced topics in Python, >numarray and Numeric that are hard to find elsewhere. > >If you are going to extensive numerical work then it is really worth it. > >Bruce > > >On 3/2/06, Zden?k Hur?k wrote: > > >>Hello, >> >>Can anybody recommend the book: >> >>Python Scripting for Computational Science >>Series: Texts in Computational Science and Engineering, Vol. 3 >>Langtangen, Hans Petter >>2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover >>ISBN: 3-540-29415-5 >> >>I am just a beginner in Python programming (but I master C, Matlab). I do >>research in applied numerical computing (oriented at control design). Is >>this a book for me? It is hard to guess just from the table contents. I >>especially doubt if the information on Numpy package is relevant at all >>with the wild development witnessed in the past recent months. However, >>peeling off these "numerical" parts, isn't buying some comprehensive Python >>book a better buy? >> >>Thanks for your tips, >>Zdenek Hurak >> >> >> >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting language >>that extends applications into web and mobile media. Attend the live webcast >>and join the prime developer group breaking into this new coding territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >???????????????????????????????????????????+?^????X???'???u????w[????x,r???)?????? ??{^????bq?b?{"??0y??vj?W?y??? mzwm??b?? m???????????)?u????^? >????y?"????a????? ?x-z??????m??????j??????F ?w????????H????n'??{?O?j?^?????????????????????????????????????6????b????*'????x%??M?jr???r?,????+-??(?? ~??{???m????X???????y?+???z????l?X??)???????v+ ??" > > > From mark at mitre.org Fri Mar 10 18:15:01 2006 From: mark at mitre.org (Mark Heslep) Date: Fri Mar 10 18:15:01 2006 Subject: [Numpy-discussion] NumPy wrapper for C output buffer of unknown size Message-ID: <4412327F.5020303@mitre.org> I'm trying to come up with a _generic_, _reusable_ numpy wrapper for a C output buffer of unknown size. That is, only the buffer pointer is available from a function call, and it's size is available elsewhere. For example, the buffer access might look like this: char * func_returns_ptr_to_allocated_buffer() { ...} And the size of the buffer is determined elsewhere by some other 'machinery'. I'm using Swig* here, and left to itself Swig will neatly map the char* to a Swig Python object, something like . That's fine unless you want access to the buffer elements via NumPy. I'm familiar with many of the fine NumPy typemaps posted here and on the net, but they all assume a known array size and use a PyArray_FromDimsandData (...SimpleFrom... w/ recent Numpy) with a dims argument provided. Instead, is there some method where I can return a numpy array of size zero (one?) that effectively captures the data pointer and then allows me to resize on the Python side? I note the the restrictions on the NumPy resize method: no previous reference, and self->base=Null. Its simple to solve this with a helper function custom to this application but its not maintainable with the app-library as it evolves. Extending the solution to any other data type is also no problem. So, as stated above, is there a generic, numpy-thonic way to solve this? Mark * Im open to other methods -ctypes, etc From mpi at osc.kiku.dk Sun Mar 12 04:22:02 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Sun Mar 12 04:22:02 2006 Subject: [Numpy-discussion] C-extensions to numpy - a little help is needed Message-ID: Hey, I am trying to understand how to write numpy extensions, in order to be able to do various things fast, that I need in my own apps. I have looked at the examples from the Numeric package, which I managed to get working with numpy, so I guess my questions holds for this forum. There's one detail, that I don't understand in the examples provided with the Numeric documentation, namely the use of Py_DECREF(...) which is used in the first example but not in the last. Let me be more specific: In the example, showing how to implement a trace() function, you access the data using array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); do some summing to find the trace and then decrement and return the result Py_DECREF(array); return PyFloat_FromDouble(sum); Now if you look at the other example for the matrix-vector multiply, the matrix and vector data are accessed using a similar approach PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; ... if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); ... vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); ... result = (PyArrayObject *) PyArray_FromDims(1, dimensions, PyArray_DOUBLE); Then calculate the DGEMV storing the result in result->data and return with return PyArray_Return(result); Finally here's my question: ---------------------------- Since the two pointers *matrix and *vector are created just like *array in the trace() example, how come they are not decremented with two calls to Py_DECREF(matrix); Py_DECREF(vector); Any help is appreciated. // Mads From Norbert.Nemec.list at gmx.de Sun Mar 12 05:59:02 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Sun Mar 12 05:59:02 2006 Subject: [Numpy-discussion] Patches for numpy.linalg and scipy.linalg Message-ID: <441428FB.4080003@gmx.de> Over the past few days I worked a bit on the linalg packages in both numpy and scipy. Attached are patches for both numpy and scipy. Both are a bit interdependent, so I'm submitting all in one cross-posted mail. I don't know about the SVN-write-access policy of numpy/scipy. If it is not kept too restrictive, it might be convenient for everyone to give me write access. I cannot promise to find much time working on the code, but linalg sure needs some cleanup and I might continue doing a bit of it. Here a few explanations about the attached patches: numpy-1-cleanup.diff just a bit of cosmetics to make the other patches cleaner numpy-2-rename-functions.diff numpy.linalg used to contain a number of aliases (like 'inv'=='inverse' and others) this is pointless and more confusing then helpful. Unless there is a really good reason, a library should offer one name only. I chose the short version of each name as the "official" name, because I think all the abbreviations are well-known and easy to understand for anyone handling numerical linear algebra. Currently all the definition that would be needed for backwards-compatibility are deactivated in a "if False:" block. If people think they should be deprecated slowly, this block could easily be re-activated. numpy-3-svd-compute_uv.diff numpy.linalg.svd now also has the compute_uv option that scipy.linalg.svd already had. Default behavior is the same as before. numpy-4-svd-bug-workaround.diff the dgesdd function of the lapack library installed on my system seems to contain a strange bug. Should probably be investigated and fixed. For the moment I just included a workaround. numpy-5-norm-copy-from-scipy.diff copied and the "norm" function from scipy.linalg and adjusted internals to work in the new environment numpy-6-norm-change-default.diff the 'scipy.linalg.norm' function was sub-optimal: for matrices, the frobenius norm is faster then the "max(svd())" by about an order of magnitude for all the test cases that I came up with. It also is invariant under orthogonal/unitary transformations, which makes it the best candidate to be the default in any case that I could come up with. The computation of the Frobenius norm can be done with the same line of code used for the vector-square norm and can even be generalized to arrays of arbitrary rank, always giving a reasonable norm for both real and complex numbers. The most straightforward way to clean up a number of inefficiencies in the definition of "norm" was to introduce a norm with "ord=None" which always calculates sqrt(sum((conjugate(x)*x).ravel())) which is generally the most efficient implementation and does not even need a check of the rank of the array. I also made this choice the default of the "norm" function. I know that this may cause hard-to-find errors in existing code if people explicitely want the "svd"-norm for a matrix and relied on the old default setting. Personally, I don't believe this is a real danger, because I don't believe there are many algorithms that depend on using the svd-norm and break if frobenius is used. If people are strictly against changing the default behavior, the patch should still be accepted and only the function header changed back. numpy-7-castCopyAndTranspose.diff simplyfied this function for purely aesthetic reasons: I believe a function that can be written cleanly in two lines should not be extended unnessecarily to 6 lines... numpy-8-dual.diff added 'norm', 'eigh' and 'eigvalsh' to the list of dual functions. scipy-1-eigh-eigvalsh.diff added two new functions to scipy.linalg. All the interfaces were there already... scipy-2-dual-norm.diff added 'norm' to the list of dual functions scipy-3-norm-change-default.diff changed the norm in the same way as "numpy.linalg.norm" was changed. See comments above. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-1-cleanup.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-2-rename-functions.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-3-svd-compute_uv.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-4-svd-bug-workaround.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-5-norm-copy-from-scipy.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-6-norm-change-default.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-7-castCopyAndTranspose.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-8-dual.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy-1-eigh-eigvalsh.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy-2-dual-norm.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy-3-norm-change-default.diff URL: From a.mcmorland at auckland.ac.nz Sun Mar 12 10:02:02 2006 From: a.mcmorland at auckland.ac.nz (Angus McMorland) Date: Sun Mar 12 10:02:02 2006 Subject: [Numpy-discussion] Numpy implementation of IDL & pdl's rebin? Message-ID: <4414620C.3040202@auckland.ac.nz> Hi, I've been interested in switching from pdl and IDL to numpy (or its predecessors) for a couple of years now, but have always been put off by the lack of a simple, built-in n-dimensional rebin command for re-sampling an array to any arbitrary dimensions. This is a feature I require frequently in these other languages. I came across this thread some time ago: http://article.gmane.org/gmane.comp.python.numeric.general/892/match=rebin which discusses various implementation strategies and encouragingly suggests the addition of a built-in command. I haven't come across any progress since then and wondered if anything had been done. At one stage I tried to implement one myself, but ran out of time and got lost trying to find the required n-dimensional interpolation routines. Could someone please tell me if (a) something has been done, or (b) if anyone with some experience in this area would be interested in collaborating to complete the task? Thanks, Angus. -- Angus McMorland email a.mcmorland at auckland.ac.nz mobile +64-21-155-4906 PhD Student, Neurophysiology / Multiphoton & Confocal Imaging Physiology, University of Auckland phone +64-9-3737-599 x89707 Armourer, Auckland University Fencing Secretary, Fencing North Inc. From oliphant.travis at ieee.org Sun Mar 12 15:04:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 12 15:04:01 2006 Subject: [Numpy-discussion] C-extensions to numpy - a little help is needed In-Reply-To: References: Message-ID: <4414A8CE.2040404@ieee.org> Mads Ipsen wrote: > Hey, > > I am trying to understand how to write numpy extensions, in order to be > able to do various things fast, that I need in my own apps. I have looked > at the examples from the Numeric package, which I managed to get working > with numpy, so I guess my questions holds for this forum. > > There's one detail, that I don't understand in the examples provided with > the Numeric documentation, namely the use of > > Py_DECREF(...) > > which is used in the first example but not in the last. Let me be more > specific: > There are several problems with the example in the documentation that lead to memory leaks. It needs to be fixed. You are correct that Py_DECREF(matrix); Py_DECREF(vector); is needed before returning control back to Python. In addition, the error-handling code (when return NULL is encountered) also needs to make sure that arrays that have been created get destroyed using Py_DECREF(...) before returning from the function. This is often done using an err: label and the goto command in C. Then, at the error label Py_XDECREF(...) is used which does nothing if the argument is NULL, but otherwise DECREF's the object (destroying it if it's the only reference). So, this example is flawed. I think that it's come up before but I don't recall the list discussion. But, obviously nothing was done to change the documentation. -Travis From cjw at sympatico.ca Sun Mar 12 15:18:00 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 12 15:18:00 2006 Subject: [Numpy-discussion] Attempt to subclass dtype Message-ID: <4414AC1B.8010107@sympatico.ca> I would like to subclass dtype. Is there a way for a user to make dtype acceptable? class d1type(_n.dtype): TypeError: Error when calling the metaclass bases type 'numpy.dtype' is not an acceptable base type >> Colin W. From Fernando.Perez at colorado.edu Sun Mar 12 16:34:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun Mar 12 16:34:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... Message-ID: <4414BDEF.70001@colorado.edu> Hi all, there are recurrent questions on the topic of low-level access to numpy, wrapping libraries, etc. Many people have home-brewed SWIG typemaps, the cookbook has some (very nice) pyrex examples, etc. I think that we could use a bit of unification work on this front, to make this particular issue easier to deal with for newcomers, and to reduce duplication of effort. 1. SWIG ======= At the last scipy conference, I collected typemaps from a number of people who had them around and all I could find on the net. I grabbed Scott Ransom's, Michel Sanner contributed his, Eric Jones gave us some as well, etc. John Hunter worked on updating Eric's (which had some nice features) to work with plain C (they were originally C++), and Bill Spotz from Sandia NL agreed to do the real work of writing some clean, documented examples of typemap use starting with this codebase. He gave me this code some time ago, and I shamefully just sat on it. I'm finally doing something about it, so today I updated this code to work with numpy instead of Numeric, and all the tests pass. You can grab it from here with an SVN client: http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpy_swig It is clearly documented, and while I'm sure it can use improvements and extensions, I think it would be a good starting point to have distributed /officially/ with numpy. That way others can improve /this/ code, rather than reimplementing numpy typemaps for the millionth time. D. Beazley (the SWIG author) has even indicated that he'd be happy to officially ship numpy typemaps with SWIG if something accepted by the community were sent to him. 2. Pyrex ======== After reading the excellent cookbook examples on pyrex usage, which I recently needed for a project, I decided to complete the code so it could be used 'out of the box'. This simply meant a bit of organization work, updating the code to use the 'modern' cimport statement and .pxd files, and writing a proper setup.py file. The result is here: http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpyx/ I also think that pyrex should be 'officially' supported. In particular, I think that the c_numpy.pxd file I put there, which is just a copy of that in mtrand, should be completed and shipped in the same headers directory as the rest of numpy's .h files. This would make it a LOT easier for newcomers to use pyrex for writing numpy extensions, since all headers would be complete and available with all numpy installations. Finally, if anyone is interested, that repository actually contains in the full checkout http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/ examples of how to wrap a more complex library which requires the creation of C structs from python objects. Getting this right took some time, as not all the details for this kind of construction are clearly documented anywhere. I hope it may be useful to others. 3. Wrapup ========= I think it would be a good idea to ship the numpy_swig example directory (and perhaps also numpyx, if somewhat expanded) listed above with the standard distribution, as well as completing the .pxd file to expose all the numpy C API for pyrex. I am not claiming to have done any of the hard work in numpyx or numpy_swig (though I did spend a lot of time on the wavelets stuff recently :), but I think we would all benefit from having this kind of 'infrastructure' code in the core. Once there, improvements can be made over time, with less effort wasted. If we agree on this, it's as simple as dropping those two directories somewhere on the numpy codebase (sandbox?, doc?). Just having them 'officially' available will be a benefit, I think. I'd like to thank Bill for the time he put into the swig module, as well as others who contributed code for this. Hopefully it will benefit everyone. Regards, f From strawman at astraw.com Sun Mar 12 16:51:06 2006 From: strawman at astraw.com (Andrew Straw) Date: Sun Mar 12 16:51:06 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414BDEF.70001@colorado.edu> References: <4414BDEF.70001@colorado.edu> Message-ID: <4414C1D5.5000108@astraw.com> Fernando Perez wrote: > At the last scipy conference, I collected typemaps from a number of > people who had them around and all I could find on the net. [snip] > I'm finally doing something about it, so today I updated this code to > work with numpy instead of Numeric, and all the tests pass. Thanks Fernando, I think this is great. > I think it would be a good starting point to have distributed > /officially/ with numpy. That way others can improve /this/ code, > rather than reimplementing numpy typemaps for the millionth time. +1 > 2. Pyrex > ======== > > After reading the excellent cookbook examples on pyrex usage, which I > recently needed for a project, I decided to complete the code so it > could be used 'out of the box'. This simply meant a bit of > organization work, updating the code to use the 'modern' cimport > statement and .pxd files, and writing a proper setup.py file. What do you mean 'modern' cimport and .pxd statement? I believe the example at http://scipy.org/Cookbook/Pyrex_and_NumPy is fundamentally no different than yours. One question, though: why have you reverted from using Python's Py_intptr_t and used intp instead? It seems to me much more likely that Py_intptr_t will be available and will do the Right Thing than relying on some libc to define it. We've seen broken implementations of C libraries before, and I say let's trust the Python devs to get this one right. > The result is here: > > http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpyx/ > > I also think that pyrex should be 'officially' supported. In > particular, I think that the c_numpy.pxd file I put there, which is > just a copy of that in mtrand, should be completed and shipped in the > same headers directory as the rest of numpy's .h files. This would > make it a LOT easier for newcomers to use pyrex for writing numpy > extensions, since all headers would be complete and available with all > numpy installations. +1 > Getting this right took some time, as not all the details for this > kind of construction are clearly documented anywhere. I hope it may > be useful to others. Note that setuptools 'automatically' deals with .pyx files: If you pass a source file that ends in .pyx, setuptools will run pyrexc on it if Pyrex is available, and otherwise use a '.c' file by the same name in the same directory. This is nice because it means you can ship .c files with your project and people can build them, even if they don't have Pyrex installed. Maybe we want to ship two setup.py files, or one that is "setuptools aware" (checks for setuptools with "if 'setuptools' in sys.modules"). Finally, a more general question: now that ctypes will become an official part of Python with 2.5, should we visit the idea of an 'official' numpy/ctypes binding? I used ctypes once about 3 years ago, so I'm not qualified to have an opinion on this, but while this stuff is in the wind... From zpincus at stanford.edu Sun Mar 12 17:04:19 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Mar 12 17:04:19 2006 Subject: [Numpy-discussion] How to tell if PyArray_FromAny has copied data? Message-ID: Hi folks, I need to figure out how to tell if PyArray_FromAny() has made a copy of the data that was provided to it. I assume that if the returned PyObject has a different address than the input object, then a copy was made somewhere along the line. Is this true? Is there a better way to tell? Before anyone asks, here's why I need to know if a copy was made. I'm writing a command to turn a numpy array into an image object suitable for sending to this 3rd party medical imaging library I've wrapped. In some cases, it is desirable to share the data buffer between an array object and an image object (and the user is responsible for ensuring that the lifetime of the image object doesn't exceed that of the array object). However, if PyArray_FromAny() has made a copy, then it is returning an entirely new object that the user will never see and so I'll need to handle the memory management differently (e.g. steal the data buffer from the new array and let the image object manage it). Alternately, I guess I could just inspect the input array object to see if it is exactly what I want, and branch on that. But running things through PyArray_FromAny seems easier, if it is possible. Zach From Fernando.Perez at colorado.edu Sun Mar 12 17:08:20 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun Mar 12 17:08:20 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C1D5.5000108@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> Message-ID: <4414C5CD.3040809@colorado.edu> Andrew Straw wrote: >> After reading the excellent cookbook examples on pyrex usage, which I >> recently needed for a project, I decided to complete the code so it could >> be used 'out of the box'. This simply meant a bit of organization work, >> updating the code to use the 'modern' cimport statement and .pxd files, >> and writing a proper setup.py file. > > > What do you mean 'modern' cimport and .pxd statement? I believe the > example at http://scipy.org/Cookbook/Pyrex_and_NumPy is fundamentally no > different than yours. Well, I'm pretty sure that when I first downloaded that code (about a month ago) it was not using cimport and pxd files, which is why I made those changes. I started using that code from a download at rev.6, which did use the 'old style', and didn't look at the wiki since: http://scipy.org/Cookbook/Pyrex_and_NumPy?action=diff&rev2=14&rev1=6 > One question, though: why have you reverted from using Python's Py_intptr_t > and used intp instead? It seems to me much more likely that Py_intptr_t > will be available and will do the Right Thing than relying on some libc to > define it. We've seen broken implementations of C libraries before, and I > say let's trust the Python devs to get this one right. Again, I was using the old code before your updates. I'm pretty sure that I did look at the wiki in February though, it's possible that my browser's cache played a trick on me, because your updates were done in February while the 'old' code seems to be from January. Anyway, it doesn't matter: what I did is very close to what you have now, I only have a more complete c_numpy file (stolen from mtrand) and a full pair of Makefile/setup.py, which can be handy for 'out of the box' usage. > Note that setuptools 'automatically' deals with .pyx files: If you pass a > source file that ends in .pyx, setuptools will run pyrexc on it if Pyrex is > available, and otherwise use a '.c' file by the same name in the same > directory. This is nice because it means you can ship .c files with your > project and people can build them, even if they don't have Pyrex installed. > Maybe we want to ship two setup.py files, or one that is "setuptools aware" > (checks for setuptools with "if 'setuptools' in sys.modules"). I certainly hope we are NOT going to make setuptools a dependency or a requirement for numpy/scipy. I've had no end of problems with it (which R. Kern has patiently dealt with in private) and I am allergic to the thing. I've seen it break plain 'setup.py install' installations with the most bizarre of errors, and I have a feeling that it's a tool whose marketing quality far outpaces its technical merits, at least in some contexts. Robert has calmly explained to me how it DOES have very useful features for certain problems, so I respect its use where needed (I've even made sure that ipython plays nicely with it, inasmuch as I can). But given how making a plain distutils-based setup.py file which knows about pyrex files (it won't call pyrexc if the .c file is present, I think) isn't that hard, I'd prefer NOT to have setuptools be a dependency. The less things numpy depends on, I think the better off we all are. Cheers, f From strawman at astraw.com Sun Mar 12 17:48:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Sun Mar 12 17:48:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C5CD.3040809@colorado.edu> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> <4414C5CD.3040809@colorado.edu> Message-ID: <4414CF35.7010200@astraw.com> Fernando Perez wrote: > Andrew Straw wrote: > >> One question, though: why have you reverted from using Python's >> Py_intptr_t >> and used intp instead? It seems to me much more likely that Py_intptr_t >> will be available and will do the Right Thing than relying on some >> libc to >> define it. We've seen broken implementations of C libraries before, >> and I >> say let's trust the Python devs to get this one right. > > Again, I was using the old code before your updates. > > Anyway, it doesn't matter: what I did is very close to what you have > now, I only have a more complete c_numpy file (stolen from mtrand) and > a full pair of Makefile/setup.py, which can be handy for 'out of the > box' usage. OK, I just hope you update your code to use Python's Py_intptr_t. I think it will be more robust. >> Note that setuptools 'automatically' deals with .pyx files: If you >> pass a >> source file that ends in .pyx, setuptools will run pyrexc on it if >> Pyrex is >> available, and otherwise use a '.c' file by the same name in the same >> directory. This is nice because it means you can ship .c files with your >> project and people can build them, even if they don't have Pyrex >> installed. >> Maybe we want to ship two setup.py files, or one that is "setuptools >> aware" >> (checks for setuptools with "if 'setuptools' in sys.modules"). > > > I certainly hope we are NOT going to make setuptools a dependency or a > requirement for numpy/scipy. I'm not proposing we do so. The 'setuptools aware' proposal above allows making use of a setuptools feature without requiring it. I agree this could be done in a few lines of code without setuptools, I was just making a suggestion for improvement to your setup.py file to make it more robust to the presence or absence of Pyrex. I like the present situation where numpy is very compatible with setuptools and I appreciate that ipython is also. I don't see what benefits numpy and scipy would get from _requiring_ setuptools, either, so we agree that there is no reason to do so. I'm sorry you've had such a bad time with setuptools but it solves many problems for me. The main developer of setuptools, Philip Eby, is very receptive to suggestions, comments, and criticisms, and given the significant chances of setuptools being incorporated into Python sooner rather than later, I'd suggest airing any remaining complaints soon or be forced to deal with them for a long time to come. From Fernando.Perez at colorado.edu Sun Mar 12 18:56:00 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun Mar 12 18:56:00 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414CF35.7010200@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> <4414C5CD.3040809@colorado.edu> <4414CF35.7010200@astraw.com> Message-ID: <4414DF08.1070707@colorado.edu> Andrew Straw wrote: > OK, I just hope you update your code to use Python's Py_intptr_t. I > think it will be more robust. Done, both on the 'real' wavelets stuff and on the exposed numpyx example. Thanks for the pointer. > I'm not proposing we do so. The 'setuptools aware' proposal above allows > making use of a setuptools feature without requiring it. I agree this > could be done in a few lines of code without setuptools, I was just > making a suggestion for improvement to your setup.py file to make it > more robust to the presence or absence of Pyrex. Thanks, that's a good point. I just fixed the setup.py file so it will not require pyrex, and added the generated C to SVN as well. This way anyone can grab the code and run it without needing anything beyond python and numpy. Cheers, f From oliphant.travis at ieee.org Sun Mar 12 19:13:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 12 19:13:02 2006 Subject: [Numpy-discussion] How to tell if PyArray_FromAny has copied data? In-Reply-To: References: Message-ID: <4414E2F6.1090000@ieee.org> Zachary Pincus wrote: > Hi folks, > > I need to figure out how to tell if PyArray_FromAny() has made a copy > of the data that was provided to it. > > I assume that if the returned PyObject has a different address than > the input object, then a copy was made somewhere along the line. Is > this true? Is there a better way to tell? > If an array object was passed in, then this will work. If, however, another object was passed in that exposes the array interface which allows data sharing, then you will get a new ArrayObject but it will share the same array data so no copying will have been done. You will be able to tell, because the returned object won't own it's own data. So, it depends on what you are going to be inputing and what kind of data sharing you are going to be checking for. > Alternately, I guess I could just inspect the input array object to > see if it is exactly what I want, and branch on that. But running > things through PyArray_FromAny seems easier, if it is possible. This actually might be easier as PyArray_FromAny can take any Python object whereas you seem to already know you have an array input. -Travis From oliphant.travis at ieee.org Sun Mar 12 19:17:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 12 19:17:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C1D5.5000108@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> Message-ID: <4414E3F9.5080905@ieee.org> Andrew Straw wrote: > What do you mean 'modern' cimport and .pxd statement? I believe the > example at http://scipy.org/Cookbook/Pyrex_and_NumPy is fundamentally no > different than yours. One question, though: why have you reverted from > using Python's Py_intptr_t and used intp instead? It seems to me much > more likely that Py_intptr_t will be available and will do the Right > Thing than relying on some libc to define it. We've seen broken > implementations of C libraries before, and I say let's trust the Python > devs to get this one right. > If you are including arrayobject.h, then intp is exactly Py_intptr_t. Since intp is what is used internally in numpy, why not use it in a numpy wrapper. So, basically, I'm not sure what your point is. I can see the case for the __array_struct__ interface which does not have to include arrayobject.h, then you need to use Py_intptr_t. -Travis From robert.kern at gmail.com Sun Mar 12 20:57:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun Mar 12 20:57:03 2006 Subject: [Numpy-discussion] Re: Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C1D5.5000108@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> Message-ID: Andrew Straw wrote: > Note that setuptools 'automatically' deals with .pyx files: If you pass > a source file that ends in .pyx, setuptools will run pyrexc on it if > Pyrex is available, and otherwise use a '.c' file by the same name in > the same directory. I'm pretty sure that numpy.distutils does this (or something like it), too. C.f. numpy/distutils/tests/pyrex_ext/. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From prabhu_r at users.sf.net Sun Mar 12 21:58:01 2006 From: prabhu_r at users.sf.net (Prabhu Ramachandran) Date: Sun Mar 12 21:58:01 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414BDEF.70001@colorado.edu> References: <4414BDEF.70001@colorado.edu> Message-ID: <17429.2130.250319.896367@prpc.aero.iitb.ac.in> >>>>> "Fernando" == Fernando Perez writes: Fernando> Hi all, there are recurrent questions on the topic of Fernando> low-level access to numpy, wrapping libraries, etc. Fernando> Many people have home-brewed SWIG typemaps, the cookbook Fernando> has some (very nice) pyrex examples, etc. I think that [...] Fernando> ... I think we would all benefit from having this kind Fernando> of 'infrastructure' code in the core. Once there, Fernando> improvements can be made over time, with less effort Fernando> wasted. +3 (one for each). Fernando> If we agree on this, it's as simple as dropping those Fernando> two directories somewhere on the numpy codebase Fernando> (sandbox?, doc?). Just having them 'officially' Fernando> available will be a benefit, I think. Yes, this will certainly be very useful. regards, prabhu From NadavH at VisionSense.com Mon Mar 13 02:03:04 2006 From: NadavH at VisionSense.com (Nadav Horesh) Date: Mon Mar 13 02:03:04 2006 Subject: [Numpy-discussion] Is there an equivalkent to APL's drop function. Message-ID: <44155ED9.40407@VisionSense.com> APL's drop is a complementary to APL's take (= numpy take). I would expect it to behave something like: an_array.drop([0,5,7],axis=0) would return an_array with elements/rows 0,5,7 missing. Nadav. From schofield at ftw.at Mon Mar 13 09:19:01 2006 From: schofield at ftw.at (Ed Schofield) Date: Mon Mar 13 09:19:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414BDEF.70001@colorado.edu> References: <4414BDEF.70001@colorado.edu> Message-ID: <4415A968.40807@ftw.at> Fernando Perez wrote: > I think it would be a good idea to ship the numpy_swig example > directory (and perhaps also numpyx, if somewhat expanded) listed above > with the standard distribution, as well as completing the .pxd file to > expose all the numpy C API for pyrex. > > I am not claiming to have done any of the hard work in numpyx or > numpy_swig (though I did spend a lot of time on the wavelets stuff > recently :), but I think we would all benefit from having this kind of > 'infrastructure' code in the core. Once there, improvements can be > made over time, with less effort wasted. Great! Thanks to you and everyone else who has worked on this. > If we agree on this, it's as simple as dropping those two directories > somewhere on the numpy codebase (sandbox?, doc?). Just having them > 'officially' available will be a benefit, I think. So, just to clarify, you think we should house the SWIG typemaps in the NumPy tree temporarily, until they're vetted, and then move them over to the SWIG tree? -- Ed From oliphant.travis at ieee.org Mon Mar 13 09:27:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 13 09:27:03 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415A968.40807@ftw.at> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> Message-ID: <4415AB45.3000702@ieee.org> Ed Schofield wrote: > Fernando Perez wrote: > > So, just to clarify, you think we should house the SWIG typemaps in the > NumPy tree temporarily, until they're vetted, and then move them over to > the SWIG tree? > I think that's what he was saying. I think it's a good idea. I've created both a pyrex and a swig directory under numpy/doc and placed the files there. More can be added, of course, like an example of how to use them. They will get "installed" to the doc directory as well: /numpy/doc At least they are in a place visible to all users. Regarding numpy 0.9.6: I made enough changes to the numpy tree that I wanted to wait a day or two to let obvious bugs I may have created rise to the surface before making another release. I'll test matplotlib today and if that goes well and I don't hear any other complaints, I'll make a release. -Travis From jdhunter at ace.bsd.uchicago.edu Mon Mar 13 09:53:04 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Mon Mar 13 09:53:04 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> (Travis Oliphant's message of "Mon, 13 Mar 2006 10:26:29 -0700") References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <87irqikzbr.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Travis" == Travis Oliphant writes: Travis> I made enough changes to the numpy tree that I wanted to Travis> wait a day or two to let obvious bugs I may have created Travis> rise to the surface before making another release. I'll Travis> test matplotlib today and if that goes well and I don't Travis> hear any other complaints, I'll make a release. Just a reminder to those who frequently update their numpy and matplotlib from src. I had a segfault today when importing matplotlib/pylab with numpy, and this was fixed simply by > cd matplotlib_src > sudo rm -rf build > sudo python setup.py install So if you experience problems, try a clean rebuild. JDH From Fernando.Perez at colorado.edu Mon Mar 13 10:20:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 13 10:20:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <4415B786.9010003@colorado.edu> Travis Oliphant wrote: > Ed Schofield wrote: > >>Fernando Perez wrote: >> >>So, just to clarify, you think we should house the SWIG typemaps in the >>NumPy tree temporarily, until they're vetted, and then move them over to >>the SWIG tree? >> > > > I think that's what he was saying. I think it's a good idea. Yes, that was my point. For example, the typemaps I sent were 1 and 2d only, and some people have posted 3-d and n-d ones as well, so this is not complete yet. But having it in the core will make sure that the contributions are actually recorded and improve our common usage tools. Once we feel they are solid enough, we can (if we think it's worthwhile) send them to the SWIG team. Though frankly, I think the most important part was having /something/ official in the numpy distribution itself: anyone needing this by definition has numpy, so we may actually be better off controlling this code rather than relying on SWIG versions and their release schedule. In any case, that can be discussed later, the point was to have this, and I'm very thankful to Travis for pulling it in. I'll delete it from the wavelets repo so we only have one copy of this code lying around. Cheers, f From strawman at astraw.com Mon Mar 13 11:21:07 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Mar 13 11:21:07 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <4415C609.6090500@astraw.com> Travis Oliphant wrote: > I made enough changes to the numpy tree that I wanted to wait a day or > two to let obvious bugs I may have created rise to the surface before > making another release. I'll test matplotlib today and if that goes > well and I don't hear any other complaints, I'll make a release. > Hi Travis, I'm getting one test failure with svn 2230 on my amd64 arch (below). Other than that, the various bits of code I've tested seem to be working OK. FAIL: check_both (numpy.core.tests.test_multiarray.test_pickling) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 235, in check_both assert_equal(self.carray, pickle.loads(self.carray.dumps())) File "/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/testing/utils.py", line 117, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/testing/utils.py", line 204, in assert_array_equal assert cond,\ AssertionError: Arrays are not equal (mismatch 66.6666666667%): Array 1: [[2 9] [7 0] [3 8]] Array 2: [[2 0] [9 3] [7 8]] ---------------------------------------------------------------------- Ran 341 tests in 0.634s FAILED (failures=1) >>> numpy.__file__ '/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/__init__.pyc' From Fernando.Perez at colorado.edu Mon Mar 13 11:23:03 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 13 11:23:03 2006 Subject: [Numpy-discussion] [Fwd: Re: [SAGEdev] faster random number generation] Message-ID: <4415C66C.2090405@colorado.edu> Hi all, I'm wondering if one of our resident experts who is more familiar than me with the details of the current build process could help us here a little. This is a message from the SAGE development list, where I am encouraging them to integrate more tightly with numpy. SAGE is a very interesting python-based system for mathematics: http://modular.ucsd.edu/sage/ After my comment below on numpy NOT needing a fortran compiler, William came back with that g77 error message. Now I'm a little confused, and before I confuse him even more, I'd prefere to clarify things with those who actually konw what's going on. Any help will be greatly appreciated. Cheers, f -------- Original Message -------- Subject: Re: [SAGEdev] faster random number generation Date: Mon, 13 Mar 2006 10:43:01 -0800 From: William Stein Reply-To: sage-devel at lists.sourceforge.net Organization: UC San Diego To: sage-devel at lists.sourceforge.net References: <4db014670603111659id0dccb9g5619a930c3ac9b27 at mail.gmail.com> <200603130111.00929.wstein at ucsd.edu> <4415B68E.4080207 at colorado.edu> On Monday 13 March 2006 10:14, Fernando Perez wrote: > No: numpy requires only standard python + a C compiler to build the > extensions. All the code in /numpy/ is pure C. Scipy, on the other hand, > /does/ need a Fortran compiler (not necessarily gcc, it's been compiled > with quite a few others, though it is not always a trivial matter to do > so). I'm confused then, because the output when building numpy (e.g., from David Joyner's build failure report) includes g77 calls. Are we building with the wrong options? building 'numpy.core._dotblas' extension compiling C sources gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/home/wdj/sagefiles/sage-1.0.6/local/include/python2.4 -fPIC' creating build/temp.linux-x86_64-2.4/numpy/core/blasdot compile options: '-DNO_ATLAS_INFO=1 -Inumpy/core/blasdot -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/home/wdj/sagefiles/sage-1.0.6/local/include/python2.4 -c' gcc: numpy/core/blasdot/_dotblas.c numpy/core/blasdot/_dotblas.c: In function 'dotblas_matrixproduct': numpy/core/blasdot/_dotblas.c:166: warning: 'ap1stride' may be used uninitialized in this function \/\/\/\/\/\/ /usr/bin/g77 build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.4/numpy/core/_dotblas.so build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_alterdot':numpy/core/blasdot/_dotblas.c:65: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:89: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_restoredot':numpy/core/blasdot/_dotblas.c:100: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:126: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_innerproduct':numpy/core/blasdot/_dotblas.c:656: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:687: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_vdot':numpy/core/blasdot/_dotblas.c:896: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:927: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:999: undefined reference to `Py_InitModule4' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `import_array':build/src/numpy/core/__multiarray_api.h:763: undefined reference to `PyImport_ImportModule' :build/src/numpy/core/__multiarray_api.h:766: undefined reference to `PyObject_GetAttrString' :build/src/numpy/core/__multiarray_api.h:768: undefined reference to `PyCObject_Type' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyExc_RuntimeError' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyErr_Format' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:1009: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':build/src/numpy/core/__multiarray_api.h:769: undefined reference to `PyCObject_AsVoidPtr' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_matrixproduct':numpy/core/blasdot/_dotblas.c:177: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:208: undefined reference to `PyTuple_New' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyErr_SetString' /usr/lib/libfrtbegin.a(frtbegin.o): In function `main': undefined reference to `MAIN__' collect2: ld returned 1 exit status build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_alterdot':numpy/core/blasdot/_dotblas.c:65: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:89: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_restoredot':numpy/core/blasdot/_dotblas.c:100: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:126: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_innerproduct':numpy/core/blasdot/_dotblas.c:656: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:687: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_vdot':numpy/core/blasdot/_dotblas.c:896: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:927: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:999: undefined reference to `Py_InitModule4' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `import_array':build/src/numpy/core/__multiarray_api.h:763: undefined reference to `PyImport_ImportModule' :build/src/numpy/core/__multiarray_api.h:766: undefined reference to `PyObject_GetAttrString' :build/src/numpy/core/__multiarray_api.h:768: undefined reference to `PyCObject_Type' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyExc_RuntimeError' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyErr_Format' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:1009: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':build/src/numpy/core/__multiarray_api.h:769: undefined reference to `PyCObject_AsVoidPtr' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_matrixproduct':numpy/core/blasdot/_dotblas.c:177: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:208: undefined reference to `PyTuple_New' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyErr_SetString' /usr/lib/libfrtbegin.a(frtbegin.o): In function `main': undefined reference to `MAIN__' collect2: ld returned 1 exit status error: Command "/usr/bin/g77 build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.4/numpy/core/_dotblas.so" failed with exit status 1 ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd_______________________________________________ sage-devel mailing list sage-devel at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sage-devel From oliphant at ee.byu.edu Mon Mar 13 13:29:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 13 13:29:03 2006 Subject: [Numpy-discussion] [Fwd: Re: [SAGEdev] faster random number generation] In-Reply-To: <4415C66C.2090405@colorado.edu> References: <4415C66C.2090405@colorado.edu> Message-ID: <4415E3F9.6070409@ee.byu.edu> Fernando Perez wrote: > Hi all, > > I'm wondering if one of our resident experts who is more familiar than > me with the details of the current build process could help us here a > little. This is a message from the SAGE development list, where I am > encouraging them to integrate more tightly with numpy. > > SAGE is a very interesting python-based system for mathematics: > > http://modular.ucsd.edu/sage/ > > After my comment below on numpy NOT needing a fortran compiler, > William came back with that g77 error message. Now I'm a little > confused, and before I confuse him even more, I'd prefere to clarify > things with those who actually konw what's going on. > > Any help will be greatly appreciated. Pearu is the expert on this. The _dotblas.c function is optimized dot using ATLAS and/or BLAS for the system. I'm not sure why g77 is being used as the linker unless it is in their configuration somehow. For example, on my system, this is the command that links _dotblas.so cc -pthread -shared build/temp.linux-i686-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib/atlas -lptf77blas -lptcblas -latlas -o build/lib.linux-i686-2.4/numpy/core/_dotblas.so Please have them post to the list to figure out their issues. I want to squash any idea that numpy *requires* a Fortran compiler as this was a major design decision not to. It's always possible that configuration issues need to be improved for certain platforms. -Travis From Fernando.Perez at colorado.edu Mon Mar 13 13:33:03 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 13 13:33:03 2006 Subject: [Numpy-discussion] [Fwd: Re: [SAGEdev] faster random number generation] In-Reply-To: <4415E3F9.6070409@ee.byu.edu> References: <4415C66C.2090405@colorado.edu> <4415E3F9.6070409@ee.byu.edu> Message-ID: <4415E4F4.60704@colorado.edu> Travis Oliphant wrote: > Fernando Perez wrote: >>After my comment below on numpy NOT needing a fortran compiler, >>William came back with that g77 error message. Now I'm a little >>confused, and before I confuse him even more, I'd prefere to clarify >>things with those who actually konw what's going on. >> >>Any help will be greatly appreciated. > > > > Pearu is the expert on this. The _dotblas.c function is optimized dot > using ATLAS and/or BLAS for the system. I'm not sure why g77 is being > used as the linker unless it is in their configuration somehow. > > For example, on my system, this is the command that links _dotblas.so > > cc -pthread -shared > build/temp.linux-i686-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib/atlas > -lptf77blas -lptcblas -latlas -o > build/lib.linux-i686-2.4/numpy/core/_dotblas.so > > Please have them post to the list to figure out their issues. I want to > squash any idea that numpy *requires* a Fortran compiler as this was a > major design decision not to. It's always possible that configuration > issues need to be improved for certain platforms. Thanks, Travis. William, care to join in the numpy list with more details? As you see, it is true that numpy is NOT supposed to require a fortran compiler, this is something that was very specifically discussed and agreed upon. But we may need you to join in to get more information about how you are doing things, before more help can be provided. Here's the list link for you to join in: http://lists.sourceforge.net/lists/listinfo/numpy-discussion Cheers, f From oliphant at ee.byu.edu Mon Mar 13 13:42:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 13 13:42:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415C609.6090500@astraw.com> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <4415C609.6090500@astraw.com> Message-ID: <4415E6EB.9070800@ee.byu.edu> Andrew Straw wrote: >Hi Travis, > >I'm getting one test failure with svn 2230 on my amd64 arch (below). >Other than that, the various bits of code I've tested seem to be working OK. > > Can you try it again. I think I found the problem (a variable declared as long instead of int). -Travis From fonnesbeck at gmail.com Mon Mar 13 13:44:02 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Mon Mar 13 13:44:02 2006 Subject: [Numpy-discussion] inconsistent dot() behaviour Message-ID: <723eb6930603131343k6f0fdcb8g180a2086ce2dc914@mail.gmail.com> I notice that the dot() function does not behave consistently across all sizes of arrays: In [7]: dot(array([1.0]), array([-0.27163628])) Out[7]: array([-0.27163628]) In [8]: dot(array([1.0,1.0]), array([-0.27163628,1.0])) Out[8]: 0.72836371999999994 Why should a 1x1 dot product produce an array, while a 2x2 (or greater) product produces a scalar? Using recent svn build of numpy. Thanks, Chris -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us From oliphant at ee.byu.edu Mon Mar 13 14:10:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 13 14:10:09 2006 Subject: [Numpy-discussion] inconsistent dot() behaviour In-Reply-To: <723eb6930603131343k6f0fdcb8g180a2086ce2dc914@mail.gmail.com> References: <723eb6930603131343k6f0fdcb8g180a2086ce2dc914@mail.gmail.com> Message-ID: <4415EDAD.5080408@ee.byu.edu> Chris Fonnesbeck wrote: >I notice that the dot() function does not behave consistently across >all sizes of arrays: > >In [7]: dot(array([1.0]), array([-0.27163628])) >Out[7]: array([-0.27163628]) > >In [8]: dot(array([1.0,1.0]), array([-0.27163628,1.0])) >Out[8]: 0.72836371999999994 > >Why should a 1x1 dot product produce an array, while a 2x2 (or >greater) product produces a scalar? > > It's an error. Basically, you can check to make sure the optimized dot function is doing what it should by running from numpy.core.multiarray import dot as olddot and checking the output of olddot against the optimized dot output. They should be the same or it's a problem. We should write a test that loops through problems of dimensions 0, 1, 2 and computes the result both ways checking the output for equivalency. Some of this is done, but obviously another case was missed. -Travis From strawman at astraw.com Mon Mar 13 14:43:08 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Mar 13 14:43:08 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415E6EB.9070800@ee.byu.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <4415C609.6090500@astraw.com> <4415E6EB.9070800@ee.byu.edu> Message-ID: <4415F552.30103@astraw.com> Travis Oliphant wrote: > Andrew Straw wrote: > >> Hi Travis, >> >> I'm getting one test failure with svn 2230 on my amd64 arch (below). >> Other than that, the various bits of code I've tested seem to be >> working OK. >> >> > Can you try it again. I think I found the problem (a variable declared > as long instead of int). Yep, all tests pass now on that machine with svn 2232. From ofnap at nus.edu.sg Mon Mar 13 15:31:04 2006 From: ofnap at nus.edu.sg (Ajith Prasad (OFN)) Date: Mon Mar 13 15:31:04 2006 Subject: [Numpy-discussion] USING NUMPY WITH EXCEL Message-ID: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg> I am interested in using NumPy with Excel - eg, entering array data in Excel ranges, doing array computations using NumPy, and presenting the results in Excel. Are there any references or templates that I could use for this purpose? Thanks in advance for any help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dd55 at cornell.edu Mon Mar 13 16:44:02 2006 From: dd55 at cornell.edu (Darren Dale) Date: Mon Mar 13 16:44:02 2006 Subject: [Numpy-discussion] gcc-4.1 Message-ID: <200603131943.53857.dd55@cornell.edu> Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 and gcc-4.1. When I try to build numpy I get warnings like "warning: build_ext: fcompiler=gnu is not available." I tried forcing numpy to use gfortran with "python setup.py config_fc --fcompiler=gnu95 build", and got the same warning for fcompiler=gnu95. Here is the output of "gfortran --version": GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) Copyright (C) 2006 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING And here is the output of "python setup.py config_fc --help-fcompiler": Running from numpy source directory. running config_fc customize CompaqFCompiler customize NoneFCompiler customize AbsoftFCompiler Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize SunFCompiler customize VastFCompiler customize GnuFCompiler customize IbmFCompiler customize Gnu95FCompiler customize IntelVisualFCompiler customize G95FCompiler customize IntelItaniumFCompiler customize PGroupFCompiler customize LaheyFCompiler customize CompaqVisualFCompiler customize MipsFCompiler customize HPUXFCompiler customize IntelItaniumVisualFCompiler customize NAGFCompiler List of available Fortran compilers: List of unavailable Fortran compilers: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaq Compaq Fortran Compiler --fcompiler=compaqv DIGITAL|Compaq Visual Fortran Compiler --fcompiler=g95 GNU Fortran 95 Compiler --fcompiler=gnu GNU Fortran Compiler --fcompiler=gnu95 GNU 95 Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun|Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler List of unimplemented Fortran compilers: --fcompiler=f Fortran Company/NAG F Compiler For compiler details, run 'config_fc --verbose' setup command. Thanks, Darren From jdhunter at ace.bsd.uchicago.edu Mon Mar 13 17:27:04 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Mon Mar 13 17:27:04 2006 Subject: [Numpy-discussion] USING NUMPY WITH EXCEL In-Reply-To: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg> ("Ajith Prasad's message of "Tue, 14 Mar 2006 07:30:15 +0800") References: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg> Message-ID: <87k6axzulj.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Ajith" == Ajith Prasad (OFN) writes: Ajith> I am interested in using NumPy with Excel - eg, entering Ajith> array data in Excel ranges, doing array computations using Ajith> NumPy, and presenting the results in Excel. Are there any Ajith> references or templates that I could use for this purpose? Ajith> Thanks in advance for any help. Mark Hammond's "Python Programming On Win32 : Help for Windows Programmers " http://packages.debian.org/unstable/python/python2.3-matplotlib covers the API for interacting with Excel using his win32 python extensions http://starship.python.net/crew/mhammond/ Here's an online tutorial http://www.markcarter.me.uk/computing/python/excel.html There is nothing numpy specific to master. Once you figure out how to get sequences from excel to python and back, you can plug numpy into the pipeline. JDH From zpincus at stanford.edu Mon Mar 13 17:28:07 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 13 17:28:07 2006 Subject: [Numpy-discussion] dot and deallocation bug? Message-ID: Hi folks, I've run across an odd error in my numpy install related to numpy.dot with python lists of floats. Now, I haven't updated from svn in a while (I'm currently using numpy to do some data analysis, so I'm not changing things until that's done). Perhaps someone could confirm whether this is still a bug in the current version? Here's how I reproduce the error (OS X 10.4.5), python 2.4.2: In [1]: import numpy In [2]: numpy.version.version Out[2]: '0.9.6.2208' In [3]: numpy.dot([0.0, 1.0], [1, 2, 3]) ValueError: matrices are not aligned In [4]: numpy.dot([0.0, 1.0], [1, 2, 3]) ValueError: matrices are not aligned In [5]: numpy.dot([0.0, 1.0], [1, 2, 3]) Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug ValueError: matrices are not aligned In [6]: numpy.dot([0.0, 1.0], [1, 2, 3]) Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug ValueError: matrices are not aligned Always the third time after I do this incorrect dot operation, the malloc errors start appearing. The more times I do the operation, the longer the list of malloc errors gets. If the input lists are not floating-point, this doesn't happen (also the ValueError states: 'objects are not aligned' instead of 'matrices are not aligned'). Any thoughts? Zach Pincus From robert.kern at gmail.com Mon Mar 13 18:08:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 13 18:08:04 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: <200603131943.53857.dd55@cornell.edu> References: <200603131943.53857.dd55@cornell.edu> Message-ID: Darren Dale wrote: > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my home > computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 and > gcc-4.1. When I try to build numpy I get warnings like "warning: build_ext: > fcompiler=gnu is not available." I tried forcing numpy to use gfortran with > "python setup.py config_fc --fcompiler=gnu95 build", and got the same warning > for fcompiler=gnu95. Building numpy shouldn't require any kind of Fortran compiler. However, the problem is that the version-recognizing regex doesn't like the version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in numpy/distutils/fcompiler/gnu.py and the class attribute Gnu95FCompiler.version_pattern . -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Fernando.Perez at colorado.edu Mon Mar 13 21:13:01 2006 From: Fernando.Perez at colorado.edu (Fernando.Perez at colorado.edu) Date: Mon Mar 13 21:13:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <1142313130.441650aad8111@webmail.colorado.edu> Quoting Travis Oliphant : > I think that's what he was saying. I think it's a good idea. I've > created both a pyrex and a swig directory under numpy/doc and placed the > files there. More can be added, of course, like an example of how to > use them. > > They will get "installed" to the doc directory as well: > /numpy/doc Just curious Travis: why did you only put in the .i/.pxd files and not the whole directories? I think that for this particular problem, there is a lot of value in the surrounding code. In both cases, the extra code provides examples of how to use each tool (swig/pyrex), there are tests, and a valid setup.py script. For a user who is not familiar with distutils, this can be quite useful, and I think in general it is a lot easier to get off the ground with a new system when you can grab an example directory which works 'out of the box' and start modifying it. If you have any concerns about the code, let me know and I'll be happy to address them, but I really think that this will be much more useful to everyone if each directory contains a complete, working example. Regards, f From oliphant.travis at ieee.org Mon Mar 13 23:18:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 13 23:18:02 2006 Subject: [Numpy-discussion] dot and deallocation bug? In-Reply-To: References: Message-ID: <44166DF9.20602@ieee.org> Zachary Pincus wrote: > Hi folks, > > I've run across an odd error in my numpy install related to numpy.dot > with python lists of floats. > > Now, I haven't updated from svn in a while (I'm currently using numpy > to do some data analysis, so I'm not changing things until that's > done). Perhaps someone could confirm whether this is still a bug in > the current version? > > Here's how I reproduce the error (OS X 10.4.5), python 2.4.2: This is a bug in the error-handling section of the optimized BLAS code. I see it too. The reason you don't see it for non floating point and/or higher than 2-d is that the there is an optimized dot function that is taking over for floating point and complex floating point code which uses the BLAS. So, you are exercising different sections of code. The optimized _dotblas.c file has been the source of quite a few bugs. The slower but less buggy version is always available as numpy.core.multiarray.dot Thanks for finding this one. -Travis From oliphant.travis at ieee.org Mon Mar 13 23:39:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 13 23:39:01 2006 Subject: [Numpy-discussion] dot and deallocation bug? In-Reply-To: <44166DF9.20602@ieee.org> References: <44166DF9.20602@ieee.org> Message-ID: <441672FF.3070904@ieee.org> Travis Oliphant wrote: >> Hi folks, >> >> I've run across an odd error in my numpy install related to numpy.dot >> with python lists of floats. >> >> Now, I haven't updated from svn in a while (I'm currently using numpy >> to do some data analysis, so I'm not changing things until that's >> done). Perhaps someone could confirm whether this is still a bug in >> the current version? >> >> Here's how I reproduce the error (OS X 10.4.5), python 2.4.2: > This is a bug in the error-handling section of the optimized BLAS > code. I see it too. > > The reason you don't see it for non floating point and/or higher than > 2-d is that the there is an optimized dot function that is taking over > for floating point and complex floating point code which uses the BLAS. > So, you are exercising different sections of code. The optimized > _dotblas.c file has been the source of quite a few bugs. The slower > but less buggy version is always available as numpy.core.multiarray.dot > > Thanks for finding this one. Update: This should be fixed in SVN now. At least the segfault I was seeing disappeared. There was a missing INCREF for a data-type descriptor. This would definitely cause those kinds of errors. -Travis From aisaac at american.edu Tue Mar 14 00:05:00 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue Mar 14 00:05:00 2006 Subject: [Numpy-discussion] USING NUMPY WITH EXCEL In-Reply-To: <87k6axzulj.fsf@peds-pc311.bsd.uchicago.edu> References: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg><87k6axzulj.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: On Mon, 13 Mar 2006, John Hunter apparently wrote: > Here's an online tutorial > http://www.markcarter.me.uk/computing/python/excel.html You may also find this useful: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbaxl10/html/xltocobjectmodelapplication.asp Please post a nice example when you produce one. Cheers, Alan Isaac From oliphant.travis at ieee.org Tue Mar 14 01:06:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 14 01:06:03 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released Message-ID: <44168767.80803@ieee.org> This post is to announce the release of NumPy 0.9.6 which fixes some important bugs and has several speed improvments. NumPy is a multi-dimensional array-package for Python that allows rapid high-level array computing with Python. It is successor to both Numeric and Numarray. More information at http://numeric.scipy.org The release notes are attached: Best regards, NumPy Developers -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: release-notes-0.9.6 URL: From wbaxter at gmail.com Tue Mar 14 03:53:00 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue Mar 14 03:53:00 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: Just wondering, does this one also require an update to scipy? And in general do numpy updates always require an update to scipy, too? Or is it only when the numpy C API interface changes? --bb On 3/14/06, Travis Oliphant wrote: > > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows rapid > high-level array computing with Python. It is successor to both Numeric > and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > > > > > > NumPy 0.9.6 is a bug-fix and optimization release with a > few new features: > > > New features (and changes): > > - bigndarray removed and support for Python2.5 ssize_t added giving > full support in Python2.5 to very-large arrays on 64-bit systems. > > - Strides can be set more arbitrarily from Python (and checking is done > to make sure memory won't be violated). > > - __array_finalize__ is now called for every array sub-class creation. > > - kron and repmat functions added > > - .round() method added for arrays > > - rint, square, reciprocal, and ones_like ufuncs added. > > - keyword arguments now possible for methods taking a single 'axis' > argument > > - Swig and Pyrex examples added in doc/swig and doc/pyrex > > - NumPy builds out of the box for cygwin > > - Different unit testing naming schemes are now supported. > > - memusage in numpy.distutils works for NT platforms > > - numpy.lib.math functions now take vectors > > - Most functions in oldnumeric now return intput class where possible > > > Speed ups: > > - x**n for integer n signficantly improved > > - array() much faster > > - .fill() method is much faster > > > Other fixes: > > - Output arrays to ufuncs works better. > > - Several ma (Masked Array) fixes. > > - umath code generation improved > > - many fixes to optimized dot function (fixes bugs in > matrix-sub-class multiply) > > - scalartype fixes > > - improvements to poly1d > > - f2py fixed to handle character arrays in common blocks > > - Scalar arithmetic improved to handle mixed-mode operation. > > - Make sure Python intYY types correspond exactly with C PyArray_INTYY > > > > > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Tue Mar 14 06:08:11 2006 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Mar 14 06:08:11 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8EF52@exchange2k.envision.co.il> There is a compatibility problem, at least with the last formal release of scipy. Should we checkout and compile scipy from svn? Nadav. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net on behalf of Bill Baxter Sent: Tue 14-Mar-06 13:52 To: numpy-discussion; SciPy Users List Cc: Subject: Re: [Numpy-discussion] [ANN] NumPy 0.9.6 released Just wondering, does this one also require an update to scipy? And in general do numpy updates always require an update to scipy, too? Or is it only when the numpy C API interface changes? --bb On 3/14/06, Travis Oliphant wrote: > > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows rapid > high-level array computing with Python. It is successor to both Numeric > and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > > > > > > NumPy 0.9.6 is a bug-fix and optimization release with a > few new features: > > > New features (and changes): > > - bigndarray removed and support for Python2.5 ssize_t added giving > full support in Python2.5 to very-large arrays on 64-bit systems. > > - Strides can be set more arbitrarily from Python (and checking is done > to make sure memory won't be violated). > > - __array_finalize__ is now called for every array sub-class creation. > > - kron and repmat functions added > > - .round() method added for arrays > > - rint, square, reciprocal, and ones_like ufuncs added. > > - keyword arguments now possible for methods taking a single 'axis' > argument > > - Swig and Pyrex examples added in doc/swig and doc/pyrex > > - NumPy builds out of the box for cygwin > > - Different unit testing naming schemes are now supported. > > - memusage in numpy.distutils works for NT platforms > > - numpy.lib.math functions now take vectors > > - Most functions in oldnumeric now return intput class where possible > > > Speed ups: > > - x**n for integer n signficantly improved > > - array() much faster > > - .fill() method is much faster > > > Other fixes: > > - Output arrays to ufuncs works better. > > - Several ma (Masked Array) fixes. > > - umath code generation improved > > - many fixes to optimized dot function (fixes bugs in > matrix-sub-class multiply) > > - scalartype fixes > > - improvements to poly1d > > - f2py fixed to handle character arrays in common blocks > > - Scalar arithmetic improved to handle mixed-mode operation. > > - Make sure Python intYY types correspond exactly with C PyArray_INTYY > > > > > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 From cjw at sympatico.ca Tue Mar 14 07:56:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 14 07:56:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: <4416E747.306@sympatico.ca> Travis Oliphant wrote: > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows > rapid high-level array computing with Python. It is successor to both > Numeric and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > - __array_finalize__ is now called for every array sub-class creation. > > Thanks. What are the parameters for this method and what does it do? How does it differ from the common Python usage of __new__ followed by __init__? [Dbg]>>> Help on built-in function __array_finalize__: __array_finalize__(...) Colin W. From oliphant at ee.byu.edu Tue Mar 14 12:26:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 14 12:26:05 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <51CE8530-E9F6-4AE1-A1CF-9FF6C6FEB25F@nrl.navy.mil> References: <44168767.80803@ieee.org> <44170C7E.6070208@ieee.org> <51CE8530-E9F6-4AE1-A1CF-9FF6C6FEB25F@nrl.navy.mil> Message-ID: <441726C7.3020207@ee.byu.edu> Paul Ray wrote: >On Mar 14, 2006, at 1:33 PM, Travis Oliphant wrote: > > > >>>Just wondering, does this one also require an update to scipy? >>>And in general do numpy updates always require an update to scipy, >>>too? >>>Or is it only when the numpy C API interface changes? >>> >>> >>It's only when the C-API interface changes that this is necessary. >>Pre-1.0 it is likely to happen at each release. >> >>After that it is less likely to happen. >> >> > >Pre-1.0, I think it is fine for the version numbering to be a free- >for-all, as it has been. There have been many times, it seems, where >the latest numpy didn't work with the latest scipy and stuff like that. > > The current SVN of numpy and the current SVN of scipy usually work with each other (perhaps with only a day lag). >After 1.0, I would like to encourage some discipline with regards to >numpy version numbering. From 1.0, the API should NOT change at all >until 1.1. > This is the plan and it is why we are being slow with getting 1.0 out. But, we *need* people to try NumPy on their codes so that we can determine whether or not API additions are needed. Without this testing we are just guessing based on the developer's own codes which do not cover the breadth. >I think that many people with codes that use Numeric or numarray are >awaiting the numpy 1.0 release as a sign that it is stable and ready >for prime time, before converting their codes. > > We need some of these people to convert earlier than that, though, so we can make sure that 1.0 really is ready for prime time. I think it's very close already or the version number wouldn't be so high. We are just waiting for more people to start using it and report any issues that they have before going to 1.0 (I'd also like scalar math to be implemented pre 1.0 as well in case that requires any C-API additions). Thanks for the feedback. What you described will be the basic behavior once 1.0 is released. Note that right now the only changes that usually need to be made are a re-compile. I know this can still be annoying when you have a deep code stack. -Travis From Paul.Ray at nrl.navy.mil Tue Mar 14 12:39:02 2006 From: Paul.Ray at nrl.navy.mil (Paul Ray) Date: Tue Mar 14 12:39:02 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <441726C7.3020207@ee.byu.edu> References: <44168767.80803@ieee.org> <44170C7E.6070208@ieee.org> <51CE8530-E9F6-4AE1-A1CF-9FF6C6FEB25F@nrl.navy.mil> <441726C7.3020207@ee.byu.edu> Message-ID: <64DF7D92-839F-4B49-AC08-D757E4753580@nrl.navy.mil> On Mar 14, 2006, at 3:25 PM, Travis Oliphant wrote: >> I think that many people with codes that use Numeric or numarray are >> awaiting the numpy 1.0 release as a sign that it is stable and ready >> for prime time, before converting their codes. >> > We need some of these people to convert earlier than that, though, > so we > can make sure that 1.0 really is ready for prime time. I think it's > very close already or the version number wouldn't be so high. We are > just waiting for more people to start using it and report any issues > that they have before going to 1.0 (I'd also like scalar math to be > implemented pre 1.0 as well in case that requires any C-API > additions). I agree. Lots of people do seem to be trying out numpy. I use it for all new code I develop, and I have converted over a bunch of older SciPy stuff that uses both the python and C APIs, including ppgplot for plotting. I also heard from the PyFITS developers that an alpha release based on numpy is coming very soon. Cheers, -- Paul From svetosch at gmx.net Tue Mar 14 13:20:03 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue Mar 14 13:20:03 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: <44173352.4020403@gmx.net> Hi, are the following issues of the new release known? (This is on winxp.) >>> import numpy as n >>> n.__version__ '0.9.6' >>> a = n.asmatrix(n.eye(3)) >>> a matrix([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> n.linalg.inverse(a) array([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]) I was asked to report if functions in linalg still don't honor the matrix input type; voila. >>> n.linalg.cholesky(a) Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\linalg\linalg.py", line 135, in cholesky_decomposition return transpose(triu(a,k=0)).copy() NameError: global name 'triu' is not defined I think I've seen this error before, was it in 0.9.4? This is a real show-stopper. Happy bug-hunting, sven Travis Oliphant schrieb: > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows rapid > high-level array computing with Python. It is successor to both Numeric > and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > > > > ------------------------------------------------------------------------ > > > NumPy 0.9.6 is a bug-fix and optimization release with a > few new features: > > > New features (and changes): > > - bigndarray removed and support for Python2.5 ssize_t added giving > full support in Python2.5 to very-large arrays on 64-bit systems. > > - Strides can be set more arbitrarily from Python (and checking is done > to make sure memory won't be violated). > > - __array_finalize__ is now called for every array sub-class creation. > > - kron and repmat functions added > > - .round() method added for arrays > > - rint, square, reciprocal, and ones_like ufuncs added. > > - keyword arguments now possible for methods taking a single 'axis' > argument > > - Swig and Pyrex examples added in doc/swig and doc/pyrex > > - NumPy builds out of the box for cygwin > > - Different unit testing naming schemes are now supported. > > - memusage in numpy.distutils works for NT platforms > > - numpy.lib.math functions now take vectors > > - Most functions in oldnumeric now return intput class where possible > > > Speed ups: > > - x**n for integer n signficantly improved > > - array() much faster > > - .fill() method is much faster > > > Other fixes: > > - Output arrays to ufuncs works better. > > - Several ma (Masked Array) fixes. > > - umath code generation improved > > - many fixes to optimized dot function (fixes bugs in > matrix-sub-class multiply) > > - scalartype fixes > > - improvements to poly1d > > - f2py fixed to handle character arrays in common blocks > > - Scalar arithmetic improved to handle mixed-mode operation. > > - Make sure Python intYY types correspond exactly with C PyArray_INTYY > > From mpi at osc.kiku.dk Tue Mar 14 13:24:11 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Tue Mar 14 13:24:11 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays Message-ID: Hey, First of all, thanks for the new release. Here's another question regarding something I cannot quite understand: Suppose you want to update bins for a histogram, you might think you could do something like: g = zeros(4,Int) x = array([0.2, 0.2]) idx = floor(x/0.1).astype(int) g[idx] += 1 Here idx becomes array([2, 2]) In this case, I would naively expect g to end up like array([0, 0, 2, 0]) (1) but instead you get array([0, 0, 1, 0]) (2) Is this intended? Just being plain novice-like naive, I would expect the slice operation g[idx] += 1 to do something like for i in range(len(I)): g[ idx[i] ] += 1 resulting in (1) and not (2). // Mads From Fernando.Perez at colorado.edu Tue Mar 14 13:41:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Mar 14 13:41:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: <4417383B.4050000@colorado.edu> Travis Oliphant wrote: > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. Just curious about the svd/singular... pair: In [2]: numpy.linalg.singular_value_decomposition?? Definition: numpy.linalg.singular_value_decomposition(A, full_matrices=0) Source: def singular_value_decomposition(A, full_matrices=0): return svd(A, full_matrices) In [3]: numpy.linalg.svd? Definition: numpy.linalg.svd(a, full_matrices=1) Do we really need both spellings to be different just on the default of the full_matrices flag? It seems to me like the kind of unnecessary special-casing that just forces people to remember little details without a clear benefit. If we want a long-spelling form, it would be cheaper to simply alias the name with a module-level singular_value_decomposition = svd so the same /object/ is referred to via both names. Perhaps I'm missing the blindingly obvious solid reason, but if this is just historical accident, I'd vote for unifying the two simply as spelling variants. I think the flipped default is more confusion-prone than beneficial. Cheers, f From robert.kern at gmail.com Tue Mar 14 14:42:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 14 14:42:07 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: <4417383B.4050000@colorado.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> Message-ID: Fernando Perez wrote: > Travis Oliphant wrote: > >> This post is to announce the release of NumPy 0.9.6 which fixes some >> important bugs and has several speed improvments. > > Just curious about the svd/singular... pair: > > In [2]: numpy.linalg.singular_value_decomposition?? > Definition: numpy.linalg.singular_value_decomposition(A, > full_matrices=0) > Source: > def singular_value_decomposition(A, full_matrices=0): > return svd(A, full_matrices) > > > In [3]: numpy.linalg.svd? > Definition: numpy.linalg.svd(a, full_matrices=1) > > > Do we really need both spellings to be different just on the default of > the full_matrices flag? It seems to me like the kind of unnecessary > special-casing that just forces people to remember little details > without a clear benefit. If we want a long-spelling form, it would be > cheaper to simply alias the name with a module-level > > singular_value_decomposition = svd > > so the same /object/ is referred to via both names. > > Perhaps I'm missing the blindingly obvious solid reason, but if this is > just historical accident, I'd vote for unifying the two simply as > spelling variants. I think the flipped default is more confusion-prone > than beneficial. I'm pretty sure that this is historical from when we wanted scipy_core's scipy.linalg to be compatible with both real scipy's scipy.linalg and backwards-compatible with Numeric's LinearAlgebra. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Tue Mar 14 14:50:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 14 14:50:06 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <4417383B.4050000@colorado.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> Message-ID: <4417487C.2080003@ee.byu.edu> Fernando Perez wrote: > Travis Oliphant wrote: > >> This post is to announce the release of NumPy 0.9.6 which fixes some >> important bugs and has several speed improvments. > > > Just curious about the svd/singular... pair: > > In [2]: numpy.linalg.singular_value_decomposition?? > Definition: numpy.linalg.singular_value_decomposition(A, > full_matrices=0) > Source: > def singular_value_decomposition(A, full_matrices=0): > return svd(A, full_matrices) > > > In [3]: numpy.linalg.svd? > Definition: numpy.linalg.svd(a, full_matrices=1) > > > Do we really need both spellings to be different just on the default > of the full_matrices flag? It seems to me like the kind of > unnecessary special-casing that just forces people to remember little > details without a clear benefit. If we want a long-spelling form, it > would be cheaper to simply alias the name with a module-level > > singular_value_decomposition = svd > > so the same /object/ is referred to via both names. > > Perhaps I'm missing the blindingly obvious solid reason, but if this > is just historical accident, I'd vote for unifying the two simply as > spelling variants. I think the flipped default is more > confusion-prone than beneficial. Backward compatibility. I don't like the singular_value_decomposition spelling but that was in Numeric (with the default given). So, it is provided to aid in code-porting. But, I think most people who compute svd expect it to produce full_matrices. Thus, the svd spelling. We make somebody unhappy either way I think. If we change anything, I would vote for changing singular_value_decomposition and adding something to convert_code to write any old code using full arguments. -Travis From robert.kern at gmail.com Tue Mar 14 15:01:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 14 15:01:00 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: <4417487C.2080003@ee.byu.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Backward compatibility. > I don't like the singular_value_decomposition spelling but that was in > Numeric (with the default given). So, it is provided to aid in > code-porting. > > But, I think most people who compute svd expect it to produce > full_matrices. Thus, the svd spelling. > > We make somebody unhappy either way I think. If we change anything, I > would vote for changing singular_value_decomposition and adding > something to convert_code to write any old code using full arguments. Alternatively, we could drop all of the Numeric compatibility forms into a separate module and have convert_code use that instead of numpy.linalg directly. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Tue Mar 14 15:04:16 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 14 15:04:16 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44173352.4020403@gmx.net> References: <44168767.80803@ieee.org> <44173352.4020403@gmx.net> Message-ID: <44174BCE.5020104@ee.byu.edu> Sven Schreiber wrote: >>>>import numpy as n >>>>n.__version__ >>>> >>>> >'0.9.6' > > >>>>a = n.asmatrix(n.eye(3)) >>>>a >>>> >>>> >matrix([[1, 0, 0], > [0, 1, 0], > [0, 0, 1]]) > > >>>>n.linalg.inverse(a) >>>> >>>> >array([[ 1., 0., 0.], > [ 0., 1., 0.], > [ 0., 0., 1.]]) > >I was asked to report if functions in linalg still don't honor the >matrix input type; voila. > > > Thanks, but of course a.I should work, right? >>>>n.linalg.cholesky(a) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python24\Lib\site-packages\numpy\linalg\linalg.py", line 135, >in cholesky_decomposition > return transpose(triu(a,k=0)).copy() >NameError: global name 'triu' is not defined > >I think I've seen this error before, was it in 0.9.4? This is a real >show-stopper. > > Thanks again, it would be nice if you could help us by checking out an SVN version before a release ;-) But, better late than never. Also, show-stopper might be a bit of an over-statement. A dumb error, yes. But, if you really depend on this, then it's a simple fix to place a from numpy.lib import * at the top of the linalg.py file. Thanks again for your help. -Travis From Fernando.Perez at colorado.edu Tue Mar 14 15:07:25 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Mar 14 15:07:25 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44174C8D.4070100@colorado.edu> Robert Kern wrote: > Travis Oliphant wrote: > > >>Backward compatibility. >>I don't like the singular_value_decomposition spelling but that was in >>Numeric (with the default given). So, it is provided to aid in >>code-porting. >> >>But, I think most people who compute svd expect it to produce >>full_matrices. Thus, the svd spelling. >> >>We make somebody unhappy either way I think. If we change anything, I >>would vote for changing singular_value_decomposition and adding >>something to convert_code to write any old code using full arguments. > > > Alternatively, we could drop all of the Numeric compatibility forms into a > separate module and have convert_code use that instead of numpy.linalg directly. +1 Let's take advantage of this opportunity to clean up as much of the cruft as we can. Porting will inevitably require some manual labor. I'd say that if backwards compatibility can be isolated to a module, the long-term win is a net beneift. Cheers, f From ndarray at mac.com Tue Mar 14 17:40:01 2006 From: ndarray at mac.com (Sasha) Date: Tue Mar 14 17:40:01 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] unique hanging? In-Reply-To: <441720A8.2040004@stanford.edu> References: <441720A8.2040004@stanford.edu> Message-ID: This really belongs to numpy-discussion. What you observe is probably the consequence of nan not being equal to itself: >>> N.equal(N.nan,N.nan) False Many algorithms would fail to terminate or produce wrong results when they encounter such an anomaly. For example: >>> N.sort([N.nan,5, N.nan, 3]) array([ nan, 5. , nan, 3. ]) This is unfortunate, but not really a bug. I would suggest that we document which numpy functions are nan safe and which are not. On 3/14/06, Jonathan Taylor wrote: > hi all, > > apologize if this is not the right list -- in the latest svn of numpy > unique seems to be taking a long time with "nan" > > reproducible error: > > ------------------------------------------------------------------------------------------- > import numpy as N > import numpy.random as R > > x = R.standard_normal((100000,)) > x = N.around(x * 100.) / 100. > > print len(N.unique(x)) > > x[0] = N.nan > print len(N.unique(x)) > > > x[0:50000] = N.nan > print 'OK' > > print len(N.unique(x)) > > > -- > ------------------------------------------------------------------------ > I'm part of the Team in Training: please support our efforts for the > Leukemia and Lymphoma Society! > > http://www.active.com/donate/tntsvmb/tntsvmbJTaylor > > GO TEAM !!! > > ------------------------------------------------------------------------ > Jonathan Taylor Tel: 650.723.9230 > Dept. of Statistics Fax: 650.725.8977 > Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo > 390 Serra Mall > Stanford, CA 94305 > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > > > > From oliphant.travis at ieee.org Tue Mar 14 18:56:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 14 18:56:00 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44178222.5010004@ieee.org> Robert Kern wrote: > Alternatively, we could drop all of the Numeric compatibility forms into a > separate module and have convert_code use that instead of numpy.linalg directly. > Good idea, and done now in SVN. -Travis From ryanlists at gmail.com Tue Mar 14 18:57:02 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Tue Mar 14 18:57:02 2006 Subject: [Numpy-discussion] matrix vs. array Message-ID: Is there a difference between matrix and array objects other than how they handle multiplication? Is there documentation somewhere about matrix objects? Are they new? Thanks, Ryan From wbaxter at gmail.com Tue Mar 14 19:09:02 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue Mar 14 19:09:02 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: Message-ID: Inspect is a really handy python module: import inspect import numpy print inspect.getsource(numpy.matrix) >From that it looks like exponentiation (__pow__) is also overloaded, and there are the additional properties: .A - get the array .T - get the transpose .H - get the conjugate transpose .I - get the inverse The other difference which isn't really obvious from looking at the source code is that matrices always have at least rank 2. I.e. .shape[0] and .shape[1] are always present for a matrix. --bb On 3/15/06, Ryan Krauss wrote: > > Is there a difference between matrix and array objects other than how > they handle multiplication? Is there documentation somewhere about > matrix objects? Are they new? > > Thanks, > > Ryan > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at stanford.edu Tue Mar 14 20:32:01 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Tue Mar 14 20:32:01 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] unique hanging? In-Reply-To: References: <441720A8.2040004@stanford.edu> Message-ID: <441798AE.7010503@stanford.edu> thanks for the reply. i could be wrong, but this problem seemed to appear only when i updated svn -- i had a very similar example before with N.nan that seemed fine.... -- jonathan Sasha wrote: >This really belongs to numpy-discussion. > >What you observe is probably the consequence of nan not being equal to itself: > > > >>>>N.equal(N.nan,N.nan) >>>> >>>> >False > >Many algorithms would fail to terminate or produce wrong results when >they encounter such an anomaly. > >For example: > > > >>>>N.sort([N.nan,5, N.nan, 3]) >>>> >>>> >array([ nan, 5. , nan, 3. ]) > >This is unfortunate, but not really a bug. I would suggest that we >document which numpy functions are nan safe and which are not. > > > >On 3/14/06, Jonathan Taylor wrote: > > >>hi all, >> >>apologize if this is not the right list -- in the latest svn of numpy >>unique seems to be taking a long time with "nan" >> >>reproducible error: >> >>------------------------------------------------------------------------------------------- >>import numpy as N >>import numpy.random as R >> >>x = R.standard_normal((100000,)) >>x = N.around(x * 100.) / 100. >> >>print len(N.unique(x)) >> >>x[0] = N.nan >>print len(N.unique(x)) >> >> >>x[0:50000] = N.nan >>print 'OK' >> >>print len(N.unique(x)) >> >> >>-- >>------------------------------------------------------------------------ >>I'm part of the Team in Training: please support our efforts for the >>Leukemia and Lymphoma Society! >> >>http://www.active.com/donate/tntsvmb/tntsvmbJTaylor >> >>GO TEAM !!! >> >>------------------------------------------------------------------------ >>Jonathan Taylor Tel: 650.723.9230 >>Dept. of Statistics Fax: 650.725.8977 >>Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo >>390 Serra Mall >>Stanford, CA 94305 >> >> >> >>_______________________________________________ >>Scipy-dev mailing list >>Scipy-dev at scipy.net >>http://www.scipy.net/mailman/listinfo/scipy-dev >> >> >> >> >> >> -- ------------------------------------------------------------------------ I'm part of the Team in Training: please support our efforts for the Leukemia and Lymphoma Society! http://www.active.com/donate/tntsvmb/tntsvmbJTaylor GO TEAM !!! ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From arnd.baecker at web.de Wed Mar 15 01:52:01 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Wed Mar 15 01:52:01 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: Message-ID: On Wed, 15 Mar 2006, Bill Baxter wrote: > Inspect is a really handy python module: > import inspect > import numpy > print inspect.getsource(numpy.matrix) Another very nice option is to use ipython: ipython In [1]: import numpy In [2]: numpy.matrix? In [2]: numpy.matrix?? One ? gives some basic information: Type: type Base Class: String Form: Namespace: Interactive File: /home/abaecker/BUILDS3/BuildDir/inst_numpy/lib/python2.4/site-packages/numpy/core/defmatrix.py (and normally a doc-string, if there is one) and ?? shows you the code. Best, Arnd From pjssilva at ime.usp.br Wed Mar 15 02:36:03 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Mar 15 02:36:03 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: Message-ID: <1142418918.25469.0.camel@localhost.localdomain> > The other difference which isn't really obvious from looking at the > source code is that matrices always have at least rank Just to make clear. They *always* have rank two. Paulo From wbaxter at gmail.com Wed Mar 15 03:40:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 15 03:40:01 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: <1142418918.25469.0.camel@localhost.localdomain> References: <1142418918.25469.0.camel@localhost.localdomain> Message-ID: On 3/15/06, Paulo J. S. Silva wrote: > > > > The other difference which isn't really obvious from looking at the > > source code is that matrices always have at least rank > > Just to make clear. They *always* have rank two. Not necessarily. >>> m = numpy.matrix('[1 2 3; 4 5 6]') >>> mm = m[numpy.matrix('[0 1]')] >>> m matrix([[1, 2, 3], [4, 5, 6]]) >>> mm matrix([[[[1, 2, 3]], [[4, 5, 6]]]]) >>> mm.shape (1, 2, 1, 3) Maybe that's not supposed to be possible, but with current numpy (well, 0.9.5 at least) you see it is possible to get something with type matrix that is not rank 2. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ferenc.Pintye at eu.decoma.com Wed Mar 15 04:10:05 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Wed Mar 15 04:10:05 2006 Subject: [Numpy-discussion] help speed up Message-ID: Hi , why is this little algorithm so too slow (10x) in Numpy than in numarray ? (i think, the if-elif-else procedure runs with numpy array indicies so slow...) # import time import os # # #####from numarray.random_array import * from numpy import * # #####vector = random(50000) vector = rand(50000) # length = len(vector) out_1 = [0,0,0] # # start = time.time() # # i = 0 while i < length-3: # # x1 = vector[i] x2 = vector[i+1] x3 = vector[i+2] x4 = vector[i+3] # if x2>x3>=x1 and x4>=x2: out_1[0] = x3 out_1[1] = x2 out_1[2] = 1 # elif x1>=x3>x2 and x2>=x4: out_1[0] = x2 out_1[1] = x3 out_1[2] = 1 else: out_1[0] = 0 out_1[1] = 0 out_1[2] = 0 # # if out_1[2] == 1: i += 1 i += 1 # # end = time.time() elapsed = end-start print "Running time : %.3f seconds" % (elapsed) # # From cjw at sympatico.ca Wed Mar 15 04:43:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 15 04:43:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <4417487C.2080003@ee.byu.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44180BC5.90604@sympatico.ca> I suggest putting the compatibility stuff in a separate module so that the user can import what he needs, instead of having the whole thing in one bundle. Over time, the separate module will get less use. For some, it will never be used. Colin W. Travis Oliphant wrote: > Fernando Perez wrote: > >> Travis Oliphant wrote: >> >>> This post is to announce the release of NumPy 0.9.6 which fixes some >>> important bugs and has several speed improvments. >> >> >> >> Just curious about the svd/singular... pair: >> >> In [2]: numpy.linalg.singular_value_decomposition?? >> Definition: numpy.linalg.singular_value_decomposition(A, >> full_matrices=0) >> Source: >> def singular_value_decomposition(A, full_matrices=0): >> return svd(A, full_matrices) >> >> >> In [3]: numpy.linalg.svd? >> Definition: numpy.linalg.svd(a, full_matrices=1) >> >> >> Do we really need both spellings to be different just on the default >> of the full_matrices flag? It seems to me like the kind of >> unnecessary special-casing that just forces people to remember little >> details without a clear benefit. If we want a long-spelling form, it >> would be cheaper to simply alias the name with a module-level >> >> singular_value_decomposition = svd >> >> so the same /object/ is referred to via both names. >> >> Perhaps I'm missing the blindingly obvious solid reason, but if this >> is just historical accident, I'd vote for unifying the two simply as >> spelling variants. I think the flipped default is more >> confusion-prone than beneficial. > > > Backward compatibility. > I don't like the singular_value_decomposition spelling but that was in > Numeric (with the default given). So, it is provided to aid in > code-porting. > > But, I think most people who compute svd expect it to produce > full_matrices. Thus, the svd spelling. > > We make somebody unhappy either way I think. If we change anything, I > would vote for changing singular_value_decomposition and adding > something to convert_code to write any old code using full arguments. > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Wed Mar 15 04:45:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 15 04:45:01 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44180C24.10703@sympatico.ca> Robert Kern wrote: >Travis Oliphant wrote: > > > >>Backward compatibility. >>I don't like the singular_value_decomposition spelling but that was in >>Numeric (with the default given). So, it is provided to aid in >>code-porting. >> >>But, I think most people who compute svd expect it to produce >>full_matrices. Thus, the svd spelling. >> >>We make somebody unhappy either way I think. If we change anything, I >>would vote for changing singular_value_decomposition and adding >>something to convert_code to write any old code using full arguments. >> >> > >Alternatively, we could drop all of the Numeric compatibility forms into a >separate module and have convert_code use that instead of numpy.linalg directly. > > > +1 This would help to reduce the name clutter. Colin W. From cjw at sympatico.ca Wed Mar 15 04:51:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 15 04:51:03 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: <1142418918.25469.0.camel@localhost.localdomain> Message-ID: <44180DB5.3030504@sympatico.ca> Bill Baxter wrote: > > On 3/15/06, *Paulo J. S. Silva* > wrote: > > > > The other difference which isn't really obvious from looking at the > > source code is that matrices always have at least rank > > Just to make clear. They *always* have rank two. > > > Not necessarily. > > >>> m = numpy.matrix('[1 2 3; 4 5 6]') > >>> mm = m[numpy.matrix('[0 1]')] > >>> m > matrix([[1, 2, 3], > [4, 5, 6]]) > >>> mm > matrix([[[[1, 2, 3]], > > [[4, 5, 6]]]]) > >>> mm.shape > (1, 2, 1, 3) > > Maybe that's not supposed to be possible, but with current numpy > (well, 0.9.5 at least) you see it is possible to get something with > type matrix that is not rank 2. A true matrix has two dimensions. The example above would appear to be a bug. Colin W. > > --bb From mfmorss at aep.com Wed Mar 15 05:05:17 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Wed Mar 15 05:05:17 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44174BCE.5020104@ee.byu.edu> Message-ID: >matrix([[1, 0, 0], > [0, 1, 0], > [0, 0, 1]]) > > >>>>n.linalg.inverse(a) >>>> >>>> >array([[ 1., 0., 0.], > [ 0., 1., 0.], > [ 0., 0., 1.]]) > >I was asked to report if functions in linalg still don't honor the >matrix input type; voila. > As someone who is just starting to experiment with numpy, may I ask, what in general is the expected result of numpy.linalg.inverse(integer_matrix)? Since not all such inverses are integer matrices, what does it mean to "honor the matrix input type?" Is there a matrix analogue of Python's integer scalar division? Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To Sent by: numpy-discussion numpy-discussion- eforge.net cc Subject 03/14/2006 06:03 Re: [Numpy-discussion] [ANN] NumPy PM 0.9.6 released Sven Schreiber wrote: >>>>import numpy as n >>>>n.__version__ >>>> >>>> >'0.9.6' > > >>>>a = n.asmatrix(n.eye(3)) >>>>a >>>> >>>> > > Thanks, but of course a.I should work, right? >>>>n.linalg.cholesky(a) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python24\Lib\site-packages\numpy\linalg\linalg.py", line 135, >in cholesky_decomposition > return transpose(triu(a,k=0)).copy() >NameError: global name 'triu' is not defined > >I think I've seen this error before, was it in 0.9.4? This is a real >show-stopper. > > Thanks again, it would be nice if you could help us by checking out an SVN version before a release ;-) But, better late than never. Also, show-stopper might be a bit of an over-statement. A dumb error, yes. But, if you really depend on this, then it's a simple fix to place a from numpy.lib import * at the top of the linalg.py file. Thanks again for your help. -Travis ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From mfmorss at aep.com Wed Mar 15 05:33:07 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Wed Mar 15 05:33:07 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <441726C7.3020207@ee.byu.edu> Message-ID: >We need some of these people to convert earlier than that, though, so we >can make sure that 1.0 really is ready for prime time. I think it's >very close already or the version number wouldn't be so high. We are >just waiting for more people to start using it and report any issues >that they have before going to 1.0 (I'd also like scalar math to be >implemented pre 1.0 as well in case that requires any C-API additions). I can understand that, but here as a potential industrial users of Numpy, we can't really afford the risk. We're looking at Numpy as a key piece of a Python replacement of commercial software for critical daily production. If we build Numpy/Scipy into our system, it has to work. We don't want to be anyone's beta testers. Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To Sent by: SciPy Users List numpy-discussion- , admin at lists.sourc numpy-discussion eforge.net cc 03/14/2006 03:25 PM Subject Re: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released Paul Ray wrote: >On Mar 14, 2006, at 1:33 PM, Travis Oliphant wrote: > > > >>>Just wondering, does this one also require an update to scipy? >>>And in general do numpy updates always require an update to scipy, >>>too? >>>Or is it only when the numpy C API interface changes? >>> >>> >>It's only when the C-API interface changes that this is necessary. >>Pre-1.0 it is likely to happen at each release. >> >>After that it is less likely to happen. >> >> > >Pre-1.0, I think it is fine for the version numbering to be a free- >for-all, as it has been. There have been many times, it seems, where >the latest numpy didn't work with the latest scipy and stuff like that. > > The current SVN of numpy and the current SVN of scipy usually work with each other (perhaps with only a day lag). >After 1.0, I would like to encourage some discipline with regards to >numpy version numbering. From 1.0, the API should NOT change at all >until 1.1. > This is the plan and it is why we are being slow with getting 1.0 out. But, we *need* people to try NumPy on their codes so that we can determine whether or not API additions are needed. Without this testing we are just guessing based on the developer's own codes which do not cover the breadth. >I think that many people with codes that use Numeric or numarray are >awaiting the numpy 1.0 release as a sign that it is stable and ready >for prime time, before converting their codes. > > We need some of these people to convert earlier than that, though, so we can make sure that 1.0 really is ready for prime time. I think it's very close already or the version number wouldn't be so high. We are just waiting for more people to start using it and report any issues that they have before going to 1.0 (I'd also like scalar math to be implemented pre 1.0 as well in case that requires any C-API additions). Thanks for the feedback. What you described will be the basic behavior once 1.0 is released. Note that right now the only changes that usually need to be made are a re-compile. I know this can still be annoying when you have a deep code stack. -Travis ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From schofield at ftw.at Wed Mar 15 06:06:05 2006 From: schofield at ftw.at (Ed Schofield) Date: Wed Mar 15 06:06:05 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: Message-ID: <44181F59.3030001@ftw.at> mfmorss at aep.com wrote: >> We need some of these people to convert earlier than that, though, so we >> can make sure that 1.0 really is ready for prime time. I think it's >> very close already or the version number wouldn't be so high. We are >> just waiting for more people to start using it and report any issues >> that they have before going to 1.0 (I'd also like scalar math to be >> implemented pre 1.0 as well in case that requires any C-API additions). >> > I can understand that, but here as a potential industrial users of Numpy, > we can't really afford the risk. We're looking at Numpy as a key piece of > a Python replacement of commercial software for critical daily production. > If we build Numpy/Scipy into our system, it has to work. We don't want to > be anyone's beta testers. > NumPy is rapidly become more stable, due the hard work of Travis and its other developers. But my opinion is that SciPy is not yet ready for "critical daily production" unless you're willing to work through bugs or missing functionality. I assume you have compelling reasons for wanting to migrate from your current commercial software to Python / NumPy / SciPy, in which case I suggest you consider a support contract with a company like Enthought. I don't have links to them myself, but the website advertises consulting services for numerical computing on these platforms, and there will be others on the scipy-user list who can give you more information. -- Ed From svetosch at gmx.net Wed Mar 15 06:55:02 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 15 06:55:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44174BCE.5020104@ee.byu.edu> References: <44168767.80803@ieee.org> <44173352.4020403@gmx.net> <44174BCE.5020104@ee.byu.edu> Message-ID: <44182A7A.7090304@gmx.net> Travis Oliphant schrieb: >>>>> n.linalg.inverse(a) >>>>> >> array([[ 1., 0., 0.], >> [ 0., 1., 0.], >> [ 0., 0., 1.]]) >> >> I was asked to report if functions in linalg still don't honor the >> matrix input type; voila. >> >> >> > Thanks, but of course a.I should work, right? Yes, I only pasted the non-working stuff I could find. > > Thanks again, it would be nice if you could help us by checking out an > SVN version before a release ;-) Ok, I'll try next time. I built scipy once, so I hope with numpy it's not more difficult than with scipy. (on win32) If somebody has good instructions how to build on win (apart from those on the wiki), please let me know. Thanks for numpy, Sven From svetosch at gmx.net Wed Mar 15 07:04:05 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 15 07:04:05 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: Message-ID: <44182CB1.6020004@gmx.net> mfmorss at aep.com schrieb: >> matrix([[1, 0, 0], >> [0, 1, 0], >> [0, 0, 1]]) >> >> >>>>> n.linalg.inverse(a) >>>>> >>>>> >> array([[ 1., 0., 0.], >> [ 0., 1., 0.], >> [ 0., 0., 1.]]) >> >> I was asked to report if functions in linalg still don't honor the >> matrix input type; voila. >> > > As someone who is just starting to experiment with numpy, may I ask, what > in general is the expected result of > numpy.linalg.inverse(integer_matrix)? Since not all such inverses are > integer matrices, what does it mean to "honor the matrix input type?" Is > there a matrix analogue of Python's integer scalar division? > Sorry for this jargon, I was just referring to "matrix" in, "array" out, which should not happen anymore for the linear algebra functions. -sven From strawman at astraw.com Wed Mar 15 09:21:18 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed Mar 15 09:21:18 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: <1142418918.25469.0.camel@localhost.localdomain> Message-ID: <44184CF0.20808@astraw.com> Bill Baxter wrote: > > On 3/15/06, *Paulo J. S. Silva* > wrote: > > > > The other difference which isn't really obvious from looking at the > > source code is that matrices always have at least rank > > Just to make clear. They *always* have rank two. > > > Not necessarily. > > >>> m = numpy.matrix('[1 2 3; 4 5 6]') > >>> mm = m[numpy.matrix('[0 1]')] > >>> m > matrix([[1, 2, 3], > [4, 5, 6]]) > >>> mm > matrix([[[[1, 2, 3]], > > [[4, 5, 6]]]]) > >>> mm.shape > (1, 2, 1, 3) > Could you enter this as a bug so it doesn't get forgotten? http://projects.scipy.org/scipy/numpy/newticket Cheers! Andrew From oliphant.travis at ieee.org Wed Mar 15 11:34:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 15 11:34:02 2006 Subject: [Numpy-discussion] help speed up In-Reply-To: References: Message-ID: <44186BEE.2060501@ieee.org> Ferenc.Pintye at eu.decoma.com wrote: > > > > Hi , > > why is this little algorithm so too slow (10x) in Numpy than in numarray ? > (i think, the if-elif-else procedure runs with numpy array indicies so > slow...) > I'm pretty sure it's scalar math. You get array scalars out of numpy arrays which don't have their own math tables, yet and so use the full umath for the moment. It's pretty high on the list of things to fix before 1.0. -Travis From oliphant.travis at ieee.org Wed Mar 15 11:35:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 15 11:35:05 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: <44184CF0.20808@astraw.com> References: <1142418918.25469.0.camel@localhost.localdomain> <44184CF0.20808@astraw.com> Message-ID: <44186C42.9080403@ieee.org> Andrew Straw wrote: > Bill Baxter wrote: > > >> On 3/15/06, *Paulo J. S. Silva* > > wrote: >> >> >> > The other difference which isn't really obvious from looking at the >> > source code is that matrices always have at least rank >> >> Just to make clear. They *always* have rank two. >> >> >> Not necessarily. >> >> >>>>> m = numpy.matrix('[1 2 3; 4 5 6]') >>>>> mm = m[numpy.matrix('[0 1]')] >>>>> m >>>>> >> matrix([[1, 2, 3], >> [4, 5, 6]]) >> >>>>> mm >>>>> >> matrix([[[[1, 2, 3]], >> >> [[4, 5, 6]]]]) >> >>>>> mm.shape >>>>> >> (1, 2, 1, 3) >> >> > Could you enter this as a bug so it doesn't get forgotten? > http://projects.scipy.org/scipy/numpy/newticket > Please check to see if it actually is still broken. I think it's fixed. -Travis > Cheers! > Andrew > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From erin.sheldon at gmail.com Wed Mar 15 12:21:04 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 12:21:04 2006 Subject: [Numpy-discussion] recarray field names Message-ID: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> Hi all- I'm new to numpy and I'm slowly weening myself off of IDL. So far the experience has been very positive. I use recarrays a lot. I often read these recarrays from fits files using pyfits (modified to work with numpy). I find myself doing the following more than I would like: if 'TAG' in rec._coldefs.names: .... It seems messy to be accessing this "hidden" attribute in this way. Is there a plan to add methods to more transparently do such things? e.g. def fieldnames(): return _coldefs.names def field_exists(fieldname): return fieldname.upper() in _coldefs.names def field_index(fieldname): if field_exists(fieldname): return _coldefs.names.index(fieldname.upper()) else: return -1 # or None maybe Thanks, Erin From perry at stsci.edu Wed Mar 15 12:34:03 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Mar 15 12:34:03 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> Message-ID: <76fbbdec3883fc386929a698021fbf8b@stsci.edu> On Mar 15, 2006, at 3:20 PM, Erin Sheldon wrote: > Hi all- > > I'm new to numpy and I'm slowly weening myself off of IDL. So far the > experience has been very positive. > > I use recarrays a lot. I often read these recarrays from fits files > using pyfits (modified to work with numpy). I find myself doing the > following more than I would like: > > if 'TAG' in rec._coldefs.names: > .... > > It seems messy to be accessing this "hidden" attribute in this way. > Is there a plan to add methods to more transparently do such things? > > e.g. > > def fieldnames(): > return _coldefs.names > > def field_exists(fieldname): > return fieldname.upper() in _coldefs.names > > def field_index(fieldname): > if field_exists(fieldname): > return _coldefs.names.index(fieldname.upper()) > else: > return -1 # or None maybe > > Thanks, > Erin > You are right that this is messy. We would like to change this sometime. But we'd like to complete the transition to numpy first before doing that so it may be some months before we can (and it may not look quite like what you suggest). But your point is very valid. Thanks, Perry From oliphant at ee.byu.edu Wed Mar 15 13:32:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 13:32:05 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: Message-ID: <441887B1.90504@ee.byu.edu> mfmorss at aep.com wrote: >>We need some of these people to convert earlier than that, though, so we >>can make sure that 1.0 really is ready for prime time. I think it's >>very close already or the version number wouldn't be so high. We are >>just waiting for more people to start using it and report any issues >>that they have before going to 1.0 (I'd also like scalar math to be >>implemented pre 1.0 as well in case that requires any C-API additions). >> >> > >I can understand that, but here as a potential industrial users of Numpy, >we can't really afford the risk. We're looking at Numpy as a key piece of >a Python replacement of commercial software for critical daily production. >If we build Numpy/Scipy into our system, it has to work. We don't want to >be anyone's beta testers. > > > That's fine. In your situation, you may need to wait until 1.0 before fully switching to NumPy. But, of course you can start using it now. We will not be changing standard interfaces in NumPy. Most of these are derived from Numeric which has been around for a long time. The closer we get to 1.0, the less willing the developers will be to see things change. On the Python-level, for example, I don't expect anything to change except for better support of array sub-classes (i.e. making sure they are preserved through more operations). On the C-level, there may be an additional C-API or two, but I don't see any significant changes to the C-API itself. Because of the way the C-API is loaded, extension modules must be re-compiled if the C-API changes. This is the only real headache at this point I think. Basically, until 1.0 you need to re-build extension modules whenever a new version of NumPy comes out (SciPy consists of a bunch of extension modules and so must be re-built). Once we hit 1.0, you will only need to rebuild on minor increment changes (i.e. 1.0 to 1.1 and bug-fixes will not change the C-API). -Travis From cookedm at physics.mcmaster.ca Wed Mar 15 13:45:01 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Mar 15 13:45:01 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <441887B1.90504@ee.byu.edu> (Travis Oliphant's message of "Wed, 15 Mar 2006 14:31:29 -0700") References: <441887B1.90504@ee.byu.edu> Message-ID: Travis Oliphant writes: > mfmorss at aep.com wrote: > >>>We need some of these people to convert earlier than that, though, so we >>>can make sure that 1.0 really is ready for prime time. I think it's >>>very close already or the version number wouldn't be so high. We are >>>just waiting for more people to start using it and report any issues >>>that they have before going to 1.0 (I'd also like scalar math to be >>>implemented pre 1.0 as well in case that requires any C-API additions). >>> >>> >> >>I can understand that, but here as a potential industrial users of Numpy, >>we can't really afford the risk. We're looking at Numpy as a key piece of >>a Python replacement of commercial software for critical daily production. >>If we build Numpy/Scipy into our system, it has to work. We don't want to >>be anyone's beta testers. >> >> >> > That's fine. In your situation, you may need to wait until 1.0 before > fully switching to NumPy. But, of course you can start using it now. > > We will not be changing standard interfaces in NumPy. Most of these > are derived from Numeric which has been around for a long time. The > closer we get to 1.0, the less willing the developers will be to see > things change. On the Python-level, for example, I don't expect > anything to change except for better support of array sub-classes > (i.e. making sure they are preserved through more operations). > > On the C-level, there may be an additional C-API or two, but I don't > see any significant changes to the C-API itself. Because of the way > the C-API is loaded, extension modules must be re-compiled if the > C-API changes. This is the only real headache at this point I think. Only if it's a change that affects data structures or previously-defined C API functions. Adding new C API functions shouldn't affect old modules (because they're added on to the end of the C API function list). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Wed Mar 15 13:52:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 13:52:02 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: <441887B1.90504@ee.byu.edu> Message-ID: <44188C54.3020502@ee.byu.edu> David M. Cooke wrote: >Only if it's a change that affects data structures or >previously-defined C API functions. Adding new C API functions >shouldn't affect old modules (because they're added on to the end of >the C API function list). > > I suppose we need to clarify when the NDARRAY_VERSION number needs a bump then, right. Because if that gets bumped "unnecessarily" it will require re-compiles that aren't really necessary. -Travis From cookedm at physics.mcmaster.ca Wed Mar 15 14:10:04 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Mar 15 14:10:04 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44188C54.3020502@ee.byu.edu> (Travis Oliphant's message of "Wed, 15 Mar 2006 14:51:16 -0700") References: <441887B1.90504@ee.byu.edu> <44188C54.3020502@ee.byu.edu> Message-ID: Travis Oliphant writes: > David M. Cooke wrote: > >>Only if it's a change that affects data structures or >>previously-defined C API functions. Adding new C API functions >>shouldn't affect old modules (because they're added on to the end of >>the C API function list). >> >> > I suppose we need to clarify when the NDARRAY_VERSION number needs a > bump then, right. Because if that gets bumped "unnecessarily" it will > require re-compiles that aren't really necessary. > > -Travis Here's things that'll make extensions using the old API ABI-incompatible (application binary interface) with a newer one: 1) semantic changes of function arguments, or of a member of a struct. Changes in type are easier to spot, but it could be possible to have the same type but different meaning. 2) adding a member to a struct before any older ones. That will change the layout. 3) adding a function to the API before any older ones. This changes the layout of C API lookup table Now, here's things that shouldn't change the ABI, and shouldn't require a bump in NDARRAY_VERSION: 1) adding a function to the C API at the end of the lookup table. The old extension won't know it's there, and won't call it :-) 2) adding a member to a struct at the end, assuming that struct is always passed as a pointer, and assuming that it doesn't make the meaning of other members differ from what the extension module expects. 3) defining new macros, constants, etc. I floated the idea a while ago of making the code generation step smarter, along with import_array(), on how to get the C API lookup table, by comparing what API functions the module wants with per-function signatures, to what can be provided, etc. Haven't done anything about it though. I'll see if I can think of something smarter before 1.0. If we're careful, we should be able to have different versions of the API. import_array() could request, say, version 1.0 of the API, and numpy could return that version (even if it's at 1.1). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From wbaxter at gmail.com Wed Mar 15 15:18:07 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 15 15:18:07 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: <44186C42.9080403@ieee.org> References: <1142418918.25469.0.camel@localhost.localdomain> <44184CF0.20808@astraw.com> <44186C42.9080403@ieee.org> Message-ID: I thought this was just expected behavior. Great. Now I can get rid of all those calls to squeeze(). Did it get logged as a bug in the first place, though, before it was fixed? Still probably worth having it in the bug tracker, though. And Travis, do you mean it's fixed in CVS or fixed in 0.9.6? Incidentally, you don't need to index a matrix with a matrix to see it, indexing a matrix with a simple python list also returns the same shape. So mm = m[numpy.matrix('[0 1]')] can also be mm = m[[0,1]] and the bug still happens in 0.9.5. Don't know about 0.9.6 or CVS though. --bb On 3/16/06, Travis Oliphant wrote: > > Andrew Straw wrote: > > Bill Baxter wrote: > > > > > >> On 3/15/06, *Paulo J. S. Silva* >> > wrote: > >> > >> > >> > The other difference which isn't really obvious from looking at > the > >> > source code is that matrices always have at least rank > >> > >> Just to make clear. They *always* have rank two. > >> > >> > >> Not necessarily. > >> > >> > >>>>> m = numpy.matrix('[1 2 3; 4 5 6]') > >>>>> mm = m[numpy.matrix('[0 1]')] > >>>>> m > >>>>> > >> matrix([[1, 2, 3], > >> [4, 5, 6]]) > >> > >>>>> mm > >>>>> > >> matrix([[[[1, 2, 3]], > >> > >> [[4, 5, 6]]]]) > >> > >>>>> mm.shape > >>>>> > >> (1, 2, 1, 3) > >> > >> > > Could you enter this as a bug so it doesn't get forgotten? > > http://projects.scipy.org/scipy/numpy/newticket > > > > Please check to see if it actually is still broken. I think it's fixed. > > > -Travis > > > Cheers! > > Andrew > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > > that extends applications into web and mobile media. Attend the live > webcast > > and join the prime developer group breaking into this new coding > territory! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From erin.sheldon at gmail.com Wed Mar 15 15:42:04 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 15:42:04 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <76fbbdec3883fc386929a698021fbf8b@stsci.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> Message-ID: <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> On 3/15/06, Perry Greenfield wrote: > You are right that this is messy. We would like to change this > sometime. But we'd like to complete the transition to numpy first > before doing that so it may be some months before we can (and it may > not look quite like what you suggest). But your point is very valid. > > Thanks, Perry OK, fair enough. Incidentally, I realized that this attribute _coldefs is not part of recarray anyway, but something added by pyfits. I see now that the names and the formats with a greater than sign concatenated on the front can be extracted from dtype: In [247]: t.dtype Out[247]: [('x', '>f4'), ('y', '>i4')] I could write my own function to extract what I need, but I thought I would ask: is there already a simpler way? And is there a function to compare this '>f4' stuff to the named types such as Float32 ('f')? Erin P.S. If it is man power that is preventing some of the simple things like this from being implemented, I could volunteer some time. From oliphant at ee.byu.edu Wed Mar 15 16:02:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 16:02:01 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> Message-ID: <4418AAB9.3080200@ee.byu.edu> Erin Sheldon wrote: >On 3/15/06, Perry Greenfield wrote: > > >>You are right that this is messy. We would like to change this >>sometime. But we'd like to complete the transition to numpy first >>before doing that so it may be some months before we can (and it may >>not look quite like what you suggest). But your point is very valid. >> >>Thanks, Perry >> >> > >OK, fair enough. > >Incidentally, I realized that this attribute >_coldefs is not part of recarray anyway, but something added by pyfits. >I see now that the names and the formats with a greater than sign >concatenated on the front can be extracted from dtype: > >In [247]: t.dtype >Out[247]: [('x', '>f4'), ('y', '>i4')] > >I could write my own function to extract what I need, but I thought I >would ask: is there already a simpler way? And is there a function to >compare this '>f4' stuff to the named types such as Float32 ('f')? > > The dtype object does contain what you want. In fact. It's the fields attribute of the dtype object that is a dictionary accessed by field name. Thus, to see if a field is a valid field itdentifier, if name in t.dtype.fields: would work (well there is a slight problem in that -1 is a special key to the dictionary that returns a list of field names ordered by offset and so would work also), but if you now that name is already a string, then no problem. -Travis From Fernando.Perez at colorado.edu Wed Mar 15 16:08:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 15 16:08:06 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418AAB9.3080200@ee.byu.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> Message-ID: <4418AC54.1060202@colorado.edu> Travis Oliphant wrote: > The dtype object does contain what you want. In fact. It's the fields > attribute of the dtype object that is a dictionary accessed by field > name. Thus, to see if a field is a valid field itdentifier, > > if name in t.dtype.fields: > > would work (well there is a slight problem in that -1 is a special key > to the dictionary that returns a list of field names ordered by offset > and so would work also), but if you now that name is already a string, > then no problem. Mmh, just curious: I wonder about the wisdom of that overloading of a 'magic' key (-1). It will make thinks like for name in t.dtype.fields: return a spurious entry (the -1), and having the sorted list accessed as for name in t.dtype.fields[-1]: reads weird. I'm sure there was a good reason behind this, but I wonder if it wouldn't be better to provide this particular functionality (the list currently associated with the special -1 key) via a different mechanism, and guaranteeing that t.dtype.fields.keys() == [ list of valid names ]. It just sounds like enforcing a bit of API orthogonality here would be a good thing in the long run, but perhaps I'm missing something (I don't claim to know the reasoning that went behind today's implementation). Best, f From erin.sheldon at gmail.com Wed Mar 15 16:15:04 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 16:15:04 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418AAB9.3080200@ee.byu.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> Message-ID: <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> On 3/15/06, Travis Oliphant wrote: > The dtype object does contain what you want. In fact. It's the fields > attribute of the dtype object that is a dictionary accessed by field > name. Thus, to see if a field is a valid field itdentifier, > > if name in t.dtype.fields: > > would work (well there is a slight problem in that -1 is a special key > to the dictionary that returns a list of field names ordered by offset > and so would work also), but if you now that name is already a string, > then no problem. Yes, I see, but I think you meant if name in t.dtype.fields.keys(): which contains the -1 as a key. In [275]: t.dtype.fields.keys: Out[275]: ('tag1','tag2',-1) So, since the last key points to the names, one can also do: In [279]: t.dtype.fields[-1] Out[279]: ('tag1', 'tag2') which is not transparent, but does what you need. For now, I'll probably just write a simple function to wrap this. Thanks, Erin From oliphant at ee.byu.edu Wed Mar 15 16:26:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 16:26:06 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> Message-ID: <4418B083.7080606@ee.byu.edu> Erin Sheldon wrote: >On 3/15/06, Travis Oliphant wrote: > > >>The dtype object does contain what you want. In fact. It's the fields >>attribute of the dtype object that is a dictionary accessed by field >>name. Thus, to see if a field is a valid field itdentifier, >> >>if name in t.dtype.fields: >> >>would work (well there is a slight problem in that -1 is a special key >>to the dictionary that returns a list of field names ordered by offset >>and so would work also), but if you now that name is already a string, >>then no problem. >> >> > >Yes, I see, but I think you meant > > if name in t.dtype.fields.keys(): > > Actually, you can use this with the dictionary itself (no need to get the keys...) name in t.dtype.fields is equivalent to name in t.dtype.fields.keys() -Travis From Fernando.Perez at colorado.edu Wed Mar 15 16:30:04 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 15 16:30:04 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> Message-ID: <4418B16D.40306@colorado.edu> Erin Sheldon wrote: > Yes, I see, but I think you meant > > if name in t.dtype.fields.keys(): No, he really meant: if name in t.dtype.fields: dictionaries are iterators, so you don't need to construct the list of keys separately. It's just a redundant waste of time and memory in most cases, unless you intend to modify the dict in your loop, case in which the iterator approach won't work and you /do/ need the explicit keys() call. In addition if name in t.dtype.fields is faster than: if name in t.dtype.fields.keys() While both are O(N) operations, the first requires a single call to the hash function on 'name' and then a C lookup in the dict's internal key table as a hash table, while the second is a direct walkthrough of a list with python-level equality testing. In [15]: nkeys = 1000000 In [16]: dct = dict(zip(keys,[None]*len(keys))) In [17]: time bool(-1 in keys) CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s Wall time: 0.01 Out[17]: False In [18]: time bool(-1 in dct) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 Out[18]: False In realistic cases for your original question you are not likely to see the difference, but it's always a good idea to be aware of the performance characteristics of various approaches. For a different problem, there may well be a real difference. Cheers, f From oliphant at ee.byu.edu Wed Mar 15 16:33:43 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 16:33:43 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418AC54.1060202@colorado.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <4418AC54.1060202@colorado.edu> Message-ID: <4418B223.5020201@ee.byu.edu> Fernando Perez wrote: > > Mmh, just curious: I wonder about the wisdom of that overloading of a > 'magic' key (-1). It will make thinks like > > for name in t.dtype.fields: No real wisdom. More organic growth. Intially I didn't have an ordered list of fields, but as more complicated data-type descriptors were supported, this became an important thing to have. I should have probably added a n additional element to the PyArray_Descr structure. Remember, it was growing out of the old PyArray_Descr already and I was a little concerned about changing it too much (there are ramifications of changing this structure in several places). So, instead of adding a "ordered_names" tuple to the dtype object, I just stuck it in the fields dictionary. I agree it's kind of odd sitting there. It probably deserves a re-factoring and pulling that out into a new attribute of the dtype object. This would mean that the PyArray_Descr structure would need a new object (names perhaps), and it would need to be tracked. Not a huge change and probably worth it before the next release. -Travis From Fernando.Perez at colorado.edu Wed Mar 15 16:37:22 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 15 16:37:22 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418B16D.40306@colorado.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> <4418B16D.40306@colorado.edu> Message-ID: <4418B320.5070200@colorado.edu> Fernando Perez wrote: > In addition > > if name in t.dtype.fields > > is faster than: > > if name in t.dtype.fields.keys() > > While both are O(N) operations, the first requires a single call to the hash > function on 'name' and then a C lookup in the dict's internal key table as a > hash table, while the second is a direct walkthrough of a list with > python-level equality testing. [ sorry, copy-pasted wrong timing run] In [1]: nkeys = 5000000 In [2]: keys=range(nkeys) In [3]: dct = dict(zip(keys,[None]*len(keys))) In [4]: time bool(-1 in dct) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 Out[4]: False In [5]: time bool(-1 in keys) CPU times: user 0.32 s, sys: 0.00 s, total: 0.32 s Wall time: 0.33 Out[5]: False Cheers, f From erin.sheldon at gmail.com Wed Mar 15 16:37:23 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 16:37:23 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418B16D.40306@colorado.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> <4418B16D.40306@colorado.edu> Message-ID: <331116dc0603151636i38f0d446n2271be64efcfc293@mail.gmail.com> Nice. Python decides to compare with the keys and not the values. The possibilities for obfuscation are endless. On 3/15/06, Fernando Perez wrote: > Erin Sheldon wrote: > > > Yes, I see, but I think you meant > > > > if name in t.dtype.fields.keys(): > > No, he really meant: > > if name in t.dtype.fields: > > dictionaries are iterators, so you don't need to construct the list of keys > separately. It's just a redundant waste of time and memory in most cases, > unless you intend to modify the dict in your loop, case in which the iterator > approach won't work and you /do/ need the explicit keys() call. > > In addition > > if name in t.dtype.fields > > is faster than: > > if name in t.dtype.fields.keys() > > While both are O(N) operations, the first requires a single call to the hash > function on 'name' and then a C lookup in the dict's internal key table as a > hash table, while the second is a direct walkthrough of a list with > python-level equality testing. > > In [15]: nkeys = 1000000 > > In [16]: dct = dict(zip(keys,[None]*len(keys))) > > In [17]: time bool(-1 in keys) > CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s > Wall time: 0.01 > Out[17]: False > > In [18]: time bool(-1 in dct) > CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s > Wall time: 0.00 > Out[18]: False > > > In realistic cases for your original question you are not likely to see the > difference, but it's always a good idea to be aware of the performance > characteristics of various approaches. For a different problem, there may > well be a real difference. > > Cheers, > > f > From robert.kern at gmail.com Wed Mar 15 16:51:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 15 16:51:25 2006 Subject: [Numpy-discussion] Re: recarray field names In-Reply-To: <331116dc0603151636i38f0d446n2271be64efcfc293@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> <4418B16D.40306@colorado.edu> <331116dc0603151636i38f0d446n2271be64efcfc293@mail.gmail.com> Message-ID: Erin Sheldon wrote: > Nice. Python decides to compare with the keys and not the values. Sure. It is a ridiculously common to ask a dictionary if it has a record for a particular key. It is much, much rarer to ask one if it has a particular value. Lists, tuples, and sets, on the other hand, only have one kind of interesting data, the values, so the __contains__ method operates on values with them. Practicality beats purity, in this case. > The possibilities for obfuscation are endless. Not in my experience. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From hurak at control.felk.cvut.cz Wed Mar 15 23:44:05 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Wed Mar 15 23:44:05 2006 Subject: [Numpy-discussion] How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) Message-ID: Hello, is anybody here working with Numpy on Gentoo Linux system? I have troubles with compiling it with ATLAS-BLAS and ATLAS-LAPACK. The trouble is perhaps that Gentoo has a little bit different approach to storing optimized BLAS and LAPACK on the system then other Linux distros. And apparently Numpy cannot find it. Numpy documentation says that I should specify the path to ATLAS in site.cfg in numpy/distutils, but I cannot find one. Thanks for your help, Zdenek Hurak ------------------------------------------------------------------ hurak at note-zhurak ~ $ ls -l /usr/lib | grep atlas -rw-r--r-- 1 root root 5293184 led 27 15:57 libatlas.a -rwxr-xr-x 1 root root 793 led 27 15:57 libatlas.la lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so -> libatlas.so.0.0.0 lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so.0 -> libatlas.so.0.0.0 -rwxr-xr-x 1 root root 2883644 led 27 15:57 libatlas.so.0.0.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libblas.so -> blas/atlas/libblas.so lrwxrwxrwx 1 root root 23 led 27 15:57 libblas.so.0 -> blas/atlas/libblas.so.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libcblas.a -> blas/atlas/libcblas.a lrwxrwxrwx 1 root root 22 led 27 15:57 libcblas.so -> blas/atlas/libcblas.so lrwxrwxrwx 1 root root 24 led 27 15:57 libcblas.so.0 -> blas/atlas/libcblas.so.0 lrwxrwxrwx 1 root root 25 led 27 21:37 liblapack.so -> lapack/atlas/liblapack.so lrwxrwxrwx 1 root root 27 led 27 21:37 liblapack.so.0 -> lapack/atlas/liblapack.so.0 ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6/numpy/distutils $ cat ../../build/lib.linux-i686-2.4/numpy/__config__.py # This file is generated by /home/hurak/tmp/install/numpy-0.9.6/setup.py # It contains system_info results at the time of building this package. __all__ = ["get_info","show"] blas_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} lapack_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} atlas_threads_info={} blas_opt_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_blas_threads_info={} lapack_opt_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_info={} lapack_mkl_info={} blas_mkl_info={} atlas_blas_info={} mkl_info={} [...] ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6 $ ls numpy/distutils/ ccompiler.py exec_command.pyc intelccompiler.py system_info.py ccompiler.pyc extension.py lib2def.py system_info.pyc command extension.pyc line_endings.py tests conv_template.py fcompiler log.py unixccompiler.py conv_template.pyc from_template.py log.pyc unixccompiler.pyc core.py from_template.pyc mingw32ccompiler.py __version__.py core.pyc info.py misc_util.py __version__.pyc cpuinfo.py info.pyc misc_util.pyc cpuinfo.pyc __init__.py setup.py exec_command.py __init__.pyc setup.pyc ------------------------------------------------------------------ From nadavh at visionsense.com Thu Mar 16 02:57:04 2006 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu Mar 16 02:57:04 2006 Subject: [Numpy-discussion] How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8EF57@exchange2k.envision.co.il> On a P4 machine my site.cfg is: [DEFAULT] library_dirs = /usr/lib:/usr/local/lib include_dirs = /usr/include:/usr/local/include src_dirs = /usr/local/src:/opt/src # search static libraries (.a) in preference to shared ones (.so) search_static_first = 0 [fftw] fftw_libs = fftw3, fftw3f fftw_opt_libs = fftw3_threaded, fftw3f_threaded # if the above aren't found, look for {s,d}fftw_libs and {s,d}fftw_opt_libs [atlas] library_dirs = /usr/lib:/usr/lib/blas/atlas # for overriding the names of the atlas libraries atlas_libs = lapack, f77blas, cblas, atlas [x11] library_dirs = /usr/X11R6/lib include_dirs = /usr/X11R6/include On a amd64 I prefer to use the acml library, which preformance is about the same as ATLAS, but contains full blas/lapack compability. Nadav. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net on behalf of Zdenek Hur?k Sent: Thu 16-Mar-06 09:44 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) Hello, is anybody here working with Numpy on Gentoo Linux system? I have troubles with compiling it with ATLAS-BLAS and ATLAS-LAPACK. The trouble is perhaps that Gentoo has a little bit different approach to storing optimized BLAS and LAPACK on the system then other Linux distros. And apparently Numpy cannot find it. Numpy documentation says that I should specify the path to ATLAS in site.cfg in numpy/distutils, but I cannot find one. Thanks for your help, Zdenek Hurak ------------------------------------------------------------------ hurak at note-zhurak ~ $ ls -l /usr/lib | grep atlas -rw-r--r-- 1 root root 5293184 led 27 15:57 libatlas.a -rwxr-xr-x 1 root root 793 led 27 15:57 libatlas.la lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so -> libatlas.so.0.0.0 lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so.0 -> libatlas.so.0.0.0 -rwxr-xr-x 1 root root 2883644 led 27 15:57 libatlas.so.0.0.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libblas.so -> blas/atlas/libblas.so lrwxrwxrwx 1 root root 23 led 27 15:57 libblas.so.0 -> blas/atlas/libblas.so.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libcblas.a -> blas/atlas/libcblas.a lrwxrwxrwx 1 root root 22 led 27 15:57 libcblas.so -> blas/atlas/libcblas.so lrwxrwxrwx 1 root root 24 led 27 15:57 libcblas.so.0 -> blas/atlas/libcblas.so.0 lrwxrwxrwx 1 root root 25 led 27 21:37 liblapack.so -> lapack/atlas/liblapack.so lrwxrwxrwx 1 root root 27 led 27 21:37 liblapack.so.0 -> lapack/atlas/liblapack.so.0 ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6/numpy/distutils $ cat ../../build/lib.linux-i686-2.4/numpy/__config__.py # This file is generated by /home/hurak/tmp/install/numpy-0.9.6/setup.py # It contains system_info results at the time of building this package. __all__ = ["get_info","show"] blas_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} lapack_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} atlas_threads_info={} blas_opt_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_blas_threads_info={} lapack_opt_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_info={} lapack_mkl_info={} blas_mkl_info={} atlas_blas_info={} mkl_info={} [...] ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6 $ ls numpy/distutils/ ccompiler.py exec_command.pyc intelccompiler.py system_info.py ccompiler.pyc extension.py lib2def.py system_info.pyc command extension.pyc line_endings.py tests conv_template.py fcompiler log.py unixccompiler.py conv_template.pyc from_template.py log.pyc unixccompiler.pyc core.py from_template.pyc mingw32ccompiler.py __version__.py core.pyc info.py misc_util.py __version__.pyc cpuinfo.py info.pyc misc_util.pyc cpuinfo.pyc __init__.py setup.py exec_command.py __init__.pyc setup.pyc ------------------------------------------------------------------ ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From rudolphv at gmail.com Thu Mar 16 05:02:11 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Thu Mar 16 05:02:11 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux Message-ID: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> I followed the instructions given on http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 (linked to from main Numpy / SciPy wiki) to try and install Numpy + Scipy on a 64bit Ubuntu Linux system, but something is not working. I get the following error message during the 'build phase of Numpy (0.9.6): /usr/bin/g77 -shared build/temp.linux-x86_64-2.4/numpy/linalg/lapack_litemodule.o -L/usr/local/lib/atlas -llapack -lptf77blas -lptcblas -latlas -lg2c-pic -o build/lib.linux-x86_64-2.4/numpy/linalg/lapack_lite.so /usr/bin/ld: /usr/local/lib/atlas/liblapack.a(dlamch.o): relocation R_X86_64_32S can not be used when making a shared object; recompile with -fPIC /usr/local/lib/atlas/liblapack.a: could not read symbols: Bad value collect2: ld returned 1 exit status The main problem seems to be this part... "relocation R_X86_64_32S can not be used when making a shared object; recompile with -fPIC". This seems to be related to the ATLAS optimized LAPACK library which I built using the standard method as laid out in the ATLAS errata. The weird thing is that I did in fact specify the -fPIC compiler flag, so I'm not sure how to proceed from here. Any thoughts? Thanks R. -- Rudolph van der Merwe From matthew.brett at gmail.com Thu Mar 16 06:22:07 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu Mar 16 06:22:07 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux In-Reply-To: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> References: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> Message-ID: <1e2af89e0603160621l1164cbfbm7fe763489a99d365@mail.gmail.com> Hi, > /usr/bin/g77 -shared > build/temp.linux-x86_64-2.4/numpy/linalg/lapack_litemodule.o > -L/usr/local/lib/atlas -llapack -lptf77blas -lptcblas -latlas > -lg2c-pic -o build/lib.linux-x86_64-2.4/numpy/linalg/lapack_lite.so > /usr/bin/ld: /usr/local/lib/atlas/liblapack.a(dlamch.o): relocation > R_X86_64_32S can not be used when making a shared object; recompile > with -fPIC I wonder if you built the lapack libraries with the -fPIC flag though? I've attached my build scripts for atlas, the include make file for the lapack libraries, and a python script to combine them - I am wondering if you need the modified include make file, which should somehow end up in your LAPACK directory as make.inc (see the build_full.py script). Best, Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: atlas_lapack_build.tgz Type: application/x-gzip Size: 3017 bytes Desc: not available URL: From Norbert.Nemec.list at gmx.de Thu Mar 16 08:01:02 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu Mar 16 08:01:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: References: Message-ID: <44198B7B.6040706@gmx.de> I have a very much related problem: Not only that the idea described by Mads Ipsen does not work, but I could generally find no efficient way to do a "counting" of elements in an array, as it is needed for a histogram. The function "histogram" contained in numpy uses a rather inefficient method, involving the sorting of the data array. What would instead be needed is a function that simply gives the count of occurances of given values in a given array: >>> [4,5,2,3,2,1,4].count([0,1,2,3,4,5]) [0,1,2,1,1,2] All the solutions that I found so far involve either sorting of the data or writing a loop in Python, both of which are unacceptable for performance. Am I missing something obvious? Mads Ipsen wrote: >Hey, > >First of all, thanks for the new release. > >Here's another question regarding something I cannot quite understand: > >Suppose you want to update bins for a histogram, you might think you >could do something like: > > g = zeros(4,Int) > x = array([0.2, 0.2]) > idx = floor(x/0.1).astype(int) > g[idx] += 1 > >Here idx becomes > > array([2, 2]) > >In this case, I would naively expect g to end up like > > array([0, 0, 2, 0]) (1) > >but instead you get > > array([0, 0, 1, 0]) (2) > >Is this intended? Just being plain novice-like naive, I would expect >the slice operation g[idx] += 1 to do something like > > for i in range(len(I)): > g[ idx[i] ] += 1 > >resulting in (1) and not (2). > >// Mads > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From robert.kern at gmail.com Thu Mar 16 08:43:10 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 16 08:43:10 2006 Subject: [Numpy-discussion] Re: How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) In-Reply-To: References: Message-ID: Zden?k Hur?k wrote: > Hello, > > is anybody here working with Numpy on Gentoo Linux system? I have troubles > with compiling it with ATLAS-BLAS and ATLAS-LAPACK. The trouble is perhaps > that Gentoo has a little bit different approach to storing optimized BLAS > and LAPACK on the system then other Linux distros. And apparently Numpy > cannot find it. Numpy documentation says that I should specify the path to > ATLAS in site.cfg in numpy/distutils, but I cannot find one. Nadav gave you a sample site.cfg. The place to put it would be next to NumPy's setup.py script. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From a.h.jaffe at gmail.com Thu Mar 16 08:45:04 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu Mar 16 08:45:04 2006 Subject: [Numpy-discussion] Status of pyfits with numpy? Message-ID: <441995D3.4070309@gmail.com> Hi All, Is a numpy-compatible version of pyfits nearing stability? I see that there is a pyfits-numpy branch visible on the trac browser -- is that available and stable enough for public use? Thanks, Andrew From rmuller at sandia.gov Thu Mar 16 09:07:07 2006 From: rmuller at sandia.gov (Rick Muller) Date: Thu Mar 16 09:07:07 2006 Subject: [Numpy-discussion] Converting a large package from Numeric to numpy Message-ID: <344D571E-58B6-4DAD-8A65-BE1A9E6F109A@sandia.gov> Before I ask for help I would first like to thank everyone on the Numpy team for the hard work that they've done. A longtime user of Numeric, I'm really impressed with what I'm seeing in Numpy thus far, particularly in the configuration routines, which seem to work wonderfully so far. I maintain a Python Quantum Chemistry package called PyQuante (http:// pyquante.sourceforge.net) that is built on top of Numeric/ LinearAlgebra. I wanted to see what would be involved in converting everything over to the new packages. The big workhorse for me is the dot routine, and I ran a few tests that implied that dot ran faster in Numpy than it did in Numeric, so I was ready to go. The conversion turned out to be easy. I wrote an interface routine called NumWrap that has a flag (use_numpy) that determines whether to import from Numeric or numpy. I then modified my code to import everything from NumWrap. So far very easy. I have a test-suite that exercises many different parts of my code. I ran the test suite using Numeric and it took 351 sec. I then ran using numpy and it took 615 sec!! Almost twice as long. (I should point out that the very good news was that the test suite ran the first time I tried it with the numpy routines without any errors!) I started digging into the differences using the python profiler, and was surprised what turned up. The big difference turned out to be routines that were taking various mathematical functions (pow, exp, even simple scalar multiplication) of array elements. This routine: def vwn_xx(x,b,c): return x*x+b*x+c which is called on many elements of a very large grid, went from taking 17 sec in the test suite to taking 68 sec in the test suite. Another routine that took a log() and a pow() of grid elements, went from 15 sec to 100 sec. I looked through Travis' new documentation. I first tried the .toscalar() function, as follows: rho = 0.5*dens[i].toscalar() but this didn't work, I got AttributeError: 'float64scalar' object has no attribute 'toscalar' So now I'm using: rho = 0.5*float(dens[i]) and this fixed the problem -- it makes the timings go back to the same (or faster) than the Numeric values. So my question is has anyone else run into problems of this sort? If so, is there a "proper", sci-pythonic way of handling conversions like this? Thanks in advance for any help you can offer, and thanks again for the great new package! Rick Rick Muller rmuller at sandia.gov From oliphant.travis at ieee.org Thu Mar 16 09:13:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 16 09:13:07 2006 Subject: [Numpy-discussion] Converting a large package from Numeric to numpy In-Reply-To: <344D571E-58B6-4DAD-8A65-BE1A9E6F109A@sandia.gov> References: <344D571E-58B6-4DAD-8A65-BE1A9E6F109A@sandia.gov> Message-ID: <44199C88.3040403@ieee.org> Rick Muller wrote: > > The conversion turned out to be easy. I wrote an interface routine > called NumWrap that has a flag (use_numpy) that determines whether to > import from Numeric or numpy. I then modified my code to import > everything from NumWrap. So far very easy. Good news. Conversion from Numeric should be really easy and conversion from numarray only a little-bit harder. > > I have a test-suite that exercises many different parts of my code. I > ran the test suite using Numeric and it took 351 sec. I then ran using > numpy and it took 615 sec!! Almost twice as long. (I should point out > that the very good news was that the test suite ran the first time I > tried it with the numpy routines without any errors!) Probably scalar math again. I'm working on that right now, so hopefully it will be faster in the next release of numpy. > > I started digging into the differences using the python profiler, and > was surprised what turned up. The big difference turned out to be > routines that were taking various mathematical functions (pow, exp, > even simple scalar multiplication) of array elements. > This routine: > > def vwn_xx(x,b,c): return x*x+b*x+c > > which is called on many elements of a very large grid, went from > taking 17 sec in the test suite to taking 68 sec in the test suite. > Another routine that took a log() and a pow() of grid elements, went > from 15 sec to 100 sec. > > I looked through Travis' new documentation. I first tried the > .toscalar() function, as follows: > rho = 0.5*dens[i].toscalar() Sorry, you may have an older version. The new spelling is .item() ;-) -Travis From perry at stsci.edu Thu Mar 16 10:44:05 2006 From: perry at stsci.edu (Perry Greenfield) Date: Thu Mar 16 10:44:05 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> Message-ID: <2cde6476418820b6b798a44cc957dedb@stsci.edu> On Mar 15, 2006, at 6:41 PM, Erin Sheldon wrote: > On 3/15/06, Perry Greenfield wrote: >> You are right that this is messy. We would like to change this >> sometime. But we'd like to complete the transition to numpy first >> before doing that so it may be some months before we can (and it may >> not look quite like what you suggest). But your point is very valid. >> >> Thanks, Perry > > > Erin > P.S. If it is man power that is preventing some of the simple things > like this from being implemented, I could volunteer some time. > Ultimately, yes. But there's a little more to it. We've felt there are a number of improvements that can be made and that these might benefit from thinking more globally about how to do them rather than add a few convenience functions. Then there is the issue of whether these changes are added to the numarray version, etc. But help would be welcomed. I'd say wait a little bit until we put out an alpha and then start some discussion on how it should be improved. The astropy mailing list is a better forum for that than this one though. Thanks, Perry From pebarrett at gmail.com Thu Mar 16 10:49:00 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Thu Mar 16 10:49:00 2006 Subject: [Numpy-discussion] Status of pyfits with numpy? In-Reply-To: <441995D3.4070309@gmail.com> References: <441995D3.4070309@gmail.com> Message-ID: <40e64fa20603161048r2c08117ay916ae897011a3b6f@mail.gmail.com> On 3/16/06, Andrew Jaffe wrote: > > Hi All, > > Is a numpy-compatible version of pyfits nearing stability? I see that > there is a pyfits-numpy branch visible on the trac browser -- is that > available and stable enough for public use? > I received the following Email from Chris Hanley, who is doing the numpy port, a few days ago. -- Paul I've finished the initial pyfits numpy port. We won't be doing an alpha release until after we finish the spring STSDAS release. We are in the process of freezing this week. I thought you might want a preview. If you do, you can grab it our of subversion with, svn co http://astropy.scipy.org/svn/pyfits/branches/pyfits-numpypyfits-numpy. That version of pyfits will support both numpy and numarray. If you have both numpy and numarray installed you can make pyfits switch between them by setting an environment variable called NUMERIX. NUMERIX = 'numarray' for numarray. NUMERIX = 'numpy' for numpy. If the variable isn't set, and both packages are installed, pyfits will default to the numarray version. If only one package is installed then pyfits will use that package. Most of the differences for image support are in syntax. However, for table support I had to completely rework the inheritance structure. Be warned, I am certain there are bugs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chanley at stsci.edu Thu Mar 16 11:12:11 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Thu Mar 16 11:12:11 2006 Subject: [Numpy-discussion] Status of pyfits with numpy? In-Reply-To: <441995D3.4070309@gmail.com> References: <441995D3.4070309@gmail.com> Message-ID: <4419B86D.3050801@stsci.edu> Andrew, We are in the process of preparing an "alpha" release of a numpy-compatible version of pyfits. I have recently merged my numpy-pyfits branch with the pyfits trunk so if you do not wish to wait for a release you can grab it our of SVN with: svn co http://astropy.scipy.org/svn/pyfits/trunk pyfits This version of pyfits is not well tested but you are welcome to try it out and send me bug reports. A couple of people have already tried an earlier version and have reported some issues with the table support. We are currently working to address those issues. If you have both numpy and numarray installed you can make pyfits switch between them by setting an environment variable called NUMERIX. NUMERIX = 'numarray' for numarray. NUMERIX = 'numpy' for numpy. If the variable isn't set, and both packages are installed, pyfits will default to the numarray version. If only one package is installed then pyfits will use that package. Chris Andrew Jaffe wrote: > Hi All, > > Is a numpy-compatible version of pyfits nearing stability? I see that > there is a pyfits-numpy branch visible on the trac browser -- is that > available and stable enough for public use? > > Thanks, > > Andrew > > From oliphant at ee.byu.edu Thu Mar 16 13:04:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 16 13:04:05 2006 Subject: [Numpy-discussion] SVN version of NumPy will require rebuild of extensions Message-ID: <4419D2A1.8020504@ee.byu.edu> This is to serve notice that I altered the C-API table in the SVN version of numpy. This change will require a rebuild of extension modules. You are supposed to get notice of that when you try to import the extension module (but that didn't work for me for some reason, importing scipy just segfaulted --- ahh.. I just figured it out... the command that gets the version number is located down in the table. This location has just changed. It makes a case that this command should always be the first entry in the C-API table). The modifications (with a few new C-API calls) will allow construction of the scalar-math tables. -Travis From oliphant at ee.byu.edu Thu Mar 16 14:55:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 16 14:55:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] nd_image In-Reply-To: <1142545376.4419dbe0b1903@webmail.ster.kuleuven.ac.be> References: <1142545376.4419dbe0b1903@webmail.ster.kuleuven.ac.be> Message-ID: <4419ECB1.6050709@ee.byu.edu> joris at ster.kuleuven.ac.be wrote: > [TO]: So, what we have in SciPy is "the official" home for the numpy > [TO]: version of his package. > >Sorry, I am a bit lost here. So nd_image was a numarray library, and is now >available via scipy where it works with numpy arrays (correct?). > nd_image was a Package that built on top of numarray and was also distributed with numarray. Exactly what goes in the base numpy backage and what goes into add-on packages is a question we will wrestle with for a while. With the existence of SciPy, I don't favor growing NumPy much beyond the (general) feature-set it has now. There are already people who grumble that it includes too-much stuff that they don't need. Other functionality should be added by other packages. We are trying to make SciPy into a large collection of packages that don't all have to be installed, so that users can pick and choose what they need if they must. >At the moment, >the library is unmaintained though, regardless whether it would be part of >scipy or numpy. Correct? If so, why exactly would it be better to put it under >scipy rather than under numpy? > > Not directly accurate. The SciPy version is maintained by the SciPy developers (not by the original author). >Some people don't need scipy for their work and are happy with numpy alone. >It used to be rather easy for them to use nd_image with numarray, but if I >understand you correctly, they would now have to install an entire new package, > > Many SciPy packages can be installed separately and nd_image (now scipy.image) is one of them. So, it is very easy to just install the scipy.image package. Hopefully, more people will run with this concept as the need arises. And, scipy is not has hard to install as you might think. Especially if you go into the setup.py file and comment out all the packages you don't actually want... -Travis From oliphant at ee.byu.edu Thu Mar 16 16:51:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 16 16:51:04 2006 Subject: [Numpy-discussion] Array scalar math ideas Message-ID: <441A07C9.8050101@ee.byu.edu> I'm starting to write the remaining array scalar math tables (so that we can speed up scalar*scalar arithmetic). Right now, all operations involving scalars are converted to 0-d arrays - use the ufunc machinery - and are converted back to scalars after the operation. The scalarmathmodule.c.src file is being written to fix this and insert type-appropriate operations for each of the (number-type) array scalars. My general strategy for the binary operations is going to be the following. I wanted to bounce it off the list to see what other ideas people had: Code outline: Convert inputs so that self is the array scalar of some type and other is the other object if (other is an array scalar of the same data-type as self) arg3 = other else if (other is an array scalar of a different data-type as self) arg3 = convert_other_to_self_data_type(other) else if (other is a Python scalar) arg3 = convert_Python_scalar_to_array_scalar(other) else if (other is a 0-d array) arg3 = convert_other_to_self_data_type_from_0-d_array(other) else return (use ufunc to calculate result). return (operation using self and arg3) if an error condition is encountered, then only at that point, the proper way to handle it will be determined by looking in the local / global / builtin scope for the error-handling variable. Tthis avoids the overhead of looking up what to do to the case of an error actually occurring --- I need to change the ufuncobject.c code to also do this and save the overhead there too --- right now what to do is looked up every time, rather than waiting until an error is actually detected.) What do people think? -Travis From cjw at sympatico.ca Thu Mar 16 17:11:13 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 16 17:11:13 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: <441A07C9.8050101@ee.byu.edu> References: <441A07C9.8050101@ee.byu.edu> Message-ID: <441A0C95.30902@sympatico.ca> Travis Oliphant wrote: > > I'm starting to write the remaining array scalar math tables (so that > we can speed up scalar*scalar arithmetic). > Right now, all operations involving scalars are converted to 0-d > arrays - use the ufunc machinery - and are converted back to scalars > after the operation. > > The scalarmathmodule.c.src file is being written to fix this and > insert type-appropriate operations for each of the (number-type) array > scalars. > > > My general strategy for the binary operations is going to be the > following. I wanted to bounce it off the list to see what other ideas > people had: > > Code outline: > > Convert inputs so that self is the array scalar of some type and other > is the other object > > if (other is an array scalar of the same data-type as self) > arg3 = other > else if (other is an array scalar of a different data-type as self) > arg3 = convert_other_to_self_data_type(other) --- Shouldn't this depend on whether self or other has the higher type? > <<< > else if (other is a Python scalar) > arg3 = convert_Python_scalar_to_array_scalar(other) > else if (other is a 0-d array) > arg3 = convert_other_to_self_data_type_from_0-d_array(other) > else > return (use ufunc to calculate result). > > return (operation using self and arg3) > > if an error condition is encountered, then only at that point, the > proper way to handle it will be determined by looking in the local / > global / builtin scope for the error-handling variable. > Tthis avoids the overhead of looking up what to do to the case of an > error actually occurring --- I need to change the ufuncobject.c code > to also do this and save the overhead there too --- right now what to > do is looked up every time, rather than waiting until an error is > actually detected.) > > What do people think? > > -Travis Would it make sense to reconsider the costs and benefits of dealing with dimensionless arrays? Colin W. From ndarray at mac.com Thu Mar 16 18:10:03 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 16 18:10:03 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: <441A0C95.30902@sympatico.ca> References: <441A07C9.8050101@ee.byu.edu> <441A0C95.30902@sympatico.ca> Message-ID: On 3/16/06, Colin J. Williams wrote: > Would it make sense to reconsider the costs and benefits of dealing with > dimensionless arrays? http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray http://aspn.activestate.com/ASPN/Mail/Message/2499954 http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3028210 This horse has suffered a lot beating ... From rudolphv at gmail.com Fri Mar 17 00:14:03 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Fri Mar 17 00:14:03 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux In-Reply-To: <44196EF6.9040602@mecha.uni-stuttgart.de> References: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> <44196EF6.9040602@mecha.uni-stuttgart.de> Message-ID: <97670e910603170013y527d1938q994419036cc383a8@mail.gmail.com> Nils, Thanks... that solved the problem. Much appreciated! Rudolph On 3/16/06, Nils Wagner wrote: > > Did you use > > # > # On 64 bit systems with GNU compiler > # > OPTS = -O2 -m64 -fPIC > NOOPT = -m64 -fPIC > > (two lines of make.inc) > > to compile lapack ? > > Also I have added -m64 -fPIC to some lines in the ATLAS generated makefile ( > Make.Linux_HAMMER64SSE3) > > F77 = /usr/bin/g77 > F77FLAGS = -fomit-frame-pointer -O -m64 -fPIC > FLINKER = $(F77) > FLINKFLAGS = $(F77FLAGS) > FCLINKFLAGS = $(FLINKFLAGS) > > > # --------------------------------------------------------------------------- > # Various C compilers, and the linker to be used when we are not linking in > # non-ATLAS BLAS (which usually necessitate using the Fortran linker). > # The C compilers recognized by ATLAS are: > # CC : Compiler to use to compile regular, non-generated code > # MCC : Compiler to use to compile generated, highly-optimized code > # XCC : Compiler to be used on the compile engine of a cross-compiler > # These will typically all be the same. An example of where this is not > # the case would be DEC ALPHA 21164, where you want to use gcc for MCC, > # because DEC's cc does not allow the programmer access to all 32 floating > # point registers. However, on normal C code, DEC's cc produces much faster > # code than gcc, so you CC set to cc. Of course, any system where you are > # cross-compiling, you will need to set XCC differently than CC & MCC. > # --------------------------------------------------------------------------- > CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS) > > GOODGCC = gcc > CC = /usr/bin/gcc > CCFLAG0 = -fomit-frame-pointer -O -mfpmath=387 -m64 -fPIC > CCFLAGS = $(CDEFS) $(CCFLAG0) > MCC = /usr/bin/gcc > MMFLAGS = -fomit-frame-pointer -O -mfpmath=387 -m64 -fPIC > XCC = /usr/bin/gcc > XCCFLAGS = $(CDEFS) -fomit-frame-pointer -O -mfpmath=387 -m64 -fPIC > CLINKER = $(CC) > CLINKFLAGS = $(CCFLAGS) > BC = $(CC) > BCFLAGS = $(CCFLAGS) > ARCHIVER = ar > ARFLAGS = r > RANLIB = echo > > > Nils > > > > > -- Rudolph van der Merwe From rudolphv at gmail.com Fri Mar 17 00:15:00 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Fri Mar 17 00:15:00 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux In-Reply-To: <1e2af89e0603160621l1164cbfbm7fe763489a99d365@mail.gmail.com> References: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> <1e2af89e0603160621l1164cbfbm7fe763489a99d365@mail.gmail.com> Message-ID: <97670e910603170014p2a915fcdka084bbdd7f2c013c@mail.gmail.com> Matthew, I see your scripts and Makefiles are optimized for a 64 bit P4 architecture. I'm running on a 64 bit AMD Opteron system though. Thanks Rudolph On 3/16/06, Matthew Brett wrote: > Hi, > > > /usr/bin/g77 -shared > > build/temp.linux-x86_64-2.4/numpy/linalg/lapack_litemodule.o > > -L/usr/local/lib/atlas -llapack -lptf77blas -lptcblas -latlas > > -lg2c-pic -o build/lib.linux-x86_64-2.4/numpy/linalg/lapack_lite.so > > /usr/bin/ld: /usr/local/lib/atlas/liblapack.a(dlamch.o): relocation > > R_X86_64_32S can not be used when making a shared object; recompile > > with -fPIC > > I wonder if you built the lapack libraries with the -fPIC flag though? > I've attached my build scripts for atlas, the include make file for > the lapack libraries, and a python script to combine them - I am > wondering if you need the modified include make file, which should > somehow end up in your LAPACK directory as make.inc (see the > build_full.py script). > > Best, > > Matthew > > > -- Rudolph van der Merwe From arnd.baecker at web.de Fri Mar 17 00:18:04 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri Mar 17 00:18:04 2006 Subject: [Numpy-discussion] strange paths on install and some warnings Message-ID: Hi, for a current svn checkout of numpy and installing with python setup.py build python setup.py install --prefix=/tmp/NUMPY this results in an install under: $HOME/.PythonLibrary/Python2.3/ ... (("""copying build/lib.linux-i686-2.3/numpy/_import_tools.py -> /home/abaecker/.PythonLibrary/Python2.3/site-packages/numpy""")) Only f2py behaves and gets installed in /tmp/NUMPY/bin. First I thought that it has something to do with a setuptools test I did some time ago, but moving ~/.PythonLibrary to a different place before installing numpy did not help. (it was also not on the PYTHONPATH before, so it should not have made a difference anyway). Even more weirdly, everything works fine on a different machine. So it looks like some configuration issue. On the other hand I did a couple of builds before (some time ago) on this machine. Before I start looking into this in more detail, I would be interested in any ideas how the above could come about and what tests I could do. In addition to the above I get a couple of warnings/errors (see below). I am not sure about their relevance (I don't remember to have seen them before). Best, Arnd Warning: not existing path in numpy/distutils: site.cfg No module named __svn_version__ F2PY Version 2_2255 blas_opt_info: blas_mkl_info: /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=['/usr/lib/sse2/libatlas.so'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=['/usr/lib/sse2/libatlas.a'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=['/usr/lib/libatlas.so'] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_info: /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c include_dirs = ['/usr/include'] /tmp/NUMPY/numpy/numpy/distutils/system_info.py:1075: FutureWarning: hex()/oct() of negative int will return a signed string in Python 2.4 and up magic = hex(hash(repr(config))) From a.h.jaffe at gmail.com Fri Mar 17 00:31:05 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri Mar 17 00:31:05 2006 Subject: [Numpy-discussion] Re: Status of pyfits with numpy? In-Reply-To: <4419B86D.3050801@stsci.edu> References: <441995D3.4070309@gmail.com> <4419B86D.3050801@stsci.edu> Message-ID: <441A7399.3000209@gmail.com> Dear Chris, Thanks -- that's great. In a quick sprint to read the new WMAP satellite data, the table support seems to work for me at first glance. Can I ask what sort of trouble has been reported? Andrew Christopher Hanley wrote: > Andrew, > > We are in the process of preparing an "alpha" release of a > numpy-compatible version of pyfits. I have recently merged my > numpy-pyfits branch with the pyfits trunk so if you do not wish to wait > for a release you can grab it our of SVN with: > > svn co http://astropy.scipy.org/svn/pyfits/trunk pyfits > > This version of pyfits is not well tested but you are welcome to try it > out and send me bug reports. > > A couple of people have already tried an earlier version and have > reported some issues with the table support. We are currently working > to address those issues. > > If you have both numpy and numarray installed you can make pyfits switch > between them by setting an environment variable called NUMERIX. NUMERIX > = 'numarray' for numarray. NUMERIX = 'numpy' for numpy. If the > variable isn't set, and both packages are installed, pyfits will default > to the numarray version. If only one package is installed then pyfits > will use that package. > > Chris > > > > Andrew Jaffe wrote: >> Hi All, >> >> Is a numpy-compatible version of pyfits nearing stability? I see that >> there is a pyfits-numpy branch visible on the trac browser -- is that >> available and stable enough for public use? >> >> Thanks, >> >> Andrew >> >> > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 From oliphant.travis at ieee.org Fri Mar 17 00:45:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 17 00:45:05 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Computing correlations with SciPy In-Reply-To: <87wteu8ju7.fsf@peds-pc311.bsd.uchicago.edu> References: <1142524171.358429.129850@i39g2000cwa.googlegroups.com> <87wteu8ju7.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: <441A76FF.9080200@ieee.org> John Hunter wrote: > The following message is a courtesy copy of an article > that has been posted to comp.lang.python as well. > > >>>>>> "tkpmep" == tkpmep writes: >>>>>> > > tkpmep> I want to compute the correlation between two sequences X > tkpmep> and Y, and tried using SciPy to do so without success.l > tkpmep> Here's what I have, how can I correct it? > > >>>> X = [1, 2, 3, 4, 5] Y = [5, 4, 3, 2, 1] import scipy > >>>> scipy.corrcoef(X,Y) > tkpmep> Traceback (most recent call last): File " tkpmep> input>", line 1, in ? File > tkpmep> "C:\Python24\Lib\site-packages\numpy\lib\function_base.py", > tkpmep> line 671, in corrcoef d = diag(c) File > tkpmep> "C:\Python24\Lib\site-packages\numpy\lib\twodim_base.py", > tkpmep> line 80, in diag raise ValueError, "Input must be 1- or > tkpmep> 2-d." ValueError: Input must be 1- or 2-d. > >>>> > > Hmm, this may be a bug in scipy. matplotlib also defines a corrcoef > function, which you may want to use until this problem gets sorted out > > The problem is now sorted out in SVN of numpy. The problem was inherited from Numeric's MLab. I revamped the cov function (which corrcoef depended on) so it should hopefully work better than it did. Thanks for the heads up. -Travis From oliphant.travis at ieee.org Fri Mar 17 01:00:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 17 01:00:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: References: Message-ID: <441A7A5B.4010701@ieee.org> Mads Ipsen wrote: > Hey, > > First of all, thanks for the new release. > > Here's another question regarding something I cannot quite understand: > > Suppose you want to update bins for a histogram, you might think you > could do something like: > > g = zeros(4,Int) > x = array([0.2, 0.2]) > idx = floor(x/0.1).astype(int) > g[idx] += 1 > > Here idx becomes > > array([2, 2]) > > In this case, I would naively expect g to end up like > > array([0, 0, 2, 0]) (1) > > but instead you get > > array([0, 0, 1, 0]) (2) > > Is this intended? Just being plain novice-like naive, I would expect > the slice operation g[idx] += 1 to do something like > Yes, this is intended (sort of --- this particular example isn't the reason for the behavior though). The issue is that the code g[idx] +=1 is equivalent in Python to g[idx] = g[idx] + 1 Then g[idx] returns array([0,0]) as it should. This new copy of the array data then gets added to 1 resulting in array([1,1]). This array is then set into element 2 of g twice just as if you had done g[2] = 1 g[2] = 1 Then end effect is you get a 1 out of the result. Perhaps a little counter to the way you were thinking of the problem, but very much how it must be due to the way Python translates the statement g[idx] += 1 in this case. There are, of course, other ways to do what you want to do. -Travis From oliphant.travis at ieee.org Fri Mar 17 01:16:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 17 01:16:03 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <44198B7B.6040706@gmx.de> References: <44198B7B.6040706@gmx.de> Message-ID: <441A7E2A.9010903@ieee.org> Norbert Nemec wrote: > I have a very much related problem: Not only that the idea described by > Mads Ipsen does not work, but I could generally find no efficient way to > do a "counting" of elements in an array, as it is needed for a histogram. > This may be something we are lacking. It depends on what you mean by efficient, I suppose. The sorting algorithms are very fast, and the histogram routines that are scattered all over the place (including the histogram function in numpy) has been in use for a long time, and you are the first person to complain of its efficiency. That isn't to say your complaint may not be valid, it's just that for most people the speed has been sufficient. > What would instead be needed is a function that simply gives the count > of occurances of given values in a given array: > I presume you are talking of "integer" arrays, since equality-comparison of floating-point values is usually not very helpful so most histograms on floating-point values are given in terms of bins. Thus, the searchsorted function uses bins for it's "counting" operation. > >>>> [4,5,2,3,2,1,4].count([0,1,2,3,4,5]) >>>> > [0,1,2,1,1,2] > > A count function for integer arrays could certainly be written using a C-loop. But, I would first just use histogram([4,5,2,3,2,1,4], [0,1,2,3,4,5]) and make sure that's really too slow, before worrying about it too much. Also, I according to the above function, the right answer is: [0, 1, 2, 1, 2, 1] Best, -Travis From perry at stsci.edu Fri Mar 17 03:59:03 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri Mar 17 03:59:03 2006 Subject: [Numpy-discussion] Re: Status of pyfits with numpy? In-Reply-To: <441A7399.3000209@gmail.com> Message-ID: Andrew Jaffe wrote: > > Dear Chris, > > Thanks -- that's great. In a quick sprint to read the new WMAP satellite > data, the table support seems to work for me at first glance. Can I ask > what sort of trouble has been reported? > > Andrew > > It's possible to construct new tables using the Coldef mechanism that will fail because Coldef allows one to associate arrays of higher precision than the column definition allows. When this used to create a table it fails since numpy doesn't allow downcasting assignments. We are changing it so that the column constructor explicitly downcasts any arrays (and raises an exception if you try to downcast integer arrays that won't fit in the new size). If you aren't trying to construct new tables from scratch, you won't encounter this. Perry From hurak at control.felk.cvut.cz Fri Mar 17 04:08:18 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Fri Mar 17 04:08:18 2006 Subject: [Numpy-discussion] Re: How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) References: Message-ID: Robert Kern wrote: > Nadav gave you a sample site.cfg. The place to put it would be next to > NumPy's setup.py script. Thank you very much, Robert. I just wander why some default (example) site.cfg file is not present in the installation package. Sorry, I am a newcomer to Python and perhaps it is a very standard thing for all Python guys to write such files before they start their installation. Best regards from Prague, Zdenek From luszczek at cs.utk.edu Fri Mar 17 06:44:02 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 06:44:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441A7A5B.4010701@ieee.org> References: <441A7A5B.4010701@ieee.org> Message-ID: <200603170943.03095.luszczek@cs.utk.edu> On Friday 17 March 2006 03:59, Travis Oliphant wrote: > Mads Ipsen wrote: > > Hey, > > > > First of all, thanks for the new release. > > > > Here's another question regarding something I cannot quite > > understand: > > > > Suppose you want to update bins for a histogram, you might think > > you could do something like: > > > > g = zeros(4,Int) > > x = array([0.2, 0.2]) > > idx = floor(x/0.1).astype(int) > > g[idx] += 1 > > > > Here idx becomes > > > > array([2, 2]) > > > > In this case, I would naively expect g to end up like > > > > array([0, 0, 2, 0]) (1) > > > > but instead you get > > > > array([0, 0, 1, 0]) (2) > > > > Is this intended? Just being plain novice-like naive, I would > > expect the slice operation g[idx] += 1 to do something like > > Yes, this is intended (sort of --- this particular example isn't the > reason for the behavior though). > > The issue is that the code g[idx] +=1 is equivalent in Python to > > g[idx] = g[idx] + 1 This is not what I read at: http://docs.python.org/ref/numeric-types.html Quote: These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). What you describe is "lack of attempt" in which case the "fall back" behavior gets triggered. > Then g[idx] returns array([0,0]) as it should. This new copy of the > array data then gets added to 1 resulting in array([1,1]). This > array is then set into element 2 of g twice just as if you had done > > g[2] = 1 > g[2] = 1 > > Then end effect is you get a 1 out of the result. > > Perhaps a little counter to the way you were thinking of the problem, > but very much how it must be due to the way Python translates the > statement g[idx] += 1 in this case. > > There are, of course, other ways to do what you want to do. Histograms are just the trivial instantiation of the problem (the problem being unintuitive behavior at least for Mads and me). If in-place arithmetic didn't make copies it would land its nicely to sparse matrix computations. Efficient, compact and in Python. And there are also other histogram-like computations but do not use '+' but other operators. Piotr > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language that extends applications into web and mobile media. Attend > the live webcast and join the prime developer group breaking into > this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121 >642 _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From arnd.baecker at web.de Fri Mar 17 09:40:03 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri Mar 17 09:40:03 2006 Subject: [Numpy-discussion] strange paths on install and some warnings In-Reply-To: References: Message-ID: On Fri, 17 Mar 2006, Arnd Baecker wrote: > Hi, > > for a current svn checkout of numpy and installing with > python setup.py build > python setup.py install --prefix=/tmp/NUMPY > this results in an install under: > $HOME/.PythonLibrary/Python2.3/ ... > (("""copying build/lib.linux-i686-2.3/numpy/_import_tools.py -> > /home/abaecker/.PythonLibrary/Python2.3/site-packages/numpy""")) > > Only f2py behaves and gets installed in /tmp/NUMPY/bin. > > First I thought that it has something to do with a setuptools > test I did some time ago, but moving ~/.PythonLibrary > to a different place before installing numpy did not help. > (it was also not on the PYTHONPATH before, so it should not > have made a difference anyway). [...] Alright, I found the reason for this, it is related to Easy Install Stuff (still mea culpa). At some point I must have followed http://peak.telecommunity.com/DevCenter/EasyInstall#traditional-pythonpath-based-installation which lead to a file .pydistutils.cfg in my homedirectory with contents: [install] install_lib = ~/.PythonLibrary/Python$py_version_short/site-packages prefix=~/.PythonLibrary/ #[easy_install] #site-dirs=/usr/local/lib/pythonX.Y/site-packages So it seems that this wins over a ``--prefix=``. And this is ok, because: """ [...] you can use Distutils configuration files to record personal or site preferences for any Distutils options. That is, any option to any command can be stored in one of two or three (depending on your platform) configuration files, which will be consulted before the command-line is parsed. This means that configuration files will override default values, and the command-line will in turn override configuration files. """ http://docs.python.org/inst/config-syntax.html Obviously, setting up a ~/.pydistutils.cfg is a good way to cause trouble at some later time because it is easily forgotten. The errors below persist... Best, Arnd > In addition to the above I get a couple of warnings/errors (see below). > I am not sure about their relevance (I don't remember to have seen > them before). > > Best, Arnd > > > Warning: not existing path in numpy/distutils: site.cfg > No module named __svn_version__ > F2PY Version 2_2255 > blas_opt_info: > blas_mkl_info: > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['mkl', 'vml', 'guide'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > NOT AVAILABLE > > atlas_blas_threads_info: > Setting PTATLAS=ATLAS > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] > found_libs=['/usr/lib/sse2/libatlas.so'] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] > found_libs=['/usr/lib/sse2/libatlas.a'] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] > found_libs=['/usr/lib/libatlas.so'] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > NOT AVAILABLE > > atlas_blas_info: > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > FOUND: > libraries = ['f77blas', 'cblas', 'atlas'] > library_dirs = ['/usr/lib/sse2'] > language = c > include_dirs = ['/usr/include'] > > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:1075: FutureWarning: > hex()/oct() of negative int will return a signed string in Python 2.4 and > up > magic = hex(hash(repr(config))) From robert.kern at gmail.com Fri Mar 17 09:46:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 17 09:46:03 2006 Subject: [Numpy-discussion] Re: How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) In-Reply-To: References: Message-ID: Zden?k Hur?k wrote: > Robert Kern wrote: > >>Nadav gave you a sample site.cfg. The place to put it would be next to >>NumPy's setup.py script. > > Thank you very much, Robert. > > I just wander why some default (example) site.cfg file is not present in the > installation package. Sorry, I am a newcomer to Python and perhaps it is a > very standard thing for all Python guys to write such files before they > start their installation. There used to be a sample_site.cfg when numpy.distutils was scipy_distutils. It got left behind in the transition probably by accident. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Mar 17 10:30:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 17 10:30:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <200603170943.03095.luszczek@cs.utk.edu> References: <441A7A5B.4010701@ieee.org> <200603170943.03095.luszczek@cs.utk.edu> Message-ID: <441AFFEE.1080508@ee.byu.edu> Piotr Luszczek wrote: >>Yes, this is intended (sort of --- this particular example isn't the >>reason for the behavior though). >> >>The issue is that the code g[idx] +=1 is equivalent in Python to >> >>g[idx] = g[idx] + 1 >> >> > >This is not what I read at: > >http://docs.python.org/ref/numeric-types.html > >Quote: > >These methods should attempt to do the operation in-place (modifying >self) and return the result (which could be, but does not have to be, >self). > >What you describe is "lack of attempt" in which case the "fall back" >behavior gets triggered. > > The problems is that this explanation is very clear when we are talking about the syntax g += 1 Then, there is a method of g that can be over-ridden to get the desired behavior. Now, what you are talking about is "indexing followed by in-place addition". i.e. at issue is how does python interpret g[idx] += 1 How does this get compiled to byte-code? There are two possibilities: 1) g[idx] creates a new object which then has 1 added to it using in-place addition. This would not produce the desired behavior as g[idx] is a copy of the data when idx is a general indexing array as it is in this case. So, you make a copy of those indices, add 1 to them and then do what with the resut? 2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 This appears to be effectively what Python actually does. Notice that there is no way for us to control this behavior because there is no __inplace_with_indexing_add__ operator to over-ride. There is no such single operation to over-ride for the object. In other words, I don't see anyay for us to even alter the object to get the behavior you want from that syntax. We can, of course, add a function or method to do that, but I we would have to extend Python to get the behavior you want here. Note, however, if idx is slice syntax, then the operation is done without making copies because, for example, g[1:10:2] returns a "view" of the data (an array that shares the same memory) as the original array. Thus, g[1:10:2] += 1 does an inplace add without data-copying. It may be possible to do what you want using zero-strided arrays, but I'll leave that for a more motivated contributor. -Travis From luszczek at cs.utk.edu Fri Mar 17 11:26:03 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 11:26:03 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441AFFEE.1080508@ee.byu.edu> References: <200603170943.03095.luszczek@cs.utk.edu> <441AFFEE.1080508@ee.byu.edu> Message-ID: <200603171424.59034.luszczek@cs.utk.edu> On Friday 17 March 2006 13:29, Travis Oliphant wrote: > Piotr Luszczek wrote: > >>Yes, this is intended (sort of --- this particular example isn't > >> the reason for the behavior though). > >> > >>The issue is that the code g[idx] +=1 is equivalent in Python to > >> > >>g[idx] = g[idx] + 1 > > > >This is not what I read at: > > > >http://docs.python.org/ref/numeric-types.html > > > >Quote: > > > >These methods should attempt to do the operation in-place (modifying > >self) and return the result (which could be, but does not have to > > be, self). > > > >What you describe is "lack of attempt" in which case the "fall back" > >behavior gets triggered. > > The problems is that this explanation is very clear when we are > talking about the syntax > > g += 1 > > Then, there is a method of g that can be over-ridden to get the > desired behavior. Now, what you are talking about is "indexing > followed by in-place addition". > > i.e. at issue is > > how does python interpret > > g[idx] += 1 > > How does this get compiled to byte-code? > > There are two possibilities: > > 1) g[idx] creates a new object which then has 1 added to it using > in-place addition. > > This would not produce the desired behavior as g[idx] is a copy > of the data when idx is a > general indexing array as it is in this case. So, you make a > copy of those indices, add 1 to them > and then do what with the resut? > > 2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > > This appears to be effectively what Python actually does. Notice > that there is no way for us to control this behavior because there is > no __inplace_with_indexing_add__ operator to over-ride. > > There is no such single operation to over-ride for the object. In > other words, I don't see anyay for us to even alter the object to get > the behavior you want from that syntax. We can, of course, add a > function or method to do that, but I we would have to extend Python > to get the behavior you want here. Hardly. At least from what I'm seeing happens on a small example. 'g[idx] += 1' becomes ('g' and 'idx' are generic objects): __getitem__(self, idx) __iadd__(1) __setitem__(result of __iadd__) By design numpy returns views from __getitem__ In this case, it would be view into 'self' and 'idx' so the __iadd__ would just use the 'idx' directly rather than a copy. Finally, __setitem__ doesn't do anything since 'self' and 'value' will be the same. Of course, this is just a quick draft. I don't know how it would work in practice and in other cases. > Note, however, if idx is slice syntax, then the operation is done > without making copies because, for example, > > g[1:10:2] > > returns a "view" of the data (an array that shares the same memory) > as the original array. Thus, > > g[1:10:2] += 1 > > does an inplace add without data-copying. > > It may be possible to do what you want using zero-strided arrays, but > I'll leave that for a more motivated contributor. > > -Travis From oliphant at ee.byu.edu Fri Mar 17 11:38:12 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 17 11:38:12 2006 Subject: [Numpy-discussion] Call for posters to gmane.comp.python.devel newsgroup Message-ID: <441B0FF8.3000606@ee.byu.edu> I'm trying to start discussions on python dev about getting a simple object into Python that at least exposes the array interface and/or has the basic C-structure of NumPy arrays. Please voice your support and comments on the newsgroup. The more people that respond, the more the python developers will see that it's not just my lonely voice asking for things to change. Perhaps it will help somebody with more time to get a PEP written up. I doubt we will make it into Python 2.5, unless somebody steps up in the next month, but it will help for Python 2.6 Thanks, -Travis From tim.hochberg at cox.net Fri Mar 17 11:44:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 11:44:04 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <200603171424.59034.luszczek@cs.utk.edu> References: <200603170943.03095.luszczek@cs.utk.edu> <441AFFEE.1080508@ee.byu.edu> <200603171424.59034.luszczek@cs.utk.edu> Message-ID: <441B114B.8010700@cox.net> Piotr Luszczek wrote: >On Friday 17 March 2006 13:29, Travis Oliphant wrote: > > >>Piotr Luszczek wrote: >> >> >>>>Yes, this is intended (sort of --- this particular example isn't >>>>the reason for the behavior though). >>>> >>>>The issue is that the code g[idx] +=1 is equivalent in Python to >>>> >>>>g[idx] = g[idx] + 1 >>>> >>>> >>>This is not what I read at: >>> >>>http://docs.python.org/ref/numeric-types.html >>> >>>Quote: >>> >>>These methods should attempt to do the operation in-place (modifying >>>self) and return the result (which could be, but does not have to >>>be, self). >>> >>>What you describe is "lack of attempt" in which case the "fall back" >>>behavior gets triggered. >>> >>> >>The problems is that this explanation is very clear when we are >>talking about the syntax >> >>g += 1 >> >>Then, there is a method of g that can be over-ridden to get the >>desired behavior. Now, what you are talking about is "indexing >>followed by in-place addition". >> >>i.e. at issue is >> >>how does python interpret >> >>g[idx] += 1 >> >>How does this get compiled to byte-code? >> >>There are two possibilities: >> >>1) g[idx] creates a new object which then has 1 added to it using >>in-place addition. >> >> This would not produce the desired behavior as g[idx] is a copy >>of the data when idx is a >> general indexing array as it is in this case. So, you make a >>copy of those indices, add 1 to them >> and then do what with the resut? >> >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 >> >> This appears to be effectively what Python actually does. Notice >>that there is no way for us to control this behavior because there is >>no __inplace_with_indexing_add__ operator to over-ride. >> >>There is no such single operation to over-ride for the object. In >>other words, I don't see anyay for us to even alter the object to get >>the behavior you want from that syntax. We can, of course, add a >>function or method to do that, but I we would have to extend Python >>to get the behavior you want here. >> >> > >Hardly. At least from what I'm seeing happens on a small example. >'g[idx] += 1' becomes ('g' and 'idx' are generic objects): >__getitem__(self, idx) >__iadd__(1) >__setitem__(result of __iadd__) > >By design numpy returns views from __getitem__ > > You are missing that idx is not a slice index. rather it is an index array (or list). In this case g[idx] does *not* return a view, it returns a copy. From this everything else follows. Conceivably g[idx] could return a psuedo-array object like flat does, but I suspect that might have bad performance characteristics. -tim >In this case, it would be view into 'self' and 'idx' so the __iadd__ >would just use the 'idx' directly rather than a copy. >Finally, __setitem__ doesn't do anything since 'self' and 'value' >will be the same. > >Of course, this is just a quick draft. I don't know how it would work >in practice and in other cases. > > > >>Note, however, if idx is slice syntax, then the operation is done >>without making copies because, for example, >> >>g[1:10:2] >> >>returns a "view" of the data (an array that shares the same memory) >>as the original array. Thus, >> >>g[1:10:2] += 1 >> >>does an inplace add without data-copying. >> >>It may be possible to do what you want using zero-strided arrays, but >>I'll leave that for a more motivated contributor. >> >>-Travis >> >> > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From robert.kern at gmail.com Fri Mar 17 11:59:05 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 17 11:59:05 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <200603171424.59034.luszczek@cs.utk.edu> References: <200603170943.03095.luszczek@cs.utk.edu> <441AFFEE.1080508@ee.byu.edu> <200603171424.59034.luszczek@cs.utk.edu> Message-ID: Piotr Luszczek wrote: > On Friday 17 March 2006 13:29, Travis Oliphant wrote: >>how does python interpret >> >>g[idx] += 1 >> >>How does this get compiled to byte-code? In [161]: c = compile('g[idx] += 1', '', 'single') In [162]: import dis In [163]: dis.dis(c) 1 0 LOAD_NAME 0 (g) 3 LOAD_NAME 1 (idx) 6 DUP_TOPX 2 9 BINARY_SUBSCR 10 LOAD_CONST 0 (1) 13 INPLACE_ADD 14 ROT_THREE 15 STORE_SUBSCR 16 LOAD_CONST 1 (None) 19 RETURN_VALUE >>There are two possibilities: >> >>1) g[idx] creates a new object which then has 1 added to it using >>in-place addition. >> >> This would not produce the desired behavior as g[idx] is a copy >>of the data when idx is a >> general indexing array as it is in this case. So, you make a >>copy of those indices, add 1 to them >> and then do what with the resut? >> >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 >> >> This appears to be effectively what Python actually does. Notice >>that there is no way for us to control this behavior because there is >>no __inplace_with_indexing_add__ operator to over-ride. >> >>There is no such single operation to over-ride for the object. In >>other words, I don't see anyay for us to even alter the object to get >>the behavior you want from that syntax. We can, of course, add a >>function or method to do that, but I we would have to extend Python >>to get the behavior you want here. > > Hardly. At least from what I'm seeing happens on a small example. > 'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > __getitem__(self, idx) > __iadd__(1) > __setitem__(result of __iadd__) > > By design numpy returns views from __getitem__ Only for slices. In [132]: a = arange(10) In [133]: idx = [2,2,3] In [134]: a[idx] Out[134]: array([2, 2, 3]) In [135]: b = a[idx] In [136]: b[-1] = 100 In [137]: a Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > In this case, it would be view into 'self' and 'idx' so the __iadd__ > would just use the 'idx' directly rather than a copy. > Finally, __setitem__ doesn't do anything since 'self' and 'value' > will be the same. No, value is the result of __iadd__ on the temporary array. 'g[idx] += 1' expands to: tmp = g.__getitem__(idx) val = tmp.__iadd__(1) g.__setitem__(idx, val) Given these class definitions: class A(object): def __getitem__(self, idx): print 'A.__getitem__(%r)' % idx return B() def __setitem__(self, idx, value): print 'A.__setitem__(%r, %r)' % (idx, value) class B(object): def __iadd__(self, x): print 'B.__iadd__(%r)' % x return self def __repr__(self): return 'B()' In [153]: a = A() In [154]: a[[0, 2, 2, 1]] += 1 A.__getitem__([0, 2, 2, 1]) B.__iadd__(1) A.__setitem__([0, 2, 2, 1], B()) > Of course, this is just a quick draft. I don't know how it would work > in practice and in other cases. Aye, there's the rub. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From luszczek at cs.utk.edu Fri Mar 17 12:14:03 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 12:14:03 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441B114B.8010700@cox.net> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> Message-ID: <200603171512.44309.luszczek@cs.utk.edu> On Friday 17 March 2006 14:43, Tim Hochberg wrote: > Piotr Luszczek wrote: > >On Friday 17 March 2006 13:29, Travis Oliphant wrote: > >>Piotr Luszczek wrote: > >>>>Yes, this is intended (sort of --- this particular example isn't > >>>>the reason for the behavior though). > >>>> > >>>>The issue is that the code g[idx] +=1 is equivalent in Python to > >>>> > >>>>g[idx] = g[idx] + 1 > >>> > >>>This is not what I read at: > >>> > >>>http://docs.python.org/ref/numeric-types.html > >>> > >>>Quote: > >>> > >>>These methods should attempt to do the operation in-place > >>> (modifying self) and return the result (which could be, but does > >>> not have to be, self). > >>> > >>>What you describe is "lack of attempt" in which case the "fall > >>> back" behavior gets triggered. > >> > >>The problems is that this explanation is very clear when we are > >>talking about the syntax > >> > >>g += 1 > >> > >>Then, there is a method of g that can be over-ridden to get the > >>desired behavior. Now, what you are talking about is "indexing > >>followed by in-place addition". > >> > >>i.e. at issue is > >> > >>how does python interpret > >> > >>g[idx] += 1 > >> > >>How does this get compiled to byte-code? > >> > >>There are two possibilities: > >> > >>1) g[idx] creates a new object which then has 1 added to it using > >>in-place addition. > >> > >> This would not produce the desired behavior as g[idx] is a > >> copy of the data when idx is a > >> general indexing array as it is in this case. So, you make a > >>copy of those indices, add 1 to them > >> and then do what with the resut? > >> > >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > >> > >> This appears to be effectively what Python actually does. > >> Notice that there is no way for us to control this behavior > >> because there is no __inplace_with_indexing_add__ operator to > >> over-ride. > >> > >>There is no such single operation to over-ride for the object. In > >>other words, I don't see anyay for us to even alter the object to > >> get the behavior you want from that syntax. We can, of course, > >> add a function or method to do that, but I we would have to extend > >> Python to get the behavior you want here. > > > >Hardly. At least from what I'm seeing happens on a small example. > >'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > >__getitem__(self, idx) > >__iadd__(1) > >__setitem__(result of __iadd__) > > > >By design numpy returns views from __getitem__ > > You are missing that idx is not a slice index. rather it is an index > array (or list). In this case g[idx] does *not* return a view, it > returns a copy. From this everything else follows. You are missing that indexing with array doesn't have to make a copy. From this everythin else follows. In other words, It's a design decision that slices give views and indirection with arrays (I don't care for lists) gives a copy. I question things on conceptual level: both objects (the array an the indexing array) are from numpy. What code would break if the operation didn't return a copy? > Conceivably g[idx] could return a psuedo-array object like flat does, > but I suspect that might have bad performance characteristics. > > -tim > > >In this case, it would be view into 'self' and 'idx' so the __iadd__ > >would just use the 'idx' directly rather than a copy. > >Finally, __setitem__ doesn't do anything since 'self' and 'value' > >will be the same. > > > >Of course, this is just a quick draft. I don't know how it would > > work in practice and in other cases. > > > >>Note, however, if idx is slice syntax, then the operation is done > >>without making copies because, for example, > >> > >>g[1:10:2] > >> > >>returns a "view" of the data (an array that shares the same memory) > >>as the original array. Thus, > >> > >>g[1:10:2] += 1 > >> > >>does an inplace add without data-copying. > >> > >>It may be possible to do what you want using zero-strided arrays, > >> but I'll leave that for a more motivated contributor. > >> > >>-Travis > > > >------------------------------------------------------- > >This SF.Net email is sponsored by xPML, a groundbreaking scripting > > language that extends applications into web and mobile media. > > Attend the live webcast and join the prime developer group breaking > > into this new coding territory! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=1 > >21642 _______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion at lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language that extends applications into web and mobile media. Attend > the live webcast and join the prime developer group breaking into > this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121 >642 _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From luszczek at cs.utk.edu Fri Mar 17 12:21:01 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 12:21:01 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: References: <200603171424.59034.luszczek@cs.utk.edu> Message-ID: <200603171519.35502.luszczek@cs.utk.edu> On Friday 17 March 2006 14:58, Robert Kern wrote: > Piotr Luszczek wrote: > > On Friday 17 March 2006 13:29, Travis Oliphant wrote: > >>how does python interpret > >> > >>g[idx] += 1 > >> > >>How does this get compiled to byte-code? > > In [161]: c = compile('g[idx] += 1', '', 'single') > > In [162]: import dis > > In [163]: dis.dis(c) > 1 0 LOAD_NAME 0 (g) > 3 LOAD_NAME 1 (idx) > 6 DUP_TOPX 2 > 9 BINARY_SUBSCR > 10 LOAD_CONST 0 (1) > 13 INPLACE_ADD > 14 ROT_THREE > 15 STORE_SUBSCR > 16 LOAD_CONST 1 (None) > 19 RETURN_VALUE This proves my point. > >>There are two possibilities: > >> > >>1) g[idx] creates a new object which then has 1 added to it using > >>in-place addition. > >> > >> This would not produce the desired behavior as g[idx] is a > >> copy of the data when idx is a > >> general indexing array as it is in this case. So, you make a > >>copy of those indices, add 1 to them > >> and then do what with the resut? > >> > >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > >> > >> This appears to be effectively what Python actually does. > >> Notice that there is no way for us to control this behavior > >> because there is no __inplace_with_indexing_add__ operator to > >> over-ride. > >> > >>There is no such single operation to over-ride for the object. In > >>other words, I don't see anyay for us to even alter the object to > >> get the behavior you want from that syntax. We can, of course, > >> add a function or method to do that, but I we would have to extend > >> Python to get the behavior you want here. > > > > Hardly. At least from what I'm seeing happens on a small example. > > 'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > > __getitem__(self, idx) > > __iadd__(1) > > __setitem__(result of __iadd__) > > > > By design numpy returns views from __getitem__ > > Only for slices. > > In [132]: a = arange(10) > > In [133]: idx = [2,2,3] > > In [134]: a[idx] > Out[134]: array([2, 2, 3]) > > In [135]: b = a[idx] > > In [136]: b[-1] = 100 > > In [137]: a > Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) Your example uses lists as indices. This is not interesting. I'm talking solely about arrays indexing other arrays. To me it is a special and very important case. > > In this case, it would be view into 'self' and 'idx' so the > > __iadd__ would just use the 'idx' directly rather than a copy. > > Finally, __setitem__ doesn't do anything since 'self' and 'value' > > will be the same. > > No, value is the result of __iadd__ on the temporary array. > > 'g[idx] += 1' expands to: > > tmp = g.__getitem__(idx) > val = tmp.__iadd__(1) > g.__setitem__(idx, val) You're missing the point. 'tmp' can be of a very specific type so that 'g.__setitem__' doesn't have to do anything: the 'add 1' was done by '__iadd__'. > Given these class definitions: > > class A(object): > def __getitem__(self, idx): > print 'A.__getitem__(%r)' % idx > return B() > def __setitem__(self, idx, value): > print 'A.__setitem__(%r, %r)' % (idx, value) > > > class B(object): > def __iadd__(self, x): > print 'B.__iadd__(%r)' % x > return self > def __repr__(self): > return 'B()' > > In [153]: a = A() > > In [154]: a[[0, 2, 2, 1]] += 1 > A.__getitem__([0, 2, 2, 1]) > B.__iadd__(1) > A.__setitem__([0, 2, 2, 1], B()) > > > Of course, this is just a quick draft. I don't know how it would > > work in practice and in other cases. > > Aye, there's the rub. Show me a code that breaks. From robert.kern at gmail.com Fri Mar 17 13:06:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 17 13:06:01 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <200603171519.35502.luszczek@cs.utk.edu> References: <200603171424.59034.luszczek@cs.utk.edu> <200603171519.35502.luszczek@cs.utk.edu> Message-ID: Piotr Luszczek wrote: > On Friday 17 March 2006 14:58, Robert Kern wrote: > >>Piotr Luszczek wrote: >>>By design numpy returns views from __getitem__ >> >>Only for slices. >> >>In [132]: a = arange(10) >> >>In [133]: idx = [2,2,3] >> >>In [134]: a[idx] >>Out[134]: array([2, 2, 3]) >> >>In [135]: b = a[idx] >> >>In [136]: b[-1] = 100 >> >>In [137]: a >>Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > Your example uses lists as indices. This is not interesting. > I'm talking solely about arrays indexing other arrays. > To me it is a special and very important case. The result is exactly the same. In [164]: a = arange(10) In [165]: idx = array([2,2,3]) In [166]: b = a[idx] In [167]: b[-1] = 100 In [168]: a Out[168]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>In this case, it would be view into 'self' and 'idx' so the >>>__iadd__ would just use the 'idx' directly rather than a copy. >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' >>>will be the same. >> >>No, value is the result of __iadd__ on the temporary array. >> >>'g[idx] += 1' expands to: >> >> tmp = g.__getitem__(idx) >> val = tmp.__iadd__(1) >> g.__setitem__(idx, val) > > You're missing the point. 'tmp' can be of a very specific type > so that 'g.__setitem__' doesn't have to do anything: the 'add 1' > was done by '__iadd__'. No, I got your point just fine; I was correcting a detail. You would have to reimplement __getitem__ to return a new kind of object that represents a non-uniformly-strided array. If you want to get anywhere, go implement that object and come back. When we have something concrete to look at instead of vague assertions, then we can start tackling the issues of integrating it into the core such that 'g[idx] += 1' works like you want it to. For example, index arrays are used in more places than in-place addition. Your new type needs to be usable in all of those places since __getitem__, __iadd__ and __setitem__ don't know that they are being called in that order and in that fashion. >>Given these class definitions: >> >> class A(object): >> def __getitem__(self, idx): >> print 'A.__getitem__(%r)' % idx >> return B() >> def __setitem__(self, idx, value): >> print 'A.__setitem__(%r, %r)' % (idx, value) >> >> >> class B(object): >> def __iadd__(self, x): >> print 'B.__iadd__(%r)' % x >> return self >> def __repr__(self): >> return 'B()' >> >>In [153]: a = A() >> >>In [154]: a[[0, 2, 2, 1]] += 1 >>A.__getitem__([0, 2, 2, 1]) >>B.__iadd__(1) >>A.__setitem__([0, 2, 2, 1], B()) >> >>>Of course, this is just a quick draft. I don't know how it would >>>work in practice and in other cases. >> >>Aye, there's the rub. > > Show me a code that breaks. Show us some code that works. I'm not interested in implementing your feature request. You are. There's plenty of work that you can do that doesn't depend on anyone else agreeing with you, so you can stop arguing and start coding. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From luszczek at cs.utk.edu Fri Mar 17 13:32:02 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 13:32:02 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: References: <200603171519.35502.luszczek@cs.utk.edu> Message-ID: <200603171630.27075.luszczek@cs.utk.edu> On Friday 17 March 2006 16:04, Robert Kern wrote: > Piotr Luszczek wrote: > > On Friday 17 March 2006 14:58, Robert Kern wrote: > >>Piotr Luszczek wrote: > >>>By design numpy returns views from __getitem__ > >> > >>Only for slices. > >> > >>In [132]: a = arange(10) > >> > >>In [133]: idx = [2,2,3] > >> > >>In [134]: a[idx] > >>Out[134]: array([2, 2, 3]) > >> > >>In [135]: b = a[idx] > >> > >>In [136]: b[-1] = 100 > >> > >>In [137]: a > >>Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > > > Your example uses lists as indices. This is not interesting. > > I'm talking solely about arrays indexing other arrays. > > To me it is a special and very important case. > > The result is exactly the same. > > In [164]: a = arange(10) > > In [165]: idx = array([2,2,3]) > > In [166]: b = a[idx] > > In [167]: b[-1] = 100 > > In [168]: a > Out[168]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > >>>In this case, it would be view into 'self' and 'idx' so the > >>>__iadd__ would just use the 'idx' directly rather than a copy. > >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' > >>>will be the same. > >> > >>No, value is the result of __iadd__ on the temporary array. > >> > >>'g[idx] += 1' expands to: > >> > >> tmp = g.__getitem__(idx) > >> val = tmp.__iadd__(1) > >> g.__setitem__(idx, val) > > > > You're missing the point. 'tmp' can be of a very specific type > > so that 'g.__setitem__' doesn't have to do anything: the 'add 1' > > was done by '__iadd__'. > > No, I got your point just fine; I was correcting a detail. > > You would have to reimplement __getitem__ to return a new kind of > object that represents a non-uniformly-strided array. If you want to > get anywhere, go implement that object and come back. When we have > something concrete to look at instead of vague assertions, then we > can start tackling the issues of integrating it into the core such > that 'g[idx] += 1' works like you want it to. For example, index > arrays are used in more places than in-place addition. Your new type > needs to be usable in all of those places since __getitem__, __iadd__ > and __setitem__ don't know that they are being called in that order > and in that fashion. This is a tough requirement but perfectly reasonable. So when my day job let's me off the hook I'll give it a try. > >>Given these class definitions: > >> > >> class A(object): > >> def __getitem__(self, idx): > >> print 'A.__getitem__(%r)' % idx > >> return B() > >> def __setitem__(self, idx, value): > >> print 'A.__setitem__(%r, %r)' % (idx, value) > >> > >> > >> class B(object): > >> def __iadd__(self, x): > >> print 'B.__iadd__(%r)' % x > >> return self > >> def __repr__(self): > >> return 'B()' > >> > >>In [153]: a = A() > >> > >>In [154]: a[[0, 2, 2, 1]] += 1 > >>A.__getitem__([0, 2, 2, 1]) > >>B.__iadd__(1) > >>A.__setitem__([0, 2, 2, 1], B()) > >> > >>>Of course, this is just a quick draft. I don't know how it would > >>>work in practice and in other cases. > >> > >>Aye, there's the rub. > > > > Show me a code that breaks. > > Show us some code that works. I'm not interested in > implementing your feature request. You are. There's plenty of work > that you can do that doesn't depend on anyone else agreeing with you, > so you can stop arguing and start coding. Arguing is good to a point and I think you're right that it's time to stop. From tim.hochberg at cox.net Fri Mar 17 13:50:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 13:50:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <200603171512.44309.luszczek@cs.utk.edu> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> Message-ID: <441B2EF7.10006@cox.net> Piotr Luszczek wrote: >On Friday 17 March 2006 14:43, Tim Hochberg wrote: > > >>Piotr Luszczek wrote: >> >> >>>On Friday 17 March 2006 13:29, Travis Oliphant wrote: >>> >>> >>>>Piotr Luszczek wrote: >>>> >>>> >>>>>>Yes, this is intended (sort of --- this particular example isn't >>>>>>the reason for the behavior though). >>>>>> >>>>>>The issue is that the code g[idx] +=1 is equivalent in Python to >>>>>> >>>>>>g[idx] = g[idx] + 1 >>>>>> >>>>>> >>>>>This is not what I read at: >>>>> >>>>>http://docs.python.org/ref/numeric-types.html >>>>> >>>>>Quote: >>>>> >>>>>These methods should attempt to do the operation in-place >>>>>(modifying self) and return the result (which could be, but does >>>>>not have to be, self). >>>>> >>>>>What you describe is "lack of attempt" in which case the "fall >>>>>back" behavior gets triggered. >>>>> >>>>> >>>>The problems is that this explanation is very clear when we are >>>>talking about the syntax >>>> >>>>g += 1 >>>> >>>>Then, there is a method of g that can be over-ridden to get the >>>>desired behavior. Now, what you are talking about is "indexing >>>>followed by in-place addition". >>>> >>>>i.e. at issue is >>>> >>>>how does python interpret >>>> >>>>g[idx] += 1 >>>> >>>>How does this get compiled to byte-code? >>>> >>>>There are two possibilities: >>>> >>>>1) g[idx] creates a new object which then has 1 added to it using >>>>in-place addition. >>>> >>>> This would not produce the desired behavior as g[idx] is a >>>>copy of the data when idx is a >>>> general indexing array as it is in this case. So, you make a >>>>copy of those indices, add 1 to them >>>> and then do what with the resut? >>>> >>>>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 >>>> >>>> This appears to be effectively what Python actually does. >>>>Notice that there is no way for us to control this behavior >>>>because there is no __inplace_with_indexing_add__ operator to >>>>over-ride. >>>> >>>>There is no such single operation to over-ride for the object. In >>>>other words, I don't see anyay for us to even alter the object to >>>>get the behavior you want from that syntax. We can, of course, >>>>add a function or method to do that, but I we would have to extend >>>>Python to get the behavior you want here. >>>> >>>> >>>Hardly. At least from what I'm seeing happens on a small example. >>>'g[idx] += 1' becomes ('g' and 'idx' are generic objects): >>>__getitem__(self, idx) >>>__iadd__(1) >>>__setitem__(result of __iadd__) >>> >>>By design numpy returns views from __getitem__ >>> >>> >>You are missing that idx is not a slice index. rather it is an index >>array (or list). In this case g[idx] does *not* return a view, it >>returns a copy. From this everything else follows. >> >> > >You are missing that indexing with array doesn't have to make >a copy. > No I'm not. > From this everythin else follows. >In other words, It's a design decision that slices give views and >indirection with arrays (I don't care for lists) gives a copy. >I question things on conceptual level: both objects (the array an >the indexing array) are from numpy. What code would break >if the operation didn't return a copy? > > > >>Conceivably g[idx] could return a psuedo-array object like flat does, >>but I suspect that might have bad performance characteristics. >> >> As I said conceivably it could be done without making a copy. However that would have performance implications; some good and some bad. It's almost certain that some code would break too, although I don't think that's much of a concern if someone got this in before numpy 1.0. We'll probably never know how well or ill this performs unless someone steps up and trys to implement this. I'm not interested. Are you? -tim >>-tim >> >> >> >>>In this case, it would be view into 'self' and 'idx' so the __iadd__ >>>would just use the 'idx' directly rather than a copy. >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' >>>will be the same. >>> >>>Of course, this is just a quick draft. I don't know how it would >>>work in practice and in other cases. >>> >>> >>> >>>>Note, however, if idx is slice syntax, then the operation is done >>>>without making copies because, for example, >>>> >>>>g[1:10:2] >>>> >>>>returns a "view" of the data (an array that shares the same memory) >>>>as the original array. Thus, >>>> >>>>g[1:10:2] += 1 >>>> >>>>does an inplace add without data-copying. >>>> >>>>It may be possible to do what you want using zero-strided arrays, >>>>but I'll leave that for a more motivated contributor. >>>> >>>>-Travis >>>> >>>> >>>------------------------------------------------------- >>>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>>language that extends applications into web and mobile media. >>>Attend the live webcast and join the prime developer group breaking >>>into this new coding territory! >>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=1 >>>21642 _______________________________________________ >>>Numpy-discussion mailing list >>>Numpy-discussion at lists.sourceforge.net >>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>> >>> >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>language that extends applications into web and mobile media. Attend >>the live webcast and join the prime developer group breaking into >>this new coding territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121 >>642 _______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > > > From luszczek at cs.utk.edu Fri Mar 17 14:20:01 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 14:20:01 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441B2EF7.10006@cox.net> References: <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> Message-ID: <200603171718.20109.luszczek@cs.utk.edu> On Friday 17 March 2006 16:49, Tim Hochberg wrote: > Piotr Luszczek wrote: > >On Friday 17 March 2006 14:43, Tim Hochberg wrote: > >>Piotr Luszczek wrote: > >>>On Friday 17 March 2006 13:29, Travis Oliphant wrote: > >>>>Piotr Luszczek wrote: > >>>>>>Yes, this is intended (sort of --- this particular example > >>>>>> isn't the reason for the behavior though). > >>>>>> > >>>>>>The issue is that the code g[idx] +=1 is equivalent in Python > >>>>>> to > >>>>>> > >>>>>>g[idx] = g[idx] + 1 > >>>>> > >>>>>This is not what I read at: > >>>>> > >>>>>http://docs.python.org/ref/numeric-types.html > >>>>> > >>>>>Quote: > >>>>> > >>>>>These methods should attempt to do the operation in-place > >>>>>(modifying self) and return the result (which could be, but does > >>>>>not have to be, self). > >>>>> > >>>>>What you describe is "lack of attempt" in which case the "fall > >>>>>back" behavior gets triggered. > >>>> > >>>>The problems is that this explanation is very clear when we are > >>>>talking about the syntax > >>>> > >>>>g += 1 > >>>> > >>>>Then, there is a method of g that can be over-ridden to get the > >>>>desired behavior. Now, what you are talking about is "indexing > >>>>followed by in-place addition". > >>>> > >>>>i.e. at issue is > >>>> > >>>>how does python interpret > >>>> > >>>>g[idx] += 1 > >>>> > >>>>How does this get compiled to byte-code? > >>>> > >>>>There are two possibilities: > >>>> > >>>>1) g[idx] creates a new object which then has 1 added to it using > >>>>in-place addition. > >>>> > >>>> This would not produce the desired behavior as g[idx] is a > >>>>copy of the data when idx is a > >>>> general indexing array as it is in this case. So, you make > >>>> a copy of those indices, add 1 to them > >>>> and then do what with the resut? > >>>> > >>>>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > >>>> > >>>> This appears to be effectively what Python actually does. > >>>>Notice that there is no way for us to control this behavior > >>>>because there is no __inplace_with_indexing_add__ operator to > >>>>over-ride. > >>>> > >>>>There is no such single operation to over-ride for the object. > >>>> In other words, I don't see anyay for us to even alter the > >>>> object to get the behavior you want from that syntax. We can, > >>>> of course, add a function or method to do that, but I we would > >>>> have to extend Python to get the behavior you want here. > >>> > >>>Hardly. At least from what I'm seeing happens on a small example. > >>>'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > >>>__getitem__(self, idx) > >>>__iadd__(1) > >>>__setitem__(result of __iadd__) > >>> > >>>By design numpy returns views from __getitem__ > >> > >>You are missing that idx is not a slice index. rather it is an > >> index array (or list). In this case g[idx] does *not* return a > >> view, it returns a copy. From this everything else follows. > > > >You are missing that indexing with array doesn't have to make > >a copy. > > No I'm not. > > > From this everythin else follows. > >In other words, It's a design decision that slices give views and > >indirection with arrays (I don't care for lists) gives a copy. > >I question things on conceptual level: both objects (the array an > >the indexing array) are from numpy. What code would break > >if the operation didn't return a copy? > > > >>Conceivably g[idx] could return a psuedo-array object like flat > >> does, but I suspect that might have bad performance > >> characteristics. > > As I said conceivably it could be done without making a copy. However > that would have performance implications; some good and some bad. > It's almost certain that some code would break too, although I don't > think that's much of a concern if someone got this in before numpy > 1.0. We'll probably never know how well or ill this performs unless > someone steps up and trys to implement this. I'm not interested. Are > you? Yes. Very much so. To me it's a very compact and intuitive way of encoding many things I do. Histogram is trivial but useful. Sparse matrix calculations are more important to me. I've raised this issue at least once before and didn't get much response. I think it's better this time around. But ultimately I agree with Robert Kern that I need to show some code and timing data to make people pay attention. > -tim > > >>-tim > >> > >>>In this case, it would be view into 'self' and 'idx' so the > >>> __iadd__ would just use the 'idx' directly rather than a copy. > >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' > >>>will be the same. > >>> > >>>Of course, this is just a quick draft. I don't know how it would > >>>work in practice and in other cases. > >>> > >>>>Note, however, if idx is slice syntax, then the operation is done > >>>>without making copies because, for example, > >>>> > >>>>g[1:10:2] > >>>> > >>>>returns a "view" of the data (an array that shares the same > >>>> memory) as the original array. Thus, > >>>> > >>>>g[1:10:2] += 1 > >>>> > >>>>does an inplace add without data-copying. > >>>> > >>>>It may be possible to do what you want using zero-strided arrays, > >>>>but I'll leave that for a more motivated contributor. > >>>> > >>>>-Travis > >>> > >>>------------------------------------------------------- > >>>This SF.Net email is sponsored by xPML, a groundbreaking scripting > >>>language that extends applications into web and mobile media. > >>>Attend the live webcast and join the prime developer group > >>> breaking into this new coding territory! > >>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat= > >>>1 21642 _______________________________________________ > >>>Numpy-discussion mailing list > >>>Numpy-discussion at lists.sourceforge.net > >>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >> > >>------------------------------------------------------- > >>This SF.Net email is sponsored by xPML, a groundbreaking scripting > >>language that extends applications into web and mobile media. > >> Attend the live webcast and join the prime developer group > >> breaking into this new coding territory! > >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=1 > >>21 642 _______________________________________________ > >>Numpy-discussion mailing list > >>Numpy-discussion at lists.sourceforge.net > >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion From tim.hochberg at cox.net Fri Mar 17 14:21:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 14:21:01 2006 Subject: [Numpy-discussion] Copy vs View for array[array] (was Histograms via indirect index arrays) In-Reply-To: <441B2EF7.10006@cox.net> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> Message-ID: <441B3632.2060402@cox.net> I just figured I'd add a couple of thought outside that needlessly contentious thread. In theory I'm all for view semantics for an array indexed by an array (I'm sure we have a good name for that, but it's escaping me). Indexing in numpy can be confusing enough without some indexing operations returning views and others copies. This is orthogonal to any issues of performance. In practice, I'm a bit skeptical. The result would need to be some sort of psuedo array object (similar to array.flat). Operations on this object would necessarily have worse performance than operations on a normal array due to the added level of indirection. In some circumstances it would also hold onto a lot of memory that might otherwise be freed since it hold a reference to the data for both the original array and the index array. How much of an effect this would have on the typical user is hard to say; any effects would certainly depend a lot on usage patterns. Here's some cases and guesses as to how I think things would work out (idx is an array): a[idx] += 1 # Improved performance and more importantly result consistent with other indexing ops c = a[idx] + b[idx] # Probably neutral; Creating the psuedo arrays should be fast, then they need to copied anyway in the add. c, d = a[idx], b[idx] del a, b e = c + d f = 2*c + d # Probably bad: Need to copy the psuedo array into a contiguous buffer multiple times. # Also holds extra memory. I'd like to see someone try it, but it's not high enough on my priority list for me to dive into it (I'm blowing all my spare cycles and then some on numexpr). -tim From cjw at sympatico.ca Fri Mar 17 14:35:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 17 14:35:03 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: References: <441A07C9.8050101@ee.byu.edu> <441A0C95.30902@sympatico.ca> Message-ID: <441B3986.9060701@sympatico.ca> Sasha wrote: >On 3/16/06, Colin J. Williams wrote: > > >>Would it make sense to reconsider the costs and benefits of dealing with >>dimensionless arrays? >> >> > >http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray > >http://aspn.activestate.com/ASPN/Mail/Message/2499954 > >http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3028210 > >This horse has suffered a lot beating ... > > > Sasha, Many thanks, this sets things out clearly. On a first scan, one thing caught my eye: I think we should implement Python scalars for the other types and then eliminate rank-0 arrays completely. I could have a student do this in a few weeks if it was agreed on. -Travis Oliphant [13-Sep-02] The Wiki sems to start with the design decision to implement zero dimension arrays. It then goes on to explore the problems created by that design decision. My contribution to the second thread you quote was: It looks as though a decision has been made. I was among those who favoured abandoning rank-0 arrays, we lost. To my mind rank-0 arrays add complexity for little benefit and make explanation more difficult. I don"t spot any discussion in the PEP of the pros and cons of the nd == 0 case. Colin W. [20-Mar-05] Ralf Juengling responded: A correction! There is, in the PEP:: Questions 1) should sequence behavior (i.e. some combination of slicing, indexing, and len) be supported for 0-dim arrays? Pros: It means that len(a) always works and returns the size of the array. Slicing code and indexing code will work for any dimension (the 0-dim array is an identity element for the operation of slicing) Cons: 0-dim arrays are really scalars. They should behave like Python scalars which do not allow sequence behavior 2) should array operations that result in a 0-dim array that is the same basic type as one of the Python scalars, return the Python scalar instead? Pros: 1) Some cases when Python expects an integer (the most dramatic is when slicing and indexing a sequence: _PyEval_SliceIndex in ceval.c) it will not try to convert it to an integer first before raising an error. Therefore it is convenient to have 0-dim arrays that are integers converted for you by the array object. 2) No risk of user confusion by having two types that are nearly but not exactly the same and whose separate existence can only be explained by the history of Python and NumPy development. 3) No problems with code that does explicit typechecks (isinstance(x, float) or type(x) == types.FloatType). Although explicit typechecks are considered bad practice in general, there are a couple of valid reasons to use them. 4) No creation of a dependency on Numeric in pickle files (though this could also be done by a special case in the pickling code for arrays) Cons: It is difficult to write generic code because scalars do not have the same methods and attributes as arrays. (such as .type or .shape). Also Python scalars have different numeric behavior as well. This results in a special-case checking that is not pleasant. Fundamentally it lets the user believe that somehow multidimensional homoegeneous arrays are something like Python lists (which except for Object arrays they are not). For me and for the end user, the (2) Pros win. I agree. Incidentally, the 0-d discussion does not appear in the 12-Jan-06 version of the PEP. I don't fully understand your proposal of 18-Feb-06 although I support Francesc's suggestion that we avoid "rank" to mean the number of dimensions. For matrices, we accept the rule that a matrix has two dimensions, not more not less. What problems would be created if we had the rule that an array has not less than one dimension? The little script below produces different results, depending on the ordering of the operands. This provides another challenge for the newcomer. Colin W. # arrayStuff.py def inArrayWorld(): import numpy as _n a= _n.arange(4) return a[2] def anotherWorld(a): import math w= 2.6 * a x= a * 2 y= 2.6 * a z= math.log10(a) return z, type(z), type(y), type(x), type(w) print anotherWorld(inArrayWorld()) From rlw at stsci.edu Fri Mar 17 14:57:10 2006 From: rlw at stsci.edu (Rick White) Date: Fri Mar 17 14:57:10 2006 Subject: [Numpy-discussion] Copy vs View for array[array] (was Histograms via indirect index arrays) In-Reply-To: <441B3632.2060402@cox.net> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> <441B3632.2060402@cox.net> Message-ID: On Fri, 17 Mar 2006, Tim Hochberg wrote: > In theory I'm all for view semantics for an array indexed by an array > (I'm sure we have a good name for that, but it's escaping me). Indexing > in numpy can be confusing enough without some indexing operations > returning views and others copies. This is orthogonal to any issues of > performance. > > In practice, I'm a bit skeptical. The result would need to be some sort > of psuedo array object (similar to array.flat). Operations on this > object would necessarily have worse performance than operations on a > normal array due to the added level of indirection. In some > circumstances it would also hold onto a lot of memory that might > otherwise be freed since it hold a reference to the data for both the > original array and the index array. Actually I think it is worse than that -- it seems to me that it actually has to make a *copy* of the index array. I don't think that we would want to keep only a reference to the index array, since if it changed then the view could respond by changing in very unexpected ways. That sounds like a nightmare side-effect to me. That's what has always made me think that this is not a good idea, even if the bookkeeping of carrying around an unevaluated array+indices could be worked out efficiently. In my applications I sometimes use very large index arrays, and I don't want to have to copy them unnecessarily. Generally I much prefer instant evaluation as in the current implementation, since that uses the minimum of memory. For what it's worth, IDL behaves exactly like the current numpy: a[idx] += 1 increments each element by 1 regardless of how many times a particular index is included in idx. From tim.hochberg at cox.net Fri Mar 17 15:16:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 15:16:06 2006 Subject: [Numpy-discussion] Copy vs View for array[array] (was Histograms via indirect index arrays) In-Reply-To: References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> <441B3632.2060402@cox.net> Message-ID: <441B4315.5040109@cox.net> Rick White wrote: >On Fri, 17 Mar 2006, Tim Hochberg wrote: > > > >>In theory I'm all for view semantics for an array indexed by an array >>(I'm sure we have a good name for that, but it's escaping me). Indexing >>in numpy can be confusing enough without some indexing operations >>returning views and others copies. This is orthogonal to any issues of >>performance. >> >>In practice, I'm a bit skeptical. The result would need to be some sort >>of psuedo array object (similar to array.flat). Operations on this >>object would necessarily have worse performance than operations on a >>normal array due to the added level of indirection. In some >>circumstances it would also hold onto a lot of memory that might >>otherwise be freed since it hold a reference to the data for both the >>original array and the index array. >> >> > >Actually I think it is worse than that -- it seems to me that it >actually has to make a *copy* of the index array. I don't think >that we would want to keep only a reference to the index array, >since if it changed then the view could respond by changing in very >unexpected ways. That sounds like a nightmare side-effect to me. > > Yeah, that would be bad. Conceivably (there's that word again) one could implement copy-on-write for the index arrays, but that would be another can of worms. Anwway, I agree that you would have to at least fake a copy somehow. >That's what has always made me think that this is not a good idea, >even if the bookkeeping of carrying around an unevaluated array+indices >could be worked out efficiently. In my applications I sometimes >use very large index arrays, and I don't want to have to copy them >unnecessarily. Generally I much prefer instant evaluation as in >the current implementation, since that uses the minimum of memory. > >For what it's worth, IDL behaves exactly like the current numpy: >a[idx] += 1 increments each element by 1 regardless of how many >times a particular index is included in idx. > > I'm not much concerned with case myself. The case that bothers me (more in the abstract than in reality since I don't use index arrays much) is the following: >>> idx1 = slice(0,None,2) >>> idx2 = [0,2,4,6,8,10] >>> idx1 = slice(0,None,2) >>> idx2 = [0,2,4,6,8] >>> a = arange(10) >>> b = arange(10) >>> ai = a[idx1] >>> bi = b[idx2] >>> ai array([0, 2, 4, 6, 8]) >>> bi array([0, 2, 4, 6, 8]) >>> ai[1] = bi[1] = 99 >>> a array([ 0, 1, 99, 3, 4, 5, 6, 7, 8, 9]) >>> b array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) The fact that 'a' and 'b' diverge here is a wart, although not one I'm sure it's worth doing anything about. -tim > > > From rmuller at sandia.gov Fri Mar 17 15:26:03 2006 From: rmuller at sandia.gov (Rick Muller) Date: Fri Mar 17 15:26:03 2006 Subject: [Numpy-discussion] Difference between eigh and Heigenvectors Message-ID: What exactly is the difference between eigh and Heigenvectors. As far as I can tell, the eigenvalues are the same, but the eigenvectors are transposes of each other. Is this a matlab compatibility thing? Thanks in advance, R. Rick Muller rmuller at sandia.gov From tim.hochberg at cox.net Fri Mar 17 15:29:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 15:29:06 2006 Subject: [Numpy-discussion] numexpr update Message-ID: <441B4619.6000306@cox.net> I just committed a whole slow of changes to numexpr for your enjoyment. First the good news. It now handles integers as well as floats and the framework is in place for adding complex numbers as well (it can actually add complex numbers, but nothing else yet). It should be pretty simple to add complex at this point, so that'll probably happen soon. Otherwise things are more or less the same, although I expect to modifying the interface a little bit here soon to better support typing of input values. Now the bad news: somewhere along the way I introduced some sort of memory bug and it occasionally segfaults on me. I'm not sure if it's reference counting or I'm scribbling outside my allocated block of memory somehow, but I'll continue hunting it down. More minor issues include int**int is a little bit bogus and I've turned off the aggressive power optimizations for the moment pending some improvements to the compiler. -tim From ndarray at mac.com Fri Mar 17 15:44:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 17 15:44:03 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: <441B3986.9060701@sympatico.ca> References: <441A07C9.8050101@ee.byu.edu> <441A0C95.30902@sympatico.ca> <441B3986.9060701@sympatico.ca> Message-ID: On 3/17/06, Colin J. Williams wrote: > Sasha wrote: > ... > >http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3028210 > ... > Many thanks, this sets things out clearly. On a first scan, one thing > caught my eye: > > I think we should implement Python scalars for the other types and then > eliminate rank-0 arrays completely. I could have a student do this in a > few weeks if it was agreed on. > > -Travis Oliphant [13-Sep-02] > I expressed my view when I started "A case for rank-0 arrays" thread. (See link above.) In summary, dimensionless arrays and scalars are different beasts and both have a right to live. > > The Wiki sems to start with the design decision to implement zero > dimension arrays. > It then goes on to explore the problems created by that design decision. > The wiki has a short section on "The Need for Zero-Rank Arrays." Please feel free to take over that page and add you criticism of the status quo. I think wiki is a better medium to discuss this issue than the mailing list. > For matrices, we accept the rule that a matrix has two dimensions, not more > not less. What problems would be created if we had the rule that an array > has not less than one dimension? > For me the main problem is the dificulty to write generic code both in C and in Python. Most issues stem from the fact that scalars are immutable while arrays (including dimentionless) are not. > The little script below produces different results, depending on the ordering > of the operands. This provides another challenge for the newcomer. I don't understand how this is related to dimensionless arrays. Looks like an array scalar feature: >>> type(2.0 * int_(1)) >>> type(int_(1) * 2.0) (I believe this can be fixed once scalar math is implemented.) Dimensionless arrays seem to be more consistent in this particular case: >>> type(array(1) * 2.0) >>> type(2.0*array(1)) I believe type(array(1) * 2.0) should be ndarray, but that's a different story. > > # arrayStuff.py > > def inArrayWorld(): > import numpy as _n > a= _n.arange(4) > return a[2] > > def anotherWorld(a): > import math > w= 2.6 * a > x= a * 2 > y= 2.6 * a > z= math.log10(a) > return z, type(z), type(y), type(x), type(w) > > print anotherWorld(inArrayWorld()) > > From schofield at ftw.at Sat Mar 18 08:27:03 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat Mar 18 08:27:03 2006 Subject: [Numpy-discussion] NumPy 0.9.6 and win32pdh Message-ID: <441C34DD.1000901@ftw.at> Hi all, I'm trying to compile Win32 binaries for a new release of SciPy. I'm getting a warning message upon import after installing numpy-0.9.6-win32-py2.4.exe: import testing -> failed: No module named win32pdh and I'm unable to run any of NumPy's unit tests unless I manually comment out the block if os.name=='nt': ... in testing\utils.py. Has anyone else seen this? Does anyone know if the win32pdh module is supposed to be installed with the python-2.4.2.msi installer from python.org? The Python 2.3 installer, numpy-0.9.6-win32-py2.3.exe, works fine, because the corresponding if statement has an additional condition (" and sys.version[:3] > '2.3'"), so the import is never attempted. -- Ed From robert.kern at gmail.com Sat Mar 18 10:18:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat Mar 18 10:18:01 2006 Subject: [Numpy-discussion] Re: NumPy 0.9.6 and win32pdh In-Reply-To: <441C34DD.1000901@ftw.at> References: <441C34DD.1000901@ftw.at> Message-ID: Ed Schofield wrote: > Hi all, > > I'm trying to compile Win32 binaries for a new release of SciPy. I'm > getting a warning message upon import after installing > numpy-0.9.6-win32-py2.4.exe: > > import testing -> failed: No module named win32pdh > > and I'm unable to run any of NumPy's unit tests unless I manually > comment out the block > > if os.name=='nt': > ... > > in testing\utils.py. > > Has anyone else seen this? Does anyone know if the win32pdh module is > supposed to be installed with the python-2.4.2.msi installer from > python.org? It's from the win32all package. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dd55 at cornell.edu Sat Mar 18 12:08:01 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat Mar 18 12:08:01 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: References: <200603131943.53857.dd55@cornell.edu> Message-ID: <200603181507.00136.dd55@cornell.edu> On Monday 13 March 2006 9:05 pm, Robert Kern wrote: > Darren Dale wrote: > > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my > > home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 > > and gcc-4.1. When I try to build numpy I get warnings like "warning: > > build_ext: fcompiler=gnu is not available." I tried forcing numpy to use > > gfortran with "python setup.py config_fc --fcompiler=gnu95 build", and > > got the same warning for fcompiler=gnu95. > > Building numpy shouldn't require any kind of Fortran compiler. > > However, the problem is that the version-recognizing regex doesn't like the > version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in > numpy/distutils/fcompiler/gnu.py and the class attribute > Gnu95FCompiler.version_pattern . Thanks Robert. May I suggest the following change, from: version_pattern = r'GNU Fortran 95 \(GCC (?P[^\s*\)]+)' to: version_pattern = r'GNU Fortran 95 \(GCC?\) (?P[^\s*\)]+)' along with the following comment to go with the debian example: # Gentoo: GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) Also, I wonder if I found a bug in numpy/distutils/ccompiler.py. if status in ok_status: m = re.match(self.version_pattern,output) if m: version = m.group('version') if not m: raise ValueError("compiler version not matched (%r)" %\ (version,)) version = LooseVersion(version) self.version = version return version That ValueError will never be raised, since it has already been established that m is not NoneType. Darren From cookedm at physics.mcmaster.ca Sat Mar 18 14:48:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sat Mar 18 14:48:02 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: <200603181507.00136.dd55@cornell.edu> (Darren Dale's message of "Sat, 18 Mar 2006 15:06:59 -0500") References: <200603131943.53857.dd55@cornell.edu> <200603181507.00136.dd55@cornell.edu> Message-ID: Darren Dale writes: > On Monday 13 March 2006 9:05 pm, Robert Kern wrote: >> Darren Dale wrote: >> > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my >> > home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 >> > and gcc-4.1. When I try to build numpy I get warnings like "warning: >> > build_ext: fcompiler=gnu is not available." I tried forcing numpy to use >> > gfortran with "python setup.py config_fc --fcompiler=gnu95 build", and >> > got the same warning for fcompiler=gnu95. >> >> Building numpy shouldn't require any kind of Fortran compiler. >> >> However, the problem is that the version-recognizing regex doesn't like the >> version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in >> numpy/distutils/fcompiler/gnu.py and the class attribute >> Gnu95FCompiler.version_pattern . > > Thanks Robert. May I suggest the following change, from: > version_pattern = r'GNU Fortran 95 \(GCC (?P[^\s*\)]+)' > to: > version_pattern = r'GNU Fortran 95 \(GCC?\) (?P[^\s*\)]+)' > along with the following comment to go with the debian example: > # Gentoo: GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) I fixed that a few days ago and forgot to check my solution in :-) Now, you can define a version_match method that can do more complex matching, instead of just a regex. The GNU Fortran compilers now use this: gfortran, for instance, checks the version output to start with 'GNU Fortran 95', then something versionish (the regex [-.\d]+). > Also, I wonder if I found a bug in numpy/distutils/ccompiler.py. > > if status in ok_status: > m = re.match(self.version_pattern,output) > if m: > version = m.group('version') > if not m: > raise ValueError("compiler version not matched (%r)" %\ > (version,)) > version = LooseVersion(version) > self.version = version > return version > > That ValueError will never be raised, since it has already been established > that m is not NoneType. This is fixed too. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From dd55 at cornell.edu Sat Mar 18 15:07:09 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat Mar 18 15:07:09 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: References: <200603131943.53857.dd55@cornell.edu> <200603181507.00136.dd55@cornell.edu> Message-ID: <200603181806.29561.dd55@cornell.edu> Hi David, I cant build numpy with the changes you just commited: Running from numpy source directory. No module named __svn_version__ F2PY Version 2_2258 blas_opt_info: blas_mkl_info: /home/darren/src/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c include_dirs = ['/usr/include/atlas'] running build_src building extension "atlas_version" sources creating build creating build/src adding 'build/src/atlas_version_-0x54967df6.c' to sources. running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext building 'atlas_version' extension compiling C sources i686-pc-linux-gnu-gcc options: '-pthread -fno-strict-aliasing -DNDEBUG -fPIC' creating build/temp.linux-i686-2.4 creating build/temp.linux-i686-2.4/build creating build/temp.linux-i686-2.4/build/src compile options: '-I/usr/include/atlas -Inumpy/core/include -I/usr/include/python2.4 -c' i686-pc-linux-gnu-gcc: build/src/atlas_version_-0x54967df6.c i686-pc-linux-gnu-gcc -pthread -shared build/temp.linux-i686-2.4/build/src/atlas_version_-0x54967df6.o -L/usr/lib -llapack -lblas -lcblas -latlas -o build/temp.linux-i686-2.4/atlas_version.so FOUND: libraries = ['lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c define_macros = [('ATLAS_INFO', '"\\"3.7.11\\""')] include_dirs = ['/usr/include/atlas'] lapack_opt_info: lapack_mkl_info: mkl_info: NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS /home/darren/src/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack_atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 include_dirs = ['/usr/include/atlas'] running build_src building extension "atlas_version" sources adding 'build/src/atlas_version_0x26de9279.c' to sources. running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext invalid syntax (gnu.py, line 225) Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelFCompiler customize LaheyFCompiler customize PGroupFCompiler customize AbsoftFCompiler customize NAGFCompiler invalid syntax (gnu.py, line 225) customize CompaqFCompiler customize IntelItaniumFCompiler invalid syntax (gnu.py, line 225) customize G95FCompiler invalid syntax (gnu.py, line 225) invalid syntax (gnu.py, line 225) ##### msg: invalid syntax (gnu.py, line 225) invalid syntax (gnu.py, line 225) FOUND: libraries = ['lapack', 'lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include/atlas'] Warning: distutils distribution has been initialized, it may be too late to add an extension lapack_lite Warning: distutils distribution has been initialized, it may be too late to add a subpackage linalg Warning: distutils distribution has been initialized, it may be too late to add an extension mtrand Warning: distutils distribution has been initialized, it may be too late to add a subpackage random Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 63, in setup_package config.add_subpackage('numpy') File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 592, in add_subpackage config_list = self.get_subpackage(subpackage_name,subpackage_path) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 582, in get_subpackage subpackage_path) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 539, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "/home/darren/src/numpy/setup.py", line 15, in configuration File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 636, in add_data_dir self.add_data_files((ds,filenames)) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 702, in add_data_files dist.data_files.extend(data_dict.items()) AttributeError: 'NoneType' object has no attribute 'extend' On Saturday 18 March 2006 5:47 pm, David M. Cooke wrote: > Darren Dale writes: > > On Monday 13 March 2006 9:05 pm, Robert Kern wrote: > >> Darren Dale wrote: > >> > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my > >> > home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 > >> > and gcc-4.1. When I try to build numpy I get warnings like "warning: > >> > build_ext: fcompiler=gnu is not available." I tried forcing numpy to > >> > use gfortran with "python setup.py config_fc --fcompiler=gnu95 build", > >> > and got the same warning for fcompiler=gnu95. > >> > >> Building numpy shouldn't require any kind of Fortran compiler. > >> > >> However, the problem is that the version-recognizing regex doesn't like > >> the version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in > >> numpy/distutils/fcompiler/gnu.py and the class attribute > >> Gnu95FCompiler.version_pattern . > > > > Thanks Robert. May I suggest the following change, from: > > version_pattern = r'GNU Fortran 95 \(GCC (?P[^\s*\)]+)' > > to: > > version_pattern = r'GNU Fortran 95 \(GCC?\) (?P[^\s*\)]+)' > > along with the following comment to go with the debian example: > > # Gentoo: GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) > > I fixed that a few days ago and forgot to check my solution in :-) > Now, you can define a version_match method that can do more complex > matching, instead of just a regex. The GNU Fortran compilers now use > this: gfortran, for instance, checks the version output to start > with 'GNU Fortran 95', then something versionish (the regex [-.\d]+). > > > Also, I wonder if I found a bug in numpy/distutils/ccompiler.py. > > > > if status in ok_status: > > m = re.match(self.version_pattern,output) > > if m: > > version = m.group('version') > > if not m: > > raise ValueError("compiler version not matched (%r)" %\ > > (version,)) > > version = LooseVersion(version) > > self.version = version > > return version > > > > That ValueError will never be raised, since it has already been > > established that m is not NoneType. > > This is fixed too. From dd55 at cornell.edu Sat Mar 18 15:19:01 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat Mar 18 15:19:01 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: <200603181806.29561.dd55@cornell.edu> References: <200603131943.53857.dd55@cornell.edu> <200603181806.29561.dd55@cornell.edu> Message-ID: <200603181818.12780.dd55@cornell.edu> On Saturday 18 March 2006 6:06 pm, Darren Dale wrote: > Hi David, > > I cant build numpy with the changes you just commited: Scratch that, sorry. I deleted my local copy of gnu.py and downloaded a fresh svn version and was able to build. Thanks! From oliphant.travis at ieee.org Sat Mar 18 20:18:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Mar 18 20:18:05 2006 Subject: [Numpy-discussion] NumPy 0.9.6 and win32pdh In-Reply-To: <441C34DD.1000901@ftw.at> References: <441C34DD.1000901@ftw.at> Message-ID: <441CDB57.6010508@ieee.org> Ed Schofield wrote: > Hi all, > > I'm trying to compile Win32 binaries for a new release of SciPy. I'm > getting a warning message upon import after installing > numpy-0.9.6-win32-py2.4.exe: > > import testing -> failed: No module named win32pdh > > and I'm unable to run any of NumPy's unit tests unless I manually > comment out the block > Hmm... I guess I though that win32pdh was part of python 2.4 (I saw this error when building Pyton 2.3 packages but not Python 2.4). But, of course it was probably that I did or didn't have win32all installed. That's unfortunate. The win32pdh dependency needs to be fixed, then. It's too bad this dependency was created without appropriate checks. No doubt we'll have several who complain about this one until numpy 0.9.7 comes out... -Travis From oliphant.travis at ieee.org Sat Mar 18 20:59:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Mar 18 20:59:03 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <200603171630.27075.luszczek@cs.utk.edu> References: <200603171519.35502.luszczek@cs.utk.edu> <200603171630.27075.luszczek@cs.utk.edu> Message-ID: <441CE4DE.6040101@ieee.org> Piotr Luszczek wrote: >> Show us some code that works. I'm not interested in >> implementing your feature request. You are. There's plenty of work >> that you can do that doesn't depend on anyone else agreeing with you, >> so you can stop arguing and start coding. >> > > Arguing is good to a point and I think you're right that it's time to > stop. > All true. However, I want to thank Piotr for illustrating an important but easy to overlook point that in-fact "advanced" indexing produces copies while "simple-indexing" produces views. The reason behind this as Tim and Rick later illustrate is that no one quite knows how to produce "view" semantics for an arbitrarily indexed array. I did actually think about this for a while because I don't completely like the different behaviors which can produce warts like the one that Tim showed us. I realized that it would be possible to extend our memory model of an array to include an index array in the array structure itself that would indicate how to advance through memory (basically a pointer or its equivalent for each array element). Rick showed that because this would have to be a "copy" of the indices and so would create too much slowness. My reason for not going there was because it seemed to add too much complication for my tastes at the time to all the code. But, one could still sub-class the ndarray and implement such a thing. I think this is the best place to explore such experimental ideas. Thanks again for the discussion. There are issues like this that it is good to get out into the open. -Travis From vzkezlxdkhy at mail.medscape.com Sun Mar 19 16:51:05 2006 From: vzkezlxdkhy at mail.medscape.com (vzkezlxdkhy) Date: Sun Mar 19 16:51:05 2006 Subject: [Numpy-discussion] Fw: Message-ID: <000c01c64bb9$08d55730$726ddf94@avelazquez> ----- Original Message ----- From: Herman Augustus To: dmqztgjn at nate.com Sent: Monday, March 13, 2006 11:16 PM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bwcexnk.gif Type: image/gif Size: 9499 bytes Desc: not available URL: From hurak at control.felk.cvut.cz Mon Mar 20 02:37:01 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Mon Mar 20 02:37:01 2006 Subject: [Numpy-discussion] Rationale behind behaviour of atleast_3d in Numpy Message-ID: Hello, As a newcommer, I am trying not only to learn to use Numpy, but also to understand it. I think I need some help with function atleast_3d. When applied to 2d array, its behaviour is different then when applied to 1d array (it is a documented feature): a = array([1,2,3]) A = array([[1,2,3],[4,5,6]]) a3d = atleast_3d(a) print a3d.shape ---> (1, 3, 1) A3d = atleast_3d(A) print A3d.shape ---> (2, 3, 1) What is the rationale behind this in my opinion unintuitive inconsistency? I especially don't understand the behaviour for 1d arrays. But even with 2d arrays, I am inclined to think (with my beginner-level of mastering Python and Numpy) that a more natural result of shape function would be (1,2,3). Thank you very much for some explanation, Zdenek Hurak From chanley at stsci.edu Mon Mar 20 06:07:07 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 20 06:07:07 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 Message-ID: <441EB6E8.8030504@stsci.edu> Solaris builds began failing yesterday with the following output: Your "cron" job on basil.stsci.edu /home/chanley/bin/numpy_update.csh produced the following output: Sun Mar 19 07:20:04 EST 2006 ### NUMPY Daily Build / Build Test ### ### Setup test environment. ### /usr/ra/pyssg/Python-2.4.2/bin/python ### Checkout, build, and install new version of numpy. ### /data/basil5 A numpy/LICENSE.txt A numpy/scipy_compatibility A numpy/TODO.txt A numpy/THANKS.txt A numpy/TEST_COMMIT A numpy/setup.py A numpy/numpy A numpy/numpy/_import_tools.py A numpy/numpy/f2py A numpy/numpy/f2py/f2py.1 A numpy/numpy/f2py/diagnose.py A numpy/numpy/f2py/rules.py A numpy/numpy/f2py/info.py A numpy/numpy/f2py/capi_maps.py A numpy/numpy/f2py/auxfuncs.py A numpy/numpy/f2py/TODO.txt A numpy/numpy/f2py/cb_rules.py A numpy/numpy/f2py/__init__.py A numpy/numpy/f2py/setup.py A numpy/numpy/f2py/src A numpy/numpy/f2py/src/test A numpy/numpy/f2py/src/test/foo.f A numpy/numpy/f2py/src/test/bar.f A numpy/numpy/f2py/src/test/foo90.f90 A numpy/numpy/f2py/src/test/foomodule.c A numpy/numpy/f2py/src/test/Makefile A numpy/numpy/f2py/src/test/wrap.f A numpy/numpy/f2py/src/fortranobject.c A numpy/numpy/f2py/src/fortranobject.h A numpy/numpy/f2py/docs A numpy/numpy/f2py/docs/HISTORY.txt A numpy/numpy/f2py/docs/pytest.py A numpy/numpy/f2py/docs/usersguide A numpy/numpy/f2py/docs/usersguide/scalar_session.dat A numpy/numpy/f2py/docs/usersguide/fib2.pyf A numpy/numpy/f2py/docs/usersguide/callback.f A numpy/numpy/f2py/docs/usersguide/calculate.f A numpy/numpy/f2py/docs/usersguide/run_main_session.dat A numpy/numpy/f2py/docs/usersguide/index.txt A numpy/numpy/f2py/docs/usersguide/var.pyf A numpy/numpy/f2py/docs/usersguide/string.f A numpy/numpy/f2py/docs/usersguide/spam.pyf A numpy/numpy/f2py/docs/usersguide/ftype_session.dat A numpy/numpy/f2py/docs/usersguide/extcallback.f A numpy/numpy/f2py/docs/usersguide/setup_example.py A numpy/numpy/f2py/docs/usersguide/array.f A numpy/numpy/f2py/docs/usersguide/common_session.dat A numpy/numpy/f2py/docs/usersguide/var_session.dat A numpy/numpy/f2py/docs/usersguide/moddata.f90 A numpy/numpy/f2py/docs/usersguide/callback_session.dat A numpy/numpy/f2py/docs/usersguide/calculate_session.dat A numpy/numpy/f2py/docs/usersguide/spam_session.dat A numpy/numpy/f2py/docs/usersguide/docutils.conf A numpy/numpy/f2py/docs/usersguide/callback2.pyf A numpy/numpy/f2py/docs/usersguide/string_session.dat A numpy/numpy/f2py/docs/usersguide/moddata_session.dat A numpy/numpy/f2py/docs/usersguide/extcallback_session.dat A numpy/numpy/f2py/docs/usersguide/array_session.dat A numpy/numpy/f2py/docs/usersguide/scalar.f A numpy/numpy/f2py/docs/usersguide/allocarr.f90 A numpy/numpy/f2py/docs/usersguide/compile_session.dat A numpy/numpy/f2py/docs/usersguide/fib1.f A numpy/numpy/f2py/docs/usersguide/fib3.f A numpy/numpy/f2py/docs/usersguide/allocarr_session.dat A numpy/numpy/f2py/docs/usersguide/ftype.f A numpy/numpy/f2py/docs/usersguide/default.css A numpy/numpy/f2py/docs/usersguide/common.f A numpy/numpy/f2py/docs/usersguide/fib1.pyf A numpy/numpy/f2py/docs/pyforttest.pyf A numpy/numpy/f2py/docs/simple_session.dat A numpy/numpy/f2py/docs/FAQ.txt A numpy/numpy/f2py/docs/THANKS.txt A numpy/numpy/f2py/docs/hello.f A numpy/numpy/f2py/docs/OLDNEWS.txt A numpy/numpy/f2py/docs/docutils.conf A numpy/numpy/f2py/docs/README.txt A numpy/numpy/f2py/docs/TESTING.txt A numpy/numpy/f2py/docs/default.css A numpy/numpy/f2py/docs/simple.f A numpy/numpy/f2py/common_rules.py A numpy/numpy/f2py/NEWS.txt A numpy/numpy/f2py/use_rules.py A numpy/numpy/f2py/f2py2e.py A numpy/numpy/f2py/setup.cfg A numpy/numpy/f2py/f90mod_rules.py A numpy/numpy/f2py/func2subr.py A numpy/numpy/f2py/tests A numpy/numpy/f2py/tests/c A numpy/numpy/f2py/tests/c/return_real.py A numpy/numpy/f2py/tests/f77 A numpy/numpy/f2py/tests/f77/return_logical.py A numpy/numpy/f2py/tests/f77/return_character.py A numpy/numpy/f2py/tests/f77/callback.py A numpy/numpy/f2py/tests/f77/return_integer.py A numpy/numpy/f2py/tests/f77/return_real.py A numpy/numpy/f2py/tests/f77/return_complex.py A numpy/numpy/f2py/tests/run_all.py A numpy/numpy/f2py/tests/array_from_pyobj A numpy/numpy/f2py/tests/array_from_pyobj/wrapmodule.c A numpy/numpy/f2py/tests/array_from_pyobj/tests A numpy/numpy/f2py/tests/array_from_pyobj/tests/test_array_from_pyobj.py A numpy/numpy/f2py/tests/array_from_pyobj/__init__.py A numpy/numpy/f2py/tests/array_from_pyobj/setup.py A numpy/numpy/f2py/tests/mixed A numpy/numpy/f2py/tests/mixed/foo_fixed.f90 A numpy/numpy/f2py/tests/mixed/foo_free.f90 A numpy/numpy/f2py/tests/mixed/foo.f A numpy/numpy/f2py/tests/mixed/run.py A numpy/numpy/f2py/tests/f90 A numpy/numpy/f2py/tests/f90/return_logical.py A numpy/numpy/f2py/tests/f90/return_character.py A numpy/numpy/f2py/tests/f90/return_integer.py A numpy/numpy/f2py/tests/f90/return_real.py A numpy/numpy/f2py/tests/f90/return_complex.py A numpy/numpy/f2py/f2py_testing.py A numpy/numpy/f2py/doc A numpy/numpy/f2py/doc/notes.tex A numpy/numpy/f2py/doc/win32_notes.txt A numpy/numpy/f2py/doc/multiarray A numpy/numpy/f2py/doc/multiarray/run.pyf A numpy/numpy/f2py/doc/multiarray/array_from_pyobj.c A numpy/numpy/f2py/doc/multiarray/bar.c A numpy/numpy/f2py/doc/multiarray/fun.pyf A numpy/numpy/f2py/doc/multiarray/foo.f A numpy/numpy/f2py/doc/multiarray/transpose.txt A numpy/numpy/f2py/doc/multiarray/fortran_array_from_pyobj.txt A numpy/numpy/f2py/doc/f2python9-final A numpy/numpy/f2py/doc/f2python9-final/structure.jpg A numpy/numpy/f2py/doc/f2python9-final/mk_ps.sh A numpy/numpy/f2py/doc/f2python9-final/mk_html.sh A numpy/numpy/f2py/doc/f2python9-final/aerostructure.jpg A numpy/numpy/f2py/doc/f2python9-final/flow.jpg A numpy/numpy/f2py/doc/f2python9-final/src A numpy/numpy/f2py/doc/f2python9-final/src/examples A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1session.txt A numpy/numpy/f2py/doc/f2python9-final/src/examples/foo.pyf A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1.f A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1mess.txt A numpy/numpy/f2py/doc/f2python9-final/src/examples/foom.pyf A numpy/numpy/f2py/doc/f2python9-final/mk_pdf.sh A numpy/numpy/f2py/doc/f2python9-final/README.txt A numpy/numpy/f2py/doc/intro.tex A numpy/numpy/f2py/doc/multiarrays.txt A numpy/numpy/f2py/doc/options.tex A numpy/numpy/f2py/doc/collectinput.py A numpy/numpy/f2py/doc/ex1 A numpy/numpy/f2py/doc/ex1/foobar.f90 A numpy/numpy/f2py/doc/ex1/runme A numpy/numpy/f2py/doc/ex1/foo.f A numpy/numpy/f2py/doc/ex1/arr.f A numpy/numpy/f2py/doc/ex1/bar.f A numpy/numpy/f2py/doc/ex1/foobar-smart.f90 A numpy/numpy/f2py/doc/ex1/foobarmodule.tex A numpy/numpy/f2py/doc/bugs.tex A numpy/numpy/f2py/doc/commands.tex A numpy/numpy/f2py/doc/signaturefile.tex A numpy/numpy/f2py/doc/fortranobject.tex A numpy/numpy/f2py/doc/apps.tex A numpy/numpy/f2py/doc/Release-1.x.txt A numpy/numpy/f2py/doc/using_F_compiler.txt A numpy/numpy/f2py/doc/Release-2.x.txt A numpy/numpy/f2py/doc/Release-3.x.txt A numpy/numpy/f2py/doc/Release-4.x.txt A numpy/numpy/f2py/doc/f2py2e.tex A numpy/numpy/f2py/doc/python9.tex A numpy/numpy/f2py/doc/index.html A numpy/numpy/f2py/doc/Makefile A numpy/numpy/f2py/doc/oldnews.html A numpy/numpy/f2py/crackfortran.py A numpy/numpy/f2py/cfuncs.py A numpy/numpy/f2py/__version__.py A numpy/numpy/f2py/README.txt A numpy/numpy/f2py/Makefile A numpy/numpy/f2py/BUGS.txt A numpy/numpy/random A numpy/numpy/random/info.py A numpy/numpy/random/mtrand A numpy/numpy/random/mtrand/Python.pxi A numpy/numpy/random/mtrand/distributions.c A numpy/numpy/random/mtrand/initarray.c A numpy/numpy/random/mtrand/mtrand.pyx A numpy/numpy/random/mtrand/mtrand.c A numpy/numpy/random/mtrand/numpy.pxi A numpy/numpy/random/mtrand/distributions.h A numpy/numpy/random/mtrand/generate_mtrand_c.py A numpy/numpy/random/mtrand/initarray.h A numpy/numpy/random/mtrand/randomkit.c A numpy/numpy/random/mtrand/randomkit.h A numpy/numpy/random/__init__.py A numpy/numpy/random/setup.py A numpy/numpy/add_newdocs.py A numpy/numpy/distutils A numpy/numpy/distutils/core.py A numpy/numpy/distutils/misc_util.py A numpy/numpy/distutils/fcompiler A numpy/numpy/distutils/fcompiler/mips.py A numpy/numpy/distutils/fcompiler/gnu.py A numpy/numpy/distutils/fcompiler/intel.py A numpy/numpy/distutils/fcompiler/vast.py A numpy/numpy/distutils/fcompiler/absoft.py A numpy/numpy/distutils/fcompiler/__init__.py A numpy/numpy/distutils/fcompiler/none.py A numpy/numpy/distutils/fcompiler/compaq.py A numpy/numpy/distutils/fcompiler/lahey.py A numpy/numpy/distutils/fcompiler/g95.py A numpy/numpy/distutils/fcompiler/hpux.py A numpy/numpy/distutils/fcompiler/nag.py A numpy/numpy/distutils/fcompiler/sun.py A numpy/numpy/distutils/fcompiler/pg.py A numpy/numpy/distutils/fcompiler/ibm.py A numpy/numpy/distutils/info.py A numpy/numpy/distutils/line_endings.py A numpy/numpy/distutils/from_template.py A numpy/numpy/distutils/__init__.py A numpy/numpy/distutils/system_info.py A numpy/numpy/distutils/conv_template.py A numpy/numpy/distutils/setup.py A numpy/numpy/distutils/cpuinfo.py A numpy/numpy/distutils/lib2def.py A numpy/numpy/distutils/intelccompiler.py A numpy/numpy/distutils/tests A numpy/numpy/distutils/tests/f2py_ext A numpy/numpy/distutils/tests/f2py_ext/tests A numpy/numpy/distutils/tests/f2py_ext/tests/test_fib2.py A numpy/numpy/distutils/tests/f2py_ext/__init__.py A numpy/numpy/distutils/tests/f2py_ext/setup.py A numpy/numpy/distutils/tests/f2py_ext/src A numpy/numpy/distutils/tests/f2py_ext/src/fib2.pyf A numpy/numpy/distutils/tests/f2py_ext/src/fib1.f A numpy/numpy/distutils/tests/pyrex_ext A numpy/numpy/distutils/tests/pyrex_ext/tests A numpy/numpy/distutils/tests/pyrex_ext/tests/test_primes.py A numpy/numpy/distutils/tests/pyrex_ext/__init__.py A numpy/numpy/distutils/tests/pyrex_ext/setup.py A numpy/numpy/distutils/tests/pyrex_ext/primes.pyx A numpy/numpy/distutils/tests/setup.py A numpy/numpy/distutils/tests/gen_ext A numpy/numpy/distutils/tests/gen_ext/tests A numpy/numpy/distutils/tests/gen_ext/tests/test_fib3.py A numpy/numpy/distutils/tests/gen_ext/__init__.py A numpy/numpy/distutils/tests/gen_ext/setup.py A numpy/numpy/distutils/tests/swig_ext A numpy/numpy/distutils/tests/swig_ext/tests A numpy/numpy/distutils/tests/swig_ext/tests/test_example.py A numpy/numpy/distutils/tests/swig_ext/tests/test_example2.py A numpy/numpy/distutils/tests/swig_ext/__init__.py A numpy/numpy/distutils/tests/swig_ext/setup.py A numpy/numpy/distutils/tests/swig_ext/src A numpy/numpy/distutils/tests/swig_ext/src/example.i A numpy/numpy/distutils/tests/swig_ext/src/zoo.cc A numpy/numpy/distutils/tests/swig_ext/src/example.c A numpy/numpy/distutils/tests/swig_ext/src/zoo.h A numpy/numpy/distutils/tests/swig_ext/src/zoo.i A numpy/numpy/distutils/tests/f2py_f90_ext A numpy/numpy/distutils/tests/f2py_f90_ext/tests A numpy/numpy/distutils/tests/f2py_f90_ext/tests/test_foo.py A numpy/numpy/distutils/tests/f2py_f90_ext/include A numpy/numpy/distutils/tests/f2py_f90_ext/include/body.f90 A numpy/numpy/distutils/tests/f2py_f90_ext/__init__.py A numpy/numpy/distutils/tests/f2py_f90_ext/setup.py A numpy/numpy/distutils/tests/f2py_f90_ext/src A numpy/numpy/distutils/tests/f2py_f90_ext/src/foo_free.f90 A numpy/numpy/distutils/tests/test_misc_util.py A numpy/numpy/distutils/extension.py A numpy/numpy/distutils/ccompiler.py A numpy/numpy/distutils/log.py A numpy/numpy/distutils/__version__.py A numpy/numpy/distutils/unixccompiler.py A numpy/numpy/distutils/exec_command.py A numpy/numpy/distutils/mingw32ccompiler.py A numpy/numpy/distutils/command A numpy/numpy/distutils/command/build_clib.py A numpy/numpy/distutils/command/egg_info.py A numpy/numpy/distutils/command/__init__.py A numpy/numpy/distutils/command/build.py A numpy/numpy/distutils/command/build_ext.py A numpy/numpy/distutils/command/config_compiler.py A numpy/numpy/distutils/command/install_data.py A numpy/numpy/distutils/command/install_headers.py A numpy/numpy/distutils/command/bdist_rpm.py A numpy/numpy/distutils/command/config.py A numpy/numpy/distutils/command/build_scripts.py A numpy/numpy/distutils/command/build_src.py A numpy/numpy/distutils/command/install.py A numpy/numpy/distutils/command/sdist.py A numpy/numpy/distutils/command/build_py.py A numpy/numpy/doc A numpy/numpy/doc/records.txt A numpy/numpy/doc/ufuncs.txt A numpy/numpy/doc/DISTUTILS.txt A numpy/numpy/doc/pyrex A numpy/numpy/doc/pyrex/numpyx.pyx A numpy/numpy/doc/pyrex/numpyx.c A numpy/numpy/doc/pyrex/c_numpy.pxd A numpy/numpy/doc/pyrex/MANIFEST A numpy/numpy/doc/pyrex/setup.py A numpy/numpy/doc/pyrex/notes A numpy/numpy/doc/pyrex/run_test.py A numpy/numpy/doc/pyrex/Makefile A numpy/numpy/doc/pyrex/c_python.pxd A numpy/numpy/doc/swig A numpy/numpy/doc/swig/numpy.i A numpy/numpy/doc/swig/series.h A numpy/numpy/doc/swig/Series.i A numpy/numpy/doc/swig/testSeries.py A numpy/numpy/doc/swig/HelperFunctions.txt A numpy/numpy/doc/swig/setup.py A numpy/numpy/doc/swig/series.cxx A numpy/numpy/doc/swig/Makefile A numpy/numpy/doc/swig/README A numpy/numpy/doc/CAPI.txt A numpy/numpy/doc/README.txt A numpy/numpy/__init__.py A numpy/numpy/lib A numpy/numpy/lib/scimath.py A numpy/numpy/lib/shape_base.py A numpy/numpy/lib/machar.py A numpy/numpy/lib/info.py A numpy/numpy/lib/twodim_base.py A numpy/numpy/lib/__init__.py A numpy/numpy/lib/src A numpy/numpy/lib/src/_compiled_base.c A numpy/numpy/lib/setup.py A numpy/numpy/lib/utils.py A numpy/numpy/lib/getlimits.py A numpy/numpy/lib/tests A numpy/numpy/lib/tests/test_shape_base.py A numpy/numpy/lib/tests/test_arraysetops.py A numpy/numpy/lib/tests/test_twodim_base.py A numpy/numpy/lib/tests/test_type_check.py A numpy/numpy/lib/tests/test_function_base.py A numpy/numpy/lib/tests/test_getlimits.py A numpy/numpy/lib/tests/test_polynomial.py A numpy/numpy/lib/tests/test_ufunclike.py A numpy/numpy/lib/tests/test_index_tricks.py A numpy/numpy/lib/mlab.py A numpy/numpy/lib/convertcode.py A numpy/numpy/lib/arraysetops.py A numpy/numpy/lib/UserArray.py A numpy/numpy/lib/type_check.py A numpy/numpy/lib/function_base.py A numpy/numpy/lib/polynomial.py A numpy/numpy/lib/ufunclike.py A numpy/numpy/lib/index_tricks.py A numpy/numpy/linalg A numpy/numpy/linalg/blas_lite.c A numpy/numpy/linalg/lapack_litemodule.c A numpy/numpy/linalg/f2c.h A numpy/numpy/linalg/info.py A numpy/numpy/linalg/zlapack_lite.c A numpy/numpy/linalg/old.py A numpy/numpy/linalg/__init__.py A numpy/numpy/linalg/setup.py A numpy/numpy/linalg/f2c_lite.c A numpy/numpy/linalg/dlamch.c A numpy/numpy/linalg/dlapack_lite.c A numpy/numpy/linalg/linalg.py A numpy/numpy/setup.py A numpy/numpy/core A numpy/numpy/core/info.py A numpy/numpy/core/defchararray.py A numpy/numpy/core/arrayprint.py A numpy/numpy/core/include A numpy/numpy/core/include/numpy A numpy/numpy/core/include/numpy/arrayobject.h A numpy/numpy/core/include/numpy/arrayscalars.h A numpy/numpy/core/include/numpy/ufuncobject.h A numpy/numpy/core/include/numpy/fenv A numpy/numpy/core/include/numpy/fenv/fenv.c A numpy/numpy/core/include/numpy/fenv/fenv.h A numpy/numpy/core/ma.py A numpy/numpy/core/__init__.py A numpy/numpy/core/setup.py A numpy/numpy/core/src A numpy/numpy/core/src/_signbit.c A numpy/numpy/core/src/multiarraymodule.c A numpy/numpy/core/src/arraytypes.inc.src A numpy/numpy/core/src/_sortmodule.c.src A numpy/numpy/core/src/arraymethods.c A numpy/numpy/core/src/ucsnarrow.c A numpy/numpy/core/src/arrayobject.c A numpy/numpy/core/src/_isnan.c A numpy/numpy/core/src/scalartypes.inc.src A numpy/numpy/core/src/ufuncobject.c A numpy/numpy/core/src/umathmodule.c.src A numpy/numpy/core/src/scalarmathmodule.c.src A numpy/numpy/core/records.py A numpy/numpy/core/oldnumeric.py A numpy/numpy/core/blasdot A numpy/numpy/core/blasdot/_dotblas.c A numpy/numpy/core/blasdot/cblas.h A numpy/numpy/core/numeric.py A numpy/numpy/core/_internal.py A numpy/numpy/core/tests A numpy/numpy/core/tests/test_multiarray.py A numpy/numpy/core/tests/test_ma.py A numpy/numpy/core/tests/test_umath.py A numpy/numpy/core/tests/test_oldnumeric.py A numpy/numpy/core/tests/test_records.py A numpy/numpy/core/tests/test_numeric.py A numpy/numpy/core/tests/test_defmatrix.py A numpy/numpy/core/tests/test_unicode.py A numpy/numpy/core/tests/test_numerictypes.py A numpy/numpy/core/tests/testdata.fits A numpy/numpy/core/memmap.py A numpy/numpy/core/code_generators A numpy/numpy/core/code_generators/generate_umath.py A numpy/numpy/core/code_generators/array_api_order.txt A numpy/numpy/core/code_generators/ufunc_api_order.txt A numpy/numpy/core/code_generators/generate_array_api.py A numpy/numpy/core/code_generators/genapi.py A numpy/numpy/core/code_generators/generate_ufunc_api.py A numpy/numpy/core/code_generators/multiarray_api_order.txt A numpy/numpy/core/defmatrix.py A numpy/numpy/core/numerictypes.py A numpy/numpy/dual.py A numpy/numpy/version.py A numpy/numpy/dft A numpy/numpy/dft/fftpack.c A numpy/numpy/dft/fftpack_litemodule.c A numpy/numpy/dft/info.py A numpy/numpy/dft/tests A numpy/numpy/dft/tests/test_helper.py A numpy/numpy/dft/fftpack.h A numpy/numpy/dft/fftpack.py A numpy/numpy/dft/__init__.py A numpy/numpy/dft/helper.py A numpy/numpy/dft/setup.py A numpy/numpy/testing A numpy/numpy/testing/numpytest.py A numpy/numpy/testing/info.py A numpy/numpy/testing/__init__.py A numpy/numpy/testing/setup.py A numpy/numpy/testing/utils.py A numpy/COMPATIBILITY A numpy/DEV_README.txt A numpy/MANIFEST.in A numpy/README.txt A numpy/benchmarks A numpy/benchmarks/sorting.py U numpy Checked out revision 2259. ### Start build. ### /data/basil5/numpy build: No such file or directory Running from numpy source directory. Warning: not existing path in numpy/distutils: site.cfg /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1264: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['blas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1273: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:1276: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack_atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1183: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1207: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:1210: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) No module named __svn_version__ F2PY Version 2_2259 blas_opt_info: blas_mkl_info: NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_info: NOT AVAILABLE blas_src_info: NOT AVAILABLE NOT AVAILABLE lapack_opt_info: lapack_mkl_info: mkl_info: NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: numpy.distutils.system_info.atlas_info NOT AVAILABLE lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/usr/local/lib'] language = f77 NOT AVAILABLE running build running config_fc running build_src building py_modules sources creating build creating build/src creating build/src/numpy creating build/src/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src/numpy/core Generating build/src/numpy/core/config.h customize SunFCompiler compiler version not matched (None) customize GnuFCompiler customize Gnu95FCompiler customize G95FCompiler customize GnuFCompiler customize Gnu95FCompiler customize SunFCompiler Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 69, in setup_package setup( **config.todict() ) File "/data/basil5/numpy/numpy/distutils/core.py", line 85, in setup return old_setup(**new_attr) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 84, in run self.build_sources() File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 99, in build_sources self.build_extension_sources(ext) File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 209, in build_extension_sources sources = self.generate_sources(sources, ext) File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 267, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 35, in generate_config_h library_dirs = default_lib_dirs) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/config.py", line 278, in try_run self._check_compiler() File "/data/basil5/numpy/numpy/distutils/command/config.py", line 35, in _check_compiler self.fcompiler.customize(self.distribution) File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line 281, in customize (conf,'f77flags')) File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line 511, in __get_flags var = command() File "/data/basil5/numpy/numpy/distutils/fcompiler/sun.py", line 27, in get_flags_f77 if (self.get_version() or '') >= '7': File "/data/basil5/numpy/numpy/distutils/ccompiler.py", line 256, in CCompiler_get_version raise ValueError("compiler version not matched (%r)" % (version,)) ValueError: compiler version not matched (None) From tim.hochberg at cox.net Mon Mar 20 07:58:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 20 07:58:18 2006 Subject: [Numpy-discussion] complex comparisons Message-ID: <441ED0EC.8030207@cox.net> I just noticed that complex arrays can be compared using [<, <=, > and >=]. Do we really want this? I'd prefer to follow numarray's lead here and only support [==, !=] on complex arrays. -tim From pfdubois at gmail.com Mon Mar 20 08:04:07 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Mon Mar 20 08:04:07 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441ED0EC.8030207@cox.net> References: <441ED0EC.8030207@cox.net> Message-ID: Plus ca change, plus c'est la meme chose. How many times have we discussed this? I'm with Tim. In fact of course using even the equality operators is often a mistake, but at least there you can think of a use case. On 20 Mar 2006 07:58:54 -0800, Tim Hochberg wrote: > > > I just noticed that complex arrays can be compared using [<, <=, > and > >=]. Do we really want this? I'd prefer to follow numarray's lead here > and only support [==, !=] on complex arrays. > > -tim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schofield at ftw.at Mon Mar 20 08:26:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Mon Mar 20 08:26:02 2006 Subject: [Numpy-discussion] NumPy 0.9.6-r1? Message-ID: <441ED79C.8010001@ftw.at> Hi Travis, I've re-built NumPy 0.9.6 for Win32 / Py2.4 after disabling the block in testing/utils.py that requires win32all. Let me know if you'd like me to upload it to the Wiki as NumPy 0.9.6-r1. It's 66kb smaller than your .exe file, which I don't quite understand, but otherwise seems to work fine. I've built it against ATLAS-3.6.0-PII from http://old.scipy.org. -- Ed From perry at stsci.edu Mon Mar 20 09:03:11 2006 From: perry at stsci.edu (Perry Greenfield) Date: Mon Mar 20 09:03:11 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: References: <441ED0EC.8030207@cox.net> Message-ID: Let me throw my 2 cents in as well. As Paul mentions this has been discussed before. It seems nonsensical to allow these comparisons since there is bound to be some confusion about their meaning no matter what is chosen. And if someone would like their code to be generic with regard to complex numbers and regular floats, then they should use .real, .imag, and abs() explicitly to be precise about what they are comparing. All these will work with both kinds of types. Perry On Mar 20, 2006, at 11:03 AM, Paul Dubois wrote: > Plus ca change, plus c'est la meme chose. How many times have we > discussed this? I'm with Tim. In fact of course using even the > equality operators is often a mistake, but at least there you can > think of a use case. > > On 20 Mar 2006 07:58:54 -0800, Tim Hochberg > wrote: >> I just noticed that complex arrays can be compared using [<, <=, > and >> >=]. Do we really want??this? I'd prefer to follow numarray's lead >> here >> and only support [==, !=] on complex arrays. >> From oliphant.travis at ieee.org Mon Mar 20 10:12:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 10:12:02 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441ED0EC.8030207@cox.net> References: <441ED0EC.8030207@cox.net> Message-ID: <441EF024.2000602@ieee.org> Tim Hochberg wrote: > > I just noticed that complex arrays can be compared using [<, <=, > and > >=]. Do we really want this? I'd prefer to follow numarray's lead > here and only support [==, !=] on complex arrays. > MATLAB has always provided complex comparisons by comparing the real part, so there is signifcant precedence for this. Anybody coming from MATLAB will appreciate it because while technically complex numbers are not ordered, often complex arrays are the result of calculations that result in real numbers but the data-type is still complex. But, now that .real and .imag work on *all* array data-types it is not an absolute necessity to support complex comparisons. I just think it's convenient and not especially confusing. -Travis From Fernando.Perez at colorado.edu Mon Mar 20 10:15:27 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 20 10:15:27 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441EF024.2000602@ieee.org> References: <441ED0EC.8030207@cox.net> <441EF024.2000602@ieee.org> Message-ID: <441EF0F2.8070009@colorado.edu> Travis Oliphant wrote: > Tim Hochberg wrote: > >>I just noticed that complex arrays can be compared using [<, <=, > and >> >>>=]. Do we really want this? I'd prefer to follow numarray's lead >> >>here and only support [==, !=] on complex arrays. >> > > MATLAB has always provided complex comparisons by comparing the real > part, so there is signifcant precedence for this. > > Anybody coming from MATLAB will appreciate it because while technically > complex numbers are not ordered, often complex arrays are the result of > calculations that result in real numbers but the data-type is still > complex. > > But, now that .real and .imag work on *all* array data-types it is not > an absolute necessity to support complex comparisons. I just think it's > convenient and not especially confusing. Well, I think this is one of those cases where we should strive more for uniformity with /Python/ than with matlab: In [18]: 1j<2j --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/fperez/ TypeError: cannot compare complex numbers using <, <=, >, >= Just my 1e-2j :) f From tim.hochberg at cox.net Mon Mar 20 10:24:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 20 10:24:02 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441EF024.2000602@ieee.org> References: <441ED0EC.8030207@cox.net> <441EF024.2000602@ieee.org> Message-ID: <441EF31C.1000307@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> I just noticed that complex arrays can be compared using [<, <=, > >> and >=]. Do we really want this? I'd prefer to follow numarray's >> lead here and only support [==, !=] on complex arrays. >> > MATLAB has always provided complex comparisons by comparing the real > part, so there is signifcant precedence for this. > Anybody coming from MATLAB will appreciate it because while > technically complex numbers are not ordered, often complex arrays are > the result of calculations that result in real numbers but the > data-type is still complex. > But, now that .real and .imag work on *all* array data-types it is not > an absolute necessity to support complex comparisons. I just think > it's convenient and not especially confusing. FWIW, it confused me. My first guess was that it was a bug. My second guess that it was comparing complex numbers by their magnitude. It was only after I dug into the code that I realized that it was comparing real, then imaginary. (I suppose I could have tried to look in the docs too, but it somehow seemed easier just to look in umathmodule). Although I use complex numbers all the time, I'm not going to use this, so in my case it's just a chance for errors to sneak into my code that would otherwise be caught. Much as I love MATLAB people, heck some of my best friends are MATLAB people, I don't think this is a place we should be catering to them. My $0.02. -tim From mahs at telcopartners.com Mon Mar 20 12:21:07 2006 From: mahs at telcopartners.com (Michael Spencer) Date: Mon Mar 20 12:21:07 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <441CE4DE.6040101@ieee.org> References: <200603171519.35502.luszczek@cs.utk.edu> <200603171630.27075.luszczek@cs.utk.edu> <441CE4DE.6040101@ieee.org> Message-ID: Travis Oliphant wrote: > Piotr Luszczek wrote: > > The reason behind this as Tim and Rick later illustrate is that no one > quite knows how to produce "view" semantics for an arbitrarily indexed > array. I did actually think about this for a while because I don't > completely like the different behaviors which can produce warts like the > one that Tim showed us. > I realized that it would be possible to extend our memory model of an > array to include an index array in the array structure itself that would > indicate how to advance through memory (basically a pointer or its > equivalent for each array element). Rick showed that because this > would have to be a "copy" of the indices and so would create too much > slowness. My reason for not going there was because it seemed to add > too much complication for my tastes at the time to all the code. > FWIW, I implemented a pure-python ndarray, independent of numpy but with a similar interface. pyarray preserves views for arbitrary indexings, by carrying around an index array as you suggest. The reason for mentioning this here, is that looking at it might help someone think about desired semantics of views. pyarray: http://svn.brownspencer.com/pyarray/trunk/ Michael From seb at ist.utl.pt Mon Mar 20 12:47:02 2006 From: seb at ist.utl.pt (Sebastien Fabbro) Date: Mon Mar 20 12:47:02 2006 Subject: [Numpy-discussion] gentoo numpy/scipy Message-ID: <441F13CF.5070009@ist.utl.pt> Hello all, For those interested, I try to maintain recent versions of numpy, scipy and some other science related python packages for Gentoo Linux in http://gentooscience.org. All of these packages will end up in the standard Gentoo tree at some point. They are experimental to test recent features or can't wait for gentoo stabilization process. On a related subject, the numpy/scipy building process could really benefit some up-to-date, documentation package maintainer friendly, e.g. a site.cfg commented example. Regards, Sebastien From chanley at stsci.edu Mon Mar 20 13:50:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 20 13:50:02 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: <441EB6E8.8030504@stsci.edu> References: <441EB6E8.8030504@stsci.edu> Message-ID: <441F2349.1050006@stsci.edu> To provide a little more information, I do not use a custom site.cfg file for this platform. All files are deleted prior to the checkout from subversion so that this is a completely clean install. The compiler I am using is the native Sun "CC" compiler from SUNWspro-6u2. Chris Christopher Hanley wrote: > Solaris builds began failing yesterday with the following output: > > Your "cron" job on basil.stsci.edu > /home/chanley/bin/numpy_update.csh > > produced the following output: > > Sun Mar 19 07:20:04 EST 2006 > ### NUMPY Daily Build / Build Test ### > > > ### Setup test environment. ### > > /usr/ra/pyssg/Python-2.4.2/bin/python > > ### Checkout, build, and install new version of numpy. ### > > /data/basil5 > A numpy/LICENSE.txt > A numpy/scipy_compatibility > A numpy/TODO.txt > A numpy/THANKS.txt > A numpy/TEST_COMMIT > A numpy/setup.py > A numpy/numpy > A numpy/numpy/_import_tools.py > A numpy/numpy/f2py > A numpy/numpy/f2py/f2py.1 > A numpy/numpy/f2py/diagnose.py > A numpy/numpy/f2py/rules.py > A numpy/numpy/f2py/info.py > A numpy/numpy/f2py/capi_maps.py > A numpy/numpy/f2py/auxfuncs.py > A numpy/numpy/f2py/TODO.txt > A numpy/numpy/f2py/cb_rules.py > A numpy/numpy/f2py/__init__.py > A numpy/numpy/f2py/setup.py > A numpy/numpy/f2py/src > A numpy/numpy/f2py/src/test > A numpy/numpy/f2py/src/test/foo.f > A numpy/numpy/f2py/src/test/bar.f > A numpy/numpy/f2py/src/test/foo90.f90 > A numpy/numpy/f2py/src/test/foomodule.c > A numpy/numpy/f2py/src/test/Makefile > A numpy/numpy/f2py/src/test/wrap.f > A numpy/numpy/f2py/src/fortranobject.c > A numpy/numpy/f2py/src/fortranobject.h > A numpy/numpy/f2py/docs > A numpy/numpy/f2py/docs/HISTORY.txt > A numpy/numpy/f2py/docs/pytest.py > A numpy/numpy/f2py/docs/usersguide > A numpy/numpy/f2py/docs/usersguide/scalar_session.dat > A numpy/numpy/f2py/docs/usersguide/fib2.pyf > A numpy/numpy/f2py/docs/usersguide/callback.f > A numpy/numpy/f2py/docs/usersguide/calculate.f > A numpy/numpy/f2py/docs/usersguide/run_main_session.dat > A numpy/numpy/f2py/docs/usersguide/index.txt > A numpy/numpy/f2py/docs/usersguide/var.pyf > A numpy/numpy/f2py/docs/usersguide/string.f > A numpy/numpy/f2py/docs/usersguide/spam.pyf > A numpy/numpy/f2py/docs/usersguide/ftype_session.dat > A numpy/numpy/f2py/docs/usersguide/extcallback.f > A numpy/numpy/f2py/docs/usersguide/setup_example.py > A numpy/numpy/f2py/docs/usersguide/array.f > A numpy/numpy/f2py/docs/usersguide/common_session.dat > A numpy/numpy/f2py/docs/usersguide/var_session.dat > A numpy/numpy/f2py/docs/usersguide/moddata.f90 > A numpy/numpy/f2py/docs/usersguide/callback_session.dat > A numpy/numpy/f2py/docs/usersguide/calculate_session.dat > A numpy/numpy/f2py/docs/usersguide/spam_session.dat > A numpy/numpy/f2py/docs/usersguide/docutils.conf > A numpy/numpy/f2py/docs/usersguide/callback2.pyf > A numpy/numpy/f2py/docs/usersguide/string_session.dat > A numpy/numpy/f2py/docs/usersguide/moddata_session.dat > A numpy/numpy/f2py/docs/usersguide/extcallback_session.dat > A numpy/numpy/f2py/docs/usersguide/array_session.dat > A numpy/numpy/f2py/docs/usersguide/scalar.f > A numpy/numpy/f2py/docs/usersguide/allocarr.f90 > A numpy/numpy/f2py/docs/usersguide/compile_session.dat > A numpy/numpy/f2py/docs/usersguide/fib1.f > A numpy/numpy/f2py/docs/usersguide/fib3.f > A numpy/numpy/f2py/docs/usersguide/allocarr_session.dat > A numpy/numpy/f2py/docs/usersguide/ftype.f > A numpy/numpy/f2py/docs/usersguide/default.css > A numpy/numpy/f2py/docs/usersguide/common.f > A numpy/numpy/f2py/docs/usersguide/fib1.pyf > A numpy/numpy/f2py/docs/pyforttest.pyf > A numpy/numpy/f2py/docs/simple_session.dat > A numpy/numpy/f2py/docs/FAQ.txt > A numpy/numpy/f2py/docs/THANKS.txt > A numpy/numpy/f2py/docs/hello.f > A numpy/numpy/f2py/docs/OLDNEWS.txt > A numpy/numpy/f2py/docs/docutils.conf > A numpy/numpy/f2py/docs/README.txt > A numpy/numpy/f2py/docs/TESTING.txt > A numpy/numpy/f2py/docs/default.css > A numpy/numpy/f2py/docs/simple.f > A numpy/numpy/f2py/common_rules.py > A numpy/numpy/f2py/NEWS.txt > A numpy/numpy/f2py/use_rules.py > A numpy/numpy/f2py/f2py2e.py > A numpy/numpy/f2py/setup.cfg > A numpy/numpy/f2py/f90mod_rules.py > A numpy/numpy/f2py/func2subr.py > A numpy/numpy/f2py/tests > A numpy/numpy/f2py/tests/c > A numpy/numpy/f2py/tests/c/return_real.py > A numpy/numpy/f2py/tests/f77 > A numpy/numpy/f2py/tests/f77/return_logical.py > A numpy/numpy/f2py/tests/f77/return_character.py > A numpy/numpy/f2py/tests/f77/callback.py > A numpy/numpy/f2py/tests/f77/return_integer.py > A numpy/numpy/f2py/tests/f77/return_real.py > A numpy/numpy/f2py/tests/f77/return_complex.py > A numpy/numpy/f2py/tests/run_all.py > A numpy/numpy/f2py/tests/array_from_pyobj > A numpy/numpy/f2py/tests/array_from_pyobj/wrapmodule.c > A numpy/numpy/f2py/tests/array_from_pyobj/tests > A numpy/numpy/f2py/tests/array_from_pyobj/tests/test_array_from_pyobj.py > A numpy/numpy/f2py/tests/array_from_pyobj/__init__.py > A numpy/numpy/f2py/tests/array_from_pyobj/setup.py > A numpy/numpy/f2py/tests/mixed > A numpy/numpy/f2py/tests/mixed/foo_fixed.f90 > A numpy/numpy/f2py/tests/mixed/foo_free.f90 > A numpy/numpy/f2py/tests/mixed/foo.f > A numpy/numpy/f2py/tests/mixed/run.py > A numpy/numpy/f2py/tests/f90 > A numpy/numpy/f2py/tests/f90/return_logical.py > A numpy/numpy/f2py/tests/f90/return_character.py > A numpy/numpy/f2py/tests/f90/return_integer.py > A numpy/numpy/f2py/tests/f90/return_real.py > A numpy/numpy/f2py/tests/f90/return_complex.py > A numpy/numpy/f2py/f2py_testing.py > A numpy/numpy/f2py/doc > A numpy/numpy/f2py/doc/notes.tex > A numpy/numpy/f2py/doc/win32_notes.txt > A numpy/numpy/f2py/doc/multiarray > A numpy/numpy/f2py/doc/multiarray/run.pyf > A numpy/numpy/f2py/doc/multiarray/array_from_pyobj.c > A numpy/numpy/f2py/doc/multiarray/bar.c > A numpy/numpy/f2py/doc/multiarray/fun.pyf > A numpy/numpy/f2py/doc/multiarray/foo.f > A numpy/numpy/f2py/doc/multiarray/transpose.txt > A numpy/numpy/f2py/doc/multiarray/fortran_array_from_pyobj.txt > A numpy/numpy/f2py/doc/f2python9-final > A numpy/numpy/f2py/doc/f2python9-final/structure.jpg > A numpy/numpy/f2py/doc/f2python9-final/mk_ps.sh > A numpy/numpy/f2py/doc/f2python9-final/mk_html.sh > A numpy/numpy/f2py/doc/f2python9-final/aerostructure.jpg > A numpy/numpy/f2py/doc/f2python9-final/flow.jpg > A numpy/numpy/f2py/doc/f2python9-final/src > A numpy/numpy/f2py/doc/f2python9-final/src/examples > A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1session.txt > A numpy/numpy/f2py/doc/f2python9-final/src/examples/foo.pyf > A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1.f > A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1mess.txt > A numpy/numpy/f2py/doc/f2python9-final/src/examples/foom.pyf > A numpy/numpy/f2py/doc/f2python9-final/mk_pdf.sh > A numpy/numpy/f2py/doc/f2python9-final/README.txt > A numpy/numpy/f2py/doc/intro.tex > A numpy/numpy/f2py/doc/multiarrays.txt > A numpy/numpy/f2py/doc/options.tex > A numpy/numpy/f2py/doc/collectinput.py > A numpy/numpy/f2py/doc/ex1 > A numpy/numpy/f2py/doc/ex1/foobar.f90 > A numpy/numpy/f2py/doc/ex1/runme > A numpy/numpy/f2py/doc/ex1/foo.f > A numpy/numpy/f2py/doc/ex1/arr.f > A numpy/numpy/f2py/doc/ex1/bar.f > A numpy/numpy/f2py/doc/ex1/foobar-smart.f90 > A numpy/numpy/f2py/doc/ex1/foobarmodule.tex > A numpy/numpy/f2py/doc/bugs.tex > A numpy/numpy/f2py/doc/commands.tex > A numpy/numpy/f2py/doc/signaturefile.tex > A numpy/numpy/f2py/doc/fortranobject.tex > A numpy/numpy/f2py/doc/apps.tex > A numpy/numpy/f2py/doc/Release-1.x.txt > A numpy/numpy/f2py/doc/using_F_compiler.txt > A numpy/numpy/f2py/doc/Release-2.x.txt > A numpy/numpy/f2py/doc/Release-3.x.txt > A numpy/numpy/f2py/doc/Release-4.x.txt > A numpy/numpy/f2py/doc/f2py2e.tex > A numpy/numpy/f2py/doc/python9.tex > A numpy/numpy/f2py/doc/index.html > A numpy/numpy/f2py/doc/Makefile > A numpy/numpy/f2py/doc/oldnews.html > A numpy/numpy/f2py/crackfortran.py > A numpy/numpy/f2py/cfuncs.py > A numpy/numpy/f2py/__version__.py > A numpy/numpy/f2py/README.txt > A numpy/numpy/f2py/Makefile > A numpy/numpy/f2py/BUGS.txt > A numpy/numpy/random > A numpy/numpy/random/info.py > A numpy/numpy/random/mtrand > A numpy/numpy/random/mtrand/Python.pxi > A numpy/numpy/random/mtrand/distributions.c > A numpy/numpy/random/mtrand/initarray.c > A numpy/numpy/random/mtrand/mtrand.pyx > A numpy/numpy/random/mtrand/mtrand.c > A numpy/numpy/random/mtrand/numpy.pxi > A numpy/numpy/random/mtrand/distributions.h > A numpy/numpy/random/mtrand/generate_mtrand_c.py > A numpy/numpy/random/mtrand/initarray.h > A numpy/numpy/random/mtrand/randomkit.c > A numpy/numpy/random/mtrand/randomkit.h > A numpy/numpy/random/__init__.py > A numpy/numpy/random/setup.py > A numpy/numpy/add_newdocs.py > A numpy/numpy/distutils > A numpy/numpy/distutils/core.py > A numpy/numpy/distutils/misc_util.py > A numpy/numpy/distutils/fcompiler > A numpy/numpy/distutils/fcompiler/mips.py > A numpy/numpy/distutils/fcompiler/gnu.py > A numpy/numpy/distutils/fcompiler/intel.py > A numpy/numpy/distutils/fcompiler/vast.py > A numpy/numpy/distutils/fcompiler/absoft.py > A numpy/numpy/distutils/fcompiler/__init__.py > A numpy/numpy/distutils/fcompiler/none.py > A numpy/numpy/distutils/fcompiler/compaq.py > A numpy/numpy/distutils/fcompiler/lahey.py > A numpy/numpy/distutils/fcompiler/g95.py > A numpy/numpy/distutils/fcompiler/hpux.py > A numpy/numpy/distutils/fcompiler/nag.py > A numpy/numpy/distutils/fcompiler/sun.py > A numpy/numpy/distutils/fcompiler/pg.py > A numpy/numpy/distutils/fcompiler/ibm.py > A numpy/numpy/distutils/info.py > A numpy/numpy/distutils/line_endings.py > A numpy/numpy/distutils/from_template.py > A numpy/numpy/distutils/__init__.py > A numpy/numpy/distutils/system_info.py > A numpy/numpy/distutils/conv_template.py > A numpy/numpy/distutils/setup.py > A numpy/numpy/distutils/cpuinfo.py > A numpy/numpy/distutils/lib2def.py > A numpy/numpy/distutils/intelccompiler.py > A numpy/numpy/distutils/tests > A numpy/numpy/distutils/tests/f2py_ext > A numpy/numpy/distutils/tests/f2py_ext/tests > A numpy/numpy/distutils/tests/f2py_ext/tests/test_fib2.py > A numpy/numpy/distutils/tests/f2py_ext/__init__.py > A numpy/numpy/distutils/tests/f2py_ext/setup.py > A numpy/numpy/distutils/tests/f2py_ext/src > A numpy/numpy/distutils/tests/f2py_ext/src/fib2.pyf > A numpy/numpy/distutils/tests/f2py_ext/src/fib1.f > A numpy/numpy/distutils/tests/pyrex_ext > A numpy/numpy/distutils/tests/pyrex_ext/tests > A numpy/numpy/distutils/tests/pyrex_ext/tests/test_primes.py > A numpy/numpy/distutils/tests/pyrex_ext/__init__.py > A numpy/numpy/distutils/tests/pyrex_ext/setup.py > A numpy/numpy/distutils/tests/pyrex_ext/primes.pyx > A numpy/numpy/distutils/tests/setup.py > A numpy/numpy/distutils/tests/gen_ext > A numpy/numpy/distutils/tests/gen_ext/tests > A numpy/numpy/distutils/tests/gen_ext/tests/test_fib3.py > A numpy/numpy/distutils/tests/gen_ext/__init__.py > A numpy/numpy/distutils/tests/gen_ext/setup.py > A numpy/numpy/distutils/tests/swig_ext > A numpy/numpy/distutils/tests/swig_ext/tests > A numpy/numpy/distutils/tests/swig_ext/tests/test_example.py > A numpy/numpy/distutils/tests/swig_ext/tests/test_example2.py > A numpy/numpy/distutils/tests/swig_ext/__init__.py > A numpy/numpy/distutils/tests/swig_ext/setup.py > A numpy/numpy/distutils/tests/swig_ext/src > A numpy/numpy/distutils/tests/swig_ext/src/example.i > A numpy/numpy/distutils/tests/swig_ext/src/zoo.cc > A numpy/numpy/distutils/tests/swig_ext/src/example.c > A numpy/numpy/distutils/tests/swig_ext/src/zoo.h > A numpy/numpy/distutils/tests/swig_ext/src/zoo.i > A numpy/numpy/distutils/tests/f2py_f90_ext > A numpy/numpy/distutils/tests/f2py_f90_ext/tests > A numpy/numpy/distutils/tests/f2py_f90_ext/tests/test_foo.py > A numpy/numpy/distutils/tests/f2py_f90_ext/include > A numpy/numpy/distutils/tests/f2py_f90_ext/include/body.f90 > A numpy/numpy/distutils/tests/f2py_f90_ext/__init__.py > A numpy/numpy/distutils/tests/f2py_f90_ext/setup.py > A numpy/numpy/distutils/tests/f2py_f90_ext/src > A numpy/numpy/distutils/tests/f2py_f90_ext/src/foo_free.f90 > A numpy/numpy/distutils/tests/test_misc_util.py > A numpy/numpy/distutils/extension.py > A numpy/numpy/distutils/ccompiler.py > A numpy/numpy/distutils/log.py > A numpy/numpy/distutils/__version__.py > A numpy/numpy/distutils/unixccompiler.py > A numpy/numpy/distutils/exec_command.py > A numpy/numpy/distutils/mingw32ccompiler.py > A numpy/numpy/distutils/command > A numpy/numpy/distutils/command/build_clib.py > A numpy/numpy/distutils/command/egg_info.py > A numpy/numpy/distutils/command/__init__.py > A numpy/numpy/distutils/command/build.py > A numpy/numpy/distutils/command/build_ext.py > A numpy/numpy/distutils/command/config_compiler.py > A numpy/numpy/distutils/command/install_data.py > A numpy/numpy/distutils/command/install_headers.py > A numpy/numpy/distutils/command/bdist_rpm.py > A numpy/numpy/distutils/command/config.py > A numpy/numpy/distutils/command/build_scripts.py > A numpy/numpy/distutils/command/build_src.py > A numpy/numpy/distutils/command/install.py > A numpy/numpy/distutils/command/sdist.py > A numpy/numpy/distutils/command/build_py.py > A numpy/numpy/doc > A numpy/numpy/doc/records.txt > A numpy/numpy/doc/ufuncs.txt > A numpy/numpy/doc/DISTUTILS.txt > A numpy/numpy/doc/pyrex > A numpy/numpy/doc/pyrex/numpyx.pyx > A numpy/numpy/doc/pyrex/numpyx.c > A numpy/numpy/doc/pyrex/c_numpy.pxd > A numpy/numpy/doc/pyrex/MANIFEST > A numpy/numpy/doc/pyrex/setup.py > A numpy/numpy/doc/pyrex/notes > A numpy/numpy/doc/pyrex/run_test.py > A numpy/numpy/doc/pyrex/Makefile > A numpy/numpy/doc/pyrex/c_python.pxd > A numpy/numpy/doc/swig > A numpy/numpy/doc/swig/numpy.i > A numpy/numpy/doc/swig/series.h > A numpy/numpy/doc/swig/Series.i > A numpy/numpy/doc/swig/testSeries.py > A numpy/numpy/doc/swig/HelperFunctions.txt > A numpy/numpy/doc/swig/setup.py > A numpy/numpy/doc/swig/series.cxx > A numpy/numpy/doc/swig/Makefile > A numpy/numpy/doc/swig/README > A numpy/numpy/doc/CAPI.txt > A numpy/numpy/doc/README.txt > A numpy/numpy/__init__.py > A numpy/numpy/lib > A numpy/numpy/lib/scimath.py > A numpy/numpy/lib/shape_base.py > A numpy/numpy/lib/machar.py > A numpy/numpy/lib/info.py > A numpy/numpy/lib/twodim_base.py > A numpy/numpy/lib/__init__.py > A numpy/numpy/lib/src > A numpy/numpy/lib/src/_compiled_base.c > A numpy/numpy/lib/setup.py > A numpy/numpy/lib/utils.py > A numpy/numpy/lib/getlimits.py > A numpy/numpy/lib/tests > A numpy/numpy/lib/tests/test_shape_base.py > A numpy/numpy/lib/tests/test_arraysetops.py > A numpy/numpy/lib/tests/test_twodim_base.py > A numpy/numpy/lib/tests/test_type_check.py > A numpy/numpy/lib/tests/test_function_base.py > A numpy/numpy/lib/tests/test_getlimits.py > A numpy/numpy/lib/tests/test_polynomial.py > A numpy/numpy/lib/tests/test_ufunclike.py > A numpy/numpy/lib/tests/test_index_tricks.py > A numpy/numpy/lib/mlab.py > A numpy/numpy/lib/convertcode.py > A numpy/numpy/lib/arraysetops.py > A numpy/numpy/lib/UserArray.py > A numpy/numpy/lib/type_check.py > A numpy/numpy/lib/function_base.py > A numpy/numpy/lib/polynomial.py > A numpy/numpy/lib/ufunclike.py > A numpy/numpy/lib/index_tricks.py > A numpy/numpy/linalg > A numpy/numpy/linalg/blas_lite.c > A numpy/numpy/linalg/lapack_litemodule.c > A numpy/numpy/linalg/f2c.h > A numpy/numpy/linalg/info.py > A numpy/numpy/linalg/zlapack_lite.c > A numpy/numpy/linalg/old.py > A numpy/numpy/linalg/__init__.py > A numpy/numpy/linalg/setup.py > A numpy/numpy/linalg/f2c_lite.c > A numpy/numpy/linalg/dlamch.c > A numpy/numpy/linalg/dlapack_lite.c > A numpy/numpy/linalg/linalg.py > A numpy/numpy/setup.py > A numpy/numpy/core > A numpy/numpy/core/info.py > A numpy/numpy/core/defchararray.py > A numpy/numpy/core/arrayprint.py > A numpy/numpy/core/include > A numpy/numpy/core/include/numpy > A numpy/numpy/core/include/numpy/arrayobject.h > A numpy/numpy/core/include/numpy/arrayscalars.h > A numpy/numpy/core/include/numpy/ufuncobject.h > A numpy/numpy/core/include/numpy/fenv > A numpy/numpy/core/include/numpy/fenv/fenv.c > A numpy/numpy/core/include/numpy/fenv/fenv.h > A numpy/numpy/core/ma.py > A numpy/numpy/core/__init__.py > A numpy/numpy/core/setup.py > A numpy/numpy/core/src > A numpy/numpy/core/src/_signbit.c > A numpy/numpy/core/src/multiarraymodule.c > A numpy/numpy/core/src/arraytypes.inc.src > A numpy/numpy/core/src/_sortmodule.c.src > A numpy/numpy/core/src/arraymethods.c > A numpy/numpy/core/src/ucsnarrow.c > A numpy/numpy/core/src/arrayobject.c > A numpy/numpy/core/src/_isnan.c > A numpy/numpy/core/src/scalartypes.inc.src > A numpy/numpy/core/src/ufuncobject.c > A numpy/numpy/core/src/umathmodule.c.src > A numpy/numpy/core/src/scalarmathmodule.c.src > A numpy/numpy/core/records.py > A numpy/numpy/core/oldnumeric.py > A numpy/numpy/core/blasdot > A numpy/numpy/core/blasdot/_dotblas.c > A numpy/numpy/core/blasdot/cblas.h > A numpy/numpy/core/numeric.py > A numpy/numpy/core/_internal.py > A numpy/numpy/core/tests > A numpy/numpy/core/tests/test_multiarray.py > A numpy/numpy/core/tests/test_ma.py > A numpy/numpy/core/tests/test_umath.py > A numpy/numpy/core/tests/test_oldnumeric.py > A numpy/numpy/core/tests/test_records.py > A numpy/numpy/core/tests/test_numeric.py > A numpy/numpy/core/tests/test_defmatrix.py > A numpy/numpy/core/tests/test_unicode.py > A numpy/numpy/core/tests/test_numerictypes.py > A numpy/numpy/core/tests/testdata.fits > A numpy/numpy/core/memmap.py > A numpy/numpy/core/code_generators > A numpy/numpy/core/code_generators/generate_umath.py > A numpy/numpy/core/code_generators/array_api_order.txt > A numpy/numpy/core/code_generators/ufunc_api_order.txt > A numpy/numpy/core/code_generators/generate_array_api.py > A numpy/numpy/core/code_generators/genapi.py > A numpy/numpy/core/code_generators/generate_ufunc_api.py > A numpy/numpy/core/code_generators/multiarray_api_order.txt > A numpy/numpy/core/defmatrix.py > A numpy/numpy/core/numerictypes.py > A numpy/numpy/dual.py > A numpy/numpy/version.py > A numpy/numpy/dft > A numpy/numpy/dft/fftpack.c > A numpy/numpy/dft/fftpack_litemodule.c > A numpy/numpy/dft/info.py > A numpy/numpy/dft/tests > A numpy/numpy/dft/tests/test_helper.py > A numpy/numpy/dft/fftpack.h > A numpy/numpy/dft/fftpack.py > A numpy/numpy/dft/__init__.py > A numpy/numpy/dft/helper.py > A numpy/numpy/dft/setup.py > A numpy/numpy/testing > A numpy/numpy/testing/numpytest.py > A numpy/numpy/testing/info.py > A numpy/numpy/testing/__init__.py > A numpy/numpy/testing/setup.py > A numpy/numpy/testing/utils.py > A numpy/COMPATIBILITY > A numpy/DEV_README.txt > A numpy/MANIFEST.in > A numpy/README.txt > A numpy/benchmarks > A numpy/benchmarks/sorting.py > U numpy > Checked out revision 2259. > > ### Start build. ### > > /data/basil5/numpy > build: No such file or directory > Running from numpy source directory. > Warning: not existing path in numpy/distutils: site.cfg > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1264: UserWarning: > Atlas (http://math-atlas.sourceforge.net/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [atlas]) or by setting > the ATLAS environment variable. > warnings.warn(AtlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['blas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1273: UserWarning: > Blas (http://www.netlib.org/blas/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [blas]) or by setting > the BLAS environment variable. > warnings.warn(BlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:1276: UserWarning: > Blas (http://www.netlib.org/blas/) sources not found. > Directories to search for the sources can be specified in the > numpy/distutils/site.cfg file (section [blas_src]) or by setting > the BLAS_SRC environment variable. > warnings.warn(BlasSrcNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['lapack_atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1183: UserWarning: > Atlas (http://math-atlas.sourceforge.net/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [atlas]) or by setting > the ATLAS environment variable. > warnings.warn(AtlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['lapack'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1207: UserWarning: > Blas (http://www.netlib.org/blas/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [blas]) or by setting > the BLAS environment variable. > warnings.warn(BlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:1210: UserWarning: > Blas (http://www.netlib.org/blas/) sources not found. > Directories to search for the sources can be specified in the > numpy/distutils/site.cfg file (section [blas_src]) or by setting > the BLAS_SRC environment variable. > warnings.warn(BlasSrcNotFoundError.__doc__) > No module named __svn_version__ > F2PY Version 2_2259 > blas_opt_info: > blas_mkl_info: > NOT AVAILABLE > > atlas_blas_threads_info: > Setting PTATLAS=ATLAS > NOT AVAILABLE > > atlas_blas_info: > NOT AVAILABLE > > blas_info: > NOT AVAILABLE > > blas_src_info: > NOT AVAILABLE > > NOT AVAILABLE > > lapack_opt_info: > lapack_mkl_info: > mkl_info: > NOT AVAILABLE > > NOT AVAILABLE > > atlas_threads_info: > Setting PTATLAS=ATLAS > numpy.distutils.system_info.atlas_threads_info > NOT AVAILABLE > > atlas_info: > numpy.distutils.system_info.atlas_info > NOT AVAILABLE > > lapack_info: > FOUND: > libraries = ['lapack'] > library_dirs = ['/usr/local/lib'] > language = f77 > > NOT AVAILABLE > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src > creating build/src/numpy > creating build/src/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src/numpy/core > Generating build/src/numpy/core/config.h > customize SunFCompiler > compiler version not matched (None) > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize SunFCompiler > Traceback (most recent call last): > File "setup.py", line 76, in ? > setup_package() > File "setup.py", line 69, in setup_package > setup( **config.todict() ) > File "/data/basil5/numpy/numpy/distutils/core.py", line 85, in setup > return old_setup(**new_attr) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/core.py", > line 149, in setup > dist.run_commands() > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 946, in run_commands > self.run_command(cmd) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/build.py", > line 112, in run > self.run_command(cmd_name) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/cmd.py", line > 333, in run_command > self.distribution.run_command(command) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 84, in run > self.build_sources() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 99, in build_sources > self.build_extension_sources(ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 209, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 267, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 35, in generate_config_h > library_dirs = default_lib_dirs) > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/config.py", > line 278, in try_run > self._check_compiler() > File "/data/basil5/numpy/numpy/distutils/command/config.py", line 35, > in _check_compiler > self.fcompiler.customize(self.distribution) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line > 281, in customize > (conf,'f77flags')) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line > 511, in __get_flags > var = command() > File "/data/basil5/numpy/numpy/distutils/fcompiler/sun.py", line 27, > in get_flags_f77 > if (self.get_version() or '') >= '7': > File "/data/basil5/numpy/numpy/distutils/ccompiler.py", line 256, in > CCompiler_get_version > raise ValueError("compiler version not matched (%r)" % (version,)) > ValueError: compiler version not matched (None) > From cookedm at physics.mcmaster.ca Mon Mar 20 14:29:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 20 14:29:06 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: <441EB6E8.8030504@stsci.edu> (Christopher Hanley's message of "Mon, 20 Mar 2006 09:06:32 -0500") References: <441EB6E8.8030504@stsci.edu> Message-ID: Christopher Hanley writes: > Solaris builds began failing yesterday with the following output: Looks like it's not matching the compiler version for your fortran compiler. What does "f90 -V" give you? It's trying to match against r'(f90|f95): (Sun|Forte Developer 7|WorkShop 6 update \d+) Fortran 95 (?P[^\s]+).*' so maybe that needs work. > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src > creating build/src/numpy > creating build/src/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src/numpy/core > Generating build/src/numpy/core/config.h > customize SunFCompiler > compiler version not matched (None) > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize SunFCompiler > Traceback (most recent call last): > File "setup.py", line 76, in ? > setup_package() > File "setup.py", line 69, in setup_package > setup( **config.todict() ) > File "/data/basil5/numpy/numpy/distutils/core.py", line 85, in setup > return old_setup(**new_attr) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/core.py", > line 149, in setup > dist.run_commands() > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 946, in run_commands > self.run_command(cmd) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/build.py", > line 112, in run > self.run_command(cmd_name) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/cmd.py", > line 333, in run_command > self.distribution.run_command(command) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 84, in run > self.build_sources() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 99, in build_sources > self.build_extension_sources(ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 209, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 267, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 35, in generate_config_h > library_dirs = default_lib_dirs) > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/config.py", > line 278, in try_run > self._check_compiler() > File "/data/basil5/numpy/numpy/distutils/command/config.py", line > 35, in _check_compiler > self.fcompiler.customize(self.distribution) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", > line 281, in customize > (conf,'f77flags')) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", > line 511, in __get_flags > var = command() > File "/data/basil5/numpy/numpy/distutils/fcompiler/sun.py", line 27, > in get_flags_f77 > if (self.get_version() or '') >= '7': > File "/data/basil5/numpy/numpy/distutils/ccompiler.py", line 256, in > CCompiler_get_version > raise ValueError("compiler version not matched (%r)" % (version,)) > ValueError: compiler version not matched (None) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Mon Mar 20 14:30:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 20 14:30:03 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441ED0EC.8030207@cox.net> (Tim Hochberg's message of "Mon, 20 Mar 2006 08:57:32 -0700") References: <441ED0EC.8030207@cox.net> Message-ID: Tim Hochberg writes: > I just noticed that complex arrays can be compared using [<, <=, > and >> =]. Do we really want this? I'd prefer to follow numarray's lead >> here > and only support [==, !=] on complex arrays. +1 -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Mon Mar 20 16:16:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 20 16:16:04 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: References: <441EB6E8.8030504@stsci.edu> Message-ID: <441F4591.2050707@ee.byu.edu> David M. Cooke wrote: >Christopher Hanley writes: > > > >>Solaris builds began failing yesterday with the following output: >> >> > >Looks like it's not matching the compiler version for your fortran >compiler. What does "f90 -V" give you? > > This should fail more gracefully, especially since a fortran compiler is not even needed for numpy to get installed. This needs to be fixed in NumPy. -Travis From zpincus at stanford.edu Mon Mar 20 16:37:02 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 20 16:37:02 2006 Subject: [Numpy-discussion] linalg.svd returns fortran arrays, causes problems Message-ID: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Hi folks, I recently updated to the SVN head of numpy, and most of my data analysis code broke! Specifically, this code involved computing the SVD of flattened multidimensional data, and then reshaping the results back to the correct dimensionality. (A very common operation.) Now these reshape operations don't generate the right results. It turns out that this is due to either (a) linalg.svd is returning fortran-style arrays when it used to return c-style arrays, or (b) numpy.reshape used to convert fortran-style arrays to c-style before reshaping, and now does not. It seems to me that (a) is more likely. If so, is this a bug or a feature? I personally would argue that returning fortran-style arrays from any functions, unless specifically requested, is a serious bug, precisely because of problems like the above. Any thoughts? Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine From robert.kern at gmail.com Mon Mar 20 16:47:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 16:47:02 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: Zachary Pincus wrote: > Hi folks, > > I recently updated to the SVN head of numpy, and most of my data > analysis code broke! Specifically, this code involved computing the SVD > of flattened multidimensional data, and then reshaping the results back > to the correct dimensionality. (A very common operation.) Now these > reshape operations don't generate the right results. Can you give us some small, self-contained code that demonstrates the problem? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Mar 20 16:52:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 16:52:03 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: Zachary Pincus wrote: > Hi folks, > > I recently updated to the SVN head of numpy, and most of my data > analysis code broke! And what version of numpy *did* work for you? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Mon Mar 20 17:15:26 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 20 17:15:26 2006 Subject: [Numpy-discussion] linalg.svd returns fortran arrays, causes problems In-Reply-To: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F537D.1070705@ee.byu.edu> Zachary Pincus wrote: > Hi folks, > > I recently updated to the SVN head of numpy, and most of my data > analysis code broke! Specifically, this code involved computing the > SVD of flattened multidimensional data, and then reshaping the > results back to the correct dimensionality. (A very common > operation.) Now these reshape operations don't generate the right > results. > The Python user should not care whether the arrays are fortran-order or c-order (it's just a matter of different striding). If it does matter, then it is a bug that must be fixed. Please show us code that is not working. If you upgraded to SVN numpy, then it's more likely to be related to the change to the flags object (now it's a builtin). -Travis From zpincus at stanford.edu Mon Mar 20 17:18:15 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 20 17:18:15 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: > Can you give us some small, self-contained code that demonstrates > the problem? u, s, vt = numpy.linalg.svd([[1,2],[3,4]]) u.flags.fortran True The problem, as per my original email, is that linalg.svd seems to be now returning fortran-style arrays. I believe that it did not do this in the past, but I am not certain. It is also possible that linalg.svd always returned fortran arrays, and the behavior of numpy.reshape with regard to fortran arrays has changed. Here is an example of how numpy.reshape differs with fortran and c-style arrays. I do not know if this is a bug, or the desired behavior. numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = False), [2, 5, 2]) array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = True), [2, 5, 2]) array([[[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]], [[11, 16], [12, 17], [13, 18], [14, 19], [15, 20]]]) In any case, there are two issues at play: (1) Should any numpy functions be allowed to return fortran arrays unless specifically requested? (2) Should all structural operations like reshape (etc.) behave the same for fortran and c-style arrays? If 2 can be guaranteed, then (1) doesn't really matter, I think. But I'm not sure if 2 should be guaranteed or not. > And what version of numpy *did* work for you? I do not recall, unfortunately. It was a svn checkout not more than two weeks old. Again, what I'm reporting isn't *necessarily* a bug, but at least one (if not two) changes in semantics which are at least problematic, and at most a bug. Zach On Mar 20, 2006, at 4:45 PM, Robert Kern wrote: > Zachary Pincus wrote: >> Hi folks, >> >> I recently updated to the SVN head of numpy, and most of my data >> analysis code broke! Specifically, this code involved computing >> the SVD >> of flattened multidimensional data, and then reshaping the >> results back >> to the correct dimensionality. (A very common operation.) Now these >> reshape operations don't generate the right results. > > Can you give us some small, self-contained code that demonstrates > the problem? > > -- > Robert Kern > robert.kern at gmail.com > > "I have come to believe that the whole world is an enigma, a > harmless enigma > that is made terrible by our own mad attempt to interpret it as > though it had > an underlying truth." > -- Umberto Eco > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From robert.kern at gmail.com Mon Mar 20 17:26:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 17:26:07 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: Zachary Pincus wrote: > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = False), [2, 5, 2]) > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = True), [2, 5, 2]) > array([[[ 1, 6], > [ 2, 7], > [ 3, 8], > [ 4, 9], > [ 5, 10]], > > [[11, 16], > [12, 17], > [13, 18], > [14, 19], > [15, 20]]]) I think this is a new bug, then. In [82]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = True), (2,5,2)) Out[82]: array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) In [83]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = False), (2,5,2)) Out[83]: array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) In [84]: numpy.__version__ Out[84]: '0.9.6.2148' -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From chanley at stsci.edu Mon Mar 20 17:35:19 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 20 17:35:19 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 Message-ID: <20060320203450.CIP50631@comet.stsci.edu> In answer to your question, f90 -V returns: basil> f90 -V f90: Sun WorkShop 6 update 2 Fortran 95 6.2 Patch 111690-10 2003/08/28 Usage: f90 [ options ] files. Use 'f90 -flags' for details However, I agree with Travis. Lack of a FORTRAN compiler should not cause the build to fail when it is not required. Thank you for your time and help. Chris >>Looks like it's not matching the compiler version for your fortran >>compiler. What does "f90 -V" give you? >> >> > >This should fail more gracefully, especially since a fortran compiler is >not even needed for numpy to get installed. This needs to be fixed in >NumPy. > >-Travis > From oliphant.travis at ieee.org Mon Mar 20 19:51:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 19:51:05 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F780F.5090602@ieee.org> Robert Kern wrote: > I think this is a new bug, then. > > In [82]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = True), (2,5,2)) > Out[82]: > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > In [83]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = False), (2,5,2)) > Out[83]: > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > Robert, can you test numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], fortran = False).reshape(2,5,2) numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], fortran = False).reshape(2,5,2) please? The reshape method has not changed since your version, but the reshape wrapper did change slightly, I believe. In a manner that could have caused this difference. -Travis From oliphant.travis at ieee.org Mon Mar 20 20:12:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 20:12:02 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F7CD9.9070800@ieee.org> Zachary Pincus wrote: >> Can you give us some small, self-contained code that demonstrates the >> problem? > > u, s, vt = numpy.linalg.svd([[1,2],[3,4]]) > u.flags.fortran > True > > The problem, as per my original email, is that linalg.svd seems to be > now returning fortran-style arrays. I believe that it did not do this > in the past, but I am not certain. I think it's always done this. It is really very easy to generate fortran-order arrays (always has been --- the transpose operation has always returned a fortran-order array). a = numpy.rand(10,20).transpose() a.flags.fortran I think the issue that has raised this discussion is that the reshape function is now not running asarray if it doesn't have to --- this was done to preserve matrices... Notice that currently the asarray function will convert Fortran-order arrays to C-order arrays (and thus make a copy). Now that I'm aware it's doing this, I actually think that's a bug. This is because Fortran order is really just a special case of striding, and other non-uniformly strided arrays are not automatically converted just because asarray is called. So, I think as a side-issue that bug should be fixed. Now, the real issue that brought this to the fore-front is what is meant by the "reshape" function. It has been the case for several months that a fortran-order array has a different scan order for it's reshape *method* than any-other strided array. The reshape method is one of those functions that you can't avoid thinking about layout of the array (so my previous statement about not concerning yourself with striding was too far-reaching). We are supporting more than one view about how layout should be done. But, we should be doing it in a backward-compatible fashion so that the standard is C-contiguous unless otherwise specified. Thus, I think the bug is in the design of the reshape method in that it uses a different layout standard depending on whether or not the array is Fortran or not. This bug has been there for awhile, I think, but only exposed by the re-working of the reshape function. I propose to change the reshape method to always expect a C-contiguous perspective unless a fortran flag is set (defaults to False), in which case the perspective is Fortran-contiguous. This is true regardless of the layout of the current array. This will fix the reshape function as well but also make it possible for someone to use reshape who has a Fortran-order array-view. > > > In any case, there are two issues at play: > (1) Should any numpy functions be allowed to return fortran arrays > unless specifically requested? > (2) Should all structural operations like reshape (etc.) behave the > same for fortran and c-style arrays? > The issue is that there are only a few shape-changing functions (ravel, reshape, etc.), that the Python user needs to have any perspective at all on how the data is layed out in memory. Other routines should not depend on the striding (or if they do they need to ensure it's contiguous first). For those routines that do require memory-layout knowledge the default must continue to be C-contiguous, but we need to add a fortran keyword to allow that perspective as well should the user choose it. This is already done for the ravel method. And it should be available for reshape (and resize---though that one will take more effort and I will not do tonight---as well). Thanks for the bug-reports. -Travis From robert.kern at gmail.com Mon Mar 20 20:13:22 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 20:13:22 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <441F780F.5090602@ieee.org> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> <441F780F.5090602@ieee.org> Message-ID: Travis Oliphant wrote: > Robert Kern wrote: > >> I think this is a new bug, then. >> >> In [82]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], >> [11,12,13,14,15,16,17,18,19,20]], fortran = True), (2,5,2)) >> Out[82]: >> array([[[ 1, 2], >> [ 3, 4], >> [ 5, 6], >> [ 7, 8], >> [ 9, 10]], >> >> [[11, 12], >> [13, 14], >> [15, 16], >> [17, 18], >> [19, 20]]]) >> >> In [83]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], >> [11,12,13,14,15,16,17,18,19,20]], fortran = False), (2,5,2)) >> Out[83]: >> array([[[ 1, 2], >> [ 3, 4], >> [ 5, 6], >> [ 7, 8], >> [ 9, 10]], >> >> [[11, 12], >> [13, 14], >> [15, 16], >> [17, 18], >> [19, 20]]]) > > Robert, can you test > > numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = False).reshape(2,5,2) > > numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = False).reshape(2,5,2) > > please? > > The reshape method has not changed since your version, but the reshape > wrapper did change slightly, I believe. In a manner that could have > caused this difference. Huh. *That* displays the bug. In [85]: numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran=False).reshape(2,5,2) Out[85]: array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) In [86]: numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran=True).reshape(2,5,2) Out[86]: array([[[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]], [[11, 16], [12, 17], [13, 18], [14, 19], [15, 20]]]) -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Mon Mar 20 20:15:17 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 20 20:15:17 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: <20060320203450.CIP50631@comet.stsci.edu> (Christopher Hanley's message of "Mon, 20 Mar 2006 20:34:50 -0500 (EST)") References: <20060320203450.CIP50631@comet.stsci.edu> Message-ID: Christopher Hanley writes: > In answer to your question, f90 -V returns: > > basil> f90 -V > f90: Sun WorkShop 6 update 2 Fortran 95 6.2 Patch 111690-10 2003/08/28 > Usage: f90 [ options ] files. Use 'f90 -flags' for details Ok, looks like the version-matching regexp was wrong. I've replaced it with a fuzzier match. The previous version wouldn't have triggered the error you saw, but it would instead return None from .get_version(). > However, I agree with Travis. Lack of a FORTRAN compiler should not > cause the build to fail when it is not required. I've replaced the thrown error with a log.warn. Fixed in svn. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant.travis at ieee.org Mon Mar 20 21:25:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 21:25:03 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F8E0C.4000901@ieee.org> Zachary Pincus wrote: >> Can you give us some small, self-contained code that demonstrates the >> problem? > > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = False), [2, 5, 2]) > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = True), [2, 5, 2]) > array([[[ 1, 6], > [ 2, 7], > [ 3, 8], > [ 4, 9], > [ 5, 10]], > > [[11, 16], > [12, 17], > [13, 18], > [14, 19], > [15, 20]]]) > > This should be fixed now in SVN. Thanks for pointing it out. -Travis From zpincus at stanford.edu Mon Mar 20 21:53:01 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 20 21:53:01 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <441F7CD9.9070800@ieee.org> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> <441F7CD9.9070800@ieee.org> Message-ID: <7D7758DF-7744-4BBB-8AB9-F3D90A1E8C50@stanford.edu> > I propose to change the reshape method to always expect a C- > contiguous perspective unless a fortran flag is set (defaults to > False), in which case the perspective is Fortran-contiguous. This > is true regardless of the layout of the current array. This seems like an optimal solution -- thanks for looking into all of this and getting the fix in the svn so quickly! Zach From dd55 at cornell.edu Tue Mar 21 08:27:34 2006 From: dd55 at cornell.edu (Darren Dale) Date: Tue Mar 21 08:27:34 2006 Subject: [Numpy-discussion] constructing char arrays with "where" Message-ID: <200603211101.00155.dd55@cornell.edu> There is some disparity between the numpy, Numeric and numarray flavors of the "where" function, as concerns creation of char arrays. I got bit by the limitation in numarray. I found a workaround for my matplotlib code that didnt use "where", but perhaps it is worth mentioning. Darren ################################################# import Numeric, numpy, numarray In [1]: numpy.where([True, False], 'T', 'F') Out[1]: array([T, F], dtype='|S1') dtype='|S1') In [2]: numpy.where([True, False], 'True', 'False') Out[2]: array([True, False], dtype='|S5') ################################################# In [3]: Numeric.where([True, False], 'T', 'F') Out[3]: array([T, F],'c') In [4]: Numeric.where([True, False], 'True', 'False') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/darren/src/matplotlib/ /usr/lib64/python2.4/site-packages/Numeric/Numeric.py in where(condition, x, y) 603 y where condition is respectively true or false. 604 """ --> 605 return choose(not_equal(condition, 0), (y, x)) 606 607 def compress(condition, m, axis=-1): ValueError: array dimensions must agree ################################################ In [23]: numarray.where([True, False], 'T', 'F') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/darren/src/matplotlib/ /usr/lib64/python2.4/site-packages/numarray/generic.py in where(condition, x, y, out) 1066 if x is None or y is None: 1067 raise ValueError("Invalid parameter list") -> 1068 return choose(ufunc.not_equal(condition, 0), (y,x), out) 1069 1070 def clip(m, m_min, m_max): /usr/lib64/python2.4/site-packages/numarray/ufunc.py in choose(selector, population, outarr, clipmode) 1988 [0, shape[i]) result in an exception. 1989 """ -> 1990 return _choose(selector, population, outarr, clipmode) 1991 1992 def _scatteredPseudos( scattered ): /usr/lib64/python2.4/site-packages/numarray/ufunc.py in __call__(self, inarr1, inarr2, outarr, clipmode) 1894 def __call__(self, inarr1, inarr2, outarr=None, clipmode=RAISE): 1895 """The standard calling interface for UFuncs""" -> 1896 computation_mode, woutarr, cfunc, ufargs = \ 1897 self._setup(inarr1, inarr2, outarr, clipmode) 1898 /usr/lib64/python2.4/site-packages/numarray/ufunc.py in _setup(self, in1, in2, outarr, clipmode) 1910 1911 if outarr is None: -> 1912 convType = _maxPopType(in2) 1913 else: 1914 convType = outarr._type /usr/lib64/python2.4/site-packages/numarray/ufunc.py in _maxPopType(xs) 1852 t = type(x) 1853 else: -> 1854 t = _nc._maxtype(x) 1855 1856 if maxT is None: /usr/lib64/python2.4/site-packages/numarray/numarraycore.py in _maxtype(args) 129 return complex 130 else: --> 131 return PyLevel2Type[_numarray._maxtype(args)] 132 133 def _storePyValueInBuffer(buffer, Ctype, index, value): TypeError: Expecting a python numeric type, got something else. From chanley at stsci.edu Tue Mar 21 08:33:59 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Tue Mar 21 08:33:59 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: References: <20060320203450.CIP50631@comet.stsci.edu> Message-ID: <44201C12.3070709@stsci.edu> David, Thank you for your time and help. Last night's build and tests completed without error. Chris > > I've replaced the thrown error with a log.warn. > > Fixed in svn. > From rhl at astro.princeton.edu Tue Mar 21 08:45:29 2006 From: rhl at astro.princeton.edu (Robert Lupton) Date: Tue Mar 21 08:45:29 2006 Subject: [Numpy-discussion] Multiple inheritance from ndarray In-Reply-To: <43FD32E4.10600@ieee.org> References: <5809AC56-B2DF-4403-B7BC-9AEEAAC78505@astro.princeton.edu> <43FD32E4.10600@ieee.org> Message-ID: I've finally found time to return to this problem. Travis's made the suggestion that I could use code along the lines of: > class Image(ndarray, actImage): > def __new__(subtype, *args) > act1 = actImage.__new__(actImage, *args) > actImage.__init__(act1, *args) > arr = array(act1.getArray(), 'd', copy=False) > self = arr.view(subtype) > > return self and this makes sense. Unfortunately, the arr.view(subtype) fails: TypeError: Cannot change descriptor for objectarray. [incidently, is that a typo for "object array"; the string's built by "C string" "concatenation"). Ideas? The C type "actImage" looks like: typedef struct actImage { int nrow; // number of rows int ncol; // number of columns void *base; // allocated memory, if owned by this object int baseRefCount; // reference counter for base double **rows; // Fluxes for pixels PyArrayObject *_array; // pointer to Numeric array; used by actNumPy } actImage; The _array is set in the C actImage constructor with: im->_array = (PyArrayObject *)PyArray_FromDimsAndData(2, dims, PyArray_DOUBLE, (void *)im->base); and getArray is a wrapper around: PyObject *actImageGetNumPy(const actImage *im) { Py_XINCREF(im->_array); return PyArray_Return(im->_array); } R > Robert Lupton wrote: >> I have a swig extension that defines a class that inherits from >> both a personal C-coded image struct (actImage), and also from >> Numeric's UserArray. This works very nicely, but I thought that >> it was about time to upgrade to numpy. >> >> The code looks like: >> >> from UserArray import * >> >> class Image(UserArray, actImage): >> def __init__(self, *args): >> actImage.__init__(self, *args) >> UserArray.__init__(self, self.getArray(), 'd', >> copy=False, savespace=False) >> >> I can't figure out how to convert this to use ndarray, as ndarray >> doesn't >> seem to have an __init__ method, merely a __new__. >> > > > Yes, the ndarray method doesn't have an __init__ method (so you > don't have to call it). > > What you need to do is write a __new__ method for your class. > However, with multiple-inheritance the details matter. You may > actually want to have your C-coded actImage class inherit (in C) > from the ndarray. If you would like help on that approach let me > know (I'll need to understand your actImage a bit better). > But, this can all be done in Python, too, but it is a bit of effort > to make sure things get created correctly. Perhaps it might make > sense to actually include a slightly modified form of the UserArray > in NumPy as a standard "container-class" (instead of a sub-class) > of the ndarray. In reality, a container class like UserArray and > a sub-class are different things. > > Here's an outline of what you need to do. This is, of course, > untested.... For example, I don't really know what actImage is. > > from numpy import ndarray, array > > class Image(ndarray, actImage): > def __new__(subtype, *args) > act1 = actImage.__new__(actImage, *args) > actImage.__init__(act1, *args) > arr = array(act1.getArray(), 'd', copy=False) > self = arr.view(subtype) > # you might need to copy attributes from act1 over to self > here... > return self > > > The problem here, is that apparently you are creating the array > first in actImage.__init__ and then passing it to UserArray. The > ndarray constructor wants to either create the array itself or use > a buffer-exposing object to use as the memory. > Keep us posted as your example is a good one that can help us all > learn. > > -Travis > > > > > From oliphant.travis at ieee.org Tue Mar 21 10:03:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 21 10:03:05 2006 Subject: [Numpy-discussion] Multiple inheritance from ndarray In-Reply-To: References: <5809AC56-B2DF-4403-B7BC-9AEEAAC78505@astro.princeton.edu> <43FD32E4.10600@ieee.org> Message-ID: <44203F91.7010505@ieee.org> Robert Lupton wrote: > I've finally found time to return to this problem. Travis's made the > suggestion that I could use code along the lines of: > >> class Image(ndarray, actImage): >> def __new__(subtype, *args) >> act1 = actImage.__new__(actImage, *args) >> actImage.__init__(act1, *args) >> arr = array(act1.getArray(), 'd', copy=False) >> self = arr.view(subtype) >> >> return self > > and this makes sense. Unfortunately, the arr.view(subtype) fails: > TypeError: Cannot change descriptor for objectarray. > [incidently, is that a typo for "object array"; the string's built > by "C string" "concatenation"). Yes, I think so. Hmm, it looks like you have object arrays floating around which seems strange. But, as I thought more about multiple inheritance I realized that there really is a case for a container class like UserArray that you can inherit from that only "wraps" the ndarray and does not try to inherit from it. The analogy I see with C-code is that a container class has a pointer to an array object structure, while a sub-class has the arrayobject structure itself as part of the new subclass. So, I added back the UserArray to numpy. Now, you can inherit from numpy.lib.UserArray.UserArray just like before. > Ideas? > > The C type "actImage" looks like: > typedef struct actImage { > int nrow; // number of rows > int ncol; // number of columns > void *base; // allocated memory, if owned by this > object > int baseRefCount; // reference counter for base > double **rows; // Fluxes for pixels > PyArrayObject *_array; // pointer to Numeric array; used > by actNumPy > } actImage; It looks like you have a "container" class by default here. You could, if you wanted make this a subclass by definining actImage as typedef struct actImage { PyArrayObject _array; int nrow; // redundant int ncol; // redundant void * base; // also seems redundant int baseRefCount; // perhaps not necessary either double **rows; // truly new information } actImage; Now, a pointer to actImage could be cast to a (PyArrayObject *) and used wherever arrayobjects are expected and work just fine. I'm not sure why you have a different pointer for allocated memory which could It seems to me that you should create a sub-type in C of the ndarray rather than a container class. But, a container class would also continue to work. With the re-addition of UserArray it would work in much the same way as well. > The _array is set in the C actImage constructor with: > im->_array = (PyArrayObject *)PyArray_FromDimsAndData(2, dims, > PyArray_DOUBLE, (void *)im->base); > For the subclass you would need to do a little more work in the constructor because PyArray_FromDims returns only a filled-in PyArrayObject structure. You could copy all the elements of this array over (this wouldn't copy data just pointers), and then delete the PyArrayObject structure (don't use DECREF, just tp_free). Then you could fill in the rest of the actImage structure. > and getArray is a wrapper around: > > PyObject *actImageGetNumPy(const actImage *im) > { > Py_XINCREF(im->_array); > return PyArray_Return(im->_array); > } > You could use the new tp_members structure here which allows you to get access to parts of your C-structure very easily. The easiest transition path is to just use the re-added UserArray as before. If you want to learn to sub-class, that could be done as well. But, there are no sub-classes in C that I know of yet, so although it is possible, there may be some subtle issues uncovered in the process. -Travis From rhl at astro.princeton.edu Tue Mar 21 11:10:02 2006 From: rhl at astro.princeton.edu (Robert Lupton) Date: Tue Mar 21 11:10:02 2006 Subject: [Numpy-discussion] Multiple inheritance from ndarray In-Reply-To: <44203F91.7010505@ieee.org> References: <5809AC56-B2DF-4403-B7BC-9AEEAAC78505@astro.princeton.edu> <43FD32E4.10600@ieee.org> <44203F91.7010505@ieee.org> Message-ID: Thanks Travis. >> Unfortunately, the arr.view(subtype) fails: >> TypeError: Cannot change descriptor for objectarray. >> [incidently, is that a typo for "object array"; the string's built >> by "C string" "concatenation")?] >> > Yes, I think so. Hmm, it looks like you have object arrays > floating around which seems strange. Do you know why I've acquired object arrays? I don't understand enough numpy guts to figure it out myself. If it's an interesting question, I can give you a complete tarball. > So, I added back the UserArray to numpy. Now, you can inherit from > numpy.lib.UserArray.UserArray just like before. This is in the latest svn version? R From pgmdevlist at mailcan.com Tue Mar 21 13:39:20 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 21 13:39:20 2006 Subject: [Numpy-discussion] Masked array - follow up Message-ID: <200603211553.43698.pgmdevlist@mailcan.com> Folks, First of all, many thanks to who implemented the ndim in the latest numpy.core.ma version. I ran into some other missing features with masked arrays. I updated the wiki page, and attached a suggestion for a patch: http://projects.scipy.org/scipy/numpy/wiki/MaskedArray The few modifications seem to work for me, but I guess they should be double/triple-checked. In any case, comments and ideas are more than welcome. Thanks again P. From haase at msg.ucsf.edu Tue Mar 21 15:15:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Mar 21 15:15:04 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" Message-ID: <200603211514.22249.haase@msg.ucsf.edu> Hi, I think I discovered another "ugly" bug: If have an array with rank 4. I thought numarray.maximum.reduce(arr, 1) would be identical to .numarray.maximum.reduce(arr, -3) But instead I get something that looks more like the transpose of the first ! I tried to check the documentation for the numarray.maximum function, but there is none at [8. Array Functions http://stsdas.stsci.edu/numarray/numarray-1.5.html/node38.html] This is a test a ran afterwards: >>> q=na.arange(8) >>> q.shape = (2,2,2) >>> q [[[0 1] [2 3]] [[4 5] [6 7]]] >>> na.maximum.reduce(q) [[4 5] [6 7]] >>> na.maximum.reduce(q,0) [[4 5] [6 7]] >>> na.maximum.reduce(q,1) [[2 3] [6 7]] >>> na.maximum.reduce(q,-1) [[1 3] [5 7]] So its not really the transpose - but in any case it's something strange... Thanks, Sebastinan Haase From pebarrett at gmail.com Wed Mar 22 05:33:09 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Wed Mar 22 05:33:09 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <200603211514.22249.haase@msg.ucsf.edu> References: <200603211514.22249.haase@msg.ucsf.edu> Message-ID: <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> On 3/21/06, Sebastian Haase wrote: > > Hi, > I think I discovered another "ugly" bug: > If have an array with rank 4. I thought > numarray.maximum.reduce(arr, 1) > would be identical to > .numarray.maximum.reduce(arr, -3) > But instead I get something that looks more like the transpose of the > first ! > I tried to check the documentation for the numarray.maximum function, > but there is none at [8. Array Functions > http://stsdas.stsci.edu/numarray/numarray-1.5.html/node38.html] > > This is a test a ran afterwards: > >>> q=na.arange(8) > >>> q.shape = (2,2,2) > >>> q > [[[0 1] > [2 3]] > [[4 5] > [6 7]]] > >>> na.maximum.reduce(q) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,0) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,1) > [[2 3] > [6 7]] > >>> na.maximum.reduce(q,-1) > [[1 3] > [5 7]] > > So its not really the transpose - but in any case it's something > strange... > The above behavior for maximum.reduce looks consistent to me. The reduce axis for examples 1 and 2 above is 0, so maximum is comparing arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. Example 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving [[2 3] [6 7]]. And the last example is comparing arrays [[0 2] [4 6]] and [[1 3] [5 7]], giving [[1 3] [5 7]]. I think it depends on how you look at the shape of the array. -- Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 22 08:20:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 22 08:20:07 2006 Subject: [Numpy-discussion] Re: numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> References: <200603211514.22249.haase@msg.ucsf.edu> <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> Message-ID: Paul Barrett wrote: > On 3/21/06, *Sebastian Haase* > wrote: > > Hi, > I think I discovered another "ugly" bug: > If have an array with rank 4. I thought > numarray.maximum.reduce(arr, 1) > would be identical to > .numarray.maximum.reduce(arr, -3) > But instead I get something that looks more like the transpose of > the first ! > I tried to check the documentation for the numarray.maximum function, > but there is none at [8. Array Functions > http://stsdas.stsci.edu/numarray/numarray-1.5.html/node38.html > ] > > This is a test a ran afterwards: > >>> q=na.arange(8) > >>> q.shape = (2,2,2) > >>> q > [[[0 1] > [2 3]] > [[4 5] > [6 7]]] > >>> na.maximum.reduce(q) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,0) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,1) > [[2 3] > [6 7]] > >>> na.maximum.reduce(q,-1) > [[1 3] > [5 7]] > > So its not really the transpose - but in any case it's something > strange... > > The above behavior for maximum.reduce looks consistent to me. > > The reduce axis for examples 1 and 2 above is 0, so maximum is comparing > arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. > Example 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving > [[2 3] [6 7]]. And the last example is comparing arrays [[0 2] [4 6]] > and [[1 3] [5 7]], giving [[1 3] [5 7]]. > > I think it depends on how you look at the shape of the array. Well, there is at least an incompatibility with numpy and Numeric. In [6]: def test(mod): ...: q = mod.reshape(mod.arange(16), (2,2,2,2)) ...: for i in range(4): ...: mask = (mod.maximum.reduce(q, i) == mod.maximum.reduce(q, i-4)) ...: print i, mod.alltrue(mod.ravel(mask)) ...: ...: In [7]: test(numpy) 0 True 1 True 2 True 3 True In [8]: test(numarray) 0 0 1 0 2 1 3 1 In [19]: test(Numeric) 0 1 1 1 2 1 3 1 In [9]: numpy.__version__ Out[9]: '0.9.6.2148' In [10]: numarray.__version__ Out[10]: '1.5.0' In [20]: Numeric.__version__ Out[20]: '24.0' -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fonnesbeck at gmail.com Wed Mar 22 08:31:01 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Mar 22 08:31:01 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's Message-ID: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> If linalg.eigenvectors is called on a matrix that contains nan values, it hangs, quickly consuming system resources. Here is the matrix: matrix = [[ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0.89827259 0. 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0.92510948 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.93319244 nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0.89827259 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0.92510948 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0. 0.92510948 0.93319244]] I would expect an exception, but I get none. Using a recent cvs build of numpy. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From fonnesbeck at gmail.com Wed Mar 22 08:31:04 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Mar 22 08:31:04 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's Message-ID: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> If linalg.eigenvectors is called on a matrix that contains nan values, it hangs, quickly consuming system resources. Here is the matrix: matrix = [[ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0.89827259 0. 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0.92510948 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.93319244 nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0.89827259 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0.92510948 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0. 0.92510948 0.93319244]] I would expect an exception, but I get none. Using a recent cvs build of numpy. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Wed Mar 22 08:39:05 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed Mar 22 08:39:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <1142313130.441650aad8111@webmail.colorado.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> Message-ID: <44217D77.7030403@ntc.zcu.cz> Fernando.Perez at colorado.edu wrote: > If you have any concerns about the code, let me know and I'll be > happy to address them, but I really think that this will be much more > useful to everyone if each directory contains a complete, working > example. Thanks for the typemaps, Fernando. Do you think you could update the 'Cookbook/SWIG and NumPy' web site section to reflect their presence? For the moment there is just my simple (yet sufficient for my purposes) attempt. cheers, r. From tgrav at mac.com Wed Mar 22 09:36:06 2006 From: tgrav at mac.com (Tommy Grav) Date: Wed Mar 22 09:36:06 2006 Subject: [Numpy-discussion] Problems installing NumPy Message-ID: <4E0ECB41-EF32-4F0E-9DAE-52B55C8AE3A0@mac.com> I am new to Python and just downloaded ActivePython 2.4.2.10 on my Mac PPC with OS X 10.4. I added the numpy package (0.9.6-py2.4) and it imports fine. But when I try pydoc random I get this error [tgrav at skathi] site-packages/numpy -> pydoc random import core -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import lib -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import linalg -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import dft -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import random -> failed: 'module' object has no attribute 'dtype' problem in ./random/__init__.py - ImportError: cannot import name add_newdoc Anyone know what is causing this problem? Is there something in my installation of either ActivePython or NumPy that needs to be fixed? Cheers Tommy tgrav at mac.com http://homepage.mac.com/tgrav/ "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction" -- Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Mar 22 09:45:59 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 09:45:59 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Fast Gauss Transform In-Reply-To: <91cf711d0603220839g2c974524h@mail.gmail.com> References: <91cf711d0603201107k7b1f38c7g@mail.gmail.com> <91cf711d0603210652u8f93965y@mail.gmail.com> <44206894.20007@gmail.com> <1142990801.30961.11.camel@localhost.localdomain> <87ek0vs20b.fsf@peds-pc311.bsd.uchicago.edu> <91cf711d0603220839g2c974524h@mail.gmail.com> Message-ID: <44218D03.4080703@ieee.org> David Huard wrote: > While fiddling with new code to compute kde, I noticed that > gaussian_kde has an unexpected behavior for 1D arrays. > > For instance : > > x = rand(1000) > g = gaussian_kde(x) > g(z) > > returns the same value no matter what z is. The problem disappears for > multidimensional data. The dot product (self.inv_cov, diff) seems to > be causing the problem, doing the product only for the first element > of diff and returning zeros for the rest. I guess this is not the > intended behavior for dot. Yet another untested branch of the optimized dot code. Apparently it's been very hard to get this one right after re-writing the scalar-multiplication portion. Scalar multiplication is used in several branches but the parameters must be set just right. You were exposing a (1,1) x (1,N) error that uses BLAS scalar multiplication at the core but the number of multiplies was set to 1 rather than N. I added a test for this case in numpy and fixed the problem. Thanks for the report. -Travis From haase at msg.ucsf.edu Wed Mar 22 09:46:01 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Mar 22 09:46:01 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> References: <200603211514.22249.haase@msg.ucsf.edu> <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> Message-ID: <200603220945.18706.haase@msg.ucsf.edu> On Wednesday 22 March 2006 05:32, Paul Barrett wrote: > On 3/21/06, Sebastian Haase wrote: > > The above behavior for maximum.reduce looks consistent to me. > > The reduce axis for examples 1 and 2 above is 0, so maximum is comparing > arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. Example > 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving [[2 3] [6 > 7]]. And the last example is comparing arrays [[0 2] [4 6]] and [[1 3] [5 > 7]], giving [[1 3] [5 7]]. > > I think it depends on how you look at the shape of the array. > > -- Paul Sorry I did the test wrong - >>> na.maximum.reduce(q,2) [[1 3] [5 7]] >>> na.maximum.reduce(q,-1) [[1 3] [5 7]] So it obviously works as advertised. The array where I noticed the transpose behaviour was really of shape 32,15,521,512 and my command was na.maximum.reduce(arr,-3) which "looked" tranpose to what I got with na.maximum.reduce(arr,1). I using numarray for image analysis and the array might have been a memmap-ed image file. I'm just worried because I found an "accidental transpose bug" on type conversion a few years back (see mailing list)... ( I forgot to say that I running Python 2.2 and numarray version '1.4.0' ) I just reran the test with python2.4 and arr=arr.copy() [which should give a non-memmap version of arr] >>> arrM = na.maximum.reduce(arr,-3) >>> arrMt = na.transpose(arrM,(0,2,1)) >>> arrM1= na.maximum.reduce(arr,1) >>> U.mmms(arrM1-arrMt) # returns min,max,mean,stddev (0, 0, 0.0, 0.0) So it definitely transposes the last two axes ! Maybe the bug shows up on rank >=4 !? - which is the case Robert tested. Thanks, Sebastian Haase From ndarray at mac.com Wed Mar 22 12:27:00 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 12:27:00 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: <200603211553.43698.pgmdevlist@mailcan.com> References: <200603211553.43698.pgmdevlist@mailcan.com> Message-ID: Pierre, Your patch proposes to add an optional argument "flag_missing=False" to the trace method. As you explain, "`flag_missing` is True and some diagonal values are masked, returns `masked`." This feature deserves some careful discussion before it is accepted. I would think this feature is unnecessary because a.fill(0).trace() is a better way to express the desired result than a.trace(flag_missing=False). (Your implementation is equivalent to a.data.trace(), which is clearly wrong because masked elements are likely to contain garbage.) -- sasha PS: BTW, your implementation is incorrect. Currently ma assumes that "masked" is a singleton, so return MaskedArray(0, int, mask=1) is not the same as return masked. Also m = getmask(a) should probably be m = getmask(d). On 3/21/06, Pierre GM wrote: > Folks, > First of all, many thanks to who implemented the ndim in the latest > numpy.core.ma version. > I ran into some other missing features with masked arrays. I updated the wiki > page, and attached a suggestion for a patch: > http://projects.scipy.org/scipy/numpy/wiki/MaskedArray > The few modifications seem to work for me, but I guess they should be > double/triple-checked. In any case, comments and ideas are more than welcome. > Thanks again > P. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ndarray at mac.com Wed Mar 22 13:43:04 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 13:43:04 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled Message-ID: In an ideal world, any function that accepts ndarray would accept ma.array and vice versa. Moreover, if the ma.array has no masked elements and the same data as ndarray, the result should be the same. Obviously current implementation falls short of this goal, but there is one feature that seems to make this goal unachievable. This feature is the "filled" method of ma.array. Pydoc for this method reports the following: | filled(self, fill_value=None) | A numeric array with masked values filled. If fill_value is None, | use self.fill_value(). | | If mask is nomask, copy data only if not contiguous. | Result is always a contiguous, numeric array. | # Is contiguous really necessary now? That is not the best possible description ("filled" is "filled"), but the essence is that the result of a.filled(value) is a contiguous ndarray obtained from the masked array by copying non-masked elements and using value for masked values. I would like to propose to add a "filled" method to ndarray. I see several possibilities and would like to hear your opinion: 1. Make filled simply return self. 2. Make filled return a contiguous copy. 3. Make filled replace nans with the fill_value if array is of floating point type. Unfortunately, adding "filled" will result is a rather confusing situation where "fill" and "filled" both exist and have very different meanings. I would like to note that "fill" is a somewhat odd ndarray method. AFAICT, it is the only non-special method that mutates the array. It appears to be just a performance trick: the same result can be achived with "a[...] = ". From pgmdevlist at mailcan.com Wed Mar 22 13:58:02 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Wed Mar 22 13:58:02 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: References: <200603211553.43698.pgmdevlist@mailcan.com> Message-ID: <200603221654.37730.pgmdevlist@mailcan.com> Sasha, I'm still not sure the `flag_missing` option is really needed. Probably not, as you noted. I intended to use it as a reminder, just in case. I agree with you, it's confusing and should probably be discarded. that maybe useful a bit later, for cumprod for example: we could have something like: if `use_missing`: a.cumprod() = MA.masked else: a.cumprod() = a.filled(1).cumprod() > I would think this > feature is unnecessary because a.fill(0).trace() is a better way to > express the desired result than a.trace(flag_missing=False). If all the elements of the diagonal of a are masked, what should a.trace() be ? 0, MA.masked, undefined ? `a.fill(0).trace` (well, `a.filled(0).trace` actually) will always give 0. The version I implemented give you MA.masked. try it. > Your implementation is equivalent to a.data.trace(), which is clearly wrong > because masked elements are likely to contain garbage.) Well, not really: if a is a MaskedArray d = diagonal(a,offset, axis1, axis2) outputs a MaskedArray, with d.data=a.diagonal().data and d.mask=d.diagonal.mask(). Check it. The version implemented actually corresponds to: a.trace() = a.diagonal().compressed().sum() > PS: BTW, your implementation is incorrect. Currently ma assumes that > "masked" is a singleton, so return MaskedArray(0, int, mask=1) is not > the same as return masked. well, I intended to use masked, but realize `masked` wasn't defined before the last line, so I just copied its definition. > Also m = getmask(a) should probably be m = > getmask(d). Well, no, not really: m is only used to check whether the initial array is masked or not. BTW, I attach the unittest I just finished writing. That passes on my box. Let me know how it works -------------- next part -------------- A non-text attachment was scrubbed... Name: testnewma.py Type: application/x-python Size: 3431 bytes Desc: not available URL: From ndarray at mac.com Wed Mar 22 14:22:06 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 14:22:06 2006 Subject: [Numpy-discussion] Bug or feature: sum vs. add.reduce Message-ID: Is this correct? >>> numpy.array([1.,.5]).sum(dtype=None) 1.5 >>> numpy.add.reduce([1.,.5],dtype=None) 1 From ndarray at mac.com Wed Mar 22 14:43:05 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 14:43:05 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: <200603221616.30402.pgmdevlist@mailcan.com> References: <200603211553.43698.pgmdevlist@mailcan.com> <200603221616.30402.pgmdevlist@mailcan.com> Message-ID: I've aplied Pierre's patch with minor modifications. See http://projects.scipy.org/scipy/numpy/changeset/2267 . For the time being trace is defined simply as diagonal().sum(). Thanks Pierre! From ndarray at mac.com Wed Mar 22 15:14:04 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 15:14:04 2006 Subject: [Numpy-discussion] Should ma.array.clip allow masked min and max? Message-ID: Pierre has implemented the clip method for ma.array, but his implementation does not seem to do the right thing if min or max is masked: >>> from numpy.core.ma import * >>> x = ones(3) >>> max = array([0,2,0],mask=[1,1,1]) >>> min = array([0,2,0],mask=[1,1,1]) >>> x.clip(min,max) array([1 1 1]) while >>> x.clip(min.data,max.data) array([0 2 0]) I believe it would be better to mask the elements in the result for which either min or max is masked. From efiring at hawaii.edu Wed Mar 22 15:38:01 2006 From: efiring at hawaii.edu (Eric Firing) Date: Wed Mar 22 15:38:01 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: References: Message-ID: <4421DF9B.6000102@hawaii.edu> Sasha wrote: > In an ideal world, any function that accepts ndarray would accept > ma.array and vice versa. Moreover, if the ma.array has no masked > elements and the same data as ndarray, the result should be the same. This would be *very* nice. > Obviously current implementation falls short of this goal, but there > is one feature that seems to make this goal unachievable. > > This feature is the "filled" method of ma.array. Pydoc for this > method reports the following: > > | filled(self, fill_value=None) > | A numeric array with masked values filled. If fill_value is None, > | use self.fill_value(). > | > | If mask is nomask, copy data only if not contiguous. > | Result is always a contiguous, numeric array. > | # Is contiguous really necessary now? > > > That is not the best possible description ("filled" is "filled"), but > the essence is that the result of a.filled(value) is a contiguous > ndarray obtained from the masked array by copying non-masked elements > and using value for masked values. > > I would like to propose to add a "filled" method to ndarray. I see > several possibilities and would like to hear your opinion: > > 1. Make filled simply return self. > > 2. Make filled return a contiguous copy. > > 3. Make filled replace nans with the fill_value if array is of > floating point type. It seems to me that any function or method that returns an array from an array should be perfectly consistent and explicit about whether it makes a copy or not. Sometimes the filled method *needs* to return a copy; therefore it should *always* return a copy, regardless of the presence or state of masking. Hence I think the filled method of ma needs to be changed in this way also. The question for your suggestion 3 is, should a nan always be the equivalent of a masked value? One loses a little flexibility, but it has an appealing simplicity to it. I could be persuaded otherwise, but right now I would vote for it. Eric > > > Unfortunately, adding "filled" will result is a rather confusing > situation where "fill" and "filled" both exist and have very different > meanings. > > I would like to note that "fill" is a somewhat odd ndarray method. > AFAICT, it is the only non-special method that mutates the array. It > appears to be just a performance trick: the same result can be achived > with "a[...] = ". > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From pgmdevlist at mailcan.com Wed Mar 22 15:42:01 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Wed Mar 22 15:42:01 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: References: <200603211553.43698.pgmdevlist@mailcan.com> <200603221616.30402.pgmdevlist@mailcan.com> Message-ID: <200603221840.51536.pgmdevlist@mailcan.com> On Wednesday 22 March 2006 17:42, Sasha wrote: > I've aplied Pierre's patch with minor modifications. See > http://projects.scipy.org/scipy/numpy/changeset/2267 . > > For the time being trace is defined simply as diagonal().sum(). Great ! Thanks a lot. I'll try to tackle some of the other missing feature very soon. From fonnesbeck at gmail.com Wed Mar 22 15:56:07 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Mar 22 15:56:07 2006 Subject: [Numpy-discussion] linalg.eigenvector hang on nan values Message-ID: <723eb6930603221555q2ff8ba7cifdbab975b4578884@mail.gmail.com> If linalg.eigenvectors is called on a matrix that contains nan values, it hangs, quickly consuming system resources. Here is the matrix: matrix = [[ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0.89827259 0. 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0.92510948 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.93319244 nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0.89827259 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0.92510948 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0. 0.92510948 0.93319244]] I would expect an exception, but I get none. Using a recent cvs build of numpy. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Wed Mar 22 15:59:23 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 15:59:23 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421DF9B.6000102@hawaii.edu> References: <4421DF9B.6000102@hawaii.edu> Message-ID: <4421E4A6.4020906@cox.net> Eric Firing wrote: > Sasha wrote: > >> In an ideal world, any function that accepts ndarray would accept >> ma.array and vice versa. Moreover, if the ma.array has no masked >> elements and the same data as ndarray, the result should be the same. > > > This would be *very* nice. > >> Obviously current implementation falls short of this goal, but there >> is one feature that seems to make this goal unachievable. >> >> This feature is the "filled" method of ma.array. Pydoc for this >> method reports the following: >> >> | filled(self, fill_value=None) >> | A numeric array with masked values filled. If fill_value is >> None, >> | use self.fill_value(). >> | >> | If mask is nomask, copy data only if not contiguous. >> | Result is always a contiguous, numeric array. >> | # Is contiguous really necessary now? >> >> >> That is not the best possible description ("filled" is "filled"), but >> the essence is that the result of a.filled(value) is a contiguous >> ndarray obtained from the masked array by copying non-masked elements >> and using value for masked values. >> >> I would like to propose to add a "filled" method to ndarray. I see >> several possibilities and would like to hear your opinion: >> >> 1. Make filled simply return self. >> >> 2. Make filled return a contiguous copy. >> >> 3. Make filled replace nans with the fill_value if array is of >> floating point type. > > > It seems to me that any function or method that returns an array from > an array should be perfectly consistent and explicit about whether it > makes a copy or not. Sometimes the filled method *needs* to return a > copy; therefore it should *always* return a copy, regardless of the > presence or state of masking. Hence I think the filled method of ma > needs to be changed in this way also. +1 (mumble mumble reshape mumble) > > The question for your suggestion 3 is, should a nan always be the > equivalent of a masked value? One loses a little flexibility, but it > has an appealing simplicity to it. I could be persuaded otherwise, > but right now I would vote for it. -0 (I'd be -1, but I don't really use masked arrays). While masked array should be convenient for masking. The basic ndarray object should be as flexible as possible since it's the building block for all the rest. -tim From tim.hochberg at cox.net Wed Mar 22 16:32:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 16:32:01 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 Message-ID: <4421EC3B.4020800@cox.net> I just updated from CVS and compiled using VC7. The resulting numpy segfaults immediately on import. Do not pass go, do not collect $200. It's been a couple of weeks probably since I updated from CVS, so I'm not sure where to start looking. I suppose I could start going back in time until it works again and then see what checkins broke it, but if anyone has any better ideas, let me know. -tim From zpincus at stanford.edu Wed Mar 22 16:45:00 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed Mar 22 16:45:00 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <4421EC3B.4020800@cox.net> References: <4421EC3B.4020800@cox.net> Message-ID: <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> I've seen this too, and been able to fix it by removing the build directory and rebuilding everything. I wonder if there's a bug in numpy's dependency checking that's causing certain files to not get rebuilt when they ought. I recently updated to the SVN head, and numpy segfaulted on import, just as you describe. After puttering around with GDB, it looked like some of the code was out of synch with other parts of the code, so I decided to rebuild from scratch. After I deleted the entire numpy build directory ('python setup.py clean' *did not* work) and re-built, things worked just fine. I've also had this problem with scipy, and the same solution applied. Anyhow, try removing all traces of build products from the numpy directory and rebuilding... maybe that will fix things. Zach On Mar 22, 2006, at 4:30 PM, Tim Hochberg wrote: > > I just updated from CVS and compiled using VC7. The resulting numpy > segfaults immediately on import. Do not pass go, do not collect > $200. It's been a couple of weeks probably since I updated from > CVS, so I'm not sure where to start looking. I suppose I could > start going back in time until it works again and then see what > checkins broke it, but if anyone has any better ideas, let me know. > > -tim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Fernando.Perez at colorado.edu Wed Mar 22 16:57:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 22 16:57:05 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> References: <4421EC3B.4020800@cox.net> <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> Message-ID: <4421F219.6080408@colorado.edu> Zachary Pincus wrote: > I've seen this too, and been able to fix it by removing the build > directory and rebuilding everything. > > I wonder if there's a bug in numpy's dependency checking that's > causing certain files to not get rebuilt when they ought. I recently > updated to the SVN head, and numpy segfaulted on import, just as you > describe. After puttering around with GDB, it looked like some of the > code was out of synch with other parts of the code, so I decided to > rebuild from scratch. > > After I deleted the entire numpy build directory ('python setup.py > clean' *did not* work) and re-built, things worked just fine. I've > also had this problem with scipy, and the same solution applied. > > Anyhow, try removing all traces of build products from the numpy > directory and rebuilding... maybe that will fix things. Just a hint from experience: do NOT trust distutils to do proper dependency management and know what to rebuild and what not to. Most of us have settled on some variation of: planck[scipy]> d /usr/local/installers/src/scipy total 16 -rwxr-xr-x 1 fperez 390 Mar 8 13:28 makepkg* drwxr-xr-x 6 fperez 4096 Mar 14 11:52 numpy/ drwxr-xr-x 4 fperez 4096 Nov 17 15:16 scipy/ -rwxr-xr-x 1 fperez 415 Jan 5 01:37 testpkg* planck[scipy]> cat makepkg #!/bin/sh PACKAGE=$1 PYTHON=python2.3 PYPREFIX=$HOME/tmp/local svn update ${PACKAGE} export PYTHONPATH=${PYPREFIX}/lib/${PYTHON}/site-packages:${PYTHONPATH}: # remove existing ${PACKAGE} to make sure the build doesn't pick up spurious things rm -rf $PYPREFIX/lib/${PYTHON}/site-packages/${PACKAGE} # make/install cd ${PACKAGE} rm -rf build $PYTHON setup.py install --prefix=$PYPREFIX ################ With that, I can just type ./makepkg numpy or ./makepkg scipy and I'm almost sure it will do the right thing. I wonder if we shouldn't add this to the wiki as the recommended procedure for builds (at least under *nix). I know it allows me to do fresh svn builds of numpy/scipy at any time without having to think (a process rarely successful within the confines of my skull). Cheers, f From Fernando.Perez at colorado.edu Wed Mar 22 17:06:14 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 22 17:06:14 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <44217D77.7030403@ntc.zcu.cz> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> Message-ID: <4421F455.2020003@colorado.edu> Robert Cimrman wrote: > Fernando.Perez at colorado.edu wrote: > >>If you have any concerns about the code, let me know and I'll be >>happy to address them, but I really think that this will be much more >> useful to everyone if each directory contains a complete, working >>example. > > > Thanks for the typemaps, Fernando. Do you think you could update the > 'Cookbook/SWIG and NumPy' web site section to reflect their presence? > For the moment there is just my simple (yet sufficient for my purposes) > attempt. Done, both for pyrex and swig. I added very simple notes, feel free to add further detail as necessary: http://scipy.org/Cookbook/Pyrex_and_NumPy http://scipy.org/Cookbook/SWIG_and_NumPy Cheers, f From robert.kern at gmail.com Wed Mar 22 17:19:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 22 17:19:06 2006 Subject: [Numpy-discussion] Re: Current SVN segfaults under VC7 In-Reply-To: <4421F219.6080408@colorado.edu> References: <4421EC3B.4020800@cox.net> <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> <4421F219.6080408@colorado.edu> Message-ID: Fernando Perez wrote: > Zachary Pincus wrote: > >> I've seen this too, and been able to fix it by removing the build >> directory and rebuilding everything. >> >> I wonder if there's a bug in numpy's dependency checking that's >> causing certain files to not get rebuilt when they ought. I recently >> updated to the SVN head, and numpy segfaulted on import, just as you >> describe. After puttering around with GDB, it looked like some of the >> code was out of synch with other parts of the code, so I decided to >> rebuild from scratch. >> >> After I deleted the entire numpy build directory ('python setup.py >> clean' *did not* work) and re-built, things worked just fine. I've >> also had this problem with scipy, and the same solution applied. >> >> Anyhow, try removing all traces of build products from the numpy >> directory and rebuilding... maybe that will fix things. > > Just a hint from experience: do NOT trust distutils to do proper > dependency management and know what to rebuild and what not to. Generally it does an okay job, I think. numpy does quite a lot with generated code, however, and distutils *does* have a hard time with that. Unfortunately, I don't think there is much we can do about that without replacing even more of distutils. > Most of > us have settled on some variation of: ... > With that, I can just type > > ./makepkg numpy > > or > > ./makepkg scipy > > and I'm almost sure it will do the right thing. > > I wonder if we shouldn't add this to the wiki as the recommended > procedure for builds (at least under *nix). I know it allows me to do > fresh svn builds of numpy/scipy at any time without having to think (a > process rarely successful within the confines of my skull). +1. At least until 1.0 when the C API settles down. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Wed Mar 22 17:20:19 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 22 17:20:19 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421E4A6.4020906@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> Message-ID: <4421F798.2070300@ee.byu.edu> Tim Hochberg wrote: >> >> It seems to me that any function or method that returns an array from >> an array should be perfectly consistent and explicit about whether it >> makes a copy or not. Sometimes the filled method *needs* to return a >> copy; therefore it should *always* return a copy, regardless of the >> presence or state of masking. Hence I think the filled method of ma >> needs to be changed in this way also. > > > +1 > > (mumble mumble reshape mumble) Tim makes a good point here. Should the reshape method be fixed to always return a copy? The semantics a.shape = (...) could still be used to re-shape contiguous arrays where possible. However, whether or not reshape returns a copy is consistent (but perhaps not explicitly explained). We will still have .ravel() which sometimes copies and sometimes doesn't. -Travis From oliphant at ee.byu.edu Wed Mar 22 17:26:14 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 22 17:26:14 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <4421EC3B.4020800@cox.net> References: <4421EC3B.4020800@cox.net> Message-ID: <4421F909.6050401@ee.byu.edu> Tim Hochberg wrote: > > I just updated from CVS and compiled using VC7. The resulting numpy > segfaults immediately on import. Do not pass go, do not collect $200. > It's been a couple of weeks probably since I updated from CVS, so I'm > not sure where to start looking. I suppose I could start going back in > time until it works again and then see what checkins broke it, but if > anyone has any better ideas, let me know. > You definitely need to remove your build directory. It is a build dependency issue (some needed generated code is not getting rebuilt when it should be). I'm not sure how to fix it, so I remove the build directory. It shows up when the C-API changes more significantly then just appending new C-API calls. It changed significantly recently as I moved the version checking C-API to the very front (so it won't move around in the future -- no matter what is done to the C-API), and I added the fixed Boolean array scalar True and False objects. The segfaults in scipy should disappear upon rebuild of numpy because the C-API now has the version checking at the very start and will gracefully exit with an error rather than rudely segfault. -Travis From pierregm at engr.uga.edu Wed Mar 22 17:37:03 2006 From: pierregm at engr.uga.edu (Pierre GM) Date: Wed Mar 22 17:37:03 2006 Subject: [Numpy-discussion] Re: Numpy-discussion digest, Vol 1 #1632 - 10 msgs In-Reply-To: <20060322234311.30FA78974B@sc8-sf-spam1.sourceforge.net> References: <20060322234311.30FA78974B@sc8-sf-spam1.sourceforge.net> Message-ID: <200603222035.08682.pierregm@engr.uga.edu> > Pierre has implemented the clip method for ma.array, but his > implementation does not seem to do the right thing if min or max is > masked: From core/oldnumeric.py: """clip(m, m_min, m_max) = every entry in m that is less than m_min is replaced by m_min, and every entry greater than m_max is replaced by m_max. """ My understanding was that m_min and m_max were two floats, not two arrays. I completely oversaw that. Your suggestion (masking where the limits are masked) makes complete sense. I'll work on that. From ndarray at mac.com Wed Mar 22 17:40:06 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 17:40:06 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421F798.2070300@ee.byu.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> Message-ID: On 3/22/06, Travis Oliphant wrote: > Tim makes a good point here. Should the reshape method be fixed to > always return a copy? The semantics a.shape = (...) could still be used > to re-shape contiguous arrays where possible. > Reshape never copies the data: >>> x = ones(4) >>> x.reshape((2,2)).__array_data__ == x.__array_data__ True The only inconsistency is that >>> x.reshape((2,2)) is x False >>> x.reshape((4,)) is x True I agree that this is unnecessary, but don't see much of a problem. +0 here > However, whether or not reshape returns a copy is consistent (but > perhaps not explicitly explained). > To me consistency means "is independent of the input." Whether or not reshape creates a new python object depends on the value of the argument. I would call it inconsistency. > We will still have .ravel() which sometimes copies and sometimes doesn't. Ravel should be a shortcut for x.reshape((x.size,)), so it is really the same question. +0 (to make ravel always return a new python object) From tim.hochberg at cox.net Wed Mar 22 17:57:09 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 17:57:09 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421F798.2070300@ee.byu.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> Message-ID: <44220001.4050606@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> It seems to me that any function or method that returns an array >>> from an array should be perfectly consistent and explicit about >>> whether it makes a copy or not. Sometimes the filled method *needs* >>> to return a copy; therefore it should *always* return a copy, >>> regardless of the presence or state of masking. Hence I think the >>> filled method of ma needs to be changed in this way also. >> >> >> >> +1 >> >> (mumble mumble reshape mumble) > > > > Tim makes a good point here. And so articulately stated too ;) > Should the reshape method be fixed to always return a copy? The > semantics a.shape = (...) could still be used to re-shape contiguous > arrays where possible. > However, whether or not reshape returns a copy is consistent (but > perhaps not explicitly explained). > > We will still have .ravel() which sometimes copies and sometimes doesn't. My opinion is that all methods and functions should either: 1. Always return a copy. 2. Always return a view 3. Return a view if possible otherwise raise an exception. So, like Sasha, I'd like to see ravel changed as well. I don't really care if it's to 1 or 3 though. -tim From tim.hochberg at cox.net Wed Mar 22 18:10:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 18:10:03 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <4421F909.6050401@ee.byu.edu> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> Message-ID: <44220366.4000004@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> I just updated from CVS and compiled using VC7. The resulting numpy >> segfaults immediately on import. Do not pass go, do not collect $200. >> It's been a couple of weeks probably since I updated from CVS, so I'm >> not sure where to start looking. I suppose I could start going back >> in time until it works again and then see what checkins broke it, but >> if anyone has any better ideas, let me know. >> > > You definitely need to remove your build directory. It is a build > dependency issue (some needed generated code is not getting rebuilt > when it should be). I'm not sure how to fix it, so I remove the build > directory. > > It shows up when the C-API changes more significantly then just > appending new C-API calls. It changed significantly recently as I > moved the version checking C-API to the very front (so it won't move > around in the future -- no matter what is done to the C-API), and I > added the fixed Boolean array scalar True and False objects. > > The segfaults in scipy should disappear upon rebuild of numpy because > the C-API now has the version checking at the very start and will > gracefully exit with an error rather than rudely segfault. This did fix things...eventually. I removed the build directory and rebuilt. Numpy still bombed the interpreter, although in a new and less exciting way. I then did a clean checkout from svn and rebuilt. Same thing. Then I manally removed numpy from site-packages and reinstalled at which points things worked and all tests passed. So in addition to needing to remove the build directory, it appears that something was hanging out in the destination directory that did not get overwritten and caused problems. Perhaps this description will help the next person who runs into this, but probably not. -tim From oliphant.travis at ieee.org Wed Mar 22 19:07:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 19:07:08 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44220B75.7030406@cox.net> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220B75.7030406@cox.net> Message-ID: <442210A7.7080802@ieee.org> Tim Hochberg wrote: > > OK, now that I've got things working, I can report a real segfault > under VC7. The following snippet of code: > > from numpy import * > a = arange(100.) > b = array(1+1j) > (a+1) ** (b+3) I can reproduce it on Linux. It looks like it's related to the power optimization stuff that was recently checked in. power(a,b+3) does not crash but >>> a**(b+3) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076248704 (LWP 1545)] 0x404162df in array_power_is_scalar (o2=0x8294200, exp=0xbfffec68) at arrayobject.c:2656 2656 arrayobject.c: No such file or directory. in arrayobject.c -Travis From ndarray at mac.com Wed Mar 22 19:20:03 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 19:20:03 2006 Subject: [Numpy-discussion] Changeset 2222 Message-ID: Changeset 2222 redefines several oldnumeric functions according to the following template: try: result = a.FUNC(...) except AttributeError: result = _wrapit(a, 'FUNC', ...) return result I believe it should be: try: FUNC = a.FUNC except AttributeError: return _wrapit(a, 'FUNC', ...) return FUNC(...) In most cases it should not matter, but some FUNCs can throw AttributeError on object arrays. From oliphant.travis at ieee.org Wed Mar 22 19:27:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 19:27:03 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44220B75.7030406@cox.net> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220B75.7030406@cox.net> Message-ID: <44221561.6060605@ieee.org> Tim Hochberg wrote: > > OK, now that I've got things working, I can report a real segfault > under VC7. The following snippet of code: > > from numpy import * > a = arange(100.) > b = array(1+1j) > (a+1) ** (b+3) > > will cause current CVS to crash. [under Window XP compiled with VC7]. > I also get a segfaults using 0.9.6 from scipy.org, which I believe is > compiled with mingw, so it must not just be a vc7 thing. It doesn't > happen if b is python scalar as opposed to a numpy scalar, FWIW. > > Can anyone reproduce this on a non-windows platform? Should be fixed in SVN, now. The problem was the use of PyArray_ISINTEGER macro on a non array object. Thanks, -Travis From oliphant.travis at ieee.org Wed Mar 22 20:09:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 20:09:15 2006 Subject: [Numpy-discussion] Changeset 2222 In-Reply-To: References: Message-ID: <44221F46.9000607@ieee.org> Sasha wrote: > Changeset 2222 redefines several oldnumeric functions according to the > following template: > > try: > result = a.FUNC(...) > except AttributeError: > result = _wrapit(a, 'FUNC', ...) > return result > > I believe it should be: > > try: > FUNC = a.FUNC > except AttributeError: > return _wrapit(a, 'FUNC', ...) > return FUNC(...) > > In most cases it should not matter, but some FUNCs can throw > AttributeError on object arrays. > Good point. Needs to be changed. -Travis From ndarray at mac.com Wed Mar 22 20:38:02 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 20:38:02 2006 Subject: [Numpy-discussion] Changeset 2222 In-Reply-To: <44221F46.9000607@ieee.org> References: <44221F46.9000607@ieee.org> Message-ID: On 3/22/06, Travis Oliphant wrote: > Good point. Needs to be changed. I'll do it. From Fernando.Perez at colorado.edu Wed Mar 22 22:22:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 22 22:22:02 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44220366.4000004@cox.net> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220366.4000004@cox.net> Message-ID: <44223E4A.9090208@colorado.edu> Tim Hochberg wrote: > This did fix things...eventually. I removed the build directory and > rebuilt. Numpy still bombed the interpreter, although in a new and less > exciting way. I then did a clean checkout from svn and rebuilt. Same > thing. Then I manally removed numpy from site-packages and reinstalled > at which points things worked and all tests passed. > > So in addition to needing to remove the build directory, it appears that > something was hanging out in the destination directory that did not get > overwritten and caused problems. > > Perhaps this description will help the next person who runs into this, > but probably not. Well, the script that I posted does precisely all of this in a mindless fashion, which is why I find it useful. You may want to give it a try (it's trivial, but it just ensures you don't forget these steps and end up wasting any time chasing the problem). Cheers, f From mpuqwpizl at mayerbrown.com Wed Mar 22 23:56:06 2006 From: mpuqwpizl at mayerbrown.com (mpuqwpizl) Date: Wed Mar 22 23:56:06 2006 Subject: [Numpy-discussion] Fw: mpuqwpizl Message-ID: <000901c64e4f$21d92660$ba0278c9@fantasy> ----- Original Message ----- From: Abraham Jeanie To: vqmqthe at wiley.com Sent: Tuesday, March 21, 2006 11:17 PM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bdftlqfrty.gif Type: image/gif Size: 16338 bytes Desc: not available URL: From cimrman3 at ntc.zcu.cz Thu Mar 23 01:51:05 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu Mar 23 01:51:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4421F455.2020003@colorado.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> <4421F455.2020003@colorado.edu> Message-ID: <44226F75.8010401@ntc.zcu.cz> Fernando Perez wrote: > Done, both for pyrex and swig. I added very simple notes, feel free to > add further detail as necessary: > > http://scipy.org/Cookbook/Pyrex_and_NumPy > http://scipy.org/Cookbook/SWIG_and_NumPy Thanks again, I have made just a little tweak to the SWIG_and_NumPy page. r. From Fernando.Perez at colorado.edu Thu Mar 23 06:27:07 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Mar 23 06:27:07 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <44220001.4050606@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> Message-ID: <4422AFF9.5000200@colorado.edu> Tim Hochberg wrote: > My opinion is that all methods and functions should either: > > 1. Always return a copy. > 2. Always return a view > 3. Return a view if possible otherwise raise an exception. Well, but is copy/view the /only/ invariant worth guaranteeing? I think there is a valid need for functions which ensure other invariants, such as contiguity. There are applications (such as passing pointers to C/Fortran libraries which don't have striding mechanisms but will not modify their inputs) which require contiguous inputs, but where one would rather make a copy only if necessary. My take on this is that we should /document/ clearly what invariants any given function satisfies, but I think the 'always view/always copy' view excludes an important usage case. There may be others beyond contiguity, but that's the one that pops immediately to mind. Cheers, f From bsouthey at gmail.com Thu Mar 23 06:34:11 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Mar 23 06:34:11 2006 Subject: [Numpy-discussion] Re: Numpy-discussion digest, Vol 1 #1632 - 10 msgs In-Reply-To: <200603222035.08682.pierregm@engr.uga.edu> References: <20060322234311.30FA78974B@sc8-sf-spam1.sourceforge.net> <200603222035.08682.pierregm@engr.uga.edu> Message-ID: Hi, I think that there are different behaviors being obtained. One occurs when a scalar value is used and another occurs when an array is being used. In the first is it just an extension of masking the whole array twice with two different values. In the second case it seems more of an element by element masking with two different values, which requires creating min and max arrays. There is also a third case of a single dimension so min and max vectors to operate on that dimension. In anycase, the values should only be masked because it allows the values to be unmasked as necessary. In which case there needs to be an appropriate unmask function such that in the extreme one of limits could be removed. Regards Bruce On 3/22/06, Pierre GM wrote: > > Pierre has implemented the clip method for ma.array, but his > > implementation does not seem to do the right thing if min or max is > > masked: > > From core/oldnumeric.py: > """clip(m, m_min, m_max) = every entry in m that is less than m_min is > replaced by m_min, and every entry greater than m_max is replaced by > m_max. > """ > My understanding was that m_min and m_max were two floats, not two arrays. I > completely oversaw that. Your suggestion (masking where the limits are > masked) makes complete sense. I'll work on that. > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From tim.hochberg at cox.net Thu Mar 23 08:03:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 08:03:07 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44223E4A.9090208@colorado.edu> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220366.4000004@cox.net> <44223E4A.9090208@colorado.edu> Message-ID: <4422C668.8020802@cox.net> Fernando Perez wrote: > Tim Hochberg wrote: > >> This did fix things...eventually. I removed the build directory and >> rebuilt. Numpy still bombed the interpreter, although in a new and >> less exciting way. I then did a clean checkout from svn and rebuilt. >> Same thing. Then I manally removed numpy from site-packages and >> reinstalled at which points things worked and all tests passed. >> >> So in addition to needing to remove the build directory, it appears >> that something was hanging out in the destination directory that did >> not get overwritten and caused problems. >> >> Perhaps this description will help the next person who runs into >> this, but probably not. > > > Well, the script that I posted does precisely all of this in a > mindless fashion, So it does. I should have looked at that more closely. > which is why I find it useful. You may want to give it a try (it's > trivial, but it just ensures you don't forget these steps and end up > wasting any time chasing the problem). Well, I'd probably have to rewrite it to get it work under windows. I may do that, although 'rm -rf' always scares me (even though it looks pretty safe here). Still, it would be nice if 'setup.py clean' and 'setup.py install' did this automagically. -tim > > Cheers, > > f > > From tim.hochberg at cox.net Thu Mar 23 08:12:09 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 08:12:09 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422AFF9.5000200@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> Message-ID: <4422C8A8.2050704@cox.net> Fernando Perez wrote: > Tim Hochberg wrote: > >> My opinion is that all methods and functions should either: >> >> 1. Always return a copy. >> 2. Always return a view >> 3. Return a view if possible otherwise raise an exception. > > > Well, but is copy/view the /only/ invariant worth guaranteeing? I > think there is a valid need for functions which ensure other > invariants, such as contiguity. There are applications (such as > passing pointers to C/Fortran libraries which don't have striding > mechanisms but will not modify their inputs) which require contiguous > inputs, but where one would rather make a copy only if necessary. This is a different case, I think. The result of this copy is not user visible except in terms of performance. I'm only concerned with functions that *return* copies or views depending on the input. I don't care if a function sometimes makes a copy under the covers but doesn't return it. > My take on this is that we should /document/ clearly what invariants > any given function satisfies, but I think the 'always view/always > copy' view excludes an important usage case. There may be others > beyond contiguity, but that's the one that pops immediately to mind. I don't think we're in disagreement here although I'm not sure. I will add, on the subject of continuity, that I think there should be a function 'ascontiguous' that parallels asarray, but assures that the result is contiguous. Although this sometimes returns a copy, I thinks that's OK since that's it's job. I would like to see all of the implicit copying pushed into functions like asarray and ascontiguous. This also helps efficiency. Imagine I have some calls to functions that require contiguous arrays and do copies under the covers if their args are not contiguous. In that case: a = ascontiguous(a) param1 = computeSomethingOnContiguousData(a) param2 = computeSomethingElseOnContiguousData(a) # etc. Will be much more efficient than the equivalent code without the ascontiguous when the initial a value is not discontiguous. Regards, -tim From robert.kern at gmail.com Thu Mar 23 08:25:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 23 08:25:11 2006 Subject: [Numpy-discussion] Re: [Numpy-user] random.randint bug ? In-Reply-To: References: Message-ID: <4422CBD1.7020606@gmail.com> Randewijk P-J wrote: > I'm having problems with randint: > > >>>>random.randint(low=1,high=10) > > 4L > >>>>random.randint(low=1,high=10) > > 7L > >>>>random.randint(low=1,high=10) > > 5L > >>>>random.randint(low=1,high=10) > > 7L > >>>>random.randint(low=1,high=10) > > 4L > >>>>random.randint(low=1,high=10) > > 0L > > The last value is wrong because it is < "low"/"1" > > The randint value should be: 1 <= value < 10 > >>>>help(random.randint) > > Help on built-in function randint: > > randint(...) > Return random integers x such that low <= x < high. > > randint(low, high=None, size=None) -> random values > > If high is None, then 0 <= x < low. > > > > Running Python 2.4.2 and NumPy 0.9.6 Fixed in SVN. Thank you. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Fernando.Perez at colorado.edu Thu Mar 23 09:02:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Mar 23 09:02:06 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422C8A8.2050704@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> Message-ID: <4422D46D.8080800@colorado.edu> Tim Hochberg wrote: > I don't think we're in disagreement here although I'm not sure. > > I will add, on the subject of continuity, that I think there should be a > function 'ascontiguous' that parallels asarray, but assures that the > result is contiguous. Although this sometimes returns a copy, I thinks > that's OK since that's it's job. I would like to see all of the implicit > copying pushed into functions like asarray and ascontiguous. I think we agree: something like ascontiguous() is precisely what I had in mind (I think that's what ravel() does today, but I'm fine if it gets a new name, as long as the functionality exists). Obviously a function like this should explicitly (docstring) say that it does NOT make any guarantees about whether its return value is a view or a copy, just that it's contiguous. Cheers, f From mfmorss at aep.com Thu Mar 23 09:05:03 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Thu Mar 23 09:05:03 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <442210A7.7080802@ieee.org> Message-ID: Oh for the days of imperative programming! y=f(x,...) x=f(x,...) was just too easy, I guess. Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To Sent by: numpy-discussion numpy-discussion- eforge.net cc Subject 03/22/2006 10:06 Re: [Numpy-discussion] Current SVN PM segfaults under VC7 Tim Hochberg wrote: > > OK, now that I've got things working, I can report a real segfault > under VC7. The following snippet of code: > > from numpy import * > a = arange(100.) > b = array(1+1j) > (a+1) ** (b+3) I can reproduce it on Linux. It looks like it's related to the power optimization stuff that was recently checked in. power(a,b+3) does not crash but >>> a**(b+3) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076248704 (LWP 1545)] 0x404162df in array_power_is_scalar (o2=0x8294200, exp=0xbfffec68) at arrayobject.c:2656 2656 arrayobject.c: No such file or directory. in arrayobject.c -Travis ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From tim.hochberg at cox.net Thu Mar 23 09:37:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 09:37:04 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422D46D.8080800@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> Message-ID: <4422DC96.9030307@cox.net> Fernando Perez wrote: > Tim Hochberg wrote: > >> I don't think we're in disagreement here although I'm not sure. >> >> I will add, on the subject of continuity, that I think there should >> be a function 'ascontiguous' that parallels asarray, but assures that >> the result is contiguous. Although this sometimes returns a copy, I >> thinks that's OK since that's it's job. I would like to see all of >> the implicit copying pushed into functions like asarray and >> ascontiguous. > > > I think we agree: something like ascontiguous() is precisely what I > had in mind (I think that's what ravel() does today, but I'm fine if > it gets a new name, as long as the functionality exists). a.ravel() seems to be equivalent to reshape(a, [-1]). That is, it returns a flattened, contiguous copy. ascontiguous(a) would be slightly different in that it would preserves the shape of a. In fact I think it would look a lot like: def ascontiguous(a): """ascontiguous(a) -> contiguous representation of a. If 'a' is allready contiguous, it is returned unchanged. Otherwise, a contiguous copy is returned. """ a = asarray(a) if not a.flags['CONTIGUOUS']: a = array(a) return a > Obviously a function like this should explicitly (docstring) say that > it does NOT make any guarantees about whether its return value is a > view or a copy, just that it's contiguous. I agree. Regards, -tim From Chris.Barker at noaa.gov Thu Mar 23 09:59:04 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 09:59:04 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4421F455.2020003@colorado.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> <4421F455.2020003@colorado.edu> Message-ID: <4422E1CA.70202@noaa.gov> Fernando Perez wrote: > Done, both for pyrex and swig. I added very simple notes, feel free to > add further detail as necessary: > > http://scipy.org/Cookbook/Pyrex_and_NumPy > http://scipy.org/Cookbook/SWIG_and_NumPy Thanks, this is great. Has anyone done this (for either SWIG or Pyrex) for the generic array interface, rather than explicitly for the numpy types? I'd love to see examples of how to write extensions that can take ANY nd-array. (I now that's only NumPy, Numeric and numarray now, but I'm hopeful it will become a standard) -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Fernando.Perez at colorado.edu Thu Mar 23 10:07:04 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Mar 23 10:07:04 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422DC96.9030307@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> Message-ID: <4422E397.6030206@colorado.edu> Tim Hochberg wrote: > a.ravel() seems to be equivalent to reshape(a, [-1]). That is, it > returns a flattened, contiguous copy. ascontiguous(a) would be slightly > different in that it would preserves the shape of a. In fact I think it > would look a lot like: > > def ascontiguous(a): > """ascontiguous(a) -> contiguous representation of a. > > If 'a' is allready contiguous, it is returned unchanged. Otherwise, > a contiguous copy > is returned. > > """ > a = asarray(a) > if not a.flags['CONTIGUOUS']: > a = array(a) > return a Yup, sorry: I've been thinking too much in terms of raw C (double*) pointers so to me, these days everything is 1-d :) I forgot about shape issues... But I think we agree on the basic point. Cheers, f From faltet at carabos.com Thu Mar 23 10:17:12 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Mar 23 10:17:12 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422E1CA.70202@noaa.gov> References: <4414BDEF.70001@colorado.edu> <4421F455.2020003@colorado.edu> <4422E1CA.70202@noaa.gov> Message-ID: <200603231916.41334.faltet@carabos.com> A Dijous 23 Mar? 2006 18:58, Christopher Barker va escriure: > > http://scipy.org/Cookbook/Pyrex_and_NumPy > > http://scipy.org/Cookbook/SWIG_and_NumPy > Has anyone done this (for either SWIG or Pyrex) for the generic array > interface, rather than explicitly for the numpy types? I'd love to see > examples of how to write extensions that can take ANY nd-array. (I now > that's only NumPy, Numeric and numarray now, but I'm hopeful it will > become a standard) I don't think this is feasible yet, although I don't think this is a big limitation. You can always compile your program against NumPy or numarray or Numeric and follow the previous recipes (adapted to the package of choice) to access to its data. The only thing that you must be aware of is to convert all the objects that foes to the extension to an object of the package of choice. For example, if you have chosen NumPy as your package to compile with, then you only have to convert Numeric/numarray objects into a NumPy one with: native_numpy_object = numpy.asarray(your_foreign_object) The array protocol will ensure that the conversion will be as efficient as possible. What's more, if the "foreign" objects are well-behaved (i.e. contiguous and native-endian ordered), then a data copy will not be made, so letting you to access to the original data address. PyTables follows this approach and I must say that's very handy. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant at ee.byu.edu Thu Mar 23 10:36:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 10:36:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422E1CA.70202@noaa.gov> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> <4421F455.2020003@colorado.edu> <4422E1CA.70202@noaa.gov> Message-ID: <4422EA6E.1070401@ee.byu.edu> Christopher Barker wrote: > Fernando Perez wrote: > >> Done, both for pyrex and swig. I added very simple notes, feel free >> to add further detail as necessary: >> >> http://scipy.org/Cookbook/Pyrex_and_NumPy >> http://scipy.org/Cookbook/SWIG_and_NumPy > > > Thanks, this is great. > > Has anyone done this (for either SWIG or Pyrex) for the generic array > interface, rather than explicitly for the numpy types? I'd love to see > examples of how to write extensions that can take ANY nd-array. (I now > that's only NumPy, Numeric and numarray now, but I'm hopeful it will > become a standard) Well, there is the __array_struct__ method that should return a C-Object pointer with all the information in it about the array. To see how to use this look at the Numeric or numpy code (do a grep on __array_struct__ and you will find the relevant sections of code). I noticed that the Pyrex cookbook had an example with using the generic interface. -Travis From rowen at cesmail.net Thu Mar 23 10:37:08 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Thu Mar 23 10:37:08 2006 Subject: [Numpy-discussion] Re: Problems installing NumPy References: <4E0ECB41-EF32-4F0E-9DAE-52B55C8AE3A0@mac.com> Message-ID: In article <4E0ECB41-EF32-4F0E-9DAE-52B55C8AE3A0 at mac.com>, Tommy Grav wrote: > I am new to Python and just downloaded ActivePython 2.4.2.10 on my > Mac PPC with OS X 10.4. > I added the numpy package (0.9.6-py2.4) and it imports fine. But when > I try pydoc random I > get this error > > [tgrav at skathi] site-packages/numpy -> pydoc random > import core -> failed: dlopen(/Library/Frameworks/Python.framework/ > Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, > 2): Symbol not found: _PyOS_ascii_strtod > Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ > lib/python2.4/site-packages/numpy/core/multiarray.so > Expected in: dynamic lookup >... > Anyone know what is causing this problem? Is there something in my > installation > of either ActivePython or NumPy that needs to be fixed? I'm not entirely clear on how what you're typing to get it to fail, but perhaps there is a problem in the package installer. If you suspect this, you can easily install numpy from source. -- Russell From tim.hochberg at cox.net Thu Mar 23 11:03:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 11:03:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4422E397.6030206@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> Message-ID: <4422F0A4.7010206@cox.net> I was just looking at the interface for array and asarray to see what other stuff should go in the interface of the hypothetical ascontiguous. There's 'dtype', which I knew about, and 'fortran', which I didn't, but which makes sense. However, there's also 'ndmin'. First off, it's not described in docstring for asarray, but I was able to find it in the docstring for array without a problem. Second, is it really necessary? It seems to be useful in an awfully narrow set of circumstances, particularly since when you are padding axes not everyone wants to pad to the left. It would seem to be more useful to ditch the ndmin and have some sort of paddims function that was more full featured (padding to either the left or the right at a minimum). I'm not entirely sure what the best interface to such a beast would look like, but a simple tactic would be to just provide leftpaddims and rightpaddims. If it's not allready clear by now (;), I prefer several narrow interfaces to one broad one. -tim From Chris.Barker at noaa.gov Thu Mar 23 11:26:16 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 11:26:16 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422AFF9.5000200@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> Message-ID: <4422F602.1060809@noaa.gov> Fernando Perez wrote: > Well, but is copy/view the /only/ invariant worth guaranteeing? I think > there is a valid need for functions which ensure other invariants, such > as contiguity. quite true. > My take on this is that we should /document/ clearly what invariants any > given function satisfies, but I think the 'always view/always copy' view > excludes an important usage case. There may be others beyond > contiguity, but that's the one that pops immediately to mind. there's asarray() of course. My feeling is that functions that may or may not return a copy should be functions, like asarray(), that ONLY exist to ensure a particular invariant. ascontiguous() asarray() I imagine there are others. What concerns me is functions like reshape() and ravel() that you might have all sorts of other reasons to use, but then can't ever know for sure if your method is going to be working with a copy or not. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Mar 23 11:32:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 11:32:06 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422F602.1060809@noaa.gov> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422F602.1060809@noaa.gov> Message-ID: <4422F79E.1040401@ee.byu.edu> There is the flatten() method which exists for precisely this reason (it *always* returns a copy). -Travis From Chris.Barker at noaa.gov Thu Mar 23 11:41:04 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 11:41:04 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <200603231916.41334.faltet@carabos.com> References: <4414BDEF.70001@colorado.edu> <4421F455.2020003@colorado.edu> <4422E1CA.70202@noaa.gov> <200603231916.41334.faltet@carabos.com> Message-ID: <4422F996.3070303@noaa.gov> Francesc Altet wrote: > A Dijous 23 Mar? 2006 18:58, Christopher Barker va escriure: >> Has anyone done this (for either SWIG or Pyrex) for the generic array >> interface, rather than explicitly for the numpy types? > I don't think this is feasible yet, um, why not? > although I don't think this is a > big limitation. You can always compile your program against NumPy or > numarray or Numeric The goal here is to be able to write code that can be compiled without the presence of numpy, or Numeric, or numarray. Indeed, no additional dependencies, but that can still efficiently use array objects passed in. My classic example is wxPython. Robin doesn't want any dependencies on other packages, so at the moment, when you pass a numpy array in, it uses python's generic sequence access to get the values, then ends up converting them to a plain old C array. This is a lot of unnecessary processing, as you can imagine. I think it can be done, and one of these days I'll figure out how, but I'd love to see an example from someone who understands these things more than myself. Travis Oliphant wrote: > Well, there is the __array_struct__ method that should return a C-Object > pointer with all the information in it about the array. > To see how to use this look at the Numeric or numpy code (do a grep on > __array_struct__ and you will find the relevant sections of code). That's exactly what I mean. Now I just need to figure out how to use it! > I noticed that the Pyrex cookbook had an example with using the generic > interface. Wonderful, I'll go look there. Do you mean the Pyrex section in the SciPy cookbook? I can't find a Pyrex cookbook. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at carabos.com Thu Mar 23 11:43:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Mar 23 11:43:06 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422EA6E.1070401@ee.byu.edu> References: <4414BDEF.70001@colorado.edu> <4422E1CA.70202@noaa.gov> <4422EA6E.1070401@ee.byu.edu> Message-ID: <200603232042.38896.faltet@carabos.com> A Dijous 23 Mar? 2006 19:35, Travis Oliphant va escriure: > Well, there is the __array_struct__ method that should return a C-Object > pointer with all the information in it about the array. > > To see how to use this look at the Numeric or numpy code (do a grep on > __array_struct__ and you will find the relevant sections of code). > > I noticed that the Pyrex cookbook had an example with using the generic > interface. I've just noticed it. That's great, but I don't understand the code quite well. In particular, if, as the introduction says, this offers "a view of the data without actually copying the data itself", then why the next line is needed?: self.data=malloc( bufsize ) Oh, well, I think I've to study more the example. Thanks! -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Thu Mar 23 11:48:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Mar 23 11:48:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422F996.3070303@noaa.gov> References: <4414BDEF.70001@colorado.edu> <200603231916.41334.faltet@carabos.com> <4422F996.3070303@noaa.gov> Message-ID: <200603232047.00745.faltet@carabos.com> A Dijous 23 Mar? 2006 20:40, Christopher Barker va escriure: > > I don't think this is feasible yet, > > um, why not? For no particular reason, except my ignorance ;-) > The goal here is to be able to write code that can be compiled without > the presence of numpy, or Numeric, or numarray. Indeed, no additional > dependencies, but that can still efficiently use array objects passed in. > > My classic example is wxPython. Robin doesn't want any dependencies on > other packages, so at the moment, when you pass a numpy array in, it > uses python's generic sequence access to get the values, then ends up > converting them to a plain old C array. This is a lot of unnecessary > processing, as you can imagine. > > I think it can be done, and one of these days I'll figure out how, but > I'd love to see an example from someone who understands these things > more than myself. Yeah, I understand you perfectly well :-) > Do you mean the Pyrex section in the SciPy cookbook? I can't find a > Pyrex cookbook. I think the referred page is here: http://scipy.org/Cookbook/ArrayStruct_and_Pyrex -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From strawman at astraw.com Thu Mar 23 12:06:02 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu Mar 23 12:06:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <200603232042.38896.faltet@carabos.com> References: <4414BDEF.70001@colorado.edu> <4422E1CA.70202@noaa.gov> <4422EA6E.1070401@ee.byu.edu> <200603232042.38896.faltet@carabos.com> Message-ID: <4422FF72.7060208@astraw.com> This example makes an __array_struct__ view of data you malloc elsewhere. In other words, it's only half of the full suite of what people may want to do. It covers elsewhere -> numpy/numarray/Numeric/__array_struct__ consumer. It does not (yet) cover numpy/__array_struct__ producer -> elsewhere. This would be a great addition, but nontrivial to cover the full spectrum of memory allocation strategies that numpy currently provides. A first step might be to support only a limited set and raise a NotImplementedError if something more complicated is encountered. I've actually been quite happy in my code doing something like allocating my own memory layouts, passing a view of them to numpy using this interface, doing any manipulation there, and then continuing on in my own code. This could, for example, be used to make a numpy view of a wx-allocated buffer... FWIW, I'm using Intel's IPP to both allocate buffers and further manipulate them, with my Python interface significantly eased by a numpy view of the data. But, like I say, I understand this is only half the solution... The wiki is open for submissions! Cheers! Andrew Francesc Altet wrote: >I've just noticed it. That's great, but I don't understand the code >quite well. In particular, if, as the introduction says, this offers >"a view of the data without actually copying the data itself", then >why the next line is needed?: > > self.data=malloc( bufsize ) > >Oh, well, I think I've to study more the example. > >Thanks! > > > From Chris.Barker at noaa.gov Thu Mar 23 12:09:05 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 12:09:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <200603232047.00745.faltet@carabos.com> References: <4414BDEF.70001@colorado.edu> <200603231916.41334.faltet@carabos.com> <4422F996.3070303@noaa.gov> <200603232047.00745.faltet@carabos.com> Message-ID: <4423003E.9@noaa.gov> Francesc Altet wrote: > I think the referred page is here: > > http://scipy.org/Cookbook/ArrayStruct_and_Pyrex Yes, I did find that. That looks like it creates an Array_Struct, for use with other code, rather than using one that is already created. Hence the malloc. It look like there is the intent to add what I am looking for: """ Using data malloced elsewhere with a Pyrex-based extension Coming soon... """ Is it soon yet Andrew? ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jimpy at oublic.org Thu Mar 23 12:12:02 2006 From: jimpy at oublic.org (Jim Carroll) Date: Thu Mar 23 12:12:02 2006 Subject: [Numpy-discussion] Newbie: Matrix form of segment-sphere intersection? Message-ID: Hi, I'm considering using matricies for some math that determines where segments intersect spheres. I have pairs of 3d coordinates P1, P2; P3, P4; P5, P6; ... Pn-1, Pn. one is inside a sphere, and the other is outside the same sphere. I want the point where segment joining the point intersects the sphere. The math is straightforward enough done in a function (for example: http://www.siggraph.org/education/materials/HyperGraph/raytrace/rtinter1.htm) Are matricies appropriate as a way of applying this sort of computation to a large number of points? Where can I find applicable tutorials or recipies? Thanks, -Jim From oliphant at ee.byu.edu Thu Mar 23 12:50:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 12:50:07 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4422F0A4.7010206@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> Message-ID: <442309CE.1090005@ee.byu.edu> Tim Hochberg wrote: > > > I was just looking at the interface for array and asarray to see what > other stuff should go in the interface of the hypothetical > ascontiguous. There's 'dtype', which I knew about, and 'fortran', > which I didn't, but which makes sense. However, there's also 'ndmin'. > First off, it's not described in docstring for asarray, but I was able > to find it in the docstring for array without a problem. Second, is it > really necessary? It seems to be useful in an awfully narrow set of > circumstances, particularly since when you are padding axes not > everyone wants to pad to the left. Padding to the left is "default" behavior for broadcasting and so it seems appropriate. This is how all lower-dimensional arrays are interpreted as "higher" dimensional arrays throughout the code. The ndmin is very handy as atested to by the uses of atleast_1d or atleast_2d in numpy library code. It was added later as an optimization step because of the number of library routines that were using it. I've since used it several times to simplify code. I think an ascontinguous on the Python level is appropriate since such a beast exists on the C-level. On the other hand, while Tim prefers narrow interfaces, the array_from_object interface is traditionally pretty broad. Thus, in my mind, the array call should get another flag keyword that forces a contiguous result. This translates easily to the C-domain, in much the same way as the fortran keyword does. -Travis From tim.hochberg at cox.net Thu Mar 23 13:47:08 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 13:47:08 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442309CE.1090005@ee.byu.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <44231741.3020308@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> >> I was just looking at the interface for array and asarray to see what >> other stuff should go in the interface of the hypothetical >> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >> which I didn't, but which makes sense. However, there's also 'ndmin'. >> First off, it's not described in docstring for asarray, but I was >> able to find it in the docstring for array without a problem. Second, >> is it really necessary? It seems to be useful in an awfully narrow >> set of circumstances, particularly since when you are padding axes >> not everyone wants to pad to the left. > > > > Padding to the left is "default" behavior for broadcasting and so it > seems appropriate. This is how all lower-dimensional arrays are > interpreted as "higher" dimensional arrays throughout the code. That makes some sense. > > The ndmin is very handy as atested to by the uses of atleast_1d or > atleast_2d in numpy library code. It was added later as an > optimization step because of the number of library routines that were > using it. I've since used it several times to simplify code. OK, I'll take your word for it. > I think an ascontinguous on the Python level is appropriate since such > a beast exists on the C-level. On the other hand, while Tim prefers > narrow interfaces, the array_from_object interface is traditionally > pretty broad. Thus, in my mind, the array call should get another flag > keyword that forces a contiguous result. This doesn't bother me since I long ago gave up any hope that the array constructor would have a narrow interface. > This translates easily to the C-domain, in much the same way as the > fortran keyword does. I'll buy that. While I accept array() needs a wide interface, I still prefer to keep as many other interfaces as possible narrow. In particular, is ndmin widely used in asarray? Or do the library routines generally use array instead. Given the choice I'd sweep as much of the dust, AKA wideness, into array90 as possible since that's irredeemably wide anyway and keep the other interfaces as narrowly focused as possible. Put another way, asarray and ascontiguous are about clarity of intent. With too much extra baggage, the intent becomes obscured. The coupling seems tight enough for dtype and fortran, but once you get to ndmin, it seems that you might as well go with the big guns and break out "array(x, copy=False, ndmin=n)". That's my $0.02 on this subject and I'll shut up about it now. -tim From travis at enthought.com Thu Mar 23 13:57:02 2006 From: travis at enthought.com (Travis N. Vaught) Date: Thu Mar 23 13:57:02 2006 Subject: [Numpy-discussion] ANN: Python Enthought Edition Version 0.9.3 Released Message-ID: <442319A6.7040108@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 0.9.3 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 0.9.3 Release Notes: -------------------- Version 0.9.3 of Python Enthought Edition includes an update to version 1.0.3 of the Enthought Tool Suite (ETS) Package-- you can look at the release notes for this ETS version here. Other major changes include: * upgrade to VTK 5.0 * addition of docutils * addition of numarray * addition of pysvn. Also, MayaVi issues should be fixed in this release. Full Release Notes are here: http://code.enthought.com/release/changelog-enthon0.9.3.shtml About Python Enthought Edition: ------------------------------- Python 2.3.5, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numeric SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com From oliphant at ee.byu.edu Thu Mar 23 13:59:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 13:59:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <44231A04.2000804@ee.byu.edu> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>Thus, in my mind, the array call should get another flag keyword that >>forces a contiguous result. >> >> > >Please don't! The fortran flag is bad enough, but has too much history >behind it. Let's not breed boolean parameters. Sooner or later >someone will use keword arguments positionally and you will end up >guessing what > > There are several boolean flags in the interface already. Adding another one won't change the current situation that you describe. There are several ways to handle this. For one, we could force the use of keyword arguments, so that the position problem does not arise. Sasha has mentioned in the past a strides array argument, but I think the default fortran and contiguous strides cases need better support then just one of many possible stridings so I wouldn't go that direction here. I'm debating whether or not the fortran flag should be used to specify both contiguous and fortran cases. Right now, the fortran argument is a three-case flag with dont-care, True, and False arguments. It seems natural to have True mean force-fortran and False mean force-contiguous with dont-care (the default) mean take an array already given (or create a C-contiguous array if we are generating a new array from another object). At any rate, if the fortran flag is there, we need to specify the contiguous case as well. So, either propose a better interface (we could change it still --- the fortran flag doesn't have that much history) to handle the situation or accept what I do ;-) -Travis From ndarray at mac.com Thu Mar 23 14:22:04 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 23 14:22:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442309CE.1090005@ee.byu.edu> References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: On 3/23/06, Travis Oliphant wrote: > Thus, in my mind, the array call should get another flag keyword that > forces a contiguous result. Please don't! The fortran flag is bad enough, but has too much history behind it. Let's not breed boolean parameters. Sooner or later someone will use keword arguments positionally and you will end up guessing what array([1,2], int8_, 1, 1, 0, 0) means. From oliphant at ee.byu.edu Thu Mar 23 14:27:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 14:27:05 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44231741.3020308@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231741.3020308@cox.net> Message-ID: <44232098.7020901@ee.byu.edu> Tim Hochberg wrote: > > That makes some sense. > >> >> The ndmin is very handy as atested to by the uses of atleast_1d or >> atleast_2d in numpy library code. It was added later as an >> optimization step because of the number of library routines that were >> using it. I've since used it several times to simplify code. > > > OK, I'll take your word for it. Not necessarily sane advice :-) --- I might be overstating things. I know that atleast_1d and atleast_2d are used all over the place in scipy. This makes sense and I can certainly understand it. I'm willing to modify things to give narrow interfaces. Right now, since the requesting both fortran and contiguous does not make sense, setting the fortran flag to false enforces C-style contiguous while setting it to True enforces fortran-style. Setting it to None (the default) specifies you don't care and the behavior will be to create C-style contiguous for new arrays and use the striding specified by the array if it's already an array. I admit that it is arguable whether or not the fortran flag should be overloaded like this or not. There are now ascontiguous and asfortran functions with fairly minimal interfaces to make it simpler. There is also a check to make sure that array is called with no more than 2 non keyword arguments. Thus, you won't be able to confuse which flag is which. -Travis From Norbert.Nemec.list at gmx.de Thu Mar 23 15:04:04 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu Mar 23 15:04:04 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's In-Reply-To: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> References: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> Message-ID: <44232949.1000606@gmx.de> The current SVN version of numpy.linalg does not contain "eigenvectors" any more (since the cleanup - patches I submitted last week), so you are obviously using something older. In any case, numpy.linalg.eig (and eigvals) do not seem to contain any check against NaNs, which probably is a bad idea. scipy.linalg.eig does contain such a check, but that module is internally very different to that in numpy. numpy.linalg certainly needs more cleanup. In that course, adding safety-checks certainly is one task. Chris Fonnesbeck wrote: > If linalg.eigenvectors is called on a matrix that contains nan values, > it hangs, quickly consuming system resources. Here is the matrix: > > matrix = [[ 0. 0. 0. 0. > 0.36658624 0. 0. > 0. 0. 0. ] > [ 0.89827259 0. 0. 0. 0. > 0. 0. > 0. 0. 0. ] > [ 0. 0.92510948 0. 0. 0. > 0. 0. > 0. 0. 0. ] > [ 0. 0. nan nan 0. > 0. 0. > 0. 0. 0. ] > [ 0. 0. nan nan 0. nan > 0. 0. 0. 0. ] > [ 0. 0. 0. 0. 0.93319244 nan > 0. 0. 0. 0. ] > [ 0. 0. 0. 0. 0.36658624 > 0. 0. > 0. 0. 0. ] > [ 0. 0. 0. 0. 0. 0. > 0.89827259 0. 0. 0. ] > [ 0. 0. 0. 0. 0. > 0. 0. > 0.92510948 0. 0. ] > [ 0. 0. 0. 0. 0. > 0. 0. > 0. 0.92510948 0.93319244]] > > I would expect an exception, but I get none. > > Using a recent cvs build of numpy. > > C. > > -- > Chris Fonnesbeck + Atlanta, GA + http://trichech.us From tim.hochberg at cox.net Thu Mar 23 15:07:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 15:07:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <442329E4.70007@cox.net> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>Thus, in my mind, the array call should get another flag keyword that >>forces a contiguous result. >> >> > >Please don't! The fortran flag is bad enough, but has too much history >behind it. Let's not breed boolean parameters. Sooner or later >someone will use keword arguments positionally and you will end up >guessing what > >array([1,2], int8_, 1, 1, 0, 0) > >means. > > We could always force keyword parameter by using **kwd. That would mess mess up help() and other introspection tools. Then again, if we want to discourage overuse of this sort of stuff, perhaps that's not a bad thing ;) But if we really need access to the C guts of an array, just allow a dictionary of flags to get plugged in. These would be the same as what somearray.flags puts out. The new interface for array would be: array(object, **flags) Where flags could contain: CONTIGUOUS : force a copy if object isn't contiguous if True. [default None] FORTRAN : force array to be fortran order if True, C order if False [default None] OWNDATA : force a copy if True [default True] WRITEABLE : Force a copy if object isn't writeable if Tru [default None] ALIGNED : Force a copy if object isn't aligned if True. [default None] UPDATEIFCOPY : Set the UPDATEIFCOPY flag? [default ???] With the exception of the FORTRAN, and possibly UPDATEIFCOPY, it would be an error to set any of these flags to False (Forcing an array to be discontiguous for instance, makes no sense). That's a thin interface and it ties together with the flags parameter nicely. On the downside, it's a little weird, particularly using OWNDATA for copy, although it is logical once you think about it. It also drops 'minnd' and 'subok'. I wouldn't miss them, but I expect someone would squawk. You could shoehorn them into flags, but then you lose one of the chief virtues of this scheme, which is that it makes a strong connection between the constructor and the flags parameter. 'subok' should be pretty well taken care of by 'asanyarray' and it would be easy enough to create an auxilliary function to replicate the 'minnd' functionality. With this kind of narrow interface, it might make sense to allow the flags parameter on all of the asX functions. That makes for a nice uniform, easy to remember, interface. I'm not positive I like this idea yet, but I thought it was interesting enough to throw into the ring anyway. Tangentially, I noticed that I can set ALIGNED to False. Isn't that going to break something somewhere? Should the ALIGNED flag be readonly? -tim From michael.sorich at gmail.com Thu Mar 23 15:27:11 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Thu Mar 23 15:27:11 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: References: Message-ID: <16761e100603231526j2a318cc8v877ff0821b76d099@mail.gmail.com> On 3/23/06, Sasha wrote: > > In an ideal world, any function that accepts ndarray would accept > ma.array and vice versa. Moreover, if the ma.array has no masked > elements and the same data as ndarray, the result should be the same. > Obviously current implementation falls short of this goal, but there > is one feature that seems to make this goal unachievable. I am just starting to use ma.array and would like to get some idea from those in the know of how close this is to reality. What percentage of functions designed for nd_arrays would work on a ma.array with no masked elements? That is if you have data with missing values, but then remove the missing values, is it necessary to convert back to a standard nd_array? The statistical language R deals with missing data fairly well. There are a number of functions for dealing with missing values (fail, omit, exclude, pass). Furthermore, there is a relatively standard way for a function to handle data with missing values, via an na.action parameter which indicates which function to call. http://spider.stat.umn.edu/R/library/stats/html/na.action.html http://spider.stat.umn.edu/R/library/stats/html/na.fail.html It would be nice to have a similar set of functions (including the fill function) for numpy. These functions could return the object without change if it is not a masked array, and if a masked array make the appropriate changes to return a nd_array or raise exception. A simple standard for indicating a function's the ability to handle masked data would be to include a mask_action parameter which holds or indicates a function for processing missing data. Also, are there any current plans to allow record type arrays to be masked? Thanks, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Mar 23 15:42:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 15:42:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442329E4.70007@cox.net> References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <442329E4.70007@cox.net> Message-ID: <4423321A.3080808@ee.byu.edu> Tim Hochberg wrote: > > Tangentially, I noticed that I can set ALIGNED to False. Isn't that > going to break something somewhere? Should the ALIGNED flag be readonly? Responding to the tangential topic :-) The ALIGNED flag can be set False because it allows one to test those sections of code that deal with misaligned data. I don't think it would break anything, because thinking that data is misaligned when it really is aligned only costs you in copy time. You can't set it TRUE if it's really mis-aligned, though, because thinking that data is aligned when it's really mis-aligned can cause segfaults on some platforms and slow code on others. -Travis From ndarray at mac.com Thu Mar 23 17:31:16 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 23 17:31:16 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44231A04.2000804@ee.byu.edu> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> Message-ID: On 3/23/06, Travis Oliphant wrote: > At any rate, if the fortran flag is there, we need to specify the > contiguous case as well. So, either propose a better interface (we > could change it still --- the fortran flag doesn't have that much > history) to handle the situation or accept what I do ;-) Let me try. I propose to eliminate the fortran flag in favor of a more general "strides" argument. This argument can be either a sequence of integers that becomes the strides, or a callable object that takes shape and dtype arguments and return a sequence that becomes the strides. For fortran and c order functions that generate appropriate stride sequences should be predefined to enable array(..., strides=fortran, ...) and array(..., strides=contiguous). From oliphant.travis at ieee.org Thu Mar 23 21:15:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 23 21:15:04 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's In-Reply-To: <44232949.1000606@gmx.de> References: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> <44232949.1000606@gmx.de> Message-ID: <4423802D.2080505@ieee.org> Norbert Nemec wrote: > The current SVN version of numpy.linalg does not contain "eigenvectors" > any more (since the cleanup - patches I submitted last week), so you are > obviously using something older. > > In any case, numpy.linalg.eig (and eigvals) do not seem to contain any > check against NaNs, which probably is a bad idea. scipy.linalg.eig does > contain such a check, but that module is internally very different to > that in numpy. > > numpy.linalg certainly needs more cleanup. In that course, adding > safety-checks certainly is one task. > Which routines don't take nans and on which platforms? It would be nice to know before checking for nans on every array. My platform (linux AMD 32-bit) seems to handle nan's fine in the linalg.eig code. Perhaps we do need to check every array for nans, but it would be nice if we could isolate it to certain platforms. -Travis From aisaac at american.edu Thu Mar 23 21:55:04 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu Mar 23 21:55:04 2006 Subject: [Numpy-discussion] Newbie: Matrix form of segment-sphere intersection? In-Reply-To: References: Message-ID: On Thu, 23 Mar 2006, (UTC) Jim Carroll apparently wrote: > Where can I find applicable tutorials or recipies? Don't know the lit, but noticed this: http://www.olympus.net/personal/7seas/contain.html fwiw, Alan Isaac From travis at enthought.com Thu Mar 23 21:57:05 2006 From: travis at enthought.com (Travis N. Vaught) Date: Thu Mar 23 21:57:05 2006 Subject: [Numpy-discussion] [wxPython-users] ANN: Python Enthought Edition Version 0.9.3 Released Message-ID: <442319A6.7040108@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 0.9.3 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 0.9.3 Release Notes: -------------------- Version 0.9.3 of Python Enthought Edition includes an update to version 1.0.3 of the Enthought Tool Suite (ETS) Package-- you can look at the release notes for this ETS version here. Other major changes include: * upgrade to VTK 5.0 * addition of docutils * addition of numarray * addition of pysvn. Also, MayaVi issues should be fixed in this release. Full Release Notes are here: http://code.enthought.com/release/changelog-enthon0.9.3.shtml About Python Enthought Edition: ------------------------------- Python 2.3.5, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numeric SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com --------------------------------------------------------------------- To unsubscribe, e-mail: wxPython-users-unsubscribe at lists.wxwidgets.org For additional commands, e-mail: wxPython-users-help at lists.wxwidgets.org From cjw at sympatico.ca Fri Mar 24 04:38:16 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 04:38:16 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <4423E7A5.3010507@sympatico.ca> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>Thus, in my mind, the array call should get another flag keyword that >>forces a contiguous result. >> >> > >Please don't! > +1 >The fortran flag is bad enough, but has too much history >behind it. Let's not breed boolean parameters. Sooner or later >someone will use keword arguments positionally and you will end up >guessing what > >array([1,2], int8_, 1, 1, 0, 0) > >means. > > The use of keyword parameters can help here. Various editors use them to provide tips and there is the help function. Colin W. From tim.hochberg at cox.net Fri Mar 24 06:33:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 06:33:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> Message-ID: <442402E5.6090901@cox.net> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>At any rate, if the fortran flag is there, we need to specify the >>contiguous case as well. So, either propose a better interface (we >>could change it still --- the fortran flag doesn't have that much >>history) to handle the situation or accept what I do ;-) >> >> > >Let me try. I propose to eliminate the fortran flag in favor of a more >general "strides" argument. This argument can be either a sequence of >integers that becomes the strides, or a callable object that takes >shape and dtype arguments and return a sequence that becomes the >strides. For fortran and c order functions that generate appropriate >stride sequences should be predefined to enable array(..., >strides=fortran, ...) and array(..., strides=contiguous). > I like the idea of being able to create an array with custom strides. The applications aren't entirely clear yet, but it does seem like it could have some interesting and useful consequences. That said, I don't think this belongs in 'array'. Historically, array has been used for all sorts of array creation activities, which is why it always seems to have a wide, somewhat incoherent interface. However, most uses of array() boil down to one thing: creating a *new* array from a python object. My preference would be to focus on that functionality for array() and spin of it's other historical uses and new uses, like this custom strided array stuff, into separate factory functions. For example (and just for example, I make no great claims for either this name or interface): a = array_from_data(a_buffer_object, dtype, dims, strides) One thing that you do make clear is that contiguous and fortran should really two values of the same flag. If you combine this with one other simplification: array() always copies, we end up with a nice thin interface: # Create a new array in 'order' order. Defaults to "C" order. array(object, dtype=None, order="C"|"FORTRAN") and # Returns an array. If object is an array and order is satisfied, return object otherwise a new array. # If order is set the returned array will be contiguous and have that ordering asarray(object, dtype=None, order=None|"C"|"FORTRAN") # Just the same, but allow subtypes. asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") You could build asarray, asanyarray, etc on top of the proposed array without problems by using type(object)==ndarray and isinstance(type, ndarray) respectively. Stuff like convenience functions for minnd would also be easy to build on top of there. This looks great to me (pre-coffee). Embrace simplicity: you have nothing to lose but your clutter;) Regards, -tim From cjw at sympatico.ca Fri Mar 24 06:50:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 06:50:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4423FB59.1060403@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> Message-ID: <44240708.5090404@sympatico.ca> Tim Hochberg wrote: > Colin J. Williams wrote: > >> Tim Hochberg wrote: >> >>> >>> >>> I was just looking at the interface for array and asarray to see >>> what other stuff should go in the interface of the hypothetical >>> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >>> which I didn't, but which makes sense. However, there's also >>> 'ndmin'. First off, it's not described in docstring for asarray, but >>> I was able to find it in the docstring for array without a problem. >>> Second, is it really necessary? It seems to be useful in an awfully >>> narrow set of circumstances, particularly since when you are padding >>> axes not everyone wants to pad to the left. >>> >>> It would seem to be more useful to ditch the ndmin and have some >>> sort of paddims function that was more full featured (padding to >>> either the left or the right at a minimum). I'm not entirely sure >>> what the best interface to such a beast would look like, but a >>> simple tactic would be to just provide leftpaddims and rightpaddims. >>> >>> If it's not allready clear by now (;), I prefer several narrow >>> interfaces to one broad one. >>> >>> -tim >>> >> >> >> >> What does ascontiguous do that copy doesn't? > > > What it doesn't do is always copy the argument. Just like asarray, it > returns it unchanged if it's contiguous. Fair enough. I guess that, for some array a, "b= ascontiguous(a)" saves a few keystrokes as compared with "b= a if a.flags.contiguous else a.copy()". The intent of the latter is clearer and probably burns fewer cycles. numarray has an iscontiguous method, which is a bit simpler than concerning the Python user with flags. > >> In numarray, transpose created a discontiguous array. copy() made >> the array contiguous. Is there sufficient benefit to justify another >> method? > > > I was proposing a function, not a method. The array object has plenty > of methods already. Since the function would only operate on ArrayType instances, would it not be better as a method? Colin W. > > -tim > >> >> Colin W. >> >> > > From cjw at sympatico.ca Fri Mar 24 07:06:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 07:06:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442402E5.6090901@cox.net> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: <44240AB6.202@sympatico.ca> Tim Hochberg wrote: > Sasha wrote: > >> On 3/23/06, Travis Oliphant wrote: >> >> >>> At any rate, if the fortran flag is there, we need to specify the >>> contiguous case as well. So, either propose a better interface (we >>> could change it still --- the fortran flag doesn't have that much >>> history) to handle the situation or accept what I do ;-) >>> >> >> Contiguity is separable from fortran: [Dbg]>>> b= _n.array([[1, 2, 3], [4, 5, 6]]) [Dbg]>>> b.flags.contiguous True [Dbg]>>> c= b.transpose() [Dbg]>>> c array([[1, 4], [2, 5], [3, 6]]) [Dbg]>>> c.flags.contiguous False [Dbg]>>> >> Let me try. I propose to eliminate the fortran flag in favor of a more >> general "strides" argument. This argument can be either a sequence of >> integers that becomes the strides, or a callable object that takes >> shape and dtype arguments and return a sequence that becomes the >> strides. For fortran and c order functions that generate appropriate >> stride sequences should be predefined to enable array(..., >> strides=fortran, ...) and array(..., strides=contiguous). >> > > I like the idea of being able to create an array with custom strides. > The applications aren't entirely clear yet, but it does seem like it > could have some interesting and useful consequences. That said, I > don't think this belongs in 'array'. Historically, array has been used > for all sorts of array creation activities, which is why it always > seems to have a wide, somewhat incoherent interface. However, most > uses of array() boil down to one thing: creating a *new* array from a > python object. My preference would be to focus on that functionality > for array() and spin of it's other historical uses and new uses, like > this custom strided array stuff, into separate factory functions. For > example (and just for example, I make no great claims for either this > name or interface): > a = array_from_data(a_buffer_object, dtype, dims, strides) [***] > > One thing that you do make clear is that contiguous and fortran should > really two values of the same flag. Please see the transpose example above. > If you combine this with one other simplification: array() always > copies, we end up with a nice thin interface: > # Create a new array in 'order' order. Defaults to "C" order. > array(object, dtype=None, order="C"|"FORTRAN") I feel that [***] above is much cleaner than this. I suggest that string constants be deprecated. > and > # Returns an array. If object is an array and order is satisfied, > return object otherwise a new array. > # If order is set the returned array will be contiguous and have > that ordering > asarray(object, dtype=None, order=None|"C"|"FORTRAN") > # Just the same, but allow subtypes. > asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") > > You could build asarray, asanyarray, etc on top of the proposed array > without problems by using type(object)==ndarray and isinstance(type, > ndarray) respectively. Stuff like convenience functions for minnd > would also be easy to build on top of there. This looks great to me > (pre-coffee). > > Embrace simplicity: you have nothing to lose but your clutter;) > > Regards, > > -tim > If [***] above were adopted, it would still be helpful to adopt numarray's iscontiguous method, or better, use a property. colin W. From boyle5 at llnl.gov Fri Mar 24 07:55:05 2006 From: boyle5 at llnl.gov (James Boyle) Date: Fri Mar 24 07:55:05 2006 Subject: [Numpy-discussion] Installing numpy 0.9.6 problems on OS X Message-ID: <1efeb18f9e8baf66d0f2e02337a2a709@llnl.gov> I am running OX 10.3.9. I installed python 2.4.2 using Universal MacPython 2.4.2.dmg. When I try to install numpy 0.9.6 it fails with the message. . . gcc: _configtest.c gcc: cannot specify -o with -c or -S and multiple compilations gcc: cannot specify -o with -c or -S and multiple compilations failure. . . I am running gcc 3.3. I note that the python 2.4.2 was built using gcc 4.0.1. Could this difference in gcc versions be the problem? At present I am at a loss as to what to try next. --Jim -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 619 bytes Desc: not available URL: From tim.hochberg at cox.net Fri Mar 24 08:22:17 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 08:22:17 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44240708.5090404@sympatico.ca> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> <44240708.5090404@sympatico.ca> Message-ID: <44241C50.40206@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Colin J. Williams wrote: >> >>> Tim Hochberg wrote: >>> >>>> >>>> >>>> I was just looking at the interface for array and asarray to see >>>> what other stuff should go in the interface of the hypothetical >>>> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >>>> which I didn't, but which makes sense. However, there's also >>>> 'ndmin'. First off, it's not described in docstring for asarray, >>>> but I was able to find it in the docstring for array without a >>>> problem. Second, is it really necessary? It seems to be useful in >>>> an awfully narrow set of circumstances, particularly since when you >>>> are padding axes not everyone wants to pad to the left. >>>> >>>> It would seem to be more useful to ditch the ndmin and have some >>>> sort of paddims function that was more full featured (padding to >>>> either the left or the right at a minimum). I'm not entirely sure >>>> what the best interface to such a beast would look like, but a >>>> simple tactic would be to just provide leftpaddims and rightpaddims. >>>> >>>> If it's not allready clear by now (;), I prefer several narrow >>>> interfaces to one broad one. >>>> >>>> -tim >>>> >>> >>> >>> >>> >>> What does ascontiguous do that copy doesn't? >> >> >> >> What it doesn't do is always copy the argument. Just like asarray, it >> returns it unchanged if it's contiguous. > > > Fair enough. I guess that, for some array a, "b= ascontiguous(a)" > saves a few keystrokes as compared > with "b= a if a.flags.contiguous else a.copy()". The intent of the > latter is clearer and probably burns fewer cycles. First, the second expression is not equivalent to ascontiguous. Second, I disagree strongly that it's clearer. Third, if cycles are actually a concern, ascontiguous if implemented in C would certainly be faster. Fourth, this can't be written in any released version of Python. Let me go into a little more detail on the first and second points. The common use case for asarray and ascontiguous is to adapt objects that you don't know much about into arrays. A typical pattern is: def func(a, b, c): a = asarray(a) b = ascontiguous(b) c = asarray(c) # Code that requires arrays for a,c and a contigous array for b. For this reason, as contiguous needs to take any object and attempt to turn it into a contiguous array if it isn't one already. So "b = ascontiguous(a)" is really equivalent to "b = asarray(a); b = b if b.flags.contiguous else b.copy()". Even if we could assume all the inputs were arrays, or we separately use as array first, the pattern you propose is error prone since it violates DRY. It would be all to easy to write "b= a if c.flags.contiguous else a.copy()" or some such. In order to verify that the expressions are correct, or in order to tell what they actually do you have to actually parse through the expression. "ascontiguous", on the other hand, is pretty much bomb proof once you know what it does. > > numarray has an iscontiguous method, which is a bit simpler than > concerning the Python user with flags. > >> >>> In numarray, transpose created a discontiguous array. copy() made >>> the array contiguous. Is there sufficient benefit to justify >>> another method? >> >> >> >> I was proposing a function, not a method. The array object has plenty >> of methods already. > > > Since the function would only operate on ArrayType instances, would it > not be better as a method? Your premise is wrong. It would operate on any object, so it has to be a function. It could also be a method, but that would be superfluous. Regards, -tim From alexander.belopolsky at gmail.com Fri Mar 24 08:28:07 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Mar 24 08:28:07 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442402E5.6090901@cox.net> References: <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: On 3/24/06, Tim Hochberg wrote: >.... For example (and just for > example, I make no great claims for either this name or interface): > a = array_from_data(a_buffer_object, dtype, dims, strides) > This looks very similar to the current ndarray "new" constructor: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) | | There are two modes of creating an array using __new__: | 1) If buffer is None, then only shape, dtype, and fortran | are used | 2) If buffer is an object exporting the buffer interface, then | all keywords are interpreted. | The dtype parameter can be any object that can be interpreted | as a numpy.dtype object. (see pydoc numpy.ndarray) I would not mind to leave array() unchanged and move discussion to streamlining ndarray.__new__ . For example, some time ago I suggested that strides should be interpreted even if buffer=None. From tim.hochberg at cox.net Fri Mar 24 08:39:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 08:39:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44240AB6.202@sympatico.ca> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> Message-ID: <44242065.9090307@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Sasha wrote: >> >>> On 3/23/06, Travis Oliphant wrote: >>> >>> >>>> At any rate, if the fortran flag is there, we need to specify the >>>> contiguous case as well. So, either propose a better interface (we >>>> could change it still --- the fortran flag doesn't have that much >>>> history) to handle the situation or accept what I do ;-) >>>> >>> >>> >>> > Contiguity is separable from fortran: > [Dbg]>>> b= _n.array([[1, 2, 3], [4, 5, 6]]) > [Dbg]>>> b.flags.contiguous > True > [Dbg]>>> c= b.transpose() > [Dbg]>>> c > array([[1, 4], > [2, 5], > [3, 6]]) > [Dbg]>>> c.flags.contiguous > False > [Dbg]>>> This is true, but irrelevant. To the best of my knowledge, the only reason to force an array to be in a specific order is to pass it to a C function that expects either FORTRAN- or C-ordered arrays. And, in that case, the array also needs to be contiguous. So, for the purpose of creating arrays (and for the purposes of ascontiguous), the only cases that matter are arrays that are both contiguous and the specified order. Thus, specifying continuity and order separately to the constructor needlessly complicates the interface. Or since I'm feeling jargon happy today, YAGNI. > >>> Let me try. I propose to eliminate the fortran flag in favor of a more >>> general "strides" argument. This argument can be either a sequence of >>> integers that becomes the strides, or a callable object that takes >>> shape and dtype arguments and return a sequence that becomes the >>> strides. For fortran and c order functions that generate appropriate >>> stride sequences should be predefined to enable array(..., >>> strides=fortran, ...) and array(..., strides=contiguous). >>> >> >> I like the idea of being able to create an array with custom strides. >> The applications aren't entirely clear yet, but it does seem like it >> could have some interesting and useful consequences. That said, I >> don't think this belongs in 'array'. Historically, array has been >> used for all sorts of array creation activities, which is why it >> always seems to have a wide, somewhat incoherent interface. However, >> most uses of array() boil down to one thing: creating a *new* array >> from a python object. My preference would be to focus on that >> functionality for array() and spin of it's other historical uses and >> new uses, like this custom strided array stuff, into separate factory >> functions. For example (and just for example, I make no great claims >> for either this name or interface): >> a = array_from_data(a_buffer_object, dtype, dims, strides) [***] >> >> One thing that you do make clear is that contiguous and fortran >> should really two values of the same flag. > > > Please see the transpose example above. > >> If you combine this with one other simplification: array() always >> copies, we end up with a nice thin interface: >> # Create a new array in 'order' order. Defaults to "C" order. >> array(object, dtype=None, order="C"|"FORTRAN") > > > I feel that [***] above is much cleaner than this. I suggest that > string constants be deprecated. I'm no huge fan of string constants myself, but I think you need to think this through more. First off, the interface I tossed off above doesn't cover the same ground as array, since it works off an already created buffer object. That means you'd have to go through all sorts of contortions and do at least one copy to get data into Fortran order. You could allow arbitrary, 1D, python sequences instead, but that doesn't help the common case of converting a 2D python object into a 2D array. You could allow N-D python objects, but then you have two ways of specifying the dims of the object and things become a big krufty mess. Compared to that string constants are great. >> and >> # Returns an array. If object is an array and order is satisfied, >> return object otherwise a new array. >> # If order is set the returned array will be contiguous and have >> that ordering >> asarray(object, dtype=None, order=None|"C"|"FORTRAN") >> # Just the same, but allow subtypes. >> asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") >> >> You could build asarray, asanyarray, etc on top of the proposed array >> without problems by using type(object)==ndarray and isinstance(type, >> ndarray) respectively. Stuff like convenience functions for minnd >> would also be easy to build on top of there. This looks great to me >> (pre-coffee). >> >> Embrace simplicity: you have nothing to lose but your clutter;) >> >> Regards, >> >> -tim >> > If [***] above were adopted, it would still be helpful to adopt > numarray's iscontiguous method, or better, use a property. -0. In my experience, 99% of my use cases would be covered for ascontiguous and for the remaining 1% I'm happy to use a.flags.contiguous. Regards, -tim From ndarray at mac.com Fri Mar 24 08:47:10 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 24 08:47:10 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: On 3/24/06, Tim Hochberg wrote: >.... For example (and just for > example, I make no great claims for either this name or interface): > a = array_from_data(a_buffer_object, dtype, dims, strides) > This looks very similar to the current ndarray "new" constructor: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) | | There are two modes of creating an array using __new__: | 1) If buffer is None, then only shape, dtype, and fortran | are used | 2) If buffer is an object exporting the buffer interface, then | all keywords are interpreted. | The dtype parameter can be any object that can be interpreted | as a numpy.dtype object. (see pydoc numpy.ndarray) I would not mind to leave array() unchanged and move discussion to streamlining ndarray.__new__ . For example, some time ago I suggested that strides should be interpreted even if buffer=None. From jimpy at oublic.org Fri Mar 24 08:48:03 2006 From: jimpy at oublic.org (Jim Carroll) Date: Fri Mar 24 08:48:03 2006 Subject: [Numpy-discussion] Re: Newbie: Matrix form of segment-sphere intersection? References: Message-ID: > On Thu, 23 Mar 2006, (UTC) Jim Carroll apparently wrote: > > Where can I find applicable tutorials or recipies? > > Don't know the lit, but noticed this: > http://www.olympus.net/personal/7seas/contain.html > Interesting... but it doesn't show the how the points are formatted in the matricies, or how a more complicated algebraic function would be done... The sphere intersection 3 step algorithm defines some values up front, then finds the discriminant, then uses the discriminant to caluclate the intersection. I'm wondering if I'd be crazy if I coded the algorithm in a C ufunc: for a table where each row has x1, y1, z1, x2, y2, z2, radius if my ufunc would return a table that's got just three columns: x3, y3, z3 representing the sphere intersection. (asuming the input is translated so the sphere is at the origin.) Does anyone do this kind of things with ufuncs? Am I on the right track, but there's an easier way to do it? Or, is there a geometric math library that does this kind of thing? (I found one a year ago, but can't remember if it's this sophisticated.) Thanks, -Jim From tim.hochberg at cox.net Fri Mar 24 08:49:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 08:49:01 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: <442422C2.2010909@cox.net> Alexander Belopolsky wrote: >On 3/24/06, Tim Hochberg wrote: > > >>.... For >> >> >example (and just for > > >>example, I make no great claims for either this name or interface): >> a = array_from_data(a_buffer_object, dtype, dims, strides) >> >> >> > >This looks very similar to the current ndarray "new" constructor: > > It sure does. > | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, > | offset=0, strides=None, fortran=False) > | > | There are two modes of creating an array using __new__: > | 1) If buffer is None, then only shape, dtype, and fortran > | are used > | 2) If buffer is an object exporting the buffer interface, then > | all keywords are interpreted. > | The dtype parameter can be any object that can be interpreted > | as a numpy.dtype object. > >(see pydoc numpy.ndarray) > >I would not mind to leave array() unchanged and move discussion to >streamlining ndarray.__new__ . For example, some time ago I suggested >that strides should be interpreted even if buffer=None. > > That does look like a good place to hang any arbitrary strides creating stuff. I'm of the opinion that arguments should *never* be ignored, so I'm all for interpreting strides even when buffer is None. I'd also contend that offset should either be respected (by overallocating) or since that's probably useless, raising a ValueError when it's nonzero. Regards, Tim From fred at ucar.edu Fri Mar 24 10:25:05 2006 From: fred at ucar.edu (Fred Clare) Date: Fri Mar 24 10:25:05 2006 Subject: [Numpy-discussion] PyArray_ISSTRING Message-ID: <2a503cbe0e0f909cb2502f799d5229c4@ucar.edu> I am running NumPy 0.9.6 on Mac OS X. When I try to use the API function PyArray_ISSTRING I get: `PyArray_UCHAR' undeclared In fact I cannot see where PyArray_UCHAR is ever declared. Fred Clare From oliphant at ee.byu.edu Fri Mar 24 11:12:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 24 11:12:00 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: <44244435.3050904@ee.byu.edu> Alexander Belopolsky wrote: >On 3/24/06, Tim Hochberg wrote: > > >>.... For >> >> >example (and just for > > >>example, I make no great claims for either this name or interface): >> a = array_from_data(a_buffer_object, dtype, dims, strides) >> >> >> > >This looks very similar to the current ndarray "new" constructor: > > | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, > | offset=0, strides=None, fortran=False) > | > | There are two modes of creating an array using __new__: > | 1) If buffer is None, then only shape, dtype, and fortran > | are used > | 2) If buffer is an object exporting the buffer interface, then > | all keywords are interpreted. > | The dtype parameter can be any object that can be interpreted > | as a numpy.dtype object. > >(see pydoc numpy.ndarray) > >I would not mind to leave array() unchanged and move discussion to >streamlining ndarray.__new__ . For example, some time ago I suggested >that strides should be interpreted even if buffer=None. > > > Here I'm much more happy about removing the fortran flag and just using the strides parameter as Sasha suggests in this routine, so please streamline as much as possible. -Travis From cjw at sympatico.ca Fri Mar 24 11:15:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 11:15:02 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44241C50.40206@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> <44240708.5090404@sympatico.ca> <44241C50.40206@cox.net> Message-ID: <4424451B.1000008@sympatico.ca> Tim Hochberg wrote: > Colin J. Williams wrote: > >> Tim Hochberg wrote: >> >>> Colin J. Williams wrote: >>> >>>> Tim Hochberg wrote: >>>> >>>>> >>>>> >>>>> I was just looking at the interface for array and asarray to see >>>>> what other stuff should go in the interface of the hypothetical >>>>> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >>>>> which I didn't, but which makes sense. However, there's also >>>>> 'ndmin'. First off, it's not described in docstring for asarray, >>>>> but I was able to find it in the docstring for array without a >>>>> problem. Second, is it really necessary? It seems to be useful in >>>>> an awfully narrow set of circumstances, particularly since when >>>>> you are padding axes not everyone wants to pad to the left. >>>>> >>>>> It would seem to be more useful to ditch the ndmin and have some >>>>> sort of paddims function that was more full featured (padding to >>>>> either the left or the right at a minimum). I'm not entirely sure >>>>> what the best interface to such a beast would look like, but a >>>>> simple tactic would be to just provide leftpaddims and rightpaddims. >>>>> >>>>> If it's not allready clear by now (;), I prefer several narrow >>>>> interfaces to one broad one. >>>>> >>>>> -tim >>>>> >>>> >>>> >>>> >>>> >>>> >>>> What does ascontiguous do that copy doesn't? >>> >>> >>> >>> >>> What it doesn't do is always copy the argument. Just like asarray, >>> it returns it unchanged if it's contiguous. >> >> >> >> Fair enough. I guess that, for some array a, "b= ascontiguous(a)" >> saves a few keystrokes as compared >> with "b= a if a.flags.contiguous else a.copy()". The intent of the >> latter is clearer and probably burns fewer cycles. > > > First, the second expression is not equivalent to ascontiguous. > Second, I disagree strongly that it's clearer. Third, if cycles are > actually a concern, ascontiguous if implemented in C would certainly > be faster. Fourth, this can't be written in any released version of > Python. > Thanks, this is illuminating. Fourth is true, but the first release of Python 2.5 is expected in eight dats. We see, in the Help: Help on built-in function array in module numpy.core.multiarray: array(...) array(object, dtype=None, copy=1, fortran=0, subok=0,ndmin=0) will return a new array formed from the given object type given. Object can be anything with an __array__ method, or any object exposing the array interface, or any (nested) sequence. If no type is given, then the type will be determined as the minimum type required to hold the objects in the sequence. If copy is zero and sequence is already an array with the right type, a reference will be returned. If the sequence is an array, type can be used only to upcast the array. For downcasting use .astype(t) method. If subok is true, then subclasses of the array may be returned. Otherwise, a base-class ndarray is returned The ndmin argument specifies how many dimensions the returned array should have as a minimum. 1's will be pre-pended to the shape as needed to meet this requirement. [Dbg]>>> help(_n.asarray) Help on function asarray in module numpy.core.numeric: asarray(a, dtype=None, fortran=False, ndmin=0) returns a as an array. Unlike array(), no copy is performed if a is already an array. Subclasses are converted to base class ndarray. It is not clear, from this, just what is acceptable, for a, by asarray. From your response, it would appear that "a" in asarray is the same as "object" in array. > Let me go into a little more detail on the first and second points. > The common use case for asarray and ascontiguous is to adapt objects > that you don't know much about into arrays. A typical pattern is: > > def func(a, b, c): > a = asarray(a) > b = ascontiguous(b) > c = asarray(c) > # Code that requires arrays for a,c and a contigous array for b. > > For this reason, as contiguous needs to take any object and attempt to > turn it into a contiguous array if it isn't one already. So "b = > ascontiguous(a)" is really equivalent to "b = asarray(a); b = b if > b.flags.contiguous else b.copy()". Even if we could assume all the > inputs were arrays, or we separately use as array first, the pattern > you propose is error prone since it violates DRY. It would be all to > easy to write "b= a if c.flags.contiguous else a.copy()" or some such. > In order to verify that the expressions are correct, or in order to > tell what they actually do you have to actually parse through the > expression. "ascontiguous", on the other hand, is pretty much bomb > proof once you know what it does. > > I would be grateful if you could clarify DRY. > >> >> numarray has an iscontiguous method, which is a bit simpler than >> concerning the Python user with flags. >> >>> >>>> In numarray, transpose created a discontiguous array. copy() made >>>> the array contiguous. Is there sufficient benefit to justify >>>> another method? >>> >>> >>> >>> >>> I was proposing a function, not a method. The array object has >>> plenty of methods already. >> >> >> >> Since the function would only operate on ArrayType instances, would >> it not be better as a method? > > > Your premise is wrong. It would operate on any object, so it has to be > a function. It could also be a method, but that would be superfluous. > > Regards, > > -tim > It would appear that you are right, ascontiguous will be permitted to operate on any object which is acceptable to array, although I can't think of a case where any object other than an array, or a sub-class could be discontiguous. Thanks for clarifying things. Colin W. From boyle5 at llnl.gov Fri Mar 24 11:17:04 2006 From: boyle5 at llnl.gov (James Boyle) Date: Fri Mar 24 11:17:04 2006 Subject: [Numpy-discussion] OS X - Universal MacPython - numpy install failure Message-ID: I am running OX 10.3.9. I installed python 2.4.2 using Universal MacPython 2.4.2.dmg. I am on a PowerPC G4. The Universal build is designed to install on the Mac PPC and Intel machines. When I run the numpy 0.9.6 install it apparently fails on the configure. The whole trace back is at the end of this posting. One thing that I noted was the line: gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' It appears that the configure routines think that I have two architectures - ppc and i386. At my low level of understanding, all I can do is speculate that the Universal install generates some confusion as to the actual cpu. Somewhere numpy is keying on information that is ambiguous in the universal install. I have installed Numeric, matplotlib and other packages without a problem. --Jim The trace back from >python setup.py build: Running from numpy source directory. Warning: not existing path in numpy/distutils: site.cfg F2PY Version 2_2236 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] running build running config_fc running build_src building py_modules sources creating build creating build/src creating build/src/numpy creating build/src/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src/numpy/core Generating build/src/numpy/core/config.h customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' compile options: '-I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' gcc: _configtest.c gcc: cannot specify -o with -c or -S and multiple compilations gcc: cannot specify -o with -c or -S and multiple compilations failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 69, in setup_package setup( **config.todict() ) File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/core.py", line 85, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 84, in run self.build_sources() File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 99, in build_sources self.build_extension_sources(ext) File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 209, in build_extension_sources sources = self.generate_sources(sources, ext) File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 267, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 37, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 4515 bytes Desc: not available URL: From tim.hochberg at cox.net Fri Mar 24 11:36:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 11:36:06 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4424451B.1000008@sympatico.ca> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> <44240708.5090404@sympatico.ca> <44241C50.40206@cox.net> <4424451B.1000008@sympatico.ca> Message-ID: <442449FA.3070503@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Colin J. Williams wrote: >> >>> Tim Hochberg wrote: >>> >>>> Colin J. Williams wrote: >>>> >>>>> Tim Hochberg wrote: >>>>> >>>>>> >>>>>> >>>>>> I was just looking at the interface for array and asarray to see >>>>>> what other stuff should go in the interface of the hypothetical >>>>>> ascontiguous. There's 'dtype', which I knew about, and >>>>>> 'fortran', which I didn't, but which makes sense. However, >>>>>> there's also 'ndmin'. First off, it's not described in docstring >>>>>> for asarray, but I was able to find it in the docstring for array >>>>>> without a problem. Second, is it really necessary? It seems to be >>>>>> useful in an awfully narrow set of circumstances, particularly >>>>>> since when you are padding axes not everyone wants to pad to the >>>>>> left. >>>>>> >>>>>> It would seem to be more useful to ditch the ndmin and have some >>>>>> sort of paddims function that was more full featured (padding to >>>>>> either the left or the right at a minimum). I'm not entirely sure >>>>>> what the best interface to such a beast would look like, but a >>>>>> simple tactic would be to just provide leftpaddims and rightpaddims. >>>>>> >>>>>> If it's not allready clear by now (;), I prefer several narrow >>>>>> interfaces to one broad one. >>>>>> >>>>>> -tim >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> What does ascontiguous do that copy doesn't? >>>> >>>> >>>> >>>> >>>> >>>> What it doesn't do is always copy the argument. Just like asarray, >>>> it returns it unchanged if it's contiguous. >>> >>> >>> >>> >>> Fair enough. I guess that, for some array a, "b= ascontiguous(a)" >>> saves a few keystrokes as compared >>> with "b= a if a.flags.contiguous else a.copy()". The intent of the >>> latter is clearer and probably burns fewer cycles. >> >> >> >> First, the second expression is not equivalent to ascontiguous. >> Second, I disagree strongly that it's clearer. Third, if cycles are >> actually a concern, ascontiguous if implemented in C would certainly >> be faster. Fourth, this can't be written in any released version of >> Python. >> > Thanks, this is illuminating. Fourth is true, but the first release > of Python 2.5 is expected in eight dats. > > We see, in the Help: > Help on built-in function array in module numpy.core.multiarray: > > array(...) > array(object, dtype=None, copy=1, fortran=0, subok=0,ndmin=0) > will return a new array formed from the given object type given. > Object can be anything with an __array__ method, or any object > exposing the array interface, or any (nested) sequence. > If no type is given, then the type will be determined as the > minimum type required to hold the objects in the sequence. > If copy is zero and sequence is already an array with the right > type, a reference will be returned. If the sequence is an array, > type can be used only to upcast the array. For downcasting > use .astype(t) method. If subok is true, then subclasses of the > array may be returned. Otherwise, a base-class ndarray is returned > The ndmin argument specifies how many dimensions the returned > array should have as a minimum. 1's will be pre-pended to the > shape as needed to meet this requirement. > [Dbg]>>> help(_n.asarray) > Help on function asarray in module numpy.core.numeric: > > asarray(a, dtype=None, fortran=False, ndmin=0) > returns a as an array. Unlike array(), > no copy is performed if a is already an array. Subclasses are > converted > to base class ndarray. > > It is not clear, from this, just what is acceptable, for a, by asarray. That could probably be clearer, it's true. > > From your response, it would appear that "a" in asarray is the same as > "object" in array. Indeed. > >> Let me go into a little more detail on the first and second points. >> The common use case for asarray and ascontiguous is to adapt objects >> that you don't know much about into arrays. A typical pattern is: >> >> def func(a, b, c): >> a = asarray(a) >> b = ascontiguous(b) >> c = asarray(c) >> # Code that requires arrays for a,c and a contigous array for b. >> >> For this reason, as contiguous needs to take any object and attempt >> to turn it into a contiguous array if it isn't one already. So "b = >> ascontiguous(a)" is really equivalent to "b = asarray(a); b = b if >> b.flags.contiguous else b.copy()". Even if we could assume all the >> inputs were arrays, or we separately use as array first, the pattern >> you propose is error prone since it violates DRY. It would be all to >> easy to write "b= a if c.flags.contiguous else a.copy()" or some >> such. In order to verify that the expressions are correct, or in >> order to tell what they actually do you have to actually parse >> through the expression. "ascontiguous", on the other hand, is pretty >> much bomb proof once you know what it does. >> >> > I would be grateful if you could clarify DRY. Don't Repeat Yourself. Basically, it's exhortation not to write out the same expression over and over again. It was on of the main motivators for getting += into Python, since: a.foo.bar[squiggly].retrieveArrayFor("Colin") = a.foo.bar[squiggly].retrieveArrayFor("Colin") + rubber_ducky.frobinate["NOW!"] Is much easier to mess up than: a.foo.bar[squiggly].retrieveArrayFor("Colin") += rubber_ducky.frobinate["NOW!"] In the former, it's easy to miss small differences between the first and second quantities. These creep in easily when cutting and pasting. Anyway, in the case in question: b= a if a.flags.contiguous else a.copy() 'a' is repeated twice. If this is getting cut and pasted all over the place it'll frequently end up that one of the a's doesn't get changed properly. For: b = ascontiguous(a) The chances are at least half as small that one gets it wrong based on numbers of chances alone. Probably much, much better than that in practice since a isn't buried in an expression, but is nicely set off by parentheses. See also: http://en.wikipedia.org/wiki/Don%27t_repeat_yourself > >> >>> >>> numarray has an iscontiguous method, which is a bit simpler than >>> concerning the Python user with flags. >>> >>>> >>>>> In numarray, transpose created a discontiguous array. copy() made >>>>> the array contiguous. Is there sufficient benefit to justify >>>>> another method? >>>> >>>> >>>> >>>> >>>> >>>> I was proposing a function, not a method. The array object has >>>> plenty of methods already. >>> >>> >>> >>> >>> Since the function would only operate on ArrayType instances, would >>> it not be better as a method? >> >> >> >> Your premise is wrong. It would operate on any object, so it has to >> be a function. It could also be a method, but that would be superfluous. >> >> Regards, >> >> -tim >> > It would appear that you are right, ascontiguous will be permitted to > operate on any object which is acceptable to array, although I can't > think of a case where any object other than an array, or a sub-class > could be discontiguous. No, that is correct. The point is I have an object I obtained from somewhere. I need a contiguous array. ascontiguous applied to that object will either give me a contiguous array or raise an exception. Thus it can be used to adapt any thing-that-can-be-turned-into-an-array into a contiguous array. > > Thanks for clarifying things. Not a problem. Regards, Tim From oliphant at ee.byu.edu Fri Mar 24 11:50:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 24 11:50:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44242065.9090307@cox.net> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> <44242065.9090307@cox.net> Message-ID: <44244D42.5020201@ee.byu.edu> Tim Hochberg wrote: >> >> >> Please see the transpose example above. >> >>> If you combine this with one other simplification: array() always >>> copies, we end up with a nice thin interface: >>> # Create a new array in 'order' order. Defaults to "C" order. >>> array(object, dtype=None, order="C"|"FORTRAN") >> I like the order flag. It's a relatively easy switch and one we should make right now. I don't mind string constants either in this case. Removing the copy flag will break a lot of code because it's been around for a long time. This is also not an "easy thing" to add to convertcode.py though I suppose 90% of the cases could be found. We would also have to re-write asarray to be an additional C-function to make it not copy but make array copy. So, for now I'm not as enthused about that idea. -Travis From ryanlists at gmail.com Fri Mar 24 12:54:01 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Fri Mar 24 12:54:01 2006 Subject: [Numpy-discussion] Re: Newbie: Matrix form of segment-sphere intersection? In-Reply-To: References: Message-ID: I would first do this in python without ufuncs and see if it is slow before going through the extra effort to put it in a ufunc. I think if you play with array broadcasting, you will be quite happy with the results for doing calculations on lots of points. Numpy operates on an element-by-element basis by default. So, for example x=arange(0,10,0.1) y=x**2 will create a vector x and then a vector of y with the elements of x squared. This sort of math on arrays I think does what you want unless there is some math function in your algorithm that doesn't take vector inputs. In that case you might need to write your own function for that. Does this help or am I missing something important? Ryan On 3/24/06, Jim Carroll wrote: > > > On Thu, 23 Mar 2006, (UTC) Jim Carroll apparently wrote: > > > Where can I find applicable tutorials or recipies? > > > > Don't know the lit, but noticed this: > > http://www.olympus.net/personal/7seas/contain.html > > > > Interesting... but it doesn't show the how the points are formatted in the > matricies, or how a more complicated algebraic function would be done... > > The sphere intersection 3 step algorithm defines some values up front, then > finds the discriminant, then uses the discriminant to caluclate the intersection. > > I'm wondering if I'd be crazy if I coded the algorithm in a C ufunc: > for a table where each row has x1, y1, z1, x2, y2, z2, radius if my ufunc > would return a table that's got just three columns: x3, y3, z3 representing the > sphere intersection. (asuming the input is translated so the sphere is at the > origin.) > > Does anyone do this kind of things with ufuncs? Am I on the right track, but > there's an easier way to do it? Or, is there a geometric math library that does > this kind of thing? (I found one a year ago, but can't remember if it's this > sophisticated.) > > Thanks, > -Jim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cookedm at physics.mcmaster.ca Fri Mar 24 13:04:07 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 24 13:04:07 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44223E4A.9090208@colorado.edu> (Fernando Perez's message of "Wed, 22 Mar 2006 23:20:58 -0700") References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220366.4000004@cox.net> <44223E4A.9090208@colorado.edu> Message-ID: Fernando Perez writes: > Tim Hochberg wrote: > >> This did fix things...eventually. I removed the build directory and >> rebuilt. Numpy still bombed the interpreter, although in a new and >> less exciting way. I then did a clean checkout from svn and rebuilt. >> Same thing. Then I manally removed numpy from site-packages and >> reinstalled at which points things worked and all tests passed. >> So in addition to needing to remove the build directory, it appears >> that something was hanging out in the destination directory that did >> not get overwritten and caused problems. >> Perhaps this description will help the next person who runs into >> this, but probably not. > > Well, the script that I posted does precisely all of this in a > mindless fashion, which is why I find it useful. You may want to give > it a try (it's trivial, but it just ensures you don't forget these > steps and end up wasting any time chasing the problem). Me, I use eggs. One advantage is, since I use the bleeding edge for for all of my code, if there's something wrong with numpy (or scipy) and I don't have the time to fix it, I can switch to an older version. I've got a make-egg script that looks like #!/bin/sh set -e PYTHON=python2.4 ${PYTHON} setup.py build ${PYTHON} -c 'import setuptools; execfile("setup.py")' bdist_egg then easy_install2.4 dist/ -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From tim.hochberg at cox.net Fri Mar 24 13:40:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 13:40:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44244D42.5020201@ee.byu.edu> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> <44242065.9090307@cox.net> <44244D42.5020201@ee.byu.edu> Message-ID: <44246715.3030401@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> >>> Please see the transpose example above. >>> >>>> If you combine this with one other simplification: array() always >>>> copies, we end up with a nice thin interface: >>>> # Create a new array in 'order' order. Defaults to "C" order. >>>> array(object, dtype=None, order="C"|"FORTRAN") >>> >>> > I like the order flag. It's a relatively easy switch and one we > should make right now. I don't mind string constants either in this > case. Great! > Removing the copy flag will break a lot of code because it's been > around for a long time. This is also not an "easy thing" to add to > convertcode.py though I suppose 90% of the cases could be found. I kinda figured on that. But I figured I'd propose my favorite and see what came of it. > We would also have to re-write asarray to be an additional C-function > to make it not copy but make array copy. I thought so too at first, but I don't this is is so. Untested, and can could probably be cleaned up some: def asarray(obj, order=None): if type(obj) == ndarray: if order: if not obj.flags.contiguous: return array(obj, order) if order == "C" and obj.flags.fortran: return array(obj, order) if order == "FORTRAN" and not obj.flags.fortran: return array(obj, order) return obj else: if order: return array(obj, order) else: return array(obj) For asanyarray, simply replace the type test with an isinstance test. > So, for now I'm not as enthused about that idea. Yeah. Without backward compatibility constraints I'm convinced that it's the right thing to do, but I realize there is a need to balance making the transistion manageable with making things "perfect". -tim From oliphant at ee.byu.edu Fri Mar 24 14:15:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 24 14:15:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44246715.3030401@cox.net> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> <44242065.9090307@cox.net> <44244D42.5020201@ee.byu.edu> <44246715.3030401@cox.net> Message-ID: <44246F50.2060801@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>>> >>>> >>>> Please see the transpose example above. >>>> >>>>> If you combine this with one other simplification: array() always >>>>> copies, we end up with a nice thin interface: >>>>> # Create a new array in 'order' order. Defaults to "C" order. >>>>> array(object, dtype=None, order="C"|"FORTRAN") >>>> >>>> >>>> >> I like the order flag. It's a relatively easy switch and one we >> should make right now. I don't mind string constants either in this >> case. > > > Great! > And this is now done... So, thankfully, the fortran= keyword is gone and replaced with the more sensible order= keyword. Tests for numpy pass, but any other code that used fortran= will need changing. Sorry about that... Thanks, -Travis From faltet at carabos.com Fri Mar 24 18:10:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Mar 24 18:10:01 2006 Subject: [Numpy-discussion] A numerical agnostic extension Message-ID: <1143252541.7569.10.camel@localhost.localdomain> Hi, After seeing the inspiring work of Andrew Straw on the implementation of a __array_struct__ Pyrex interface for the array protocol, I wasn't able to relax and finally implemented another extension (heavily based on Andrew's work) that takes Numeric/numarray/NumPy objects and exposes the same interface to access to their data and metadata from both Python and Pyrex (and hence, C) spaces. All this *without* a need to compile against a specific numerical package. I've created a new recipe for this extension in: http://www.scipy.org/Cookbook/A_Pyrex_Agnostic_Class I'm pretty excited about all the possibilities that opens this relatively small class: numerical agnosticism is closer than ever :-) -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From ted.horst at earthlink.net Fri Mar 24 18:25:13 2006 From: ted.horst at earthlink.net (Ted Horst) Date: Fri Mar 24 18:25:13 2006 Subject: [Numpy-discussion] OS X - Universal MacPython - numpy install failure In-Reply-To: References: Message-ID: <8002A455-98BF-42C1-AFF3-678E13E0D9E1@earthlink.net> I could be wrong, but I don't think 10.3.9 supports universal binaries or 10.4 sdk. I think you need a different pythn install. Ted On Mar 24, 2006, at 13:16, James Boyle wrote: > I am running OX 10.3.9. I installed python 2.4.2 using Universal > MacPython 2.4.2.dmg. > I am on a PowerPC G4. The Universal build is designed to install on > the Mac PPC and Intel machines. > > When I run the numpy 0.9.6 install it apparently fails on the > configure. The whole trace back is at the end of this posting. > One thing that I noted was the line: > > gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/ > MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp- > precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' > > It appears that the configure routines think that I have two > architectures - ppc and i386. > At my low level of understanding, all I can do is speculate that > the Universal install generates some confusion as to the actual cpu. > Somewhere numpy is keying on information that is ambiguous in the > universal install. > > I have installed Numeric, matplotlib and other packages without a > problem. > > --Jim > > > The trace back from >python setup.py build: > > Running from numpy source directory. > Warning: not existing path in numpy/distutils: site.cfg > F2PY Version 2_2236 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec', '-I/System/Library/ > Frameworks/vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec'] > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src > creating build/src/numpy > creating build/src/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src/numpy/core > Generating build/src/numpy/core/config.h > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > customize GnuFCompiler > customize GnuFCompiler > customize GnuFCompiler using config > gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/ > MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp- > precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' > compile options: '-I/Library/Frameworks/Python.framework/Versions/ > 2.4/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/ > Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' > gcc: _configtest.c > gcc: cannot specify -o with -c or -S and multiple compilations > gcc: cannot specify -o with -c or -S and multiple compilations > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 76, in ? > setup_package() > File "setup.py", line 69, in setup_package > setup( **config.todict() ) > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/ > core.py", line 85, in setup > return old_setup(**new_attr) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/core.py", line 149, in setup > dist.run_commands() > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/dist.py", line 946, in run_commands > self.run_command(cmd) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/command/build.py", line 112, in run > self.run_command(cmd_name) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 84, in run > self.build_sources() > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 99, in build_sources > self.build_extension_sources(ext) > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 209, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 267, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 37, in generate_config_h > raise "ERROR: Failed to test configuration" > ERROR: Failed to test configuration From tim.hochberg at cox.net Fri Mar 24 18:30:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 18:30:02 2006 Subject: [Numpy-discussion] Ransom Proposals Message-ID: <4424AB16.6000007@cox.net> Now that the fortran keyword is dead, I'll take time off from rejoicing, to briefly go over other proposals that were involved in that thread lest they get lost in the shuffle. * reshape -- The origination of that whole, enormous thread was with reshape and what its return behaviour should be. Currently it returns a view if possible, otherwise it returns a copy. That's evil behaviour. I believe that it should either always return a copy or return a view if possible and otherwise raise an exception. I think I actually prefer the later, although it's probably a more disruptive change, since its more flexible when combined with asarray and friends. One possible compromise would be to have numpy.reshape always copy, while array.reshape always returns a view. * ascontiguous -- In private email Chris Barker mentioned that the name ascontiguous was confusing, or at least not to his taste and suggested "something like" ascontiguous_array. I don't like that one, but it might worth considering something that matches asarray and asanyarray. ascontigarray looks okay to me, but it's quite possible that I've staring at this too long and that's just cryptic. * ndmin -- I still think ndmin should be spun off to a separate function. It's trivial to implement a fuction, call it paddims for lack of a better name (asatleastndimensionarray seems kind of long and cryptic!). It should have minimal performance since no data gets copied, and if it makes a difference I would be willing to implement it in C in need be so that that performance impact was minimized. def paddims(a, n): "return a view of 'a' with at least 'n' dimensions" dims = a.shape b = a.view() b.shape = (1,)*(n - len(dims)) + dims return b * copy -- Yes, I understand that the copy flag is probably not going to change for backward compatibility reasons if nothing else, but there was one last point I wanted to make about the copy flag. One of the warts that results from the copy flag, and I think that this is a common problem for functions that take parameters that switch their mode of operation, is that some combinations of input become nonsensical. Consider these five possibilities: array(anarray, dtype=matchingtype, order=matchingorder, copy=True) # OK; copied array(anarray, dtype=matchingtype, order=matchingorder, copy=False) # OK; not copied array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, copy=True) # OK; copied array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, copy=False) # Ugly; copied array(nonarray, dtype=whatever, order=whatever, copy=False) # Ugly; copied [Note that I've folded nonmatchingtype and nonmatchingorder together since they have the same effect] Of these five possibilities, two have results where the arguments and the action taken become uncoupled. One way to address this would be to change the name of the copy flag to something that matches reality: force_copy. However, that seems kind of pointless, since it still introduces as the underlying problem that some of the modes the array function can operate in are kind of bogus. Compare this to the case where the two primitives are array and asarray: array(anarray, dtype=matchingtype, order=matchingorder) # copied array(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied asarray(anarray, dtype=matchingtype, order=matchingorder) # not copied asarray(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied asarray(nonarray, dtype=whatever, order=whatever, copy=False) # copied There's still five cases, so the interface hasn't narrowed any[*], but all the possible argument combinations make sense (or raise a straightforward error). And think how much easier this behaviour is to explain! Anyway that's it for now and hopefully for a while. Regards, -tim [*] In reality it does narrow the interface because we already have asarray, but now you really need as array whereas before it was really just shorthand for array(copy=False). From tim.hochberg at cox.net Fri Mar 24 20:36:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 20:36:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4424AB16.6000007@cox.net> References: <4424AB16.6000007@cox.net> Message-ID: <4424C889.7090706@cox.net> Hah! This was supposed to be "Random Proposals", but Ransom is pretty good too! Tim Hochberg wrote: > > Now that the fortran keyword is dead, I'll take time off from > rejoicing, to briefly go over other proposals that were involved in > that thread lest they get lost in the shuffle. > > * reshape -- The origination of that whole, enormous thread was with > reshape and what its return behaviour should be. Currently it returns > a view if possible, otherwise it returns a copy. That's evil > behaviour. I believe that it should either always return a copy or > return a view if possible and otherwise raise an exception. I think I > actually prefer the later, although it's probably a more disruptive > change, since its more flexible when combined with asarray and > friends. One possible compromise would be to have numpy.reshape > always copy, while array.reshape always returns a view. > > * ascontiguous -- In private email Chris Barker mentioned that the > name ascontiguous was confusing, or at least not to his taste and > suggested "something like" ascontiguous_array. I don't like that one, > but it might worth considering something that matches asarray and > asanyarray. ascontigarray looks okay to me, but it's quite possible > that I've staring at this too long and that's just cryptic. > > * ndmin -- I still think ndmin should be spun off to a separate > function. It's trivial to implement a fuction, call it paddims for > lack of a better name (asatleastndimensionarray seems kind of long and > cryptic!). It should have minimal performance since no data gets > copied, and if it makes a difference I would be willing to implement > it in C in need be so that that performance impact was minimized. > > def paddims(a, n): > "return a view of 'a' with at least 'n' dimensions" > dims = a.shape > b = a.view() > b.shape = (1,)*(n - len(dims)) + dims > return b > > * copy -- Yes, I understand that the copy flag is probably not going > to change for backward compatibility reasons if nothing else, but > there was one last point I wanted to make about the copy flag. One of > the warts that results from the copy flag, and I think that this is a > common problem for functions that take parameters that switch their > mode of operation, is that some combinations of input become > nonsensical. Consider these five possibilities: > > array(anarray, dtype=matchingtype, order=matchingorder, copy=True) # > OK; copied > array(anarray, dtype=matchingtype, order=matchingorder, copy=False) # > OK; not copied > array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, > copy=True) # OK; copied > array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, > copy=False) # Ugly; copied > array(nonarray, dtype=whatever, order=whatever, copy=False) # Ugly; > copied > > [Note that I've folded nonmatchingtype and nonmatchingorder together > since they have the same effect] > > Of these five possibilities, two have results where the arguments and > the action taken become uncoupled. One way to address this would be to > change the name of the copy flag to something that matches reality: > force_copy. However, that seems kind of pointless, since it still > introduces as the underlying problem that some of the modes the array > function can operate in are kind of bogus. Compare this to the case > where the two primitives are array and asarray: > > array(anarray, dtype=matchingtype, order=matchingorder) # copied > array(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied > > asarray(anarray, dtype=matchingtype, order=matchingorder) # not copied > asarray(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied > asarray(nonarray, dtype=whatever, order=whatever, copy=False) # copied > > There's still five cases, so the interface hasn't narrowed any[*], but > all the possible argument combinations make sense (or raise a > straightforward error). And think how much easier this behaviour is to > explain! > > Anyway that's it for now and hopefully for a while. > > Regards, > > -tim > > > [*] In reality it does narrow the interface because we already have > asarray, but now you really need as array whereas before it was really > just shorthand for array(copy=False). > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From ofnap at nus.edu.sg Sat Mar 25 05:00:03 2006 From: ofnap at nus.edu.sg (Ajith Prasad (OFN)) Date: Sat Mar 25 05:00:03 2006 Subject: [Numpy-discussion] USING NUMPY FOR AN ENROLLMENT PROJECTION PROBLEM Message-ID: <502D9B13021C2D42B05F32C5C3F4C1DD2875FF@MBX01.stf.nus.edu.sg> I posted this problem - arising from my work at the local university - some years back in comp.lang.python and received a solution that did not require the use of matrix algebra. The solution was: def enrollment(year, intake, survivalrate): return sum([intake[year-i]*rate for i, rate in enumerate(survivalrate)]) I would welcome help in formulating a solution using NumPy. Thanks in advance for any suggestion. The Problem: The enrolment E(n) of an institution at the beginning of year n is the sum of the intake for year n, I(n), and the survivors from the intakes of previous r years. Thus, if s(1) is the 1-year intake survival rate, s(2) is the 2-year survival rate, etc, we have: E(n)= I(n)+I(n-1)*s(1)+ I(n-2)*s(2)+...+I(n-r)*s(r) E(n+1)= I(n+1)+I(n)*s(1)+I(n-1)*s(2)+... +I(n-r-1)*s(r) . . . E(n+k)= I(n+k)+I(n+k-1)*s(1)+I(n+k-2)*s(2)+...+I(n+k-r)*s(r) Given: (a) the actual intakes for the current and previous r years, I(n), I(n-1),I(n-2),..,I(n-r), and the planned intakes for the next n+k years: I(n+1), I(n+2),..., I(n+k), we have the intake vector I = (I(n-r), I(n-r-1),...,I(n),I(n+1),..., I(n+k)); and (b) the survival rate vector, s = (1,s(1), s(2),...,s(r)) Find: The k*1 enrolment projection column vector, E = (E(n+1),E(n+2),...,E(n+k)) in terms of a k*(r+1) matrix P (derived from I) and the (r+1)*1 column vector, s. I = P*s Is there a compact Python representation of the relevant matrix P where: P = [I(n+1) I(n) I(n-1).. . I(n-r) I(n+2) I(n+1) I(n)... I(n-r-1) . . I(n+k) I(n+k-1) I(n+k-2)... I(n+k-r)] From ndarray at mac.com Sat Mar 25 09:54:01 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 09:54:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4424AB16.6000007@cox.net> References: <4424AB16.6000007@cox.net> Message-ID: On 3/24/06, Tim Hochberg wrote: > > Now that the fortran keyword is dead, I'll take time off from rejoicing, > to briefly go over other proposals that were involved in that thread > lest they get lost in the shuffle. > Many thanks for the effort. That thread actually started as "ndarray.fill and ma.array.filled", but it quickly went off the original topic and forked at your annalistic "(mumble mumble reshape mumble)." :-) > * reshape -- The origination of that whole, enormous thread was with > reshape and what its return behaviour should be. Currently it returns a > view if possible, otherwise it returns a copy. That's evil behaviour. I > believe that it should either always return a copy or return a view if > possible and otherwise raise an exception. I think I actually prefer the > later, although it's probably a more disruptive change, since its more > flexible when combined with asarray and friends. One possible > compromise would be to have numpy.reshape always copy, while > array.reshape always returns a view. > Could you, please provide some examples of the current behavior that you find undesired and how yuo would fix it? I believe the inconsistency is that x.reshape(shape) is x depends on shape. This is easy to fix. Do you also suggest that x.reshape(...).__array_data__ is x.__array_data__ should be False? > * ascontiguous -- In private email Chris Barker mentioned that the name > ascontiguous was confusing, or at least not to his taste and suggested > "something like" ascontiguous_array. I don't like that one, but it might > worth considering something that matches asarray and asanyarray. > ascontigarray looks okay to me, but it's quite possible that I've > staring at this too long and that's just cryptic. > +0 (+1 for "ascontiguousarray") > * ndmin -- I still think ndmin should be spun off to a separate > function. It's trivial to implement a fuction, call it paddims for lack > of a better name (asatleastndimensionarray seems kind of long and > cryptic!). It should have minimal performance since no data gets copied, > and if it makes a difference I would be willing to implement it in C in > need be so that that performance impact was minimized. > > def paddims(a, n): > "return a view of 'a' with at least 'n' dimensions" > dims = a.shape > b = a.view() > b.shape = (1,)*(n - len(dims)) + dims > return b +1 (Boolean parameter is almost always a sign that a function is overloaded.) > > * copy -- Yes, I understand that the copy flag is probably not going to > change for backward compatibility reasons if nothing else, but there was > one last point I wanted to make about the copy flag. I am all for eliminating boolean parameters no matter what the consequences are. For me foo(bool, ...) is always a sign that it should be refactored to foo(...) an bar(...), with shared code possibly going to _baz(...). From ndarray at mac.com Sat Mar 25 10:13:05 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 10:13:05 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] masked_array drops fill_value In-Reply-To: References: Message-ID: Please post questions about "ma" on numpy-discussion. I don't like per-array fill_value feature. I believe fill_value should be a mandatory argument to the filled method and default fill_values only used internally to avoid passing bad data to ufuncs. The behavior that you describe seem to be historical. I would like to hear from other people who rely on per-array fill values before making any change. On 3/25/06, Tim Leslie wrote: > Hi all, > > In ma.py the masked_array function the fill_value of the array passed in is > not used. The code currently looks like this: > > def masked_array (a, mask=nomask, fill_value=None): > """masked_array(a, mask=nomask) = > array(a, mask=mask, copy=0, fill_value=fill_value) > """ > return array(a, mask=mask, copy=0, fill_value=fill_value) > > > It seems to me that using the fill_value from a (if it is a MaskedArray) > would be the sane thing to do? Something like > > if fill_value == None and isinstance(a, MaskedArray): > fill_value = a.fill_value() > return array(a, mask=mask, copy=0, fill_value=fill_value) > > As it stands, all the ma functions such as transpose, reshape, etc lose the > fill_value which seems wrong to me. > > Tim > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > > > From tim.hochberg at cox.net Sat Mar 25 15:29:13 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 25 15:29:13 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> Message-ID: <4425D222.7010301@cox.net> Sasha wrote: >On 3/24/06, Tim Hochberg wrote: > > [SNIP] > > >>* reshape -- The origination of that whole, enormous thread was with >>reshape and what its return behaviour should be. Currently it returns a >>view if possible, otherwise it returns a copy. That's evil behaviour. I >>believe that it should either always return a copy or return a view if >>possible and otherwise raise an exception. I think I actually prefer the >>later, although it's probably a more disruptive change, since its more >>flexible when combined with asarray and friends. One possible >>compromise would be to have numpy.reshape always copy, while >>array.reshape always returns a view. >> >> >> >Could you, please provide some examples of the current behavior that >you find undesired and how yuo would fix it? I believe the >inconsistency is that x.reshape(shape) is x depends on shape. This is >easy to fix. Do you also suggest that x.reshape(...).__array_data__ >is x.__array_data__ should be False? > > Hmmm. I should defer discussion of x.reshape(shape) because I've never used it . I only use the function reshape, not the method; I'm old school ;) If this thread keeps dragging on I imagine I'll have to go look into exactly what x.reshape does. As for the function, the issue I have with it is that: reshape(obj, shape) may return a view or a copy of 'obj'. Which depends at least on what 'obj' is and may depend on shape as well. I admit I don't remember -- nor should I have to. Here's a simple example. Suppose I have some image data that is stored in a flat array [R, G, B, R, G, B, ....]. Now suppose I have a function that converts this to grey scale in place by averaging the data def suck_color_from(obj): rbg = reshape(obj, [-1. 3]) grey = (rgb[:,0] + rgb[:,1] + rgb[:,2]) / 3.0 rgb[:,0] = rgb[:,1] = rgb[:,2] = grey Now, this will work find as long as reshape is returning a view. However, if reshape does not return a view, let's say obj is a list, it will fail. And not *just* fail, it will fail silently. There are obvious ways to fix this program (set obj.shape instead of using reshape, for example), but the fewer perils I need to remember the better. That's why I'd like to sweep all the view/shape indeterminacy into a few asXXXarray functions. Everything else should always return a view or always return a copy. Here's another example, in this case we are returning a new rgb object: def make_bland_picture_from(obj): obj = copy(obj) rbg = reshape(obj, [-1. 3]) grey = (rgb[:,0] + rgb[:,1] + rgb[:,2]) / 3.0 rgb[:,0] = rgb[:,1] = rgb[:,2] = grey return obj Again this works fine until someone passes in a list or other nonarray sequence. Then, again, it fails silently. If reshape either always returned a copy, or always returned a view, this kind of trap would not be available. I believe numpy.transpose also suffers from this behaviour. My preferred behaviour is that reshape and transpose always returns a view. That is, reshape(somelist) would become a type error. Always returning a copy makes it a little more tedius to write some kinds of code efficiently, however I still think it would be better than the current situation. As for x.copy(), while I don't know what the current situation is, I think it should always return a copy. Regards, Tim From tim.hochberg at cox.net Sat Mar 25 16:21:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 25 16:21:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> Message-ID: <4425DE36.60203@cox.net> Sasha wrote: >On 3/24/06, Tim Hochberg wrote: > > > [SNIP] > >>* copy -- Yes, I understand that the copy flag is probably not going to >>change for backward compatibility reasons if nothing else, but there was >>one last point I wanted to make about the copy flag. >> >> > >I am all for eliminating boolean parameters no matter what the >consequences are. For me foo(bool, ...) is always a sign that it >should be refactored to foo(...) an bar(...), with shared code >possibly going to _baz(...). > > > Regarding the backwards compatibility issues, on some reflection, they really don't seem that bad. Assuming the conversion programs I've heard about can take care of the other issues, for this one all that needs to be done is to search for "array(\(.*\), copy=0|False, \(.*\))" and replace it with "asarray(\1, \3)". [Those aren't real regex's, they're made up psuedo regex gobbeldy gook, but I'll come up with the real deal if there's any chance that'll it'd help push things along.] I'd expect the rate of false positives from this to be extremely low. Regards, -tim From ndarray at mac.com Sat Mar 25 17:45:05 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 17:45:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425D222.7010301@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> Message-ID: On 3/25/06, Tim Hochberg wrote: > ... > However, if reshape does not return a view, let's say obj is a list, it > will fail. And not *just* fail, it will fail silently. There are obvious > ways to fix this program (set obj.shape instead of using reshape, for > example), but the fewer perils I need to remember the better. Or, you can use the reshape method instead of function. I believe numpy advocates use of methods instead of functions. What you observe is just another reason for that. Since functions like reshape remain in numpy primarily for backwards compatibility, I would be against any change in semantics. From Fernando.Perez at colorado.edu Sat Mar 25 17:50:07 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sat Mar 25 17:50:07 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> Message-ID: <4425F326.5010904@colorado.edu> Sasha wrote: > On 3/25/06, Tim Hochberg wrote: > >>... >>However, if reshape does not return a view, let's say obj is a list, it >>will fail. And not *just* fail, it will fail silently. There are obvious >>ways to fix this program (set obj.shape instead of using reshape, for >>example), but the fewer perils I need to remember the better. > > > Or, you can use the reshape method instead of function. I believe > numpy advocates use of methods instead of functions. What you observe > is just another reason for that. Since functions like reshape remain > in numpy primarily for backwards compatibility, I would be against any > change in semantics. Mmh. I bet many people will continue to use the functional interface for a long time. I'd vote for uniform semantics before 1.0. Really, the whole 'reshape(foo) and foo.reshape() have different view/copy behavior' thing is horrible. WAY too easy to forget/confuse. Special cases are /almost never/ special enough to warrant this kind of extra mental overhead. At least I know /I/ will forget, get confused, and make mistakes. So I'd like to ask for as-uniform-as possible behavior. Cheers, f From strang at nmr.mgh.harvard.edu Sat Mar 25 18:01:02 2006 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Sat Mar 25 18:01:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: >> Or, you can use the reshape method instead of function. I believe >> numpy advocates use of methods instead of functions. What you observe >> is just another reason for that. Since functions like reshape remain >> in numpy primarily for backwards compatibility, I would be against any >> change in semantics. > > Mmh. I bet many people will continue to use the functional interface for a > long time. I'd vote for uniform semantics before 1.0. Really, the whole > 'reshape(foo) and foo.reshape() have different view/copy behavior' thing is > horrible. WAY too easy to forget/confuse. Special cases are /almost never/ > special enough to warrant this kind of extra mental overhead. +1 from a strongly interested user > At least I know /I/ will forget, get confused, and make mistakes. Me too. Probably at least once/week. ;-) > So I'd like to ask for as-uniform-as possible behavior. +1 -best Gary From ndarray at mac.com Sat Mar 25 18:05:01 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 18:05:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: On 3/25/06, Fernando Perez wrote: > At least I know /I/ will forget, get confused, and make mistakes. So I'd like > to ask for as-uniform-as possible behavior. Do I understand correctly that you and Tim propose to make reshape(obj, shape) return a view if obj is an instance of ndarray and throw an exception otherwise? If this is the case, it is probably beter to remove the reshape function alltogether. This way reshape(obj, shape) will very consistently throw a name error and will have to be replaced with obj.reshape(shape). From Fernando.Perez at colorado.edu Sat Mar 25 18:14:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sat Mar 25 18:14:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <4425F8BA.7050903@colorado.edu> Sasha wrote: > On 3/25/06, Fernando Perez wrote: > >>At least I know /I/ will forget, get confused, and make mistakes. So I'd like >>to ask for as-uniform-as possible behavior. > > > Do I understand correctly that you and Tim propose to make > reshape(obj, shape) return a view if obj is an instance of ndarray and > throw an exception otherwise? I don't propose anything in particular. I just don't want the reshape(foo) is a view on the even days of the week and a copy on the odd ones thingie :) All I'm asking is for an API that has as few special cases as possible, so that we can both learn (and teach) a small set of rules that apply everywhere. For example, the 'foo[a:b:c] for arrays is a view while a copy for a slice', while a special case, is justifiable to new users as necessary given the performance requirements of numpy (though there's been musings on python-dev of changing to such semantics in py3k, which would be great for us). But such special rules, whether regarding deviations against the core language or internally within numpy, should (IMHO) be kept to an absolute minimum. We should treat them as expensive specialties we must pay for with a very tight budget (the willingness and ability of users to keep track of the exponential rise in complexity induced by interlocking special cases). Cheers, f From Fernando.Perez at colorado.edu Sat Mar 25 18:19:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sat Mar 25 18:19:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F8BA.7050903@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <4425F8BA.7050903@colorado.edu> Message-ID: <4425F9FC.7050208@colorado.edu> Fernando Perez wrote: > For example, the 'foo[a:b:c] for arrays is a view while a copy for a slice', ^^^^^ That was 'list', of course. f From dhjjj at 455.com Sat Mar 25 18:49:02 2006 From: dhjjj at 455.com (fjkkk) Date: Sat Mar 25 18:49:02 2006 Subject: [Numpy-discussion] =?GB2312?B?uN+8ts7Ew9jWsNK1u6/RtS3Btw==?= Message-ID: ?? ?? 0755/82129135 82128600 82128650 ??????????????????????????????????????? ??????????????? ? ? ??????? ? ?2006?4?1-2??2006?4?8-9?? ??????????? ??????????????? ??????????????????????????????????????? ? ?? ????????????????????????????????????? ??????????????????????????????????????? ??????????????????????????????????????? ?????????????????? ????????????????????????????????????? ??????????????????????? ? ???? ? ??????????????????? ? ?????????????????????????????? ? ???????????????????? ? ???????????????? ? ?????????????????????? ? ????????????????????? ? ??????????????????????????? ? ?????????????????????????? ? ???????????????????????? ??????????????????????????????????????? ???? ????????????????? ? ???????????? ? ?????????? ? ??????????? ? ?????????????? ? ??????????????? ? ??????????? ? ??????????? ? ????? ? ?????????????? ??????????? ? ???? ? ????? ? ???????????? ? ???????? ? 13??????? ? ???? ? ?????????????? ? SMART???SWOT?? ? ???????? ????????? ? ????????? ? ??????????????????? ? ???? ? ????? ? ?????????? ? ????????? ? ????????? ? ???????????????????? ? ????????????? ? ????????????? ??????????? ? ?????? ? ??????? ? ????? ? ??????? ? ?????? ? ????? ? ????????????????? ???????? ? ???????? ? ??????? ? ????? ? ???????? ? ????????? ? ????????? ????????????? ? ????????? ? ??????????? ? ????????? ? ????????????????????? ? ??????????????? ? ??????????????? ? ?????????????? ? ?????????????? ? ????????????????? ? ?????????????? ? ???????????????????? ? ?????????????????????? ?????5S??????? ? ?????????????? ? ??????? ? ????????? ? ????????????? ????????????? ? ??????? ? ??????? ??????80/20?? ?????????? ????????????????? ? ????????? ? ????????? ? ??????????? ?????? ? ?????? ? ??????????? ? ?????? ? ????????? ???????????? ? ?????????? ? ???????????? ? ??????????? ? ?????????????? ? ??????????? ? ??????? ?????????????? 1?????????? ? ???TPO?? ? ??????? ? ??????? ? ?????? ? ????? ? ?????????? ? ??????? 2? ?????? ? ??????? ? ????? ? ????? ? ???????????? ? ???????????????? 3? ?????? ? ???? ? ???? ? ????? ? ?????? 4? ???? ? ????????? ? ????????? ? ????????? ??????????????????????????????????????? ???????? ????????????????????????????????????? ??????????????????????????????????????? ???????????????????????????????????????? ??????????????????????????????????????? ???????????????????????????????? ????????????????????????????????????? ?????????????????????????????????????? ???? ???????????????????????????????????? ???????????????????????? ???????????????????????????????????? ?????????????????????????????????????? ?????????????????????????????????????? ????????????? ???????????????????????????????????????? ? ???? ?????????????????????????????????????? ??????????????????????? ????? ????2006?3?31? ????? 4?1-2?(??) 4?8-9?(??) ????? ?????? ?????? ????? 1800?/?(???????????????????) ?????? ??????? ?? ?? 0755/82129135 82128600 82128650 ?? ?? 0755/61351396 82129209 ???xiaoyu9966333 at 126.com ?? ? ?? ??? ??? ???????????????????????????????????????? From tim.hochberg at cox.net Sat Mar 25 22:35:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 25 22:35:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> Message-ID: <442635F6.1050109@cox.net> Sasha wrote: >On 3/25/06, Tim Hochberg wrote: > > >>... >>However, if reshape does not return a view, let's say obj is a list, it >>will fail. And not *just* fail, it will fail silently. There are obvious >>ways to fix this program (set obj.shape instead of using reshape, for >>example), but the fewer perils I need to remember the better. >> >> > >Or, you can use the reshape method instead of function. I believe >numpy advocates use of methods instead of functions. What you observe >is just another reason for that. Since functions like reshape remain >in numpy primarily for backwards compatibility, I would be against any >change in semantics. > > That's not unreasonable. How would you feel about relegating reshape and transpose and whatever else is found to have this unfortunate characteristics to a backwards compatibility module so that old code can use it, but new code would not see it. That brings up another question: is the plan to keep oldnumeric around forever, or is it going away eventually? If it is going away, then the place to put these would be oldnumeric. Actually, it's OK if it sticks around as long as it doesn't end up in the numpy namespace by default. I agree with Fernando: I don't have a specific behaviour that I'm looking for in these as long as it's consistent. Just leaving it alone, but deprecating it and shunting it off to a compatibility module would be fine. Fine, as long as x.reshape behaves sensibly: does it always return a view? You (Sasha) said something about it maybe sometimes returning a copy. If so, that should be fixed. -tim From schofield at ftw.at Sun Mar 26 05:45:04 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun Mar 26 05:45:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442635F6.1050109@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: <057A5613-8FDC-49F8-B1C7-BBEBFDD9D8C0@ftw.at> On 26/03/2006, at 8:34 AM, Tim Hochberg wrote: > That brings up another question: is the plan to keep oldnumeric > around forever, or is it going away eventually? If it is going > away, then the place to put these would be oldnumeric. Actually, > it's OK if it sticks around as long as it doesn't end up in the > numpy namespace by default. I think this is a great idea. Let's remove the functions in oldnumeric.py from the numpy namespace by default. People migrating from Numeric could add "from numpy.core.oldnumeric import *" at the top of each source file for compatibility. This would send a clear message that the functions in this module are deprecated and there for compatibility reasons only, and that the methods are the preferred interface. A clear division like this between the old and the new would also simplify decisions on when to leave the existing behaviour alone and when it should be changed for the better. Then the problem Fernando sees with the reshape function and method having different copy / view behaviour would be easier to explain and understand -- the functions just do the old Numeric thing, whereas the methods are consistent with each other. I'm now a fan of method interfaces in Python. One reason is that, with .reshape(), .sum() etc. as methods, rather than functions, it's possible for me to make the behaviour of SciPy's sparse matrices highly consistent with NumPy's arrays and matrices, without NumPy needing hooks for this. But making numpy.sum etc accept either dense or sparse matrices would require extensive modification to NumPy. -- Ed From ndarray at mac.com Sun Mar 26 05:54:01 2006 From: ndarray at mac.com (Sasha) Date: Sun Mar 26 05:54:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442635F6.1050109@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: On 3/26/06, Tim Hochberg wrote: > ... > That brings up another question: is the plan to keep oldnumeric around > forever, or is it going away eventually? If it is going away, then the > place to put these would be oldnumeric. Actually, it's OK if it sticks > around as long as it doesn't end up in the numpy namespace by default. > It *is* in oldnumeric. However, I don't really know what plans for oldnumeric are. Travis? > ... > be fine. Fine, as long as x.reshape behaves sensibly: does it always > return a view? You (Sasha) said something about it maybe sometimes > returning a copy. If so, that should be fixed. I never said that. What I said was x.reshape(shape) returns x itself if shape==x.shape and a view othrwise. This should probably be fixed because it may lead to inconsistent behavior of functions that manipulate metadata after reshape. From cjw at sympatico.ca Sun Mar 26 06:16:06 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 26 06:16:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <057A5613-8FDC-49F8-B1C7-BBEBFDD9D8C0@ftw.at> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <057A5613-8FDC-49F8-B1C7-BBEBFDD9D8C0@ftw.at> Message-ID: <4426A22A.5020503@sympatico.ca> Ed Schofield wrote: > > On 26/03/2006, at 8:34 AM, Tim Hochberg wrote: > >> That brings up another question: is the plan to keep oldnumeric >> around forever, or is it going away eventually? If it is going away, >> then the place to put these would be oldnumeric. Actually, it's OK >> if it sticks around as long as it doesn't end up in the numpy >> namespace by default. > > > I think this is a great idea. Let's remove the functions in > oldnumeric.py from the numpy namespace by default. People migrating > from Numeric could add "from numpy.core.oldnumeric import *" at the > top of each source file for compatibility. This would send a clear > message that the functions in this module are deprecated and there > for compatibility reasons only, and that the methods are the > preferred interface. A clear division like this between the old and > the new would also simplify decisions on when to leave the existing > behaviour alone and when it should be changed for the better. Then > the problem Fernando sees with the reshape function and method having > different copy / view behaviour would be easier to explain and > understand -- the functions just do the old Numeric thing, whereas > the methods are consistent with each other. > +1 > I'm now a fan of method interfaces in Python. One reason is that, > with .reshape(), .sum() etc. as methods, rather than functions, it's > possible for me to make the behaviour of SciPy's sparse matrices > highly consistent with NumPy's arrays and matrices, without NumPy > needing hooks for this. But making numpy.sum etc accept either dense > or sparse matrices would require extensive modification to NumPy. > Can sparse matrices not be handled in a sub-class? Colin W. > -- Ed From cjw at sympatico.ca Sun Mar 26 06:17:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 26 06:17:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <442694A9.9080108@sympatico.ca> Fernando Perez wrote: > Sasha wrote: > >> On 3/25/06, Tim Hochberg wrote: >> >>> ... >>> However, if reshape does not return a view, let's say obj is a list, it >>> will fail. And not *just* fail, it will fail silently. There are >>> obvious >>> ways to fix this program (set obj.shape instead of using reshape, for >>> example), but the fewer perils I need to remember the better. >> >> >> >> Or, you can use the reshape method instead of function. I believe >> numpy advocates use of methods instead of functions. What you observe >> is just another reason for that. Since functions like reshape remain >> in numpy primarily for backwards compatibility, I would be against any >> change in semantics. > > > Mmh. I bet many people will continue to use the functional interface > for a long time. I'd vote for uniform semantics before 1.0. Really, > the whole 'reshape(foo) and foo.reshape() have different view/copy > behavior' thing is horrible. WAY too easy to forget/confuse. Special > cases are /almost never/ special enough to warrant this kind of extra > mental overhead. > > At least I know /I/ will forget, get confused, and make mistakes. So > I'd like to ask for as-uniform-as possible behavior. > +1 Perhaps most of the functions could be in a compatibility module, which the user would import as needed, with most cases defined as something like: reshape(arr, d1, d2, ...)= numpy.ndarray.reshape(arr,d1, d2, ...) Colin W. > Cheers, > > f > From bill_lang at ir.hit.edu.cn Sun Mar 26 06:17:05 2006 From: bill_lang at ir.hit.edu.cn (Bill_Lang) Date: Sun Mar 26 06:17:05 2006 Subject: [Numpy-discussion] Problem on install Numpy. Thanks~! Message-ID: <20060326141249.D949D1B005C@ir.hit.edu.cn> Hi Numpy list members, Just now, I have run numpy-0.9.6r1.win32-py2.4.exe which was download from sourceforge. Then there is a new dictionary under "C:\Python24\Lib\site-packages\". And I checked the sys.path. the numby path is included in sys.path. But when I import numeric. There is a hint that "ImportError: No module named numeric". Is there anybody could help me to solve this problem? Thanks a lot~! BTW: I am a freshman of Python. Yet, I love Python now. Best wishes; Bill_Lang -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Sun Mar 26 06:28:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 26 06:28:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: <4426A4EF.4080603@sympatico.ca> Sasha wrote: >On 3/26/06, Tim Hochberg wrote: > > >>... >>That brings up another question: is the plan to keep oldnumeric around >>forever, or is it going away eventually? If it is going away, then the >>place to put these would be oldnumeric. Actually, it's OK if it sticks >>around as long as it doesn't end up in the numpy namespace by default. >> >> >> > >It *is* in oldnumeric. However, I don't really know what plans for >oldnumeric are. Travis? > > An essential part of Ed's suggestion, as I understood it, was that the names in oldnumeric would not be hauled into numpy. The user could import oldnumeric as needed. As time goes by, the need will decline. Some consideration might be given to extending this to other non-core packages. This would reduce time required to handle basic core startup i.e. the elements being proposed for inclusion in core Python 2.6. Colin W. >>... >>be fine. Fine, as long as x.reshape behaves sensibly: does it always >>return a view? You (Sasha) said something about it maybe sometimes >>returning a copy. If so, that should be fixed. >> >> > >I never said that. What I said was x.reshape(shape) returns x itself >if shape==x.shape and a view othrwise. This should probably be fixed >because it may lead to inconsistent behavior of functions that >manipulate metadata after reshape. > > > From aisaac at american.edu Sun Mar 26 07:50:01 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun Mar 26 07:50:01 2006 Subject: [Numpy-discussion] Problem on install Numpy. Thanks~! In-Reply-To: <20060326141249.D949D1B005C@ir.hit.edu.cn> References: <20060326141249.D949D1B005C@ir.hit.edu.cn> Message-ID: On Sun, 26 Mar 2006, Bill_Lang apparently wrote: > But when I import numeric. There is a hint that > "ImportError: No module named numeric". Is there anybody > could help me to solve this problem? Just looking for the most obvious possibility ... Do you import numpy ??? (That's not what you said.) Cheers, Alan Isaac From efiring at hawaii.edu Sun Mar 26 11:00:04 2006 From: efiring at hawaii.edu (Eric Firing) Date: Sun Mar 26 11:00:04 2006 Subject: [Numpy-discussion] Problem on install Numpy. Thanks~! In-Reply-To: <20060326141249.D949D1B005C@ir.hit.edu.cn> References: <20060326141249.D949D1B005C@ir.hit.edu.cn> Message-ID: <4426E475.7070601@hawaii.edu> Bill_Lang, The module is called "numpy", not "numeric", so you need to do: import numpy Eric Bill_Lang wrote: > Hi Numpy list members, > > Just now, I have run numpy-0.9.6r1.win32-py2.4.exe which was > download from sourceforge. Then there is a new dictionary under > "C:\Python24\Lib\site-packages\". And I checked the sys.path. the numby > path is included in sys.path. > > But when I import numeric. There is a hint that "ImportError: No module > named numeric". From zpincus at stanford.edu Sun Mar 26 17:11:21 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Mar 26 17:11:21 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442694A9.9080108@sympatico.ca> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <442694A9.9080108@sympatico.ca> Message-ID: <3A2F069E-5806-4822-A9C0-E32AC5F0510C@stanford.edu> Speaking of differences between numpy functions and ndarray methods, numpy.resize(...) pads out enlarged arrays with copies of the contents of that array, while ndarray.resize(...) pads out enlarged regions with zeros. Should these be reconciled as well? I'm partial to the former case, but if there was (perhaps there already is?) another simple way to achieve this result, then it wouldn't matter which semantics were chosen. To retain compatibility with Numeric, either numpy.resize would need to retain its current semantics, or a function that does the same task (expanding and padding out an array with copies of its contents) would need to be added, so that references to resize in old code could be replaced by calls to that function. Zach >> Mmh. I bet many people will continue to use the functional >> interface for a long time. I'd vote for uniform semantics before >> 1.0. Really, the whole 'reshape(foo) and foo.reshape() have >> different view/copy behavior' thing is horrible. WAY too easy to >> forget/confuse. Special cases are /almost never/ special enough >> to warrant this kind of extra mental overhead. >> >> At least I know /I/ will forget, get confused, and make mistakes. >> So I'd like to ask for as-uniform-as possible behavior. >> > +1 > Perhaps most of the functions could be in a compatibility module, > which the user would import as needed, with most cases defined as > something like: > reshape(arr, d1, d2, ...)= numpy.ndarray.reshape(arr,d1, d2, ...) From lapataia28-northern at yahoo.com Sun Mar 26 17:51:14 2006 From: lapataia28-northern at yahoo.com (Lalo) Date: Sun Mar 26 17:51:14 2006 Subject: [Numpy-discussion] Cygwin Message-ID: <20060327015015.2402.qmail@web36208.mail.mud.yahoo.com> Hi, I tried to compile numpy 0.9.6 on a standard Cygwin installation but got some errors. Do I need to modify the site.cfg defaults? Thank you for your help. Lalo ------------------------------------------------------ Running from numpy source directory. /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=['/usr/lib/liblapack.a'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=['/usr/lib/liblapack.dll.a'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:1264: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['blas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack_atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:1183: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ F2PY Version 2_2236 blas_opt_info: blas_mkl_info: NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_info: FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: numpy.distutils.system_info.atlas_info NOT AVAILABLE lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running install running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources adding 'build/src/numpy/core/config.h' to sources. adding 'build/src/numpy/core/__multiarray_api.h' to sources. adding 'build/src/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src/numpy/core/src/scalartypes.inc', 'build/src/numpy/core/src/arraytypes.inc', 'build/src/numpy/core/config.h', 'build/src/numpy/core/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src/numpy/core/config.h' to sources. adding 'build/src/numpy/core/__ufunc_api.h' to sources. adding 'build/src/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src/numpy/core/src/scalartypes.inc', 'build/src/numpy/core/src/arraytypes.inc', 'build/src/numpy/core/config.h', 'build/src/numpy/core/__ufunc_api.h'] building extension "numpy.core._sort" sources adding 'build/src/numpy/core/config.h' to sources. adding 'build/src/numpy/core/__multiarray_api.h' to sources. numpy.core - nothing done with h_files= ['build/src/numpy/core/config.h', 'build/src/numpy/core/__multiarray_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.dft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. building extension "numpy.random.mtrand" sources Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes' compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c:7:2: #error No _WIN32 _configtest.c:7:2: #error No _WIN32 failure. removing: _configtest.c _configtest.o building data_files sources running build_py copying build/src/numpy/__config__.py -> build/lib.cygwin-1.5.19-i686-2.4/numpy copying build/src/numpy/distutils/__config__.py -> build/lib.cygwin-1.5.19-i686-2.4/numpy/distutils running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using build_ext building 'numpy.core.multiarray' extension compiling C sources gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes' compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.19-i686-2.4/numpy/core/src/multiarraymodule.o -L. -lpython2.4 -o build/lib.cygwin-1.5.19-i686-2.4/numpy/core/multiarray.dll /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython2.4 collect2: ld returned 1 exit status /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython2.4 collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.19-i686-2.4/numpy/core/src/multiarraymodule.o -L. -lpython2.4 -o build/lib.cygwin-1.5.19-i686-2.4/numpy/core/multiarray.dll" failed with exit status 1 From wbaxter at gmail.com Sun Mar 26 18:15:03 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sun Mar 26 18:15:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: On 3/26/06, Fernando Perez wrote: > > > Or, you can use the reshape method instead of function. I believe > > numpy advocates use of methods instead of functions. What you observe > > is just another reason for that. Since functions like reshape remain > > in numpy primarily for backwards compatibility, I would be against any > > change in semantics. > > Mmh. I bet many people will continue to use the functional interface for > a > long time. And maybe they'd be right to want a functional interface, too: http://www.gotw.ca/gotw/084.htm The author argues that in general, you should make as few functions as possible be members of a class, always preferring non-member functions wherever possible. It's about C++, but the principle should apply to Python as well. It does make a certain kind of sense, although the article seems to completely ignore polymorphism, which seems like it would be another good reason to have functions be members, at least in C++. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sun Mar 26 18:53:10 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 26 18:53:10 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: <4427535E.4060609@ieee.org> Sasha wrote: > I never said that. What I said was x.reshape(shape) returns x itself > if shape==x.shape and a view othrwise. Is this true? I just checked on it and it didn't seem to be the case. Perhaps this was "fixed" during the recent reshape bug-fix. -Travis From tim.hochberg at cox.net Sun Mar 26 19:02:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 26 19:02:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4427535E.4060609@ieee.org> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <4427535E.4060609@ieee.org> Message-ID: <44275585.6010309@cox.net> Travis Oliphant wrote: > Sasha wrote: > >> I never said that. What I said was x.reshape(shape) returns x itself >> if shape==x.shape and a view othrwise. > > Is this true? I just checked on it and it didn't seem to be the > case. Perhaps this was "fixed" during the recent reshape bug-fix. It was at least true in the fairly recent past: >>> numpy.__version__ '0.9.7.2275' >>> a = numpy.arange(9) >>> b = a.reshape(a.shape) >>> b is a True >>> c = a.reshape((3,3)) >>> c is a False I can't get through to svn.scipy.org right now to do an update, so I can't check on current SVN, but I believe that this version is less than a week old. -tim From tim.hochberg at cox.net Sun Mar 26 20:34:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 26 20:34:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <44276AF0.5070405@cox.net> Bill Baxter wrote: > > > On 3/26/06, *Fernando Perez* > wrote: > > > Or, you can use the reshape method instead of function. I believe > > numpy advocates use of methods instead of functions. What you > observe > > is just another reason for that. Since functions like reshape > remain > > in numpy primarily for backwards compatibility, I would be > against any > > change in semantics. > > Mmh. I bet many people will continue to use the functional > interface for a > long time. > > > > And maybe they'd be right to want a functional interface, too: > http://www.gotw.ca/gotw/084.htm > > > The author argues that in general, you should make as few functions as > possible be members of a class, always preferring non-member functions > wherever possible. It's about C++, but the principle should apply to > Python as well. It does make a certain kind of sense, although the > article seems to completely ignore polymorphism, which seems like it > would be another good reason to have functions be members, at least in > C++. Personally, I've always prefered the functional interface, but it appears that for historical / backwards compatibility reasons our choice is between a functional interface that is broken and a working method based interface. For that reason alone, I'm now in favor of ditching the functional interface to 'reshape' and 'transpose'. As to whether 'reshape' and 'transpose' should be functions or methods in an ideal world, I think it's worth considering them together. It's straightforward to come up with a version of 'reshape' that's fairly generic: def reshape(obj, shape): val = obj.view() val.shape = shape return val [Unlike the current version of 'reshape', this one always returns a view; that is on purpose] This will reshape any object that supports both the 'view' method and the 'shape' attribute. Writing something equivalent for 'transpose' does not appear possible. I could do something like: def transpose(obj, axes=None): if axes == None: axes = reversed(range(len(obj.shape))) else: axes = list(axes) shape = numpy.asarray(obj.shape)[axes] strides = numpy.asarray(obj.strides)[axes] val = obj.view() val.shape = shape val.strides = strides return val However, while I consider it reasonable to require all array-like objects to have 'view' and 'shape', I doubt that it's reasonable to have them need 'strides' as well. What would 'strides' mean for a sparse array for example? Therefore, 'transpose' should be a method. Since I've always considered 'reshape' and 'transpose' a pair and would find it disconcerting if one were a method while the other was a function (this may be a personal foible however), so I'm coming to the view that they should probably have been methods all along. Regards, -tim From charlesr.harris at gmail.com Sun Mar 26 20:48:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun Mar 26 20:48:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: On 3/25/06, Fernando Perez wrote: > > Sasha wrote: > > On 3/25/06, Tim Hochberg wrote: > > > >>... > >>However, if reshape does not return a view, let's say obj is a list, it > >>will fail. And not *just* fail, it will fail silently. There are obvious > >>ways to fix this program (set obj.shape instead of using reshape, for > >>example), but the fewer perils I need to remember the better. > > > > > > Or, you can use the reshape method instead of function. I believe > > numpy advocates use of methods instead of functions. What you observe > > is just another reason for that. Since functions like reshape remain > > in numpy primarily for backwards compatibility, I would be against any > > change in semantics. > > Mmh. I bet many people will continue to use the functional interface for > a > long time. I'd vote for uniform semantics before 1.0. Really, the whole > 'reshape(foo) and foo.reshape() have different view/copy behavior' thing > is > horrible. WAY too easy to forget/confuse. Special cases are /almost > never/ > special enough to warrant this kind of extra mental overhead. Pimpf. I just stopped using reshape. If I wanted a different shape I assigned the tuple. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gruben at bigpond.net.au Mon Mar 27 04:46:03 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Mon Mar 27 04:46:03 2006 Subject: [Numpy-discussion] (offlist) Numpy implementation of IDL & pdl's rebin? In-Reply-To: <441DF726.1080603@auckland.ac.nz> References: <4414620C.3040202@auckland.ac.nz> <441A7831.7010704@bigpond.net.au> <441DF726.1080603@auckland.ac.nz> Message-ID: <4427E9B0.3040807@bigpond.net.au> Recently a question was asked about whether numpy/scipy had a data rebin command similar to IDL's rebin. IDL's rebin can rebin data to smaller or larger arrays whose dimensions are factors or multiples of the original size. I have had a play and placed some code which takes care of the reducing case for ndarrays on the wiki here: Gary R. From cimrman3 at ntc.zcu.cz Mon Mar 27 07:16:02 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon Mar 27 07:16:02 2006 Subject: [Numpy-discussion] scipy.sparse + umfpack + system_info Message-ID: <44280161.4030708@ntc.zcu.cz> Hi all, I have added a basic umfpack detection support to numpy.distutils.system_info() - see example site.cfg below (and replace ...). The Umfpack wrapper still resides in the sandbox, but its setup.py now uses the detection stuff. Now I would like to move the wrapper somewhere under scipy.sparse and I hit a problem: umpfack.py depends on scipy.sparse, since it uses the CSR/CSC matrix formats, but also scipy.sparse uses umfpack.py for solving linear equations.if present. How to get out of this circular dependency problem? Do you have any suggestions on how to organize the files? r. -- site.cfg: [umfpack] library_dirs = /UMFPACK/Lib:/AMD/Lib include_dirs = /UMFPACK/Include umfpack_libs = umfpack, amd From anewgene at gmail.com Mon Mar 27 08:37:04 2006 From: anewgene at gmail.com (CL) Date: Mon Mar 27 08:37:04 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array Message-ID: <44281477.9080500@hotpop.com> Hi, group, I need to get the "rank" of an 1-D array (ie. a vector). Note that "rank" here is not the value returned from "rank(a_array)". It is the order of the item in its sorted arrray. For example, I have a python function called "listrank" to return the "rank" as below: In [19]: x Out[19]: array([1, 2, 5, 3, 3, 2]) In [20]: listrank(x) Out[20]: [6, 4, 1, 2, 2, 4] Somebody suggested me to use "argsort(argsort(x))". But the problem is it does not handle ties. See the output: In [21]: argsort(argsort(x)) Out[21]: array([0, 1, 5, 3, 4, 2]) I am wondering if there is a solution in numpy/numarray/numeric to get this done nicely. Thanks. CL From cjw at sympatico.ca Mon Mar 27 08:49:12 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Mar 27 08:49:12 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44276AF0.5070405@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> Message-ID: <4428177F.4030602@sympatico.ca> Tim Hochberg wrote: > Bill Baxter wrote: > >> >> >> On 3/26/06, *Fernando Perez* > > wrote: >> >> > Or, you can use the reshape method instead of function. I believe >> > numpy advocates use of methods instead of functions. What you >> observe >> > is just another reason for that. Since functions like reshape >> remain >> > in numpy primarily for backwards compatibility, I would be >> against any >> > change in semantics. >> >> Mmh. I bet many people will continue to use the functional >> interface for a >> long time. >> >> >> And maybe they'd be right to want a functional interface, too: >> http://www.gotw.ca/gotw/084.htm >> >> >> The author argues that in general, you should make as few functions >> as possible be members of a class, always preferring non-member >> functions wherever possible. It's about C++, but the principle >> should apply to Python as well. It does make a certain kind of >> sense, although the article seems to completely ignore polymorphism, >> which seems like it would be another good reason to have functions be >> members, at least in C++. > > > > Personally, I've always prefered the functional interface, but it > appears that for historical / backwards compatibility reasons our > choice is between a functional interface that is broken and a working > method based interface. For that reason alone, I'm now in favor of > ditching the functional interface to 'reshape' and 'transpose'. The transpose function handles a list and returns an array. Of course the case could be made that things would be clearer if the user wrote the longer: a= array([[1, 2]]).transpose() in place of: a=transpose([[1, 2]]) Perhaps extension of the T property to the array would simplify things: a=array([[1, 2]]).T Colin W. > > As to whether 'reshape' and 'transpose' should be functions or methods > in an ideal world, I think it's worth considering them together. It's > straightforward to come up with a version of 'reshape' that's fairly > generic: > > def reshape(obj, shape): > val = obj.view() > val.shape = shape > return val > > [Unlike the current version of 'reshape', this one always returns a > view; that is on purpose] This will reshape any object that supports > both the 'view' method and the 'shape' attribute. Writing something > equivalent for 'transpose' does not appear possible. I could do > something like: > > def transpose(obj, axes=None): > if axes == None: > axes = reversed(range(len(obj.shape))) > else: > axes = list(axes) > shape = numpy.asarray(obj.shape)[axes] > strides = numpy.asarray(obj.strides)[axes] > val = obj.view() > val.shape = shape > val.strides = strides > return val > > However, while I consider it reasonable to require all array-like > objects to have 'view' and 'shape', I doubt that it's reasonable to > have them need 'strides' as well. What would 'strides' mean for a > sparse array for example? Therefore, 'transpose' should be a method. > Since I've always considered 'reshape' and 'transpose' a pair and > would find it disconcerting if one were a method while the other was a > function (this may be a personal foible however), so I'm coming to the > view that they should probably have been methods all along. > > Regards, > > -tim > > > > > > > > > From tim.hochberg at cox.net Mon Mar 27 09:13:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 09:13:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428177F.4030602@sympatico.ca> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> Message-ID: <44281CE3.8080800@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Bill Baxter wrote: >> >>> >>> >>> On 3/26/06, *Fernando Perez* >> > wrote: >>> >>> > Or, you can use the reshape method instead of function. I >>> believe >>> > numpy advocates use of methods instead of functions. What you >>> observe >>> > is just another reason for that. Since functions like reshape >>> remain >>> > in numpy primarily for backwards compatibility, I would be >>> against any >>> > change in semantics. >>> >>> Mmh. I bet many people will continue to use the functional >>> interface for a >>> long time. >>> >>> >>> And maybe they'd be right to want a functional interface, too: >>> http://www.gotw.ca/gotw/084.htm >>> >>> >>> The author argues that in general, you should make as few functions >>> as possible be members of a class, always preferring non-member >>> functions wherever possible. It's about C++, but the principle >>> should apply to Python as well. It does make a certain kind of >>> sense, although the article seems to completely ignore polymorphism, >>> which seems like it would be another good reason to have functions >>> be members, at least in C++. >> >> >> >> >> Personally, I've always prefered the functional interface, but it >> appears that for historical / backwards compatibility reasons our >> choice is between a functional interface that is broken and a working >> method based interface. For that reason alone, I'm now in favor of >> ditching the functional interface to 'reshape' and 'transpose'. > > > The transpose function handles a list and returns an array. Yes. And this is exactly the same property that I've been ranting about incessantly for the past several days in the case of reshape. It seems convenient, but it's evil. It leads to intermittent, hard to find bugs, since when passed a list it necessarily creates a new array object, but when passed an array, it returns a view. > Of course the case could be made that things would be clearer if the > user wrote the longer: > a= array([[1, 2]]).transpose() > in place of: > a=transpose([[1, 2]]) The best way to do deal with this is to make sure everything is an array at the boundaries of your function. def frobinate(a, b): a = asarray(a) b = asarray(b) bar = a.transpose() stool = a.reshape([1,2,3]) # etc, c = asarray(get_data_from_somewhere_that_might_not_return_an_array()) # etc. It would easy enough to automate this using decorators if the extra lines bothers people. Thus the above could become: @array_arguments def frobinate(a, b): bar = a.transpose() stool = a.reshape([1,2,3]) # etc, c = asarray(get_data_from_somewhere_that_might_not_return_an_array()) # etc. In either case, then you can use all of the array functions without worry, and with maximum efficiency, in the body of your function. > Perhaps extension of the T property to the array would simplify things: > a=array([[1, 2]]).T I tend to think that the 'T' property should stay put. 'T' is really specific to two dimensional arrays and looses much of its appeal when applied to multidimensional arrays since it takes no arguments. I think it adds more clutter than functionality, so I'm -1 on that idea. Regards, -tim From tim.hochberg at cox.net Mon Mar 27 09:51:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 09:51:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4427535E.4060609@ieee.org> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <4427535E.4060609@ieee.org> Message-ID: <442825C6.6060707@cox.net> Travis Oliphant wrote: > Sasha wrote: > >> I never said that. What I said was x.reshape(shape) returns x itself >> if shape==x.shape and a view othrwise. > > Is this true? I just checked on it and it didn't seem to be the > case. Perhaps this was "fixed" during the recent reshape bug-fix. > It looks like the bug is in PyArray_Ravel. array_reshape calls PyArray_Ravel if the newshape has dimension-1 (line 77 of array_methods.c): if (newshape.len == 1) { PyDimMem_FREE(newshape.ptr); return PyArray_Ravel(self, 0); } Ravel in turn, returns self if the array is already flat (line 187 of multiarray_module.c): if (!fortran && PyArray_ISCONTIGUOUS(a)) { if (a->nd == 1) { Py_INCREF(a); return (PyObject *)a; } return PyArray_Newshape(a, &newdim, PyArray_CORDER); } It looks like ripping out the second if statement would fix everything, but I haven't tried it yet. -tim From tim.hochberg at cox.net Mon Mar 27 10:09:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 10:09:07 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array In-Reply-To: <44281477.9080500@hotpop.com> References: <44281477.9080500@hotpop.com> Message-ID: <442829EE.7050204@cox.net> CL wrote: > Hi, group, > I need to get the "rank" of an 1-D array (ie. a vector). > Note that "rank" here is not the value returned from "rank(a_array)". > It is the order of the item in its sorted arrray. For example, I have > a python function called "listrank" to return the "rank" as below: In the future, please include the relevant function. This saves us (me anyway) time reverse engineering said function from the description you give. Is the function below equivalent to your listrank function? def listrank(v): rank = {} for i, x in enumerate(reversed(sorted(v))): if x not in rank: rank[x] = i return [rank[x]+1 for x in v] > > In [19]: x > Out[19]: array([1, 2, 5, 3, 3, 2]) > > In [20]: listrank(x) > Out[20]: [6, 4, 1, 2, 2, 4] > > Somebody suggested me to use "argsort(argsort(x))". But the problem is > it does not handle ties. See the output: > > In [21]: argsort(argsort(x)) > Out[21]: array([0, 1, 5, 3, 4, 2]) > > I am wondering if there is a solution in numpy/numarray/numeric to get > this done nicely. Unfortunately, nothing comes to mind immediately. This kind of problem, where the values at one index depend on the values at a different index is often hard to deal with in the array framework. How large of vectors are you typically dealing with? If they are not extremely large or this isn't a performance critical a python solution like above, possibly somewhat optimized, may well be sufficient. Perhaps someone else will come up with something though. Regards, -tim From oliphant at ee.byu.edu Mon Mar 27 10:46:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 10:46:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44281CE3.8080800@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> Message-ID: <4428329B.7050300@ee.byu.edu> Tim Hochberg wrote: >> >> >> The transpose function handles a list and returns an array. > > > Yes. And this is exactly the same property that I've been ranting > about incessantly for the past several days in the case of reshape. It > seems convenient, but it's evil. It leads to intermittent, hard to > find bugs, since when passed a list it necessarily creates a new array > object, but when passed an array, it returns a view. > And this is also the area where Tim is going to find people that disagree. The functional interfaces have always been there so that other objects could be manipulated as arrays. There is *a lot* of precendent behind the functional interfaces. It seems like a big change to simply get rid of them . Yes, having this ability means that you have to think about it a bit if you are going to use the functional interface and try to do in-place operations. But, I would argue that this is an "advanced" usage which is implemented to save space and time. I fully support such "advanced" usages, but believe the person using them should know what they are doing. Understanding why the functional interface may return a copy does not seem to me to be a big deal for somebody thinking about "in-place" operations. I'm -1 right now on removing (or even deprecating) the functional interfaces. I'm +1 on educating users on the advantages of using methods and that the functional interfaces are just basic wrappers around them. -Travis From oliphant at ee.byu.edu Mon Mar 27 10:48:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 10:48:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442825C6.6060707@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <4427535E.4060609@ieee.org> <442825C6.6060707@cox.net> Message-ID: <44283339.8080007@ee.byu.edu> > > Ravel in turn, returns self if the array is already flat (line 187 of > multiarray_module.c): > > if (!fortran && PyArray_ISCONTIGUOUS(a)) { > if (a->nd == 1) { > Py_INCREF(a); > return (PyObject *)a; > } > return PyArray_Newshape(a, &newdim, PyArray_CORDER); > } I just eliminated this so that .ravel() always returns a new view. I can see this optimization leads to an undesirable inconsistency. I also added a special case for FORTRANORDER arrays so that is consistent with the CORDER case as well. -Travis From oliphant at ee.byu.edu Mon Mar 27 10:58:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 10:58:00 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <3A2F069E-5806-4822-A9C0-E32AC5F0510C@stanford.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <442694A9.9080108@sympatico.ca> <3A2F069E-5806-4822-A9C0-E32AC5F0510C@stanford.edu> Message-ID: <44283577.7010809@ee.byu.edu> Zachary Pincus wrote: > Speaking of differences between numpy functions and ndarray methods, > numpy.resize(...) pads out enlarged arrays with copies of the > contents of that array, while ndarray.resize(...) pads out enlarged > regions with zeros. Yep, this is a documented difference. I can see desireability of both behaviors, but I agree that using the same name for different behaviors (without a keyword flag) is not a good idea. > > Should these be reconciled as well? I'm partial to the former case, > but if there was (perhaps there already is?) another simple way to > achieve this result, then it wouldn't matter which semantics were > chosen. > > To retain compatibility with Numeric, either numpy.resize would need > to retain its current semantics, or a function that does the same > task (expanding and padding out an array with copies of its contents) > would need to be added, so that references to resize in old code > could be replaced by calls to that function. I think for consistency we should make a new function and then add it to the list of replaced functions in convertcode.py The other possibility that I'm liking more is to use a keyword argument that specifies a fill-value. The default for the method could be 0 while the default for the resize-function could be the array itself. True, different default keywords may not be convenient (but that exists already with the axis keyword for many methods and is done for backwards compatibility). All the code review is great. Please continue it. -Travis From Chris.Barker at noaa.gov Mon Mar 27 11:08:00 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Mar 27 11:08:00 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <44283805.40101@noaa.gov> Sasha wrote: > If this is the case, it is probably better to remove the reshape > function altogether. +1 I'm a big fan of OO syntax -- it really cleans up name spaces. Colin J. Williams wrote: > Perhaps most of the functions could be in a compatibility module, which > the user would import as needed, with most cases defined as something like: > reshape(arr, d1, d2, ...)= numpy.ndarray.reshape(arr,d1, d2, ...) +1 Ed Schofield wrote: > I'm now a fan of method interfaces in Python. One reason is that, with > .reshape(), .sum() etc. as methods, rather than functions, it's possible > for me to make the behaviour of SciPy's sparse matrices highly > consistent with NumPy's arrays and matrices Another good reason for an OO interface! Bill Baxter wrote: > And maybe they'd be right to want a functional interface, too: > http://www.gotw.ca/gotw/084.htm I can see that this applies to sin(), cos() etc, but from that doc: """Prefer to make it a member if it needs access to internals:""" I think reshape() and friends fall into this category. I always thought there was an odd argument in the old Numeric docs: """ Choosing functions means that the same procedures can be applied to arbitrary python sequences, not just to arrays. For example, while: >> transpose([[1,2], [3,4]]) works just fine, >> [[1,2], [3,4]].transpose() can't work. """ For me, that argument would only hold if transposing a list returned a transposed list, but it doesn't: it returns an array. What's going on under the hood is something like: return asarray(TheInput).transpose() The user can not use transpose() without committing to arrays, so why not just have them make the asarray call. All you're doing is creating a less explicit interface and cluttering the name space to save some typing. As far I'm concerned, most functions that have a signature like: Array = function(Array, whatever) should probably be methods. I know there's history to consider, so I don't actually expect this to change, and I'm very happy with the new methods introduced by numpy. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tim.hochberg at cox.net Mon Mar 27 11:08:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 11:08:03 2006 Subject: [Numpy-discussion] order flag again Message-ID: <442837E4.8020303@cox.net> I notice that some methods of the array object, notably reshape, take the order flag. I think this is a mistake. At this point, x.reshape() should always be returning a view. That means it's a cheap operation. Similarly, setting the order flag is also a cheap operation[*]. For that reason there's not a pressing efficiency reason to combine to the two operations and I feel that they should be broken apart. Thus we'd gain an operation 'setorder' or similar and lose the order flag everywhere but array and the various asXXXarray[**] functions since they need it to efficiently construct a correctly ordered array. Thus: b = a.reshape(newshape, order=neworder) would become: b = a.reshape(newshape).setorder(neworder) That doesn't look substantially better in that form, but compare: b = a.reshape(weeble, order=wobble) with: b = a.reshape(weeble).setorder(wobble) And the superiority of the second form begins to becomes apparent. The requirement that order be keyword parameter helps, but not enough IMO. The second form is orthogonal and self documenting and just all around more swell ;) Regards, -tim [*] I know this even though I haven't thought through what reshape plus the order flag is doing because reshape isn't allowed to return a copy. Thus setting the order flag in this context must amount to just rejiggering shape and strides. [**] And I suppose, maybe, ravel. However, ravel is yet another evil function, or rather method in this case, that sometimes returns a view and sometimes a copy. Ravels is really unnecessary, and I don't have the energy to wage a campaign against its evil ways (much to everyones relief I imagine), so I plan to just ignore it and recomend that no one use it . Ever. From anewgene at gmail.com Mon Mar 27 11:25:11 2006 From: anewgene at gmail.com (CL) Date: Mon Mar 27 11:25:11 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array In-Reply-To: <442829EE.7050204@cox.net> References: <44281477.9080500@hotpop.com> <442829EE.7050204@cox.net> Message-ID: <44283BEE.5090802@hotpop.com> Thanks, Tim. Your function of "listrank" is indeed equivalent with mine. My typical vector size is 150K and there are thousands of such vectors need to be processed. I think the performace would be boosted if there is any way other than pure python fucntion "listrank". Thanks again, CL Tim Hochberg wrote: > CL wrote: > >> Hi, group, >> I need to get the "rank" of an 1-D array (ie. a vector). >> Note that "rank" here is not the value returned from "rank(a_array)". >> It is the order of the item in its sorted arrray. For example, I have >> a python function called "listrank" to return the "rank" as below: > > > In the future, please include the relevant function. This saves us (me > anyway) time reverse engineering said function from the description > you give. Is the function below equivalent to your listrank function? > > def listrank(v): > rank = {} > for i, x in enumerate(reversed(sorted(v))): > if x not in rank: > rank[x] = i > return [rank[x]+1 for x in v] > >> >> In [19]: x >> Out[19]: array([1, 2, 5, 3, 3, 2]) >> >> In [20]: listrank(x) >> Out[20]: [6, 4, 1, 2, 2, 4] >> >> Somebody suggested me to use "argsort(argsort(x))". But the problem >> is it does not handle ties. See the output: >> >> In [21]: argsort(argsort(x)) >> Out[21]: array([0, 1, 5, 3, 4, 2]) >> >> I am wondering if there is a solution in numpy/numarray/numeric to >> get this done nicely. > > > Unfortunately, nothing comes to mind immediately. This kind of > problem, where the values at one index depend on the values at a > different index is often hard to deal with in the array framework. How > large of vectors are you typically dealing with? If they are not > extremely large or this isn't a performance critical a python solution > like above, possibly somewhat optimized, may well be sufficient. > > Perhaps someone else will come up with something though. > > Regards, > > -tim > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From tim.hochberg at cox.net Mon Mar 27 11:43:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 11:43:04 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array In-Reply-To: <44283BEE.5090802@hotpop.com> References: <44281477.9080500@hotpop.com> <442829EE.7050204@cox.net> <44283BEE.5090802@hotpop.com> Message-ID: <44284001.2080108@cox.net> CL wrote: > Thanks, Tim. Your function of "listrank" is indeed equivalent with > mine. My typical vector size is 150K and there are thousands of such > vectors need to be processed. Ouch! > I think the performace would be boosted if there is any way other than > pure python fucntion "listrank". OK. Well still no great ideas on this end. If you have some a priori knowledge about the vector, there might be some tricks using take. But that would require that the values in 'v; be integers that are on the order of len(v): is that the case? In any event, the following is considerably faster than the previous version: def listrank_4(v): rank = {} for i, x in enumerate(numpy.sort(v)[::-1]): if x not in rank: rank[x] = i return numpy.array([rank[x] for x in v]) + 1 The execution time here appears to be dominated by the time it takes to insert items in the dictionary. If we know enough about the items of 'v', we could potentially replace the dictionary with a vector and speed things about quite a bit more. Regards, -tim > > Thanks again, > > CL > > Tim Hochberg wrote: > >> CL wrote: >> >>> Hi, group, >>> I need to get the "rank" of an 1-D array (ie. a vector). >>> Note that "rank" here is not the value returned from >>> "rank(a_array)". It is the order of the item in its sorted arrray. >>> For example, I have a python function called "listrank" to return >>> the "rank" as below: >> >> >> >> In the future, please include the relevant function. This saves us >> (me anyway) time reverse engineering said function from the >> description you give. Is the function below equivalent to your >> listrank function? >> >> def listrank(v): >> rank = {} >> for i, x in enumerate(reversed(sorted(v))): >> if x not in rank: >> rank[x] = i >> return [rank[x]+1 for x in v] >> >>> >>> In [19]: x >>> Out[19]: array([1, 2, 5, 3, 3, 2]) >>> >>> In [20]: listrank(x) >>> Out[20]: [6, 4, 1, 2, 2, 4] >>> >>> Somebody suggested me to use "argsort(argsort(x))". But the problem >>> is it does not handle ties. See the output: >>> >>> In [21]: argsort(argsort(x)) >>> Out[21]: array([0, 1, 5, 3, 4, 2]) >>> >>> I am wondering if there is a solution in numpy/numarray/numeric to >>> get this done nicely. >> >> >> >> Unfortunately, nothing comes to mind immediately. This kind of >> problem, where the values at one index depend on the values at a >> different index is often hard to deal with in the array framework. >> How large of vectors are you typically dealing with? If they are not >> extremely large or this isn't a performance critical a python >> solution like above, possibly somewhat optimized, may well be >> sufficient. >> >> Perhaps someone else will come up with something though. >> >> Regards, >> >> -tim >> >> >> >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by xPML, a groundbreaking scripting >> language >> that extends applications into web and mobile media. Attend the live >> webcast >> and join the prime developer group breaking into this new coding >> territory! >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Mon Mar 27 11:49:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 11:49:05 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <442837E4.8020303@cox.net> References: <442837E4.8020303@cox.net> Message-ID: <4428418A.1070606@ee.byu.edu> Tim Hochberg wrote: > > I notice that some methods of the array object, notably reshape, take > the order flag. I think this is a mistake. At this point, x.reshape() > should always be returning a view. The reason for the order flag is that it is needed to interpret what you think a reshape operation should do. How should the reshape method think the data is stored. That is the intended meaning of the reshape flag. The FORTRAN flag on the array means the strides are actually equivalent to FORTRAN order The CONTIGUOUS flag on the array means the strides are actually equilvalent to C-CONTIGUOUS order. For a non-contiguous array, you may still have a sense of which dimension you think ought to vary the fastest. This kind of "sense" is not needed all the time --- only on certain operations. For these operations, the order keyword needs to be present. To do what Tim suggests would require the addition of an additional flag (call it the ORDER flag) on the array that would be carried around to tell everybody how to think about this array as a one-dimensional sequence of bytes. Such an approach is possible, of course. -Travis From zpincus at stanford.edu Mon Mar 27 12:14:56 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 27 12:14:56 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428418A.1070606@ee.byu.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> Message-ID: <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> > The reason for the order flag is that it is needed to interpret > what you think a reshape operation should do. How should the > reshape method think the data is stored. That is the intended > meaning of the reshape flag. > The FORTRAN flag on the array means the strides are actually > equivalent to FORTRAN order > The CONTIGUOUS flag on the array means the strides are actually > equilvalent to C-CONTIGUOUS order. > [...] > To do what Tim suggests would require the addition of an additional > flag (call it the ORDER flag) on the array that would be carried > around to tell everybody how to think about this array as a one- > dimensional sequence of bytes. > Such an approach is possible, of course. Does this mean that if I create a new array with FORTRAN order from the numpy.array() function, every time I reshape that array I need to tell reshape to use FORTRAN order? That seems a bit odd... Yet the text above seems to suggests that arrays will not know what order their own bytes are in, and thus would need to be told whether they should be reshaped like FORTRAN or C memory blocks. I assume I'm misinterpreting something here? Zach From tim.hochberg at cox.net Mon Mar 27 12:18:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 12:18:07 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428329B.7050300@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> Message-ID: <4428484F.2030604@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> >>> The transpose function handles a list and returns an array. >> >> >> >> Yes. And this is exactly the same property that I've been ranting >> about incessantly for the past several days in the case of reshape. >> It seems convenient, but it's evil. It leads to intermittent, hard to >> find bugs, since when passed a list it necessarily creates a new >> array object, but when passed an array, it returns a view. >> > > And this is also the area where Tim is going to find people that > disagree. The functional interfaces have always been there so that > other objects could be manipulated as arrays. There is *a lot* of > precendent behind the functional interfaces. It seems like a big > change to simply get rid of them . Yes. I certainly hope to get disagreement. While I'm certain that I'm right, this is a big change and I would feel uncomfortable if it were adopted without being thoroughly thrashed out. This way, if it does get adopted, I'll feel comfortable that it was fully vetted. And if doesn't, I get to say I told you so every time it bites someone ;) And who knows perhaps you can change my mind. > > Yes, having this ability means that you have to think about it a bit > if you are going to use the functional interface and try to do > in-place operations. But, I would argue that this is an "advanced" > usage which is implemented to save space and time. How is this true though? In what way, for instance, is: b = asarray(a).reshape(newshape) slower or less space efficient than todays: b = reshape(a) ? If the answer is just the overhead of the call to asarray, I'll rewrite it in C for you at which point the overhead will truly be negligible. [Note I don't encourage the former style. In general I think all objects you want to use as arrays should be converted at the function boundaries.] In fact, I'd argue just the opposite. The function encourage people to *not* convert objects to arrays at the boundaries of their functions. As a result, in addition to getting subtle, intermitent bugs, they tend to end up converting from objects to arrays multiple times inside a given function; wasting both space and time. Let me muddy the waters a bit by saying that the subtle bugs you can get from reshape accepting lists pale next to the issue that whether you get a view or a copy depends on what the strides of the array are. Who keeps track of strides? I'd forgotten about that until I was looking at ravel, at which point I recalled the true horror of the function that is reshape. For those of you playing along at home: >>> a = numpy.arange(9).reshape([3,3]) >>> b = a[1:,1:] >>> flat_a = a.reshape(-1) >>> flat_b = b.reshape(-1) >>> flat_a[0] = 999 >>> flat_b[0] = 999 >>> a array([[999, 1, 2], [ 3, 4, 5], [ 6, 7, 8]]) >>> b array([[4, 5], [7, 8]]) !!! > I fully support such "advanced" usages, but believe the person using > them should know what they are doing. Understanding why the > functional interface may return a copy does not seem to me to be a big > deal for somebody thinking about "in-place" operations. If I truly saw any benefit to these, other than saving the typing of a few characters, I'd agree. However, I don't see such benefit. I don't see them advanced either as they offer no benefit over the corresponding "basic" usages other than marginally less typing. And you pay for that as it encourages bad practices and results in subtle bugs. > > I'm -1 right now on removing (or even deprecating) the functional > interfaces. How about just segregating them. You can even call the submodule "advanced" if you like, although I'd prefer "danger_danger_warning", so perhaps "functions" would be a good compromise. Then I, and other like-minded numpy users, can simply ban the use of the "functions" submodule from our code and live blissful, efficient, bug free existences. Or something like that. > I'm +1 on educating users on the advantages of using methods and > that the functional interfaces are just basic wrappers around them. Let me restate that I'm not method-ist (nor a Methodist for that matter), I'd be perfectly happy with functions. However, it seems impossible for backwards compatibility and cultural reasons to have non-broken function semantics. So for that reason I'm in favor of deprecating or segregating the borked functions. I'm not actually in favor of removing them altogether, as that would unnecessarily complicate migration from Numeric, but I am in favor of booting them out of the main numpy namespace. Regards, Tim From tim.hochberg at cox.net Mon Mar 27 12:48:51 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 12:48:51 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428418A.1070606@ee.byu.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> Message-ID: <44284F3E.1080102@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> I notice that some methods of the array object, notably reshape, take >> the order flag. I think this is a mistake. At this point, x.reshape() >> should always be returning a view. > > > The reason for the order flag is that it is needed to interpret what > you think a reshape operation should do. How should the reshape > method think the data is stored. That is the intended meaning of the > reshape flag. > The FORTRAN flag on the array means the strides are actually > equivalent to FORTRAN order > The CONTIGUOUS flag on the array means the strides are actually > equilvalent to C-CONTIGUOUS order. > > For a non-contiguous array, you may still have a sense of which > dimension you think ought to vary the fastest. This kind of "sense" > is not needed all the time --- only on certain operations. For these > operations, the order keyword needs to be present. > To do what Tim suggests would require the addition of an additional > flag (call it the ORDER flag) on the array that would be carried > around to tell everybody how to think about this array as a > one-dimensional sequence of bytes. > Such an approach is possible, of course. OK, so the order flag is needed anytime you might want to copy the data, correct? What x.reshape() should do is still up in the air, but I feel strongly that it shouldn't be copying *sometimes*. I'm going to assume for the moment that my previous proposal regarding reshape and ravel gets adopted. We have three use cases: 1. Want a reshaped view: b = a.reshape(*newshape) 2. Want a reshaped copy b = a.ravel(*newshape=[-1], order=neworder) 3. Want a reshaped view if possible or a copy: b = ascontiguousarray(a, order=neworder).reshape(*newshape) Reshape doesn't need the order flag since it's not in the business of making copies. If you want a copy, use the new, revamped ravel which makes a copy and respects the order[*]. Now this third case is what I would consider an advanced usage as described in Travis' previous email. You want a contiguous array, probably flat, and you want it as efficiently as possible. However it's also a potential source of subtle bugs since you may or may not get a view. For this reason, I think that requiring ascontiguousarray to support this case is not only acceptable, it's a very good thing. As I've mentioned before, I'd like to sweep all of the operations that may return a view or a copy into a few asXXXarray functions so that these potentially problematic operations are localized and easily identifiable. Regards, -tim [*] I've never been convinced of the need for ravel myself, since it's always been possible to write it efficiently some other way. However, it's more useful under the current proposal and I have no desire to get rid of it as long as it's behaviour stops being evil! From Fernando.Perez at colorado.edu Mon Mar 27 12:57:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 27 12:57:05 2006 Subject: [Numpy-discussion] order flag again Message-ID: <4428516E.4070308@colorado.edu> On 01:46 PM Tim Hochberg wrote: > As I've mentioned before, I'd like to sweep all of the operations that may > return a view or a copy into a few asXXXarray functions so that these > potentially problematic operations are localized and easily identifiable. +1 on the idea that: asXXXarray: functions which take their input and return always an array, but trying to avoid copies if at all possible. This means that code calling such functions should treat their returned values strictly as /inputs/, never modifying them in place. These functions are provided by numpy strictly for performance reasons. Or something like that. I like being able to tell to users that this kind of problem is solved by a unified familiy of functions in the API, with a clear naming convention. There is a need for them, but they are also a bit dangerous, so they should be clearly labeled. Painting them all in the same bright red helps indicate their 'sharp blade' nature. Cheers, f From charlesr.harris at gmail.com Mon Mar 27 13:06:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 13:06:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428484F.2030604@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> Message-ID: On 3/27/06, Tim Hochberg wrote: > > Travis Oliphant wrote: > > > Tim Hochberg wrote: > > > >>> > >>> > >>> The transpose function handles a list and returns an array. > >> > >> > >> > >> Yes. And this is exactly the same property that I've been ranting > >> about incessantly for the past several days in the case of reshape. > >> It seems convenient, but it's evil. It leads to intermittent, hard to > >> find bugs, since when passed a list it necessarily creates a new > >> array object, but when passed an array, it returns a view. > >> > > > > And this is also the area where Tim is going to find people that > > disagree. The functional interfaces have always been there so that > > other objects could be manipulated as arrays. There is *a lot* of > > precendent behind the functional interfaces. It seems like a big > > change to simply get rid of them . > > Yes. I certainly hope to get disagreement. While I'm certain that I'm > right, this is a big change and I would feel uncomfortable if it were > adopted without being thoroughly thrashed out. This way, if it does get > adopted, I'll feel comfortable that it was fully vetted. And if doesn't, > I get to say I told you so every time it bites someone ;) And who knows > perhaps you can change my mind. > > > > > Yes, having this ability means that you have to think about it a bit > > if you are going to use the functional interface and try to do > > in-place operations. But, I would argue that this is an "advanced" > > usage which is implemented to save space and time. > > How is this true though? In what way, for instance, is: > > b = asarray(a).reshape(newshape) > > slower or less space efficient than todays: > > b = reshape(a) > > ? If the answer is just the overhead of the call to asarray, I'll > rewrite it in C for you at which point the overhead will truly be > negligible. [Note I don't encourage the former style. In general I think > all objects you want to use as arrays should be converted at the > function boundaries.] > > In fact, I'd argue just the opposite. The function encourage people to > *not* convert objects to arrays at the boundaries of their functions. As > a result, in addition to getting subtle, intermitent bugs, they tend to > end up converting from objects to arrays multiple times inside a given > function; wasting both space and time. > > Let me muddy the waters a bit by saying that the subtle bugs you can get > from reshape accepting lists pale next to the issue that whether you get > a view or a copy depends on what the strides of the array are. Who keeps > track of strides? I'd forgotten about that until I was looking at ravel, > at which point I recalled the true horror of the function that is > reshape. For those of you playing along at home: > > >>> a = numpy.arange(9).reshape([3,3]) > >>> b = a[1:,1:] > >>> flat_a = a.reshape(-1) > >>> flat_b = b.reshape(-1) > >>> flat_a[0] = 999 > >>> flat_b[0] = 999 > >>> a > array([[999, 1, 2], > [ 3, 4, 5], > [ 6, 7, 8]]) > >>> b > array([[4, 5], > [7, 8]]) > > !!! > > > I fully support such "advanced" usages, but believe the person using > > them should know what they are doing. Understanding why the > > functional interface may return a copy does not seem to me to be a big > > deal for somebody thinking about "in-place" operations. > > If I truly saw any benefit to these, other than saving the typing of a > few characters, I'd agree. However, I don't see such benefit. I don't > see them advanced either as they offer no benefit over the corresponding > "basic" usages other than marginally less typing. And you pay for that > as it encourages bad practices and results in subtle bugs. > > > > > I'm -1 right now on removing (or even deprecating) the functional > > interfaces. > > How about just segregating them. You can even call the submodule > "advanced" if you like, although I'd prefer "danger_danger_warning", so > perhaps "functions" would be a good compromise. Then I, and other > like-minded numpy users, can simply ban the use of the "functions" > submodule from our code and live blissful, efficient, bug free > existences. Or something like that. > > > I'm +1 on educating users on the advantages of using methods and > > that the functional interfaces are just basic wrappers around them. > > Let me restate that I'm not method-ist (nor a Methodist for that > matter), I'd be perfectly happy with functions. However, it seems > impossible for backwards compatibility and cultural reasons to have > non-broken function semantics. So for that reason I'm in favor of > deprecating or segregating the borked functions. I'm not actually in > favor of removing them altogether, as that would unnecessarily > complicate migration from Numeric, but I am in favor of booting them out > of the main numpy namespace. How about functions always return a copy unless they are carefully documented interface helper functions like asarray. That's how I generally think of them anyway. Of course, the main virtue of asarray is that it helps write functions that do precisely what Tim is arguing against: mixing list and array types and returning copies in the first case, views in the second. So if we throw out functions we should throw out asarray also and make a rule that numpy functions don't take lists. And then there are scalars... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Mar 27 13:19:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 13:19:02 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428516E.4070308@colorado.edu> References: <4428516E.4070308@colorado.edu> Message-ID: On 3/27/06, Fernando Perez wrote: > > On 01:46 PM Tim Hochberg wrote: > > > As I've mentioned before, I'd like to sweep all of the operations that > may > > return a view or a copy into a few asXXXarray functions so that these > > potentially problematic operations are localized and easily > identifiable. > > +1 on the idea that: > > asXXXarray: functions which take their input and return always an array, > but > trying to avoid copies if at all possible. This means that code calling > such > functions should treat their returned values strictly as /inputs/, never > modifying them in place. These functions are provided by numpy strictly > for > performance reasons. How about Tim's example: b = asarray(a).reshape(newshape) Or something like that. I think the problem is mixing python list types with arrays. It is convenient but we get into trouble. What purpose does ascontiguous serve? I like being able to tell to users that this kind of > problem is solved by a unified familiy of functions in the API, with a > clear > naming convention. There is a need for them, but they are also a bit > dangerous, so they should be clearly labeled. Painting them all in the > same > bright red helps indicate their 'sharp blade' nature. > > Cheers, > > f > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Mon Mar 27 13:29:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 13:29:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> Message-ID: <442858F5.6030001@cox.net> Charles R Harris wrote: [CHOP] > > How about functions always return a copy unless they are carefully > documented interface helper functions like asarray. That's how I > generally think of them anyway. I don't really care as long as they (a) stop being evil and (b) there's some way to do stuff efficiently. Functions that always return copies seem sort of useless to me, but I'd be happy to move to methods if that was the case. However, I suspect you won't make much headway with this since my impression is that many of the people who do care about functions also care passionately about efficiency as well. > Of course, the main virtue of asarray is that it helps write functions > that do precisely what Tim is arguing against: mixing list and array > types and returning copies in the first case, views in the second. So > if we throw out functions we should throw out asarray also and make a > rule that numpy functions don't take lists. This isn't right. I think it's perfectly fine, desirable in fact, for functions to operate on both list and array types. The problem only arises when you return ether a view or copy of one of the original inputs depending on what it was. In my experience, this is relatively hard to do in all but the most trivial of functions since most operations return a new array. However, it is true that I think people should be strongly discouraged from doing this. For example: def add(a, b): a = asarray(a) b = asarray(b) return a + b is fine, but: def iadd(a, b): a = asarray(a) b = asarray(b) a += b return a is not and should be rewritten. Possibly like: def iadd(a, b): if not isinstance(a, ndarray): raise TypeError("'a' must be an array") b = asarray(b) a += b return a since += is a bit squirley. (Try somelist += somearray, to see what I mean). > And then there are scalars... I'm curious what problems you forsee with scalars. Regards, -tim From Fernando.Perez at colorado.edu Mon Mar 27 13:31:12 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 27 13:31:12 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: References: <4428516E.4070308@colorado.edu> Message-ID: <44285970.5080008@colorado.edu> Charles R Harris wrote: > What purpose does ascontiguous serve? One I can think of: making sure that C code of the type double foo(double *array,long num_elements) { ... } can be called in C as foo((double *)numpy_arr->data,numpy_arr->dimensions[0]); (the dims name in C may be different, I'm not looking at code right now) and be OK with walking the given memory region as a contiguous chunk of doubles, num_elements of them. As long as foo() doesn't modify its input, this works fine and it's nice to have such guarantees: it makes writing extension code a lot easier. Cheers, f From oliphant at ee.byu.edu Mon Mar 27 13:43:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 13:43:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428484F.2030604@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> Message-ID: <44285C3D.9080802@ee.byu.edu> Tim Hochberg wrote: >> >> Yes, having this ability means that you have to think about it a bit >> if you are going to use the functional interface and try to do >> in-place operations. But, I would argue that this is an "advanced" >> usage which is implemented to save space and time. > > > How is this true though? In what way, for instance, is: > > b = asarray(a).reshape(newshape) > > slower or less space efficient than todays: > > b = reshape(a) Well, the big difference is that b=reshape(a) is actually try: reshape = a.reshape except AttributeError: return a.__array_wrap__(asarray(a).reshape(newshape)) return reshape(newshape) So, it handles more cases, more cleanly then the asarray(a).rehsape(newshape) approach does. -Travis From chanley at stsci.edu Mon Mar 27 13:48:08 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 27 13:48:08 2006 Subject: [Numpy-discussion] problem with ndarray attributes Message-ID: <44285D68.1060908@stsci.edu> Hi Travis, I have run across a feature in numpy which I am not sure how to handle. One of the new features of the ndarray is that named fields can be accessed as attributes. So, if a user created a recarray with fields named "address", "phone", and "names" they would be able to access all of the names in the recarray objects with "recarrayObject.names". I have found that this can cause problems when a user creates a table with a field name that happen to coincide with an existing class attribute or method. In particular, I am having a problem in pyfits where a user has a table with a column named "field". This column name is causing an attribute "field" to be created that overrides the existing class field method. It would seem to me that existing class methods and attributes should take precedence over the attributes created to correspond to column names. Is there a way to implement this behavior in ndarray? Thank you for your time and help, Chris From oliphant at ee.byu.edu Mon Mar 27 14:03:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 14:03:05 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <44285D68.1060908@stsci.edu> References: <44285D68.1060908@stsci.edu> Message-ID: <4428610B.4040307@ee.byu.edu> Christopher Hanley wrote: > Hi Travis, > > I have run across a feature in numpy which I am not sure how to > handle. One of the new features of the ndarray is that named fields > can be accessed as attributes. So, if a user created a recarray with > fields named "address", "phone", and "names" they would be able to > access all of the names in the recarray objects with > "recarrayObject.names". To be clear. This is only true for a recarray subclass. Standard ndarray's cannot look-up fields with attribute access. > > I have found that this can cause problems when a user creates a table > with a field name that happen to coincide with an existing class > attribute or method. In particular, I am having a problem in pyfits > where a user has a table with a column named "field". This column > name is causing an attribute "field" to be created that overrides the > existing class field method. This is a standard problem with using attribute access. I suppose it would be better to always lookup the attributes first on the object and then if that fails use field access. That way library code would not break if a user happened to name their fields after a method or property. Users would have to change code if new methods or properties were added (although that won't be as common). > > It would seem to me that existing class methods and attributes should > take precedence over the attributes created to correspond to column > names. Is there a way to implement this behavior in ndarray? I think this is the right thing to do. Are there any objections? -Travis From tim.hochberg at cox.net Mon Mar 27 14:09:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:09:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44285C3D.9080802@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> Message-ID: <44286266.9050503@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> Yes, having this ability means that you have to think about it a bit >>> if you are going to use the functional interface and try to do >>> in-place operations. But, I would argue that this is an "advanced" >>> usage which is implemented to save space and time. >> >> >> >> How is this true though? In what way, for instance, is: >> >> b = asarray(a).reshape(newshape) >> >> slower or less space efficient than todays: >> >> b = reshape(a) > > > > Well, the big difference is that b=reshape(a) is actually > > try: > reshape = a.reshape > except AttributeError: > return a.__array_wrap__(asarray(a).reshape(newshape)) > > return reshape(newshape) > > So, it handles more cases, more cleanly then the > asarray(a).rehsape(newshape) approach does. OK. Although I should have said asanyarray(a).reshape(newshape), I still see how that this handles more cases. I do not agree that it is worth the attendant drawbacks and I would still like to see these functions spun off into a separate namespace. I need to go think about this case some more. This is not something that I've run into in practice, but I can see I'll have a stronger case if I can come up with a alternative to this in terms of safe functions. Do you have some examples of objects that do not have 'reshape', but do have '__array_wrap__'? -tim From perry at stsci.edu Mon Mar 27 14:11:05 2006 From: perry at stsci.edu (Perry Greenfield) Date: Mon Mar 27 14:11:05 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <4428610B.4040307@ee.byu.edu> References: <44285D68.1060908@stsci.edu> <4428610B.4040307@ee.byu.edu> Message-ID: <1e7e648a95a431db8c5a3844781d3211@stsci.edu> On Mar 27, 2006, at 5:02 PM, Travis Oliphant wrote: > > This is a standard problem with using attribute access. I suppose it > would be better to always lookup the attributes first on the object > and then if that fails use field access. That way library code would > not break if a user happened to name their fields after a method or > property. Users would have to change code if new methods or > properties were added (although that won't be as common). I think some mechanism must be built in to make it easy to add new attributes and methods (e.g., some method to append new object attributes that are protected?). That way subclassing record arrays wouldn't have to do their own checks on this but rather register their new methods and attributes as things that would not get overridden (or is there some more magic way of doing this that doesn't require something this explicit?) >> >> It would seem to me that existing class methods and attributes should >> take precedence over the attributes created to correspond to column >> names. Is there a way to implement this behavior in ndarray? > > > I think this is the right thing to do. Are there any objections? > Seems like the obvious thing. Otherwise any tool one builds that allows a user to use arbitrary field names can suddenly break if a user uses the wrong name. With the proposed change, there is always the field method to get around any name collisions. Perry From oliphant at ee.byu.edu Mon Mar 27 14:15:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 14:15:00 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <44285D68.1060908@stsci.edu> References: <44285D68.1060908@stsci.edu> Message-ID: <442863A7.1080803@ee.byu.edu> Christopher Hanley wrote: > Hi Travis, > > I have run across a feature in numpy which I am not sure how to > handle. One of the new features of the ndarray is that named fields > can be accessed as attributes. So, if a user created a recarray with > fields named "address", "phone", and "names" they would be able to > access all of the names in the recarray objects with > "recarrayObject.names". > > I have found that this can cause problems when a user creates a table > with a field name that happen to coincide with an existing class > attribute or method. In particular, I am having a problem in pyfits > where a user has a table with a column named "field". This column > name is causing an attribute "field" to be created that overrides the > existing class field method. > > It would seem to me that existing class methods and attributes should > take precedence over the attributes created to correspond to column > names. Is there a way to implement this behavior in ndarray? > Now that I've written that I think the class methods and attributes should come first, I should mention, however, that a library writer could write ndarray.__getattribute__(a,'field')(*args) to make sure that the method a.field(*args) always gets called. This seems like quite a burden to force on library writers, though. I'd rather have users be aware of what they are using as field names since the methods and attributes on arrays should not be changing excessively. -Travis From charlesr.harris at gmail.com Mon Mar 27 14:16:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 14:16:02 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <44285970.5080008@colorado.edu> References: <4428516E.4070308@colorado.edu> <44285970.5080008@colorado.edu> Message-ID: On 3/27/06, Fernando Perez wrote: > > Charles R Harris wrote: > > > What purpose does ascontiguous serve? > > One I can think of: making sure that C code of the type > > double foo(double *array,long num_elements) { ... } > > can be called in C as > > foo((double *)numpy_arr->data,numpy_arr->dimensions[0]); > > (the dims name in C may be different, I'm not looking at code right now) > > and be OK with walking the given memory region as a contiguous chunk of > doubles, num_elements of them. > > As long as foo() doesn't modify its input, this works fine and it's nice > to > have such guarantees: it makes writing extension code a lot easier. > OK, it's an interface. Since getting stuff back from C will require a copy if the original is not contiquous, a test involving a.iscontiguous will be needed and ascontiguous can be dispensed with. Isn't there a C version of this anyway? ISTR that numarray had one. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 14:25:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 14:25:03 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> Message-ID: <4428660A.2020607@ee.byu.edu> Zachary Pincus wrote: >> The reason for the order flag is that it is needed to interpret what >> you think a reshape operation should do. How should the reshape >> method think the data is stored. That is the intended meaning of >> the reshape flag. >> The FORTRAN flag on the array means the strides are actually >> equivalent to FORTRAN order >> The CONTIGUOUS flag on the array means the strides are actually >> equilvalent to C-CONTIGUOUS order. >> [...] >> To do what Tim suggests would require the addition of an additional >> flag (call it the ORDER flag) on the array that would be carried >> around to tell everybody how to think about this array as a one- >> dimensional sequence of bytes. >> Such an approach is possible, of course. > > > > Does this mean that if I create a new array with FORTRAN order from > the numpy.array() function, every time I reshape that array I need to > tell reshape to use FORTRAN order? That seems a bit odd... Yes, that is what it means. Doing anything else leads to a host of problems. Let me illustrate. a = rand(10,3) b = a.transpose() b.flags # b is now in Fortran order b.ravel() # do you really want this to now be interpreted in FORTRAN-order. I didn't think so. What this illustrates is that the FORTRAN and CONTIGUOUS flags on the array are just special cases of the strides (and in fact can be updated from the strides at any time). But, an ORDER flag would be an independent concept that could be set and reset at will. The ORDER flag is only necessary when somebody is interpreting the array as a linear sequence of bytes. If the ORDER flag is added, then the Array iterator would also need to be updated, to recognize and respect the order flag. Right now, the array iterator (a.flat) is always C-contiguous. > > Yet the text above seems to suggests that arrays will not know what > order their own bytes are in, and thus would need to be told whether > they should be reshaped like FORTRAN or C memory blocks. > > I assume I'm misinterpreting something here? It wasn't until a few weeks ago that I think I really understood the difference between ORDER='FORTRAN' and flags & FORTRAN == True. The order information is really an additional but related piece of information . The example above illustrates how you could have flags & FORTRAN == True but order == 'C' -Travis From tim.hochberg at cox.net Mon Mar 27 14:45:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:45:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> Message-ID: <44286AAE.3030204@cox.net> Charles R Harris wrote: > > > On 3/27/06, *Tim Hochberg* > wrote: > > Charles R Harris wrote: > [CHOP] > > > > > How about functions always return a copy unless they are carefully > > documented interface helper functions like asarray. That's how I > > generally think of them anyway. > > I don't really care as long as they (a) stop being evil and (b) > there's > some way to do stuff efficiently. Functions that always return copies > seem sort of useless to me, but I'd be happy to move to methods if > that > was the case. However, I suspect you won't make much headway with this > since my impression is that many of the people who do care about > functions also care passionately about efficiency as well. > > > Of course, the main virtue of asarray is that it helps write > functions > > that do precisely what Tim is arguing against: mixing list and array > > types and returning copies in the first case, views in the > second. So > > if we throw out functions we should throw out asarray also and > make a > > rule that numpy functions don't take lists. > > This isn't right. I think it's perfectly fine, desirable in fact, for > functions to operate on both list and array types. The problem only > arises when you return ether a view or copy of one of the original > inputs depending on what it was. In my experience, this is relatively > hard to do in all but the most trivial of functions since most > operations return a new array. However, it is true that I think people > should be strongly discouraged from doing this. For example: > > def add(a, b): > a = asarray(a) > b = asarray(b) > return a + b > > is fine, but: > > def iadd(a, b): > a = asarray(a) > b = asarray(b) > a += b > return a > > is not and should be rewritten. Possibly like: > > def iadd(a, b): > if not isinstance(a, ndarray): raise TypeError("'a' must be an > array") > b = asarray(b) > a += b > return a > > since += is a bit squirley. (Try somelist += somearray, to see > what I mean). > > > It's a syntax error, which I believe is the right thing. Really? If so, this is a version thing. This is what I get running on Python 2.4.1 and current numpy SVN (0.9.7.2286): >>> l = list(a) >>> l [999, 1, 2, 3, 4, 5, 6, 7, 8] >>> a array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>> l += a >>> l array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>> a array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > And then there are scalars... > > I'm curious what problems you forsee with scalars. > > > There was/is the whole problem of when to return scalars and if they > should be python scalars. The problem arises from trying to be > compatible with normal python stuff. Matlab got around this by making > everything 1) an array, and 2) double precision. On the other hand, > Matlab is not nearly is easy to use for nonstandard types. \ I think most of this problem will go away in Python with the introduction of the __index__ protocol, so for the moment at least I'm not worried about it. Regards, -tim From tim.hochberg at cox.net Mon Mar 27 14:51:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:51:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286AAE.3030204@cox.net> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: <44286C1C.5010605@cox.net> Tim Hochberg wrote: > Charles R Harris wrote: > >> >> >> On 3/27/06, *Tim Hochberg* > > wrote: >> >> Charles R Harris wrote: >> [CHOP] >> >> > >> > How about functions always return a copy unless they are carefully >> > documented interface helper functions like asarray. That's how I >> > generally think of them anyway. >> >> I don't really care as long as they (a) stop being evil and (b) >> there's >> some way to do stuff efficiently. Functions that always return >> copies >> seem sort of useless to me, but I'd be happy to move to methods if >> that >> was the case. However, I suspect you won't make much headway with >> this >> since my impression is that many of the people who do care about >> functions also care passionately about efficiency as well. >> >> > Of course, the main virtue of asarray is that it helps write >> functions >> > that do precisely what Tim is arguing against: mixing list and >> array >> > types and returning copies in the first case, views in the >> second. So >> > if we throw out functions we should throw out asarray also and >> make a >> > rule that numpy functions don't take lists. >> >> This isn't right. I think it's perfectly fine, desirable in fact, >> for >> functions to operate on both list and array types. The problem only >> arises when you return ether a view or copy of one of the original >> inputs depending on what it was. In my experience, this is >> relatively >> hard to do in all but the most trivial of functions since most >> operations return a new array. However, it is true that I think >> people >> should be strongly discouraged from doing this. For example: >> >> def add(a, b): >> a = asarray(a) >> b = asarray(b) >> return a + b >> >> is fine, but: >> >> def iadd(a, b): >> a = asarray(a) >> b = asarray(b) >> a += b >> return a >> >> is not and should be rewritten. Possibly like: >> >> def iadd(a, b): >> if not isinstance(a, ndarray): raise TypeError("'a' must >> be an >> array") >> b = asarray(b) >> a += b >> return a >> >> since += is a bit squirley. (Try somelist += somearray, to see >> what I mean). >> >> >> It's a syntax error, which I believe is the right thing. > > > Really? If so, this is a version thing. This is what I get running on > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > >>> l = list(a) > >>> l > [999, 1, 2, 3, 4, 5, 6, 7, 8] > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > >>> l += a > >>> l > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) Let me add that I think that this is pretty dubious, so if this is a new feature, perhaps we should revert it before it becomes entrenched. Regards, -tim > >> >> > And then there are scalars... >> >> I'm curious what problems you forsee with scalars. >> >> There was/is the whole problem of when to return scalars and if they >> should be python scalars. The problem arises from trying to be >> compatible with normal python stuff. Matlab got around this by making >> everything 1) an array, and 2) double precision. On the other hand, >> Matlab is not nearly is easy to use for nonstandard types. \ > > > I think most of this problem will go away in Python with the > introduction of the __index__ protocol, so for the moment at least I'm > not worried about it. > From tim.hochberg at cox.net Mon Mar 27 14:54:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:54:02 2006 Subject: [Numpy-discussion] arrays and iterators. Message-ID: <44286CCB.3030704@cox.net> Lest everybody think that I'm only about ripping things out, let me propose adding a feature. Well, sort of. I propose that we make: array(someiterator) an value error for right now instead of returning a rank-0 object array. Then, in Python 2.5, we can use the _length_hint protocol to efficiently create arrays from iterator objects without any extra copying. The exact details don't need to be worked out right now, after all we're all too tied up with reshape, but I'd like to reserve that syntax for that purpose. Regards, -tim From webb.sprague at gmail.com Mon Mar 27 15:07:10 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Mon Mar 27 15:07:10 2006 Subject: [Numpy-discussion] Bug in scipy.linalg.svd? Message-ID: Hi all, scipy.__version__: 0.3.2 numpy.__version__: 0.9.4 OS: Gentoo. Maybe I am missing something key here, but my Numerical Recipes book says: "Any M x N matrix A whose number of rows M is greater than or equal to its number of columns N, can be written as the product of of an *M x N* column orthogonal matrix U, an N x N diagonal matrix W, ... and the transpose of an N x N orthogh matrix V." So... why do I get the following behavior: (u, x, v) = scipy.linalg.svd(RandomArray.normal(0,1,[10,4]) u.shape == (10,10) By the way, if you run this from Numeric, U[:, 0:4] are the same, but Numeric doesn't generate the additional columns. I think we should write a unit test such that matrixmultipy(matrixmultiply(u,diag(x)), transpose(v)) == A. From robert.kern at gmail.com Mon Mar 27 15:20:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 27 15:20:06 2006 Subject: [Numpy-discussion] Re: Bug in scipy.linalg.svd? In-Reply-To: References: Message-ID: Webb Sprague wrote: > Hi all, > > scipy.__version__: 0.3.2 > > numpy.__version__: 0.9.4 > > OS: Gentoo. > > Maybe I am missing something key here, but my Numerical Recipes book says: > > "Any M x N matrix A whose number of rows M is greater than or equal to > its number of columns N, can be written as the product of of an *M x > N* column orthogonal matrix U, an N x N diagonal matrix W, ... and the > transpose of an N x N orthogh matrix V." This is only a convention. It is frequently called computing the "thin SVD." Another convention is to return the fully expanded U matrix. scipy.linalg.svd does the latter. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From michael.sorich at gmail.com Mon Mar 27 15:37:12 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Mon Mar 27 15:37:12 2006 Subject: [Numpy-discussion] making a character array from numeric array Message-ID: <16761e100603271536j79da6963gc4eadf889a8d3c43@mail.gmail.com> Hi, I am trying to convert an int or float ndarray into a string ndarray. Using astype this is possible, but only if you explicity set the number of characters (see eg below). It would be nice if just setting type='S' (or N.str_) would automatically make an a character array of sufficient size to hold the string representation. Is there a way to do this? thanks, michael import numpy as N N.version.version '0.9.6' N.arange(20).astype(N.str_) array([, , , , , , , , , , , , , , , , , , , ], dtype='|S0') N.arange(20).astype('S') array([, , , , , , , , , , , , , , , , , , , ], dtype='|S0') N.arange(20).astype('S1') array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype='|S1') N.arange(20).astype('S2') array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], dtype='|S2') -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 15:41:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 15:41:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286C1C.5010605@cox.net> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> Message-ID: <442877DF.4050609@ee.byu.edu> Tim Hochberg wrote: >> Charles R Harris wrote: >> >> >>> l = list(a) >> >>> l >> [999, 1, 2, 3, 4, 5, 6, 7, 8] >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >>> l += a >> >>> l >> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > > Let me add that I think that this is pretty dubious, so if this is a > new feature, perhaps we should revert it before it becomes entrenched. I don't think it's a new feature, but it's simply the result of l += a being translated to l = l + a # lists don't have in-place add's Numeric has this behavior as well. -Travis From charlesr.harris at gmail.com Mon Mar 27 15:47:05 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 15:47:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286AAE.3030204@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: On 3/27/06, Tim Hochberg wrote: > > Charles R Harris wrote: > > > > > > > On 3/27/06, *Tim Hochberg* > > wrote: > > > > Charles R Harris wrote: > > [CHOP] > > > > > > > > How about functions always return a copy unless they are carefully > > > documented interface helper functions like asarray. That's how I > > > generally think of them anyway. > > > > I don't really care as long as they (a) stop being evil and (b) > > there's > > some way to do stuff efficiently. Functions that always return > copies > > seem sort of useless to me, but I'd be happy to move to methods if > > that > > was the case. However, I suspect you won't make much headway with > this > > since my impression is that many of the people who do care about > > functions also care passionately about efficiency as well. > > > > > Of course, the main virtue of asarray is that it helps write > > functions > > > that do precisely what Tim is arguing against: mixing list and > array > > > types and returning copies in the first case, views in the > > second. So > > > if we throw out functions we should throw out asarray also and > > make a > > > rule that numpy functions don't take lists. > > > > This isn't right. I think it's perfectly fine, desirable in fact, > for > > functions to operate on both list and array types. The problem only > > arises when you return ether a view or copy of one of the original > > inputs depending on what it was. In my experience, this is > relatively > > hard to do in all but the most trivial of functions since most > > operations return a new array. However, it is true that I think > people > > should be strongly discouraged from doing this. For example: > > > > def add(a, b): > > a = asarray(a) > > b = asarray(b) > > return a + b > > > > is fine, but: > > > > def iadd(a, b): > > a = asarray(a) > > b = asarray(b) > > a += b > > return a > > > > is not and should be rewritten. Possibly like: > > > > def iadd(a, b): > > if not isinstance(a, ndarray): raise TypeError("'a' must be > an > > array") > > b = asarray(b) > > a += b > > return a > > > > since += is a bit squirley. (Try somelist += somearray, to see > > what I mean). > > > > > > It's a syntax error, which I believe is the right thing. Really? If so, this is a version thing. This is what I get running on > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > >>> l = list(a) > >>> l > [999, 1, 2, 3, 4, 5, 6, 7, 8] > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > >>> l += a > >>> l > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) Ugh. IMHO, this should throw an error. For these sort of assignment operators I think the right hand side should be cast to the type of the left, which I think of as some sort of fancy register variable. If anything, the list should be appended to. I'll have to try the latest SVN just to appreciate this. > -tim > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 15:48:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 15:48:05 2006 Subject: [Numpy-discussion] making a character array from numeric array In-Reply-To: <16761e100603271536j79da6963gc4eadf889a8d3c43@mail.gmail.com> References: <16761e100603271536j79da6963gc4eadf889a8d3c43@mail.gmail.com> Message-ID: <44287987.90507@ee.byu.edu> Michael Sorich wrote: > Hi, > > I am trying to convert an int or float ndarray into a string ndarray. > Using astype this is possible, but only if you explicity set the > number of characters (see eg below). It would be nice if just setting > type='S' (or N.str_) would automatically make an a character array of > sufficient size to hold the string representation. Is there a way to > do this? Not with astype directly because to do what you want requires doing the conversion into variable-length strings and then finding the largest. But, you could do it this way: def convert_to_string(arr): fl = arr.flat l = [str(x) for x in fl] return array(l).reshape(arr.shape) Or as a one-liner array([str(x) for x in arr.flat], dtype='S').reshape(arr.shape) (The dtype='S' is technically unnecessary but should be a bit faster) -Travis From robert.kern at gmail.com Mon Mar 27 15:52:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 27 15:52:06 2006 Subject: [Numpy-discussion] Re: Ransom Proposals In-Reply-To: <442877DF.4050609@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> <442877DF.4050609@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Tim Hochberg wrote: > >>> Charles R Harris wrote: >>> >>> >>> l = list(a) >>> >>> l >>> [999, 1, 2, 3, 4, 5, 6, 7, 8] >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>> >>> l += a >>> >>> l >>> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >> Let me add that I think that this is pretty dubious, so if this is a >> new feature, perhaps we should revert it before it becomes entrenched. > > I don't think it's a new feature, but it's simply the result of > > l += a being translated to > > l = l + a # lists don't have in-place add's > > Numeric has this behavior as well. Lists do have in-place adds. In [1]: a = range(10) In [2]: b = range(10, 20) In [3]: id(a) Out[3]: 92350264 In [4]: a += b In [5]: id(a) Out[5]: 92350264 In [6]: a Out[6]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Mar 27 16:00:08 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 16:00:08 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: Hmm, On 3/27/06, Charles R Harris wrote: > > > > On 3/27/06, Tim Hochberg wrote: > > > > Charles R Harris wrote: > > > > > > > > > > > On 3/27/06, *Tim Hochberg* > > > wrote: > > > > > > Charles R Harris wrote: > > > [CHOP] > > > > > > > > > > > How about functions always return a copy unless they are > > carefully > > > > documented interface helper functions like asarray. That's how I > > > > > > generally think of them anyway. > > > > > > I don't really care as long as they (a) stop being evil and (b) > > > there's > > > some way to do stuff efficiently. Functions that always return > > copies > > > seem sort of useless to me, but I'd be happy to move to methods if > > > that > > > was the case. However, I suspect you won't make much headway with > > this > > > since my impression is that many of the people who do care about > > > functions also care passionately about efficiency as well. > > > > > > > Of course, the main virtue of asarray is that it helps write > > > functions > > > > that do precisely what Tim is arguing against: mixing list and > > array > > > > types and returning copies in the first case, views in the > > > second. So > > > > if we throw out functions we should throw out asarray also and > > > make a > > > > rule that numpy functions don't take lists. > > > > > > This isn't right. I think it's perfectly fine, desirable in fact, > > for > > > functions to operate on both list and array types. The problem > > only > > > arises when you return ether a view or copy of one of the original > > > > > inputs depending on what it was. In my experience, this is > > relatively > > > hard to do in all but the most trivial of functions since most > > > operations return a new array. However, it is true that I think > > people > > > should be strongly discouraged from doing this. For example: > > > > > > def add(a, b): > > > a = asarray(a) > > > b = asarray(b) > > > return a + b > > > > > > is fine, but: > > > > > > def iadd(a, b): > > > a = asarray(a) > > > b = asarray(b) > > > a += b > > > return a > > > > > > is not and should be rewritten. Possibly like: > > > > > > def iadd(a, b): > > > if not isinstance(a, ndarray): raise TypeError("'a' must > > be an > > > array") > > > b = asarray(b) > > > a += b > > > return a > > > > > > since += is a bit squirley. (Try somelist += somearray, to see > > > what I mean). > > > > > > > > > It's a syntax error, which I believe is the right thing. > > > > > Really? If so, this is a version thing. This is what I get running on > > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > > > >>> l = list(a) > > >>> l > > [999, 1, 2, 3, 4, 5, 6, 7, 8] > > >>> a > > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > >>> l += a > > >>> l > > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > > >>> a > > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > Ugh. IMHO, this should throw an error. For these sort of assignment > operators I think the right hand side should be cast to the type of the > left, which I think of as some sort of fancy register variable. If anything, > the list should be appended to. I'll have to try the latest SVN just to > appreciate this. > > It was pilot error. On the other hand: >>> b = [1,2,3] >>> b += b >>> b [1, 2, 3, 1, 2, 3] Which makes sense from the list perspective. From what Travis says, += is actually a phony in place operator, so if [1,2,3] + array([1,2,3]) is supported and returns an array, the result is a given. This is what comes from mixing types and trying to treat them all as interconvertable. Lists aren't arrays, they are just a convenient way to specify the array entries for initialization. I think in these cases one should be forced to be explicit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:05:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:05:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286266.9050503@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> Message-ID: <44287D7D.8010604@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>>> >>>> Yes, having this ability means that you have to think about it a >>>> bit if you are going to use the functional interface and try to do >>>> in-place operations. But, I would argue that this is an "advanced" >>>> usage which is implemented to save space and time. >>> >>> >>> >>> >>> How is this true though? In what way, for instance, is: >>> >>> b = asarray(a).reshape(newshape) >>> >>> slower or less space efficient than todays: >>> >>> b = reshape(a) >> >> >> >> >> Well, the big difference is that b=reshape(a) is actually >> >> try: >> reshape = a.reshape >> except AttributeError: >> return a.__array_wrap__(asarray(a).reshape(newshape)) >> >> return reshape(newshape) >> >> So, it handles more cases, more cleanly then the >> asarray(a).rehsape(newshape) approach does. > > > OK. Although I should have said asanyarray(a).reshape(newshape), I > still see how that this handles more cases. Because you can define objects that aren't sub-classes of the array at all but just define the method reshape and still have the functional interface work. > > I need to go think about this case some more. This is not something > that I've run into in practice, but I can see I'll have a stronger > case if I can come up with a alternative to this in terms of safe > functions. Do you have some examples of objects that do not have > 'reshape', but do have '__array_wrap__'? Not presently, the __array_wrap__ method is new. It's a mechanism for letting functions that need arrays internally to be more polymorphic and deal with different kinds of objects. It would seem that most of Tim's complaints are directly against polymorphism. I would argue that getting rid of polymorphism is not something we should be trying to do in NumPy. Tim is pointing out that if you use polymorphic functions then you can't assume things like "in-place" have any meaning. But, then Python has this same problem, because l += a doesn't do 'in-place' for the list, but does do inplace if 'l' were an array. In my mind, this problem and Tim's concern over 'view' and 'copy' behavior is one and the same. I don't view it as a problem of function behavior as much as problem with documentation and "mismatch between what a particular user expects and what is actually done." My attitude is that to write 'in-place' functions (what seems to be the driving example that Tim brings up), you need to know some details about what kind of object you are dealing with anyway, so you can't write polymorphic in-place functions very easily. So, I guess from one perspective, Tim is arguing against things that are at the very core of Python itself. I'm very resistant to his desire to remove or significantly alter the functional behavior. I also think there is a very good reason to have a few methods that return either a view or a copy. The reason is that there are some arrays that the method can't be implemented unless a copy is made. The only other possibility --- "raising an error" --- is going to confuse a lot of new users and force them to deal with the underlying memory-layout of an array before they really need to. I think the current compromise is practical and disagree strongly that it is somehow "evil." It's only evil to somebody trying to do in-place work. If you are doing that in a polymorphic language like Python, then you need to understand the actual objects you are dealing with and should be expected to have a more mature understanding of NumPy. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:07:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:07:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: <44287DFC.6050700@ee.byu.edu> Charles R Harris wrote: > Really? If so, this is a version thing. This is what I get running on > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > >>> l = list(a) > >>> l > [999, 1, 2, 3, 4, 5, 6, 7, 8] > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > >>> l += a > >>> l > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > Ugh. IMHO, this should throw an error. For these sort of assignment > operators I think the right hand side should be cast to the type of > the left, which I think of as some sort of fancy register variable. If > anything, the list should be appended to. I'll have to try the latest > SVN just to appreciate this. > Again, this is a Python thing. Numeric has this behavior as well. Depending on your point of view it's either a great thing or a wart. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:10:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:10:04 2006 Subject: [Numpy-discussion] arrays and iterators. In-Reply-To: <44287C07.6010805@cox.net> References: <44286CCB.3030704@cox.net> <44287805.5040600@ee.byu.edu> <44287C07.6010805@cox.net> Message-ID: <44287ECE.80602@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>> >>> Lest everybody think that I'm only about ripping things out, let me >>> propose adding a feature. Well, sort of. I propose that we make: >>> >>> array(someiterator) >>> >>> an value error for right now instead of returning a rank-0 object >>> array. >> >> >> >> I'm O.K. with that. >> -Travis > > > OK, if no one else objects in the next day or so, I'll look into doing > it. Just to be clear. You are only going to change the case which does presently return a rank-0 object array, right? someiterator could be an array, or a list in which case array(someiterator) already does an intelligible thing. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:14:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:14:00 2006 Subject: [Numpy-discussion] arrays and iterators. In-Reply-To: <44287C07.6010805@cox.net> References: <44286CCB.3030704@cox.net> <44287805.5040600@ee.byu.edu> <44287C07.6010805@cox.net> Message-ID: <44287F8F.7090502@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>> >>> Lest everybody think that I'm only about ripping things out, let me >>> propose adding a feature. Well, sort of. I propose that we make: >>> >>> array(someiterator) >>> >>> an value error for right now instead of returning a rank-0 object >>> array. >> >> >> >> I'm O.K. with that. >> -Travis > > > OK, if no one else objects in the next day or so, I'll look into doing > it. You might also look at having array(generator object) actually run the generator instead of return a rank-0 object array. -Travis From michael.sorich at gmail.com Mon Mar 27 16:17:13 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Mon Mar 27 16:17:13 2006 Subject: [Numpy-discussion] precedence of '&' and '>' operators Message-ID: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> It seems that the '&' operator has a higher precedence than a comparison operator such as '>'. This does not seem so intuitive to me. Is this correct? eg >>>import numpy as N >>>N.version.version '0.9.6' >>>a = N.arange(10) >>>a>1 & a<4 Traceback (most recent call last): File "", line 1, in ? ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() >>>(a>1) & (a<4) array([False, False, True, True, False, False, False, False, False, False], dtype=bool) Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:24:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:24:02 2006 Subject: [Numpy-discussion] precedence of '&' and '>' operators In-Reply-To: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> References: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> Message-ID: <442881FD.7010005@ee.byu.edu> Michael Sorich wrote: > It seems that the '&' operator has a higher precedence than a > comparison operator such as '>'. This does not seem so intuitive to > me. Is this correct? Yes, this is correct. See http://docs.python.org/ref/summary.html The problem really is that "and" (which has lower precedence) does not allow being over-written. So, we must use the bit-wise operator. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:29:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:29:06 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: References: <4428516E.4070308@colorado.edu> <44285970.5080008@colorado.edu> Message-ID: <44288339.5090507@ee.byu.edu> Charles R Harris wrote: > > > On 3/27/06, *Fernando Perez* > wrote: > > Charles R Harris wrote: > > > What purpose does ascontiguous serve? > > One I can think of: making sure that C code of the type > > double foo(double *array,long num_elements) { ... } > > can be called in C as > > foo((double *)numpy_arr->data,numpy_arr->dimensions[0]); > > (the dims name in C may be different, I'm not looking at code > right now) > > and be OK with walking the given memory region as a contiguous > chunk of > doubles, num_elements of them. > > As long as foo() doesn't modify its input, this works fine and > it's nice to > have such guarantees: it makes writing extension code a lot easier. > > > OK, it's an interface. Since getting stuff back from C will require a > copy if the original is not contiquous, a test involving > a.iscontiguous will be needed and ascontiguous can be dispensed with. > Isn't there a C version of this anyway? ISTR that numarray had one. > It think you are talking about the UPDATEIFCOPY requirements flag on the PyArray_FromAny C-API. Most aren't using this yet (and it's not directly exposed to Python). But it's how you would get a copy of the array with the correct requirements, operate on it and then once you delete your reference have the information go back into the original array (this is true whether or not the array is contiguous). The original array is also set to be read-only during the time that the copy is available so that you can't make changes to it (that are going to be lost). Perhaps what we need to do is make this scheme available to the Python user as well. -Travis From michael.sorich at gmail.com Mon Mar 27 16:31:03 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Mon Mar 27 16:31:03 2006 Subject: [Numpy-discussion] using boolean arrays as indices Message-ID: <16761e100603271630v6a46dbabq686ef548bc82aadf@mail.gmail.com> If I have a 2D array and I want to use a boolean array to select the rows. If I use a[boolarray] it works, but if I use a[boolarray, :] an exception is raised. Is there a reason for this? eg >>>N.__version__ '0.9.6' >>>a = N.arange(10).reshape((2,5)) >>>a array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) >>>a[N.array([True, False])] array([[0, 1, 2, 3, 4]]) >>>a[N.array([True, False]),:] Traceback (most recent call last): File "", line 1, in ? IndexError: arrays used as indices must be of integer type -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:37:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:37:01 2006 Subject: [Numpy-discussion] Re: Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> <442877DF.4050609@ee.byu.edu> Message-ID: <442884F3.7060807@ee.byu.edu> Robert Kern wrote: >Travis Oliphant wrote: > > >>Tim Hochberg wrote: >> >> >> >>>>Charles R Harris wrote: >>>> >>>> >>> l = list(a) >>>> >>> l >>>> [999, 1, 2, 3, 4, 5, 6, 7, 8] >>>> >>> a >>>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>>> >>> l += a >>>> >>> l >>>> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>>> >>> a >>>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>>> >>>> >>>Let me add that I think that this is pretty dubious, so if this is a >>>new feature, perhaps we should revert it before it becomes entrenched. >>> >>> >>I don't think it's a new feature, but it's simply the result of >> >>l += a being translated to >> >>l = l + a # lists don't have in-place add's >> >>Numeric has this behavior as well. >> >> > >Lists do have in-place adds. > >In [1]: a = range(10) > >In [2]: b = range(10, 20) > >In [3]: id(a) >Out[3]: 92350264 > >In [4]: a += b > >In [5]: id(a) >Out[5]: 92350264 > >In [6]: a >Out[6]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] > > > True, true. So, why this behavior? This seems like a Python issue. Compare a = range(10) b = arange(10,20) a.__iadd__(b) with a += b Shouldn't these be equivalent? -Travis From tim.leslie at gmail.com Mon Mar 27 16:46:04 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Mon Mar 27 16:46:04 2006 Subject: [Numpy-discussion] precedence of '&' and '>' operators In-Reply-To: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> References: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> Message-ID: On 3/28/06, Michael Sorich wrote: > > It seems that the '&' operator has a higher precedence than a comparison > operator such as '>'. This does not seem so intuitive to me. Is this > correct? > Yep. http://docs.python.org/ref/summary.html Cheers, Timl eg > >>>import numpy as N > >>>N.version.version > '0.9.6' > >>>a = N.arange(10) > >>>a>1 & a<4 > Traceback (most recent call last): > File "", line 1, in ? > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > >>>(a>1) & (a<4) > array([False, False, True, True, False, False, False, False, False, > False], dtype=bool) > > Michael > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:57:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:57:02 2006 Subject: [Numpy-discussion] using boolean arrays as indices In-Reply-To: <16761e100603271630v6a46dbabq686ef548bc82aadf@mail.gmail.com> References: <16761e100603271630v6a46dbabq686ef548bc82aadf@mail.gmail.com> Message-ID: <442889AB.9020508@ee.byu.edu> Michael Sorich wrote: > If I have a 2D array and I want to use a boolean array to select the > rows. If I use a[boolarray] it works, but if I use a[boolarray, :] an > exception is raised. Is there a reason for this? Yes. Currently, you can only use Boolean indexing using X[obj] which is equivalent to (but faster than) X[obj.nonzero()] There is no way currently to use Boolean indexing on a single axis. A useful specification would need to be made that is consistent with the current definition along with an implementation. You could use instead a.compress(condition, axis=0) This selects out sub-arrays along a particular axis. -Travis From zpincus at stanford.edu Mon Mar 27 17:14:18 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 27 17:14:18 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428660A.2020607@ee.byu.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> <4428660A.2020607@ee.byu.edu> Message-ID: <3964FC31-4753-4E3A-B69C-9A6CD88C9202@stanford.edu> >> Does this mean that if I create a new array with FORTRAN order >> from the numpy.array() function, every time I reshape that array >> I need to tell reshape to use FORTRAN order? That seems a bit odd... > > Yes, that is what it means. Doing anything else leads to a host > of problems. Let me illustrate. > > a = rand(10,3) > b = a.transpose() > b.flags # b is now in Fortran order > b.ravel() # do you really want this to now be interpreted in > FORTRAN-order. I didn't think so. > > What this illustrates is that the FORTRAN and CONTIGUOUS flags on > the array are just special cases of the strides (and in fact can be > updated from the strides at any time). But, an ORDER flag would > be an independent concept that could be set and reset at will. The > ORDER flag is only necessary when somebody is interpreting the > array as a linear sequence of bytes. I think I understand now. Thanks. Now a question concerning a common use-case: Frequently I have either a string or a buffer protocol object that is basically a chunk of memory returned from some wrapped C library function. Sometimes this string/object is in fortran-strided memory order, and I need to tell the ndarray constructor that so that indexing works correctly. Clearly, fortran-strided memory order is just a special case of the strides, as you point out. So what I need to do in this case is provide an appropriate strides argument to the ndarray constructor. But that's harder (and less transparent for someone reading the code) than just setting a flag like Fortran=True or Order='FORTRAN'. So from this perspective it would be great if the ndarray constructor allowed something like strides='FORTRAN'. Of course, if an order flag is provided, then the ndarray constructor would have two different and orthogonal things that could/should accept a parameter of 'FORTRAN' -- Order and Strides. Now that's confusing! I guess the problem is (as you demonstrate clearly) that there are two semi-orthogonal issues around FORTRAN-ness. First is the issue of how a 1D memory region is to be indexed as a multidimensional array. (e.g. when constructing an array from a buffer.) The second is the issue of how a multidimensional array is to be treated as a 1D memory region. (e.g. when using ravel or resize [if resize is padding with the array contents]). This is very confusing, and could certainly benefit from some notational clarity. Does someone have any good ideas for what to call each of these properties? Strides and Order (respectively) seems to be what people are using in this email thread, but I'm not sure if there could be something better... Zach From tim.hochberg at cox.net Mon Mar 27 17:53:14 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 17:53:14 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442877DF.4050609@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> <442877DF.4050609@ee.byu.edu> Message-ID: <442896E1.8070207@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> Charles R Harris wrote: >>> >>> >>> l = list(a) >>> >>> l >>> [999, 1, 2, 3, 4, 5, 6, 7, 8] >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>> >>> l += a >>> >>> l >>> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >> >> >> >> Let me add that I think that this is pretty dubious, so if this is a >> new feature, perhaps we should revert it before it becomes entrenched. > > > > I don't think it's a new feature, but it's simply the result of > > l += a being translated to > > l = l + a # lists don't have in-place add's > > Numeric has this behavior as well. That's what I would expect, but Charles claimed it was resulted in an error of some sort, so I was wondering if it was new. I was also wondering if we could disable it somehow by proper finagaling of the various add and iadd ops, but I can't really see how. However, I did stumble on this odd behaviour: class reporter(object): def __init__(self, name): self.name = name def __add__(self, other): print "add called on", self.name, other.name return NotImplemented def __radd__(self, other): print "radd called on", self.name, other.name return NotImplemented def __iadd__(self, other): print "iadd called on", self.name, other.name return NotImplemented class reporter2(reporter): pass a = reporter("A") b = reporter2("B") a += b print a ==> iadd called on A B add called on A B radd called on B A iadd called on A B NotImplemented Shouldn't that raise TypeError? Yes? No? I'll go enter it as a bug, but I want to make sure I'm not missing something stupid. -tim From tim.hochberg at cox.net Mon Mar 27 18:41:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 18:41:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44287D7D.8010604@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> Message-ID: <4428A228.7040806@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> Travis Oliphant wrote: >> >>> Tim Hochberg wrote: >>> >>>>> >>>>> Yes, having this ability means that you have to think about it a >>>>> bit if you are going to use the functional interface and try to do >>>>> in-place operations. But, I would argue that this is an >>>>> "advanced" usage which is implemented to save space and time. >>>> >>>> >>>> >>>> >>>> >>>> How is this true though? In what way, for instance, is: >>>> >>>> b = asarray(a).reshape(newshape) >>>> >>>> slower or less space efficient than todays: >>>> >>>> b = reshape(a) >>> >>> >>> >>> >>> >>> Well, the big difference is that b=reshape(a) is actually >>> >>> try: >>> reshape = a.reshape >>> except AttributeError: >>> return a.__array_wrap__(asarray(a).reshape(newshape)) >>> >>> return reshape(newshape) >>> >>> So, it handles more cases, more cleanly then the >>> asarray(a).rehsape(newshape) approach does. >> >> >> >> OK. Although I should have said asanyarray(a).reshape(newshape), I >> still see how that this handles more cases. > > > Because you can define objects that aren't sub-classes of the array at > all but just define the method reshape and still have the functional > interface work. > >> >> I need to go think about this case some more. This is not something >> that I've run into in practice, but I can see I'll have a stronger >> case if I can come up with a alternative to this in terms of safe >> functions. Do you have some examples of objects that do not have >> 'reshape', but do have '__array_wrap__'? > > > > Not presently, the __array_wrap__ method is new. It's a mechanism for > letting functions that need arrays internally to be more polymorphic > and deal with different kinds of objects. > > > It would seem that most of Tim's complaints are directly against > polymorphism. I would argue that getting rid of polymorphism is not > something we should be trying to do in NumPy. I'm certainly not against polymorphism in general. However, I do believe that the intersection of polymorphism with view/copy behaviour can be dangerous in certain circumstances, and we should limit those circumstances to a few, visible functions. The cases I've been complaining about are all instances of this pattern: b = func(a, *other_args, **other_kwargs) With the added complication that b may be a view of a or it may be a new array object. This particular case is 'sharp blade' and should be treated with respect. Here's how I would probably write reshape if I were starting a Numeric like language from scratch: def reshape(a, shape): b = a.view() b.shape = shape return b That's no less polymorphic than the current reshape, but it's polymorphic in a different way. It would work on anything that has view method and a shape parameter. This as opposed to anything that has reshape or has __arrary_wrap__ and can be turned into an array. This is a nonstarter for backwards compatibility reasons however. > > Tim is pointing out that if you use polymorphic functions then you > can't assume things like "in-place" have any meaning. But, then > Python has this same problem, because > > l += a > > doesn't do 'in-place' for the list, but does do inplace if 'l' were an > array. In my mind, this problem and Tim's concern over 'view' and > 'copy' behavior is one and the same. I agree. > I don't view it as a problem of function behavior as much as problem > with documentation and "mismatch between what a particular user > expects and what is actually done." I disagree. Fernando stated this much better than I could: We should treat them as expensive specialties we must pay for with a very tight budget (the willingness and ability of users to keep track of the exponential rise in complexity induced by interlocking special cases). > > My attitude is that to write 'in-place' functions (what seems to be > the driving example that Tim brings up), you need to know some details > about what kind of object you are dealing with anyway, so you can't > write polymorphic in-place functions very easily. Correct. I doubt writing polymorphic in place functions will ever be easy. But the choice of tools we provide can influence how often they end up being correct. > > So, I guess from one perspective, Tim is arguing against things that > are at the very core of Python itself. I'm very resistant to his > desire to remove or significantly alter the functional behavior. > > I also think there is a very good reason to have a few methods that > return either a view or a copy. The reason is that there are some > arrays that the method can't be implemented unless a copy is made. > The only other possibility --- "raising an error" --- is going to > confuse a lot of new users and force them to deal with the underlying > memory-layout of an array before they really need to. I don't think this does anyone any favors myself, it just lets people get into habits early on that are going to bite them later. > I think the current compromise is practical and disagree strongly that > it is somehow "evil." It's only evil to somebody trying to do > in-place work. I think *most* of the current compromise is good. > If you are doing that in a polymorphic language like Python, then you > need to understand the actual objects you are dealing with and should > be expected to have a more mature understanding of NumPy. FWIW, I've been using Numeric and it's successors since Numeric was pre-alpha, when Jim Huginin was still around, when giants walked the earth. My understanding of numerapy [*] may be flawed, but it's definitely mature. [*] Referring to the whole Numeric/numarray/numpy family In general, when balancing safety, power and speed, numerapy has leaned heavily toward speed and power. I think that is the right choice. However, in the case at hand, there is no speed tradeoff: you'll note, for example, that I have consistently avoided advocating that reshape and friends return copies. The trade off has been between safety and power, and in this case the power is purely theoretical at this point since there aren't yet any clients to __array_wrap__ that don't define reshape. I would think that is a very narrow slice of potential object space. That being said I have a suggestion that *might* satisfy everyone. Set the WRITEABLE flag to false if reshape creates a new array: def _viewwrapit(obj, method, *args, **kwds): try: wrap = obj.__array_wrap__ except AttributeError: wrap = None if type(obj) is ndarray: writeable = True else: writeable = False obj = asarray(obj) result = getattr(obj, method)(*args, **kwds) result.flags.writeable = writeable if wrap: result = wrap(result) return result def reshape(a, newshape, order=False): try: reshape = a.reshape except AttributeError: return _viewwrapit(a, 'reshape', newshape, order=order) else: return reshape(newshape, order=order) a = arange(4) a2 = reshape(a, [2,2]) a2[0,1] = 99 print "a2 =", a2 print "a = ", a l = [0, 1, 2, 3] l2 = reshape(l, [2,2]) l2[0,1] = 99 ==> a2 = [[ 0 99] [ 2 3]] a = [ 0 99 2 3] Traceback (most recent call last): File "scratch.py", line 37, in ? l2[0,1] = 99 RuntimeError: array is not writeable Thus we get the same semantics as we have now exept in the one case that causes trouble: modifying the result of a function that returns both copies and views when it has returned a copy. The only catch is that clients of __array_wrap__ would need to be strongly encouraged to respect the writeable flag. However, since __array_wrap__ is new, this is probably not a big deal. This could also help solve the problem of how to deal with a.reshape(newshape) when the result would be a copy; simple make the copy nonwriteable. However, that case is more complex and requires more thought. No power is lost; 'advanced' users can simply set the writeable flag back to True. If we need to be able to do this in one line, setflags could be modified to return self, so that: reshape(a, newshape).setflags(write=True) is essentially equivalent to the current reshape. [A side note, it kind of looks like setflags may need some revamping; shouldn't the flag values in setflags match those in flags?] The result of reshape(a, newshape), when a new object is required can then *almost* be regarded as a view of a constant object. This is because modifying the operand and expecting it to change the result is extremely rare. Unlike modifying the result and expecting it to change the operand, which while not exactly common ,is not that rare and is how one gets into trouble. Regards, -tim From oliphant.travis at ieee.org Mon Mar 27 21:05:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 27 21:05:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428A228.7040806@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> Message-ID: <4428C399.1080002@ieee.org> >> >> I also think there is a very good reason to have a few methods that >> return either a view or a copy. The reason is that there are some >> arrays that the method can't be implemented unless a copy is made. >> The only other possibility --- "raising an error" --- is going to >> confuse a lot of new users and force them to deal with the underlying >> memory-layout of an array before they really need to. > > I don't think this does anyone any favors myself, it just lets people > get into habits early on that are going to bite them later. > >> I think the current compromise is practical and disagree strongly >> that it is somehow "evil." It's only evil to somebody trying to do >> in-place work. > > I think *most* of the current compromise is good. > >> If you are doing that in a polymorphic language like Python, then you >> need to understand the actual objects you are dealing with and should >> be expected to have a more mature understanding of NumPy. > > FWIW, I've been using Numeric and it's successors since Numeric was > pre-alpha, when Jim Huginin was still around, when giants walked the > earth. My understanding of numerapy [*] may be flawed, but it's > definitely mature. > Weren't you involved in a port of Numeric to Jython. I seem to remember that you wrote that. So, I know you're an "old-timer" and I really appreciate that you are trying so hard to advocate your view. There are compromises all over the place and not all of them have anything remotely-approaching a "solid-reason" behind them other than that's just what got written. So, pre 1.0 I'm all for fixing things that can be fixed so we don't saddle ourselves with problems due to "muddy concepts". Of course once Python 3000 comes out (I'm guessing in 3-5 years) we'll have a chance to do this again ;-) > > In general, when balancing safety, power and speed, numerapy has > leaned heavily toward speed and power. I think that is the right > choice. However, in the case at hand, there is no speed tradeoff: > you'll note, for example, that I have consistently avoided advocating > that reshape and friends return copies. The trade off has been between > safety and power, and in this case the power is purely theoretical at > this point since there aren't yet any clients to __array_wrap__ that > don't define reshape. I would think that is a very narrow slice of > potential object space. The trade off seems to be in this case between safety and new-user-friendliness. > > That being said I have a suggestion that *might* satisfy everyone. Set > the WRITEABLE flag to false if reshape creates a new array: This is an interesting proposition. Definitely something that could be done. It would certainly serve the purpose of warning somebody that a new copy has been made. > Thus we get the same semantics as we have now exept in the one case > that causes trouble: modifying the result of a function that returns > both copies and views when it has returned a copy. The only catch is > that clients of __array_wrap__ would need to be strongly encouraged > to respect the writeable flag. However, since __array_wrap__ is new, > this is probably not a big deal. This could also help solve the > problem of how to deal with a.reshape(newshape) when the result would > be a copy; simple make the copy nonwriteable. > However, that case is more complex and requires more thought. > > No power is lost; 'advanced' users can simply set the writeable flag > back to True. If we need to be able to do this in one line, setflags > could be modified to return self, so that: > > reshape(a, newshape).setflags(write=True) Except it's the "young" user who would expect to be able to modify the array they just reshaped (and who wouldn't care that they got a copy in the first place). So, this puts more work for the "novice" user than the "advanced" user. But as you argue, having users not learn the difference between copy and view can certainly bite them. And, I can see that having somebody modify the array after reshaping is less likely --- most of the time it is just used in a later calculation. > > is essentially equivalent to the current reshape. [A side note, it > kind of looks like setflags may need some revamping; shouldn't the > flag values in setflags match those in flags?] Are you referring to keyword arguments? Then I suppose that would be true. > > The result of reshape(a, newshape), when a new object is required can > then *almost* be regarded as a view of a constant object. This is > because modifying the operand and expecting it to change the result is > extremely rare. Unlike modifying the result and expecting it to change > the operand, which while not exactly common ,is not that rare and is > how one gets into trouble. > One could try setting the writeable flag to false for copy-like behavior when both views and copies are possible and see what happens with dependent code like scipy. -Travis From charlesr.harris at gmail.com Mon Mar 27 21:40:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 21:40:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428A228.7040806@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> Message-ID: Tim, > > That being said I have a suggestion that *might* satisfy everyone. Set > the WRITEABLE flag to false if reshape creates a new array: I think this will cause problems as I myself usually want to modify the contents after a reshape. Why not in this case have a.reshape(...) return a view when possible and raise an error otherwise, and have the functional form make a copy. Reshape the method then provides a path to efficient computing, while reshape the function promotes safe computing. I like to write classes in C++ where I can do things like b.mul(c).add(d).div(e) all inplace for efficiency and the current situation strikes me as similar. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Mar 27 23:20:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 27 23:20:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> Message-ID: <4428E36E.4000908@ieee.org> Charles R Harris wrote: > Tim, > > > That being said I have a suggestion that *might* satisfy everyone. Set > the WRITEABLE flag to false if reshape creates a new array: > > > I think this will cause problems as I myself usually want to modify > the contents after a reshape. Why not in this case have a.reshape(...) > return a view when possible and raise an error otherwise, and have the > functional form make a copy. Reshape the method then provides a path > to efficient computing, while reshape the function promotes safe > computing. I like to write classes in C++ where I can do things like > b.mul(c).add(d).div(e) all inplace for efficiency and the current > situation strikes me as similar. > This is the direction I could support. If we did this, then the ravel method would also have to change to be consistent. The functions would be copy-if-necessary and would *not* be deprecated. To summarize on making methods that return views raise an error when impossible and not changing and/or deprecating functions (i.e. the functions are simple wrappers around the methods). -1 on making methods that return views raise an error when impossible and changing the function to make a copy +0 -Travis From pgmdevlist at mailcan.com Tue Mar 28 00:19:06 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 28 00:19:06 2006 Subject: [Numpy-discussion] More about booleans as indices Message-ID: <200603280318.19036.pgmdevlist@mailcan.com> Folks, I was playing with the compress method and extended slicing techniques, when I ran into this problem: I want to select the records of rec.array (`arec`) for which a given tag (`tag`) has some given values, as listed in `specialtaglist`. A basic arec[arec.tag in specialtaglist] raises a ValueError exception (ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()). A workaround I found was: arec[ [arec.tag[k] in specialtaglist for k in range(len(arec))] ] I was wondering whether there was a better/faster/simpler method ? Thx in advance From pgmdevlist at mailcan.com Tue Mar 28 01:28:35 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 28 01:28:35 2006 Subject: [Numpy-discussion] New patch for MA Message-ID: <200603280427.52789.pgmdevlist@mailcan.com> Folks, You can find a new patch for MA on the wiki http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/ma-200603280900.patch along with a test suite. The 'clip' method should now work with array arguments. Were also added cumsum, cumprod, std, var and squeeze. I'll deal with flags, setflags, setfield, dump and others when I'll have a better idea of how it works -- which probably won't happen anytime soon, as I don't really have time to dig in the code for these functions. AAMOF, I'm more interested in checking/patching some other aspects of numpy for MA (eg, mlab...) Once again, please send me your comments and suggestions. Thx for everything P. From cjw at sympatico.ca Tue Mar 28 06:02:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 28 06:02:04 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <4428610B.4040307@ee.byu.edu> References: <44285D68.1060908@stsci.edu> <4428610B.4040307@ee.byu.edu> Message-ID: <442941B6.4050207@sympatico.ca> Travis Oliphant wrote: > Christopher Hanley wrote: > >> Hi Travis, >> >> I have run across a feature in numpy which I am not sure how to >> handle. One of the new features of the ndarray is that named fields >> can be accessed as attributes. So, if a user created a recarray with >> fields named "address", "phone", and "names" they would be able to >> access all of the names in the recarray objects with >> "recarrayObject.names". > > > > To be clear. This is only true for a recarray subclass. Standard > ndarray's cannot look-up fields with attribute access. Is any consideration being given to integrating recarray into ndarray? It seems to me that attribute access is desirable, providing that there is some check to ensure that the name does not duplicate an existing attribute. Colin W. > >> >> I have found that this can cause problems when a user creates a table >> with a field name that happen to coincide with an existing class >> attribute or method. In particular, I am having a problem in pyfits >> where a user has a table with a column named "field". This column >> name is causing an attribute "field" to be created that overrides the >> existing class field method. > > > > This is a standard problem with using attribute access. I suppose it > would be better to always lookup the attributes first on the object > and then if that fails use field access. That way library code would > not break if a user happened to name their fields after a method or > property. Users would have to change code if new methods or > properties were added (although that won't be as common). > >> >> It would seem to me that existing class methods and attributes should >> take precedence over the attributes created to correspond to column >> names. Is there a way to implement this behavior in ndarray? > > > > I think this is the right thing to do. Are there any objections? > > -Travis > From bblais at bryant.edu Tue Mar 28 06:03:14 2006 From: bblais at bryant.edu (Brian Blais) Date: Tue Mar 28 06:03:14 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? Message-ID: <4429414F.1060906@bryant.edu> Hello, I thought I had read somewhere that the values in a numpy contiguous array are stored column-wise (moving down from the upper left to the lower left first) like C as opposed to row-wise (moving right from the upper left to the upper right first). However, when I write a simply Pyrex function to access the data pointer and print it out, it seems to go across the row. Is this correct? I am using numpy 0.9.3 and Python 2.3 on Linux. This also occurs for Numeric. I post the code below, with the sample output. thanks, bb -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais #test_numpy.pyx: cdef extern from "numpy/arrayobject.h": struct PyArray_Descr: int type_num, elsize char type ctypedef class numpy.ArrayType [object PyArrayObject]: cdef char *data cdef int nd cdef int *dimensions, *strides cdef object base cdef PyArray_Descr *descr cdef int flags def test_numpy(ArrayType x): cdef int i cdef double *data cdef int length data=x.data length=x.dimensions[0]*x.dimensions[1] for i from 0 <= i < length: print data[i] #------------------------------------------------------ #testit.py: import numpy from test_numpy import * a=numpy.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12]],numpy.Float64) print a test_numpy(a) #------------------------------------------------------ #output from program: >>> execfile('testit.py') [[ 1. 2. 3.] [ 4. 5. 6.] [ 7. 8. 9.] [ 10. 11. 12.]] 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 11.0 12.0 From cjw at sympatico.ca Tue Mar 28 06:11:08 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 28 06:11:08 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44287DFC.6050700@ee.byu.edu> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44287DFC.6050700@ee.byu.edu> Message-ID: <442943EF.8070606@sympatico.ca> Travis Oliphant wrote: > Charles R Harris wrote: > >> Really? If so, this is a version thing. This is what I get >> running on >> Python 2.4.1 and current numpy SVN (0.9.7.2286): >> >> >>> l = list(a) >> >>> l >> [999, 1, 2, 3, 4, 5, 6, 7, 8] >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >>> l += a >> >>> l >> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >> >> Ugh. IMHO, this should throw an error. For these sort of assignment >> operators I think the right hand side should be cast to the type of >> the left, which I think of as some sort of fancy register variable. >> If anything, the list should be appended to. I'll have to try the >> latest SVN just to appreciate this. >> > > Again, this is a Python thing. Numeric has this behavior as well. > Depending on your point of view it's either a great thing or a wart. A wart +1 Colin W. From faltet at carabos.com Tue Mar 28 07:19:25 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Mar 28 07:19:25 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? In-Reply-To: <4429414F.1060906@bryant.edu> References: <4429414F.1060906@bryant.edu> Message-ID: <200603281648.40265.faltet@carabos.com> A Dimarts 28 Mar? 2006 15:59, Brian Blais va escriure: > Hello, > > I thought I had read somewhere that the values in a numpy contiguous array > are stored column-wise (moving down from the upper left to the lower left > first) like C as opposed to row-wise (moving right from the upper left to > the upper right first). However, when I write a simply Pyrex function to > access the data pointer and print it out, it seems to go across the row. > Is this correct? Nope. C arrangement of values is row-wise and Fortran is column-wise. You are seeing row-wise ordering, so everything behaves as it should. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tim.hochberg at cox.net Tue Mar 28 07:36:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 07:36:18 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428E36E.4000908@ieee.org> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> Message-ID: <442957A4.7070301@cox.net> Travis Oliphant wrote: > Charles R Harris wrote: > >> Tim, >> >> >> That being said I have a suggestion that *might* satisfy >> everyone. Set >> the WRITEABLE flag to false if reshape creates a new array: >> >> I think this will cause problems as I myself usually want to modify >> the contents after a reshape. Why not in this case have >> a.reshape(...) return a view when possible and raise an error >> otherwise, and have the functional form make a copy. Reshape the >> method then provides a path to efficient computing, while reshape the >> function promotes safe computing. I like to write classes in C++ >> where I can do things like >> b.mul(c).add(d).div(e) all inplace for efficiency and the current >> situation strikes me as similar. >> > This is the direction I could support. If we did this, then the > ravel method would also have to change to be consistent. > > The functions would be copy-if-necessary and would *not* be deprecated. > > To summarize > > on making methods that return views raise an error when impossible and > not changing and/or deprecating functions (i.e. the functions are > simple wrappers around the methods). > > -1 > > on making methods that return views raise an error when impossible and > changing the function to make a copy > > +0 +1 Assuming I understand all of the implications. * When doing automated conversion from Numeric/numarray to numpy, "reshape(obj) => asarray(obj).reshape()". Doing anything else is going to cause breakage and/or massive slowdowns. Can the conversion tool handle this? I would guess yes, but that's just a guess. * This means that "somenoncontiguousarray.reshape(newshape)" will oftern raise an error. That's a good thing, but I figured I'd get it out in the open now so no one is suprised. * Since x.reshape would no longer return copies, the order flag is unnecessary. * numpy.reshape(obj, newshape) on the other hand, always copies, so it *could* grow an order flag. I think that I'd prefer that it didn't; if you need that kind of control of the resulting arrays you should be using array to copy your matrices. * What does fixing ravel to be consistent mean? Always return a view? Or always return a copy? The latter seems more useful, as the first is already covered by both obj.reshape(-1) and obj.flat. Regards, -tim From tim.hochberg at cox.net Tue Mar 28 07:40:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 07:40:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442957A4.7070301@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> Message-ID: <44295897.9030307@cox.net> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Charles R Harris wrote: >> >>> Tim, >>> >>> >>> That being said I have a suggestion that *might* satisfy >>> everyone. Set >>> the WRITEABLE flag to false if reshape creates a new array: >>> >>> I think this will cause problems as I myself usually want to modify >>> the contents after a reshape. Why not in this case have >>> a.reshape(...) return a view when possible and raise an error >>> otherwise, and have the functional form make a copy. Reshape the >>> method then provides a path to efficient computing, while reshape >>> the function promotes safe computing. I like to write classes in C++ >>> where I can do things like >>> b.mul(c).add(d).div(e) all inplace for efficiency and the current >>> situation strikes me as similar. >>> >> This is the direction I could support. If we did this, then the >> ravel method would also have to change to be consistent. >> >> The functions would be copy-if-necessary and would *not* be deprecated. >> >> To summarize >> >> on making methods that return views raise an error when impossible >> and not changing and/or deprecating functions (i.e. the functions are >> simple wrappers around the methods). >> >> -1 >> >> on making methods that return views raise an error when impossible >> and changing the function to make a copy >> >> +0 > > > +1 Assuming I understand all of the implications. > > * When doing automated conversion from Numeric/numarray to numpy, > "reshape(obj) => asarray(obj).reshape()". Doing anything else is going > to cause breakage and/or massive slowdowns. Can the conversion tool > handle this? I would guess yes, but that's just a guess. Actually, that should be: reshape(obj, newshape) => ascontiguousarray(obj).reshape(newshape) > > * This means that "somenoncontiguousarray.reshape(newshape)" will > oftern raise an error. That's a good thing, but I figured I'd get it > out in the open now so no one is suprised. > > * Since x.reshape would no longer return copies, the order flag is > unnecessary. > > * numpy.reshape(obj, newshape) on the other hand, always copies, so it > *could* grow an order flag. I think that I'd prefer that it didn't; if > you need that kind of control of the resulting arrays you should be > using array to copy your matrices. > > * What does fixing ravel to be consistent mean? Always return a view? > Or always return a copy? The latter seems more useful, as the first is > already covered by both obj.reshape(-1) and obj.flat. > > Regards, > > -tim > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From bblais at bryant.edu Tue Mar 28 08:41:04 2006 From: bblais at bryant.edu (Brian Blais) Date: Tue Mar 28 08:41:04 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? In-Reply-To: <200603281648.40265.faltet@carabos.com> References: <4429414F.1060906@bryant.edu> <200603281648.40265.faltet@carabos.com> Message-ID: <4429662A.2070307@bryant.edu> Francesc Altet wrote: > > Nope. C arrangement of values is row-wise and Fortran is column-wise. > You are seeing row-wise ordering, so everything behaves as it should. > interesting. I am coming from Matlab, where things are definitely column-wise organized. I always assumed this was the C standard, but now that I think of it the C double array [][] is row-wise. thanks, bb -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From charlesr.harris at gmail.com Tue Mar 28 09:21:10 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 09:21:10 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? In-Reply-To: <4429662A.2070307@bryant.edu> References: <4429414F.1060906@bryant.edu> <200603281648.40265.faltet@carabos.com> <4429662A.2070307@bryant.edu> Message-ID: Hi Brian, On 3/28/06, Brian Blais wrote: > > Francesc Altet wrote: > > > > Nope. C arrangement of values is row-wise and Fortran is column-wise. > > You are seeing row-wise ordering, so everything behaves as it should. > > > > interesting. I am coming from Matlab, where things are definitely > column-wise > organized. I always assumed this was the C standard, but now that I think > of it the > C double array [][] is row-wise. Matlab was originally written in Fortran by Cleve Moler as an easy to use interface to LINPACK and EISPACK. If you google around you can probably find the original free Fortran version out on the net somewhere. Matlab was rewritten in C when John Little and Cleve Moler founded MathWorks but I imagine the new version still called many Fortran subroutines for the heavy lifting . You can find many other Fortran traces in Matlab. For instance, Matlab array indices start at 1 in the traditional Fortran way. On a side note, I believe that John Little's parents moved into the house that my family built in Lincoln, MA. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Mar 28 10:45:05 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue Mar 28 10:45:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428C399.1080002@ieee.org> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428C399.1080002@ieee.org> Message-ID: <4429841F.4040105@noaa.gov> I want to start by saying a am +1 on everything Tim is advocating, just so you all know it's not just him. I'm not writing any code, however. A few comments: Travis Oliphant wrote: > It would seem that most of Tim's complaints are directly against > polymorphism. I don't think so. I think Tim (and I) just want it to be really clear what the polymorphic functions are and what aren't. Also, for polymorphism to work, there always has to be some definition of what "similar" objects are: in static languages, this means they are derived from the same class. In duck-typed languages, it means they share the same behavior, so all we're doing here is deciding exactly how much behavior they need to share. > But, then Python has > this same problem, because > > l += a > > doesn't do 'in-place' for the list, but does do inplace if 'l' were an > array. Yes, it does do it for the list, but differently. However, maybe a tuple is a better example: >>> a = (1,2,3) >>> id(a) 331328 >>> a += (3,4,5) >>> a (1, 2, 3, 3, 4, 5) >>> id(a) 236712 Whereas with numpy: >>> import numpy as N >>> a = N.array((1,2,3)) >>> id(a) 6436864 >>> a += (3,4,5) >>> a array([4, 6, 8]) >>> id(a) 6436864 So there are two differences: addition is defined differently, and one is in-place, and one isn't. Personally, I've always thought that the "in place" operators should never work with immutable types: in=place should mean in-place,, not "in-place if possible, but not if not possible. and yes, this is completely analogous to the issue at hand. And then we have this (already pointed out in this thread) >>> l = [1,2,3] >>> l [1, 2, 3] >>> a array([4, 6, 8]) >>> l += a >>> l array([ 5, 8, 11]) >>> b [4, 6, 8] >>> l = [1,2,3] >>> l += b >>> l [1, 2, 3, 4, 6, 8] >>> l = [1,2,3] >>> t = (3,4,5) >>> l+=t >>> l [1, 2, 3, 3, 4, 5] How this is anything but a bug is beyond me! Lists do define the in-place operator, and really do it in-place. That being the case, I would certainly never expect += to change the type of a list! It doesn't with a tuple. Is this a numpy issue or a Python one? > I don't view it as a problem of > function behavior as much as problem with documentation and "mismatch > between what a particular user expects and what is actually done." Quite true. However, what I advocate is that to keep that mismatch as rare as possible, the ONLY functions that should have the "sometimes a view and sometimes a copy" behavior should be functions that explicitly exist to provide that behavior, such as asarray(). All it does is save typing and increase the potential for confusion and errors to built it in to functions that have other uses. Travis Oliphant wrote: > on making methods that return views raise an error when impossible and > not changing and/or deprecating functions (i.e. the functions are simple > wrappers around the methods). +1 > on making methods that return views raise an error when impossible and > changing the function to make a copy +1 : Only if the functions are moved to a separate "backwards compatible" name space. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Tue Mar 28 11:05:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 11:05:01 2006 Subject: [Numpy-discussion] Re: Ransom Proposals In-Reply-To: <4429841F.4040105@noaa.gov> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428C399.1080002@ieee.org> <4429841F.4040105@noaa.gov> Message-ID: Christopher Barker wrote: > And then we have this (already pointed out in this thread) > >>>> l = [1,2,3] >>>> l > [1, 2, 3] >>>> a > array([4, 6, 8]) >>>> l += a >>>> l > array([ 5, 8, 11]) > >>>> b > [4, 6, 8] >>>> l = [1,2,3] >>>> l += b >>>> l > [1, 2, 3, 4, 6, 8] > >>>> l = [1,2,3] >>>> t = (3,4,5) >>>> l+=t >>>> l > [1, 2, 3, 3, 4, 5] > > How this is anything but a bug is beyond me! Lists do define the > in-place operator, and really do it in-place. That being the case, I > would certainly never expect += to change the type of a list! It doesn't > with a tuple. Is this a numpy issue or a Python one? A Python one. http://mail.python.org/pipermail/python-dev/2006-March/062889.html -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at cox.net Tue Mar 28 11:22:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 11:22:07 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4429841F.4040105@noaa.gov> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428C399.1080002@ieee.org> <4429841F.4040105@noaa.gov> Message-ID: <44298CB6.90007@cox.net> Christopher Barker wrote: > I want to start by saying a am +1 on everything Tim is advocating, > just so you all know it's not just him. I'm not writing any code, > however. Thanks Chris. > > A few comments: > > Travis Oliphant wrote: > >> It would seem that most of Tim's complaints are directly against >> polymorphism. > > > I don't think so. I think Tim (and I) just want it to be really clear > what the polymorphic functions are and what aren't. Also, for > polymorphism to work, there always has to be some definition of what > "similar" objects are: in static languages, this means they are > derived from the same class. In duck-typed languages, it means they > share the same behavior, so all we're doing here is deciding exactly > how much behavior they need to share. > > > But, then Python has > >> this same problem, because >> >> l += a >> >> doesn't do 'in-place' for the list, but does do inplace if 'l' were >> an array. > > > Yes, it does do it for the list, but differently. However, maybe a > tuple is a better example: > > >>> a = (1,2,3) > >>> id(a) > 331328 > >>> a += (3,4,5) > >>> a > (1, 2, 3, 3, 4, 5) > >>> id(a) > 236712 > > Whereas with numpy: > > >>> import numpy as N > >>> a = N.array((1,2,3)) > >>> id(a) > 6436864 > >>> a += (3,4,5) > >>> a > array([4, 6, 8]) > >>> id(a) > 6436864 > > So there are two differences: addition is defined differently, and one > is in-place, and one isn't. Personally, I've always thought that the > "in place" operators should never work with immutable types: in=place > should mean in-place,, not "in-place if possible, but not if not > possible. and yes, this is completely analogous to the issue at hand. The problem with this is that then inplace add doesn't work with numbers; a huge use case that can't be neglected. Really though, in-place addition for immutable types works fine unless one is overusing 'is'. Issues can come up with mixing mutable and imutatble types though, or two mutable types if the left handed one doesn't define iadd. > And then we have this (already pointed out in this thread) > > >>> l = [1,2,3] > >>> l > [1, 2, 3] > >>> a > array([4, 6, 8]) > >>> l += a > >>> l > array([ 5, 8, 11]) It is a bug. Travis brought this up on PyDev and they agreed that there is a problem with PyNumber_InplaceAdd. Armin's looking at this now in the hope of getting a fix in for 2.5. The result should be a list: [1,2,3,5,8,11]. > >>> b > [4, 6, 8] > >>> l = [1,2,3] > >>> l += b > >>> l > [1, 2, 3, 4, 6, 8] > > >>> l = [1,2,3] > >>> t = (3,4,5) > >>> l+=t > >>> l > [1, 2, 3, 3, 4, 5] > > How this is anything but a bug is beyond me! Lists do define the > in-place operator, and really do it in-place. That being the case, I > would certainly never expect += to change the type of a list! It > doesn't with a tuple. Is this a numpy issue or a Python one? A Python one. However, I'll be writing more on this in a moment; the behaviour that causes the issue with list is slated to be fixed, but I think that there's some other cases to consider. > >> I don't view it as a problem of function behavior as much as problem >> with documentation and "mismatch between what a particular user >> expects and what is actually done." > > > Quite true. However, what I advocate is that to keep that mismatch as > rare as possible, the ONLY functions that should have the "sometimes a > view and sometimes a copy" behavior should be functions that > explicitly exist to provide that behavior, such as asarray(). All it > does is save typing and increase the potential for confusion and > errors to built it in to functions that have other uses. > > Travis Oliphant wrote: > >> on making methods that return views raise an error when impossible >> and not changing and/or deprecating functions (i.e. the functions are >> simple wrappers around the methods). > > > +1 > >> on making methods that return views raise an error when impossible >> and changing the function to make a copy > > > +1 : Only if the functions are moved to a separate "backwards > compatible" name space. > FWIW, I'm fine if the functions always copy and methods always returns views, since this gets rid of the view/copy dilema. I believe that Sasha previously raised objections to this on backwards compatibility grounds. I believe Sasha preferred leaving the functions alone and just not recomending their use. I was fine with that too as long as the functions got segregated. Travis didn't like that, which loops us back around to the current proposal. Regards, -tim From tim.hochberg at cox.net Tue Mar 28 12:18:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 12:18:04 2006 Subject: [Numpy-discussion] inplace add Message-ID: <442999E3.8030102@cox.net> Travis recently brought up the odd behaviour of adding an array to a list using inplace add, notably: >>> l = range(4) >>> a = arange(4) >>> l += a >>> l array([0, 2, 4, 6]) It was agreed that this was a deficiency in PyNumber_InplaceAdd and this is slated to be fixed. Hopefully for 2.5. Thanks Travis, Robert Kern and Armin Rigo for taking care of this. I had planned to write something here about, how, although this fixes the problem with lists, there is a similar problem with tuples, and since tuples don't have __iadd__, we could still expect to see this problem there. However, I now think that's not the case. The proposed resolution order will work something like: def iadd(a, b): if hasattr(a, '__iadd__'): tmp = a.__iadd__(b) if tmp is not NotImplemented: return tmp if hasattr(a, '__add__'): tmp = a.__add__(b) if tmp is not NotImplemented: return tmp if hasattr(b, '__radd__'): temp = b.__radd__ if tmp is not NotImplemented: return tmp raise TypeError("can't add %r and %r" % (type(a), type(b))) You'll notice if you try this, that "iadd(atuple, anarray)" throws an error because atuple refuses to be added to anything other than another tuple.That raises an error at "__add__" and prevents any oddities. So, we should be safe from both tuples and lists, and really they are the only types that are prevalent enough to worry about in this context. Still, I have a niggling feeling that iadd should really not be calling radd at all, but nothing concrete. -tim From oliphant.travis at ieee.org Tue Mar 28 12:41:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 12:41:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442957A4.7070301@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> Message-ID: <4429944C.9060103@ieee.org> Tim Hochberg wrote: > > +1 Assuming I understand all of the implications. > > * When doing automated conversion from Numeric/numarray to numpy, > "reshape(obj) => asarray(obj).reshape()". Doing anything else is going > to cause breakage and/or massive slowdowns. Can the conversion tool > handle this? I would guess yes, but that's just a guess. Why would we need to change this --- even to ascontiguousarray(obj).reshape()? > > * This means that "somenoncontiguousarray.reshape(newshape)" will > oftern raise an error. That's a good thing, but I figured I'd get it > out in the open now so no one is suprised. I'm still not sold that raising an error here is a good thing, which is why I think the function should work as it always as and *not* be deprecated or hard to get at even if the method is changed to raise an error if the reshape cannot be accomplished without a copy. Discontiguous arrays actually show up quite a bit. I'm not convinced that Tim's concern over "confusion" between when something returns a view and when it returns a copy is that big of a deal. I maintain it's only disconcerting for someone who wants to do something "advanced". That person will know how to figure it out. If things are "fixed" then new users will get bitten by slow code or errors that don't make sense. I'm opposed to making things unnecessarily hard for new users. Let's discuss this with a particular example in mind that is not contrived to do 'in-place' operations. I think this will exemplify the crux of the matter for those who may not quite understand what this discussion is about. a = rand(4,5,3) b = a.transpose().reshape(15,4) c = a.reshape(15,4) In this example b is now a copy of the data in a and c is a "view" of the data in a. The reason for this is because a.transpose() returns a "view" of the data which is not C-style contiguous. The reshape function needs to know how to go between a linear sequence of elements and a multidimensional array --- the default is current C-style contiguous (last-index varies the fastest). If I understand correctly, Tim want's the second line to raise an error so that the reshape method *always* returns a view. I'm not enthused about having the second line raise an error because it is an example of common code (perhaps a transpose does not precede the reshape but a slice selection --- there are lots of ways to get discontiguous arrays). As it stands, I think the only problems that arise are when the user wants to modify b and/or c in-place. Then, one is not sure whether or not b (or c) is pointing to the data of a or not. My argument is that the reshape method documentation should state that you should not modify the contents of the return value of reshape unless it doesn't matter whether or not you have a copy of the data. I would even be willing to make 'views' un-writeable by default so that there was no way to "inadvertently" modify the data in a, unless you know you want to. We had this discussion for ravel months ago and the flatten method which always returns copies was created. The ravel method is another one that returns views when it can (contiguous) and copies when it can't. I think reshape and ravel are the only methods that behave this way. I'm not convinced that changing their current, standard behavior is going to buy us anything but more confusion. > > * Since x.reshape would no longer return copies, the order flag is > unnecessary. I don't believe this. See my other posts on the difference between the array striding and how you want to interpret the array. > > * numpy.reshape(obj, newshape) on the other hand, always copies, so it > *could* grow an order flag. I think that I'd prefer that it didn't; if > you need that kind of control of the resulting arrays you should be > using array to copy your matrices. No, this is not what I understand. the reshape function would not *always* copy but would behave as it currently does. Again, you haven't sold me on the need to absolutely, all the time, return either views or copies. I think such a dogmatic view is too limiting in this case. I have not been bitten by this in over 10 years. Why is this such an issue now? > > * What does fixing ravel to be consistent mean? Always return a view? > Or always return a copy? The latter seems more useful, as the first is > already covered by both obj.reshape(-1) and obj.flat. I meant that whatever the behavior is for .reshape should be the same for .ravel since it is syntactic sugar for obj.reshape(-1) And obj.flat is not even in the same ball-park since it returns an array_iterator object and not an array. Best, -Travis From jelleferinga at gmail.com Tue Mar 28 13:21:05 2006 From: jelleferinga at gmail.com (Jelle Feringa / EZCT Architecture & Design Research) Date: Tue Mar 28 13:21:05 2006 Subject: [Numpy-discussion] scipybooksample | dtype Message-ID: <37fac3b90603281320v365e9543t55f96abc7d495e4b@mail.gmail.com> Hi all, I'm familiarizing myself with scipy, and working my way through the http://numeric.scipy.org/scipybooksample.pdf tutorial. the tutorial states: ------------------------------------- >>> a = array([[1,2,3],[4,5,6]]) >>> a.shape (2, 3) >>> a.dtype ------------------------------------- when i run this code on my machine, having numpy 0.9.6 installed, i get the following: >>> a = numpy.array([[1,2,3],[4,5,6]]) >>> a.dtype dtype('>> numpy.version.version '0.9.6' >>> >>> seems that the way dtypes are being printed changed since the scipybooksample.pdf was released, would anyone of you be able to point me out where I can read up to what the updated dtypes signify? it's a bit cryptic to the uninitiated. i do understand that at the rate scipy/numpy is progressing that it can be hard to get the documentation consist with the current version. //btw: the new ndimage module is a great piece of work, thanks so much for all your efforts! -jelle -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Mar 28 13:24:04 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 13:24:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4429944C.9060103@ieee.org> References: <4424AB16.6000007@cox.net> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> <4429944C.9060103@ieee.org> Message-ID: Travis, On 3/28/06, Travis Oliphant wrote: > > Tim Hochberg wrote: a = rand(4,5,3) > b = a.transpose().reshape(15,4) > c = a.reshape(15,4) I would do this: >> b = reshape(a.transpose(), (15,4)) or, if the functional version of transpose would always return a copy: >> b = transpose(a).reshape(15,4) As I said before, I simply stopped using reshape because of its ambiguous behaviour. There were clearer ways to get what I wanted. But in any case, the casual user will hardly notice the inefficiency of making a copy. Users concerned with speed can use methods and should know enough to understand what is going on, but they should be warned if their attemps at efficiency aren't going to pan out in a particular case. > My argument is that the reshape method documentation should state that > you should not modify the contents of the return value of reshape unless > it doesn't matter whether or not you have a copy of the data. Who reads documentation? Why should they have to rummage through the footnotes or run a bunch of test cases to discover why things aren't behaving? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Mar 28 13:38:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 13:38:02 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error Message-ID: <4429AC89.6090301@ieee.org> It look's like Tim's arguments have won out for the most part. The .reshape and .ravel methods of arrays now return views only or raise an error if that is not possible. However, both of these methods still have an order argument because I don't think it is prudent to have a.transpose().ravel() automatically column-scan --- you have to tell ravel you want a Fortran-order result. The reshape and ravel functions behave exactly as they used to (view if possible, copy otherwise). I don't think these functions should be deprecated. I don't think any of the functional interfaces should be deprecated, either. Part of the reason for the change is that it was not disruptive. None of the NumPy code broke (except for a small piece of arrayprint) after the change. None of the SciPy code broke either (as far as I can tell). Hopefully we can put this to rest and move on to more important issues... -Travis From bblais at bryant.edu Tue Mar 28 14:34:02 2006 From: bblais at bryant.edu (Brian Blais) Date: Tue Mar 28 14:34:02 2006 Subject: [Numpy-discussion] getting data pointer to numpy object in Pyrex Message-ID: <4429B89E.30100@bryant.edu> Hello, I posted this on the Pyrex list, but I figured someone here may be able to answer it too. Is here an easy way, in Pyrex, to get the data pointer to a numpy object which is *not* passed as a parameter? For example, I know I can do: #--------------- cdef extern from "numpy/arrayobject.h": struct PyArray_Descr: int type_num, elsize char type ctypedef class numpy.ArrayType [object PyArrayObject]: cdef char *data cdef int nd cdef int *dimensions, *strides cdef object base cdef PyArray_Descr *descr cdef int flags def test1(ArrayType w): # pass a numpy object cdef double *w_p # get the pointers w_p=w.data #--------------- ...but I would like to do something like... #------------------------------- def test2(d): # pass a python dict w=d['w'] # get the numpy object cdef double *w_p # get the pointers w_p=w.data #------------------------------- Is there a way to do this? thanks, Brian Blais -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From tim.hochberg at cox.net Tue Mar 28 14:40:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 14:40:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4429944C.9060103@ieee.org> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> <4429944C.9060103@ieee.org> Message-ID: <4429BAF4.6070801@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> +1 Assuming I understand all of the implications. >> >> * When doing automated conversion from Numeric/numarray to numpy, >> "reshape(obj) => asarray(obj).reshape()". Doing anything else is >> going to cause breakage and/or massive slowdowns. Can the conversion >> tool handle this? I would guess yes, but that's just a guess. > > Why would we need to change this --- even to > ascontiguousarray(obj).reshape()? Because things could break or get very very slow. Perhaps I'm the only one who's ever used reshape to excess, so perhaps it won't really hurt anyone. You can see the code that originally inspired my loathing for reshape here: http://starship.python.net/~hochberg/fourier.py.html Look at fft_base2. I don't *think* that this code would break under this change, however it would certainly start running very slow if reshape started returning copies all the time. On the other hand, asarray(x).reshape would work about the same. It is true that I have nothing comparable in my active codebase, so maybe such stuff is too rare to worry about. [The code above is quite old and may not even work on current Numeric. It was quite cool though -- running within a factor of two in speed of Numeric's built in FFT for large power of two length arrays]. So in summary, some things could break. Or they could become really slow. I don't know how much of a problem, if any, that this will be in practice. I didn't find anything in my active codebase where it would be a problem, so I guess it doesn't matter to me. >> >> * This means that "somenoncontiguousarray.reshape(newshape)" will >> oftern raise an error. That's a good thing, but I figured I'd get it >> out in the open now so no one is suprised. > > I'm still not sold that raising an error here is a good thing, which > is why I think the function should work as it always as and *not* be > deprecated or hard to get at even if the method is changed to raise an > error if the reshape cannot be accomplished without a copy. I'm not sure what you are saying here. My bullet point above only relates to methods. It sounds like your on board with methods always returning views. Yet you disagree. Are we talking past each other here? > Discontiguous arrays actually show up quite a bit. I'm not convinced > that Tim's concern over "confusion" between when something returns a > view and when it returns a copy is that big of a deal. I maintain > it's only disconcerting for someone who wants to do something > "advanced". That person will know how to figure it out. If things > are "fixed" then new users will get bitten by slow code or errors that > don't make sense. I'm opposed to making things unnecessarily hard for > new users. Me too. But I have come to opposite conclusion from you here. If the functions always returned a copy, they are safe and easy to use. And the error message for the method reshape could be made quite informative: """cannot reshape a discontiguous array with the method reshape use the function reshape instead or use ascontiguousarray to get a contiguous array""" With something like that in place, I think it would be hard to get stuck to too long if one ran into an error reshaping a discontiguous array. > > Let's discuss this with a particular example in mind that is not > contrived to do 'in-place' operations. I think this will exemplify > the crux of the matter for those who may not quite understand what > this discussion is about. > > a = rand(4,5,3) > b = a.transpose().reshape(15,4) > c = a.reshape(15,4) > > In this example b is now a copy of the data in a and c is a "view" of > the data in a. The reason for this is because a.transpose() returns > a "view" of the data which is not C-style contiguous. The reshape > function needs to know how to go between a linear sequence of elements > and a multidimensional array --- the default is current C-style > contiguous (last-index varies the fastest). > If I understand correctly, Tim want's the second line to raise an > error so that the reshape method *always* returns a view. Yes. Well that's one of my many positions. I've stated my primary goal many times, which is that with a few careful exceptions, functions should always return views or always return copys. There are many ways to get closer to that goal, but each one has someone who dislikes it. This approach is one of those ways. > I'm not enthused about having the second line raise an error because > it is an example of common code (perhaps a transpose does not precede > the reshape but a slice selection --- there are lots of ways to get > discontiguous arrays). As it stands, I think the only problems that > arise are when the user wants to modify b and/or c in-place. Then, > one is not sure whether or not b (or c) is pointing to the data of a > or not. > My argument is that the reshape method documentation should state that > you should not modify the contents of the return value of reshape > unless it doesn't matter whether or not you have a copy of the data. > I would even be willing to make 'views' un-writeable by default so > that there was no way to "inadvertently" modify the data in a, unless > you know you want to. I weeble and wobble on this suggestion. It is appealing, but it is a little weird. To make it truly safe you'd need to lock both the operand and the result, which I expect would make people squawk.. > > > We had this discussion for ravel months ago and the flatten method > which always returns copies was created. The ravel method is another > one that returns views when it can (contiguous) and copies when it can't. > I think reshape and ravel are the only methods that behave this way. > I'm not convinced that changing their current, standard behavior is > going to buy us anything but more confusion. Wow! So we have: reshape(-1) ravel() flatten() flat And that's just the methods. That's not on todays agenda however. >> >> * Since x.reshape would no longer return copies, the order flag is >> unnecessary. > > I don't believe this. See my other posts on the difference between > the array striding and how you want to interpret the array. I think I see the root of our difference here. My thinking has been that the underlying data could be stored in either Fortran-order or C-order, but the numarray objects themselves were always C-order in the sense that the last axes were viewed as varying fastest. Thus you only need to know the order when you copy it since that controls the layout of the underlying data structure. On the other hand, your view is that the entire numarray object can be viewed as either C-order or Fortran-order. Thus, every reshaping operation needs to know the order of the matrix. Is that correct? I'm not sure how I feel about that, but at least I think I've got it now. And if we view the arrays that way, doesn't that mean flat is broken? >> >> * numpy.reshape(obj, newshape) on the other hand, always copies, so >> it *could* grow an order flag. I think that I'd prefer that it >> didn't; if you need that kind of control of the resulting arrays you >> should be using array to copy your matrices. > > No, this is not what I understand. the reshape function would not > *always* copy but would behave as it currently does. Again, you > haven't sold me on the need to absolutely, all the time, return either > views or copies. I think such a dogmatic view is too limiting in > this case. > > I have not been bitten by this in over 10 years. Why is this such an > issue now? Because it's probably the only chance we'll ever get to change some of these things. >> >> * What does fixing ravel to be consistent mean? Always return a view? >> Or always return a copy? The latter seems more useful, as the first >> is already covered by both obj.reshape(-1) and obj.flat. > > > I meant that whatever the behavior is for .reshape should be the same > for .ravel since it is syntactic sugar for obj.reshape(-1) > > And obj.flat is not even in the same ball-park since it returns an > array_iterator object and not an array. > Regards, -tim From robert.kern at gmail.com Tue Mar 28 15:06:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 15:06:02 2006 Subject: [Numpy-discussion] Re: getting data pointer to numpy object in Pyrex In-Reply-To: <4429B89E.30100@bryant.edu> References: <4429B89E.30100@bryant.edu> Message-ID: Brian Blais wrote: > Hello, > > I posted this on the Pyrex list, but I figured someone here may be able > to answer it too. > > Is here an easy way, in Pyrex, to get the data pointer to a numpy object > which is *not* passed as a parameter? For example, I know I can do: > > #--------------- > cdef extern from "numpy/arrayobject.h": > > struct PyArray_Descr: > int type_num, elsize > char type > > ctypedef class numpy.ArrayType [object PyArrayObject]: > cdef char *data > cdef int nd > cdef int *dimensions, *strides > cdef object base > cdef PyArray_Descr *descr > cdef int flags > > def test1(ArrayType w): # pass a numpy object > > cdef double *w_p > > # get the pointers > w_p=w.data > #--------------- > > ...but I would like to do something like... > > #------------------------------- > def test2(d): # pass a python dict > > w=d['w'] # get the numpy object > > cdef double *w_p > > # get the pointers > w_p=w.data > #------------------------------- > > > Is there a way to do this? You will probably want to use the c_numpy.pxd file in numpy/doc/pyrex/. It has exposed a bit more of the API than you have. If you have any contributions, we would like to fold them in. In general, you would just make a cast like you would for any other C type. However, you must be sure that the object actually is an array. I think the following code would work as expected, including the case where d['w'] can't be turned into an array, but I have not tested it. cimport c_numpy import numpy c_numpy.import_array() def test2(d): cdef c_numpy.ndarray w w = numpy.asarray(d['w']) # ... -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Tue Mar 28 15:14:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 15:14:00 2006 Subject: [Numpy-discussion] Re: [SciPy-user] multiplying sparse matrices in scipy-0.4.8? In-Reply-To: <98790998-659E-42C3-A2AD-005AB68CCA12@ftw.at> References: <87odzx3x6r.fsf@hmm.lanl.gov> <98790998-659E-42C3-A2AD-005AB68CCA12@ftw.at> Message-ID: <4429C30A.9040206@ieee.org> Ed Schofield wrote: > > > Yes, this is a bug. > > Travis, could you please take a look at this? The FORTRAN functions > dcscmucsc and dcscmucsr both seem to be returning incorrect values in > indptr. In fact, could you please explain the Python code in matmat() > that calls these functions? I'd like to understand what the "while 1" > loop is for, and in particular why we have > c, rowc, ptrc, irow, kcol, ierr = func(*args) > when c, rowc, etc are part of *args anyway. The arguments are input and output arguments because that is my understanding of f2py's preference if something is both input and ouput rather than using an in-place argument. The while True: loop is so that resizing can occur if the guess on how many non-zero elements to reserve for the output matrix is wrong. There were some problems with the re-entry that I've now fixed. The tests you added to SciPy now pass. -Travis From tim.hochberg at cox.net Tue Mar 28 15:36:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 15:36:01 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429AC89.6090301@ieee.org> References: <4429AC89.6090301@ieee.org> Message-ID: <4429C812.90109@cox.net> Travis Oliphant wrote: > > It look's like Tim's arguments have won out for the most part. Hooray. Now let's hope I'm right! > > The .reshape and .ravel methods of arrays now return views only or > raise an error if that is not possible. However, both of these > methods still have an order argument because I don't think it is > prudent to have a.transpose().ravel() automatically column-scan --- > you have to tell ravel you want a Fortran-order result. OK, I think I understand why you want the order flag now. But isn't flat in a weird state now? All of the other reshape methods take an order parameter, but flat is inherently C-ordered, is it not? Also, is it useful to view an array as fortran ordered? I suppose if you are working with multidimensional Fortran data you want to keep it in the correct order. Still, something seems not quite right about the current state of affairs. Let me ask a question about the order flag: does it mean anything for a noncontiguous array? As I interpret it, right now: CONTIGUOUS = True, FORTRAN = True => Strides are Fortran order CONTIGUOUS = True, FORTRAN = False=> Strides are C order CONTIGUOUS = False, FORTRAN = Any=> Matrix is Discontiguous Is that correct? Or is it possible to have strides that neither C-order not Fortran-Order in a contiguous matrix? I'm trying to find a way to shoehorn an order flag in here without breaking anything, but I don't think it'll fit. So you'd need a separate flag to indicate the reshape-order if one wanted to do that. Mostly this would make life simpler for everyone, in normal use, the reshape-order flag gets set at array creation and everything just works. flat works properly, the order flag is only needed in construction. However, since the behaviour of the array is different with respect to reshape, you'd probably want to tag the repr somehow ('farray' instead of 'array' perhaps). In practice, this would be essentially the same as having a sister class to ndarray that worked on Fortran ordered data. Everything would behave the same except for how reshape works. I'm not sure if any of that is a good idea, but I'm not entirely convinced that the order flag, although a big improvement over what it replaced, is the best approach, so I'm musing. > The reshape and ravel functions behave exactly as they used to (view > if possible, copy otherwise). I don't think these functions should be > deprecated. I don't think any of the functional interfaces should be > deprecated, either. The good thing about functions is that I can always hot patch numpy with safer versions if I like. So I'll call this good. When I get around to it, I'll probably try patching in versions of reshape and transpose that lock either views or copies and see how that works out. I'll report back on how it goes. > > Part of the reason for the change is that it was not disruptive. None > of the NumPy code broke (except for a small piece of arrayprint) after > the change. None of the SciPy code broke either (as far as I can tell). > Hopefully we can put this to rest and move on to more important issues... Hooray! -tim From charlesr.harris at gmail.com Tue Mar 28 16:58:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 16:58:02 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429C812.90109@cox.net> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> Message-ID: On 3/28/06, Tim Hochberg wrote: > > Travis Oliphant wrote: > > > > > It look's like Tim's arguments have won out for the most part. > > Hooray. Now let's hope I'm right! +1. Now I'm nervous. > > > > Part of the reason for the change is that it was not disruptive. None > > of the NumPy code broke (except for a small piece of arrayprint) after > > the change. None of the SciPy code broke either (as far as I can tell). > > Hopefully we can put this to rest and move on to more important > issues... But what could be more important to safe programming than having a sure grasp of when you are working with a view or a copy? O_- Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Mar 28 17:02:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 17:02:01 2006 Subject: [Numpy-discussion] NetCDF module that links against NumPy Message-ID: <4429DC3D.1010000@ieee.org> I've adapted Konrad Hinsen's Scientific.IO.NetCDF module to work with NumPy. For now it's available at http://sourceforge.net/project/showfiles.php?group_id=1315&package_id=185504&release_id=405511 I would like to put this module in SciPy but don't understand whether or not the License is compatible. It sounds GPL-ish. Would it be O.K. to place in the SciPy/sandbox? Regards, -Travis From zpincus at stanford.edu Tue Mar 28 17:36:11 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 17:36:11 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> Message-ID: <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> Uh oh, Just updated to new numpy so that dot() wouldn't be broken. What fun to play with the new order/ravel/reshape stuff. Unfortunately, the ravel/reshape methods seem to not be quite worked out properly yet. Basically (and as expected) the ravel method doesn't work anymore on fortran-strided arrays. But since fortran- strided arrays happen *all-the-time* (e.g. after a transpose operation), things get confusing fast. In fact, absolutely no reshape operation (other than the identity reshape) will work on fortran-strided arrays (see below). Also, the errors could be more helpful, and anyway ravel shouldn't fail with an error about reshape fortran-strided. This can't be the optimal situation, since it makes the method interface next-to-useless (because exceptions might be thrown at any moment) unless a *lot* of care is taken to properly sanitize the arrays that are used (in which case any speed gains are probably lost). In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.7.2291' In [3]: a = numpy.array([[1,2],[3,4],[5,6]]) In [4]: b = a.transpose() In [5]: a.ravel() Out[5]: array([1, 2, 3, 4, 5, 6]) In [6]: b.ravel() ValueError: cannot return a view from reshape. In [7]: b.shape Out[7]: (2, 3) In [8]: b.reshape((1,6)) ValueError: cannot return a view from reshape. In [9]: b.reshape((6,1)) ValueError: cannot return a view from reshape. In [10]: b.reshape((3,2)) ValueError: cannot return a view from reshape. This state of affairs seems pretty hostile to both advanced users and basic ones. Now I need to go through all of my code and eliminate the use of array methods, since about half of the time I'll be calling things like ravel on arrays that have been transposed. What a waste -- is there something better I can do or something better numpy can do? At the very least, a strong convention really needs to be established and made clear about what the methods do (never return copies) and what the functions do (sometimes return copies? hardly a strong convention), and when to choose one or the other. Zach From zpincus at stanford.edu Tue Mar 28 17:55:05 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 17:55:05 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> Message-ID: <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Let me give a more concrete example, pulled from some code I was actually using. The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 ) It appears that to be safe, a function like this must be rewritten as: def distance_squared(a, b): return numpy.sum( (numpy.ravel(a) - numpy.ravel(b))**2 ) At the very least, it seems surprising (to a novice especially, but to me too, and I don't consider myself a total newb at this) that for something as simple as this, that the ravel method might not work in some cases, especially since one of the exciting new features of numpy is (finally!) having useful array methods (makes life in ipython with tab completion much easier). Zach PS. I know that the "proper" solution would look more like the below. I guess the question is how tolerant should numpy be of code that deviates from "proper". def distance_squared(a, b): return ((a - b)**2).sum() On Mar 28, 2006, at 5:35 PM, Zachary Pincus wrote: > Uh oh, > > Just updated to new numpy so that dot() wouldn't be broken. What > fun to play with the new order/ravel/reshape stuff. > > Unfortunately, the ravel/reshape methods seem to not be quite > worked out properly yet. Basically (and as expected) the ravel > method doesn't work anymore on fortran-strided arrays. But since > fortran-strided arrays happen *all-the-time* (e.g. after a > transpose operation), things get confusing fast. > > In fact, absolutely no reshape operation (other than the identity > reshape) will work on fortran-strided arrays (see below). Also, the > errors could be more helpful, and anyway ravel shouldn't fail with > an error about reshape fortran-strided. > > This can't be the optimal situation, since it makes the method > interface next-to-useless (because exceptions might be thrown at > any moment) unless a *lot* of care is taken to properly sanitize > the arrays that are used (in which case any speed gains are > probably lost). > > In [1]: import numpy > In [2]: numpy.__version__ > Out[2]: '0.9.7.2291' > In [3]: a = numpy.array([[1,2],[3,4],[5,6]]) > In [4]: b = a.transpose() > In [5]: a.ravel() > Out[5]: array([1, 2, 3, 4, 5, 6]) > In [6]: b.ravel() > ValueError: cannot return a view from reshape. > In [7]: b.shape > Out[7]: (2, 3) > In [8]: b.reshape((1,6)) > ValueError: cannot return a view from reshape. > In [9]: b.reshape((6,1)) > ValueError: cannot return a view from reshape. > In [10]: b.reshape((3,2)) > ValueError: cannot return a view from reshape. > > This state of affairs seems pretty hostile to both advanced users > and basic ones. Now I need to go through all of my code and > eliminate the use of array methods, since about half of the time > I'll be calling things like ravel on arrays that have been > transposed. What a waste -- is there something better I can do or > something better numpy can do? > > At the very least, a strong convention really needs to be > established and made clear about what the methods do (never return > copies) and what the functions do (sometimes return copies? hardly > a strong convention), and when to choose one or the other. > > Zach > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Tue Mar 28 18:20:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 18:20:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> Message-ID: <4429EEB2.7000803@ieee.org> Zachary Pincus wrote: > Uh oh, > > Just updated to new numpy so that dot() wouldn't be broken. What fun > to play with the new order/ravel/reshape stuff. > > Unfortunately, the ravel/reshape methods seem to not be quite worked > out properly yet. Basically (and as expected) the ravel method doesn't > work anymore on fortran-strided arrays. But since fortran-strided > arrays happen *all-the-time* (e.g. after a transpose operation), > things get confusing fast. > You've demonstrated my biggest concern with adopting Tim's approach. Because you can very easily get arrays that are not C-contiguous, you end up with a stituation in which the reshape and ravel methods are not very useful in my mind. > In fact, absolutely no reshape operation (other than the identity > reshape) will work on fortran-strided arrays (see below). Also, the > errors could be more helpful, and anyway ravel shouldn't fail with an > error about reshape fortran-strided. There is one more problem with what I checked in. The PyArray_Ravel() function in C is now completely different and is used in _check_axis and therefore creates a segfault. So, I've decided to compromise. .reshape maintains it's view-only behavior .ravel() goes back to copy-if-necessary behavior (If you don't like it then use .reshape(-1) ). This should fix most of Zach's concerns and also fixes the segfault you get with rand(10,3).transpose().sum() -Travis From webb.sprague at gmail.com Tue Mar 28 18:28:08 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Tue Mar 28 18:28:08 2006 Subject: [Numpy-discussion] round_() problems Message-ID: Hi all, numpy.__version__ = 0.9.4 (After my svd question, I hope to follow up with something a tad less silly. Unfortunately, this a lot vaguer.) I am getting the following backtrace when I try to round a matrix, but I am not sure why (it works in other places...) Has anyone seen this error before, any explanation? Anyway to get more information for diagnosis? The function the traceback hits only sets up an html table of rounded matrix items, and it works for some matrices. There is a possibility the matrix I am feeding it is messed up, which is why I wanted to print it.... Thanks, feel free to cc me as I read the list as a digest. Begin traceback: (import numpy as N) File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table outStr += '' + str(N.round_(item,prec)) + '' File "/usr/lib/python2.4/site-packages/numpy/core/oldnumeric.py", line 461, in round_ a = absolute(multiply(a, 10.**decimals)) TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'numpy.ndarray' From zpincus at stanford.edu Tue Mar 28 18:44:03 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 18:44:03 2006 Subject: [Numpy-discussion] segfault with min() method on fortran-strided arrays. Message-ID: Hi, I found a segfault in numpy that seems to have to do with the new 'order' stuff. Steps to reproduce and gdb backtrace below. This is new since numpy 0.9.7.2262, which is what I updated from today. Zach In [1]: import numpy In [2]: a = numpy.array([[1,2],[3,4],[5,6]]) In [3]: b = numpy.transpose(a) In [4]: a.min() Out[4]: 1 In [5]: b.min() Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x0000000c _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/core/ src/arrayobject.c:3505 3505 *axis = PyArray_NDIM(temp)-1; (gdb) backtrace #0 _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/ core/src/arrayobject.c:3505 #1 0x01170594 in array_min (self=0x6776b0, args=0x139eba0, kwds=0x0) at numpy/core/src/multiarraymodule.c:2909 #2 0x0028b984 in PyEval_EvalFrame (f=0x1868c10) at Python/ceval.c:3558 [...python stack frames snipped...] (gdb) info locals newdim = { ptr = 0xbfffeaa8, len = 1 } val = {-1} temp = (PyObject *) 0x6776b0 n = 2 (gdb) info args arr = (PyArrayObject *) 0x76696577 axis = (int *) 0xbfffeb0c flags = 0 From zpincus at stanford.edu Tue Mar 28 19:06:05 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 19:06:05 2006 Subject: [Numpy-discussion] segfault with min() method on fortran-strided arrays. In-Reply-To: References: Message-ID: <8B8C9DFE-430F-41CD-B170-0545DE57A4F4@stanford.edu> Already fixed in svn -- thanks Travis. Zach On Mar 28, 2006, at 6:43 PM, Zachary Pincus wrote: > Hi, > > I found a segfault in numpy that seems to have to do with the new > 'order' stuff. Steps to reproduce and gdb backtrace below. This is > new since numpy 0.9.7.2262, which is what I updated from today. > > Zach > > > In [1]: import numpy > In [2]: a = numpy.array([[1,2],[3,4],[5,6]]) > In [3]: b = numpy.transpose(a) > In [4]: a.min() > Out[4]: 1 > In [5]: b.min() > > Program received signal EXC_BAD_ACCESS, Could not access memory. > Reason: KERN_PROTECTION_FAILURE at address: 0x0000000c > _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/ > core/src/arrayobject.c:3505 > 3505 *axis = PyArray_NDIM(temp)-1; > (gdb) backtrace > #0 _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/ > core/src/arrayobject.c:3505 > #1 0x01170594 in array_min (self=0x6776b0, args=0x139eba0, > kwds=0x0) at numpy/core/src/multiarraymodule.c:2909 > #2 0x0028b984 in PyEval_EvalFrame (f=0x1868c10) at Python/ceval.c: > 3558 > [...python stack frames snipped...] > (gdb) info locals > newdim = { > ptr = 0xbfffeaa8, > len = 1 > } > val = {-1} > temp = (PyObject *) 0x6776b0 > n = 2 > (gdb) info args > arr = (PyArrayObject *) 0x76696577 > axis = (int *) 0xbfffeb0c > flags = 0 > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From pgmdevlist at mailcan.com Tue Mar 28 19:31:09 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 28 19:31:09 2006 Subject: [Numpy-discussion] round_() problems In-Reply-To: References: Message-ID: <200603282230.32210.pgmdevlist@mailcan.com> > File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table > outStr += '' + str(N.round_(item,prec)) + '' Have you tried to use around instead of round ? That way, I think you wouldn't be using oldnumeric and stick to ndarrays... From efiring at hawaii.edu Tue Mar 28 20:44:05 2006 From: efiring at hawaii.edu (Eric Firing) Date: Tue Mar 28 20:44:05 2006 Subject: [Numpy-discussion] NetCDF module that links against NumPy In-Reply-To: <4429DC3D.1010000@ieee.org> References: <4429DC3D.1010000@ieee.org> Message-ID: <442A1075.1030506@hawaii.edu> Travis, Thanks--that is an important addition. In case you or others are not already aware of it, below is the address of Jeff Whitaker's NetCDF4 module for numpy, with NetCDF3 read/write capability. It requires HDF5 as well as NetCDF4 so it is not the easiest module to install, but it is nice to know that NetCDF4 support is already in place for numpy, before NetCDF4 has even been released for general use! http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/python/netCDF4.html Eric Travis Oliphant wrote: > > I've adapted Konrad Hinsen's Scientific.IO.NetCDF module to work with > NumPy. > > For now it's available at > > http://sourceforge.net/project/showfiles.php?group_id=1315&package_id=185504&release_id=405511 > > > I would like to put this module in SciPy but don't understand whether or > not the License is compatible. It sounds GPL-ish. > > Would it be O.K. to place in the SciPy/sandbox? > > > Regards, > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From webb.sprague at gmail.com Tue Mar 28 21:47:03 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Tue Mar 28 21:47:03 2006 Subject: [Numpy-discussion] round_() problems Message-ID: > > > File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table > > outStr += '' + str(N.round_(item,prec)) + '' > > > Have you tried to use around instead of round ? That way, I think you wouldn't > be using oldnumeric and stick to ndarrays... Yes. Same result. From robert.kern at gmail.com Tue Mar 28 21:51:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 21:51:02 2006 Subject: [Numpy-discussion] Re: round_() problems In-Reply-To: References: Message-ID: Webb Sprague wrote: > Hi all, > > numpy.__version__ = 0.9.4 > > (After my svd question, I hope to follow up with something a tad less > silly. Unfortunately, this a lot vaguer.) > > I am getting the following backtrace when I try to round a matrix, but > I am not sure why (it works in other places...) Has anyone seen this > error before, any explanation? Anyway to get more information for > diagnosis? > > The function the traceback hits only sets up an html table of rounded > matrix items, and it works for some matrices. There is a possibility > the matrix I am feeding it is messed up, which is why I wanted to > print it.... > > Thanks, feel free to cc me as I read the list as a digest. > > Begin traceback: > > (import numpy as N) > > File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table > outStr += '' + str(N.round_(item,prec)) + '' > > File "/usr/lib/python2.4/site-packages/numpy/core/oldnumeric.py", > line 461, in round_ > a = absolute(multiply(a, 10.**decimals)) > > TypeError: unsupported operand type(s) for ** or pow(): 'float' and > 'numpy.ndarray' Well, I haven't found an input that reproduces the bug. Can you try to find a (small) input that does fail and post that? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From baecker at physik.tu-dresden.de Tue Mar 28 21:54:02 2006 From: baecker at physik.tu-dresden.de (Arnd Baecker) Date: Tue Mar 28 21:54:02 2006 Subject: [Numpy-discussion] 64bit check_manyways segfault. Message-ID: Hi, with current svn I get a segfault for numpy.test(10, 10): check_manyways (numpy.lib.tests.test_arraysetops.test_aso)Segmentation fault There are no "incompatible pointer type" during the build of numpy. The backtrace is below. I hope this has not been reported during the last two days (I hardly manage to keep up with the traffic during the week ;-). Best, Arnd Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 46912507335168 (LWP 12055)] 0x00002aaaabaa4513 in PyArray_NewFromDescr (subtype=0x2aaaabbef320, descr=0x2aaaabbeba80, nd=1, dims=0x0, strides=0x0, data=0x0, flags=0, obj=0x0) at arrayobject.c:3969 3969 if (dims[i] < 0) { (gdb) bt #0 0x00002aaaabaa4513 in PyArray_NewFromDescr (subtype=0x2aaaabbef320, descr=0x2aaaabbeba80, nd=1, dims=0x0, strides=0x0, data=0x0, flags=0, obj=0x0) at arrayobject.c:3969 #1 0x00002aaaabaa6b2e in PyArray_Zeros (nd=1, dims=0x0, type=0x2aaaabad4c40, fortran_=0) at multiarraymodule.c:4679 #2 0x00002aaaabacebd5 in array_zeros (ignored=0x2aaaabbef320, args=0x2aaaabbef320, kwds=0x2aaaabad4c40) at multiarraymodule.c:4723 #3 0x0000000000478cfb in PyEval_EvalFrame (f=0x95dd30) at ceval.c:3558 #4 0x0000000000479fb1 in PyEval_EvalFrame (f=0x701c80) at ceval.c:3640 #5 0x0000000000479fb1 in PyEval_EvalFrame (f=0x716970) at ceval.c:3640 #6 0x0000000000479fb1 in PyEval_EvalFrame (f=0x7178c0) at ceval.c:3640 #7 0x0000000000479fb1 in PyEval_EvalFrame (f=0x6e4740) at ceval.c:3640 #8 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab948d50, globals=0x2aaaabad4c40, locals=0x0, args=0x6e4740, argcount=2, kws=0x91bad0, kwcount=0, defs=0x2aaaab950ce8, defcount=1, closure=0x0) at ceval.c:2736 #9 0x00000000004c6099 in function_call (func=0x2aaaab95e758, arg=0x2aaaae026758, kw=0x91b1e0) at funcobject.c:548 #10 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #11 0x00000000004772ea in PyEval_EvalFrame (f=0x6e7bb0) at ceval.c:3835 #12 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab948dc0, globals=0x2aaaabad4c40, locals=0x0, args=0x6e7bb0, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #13 0x00000000004c6099 in function_call (func=0x2aaaab95e7d0, arg=0x2aaaae0265f0, kw=0x0) at funcobject.c:548 #14 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #15 0x0000000000420ee0 in instancemethod_call (func=0x2aaaabbef320, arg=0x2aaaae0265f0, kw=0x0) at classobject.c:2447 #16 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #17 0x00000000004777d9 in PyEval_EvalFrame (f=0x80c070) at ceval.c:3766 #18 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab93a340, globals=0x2aaaabad4c40, locals=0x0, args=0x80c070, argcount=2, kws=0x0, kwcount=0, defs=0x2aaaab9607a8, defcount=1, closure=0x0) at ceval.c:2736 #19 0x00000000004c6099 in function_call (func=0x2aaaab961cf8, arg=0x2aaaae026878, kw=0x0) at funcobject.c:548 #20 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #21 0x0000000000420ee0 in instancemethod_call (func=0x2aaaabbef320, arg=0x2aaaae026878, kw=0x0) at classobject.c:2447 #22 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #23 0x000000000044fd80 in slot_tp_call (self=0x2aaaad97d890, args=0x2aaaae04e450, kwds=0x0) at typeobject.c:4536 #24 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #25 0x00000000004777d9 in PyEval_EvalFrame (f=0x739bd0) at ceval.c:3766 #26 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab94c810, globals=0x2aaaabad4c40, locals=0x0, args=0x739bd0, argcount=2, kws=0x8af160, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #27 0x00000000004c6099 in function_call (func=0x2aaaab95f050, arg=0x2aaaae029248, kw=0x91afa0) at funcobject.c:548 #28 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #29 0x00000000004772ea in PyEval_EvalFrame (f=0x7301d0) at ceval.c:3835 #30 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab94c880, globals=0x2aaaabad4c40, locals=0x0, args=0x7301d0, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #31 0x00000000004c6099 in function_call (func=0x2aaaab95f0c8, arg=0x2aaaae026d40, kw=0x0) at funcobject.c:548 #32 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #33 0x0000000000420ee0 in instancemethod_call (func=0x2aaaabbef320, arg=0x2aaaae026d40, kw=0x0) at classobject.c:2447 #34 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #35 0x000000000044fd80 in slot_tp_call (self=0x2aaaaca7acd0, args=0x2aaaae022c50, kwds=0x0) at typeobject.c:4536 #36 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #37 0x00000000004777d9 in PyEval_EvalFrame (f=0x7760a0) at ceval.c:3766 #38 0x0000000000479fb1 in PyEval_EvalFrame (f=0x6ec260) at ceval.c:3640 #39 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab93adc0, globals=0x2aaaabad4c40, locals=0x0, args=0x6ec260, argcount=3, kws=0x6d26d0, kwcount=0, defs=0x2aaaab963578, defcount=2, closure=0x0) at ceval.c:2736 #40 0x00000000004788f7 in PyEval_EvalFrame (f=0x6d2520) at ceval.c:3650 #41 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaaab23dc0, globals=0x2aaaabad4c40, locals=0x0, args=0x6d2520, argcount=2, kws=0x66c900, kwcount=0, defs=0x2aaaaca84e30, defcount=2, closure=0x0) at ceval.c:2736 #42 0x00000000004788f7 in PyEval_EvalFrame (f=0x66c760) at ceval.c:3650 ---Type to continue, or q to quit--- #43 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab948030, globals=0x2aaaabad4c40, locals=0x0, args=0x66c760, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #44 0x000000000047af72 in PyEval_EvalCode (co=0x2aaaabbef320, globals=0x2aaaabad4c40, locals=0x0) at ceval.c:484 #45 0x00000000004a1c72 in PyRun_InteractiveOneFlags (fp=0x2aaaaab132f0, filename=0x4cbf24 "", flags=0x7fffff82fcfc) at pythonrun.c:1265 #46 0x00000000004a1e04 in PyRun_InteractiveLoopFlags (fp=0x2aaaab556b00, filename=0x4cbf24 "", flags=0x7fffff82fcfc) at pythonrun.c:695 #47 0x00000000004a2350 in PyRun_AnyFileExFlags (fp=0x2aaaab556b00, filename=0x2aaaabad4c40 "number of dimensions must be >=0", closeit=0, flags=0x7fffff82fcfc) at pythonrun.c:658 #48 0x0000000000410788 in Py_Main (argc=0, argv=0x7fffff832721) at main.c:484 #49 0x00002aaaab34d5aa in __libc_start_main () from /lib64/tls/libc.so.6 #50 0x000000000040fdfa in _start () at start.S:113 #51 0x00007fffff82fdf8 in ?? () #52 0x00002aaaaabc19c0 in rtld_errno () from /lib64/ld-linux-x86-64.so.2 #53 0x0000000000000001 in ?? () From charlesr.harris at gmail.com Tue Mar 28 22:01:03 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 22:01:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: On 3/28/06, Zachary Pincus wrote: > > Let me give a more concrete example, pulled from some code I was > actually using. > > The code is obviously a bit flawed, but it worked fine with numpy up > until today. Now it breaks when a and b aren't both c-strided: > def distance_squared(a, b): > return numpy.sum( (a.ravel() - b.ravel())**2 ) def distance_squared(a,b) return ((a - b)**2).sum() The function should probably throw an error if a and b don't have the same shape and order and now it does. I note that your original function was written for the old functional sum and attempts to be efficient. The new Numpy sum method make it much easier to do what you want. Thanks, Travis. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Mar 28 22:03:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 22:03:02 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <7F869A35-1EC7-41F5-BA60-FBF65F6937D5@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> <7F869A35-1EC7-41F5-BA60-FBF65F6937D5@stanford.edu> Message-ID: <442A22F0.7080506@ieee.org> Zachary Pincus wrote: >>> Unfortunately, the ravel/reshape methods seem to not be quite worked >>> out properly yet. Basically (and as expected) the ravel method >>> doesn't work anymore on fortran-strided arrays. But since >>> fortran-strided arrays happen *all-the-time* (e.g. after a transpose >>> operation), things get confusing fast. >>> > > > One question: was the issue in my example code where fortran-strided > arrays couldn't be reshaped at all a feature or a bug in the reshape > method? It's now a feature. Reshape will only work if it can produce a view of the array. Actually, I can't claim that I've thoroughly analyzed every possibility to determine if a view is possible. I'm sure there are cases where a view is technically feasible. The simple rules are: 1) If you are only adding or deleting ones from the shape, then it always succeeds. 2) If you aren't actually changing the shape it always succeeds (a -1 present means you are changing the shape) Other than that it only succeeds if you are reshaping a C-contiguous array using C-ordering or you are reshaping a Fortran-contiguous array using Fortran-ordering. -Travis From robert.kern at gmail.com Tue Mar 28 22:11:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 22:11:01 2006 Subject: [Numpy-discussion] Re: reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: Charles R Harris wrote: > > > On 3/28/06, *Zachary Pincus* > wrote: > > Let me give a more concrete example, pulled from some code I was > actually using. > > The code is obviously a bit flawed, but it worked fine with numpy up > until today. Now it breaks when a and b aren't both c-strided: > def distance_squared(a, b): > return numpy.sum( (a.ravel() - b.ravel())**2 ) > > def distance_squared(a,b) > return ((a - b)**2).sum() > > The function should probably throw an error if a and b don't have the > same shape and order and now it does. Does it? In [55]: def distance_squared(a, b): ....: return ((a - b) ** 2).sum() ....: In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) Out[56]: 55 -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at cox.net Tue Mar 28 22:27:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 22:27:05 2006 Subject: [Numpy-discussion] Re: reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: <442A2880.90409@cox.net> Robert Kern wrote: >Charles R Harris wrote: > > >>On 3/28/06, *Zachary Pincus* >> wrote: >> >> Let me give a more concrete example, pulled from some code I was >> actually using. >> >> The code is obviously a bit flawed, but it worked fine with numpy up >> until today. Now it breaks when a and b aren't both c-strided: >> def distance_squared(a, b): >> return numpy.sum( (a.ravel() - b.ravel())**2 ) >> >>def distance_squared(a,b) >> return ((a - b)**2).sum() >> >>The function should probably throw an error if a and b don't have the >>same shape and order and now it does. >> >> > >Does it? > >In [55]: def distance_squared(a, b): > ....: return ((a - b) ** 2).sum() > ....: > >In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) >Out[56]: 55 > > Right. It will work with any two arrays that can be broadcast to the same shape. Given numpy's rules, that's arguably the right thing to do in general although whether it's the right thing to do here depends on the specific application, which I don't know. Still, this is almost certainly better than the original case, which will blithely compute the distance_squared for any two arrays with the same total size. Then again, it's possible that's what the author really wanted. Unlikely, I think, but possible. Regards, -tim From charlesr.harris at gmail.com Tue Mar 28 22:34:03 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 22:34:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429EEB2.7000803@ieee.org> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> Message-ID: On 3/28/06, Travis Oliphant wrote: > > > rand(10,3).transpose().sum() Shouldn't sum be fixed instead? I admit that a quick look at the code leaves me no more enlightened. It is going to take more than a little time to figure out how numpy is put together. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnd.baecker at web.de Tue Mar 28 22:44:06 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Tue Mar 28 22:44:06 2006 Subject: [Numpy-discussion] 64bit check_manyways segfault. In-Reply-To: <442A26BF.7080008@ieee.org> References: <442A26BF.7080008@ieee.org> Message-ID: On Tue, 28 Mar 2006, Travis Oliphant wrote: > Arnd Baecker wrote: > > Hi, > > > > with current svn I get a segfault for numpy.test(10, 10): > > > > check_manyways (numpy.lib.tests.test_arraysetops.test_aso)Segmentation > > fault > > > Thanks for the test. Nobody had caught that one yet. > > Should be fixed now in SVN. That was quick, indeed! Works fine now. So presently the only problems on 64 Bit are the ndimage ones for scipy - I still have found no time (and won't during the next two weeks at least) to look into that in a bit more detail - sorry... Best, Arnd From michael.sorich at gmail.com Tue Mar 28 22:47:04 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Tue Mar 28 22:47:04 2006 Subject: [Numpy-discussion] stringscalar comparisons with a chararray Message-ID: <16761e100603282240h51c9774ax8e51eafe9fc02058@mail.gmail.com> As shown in the example below, chararray comparisons do not seem to work well when stringscalars are used. In the example below I am also curious as to whether the the length of c[0] should be 2 instead of 5. numpy.__version__ 0.9.6 >>> from numpy import * >>> c =char.array(['on','two','three']) >>> c chararray([on, two, three], dtype='|S5') >>> c=='on' array([True, False, False], dtype=bool) >>> c==c[0] array([False, False, False], dtype=bool) >>> c==str(c[0]) array([True, False, False], dtype=bool) >>> c[0] 'on' >>> len(c[0]) 5 >>> c.dtype dtype('|S5') >>> c==c[2] array([False, False, True], dtype=bool) -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Mar 28 22:49:05 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 22:49:05 2006 Subject: [Numpy-discussion] Re: reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: Robert, On 3/28/06, Robert Kern wrote: > > Charles R Harris wrote: > > > > > > On 3/28/06, *Zachary Pincus* > > wrote: > > > > Let me give a more concrete example, pulled from some code I was > > actually using. > > > > The code is obviously a bit flawed, but it worked fine with numpy up > > until today. Now it breaks when a and b aren't both c-strided: > > def distance_squared(a, b): > > return numpy.sum( (a.ravel() - b.ravel())**2 ) > > > > def distance_squared(a,b) > > return ((a - b)**2).sum() > > > > The function should probably throw an error if a and b don't have the > > same shape and order and now it does. > > Does it? > > In [55]: def distance_squared(a, b): > ....: return ((a - b) ** 2).sum() > ....: > > In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) > Out[56]: 55 Oops! Guess there should be a test for shape equality in there somewhere. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Wed Mar 29 00:01:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Mar 29 00:01:08 2006 Subject: [Numpy-discussion] getting data pointer to numpy object in Pyrex In-Reply-To: <4429B89E.30100@bryant.edu> References: <4429B89E.30100@bryant.edu> Message-ID: <200603291000.10625.faltet@carabos.com> A Dimecres 29 Mar? 2006 00:28, Brian Blais va escriure: > ...but I would like to do something like... > > #------------------------------- > def test2(d): # pass a python dict > > w=d['w'] # get the numpy object > > cdef double *w_p > > # get the pointers > w_p=w.data > #------------------------------- > > > Is there a way to do this? The next works for any numerical package that supports the array protocol: data_address = int(w.__array_data__[0], 16) w_p = PyInt_AsLong(data_address) and declare PyInt_AsLong as: cdef extern from "Python.h": long PyInt_AsLong(object) It is important to note that the above solution does not require to compile against the numpy (or any other numerical package) headers at all. See "A Pyrex Agnostic Class" recipe in the SciPy cookbook (http://www.scipy.org/Cookbook) for a more complete example. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From georgi at molgen.mpg.de Wed Mar 29 01:51:17 2006 From: georgi at molgen.mpg.de (Benjamin Georgi) Date: Wed Mar 29 01:51:17 2006 Subject: [Numpy-discussion] verbosity control for warnings Message-ID: Hi, my application frequently requires to take logarithms of zero, i.e. >>> import numarray >>> numarray.log(0.0) Warning: Encountered divide by zero(s) in log -inf Now, while the result of -inf works just fine, any direct application of numarray.log produces tons of warnings. Right now I'm masking the input arrays to avoid the warnings, which is rather inefficient. So, the question is whether there is some kind of verbosity control I could use to deactivate these warnings. If there is no such functionality, I would like to suggest looking into the build-in Python logging package for that purpose. Best, Benjamin From bblais at bryant.edu Wed Mar 29 05:44:29 2006 From: bblais at bryant.edu (Brian Blais) Date: Wed Mar 29 05:44:29 2006 Subject: [Numpy-discussion] Re: getting data pointer to numpy object in Pyrex In-Reply-To: References: <4429B89E.30100@bryant.edu> Message-ID: <442A8E61.7090605@bryant.edu> Robert Kern wrote: > Brian Blais wrote: >> Is here an easy way, in Pyrex, to get the data pointer to a numpy object >> which is *not* passed as a parameter? For example, I know I can do: >> > You will probably want to use the c_numpy.pxd file in numpy/doc/pyrex/. It has > exposed a bit more of the API than you have. If you have any contributions, we > would like to fold them in. > > In general, you would just make a cast like you would for any other C type. > However, you must be sure that the object actually is an array. I think the > following code would work as expected, including the case where d['w'] can't be > turned into an array, but I have not tested it. > > > > cimport c_numpy > import numpy > > c_numpy.import_array() > > def test2(d): > cdef c_numpy.ndarray w > > w = numpy.asarray(d['w']) > # ... > > > I tried this, and got some warnings, and the code didn't work. I get the compile-time warning of: warning: use of cast expressions as lvalues is deprecated (along with all of the other, normal, Pyrex compile warnings) I am including the code here: #test3.pyx cimport c_numpy import numpy c_numpy.import_array() cdef extern from "numpy/arrayobject.h": struct PyArray_Descr: int type_num, elsize char type ctypedef class numpy.ArrayType [object PyArrayObject]: cdef char *data cdef int nd cdef int *dimensions, *strides cdef object base cdef PyArray_Descr *descr cdef int flags def fun1(d): cdef ArrayType W cdef double *w cdef int i,l W=d['w'] w=W.data l=W.dimensions[0]*W.dimensions[1] for i from 0 <= i < l: print "%.4e" % w[i] def fun2(d): cdef c_numpy.ndarray W cdef double *w cdef int i,l W = numpy.asarray(d['w']) w=W.data l=W.dimensions[0]*W.dimensions[1] for i from 0 <= i < l: print "%.4e" % w[i] #================================================= #run_test3.py import numpy import test3 w=numpy.arange(0,2,.5) d={"w":w} test3.fun1(d) print "===================" test3.fun2(d) #=================================================== #output from program: >>> execfile('run_test3.py') 0.0000e+00 5.0000e-01 1.0000e+00 1.5000e+00 8.3394e-312 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 1.9354e+01 5.3216e-315 0.0000e+00 2.1127e+01 5.3428e-315 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 -5.1923e+00 2.5548e+02 =================== 0.0000e+00 5.0000e-01 1.0000e+00 1.5000e+00 8.3394e-312 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 1.9354e+01 5.3216e-315 0.0000e+00 2.1127e+01 5.3428e-315 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 -5.1923e+00 2.5548e+02 -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From perry at stsci.edu Wed Mar 29 06:01:10 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Mar 29 06:01:10 2006 Subject: [Numpy-discussion] verbosity control for warnings In-Reply-To: References: Message-ID: <00d31387aa3cf09970ef1d4517dcb52d@stsci.edu> On Mar 29, 2006, at 4:50 AM, Benjamin Georgi wrote: > > Hi, > > my application frequently requires to take logarithms of zero, i.e. > >>>> import numarray >>>> numarray.log(0.0) > Warning: Encountered divide by zero(s) in log > -inf > > Now, while the result of -inf works just fine, any direct application > of numarray.log produces tons of warnings. Right now I'm masking the > input arrays to avoid the warnings, which is rather inefficient. > > So, the question is whether there is some kind of verbosity control I > could use to deactivate these warnings. If there is no such > functionality, I would like to suggest looking into the build-in > Python logging package for that purpose. > Yes, look at section 6.2 of the numarray manual. Perry Greenfield From benjamin at decideur.info Wed Mar 29 06:20:23 2006 From: benjamin at decideur.info (Benjamin Thyreau) Date: Wed Mar 29 06:20:23 2006 Subject: [Numpy-discussion] distutils.misc_util Configuration error Message-ID: <200603291418.k2TEILOG018907@decideur.info> Hi, when trying to create an distutils.misc_util.Configuration instance, since numpy 0.9.5, i got an Python 2.4.1 (#1, May 2 2005, 15:06:50) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy.distutils.misc_util >>> numpy.distutils.misc_util.Configuration() Traceback (most recent call last): File "", line 1, in ? File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 450, in __init__ f = get_frame(i) File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 359, in get_frame return sys._getframe(level+1) ValueError: call stack is not deep enough Error. Strangely enough (or not), when running from ipython, it seems to work. On jan 31th 2006, pearu commited: for i in range(1,3): f = get_frame(i) try: caller_instance = eval('self',f.f_globals,f.f_locals) break except NameError: caller_instance = None if isinstance(caller_instance, self.__class__): if caller_instance.options['delegate_options_to_subpackages']: self.set_options(**caller_instance.options) on misc_util.py, at the end of the Configuration constructor. I don't exactly understand what that code do, but it prevent my setup.py to run since then. Thanks for your help -- Benjamin Thyreau decideur.info From pearu at scipy.org Wed Mar 29 06:33:05 2006 From: pearu at scipy.org (Pearu Peterson) Date: Wed Mar 29 06:33:05 2006 Subject: [Numpy-discussion] distutils.misc_util Configuration error In-Reply-To: <200603291418.k2TEILOG018907@decideur.info> References: <200603291418.k2TEILOG018907@decideur.info> Message-ID: On Wed, 29 Mar 2006, Benjamin Thyreau wrote: > Hi, > when trying to create an distutils.misc_util.Configuration instance, since > numpy 0.9.5, i got an > > Python 2.4.1 (#1, May 2 2005, 15:06:50) > [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy.distutils.misc_util >>>> numpy.distutils.misc_util.Configuration() > Traceback (most recent call last): > File "", line 1, in ? > File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 450, in __init__ > f = get_frame(i) > File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 359, in get_frame > return sys._getframe(level+1) > ValueError: call stack is not deep enough Fixed in numpy svn. Thanks, Pearu From cjw at sympatico.ca Wed Mar 29 06:49:10 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 29 06:49:10 2006 Subject: [Numpy-discussion] round_() problems and arange incompatibility In-Reply-To: <200603282230.32210.pgmdevlist@mailcan.com> References: <200603282230.32210.pgmdevlist@mailcan.com> Message-ID: <442A9E4C.8060403@sympatico.ca> Pierre GM wrote: >> File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table >> outStr += '' + str(N.round_(item,prec)) + '' >> >> > > >Have you tried to use around instead of round ? That way, I think you wouldn't >be using oldnumeric and stick to ndarrays... > > > "round" seems a better word to use than "around". Would it not be better to have some sort of compatibility switch for those who wish to use names with their old meanings, rather than prepending "a" to indicate the new way? Incidentally, particularly for testing, I found the shape parameter in arange to be useful: [Dbg]>>> import numarray as _na [Dbg]>>> help(_na.arange) Help on function arange in module numarray.numarraycore: arange(a1, a2=None, stride=1, type=None, shape=None, typecode=None, dtype=None) Returns an array of numbers in sequence over the specified range. With numpy, we have: [Dbg]>>> a= _n.arange(6, shape=(3, 2), dtype= _n.ArrayType) Traceback (most recent call last): File "", line 1, in ? TypeError: 'shape' is an invalid keyword argument for this function [Dbg]>>> help(_n.arange) Help on built-in function arange in module numpy.core.multiarray: arange(...) arange([start,] stop[, step,], dtype=None) For integer arguments, just like range() except it returns an array whose type can be specified by the keyword argument dtype. If dtype is not specified, the type of the result is deduced from the type of the arguments. For floating point arguments, the length of the result is ceil((stop - start)/step). This rule may result in the last element of the result be greater than stop. Colin W. From zingale at gmail.com Wed Mar 29 07:02:07 2006 From: zingale at gmail.com (Mike Zingale) Date: Wed Mar 29 07:02:07 2006 Subject: [Numpy-discussion] numarray 1.5.1 self test failure Message-ID: <885402e30603290700s2b85e98aucc16eb5f9034111@mail.gmail.com> Hi, I just installed numarray 1.5.1 on my Fedora Core 4 box (python version 2.4.1). Right afterwards, I did: import numarray.testall as testall testall.test() at the python prompt, and I got 1 failure. The full output is below. Is this a known failure? My machine is a dual Xeon, and I am running the 32bit version of Fedora. Mike >>> import numarray.testall as testall >>> testall.test() Testing numarray 1.5.1 on normal Python (2, 4, 1, 'final', 0) on platform linux2 with builtin linear algebra support numarray.numtest: 2.41 ((0, 1208), (0, 1208)) numarray.ieeespecial: 0.06 (0, 86) numarray.records: 0.05 (0, 48) numarray.strings: 0.16 (0, 189) numarray.memmap: 0.07 (0, 82) numarray.objects: 0.12 (0, 105) numarray.memorytest: 0.01 (0, 16) numarray.examples.convolve: 0.09 ((0, 20), (0, 20), (0, 20), (0, 20)) numarray.convolve: 0.07 (0, 45) numarray.fft: 0.13 (0, 75) numarray.linear_algebra: 1.55 ((0, 46), (0, 51)) numarray.image: 0.05 (0, 37) numarray.nd_image: 2.76 (0, 397) numarray.random_array: 0.10 (0, 53) numarray.ma: 1.42 (0, 674) numarray.matrix: 0.03 (0, 11) ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 32, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na, copy=1) num2 = Numeric.array([1,2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 49, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na, copy=0) num2 = Numeric.array([1,2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 66, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[::2], copy=0) num2 = Numeric.array([1,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 83, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[::2], copy=1) num2 = Numeric.array([1,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 100, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[1:], copy=1) num2 = Numeric.array([2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 117, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[1:], copy=0) num2 = Numeric.array([2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 208, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num, copy=1) nb = numarray.array([1,2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 i [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 i [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 226, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num, copy=0) nb = numarray.array([1,2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 i [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 i [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 244, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[::2], copy=1) nb = numarray.array([1,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 262, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[::2], copy=0) nb = numarray.array([1,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 280, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[1:], copy=1) nb = numarray.array([2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 298, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[1:], copy=0) nb = numarray.array([2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** 1 items had failures: 12 of 18 in numarray.array_protocol.test_Numeric ***Test Failed*** 12 failures. numarray.array_protocol: 0.09 (12, 18) numarray.dtype: 0.02 (0, 15) Total time: 9.27 > From jmiller at stsci.edu Wed Mar 29 07:11:10 2006 From: jmiller at stsci.edu (Todd Miller) Date: Wed Mar 29 07:11:10 2006 Subject: [Numpy-discussion] numarray 1.5.1 self test failure In-Reply-To: <885402e30603290700s2b85e98aucc16eb5f9034111@mail.gmail.com> References: <885402e30603290700s2b85e98aucc16eb5f9034111@mail.gmail.com> Message-ID: <442AA36D.3060505@stsci.edu> Hi Mike, The portion of the numarray self-test that is failing requires Numeric-24.2 to succeed because it is testing the new "array interface" which did not exist until 24.2. Without 24.2, numarray sees Numeric arrays as more general Python sequences and therefore casts to the default array type for that Python scalar kind. This results in type mismatches. Regards, Todd Mike Zingale wrote: > Hi, I just installed numarray 1.5.1 on my Fedora Core 4 box (python > version 2.4.1). Right afterwards, I did: > > import numarray.testall as testall > testall.test() > > at the python prompt, and I got 1 failure. The full output is below. > > Is this a known failure? My machine is a dual Xeon, and I am running > the 32bit version of Fedora. > > Mike > > > >>>> import numarray.testall as testall >>>> testall.test() >>>> > Testing numarray 1.5.1 on normal Python (2, 4, 1, 'final', 0) on > platform linux2 with builtin linear algebra support > numarray.numtest: 2.41 ((0, 1208), (0, 1208)) > numarray.ieeespecial: 0.06 (0, 86) > numarray.records: 0.05 (0, 48) > numarray.strings: 0.16 (0, 189) > numarray.memmap: 0.07 (0, 82) > numarray.objects: 0.12 (0, 105) > numarray.memorytest: 0.01 (0, 16) > numarray.examples.convolve: 0.09 ((0, 20), (0, 20), > (0, 20), (0, 20)) > numarray.convolve: 0.07 (0, 45) > numarray.fft: 0.13 (0, 75) > numarray.linear_algebra: 1.55 ((0, 46), (0, 51)) > numarray.image: 0.05 (0, 37) > numarray.nd_image: 2.76 (0, 397) > numarray.random_array: 0.10 (0, 53) > numarray.ma: 1.42 (0, 674) > numarray.matrix: 0.03 (0, 11) > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 32, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na, copy=1) > num2 = Numeric.array([1,2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 49, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na, copy=0) > num2 = Numeric.array([1,2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 66, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[::2], copy=0) > num2 = Numeric.array([1,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 83, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[::2], copy=1) > num2 = Numeric.array([1,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 100, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[1:], copy=1) > num2 = Numeric.array([2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 117, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[1:], copy=0) > num2 = Numeric.array([2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 208, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num, copy=1) > nb = numarray.array([1,2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > i [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > i [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 226, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num, copy=0) > nb = numarray.array([1,2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > i [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > i [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 244, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[::2], copy=1) > nb = numarray.array([1,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 262, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[::2], copy=0) > nb = numarray.array([1,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 280, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[1:], copy=1) > nb = numarray.array([2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 298, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[1:], copy=0) > nb = numarray.array([2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > 1 items had failures: > 12 of 18 in numarray.array_protocol.test_Numeric > ***Test Failed*** 12 failures. > numarray.array_protocol: 0.09 (12, 18) > numarray.dtype: 0.02 (0, 15) > Total time: 9.27 > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Mar 29 10:11:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 29 10:11:07 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <442A2572.9000800@cox.net> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> <442A2572.9000800@cox.net> Message-ID: <442ACD95.60101@ieee.org> Tim Hochberg wrote: > > Hi Travis, > > Ouch! > > In a sense, the problems that Zachary is having illustrates both of > our points perfectly: he's now put in a position of going through a > bunch of needless pain. On the other hand, the implicit copying has > resulted in a bunch of code that does a bunch of uneeded copies, > probably without the original author knowing it, resulting in both > speed an memory inefficiency. Yeah. I really see your point. I'm just uncomfortable breaking people's code too much. A sub-class is a good idea. It's a good place to experiment with new ideas. It's one of the reasons we wanted the array to be sub-classable. Making a sub-class a super-class is also possible once it's working. Best, -Travis From oliphant.travis at ieee.org Wed Mar 29 10:12:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 29 10:12:11 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> Message-ID: <442ACDD1.7060501@ieee.org> Charles R Harris wrote: > > > On 3/28/06, *Travis Oliphant* > wrote: > > > rand(10,3).transpose().sum() > > > Shouldn't sum be fixed instead? I admit that a quick look at the code > leaves me no more enlightened. It is going to take more than a little > time to figure out how numpy is put together. > FWIW: I also fixed the _check_axis code (it was ignoring errors and shouldn't have been). -Travis From Chris.Barker at noaa.gov Wed Mar 29 12:11:12 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed Mar 29 12:11:12 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429EEB2.7000803@ieee.org> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> Message-ID: <442AE9AA.2090007@noaa.gov> >> Since fortran-strided >> arrays happen *all-the-time* (e.g. after a transpose operation), I have to wonder if it's really a good idea to do this. Why not just make a re-ordered copy on transpose, rather than making it Fortran strided? Is the optimization really that important? I had always thought of Fortran strided arrays as being a special case exception (for working with external Fortran codes), rather than something that happens all the time. I suppose if all these issues are worked out, it'd be transparent to me, but it strikes me as more complication that necessary. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Wed Mar 29 13:14:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 29 13:14:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <442AE9AA.2090007@noaa.gov> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> <442AE9AA.2090007@noaa.gov> Message-ID: <442AF853.7050509@ee.byu.edu> Christopher Barker wrote: >>> Since fortran-strided arrays happen *all-the-time* (e.g. after a >>> transpose operation), >> > > I have to wonder if it's really a good idea to do this. Why not just > make a re-ordered copy on transpose, rather than making it Fortran > strided? Is the optimization really that important? And optimization is important. The _dotblas.c function for example now works much faster with very common transpose operations because the underlying blas knows how to deal with striding so it was a complete wasted to always use C-ordering and get a copy of a Fortran-strided array. On large arrays, avoiding the unnecessary copy can be very significant. -Travis From david.huard at gmail.com Wed Mar 29 14:21:21 2006 From: david.huard at gmail.com (David Huard) Date: Wed Mar 29 14:21:21 2006 Subject: [Numpy-discussion] Freezing numpy code Message-ID: <91cf711d0603291420g7068c84y623b43d36a83c850@mail.gmail.com> Hi, I would like to freeze some code in order to run it on machines without installing all the modules I use. I compiled Python 2.4.3 from source and rebuilt numpy. When I try to freeze the following code : #test.py from numpy import mean, rand x = rand(1000) print mean(x) with python /usr/share/doc/python2.4/examples/Tools/freeze/freeze.py test.py make I get the following error message : No scipy-style subpackage 'testing' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'core' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'lib' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'linalg' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'dft' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'random' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'f2py' found in n:u:m:p:y. Ignoring. Traceback (most recent call last): File "test.py", line 1, in ? from numpy import mean, rand File "/usr/lib/python2.4/site-packages/numpy/__init__.py", line 44, in ? __doc__ += pkgload.get_pkgdocs() File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 320, in get_pkgdocs retstr = self._format_titles(titles) +\ File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 283, in _format_titles max_length = max(lengths) ValueError: max() arg is an empty sequence Does anybody know what I'm missing ? I'm using the latest svn. Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From zpincus at stanford.edu Wed Mar 29 16:04:05 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed Mar 29 16:04:05 2006 Subject: [Numpy-discussion] numpy.linalg: eig versus eigh? Message-ID: <5CFF089A-5B17-4B4F-83BA-2B75CC2E7BDF@stanford.edu> Can anyone tell me what the difference between numpy.linalg.eig and numpy.linalg.eigh is? (Ditto for eigvals and eigvalsh.) Thanks, Zach From aeronautic at hrvatskamail.com Wed Mar 29 19:22:07 2006 From: aeronautic at hrvatskamail.com (=?windows-1251?B?NS02IODv8OXr/w==?=) Date: Wed Mar 29 19:22:07 2006 Subject: [Numpy-discussion] =?windows-1251?B?ysDE0M7CzsUgxMXLzs/QzsjHws7E0dLCzg==?= Message-ID: <8de501c652a1$0761d935$c92b9c86@hrvatskamail.com> ?????????? ??? ? ????? ?????? ??????? ??????? ? ???????????? ?????????: 5-6 ?????? ???????? ???????????????? ? ??????????? ???????????:???????????? ? ?????? ? ????. 6 ?????? - ??????????? ? ????????????? ? ?? ??????????? ??????? ???? . ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. 232-2133 ???????? ???????????????? ? ??????????? ???????????:???????????? ? ?????? ? ????. 5-6 ?????? 2006 ???? 1. ?????? ???????????? ??????????? ???????? ?????? ???????? ?????? ?????????????. ????? ?? ???????? ???? ??????? 2. ???????? ????????????????, ??? ??????????? ??????????. ?????????, ???????????? ??? ??????? ? ???????????? ?????????? ?? ??????????????-???????? ?????: ?????? ???????? ???????????? ??????????. ???? ????????? ?? ???????, ??? ??? ????? ???????? ??? ????? ????????? ???? ??????????? ??????????????? ????????. ??? ????? ???????? ?? ?????????? ?????????? 3. ??????????? ???????????????? ? ????. ?????? ???????? ?????????? ??????? ????????? ???????????????? ? ????????. ?????????? ??????????????? ???????: ???????????? ???????? ????????. 4. ?????????? ?????????? ????????? ?????, ???????????? ?????? ????????. ??????? ??????????? ????????? ??????????, ??????????? ??????????, ????????? ? ?????? ???????????? ??????, ?????????? ?? ?????? ????? ? ??. ???????????? ?????????? ? ??????????? ?????????? ???????????? ?????? 5. ??? ????????? ??????? ?????? ? ???????? ???????????? ????? ???????, ????? ??? ??? ??????????? ???????? ?????????? ? ?????? ???? ????????????? ???????? ???????. ??????? ???????. ??????? ???????. ???????????????? ????????????? ?????????????? ?????. 6. ???????? ?????????? ???????? ?????????? ??? ???????? ?????????. ?????????? ?????????? ??? ??????, ????????, ???????, ??????????. ?????????? ?????????????? ???????????? ? ??. ????????????? ??????? ??????????????? ????. ???????????????? ?????????????? ????????? ?????????. 7. ??????????? ?????????? ??????. ???? ??????????? ??? ????? ??????? ?????????????, ? ????????? ?? ??????, ??? ???? ???????? ?????-?? ?????????, ?? ???, ????? ?????? ?????? ???????????? 8. ?????????????? ???????????????? ? ?????? ?? ?????????? ??????? ??????????. ????? ?????????? ???????? ? 10.00 ?? 17.00. ? ????????? ??????: ????, ???????????? ?????????, + 3 ??? ???????????????? ?????????. ????????? 9500 ???. + ???. ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. 232-2133 ?????????? ? ?????????????? ?? ??????????? ??????? ????6 ?????? 2006 ???? 1. ????? ???????????????? ? ????? ??????????? ??????????? ??????? ????. 2. ??????????????? ???????????? ?? ????????? ???????????? ????????????????. ???????????????? ??????????????? ??????????? ???????????????? ??????????????? ??????????? ??? ???????????????? ??????????????? ?????????? ??? ?????? ??????????? ????????? 3. ???????? ????????? ??????????? ?????????? ? ?? ??????? ??????-?????? ??????????? ???? ? ?????????? ??????????????? ??????????? ????????? ???????????????? ?? ??????????? (????? ???????? ???????, ?????? ?????, ????????????, ?????? ?????, ?????????? ???????????, ???????????????) ??????????? ??????????? ?? ?????? ??????? ??? ????? ?? ?????? ??????? ?????? - ?? ????????? ????????????? ??????????-???????? ????????? ? ??????????? ?????? ? ?????. 4. ??????????? ?????????????, ???????????? ???? ??????????? ????????? ?????????? ?? ??????????? ??????????? ??????? ???? ?????????? ????????????? ?? ????? ?????? ?????????, ??????? ?? ????????? ?????????? ?? ??????????, ?? ????????????? ???????? ???????????? ????????????? ???????????????? ?.?????? - ???????? ????????? 5. ?????????????? ?????? ? ?? ??????????? ??????????????? ????? ??????????? ???????????? ?????? ??? ?.?????? ???? ?.?????? ?????????? ?? ????? ???????? ??????????????? ????????? ????? ? ?.?????? 6. ?????????????? ???????????? ????????? ???????: 5800 ???. + ??? ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. 232-2133 -------------- next part -------------- An HTML attachment was scrubbed... URL: From luszczek at cs.utk.edu Wed Mar 29 19:26:03 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Wed Mar 29 19:26:03 2006 Subject: [Numpy-discussion] numpy.linalg: eig versus eigh? In-Reply-To: <5CFF089A-5B17-4B4F-83BA-2B75CC2E7BDF@stanford.edu> References: <5CFF089A-5B17-4B4F-83BA-2B75CC2E7BDF@stanford.edu> Message-ID: <200603292224.09131.luszczek@cs.utk.edu> On Wednesday 29 March 2006 19:03, Zachary Pincus wrote: > Can anyone tell me what the difference between numpy.linalg.eig and > numpy.linalg.eigh is? (Ditto for eigvals and eigvalsh.) > > Thanks, > > Zach eig() is for nonsymmetric matrices and eigh() is for symmetric (or hermitian matrices). The former most likely will return complex eigen values. The latter always returns real eigen values. eig() is slower and numerically less reliable (it calls LAPACK's DGEEV and ZGEEV). Even for symmetric matrices it can produce complex eigen pairs due to numerical round-off. eigh() can be called with an optional parameter UPLO that tells the routine which triangle of the matrix it should operate on. That triangle is destroyed and the other triangle is left untouched. Piotr From haase at msg.ucsf.edu Wed Mar 29 20:54:01 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Mar 29 20:54:01 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <200603220945.18706.haase@msg.ucsf.edu> References: <200603211514.22249.haase@msg.ucsf.edu> <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> <200603220945.18706.haase@msg.ucsf.edu> Message-ID: <442B6460.5030909@msg.ucsf.edu> Hi, Please consider this a bug report (I have never filed a bug with SF...) I just don't want this falling through the cracks... Again: na.maximum.reduce(arr,-3) returned the transpose of what it should have given ( na.maximum.reduce(arr,1) in my case - see below for details ) Thanks, Sebastian haase Sebastian Haase wrote: > On Wednesday 22 March 2006 05:32, Paul Barrett wrote: >> On 3/21/06, Sebastian Haase wrote: > >> The above behavior for maximum.reduce looks consistent to me. >> >> The reduce axis for examples 1 and 2 above is 0, so maximum is comparing >> arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. Example >> 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving [[2 3] [6 >> 7]]. And the last example is comparing arrays [[0 2] [4 6]] and [[1 3] [5 >> 7]], giving [[1 3] [5 7]]. >> >> I think it depends on how you look at the shape of the array. >> >> -- Paul > Sorry > I did the test wrong - >>>> na.maximum.reduce(q,2) > [[1 3] > [5 7]] >>>> na.maximum.reduce(q,-1) > [[1 3] > [5 7]] > So it obviously works as advertised. > > The array where I noticed the transpose behaviour was really of shape > 32,15,521,512 and my command was na.maximum.reduce(arr,-3) which "looked" > tranpose to what I got with na.maximum.reduce(arr,1). > I using numarray for image analysis and the array might have been a memmap-ed > image file. I'm just worried because I found an "accidental transpose bug" > on type conversion a few years back (see mailing list)... > ( I forgot to say that I running Python 2.2 and numarray version '1.4.0' ) > > I just reran the test with python2.4 and arr=arr.copy() [which should give a > non-memmap version of arr] > >>>> arrM = na.maximum.reduce(arr,-3) >>>> arrMt = na.transpose(arrM,(0,2,1)) >>>> arrM1= na.maximum.reduce(arr,1) >>>> U.mmms(arrM1-arrMt) # returns min,max,mean,stddev > (0, 0, 0.0, 0.0) > > So it definitely transposes the last two axes ! Maybe the bug shows up on > rank >=4 !? - which is the case Robert tested. > > Thanks, > Sebastian Haase > From pearu at scipy.org Thu Mar 30 00:38:01 2006 From: pearu at scipy.org (Pearu Peterson) Date: Thu Mar 30 00:38:01 2006 Subject: [Numpy-discussion] Freezing numpy code In-Reply-To: <91cf711d0603291420g7068c84y623b43d36a83c850@mail.gmail.com> References: <91cf711d0603291420g7068c84y623b43d36a83c850@mail.gmail.com> Message-ID: On Wed, 29 Mar 2006, David Huard wrote: > I would like to freeze some code in order to run it on machines without > installing all the modules I use. I compiled Python 2.4.3 from source and > rebuilt numpy. When I try to freeze the following code : > > #test.py > from numpy import mean, rand > x = rand(1000) > print mean(x) > > with > > python /usr/share/doc/python2.4/examples/Tools/freeze/freeze.py test.py > make > > I get the following error message : > > No scipy-style subpackage 'testing' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'core' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'lib' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'linalg' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'dft' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'random' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'f2py' found in n:u:m:p:y. Ignoring. > Traceback (most recent call last): > File "test.py", line 1, in ? > from numpy import mean, rand > File "/usr/lib/python2.4/site-packages/numpy/__init__.py", line 44, in ? > __doc__ += pkgload.get_pkgdocs() > File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 320, > in get_pkgdocs > retstr = self._format_titles(titles) +\ > File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 283, > in _format_titles > max_length = max(lengths) > ValueError: max() arg is an empty sequence I have fixed these errors in svn. Now I get File "/usr/local/lib/python2.4/site-packages/numpy/core/numerictypes.py", line 85, in ? from multiarray import typeinfo, ndarray, array, empty ImportError: cannot import name typeinfo I checked that given multiarray comes from Numeric. It looks like freeze is not able to freeze extension modules that are not in path. Pearu From vmas at carabos.com Thu Mar 30 03:46:06 2006 From: vmas at carabos.com (Vicent Mas (V+)) Date: Thu Mar 30 03:46:06 2006 Subject: [Numpy-discussion] USING NUMPY FOR AN ENROLLMENT PROJECTION PROBLEM In-Reply-To: <502D9B13021C2D42B05F32C5C3F4C1DD2875FF@MBX01.stf.nus.edu.sg> References: <502D9B13021C2D42B05F32C5C3F4C1DD2875FF@MBX01.stf.nus.edu.sg> Message-ID: <200603301345.39880.vmas@carabos.com> El S?bado 25 Marzo 2006 13:59, Ajith Prasad (OFN) escribi?: > I posted this problem - arising from my work at the local university > - some years back in comp.lang.python and received a solution that > did not require the use of matrix algebra. The solution was: > > def enrollment(year, intake, survivalrate): > return sum([intake[year-i]*rate > for i, rate in enumerate(survivalrate)]) > > > I would welcome help in formulating a solution using NumPy. Thanks in > advance for any suggestion. > > > The Problem: > > > The enrolment E(n) of an institution at the beginning of year n is > the sum of the intake for year n, I(n), and the survivors from the > intakes of previous r years. Thus, if s(1) is the 1-year intake > survival rate, s(2) is the 2-year survival rate, etc, we have: > > > E(n)= I(n)+I(n-1)*s(1)+ I(n-2)*s(2)+...+I(n-r)*s(r) > E(n+1)= I(n+1)+I(n)*s(1)+I(n-1)*s(2)+... +I(n-r-1)*s(r) > . > . > . > E(n+k)= I(n+k)+I(n+k-1)*s(1)+I(n+k-2)*s(2)+...+I(n+k-r)*s(r) > > > Given: > (a) the actual intakes for the current and previous r years, I(n), > I(n-1),I(n-2),..,I(n-r), and the planned intakes for the next n+k > years: I(n+1), I(n+2),..., I(n+k), we have the intake vector I = > (I(n-r), I(n-r-1),...,I(n),I(n+1),..., I(n+k)); and > (b) the survival rate vector, s = (1,s(1), s(2),...,s(r)) > Find: > The k*1 enrolment projection column vector, E = > (E(n+1),E(n+2),...,E(n+k)) in terms of a k*(r+1) matrix P (derived > from I) and the (r+1)*1 column vector, s. > > > I = P*s > > > Is there a compact Python representation of the relevant matrix P > where: > > > P = [I(n+1) I(n) I(n-1).. . I(n-r) > I(n+2) I(n+1) I(n)... I(n-r-1) > . > . > I(n+k) I(n+k-1) I(n+k-2)... I(n+k-r)] > > > If I've understood properly you simply want to multiply two matrices. Just store your intakes in an array, your survival rates in another one and call matrixmultiply (or dot). The following interactive session shows it: In [1]: import numarray In [2]: p_matrix = numarray.array([1,2,3,4,5,10,1,2,3,4,20,10,1,2,3,30,20,10,1,2], ...: shape=(4, 5), type="Int8") In [3]: sr_array = numarray.array([1, 0.2, 0.15, 0.25, 0.3]) In [4]: enrolment_array = numarray.matrixmultiply(p_matrix, sr_array) In [5]: print p_matrix [[ 1 2 3 4 5] [10 1 2 3 4] [20 10 1 2 3] [30 20 10 1 2]] In [6]: print sr_array [ 1. 0.2 0.15 0.25 0.3 ] In [7]: print enrolment_array [ 4.35 12.45 23.55 36.35] Regards -- :: \ / Vicent Mas http://www.carabos.com 0;0 / \ C?rabos Coop. Enjoy Data V V " " -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From Norbert.Nemec.list at gmx.de Thu Mar 30 06:14:17 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu Mar 30 06:14:17 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441A7E2A.9010903@ieee.org> References: <44198B7B.6040706@gmx.de> <441A7E2A.9010903@ieee.org> Message-ID: <442BE77D.7090001@gmx.de> Sorry, Travis, I missed your answer for a week... I confess, my efficiency concerns about the histogram routine are to some extent, of theoretical nature. However, given the widespread use of the function and considering the simplicity of the solution, I think it is still worth thinking about it: The current histogram routine is based on a "sort" of the incoming data. Quicksort is usually O(N log N), worst case O(N**2). Doing the histogram by simply counting is a simple O(N) procedure. The absolute cost of the quicksort algorithm is low, but the simple counting should be even lower, if done efficiently in C. What probably weighs heavier though, is the memory usage: sorting can either be done in place, destroying the original data, or requires a copy. Counting could be done even on an iterator without storing all the data. Lacking a comparable implementation of the counting algorithm, I have not done any actual speed comparisons. Why nobody ever found this inefficiency, I cannot say. Probably nobody ever had a real comparison how fast the histogram could be. Unless you are dealing with really huge data, the difference will not show up. Anyway, as I said, a solution should be simple to code once it is decided how to do it. Simply recoding the histogram routine itself in C would be the minimal solution. Implementing a more flexible counting routine that can be used in other contexts as well would certainly be more desirable, but I have no clear vision what that should look like. Greetings, Norbert Travis Oliphant wrote: > Norbert Nemec wrote: > >> I have a very much related problem: Not only that the idea described by >> Mads Ipsen does not work, but I could generally find no efficient way to >> do a "counting" of elements in an array, as it is needed for a >> histogram. >> > > This may be something we are lacking. > > It depends on what you mean by efficient, I suppose. The sorting > algorithms are very fast, and the histogram routines that are > scattered all over the place (including the histogram function in > numpy) has been in use for a long time, and you are the first person > to complain of its efficiency. That isn't to say your complaint may > not be valid, it's just that for most people the speed has been > sufficient. > >> What would instead be needed is a function that simply gives the count >> of occurances of given values in a given array: >> > > I presume you are talking of "integer" arrays, since > equality-comparison of floating-point values is usually not very > helpful so most histograms on floating-point values are given in terms > of bins. Thus, the searchsorted function uses bins for it's > "counting" operation. > >> >> >>>>> [4,5,2,3,2,1,4].count([0,1,2,3,4,5]) >>>>> >>>> >> [0,1,2,1,1,2] >> >> > > > A count function for integer arrays could certainly be written using a > C-loop. But, I would first just use histogram([4,5,2,3,2,1,4], > [0,1,2,3,4,5]) and make sure that's really too slow, before worrying > about it too much. > > Also, I according to the above function, the right answer is: > > [0, 1, 2, 1, 2, 1] > > > Best, > > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From tim.hochberg at cox.net Thu Mar 30 08:53:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 30 08:53:05 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <442BE77D.7090001@gmx.de> References: <44198B7B.6040706@gmx.de> <441A7E2A.9010903@ieee.org> <442BE77D.7090001@gmx.de> Message-ID: <442C0CD6.3030105@cox.net> Norbert Nemec wrote: >Sorry, Travis, I missed your answer for a week... > >I confess, my efficiency concerns about the histogram routine are to >some extent, of theoretical nature. However, given the widespread use of >the function and considering the simplicity of the solution, I think it >is still worth thinking about it: > >The current histogram routine is based on a "sort" of the incoming data. >Quicksort is usually O(N log N), worst case O(N**2). Doing the histogram >by simply counting is a simple O(N) procedure. The absolute cost of the >quicksort algorithm is low, but the simple counting should be even >lower, if done efficiently in C. > > I'm not so sure about this analysis: histogram allows aribtrarily spaced bins, so I think the actual comparison is O(N log N) versus O(N log M) where M is the number of bins. The number of bins will typically be much lower than N it's true, but the log will tend to wash that out some. Take, for example, 1024 items into 32 bins. Then log N is 10 versus log M being 5. This difference is small enough that probably the differences in constant factors will dominate. >What probably weighs heavier though, is the memory usage: sorting can >either be done in place, destroying the original data, or requires a >copy. > I wouldn't worry about the memory usage itself so much. We all have oodles of RAM and virtual memory these days anyway. In addition, you'll probably end up copying discontiguous data anyway, which one ends up with rather frequently. However, the counting algorithm should be more cache friendly than the sort based on. I find that operations on matrices that don't fit in the cache are dominated by memory access time, not CPU time. On my machine that happens somewhere between 10k and 100k elements. For arrays with 100k or so elements, the counting algorithm might be a big win. >Counting could be done even on an iterator without storing all the >data. > > This is true. However, if you go through the iterator interface, it will be much slower than directly addressing the array memory. Such a thing might be cool to have somewhere (itertools or something), but probably isn't appropriate for numpy. >Lacking a comparable implementation of the counting algorithm, I have >not done any actual speed comparisons. Why nobody ever found this >inefficiency, I cannot say. Probably nobody ever had a real comparison >how fast the histogram could be. Unless you are dealing with really huge >data, the difference will not show up. > > This is the crux of the issue I think. The implementaion of histogram looks fine for what it does. For reasonably large arrays, the overhead of it being implemented in Python will be negligible. Without someone who actually needs something different, a new implementation is liable to wander off into the realm of pointless microoptimization. So, my advice is to leave it alone till it actually causes you problems. >Anyway, as I said, a solution should be simple to code once it is >decided how to do it. Simply recoding the histogram routine itself in C >would be the minimal solution. Implementing a more flexible counting >routine that can be used in other contexts as well would certainly be >more desirable, but I have no clear vision what that should look like. > > Well if you were going to do this. I would code a drop in replacement for the current histogram. No iterators (they belong in a different library). Variable spaces bins allowed, etc. It doesn't sound like we need a different histogram function and the more overlapping functions that there are, the less each individual one gets used and tested and the more likely that bugs and bitrot set in. What I'd acutally do is implent something that behaved like this: def histogram_core(data, bins): # I hope I grabbed the right stuff out histogram_core # bins is an actual list of bins edges as per histogram # which could really us a docstring n = sort(a).searchsorted(bins) n = concatenate([n, [len(a)]]) n = n[1:]-n[:-1] return n Except written in C and using a counting algorithm. Then the histogram can be just a thin wrapper over histogram_core. Then test it with a bunch of different array sizes and bin counts and see how it does. My guess is that the sort based algoritm will do better in some cases (arrays with 10k or fewer elements and lots of bins) while the counting algorithm does better in others (few bins or 100k or more elements). But we'll see. Or maybe we won't. Regards, -tim >Greetings, >Norbert > > > >Travis Oliphant wrote: > > > >>Norbert Nemec wrote: >> >> >> >>>I have a very much related problem: Not only that the idea described by >>>Mads Ipsen does not work, but I could generally find no efficient way to >>>do a "counting" of elements in an array, as it is needed for a >>>histogram. >>> >>> >>> >>This may be something we are lacking. >> >>It depends on what you mean by efficient, I suppose. The sorting >>algorithms are very fast, and the histogram routines that are >>scattered all over the place (including the histogram function in >>numpy) has been in use for a long time, and you are the first person >>to complain of its efficiency. That isn't to say your complaint may >>not be valid, it's just that for most people the speed has been >>sufficient. >> >> >> >>>What would instead be needed is a function that simply gives the count >>>of occurances of given values in a given array: >>> >>> >>> >>I presume you are talking of "integer" arrays, since >>equality-comparison of floating-point values is usually not very >>helpful so most histograms on floating-point values are given in terms >>of bins. Thus, the searchsorted function uses bins for it's >>"counting" operation. >> >> >> >>> >>> >>> >>> >>>>>>[4,5,2,3,2,1,4].count([0,1,2,3,4,5]) >>>>>> >>>>>> >>>>>> >>>[0,1,2,1,1,2] >>> >>> >>> >>> >>A count function for integer arrays could certainly be written using a >>C-loop. But, I would first just use histogram([4,5,2,3,2,1,4], >>[0,1,2,3,4,5]) and make sure that's really too slow, before worrying >>about it too much. >> >>Also, I according to the above function, the right answer is: >> >>[0, 1, 2, 1, 2, 1] >> >> >>Best, >> >> >>-Travis >> >> >> >> >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>language >>that extends applications into web and mobile media. Attend the live >>webcast >>and join the prime developer group breaking into this new coding >>territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> > > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From david.huard at gmail.com Thu Mar 30 09:10:06 2006 From: david.huard at gmail.com (David Huard) Date: Thu Mar 30 09:10:06 2006 Subject: [Numpy-discussion] Re: Freezing numpy code Message-ID: <91cf711d0603300844n1fdc7227ud5bc14533ee69ed4@mail.gmail.com> Thanks Pearu, >From the documentation in freeze.py, it is possible to tell freeze the explicit path to modules. freeze [options...] script [module] where module is an absolute path. I'll try this out. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Thu Mar 30 11:22:11 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 30 11:22:11 2006 Subject: [Numpy-discussion] Problem with _wrapit Message-ID: <442C2FBA.5080705@cox.net> There's a problem with the _wrapit function. If the function that it wraps, returns a scalar, it blows up. Here's an example: import numpy class narray(numpy.ndarray): def get_argmax(self): raise AttributeError argmax = property(get_argmax) a = narray([3], int, numpy.arange(3)) print type(a), isinstance(a, numpy.ndarray) print a print numpy.argmax(a) ==> True [0 1 2] Traceback (most recent call last): File "find_bug.py", line 13, in ? print numpy.argmax(a) File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", line 278, in argmax return _wrapit(a, 'argmax', axis) File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", line 170, in _wrapit result = wrap(result) TypeError: can only be called with ndarray object A possible fix is to do another isinstance check on the way out: def _wrapit(obj, method, *args, **kwds): try: wrap = obj.__array_wrap__ except AttributeError: wrap = None result = getattr(asarray(obj),method)(*args, **kwds) if wrap and isinstance(result, mu.ndarray): result = wrap(result) return result That fixes this problem, and I think it should be OK in general, but I'd like a second opinion before comitting it. regards, -tim From simon at arrowtheory.com Thu Mar 30 14:50:01 2006 From: simon at arrowtheory.com (Simon Burton) Date: Thu Mar 30 14:50:01 2006 Subject: [Numpy-discussion] ufunc broadcast for object arrays Message-ID: <20060331094857.241d9485.simon@arrowtheory.com> >>> a=numpy.array(list('foabcdeef'),dtype=numpy.dtype("O")) >>> numpy.equal(a,'f') NotImplemented So where do I start implementing this ? core/src/ufuncobject.c ? Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From ndarray at mac.com Thu Mar 30 19:55:01 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 30 19:55:01 2006 Subject: [Numpy-discussion] ufunc broadcast for object arrays In-Reply-To: <20060331094857.241d9485.simon@arrowtheory.com> References: <20060331094857.241d9485.simon@arrowtheory.com> Message-ID: On 3/30/06, Simon Burton wrote: > > >>> a=numpy.array(list('foabcdeef'),dtype=numpy.dtype("O")) > >>> numpy.equal(a,'f') > NotImplemented > > So where do I start implementing this ? > core/src/ufuncobject.c ? > Yes, 879 if (PyTypeNum_ISFLEXIBLE(arg_types[i])) { 880 loop->notimplemented = 1; 881 return nargs; 882 } (line numbers from r2310) From oliphant at ee.byu.edu Thu Mar 30 19:55:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 30 19:55:03 2006 Subject: [Numpy-discussion] ufunc broadcast for object arrays In-Reply-To: <20060331094857.241d9485.simon@arrowtheory.com> References: <20060331094857.241d9485.simon@arrowtheory.com> Message-ID: <442C6F98.3060601@ee.byu.edu> Simon Burton wrote: >>>>a=numpy.array(list('foabcdeef'),dtype=numpy.dtype("O")) >>>>numpy.equal(a,'f') >>>> >>>> A couple of things: 1) dtype=numpy.dtype("O") is equivalent to dtype="O" 2) Because you have an object array this should work. You can work around the bug using numpy.equal(a,array('f','O')) -Travis From pgmdevlist at mailcan.com Thu Mar 30 19:55:05 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Thu Mar 30 19:55:05 2006 Subject: [Numpy-discussion] Dumping record arrays Message-ID: <200603302127.24231.pgmdevlist@mailcan.com> Folks, I'd like to dump/pickle some record arrays. The pickling works, the unpickling raises a ValueError (on my version of numpy 0.9.6). (cf below). Is this already corrected in the svn version ? Thx ########################################################################### # x1 = array([21,32,14]) x2 = array(['my','first','name']) x3 = array([3.1, 4.5, 6.2]) r = rec.fromarrays([x1,x2,x3], names='id, word, number') r.dump('dumper') rb=load('dumper') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/backtopop/Work/workspace-python/pyflows/src/ /usr/lib64/python2.4/site-packages/numpy/core/numeric.py in load(file) 331 if isinstance(file, type("")): 332 file = _file(file,"rb") --> 333 return _cload(file) 334 335 # These are all essentially abbreviations /usr/lib64/python2.4/site-packages/numpy/core/_internal.py in _reconstruct(subtype, shape, dtype) 251 252 def _reconstruct(subtype, shape, dtype): --> 253 return ndarray.__new__(subtype, shape, dtype) 254 255 ValueError: ('data-type with unspecified variable length', , (, (0,), 'V')) From tim.hochberg at cox.net Thu Mar 30 19:55:13 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 30 19:55:13 2006 Subject: [Numpy-discussion] Segfault! Message-ID: <442C670A.1070209@cox.net> The following snippet utilizing a rank-0 array will segfault numpy: import numpy a = numpy.array(4) b = numpy.arange(10) a *= b (Build '0.9.7.2301' using VC7 on windows). I'll file a better bug report tomorrow, assuming this hasn't been fixed, but I'm on my way out the door. -tim From ndarray at mac.com Thu Mar 30 21:20:02 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 30 21:20:02 2006 Subject: [Numpy-discussion] Segfault! In-Reply-To: <442C670A.1070209@cox.net> References: <442C670A.1070209@cox.net> Message-ID: Fixed. On 3/30/06, Tim Hochberg wrote: > > The following snippet utilizing a rank-0 array will segfault numpy: > > import numpy > a = numpy.array(4) > b = numpy.arange(10) > a *= b > > (Build '0.9.7.2301' using VC7 on windows). > > I'll file a better bug report tomorrow, assuming this hasn't been fixed, > but I'm on my way out the door. > > -tim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aberrant at takas.lt Fri Mar 31 01:40:29 2006 From: aberrant at takas.lt (=?windows-1251?B?wu3o7ODt6OUg08PQzsfA?=) Date: Fri Mar 31 01:40:29 2006 Subject: [Numpy-discussion] =?windows-1251?B?we7w/OHgIPEg6u7w7+Dw4PLo4u377CDs7vjl7e3o9+Xx8uLu7A==?= Message-ID: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 10 13 ?????? 2006 ???? ???? ????????????? ?????????, ???????? ?????????? ???????????, ???????? ??????????? ? ?????? ??????? ???????????? ????????? ???? ?????????????? ?????????? ? ????????, ????? ?? ???????????? ???????? ???????? ?????????? ????????. ??? ??????? ????? ????????????? ? ?????????? ?????????? ??????????? ?? ???????????? ??????? ????????? ??? ???? ? ????? ? ????????? ???????. ?????? ?? ? ??? ???????? ????? ???????? ?????????? ?????? ??????????? ????????, ??????? ???????? ??????????? ??????? ???????????? ?????, ????????? ??????????????????? ? ???????????????? ?????? ? ?????? ? ????????? ????????????? ????????? ?????????? ???????, ???????? ??? ??????????????, ?????????? ? ????????????????? ?? ??????? ???????????. ??????? ????????? ?? ????????????? ?????????? ??????????? ???????, ?????? ??????? ?????????? ? ??????????? ????????, ? ????? ????????????? ?????????? ?????. ?????????? ??????????, ??????????? ? ???????????? ?????????? ? ????????? ??????????????? ??????????????????? ????????-??????????: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 10 13 ?????? 2006 ???? ???? ?????: ??????????? ??????????? ??????? ? ?????????? ??????????? ??????????? ??????????? ? ????????? ? ??????????? ???????? ??????????? ????????, ??????? ???????? ?????????? ????? ?????????????? ??????? ????? ???????? ?????? ??????????????? ???????????? ? ?????????????? ?????????. ????????? ?? ??? ?????????????? ????????????, ????? ???????? ???????? ???????? ???????? ???????? ? ??????? ??????????? ? ????? ??????? ???????? ? ??????????? ????????????? ????? ? ???????? ???????? ?????????? ???????? ?? ???????? ?????????? ? ?????????? ????????. ?? ???????? ????? ???????? ??????-????? ????????????? ??????????? ??????????? ???????? ??? ?????? ?? ????????????? ???????? ?????????????????? ? ?????????????? ???????, ? ????? ??????????? ? ?????? ?????????? ???????? ?? ???????????? ???????? ????????. ????? ??????? ??????? ? ????? ??? ???????? ????????? ??????????, ????????? ??? ?? ?????????: (495) 98O-65-16, 98?-65-?9 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Fri Mar 31 04:24:05 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri Mar 31 04:24:05 2006 Subject: [Numpy-discussion] Segfault! In-Reply-To: Message-ID: <009b01c654bd$dbc746a0$0a84a8c0@dsp.sun.ac.za> Hello all I've noticed a lot of fixes going into NumPy during the past couple of weeks and I'm very excited about the progress that is being made. However, it seems many changes are being made without the addition of test cases to make sure that things don't break again in the future. The code snippet Tim provided to reproduce the bug mentioned below would be a perfect test, but it doesn't seem as if a test for this bug was added in the changeset where the bug was fixed or any changeset since. Any comments? Regards, Albert > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy- > discussion-admin at lists.sourceforge.net] On Behalf Of Sasha > Sent: 31 March 2006 07:20 > To: tim.hochberg at ieee.org > Cc: numpy-discussion > Subject: Re: [Numpy-discussion] Segfault! > > Fixed. > > On 3/30/06, Tim Hochberg wrote: > > > > The following snippet utilizing a rank-0 array will segfault numpy: > > > > import numpy > > a = numpy.array(4) > > b = numpy.arange(10) > > a *= b > > > > (Build '0.9.7.2301' using VC7 on windows). > > > > I'll file a better bug report tomorrow, assuming this hasn't been fixed, > > but I'm on my way out the door. > > > > -tim From ndarray at mac.com Fri Mar 31 06:26:04 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 31 06:26:04 2006 Subject: [Numpy-discussion] Segfault! In-Reply-To: <009b01c654bd$dbc746a0$0a84a8c0@dsp.sun.ac.za> References: <009b01c654bd$dbc746a0$0a84a8c0@dsp.sun.ac.za> Message-ID: Point taken. On 3/31/06, Albert Strasheim wrote: > ... > However, it seems many changes are being made without the addition of test > cases to make sure that things don't break again in the future. The code > snippet Tim provided to reproduce the bug mentioned below would be a perfect > test, but it doesn't seem as if a test for this bug was added in the > changeset where the bug was fixed or any changeset since. From pebarrett at gmail.com Fri Mar 31 06:43:04 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri Mar 31 06:43:04 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX Message-ID: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> I'm having trouble build Numpy on HP-UX. I get the following error message when using the native 'cc' compiler: From ndarray at mac.com Fri Mar 31 06:48:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 31 06:48:03 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX In-Reply-To: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> References: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> Message-ID: Try to remove the "build" directory. On 3/31/06, Paul Barrett wrote: > I'm having trouble build Numpy on HP-UX. > > I get the following error message when using the native 'cc' compiler: > > cc: numpy/core/src/multiarraymodule.c > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: Illegal rede > claration for identifier "ushort". > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: Illegal rede > claration for identifier "uint". > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: Redeclaration of "Py > Array_SetNumericOps" with a different storage class specifier: "PyArray_SetNumer > icOps" will have internal linkage. > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: Illegal rede > claration for identifier "ushort". > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: Illegal rede > claration for identifier "uint". > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: Redeclaration of "Py > Array_SetNumericOps" with a different storage class specifier: "PyArray_SetNumer > icOps" will have internal linkage. > error: Command "cc -Ae -DNDEBUG -O +z -Ibuild/src/numpy/core/src -Inumpy/core/in > clude -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/ > include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.hp-ux-B.11. > 11-9000/785-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 > > And this error message when trying to use 'gcc' instead: > > error: don't know how to compile C/C++ code on platform 'posix' with 'gcc' comp > iler > > Has this been fixed in the current SVN or should can someone suggest a fix? > > -- Paul > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From pebarrett at gmail.com Fri Mar 31 06:51:01 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri Mar 31 06:51:01 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX In-Reply-To: References: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> Message-ID: <40e64fa20603310649v61b9f9d8wd32f4aa818753ea6@mail.gmail.com> Yes, I've done this numerous times without success. It looks like a configuration problem. -- Paul On 3/31/06, Sasha wrote: > > Try to remove the "build" directory. > > On 3/31/06, Paul Barrett wrote: > > I'm having trouble build Numpy on HP-UX. > > > > I get the following error message when using the native 'cc' compiler: > > > > cc: numpy/core/src/multiarraymodule.c > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > error: Command "cc -Ae -DNDEBUG -O +z -Ibuild/src/numpy/core/src > -Inumpy/core/in > > clude -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include > -I/usr/local/ > > include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.hp- > ux-B.11. > > 11-9000/785-2.4/numpy/core/src/multiarraymodule.o" failed with exit > status 1 > > > > And this error message when trying to use 'gcc' instead: > > > > error: don't know how to compile C/C++ code on platform 'posix' with > 'gcc' comp > > iler > > > > Has this been fixed in the current SVN or should can someone suggest a > fix? > > > > -- Paul > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > > that extends applications into web and mobile media. Attend the live > webcast > > and join the prime developer group breaking into this new coding > territory! > > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pebarrett at gmail.com Fri Mar 31 06:51:05 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri Mar 31 06:51:05 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX In-Reply-To: References: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> Message-ID: <40e64fa20603310649v61b9f9d8wd32f4aa818753ea6@mail.gmail.com> Yes, I've done this numerous times without success. It looks like a configuration problem. -- Paul On 3/31/06, Sasha wrote: > > Try to remove the "build" directory. > > On 3/31/06, Paul Barrett wrote: > > I'm having trouble build Numpy on HP-UX. > > > > I get the following error message when using the native 'cc' compiler: > > > > cc: numpy/core/src/multiarraymodule.c > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > error: Command "cc -Ae -DNDEBUG -O +z -Ibuild/src/numpy/core/src > -Inumpy/core/in > > clude -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include > -I/usr/local/ > > include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.hp- > ux-B.11. > > 11-9000/785-2.4/numpy/core/src/multiarraymodule.o" failed with exit > status 1 > > > > And this error message when trying to use 'gcc' instead: > > > > error: don't know how to compile C/C++ code on platform 'posix' with > 'gcc' comp > > iler > > > > Has this been fixed in the current SVN or should can someone suggest a > fix? > > > > -- Paul > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > > that extends applications into web and mobile media. Attend the live > webcast > > and join the prime developer group breaking into this new coding > territory! > > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Fri Mar 31 08:32:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 08:32:03 2006 Subject: [Numpy-discussion] Re: Problem with _wrapit In-Reply-To: <442C2FBA.5080705@cox.net> References: <442C2FBA.5080705@cox.net> Message-ID: <442D5949.9010907@cox.net> Well, I got no negative opinions (no opinions at all for that matter), so I went ahead and commited this change (with test cases -- thanks for the push Albert). -tim Tim Hochberg wrote: > > There's a problem with the _wrapit function. If the function that it > wraps, returns a scalar, it blows up. Here's an example: > > import numpy > > class narray(numpy.ndarray): > def get_argmax(self): > raise AttributeError > argmax = property(get_argmax) > a = narray([3], int, numpy.arange(3)) > print type(a), isinstance(a, numpy.ndarray) > > print a > print numpy.argmax(a) > > ==> > > True > [0 1 2] > Traceback (most recent call last): > File "find_bug.py", line 13, in ? > print numpy.argmax(a) > File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", > line 278, in argmax > return _wrapit(a, 'argmax', axis) > File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", > line 170, in _wrapit > result = wrap(result) > TypeError: can only be called with ndarray object > > > A possible fix is to do another isinstance check on the way out: > > def _wrapit(obj, method, *args, **kwds): > try: > wrap = obj.__array_wrap__ > except AttributeError: > wrap = None > result = getattr(asarray(obj),method)(*args, **kwds) > if wrap and isinstance(result, mu.ndarray): > result = wrap(result) > return result > > That fixes this problem, and I think it should be OK in general, but > I'd like a second opinion before comitting it. > > regards, > > -tim > > > > > > From filip at ftv.pl Fri Mar 31 10:13:05 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Fri Mar 31 10:13:05 2006 Subject: [Numpy-discussion] Confused with Py_ssize_t and Py_intptr_t (intp) Message-ID: <274382036.20060331201149@ftv.pl> Hello, I'm trying to figure out what is the proper type for indexing arrays. Is it safe to use Py_ssize_t although PyArrayObject dimensions are of Py_intptr_t (intp) type or use intp instead of Py_ssize_t? Is there any platform on which these two types has different size or range? References: [1] http://www.python.org/dev/peps/pep-0353/#why-not-py-intptr-t [2] numpy 0.9.6, arrayobject.c: static _int_or_ssize_t array_length(PyArrayObject *self) { if (self->nd != 0) { return self->dimensions[0]; } else { PyErr_SetString(PyExc_TypeError, "len() of unsized object"); return -1; } } // self->dimensions is of intp* type [3] http://svn.python.org/projects/python/branches/ssize_t/Include/abstract.h PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *obj, const void **buffer, Py_ssize_t *buffer_len); -- best, fw From haase at msg.ucsf.edu Fri Mar 31 12:30:02 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Mar 31 12:30:02 2006 Subject: [Numpy-discussion] first impressions with numpy Message-ID: <442D9124.5020905@msg.ucsf.edu> Hi, I'm a long time user of numarray. Now I downloaded numpy for the first time - and am quite excited to maybe soon being able to use things like weave ! Thanks for all the good work ! 1) I was following the discussion about dtype with interest, and so it was one of the first things to check... >>> repr(aa.dtype) 'dtype('>> str(aa.dtype) ''>> aa.dtype.name 'int32' Would it be possible to change str(aa.dtype) to aa.dtype.name ?? I think it would be much more readable ! I'm mainly thinking of (some "less involved") people that just want to know that it's a '32bit integer' ; ' References: <442D9124.5020905@msg.ucsf.edu> Message-ID: <442D9AC8.9090400@stsci.edu> Sebastian Haase wrote: > Hi, > I'm a long time user of numarray. Now I downloaded numpy for the first > time - and am quite excited to maybe soon being able to use things > like weave ! > Thanks for all the good work ! > > 1) > I was following the discussion about dtype with interest, and so it > was one of the first things to check... > >>> repr(aa.dtype) > 'dtype(' >>> str(aa.dtype) > '' >>> aa.dtype.name > 'int32' > > Would it be possible to change str(aa.dtype) to aa.dtype.name ?? > I think it would be much more readable ! > I'm mainly thinking of (some "less involved") people that just want to > know that it's a '32bit integer' ; ' > > 2) > This is probably more numarray related: > Why does numarray.asarray( numpy.array([1]) ) return a numpy array, > not a numarray ??? This is even true for numarray.array( > numpy.array([1]) ) !! This is what I get for numarray-CVS and numpy-CVS: >>> a = numarray.array(numpy.array([1])) >>> type(a) >>> a = numarray.asarray(numpy.array([1])) >>> type(a) So... I don't see a problem. Also: I logged the numarray.maximum.reduce axis=-3 transpose problem on SF. It should get fixed for numarray-1.5.2 but that may be a while. Todd > > I'll keep exploring... > Thanks, > Sebastian Haase > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant at ee.byu.edu Fri Mar 31 14:24:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 31 14:24:03 2006 Subject: [Numpy-discussion] first impressions with numpy In-Reply-To: <442D9124.5020905@msg.ucsf.edu> References: <442D9124.5020905@msg.ucsf.edu> Message-ID: <442DABA8.20003@ee.byu.edu> Sebastian Haase wrote: > Hi, > I'm a long time user of numarray. Now I downloaded numpy for the first > time - and am quite excited to maybe soon being able to use things > like weave ! > Thanks for all the good work ! > > 1) > I was following the discussion about dtype with interest, and so it > was one of the first things to check... > >>> repr(aa.dtype) > 'dtype(' >>> str(aa.dtype) > '' >>> aa.dtype.name > 'int32' > > Would it be possible to change str(aa.dtype) to aa.dtype.name ?? It's definitely possible and probably a good idea for some of the builtin types. -Travis From pgmdevlist at mailcan.com Fri Mar 31 15:37:03 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Fri Mar 31 15:37:03 2006 Subject: [Numpy-discussion] Pb with vectorizing Message-ID: <200603311824.38683.pgmdevlist@mailcan.com> Folks, I'm trying to use `vectorize` on the following lambda function, taking a scalar as input, and output a 10-char string fnc = lambda x: datetime.date.fromordinal(x).strftime("%Y-%m-%d") I can't find a way to get an actual 'S10' chararray as output. OIn top of that, it seems that vectorize still uses the oldnumeric typecode: is this on purpose (and then which one), or just because it has been updated yet ? Here's what I tried so far. Any idea/comment (even a 'that can't work, forget it) more than welcome, as usual Thx P. ---------------------------------------------------------- x = array([719893,719894, 719895]) fnc(x[0]) >>> '1972-01-01' fnc(x[:2]) >>> TypeError: only length-1 arrays can be converted to Python scalars (OK, that wasn't suppose to work) vectorize(fnc)(x[:2]) >>> array([, ], dtype='|S0') (not quite) vectorize(fnc,otypes="S")(x[:2]) >>> array([, ], dtype='|S0') (ditto) vectorize(fnc,otypes="|S10")(x[:2]) >>> ValueError: invalid typecode specified asarray(vectorize(fnc)(newddatesordinal[:2]),dtype='|S10') >>> array([?6z??*, ?6z??*], dtype='|S10') From tim.hochberg at cox.net Fri Mar 31 17:25:10 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 17:25:10 2006 Subject: [Numpy-discussion] first impressions with numpy In-Reply-To: <442DB91F.9030103@msg.ucsf.edu> References: <442D9124.5020905@msg.ucsf.edu> <442D9695.2050900@cox.net> <442DB655.2050203@cox.net> <442DB91F.9030103@msg.ucsf.edu> Message-ID: <442DD638.60706@cox.net> Sebastian Haase wrote: > Thanks Tim, > that's OK - I got the idea... > BTW, is there a (policy) reason that you sent the first email just to > me and not the mailing list !? No. Just clumsy fingers. Probably the same reason the functions got all garbled! > > I would really be more interested in comments to my first point ;-) > I think it's important that numpy will not be to cryptic and only for > "hackers", but nice to look at ... (hope you get what I mean ;-) Well, I think it's probably a good idea and it sounds like Travis like the idea " for some of the builtin types". I suspect that's code for "not types for which it doesn't make sense, like recarrays". > I'm have developed an "image analysis algorithm development" platform > (based on wxPython + PyShell) that more and more people in our lab are > using (instead of Matlab !) and I changed the default sys.displayhook > to print str(...) instead of repr(...) > mainly to get .3 instead of .2999999999998 > but seing int32 instead of Thanks for your great work .. Oh, I'm not doing much in the way of great work. I'm mostly just causing Travis headaches. -tim > Sebastian > > > Tim Hochberg wrote: > >> Tim Hochberg wrote: >> >>> Sebastian Haase wrote: >>> >>>> Hi, >>>> I'm a long time user of numarray. Now I downloaded numpy for the >>>> first time - and am quite excited to maybe soon being able to use >>>> things like weave ! >>>> Thanks for all the good work ! >>>> >>> [SNIP] >>> >>>> >>>> >>>> 2) >>>> This is probably more numarray related: >>>> Why does numarray.asarray( numpy.array([1]) ) return a numpy array, >>>> not a numarray ??? This is even true for numarray.array( >>>> numpy.array([1]) ) !! >>>> >>> I expect that numarray is grabbing the output of __array__() and >>> using that for asarray. What's happening in the second case is hard >>> to know, but I bet it's some snafu with assuming that anything that >>> isn't a numarray, but implements __array__ must be returning a new >>> numarray. That's just a guess. FWIW, I'm using the following two >>> functions to go back and forth between numarray and numpy. So far >>> they seem to work and they're faster (or actually work in the case >>> of asnumarray). You may want to rename them to suit your personal >>> preferences. If you use them, please let me know if you find any >>> problems. >>> >>> Regards, >>> >> I see this got thoroughly mangled somehow. If you want a copy, let me >> know and I'll just send you it as a file. >> >> -tim >> > > > From tim.hochberg at cox.net Fri Mar 31 18:38:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 18:38:07 2006 Subject: [Numpy-discussion] numpy error handling Message-ID: <442DE773.4060104@cox.net> I've just been looking at how numpy handles changing the behaviour that is triggered when there are numeric error conditions (overflow, underflow, etc.). If I understand it correctly, and that's a big if, I don't think I like it nearly as much as the what numarray has in place. It appears that numpy uses the two functions, seterr and geterr, to set and query the error handling. These set/read a secret variable stored in the local scope. I assume that the various ufuncs then examine that value to determine how to handle errors. The secret variable approach is a little clunky, but that's not what concerns me. What concerns me is that this approach is *only* useful for built in numpy functions and falls down if we call any user defined functions. Suppose we want to be warned on underflow. Setting this is as simple as: def func(*args): numpy.seterr(under='warn') # do stuff with args return result Since seterr is local to the function, we don't have to reset the error handling at the end, which is convenient. And, this works fine if all we are doing is calling numpy functions and methods. However, if we are calling a function of our own devising we're out of luck since the called function will not inherit the error settings that we have set. Thus we have no way to influence the error settings of functions downstream from us. Compare this to the numarray approach, where you write code like the following: def func(*args) numarray.Error.pushMode(underflow='warn') try: # do stuff with args return result finally: numarray.Error.popMode() At first glance this looks a lot more clunky. However, it's also a lot more useful. In this case, pushMode pushes a new error mode value onto a global[1] stack of mode values. Thus, any functions that func calls, will inherit it's error settings, unless they explicitly override them. As for the clunkiness factor, that's partly just the choice of camelCase versus smashwords, but also the try-finally. The second, however, will be addressed in Python 2.5 with the 'with' statement. At that point, we could have a numpy equivalent to the above numarray code that looked something like: def func(*args) with numpy.errors(underflow='warn'): # do stuff with args return result I believe we should rework the error handling to follow the numarray model before things get set in stone at 1.0. The 'with' statement support can be worked in a bit later if need be, although I suspect that will be easy. I also would prefer more verbose keys ala numarray (underflow, overflow, dicidebyzero and invalid) than those currently used by numpy (under, over, divide and invalid). And (will he never stop) I like numarrays defaults better here too: overflow='warn', underflow='ignore', dividebyzero='warn', invalid='warn'. Currently, numpy defaults to ignore for all cases. These last points are relatively minor though. Regards, -tim [1] I believe it's actually thread local, and if it's not it should be, but that detail is not important for our discussion here. From zpincus at stanford.edu Fri Mar 31 18:52:04 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri Mar 31 18:52:04 2006 Subject: [Numpy-discussion] array_arguments decorator Message-ID: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> Hi folks - I had seen some talk on this list about the utility of a decorator for functions that need to convert their arguments to numpy arrays. This would help eliminate boilerplate calls to 'asarray' like: def distance_squared(a, b): a = numpy.asarray(a) b = numpy.asarray(b) return ((a - b)**2).sum() Here is a trivial decorator I was thinking of adding to the wiki -- does this cover enough cases to be useful? In a bigger sense, would it be worthwhile to add some decorators like this to numpy itself? (I'm not sure I'm in favor of this, since I kind of like smaller APIs over bigger ones.) def array_arguments(f): """Wrap a function such that any positional arguments are converted into numpy arrays before the function is called.""" def convert_arg_wrapper(*args, **kwargs): array_args = [numpy.asarray(a) for a in args] return f(*array_args, **kwargs) return convert_arg_wrapper now distance_squared can look like: @array_arguments def distance_squared(a, b): return ((a - b)**2).sum() if using python 2.4, or if not so using: def distance_squared(a, b): return ((a - b)**2).sum() distance_squared = array_arguments(distance_squared) Zach From tim.hochberg at cox.net Fri Mar 31 19:49:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 19:49:03 2006 Subject: [Numpy-discussion] array_arguments decorator In-Reply-To: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> References: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> Message-ID: <442DF814.8020806@cox.net> Zachary Pincus wrote: > Hi folks - > > I had seen some talk on this list about the utility of a decorator > for functions that need to convert their arguments to numpy arrays. > This would help eliminate boilerplate calls to 'asarray' like: > > def distance_squared(a, b): > a = numpy.asarray(a) > b = numpy.asarray(b) > return ((a - b)**2).sum() > > Here is a trivial decorator I was thinking of adding to the wiki -- > does this cover enough cases to be useful? In a bigger sense, would > it be worthwhile to add some decorators like this to numpy itself? > (I'm not sure I'm in favor of this, since I kind of like smaller APIs > over bigger ones.) > > def array_arguments(f): > """Wrap a function such that any positional arguments are > converted into numpy arrays before the function is called.""" > def convert_arg_wrapper(*args, **kwargs): > array_args = [numpy.asarray(a) for a in args] > return f(*array_args, **kwargs) > return convert_arg_wrapper > > now distance_squared can look like: > @array_arguments > def distance_squared(a, b): > return ((a - b)**2).sum() > > if using python 2.4, or if not so using: > def distance_squared(a, b): > return ((a - b)**2).sum() > distance_squared = array_arguments(distance_squared) Great minds think alike. Or at least our minds think alike ;) I also wrote up a decorator for this same purpose. Then I got distracted and forgot to post it. Mine has more features at the expense of being more complicated. The main extra feature is that it allows you to decide both which args get checked and what there types should be. It also preserves the signature of the original function. Some of this stuff is accomplished using the decorator decorator, which you can find here: http://www.phyast.pitt.edu/~micheles/python/ The upshot of all of this is that you can do stuff like: @array_function(a=float, b=None, c=complex) def foo(a, b, c=1, d=None): print repr(a), repr(b), repr(c), repr(d) And it will convert 'a' and 'c' to float and complex arrays respectively and convert 'b' to some type of array. Arguments not mentioned don't get touched, unless you specify no arguments, in which case all of the positional arguments get converted (*args and **kwargs are not touched in this case). I'm not certain that this is the world's best interface, nor am I certain that the extra complexity is worth it -- yours is certainly easier to understand. However, it was fun to write. I am unlikely to find the time to do anything with it anytime soon, so should you desire to revamp and place it on the wiki, feel free. Or, if you want to ignore it, feel free to do that as well. The code is below: Regards, -tim import inspect, decorator, numpy def array_function(**signature): def deco(func): regargs, varargs, varkwargs, defaults = inspect.getargspec(func) if not signature: signature.update((name, None) for x in regargs) def caller(func, *args, **kwargs): args = list(args) for i, (name, value) in enumerate(zip(regargs, args)): if name in signature: args[i] = numpy.asarray(value, signature[name]) for name, value in kwargs.items(): if name in signature: kwargs[name] = numpy.asarray(value, signature[name]) return func(*args, **kwargs) return decorator._decorate(func, caller) return deco From zpincus at stanford.edu Fri Mar 31 23:41:06 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri Mar 31 23:41:06 2006 Subject: [Numpy-discussion] array_arguments decorator In-Reply-To: <442DF814.8020806@cox.net> References: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> <442DF814.8020806@cox.net> Message-ID: <0B289EAA-C75B-4A3C-96C8-59C98EBA37E5@stanford.edu> Hi Tim - Thanks for posting your featureful decorator. I didn't know that some of that stuff was possible -- very cool. Hopefully I'll find some time to put these up on the wiki. Zach On Mar 31, 2006, at 7:48 PM, Tim Hochberg wrote: > Zachary Pincus wrote: > >> Hi folks - >> >> I had seen some talk on this list about the utility of a >> decorator for functions that need to convert their arguments to >> numpy arrays. This would help eliminate boilerplate calls to >> 'asarray' like: >> >> def distance_squared(a, b): >> a = numpy.asarray(a) >> b = numpy.asarray(b) >> return ((a - b)**2).sum() >> >> Here is a trivial decorator I was thinking of adding to the wiki >> -- does this cover enough cases to be useful? In a bigger sense, >> would it be worthwhile to add some decorators like this to numpy >> itself? (I'm not sure I'm in favor of this, since I kind of like >> smaller APIs over bigger ones.) >> >> def array_arguments(f): >> """Wrap a function such that any positional arguments are >> converted into numpy arrays before the function is called.""" >> def convert_arg_wrapper(*args, **kwargs): >> array_args = [numpy.asarray(a) for a in args] >> return f(*array_args, **kwargs) >> return convert_arg_wrapper >> >> now distance_squared can look like: >> @array_arguments >> def distance_squared(a, b): >> return ((a - b)**2).sum() >> >> if using python 2.4, or if not so using: >> def distance_squared(a, b): >> return ((a - b)**2).sum() >> distance_squared = array_arguments(distance_squared) > > Great minds think alike. Or at least our minds think alike ;) I > also wrote up a decorator for this same purpose. Then I got > distracted and forgot to post it. Mine has more features at the > expense of being more complicated. The main extra feature is that > it allows you to decide both which args get checked and what there > types should be. It also preserves the signature of the original > function. Some of this stuff is accomplished using the decorator > decorator, which you can find here: > > http://www.phyast.pitt.edu/~micheles/python/ > > The upshot of all of this is that you can do stuff like: > > @array_function(a=float, b=None, c=complex) > def foo(a, b, c=1, d=None): > print repr(a), repr(b), repr(c), repr(d) > > And it will convert 'a' and 'c' to float and complex arrays > respectively and convert 'b' to some type of array. Arguments not > mentioned don't get touched, unless you specify no arguments, in > which case all of the positional arguments get converted (*args and > **kwargs are not touched in this case). > > I'm not certain that this is the world's best interface, nor am I > certain that the extra complexity is worth it -- yours is certainly > easier to understand. However, it was fun to write. I am unlikely > to find the time to do anything with it anytime soon, so should you > desire to revamp and place it on the wiki, feel free. Or, if you > want to ignore it, feel free to do that as well. > > The code is below: > > Regards, > > -tim > > > > import inspect, decorator, numpy > > def array_function(**signature): > def deco(func): > regargs, varargs, varkwargs, defaults = inspect.getargspec > (func) > if not signature: > signature.update((name, None) for x in regargs) > def caller(func, *args, **kwargs): > args = list(args) > for i, (name, value) in enumerate(zip(regargs, args)): > if name in signature: > args[i] = numpy.asarray(value, signature[name]) > for name, value in kwargs.items(): > if name in signature: > kwargs[name] = numpy.asarray(value, signature > [name]) > return func(*args, **kwargs) > return decorator._decorate(func, caller) > return deco > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Fri Mar 31 23:44:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 31 23:44:01 2006 Subject: [Numpy-discussion] numpy error handling In-Reply-To: <442DE773.4060104@cox.net> References: <442DE773.4060104@cox.net> Message-ID: <442E2F05.5080809@ieee.org> Tim Hochberg wrote: > > I've just been looking at how numpy handles changing the behaviour > that is triggered when there are numeric error conditions (overflow, > underflow, etc.). If I understand it correctly, and that's a big if, I > don't think I like it nearly as much as the what numarray has in place. > > It appears that numpy uses the two functions, seterr and geterr, to > set and query the error handling. These set/read a secret variable > stored in the local scope. This approach was decided on after discussions with Guido who didn't like the idea of pushing and popping from a global stack. I'm not sure I'm completely in love with it my self, but it is actually more flexible then the numarray approach. You can get the numarray approach back simply by setting the error in the builtin scope (instead of in the local scope which is done by default. Then, at the end of the function, you can restore it. If it was felt useful to create a stack to handle this on the builtin level then that is easily done as well. > I assume that the various ufuncs then examine that value to determine > how to handle errors. The secret variable approach is a little clunky, > but that's not what concerns me. What concerns me is that this > approach is *only* useful for built in numpy functions and falls down > if we call any user defined functions. > > Suppose we want to be warned on underflow. Setting this is as simple as: > > def func(*args): > numpy.seterr(under='warn') > # do stuff with args > return result > > Since seterr is local to the function, we don't have to reset the > error handling at the end, which is convenient. And, this works fine > if all we are doing is calling numpy functions and methods. However, > if we are calling a function of our own devising we're out of luck > since the called function will not inherit the error settings that we > have set. Again, you have control over where you set the "secret" variable (local, global (module), and builtin). I also don't see how that's anymore clunky then a "secret" stack. You may set the error in the builtin scope --- in fact it would probably be trivial to implement a stack based on this and implement the pushMode popMode interface of numarray. But, I think this question does deserve a bit of debate. I don't think there has been a serious discussion over the method. To help Tim and others understand what happens: When a ufunc is called, a specific variable name is searched for in the following name-spaces in the following order: 1) local 2) global 3) builtin (There is a bit of an optimization in that when the error mode is the default mode --- do nothing, a global flag is set which by-passes the search for the name). The first time the variable name is found, the error mode is read from that variable. This error mode is placed as part of the ufunc loop object. At the end of each 1-d loop the IEEE error mode flags are checked (depending on the state of the error mode) and appropriate action taken. By the way, it would not be too difficult to change how the error mode is set (probably an hour's worth of work). So, concern over implementation changes should not be a factor right now. Currently the error mode is read from a variable using standard scoping rules. It would save the (not insignificant) name-space lookup time to instead use a global stack (i.e. a Python list) and just get the error mode from the top of that stack. > Thus we have no way to influence the error settings of functions > downstream from us. Of course, there is a way to do this by setting the variable in the global or builtin scope as I've described above. What's really the argument here, is whether having the flexibility at the local and global name-spaces really worth the extra name-lookups for each ufunc. I've argued that the numarray behavior can result from using the builtin namespace for the error control. (perhaps with better Python-side support for setting and retrieving it). What numpy has is control at the global and local namespace level as well which can override the builtin name-space behavior. So, we should at least frame the discussion in terms of what is actually possible. > > I also would prefer more verbose keys ala numarray (underflow, > overflow, dicidebyzero and invalid) than those currently used by numpy > (under, over, divide and invalid). In my mind, verbose keys are just extra baggage unless they are really self documenting. You just need reminders and clues. It seems to be a preference thing. I guess I hate typing long strings when only the first few letters clue me in to what is being talked about. > And (will he never stop) I like numarrays defaults better here too: > overflow='warn', underflow='ignore', dividebyzero='warn', > invalid='warn'. Currently, numpy defaults to ignore for all cases. > These last points are relatively minor though. This has optimization issues the way the code is written now. The defaults are there to produce the fastest loops. So, I'm hesitant to change them based only on ambiguous preferences. Good feedback. Thanks again for taking the time to look at this and offer review. -Travis > > Regards, > > -tim > > > [1] I believe it's actually thread local, and if it's not it should > be, but that detail is not important for our discussion here. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From zpincus at stanford.edu Wed Mar 1 01:07:02 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed Mar 1 01:07:02 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? Message-ID: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> Hi folks, I'm using numpy 0.95 and came across an odd error with numpy.dot and degenerate 2D floating-point arrays. See below for an illustration. In [1]: import numpy In [2]: numpy.version.version Out[2]: '0.9.5' In [3]: numpy.dot([[5]],[[1,1,1]]) # 2D int case OK Out[3]: array([[5, 5, 5]]) In [4]: numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? Out[4]: array([[ 5.00000000e+000, 4.46601253e-316, 5.42111867e-306]]) In [5]: numpy.dot([5.],[1,1,1]) # 1D float case OK Out[5]: array([ 5., 5., 5.]) In [6]: numpy.dot([[[5.]]],[[[1,1,1]]]) # 3D float case OK Out[6]: array([[[[ 5., 5., 5.]]]]) Zach From Mithrandir42 at web.de Wed Mar 1 03:41:02 2006 From: Mithrandir42 at web.de (Niklas Volbers) Date: Wed Mar 1 03:41:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array Message-ID: <1307204529@web.de> [[Oops, I accidentally sent my post to scipy-users, but it was intended for numpy-discussion, so here's another attempt]] Hey Michael, take a look at my attempt of such a Table implementation! The newest release 0.5.2 of my plotting project http://developer.berlios.de/projects/sloppyplot contains a Table class (in Sloppy.Base.dataset) which wraps a heterogeneous numpy array. The class should be fairly self-documenting, at least I hope so. Don't get confused by the 'undolist' stuff, this is my private undo implementation which could be easily removed from the code. If you want a similar implementation using a list of 1-dimensional arrays, then download the previous release 0.5.1 (which uses Numeric). The reason I switched over to the heterogeneous approach was that it is easier to provide similar wrappers for 2-d table like data (using a 2d heterogeneous array) and for 2-d matrix like data (using a 2d homogeneous array). Using a list of arrays gives you some problems when you would like to access the rows, because then you are in charge of creating a new 1-d array that represents the row, while with the heterogeneous array you can access both the columns and the rows quite naturally. By the way, I had first planned to subclass ndarray, but I did not know how to resize the array and still keep the array as such persistent. This is why I wrapped the array into a class called 'Dataset' which you can consider constant. If you need some more help on this, feel free to ask, Niklas Volbers. Travis Oliphant schrieb am 01.03.06 08:16:15: > > Michael Sorich wrote: > > > Hi, > > > > I am looking for a table like array. Something like a 'data frame' > > object to those familiar with the statistical languages R and Splus. > > This is mainly to hold and manipulate 2D spreadsheet like data, which > > tends to be of relatively small size (compared to what many people > > seem to use numpy for), heterogenous, have column and row names, and > > often contains missing data. > > You could subclass the ndarray to produce one of these fairly easily, I > think. The missing data item could be handled by a mask stored along > with the array (or even in the array itself). Or you could use a masked > array as your core object (though I'm not sure how it handles the > arbitrary (i.e. record-like) data-types yet). > > Alternatively, and probably the easiest way to get started, you could > just create your own table-like class and use simple 1-d arrays or 1-d > masked arrays for each of the columns --- This has always been a way to > store record-like tables. > > It really depends what you want the data-frames to be able to do and > what you want them to "look-like." > > > A RecArray seems potentially useful, as it allows different fields to > > have different data types and holds the name of the field. However it > > doesn't seem easy to manipulate the data. Or perhaps I am simply > > having difficulty finding documentation on there features. > > Adding a new column/field means basically creating a new array with a > new data-type and copying data over into the already-defined fields. > Data-types always have a fixed number of bytes per item. What those > bytes represent can be quite arbitrary but it's always fixed. So, it > is always "more work" to insert a new column. You could make that > seamless in your table class so the user doesn't see it though. > > You'll want to thoroughly understand the dtype object including it's > attributes and methods. Particularly the fields attribute of the dtype > object. > > > eg > > adding a new column/field (and to a lesser extent a new row/record) to > > the recarray > > Adding a new row or record is actually similar because once an array is > created it is usually resized by creating another array and copying the > old array into it in the right places. > > > Changing the field/column names > > make a new table by selecting a subset of fields/columns. (you can > > select a single field/column, but not multiple). > > Right. So far you can't select multiple columns. It would be possible > to add this feature with a little-bit of effort if there were a strong > demand for it, but it would be much easier to do it in your subclass > and/or container class. > > How many people would like to see x['f1','f2','f5'] return a new array > with a new data-type descriptor constructed from the provided fields? > > > It would also be nice for the table to be able to deal easily with > > masked data (I have not tried this with recarray yet) and perhaps also > > to be able to give the rows/records unique ids that could be used to > > select the rows/records (in addition to the row/record index), in the > > same way that the fieldnames can select the fields. > > Adding fieldnames to the "rows" is definitely something that a subclass > would be needed for. I'm not sure how you would even propose to select > using row names. Would you also use getitem semantics? > > > Can anyone comment on this issue? Particularly whether code exists for > > this purpose, and if not ideas about how best to go about developing > > such a Table like array (this would need to be limited to python > > programing as my ability to program in c is very limited). > > I don't know of code that already exists for this, but I don't think it > would be too hard to construct your own data-frame object. > > I would probably start with an implementation that just used standard > arrays of a particular type to represent the internal columns and then > handle the indexing using your own over-riding of the __getitem__ and > __setitem__ special methods. This would be the easiest to get working, > I think. > > -Travis ______________________________________________________________ Verschicken Sie romantische, coole und witzige Bilder per SMS! Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193 From pjssilva at ime.usp.br Wed Mar 1 05:10:09 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Wed Mar 1 05:10:09 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> Message-ID: <1141218124.8183.0.camel@localhost.localdomain> Em Qua, 2006-03-01 ?s 01:05 -0800, Zachary Pincus escreveu: > In [4]: numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? > Out[4]: array([[ 5.00000000e+000, 4.46601253e-316, > 5.42111867e-306]]) It seems fixed in the repository In [1]:import numpy In [2]:numpy.dot([[5]],[[1,1,1]]) # 2D int case OK Out[2]:array([[5, 5, 5]]) In [3]:numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? Out[3]:array([[ 5., 5., 5.]]) In [4]:numpy.version.version Out[4]:'0.9.6.2179' Best, Paulo From pebarrett at gmail.com Wed Mar 1 06:45:02 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Wed Mar 1 06:45:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array In-Reply-To: <44054A19.6040202@ieee.org> References: <16761e100602282240y5bcf869fme9dd2f42771066c4@mail.gmail.com> <44054A19.6040202@ieee.org> Message-ID: <40e64fa20603010644x129d3f63ka07ab0b061469fd4@mail.gmail.com> On 3/1/06, Travis Oliphant wrote: > > > How many people would like to see x['f1','f2','f5'] return a new array > with a new data-type descriptor constructed from the provided fields? > > +1 I'm surprised that it's not already available. -- Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at utoronto.ca Wed Mar 1 09:03:10 2006 From: jonathan.taylor at utoronto.ca (Jonathan Taylor) Date: Wed Mar 1 09:03:10 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <1141218124.8183.0.camel@localhost.localdomain> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> Message-ID: <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> Is there a test for this in the test suite.? It looks really bad for numpy if ppl cant do matrix multiplication properly on released versions. :( J. On 3/1/06, Paulo Jose da Silva e Silva wrote: > Em Qua, 2006-03-01 ?s 01:05 -0800, Zachary Pincus escreveu: > > > > In [4]: numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? > > Out[4]: array([[ 5.00000000e+000, 4.46601253e-316, > > 5.42111867e-306]]) > > It seems fixed in the repository > > In [1]:import numpy > > In [2]:numpy.dot([[5]],[[1,1,1]]) # 2D int case OK > Out[2]:array([[5, 5, 5]]) > > In [3]:numpy.dot([[5.]],[[1,1,1]]) # 2D float case: what??? > Out[3]:array([[ 5., 5., 5.]]) > > In [4]:numpy.version.version > Out[4]:'0.9.6.2179' > > Best, > > Paulo > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Mar 1 09:33:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 1 09:33:05 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> Message-ID: <4405DA9B.9000307@ieee.org> Jonathan Taylor wrote: >Is there a test for this in the test suite.? It looks really bad for >numpy if ppl cant do matrix multiplication properly on released >versions. :( > > There are tests for matrix multiplication, just not one for this particular path through the code which is scalar multiplication by a row vector. (This is the optimized _dotblas.c code by the way which was just changed in 0.9.5 to try and make it faster -- obviously some bugs crept in at the same time). But, these bugs were quickly fixed when they were made apparent. That's why we are still not at 1.0 yet. Feel free to contribute more tests. -Travis From bsouthey at gmail.com Wed Mar 1 10:12:06 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Wed Mar 1 10:12:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array In-Reply-To: <1307204529@web.de> References: <1307204529@web.de> Message-ID: Hi, Thanks for your link, Niklas, as it looks interesting. The ability to create some sort of data array object would be very useful for me. The record array was not sufficiently flexible when I looked especially for missing values. The approach that I used was a class that defined, amongst other things, a masked array to be able to handle missing values and a dictionary to recode alphanumeric values into numeric values. The downside is that there is overhead in converting this to use in other functions and in some cases writing new functions. Bruce On 3/1/06, Niklas Volbers wrote: > [[Oops, I accidentally sent my post to scipy-users, but it was intended for numpy-discussion, so here's another attempt]] > > > Hey Michael, > > take a look at my attempt of such a Table implementation! > > The newest release 0.5.2 of my plotting project > > http://developer.berlios.de/projects/sloppyplot > > contains a Table class (in Sloppy.Base.dataset) which wraps a heterogeneous numpy array. The class should be fairly self-documenting, at least I hope so. Don't get confused by the 'undolist' stuff, this is my private undo implementation which could be easily removed from the code. > > If you want a similar implementation using a list of 1-dimensional arrays, then download the previous release 0.5.1 (which uses Numeric). > > The reason I switched over to the heterogeneous approach was that it is easier to provide similar wrappers for 2-d table like data (using a 2d heterogeneous array) and for 2-d matrix like data (using a 2d homogeneous array). Using a list of arrays gives you some problems when you would like to access the rows, because then you are in charge of creating a new 1-d array that represents the row, while with the heterogeneous array you can access both the columns and the rows quite naturally. > > By the way, I had first planned to subclass ndarray, but I did not know how to resize the array and still keep the array as such persistent. This is why I wrapped the array into a class called 'Dataset' which you can consider constant. > > If you need some more help on this, feel free to ask, > > Niklas Volbers. > > > > > > Travis Oliphant schrieb am 01.03.06 08:16:15: > > > > Michael Sorich wrote: > > > > > Hi, > > > > > > I am looking for a table like array. Something like a 'data frame' > > > object to those familiar with the statistical languages R and Splus. > > > This is mainly to hold and manipulate 2D spreadsheet like data, which > > > tends to be of relatively small size (compared to what many people > > > seem to use numpy for), heterogenous, have column and row names, and > > > often contains missing data. > > > > You could subclass the ndarray to produce one of these fairly easily, I > > think. The missing data item could be handled by a mask stored along > > with the array (or even in the array itself). Or you could use a masked > > array as your core object (though I'm not sure how it handles the > > arbitrary (i.e. record-like) data-types yet). > > > > Alternatively, and probably the easiest way to get started, you could > > just create your own table-like class and use simple 1-d arrays or 1-d > > masked arrays for each of the columns --- This has always been a way to > > store record-like tables. > > > > It really depends what you want the data-frames to be able to do and > > what you want them to "look-like." > > > > > A RecArray seems potentially useful, as it allows different fields to > > > have different data types and holds the name of the field. However it > > > doesn't seem easy to manipulate the data. Or perhaps I am simply > > > having difficulty finding documentation on there features. > > > > Adding a new column/field means basically creating a new array with a > > new data-type and copying data over into the already-defined fields. > > Data-types always have a fixed number of bytes per item. What those > > bytes represent can be quite arbitrary but it's always fixed. So, it > > is always "more work" to insert a new column. You could make that > > seamless in your table class so the user doesn't see it though. > > > > You'll want to thoroughly understand the dtype object including it's > > attributes and methods. Particularly the fields attribute of the dtype > > object. > > > > > eg > > > adding a new column/field (and to a lesser extent a new row/record) to > > > the recarray > > > > Adding a new row or record is actually similar because once an array is > > created it is usually resized by creating another array and copying the > > old array into it in the right places. > > > > > Changing the field/column names > > > make a new table by selecting a subset of fields/columns. (you can > > > select a single field/column, but not multiple). > > > > Right. So far you can't select multiple columns. It would be possible > > to add this feature with a little-bit of effort if there were a strong > > demand for it, but it would be much easier to do it in your subclass > > and/or container class. > > > > How many people would like to see x['f1','f2','f5'] return a new array > > with a new data-type descriptor constructed from the provided fields? > > > > > It would also be nice for the table to be able to deal easily with > > > masked data (I have not tried this with recarray yet) and perhaps also > > > to be able to give the rows/records unique ids that could be used to > > > select the rows/records (in addition to the row/record index), in the > > > same way that the fieldnames can select the fields. > > > > Adding fieldnames to the "rows" is definitely something that a subclass > > would be needed for. I'm not sure how you would even propose to select > > using row names. Would you also use getitem semantics? > > > > > Can anyone comment on this issue? Particularly whether code exists for > > > this purpose, and if not ideas about how best to go about developing > > > such a Table like array (this would need to be limited to python > > > programing as my ability to program in c is very limited). > > > > I don't know of code that already exists for this, but I don't think it > > would be too hard to construct your own data-frame object. > > > > I would probably start with an implementation that just used standard > > arrays of a particular type to represent the internal columns and then > > handle the indexing using your own over-riding of the __getitem__ and > > __setitem__ special methods. This would be the easiest to get working, > > I think. > > > > -Travis > > ______________________________________________________________ > Verschicken Sie romantische, coole und witzige Bilder per SMS! > Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193 > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Wed Mar 1 11:03:05 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 1 11:03:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Table like array In-Reply-To: <40e64fa20603010644x129d3f63ka07ab0b061469fd4@mail.gmail.com> References: <16761e100602282240y5bcf869fme9dd2f42771066c4@mail.gmail.com> <44054A19.6040202@ieee.org> <40e64fa20603010644x129d3f63ka07ab0b061469fd4@mail.gmail.com> Message-ID: <4405EFE6.3020201@sympatico.ca> Paul Barrett wrote: > > On 3/1/06, *Travis Oliphant* > wrote: > > > How many people would like to see x['f1','f2','f5'] return a new > array > with a new data-type descriptor constructed from the provided fields? > > > +1 > > I'm surprised that it's not already available. > > -- Paul I would like some clarification as to just what is proposed. The expression looks like a subscription but the purpose is closer to that of a function or method. Is this a new method for something which can be done easily in the standard way? -1 Colin W From ndarray at mac.com Wed Mar 1 11:29:10 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 1 11:29:10 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: <4405DA9B.9000307@ieee.org> References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> <4405DA9B.9000307@ieee.org> Message-ID: On 3/1/06, Travis Oliphant wrote: > ... > Feel free to contribute more tests. There are some things that we can do to help users contribute tests. Before I make specific suggestions, let me just express an opinion that developers are less likely to write effective tests than users because they tend to overlook in testing exactly the same cases that they get wrong in coding. Here are the things that we can do: 1. Documentation. Tell users how to submit bug reports that can be immediately converted into unit tests. 2. Improve NumpyTest: add facilities to generate test data with various shapes and datatypes and filled with wide range of values: extremely large/small numbers, IEEE special values, etc. Add routines to automatically test universal functions on all datatypes. Automate consistency checks - when appropriate check that computations on different datatypes produce similar results. 3. Add support for doctests. This is probably the simplest type of test that a user can produce - just cut and paste from python session. Doctests can be placed in separate files and don't need to clutter real documentation. Python 2.4 contains code integrating doctests and unit tests and that code has 95% of what is needed. If we have to support 2.3, it may be appropriate to distribute a backport with numpy. The remaining 5% include support for ipython sessions, maybe better output checking for printed arrays (such as scan through to a closing paren rather than an empty line etc.). Any volunteers? I guess the first step would be to put up a wiki page. Since this is really a developers/advanced users project it belongs to the "hard hat" area. From cookedm at physics.mcmaster.ca Wed Mar 1 11:42:07 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Mar 1 11:42:07 2006 Subject: [Numpy-discussion] numpy.dot gives wrong results with floats? In-Reply-To: References: <5574CC2B-5756-4B63-8381-A18D58183EB5@stanford.edu> <1141218124.8183.0.camel@localhost.localdomain> <463e11f90603010902v6d2729a9gd53e13320c361992@mail.gmail.com> <4405DA9B.9000307@ieee.org> Message-ID: <20060301194050.GA31179@arbutus.physics.mcmaster.ca> On Wed, Mar 01, 2006 at 02:28:28PM -0500, Sasha wrote: > On 3/1/06, Travis Oliphant wrote: > > ... > > Feel free to contribute more tests. > 3. Add support for doctests. This is probably the simplest type of > test that a user can produce - just cut and paste from python session. > Doctests can be placed in separate files and don't need to clutter > real documentation. Python 2.4 contains code integrating doctests and > unit tests and that code has 95% of what is needed. If we have to > support 2.3, it may be appropriate to distribute a backport with > numpy. The remaining 5% include support for ipython sessions, maybe > better output checking for printed arrays (such as scan through to a > closing paren rather than an empty line etc.). Already there. See the tests in numpy/lib/tests/test_polynomial.py -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From skip at pobox.com Wed Mar 1 16:03:04 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed Mar 1 16:03:04 2006 Subject: [Numpy-discussion] fold distutils back into Python distro? Message-ID: <17414.13849.523893.552309@montanaro.dyndns.org> As I understand it, numpy (or maybe it's scipy) has its own fork of distutils. What would it take to fold it back into the main distribution? Skip From oliphant at ee.byu.edu Wed Mar 1 16:14:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 1 16:14:03 2006 Subject: [Numpy-discussion] fold distutils back into Python distro? In-Reply-To: <17414.13849.523893.552309@montanaro.dyndns.org> References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: <440638AE.6020504@ee.byu.edu> skip at pobox.com wrote: >As I understand it, numpy (or maybe it's scipy) has its own fork of >distutils. What would it take to fold it back into the main distribution? > > > I guess a PEP that nobody has been willing to write. I suspect that not all of the numpy extensions are going to be seen as necessary by the Python people, and I don't know of anybody that's been willing to try and wrestle with that legwork. And just for clarification. As far as I understand it, it's not technically a fork of distutils. It is based on distutils but just extends it in many ways (i.e. new subclasses that extend from the Distutils classes). Pearu is the expert on it, however. -Travis From wbaxter at gmail.com Wed Mar 1 16:22:02 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 1 16:22:02 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function Message-ID: The NumPy for Matlab Users's wiki is currently pleading to have someone fill in "*the idiom for supporting both matrices and arrays in a function". *Does anyone know what this is? My guess is it depends first off on whether your goal is - to treat everything as a matrix, but also accept 2D arrays and do the conversion to matrix internally (asmatrix()) - to treat everything as a general array, but also accept matrices as a special case of 2D array, converting to arrays I guess it's likely that you'd also like to make it so the type returned is the same as the type input. Is there some slick way to do that? It doesn't seem to me that the latter case should really be specific to matrix/array since matrix is just one particular subclass of array. Is this what those UFuncs are for? Is there some way to write functions that will accept any object that supports the "array protocal"? This comment about the idiom for matrix/array functions is the last major wart remaining on that wiki page, so I'd like to wipe it out. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 1 16:23:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 1 16:23:02 2006 Subject: [Numpy-discussion] Re: fold distutils back into Python distro? In-Reply-To: <17414.13849.523893.552309@montanaro.dyndns.org> References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > As I understand it, numpy (or maybe it's scipy) has its own fork of > distutils. What would it take to fold it back into the main distribution? numpy.distutils (nee scipy_distutils) is not really a fork. It's just a set of extensions to distutils. One of the primary features is handling extensions with FORTRAN code. If you can convince Guido that standard distutils ought to handle FORTRAN, then we might have a deal. Some of the other features might be rolled into the standard library, but probably not on the 2.5 time frame. The useful ones might be the system_info stuff (which you have recently become far too familiar with :-)), the build_src capabilities, colored output, and the Configuration object. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant at ee.byu.edu Wed Mar 1 16:46:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 1 16:46:02 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: References: Message-ID: <4406400E.6080600@ee.byu.edu> Bill Baxter wrote: > The NumPy for Matlab Users's wiki is currently pleading to have > someone fill in "/*the idiom for supporting both matrices and arrays > in a function". > */Does anyone know what this is? I'm not sure what they want, exactly. Since a matrix is a subclass of an array you can treat them the same for the most part. The * and ** operators are the two that act differently on arrays and matrices so if you want to be sure then you use the functions dot and umath.multiply instead of the infix notation. If you want to convert arbitrary objects to "some-kind of array" then asanyarray(...) allows sub-classes of the array to pass to your function. To be perfectly careful, however, you will need to use explicity functions to perform any operation you may need. The other approach is to store the __array_wrap__ method of the input object (if there is any), convert everything to arrays using asarray() and then wrap the final result --- this is what ufuncs do internally. Again, I'm not sure what they mean? -Travis From robert.kern at gmail.com Wed Mar 1 16:52:32 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 1 16:52:32 2006 Subject: [Numpy-discussion] Re: The idiom for supporting matrices and arrays in a function In-Reply-To: References: Message-ID: Bill Baxter wrote: > The NumPy for Matlab Users's wiki is currently pleading to have someone > fill in "/*the idiom for supporting both matrices and arrays in a > function". > * /Does anyone know what this is? The subclassing functionality is rather new, so I don't know if the proper idioms have really been discovered. I would suggest converting to arrays using asarray() right at the beginning of the function. If you want to spit out matrices/what-have-you out again, then you will need to do some more work. I would suggest looking at the procedure that ufuncs do with __array_finalize__ and __array_priority__ and creating a pure Python reference implementation that others could use. It's possible that you would be able to turn it into a decorator. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant at ee.byu.edu Wed Mar 1 17:10:38 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 1 17:10:38 2006 Subject: [Numpy-discussion] Re: The idiom for supporting matrices and arrays in a function In-Reply-To: References: Message-ID: <440645D0.80701@ee.byu.edu> Robert Kern wrote: >Bill Baxter wrote: > > >>The NumPy for Matlab Users's wiki is currently pleading to have someone >>fill in "/*the idiom for supporting both matrices and arrays in a >>function". >>* /Does anyone know what this is? >> >> > >The subclassing functionality is rather new, so I don't know if the proper >idioms have really been discovered. I would suggest converting to arrays using >asarray() right at the beginning of the function. If you want to spit out >matrices/what-have-you out again, then you will need to do some more work. I >would suggest looking at the procedure that ufuncs do with __array_finalize__ >and __array_priority__ and creating a pure Python reference implementation that >others could use. It's possible that you would be able to turn it into a decorator. > > This is kind of emerging as the right strategy (although it's the __array_wrap__ and __array_priority__ methods that are actually relevant here. But, it is all rather new so we can forgive Robert this time ;-) ) The __array_priority__ is important if you've got "competing" wrappers (i.e. a 2-input, 1-output function) and don't know which __array_wrap__ to use for the output. In matlab you have one basic type of object. In NumPy you have a basic ndarray object that can be sub-classed which gives rise to all kinds of possibilities. I think we are still figuring out what the best thing to do is. Ufuncs convert everything to base-class arrays (what asarray does) and then use the __array_wrap__ of the input with the highest __array_priority__ to wrap all of the outputs (except that now output arrays passed in use their own __array_wrap__). So, I suppose following their example is a reasonable approach. Look in numpy/linlag/linlag.py for examples of using the asarray/__array_wrap__ strategy... The other approach is to let sub-classes through the array conversion (asanyarray), but this approach could rise to later errors if the sub-class redefines an operation you didn't expect it to, so is probably not safe unless all of your operations are functions that can handle any array subclass. And yes, I think a decorator could be written that would manage this. I'll leave that for the motivated... -Travis From skip at pobox.com Wed Mar 1 17:14:18 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed Mar 1 17:14:18 2006 Subject: [Numpy-discussion] fold distutils back into Python distro? In-Reply-To: <440638AE.6020504@ee.byu.edu> References: <17414.13849.523893.552309@montanaro.dyndns.org> <440638AE.6020504@ee.byu.edu> Message-ID: <17414.18081.540554.463915@montanaro.dyndns.org> Travis> And just for clarification. As far as I understand it, it's not Travis> technically a fork of distutils. It is based on distutils but Travis> just extends it in many ways (i.e. new subclasses that extend Travis> from the Distutils classes). Ah, okay. That's different. I was thinking it was a fork. Skip From skip at pobox.com Wed Mar 1 17:16:19 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed Mar 1 17:16:19 2006 Subject: [Numpy-discussion] Re: fold distutils back into Python distro? In-Reply-To: References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: <17414.18223.900432.86652@montanaro.dyndns.org> Robert> numpy.distutils (nee scipy_distutils) is not really a fork. It's Robert> just a set of extensions to distutils. One of the primary Robert> features is handling extensions with FORTRAN code. If you can Robert> convince Guido that standard distutils ought to handle FORTRAN, Robert> then we might have a deal. I see no reason why that wouldn't be possible in principle. After all, the bdist_* stuff keeps growing to handle a number of different package formats. Robert> Some of the other features might be rolled into the standard Robert> library, but probably not on the 2.5 time frame. The useful ones Robert> might be the system_info stuff (which you have recently become Robert> far too familiar with :-)), the build_src capabilities, colored Robert> output, and the Configuration object. Any idea if patches for each could be created fairly easily for each of these? Skip From haase at msg.ucsf.edu Wed Mar 1 18:14:02 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Mar 1 18:14:02 2006 Subject: [Numpy-discussion] numarray: need Float32 abs from array of type na.Complex64 or na.Complex32 Message-ID: <200603011813.40773.haase@msg.ucsf.edu> Hi, I just was hunting a very strange bug: I got weird results .... anyway. I found that I was using this line of code: if img.type() in (na.Complex64, na.Complex32): img = abs(na.asarray(img, na.Float32)) Now I don't understand how this code worked at all ? But I only saw those "weird results" after looking at the "dimmest corners" in my image and otherwise all images looked O.K. The goal here was to get a single precision absolute for either a complex64 or complex32 valued image array WITHOUT creating a temporary array. My thinking was that: img = na.abs( na.asarray(img, na.Complex32) ) would create a complete temporary version of img starts out being Complex64 Is this really the case ? Thanks for any hints, Sebastian Haase From gerard.vermeulen at grenoble.cnrs.fr Thu Mar 2 00:30:02 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Thu Mar 2 00:30:02 2006 Subject: [Numpy-discussion] Re: fold distutils back into Python distro? In-Reply-To: References: <17414.13849.523893.552309@montanaro.dyndns.org> Message-ID: <20060302092859.70abb8ce.gerard.vermeulen@grenoble.cnrs.fr> On Wed, 01 Mar 2006 18:21:41 -0600 Robert Kern wrote: > Some of the other features might be rolled into the standard library, but > probably not on the 2.5 time frame. The useful ones might be the system_info > stuff (which you have recently become far too familiar with :-)), the build_src > capabilities, colored output, and the Configuration object. IMO, the colored output should be an optional feature. My emacs buffers do not understand it and the only way to disable it is modifying the source as far as I remember. Gerard From hurak at control.felk.cvut.cz Thu Mar 2 03:23:03 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Thu Mar 2 03:23:03 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. Message-ID: Hello, Can anybody recommend the book: Python Scripting for Computational Science Series: Texts in Computational Science and Engineering, Vol. 3 Langtangen, Hans Petter 2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover ISBN: 3-540-29415-5 I am just a beginner in Python programming (but I master C, Matlab). I do research in applied numerical computing (oriented at control design). Is this a book for me? It is hard to guess just from the table contents. I especially doubt if the information on Numpy package is relevant at all with the wild development witnessed in the past recent months. However, peeling off these "numerical" parts, isn't buying some comprehensive Python book a better buy? Thanks for your tips, Zdenek Hurak From vinicius.lobosco at gmail.com Thu Mar 2 03:47:01 2006 From: vinicius.lobosco at gmail.com (Vinicius Lobosco) Date: Thu Mar 2 03:47:01 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. In-Reply-To: References: Message-ID: <200603021246.08706.vinicius.lobosco@gmail.com> Hi Zdenek! I have this book, and I like this a lot as the other book of Langtangen about Diffpack. I think he is very good in collecting in one place most of the aspects one would like to use within this subject. So if you think about implementing your software as a web service, to implement a GUI or to glue you C, C++ or Fortran software with Python or something else, this is the right book. And it is very well written. On the other hand, if you're just looking for information regarding NumPy (from a Matlab user perspective), it would be better to borrow it in the library and copy the only chapter about it. There is however, no book alternative. The best material is available on-line, to my knowledge. Second, you're also right that some stuffs are not updated anymore. Best regards, Vinicius On Thursday 02 March 2006 12.21, Zden?k Hur?k wrote: > Hello, > > Can anybody recommend the book: > > Python Scripting for Computational Science > Series: Texts in Computational Science and Engineering, Vol. 3 > Langtangen, Hans Petter > 2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover > ISBN: 3-540-29415-5 > > I am just a beginner in Python programming (but I master C, Matlab). I do > research in applied numerical computing (oriented at control design). Is > this a book for me? It is hard to guess just from the table contents. I > especially doubt if the information on Numpy package is relevant at all > with the wild development witnessed in the past recent months. However, > peeling off these "numerical" parts, isn't buying some comprehensive Python > book a better buy? > > Thanks for your tips, > Zdenek Hurak ------------------------ Vinicius Lobosco, PhD Process Intelligence www.paperplat.com +46 8 612 7803 +46 73 925 8476 (cell phone) Bj?rnn?sv?gen 21 SE-113 47 Stockholm, Sweden From cjw at sympatico.ca Thu Mar 2 05:39:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 2 05:39:01 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: <4406400E.6080600@ee.byu.edu> References: <4406400E.6080600@ee.byu.edu> Message-ID: <4406F52E.8000603@sympatico.ca> Travis Oliphant wrote: > Bill Baxter wrote: > >> The NumPy for Matlab Users's wiki is currently pleading to have >> someone fill in "/*the idiom for supporting both matrices and arrays >> in a function". */Does anyone know what this is? > > > I'm not sure what they want, exactly. > > Since a matrix is a subclass of an array you can treat them the same > for the most part. The * and ** operators are the two that act > differently on arrays and matrices so if you want to be sure then you > use the functions dot and umath.multiply instead of the infix notation. > > If you want to convert arbitrary objects to "some-kind of array" then > > asanyarray(...) How does asanyyarray() differ from asarray()? Colin W. > > allows sub-classes of the array to pass to your function. To be > perfectly careful, however, you will need to use explicity functions > to perform any operation you may need. > > The other approach is to store the __array_wrap__ method of the input > object (if there is any), convert everything to arrays using asarray() > and then wrap the final result --- this is what ufuncs do internally. > > Again, I'm not sure what they mean? > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From bsouthey at gmail.com Thu Mar 2 05:50:03 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Mar 2 05:50:03 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. In-Reply-To: References: Message-ID: Hi, I have the first edition which is very comprehensive on Python, numarray and Numeric - note that by NumPy he is means only Numeric and numarray and not the new numpy. If you have the basic of Python and numerical computing then it is a great book as it cover many different topics. For example, there is considerable detail on using Fortran and C/C++ with Numeric and numarray including examples. For the majority of the material in the book is still valid for the new Numpy. He does provide a number of advanced topics in Python, numarray and Numeric that are hard to find elsewhere. If you are going to extensive numerical work then it is really worth it. Bruce On 3/2/06, Zden?k Hur?k wrote: > Hello, > > Can anybody recommend the book: > > Python Scripting for Computational Science > Series: Texts in Computational Science and Engineering, Vol. 3 > Langtangen, Hans Petter > 2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover > ISBN: 3-540-29415-5 > > I am just a beginner in Python programming (but I master C, Matlab). I do > research in applied numerical computing (oriented at control design). Is > this a book for me? It is hard to guess just from the table contents. I > especially doubt if the information on Numpy package is relevant at all > with the wild development witnessed in the past recent months. However, > peeling off these "numerical" parts, isn't buying some comprehensive Python > book a better buy? > > Thanks for your tips, > Zdenek Hurak > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aisaac at american.edu Thu Mar 2 06:59:00 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu Mar 2 06:59:00 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: <4406F52E.8000603@sympatico.ca> References: <4406400E.6080600@ee.byu.edu><4406F52E.8000603@sympatico.ca> Message-ID: On Thu, 02 Mar 2006, "Colin J. Williams" apparently wrote: > How does asanyyarray() differ from asarray()? It leaves alone subclasses of ndarray. See example below. Cheers, Alan Isaac Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as N >>> y=N.mat([[1,2],[3,4]]) >>> z1=N.asarray(y) >>> z2=N.asanyarray(y) >>> z1 array([[1, 2], [3, 4]]) >>> z2 matrix([[1, 2], [3, 4]]) >>> From jmiller at stsci.edu Thu Mar 2 07:14:03 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Mar 2 07:14:03 2006 Subject: [Numpy-discussion] numarray: need Float32 abs from array of type na.Complex64 or na.Complex32 In-Reply-To: <200603011813.40773.haase@msg.ucsf.edu> References: <200603011813.40773.haase@msg.ucsf.edu> Message-ID: <44070B9C.4040805@stsci.edu> Sebastian Haase wrote: > Hi, > I just was hunting a very strange bug: I got weird results .... anyway. > > I found that I was using this line of code: > if img.type() in (na.Complex64, na.Complex32): > img = abs(na.asarray(img, na.Float32)) > > Now I don't understand how this code worked at all ? But I only saw those > "weird results" after looking at the "dimmest corners" in my image and > otherwise all images looked O.K. > > The goal here was to get a single precision absolute for either a complex64 or > complex32 valued image array WITHOUT creating a temporary array. > This idea sounds inconsistent to me. If the array is Complex64 and you pass it into asarray(,Float32), you're going to get a temporary. AND you're also truncating the imaginary part as you downcast to Float32... you're not getting a complex magnitude. Since abs() transforms from complex to real, you're going to get a temporary unless you get a little fancy, maybe like this: na.abs(img, img.real) # stores the abs() into the real component of the original array. img.real is a view not a copy. img.imag = 0 # optional step which makes the complex img array a real valued array with complex storage. img = img.real # just forget that img is using complex storage. Note that img is now discontiguous since no copy was made and there are still interleaved 0-valued imaginary components. So, I think there are two points to avoiding a copy here: (1) use the optional ufunc output parameter (2) store the abs() result into the .real view of the original complex array. > My thinking was that: > img = na.abs( na.asarray(img, na.Complex32) ) > would create a complete temporary version of img starts out being Complex64 > Is this really the case ? > Yes. From stefan at sun.ac.za Thu Mar 2 08:14:04 2006 From: stefan at sun.ac.za (Stefan van der Walt) Date: Thu Mar 2 08:14:04 2006 Subject: [Numpy-discussion] setting package path In-Reply-To: References: <20060228221937.GE10590@alpha> Message-ID: <20060302161325.GC12227@alpha> On Tue, Feb 28, 2006 at 11:34:43PM -0600, Pearu Peterson wrote: > >which calculates "somepath". Which is wrong: the docstring, the code > >or my interpretation? > > You have to read also following code: > > d1 = > os.path.join(d,'build','lib.%s-%s'%(get_platform(),sys.version[:3])) > if not os.path.isdir(d1): > d1 = os.path.dirname(d) # <- here we get "somepath/.." > sys.path.insert(0,d1) Pearu, Thanks for pointing this out. I was trying to figure out why my module's tests didn't run when I did, for example: $ python test_geom.py But now I realise it is because typical package layout is "somepath/some_dir/tests/test_file.py", not "somepath/some_dir/test_file.py". For example, we have "numpy/core/tests/test_umath.py". As a temporary workaround, set_local_path("../../..") works for me. Regards St?fan From jcollins_boulder at earthlink.net Thu Mar 2 09:01:08 2006 From: jcollins_boulder at earthlink.net (Jeffery D. Collins) Date: Thu Mar 2 09:01:08 2006 Subject: [Numpy-discussion] Numpy licenses Message-ID: <440724A4.4010804@earthlink.net> The SF site claims the following for Numeric/Numpy: License : OSI-Approved Open Source , GNU General Public License (GPL) Can this be changed to reflect LICENSE.TXT? Numpy (and the older Numeric-*) has never been GPL'd, correct? Also, a previous thread mentioned that the licensing terms for f2py will be changed from LGPL. In the svn trunk, some of its documentation still refers to LGPL, particularly f2py2e.py. Will this be changed before the next release? Thanks! Jeff From mfmorss at aep.com Thu Mar 2 09:03:02 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Thu Mar 2 09:03:02 2006 Subject: [Numpy-discussion] problems installing numpy on AIX Message-ID: I returned to this task. After some minor tweaks discussed here earlier, the build appears to succeed. There are two issues. Issue 1 is, to me, of unknown significance; it does not prevent the build. Issue 2 is that numpy can't be imported. There are problems with newdocs, type_check and numerictypes. The symptoms are shown below. Issue 1. creating build/temp.aix-5.2-2.4/build/src/numpy/core/src compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/app/sandbox/s625662/installed/include/python2.4 -c' cc_r: build/src/numpy/core/src/umathmodule.c "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. Issue 2. $ python Python 2.4.2 (#2, Feb 22 2006, 08:38:08) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy import core -> failed: import random -> failed: 'module' object has no attribute 'dtype' import lib -> failed: import linalg -> failed: import dft -> failed: Traceback (most recent call last): File "", line 1, in ? File "/mydirectory/lib/python2.4/site-packages/numpy/__init__.py", line 45, in ? import add_newdocs File "/mydirectory/lib/python2.4/site-packages/numpy/add_newdocs.py", line 2, in ? from lib import add_newdoc File "/mydirectory/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in ? from type_check import * File "/mydirectory/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in ? import numpy.core.numeric as _nx File "/mydirectory/lib/python2.4/site-packages/numpy/core/__init__.py", line 7, in ? import numerictypes as nt File "/mydirectory/lib/python2.4/site-packages/numpy/core/numerictypes.py", line 371, in ? _unicodesize = array('u','U').itemsize MemoryError >>> Mark F. Morss Principal Analyst, Market Risk American Electric Power From pfdubois at gmail.com Thu Mar 2 09:07:04 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Thu Mar 2 09:07:04 2006 Subject: [Numpy-discussion] Numpy licenses In-Reply-To: <440724A4.4010804@earthlink.net> References: <440724A4.4010804@earthlink.net> Message-ID: Numeric and numpy are not related from a licensing point of view. Numeric was released under an LLNL-written open source license. Travis wrote numpy and he can put any license he wants on that. On 02 Mar 2006 09:03:12 -0800, Jeffery D. Collins < jcollins_boulder at earthlink.net> wrote: > > The SF site claims the following for Numeric/Numpy: > > License : OSI-Approved Open Source > , GNU > General Public License (GPL) > > > > Can this be changed to reflect LICENSE.TXT? Numpy (and the older > Numeric-*) has never been GPL'd, correct? > > Also, a previous thread mentioned that the licensing terms for f2py will > be changed from LGPL. In the svn trunk, some of its documentation still > refers to LGPL, particularly f2py2e.py. Will this be changed before > the next release? > > Thanks! > > Jeff > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfmorss at aep.com Thu Mar 2 09:20:01 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Thu Mar 2 09:20:01 2006 Subject: [Numpy-discussion] problems installing numpy on AIX In-Reply-To: Message-ID: Minor correction. Mark F. Morss Principal Analyst, Market Risk 614-583-6757 Audinet 220-6757 mfmorss at aep.com Sent by: numpy-discussion- To admin at lists.sourc numpy-discussion eforge.net cc 03/02/2006 11:59 AM Subject [Numpy-discussion] problems installing numpy on AIX I returned to this task. After some minor tweaks discussed here earlier, the build appears to succeed. There are two issues. Issue 1 is, to me, of unknown significance; it does not prevent the build. Issue 2 is that numpy can't be imported. There are problems with newdocs, type_check and numerictypes. The symptoms are shown below. Issue 1. creating build/temp.aix-5.2-2.4/build/src/numpy/core/src compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/mydirectory/include/python2.4 -c' cc_r: build/src/numpy/core/src/umathmodule.c "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. Issue 2. $ python Python 2.4.2 (#2, Feb 22 2006, 08:38:08) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy import core -> failed: import random -> failed: 'module' object has no attribute 'dtype' import lib -> failed: import linalg -> failed: import dft -> failed: Traceback (most recent call last): File "", line 1, in ? File "/mydirectory/lib/python2.4/site-packages/numpy/__init__.py", line 45, in ? import add_newdocs File "/mydirectory/lib/python2.4/site-packages/numpy/add_newdocs.py", line 2, in ? from lib import add_newdoc File "/mydirectory/lib/python2.4/site-packages/numpy/lib/__init__.py", line 5, in ? from type_check import * File "/mydirectory/lib/python2.4/site-packages/numpy/lib/type_check.py", line 8, in ? import numpy.core.numeric as _nx File "/mydirectory/lib/python2.4/site-packages/numpy/core/__init__.py", line 7, in ? import numerictypes as nt File "/mydirectory/lib/python2.4/site-packages/numpy/core/numerictypes.py", line 371, in ? _unicodesize = array('u','U').itemsize MemoryError >>> Mark F. Morss Principal Analyst, Market Risk American Electric Power ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From robert.kern at gmail.com Thu Mar 2 09:26:10 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 2 09:26:10 2006 Subject: [Numpy-discussion] Re: Numpy licenses In-Reply-To: <440724A4.4010804@earthlink.net> References: <440724A4.4010804@earthlink.net> Message-ID: Jeffery D. Collins wrote: > The SF site claims the following for Numeric/Numpy: > > License : OSI-Approved Open Source > , GNU > General Public License (GPL) > > > > Can this be changed to reflect LICENSE.TXT? Numpy (and the older > Numeric-*) has never been GPL'd, correct? It's possible that there used to be some code checked into the repository that was GPLed. It was certainly never a part of Numeric, though. What worries me more is that someone thought Numeric et al. belong to the Multi-User Dungeons topic. I would like to stress, though, that Sourceforge is *not* the place for information on the new NumPy. The only remaining use for it is the mailing list and downloads, and I hope that we can start using PyPI instead for placing our distributions. > Also, a previous thread mentioned that the licensing terms for f2py will > be changed from LGPL. In the svn trunk, some of its documentation still > refers to LGPL, particularly f2py2e.py. Will this be changed before > the next release? Done. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From cookedm at physics.mcmaster.ca Thu Mar 2 10:13:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Mar 2 10:13:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <440530AC.2010000@enthought.com> (eric jones's message of "Tue, 28 Feb 2006 23:27:08 -0600") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: eric jones writes: > Travis Oliphant wrote: > >> Weave can still help with the "auto-compilation" of the specific >> library for your type. Ultimately such code will be faster than >> NumPy can every be. > > Yes. weave.blitz() can be used to do the equivalent of this lazy > evaluation for you in many cases without much effort. For example: > > import weave > from scipy import arange > > a = arange(1e7) > b = arange(1e7) > c=2.0*a+3.0*b > > # or with weave > weave.blitz("c=2.0*a+3.0*b") > > As Paul D. mentioned.what Tim outlined is essentially template > expressions in C++. blitz++ (http://www.oonumerics.org/blitz/) is a > C++ template expressions library for array operations, and weave.blitz > translates a Numeric expression into C++ blitz code. For the example > above on large arrays, you get about a factor of 4 speed up on large > arrays. (Notice, the first time you run the example it will be much > slower because of compile time. Use timings from subsequent runs.) > > C:\temp>weave_time.py > Expression: c=2.0*a+3.0*b > Numeric: 0.678311899322 > Weave: 0.162177084984 > Speed-up: 4.18253848494 > > All this to say, I think weave basically accomplishes what Tim wants > with a different mechanism (letting C++ compilers do the optimization > instead of writing this optimization at the python level). It does > require a compiler on client machines in its current form (even that > can be fixed...), but I think it might prove faster than > re-implementing a numeric expression compiler at the python level > (though that sounds fun as well). I couldn't leave it at that :-), so I wrote my bytecode idea up. You can grab it at http://arbutus.mcmaster.ca/dmc/software/numexpr-0.1.tar.gz Here are the performance numbers (note I use 1e6 elements instead of 1e7) cookedm at arbutus$ py -c 'import numexpr.timing; numexpr.timing.compare()' Expression: b*c+d*e numpy: 0.0934900999069 Weave: 0.0459051132202 Speed-up of weave over numpy: 2.03659447388 numexpr: 0.0467489004135 Speed-up of numexpr over numpy: 1.99983527056 Expression: 2*a+3*b numpy: 0.0784019947052 Weave: 0.0292909860611 Speed-up of weave over numpy: 2.67665945222 numexpr: 0.0323888063431 Speed-up of numexpr over numpy: 2.42065094572 Wow. On par with weave.blitz, and no C++ compiler!!! :-) You use it like this: from numexpr import numexpr func = numexpr("2*a+3*b") a = arange(1e6) b = arange(1e6) c = func(a, b) Alternatively, you can use it like weave.blitz, like this: from numexpr import evaluate a = arange(1e6) b = arange(1e6) c = evaluate("2*a+3*b") How it works ============ Well, first of all, it only works with the basic operations (+, -, *, and /), and only on real constants and arrays of doubles. (These restrictions are mainly for lack of time, of course.) Given an expression, it first compiles it to a program written in a small bytecode. Here's what it looks like: In [1]: from numexpr import numexpr In [2]: numexpr("2*a+3*b", precompiled=True) # precompiled=True just returns the program before it's made into a # bytecode object, so you can what it looks like Out[2]: [('mul_c', Register(0), Register(1, a), Constant(0, 2)), ('mul_c', Temporary(3), Register(2, b), Constant(1, 3)), ('add', Register(0), Register(0), Temporary(3))] In [3]: c = numexpr("2*a+3*b") In [4]: c.program Out[4]: '\x08\x00\x01\x00\x08\x03\x02\x01\x02\x00\x00\x03' In [5]: c.constants Out[5]: array([ 2., 3.]) In [6]: c.n_temps Out[6]: 1 In [7]: type(c) Out[7]: The bytecode program is a string of 4-character groups, where the first character of each group is the opcode, the second is the "register" to store in, and the third and fourth are the register numbers to use as arguments. Inputs and outputs are assigned to registers (so in the example above, the output is Register(0), and the two inputs are Register(1) and Register(2)), and temporaries are in remaining registers. The group of registers is actually an array of double*. The secret behind the speed is that the bytecode program is run blocked: it's run repeatedly, operating on 128 elements of each input array at at time (then 8 elements then 1 element for the leftover). This does a tradeoff between cache misses (the arguments for each time through the program will most likely be able to fit in the cache each time), and the overhead of the branching from evaluating the program. It's like doing this in numpy: c[0:128] = 2*a[0:128] + 3*b[0:128] c[128:256] = 2*a[128:256] + 3*b[128:256] etc. I could see this getting a nice speedup from using a CPU's vector instructions. For instance, it should be pretty easy to use liboil for the intermediate vector evaluations. If people think it's useful enough, I can check it into scipy's sandbox. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Chris.Barker at noaa.gov Thu Mar 2 11:41:00 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 2 11:41:00 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <440530AC.2010000@enthought.com> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <44074A05.30700@noaa.gov> eric jones wrote: > All this to say, I think weave basically accomplishes what Tim wants > with a different mechanism (letting C++ compilers do the optimization > instead of writing this optimization at the python level). It does > require a compiler on client machines in its current form (even that can > be fixed...) Perhaps by teaching Psyco about nd-arrays? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Mar 2 13:13:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 2 13:13:04 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <44075FA9.5040104@ee.byu.edu> David M. Cooke wrote: >I couldn't leave it at that :-), so I wrote my bytecode idea up. > >You can grab it at http://arbutus.mcmaster.ca/dmc/software/numexpr-0.1.tar.gz > > Awesome. Please do check it in to the sandbox. -Travis From schofield at ftw.at Thu Mar 2 13:25:01 2006 From: schofield at ftw.at (Ed Schofield) Date: Thu Mar 2 13:25:01 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <3FAADB1F-9C8C-4E92-91AA-244D15E32C3B@ftw.at> On 02/03/2006, at 7:12 PM, David M. Cooke wrote: > I couldn't leave it at that :-), so I wrote my bytecode idea up. > > You can grab it at http://arbutus.mcmaster.ca/dmc/software/ > numexpr-0.1.tar.gz I'd had a brief look at the code, and I think it's great! I particularly like the power available with the "numexpr.evaluate (string_expression)" syntax. I'd fully support your adding it to the scipy sandbox. I'm sure we could fill it out with more operators and data types quite fast. Great work! -- Ed From tim.hochberg at cox.net Thu Mar 2 15:21:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 2 15:21:04 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <44074A05.30700@noaa.gov> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <44074A05.30700@noaa.gov> Message-ID: <44077DA3.70801@cox.net> Christopher Barker wrote: > eric jones wrote: > >> All this to say, I think weave basically accomplishes what Tim wants >> with a different mechanism (letting C++ compilers do the optimization >> instead of writing this optimization at the python level). It does >> require a compiler on client machines in its current form (even that >> can be fixed...) > > > Perhaps by teaching Psyco about nd-arrays? Psyco is unlikely to ever be much good for floating point stuff. This is because it only knows about longs/pointers and arrays of longs/pointers. The floating point support it has at present involves breaking a double into two long sized chunks. Then whenever you need to manipulate the float you have to find the pieces and put them back together[1]. All of this means that float support is much poorer then integer support in Psyco. In theory this could be fixed, but since Armin's no longer actively working on Psyco, just maintaining it, I don't see this changing. Perhaps when Psyco like technology gets incorporated in PyPy it will include better support for floating point. Regards, -tim [1] This has all gotten rather fuzzy too me now, it's been a while since I looked at it and my understanding was at best fragile anyway. From tim.hochberg at cox.net Thu Mar 2 15:49:11 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 2 15:49:11 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <4407843A.5010300@cox.net> David M. Cooke wrote: >eric jones writes: > > > >>Travis Oliphant wrote: >> >> >> >>>Weave can still help with the "auto-compilation" of the specific >>>library for your type. Ultimately such code will be faster than >>>NumPy can every be. >>> >>> >>Yes. weave.blitz() can be used to do the equivalent of this lazy >>evaluation for you in many cases without much effort. For example: >> >>import weave >>from scipy import arange >> >>a = arange(1e7) >>b = arange(1e7) >>c=2.0*a+3.0*b >> >># or with weave >>weave.blitz("c=2.0*a+3.0*b") >> >>As Paul D. mentioned.what Tim outlined is essentially template >>expressions in C++. blitz++ (http://www.oonumerics.org/blitz/) is a >>C++ template expressions library for array operations, and weave.blitz >>translates a Numeric expression into C++ blitz code. For the example >>above on large arrays, you get about a factor of 4 speed up on large >>arrays. (Notice, the first time you run the example it will be much >>slower because of compile time. Use timings from subsequent runs.) >> >>C:\temp>weave_time.py >>Expression: c=2.0*a+3.0*b >>Numeric: 0.678311899322 >>Weave: 0.162177084984 >>Speed-up: 4.18253848494 >> >>All this to say, I think weave basically accomplishes what Tim wants >>with a different mechanism (letting C++ compilers do the optimization >>instead of writing this optimization at the python level). It does >>require a compiler on client machines in its current form (even that >>can be fixed...), but I think it might prove faster than >>re-implementing a numeric expression compiler at the python level >>(though that sounds fun as well). >> >> > >I couldn't leave it at that :-), so I wrote my bytecode idea up. > >You can grab it at http://arbutus.mcmaster.ca/dmc/software/numexpr-0.1.tar.gz > >Here are the performance numbers (note I use 1e6 elements instead of >1e7) > >cookedm at arbutus$ py -c 'import numexpr.timing; numexpr.timing.compare()' >Expression: b*c+d*e >numpy: 0.0934900999069 >Weave: 0.0459051132202 >Speed-up of weave over numpy: 2.03659447388 >numexpr: 0.0467489004135 >Speed-up of numexpr over numpy: 1.99983527056 > >Expression: 2*a+3*b >numpy: 0.0784019947052 >Weave: 0.0292909860611 >Speed-up of weave over numpy: 2.67665945222 >numexpr: 0.0323888063431 >Speed-up of numexpr over numpy: 2.42065094572 > >Wow. On par with weave.blitz, and no C++ compiler!!! :-) > > That's awesome! I was also tempted by this, but I never got beyond prototyping some stuff in Python. >You use it like this: > >from numexpr import numexpr > >func = numexpr("2*a+3*b") > >a = arange(1e6) >b = arange(1e6) >c = func(a, b) > > Does this just uses the order that variable are initially used in the expression to determine the input order? I'm not sure I like that. It is very convenient for simple expressions though. >Alternatively, you can use it like weave.blitz, like this: > >from numexpr import evaluate > >a = arange(1e6) >b = arange(1e6) >c = evaluate("2*a+3*b") > > That's pretty sweet. And I see that you cache the expressions, so it should be pretty fast if you need to loop. [snip details] >If people think it's useful enough, I can check it into scipy's sandbox. > > Definitely check it in. It won't compile here with VC7, but I'll see if I can figure out why. This is probably thinking two far ahead, but an interesting thing to try would be adding conditional expressions: c = evaluate("(2*a + b) if (a > b) else (2*b + a)") If that could be made to work, and work fast, it would save both memory and time in those cases where you have to vary the computation based on the value. At present I end up computing the full arrays for each case and then choosing which result to use based on a mask, so it takes three times as much space as doing it element by element. -tim From oliphant at ee.byu.edu Thu Mar 2 16:05:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 2 16:05:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> Message-ID: <4407881A.6090108@ee.byu.edu> David M. Cooke wrote: >I couldn't leave it at that :-), so I wrote my bytecode idea up. > > > I like what you've done. It makes me wonder what could be done with just a Python-level solution (that might be easier to generalize to more data-types and operators). You've definitely given us all some great stuff to play with. -Travis From cookedm at physics.mcmaster.ca Thu Mar 2 16:20:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Mar 2 16:20:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4407843A.5010300@cox.net> (Tim Hochberg's message of "Thu, 02 Mar 2006 16:48:10 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> Message-ID: Tim Hochberg writes: > David M. Cooke wrote: > >>Wow. On par with weave.blitz, and no C++ compiler!!! :-) >> >> > That's awesome! I was also tempted by this, but I never got beyond > prototyping some stuff in Python. > >>You use it like this: >> >>from numexpr import numexpr >> >>func = numexpr("2*a+3*b") >> >>a = arange(1e6) >>b = arange(1e6) >>c = func(a, b) >> >> > Does this just uses the order that variable are initially used in the > expression to determine the input order? I'm not sure I like that. It > is very convenient for simple expressions though. By default, it does lexical order (whatever .sort() on the list of variable names gives). You can specify the order with func = numexpr("2*a+3*b", input_order=('a', 'b')) >>Alternatively, you can use it like weave.blitz, like this: >> >>from numexpr import evaluate >> >>a = arange(1e6) >>b = arange(1e6) >>c = evaluate("2*a+3*b") >> >> > That's pretty sweet. And I see that you cache the expressions, so it > should be pretty fast if you need to loop. > > [snip details] > >>If people think it's useful enough, I can check it into scipy's sandbox. >> >> > Definitely check it in. It won't compile here with VC7, but I'll see > if I can figure out why. It's in. Check the setup.py -- I left some gcc'isms in there (-funroll-loops). That should be conditional on the compiler. > This is probably thinking two far ahead, but an interesting thing to > try would be adding conditional expressions: > > c = evaluate("(2*a + b) if (a > b) else (2*b + a)") > > If that could be made to work, and work fast, it would save both > memory and time in those cases where you have to vary the computation > based on the value. At present I end up computing the full arrays for > each case and then choosing which result to use based on a mask, so it > takes three times as much space as doing it element by element. It's on my list of things to think about, along with implementing reductions (sum, in particular). It'd probably look more like c = evaluate("where(a > b, 2*a+b, 2*b+a)") because of the vector nature of the code. Doing the "if" elementwise would mean the bytecode program would have to be run for each element, instead of on a vector of some fixed length. I wrote up a bit on my blog at http://isobaric.blogspot.com/ (finally, something to blog about :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant.travis at ieee.org Thu Mar 2 17:05:19 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 2 17:05:19 2006 Subject: [Numpy-discussion] Re: problems installing numpy on AIX In-Reply-To: References: Message-ID: mfmorss at aep.com wrote: > > I returned to this task. > > After some minor tweaks discussed here earlier, the build appears to > succeed. There are two issues. Issue 1 is, to me, of unknown > significance; it does not prevent the build. These errors seem to indicate that while the build finished it did not result in useful code. Could you post the output of the build log? It may contain warnings that are relevant. I don't understand why you are getting the errors you are getting. They seem unusual. Make sure you are running from outside the directory you built in (i.e. don't run in the numpy source directory). > > Issue 2 is that numpy can't be imported. There are problems with newdocs, > type_check and numerictypes. > > The symptoms are shown below. > > > Issue 1. > > creating build/temp.aix-5.2-2.4/build/src/numpy/core/src > compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include > -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include > -I/mydirectory/include/python2.4 -c' > cc_r: build/src/numpy/core/src/umathmodule.c > "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) > Function argument assignment between types "long double*" and "double*" is > not allowed. This is a strange as it seems your math library has the wrong definition for modfl. Can you verify what section of code this is balking on? Thanks for the attempts, -Travis From wbaxter at gmail.com Thu Mar 2 17:23:23 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu Mar 2 17:23:23 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: References: <4406400E.6080600@ee.byu.edu> <4406F52E.8000603@sympatico.ca> Message-ID: Thanks for all the info folks. This all sounds a little more complicated than is warranted for the "Numpy for Matlab Users" wiki page, though. So I think I'll just comment on the presence of asarray, asanyarray, and asmatrix for type conversions. --bb On 3/3/06, Alan G Isaac wrote: > > On Thu, 02 Mar 2006, "Colin J. Williams" apparently wrote: > > How does asanyyarray() differ from asarray()? > > It leaves alone subclasses of ndarray. See example below. > Cheers, > Alan Isaac > > Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit > (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy as N > >>> y=N.mat([[1,2],[3,4]]) > >>> z1=N.asarray(y) > >>> z2=N.asanyarray(y) > >>> z1 > array([[1, 2], > [3, 4]]) > >>> z2 > matrix([[1, 2], > [3, 4]]) > >>> > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Mar 2 20:18:03 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Mar 2 20:18:03 2006 Subject: [Numpy-discussion] Matlab page on scipy wiki In-Reply-To: <20060210122242.GA21950@sun.ac.za> References: <20060210122242.GA21950@sun.ac.za> Message-ID: <4407C359.3030601@msg.ucsf.edu> Hi, I noted on http://www.scipy.org/Wiki/NumPy_for_Matlab_Users matlab a(1:5,:) numpy a[0:4] or a[0:4,:] a[0:4] or a[0:4,:] the first five rows of a I think this is wrong! in numpy it would be: a[:5] (or a[0:5] ) for the first five elements To the best of my knowledge (I have never used Matlab myself!) this is one of the biggest points of confusion for Matlab users !! WHY DOES a[4:6] NOT INCLUDE THE ELEMENTS 4,5 *AND* 6 ??? The only explanation I have is that its a) like a C/C++-for-loop (for(i=4;i<6;i++) --> note the '<' (not '<=') b) it then also always is true that: "last" minus first is equal to number-of-elements (example: 6-4 = 2) c) it turns out from experience that this convention WILL save you lots of '-1' and '+1' in your code (actually almost all of them) [if I see a "+1" in a Matlab person's numpy code I can almost always guess that he/she made a mistake !] Maybe this paragraph could be added to the wiki ... Thanks for this wiki page - I think I looks great - Sebastian Haase From jh at oobleck.astro.cornell.edu Thu Mar 2 20:24:02 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Thu Mar 2 20:24:02 2006 Subject: [Numpy-discussion] Numpy licenses In-Reply-To: <20060302170802.597DD88A09@sc8-sf-spam1.sourceforge.net> (numpy-discussion-request@lists.sourceforge.net) References: <20060302170802.597DD88A09@sc8-sf-spam1.sourceforge.net> Message-ID: <200603030423.k234NgCD004844@oobleck.astro.cornell.edu> > Travis wrote numpy and he can put any license he wants on that. Really, he didn't take any lines of code from its predecessors? My understanding was that much borrowing occurred. This is good, and is the strength of OSS, but it does require attention to licensing for derived works. Travis, if I'm mistaken, please correct me. --jh-- From wbaxter at gmail.com Thu Mar 2 20:48:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Thu Mar 2 20:48:01 2006 Subject: [Numpy-discussion] Matlab page on scipy wiki In-Reply-To: <4407C359.3030601@msg.ucsf.edu> References: <20060210122242.GA21950@sun.ac.za> <4407C359.3030601@msg.ucsf.edu> Message-ID: Ah, right you are. Fixed. Thanks. I guess the python version could also be written a[:5]. --bb On 3/3/06, Sebastian Haase wrote: > > Hi, > I noted on > http://www.scipy.org/Wiki/NumPy_for_Matlab_Users > > matlab a(1:5,:) > numpy a[0:4] or a[0:4,:] a[0:4] or a[0:4,:] > the first five rows of a > > I think this is wrong! > > in numpy it would be: a[:5] (or a[0:5] ) for the first five elements > > To the best of my knowledge (I have never used Matlab myself!) this is > one of the biggest points of confusion for Matlab users !! > WHY DOES a[4:6] NOT INCLUDE THE ELEMENTS 4,5 *AND* 6 ??? > > The only explanation I have is that its > a) like a C/C++-for-loop (for(i=4;i<6;i++) --> note the '<' (not '<=') > b) it then also always is true that: "last" minus first is equal to > number-of-elements (example: 6-4 = 2) > c) it turns out from experience that this convention WILL save you lots > of '-1' and '+1' in your code (actually almost all of them) > [if I see a "+1" in a Matlab person's numpy code I can almost always > guess that he/she made a mistake !] > > Maybe this paragraph could be added to the wiki ... > > Thanks for this wiki page - I think I looks great > > - Sebastian Haase > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From vidar+list at 37mm.no Fri Mar 3 00:29:06 2006 From: vidar+list at 37mm.no (Vidar Gundersen) Date: Fri Mar 3 00:29:06 2006 Subject: [Numpy-discussion] Matlab page on scipy wiki In-Reply-To: <4407C359.3030601@msg.ucsf.edu> (Sebastian Haase's message of "Thu, 02 Mar 2006 20:17:29 -0800") References: <20060210122242.GA21950@sun.ac.za> <4407C359.3030601@msg.ucsf.edu> Message-ID: ===== Original message from Sebastian Haase | 3 Mar 2006: > I noted on > http://www.scipy.org/Wiki/NumPy_for_Matlab_Users there is one other difference, which i think is not so easily seen, and not noted on the above mentioned page: a(:) in Matlab, will flatten the array by following each column, whereas a.flatten() in NumPy will walk through the matrix like we read comics, putting each row after each other: In [1]: a = arange(9).reshape(3,-1) In [2]: a Out[2]: array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) In [3]: a.flatten() Out[3]: array([0, 1, 2, 3, 4, 5, 6, 7, 8]) In [4]: a.flatten(1) # same output as a(:) in Matlab Out[4]: array([0, 3, 6, 1, 4, 7, 2, 5, 8]) the user have to be aware of this when reshaping matrices also. I've updated my similar Matlab-Python reference to cover NumPy: http://37mm.no/mpy/matlab-numpy.html (example plot graphics is not yet included in the HTML version.) See other command equivalents, like IDL or R commands in NumPy, here: http://37mm.no/matlab-python-xref.html Vidar___ From gerard.vermeulen at grenoble.cnrs.fr Fri Mar 3 00:45:01 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Fri Mar 3 00:45:01 2006 Subject: [Numpy-discussion] polynomial subtraction buglet in 0.9.6.2193 Message-ID: <20060303094406.1d8b2a5e.gerard.vermeulen@grenoble.cnrs.fr> I found the following buglet in the polynomial subtraction of numpy-0.9.6.2193: diff -u polynomial.py.dtype polynomial.py --- polynomial.py.dtype 2006-03-03 09:34:40.000000000 +0100 +++ polynomial.py 2006-03-03 09:36:29.000000000 +0100 @@ -265,10 +265,10 @@ if diff == 0: return a1 - a2 elif diff > 0: - zr = NX.zeros(diff, a1) + zr = NX.zeros(diff, a1.dtype) val = NX.concatenate((zr, a1)) - a2 else: - zr = NX.zeros(abs(diff), a2) + zr = NX.zeros(abs(diff), a2.dtype) val = a1 - NX.concatenate((zr, a2)) if truepoly: val = poly1d(val) Gerard From oliphant.travis at ieee.org Fri Mar 3 08:28:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 3 08:28:02 2006 Subject: [Numpy-discussion] Numpy licenses In-Reply-To: <200603030423.k234NgCD004844@oobleck.astro.cornell.edu> References: <20060302170802.597DD88A09@sc8-sf-spam1.sourceforge.net> <200603030423.k234NgCD004844@oobleck.astro.cornell.edu> Message-ID: <44086E3F.9020302@ieee.org> Joe Harrington wrote: >> Travis wrote numpy and he can put any license he wants on that. >> > > Joe is basically right. I started from the Numeric code and then changed most of it at least a little bit :-), but there are still recognizable chunks from the old Numeric code base. Then, I borrowed mostly ideas from Numarray except for the _sortmodule.c which I pretty-much grabbed whole-cloth. So, I was always under the impression that I was modifying already produced code. The license should continue to be similar to what it was. -Travis From tim.hochberg at cox.net Fri Mar 3 09:24:10 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 09:24:10 2006 Subject: [Numpy-discussion] Truth values of arrays: Message-ID: <44087B9E.5020903@cox.net> I'm not a fan of the following behaviour, which I just noticed: >>> a = arange(1); b = arange(1) >>> cmp(a,b) 0 >>> not (a == b) False My feeling is that only shape-() arrays should be usable as truth values. Allowing size-1 arrays to be compared as well is just adding a weird, not particularly useful, corner case that's going to bite someone eventually. -tim From mfmorss at aep.com Fri Mar 3 09:26:05 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Fri Mar 3 09:26:05 2006 Subject: [Numpy-discussion] Re: problems installing numpy on AIX In-Reply-To: <440870AF.5000308@ieee.org> Message-ID: Well, thank you Travis, that was most exceedingly helpful! With that, numpy installs, and python imports it without complaint. Next: SciPy. Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To mfmorss at aep.com 03/03/2006 11:37 cc AM Subject Re: [Numpy-discussion] Re: problems installing numpy on AIX mfmorss at aep.com wrote: > (See attached file: Numpy_install.txt) > > Try changing the line 371 in numerictypes.py to _unicodesize = array('u','U1').itemsize and see what that produces... -Travis From tim.hochberg at cox.net Fri Mar 3 09:39:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 09:39:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4407881A.6090108@ee.byu.edu> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407881A.6090108@ee.byu.edu> Message-ID: <44087F21.1000203@cox.net> Travis Oliphant wrote: > David M. Cooke wrote: > >> I couldn't leave it at that :-), so I wrote my bytecode idea up. >> >> >> > I like what you've done. It makes me wonder what could be done with > just a Python-level solution (that might be easier to generalize to > more data-types and operators). I tried hand coding stuff at the pure Python level to see how close I could come to David's results. I performed the operations in chunks (larger chunks than he did so that Python overhead didn't kill me) For the second case he presents (2*a+3*b), I managed to get a 70% speedup. While not negligible, it's a far cry from the 180% speedup I saw with David's code for the same result. So while doing things at the Python level would certainly be easier, the payoff isn't likely to be nearly as big. > You've definitely given us all some great stuff to play with. Indeed. From oliphant at ee.byu.edu Fri Mar 3 09:54:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 3 09:54:05 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <44087F21.1000203@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407881A.6090108@ee.byu.edu> <44087F21.1000203@cox.net> Message-ID: <44088298.7020606@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> David M. Cooke wrote: >> >>> I couldn't leave it at that :-), so I wrote my bytecode idea up. >>> >>> >>> >> I like what you've done. It makes me wonder what could be done with >> just a Python-level solution (that might be easier to generalize to >> more data-types and operators). > > > I tried hand coding stuff at the pure Python level to see how close I > could come to David's results. I performed the operations in chunks > (larger chunks than he did so that Python overhead didn't kill me) For > the second case he presents (2*a+3*b), I managed to get a 70% speedup. > While not negligible, it's a far cry from the 180% speedup I saw with > David's code for the same result. So while doing things at the Python > level would certainly be easier, the payoff isn't likely to be nearly > as big. That's good information. Still, it might be useful for more general-operators and data-types to have something that does essentially this in Python only. Then, if you have a simpler expression with a few select data-types you could have the big-speed up with something more specific. -Travis From tim.hochberg at cox.net Fri Mar 3 10:20:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 10:20:01 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> Message-ID: <44088897.6070802@cox.net> David M. Cooke wrote: >Tim Hochberg writes: > > > >>David M. Cooke wrote: >> >> > >It's in. Check the setup.py -- I left some gcc'isms in there >(-funroll-loops). That should be conditional on the compiler. > > I does whine about that, but it ignores it, so I did to ;). The misplaced declaration of dims I mentioned earlier was the real culprit. >>This is probably thinking two far ahead, but an interesting thing to >>try would be adding conditional expressions: >> >>c = evaluate("(2*a + b) if (a > b) else (2*b + a)") >> >>If that could be made to work, and work fast, it would save both >>memory and time in those cases where you have to vary the computation >>based on the value. At present I end up computing the full arrays for >>each case and then choosing which result to use based on a mask, so it >>takes three times as much space as doing it element by element. >> >> > >It's on my list of things to think about, along with implementing >reductions (sum, in particular). It'd probably look more like > >c = evaluate("where(a > b, 2*a+b, 2*b+a)") > >because of the vector nature of the code. Doing the "if" elementwise >would mean the bytecode program would have to be run for each element, >instead of on a vector of some fixed length. > > That makes sense. One thought I had with respect to the various numpy functions (sin, cos, pow, etc) was to just have the bytecodes: call_unary_function, function_id, store_in, source call_binary_function, function_id, store_in, source1, source2 call_trinary_function, function_id, store_in, source1, source2, source3 Then just store pointers to the functions in relevant tables. In it's most straightforward form, you'd need 6 character chunks of bytecode instead of four. However, if that turns out to slow everything else down I think it could be packed down to 4 again. The function_ids could probably be packed into the opcode (as long as we stay below 200 or so functions, which is probably safe), the other way to pack things down is to require that one of the sources for trinary functions is always a certain register (say register-0). That requires a bit more cleverness at the compiler level, but is probably feasible. OT, stupid subversion question: where do I find the sandbox or whereever this got checked in? -tim From ndarray at mac.com Fri Mar 3 10:36:01 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 3 10:36:01 2006 Subject: [Numpy-discussion] Truth values of arrays: In-Reply-To: <44087B9E.5020903@cox.net> References: <44087B9E.5020903@cox.net> Message-ID: On 3/3/06, Tim Hochberg wrote: > ... > My feeling is that only shape-() arrays should be usable as truth > values. Allowing size-1 arrays to be compared as well is just adding a > weird, not particularly useful, corner case that's going to bite someone > eventually. I agree, but size-1 arrays are also special with respect to broadcasting where they behave much like scalars: >>> x = arange(5) >>> x + [1] array([1, 2, 3, 4, 5]) From tim.hochberg at cox.net Fri Mar 3 10:36:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 10:36:06 2006 Subject: [Numpy-discussion] Truth values of arrays: In-Reply-To: References: <44087B9E.5020903@cox.net> Message-ID: <44088C83.3090703@cox.net> Sasha wrote: >On 3/3/06, Tim Hochberg wrote: > > >> ... >>My feeling is that only shape-() arrays should be usable as truth >>values. Allowing size-1 arrays to be compared as well is just adding a >>weird, not particularly useful, corner case that's going to bite someone >>eventually. >> >> > >I agree, but size-1 arrays are also special with respect to >broadcasting where they behave much like scalars: > > > >>>>x = arange(5) >>>>x + [1] >>>> >>>> >array([1, 2, 3, 4, 5]) > > True, but there's a good reason for that in that there's really not a good way to say "broadcast on this dimension", so being of dimension 1 has to stand in for that. There are plenty of good ways to spell compare the first (and only) element of this array however. The obvious being (a[0] References: <20060210122242.GA21950@sun.ac.za> <4407C359.3030601@msg.ucsf.edu> Message-ID: <44089706.2020700@noaa.gov> Bill Baxter wrote: > Ah, right you are. Fixed. Thanks. > I guess the python version could also be written a[:5]. I've just added that -- it's a very useful idiom. by the way, to prevent [:5] from being interpreted as a link by the Wiki software, I needed to wrap it in ``, to make it look like code. I think that's a good idea for all the code in that table. Maybe we can go in a bit at a time and add the ``. I did a few lines near the one I was working on anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tim.hochberg at cox.net Fri Mar 3 12:13:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 12:13:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> Message-ID: <4408A308.3010003@cox.net> FYI, In my local copy of numexpr, I added a few unary functions just to see how it would be. Pretty easy reall. I think binary functions will also be easy. Trinary functions (e.g., where) will be a bit harder unless we move to 5 byte opcodes. At present I'm not doing anything fancy like looking up functions in a table, I'm just giving each it's own opcodes. If the code gets big enough that it starts falling out of the cache, we might consider that, but adding opcodes is simpler. I also fixed a couple places where constant folding seemed to be not completely working. Once I figure out where the sandbox is I'll upload the changes.... Expression: 2*a+(cos(3)+5)*sin(b) numpy: 0.224821311088 numexpr: 0.112924547673 Speed-up of numexpr over numpy: 1.99089848683 -tim From robert.kern at gmail.com Fri Mar 3 12:24:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 3 12:24:02 2006 Subject: [Numpy-discussion] Re: Numpy and PEP 343 In-Reply-To: <44088897.6070802@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: Tim Hochberg wrote: > OT, stupid subversion question: where do I find the sandbox or > whereever this got checked in? http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/numexpr/ Our sandbox is actually scipy.sandbox. Uncomment out the appropriate line in Lib/sandbox/setup.py to build. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From cookedm at physics.mcmaster.ca Fri Mar 3 12:59:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 3 12:59:03 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <44088897.6070802@cox.net> (Tim Hochberg's message of "Fri, 03 Mar 2006 11:19:03 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: Tim Hochberg writes: > That makes sense. One thought I had with respect to the various numpy > functions (sin, cos, pow, etc) was to just have the bytecodes: > > call_unary_function, function_id, store_in, source > call_binary_function, function_id, store_in, source1, source2 > call_trinary_function, function_id, store_in, source1, source2, source3 > > Then just store pointers to the functions in relevant tables. In it's > most straightforward form, you'd need 6 character chunks of bytecode > instead of four. However, if that turns out to slow everything else > down I think it could be packed down to 4 again. The function_ids > could probably be packed into the opcode (as long as we stay below 200 > or so functions, which is probably safe), the other way to pack things > down is to require that one of the sources for trinary functions is > always a certain register (say register-0). That requires a bit more > cleverness at the compiler level, but is probably feasible. That's along the lines I'm thinking of. It seems to me that if evaluating the function requires a function call (and not an inlined machine instruction like the basic ops), then we may as well dispatch like this (plus, it's easier :). This could also allow for user extensions. Binary and trinary (how many of those do we have??) could maybe be handled by storing the extra arguments in a separate array. I'm going to look at adding more smarts to the compiler, too. Got a couple books on them :-) Different data types could be handled by separate input arrays, and a conversion opcode ('int2float', say). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From aisaac at american.edu Fri Mar 3 13:02:01 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri Mar 3 13:02:01 2006 Subject: [Numpy-discussion] The idiom for supporting matrices and arrays in a function In-Reply-To: References: <4406400E.6080600@ee.byu.edu> <4406F52E.8000603@sympatico.ca> Message-ID: On Fri, 3 Mar 2006, Bill Baxter apparently wrote: > I'll just comment on the presence of asarray, asanyarray, > and asmatrix for type conversions. By the way, I believe asmatrix is currently not documented in Guide to NumPy. Cheers, Alan Isaac From tim.hochberg at cox.net Fri Mar 3 13:36:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 13:36:01 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: <4408B679.8080608@cox.net> David M. Cooke wrote: >Tim Hochberg writes: > > > >>That makes sense. One thought I had with respect to the various numpy >>functions (sin, cos, pow, etc) was to just have the bytecodes: >> >>call_unary_function, function_id, store_in, source >>call_binary_function, function_id, store_in, source1, source2 >>call_trinary_function, function_id, store_in, source1, source2, source3 >> >>Then just store pointers to the functions in relevant tables. In it's >>most straightforward form, you'd need 6 character chunks of bytecode >>instead of four. However, if that turns out to slow everything else >>down I think it could be packed down to 4 again. The function_ids >>could probably be packed into the opcode (as long as we stay below 200 >>or so functions, which is probably safe), the other way to pack things >>down is to require that one of the sources for trinary functions is >>always a certain register (say register-0). That requires a bit more >>cleverness at the compiler level, but is probably feasible. >> >> > >That's along the lines I'm thinking of. It seems to me that if >evaluating the function requires a function call (and not an inlined >machine instruction like the basic ops), then we may as well dispatch >like this (plus, it's easier :). This could also allow for user >extensions. > Hmm. If we are going to allow user extensions, we should think about this now. I was thinking of just putting all of the functions in a big table and doing a lookup based on bytecode. I'm not sure how this should work if we are allowing user extensions though. >Binary and trinary (how many of those do we have??) > I'm sure there are more that I'm not thinking of, but the main function that I am concerned with is arctan2. For trinary, the main one is which. > could >maybe be handled by storing the extra arguments in a separate array. > > If there really are only a few functions, we could give them their own opcodes. That means binary function still fit in 4 characters without a problem. That is: OP_ATAN2 R0, R1, R2 would compute: R0[:] = arctan2(R1, R2) Trinary could be defined as using their input as one of the arguments. This would sometimes require an extra copy and some cleverness in the compiler, but I think it would be reasonable. Thus OP_WHICH R0, R1, R2 would compute: R0[:] = which(R0, R1, R2) since the first argument of 'which' is often thrown away anyway, for instance 'which (a < 1, b, c)', the extra copy could often be avoided if we're smart. This might be a bad idea if there turns out to be several more trinary functions that I'm not thinking of or if there are quaternary functions we want to support. (I can't think of any though). >I'm going to look at adding more smarts to the compiler, too. Got a >couple books on them :-) > > Do you have any recomendations? >Different data types could be handled by separate input arrays, and a >conversion opcode ('int2float', say). > > That'd be cool. At some point I want to handle complex, but I'm going to wait till the main issues with floats shake themselves out. -tim From tim.hochberg at cox.net Fri Mar 3 13:41:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 13:41:05 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4408B679.8080608@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408B679.8080608@cox.net> Message-ID: <4408B7C7.2040304@cox.net> Tim Hochberg wrote: >> > If there really are only a few functions, we could give them their own > opcodes. That means binary function still fit in 4 characters without > a problem. That is: > > OP_ATAN2 R0, R1, R2 > > would compute: > R0[:] = arctan2(R1, R2) > > Trinary could be defined as using their input as one of the arguments. > This would sometimes require an extra copy and some cleverness in the > compiler, but I think it would be reasonable. Thus > > OP_WHICH R0, R1, R2 > > would compute: > R0[:] = which(R0, R1, R2) > > since the first argument of 'which' is often thrown away anyway, for > instance 'which (a < 1, b, c)', the extra copy could often be avoided > if we're smart. This might be a bad idea if there turns out to be > several more trinary functions that I'm not thinking of or if there > are quaternary functions we want to support. (I can't think of any > though). > Then again, it might be better to just do a simple, general solution to this case and pass in the arguments in an array as you say. The above might be pointless overengineering. -tim From tim.hochberg at cox.net Fri Mar 3 14:36:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 3 14:36:04 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: <4408C4B5.4090503@cox.net> David M. Cooke wrote: >Tim Hochberg writes: > > > >>That makes sense. One thought I had with respect to the various numpy >>functions (sin, cos, pow, etc) was to just have the bytecodes: >> >>call_unary_function, function_id, store_in, source >>call_binary_function, function_id, store_in, source1, source2 >>call_trinary_function, function_id, store_in, source1, source2, source3 >> >>Then just store pointers to the functions in relevant tables. In it's >>most straightforward form, you'd need 6 character chunks of bytecode >>instead of four. However, if that turns out to slow everything else >>down I think it could be packed down to 4 again. The function_ids >>could probably be packed into the opcode (as long as we stay below 200 >>or so functions, which is probably safe), the other way to pack things >>down is to require that one of the sources for trinary functions is >>always a certain register (say register-0). That requires a bit more >>cleverness at the compiler level, but is probably feasible. >> >> > >That's along the lines I'm thinking of. It seems to me that if >evaluating the function requires a function call (and not an inlined >machine instruction like the basic ops), then we may as well dispatch >like this (plus, it's easier :). This could also allow for user >extensions. Binary and trinary (how many of those do we have??) could >maybe be handled by storing the extra arguments in a separate array. > > We may want to reconsider this at least partially. I tried implementing a few 1-argument functions two way. First as a table lookup and second using a dedicated opcode. The first gave me a speedup of almost 60%, but the latter gave me a speedup of 100%. The difference suprised me, but I suspect it's do to the fact that the x86 supports some functions directly, so the function call gets optimized away for sin and cos just as it does for +-*/. That implies that some functions should get there own opcodes, while others are not worthy. This little quote from wikipedia lists the functions we would want to give there own opcodes to: x86 (since the 80486DX processor) assembly language includes a stack based floating point unit which can perform addition, subtraction, negation, multiplication, division, remainder, square roots, integer truncation, fraction truncation, and scale by power of two. The operations also include conversion instructions which can load or store a value from memory in any of the following formats: Binary coded decimal, 32-bit integer, 64-bit integer, 32-bit floating point, 64-bit floating point or 80-bit floating point (upon loading, the value is converted to the currently floating point mode). The x86 also includes a number of transcendental functions including sine, cosine, tangent, arctangent, exponentiation with the base 2 and logarithms to bases 2, 10, or e . So, that's my new proposal: some functions get there own opcodes (sin, cos, ln, log10, etc), while others get shunted to table lookup (not sure what's in that list yet, but I'm sure there's lots). -tim >I'm going to look at adding more smarts to the compiler, too. Got a >couple books on them :-) > >Different data types could be handled by separate input arrays, and a >conversion opcode ('int2float', say). > > > From oliphant at ee.byu.edu Fri Mar 3 14:42:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 3 14:42:05 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4408C4B5.4090503@cox.net> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: <4408C616.5040202@ee.byu.edu> Tim Hochberg wrote: > We may want to reconsider this at least partially. I tried > implementing a few 1-argument functions two way. First as a table > lookup and second using a dedicated opcode. The first gave me a > speedup of almost 60%, but the latter gave me a speedup of 100%. The > difference suprised me, but I suspect it's do to the fact that the x86 > supports some functions directly, so the function call gets optimized > away for sin and cos just as it does for +-*/. That implies that some > functions should get there own opcodes, while others are not worthy. > This little quote from wikipedia lists the functions we would want to > give there own opcodes to: > > x86 (since the 80486DX processor) assembly language includes a stack > based floating point unit which can perform addition, subtraction, > negation, multiplication, division, remainder, square roots, integer > truncation, fraction truncation, and scale by power of two. The > operations also include conversion instructions which can load or > store a value from memory in any of the following formats: Binary > coded decimal, 32-bit integer, 64-bit integer, 32-bit floating > point, 64-bit floating point or 80-bit floating point (upon loading, > the value is converted to the currently floating point mode). The > x86 also includes a number of transcendental functions including > sine, cosine, tangent, arctangent, exponentiation with the base 2 > and logarithms to bases 2, 10, or e > > . > > > > So, that's my new proposal: some functions get there own opcodes (sin, > cos, ln, log10, etc), while others get shunted to table lookup (not > sure what's in that list yet, but I'm sure there's lots). > For the same reason, I think these same functions should get their own ufunc loops instead of using the default loop with function pointers. Thanks for providing this link. It's a useful list. -Travis From cookedm at physics.mcmaster.ca Fri Mar 3 14:45:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 3 14:45:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: <4408C4B5.4090503@cox.net> (Tim Hochberg's message of "Fri, 03 Mar 2006 15:35:33 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: Tim Hochberg writes: >> > We may want to reconsider this at least partially. I tried > implementing a few 1-argument functions two way. First as a table > lookup and second using a dedicated opcode. The first gave me a > speedup of almost 60%, but the latter gave me a speedup of 100%. The > difference suprised me, but I suspect it's do to the fact that the x86 > supports some functions directly, so the function call gets optimized > away for sin and cos just as it does for +-*/. That implies that some > functions should get there own opcodes, while others are not worthy. Yeah, I was going to stare at the assembly output from gcc before deciding. Note that some functions that are supported directly by the CPU may still be a function call because they aren't good enough. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Fri Mar 3 15:08:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 3 15:08:06 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: (David M. Cooke's message of "Fri, 03 Mar 2006 17:44:28 -0500") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: cookedm at physics.mcmaster.ca (David M. Cooke) writes: > Tim Hochberg writes: >>> >> We may want to reconsider this at least partially. I tried >> implementing a few 1-argument functions two way. First as a table >> lookup and second using a dedicated opcode. The first gave me a >> speedup of almost 60%, but the latter gave me a speedup of 100%. The >> difference suprised me, but I suspect it's do to the fact that the x86 >> supports some functions directly, so the function call gets optimized >> away for sin and cos just as it does for +-*/. That implies that some >> functions should get there own opcodes, while others are not worthy. > > Yeah, I was going to stare at the assembly output from gcc before > deciding. > > Note that some functions that are supported directly by the CPU may > still be a function call because they aren't good enough. ... and a quick test shows me that with gcc 4 under linux, only sqrt is inlined (and it does a test for NaN (I think) and calls the C library sqrt on that). On PowerPC under OS X, even sqrt is a function call. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Fri Mar 3 15:45:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 3 15:45:02 2006 Subject: [Numpy-discussion] Numpy and PEP 343 In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> Message-ID: <4408D4FA.6050205@ee.byu.edu> David M. Cooke wrote: >cookedm at physics.mcmaster.ca (David M. Cooke) writes: > > >>Yeah, I was going to stare at the assembly output from gcc before >>deciding. >> >>Note that some functions that are supported directly by the CPU may >>still be a function call because they aren't good enough. >> >> > >... and a quick test shows me that with gcc 4 under linux, only sqrt >is inlined (and it does a test for NaN (I think) and calls the C >library sqrt on that). On PowerPC under OS X, even sqrt is a function >call. > > There were some posts earlier showing that numarray had faster sin(x) functions on an INTEL platform (with a particular compiler that I'm not sure was specified). I'm pretty sure the difference was the function call versus the inline sin computation that the compiler used. I bet it depends on the compiler. But, you bring up a good point that the on-chip provided functions might not be "math-lib" worthy according to different compilers. It would be nice if there was still some way to use them though letting the user decide if they were "good enough" -Travis From zpincus at stanford.edu Fri Mar 3 17:29:16 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri Mar 3 17:29:16 2006 Subject: [Numpy-discussion] numpy.linalg prevents use of scipy.linalg? Message-ID: Hi folks, I can't seem to use any of the functions in scipy.linalg because numpy defines its own linalg which shadows that of scipy! Specifically, scipy.linalg defines 'norm' (in linalg/basic.py), and numpy doesn't. (This among other differences, I assume.) In [1]: import scipy In [2]: scipy.linalg.norm AttributeError: 'module' object has no attribute 'norm' In [3]: scipy.linalg.__file__ Out[3]: '[...]/python2.4/site-packages/numpy/linalg/__init__.pyc' ^^^^^^^ Now, look what happens when we import scipy.linalg directly: In [1]: import scipy.linalg In [2]: scipy.linalg.norm Out[2]: In [3]: scipy.linalg.__file__ Out[3]: '[...]/python2.4/site-packages/scipy/linalg/__init__.pyc' ^^^^^^^ Something needs to be done to fix this -- but what? Scipy historically imports * from numpy. Scipy historically has a linalg module. The new thing is that numpy has a linalg module, too, which is loaded by default into the numpy namespace. (Compared to Numeric's LinearAlgebra module.) The only thing I can think of is to fold all of scipy.linalg into numpy.linalg, and remove the former. This provides for backwards compatibility for everyone, but perhaps a bit of work. Zach From robert.kern at gmail.com Fri Mar 3 17:49:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 3 17:49:01 2006 Subject: [Numpy-discussion] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: References: Message-ID: Zachary Pincus wrote: > Hi folks, > > I can't seem to use any of the functions in scipy.linalg because > numpy defines its own linalg which shadows that of scipy! I believe I've already fixed this. http://projects.scipy.org/scipy/scipy/changeset/1617 What version of scipy are you using? Before reporting a bug, please make sure it exists in the current SVN revision. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From schofield at ftw.at Sat Mar 4 05:15:04 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat Mar 4 05:15:04 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: <44096620.8080505@gmail.com> References: <44096620.8080505@gmail.com> Message-ID: <440992C9.6090801@ftw.at> Robert Kern wrote: > Zachary Pincus wrote: > >>> What version of scipy are you using? Before reporting a bug, please >>> make sure it >>> exists in the current SVN revision. >>> >> Sorry I didn't include the version (0.4.6, by the way) in my first >> email. My oversight. >> >> As to your latter statement, are you sure about that? You're only >> interested in getting bug reports from people who rebuild scipy from >> the svn every night, or who have tracked down the problem to the line >> so they know where in the svn to see if it's fixed? >> > > No, I don't expect everyone to rebuild scipy every day. However, when you think > you've found a bug, it is standard procedure for you to build the most recent > version to test it and see if the bug is still there. So yes, I do expect people > to build from SVN when they're about to report a bug, but no more often. > I don't expect this. I think a bug report against the latest released version is better than no bug report. If we think we've seen and fixed the bug in SVN, it takes us less time to reply "Hmmm, check this with the latest SVN" than it would take Zachary or another would-be bug reporter to check out and build the whole tree from SVN, just on the off-chance we've fixed it since then. This lowers the barrier to entry for helpers. Reading bug reports about the current released version can also be a helpful reminder to us of which bugs are still outstanding in the most recent release. -- Ed From tim.hochberg at cox.net Sat Mar 4 06:59:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 4 06:59:04 2006 Subject: [Fwd: Re: [Numpy-discussion] Numpy and PEP 343] Message-ID: <4409AB20.7060201@cox.net> Check this out: Expression: where(0.1*a > arctan2(a, b), 2*a, arctan2(a,b)) numpy: 0.706635284652 Skipping weave timing numexpr: 0.19127785794 Speed-up of numexpr over numpy: 3.69428689898 270% speed up! Pretty cool. Unfortunately, my implementation of two and three argument functions is still buggy if one of the arguments is a constant. I'm not sure what the best tactic is to deal with that. My strategy for dealing with opcodes that require more than three arguments (just 'where' at present) was to just use two opcode blocks. The second opcode block had the opcode OP_NOOP (which does nothing), but the first opcode steals arguments from it. It's pretty simple and more or less infinitely expandable. I'm off to bed now, perhaps the answer to the constant dilemma will be clear in the morning. [next morning] OK, I know how to fix it, I just need to scrape together a few minutes to do it. -tim From tim.hochberg at cox.net Sat Mar 4 07:00:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 4 07:00:03 2006 Subject: [Fwd: [Fwd: Re: [Numpy-discussion] Numpy and PEP 343]] Message-ID: <4409AB3A.8090908@cox.net> Check this out: Expression: where(0.1*a > arctan2(a, b), 2*a, arctan2(a,b)) numpy: 0.706635284652 Skipping weave timing numexpr: 0.19127785794 Speed-up of numexpr over numpy: 3.69428689898 270% speed up! Pretty cool. Unfortunately, my implementation of two and three argument functions is still buggy if one of the arguments is a constant. I'm not sure what the best tactic is to deal with that. My strategy for dealing with opcodes that require more than three arguments (just 'where' at present) was to just use two opcode blocks. The second opcode block had the opcode OP_NOOP (which does nothing), but the first opcode steals arguments from it. It's pretty simple and more or less infinitely expandable. I'm off to bed now, perhaps the answer to the constant dilemma will be clear in the morning. [next morning] OK, I know how to fix it, I just need to scrape together a few minutes to do it. -tim From oliphant.travis at ieee.org Sat Mar 4 09:08:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Mar 4 09:08:04 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: <440992C9.6090801@ftw.at> References: <44096620.8080505@gmail.com> <440992C9.6090801@ftw.at> Message-ID: <4409C944.1020806@ieee.org> Ed Schofield wrote: >> No, I don't expect everyone to rebuild scipy every day. However, when you think >> you've found a bug, it is standard procedure for you to build the most recent >> version to test it and see if the bug is still there. So yes, I do expect people >> to build from SVN when they're about to report a bug, but no more often. >> >> > > I don't expect this. I think a bug report against the latest released > version is better than no bug report. If we think we've seen and fixed > the bug in SVN, it takes us less time to reply "Hmmm, check this with > the latest SVN" than it would take Zachary or another would-be bug > reporter to check out and build the whole tree from SVN, just on the > off-chance we've fixed it since then. This lowers the barrier to entry > for helpers. Reading bug reports about the current released version can > also be a helpful reminder to us of which bugs are still outstanding in > the most recent release. > > I don't want potential bug reports not to get filed, either. This is another advantage of the ticket system on the trac pages. The bug-reporter could just check that page to see if the problem has been fixed. Alternatively, the bug-reporter can do a search through the newsgroup to see if the problem has been talked about and reported as fixed. I think requiring all bug-reporters to build from SVN is a bit of a barrier that we don't want to push at this stage. Especially, when there may be many little bugs lurking from the transition. -Travis From robert.kern at gmail.com Sat Mar 4 11:45:13 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat Mar 4 11:45:13 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Re: numpy.linalg prevents use of scipy.linalg? In-Reply-To: <4409C944.1020806@ieee.org> References: <44096620.8080505@gmail.com> <440992C9.6090801@ftw.at> <4409C944.1020806@ieee.org> Message-ID: Travis Oliphant wrote: > I don't want potential bug reports not to get filed, either. This is > another advantage of the ticket system on the trac pages. The > bug-reporter could just check that page to see if the problem has been > fixed. Alternatively, the bug-reporter can do a search through the > newsgroup to see if the problem has been talked about and reported as > fixed. > > I think requiring all bug-reporters to build from SVN is a bit of a > barrier that we don't want to push at this stage. Especially, when > there may be many little bugs lurking from the transition. I worded myself poorly. I don't want people to forgo entering bugs in the tracker if they can't find the time to rebuild from SVN. But I do want them to try! -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From tim.hochberg at cox.net Sun Mar 5 16:30:53 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 5 16:30:53 2006 Subject: [Numpy-discussion] numexpr In-Reply-To: <44095626.50604@ieee.org> References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> <4408D4FA.6050205@ee.byu.edu> <44092A05.5080905@cox.net> <44095626.50604@ieee.org> Message-ID: <440A6583.4040609@cox.net> I just commited a bunch of changes to numexpr in the sandbox (BTW, thanks to Robert Kern for pointing me to the right location for that). At this point, there's a fairly complete framework for evaluationg numeric expressions (as long as they're not complex, and have unit stride and are all the same size etc, OK maybe not so complete, but potentially very useful nonetheless). A couple basic operations (** and %) need to be filled in as well as a bunch of functions. Still most basic operaions work. A bunch of 2- and 3-arg functions work as well as 'where'. I'll wait to hear from David Cooke to see how he likes the current function framework -- if he likes it, I'll go ahead and fill in the other common functions. It would also be nice to support sum and product as David mentioned, but I haven't though about that at all yet. At some point I'd like to add support for complex numbers as well, but I'd like to let the floating point stuff settle a bit first. -tim From pfpntdfhcz at montgomery.com Sun Mar 5 16:47:40 2006 From: pfpntdfhcz at montgomery.com (pfpntdfhcz) Date: Sun Mar 5 16:47:40 2006 Subject: [Numpy-discussion] Fw: numpy-discussion Message-ID: <000f01c64058$c8ec8000$1a041253@malpikrz008dft> ----- Original Message ----- From: Bass Aileen To: sgnptwk at southtrust.com Sent: Saturday, March 4, 2006 8:21 AM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: wucpxgkjjc.gif Type: image/gif Size: 9405 bytes Desc: not available URL: From tim.hochberg at cox.net Sun Mar 5 16:51:47 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 5 16:51:47 2006 Subject: [Numpy-discussion] numexpr thoughts Message-ID: <440AF97C.5040106@cox.net> 1. David already mentioned opcodes to copy from an integer array into a float register. Another idea would be to have an opcode to copy from a strided array into a register. This would save a copy for noncontiguous matrices and opens up the door to doing broadcasting. I think we'd have to tweak the main interpreter loop a bit, but I think it's doable. Depending how crunched we feel for opcode space, the casting array and this array could overlap (OP_CAST_AND_COPY for instance). 2. Something I noticed while writing tests is that the rearangement of operations for 'a/const' to '(1/const)*a' changes the results slightly (try const=3). I don't think I care -- I just thought I'd point it out. 3. It may simplify things to use copy_c to eliminate a bunch of the extra bytecodes for operating on functions of more than one argument. I need to check the timings on this -- I don't know how much of a speed hit using copy_c will cause. 4. At some point I plan to add complex operations. My original thought was to just duplicate the current, floating-point interpreter for complex numbers. However, if we have opcodes for casting, it might instead make sense to have one interpreter that works with both. This would be more efficient for mixed expressions since we wouldnt' have to case everything complex right away. It looks like there's enough opcode space, although I worry a bit whether making that main switch loop huge is going to slow things down. Probably not, but it's something to keep an eye on. 5. Same thoughts as 4., but for integer operations. Potentially the interpreter could work with longs, doubles and cdoubles, downcasting as appropriate on the way out. It's looking like numexpr could come pretty close to evaluating most numpy expressions if we put enough work into it. None of the required changes look like they should slow down the original, simple, fast case. However we should keep an eye on that as we go along. It's all very cool; I'd like to thank David for providing such a fun toy. -tim From ndarray at mac.com Sun Mar 5 17:02:05 2006 From: ndarray at mac.com (Sasha) Date: Sun Mar 5 17:02:05 2006 Subject: [Numpy-discussion] Tests fail in svn r2195 Message-ID: Reproduced on Linux with python 2.4 and on MacOS X with python 2.3. ====================================================================== ERROR: Check creation from tuples ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 151, in check_tuple h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check creation from tuples ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 151, in check_tuple h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check reading the top fields of a nested array ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 244, in check_access_top_fields h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check reading the nested fields of a nested array (1st level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 262, in check_nested1_acessors h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check access nested descriptors of a nested array (1st level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 310, in check_nested1_descriptor h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check reading the nested fields of a nested array (2nd level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 292, in check_nested2_acessors h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: Check access nested descriptors of a nested array (2nd level) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 318, in check_nested2_descriptor h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ====================================================================== ERROR: check_access_fields (numpy.core.tests.test_numerictypes.test_read_values_plain_single) ---------------------------------------------------------------------- Traceback (most recent call last): File "/bnv/sandbox/numpy-dev/lib/python2.4/site-packages/numpy/core/tests/test_numerictypes.py", line 210, in check_access_fields h = array(self._buffer, dtype=self._descr) TypeError: expected a readable buffer object ---------------------------------------------------------------------- Ran 337 tests in 1.131s FAILED (errors=8) From charlesr.harris at gmail.com Sun Mar 5 17:03:34 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun Mar 5 17:03:34 2006 Subject: [Numpy-discussion] Thoughts on an ndarray super-class In-Reply-To: <43FD5914.4060506@ieee.org> References: <43FD5914.4060506@ieee.org> Message-ID: On 2/22/06, Travis Oliphant wrote: > > The bigndarray class is going to disappear (probably in the next release > of NumPy). It was a stop-gap measure as the future of 64-bit fixes in > Python was unclear. Python 2.5 will have removed the 64-bit limitations > that led to the bigndarray and so it will be removed. > > I have been thinking, however, of replacing it with a super-class that > does not define the dimensions or strides. I use arrays a lot as buffers to pass binary data between C programs. At the moment I use the buffer interface, but if the superclass was just as convenient that would do. Mostly I just want simple. Are you thinking of keeping the dtype? Chuck From ndarray at mac.com Mon Mar 6 08:49:01 2006 From: ndarray at mac.com (Sasha) Date: Mon Mar 6 08:49:01 2006 Subject: [Numpy-discussion] Code cleanup patches Message-ID: I would like to thank timl for the code cleanup patches. I've commited the ma.py patch to svn and the other patches seem to be acceptable as well. I assume you used some kind of automated code checking tool to spot the irregularities that you've fixed. It would be nice if you could describe your procedure and the tool that you used somewere on the developers' wiki. I've created a stub at http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines . I assume that for the most part numpy can simply follow general python guidelines, but as more people start contributing code it maybe useful to document some guidelines. From mpi at osc.kiku.dk Mon Mar 6 13:04:02 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Mon Mar 6 13:04:02 2006 Subject: [Numpy-discussion] Is sum() slow ? In-Reply-To: References: Message-ID: Hi, Here are some timings that puzzle me a bit. Sum the two rows of a 2xn matrix, where n is some large number python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x.sum(0)" 10000 loops, best of 3: 36.2 usec per loop python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x[0] + x[1]" 100000 loops, best of 3: 5.35 usec per loop The two calculations are completely equivalent (at least from the numerical results point of view). But using the built-in sum() function is app. 6 times slower. Just for the reference - using Numeric gives python -m timeit -s "from Numeric import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "sum(x)" 100000 loops, best of 3: 7.08 usec per loop Any suggestions to why this might be - or am I doing something wrong here? Thanks // Mads PS. Thanks to everybody that responded to my previous posting on problems and questions regarding rint() and related coding issues. Your replies were very helpful! From cookedm at physics.mcmaster.ca Mon Mar 6 15:00:04 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 6 15:00:04 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440AF97C.5040106@cox.net> (Tim Hochberg's message of "Sun, 05 Mar 2006 07:45:16 -0700") References: <440AF97C.5040106@cox.net> Message-ID: Tim Hochberg writes: > 1. David already mentioned opcodes to copy from an integer array into > a float register. Another idea would be to have an opcode to copy from > a strided array into a register. This would save a copy for > noncontiguous matrices and opens up the door to doing broadcasting. I > think we'd have to tweak the main interpreter loop a bit, but I think > it's doable. Depending how crunched we feel for opcode space, the > casting array and this array could overlap (OP_CAST_AND_COPY for > instance). Hadn't thought about doing strided arrays like that; it should work. > 2. Something I noticed while writing tests is that the rearangement of > operations for 'a/const' to '(1/const)*a' changes the results > slightly (try const=3). I don't think I care -- I just thought I'd > point it out. I don't care either :-) Although it may be worthwile throwing in a compiler option to keep a/const. > 3. It may simplify things to use copy_c to eliminate a bunch of the > extra bytecodes for operating on functions of more than one argument. > I need to check the timings on this -- I don't know how much of a > speed hit using copy_c will cause. I'm implementing another solution: I'm getting rid of individual constant operators, like 'add_c'. Constants will be stored in the temps array, as 128 repetitions of the constant. That way 'add' (for instance) can operate on both vector + vector and vector + scalar: it would take the register numbers for the two vectors for the first one, and for the second case, the scalar argument would be the register number for the repeated constant. Timings on my iBook show this is actually slightly faster. I didn't expect that :-) > 4. At some point I plan to add complex operations. My original thought > was to just duplicate the current, floating-point interpreter for > complex numbers. However, if we have opcodes for casting, it might > instead make sense to have one interpreter that works with both. This > would be more efficient for mixed expressions since we wouldnt' have > to case everything complex right away. It looks like there's enough > opcode space, although I worry a bit whether making that main switch > loop huge is going to slow things down. Probably not, but it's > something to keep an eye on. The way I see it, there should be two machines: one for single precision, and one for double precision (maybe one for longdoubles, but I'm not worrying about that now). That's simple enough. Each machine should work with floats, and the complex form of that float. Potentially, cast opcodes could be used to convert single to double, for instance. I'm still thinking about how to fit that. > 5. Same thoughts as 4., but for integer operations. Potentially the > interpreter could work with longs, doubles and cdoubles, downcasting > as appropriate on the way out. I suppose this could be considered a "reduction" operation: instead of collapsing the array, we're collapsing the precision. > It's looking like numexpr could come pretty close to evaluating most > numpy expressions if we put enough work into it. None of the required > changes look like they should slow down the original, simple, fast > case. However we should keep an eye on that as we go along. It's all > very cool; I'd like to thank David for providing such a fun toy. thanks Right now, here's my thoughts on where to go: 1. I'm rewriting the compiler to be easier to play with. Top things on the list are better register allocation, and common subexpression elimination. 2. Replace individual constants with "constant arrays": repeations of the constant in the vector registers. 3. Reduction. I figure this could be done at the end of the program in each loop: sum or multiply the output register. Downcasting the output could be done here too. 4. Add complex numbers. If it doesn't look really messy, we could add them to the current machine. Otherwise, we could make a machine that works on complex numbers only. 5. Currently, we use a big switch statement. There are ways (taken from Forth) that are better: indirect and direct threading. Unfortunately, it looks the easy way to do these uses GCC's capability to take the address of local labels. I'll add that if I can refactor the machine enough so that both variants can be produced. Have a look at http://www.complang.tuwien.ac.at/anton/vmgen/ which is the virtual machine generator used for gforth (but applicable to other things). I may use this. 6. Replace register numbers in the program with actual pointers to the correct register. That should remove a layer of indirection. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Mon Mar 6 15:00:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 6 15:00:06 2006 Subject: [Numpy-discussion] numexpr In-Reply-To: <440A6583.4040609@cox.net> (Tim Hochberg's message of "Sat, 04 Mar 2006 21:13:55 -0700") References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <4408C4B5.4090503@cox.net> <4408D4FA.6050205@ee.byu.edu> <44092A05.5080905@cox.net> <44095626.50604@ieee.org> <440A6583.4040609@cox.net> Message-ID: Tim Hochberg writes: > I just commited a bunch of changes to numexpr in the sandbox (BTW, > thanks to Robert Kern for pointing me to the right location for that). > > At this point, there's a fairly complete framework for evaluationg > numeric expressions (as long as they're not complex, and have unit > stride and are all the same size etc, OK maybe not so complete, but > potentially very useful nonetheless). A couple basic operations (** > and %) need to be filled in as well as a bunch of functions. Still > most basic operaions work. A bunch of 2- and 3-arg functions work as > well > as 'where'. > > I'll wait to hear from David Cooke to see how he likes the current > function framework -- if he likes it, I'll go ahead and fill in the > other common functions. It would also be nice to support sum and > product as David mentioned, but I haven't though about that at all > yet. Looks good. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From robert.kern at gmail.com Mon Mar 6 15:32:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 6 15:32:02 2006 Subject: [Numpy-discussion] Re: numexpr thoughts In-Reply-To: References: <440AF97C.5040106@cox.net> Message-ID: David M. Cooke wrote: > 5. Currently, we use a big switch statement. There are ways (taken > from Forth) that are better: indirect and direct threading. > Unfortunately, it looks the easy way to do these uses GCC's > capability to take the address of local labels. I'll add that if I > can refactor the machine enough so that both variants can be > produced. Have a look at > http://www.complang.tuwien.ac.at/anton/vmgen/ > which is the virtual machine generator used for gforth (but > applicable to other things). I may use this. Hmmm. If LLVM weren't so huge and such a pain to install, I might recommend looking at using it. It could make a fun experiment, though. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Mon Mar 6 15:39:05 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 6 15:39:05 2006 Subject: [Numpy-discussion] Re: numexpr thoughts In-Reply-To: (Robert Kern's message of "Mon, 06 Mar 2006 17:30:36 -0600") References: <440AF97C.5040106@cox.net> Message-ID: Robert Kern writes: > David M. Cooke wrote: > >> 5. Currently, we use a big switch statement. There are ways (taken >> from Forth) that are better: indirect and direct threading. >> Unfortunately, it looks the easy way to do these uses GCC's >> capability to take the address of local labels. I'll add that if I >> can refactor the machine enough so that both variants can be >> produced. Have a look at >> http://www.complang.tuwien.ac.at/anton/vmgen/ >> which is the virtual machine generator used for gforth (but >> applicable to other things). I may use this. > > Hmmm. If LLVM weren't so huge and such a pain to install, I might recommend > looking at using it. It could make a fun experiment, though. Yeah, I had a look at that. PyPy is using it, so things could be stolen from that. Fortunately, our virtual machine can be simpler than most, because we don't have conditionals or jumps :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From tim.hochberg at cox.net Mon Mar 6 19:24:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 6 19:24:01 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: References: <440AF97C.5040106@cox.net> Message-ID: <440CFCA1.70007@cox.net> David M. Cooke wrote: >1. David already mentioned opcodes to copy from an integer array into >> a float register. Another idea would be to have an opcode to copy from >> a strided array into a register. This would save a copy for >> noncontiguous matrices and opens up the door to doing broadcasting. I >> think we'd have to tweak the main interpreter loop a bit, but I think >> it's doable. Depending how crunched we feel for opcode space, the >> casting array and this array could overlap (OP_CAST_AND_COPY for >> instance). > > > >Hadn't thought about doing strided arrays like that; it should work. > > This opens another interesting can of worms: two stage compilation. If we aren't copying the arrays on the way in, it might make sense to have an abstract opcode OP_COPY_ANY, that copied data from any allowable source. This would be produced during the compilation stage. When numexpr was called, before the loop was executed, OP_COPY_ANY would be replaced depending on the input value. For example: OP_COPY_ANY destreg input would translate to: OP_COPY_INT16ARR destreg input # input is an int array OP_COPY_FLT32SCL destreg input # input is an FP scalar etc. The data would then be copied from the array into a register with appropriate striding and casting. If 'input' was already a contiguous double array, then it could translate into simply setting a pointer as is done now (OP_COPY_SIMPLE perhaps). [SNIP] >Right now, here's my thoughts on where to go: > >1. I'm rewriting the compiler to be easier to play with. Top things > on the list are better register allocation, and common > subexpression elimination. > > Cool. >2. Replace individual constants with "constant arrays": repeations of > the constant in the vector registers. > > I'm already using this trick to a certain extent to reduce the number op func_xy opcodes. I copy the constants into arrays using OP_COPY_C. This sounds like essentially the same thing as what you are doing. >3. Reduction. I figure this could be done at the end of the program in > each loop: sum or multiply the output register. Downcasting the > output could be done here too. > > Do you mean that sum() and friends could only occur as the outermost function. That is: "sum(a+3*b)" would work, but: "where(a, sum(a+3*b), c)" would not? Or am I misunderstanding you here? I don't think I like that limitation if that's the case. I don' think it should be necessary either. >4. Add complex numbers. If it doesn't look really messy, we could add > them to the current machine. Otherwise, we could make a machine that > works on complex numbers only. > > It doesn't seem like it should be bad at all. The slickest thing would be to use to adjacent float buffers for a complex buffer. That would make the compiler a bit more complex, but it would keep the intrepeter simpler as there would only be a single buffer type. All that needs to be supported are the basic operations (+,-,*,/,// and **); comparisons don't work for complex numbers anyway and all of the functions can go through the function pointers since they're slow anyway. The one exception is where, which would be mixed complex and float and should be fast. The other question is integers. There would be some advantages to supporting mixed integer and floating point operations. This adds a larger crop of operators though; The basic ones as above, plus comparisons, plus bitwise operators. >5. Currently, we use a big switch statement. There are ways (taken > from Forth) that are better: indirect and direct threading. > Unfortunately, it looks the easy way to do these uses GCC's > capability to take the address of local labels. I'll add that if I > can refactor the machine enough so that both variants can be > produced. Have a look at > http://www.complang.tuwien.ac.at/anton/vmgen/ > which is the virtual machine generator used for gforth (but > applicable to other things). I may use this. > > Better as in lower overhead? Or better as in simpler to write? The current code is competitive with weave already which would seem to indicate that we're already dominated by the overhead of the FP math, not the big switch statement. I'd only rewrite things if it's going to be simpler to work with. >6. Replace register numbers in the program with actual pointers to the > correct register. That should remove a layer of indirection. > > This would be done at function calling time as well? So this is more two-stage compilation stuff? -tim From mpi at osc.kiku.dk Mon Mar 6 23:16:02 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Mon Mar 6 23:16:02 2006 Subject: [Numpy-discussion] Is sum() slow? Message-ID: Hi, Sorry if this is a double post - unsure if it made it the first time: Here are some timings that puzzle me a bit. Sum the two rows of a 2xn matrix, where n is some large number python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x.sum(0)" 10000 loops, best of 3: 36.2 usec per loop python -m timeit -s "from numpy import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "x[0] + x[1]" 100000 loops, best of 3: 5.35 usec per loop The two calculations are completely equivalent (at least from the numerical results point of view). But using the built-in sum() function is app. 6 times slower. Just for the reference - using Numeric gives python -m timeit -s "from Numeric import array,sum,reshape; x = array([1.5]*1000); x = reshape(x,(2,500))" "sum(x)" 100000 loops, best of 3: 7.08 usec per loop Any suggestions to why this might be - or am I doing something wrong here? Thanks // Mads PS. Thanks to everybody that responded to my previous posting on problems and questions regarding rint() and related coding issues. Your replies were very helpful! From tim.hochberg at cox.net Tue Mar 7 10:35:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 7 10:35:02 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440CFCA1.70007@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> Message-ID: <440DD209.5060900@cox.net> Tim Hochberg wrote: >> 3. Reduction. I figure this could be done at the end of the program in >> each loop: sum or multiply the output register. Downcasting the >> output could be done here too. >> >> > Do you mean that sum() and friends could only occur as the outermost > function. That is: > "sum(a+3*b)" > would work, but: > "where(a, sum(a+3*b), c)" > would not? Or am I misunderstanding you here? I don't think I like > that limitation if that's the case. I don' think it should be > necessary either. OK, I thought about this some more and I think I was mostly wrong. I suspect that reduction does need to happen as the last step. Still it would be nice for "where(a, sum(a+3*b), c)" to work. This could be done by doing the following transformation: a = evaluate("where(a, sum(a+3*b), c)") => temp=evaluate("sum(a+3*b)"); a = evaluate("where(a, temp, c)") I suspect that this this would be fairly easy to do automagically as long as it was at the Python level. That is, numexpr would return a python object that would call the lower level interpreter.numexpr appropriately. This would have some other advantages as well -- it would make it easy to deal with keyword arguments for one. It would also make it easy to do the bytecode rewriting if we decide to go that route. It does add more call overhead, but if that turns out to be we can move stuff into C later. I'm still not excited about summing over the whole output buffer though. That ends up allocating and scanning through a whole extra buffer which may result in a signifigant speed and memory hit for large arrays. Since if we're only doing this on the way out, there should be no problem just allocating a single double (or complex) to do the sum in. On the way in, this could be set to zero or one based on what the last opcode is (sum or product). Then the SUM opcode could simply do something like: BTW, the cleanup of the interpreter looks pretty slick. I'm going to look at timings for using COPY_C versus using add directly and see about reducing the number of opcodes. If this works out OK, the number comparison opcodes could be reduced a lot. Sorry about munging the line endings. -tim From oliphant at ee.byu.edu Tue Mar 7 10:38:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 7 10:38:03 2006 Subject: [Numpy-discussion] Is sum() slow? In-Reply-To: References: Message-ID: <440DD2FA.4060301@ee.byu.edu> Mads Ipsen wrote: >Hi, > >Sorry if this is a double post - unsure if it made it the first time: > >Here are some timings that puzzle me a bit. Sum the two rows of a 2xn >matrix, where n is some large number > >python -m timeit -s "from numpy import array,sum,reshape; x = >array([1.5]*1000); x = reshape(x,(2,500))" "x.sum(0)" >10000 loops, best of 3: 36.2 usec per loop > >python -m timeit -s "from numpy import array,sum,reshape; x = >array([1.5]*1000); x = reshape(x,(2,500))" "x[0] + x[1]" >100000 loops, best of 3: 5.35 usec per loop > > This is probably reasonable. There is overhead in the looping construct (basically what happens is that the first element is copied into the output and then a function called --- which in this case has a loop of size 1 to compute the sum). This is then repeated 500 times. So, you have 500 C-function pointer calls in the first case. In the second case you have basically a single call to the same function where the 500-element loop is done. I'm a little surprised that Numeric is so much faster for this case as you show later. The sum code is actually add.reduce... which uses a generic reduction concept in ufuncs. It has overhead over what you might do using some less general approach. If anyone can figure out how to make the NOBUFFER secion in GenericReduce faster in ufuncobject.c it will be greatly welcomed. Speed improvements are always welcome. -Travis From cookedm at physics.mcmaster.ca Tue Mar 7 10:53:05 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 7 10:53:05 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440DD209.5060900@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> Message-ID: <20060307185127.GA31063@arbutus.physics.mcmaster.ca> On Tue, Mar 07, 2006 at 11:33:45AM -0700, Tim Hochberg wrote: > Tim Hochberg wrote: > > >>3. Reduction. I figure this could be done at the end of the program in > >> each loop: sum or multiply the output register. Downcasting the > >> output could be done here too. > >> > > I'm still not excited about summing over the whole output buffer though. > That ends up allocating and scanning through a whole extra buffer which > may result in a signifigant speed and memory hit for large arrays. Since > if we're only doing this on the way out, there should be no problem just > allocating a single double (or complex) to do the sum in. On the way > in, this could be set to zero or one based on what the last opcode is > (sum or product). Then the SUM opcode could simply do something like: No, no, we'd just sum over the 128 element output vector (mem[0]), and add the result to cumulative sum. That vector should already be in cache, as the last op would put it there. > BTW, the cleanup of the interpreter looks pretty slick. Not finished yet :-) Look for a checkin today (if I have time). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From tim.hochberg at cox.net Tue Mar 7 11:09:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 7 11:09:02 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <20060307185127.GA31063@arbutus.physics.mcmaster.ca> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> Message-ID: <440DDA10.4020904@cox.net> David M. Cooke wrote: >On Tue, Mar 07, 2006 at 11:33:45AM -0700, Tim Hochberg wrote: > > >>Tim Hochberg wrote: >> >> >> >>>>3. Reduction. I figure this could be done at the end of the program in >>>> each loop: sum or multiply the output register. Downcasting the >>>> output could be done here too. >>>> >>>> >>>> >>I'm still not excited about summing over the whole output buffer though. >>That ends up allocating and scanning through a whole extra buffer which >>may result in a signifigant speed and memory hit for large arrays. Since >>if we're only doing this on the way out, there should be no problem just >>allocating a single double (or complex) to do the sum in. On the way >>in, this could be set to zero or one based on what the last opcode is >>(sum or product). Then the SUM opcode could simply do something like: >> >> > >No, no, we'd just sum over the 128 element output vector (mem[0]), and >add the result to cumulative sum. That vector should already be in >cache, as the last op would put it there. > > Ah! Ok then. That's what I was thinking of too. For some reason I thought you were proposing building the whole result vector then summing it. Here's another wrinkle: how do we deal with: >>> a = reshape(arange(9), (3,3)) >>> sum(a) array([ 9, 12, 15]) Just forbid it? For the time being at least? > > >>BTW, the cleanup of the interpreter looks pretty slick. >> >> > >Not finished yet :-) Look for a checkin today (if I have time). > > They didn't seem to have any speed advantage, so I ripped out all the compare with constant opcodes amd used COPY_C instead. I'm probably going to rip out OP_WHERE_XXC and OP_WHERE_XCX depending on the timings there. Should I also kill OP_ADD_C and friends as well while I'm at it? -tim From cookedm at physics.mcmaster.ca Tue Mar 7 12:36:01 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 7 12:36:01 2006 Subject: [Numpy-discussion] numexpr thoughts In-Reply-To: <440DDA10.4020904@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> <440DDA10.4020904@cox.net> Message-ID: <20060307203412.GA31277@arbutus.physics.mcmaster.ca> On Tue, Mar 07, 2006 at 12:08:00PM -0700, Tim Hochberg wrote: > David M. Cooke wrote: > > >On Tue, Mar 07, 2006 at 11:33:45AM -0700, Tim Hochberg wrote: > > > > > >>Tim Hochberg wrote: > >> > >> > >> > >>>>3. Reduction. I figure this could be done at the end of the program in > >>>>each loop: sum or multiply the output register. Downcasting the > >>>>output could be done here too. > >>>> > >>>> > >>>> > >>I'm still not excited about summing over the whole output buffer though. > >>That ends up allocating and scanning through a whole extra buffer which > >>may result in a signifigant speed and memory hit for large arrays. Since > >>if we're only doing this on the way out, there should be no problem just > >>allocating a single double (or complex) to do the sum in. On the way > >>in, this could be set to zero or one based on what the last opcode is > >>(sum or product). Then the SUM opcode could simply do something like: > >> > >> > > > >No, no, we'd just sum over the 128 element output vector (mem[0]), and > >add the result to cumulative sum. That vector should already be in > >cache, as the last op would put it there. > > > > > > Ah! Ok then. That's what I was thinking of too. For some reason I > thought you were proposing building the whole result vector then summing it. > > Here's another wrinkle: how do we deal with: > > >>> a = reshape(arange(9), (3,3)) > >>> sum(a) > array([ 9, 12, 15]) > > > Just forbid it? For the time being at least? Well, I think for now, sum(a) = sum(a.ravel()). Doing differently will mean working with dimensions, which is something else to figure out. > They didn't seem to have any speed advantage, so I ripped out all the > compare with constant opcodes amd used COPY_C instead. I'm probably > going to rip out OP_WHERE_XXC and OP_WHERE_XCX depending on the timings > there. Should I also kill OP_ADD_C and friends as well while I'm at it? Don't bother; the version I'm going to check in already has them gone. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cjw at sympatico.ca Tue Mar 7 14:17:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 7 14:17:03 2006 Subject: [Numpy-discussion] numpy doc strings - a tool to list problem cases Message-ID: <440E0643.40904@sympatico.ca> Some of the docstrings have lines which are too long. Some names have no associated docstring. The effects of this are that, for the long lines, the PyDoc generated documentation is difficult to read on a browser and, for the missing cases, the generated docs are incomplete. The question of how long is too long is a matter of personal choice. I used 78 characters in the little script below that lists problem cases. # to check numpy docs, for long and missing lines import numpy lst= dir(numpy) lnMax= 78 # maximum number characters per line print 'Report of missing docstrings and docstrings with lines that are too long' print for nm in lst: txt= eval('numpy.' + nm + '.__doc__') if txt is None: print nm, 'No docstring' continue lns= txt.split('\n') for lnNo in range(len(lns)): ln= lns[lnNo] if len(ln) > lnMax: print nm, 'Line#:', lnNo, 'Line Length:', len(ln) z= 1 z= 1 Colin W. From cookedm at physics.mcmaster.ca Tue Mar 7 14:59:04 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 7 14:59:04 2006 Subject: [Numpy-discussion] Power optimization branch merged to Numpy trunk Message-ID: Since Tim and I are now fiddling around with numexpr, I've merged what we've got with the power optimization to the Numpy trunk. This includes: * x**2 is _as_fast_ as x*x now * optimization of x**n for x as an array and n as a Python scalar. * x**2, x**(-1), x**0 are optimized to use the new square, reciprocal, and ones_like ufuncs * complex integer powers should be faster Also, PyArray_FromAny has been speeded up when converting Python scalars. A side effect of this change is that the array protocol won't be looked for on subclasses of Python scalars (int, long, float, complex). I don't expect that to be a problem. Some things we hadn't got around to: - integer arrays to a power (currently, these use pow()) - more optimization of integer powers enjoy the speed :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From fred at ucar.edu Tue Mar 7 16:48:03 2006 From: fred at ucar.edu (Fred Clare) Date: Tue Mar 7 16:48:03 2006 Subject: [Numpy-discussion] NumPy/Numeric/PyArray_Check Message-ID: <1d36eb09073891383f5397ac3b191c09@ucar.edu> I have been a long-time Numeric user and have just started to use numpy-0.9.5 on a Mac running OS X Version 10.3.9. I installed NumPy in the same Python that has Numeric 24.2 installed. In using Numeric in my Python codes I have used only "import Numeric" for the import. After having installed "numpy" I replaced "import Numeric" with "import numpy as Numeric". All seems to go O.K. until I call a function in a module that uses Numeric (and not numpy) to create an array in my Python script. When I pass this Numeric array to a function in an extension module (that was compiled using the arrayobject.h from the NumPy distribution) and do a "PyArray_Check" on the Numeric array, it returns false. Is here a way around this? Based on the statement in the "Guide to NumPy" that Numeric and NumPy can be used simultaneously I was hoping that this would work. Fred Clare From hgamboa at gmail.com Wed Mar 8 04:25:03 2006 From: hgamboa at gmail.com (Hugo Gamboa) Date: Wed Mar 8 04:25:03 2006 Subject: [Numpy-discussion] Power optimization branch merged to Numpy trunk In-Reply-To: References: Message-ID: <86522b1a0603080424l7a696a8enfae90393eeed7829@mail.gmail.com> Since you are mentioning some improvements (I have confirmed them!) I would like to ask what code are you using for benchmarking numpy. I wanted to know what code was faster: sum(|x|)/N or sqrt(sum(x**2))/sqrt(N) I wrote a timeit test and discovered that using the function dot(x,x) to compute sum(x**2) gives the best result and is one order of magnitude faster than sum(|x|). I found that sum and absolute are approx. 5 times slower than dot. I also noticed that the std function is slightly slower than a python implementation code. These are the results for some operations in a randn vector of size 1000: 48.9+-22.6us: np.dot(v,v) 628.8+-87.3us: np.sum(np.absolute(v)) 518.3+-65.9us: np.sum(v*v) 118.0+-1.4us: v*v 234.4+-56.5us: np.absolute(v) 234.3+-56.2us: v.sum() 1175.2+-58.6us: v.std() 969.6+-77.4us: vv=np.mean(v)-v;(np.sqrt(np.dot(vv,vv)/(len(v)-1))) The code for timming the numpy operations: ################# import timeit import numpy as np p= """ import numpy as np vsize=1000 v=np.randn(vsize) """ vs=[ 'np.dot(v,v)', 'np.sum(np.absolute(v))', 'np.sum(v*v)', 'v*v', 'np.absolute(v)', 'v.sum()', 'v.std()', 'vv=np.mean(v)-v;(np.sqrt(np.dot(vv,vv)/(len(v)-1)))'] ntests=1000 tries=5 for s in vs: t = timeit.Timer(s,p) r=(np.array(t.repeat(tries,ntests))/ntests) * 10.0e6 print ("%10.1f+-%3.1fus: %s " % (np.mean(r), np.std(r) ,s)) ################# If some one have a complete benchmarking set of functions I would like to use them. Thanks. Hugo Gamboa -------------- next part -------------- An HTML attachment was scrubbed... URL: From nikhil.padmanabhan at gmail.com Wed Mar 8 08:14:07 2006 From: nikhil.padmanabhan at gmail.com (Nikhil Padmanabhan) Date: Wed Mar 8 08:14:07 2006 Subject: [Numpy-discussion] asin, acos, etc... Message-ID: Hi, Is there support for the inverse trigonometric functions in numpy? I haven't been able to find any references in the documentation, but I might have not been looking hard enough. I am using v0.9.4. Thanks in advance, -- Nikhil ------------------------------------ Nikhil Padmanabhan nikhil.padmanabhan at gmail.com From ryanlists at gmail.com Wed Mar 8 08:17:08 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Wed Mar 8 08:17:08 2006 Subject: [Numpy-discussion] asin, acos, etc... In-Reply-To: References: Message-ID: They all begin with arc In [4]: numpy.arc numpy.arccos numpy.arcsin numpy.arctan numpy.arctanh numpy.arccosh numpy.arcsinh numpy.arctan2 In [4]: numpy.arccos(0) Out[4]: 1.5707963267948966 On 3/8/06, Nikhil Padmanabhan wrote: > Hi, > > Is there support for the inverse trigonometric functions in numpy? I > haven't been able to find any references in the documentation, but I > might have not been looking hard enough. > > I am using v0.9.4. > > Thanks in advance, > -- Nikhil > ------------------------------------ > Nikhil Padmanabhan > nikhil.padmanabhan at gmail.com > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From nikhil.padmanabhan at gmail.com Wed Mar 8 08:19:04 2006 From: nikhil.padmanabhan at gmail.com (Nikhil Padmanabhan) Date: Wed Mar 8 08:19:04 2006 Subject: [Numpy-discussion] asin, acos, etc... In-Reply-To: References: Message-ID: <3B1D8207-9332-4904-A67B-7573104FB2EE@gmail.com> Thanks! -- Nikhil ------------------------------------ Nikhil Padmanabhan npadmana at princeton.edu http://wwwphy.princeton.edu/~npadmana nikhil@(609) 258-4355 On Mar 8, 2006, at 11:16 AM, Ryan Krauss wrote: > They all begin with arc > > In [4]: numpy.arc > numpy.arccos numpy.arcsin numpy.arctan numpy.arctanh > numpy.arccosh numpy.arcsinh numpy.arctan2 > > In [4]: numpy.arccos(0) > Out[4]: 1.5707963267948966 > > > On 3/8/06, Nikhil Padmanabhan wrote: >> Hi, >> >> Is there support for the inverse trigonometric functions in numpy? I >> haven't been able to find any references in the documentation, but I >> might have not been looking hard enough. >> >> I am using v0.9.4. >> >> Thanks in advance, >> -- Nikhil >> ------------------------------------ >> Nikhil Padmanabhan >> nikhil.padmanabhan at gmail.com >> >> >> >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by xPML, a groundbreaking scripting >> language >> that extends applications into web and mobile media. Attend the >> live webcast >> and join the prime developer group breaking into this new coding >> territory! >> http://sel.as-us.falkag.net/sel? >> cmd=lnk&kid=110944&bid=241720&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> From svetosch at gmx.net Wed Mar 8 09:00:08 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 8 09:00:08 2006 Subject: [Numpy-discussion] Matrix times scalar is wacky In-Reply-To: <43FEE539.9010606@ieee.org> References: <43FEE539.9010606@ieee.org> Message-ID: <440F0D78.4050406@gmx.net> Travis Oliphant schrieb: > Bill Baxter wrote: > >> Multiplying a matrix times a scalar seems to return junk for some reason: >> >> >>> A = numpy.asmatrix(numpy.rand(1,2)) >> >>> A >> matrix([[ 0.30604211, 0.98475225]]) >> >>> A * 0.2 >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) >> >>> 0.2 * A >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) >> >>> numpy.__version__ >> '0.9.5' >> > This should be fixed in SVN. > > I have just been bitten by this bug, so I would like to ask when to expect the next release. And/or are there any workarounds? Thanks, Sven From oliphant at ee.byu.edu Wed Mar 8 14:54:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 8 14:54:06 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F0D78.4050406@gmx.net> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> Message-ID: <440F606B.8010604@ee.byu.edu> I think it's time for a new release of NumPy. There have been several important bug-fixes and speed improvements recently. Please log any outstanding issues to the ticket system on http://projects.scipy.org/scipy/numpy/timeline I'd like to put another release out over the weekend. -Travis From svetosch at gmx.net Wed Mar 8 15:16:01 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 8 15:16:01 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F606B.8010604@ee.byu.edu> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> <440F606B.8010604@ee.byu.edu> Message-ID: <440F656E.1010206@gmx.net> Travis Oliphant schrieb: > I think it's time for a new release of NumPy. There have been several > important bug-fixes and speed improvements recently. > > Please log any outstanding issues to the ticket system on > http://projects.scipy.org/scipy/numpy/timeline > Don't know if it qualifies as an outstanding issue, but may I ask about the status of making matrix decompositions (and ideally some other functions as well, like lstsq for example) preserve matrix types? Thanks much, Sven From wbaxter at gmail.com Wed Mar 8 15:40:03 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 8 15:40:03 2006 Subject: [Numpy-discussion] Matrix times scalar is wacky In-Reply-To: <440F0D78.4050406@gmx.net> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> Message-ID: Workarounds I know of are: asmatrix(scalar * m.A) or I noticed the other day that scalar division works ok, so if you know scalar is != 0, m / (1.0/scalar) where m is a numpy.matrix, of course. I find myself repeatedly getting bitten by this, since I'm writing code that does lots of lerping between vectors. This is pretty dang ugly to see all over the place: vlerp = asmatrix((1-t) * v1.A + t * v2.A) when it should just be vlerp = (1-t)*v1 + t*v2 yeh yeh, I should write a function... --bb On 3/9/06, Sven Schreiber wrote: > > Travis Oliphant schrieb: > > Bill Baxter wrote: > > > >> Multiplying a matrix times a scalar seems to return junk for some > reason: > >> > >> >>> A = numpy.asmatrix(numpy.rand(1,2)) > >> >>> A > >> matrix([[ 0.30604211, 0.98475225]]) > >> >>> A * 0.2 > >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) > >> >>> 0.2 * A > >> matrix([[ 6.12084210e-002, 7.18482614e-290]]) > >> >>> numpy.__version__ > >> '0.9.5' > >> > > This should be fixed in SVN. > > > > > I have just been bitten by this bug, so I would like to ask when to > expect the next release. And/or are there any workarounds? > > Thanks, > Sven > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Wed Mar 8 15:45:16 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 8 15:45:16 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F656E.1010206@gmx.net> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> <440F606B.8010604@ee.byu.edu> <440F656E.1010206@gmx.net> Message-ID: <440F6C65.80005@ee.byu.edu> Sven Schreiber wrote: >Travis Oliphant schrieb: > > >>I think it's time for a new release of NumPy. There have been several >>important bug-fixes and speed improvements recently. >> >>Please log any outstanding issues to the ticket system on >>http://projects.scipy.org/scipy/numpy/timeline >> >> >> > >Don't know if it qualifies as an outstanding issue, but may I ask about >the status of making matrix decompositions (and ideally some other >functions as well, like lstsq for example) preserve matrix types? > > > We should be doing better in this regard. The __array_wrap__ is being applied (correctly now) in all the linalg functions, so that matrices are returned if they are used. All the ufuncs already preserve arguments. But, we should also peruse the code for asarray calls, save the __array_wrap__ method if available and wrap back the results so that types can be preserved. Someone spoke of writing a decorator to do that automatically which would indeed be nice. If you have examples of functions that are not being preserved, please report them. -Travis From Fernando.Perez at colorado.edu Wed Mar 8 16:17:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 8 16:17:02 2006 Subject: [Numpy-discussion] New release coming In-Reply-To: <440F6C65.80005@ee.byu.edu> References: <43FEE539.9010606@ieee.org> <440F0D78.4050406@gmx.net> <440F606B.8010604@ee.byu.edu> <440F656E.1010206@gmx.net> <440F6C65.80005@ee.byu.edu> Message-ID: <440F73E3.3090503@colorado.edu> Travis Oliphant wrote: > Someone spoke of writing a decorator to do that automatically which > would indeed be nice. As long as it's used in the 'old-fashioned' way (i.e., as func = decorator(func) and not @decorator func so that numpy remains 2.3-compatible. Just a heads-up so that 2.4-isms don't appear unintentionally. Cheers, f From tim.hochberg at cox.net Wed Mar 8 17:12:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 8 17:12:05 2006 Subject: [Numpy-discussion] numexpr In-Reply-To: <20060307185127.GA31063@arbutus.physics.mcmaster.ca> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> Message-ID: <440F80A3.8070006@cox.net> I made some more changes to numexp including adding the optimiztions to pow that we earlier added to numpy. It would be nice to have some way to select the level of optimization. That way, we could work on very aggressive optimization without worrying about messing someone up later who would can't tolerate the moderate accuracy loss. The only idea I have thus far is to create a new Expression each time, instead of just using the precreated 'E'. This would allow us to pass in a context to Expression and this could be passed onto the Nodes that were subsequently created from it so that it would be available where it's needed. That may not make any sense to anyone but David, but I invite you look numexpr/expression.py and numexpr/compiler.py. Then it'll be clear as,..., well clear as something. -tim From dimgmdrp at eaton.com Wed Mar 8 17:15:13 2006 From: dimgmdrp at eaton.com (dimgmdrp) Date: Wed Mar 8 17:15:13 2006 Subject: [Numpy-discussion] Fw: numpy-discussion Message-ID: <000701c64316$b5631970$94097a57@winzix> ----- Original Message ----- From: Burr Colin To: uemyevc at flowserve.com Sent: Monday, March 6, 2006 9:30 AM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kljrnwrziyaflo.gif Type: image/gif Size: 9582 bytes Desc: not available URL: From alexander.belopolsky at gmail.com Wed Mar 8 19:31:02 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed Mar 8 19:31:02 2006 Subject: [Numpy-discussion] Is sum() slow? In-Reply-To: <440DD2FA.4060301@ee.byu.edu> References: <440DD2FA.4060301@ee.byu.edu> Message-ID: On 3/7/06, Travis Oliphant wrote: > ... > I'm a little surprised that Numeric is so much faster for this case as > you show later. > Here is another puzzle: > python -m timeit -s "from Numeric import zeros,sum; x = zeros((2,500),'f')" "sum(x,0)" 100000 loops, best of 3: 6.05 usec per loop > python -m timeit -s "from Numeric import zeros,sum; x = zeros((500,2),'f')" "sum(x,1)" 10000 loops, best of 3: 26.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((2,500),'f')" "x.sum(0)" 10000 loops, best of 3: 22.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((500,2),'f')" "x.sum(1)" 10000 loops, best of 3: 23.6 usec per loop Numpy and Numeric perform very similarly when reducing along the second axis. > If anyone can figure out how to make the NOBUFFER secion in > GenericReduce faster in ufuncobject.c it will be greatly welcomed. Here is my $.02: BTW, I've tried to take loop->... redirections out of the loop: no effect with gcc -O3 . From ndarray at mac.com Wed Mar 8 19:33:02 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 8 19:33:02 2006 Subject: [Numpy-discussion] Is sum() slow? In-Reply-To: <440DD2FA.4060301@ee.byu.edu> References: <440DD2FA.4060301@ee.byu.edu> Message-ID: On 3/7/06, Travis Oliphant wrote: > ... > I'm a little surprised that Numeric is so much faster for this case as > you show later. > Here is another puzzle: > python -m timeit -s "from Numeric import zeros,sum; x = zeros((2,500),'f')" "sum(x,0)" 100000 loops, best of 3: 6.05 usec per loop > python -m timeit -s "from Numeric import zeros,sum; x = zeros((500,2),'f')" "sum(x,1)" 10000 loops, best of 3: 26.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((2,500),'f')" "x.sum(0)" 10000 loops, best of 3: 22.8 usec per loop > python -m timeit -s "from numpy import zeros; x = zeros((500,2),'f')" "x.sum(1)" 10000 loops, best of 3: 23.6 usec per loop Numpy and Numeric perform very similarly when reducing along the second axis. > If anyone can figure out how to make the NOBUFFER secion in > GenericReduce faster in ufuncobject.c it will be greatly welcomed. Here is my $.02: BTW, I've tried to take loop->... redirections out of the loop: no effect with gcc -O3 . From tim.hochberg at cox.net Wed Mar 8 20:50:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 8 20:50:03 2006 Subject: [Numpy-discussion] numexpr: optimizing pow In-Reply-To: <440F80A3.8070006@cox.net> References: <440AF97C.5040106@cox.net> <440CFCA1.70007@cox.net> <440DD209.5060900@cox.net> <20060307185127.GA31063@arbutus.physics.mcmaster.ca> <440F80A3.8070006@cox.net> Message-ID: <440FB3C7.7070704@cox.net> I just checked in some changes that do aggressive optimization on the pow operator in numexpr. Now all integral and half integral powers between [-50 and 50] are computed using multiples and sqrt. (Empirically 50 seemed to be the closest round number to the breakeven point.) I mention this primarily because I think it's cool. But also, it's the kind of optimization that I don't think would be feasible in numpy itself short of defining a whole pile of special cases, either separate ufuncs or separate loops within a single ufunc, one for each case that needed optimizing. Otherwise the bookkeeping overhead would overwhelm the savings of replacing pow with multiplies. Now all of the bookkeeping is done in Python, which makes it easy; and done once ahead of time and translated into bytecode, which makes it fast. The actual code that does the optimization is included below for those of you interested enough to care, but not interested enough to check it out of the sandbox. It could be made simpler, but I jump through some hoops to avoid unnecessary mulitplies. For instance, starting 'r' as 'OpNode('ones_like', [a])' would simplify things signifigantly, but at the cost of adding an extra multiply in most cases. That brings up an interesting idea. If 'mul' were made smarter, so that it recognized OpNode('ones_like', [a]) and ConstantNode(1), then not only would that speed some 'mul' cases up, it would simplify the code for 'pow' as well. I'll have to look into that tomorrow. Regards, -tim if True: # Aggressive RANGE = 50 # Approximate break even point with pow(x,y) # Optimize all integral and half integral powers in [-RANGE, RANGE] # Note: for complex numbers RANGE would be larger. if (int(2*x) == 2*x) and (-RANGE <= abs(x) <= RANGE): n = int(abs(x)) ishalfpower = int(abs(2*x)) % 2 r = None p = a mask = 1 while True: if (n & mask): if r is None: r = p else: r = OpNode('mul', [r,p]) mask <<= 1 if mask > n: break p = OpNode('mul', [p,p]) if ishalfpower: sqrta = OpNode('sqrt', [a]) if r is None: r = sqrta else: r = OpNode('mul', [r, sqrta]) if r is None: r = OpNode('ones_like', [a]) if x < 0: r = OpNode('div', [ConstantNode(1), r]) return r From alodiaswint at varingen.no Thu Mar 9 05:46:02 2006 From: alodiaswint at varingen.no (Alodia Swint) Date: Thu Mar 9 05:46:02 2006 Subject: [Numpy-discussion] Re: PTaramcy news Message-ID: <000001c6437f$a0565d80$781fa8c0@iyw43> sD qVxiaaugsrka $ a 69,95 (10 sH t Cd abIets) iVzauIeieuym $ n 105,45 (3 DY 0 t ep abIets) fCiilalImits $ 9 u 9,95 (1 cD 0 tab XL Iets) MO And many oth rY er http://cig56.hospotte.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Thu Mar 9 05:58:00 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 9 05:58:00 2006 Subject: [Numpy-discussion] ndarray - reshaping a sub-class Message-ID: <44103429.4070502@sympatico.ca> Is this a bug which should be reported to the Tracker? Colin W. http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3038233 From ndarray at mac.com Thu Mar 9 06:27:12 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 9 06:27:12 2006 Subject: [Numpy-discussion] ndarray - reshaping a sub-class In-Reply-To: <44103429.4070502@sympatico.ca> References: <44103429.4070502@sympatico.ca> Message-ID: I would say yes, report it. It is trivial to fix: just change asarray to asanyarray in reshape definition in oldnumeric.py, but I am not sure this is desirable. Oldnumeric is a backward compatibility module and probably matches old Numeric behavior in this respect. On 3/9/06, Colin J. Williams wrote: > Is this a bug which should be reported to the Tracker? > > Colin W. > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3038233 > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Thu Mar 9 10:46:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 9 10:46:04 2006 Subject: [Numpy-discussion] Archives Message-ID: <441077E1.7070007@sympatico.ca> The SourceForge archive for Numpy seemed to fizzle late in February but the Active State archive appears current. Colin W. "http://sourceforge.net/mailarchive/forum.php?forum_id=4890&max_rows=25&style=flat&viewmonth=200602&viewday=25" From efiring at hawaii.edu Thu Mar 9 10:50:02 2006 From: efiring at hawaii.edu (Eric Firing) Date: Thu Mar 9 10:50:02 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: References: <4404A71B.10600@cox.net> <4404F6B3.9080809@pfdubois.com> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> Message-ID: <441078AB.5090003@hawaii.edu> Travis et al., Jeff Whitaker found that the imaged_masked.py example (with the colorbar line uncommented) in matplotlib 0.87 triggered a numpy bug--the script works normally with Numeric and numarray. He committed a workaround to mpl svn, but I suspect you may want to track it down and squash it in numpy before the next release. It is not clear to me whether it is really in ma, or merely revealed by ma. The attached script triggers the bug. Thanks. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy_ma_bug.py Type: text/x-python Size: 144 bytes Desc: not available URL: From dav at alum.mit.edu Thu Mar 9 11:40:02 2006 From: dav at alum.mit.edu (Dav Clark) Date: Thu Mar 9 11:40:02 2006 Subject: [Numpy-discussion] numpy with mod_python... Message-ID: <200603091139.11872.dav@alum.mit.edu> So, This seems like an obvious thing, but there are so many levels of things involved that I'd really appreciate some guidance. 'import numpy' works fine most places, but it strangely fails when done from ../site-packages/numpy. That, I don't care so much, but I found that problem in persuing why 'import numpy' fails when called from a script within mod_python. In this case, the cwd is '/'. The precise error (in both cases) is: --- /usr/lib/python2.4/site-packages/numpy/core/__init__.py 20 21 __all__ = ['char','rec','memmap','ma'] ---> 22 __all__ += numeric.__all__ 23 __all__ += oldnumeric.__all__ 24 __all__ += defmatrix.__all__ NameError: name 'numeric' is not defined --- The obvious solution / hack to get this working in mod_python is to simply os.chdir() before running an import. Except that doesn't work. So, ideas about how to hack mod_python or numpy - or even better how to _fix_ numpy - would be greatly appreciated! Thanks, -- Dav Clark Holistic Neuroscientist 917-544-8408 ))<>(( From robert.kern at gmail.com Thu Mar 9 13:47:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 9 13:47:02 2006 Subject: [Numpy-discussion] Re: numpy with mod_python... In-Reply-To: <200603091139.11872.dav@alum.mit.edu> References: <200603091139.11872.dav@alum.mit.edu> Message-ID: Dav Clark wrote: > So, > > This seems like an obvious thing, but there are so many levels of things > involved that I'd really appreciate some guidance. > > 'import numpy' works fine most places, but it strangely fails when done > from ../site-packages/numpy. > > That, I don't care so much, but I found that problem in persuing why 'import > numpy' fails when called from a script within mod_python. In this case, the > cwd is '/'. > > The precise error (in both cases) is: > > > --- > /usr/lib/python2.4/site-packages/numpy/core/__init__.py > 20 > 21 __all__ = ['char','rec','memmap','ma'] > ---> 22 __all__ += numeric.__all__ > 23 __all__ += oldnumeric.__all__ > 24 __all__ += defmatrix.__all__ > > NameError: name 'numeric' is not defined > --- > > The obvious solution / hack to get this working in mod_python is to simply > os.chdir() before running an import. Except that doesn't work. > > So, ideas about how to hack mod_python or numpy - or even better how to _fix_ > numpy - would be greatly appreciated! Before you import numpy, what is your sys.path? Where is numpy/? What files are in numpy/core/? What version of numpy are you using? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zpincus at stanford.edu Thu Mar 9 15:55:04 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu Mar 9 15:55:04 2006 Subject: [Numpy-discussion] argmax() return type can't index python lists Message-ID: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> Hi folks, I'm using a fresh build of numpy and python 2.5, both checked out from SVN yesterday. It seems that numpy.argmax() and ndarray.argmax() are now returning 'int32scalar' types instead of 'int' types. Despite the fact that the former seems to inherit from the latter, 'int32scalar' types do not appear to be usable as indices into python lists. Anyhow, I'm not sure if this is a python 2.5 regression, or a problem in all versions of python that's exposed by this behavior, or a numpy problem. Any thoughts? Zach Test case: In [1]: import numpy In [2]: l = [1,2,3] In [3]: l[numpy.argmax(l)] : list indices must be integers In [4]: type(numpy.argmax(l)) Out[4]: In [5]: print numpy.version.version 0.9.6.2208 In [6]: import sys; print sys.version 2.5a0 (trunk:42924M, Mar 8 2006, 19:29:24) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] From dbrown at ucar.edu Thu Mar 9 15:57:04 2006 From: dbrown at ucar.edu (David Ian Brown) Date: Thu Mar 9 15:57:04 2006 Subject: [Numpy-discussion] numpy C API bugs? Message-ID: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> Hi Travis and NumPy developers, I have recently been engaged in updating the PyNIO module (http://www.pyngl.ucar.edu/Nio.html) to work with the NumPy C API. It was originally coded using the Numeric's C API. In the process I have encountered what appear to me to be a couple of bugs in the interface. They were relatively minor and not difficult to work around, but you would probably like to know of them if you don't already. In the NumPy version of arrayobject.h the macros PyArray_GETITEM and PyArray_SETITEM are defined in way that does not work. Here is the GETITEM macro: #define PyArray_GETITEM(obj,itemptr) \ ((char *)itemptr, \ (PyArrayObject *)obj); When I try to use this macro in my code I get a compile error, because 'getitem' is not a member of NumPy's PyArray_Descr struct. Instead it is further nested within the struct member 'PyArray_ArrFuncs *f'. So, unless I misunderstand, the macro needs to access the 'getitem' function as ((PyArrayObject *)(obj))->descr->f->getitem(... The other issue with 'getitem', apparently the actual method, not just this macro, is that the 2 arguments are ordered oppositely from what is stated in "Guide To NumPy", as well as from the way this method was implemented in Numeric. Notice in the macro (and in NumPy's implementation), 'itemptr' is first and the PyArrayObject pointer is second. The documentation and Numeric has it the other way around. I hope this is useful. -dave From dbrown at ucar.edu Thu Mar 9 16:59:01 2006 From: dbrown at ucar.edu (David Ian Brown) Date: Thu Mar 9 16:59:01 2006 Subject: [Numpy-discussion] numpy C API bugs? In-Reply-To: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> References: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> Message-ID: <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> On Mar 9, 2006, at 4:56 PM, David Ian Brown wrote: > > Hi Travis and NumPy developers, > > I have recently been engaged in updating the PyNIO module > (http://www.pyngl.ucar.edu/Nio.html) > to work with the NumPy C API. It was originally coded using the > Numeric's C API. > > In the process I have encountered what appear to me to be a couple of > bugs in the interface. They were relatively minor and not difficult > to work around, but you would probably like to know of them if you > don't already. > > In the NumPy version of arrayobject.h the macros PyArray_GETITEM and > PyArray_SETITEM > are defined in way that does not work. Here is the GETITEM macro: > > #define PyArray_GETITEM(obj,itemptr) \ > ((char *)itemptr, \ > (PyArrayObject *)obj); > > When I try to use this macro in my code I get a compile error, because > 'getitem' is not a member of NumPy's > PyArray_Descr struct. Instead it is further nested within the struct > member 'PyArray_ArrFuncs *f'. > So, unless I misunderstand, the macro needs to access the 'getitem' > function as > > ((PyArrayObject *)(obj))->descr->f->getitem(... > > The other issue with 'getitem', apparently the actual method, not just > this macro, is that the 2 arguments are > ordered oppositely from what is stated in "Guide To NumPy", as well > as from the way this method was > implemented in Numeric. Notice in the macro (and in NumPy's > implementation), 'itemptr' is first and > the PyArrayObject pointer is second. The documentation and Numeric has > it the other way around. Correction: actually "Guide to NumPy" has the 'getitem' function documented with the arguments ordered as implemented in NumPy (but still opposite from Numeric). Only the macro is documented opposite from its implementation. > > I hope this is useful. > -dave > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ndarray at mac.com Thu Mar 9 17:33:01 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 9 17:33:01 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: <441078AB.5090003@hawaii.edu> References: <4404A71B.10600@cox.net> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <441078AB.5090003@hawaii.edu> Message-ID: I would say it is an ma bug, but don't know how to fix it properly without changing numpy scalar arithmetic to call an enhanced version of __array__ and pass context in. The core problem can be demonstrated by the following session: >>> from numpy import * >>> x = ma.array([1],mask=[1]) >>> int_(1)*x Traceback (most recent call last): File "", line 1, in ? File ".../numpy/core/ma.py", line 622, in __array__ raise MAError, \ numpy.core.ma.MAError: Cannot automatically convert masked array to numeric because data is masked in one or more locations. Note that x*int_(1) works as expected. This is so because python dispatches multiplication to int.__mul__ rather than ma.array.__mul__ if int_(1) is the first multiplicand. I've fixed a similar problem for array*ma.array case and array(1)*x works in the current version of numpy. I will not have time to work on this before the weekend, so if someone is motivated enought to fix this bug before the upcoming release, please take a look at http://projects.scipy.org/scipy/numpy/wiki/MaskedArray ("Ufuncs and Masked Arrays"). It should be straightforward to generalize that approach for array scalars. On 3/9/06, Eric Firing wrote: > Travis et al., > > Jeff Whitaker found that the imaged_masked.py example (with the colorbar > line uncommented) in matplotlib 0.87 triggered a numpy bug--the script > works normally with Numeric and numarray. He committed a workaround to > mpl svn, but I suspect you may want to track it down and squash it in > numpy before the next release. It is not clear to me whether it is > really in ma, or merely revealed by ma. The attached script triggers > the bug. > > Thanks. > > Eric > > > > From oliphant.travis at ieee.org Thu Mar 9 21:08:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 9 21:08:00 2006 Subject: [Numpy-discussion] argmax() return type can't index python lists In-Reply-To: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> References: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> Message-ID: <4411096A.50203@ieee.org> Zachary Pincus wrote: > Hi folks, > > I'm using a fresh build of numpy and python 2.5, both checked out from > SVN yesterday. It seems that numpy.argmax() and ndarray.argmax() are > now returning 'int32scalar' types instead of 'int' types. This has always been the case for numpy so it's not new. > Despite the fact that the former seems to inherit from the latter, > 'int32scalar' types do not appear to be usable as indices into python > lists. This is new and is a Python 2.5 problem it would appear. > > Anyhow, I'm not sure if this is a python 2.5 regression, or a problem > in all versions of python that's exposed by this behavior, or a numpy > problem. > Note that the __index__ PEP I wrote was just accepted and checked-in so in fact *all* numpy scalar types will be acceptable as indices in Python2.5 (but we have to write the code for that still in numpy). So, it's possible that something in Python 2.5 changed is causing it not to work. This very-well could be a bug in the new implementation of __index__. -Travis From oliphant.travis at ieee.org Thu Mar 9 21:29:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 9 21:29:02 2006 Subject: [Numpy-discussion] numpy C API bugs? In-Reply-To: <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> References: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> Message-ID: <44110E6F.4080703@ieee.org> David Ian Brown wrote: >> >> I have recently been engaged in updating the PyNIO module >> (http://www.pyngl.ucar.edu/Nio.html) >> to work with the NumPy C API. It was originally coded using the >> Numeric's C API. >> >> In the process I have encountered what appear to me to be a couple >> of bugs in the interface. They were relatively minor and not >> difficult to work around, but you would probably like to know of them >> if you don't already. >> >> In the NumPy version of arrayobject.h the macros PyArray_GETITEM and >> PyArray_SETITEM >> are defined in way that does not work. Here is the GETITEM macro: >> >> #define PyArray_GETITEM(obj,itemptr) \ >> ((char *)itemptr, \ >> (PyArrayObject *)obj); >> Thank you very much for catching this. The introduction of 'f' was more recent then the writing of these macros and so this error crept in. I'm definitely glad to fix it. > Correction: actually "Guide to NumPy" has the 'getitem' function > documented with the arguments ordered > as implemented in NumPy (but still opposite from Numeric). Only the > macro is documented opposite from > its implementation. I think further clarification is needed. First of all Numeric did not have macros. They just had function pointers to getitem. Second, Numpy's function pointers to getitem and setitem are almost exactly the same. The only difference is the *addition* of another argument to both functions to allow the array to be passed in (so that the size of the array and whether or not it's well-behaved can be known). So, the order is *not* different in NumPy and Numeric, there is just another argument needed. The new macros introduced do have a different order (and I don't think it's documented oppositely --- it's perhaps not documented well at all, however :-) and so may be confusing). It is recommended to use the macros. They are PyArray_GETITEM(array, itemptr) Return a Python object corresponding to the item in the array at the location itemptr. PyArray_SETITEM(array, itemptr, obj) Set object into the array at the memory pointed to by itemptr (an attempt is made to convert to the right type if that is possible). Thanks for the report. -Travis From zpincus at stanford.edu Thu Mar 9 21:34:02 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Thu Mar 9 21:34:02 2006 Subject: [Numpy-discussion] argmax() return type can't index python lists In-Reply-To: <4411096A.50203@ieee.org> References: <753FA7E5-C708-4F38-8F24-269D3D45EA5B@stanford.edu> <4411096A.50203@ieee.org> Message-ID: <9F90F4AA-70DF-48AE-B295-1AEDE1BB172C@stanford.edu> Thanks for your reply, Travis. > Note that the __index__ PEP I wrote was just accepted and checked- > in so in fact *all* numpy scalar types will be acceptable as > indices in Python2.5 (but we have to write the code for that still > in numpy). So, it's possible that something in Python 2.5 changed > is causing it not to work. > > This very-well could be a bug in the new implementation of __index__. The problem is that I can't reproduce this bug outside of numpy. If I go to the python people, they'll most likely say it's a numpy problem, unless I can show some other test case. Right now, all of the following classes work as indices in 2.5. The only thing that doesn't work is a raw numpy.int32 value. I'd be happy to file a bug report on python if I could figure out a test case. Zach Cases which work fine: class int_sublcass(int): pass class empty(object): pass class int_1(int, empty): pass class int_2(empty, int): pass class int_numpy_1(numpy.signedinteger, int): def __init__(self, i): int.__init__(self, i) class int_numpy_2(int, numpy.signedinteger): def __init__(self, i): int.__init__(self, i) l = [1,2,3] l[int_sublcass(1)] l[int_1(1)] l[int_2(1)] l[int_numpy_1(1)] l[int_numpy_2(1)] From oliphant.travis at ieee.org Fri Mar 10 02:32:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 10 02:32:04 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: References: <4404A71B.10600@cox.net> <44051AC1.9000908@ieee.org> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <441078AB.5090003@hawaii.edu> Message-ID: <4411557B.4020006@ieee.org> Sasha wrote: > I would say it is an ma bug, but don't know how to fix it properly > without changing numpy scalar arithmetic to call an enhanced version > of __array__ and pass context in. > I think it was an array scalar bug. I've fixed it by always calling the ufunc (which handles conversions to and from other objects better than the array scalars were doing in the generic arithmetic code). The result is shorter code and symmetric handling of the * vs. * case. I also changed the error to a warning for backward compatibility. Calling __array__() on a masked array should succeed and we will assume the user knows what they are doing. -Travis From pfdubois at gmail.com Fri Mar 10 07:07:12 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Fri Mar 10 07:07:12 2006 Subject: [Numpy-discussion] numpy.ma bug In-Reply-To: <4411557B.4020006@ieee.org> References: <4404A71B.10600@cox.net> <440530AC.2010000@enthought.com> <4407843A.5010300@cox.net> <44088897.6070802@cox.net> <441078AB.5090003@hawaii.edu> <4411557B.4020006@ieee.org> Message-ID: It was a design decision: if an automatic conversion of a masked array to a numeric one is attempted, and there actually are masked values, it is logical to either: a. raise an exception, because there is no equivalent numeric array, or; b. fill the masked value with the fill value (b) suffers from a problem: the default fill value might not be appropriate for the use that is about to occur. My original conclusion was that it was better to choose (a) to force the user to explicitly do the conversion with x.fill(value) and pass THAT to whatever was going to consume it. I don't think I've changed my mind. On 10 Mar 2006 02:32:19 -0800, Travis Oliphant wrote: > > Sasha wrote: > > I would say it is an ma bug, but don't know how to fix it properly > > without changing numpy scalar arithmetic to call an enhanced version > > of __array__ and pass context in. > > > I think it was an array scalar bug. I've fixed it by always calling the > ufunc (which handles conversions to and from other objects better than > the array scalars were doing in the generic arithmetic code). > > The result is shorter code and symmetric handling of the > * vs. * case. > > I also changed the error to a warning for backward compatibility. > Calling __array__() on a masked array should succeed and we will assume > the user knows what they are doing. > > -Travis > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Mar 10 10:20:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Mar 10 10:20:03 2006 Subject: [Numpy-discussion] Key 'i' disappeared in numpy.typeNA Message-ID: <200603101919.40714.faltet@carabos.com> Hi, I'm in a hurry, so not able to track this right down, but I have just checked the numpy SVN version of numpy and discovered that the 'i' entry for typeNA has disappeared: >>> numpy.__version__ '0.9.6.2214' >>> numpy.typeNA['i'] Traceback (most recent call last): File "", line 1, in ? KeyError: 'i' >>> numpy.typeNA['l'] 'Int32' Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From dbrown at ucar.edu Fri Mar 10 11:52:07 2006 From: dbrown at ucar.edu (David Ian Brown) Date: Fri Mar 10 11:52:07 2006 Subject: [Numpy-discussion] numpy C API bugs? In-Reply-To: <44110E6F.4080703@ieee.org> References: <062955d983f4b4a829f339cbefc65eeb@ucar.edu> <840ed2224de5fbaff76725bfbccb5a1f@ucar.edu> <44110E6F.4080703@ieee.org> Message-ID: <10dbcbe9aebaf94303d3194a1f97dde4@ucar.edu> Hi Travis, Yes, now I see that I just wasn't looking closely enough at the the docs, that Numeric's getitem did have only one argument, and the docs have the arguments in the correct order both for the function and the macro. I was just confused by the fact that the macro has the arguments in the opposite order of the function. So the only problem was the missing 'f' in the macro. -dave On Mar 9, 2006, at 10:28 PM, Travis Oliphant wrote: > David Ian Brown wrote: >>> >>> I have recently been engaged in updating the PyNIO module >>> (http://www.pyngl.ucar.edu/Nio.html) >>> to work with the NumPy C API. It was originally coded using the >>> Numeric's C API. >>> >>> In the process I have encountered what appear to me to be a couple >>> of bugs in the interface. They were relatively minor and not >>> difficult to work around, but you would probably like to know of >>> them if you don't already. >>> >>> In the NumPy version of arrayobject.h the macros PyArray_GETITEM and >>> PyArray_SETITEM >>> are defined in way that does not work. Here is the GETITEM macro: >>> >>> #define PyArray_GETITEM(obj,itemptr) \ >>> ((char *)itemptr, \ >>> (PyArrayObject *)obj); >>> > > Thank you very much for catching this. The introduction of 'f' was > more recent then the writing of these macros and so this error crept > in. I'm definitely glad to fix it. > >> Correction: actually "Guide to NumPy" has the 'getitem' function >> documented with the arguments ordered >> as implemented in NumPy (but still opposite from Numeric). Only the >> macro is documented opposite from >> its implementation. > > I think further clarification is needed. > First of all Numeric did not have macros. They just had function > pointers to getitem. Second, Numpy's function pointers to getitem and > setitem are almost exactly the same. The only difference is the > *addition* of another argument to both functions to allow the array to > be passed in (so that the size of the array and whether or not it's > well-behaved can be known). > So, the order is *not* different in NumPy and Numeric, there is just > another argument needed. > The new macros introduced do have a different order (and I don't think > it's documented oppositely --- it's perhaps not documented well at > all, however :-) and so may be confusing). > > It is recommended to use the macros. They are > > PyArray_GETITEM(array, itemptr) > > Return a Python object corresponding to the item in the array at the > location itemptr. > > PyArray_SETITEM(array, itemptr, obj) > > Set object into the array at the memory pointed to by itemptr (an > attempt is made to convert to the right type if that is possible). > > Thanks for the report. > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From dav at alum.mit.edu Fri Mar 10 12:22:03 2006 From: dav at alum.mit.edu (Dav Clark) Date: Fri Mar 10 12:22:03 2006 Subject: [Numpy-discussion] Further info on mod_python / numpy crashing Message-ID: <4411DFB4.8000103@alum.mit.edu> First - I realized I failed to include version numbers. I'm using numpy 0.9.5, and mod_python 3.1.4 and apache 2.0.54. Standard (I think) for red hat FC4 on an intel box. So, I've spent some time tracing through mod_python... the bad code is here, from the import_module def in mod_python/apache.py: parts = module_name.split('.') for i in range(len(parts)): f, p, d = imp.find_module(parts[i], path) try: mname = ".".join(parts[:i+1]) module = imp.load_module(mname, f, p, d) finally: if f: f.close() if hasattr(module, "__path__"): path = module.__path__ From a mod_python interpreter - all calls to import get replaced with import_module (to prevent reloads). The setting of the 'path' parameter in this way is (I think) invoking the same bug that causes the crash when imoporting numpy from ../site-packages/numpy. Can anyone even replicate this? Should I cross-post this to mod_python? Cheers, Dav From dav at alum.mit.edu Fri Mar 10 14:58:03 2006 From: dav at alum.mit.edu (Dav Clark) Date: Fri Mar 10 14:58:03 2006 Subject: [Numpy-discussion] solution for mod_python / numpy interaction Message-ID: <44120475.7020706@alum.mit.edu> So mod_python 3.2(.8) included a major overhaul of the import mechanism. I'm still not sure what the problem is, and I think an 'import numpy' from site-packages/numpy _shouldn't_ cause an error (though it does). But I can at least do my work. DC From mark at mitre.org Fri Mar 10 17:44:05 2006 From: mark at mitre.org (Mark Heslep) Date: Fri Mar 10 17:44:05 2006 Subject: [Numpy-discussion] Opinions on the book H.P. Langtangen: Python Scripting for Computational Science, 2nd ed. In-Reply-To: References: Message-ID: <44122B58.1010707@mitre.org> +1 Chapter 8 Advanced Py is one of the best Python howto's Ive seen, and the other chapters covering the Numpy API and especially howto use are excellent. Bruce Southey wrote: >Hi, >I have the first edition which is very comprehensive on Python, >numarray and Numeric - note that by NumPy he is means only Numeric and >numarray and not the new numpy. If you have the basic of Python and >numerical computing then it is a great book as it cover many different >topics. For example, there is considerable detail on using Fortran and >C/C++ with Numeric and numarray including examples. > >For the majority of the material in the book is still valid for the >new Numpy. He does provide a number of advanced topics in Python, >numarray and Numeric that are hard to find elsewhere. > >If you are going to extensive numerical work then it is really worth it. > >Bruce > > >On 3/2/06, Zden?k Hur?k wrote: > > >>Hello, >> >>Can anybody recommend the book: >> >>Python Scripting for Computational Science >>Series: Texts in Computational Science and Engineering, Vol. 3 >>Langtangen, Hans Petter >>2nd ed., 2006, XXIV, 736 p. 33 illus., Hardcover >>ISBN: 3-540-29415-5 >> >>I am just a beginner in Python programming (but I master C, Matlab). I do >>research in applied numerical computing (oriented at control design). Is >>this a book for me? It is hard to guess just from the table contents. I >>especially doubt if the information on Numpy package is relevant at all >>with the wild development witnessed in the past recent months. However, >>peeling off these "numerical" parts, isn't buying some comprehensive Python >>book a better buy? >> >>Thanks for your tips, >>Zdenek Hurak >> >> >> >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting language >>that extends applications into web and mobile media. Attend the live webcast >>and join the prime developer group breaking into this new coding territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >???????????????????????????????????????????+?^????X???'???u????w[????x,r???)?????? ??{^????bq?b?{"??0y??vj?W?y??? mzwm??b?? m???????????)?u????^? >????y?"????a????? ?x-z??????m??????j??????F ?w????????H????n'??{?O?j?^?????????????????????????????????????6????b????*'????x%??M?jr???r?,????+-??(?? ~??{???m????X???????y?+???z????l?X??)???????v+ ??" > > > From mark at mitre.org Fri Mar 10 18:15:01 2006 From: mark at mitre.org (Mark Heslep) Date: Fri Mar 10 18:15:01 2006 Subject: [Numpy-discussion] NumPy wrapper for C output buffer of unknown size Message-ID: <4412327F.5020303@mitre.org> I'm trying to come up with a _generic_, _reusable_ numpy wrapper for a C output buffer of unknown size. That is, only the buffer pointer is available from a function call, and it's size is available elsewhere. For example, the buffer access might look like this: char * func_returns_ptr_to_allocated_buffer() { ...} And the size of the buffer is determined elsewhere by some other 'machinery'. I'm using Swig* here, and left to itself Swig will neatly map the char* to a Swig Python object, something like . That's fine unless you want access to the buffer elements via NumPy. I'm familiar with many of the fine NumPy typemaps posted here and on the net, but they all assume a known array size and use a PyArray_FromDimsandData (...SimpleFrom... w/ recent Numpy) with a dims argument provided. Instead, is there some method where I can return a numpy array of size zero (one?) that effectively captures the data pointer and then allows me to resize on the Python side? I note the the restrictions on the NumPy resize method: no previous reference, and self->base=Null. Its simple to solve this with a helper function custom to this application but its not maintainable with the app-library as it evolves. Extending the solution to any other data type is also no problem. So, as stated above, is there a generic, numpy-thonic way to solve this? Mark * Im open to other methods -ctypes, etc From mpi at osc.kiku.dk Sun Mar 12 04:22:02 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Sun Mar 12 04:22:02 2006 Subject: [Numpy-discussion] C-extensions to numpy - a little help is needed Message-ID: Hey, I am trying to understand how to write numpy extensions, in order to be able to do various things fast, that I need in my own apps. I have looked at the examples from the Numeric package, which I managed to get working with numpy, so I guess my questions holds for this forum. There's one detail, that I don't understand in the examples provided with the Numeric documentation, namely the use of Py_DECREF(...) which is used in the first example but not in the last. Let me be more specific: In the example, showing how to implement a trace() function, you access the data using array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); do some summing to find the trace and then decrement and return the result Py_DECREF(array); return PyFloat_FromDouble(sum); Now if you look at the other example for the matrix-vector multiply, the matrix and vector data are accessed using a similar approach PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; ... if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); ... vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); ... result = (PyArrayObject *) PyArray_FromDims(1, dimensions, PyArray_DOUBLE); Then calculate the DGEMV storing the result in result->data and return with return PyArray_Return(result); Finally here's my question: ---------------------------- Since the two pointers *matrix and *vector are created just like *array in the trace() example, how come they are not decremented with two calls to Py_DECREF(matrix); Py_DECREF(vector); Any help is appreciated. // Mads From Norbert.Nemec.list at gmx.de Sun Mar 12 05:59:02 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Sun Mar 12 05:59:02 2006 Subject: [Numpy-discussion] Patches for numpy.linalg and scipy.linalg Message-ID: <441428FB.4080003@gmx.de> Over the past few days I worked a bit on the linalg packages in both numpy and scipy. Attached are patches for both numpy and scipy. Both are a bit interdependent, so I'm submitting all in one cross-posted mail. I don't know about the SVN-write-access policy of numpy/scipy. If it is not kept too restrictive, it might be convenient for everyone to give me write access. I cannot promise to find much time working on the code, but linalg sure needs some cleanup and I might continue doing a bit of it. Here a few explanations about the attached patches: numpy-1-cleanup.diff just a bit of cosmetics to make the other patches cleaner numpy-2-rename-functions.diff numpy.linalg used to contain a number of aliases (like 'inv'=='inverse' and others) this is pointless and more confusing then helpful. Unless there is a really good reason, a library should offer one name only. I chose the short version of each name as the "official" name, because I think all the abbreviations are well-known and easy to understand for anyone handling numerical linear algebra. Currently all the definition that would be needed for backwards-compatibility are deactivated in a "if False:" block. If people think they should be deprecated slowly, this block could easily be re-activated. numpy-3-svd-compute_uv.diff numpy.linalg.svd now also has the compute_uv option that scipy.linalg.svd already had. Default behavior is the same as before. numpy-4-svd-bug-workaround.diff the dgesdd function of the lapack library installed on my system seems to contain a strange bug. Should probably be investigated and fixed. For the moment I just included a workaround. numpy-5-norm-copy-from-scipy.diff copied and the "norm" function from scipy.linalg and adjusted internals to work in the new environment numpy-6-norm-change-default.diff the 'scipy.linalg.norm' function was sub-optimal: for matrices, the frobenius norm is faster then the "max(svd())" by about an order of magnitude for all the test cases that I came up with. It also is invariant under orthogonal/unitary transformations, which makes it the best candidate to be the default in any case that I could come up with. The computation of the Frobenius norm can be done with the same line of code used for the vector-square norm and can even be generalized to arrays of arbitrary rank, always giving a reasonable norm for both real and complex numbers. The most straightforward way to clean up a number of inefficiencies in the definition of "norm" was to introduce a norm with "ord=None" which always calculates sqrt(sum((conjugate(x)*x).ravel())) which is generally the most efficient implementation and does not even need a check of the rank of the array. I also made this choice the default of the "norm" function. I know that this may cause hard-to-find errors in existing code if people explicitely want the "svd"-norm for a matrix and relied on the old default setting. Personally, I don't believe this is a real danger, because I don't believe there are many algorithms that depend on using the svd-norm and break if frobenius is used. If people are strictly against changing the default behavior, the patch should still be accepted and only the function header changed back. numpy-7-castCopyAndTranspose.diff simplyfied this function for purely aesthetic reasons: I believe a function that can be written cleanly in two lines should not be extended unnessecarily to 6 lines... numpy-8-dual.diff added 'norm', 'eigh' and 'eigvalsh' to the list of dual functions. scipy-1-eigh-eigvalsh.diff added two new functions to scipy.linalg. All the interfaces were there already... scipy-2-dual-norm.diff added 'norm' to the list of dual functions scipy-3-norm-change-default.diff changed the norm in the same way as "numpy.linalg.norm" was changed. See comments above. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-1-cleanup.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-2-rename-functions.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-3-svd-compute_uv.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-4-svd-bug-workaround.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-5-norm-copy-from-scipy.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-6-norm-change-default.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-7-castCopyAndTranspose.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numpy-8-dual.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy-1-eigh-eigvalsh.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy-2-dual-norm.diff URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: scipy-3-norm-change-default.diff URL: From a.mcmorland at auckland.ac.nz Sun Mar 12 10:02:02 2006 From: a.mcmorland at auckland.ac.nz (Angus McMorland) Date: Sun Mar 12 10:02:02 2006 Subject: [Numpy-discussion] Numpy implementation of IDL & pdl's rebin? Message-ID: <4414620C.3040202@auckland.ac.nz> Hi, I've been interested in switching from pdl and IDL to numpy (or its predecessors) for a couple of years now, but have always been put off by the lack of a simple, built-in n-dimensional rebin command for re-sampling an array to any arbitrary dimensions. This is a feature I require frequently in these other languages. I came across this thread some time ago: http://article.gmane.org/gmane.comp.python.numeric.general/892/match=rebin which discusses various implementation strategies and encouragingly suggests the addition of a built-in command. I haven't come across any progress since then and wondered if anything had been done. At one stage I tried to implement one myself, but ran out of time and got lost trying to find the required n-dimensional interpolation routines. Could someone please tell me if (a) something has been done, or (b) if anyone with some experience in this area would be interested in collaborating to complete the task? Thanks, Angus. -- Angus McMorland email a.mcmorland at auckland.ac.nz mobile +64-21-155-4906 PhD Student, Neurophysiology / Multiphoton & Confocal Imaging Physiology, University of Auckland phone +64-9-3737-599 x89707 Armourer, Auckland University Fencing Secretary, Fencing North Inc. From oliphant.travis at ieee.org Sun Mar 12 15:04:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 12 15:04:01 2006 Subject: [Numpy-discussion] C-extensions to numpy - a little help is needed In-Reply-To: References: Message-ID: <4414A8CE.2040404@ieee.org> Mads Ipsen wrote: > Hey, > > I am trying to understand how to write numpy extensions, in order to be > able to do various things fast, that I need in my own apps. I have looked > at the examples from the Numeric package, which I managed to get working > with numpy, so I guess my questions holds for this forum. > > There's one detail, that I don't understand in the examples provided with > the Numeric documentation, namely the use of > > Py_DECREF(...) > > which is used in the first example but not in the last. Let me be more > specific: > There are several problems with the example in the documentation that lead to memory leaks. It needs to be fixed. You are correct that Py_DECREF(matrix); Py_DECREF(vector); is needed before returning control back to Python. In addition, the error-handling code (when return NULL is encountered) also needs to make sure that arrays that have been created get destroyed using Py_DECREF(...) before returning from the function. This is often done using an err: label and the goto command in C. Then, at the error label Py_XDECREF(...) is used which does nothing if the argument is NULL, but otherwise DECREF's the object (destroying it if it's the only reference). So, this example is flawed. I think that it's come up before but I don't recall the list discussion. But, obviously nothing was done to change the documentation. -Travis From cjw at sympatico.ca Sun Mar 12 15:18:00 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 12 15:18:00 2006 Subject: [Numpy-discussion] Attempt to subclass dtype Message-ID: <4414AC1B.8010107@sympatico.ca> I would like to subclass dtype. Is there a way for a user to make dtype acceptable? class d1type(_n.dtype): TypeError: Error when calling the metaclass bases type 'numpy.dtype' is not an acceptable base type >> Colin W. From Fernando.Perez at colorado.edu Sun Mar 12 16:34:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun Mar 12 16:34:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... Message-ID: <4414BDEF.70001@colorado.edu> Hi all, there are recurrent questions on the topic of low-level access to numpy, wrapping libraries, etc. Many people have home-brewed SWIG typemaps, the cookbook has some (very nice) pyrex examples, etc. I think that we could use a bit of unification work on this front, to make this particular issue easier to deal with for newcomers, and to reduce duplication of effort. 1. SWIG ======= At the last scipy conference, I collected typemaps from a number of people who had them around and all I could find on the net. I grabbed Scott Ransom's, Michel Sanner contributed his, Eric Jones gave us some as well, etc. John Hunter worked on updating Eric's (which had some nice features) to work with plain C (they were originally C++), and Bill Spotz from Sandia NL agreed to do the real work of writing some clean, documented examples of typemap use starting with this codebase. He gave me this code some time ago, and I shamefully just sat on it. I'm finally doing something about it, so today I updated this code to work with numpy instead of Numeric, and all the tests pass. You can grab it from here with an SVN client: http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpy_swig It is clearly documented, and while I'm sure it can use improvements and extensions, I think it would be a good starting point to have distributed /officially/ with numpy. That way others can improve /this/ code, rather than reimplementing numpy typemaps for the millionth time. D. Beazley (the SWIG author) has even indicated that he'd be happy to officially ship numpy typemaps with SWIG if something accepted by the community were sent to him. 2. Pyrex ======== After reading the excellent cookbook examples on pyrex usage, which I recently needed for a project, I decided to complete the code so it could be used 'out of the box'. This simply meant a bit of organization work, updating the code to use the 'modern' cimport statement and .pxd files, and writing a proper setup.py file. The result is here: http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpyx/ I also think that pyrex should be 'officially' supported. In particular, I think that the c_numpy.pxd file I put there, which is just a copy of that in mtrand, should be completed and shipped in the same headers directory as the rest of numpy's .h files. This would make it a LOT easier for newcomers to use pyrex for writing numpy extensions, since all headers would be complete and available with all numpy installations. Finally, if anyone is interested, that repository actually contains in the full checkout http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/ examples of how to wrap a more complex library which requires the creation of C structs from python objects. Getting this right took some time, as not all the details for this kind of construction are clearly documented anywhere. I hope it may be useful to others. 3. Wrapup ========= I think it would be a good idea to ship the numpy_swig example directory (and perhaps also numpyx, if somewhat expanded) listed above with the standard distribution, as well as completing the .pxd file to expose all the numpy C API for pyrex. I am not claiming to have done any of the hard work in numpyx or numpy_swig (though I did spend a lot of time on the wavelets stuff recently :), but I think we would all benefit from having this kind of 'infrastructure' code in the core. Once there, improvements can be made over time, with less effort wasted. If we agree on this, it's as simple as dropping those two directories somewhere on the numpy codebase (sandbox?, doc?). Just having them 'officially' available will be a benefit, I think. I'd like to thank Bill for the time he put into the swig module, as well as others who contributed code for this. Hopefully it will benefit everyone. Regards, f From strawman at astraw.com Sun Mar 12 16:51:06 2006 From: strawman at astraw.com (Andrew Straw) Date: Sun Mar 12 16:51:06 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414BDEF.70001@colorado.edu> References: <4414BDEF.70001@colorado.edu> Message-ID: <4414C1D5.5000108@astraw.com> Fernando Perez wrote: > At the last scipy conference, I collected typemaps from a number of > people who had them around and all I could find on the net. [snip] > I'm finally doing something about it, so today I updated this code to > work with numpy instead of Numeric, and all the tests pass. Thanks Fernando, I think this is great. > I think it would be a good starting point to have distributed > /officially/ with numpy. That way others can improve /this/ code, > rather than reimplementing numpy typemaps for the millionth time. +1 > 2. Pyrex > ======== > > After reading the excellent cookbook examples on pyrex usage, which I > recently needed for a project, I decided to complete the code so it > could be used 'out of the box'. This simply meant a bit of > organization work, updating the code to use the 'modern' cimport > statement and .pxd files, and writing a proper setup.py file. What do you mean 'modern' cimport and .pxd statement? I believe the example at http://scipy.org/Cookbook/Pyrex_and_NumPy is fundamentally no different than yours. One question, though: why have you reverted from using Python's Py_intptr_t and used intp instead? It seems to me much more likely that Py_intptr_t will be available and will do the Right Thing than relying on some libc to define it. We've seen broken implementations of C libraries before, and I say let's trust the Python devs to get this one right. > The result is here: > > http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpyx/ > > I also think that pyrex should be 'officially' supported. In > particular, I think that the c_numpy.pxd file I put there, which is > just a copy of that in mtrand, should be completed and shipped in the > same headers directory as the rest of numpy's .h files. This would > make it a LOT easier for newcomers to use pyrex for writing numpy > extensions, since all headers would be complete and available with all > numpy installations. +1 > Getting this right took some time, as not all the details for this > kind of construction are clearly documented anywhere. I hope it may > be useful to others. Note that setuptools 'automatically' deals with .pyx files: If you pass a source file that ends in .pyx, setuptools will run pyrexc on it if Pyrex is available, and otherwise use a '.c' file by the same name in the same directory. This is nice because it means you can ship .c files with your project and people can build them, even if they don't have Pyrex installed. Maybe we want to ship two setup.py files, or one that is "setuptools aware" (checks for setuptools with "if 'setuptools' in sys.modules"). Finally, a more general question: now that ctypes will become an official part of Python with 2.5, should we visit the idea of an 'official' numpy/ctypes binding? I used ctypes once about 3 years ago, so I'm not qualified to have an opinion on this, but while this stuff is in the wind... From zpincus at stanford.edu Sun Mar 12 17:04:19 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Mar 12 17:04:19 2006 Subject: [Numpy-discussion] How to tell if PyArray_FromAny has copied data? Message-ID: Hi folks, I need to figure out how to tell if PyArray_FromAny() has made a copy of the data that was provided to it. I assume that if the returned PyObject has a different address than the input object, then a copy was made somewhere along the line. Is this true? Is there a better way to tell? Before anyone asks, here's why I need to know if a copy was made. I'm writing a command to turn a numpy array into an image object suitable for sending to this 3rd party medical imaging library I've wrapped. In some cases, it is desirable to share the data buffer between an array object and an image object (and the user is responsible for ensuring that the lifetime of the image object doesn't exceed that of the array object). However, if PyArray_FromAny() has made a copy, then it is returning an entirely new object that the user will never see and so I'll need to handle the memory management differently (e.g. steal the data buffer from the new array and let the image object manage it). Alternately, I guess I could just inspect the input array object to see if it is exactly what I want, and branch on that. But running things through PyArray_FromAny seems easier, if it is possible. Zach From Fernando.Perez at colorado.edu Sun Mar 12 17:08:20 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun Mar 12 17:08:20 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C1D5.5000108@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> Message-ID: <4414C5CD.3040809@colorado.edu> Andrew Straw wrote: >> After reading the excellent cookbook examples on pyrex usage, which I >> recently needed for a project, I decided to complete the code so it could >> be used 'out of the box'. This simply meant a bit of organization work, >> updating the code to use the 'modern' cimport statement and .pxd files, >> and writing a proper setup.py file. > > > What do you mean 'modern' cimport and .pxd statement? I believe the > example at http://scipy.org/Cookbook/Pyrex_and_NumPy is fundamentally no > different than yours. Well, I'm pretty sure that when I first downloaded that code (about a month ago) it was not using cimport and pxd files, which is why I made those changes. I started using that code from a download at rev.6, which did use the 'old style', and didn't look at the wiki since: http://scipy.org/Cookbook/Pyrex_and_NumPy?action=diff&rev2=14&rev1=6 > One question, though: why have you reverted from using Python's Py_intptr_t > and used intp instead? It seems to me much more likely that Py_intptr_t > will be available and will do the Right Thing than relying on some libc to > define it. We've seen broken implementations of C libraries before, and I > say let's trust the Python devs to get this one right. Again, I was using the old code before your updates. I'm pretty sure that I did look at the wiki in February though, it's possible that my browser's cache played a trick on me, because your updates were done in February while the 'old' code seems to be from January. Anyway, it doesn't matter: what I did is very close to what you have now, I only have a more complete c_numpy file (stolen from mtrand) and a full pair of Makefile/setup.py, which can be handy for 'out of the box' usage. > Note that setuptools 'automatically' deals with .pyx files: If you pass a > source file that ends in .pyx, setuptools will run pyrexc on it if Pyrex is > available, and otherwise use a '.c' file by the same name in the same > directory. This is nice because it means you can ship .c files with your > project and people can build them, even if they don't have Pyrex installed. > Maybe we want to ship two setup.py files, or one that is "setuptools aware" > (checks for setuptools with "if 'setuptools' in sys.modules"). I certainly hope we are NOT going to make setuptools a dependency or a requirement for numpy/scipy. I've had no end of problems with it (which R. Kern has patiently dealt with in private) and I am allergic to the thing. I've seen it break plain 'setup.py install' installations with the most bizarre of errors, and I have a feeling that it's a tool whose marketing quality far outpaces its technical merits, at least in some contexts. Robert has calmly explained to me how it DOES have very useful features for certain problems, so I respect its use where needed (I've even made sure that ipython plays nicely with it, inasmuch as I can). But given how making a plain distutils-based setup.py file which knows about pyrex files (it won't call pyrexc if the .c file is present, I think) isn't that hard, I'd prefer NOT to have setuptools be a dependency. The less things numpy depends on, I think the better off we all are. Cheers, f From strawman at astraw.com Sun Mar 12 17:48:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Sun Mar 12 17:48:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C5CD.3040809@colorado.edu> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> <4414C5CD.3040809@colorado.edu> Message-ID: <4414CF35.7010200@astraw.com> Fernando Perez wrote: > Andrew Straw wrote: > >> One question, though: why have you reverted from using Python's >> Py_intptr_t >> and used intp instead? It seems to me much more likely that Py_intptr_t >> will be available and will do the Right Thing than relying on some >> libc to >> define it. We've seen broken implementations of C libraries before, >> and I >> say let's trust the Python devs to get this one right. > > Again, I was using the old code before your updates. > > Anyway, it doesn't matter: what I did is very close to what you have > now, I only have a more complete c_numpy file (stolen from mtrand) and > a full pair of Makefile/setup.py, which can be handy for 'out of the > box' usage. OK, I just hope you update your code to use Python's Py_intptr_t. I think it will be more robust. >> Note that setuptools 'automatically' deals with .pyx files: If you >> pass a >> source file that ends in .pyx, setuptools will run pyrexc on it if >> Pyrex is >> available, and otherwise use a '.c' file by the same name in the same >> directory. This is nice because it means you can ship .c files with your >> project and people can build them, even if they don't have Pyrex >> installed. >> Maybe we want to ship two setup.py files, or one that is "setuptools >> aware" >> (checks for setuptools with "if 'setuptools' in sys.modules"). > > > I certainly hope we are NOT going to make setuptools a dependency or a > requirement for numpy/scipy. I'm not proposing we do so. The 'setuptools aware' proposal above allows making use of a setuptools feature without requiring it. I agree this could be done in a few lines of code without setuptools, I was just making a suggestion for improvement to your setup.py file to make it more robust to the presence or absence of Pyrex. I like the present situation where numpy is very compatible with setuptools and I appreciate that ipython is also. I don't see what benefits numpy and scipy would get from _requiring_ setuptools, either, so we agree that there is no reason to do so. I'm sorry you've had such a bad time with setuptools but it solves many problems for me. The main developer of setuptools, Philip Eby, is very receptive to suggestions, comments, and criticisms, and given the significant chances of setuptools being incorporated into Python sooner rather than later, I'd suggest airing any remaining complaints soon or be forced to deal with them for a long time to come. From Fernando.Perez at colorado.edu Sun Mar 12 18:56:00 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sun Mar 12 18:56:00 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414CF35.7010200@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> <4414C5CD.3040809@colorado.edu> <4414CF35.7010200@astraw.com> Message-ID: <4414DF08.1070707@colorado.edu> Andrew Straw wrote: > OK, I just hope you update your code to use Python's Py_intptr_t. I > think it will be more robust. Done, both on the 'real' wavelets stuff and on the exposed numpyx example. Thanks for the pointer. > I'm not proposing we do so. The 'setuptools aware' proposal above allows > making use of a setuptools feature without requiring it. I agree this > could be done in a few lines of code without setuptools, I was just > making a suggestion for improvement to your setup.py file to make it > more robust to the presence or absence of Pyrex. Thanks, that's a good point. I just fixed the setup.py file so it will not require pyrex, and added the generated C to SVN as well. This way anyone can grab the code and run it without needing anything beyond python and numpy. Cheers, f From oliphant.travis at ieee.org Sun Mar 12 19:13:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 12 19:13:02 2006 Subject: [Numpy-discussion] How to tell if PyArray_FromAny has copied data? In-Reply-To: References: Message-ID: <4414E2F6.1090000@ieee.org> Zachary Pincus wrote: > Hi folks, > > I need to figure out how to tell if PyArray_FromAny() has made a copy > of the data that was provided to it. > > I assume that if the returned PyObject has a different address than > the input object, then a copy was made somewhere along the line. Is > this true? Is there a better way to tell? > If an array object was passed in, then this will work. If, however, another object was passed in that exposes the array interface which allows data sharing, then you will get a new ArrayObject but it will share the same array data so no copying will have been done. You will be able to tell, because the returned object won't own it's own data. So, it depends on what you are going to be inputing and what kind of data sharing you are going to be checking for. > Alternately, I guess I could just inspect the input array object to > see if it is exactly what I want, and branch on that. But running > things through PyArray_FromAny seems easier, if it is possible. This actually might be easier as PyArray_FromAny can take any Python object whereas you seem to already know you have an array input. -Travis From oliphant.travis at ieee.org Sun Mar 12 19:17:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 12 19:17:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C1D5.5000108@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> Message-ID: <4414E3F9.5080905@ieee.org> Andrew Straw wrote: > What do you mean 'modern' cimport and .pxd statement? I believe the > example at http://scipy.org/Cookbook/Pyrex_and_NumPy is fundamentally no > different than yours. One question, though: why have you reverted from > using Python's Py_intptr_t and used intp instead? It seems to me much > more likely that Py_intptr_t will be available and will do the Right > Thing than relying on some libc to define it. We've seen broken > implementations of C libraries before, and I say let's trust the Python > devs to get this one right. > If you are including arrayobject.h, then intp is exactly Py_intptr_t. Since intp is what is used internally in numpy, why not use it in a numpy wrapper. So, basically, I'm not sure what your point is. I can see the case for the __array_struct__ interface which does not have to include arrayobject.h, then you need to use Py_intptr_t. -Travis From robert.kern at gmail.com Sun Mar 12 20:57:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sun Mar 12 20:57:03 2006 Subject: [Numpy-discussion] Re: Access to C/C++, typemaps, pyrex... In-Reply-To: <4414C1D5.5000108@astraw.com> References: <4414BDEF.70001@colorado.edu> <4414C1D5.5000108@astraw.com> Message-ID: Andrew Straw wrote: > Note that setuptools 'automatically' deals with .pyx files: If you pass > a source file that ends in .pyx, setuptools will run pyrexc on it if > Pyrex is available, and otherwise use a '.c' file by the same name in > the same directory. I'm pretty sure that numpy.distutils does this (or something like it), too. C.f. numpy/distutils/tests/pyrex_ext/. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From prabhu_r at users.sf.net Sun Mar 12 21:58:01 2006 From: prabhu_r at users.sf.net (Prabhu Ramachandran) Date: Sun Mar 12 21:58:01 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414BDEF.70001@colorado.edu> References: <4414BDEF.70001@colorado.edu> Message-ID: <17429.2130.250319.896367@prpc.aero.iitb.ac.in> >>>>> "Fernando" == Fernando Perez writes: Fernando> Hi all, there are recurrent questions on the topic of Fernando> low-level access to numpy, wrapping libraries, etc. Fernando> Many people have home-brewed SWIG typemaps, the cookbook Fernando> has some (very nice) pyrex examples, etc. I think that [...] Fernando> ... I think we would all benefit from having this kind Fernando> of 'infrastructure' code in the core. Once there, Fernando> improvements can be made over time, with less effort Fernando> wasted. +3 (one for each). Fernando> If we agree on this, it's as simple as dropping those Fernando> two directories somewhere on the numpy codebase Fernando> (sandbox?, doc?). Just having them 'officially' Fernando> available will be a benefit, I think. Yes, this will certainly be very useful. regards, prabhu From NadavH at VisionSense.com Mon Mar 13 02:03:04 2006 From: NadavH at VisionSense.com (Nadav Horesh) Date: Mon Mar 13 02:03:04 2006 Subject: [Numpy-discussion] Is there an equivalkent to APL's drop function. Message-ID: <44155ED9.40407@VisionSense.com> APL's drop is a complementary to APL's take (= numpy take). I would expect it to behave something like: an_array.drop([0,5,7],axis=0) would return an_array with elements/rows 0,5,7 missing. Nadav. From schofield at ftw.at Mon Mar 13 09:19:01 2006 From: schofield at ftw.at (Ed Schofield) Date: Mon Mar 13 09:19:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4414BDEF.70001@colorado.edu> References: <4414BDEF.70001@colorado.edu> Message-ID: <4415A968.40807@ftw.at> Fernando Perez wrote: > I think it would be a good idea to ship the numpy_swig example > directory (and perhaps also numpyx, if somewhat expanded) listed above > with the standard distribution, as well as completing the .pxd file to > expose all the numpy C API for pyrex. > > I am not claiming to have done any of the hard work in numpyx or > numpy_swig (though I did spend a lot of time on the wavelets stuff > recently :), but I think we would all benefit from having this kind of > 'infrastructure' code in the core. Once there, improvements can be > made over time, with less effort wasted. Great! Thanks to you and everyone else who has worked on this. > If we agree on this, it's as simple as dropping those two directories > somewhere on the numpy codebase (sandbox?, doc?). Just having them > 'officially' available will be a benefit, I think. So, just to clarify, you think we should house the SWIG typemaps in the NumPy tree temporarily, until they're vetted, and then move them over to the SWIG tree? -- Ed From oliphant.travis at ieee.org Mon Mar 13 09:27:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 13 09:27:03 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415A968.40807@ftw.at> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> Message-ID: <4415AB45.3000702@ieee.org> Ed Schofield wrote: > Fernando Perez wrote: > > So, just to clarify, you think we should house the SWIG typemaps in the > NumPy tree temporarily, until they're vetted, and then move them over to > the SWIG tree? > I think that's what he was saying. I think it's a good idea. I've created both a pyrex and a swig directory under numpy/doc and placed the files there. More can be added, of course, like an example of how to use them. They will get "installed" to the doc directory as well: /numpy/doc At least they are in a place visible to all users. Regarding numpy 0.9.6: I made enough changes to the numpy tree that I wanted to wait a day or two to let obvious bugs I may have created rise to the surface before making another release. I'll test matplotlib today and if that goes well and I don't hear any other complaints, I'll make a release. -Travis From jdhunter at ace.bsd.uchicago.edu Mon Mar 13 09:53:04 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Mon Mar 13 09:53:04 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> (Travis Oliphant's message of "Mon, 13 Mar 2006 10:26:29 -0700") References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <87irqikzbr.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Travis" == Travis Oliphant writes: Travis> I made enough changes to the numpy tree that I wanted to Travis> wait a day or two to let obvious bugs I may have created Travis> rise to the surface before making another release. I'll Travis> test matplotlib today and if that goes well and I don't Travis> hear any other complaints, I'll make a release. Just a reminder to those who frequently update their numpy and matplotlib from src. I had a segfault today when importing matplotlib/pylab with numpy, and this was fixed simply by > cd matplotlib_src > sudo rm -rf build > sudo python setup.py install So if you experience problems, try a clean rebuild. JDH From Fernando.Perez at colorado.edu Mon Mar 13 10:20:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 13 10:20:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <4415B786.9010003@colorado.edu> Travis Oliphant wrote: > Ed Schofield wrote: > >>Fernando Perez wrote: >> >>So, just to clarify, you think we should house the SWIG typemaps in the >>NumPy tree temporarily, until they're vetted, and then move them over to >>the SWIG tree? >> > > > I think that's what he was saying. I think it's a good idea. Yes, that was my point. For example, the typemaps I sent were 1 and 2d only, and some people have posted 3-d and n-d ones as well, so this is not complete yet. But having it in the core will make sure that the contributions are actually recorded and improve our common usage tools. Once we feel they are solid enough, we can (if we think it's worthwhile) send them to the SWIG team. Though frankly, I think the most important part was having /something/ official in the numpy distribution itself: anyone needing this by definition has numpy, so we may actually be better off controlling this code rather than relying on SWIG versions and their release schedule. In any case, that can be discussed later, the point was to have this, and I'm very thankful to Travis for pulling it in. I'll delete it from the wavelets repo so we only have one copy of this code lying around. Cheers, f From strawman at astraw.com Mon Mar 13 11:21:07 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Mar 13 11:21:07 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <4415C609.6090500@astraw.com> Travis Oliphant wrote: > I made enough changes to the numpy tree that I wanted to wait a day or > two to let obvious bugs I may have created rise to the surface before > making another release. I'll test matplotlib today and if that goes > well and I don't hear any other complaints, I'll make a release. > Hi Travis, I'm getting one test failure with svn 2230 on my amd64 arch (below). Other than that, the various bits of code I've tested seem to be working OK. FAIL: check_both (numpy.core.tests.test_multiarray.test_pickling) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 235, in check_both assert_equal(self.carray, pickle.loads(self.carray.dumps())) File "/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/testing/utils.py", line 117, in assert_equal return assert_array_equal(actual, desired, err_msg) File "/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/testing/utils.py", line 204, in assert_array_equal assert cond,\ AssertionError: Arrays are not equal (mismatch 66.6666666667%): Array 1: [[2 9] [7 0] [3 8]] Array 2: [[2 0] [9 3] [7 8]] ---------------------------------------------------------------------- Ran 341 tests in 0.634s FAILED (failures=1) >>> numpy.__file__ '/home/astraw/py2.3-linux-x86_64/lib/python2.3/site-packages/numpy-0.9.6.2230-py2.3-linux-x86_64.egg/numpy/__init__.pyc' From Fernando.Perez at colorado.edu Mon Mar 13 11:23:03 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 13 11:23:03 2006 Subject: [Numpy-discussion] [Fwd: Re: [SAGEdev] faster random number generation] Message-ID: <4415C66C.2090405@colorado.edu> Hi all, I'm wondering if one of our resident experts who is more familiar than me with the details of the current build process could help us here a little. This is a message from the SAGE development list, where I am encouraging them to integrate more tightly with numpy. SAGE is a very interesting python-based system for mathematics: http://modular.ucsd.edu/sage/ After my comment below on numpy NOT needing a fortran compiler, William came back with that g77 error message. Now I'm a little confused, and before I confuse him even more, I'd prefere to clarify things with those who actually konw what's going on. Any help will be greatly appreciated. Cheers, f -------- Original Message -------- Subject: Re: [SAGEdev] faster random number generation Date: Mon, 13 Mar 2006 10:43:01 -0800 From: William Stein Reply-To: sage-devel at lists.sourceforge.net Organization: UC San Diego To: sage-devel at lists.sourceforge.net References: <4db014670603111659id0dccb9g5619a930c3ac9b27 at mail.gmail.com> <200603130111.00929.wstein at ucsd.edu> <4415B68E.4080207 at colorado.edu> On Monday 13 March 2006 10:14, Fernando Perez wrote: > No: numpy requires only standard python + a C compiler to build the > extensions. All the code in /numpy/ is pure C. Scipy, on the other hand, > /does/ need a Fortran compiler (not necessarily gcc, it's been compiled > with quite a few others, though it is not always a trivial matter to do > so). I'm confused then, because the output when building numpy (e.g., from David Joyner's build failure report) includes g77 calls. Are we building with the wrong options? building 'numpy.core._dotblas' extension compiling C sources gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/home/wdj/sagefiles/sage-1.0.6/local/include/python2.4 -fPIC' creating build/temp.linux-x86_64-2.4/numpy/core/blasdot compile options: '-DNO_ATLAS_INFO=1 -Inumpy/core/blasdot -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/home/wdj/sagefiles/sage-1.0.6/local/include/python2.4 -c' gcc: numpy/core/blasdot/_dotblas.c numpy/core/blasdot/_dotblas.c: In function 'dotblas_matrixproduct': numpy/core/blasdot/_dotblas.c:166: warning: 'ap1stride' may be used uninitialized in this function \/\/\/\/\/\/ /usr/bin/g77 build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.4/numpy/core/_dotblas.so build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_alterdot':numpy/core/blasdot/_dotblas.c:65: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:89: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_restoredot':numpy/core/blasdot/_dotblas.c:100: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:126: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_innerproduct':numpy/core/blasdot/_dotblas.c:656: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:687: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_vdot':numpy/core/blasdot/_dotblas.c:896: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:927: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:999: undefined reference to `Py_InitModule4' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `import_array':build/src/numpy/core/__multiarray_api.h:763: undefined reference to `PyImport_ImportModule' :build/src/numpy/core/__multiarray_api.h:766: undefined reference to `PyObject_GetAttrString' :build/src/numpy/core/__multiarray_api.h:768: undefined reference to `PyCObject_Type' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyExc_RuntimeError' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyErr_Format' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:1009: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':build/src/numpy/core/__multiarray_api.h:769: undefined reference to `PyCObject_AsVoidPtr' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_matrixproduct':numpy/core/blasdot/_dotblas.c:177: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:208: undefined reference to `PyTuple_New' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyErr_SetString' /usr/lib/libfrtbegin.a(frtbegin.o): In function `main': undefined reference to `MAIN__' collect2: ld returned 1 exit status build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_alterdot':numpy/core/blasdot/_dotblas.c:65: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:89: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_restoredot':numpy/core/blasdot/_dotblas.c:100: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:126: undefined reference to `_Py_NoneStruct' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_innerproduct':numpy/core/blasdot/_dotblas.c:656: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:717: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:687: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_vdot':numpy/core/blasdot/_dotblas.c:896: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:946: undefined reference to `PyErr_SetString' :numpy/core/blasdot/_dotblas.c:927: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:999: undefined reference to `Py_InitModule4' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `import_array':build/src/numpy/core/__multiarray_api.h:763: undefined reference to `PyImport_ImportModule' :build/src/numpy/core/__multiarray_api.h:766: undefined reference to `PyObject_GetAttrString' :build/src/numpy/core/__multiarray_api.h:768: undefined reference to `PyCObject_Type' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyExc_RuntimeError' :build/src/numpy/core/__multiarray_api.h:776: undefined reference to `PyErr_Format' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':numpy/core/blasdot/_dotblas.c:1009: undefined reference to `PyTuple_New' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `init_dotblas':build/src/numpy/core/__multiarray_api.h:769: undefined reference to `PyCObject_AsVoidPtr' build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o: In function `dotblas_matrixproduct':numpy/core/blasdot/_dotblas.c:177: undefined reference to `PyArg_ParseTuple' :numpy/core/blasdot/_dotblas.c:208: undefined reference to `PyTuple_New' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyExc_ValueError' :numpy/core/blasdot/_dotblas.c:263: undefined reference to `PyErr_SetString' /usr/lib/libfrtbegin.a(frtbegin.o): In function `main': undefined reference to `MAIN__' collect2: ld returned 1 exit status error: Command "/usr/bin/g77 build/temp.linux-x86_64-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib -lblas -lg2c -o build/lib.linux-x86_64-2.4/numpy/core/_dotblas.so" failed with exit status 1 ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd_______________________________________________ sage-devel mailing list sage-devel at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sage-devel From oliphant at ee.byu.edu Mon Mar 13 13:29:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 13 13:29:03 2006 Subject: [Numpy-discussion] [Fwd: Re: [SAGEdev] faster random number generation] In-Reply-To: <4415C66C.2090405@colorado.edu> References: <4415C66C.2090405@colorado.edu> Message-ID: <4415E3F9.6070409@ee.byu.edu> Fernando Perez wrote: > Hi all, > > I'm wondering if one of our resident experts who is more familiar than > me with the details of the current build process could help us here a > little. This is a message from the SAGE development list, where I am > encouraging them to integrate more tightly with numpy. > > SAGE is a very interesting python-based system for mathematics: > > http://modular.ucsd.edu/sage/ > > After my comment below on numpy NOT needing a fortran compiler, > William came back with that g77 error message. Now I'm a little > confused, and before I confuse him even more, I'd prefere to clarify > things with those who actually konw what's going on. > > Any help will be greatly appreciated. Pearu is the expert on this. The _dotblas.c function is optimized dot using ATLAS and/or BLAS for the system. I'm not sure why g77 is being used as the linker unless it is in their configuration somehow. For example, on my system, this is the command that links _dotblas.so cc -pthread -shared build/temp.linux-i686-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib/atlas -lptf77blas -lptcblas -latlas -o build/lib.linux-i686-2.4/numpy/core/_dotblas.so Please have them post to the list to figure out their issues. I want to squash any idea that numpy *requires* a Fortran compiler as this was a major design decision not to. It's always possible that configuration issues need to be improved for certain platforms. -Travis From Fernando.Perez at colorado.edu Mon Mar 13 13:33:03 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 13 13:33:03 2006 Subject: [Numpy-discussion] [Fwd: Re: [SAGEdev] faster random number generation] In-Reply-To: <4415E3F9.6070409@ee.byu.edu> References: <4415C66C.2090405@colorado.edu> <4415E3F9.6070409@ee.byu.edu> Message-ID: <4415E4F4.60704@colorado.edu> Travis Oliphant wrote: > Fernando Perez wrote: >>After my comment below on numpy NOT needing a fortran compiler, >>William came back with that g77 error message. Now I'm a little >>confused, and before I confuse him even more, I'd prefere to clarify >>things with those who actually konw what's going on. >> >>Any help will be greatly appreciated. > > > > Pearu is the expert on this. The _dotblas.c function is optimized dot > using ATLAS and/or BLAS for the system. I'm not sure why g77 is being > used as the linker unless it is in their configuration somehow. > > For example, on my system, this is the command that links _dotblas.so > > cc -pthread -shared > build/temp.linux-i686-2.4/numpy/core/blasdot/_dotblas.o -L/usr/lib/atlas > -lptf77blas -lptcblas -latlas -o > build/lib.linux-i686-2.4/numpy/core/_dotblas.so > > Please have them post to the list to figure out their issues. I want to > squash any idea that numpy *requires* a Fortran compiler as this was a > major design decision not to. It's always possible that configuration > issues need to be improved for certain platforms. Thanks, Travis. William, care to join in the numpy list with more details? As you see, it is true that numpy is NOT supposed to require a fortran compiler, this is something that was very specifically discussed and agreed upon. But we may need you to join in to get more information about how you are doing things, before more help can be provided. Here's the list link for you to join in: http://lists.sourceforge.net/lists/listinfo/numpy-discussion Cheers, f From oliphant at ee.byu.edu Mon Mar 13 13:42:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 13 13:42:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415C609.6090500@astraw.com> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <4415C609.6090500@astraw.com> Message-ID: <4415E6EB.9070800@ee.byu.edu> Andrew Straw wrote: >Hi Travis, > >I'm getting one test failure with svn 2230 on my amd64 arch (below). >Other than that, the various bits of code I've tested seem to be working OK. > > Can you try it again. I think I found the problem (a variable declared as long instead of int). -Travis From fonnesbeck at gmail.com Mon Mar 13 13:44:02 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Mon Mar 13 13:44:02 2006 Subject: [Numpy-discussion] inconsistent dot() behaviour Message-ID: <723eb6930603131343k6f0fdcb8g180a2086ce2dc914@mail.gmail.com> I notice that the dot() function does not behave consistently across all sizes of arrays: In [7]: dot(array([1.0]), array([-0.27163628])) Out[7]: array([-0.27163628]) In [8]: dot(array([1.0,1.0]), array([-0.27163628,1.0])) Out[8]: 0.72836371999999994 Why should a 1x1 dot product produce an array, while a 2x2 (or greater) product produces a scalar? Using recent svn build of numpy. Thanks, Chris -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us From oliphant at ee.byu.edu Mon Mar 13 14:10:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 13 14:10:09 2006 Subject: [Numpy-discussion] inconsistent dot() behaviour In-Reply-To: <723eb6930603131343k6f0fdcb8g180a2086ce2dc914@mail.gmail.com> References: <723eb6930603131343k6f0fdcb8g180a2086ce2dc914@mail.gmail.com> Message-ID: <4415EDAD.5080408@ee.byu.edu> Chris Fonnesbeck wrote: >I notice that the dot() function does not behave consistently across >all sizes of arrays: > >In [7]: dot(array([1.0]), array([-0.27163628])) >Out[7]: array([-0.27163628]) > >In [8]: dot(array([1.0,1.0]), array([-0.27163628,1.0])) >Out[8]: 0.72836371999999994 > >Why should a 1x1 dot product produce an array, while a 2x2 (or >greater) product produces a scalar? > > It's an error. Basically, you can check to make sure the optimized dot function is doing what it should by running from numpy.core.multiarray import dot as olddot and checking the output of olddot against the optimized dot output. They should be the same or it's a problem. We should write a test that loops through problems of dimensions 0, 1, 2 and computes the result both ways checking the output for equivalency. Some of this is done, but obviously another case was missed. -Travis From strawman at astraw.com Mon Mar 13 14:43:08 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Mar 13 14:43:08 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415E6EB.9070800@ee.byu.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <4415C609.6090500@astraw.com> <4415E6EB.9070800@ee.byu.edu> Message-ID: <4415F552.30103@astraw.com> Travis Oliphant wrote: > Andrew Straw wrote: > >> Hi Travis, >> >> I'm getting one test failure with svn 2230 on my amd64 arch (below). >> Other than that, the various bits of code I've tested seem to be >> working OK. >> >> > Can you try it again. I think I found the problem (a variable declared > as long instead of int). Yep, all tests pass now on that machine with svn 2232. From ofnap at nus.edu.sg Mon Mar 13 15:31:04 2006 From: ofnap at nus.edu.sg (Ajith Prasad (OFN)) Date: Mon Mar 13 15:31:04 2006 Subject: [Numpy-discussion] USING NUMPY WITH EXCEL Message-ID: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg> I am interested in using NumPy with Excel - eg, entering array data in Excel ranges, doing array computations using NumPy, and presenting the results in Excel. Are there any references or templates that I could use for this purpose? Thanks in advance for any help. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dd55 at cornell.edu Mon Mar 13 16:44:02 2006 From: dd55 at cornell.edu (Darren Dale) Date: Mon Mar 13 16:44:02 2006 Subject: [Numpy-discussion] gcc-4.1 Message-ID: <200603131943.53857.dd55@cornell.edu> Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 and gcc-4.1. When I try to build numpy I get warnings like "warning: build_ext: fcompiler=gnu is not available." I tried forcing numpy to use gfortran with "python setup.py config_fc --fcompiler=gnu95 build", and got the same warning for fcompiler=gnu95. Here is the output of "gfortran --version": GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) Copyright (C) 2006 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING And here is the output of "python setup.py config_fc --help-fcompiler": Running from numpy source directory. running config_fc customize CompaqFCompiler customize NoneFCompiler customize AbsoftFCompiler Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelFCompiler Could not locate executable g77 Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize SunFCompiler customize VastFCompiler customize GnuFCompiler customize IbmFCompiler customize Gnu95FCompiler customize IntelVisualFCompiler customize G95FCompiler customize IntelItaniumFCompiler customize PGroupFCompiler customize LaheyFCompiler customize CompaqVisualFCompiler customize MipsFCompiler customize HPUXFCompiler customize IntelItaniumVisualFCompiler customize NAGFCompiler List of available Fortran compilers: List of unavailable Fortran compilers: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaq Compaq Fortran Compiler --fcompiler=compaqv DIGITAL|Compaq Visual Fortran Compiler --fcompiler=g95 GNU Fortran 95 Compiler --fcompiler=gnu GNU Fortran Compiler --fcompiler=gnu95 GNU 95 Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun|Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler List of unimplemented Fortran compilers: --fcompiler=f Fortran Company/NAG F Compiler For compiler details, run 'config_fc --verbose' setup command. Thanks, Darren From jdhunter at ace.bsd.uchicago.edu Mon Mar 13 17:27:04 2006 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Mon Mar 13 17:27:04 2006 Subject: [Numpy-discussion] USING NUMPY WITH EXCEL In-Reply-To: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg> ("Ajith Prasad's message of "Tue, 14 Mar 2006 07:30:15 +0800") References: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg> Message-ID: <87k6axzulj.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Ajith" == Ajith Prasad (OFN) writes: Ajith> I am interested in using NumPy with Excel - eg, entering Ajith> array data in Excel ranges, doing array computations using Ajith> NumPy, and presenting the results in Excel. Are there any Ajith> references or templates that I could use for this purpose? Ajith> Thanks in advance for any help. Mark Hammond's "Python Programming On Win32 : Help for Windows Programmers " http://packages.debian.org/unstable/python/python2.3-matplotlib covers the API for interacting with Excel using his win32 python extensions http://starship.python.net/crew/mhammond/ Here's an online tutorial http://www.markcarter.me.uk/computing/python/excel.html There is nothing numpy specific to master. Once you figure out how to get sequences from excel to python and back, you can plug numpy into the pipeline. JDH From zpincus at stanford.edu Mon Mar 13 17:28:07 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 13 17:28:07 2006 Subject: [Numpy-discussion] dot and deallocation bug? Message-ID: Hi folks, I've run across an odd error in my numpy install related to numpy.dot with python lists of floats. Now, I haven't updated from svn in a while (I'm currently using numpy to do some data analysis, so I'm not changing things until that's done). Perhaps someone could confirm whether this is still a bug in the current version? Here's how I reproduce the error (OS X 10.4.5), python 2.4.2: In [1]: import numpy In [2]: numpy.version.version Out[2]: '0.9.6.2208' In [3]: numpy.dot([0.0, 1.0], [1, 2, 3]) ValueError: matrices are not aligned In [4]: numpy.dot([0.0, 1.0], [1, 2, 3]) ValueError: matrices are not aligned In [5]: numpy.dot([0.0, 1.0], [1, 2, 3]) Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug ValueError: matrices are not aligned In [6]: numpy.dot([0.0, 1.0], [1, 2, 3]) Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug Python(5957) malloc: *** Deallocation of a pointer not malloced: 0x1188bec; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug ValueError: matrices are not aligned Always the third time after I do this incorrect dot operation, the malloc errors start appearing. The more times I do the operation, the longer the list of malloc errors gets. If the input lists are not floating-point, this doesn't happen (also the ValueError states: 'objects are not aligned' instead of 'matrices are not aligned'). Any thoughts? Zach Pincus From robert.kern at gmail.com Mon Mar 13 18:08:04 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 13 18:08:04 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: <200603131943.53857.dd55@cornell.edu> References: <200603131943.53857.dd55@cornell.edu> Message-ID: Darren Dale wrote: > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my home > computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 and > gcc-4.1. When I try to build numpy I get warnings like "warning: build_ext: > fcompiler=gnu is not available." I tried forcing numpy to use gfortran with > "python setup.py config_fc --fcompiler=gnu95 build", and got the same warning > for fcompiler=gnu95. Building numpy shouldn't require any kind of Fortran compiler. However, the problem is that the version-recognizing regex doesn't like the version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in numpy/distutils/fcompiler/gnu.py and the class attribute Gnu95FCompiler.version_pattern . -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Fernando.Perez at colorado.edu Mon Mar 13 21:13:01 2006 From: Fernando.Perez at colorado.edu (Fernando.Perez at colorado.edu) Date: Mon Mar 13 21:13:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4415AB45.3000702@ieee.org> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> Message-ID: <1142313130.441650aad8111@webmail.colorado.edu> Quoting Travis Oliphant : > I think that's what he was saying. I think it's a good idea. I've > created both a pyrex and a swig directory under numpy/doc and placed the > files there. More can be added, of course, like an example of how to > use them. > > They will get "installed" to the doc directory as well: > /numpy/doc Just curious Travis: why did you only put in the .i/.pxd files and not the whole directories? I think that for this particular problem, there is a lot of value in the surrounding code. In both cases, the extra code provides examples of how to use each tool (swig/pyrex), there are tests, and a valid setup.py script. For a user who is not familiar with distutils, this can be quite useful, and I think in general it is a lot easier to get off the ground with a new system when you can grab an example directory which works 'out of the box' and start modifying it. If you have any concerns about the code, let me know and I'll be happy to address them, but I really think that this will be much more useful to everyone if each directory contains a complete, working example. Regards, f From oliphant.travis at ieee.org Mon Mar 13 23:18:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 13 23:18:02 2006 Subject: [Numpy-discussion] dot and deallocation bug? In-Reply-To: References: Message-ID: <44166DF9.20602@ieee.org> Zachary Pincus wrote: > Hi folks, > > I've run across an odd error in my numpy install related to numpy.dot > with python lists of floats. > > Now, I haven't updated from svn in a while (I'm currently using numpy > to do some data analysis, so I'm not changing things until that's > done). Perhaps someone could confirm whether this is still a bug in > the current version? > > Here's how I reproduce the error (OS X 10.4.5), python 2.4.2: This is a bug in the error-handling section of the optimized BLAS code. I see it too. The reason you don't see it for non floating point and/or higher than 2-d is that the there is an optimized dot function that is taking over for floating point and complex floating point code which uses the BLAS. So, you are exercising different sections of code. The optimized _dotblas.c file has been the source of quite a few bugs. The slower but less buggy version is always available as numpy.core.multiarray.dot Thanks for finding this one. -Travis From oliphant.travis at ieee.org Mon Mar 13 23:39:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 13 23:39:01 2006 Subject: [Numpy-discussion] dot and deallocation bug? In-Reply-To: <44166DF9.20602@ieee.org> References: <44166DF9.20602@ieee.org> Message-ID: <441672FF.3070904@ieee.org> Travis Oliphant wrote: >> Hi folks, >> >> I've run across an odd error in my numpy install related to numpy.dot >> with python lists of floats. >> >> Now, I haven't updated from svn in a while (I'm currently using numpy >> to do some data analysis, so I'm not changing things until that's >> done). Perhaps someone could confirm whether this is still a bug in >> the current version? >> >> Here's how I reproduce the error (OS X 10.4.5), python 2.4.2: > This is a bug in the error-handling section of the optimized BLAS > code. I see it too. > > The reason you don't see it for non floating point and/or higher than > 2-d is that the there is an optimized dot function that is taking over > for floating point and complex floating point code which uses the BLAS. > So, you are exercising different sections of code. The optimized > _dotblas.c file has been the source of quite a few bugs. The slower > but less buggy version is always available as numpy.core.multiarray.dot > > Thanks for finding this one. Update: This should be fixed in SVN now. At least the segfault I was seeing disappeared. There was a missing INCREF for a data-type descriptor. This would definitely cause those kinds of errors. -Travis From aisaac at american.edu Tue Mar 14 00:05:00 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue Mar 14 00:05:00 2006 Subject: [Numpy-discussion] USING NUMPY WITH EXCEL In-Reply-To: <87k6axzulj.fsf@peds-pc311.bsd.uchicago.edu> References: <502D9B13021C2D42B05F32C5C3F4C1DD0261C4@MBX01.stf.nus.edu.sg><87k6axzulj.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: On Mon, 13 Mar 2006, John Hunter apparently wrote: > Here's an online tutorial > http://www.markcarter.me.uk/computing/python/excel.html You may also find this useful: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbaxl10/html/xltocobjectmodelapplication.asp Please post a nice example when you produce one. Cheers, Alan Isaac From oliphant.travis at ieee.org Tue Mar 14 01:06:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 14 01:06:03 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released Message-ID: <44168767.80803@ieee.org> This post is to announce the release of NumPy 0.9.6 which fixes some important bugs and has several speed improvments. NumPy is a multi-dimensional array-package for Python that allows rapid high-level array computing with Python. It is successor to both Numeric and Numarray. More information at http://numeric.scipy.org The release notes are attached: Best regards, NumPy Developers -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: release-notes-0.9.6 URL: From wbaxter at gmail.com Tue Mar 14 03:53:00 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue Mar 14 03:53:00 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: Just wondering, does this one also require an update to scipy? And in general do numpy updates always require an update to scipy, too? Or is it only when the numpy C API interface changes? --bb On 3/14/06, Travis Oliphant wrote: > > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows rapid > high-level array computing with Python. It is successor to both Numeric > and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > > > > > > NumPy 0.9.6 is a bug-fix and optimization release with a > few new features: > > > New features (and changes): > > - bigndarray removed and support for Python2.5 ssize_t added giving > full support in Python2.5 to very-large arrays on 64-bit systems. > > - Strides can be set more arbitrarily from Python (and checking is done > to make sure memory won't be violated). > > - __array_finalize__ is now called for every array sub-class creation. > > - kron and repmat functions added > > - .round() method added for arrays > > - rint, square, reciprocal, and ones_like ufuncs added. > > - keyword arguments now possible for methods taking a single 'axis' > argument > > - Swig and Pyrex examples added in doc/swig and doc/pyrex > > - NumPy builds out of the box for cygwin > > - Different unit testing naming schemes are now supported. > > - memusage in numpy.distutils works for NT platforms > > - numpy.lib.math functions now take vectors > > - Most functions in oldnumeric now return intput class where possible > > > Speed ups: > > - x**n for integer n signficantly improved > > - array() much faster > > - .fill() method is much faster > > > Other fixes: > > - Output arrays to ufuncs works better. > > - Several ma (Masked Array) fixes. > > - umath code generation improved > > - many fixes to optimized dot function (fixes bugs in > matrix-sub-class multiply) > > - scalartype fixes > > - improvements to poly1d > > - f2py fixed to handle character arrays in common blocks > > - Scalar arithmetic improved to handle mixed-mode operation. > > - Make sure Python intYY types correspond exactly with C PyArray_INTYY > > > > > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nadavh at visionsense.com Tue Mar 14 06:08:11 2006 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Mar 14 06:08:11 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8EF52@exchange2k.envision.co.il> There is a compatibility problem, at least with the last formal release of scipy. Should we checkout and compile scipy from svn? Nadav. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net on behalf of Bill Baxter Sent: Tue 14-Mar-06 13:52 To: numpy-discussion; SciPy Users List Cc: Subject: Re: [Numpy-discussion] [ANN] NumPy 0.9.6 released Just wondering, does this one also require an update to scipy? And in general do numpy updates always require an update to scipy, too? Or is it only when the numpy C API interface changes? --bb On 3/14/06, Travis Oliphant wrote: > > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows rapid > high-level array computing with Python. It is successor to both Numeric > and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > > > > > > NumPy 0.9.6 is a bug-fix and optimization release with a > few new features: > > > New features (and changes): > > - bigndarray removed and support for Python2.5 ssize_t added giving > full support in Python2.5 to very-large arrays on 64-bit systems. > > - Strides can be set more arbitrarily from Python (and checking is done > to make sure memory won't be violated). > > - __array_finalize__ is now called for every array sub-class creation. > > - kron and repmat functions added > > - .round() method added for arrays > > - rint, square, reciprocal, and ones_like ufuncs added. > > - keyword arguments now possible for methods taking a single 'axis' > argument > > - Swig and Pyrex examples added in doc/swig and doc/pyrex > > - NumPy builds out of the box for cygwin > > - Different unit testing naming schemes are now supported. > > - memusage in numpy.distutils works for NT platforms > > - numpy.lib.math functions now take vectors > > - Most functions in oldnumeric now return intput class where possible > > > Speed ups: > > - x**n for integer n signficantly improved > > - array() much faster > > - .fill() method is much faster > > > Other fixes: > > - Output arrays to ufuncs works better. > > - Several ma (Masked Array) fixes. > > - umath code generation improved > > - many fixes to optimized dot function (fixes bugs in > matrix-sub-class multiply) > > - scalartype fixes > > - improvements to poly1d > > - f2py fixed to handle character arrays in common blocks > > - Scalar arithmetic improved to handle mixed-mode operation. > > - Make sure Python intYY types correspond exactly with C PyArray_INTYY > > > > > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 From cjw at sympatico.ca Tue Mar 14 07:56:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 14 07:56:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: <4416E747.306@sympatico.ca> Travis Oliphant wrote: > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows > rapid high-level array computing with Python. It is successor to both > Numeric and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > - __array_finalize__ is now called for every array sub-class creation. > > Thanks. What are the parameters for this method and what does it do? How does it differ from the common Python usage of __new__ followed by __init__? [Dbg]>>> Help on built-in function __array_finalize__: __array_finalize__(...) Colin W. From oliphant at ee.byu.edu Tue Mar 14 12:26:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 14 12:26:05 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <51CE8530-E9F6-4AE1-A1CF-9FF6C6FEB25F@nrl.navy.mil> References: <44168767.80803@ieee.org> <44170C7E.6070208@ieee.org> <51CE8530-E9F6-4AE1-A1CF-9FF6C6FEB25F@nrl.navy.mil> Message-ID: <441726C7.3020207@ee.byu.edu> Paul Ray wrote: >On Mar 14, 2006, at 1:33 PM, Travis Oliphant wrote: > > > >>>Just wondering, does this one also require an update to scipy? >>>And in general do numpy updates always require an update to scipy, >>>too? >>>Or is it only when the numpy C API interface changes? >>> >>> >>It's only when the C-API interface changes that this is necessary. >>Pre-1.0 it is likely to happen at each release. >> >>After that it is less likely to happen. >> >> > >Pre-1.0, I think it is fine for the version numbering to be a free- >for-all, as it has been. There have been many times, it seems, where >the latest numpy didn't work with the latest scipy and stuff like that. > > The current SVN of numpy and the current SVN of scipy usually work with each other (perhaps with only a day lag). >After 1.0, I would like to encourage some discipline with regards to >numpy version numbering. From 1.0, the API should NOT change at all >until 1.1. > This is the plan and it is why we are being slow with getting 1.0 out. But, we *need* people to try NumPy on their codes so that we can determine whether or not API additions are needed. Without this testing we are just guessing based on the developer's own codes which do not cover the breadth. >I think that many people with codes that use Numeric or numarray are >awaiting the numpy 1.0 release as a sign that it is stable and ready >for prime time, before converting their codes. > > We need some of these people to convert earlier than that, though, so we can make sure that 1.0 really is ready for prime time. I think it's very close already or the version number wouldn't be so high. We are just waiting for more people to start using it and report any issues that they have before going to 1.0 (I'd also like scalar math to be implemented pre 1.0 as well in case that requires any C-API additions). Thanks for the feedback. What you described will be the basic behavior once 1.0 is released. Note that right now the only changes that usually need to be made are a re-compile. I know this can still be annoying when you have a deep code stack. -Travis From Paul.Ray at nrl.navy.mil Tue Mar 14 12:39:02 2006 From: Paul.Ray at nrl.navy.mil (Paul Ray) Date: Tue Mar 14 12:39:02 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <441726C7.3020207@ee.byu.edu> References: <44168767.80803@ieee.org> <44170C7E.6070208@ieee.org> <51CE8530-E9F6-4AE1-A1CF-9FF6C6FEB25F@nrl.navy.mil> <441726C7.3020207@ee.byu.edu> Message-ID: <64DF7D92-839F-4B49-AC08-D757E4753580@nrl.navy.mil> On Mar 14, 2006, at 3:25 PM, Travis Oliphant wrote: >> I think that many people with codes that use Numeric or numarray are >> awaiting the numpy 1.0 release as a sign that it is stable and ready >> for prime time, before converting their codes. >> > We need some of these people to convert earlier than that, though, > so we > can make sure that 1.0 really is ready for prime time. I think it's > very close already or the version number wouldn't be so high. We are > just waiting for more people to start using it and report any issues > that they have before going to 1.0 (I'd also like scalar math to be > implemented pre 1.0 as well in case that requires any C-API > additions). I agree. Lots of people do seem to be trying out numpy. I use it for all new code I develop, and I have converted over a bunch of older SciPy stuff that uses both the python and C APIs, including ppgplot for plotting. I also heard from the PyFITS developers that an alpha release based on numpy is coming very soon. Cheers, -- Paul From svetosch at gmx.net Tue Mar 14 13:20:03 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Tue Mar 14 13:20:03 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: <44173352.4020403@gmx.net> Hi, are the following issues of the new release known? (This is on winxp.) >>> import numpy as n >>> n.__version__ '0.9.6' >>> a = n.asmatrix(n.eye(3)) >>> a matrix([[1, 0, 0], [0, 1, 0], [0, 0, 1]]) >>> n.linalg.inverse(a) array([[ 1., 0., 0.], [ 0., 1., 0.], [ 0., 0., 1.]]) I was asked to report if functions in linalg still don't honor the matrix input type; voila. >>> n.linalg.cholesky(a) Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\linalg\linalg.py", line 135, in cholesky_decomposition return transpose(triu(a,k=0)).copy() NameError: global name 'triu' is not defined I think I've seen this error before, was it in 0.9.4? This is a real show-stopper. Happy bug-hunting, sven Travis Oliphant schrieb: > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. > > NumPy is a multi-dimensional array-package for Python that allows rapid > high-level array computing with Python. It is successor to both Numeric > and Numarray. More information at http://numeric.scipy.org > > The release notes are attached: > > Best regards, > > NumPy Developers > > > > > ------------------------------------------------------------------------ > > > NumPy 0.9.6 is a bug-fix and optimization release with a > few new features: > > > New features (and changes): > > - bigndarray removed and support for Python2.5 ssize_t added giving > full support in Python2.5 to very-large arrays on 64-bit systems. > > - Strides can be set more arbitrarily from Python (and checking is done > to make sure memory won't be violated). > > - __array_finalize__ is now called for every array sub-class creation. > > - kron and repmat functions added > > - .round() method added for arrays > > - rint, square, reciprocal, and ones_like ufuncs added. > > - keyword arguments now possible for methods taking a single 'axis' > argument > > - Swig and Pyrex examples added in doc/swig and doc/pyrex > > - NumPy builds out of the box for cygwin > > - Different unit testing naming schemes are now supported. > > - memusage in numpy.distutils works for NT platforms > > - numpy.lib.math functions now take vectors > > - Most functions in oldnumeric now return intput class where possible > > > Speed ups: > > - x**n for integer n signficantly improved > > - array() much faster > > - .fill() method is much faster > > > Other fixes: > > - Output arrays to ufuncs works better. > > - Several ma (Masked Array) fixes. > > - umath code generation improved > > - many fixes to optimized dot function (fixes bugs in > matrix-sub-class multiply) > > - scalartype fixes > > - improvements to poly1d > > - f2py fixed to handle character arrays in common blocks > > - Scalar arithmetic improved to handle mixed-mode operation. > > - Make sure Python intYY types correspond exactly with C PyArray_INTYY > > From mpi at osc.kiku.dk Tue Mar 14 13:24:11 2006 From: mpi at osc.kiku.dk (Mads Ipsen) Date: Tue Mar 14 13:24:11 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays Message-ID: Hey, First of all, thanks for the new release. Here's another question regarding something I cannot quite understand: Suppose you want to update bins for a histogram, you might think you could do something like: g = zeros(4,Int) x = array([0.2, 0.2]) idx = floor(x/0.1).astype(int) g[idx] += 1 Here idx becomes array([2, 2]) In this case, I would naively expect g to end up like array([0, 0, 2, 0]) (1) but instead you get array([0, 0, 1, 0]) (2) Is this intended? Just being plain novice-like naive, I would expect the slice operation g[idx] += 1 to do something like for i in range(len(I)): g[ idx[i] ] += 1 resulting in (1) and not (2). // Mads From Fernando.Perez at colorado.edu Tue Mar 14 13:41:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Mar 14 13:41:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44168767.80803@ieee.org> References: <44168767.80803@ieee.org> Message-ID: <4417383B.4050000@colorado.edu> Travis Oliphant wrote: > This post is to announce the release of NumPy 0.9.6 which fixes some > important bugs and has several speed improvments. Just curious about the svd/singular... pair: In [2]: numpy.linalg.singular_value_decomposition?? Definition: numpy.linalg.singular_value_decomposition(A, full_matrices=0) Source: def singular_value_decomposition(A, full_matrices=0): return svd(A, full_matrices) In [3]: numpy.linalg.svd? Definition: numpy.linalg.svd(a, full_matrices=1) Do we really need both spellings to be different just on the default of the full_matrices flag? It seems to me like the kind of unnecessary special-casing that just forces people to remember little details without a clear benefit. If we want a long-spelling form, it would be cheaper to simply alias the name with a module-level singular_value_decomposition = svd so the same /object/ is referred to via both names. Perhaps I'm missing the blindingly obvious solid reason, but if this is just historical accident, I'd vote for unifying the two simply as spelling variants. I think the flipped default is more confusion-prone than beneficial. Cheers, f From robert.kern at gmail.com Tue Mar 14 14:42:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 14 14:42:07 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: <4417383B.4050000@colorado.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> Message-ID: Fernando Perez wrote: > Travis Oliphant wrote: > >> This post is to announce the release of NumPy 0.9.6 which fixes some >> important bugs and has several speed improvments. > > Just curious about the svd/singular... pair: > > In [2]: numpy.linalg.singular_value_decomposition?? > Definition: numpy.linalg.singular_value_decomposition(A, > full_matrices=0) > Source: > def singular_value_decomposition(A, full_matrices=0): > return svd(A, full_matrices) > > > In [3]: numpy.linalg.svd? > Definition: numpy.linalg.svd(a, full_matrices=1) > > > Do we really need both spellings to be different just on the default of > the full_matrices flag? It seems to me like the kind of unnecessary > special-casing that just forces people to remember little details > without a clear benefit. If we want a long-spelling form, it would be > cheaper to simply alias the name with a module-level > > singular_value_decomposition = svd > > so the same /object/ is referred to via both names. > > Perhaps I'm missing the blindingly obvious solid reason, but if this is > just historical accident, I'd vote for unifying the two simply as > spelling variants. I think the flipped default is more confusion-prone > than beneficial. I'm pretty sure that this is historical from when we wanted scipy_core's scipy.linalg to be compatible with both real scipy's scipy.linalg and backwards-compatible with Numeric's LinearAlgebra. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Tue Mar 14 14:50:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 14 14:50:06 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <4417383B.4050000@colorado.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> Message-ID: <4417487C.2080003@ee.byu.edu> Fernando Perez wrote: > Travis Oliphant wrote: > >> This post is to announce the release of NumPy 0.9.6 which fixes some >> important bugs and has several speed improvments. > > > Just curious about the svd/singular... pair: > > In [2]: numpy.linalg.singular_value_decomposition?? > Definition: numpy.linalg.singular_value_decomposition(A, > full_matrices=0) > Source: > def singular_value_decomposition(A, full_matrices=0): > return svd(A, full_matrices) > > > In [3]: numpy.linalg.svd? > Definition: numpy.linalg.svd(a, full_matrices=1) > > > Do we really need both spellings to be different just on the default > of the full_matrices flag? It seems to me like the kind of > unnecessary special-casing that just forces people to remember little > details without a clear benefit. If we want a long-spelling form, it > would be cheaper to simply alias the name with a module-level > > singular_value_decomposition = svd > > so the same /object/ is referred to via both names. > > Perhaps I'm missing the blindingly obvious solid reason, but if this > is just historical accident, I'd vote for unifying the two simply as > spelling variants. I think the flipped default is more > confusion-prone than beneficial. Backward compatibility. I don't like the singular_value_decomposition spelling but that was in Numeric (with the default given). So, it is provided to aid in code-porting. But, I think most people who compute svd expect it to produce full_matrices. Thus, the svd spelling. We make somebody unhappy either way I think. If we change anything, I would vote for changing singular_value_decomposition and adding something to convert_code to write any old code using full arguments. -Travis From robert.kern at gmail.com Tue Mar 14 15:01:00 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 14 15:01:00 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: <4417487C.2080003@ee.byu.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Backward compatibility. > I don't like the singular_value_decomposition spelling but that was in > Numeric (with the default given). So, it is provided to aid in > code-porting. > > But, I think most people who compute svd expect it to produce > full_matrices. Thus, the svd spelling. > > We make somebody unhappy either way I think. If we change anything, I > would vote for changing singular_value_decomposition and adding > something to convert_code to write any old code using full arguments. Alternatively, we could drop all of the Numeric compatibility forms into a separate module and have convert_code use that instead of numpy.linalg directly. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Tue Mar 14 15:04:16 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Mar 14 15:04:16 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44173352.4020403@gmx.net> References: <44168767.80803@ieee.org> <44173352.4020403@gmx.net> Message-ID: <44174BCE.5020104@ee.byu.edu> Sven Schreiber wrote: >>>>import numpy as n >>>>n.__version__ >>>> >>>> >'0.9.6' > > >>>>a = n.asmatrix(n.eye(3)) >>>>a >>>> >>>> >matrix([[1, 0, 0], > [0, 1, 0], > [0, 0, 1]]) > > >>>>n.linalg.inverse(a) >>>> >>>> >array([[ 1., 0., 0.], > [ 0., 1., 0.], > [ 0., 0., 1.]]) > >I was asked to report if functions in linalg still don't honor the >matrix input type; voila. > > > Thanks, but of course a.I should work, right? >>>>n.linalg.cholesky(a) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python24\Lib\site-packages\numpy\linalg\linalg.py", line 135, >in cholesky_decomposition > return transpose(triu(a,k=0)).copy() >NameError: global name 'triu' is not defined > >I think I've seen this error before, was it in 0.9.4? This is a real >show-stopper. > > Thanks again, it would be nice if you could help us by checking out an SVN version before a release ;-) But, better late than never. Also, show-stopper might be a bit of an over-statement. A dumb error, yes. But, if you really depend on this, then it's a simple fix to place a from numpy.lib import * at the top of the linalg.py file. Thanks again for your help. -Travis From Fernando.Perez at colorado.edu Tue Mar 14 15:07:25 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Mar 14 15:07:25 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44174C8D.4070100@colorado.edu> Robert Kern wrote: > Travis Oliphant wrote: > > >>Backward compatibility. >>I don't like the singular_value_decomposition spelling but that was in >>Numeric (with the default given). So, it is provided to aid in >>code-porting. >> >>But, I think most people who compute svd expect it to produce >>full_matrices. Thus, the svd spelling. >> >>We make somebody unhappy either way I think. If we change anything, I >>would vote for changing singular_value_decomposition and adding >>something to convert_code to write any old code using full arguments. > > > Alternatively, we could drop all of the Numeric compatibility forms into a > separate module and have convert_code use that instead of numpy.linalg directly. +1 Let's take advantage of this opportunity to clean up as much of the cruft as we can. Porting will inevitably require some manual labor. I'd say that if backwards compatibility can be isolated to a module, the long-term win is a net beneift. Cheers, f From ndarray at mac.com Tue Mar 14 17:40:01 2006 From: ndarray at mac.com (Sasha) Date: Tue Mar 14 17:40:01 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] unique hanging? In-Reply-To: <441720A8.2040004@stanford.edu> References: <441720A8.2040004@stanford.edu> Message-ID: This really belongs to numpy-discussion. What you observe is probably the consequence of nan not being equal to itself: >>> N.equal(N.nan,N.nan) False Many algorithms would fail to terminate or produce wrong results when they encounter such an anomaly. For example: >>> N.sort([N.nan,5, N.nan, 3]) array([ nan, 5. , nan, 3. ]) This is unfortunate, but not really a bug. I would suggest that we document which numpy functions are nan safe and which are not. On 3/14/06, Jonathan Taylor wrote: > hi all, > > apologize if this is not the right list -- in the latest svn of numpy > unique seems to be taking a long time with "nan" > > reproducible error: > > ------------------------------------------------------------------------------------------- > import numpy as N > import numpy.random as R > > x = R.standard_normal((100000,)) > x = N.around(x * 100.) / 100. > > print len(N.unique(x)) > > x[0] = N.nan > print len(N.unique(x)) > > > x[0:50000] = N.nan > print 'OK' > > print len(N.unique(x)) > > > -- > ------------------------------------------------------------------------ > I'm part of the Team in Training: please support our efforts for the > Leukemia and Lymphoma Society! > > http://www.active.com/donate/tntsvmb/tntsvmbJTaylor > > GO TEAM !!! > > ------------------------------------------------------------------------ > Jonathan Taylor Tel: 650.723.9230 > Dept. of Statistics Fax: 650.725.8977 > Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo > 390 Serra Mall > Stanford, CA 94305 > > > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > > > > From oliphant.travis at ieee.org Tue Mar 14 18:56:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 14 18:56:00 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44178222.5010004@ieee.org> Robert Kern wrote: > Alternatively, we could drop all of the Numeric compatibility forms into a > separate module and have convert_code use that instead of numpy.linalg directly. > Good idea, and done now in SVN. -Travis From ryanlists at gmail.com Tue Mar 14 18:57:02 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Tue Mar 14 18:57:02 2006 Subject: [Numpy-discussion] matrix vs. array Message-ID: Is there a difference between matrix and array objects other than how they handle multiplication? Is there documentation somewhere about matrix objects? Are they new? Thanks, Ryan From wbaxter at gmail.com Tue Mar 14 19:09:02 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue Mar 14 19:09:02 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: Message-ID: Inspect is a really handy python module: import inspect import numpy print inspect.getsource(numpy.matrix) >From that it looks like exponentiation (__pow__) is also overloaded, and there are the additional properties: .A - get the array .T - get the transpose .H - get the conjugate transpose .I - get the inverse The other difference which isn't really obvious from looking at the source code is that matrices always have at least rank 2. I.e. .shape[0] and .shape[1] are always present for a matrix. --bb On 3/15/06, Ryan Krauss wrote: > > Is there a difference between matrix and array objects other than how > they handle multiplication? Is there documentation somewhere about > matrix objects? Are they new? > > Thanks, > > Ryan > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan.taylor at stanford.edu Tue Mar 14 20:32:01 2006 From: jonathan.taylor at stanford.edu (Jonathan Taylor) Date: Tue Mar 14 20:32:01 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] unique hanging? In-Reply-To: References: <441720A8.2040004@stanford.edu> Message-ID: <441798AE.7010503@stanford.edu> thanks for the reply. i could be wrong, but this problem seemed to appear only when i updated svn -- i had a very similar example before with N.nan that seemed fine.... -- jonathan Sasha wrote: >This really belongs to numpy-discussion. > >What you observe is probably the consequence of nan not being equal to itself: > > > >>>>N.equal(N.nan,N.nan) >>>> >>>> >False > >Many algorithms would fail to terminate or produce wrong results when >they encounter such an anomaly. > >For example: > > > >>>>N.sort([N.nan,5, N.nan, 3]) >>>> >>>> >array([ nan, 5. , nan, 3. ]) > >This is unfortunate, but not really a bug. I would suggest that we >document which numpy functions are nan safe and which are not. > > > >On 3/14/06, Jonathan Taylor wrote: > > >>hi all, >> >>apologize if this is not the right list -- in the latest svn of numpy >>unique seems to be taking a long time with "nan" >> >>reproducible error: >> >>------------------------------------------------------------------------------------------- >>import numpy as N >>import numpy.random as R >> >>x = R.standard_normal((100000,)) >>x = N.around(x * 100.) / 100. >> >>print len(N.unique(x)) >> >>x[0] = N.nan >>print len(N.unique(x)) >> >> >>x[0:50000] = N.nan >>print 'OK' >> >>print len(N.unique(x)) >> >> >>-- >>------------------------------------------------------------------------ >>I'm part of the Team in Training: please support our efforts for the >>Leukemia and Lymphoma Society! >> >>http://www.active.com/donate/tntsvmb/tntsvmbJTaylor >> >>GO TEAM !!! >> >>------------------------------------------------------------------------ >>Jonathan Taylor Tel: 650.723.9230 >>Dept. of Statistics Fax: 650.725.8977 >>Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo >>390 Serra Mall >>Stanford, CA 94305 >> >> >> >>_______________________________________________ >>Scipy-dev mailing list >>Scipy-dev at scipy.net >>http://www.scipy.net/mailman/listinfo/scipy-dev >> >> >> >> >> >> -- ------------------------------------------------------------------------ I'm part of the Team in Training: please support our efforts for the Leukemia and Lymphoma Society! http://www.active.com/donate/tntsvmb/tntsvmbJTaylor GO TEAM !!! ------------------------------------------------------------------------ Jonathan Taylor Tel: 650.723.9230 Dept. of Statistics Fax: 650.725.8977 Sequoia Hall, 137 www-stat.stanford.edu/~jtaylo 390 Serra Mall Stanford, CA 94305 From arnd.baecker at web.de Wed Mar 15 01:52:01 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Wed Mar 15 01:52:01 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: Message-ID: On Wed, 15 Mar 2006, Bill Baxter wrote: > Inspect is a really handy python module: > import inspect > import numpy > print inspect.getsource(numpy.matrix) Another very nice option is to use ipython: ipython In [1]: import numpy In [2]: numpy.matrix? In [2]: numpy.matrix?? One ? gives some basic information: Type: type Base Class: String Form: Namespace: Interactive File: /home/abaecker/BUILDS3/BuildDir/inst_numpy/lib/python2.4/site-packages/numpy/core/defmatrix.py (and normally a doc-string, if there is one) and ?? shows you the code. Best, Arnd From pjssilva at ime.usp.br Wed Mar 15 02:36:03 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Mar 15 02:36:03 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: Message-ID: <1142418918.25469.0.camel@localhost.localdomain> > The other difference which isn't really obvious from looking at the > source code is that matrices always have at least rank Just to make clear. They *always* have rank two. Paulo From wbaxter at gmail.com Wed Mar 15 03:40:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 15 03:40:01 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: <1142418918.25469.0.camel@localhost.localdomain> References: <1142418918.25469.0.camel@localhost.localdomain> Message-ID: On 3/15/06, Paulo J. S. Silva wrote: > > > > The other difference which isn't really obvious from looking at the > > source code is that matrices always have at least rank > > Just to make clear. They *always* have rank two. Not necessarily. >>> m = numpy.matrix('[1 2 3; 4 5 6]') >>> mm = m[numpy.matrix('[0 1]')] >>> m matrix([[1, 2, 3], [4, 5, 6]]) >>> mm matrix([[[[1, 2, 3]], [[4, 5, 6]]]]) >>> mm.shape (1, 2, 1, 3) Maybe that's not supposed to be possible, but with current numpy (well, 0.9.5 at least) you see it is possible to get something with type matrix that is not rank 2. --bb -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ferenc.Pintye at eu.decoma.com Wed Mar 15 04:10:05 2006 From: Ferenc.Pintye at eu.decoma.com (Ferenc.Pintye at eu.decoma.com) Date: Wed Mar 15 04:10:05 2006 Subject: [Numpy-discussion] help speed up Message-ID: Hi , why is this little algorithm so too slow (10x) in Numpy than in numarray ? (i think, the if-elif-else procedure runs with numpy array indicies so slow...) # import time import os # # #####from numarray.random_array import * from numpy import * # #####vector = random(50000) vector = rand(50000) # length = len(vector) out_1 = [0,0,0] # # start = time.time() # # i = 0 while i < length-3: # # x1 = vector[i] x2 = vector[i+1] x3 = vector[i+2] x4 = vector[i+3] # if x2>x3>=x1 and x4>=x2: out_1[0] = x3 out_1[1] = x2 out_1[2] = 1 # elif x1>=x3>x2 and x2>=x4: out_1[0] = x2 out_1[1] = x3 out_1[2] = 1 else: out_1[0] = 0 out_1[1] = 0 out_1[2] = 0 # # if out_1[2] == 1: i += 1 i += 1 # # end = time.time() elapsed = end-start print "Running time : %.3f seconds" % (elapsed) # # From cjw at sympatico.ca Wed Mar 15 04:43:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 15 04:43:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <4417487C.2080003@ee.byu.edu> References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44180BC5.90604@sympatico.ca> I suggest putting the compatibility stuff in a separate module so that the user can import what he needs, instead of having the whole thing in one bundle. Over time, the separate module will get less use. For some, it will never be used. Colin W. Travis Oliphant wrote: > Fernando Perez wrote: > >> Travis Oliphant wrote: >> >>> This post is to announce the release of NumPy 0.9.6 which fixes some >>> important bugs and has several speed improvments. >> >> >> >> Just curious about the svd/singular... pair: >> >> In [2]: numpy.linalg.singular_value_decomposition?? >> Definition: numpy.linalg.singular_value_decomposition(A, >> full_matrices=0) >> Source: >> def singular_value_decomposition(A, full_matrices=0): >> return svd(A, full_matrices) >> >> >> In [3]: numpy.linalg.svd? >> Definition: numpy.linalg.svd(a, full_matrices=1) >> >> >> Do we really need both spellings to be different just on the default >> of the full_matrices flag? It seems to me like the kind of >> unnecessary special-casing that just forces people to remember little >> details without a clear benefit. If we want a long-spelling form, it >> would be cheaper to simply alias the name with a module-level >> >> singular_value_decomposition = svd >> >> so the same /object/ is referred to via both names. >> >> Perhaps I'm missing the blindingly obvious solid reason, but if this >> is just historical accident, I'd vote for unifying the two simply as >> spelling variants. I think the flipped default is more >> confusion-prone than beneficial. > > > Backward compatibility. > I don't like the singular_value_decomposition spelling but that was in > Numeric (with the default given). So, it is provided to aid in > code-porting. > > But, I think most people who compute svd expect it to produce > full_matrices. Thus, the svd spelling. > > We make somebody unhappy either way I think. If we change anything, I > would vote for changing singular_value_decomposition and adding > something to convert_code to write any old code using full arguments. > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Wed Mar 15 04:45:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 15 04:45:01 2006 Subject: [Numpy-discussion] Re: [ANN] NumPy 0.9.6 released In-Reply-To: References: <44168767.80803@ieee.org> <4417383B.4050000@colorado.edu> <4417487C.2080003@ee.byu.edu> Message-ID: <44180C24.10703@sympatico.ca> Robert Kern wrote: >Travis Oliphant wrote: > > > >>Backward compatibility. >>I don't like the singular_value_decomposition spelling but that was in >>Numeric (with the default given). So, it is provided to aid in >>code-porting. >> >>But, I think most people who compute svd expect it to produce >>full_matrices. Thus, the svd spelling. >> >>We make somebody unhappy either way I think. If we change anything, I >>would vote for changing singular_value_decomposition and adding >>something to convert_code to write any old code using full arguments. >> >> > >Alternatively, we could drop all of the Numeric compatibility forms into a >separate module and have convert_code use that instead of numpy.linalg directly. > > > +1 This would help to reduce the name clutter. Colin W. From cjw at sympatico.ca Wed Mar 15 04:51:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 15 04:51:03 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: <1142418918.25469.0.camel@localhost.localdomain> Message-ID: <44180DB5.3030504@sympatico.ca> Bill Baxter wrote: > > On 3/15/06, *Paulo J. S. Silva* > wrote: > > > > The other difference which isn't really obvious from looking at the > > source code is that matrices always have at least rank > > Just to make clear. They *always* have rank two. > > > Not necessarily. > > >>> m = numpy.matrix('[1 2 3; 4 5 6]') > >>> mm = m[numpy.matrix('[0 1]')] > >>> m > matrix([[1, 2, 3], > [4, 5, 6]]) > >>> mm > matrix([[[[1, 2, 3]], > > [[4, 5, 6]]]]) > >>> mm.shape > (1, 2, 1, 3) > > Maybe that's not supposed to be possible, but with current numpy > (well, 0.9.5 at least) you see it is possible to get something with > type matrix that is not rank 2. A true matrix has two dimensions. The example above would appear to be a bug. Colin W. > > --bb From mfmorss at aep.com Wed Mar 15 05:05:17 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Wed Mar 15 05:05:17 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44174BCE.5020104@ee.byu.edu> Message-ID: >matrix([[1, 0, 0], > [0, 1, 0], > [0, 0, 1]]) > > >>>>n.linalg.inverse(a) >>>> >>>> >array([[ 1., 0., 0.], > [ 0., 1., 0.], > [ 0., 0., 1.]]) > >I was asked to report if functions in linalg still don't honor the >matrix input type; voila. > As someone who is just starting to experiment with numpy, may I ask, what in general is the expected result of numpy.linalg.inverse(integer_matrix)? Since not all such inverses are integer matrices, what does it mean to "honor the matrix input type?" Is there a matrix analogue of Python's integer scalar division? Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To Sent by: numpy-discussion numpy-discussion- eforge.net cc Subject 03/14/2006 06:03 Re: [Numpy-discussion] [ANN] NumPy PM 0.9.6 released Sven Schreiber wrote: >>>>import numpy as n >>>>n.__version__ >>>> >>>> >'0.9.6' > > >>>>a = n.asmatrix(n.eye(3)) >>>>a >>>> >>>> > > Thanks, but of course a.I should work, right? >>>>n.linalg.cholesky(a) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python24\Lib\site-packages\numpy\linalg\linalg.py", line 135, >in cholesky_decomposition > return transpose(triu(a,k=0)).copy() >NameError: global name 'triu' is not defined > >I think I've seen this error before, was it in 0.9.4? This is a real >show-stopper. > > Thanks again, it would be nice if you could help us by checking out an SVN version before a release ;-) But, better late than never. Also, show-stopper might be a bit of an over-statement. A dumb error, yes. But, if you really depend on this, then it's a simple fix to place a from numpy.lib import * at the top of the linalg.py file. Thanks again for your help. -Travis ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From mfmorss at aep.com Wed Mar 15 05:33:07 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Wed Mar 15 05:33:07 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <441726C7.3020207@ee.byu.edu> Message-ID: >We need some of these people to convert earlier than that, though, so we >can make sure that 1.0 really is ready for prime time. I think it's >very close already or the version number wouldn't be so high. We are >just waiting for more people to start using it and report any issues >that they have before going to 1.0 (I'd also like scalar math to be >implemented pre 1.0 as well in case that requires any C-API additions). I can understand that, but here as a potential industrial users of Numpy, we can't really afford the risk. We're looking at Numpy as a key piece of a Python replacement of commercial software for critical daily production. If we build Numpy/Scipy into our system, it has to work. We don't want to be anyone's beta testers. Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To Sent by: SciPy Users List numpy-discussion- , admin at lists.sourc numpy-discussion eforge.net cc 03/14/2006 03:25 PM Subject Re: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released Paul Ray wrote: >On Mar 14, 2006, at 1:33 PM, Travis Oliphant wrote: > > > >>>Just wondering, does this one also require an update to scipy? >>>And in general do numpy updates always require an update to scipy, >>>too? >>>Or is it only when the numpy C API interface changes? >>> >>> >>It's only when the C-API interface changes that this is necessary. >>Pre-1.0 it is likely to happen at each release. >> >>After that it is less likely to happen. >> >> > >Pre-1.0, I think it is fine for the version numbering to be a free- >for-all, as it has been. There have been many times, it seems, where >the latest numpy didn't work with the latest scipy and stuff like that. > > The current SVN of numpy and the current SVN of scipy usually work with each other (perhaps with only a day lag). >After 1.0, I would like to encourage some discipline with regards to >numpy version numbering. From 1.0, the API should NOT change at all >until 1.1. > This is the plan and it is why we are being slow with getting 1.0 out. But, we *need* people to try NumPy on their codes so that we can determine whether or not API additions are needed. Without this testing we are just guessing based on the developer's own codes which do not cover the breadth. >I think that many people with codes that use Numeric or numarray are >awaiting the numpy 1.0 release as a sign that it is stable and ready >for prime time, before converting their codes. > > We need some of these people to convert earlier than that, though, so we can make sure that 1.0 really is ready for prime time. I think it's very close already or the version number wouldn't be so high. We are just waiting for more people to start using it and report any issues that they have before going to 1.0 (I'd also like scalar math to be implemented pre 1.0 as well in case that requires any C-API additions). Thanks for the feedback. What you described will be the basic behavior once 1.0 is released. Note that right now the only changes that usually need to be made are a re-compile. I know this can still be annoying when you have a deep code stack. -Travis ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From schofield at ftw.at Wed Mar 15 06:06:05 2006 From: schofield at ftw.at (Ed Schofield) Date: Wed Mar 15 06:06:05 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: Message-ID: <44181F59.3030001@ftw.at> mfmorss at aep.com wrote: >> We need some of these people to convert earlier than that, though, so we >> can make sure that 1.0 really is ready for prime time. I think it's >> very close already or the version number wouldn't be so high. We are >> just waiting for more people to start using it and report any issues >> that they have before going to 1.0 (I'd also like scalar math to be >> implemented pre 1.0 as well in case that requires any C-API additions). >> > I can understand that, but here as a potential industrial users of Numpy, > we can't really afford the risk. We're looking at Numpy as a key piece of > a Python replacement of commercial software for critical daily production. > If we build Numpy/Scipy into our system, it has to work. We don't want to > be anyone's beta testers. > NumPy is rapidly become more stable, due the hard work of Travis and its other developers. But my opinion is that SciPy is not yet ready for "critical daily production" unless you're willing to work through bugs or missing functionality. I assume you have compelling reasons for wanting to migrate from your current commercial software to Python / NumPy / SciPy, in which case I suggest you consider a support contract with a company like Enthought. I don't have links to them myself, but the website advertises consulting services for numerical computing on these platforms, and there will be others on the scipy-user list who can give you more information. -- Ed From svetosch at gmx.net Wed Mar 15 06:55:02 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 15 06:55:02 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44174BCE.5020104@ee.byu.edu> References: <44168767.80803@ieee.org> <44173352.4020403@gmx.net> <44174BCE.5020104@ee.byu.edu> Message-ID: <44182A7A.7090304@gmx.net> Travis Oliphant schrieb: >>>>> n.linalg.inverse(a) >>>>> >> array([[ 1., 0., 0.], >> [ 0., 1., 0.], >> [ 0., 0., 1.]]) >> >> I was asked to report if functions in linalg still don't honor the >> matrix input type; voila. >> >> >> > Thanks, but of course a.I should work, right? Yes, I only pasted the non-working stuff I could find. > > Thanks again, it would be nice if you could help us by checking out an > SVN version before a release ;-) Ok, I'll try next time. I built scipy once, so I hope with numpy it's not more difficult than with scipy. (on win32) If somebody has good instructions how to build on win (apart from those on the wiki), please let me know. Thanks for numpy, Sven From svetosch at gmx.net Wed Mar 15 07:04:05 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Mar 15 07:04:05 2006 Subject: [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: Message-ID: <44182CB1.6020004@gmx.net> mfmorss at aep.com schrieb: >> matrix([[1, 0, 0], >> [0, 1, 0], >> [0, 0, 1]]) >> >> >>>>> n.linalg.inverse(a) >>>>> >>>>> >> array([[ 1., 0., 0.], >> [ 0., 1., 0.], >> [ 0., 0., 1.]]) >> >> I was asked to report if functions in linalg still don't honor the >> matrix input type; voila. >> > > As someone who is just starting to experiment with numpy, may I ask, what > in general is the expected result of > numpy.linalg.inverse(integer_matrix)? Since not all such inverses are > integer matrices, what does it mean to "honor the matrix input type?" Is > there a matrix analogue of Python's integer scalar division? > Sorry for this jargon, I was just referring to "matrix" in, "array" out, which should not happen anymore for the linear algebra functions. -sven From strawman at astraw.com Wed Mar 15 09:21:18 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed Mar 15 09:21:18 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: References: <1142418918.25469.0.camel@localhost.localdomain> Message-ID: <44184CF0.20808@astraw.com> Bill Baxter wrote: > > On 3/15/06, *Paulo J. S. Silva* > wrote: > > > > The other difference which isn't really obvious from looking at the > > source code is that matrices always have at least rank > > Just to make clear. They *always* have rank two. > > > Not necessarily. > > >>> m = numpy.matrix('[1 2 3; 4 5 6]') > >>> mm = m[numpy.matrix('[0 1]')] > >>> m > matrix([[1, 2, 3], > [4, 5, 6]]) > >>> mm > matrix([[[[1, 2, 3]], > > [[4, 5, 6]]]]) > >>> mm.shape > (1, 2, 1, 3) > Could you enter this as a bug so it doesn't get forgotten? http://projects.scipy.org/scipy/numpy/newticket Cheers! Andrew From oliphant.travis at ieee.org Wed Mar 15 11:34:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 15 11:34:02 2006 Subject: [Numpy-discussion] help speed up In-Reply-To: References: Message-ID: <44186BEE.2060501@ieee.org> Ferenc.Pintye at eu.decoma.com wrote: > > > > Hi , > > why is this little algorithm so too slow (10x) in Numpy than in numarray ? > (i think, the if-elif-else procedure runs with numpy array indicies so > slow...) > I'm pretty sure it's scalar math. You get array scalars out of numpy arrays which don't have their own math tables, yet and so use the full umath for the moment. It's pretty high on the list of things to fix before 1.0. -Travis From oliphant.travis at ieee.org Wed Mar 15 11:35:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 15 11:35:05 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: <44184CF0.20808@astraw.com> References: <1142418918.25469.0.camel@localhost.localdomain> <44184CF0.20808@astraw.com> Message-ID: <44186C42.9080403@ieee.org> Andrew Straw wrote: > Bill Baxter wrote: > > >> On 3/15/06, *Paulo J. S. Silva* > > wrote: >> >> >> > The other difference which isn't really obvious from looking at the >> > source code is that matrices always have at least rank >> >> Just to make clear. They *always* have rank two. >> >> >> Not necessarily. >> >> >>>>> m = numpy.matrix('[1 2 3; 4 5 6]') >>>>> mm = m[numpy.matrix('[0 1]')] >>>>> m >>>>> >> matrix([[1, 2, 3], >> [4, 5, 6]]) >> >>>>> mm >>>>> >> matrix([[[[1, 2, 3]], >> >> [[4, 5, 6]]]]) >> >>>>> mm.shape >>>>> >> (1, 2, 1, 3) >> >> > Could you enter this as a bug so it doesn't get forgotten? > http://projects.scipy.org/scipy/numpy/newticket > Please check to see if it actually is still broken. I think it's fixed. -Travis > Cheers! > Andrew > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From erin.sheldon at gmail.com Wed Mar 15 12:21:04 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 12:21:04 2006 Subject: [Numpy-discussion] recarray field names Message-ID: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> Hi all- I'm new to numpy and I'm slowly weening myself off of IDL. So far the experience has been very positive. I use recarrays a lot. I often read these recarrays from fits files using pyfits (modified to work with numpy). I find myself doing the following more than I would like: if 'TAG' in rec._coldefs.names: .... It seems messy to be accessing this "hidden" attribute in this way. Is there a plan to add methods to more transparently do such things? e.g. def fieldnames(): return _coldefs.names def field_exists(fieldname): return fieldname.upper() in _coldefs.names def field_index(fieldname): if field_exists(fieldname): return _coldefs.names.index(fieldname.upper()) else: return -1 # or None maybe Thanks, Erin From perry at stsci.edu Wed Mar 15 12:34:03 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Mar 15 12:34:03 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> Message-ID: <76fbbdec3883fc386929a698021fbf8b@stsci.edu> On Mar 15, 2006, at 3:20 PM, Erin Sheldon wrote: > Hi all- > > I'm new to numpy and I'm slowly weening myself off of IDL. So far the > experience has been very positive. > > I use recarrays a lot. I often read these recarrays from fits files > using pyfits (modified to work with numpy). I find myself doing the > following more than I would like: > > if 'TAG' in rec._coldefs.names: > .... > > It seems messy to be accessing this "hidden" attribute in this way. > Is there a plan to add methods to more transparently do such things? > > e.g. > > def fieldnames(): > return _coldefs.names > > def field_exists(fieldname): > return fieldname.upper() in _coldefs.names > > def field_index(fieldname): > if field_exists(fieldname): > return _coldefs.names.index(fieldname.upper()) > else: > return -1 # or None maybe > > Thanks, > Erin > You are right that this is messy. We would like to change this sometime. But we'd like to complete the transition to numpy first before doing that so it may be some months before we can (and it may not look quite like what you suggest). But your point is very valid. Thanks, Perry From oliphant at ee.byu.edu Wed Mar 15 13:32:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 13:32:05 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: Message-ID: <441887B1.90504@ee.byu.edu> mfmorss at aep.com wrote: >>We need some of these people to convert earlier than that, though, so we >>can make sure that 1.0 really is ready for prime time. I think it's >>very close already or the version number wouldn't be so high. We are >>just waiting for more people to start using it and report any issues >>that they have before going to 1.0 (I'd also like scalar math to be >>implemented pre 1.0 as well in case that requires any C-API additions). >> >> > >I can understand that, but here as a potential industrial users of Numpy, >we can't really afford the risk. We're looking at Numpy as a key piece of >a Python replacement of commercial software for critical daily production. >If we build Numpy/Scipy into our system, it has to work. We don't want to >be anyone's beta testers. > > > That's fine. In your situation, you may need to wait until 1.0 before fully switching to NumPy. But, of course you can start using it now. We will not be changing standard interfaces in NumPy. Most of these are derived from Numeric which has been around for a long time. The closer we get to 1.0, the less willing the developers will be to see things change. On the Python-level, for example, I don't expect anything to change except for better support of array sub-classes (i.e. making sure they are preserved through more operations). On the C-level, there may be an additional C-API or two, but I don't see any significant changes to the C-API itself. Because of the way the C-API is loaded, extension modules must be re-compiled if the C-API changes. This is the only real headache at this point I think. Basically, until 1.0 you need to re-build extension modules whenever a new version of NumPy comes out (SciPy consists of a bunch of extension modules and so must be re-built). Once we hit 1.0, you will only need to rebuild on minor increment changes (i.e. 1.0 to 1.1 and bug-fixes will not change the C-API). -Travis From cookedm at physics.mcmaster.ca Wed Mar 15 13:45:01 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Mar 15 13:45:01 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <441887B1.90504@ee.byu.edu> (Travis Oliphant's message of "Wed, 15 Mar 2006 14:31:29 -0700") References: <441887B1.90504@ee.byu.edu> Message-ID: Travis Oliphant writes: > mfmorss at aep.com wrote: > >>>We need some of these people to convert earlier than that, though, so we >>>can make sure that 1.0 really is ready for prime time. I think it's >>>very close already or the version number wouldn't be so high. We are >>>just waiting for more people to start using it and report any issues >>>that they have before going to 1.0 (I'd also like scalar math to be >>>implemented pre 1.0 as well in case that requires any C-API additions). >>> >>> >> >>I can understand that, but here as a potential industrial users of Numpy, >>we can't really afford the risk. We're looking at Numpy as a key piece of >>a Python replacement of commercial software for critical daily production. >>If we build Numpy/Scipy into our system, it has to work. We don't want to >>be anyone's beta testers. >> >> >> > That's fine. In your situation, you may need to wait until 1.0 before > fully switching to NumPy. But, of course you can start using it now. > > We will not be changing standard interfaces in NumPy. Most of these > are derived from Numeric which has been around for a long time. The > closer we get to 1.0, the less willing the developers will be to see > things change. On the Python-level, for example, I don't expect > anything to change except for better support of array sub-classes > (i.e. making sure they are preserved through more operations). > > On the C-level, there may be an additional C-API or two, but I don't > see any significant changes to the C-API itself. Because of the way > the C-API is loaded, extension modules must be re-compiled if the > C-API changes. This is the only real headache at this point I think. Only if it's a change that affects data structures or previously-defined C API functions. Adding new C API functions shouldn't affect old modules (because they're added on to the end of the C API function list). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Wed Mar 15 13:52:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 13:52:02 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: References: <441887B1.90504@ee.byu.edu> Message-ID: <44188C54.3020502@ee.byu.edu> David M. Cooke wrote: >Only if it's a change that affects data structures or >previously-defined C API functions. Adding new C API functions >shouldn't affect old modules (because they're added on to the end of >the C API function list). > > I suppose we need to clarify when the NDARRAY_VERSION number needs a bump then, right. Because if that gets bumped "unnecessarily" it will require re-compiles that aren't really necessary. -Travis From cookedm at physics.mcmaster.ca Wed Mar 15 14:10:04 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Mar 15 14:10:04 2006 Subject: [SciPy-user] [Numpy-discussion] [ANN] NumPy 0.9.6 released In-Reply-To: <44188C54.3020502@ee.byu.edu> (Travis Oliphant's message of "Wed, 15 Mar 2006 14:51:16 -0700") References: <441887B1.90504@ee.byu.edu> <44188C54.3020502@ee.byu.edu> Message-ID: Travis Oliphant writes: > David M. Cooke wrote: > >>Only if it's a change that affects data structures or >>previously-defined C API functions. Adding new C API functions >>shouldn't affect old modules (because they're added on to the end of >>the C API function list). >> >> > I suppose we need to clarify when the NDARRAY_VERSION number needs a > bump then, right. Because if that gets bumped "unnecessarily" it will > require re-compiles that aren't really necessary. > > -Travis Here's things that'll make extensions using the old API ABI-incompatible (application binary interface) with a newer one: 1) semantic changes of function arguments, or of a member of a struct. Changes in type are easier to spot, but it could be possible to have the same type but different meaning. 2) adding a member to a struct before any older ones. That will change the layout. 3) adding a function to the API before any older ones. This changes the layout of C API lookup table Now, here's things that shouldn't change the ABI, and shouldn't require a bump in NDARRAY_VERSION: 1) adding a function to the C API at the end of the lookup table. The old extension won't know it's there, and won't call it :-) 2) adding a member to a struct at the end, assuming that struct is always passed as a pointer, and assuming that it doesn't make the meaning of other members differ from what the extension module expects. 3) defining new macros, constants, etc. I floated the idea a while ago of making the code generation step smarter, along with import_array(), on how to get the C API lookup table, by comparing what API functions the module wants with per-function signatures, to what can be provided, etc. Haven't done anything about it though. I'll see if I can think of something smarter before 1.0. If we're careful, we should be able to have different versions of the API. import_array() could request, say, version 1.0 of the API, and numpy could return that version (even if it's at 1.1). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From wbaxter at gmail.com Wed Mar 15 15:18:07 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Wed Mar 15 15:18:07 2006 Subject: [Numpy-discussion] matrix vs. array In-Reply-To: <44186C42.9080403@ieee.org> References: <1142418918.25469.0.camel@localhost.localdomain> <44184CF0.20808@astraw.com> <44186C42.9080403@ieee.org> Message-ID: I thought this was just expected behavior. Great. Now I can get rid of all those calls to squeeze(). Did it get logged as a bug in the first place, though, before it was fixed? Still probably worth having it in the bug tracker, though. And Travis, do you mean it's fixed in CVS or fixed in 0.9.6? Incidentally, you don't need to index a matrix with a matrix to see it, indexing a matrix with a simple python list also returns the same shape. So mm = m[numpy.matrix('[0 1]')] can also be mm = m[[0,1]] and the bug still happens in 0.9.5. Don't know about 0.9.6 or CVS though. --bb On 3/16/06, Travis Oliphant wrote: > > Andrew Straw wrote: > > Bill Baxter wrote: > > > > > >> On 3/15/06, *Paulo J. S. Silva* >> > wrote: > >> > >> > >> > The other difference which isn't really obvious from looking at > the > >> > source code is that matrices always have at least rank > >> > >> Just to make clear. They *always* have rank two. > >> > >> > >> Not necessarily. > >> > >> > >>>>> m = numpy.matrix('[1 2 3; 4 5 6]') > >>>>> mm = m[numpy.matrix('[0 1]')] > >>>>> m > >>>>> > >> matrix([[1, 2, 3], > >> [4, 5, 6]]) > >> > >>>>> mm > >>>>> > >> matrix([[[[1, 2, 3]], > >> > >> [[4, 5, 6]]]]) > >> > >>>>> mm.shape > >>>>> > >> (1, 2, 1, 3) > >> > >> > > Could you enter this as a bug so it doesn't get forgotten? > > http://projects.scipy.org/scipy/numpy/newticket > > > > Please check to see if it actually is still broken. I think it's fixed. > > > -Travis > > > Cheers! > > Andrew > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > > that extends applications into web and mobile media. Attend the live > webcast > > and join the prime developer group breaking into this new coding > territory! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- William V. Baxter III OLM Digital Kono Dens Building Rm 302 1-8-8 Wakabayashi Setagaya-ku Tokyo, Japan 154-0023 +81 (3) 3422-3380 -------------- next part -------------- An HTML attachment was scrubbed... URL: From erin.sheldon at gmail.com Wed Mar 15 15:42:04 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 15:42:04 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <76fbbdec3883fc386929a698021fbf8b@stsci.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> Message-ID: <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> On 3/15/06, Perry Greenfield wrote: > You are right that this is messy. We would like to change this > sometime. But we'd like to complete the transition to numpy first > before doing that so it may be some months before we can (and it may > not look quite like what you suggest). But your point is very valid. > > Thanks, Perry OK, fair enough. Incidentally, I realized that this attribute _coldefs is not part of recarray anyway, but something added by pyfits. I see now that the names and the formats with a greater than sign concatenated on the front can be extracted from dtype: In [247]: t.dtype Out[247]: [('x', '>f4'), ('y', '>i4')] I could write my own function to extract what I need, but I thought I would ask: is there already a simpler way? And is there a function to compare this '>f4' stuff to the named types such as Float32 ('f')? Erin P.S. If it is man power that is preventing some of the simple things like this from being implemented, I could volunteer some time. From oliphant at ee.byu.edu Wed Mar 15 16:02:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 16:02:01 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> Message-ID: <4418AAB9.3080200@ee.byu.edu> Erin Sheldon wrote: >On 3/15/06, Perry Greenfield wrote: > > >>You are right that this is messy. We would like to change this >>sometime. But we'd like to complete the transition to numpy first >>before doing that so it may be some months before we can (and it may >>not look quite like what you suggest). But your point is very valid. >> >>Thanks, Perry >> >> > >OK, fair enough. > >Incidentally, I realized that this attribute >_coldefs is not part of recarray anyway, but something added by pyfits. >I see now that the names and the formats with a greater than sign >concatenated on the front can be extracted from dtype: > >In [247]: t.dtype >Out[247]: [('x', '>f4'), ('y', '>i4')] > >I could write my own function to extract what I need, but I thought I >would ask: is there already a simpler way? And is there a function to >compare this '>f4' stuff to the named types such as Float32 ('f')? > > The dtype object does contain what you want. In fact. It's the fields attribute of the dtype object that is a dictionary accessed by field name. Thus, to see if a field is a valid field itdentifier, if name in t.dtype.fields: would work (well there is a slight problem in that -1 is a special key to the dictionary that returns a list of field names ordered by offset and so would work also), but if you now that name is already a string, then no problem. -Travis From Fernando.Perez at colorado.edu Wed Mar 15 16:08:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 15 16:08:06 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418AAB9.3080200@ee.byu.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> Message-ID: <4418AC54.1060202@colorado.edu> Travis Oliphant wrote: > The dtype object does contain what you want. In fact. It's the fields > attribute of the dtype object that is a dictionary accessed by field > name. Thus, to see if a field is a valid field itdentifier, > > if name in t.dtype.fields: > > would work (well there is a slight problem in that -1 is a special key > to the dictionary that returns a list of field names ordered by offset > and so would work also), but if you now that name is already a string, > then no problem. Mmh, just curious: I wonder about the wisdom of that overloading of a 'magic' key (-1). It will make thinks like for name in t.dtype.fields: return a spurious entry (the -1), and having the sorted list accessed as for name in t.dtype.fields[-1]: reads weird. I'm sure there was a good reason behind this, but I wonder if it wouldn't be better to provide this particular functionality (the list currently associated with the special -1 key) via a different mechanism, and guaranteeing that t.dtype.fields.keys() == [ list of valid names ]. It just sounds like enforcing a bit of API orthogonality here would be a good thing in the long run, but perhaps I'm missing something (I don't claim to know the reasoning that went behind today's implementation). Best, f From erin.sheldon at gmail.com Wed Mar 15 16:15:04 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 16:15:04 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418AAB9.3080200@ee.byu.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> Message-ID: <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> On 3/15/06, Travis Oliphant wrote: > The dtype object does contain what you want. In fact. It's the fields > attribute of the dtype object that is a dictionary accessed by field > name. Thus, to see if a field is a valid field itdentifier, > > if name in t.dtype.fields: > > would work (well there is a slight problem in that -1 is a special key > to the dictionary that returns a list of field names ordered by offset > and so would work also), but if you now that name is already a string, > then no problem. Yes, I see, but I think you meant if name in t.dtype.fields.keys(): which contains the -1 as a key. In [275]: t.dtype.fields.keys: Out[275]: ('tag1','tag2',-1) So, since the last key points to the names, one can also do: In [279]: t.dtype.fields[-1] Out[279]: ('tag1', 'tag2') which is not transparent, but does what you need. For now, I'll probably just write a simple function to wrap this. Thanks, Erin From oliphant at ee.byu.edu Wed Mar 15 16:26:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 16:26:06 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> Message-ID: <4418B083.7080606@ee.byu.edu> Erin Sheldon wrote: >On 3/15/06, Travis Oliphant wrote: > > >>The dtype object does contain what you want. In fact. It's the fields >>attribute of the dtype object that is a dictionary accessed by field >>name. Thus, to see if a field is a valid field itdentifier, >> >>if name in t.dtype.fields: >> >>would work (well there is a slight problem in that -1 is a special key >>to the dictionary that returns a list of field names ordered by offset >>and so would work also), but if you now that name is already a string, >>then no problem. >> >> > >Yes, I see, but I think you meant > > if name in t.dtype.fields.keys(): > > Actually, you can use this with the dictionary itself (no need to get the keys...) name in t.dtype.fields is equivalent to name in t.dtype.fields.keys() -Travis From Fernando.Perez at colorado.edu Wed Mar 15 16:30:04 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 15 16:30:04 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> Message-ID: <4418B16D.40306@colorado.edu> Erin Sheldon wrote: > Yes, I see, but I think you meant > > if name in t.dtype.fields.keys(): No, he really meant: if name in t.dtype.fields: dictionaries are iterators, so you don't need to construct the list of keys separately. It's just a redundant waste of time and memory in most cases, unless you intend to modify the dict in your loop, case in which the iterator approach won't work and you /do/ need the explicit keys() call. In addition if name in t.dtype.fields is faster than: if name in t.dtype.fields.keys() While both are O(N) operations, the first requires a single call to the hash function on 'name' and then a C lookup in the dict's internal key table as a hash table, while the second is a direct walkthrough of a list with python-level equality testing. In [15]: nkeys = 1000000 In [16]: dct = dict(zip(keys,[None]*len(keys))) In [17]: time bool(-1 in keys) CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s Wall time: 0.01 Out[17]: False In [18]: time bool(-1 in dct) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 Out[18]: False In realistic cases for your original question you are not likely to see the difference, but it's always a good idea to be aware of the performance characteristics of various approaches. For a different problem, there may well be a real difference. Cheers, f From oliphant at ee.byu.edu Wed Mar 15 16:33:43 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 15 16:33:43 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418AC54.1060202@colorado.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <4418AC54.1060202@colorado.edu> Message-ID: <4418B223.5020201@ee.byu.edu> Fernando Perez wrote: > > Mmh, just curious: I wonder about the wisdom of that overloading of a > 'magic' key (-1). It will make thinks like > > for name in t.dtype.fields: No real wisdom. More organic growth. Intially I didn't have an ordered list of fields, but as more complicated data-type descriptors were supported, this became an important thing to have. I should have probably added a n additional element to the PyArray_Descr structure. Remember, it was growing out of the old PyArray_Descr already and I was a little concerned about changing it too much (there are ramifications of changing this structure in several places). So, instead of adding a "ordered_names" tuple to the dtype object, I just stuck it in the fields dictionary. I agree it's kind of odd sitting there. It probably deserves a re-factoring and pulling that out into a new attribute of the dtype object. This would mean that the PyArray_Descr structure would need a new object (names perhaps), and it would need to be tracked. Not a huge change and probably worth it before the next release. -Travis From Fernando.Perez at colorado.edu Wed Mar 15 16:37:22 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 15 16:37:22 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418B16D.40306@colorado.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> <4418B16D.40306@colorado.edu> Message-ID: <4418B320.5070200@colorado.edu> Fernando Perez wrote: > In addition > > if name in t.dtype.fields > > is faster than: > > if name in t.dtype.fields.keys() > > While both are O(N) operations, the first requires a single call to the hash > function on 'name' and then a C lookup in the dict's internal key table as a > hash table, while the second is a direct walkthrough of a list with > python-level equality testing. [ sorry, copy-pasted wrong timing run] In [1]: nkeys = 5000000 In [2]: keys=range(nkeys) In [3]: dct = dict(zip(keys,[None]*len(keys))) In [4]: time bool(-1 in dct) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 Out[4]: False In [5]: time bool(-1 in keys) CPU times: user 0.32 s, sys: 0.00 s, total: 0.32 s Wall time: 0.33 Out[5]: False Cheers, f From erin.sheldon at gmail.com Wed Mar 15 16:37:23 2006 From: erin.sheldon at gmail.com (Erin Sheldon) Date: Wed Mar 15 16:37:23 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <4418B16D.40306@colorado.edu> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> <4418B16D.40306@colorado.edu> Message-ID: <331116dc0603151636i38f0d446n2271be64efcfc293@mail.gmail.com> Nice. Python decides to compare with the keys and not the values. The possibilities for obfuscation are endless. On 3/15/06, Fernando Perez wrote: > Erin Sheldon wrote: > > > Yes, I see, but I think you meant > > > > if name in t.dtype.fields.keys(): > > No, he really meant: > > if name in t.dtype.fields: > > dictionaries are iterators, so you don't need to construct the list of keys > separately. It's just a redundant waste of time and memory in most cases, > unless you intend to modify the dict in your loop, case in which the iterator > approach won't work and you /do/ need the explicit keys() call. > > In addition > > if name in t.dtype.fields > > is faster than: > > if name in t.dtype.fields.keys() > > While both are O(N) operations, the first requires a single call to the hash > function on 'name' and then a C lookup in the dict's internal key table as a > hash table, while the second is a direct walkthrough of a list with > python-level equality testing. > > In [15]: nkeys = 1000000 > > In [16]: dct = dict(zip(keys,[None]*len(keys))) > > In [17]: time bool(-1 in keys) > CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s > Wall time: 0.01 > Out[17]: False > > In [18]: time bool(-1 in dct) > CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s > Wall time: 0.00 > Out[18]: False > > > In realistic cases for your original question you are not likely to see the > difference, but it's always a good idea to be aware of the performance > characteristics of various approaches. For a different problem, there may > well be a real difference. > > Cheers, > > f > From robert.kern at gmail.com Wed Mar 15 16:51:25 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 15 16:51:25 2006 Subject: [Numpy-discussion] Re: recarray field names In-Reply-To: <331116dc0603151636i38f0d446n2271be64efcfc293@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> <4418AAB9.3080200@ee.byu.edu> <331116dc0603151614x721fa159w80fea300be8e302b@mail.gmail.com> <4418B16D.40306@colorado.edu> <331116dc0603151636i38f0d446n2271be64efcfc293@mail.gmail.com> Message-ID: Erin Sheldon wrote: > Nice. Python decides to compare with the keys and not the values. Sure. It is a ridiculously common to ask a dictionary if it has a record for a particular key. It is much, much rarer to ask one if it has a particular value. Lists, tuples, and sets, on the other hand, only have one kind of interesting data, the values, so the __contains__ method operates on values with them. Practicality beats purity, in this case. > The possibilities for obfuscation are endless. Not in my experience. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From hurak at control.felk.cvut.cz Wed Mar 15 23:44:05 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Wed Mar 15 23:44:05 2006 Subject: [Numpy-discussion] How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) Message-ID: Hello, is anybody here working with Numpy on Gentoo Linux system? I have troubles with compiling it with ATLAS-BLAS and ATLAS-LAPACK. The trouble is perhaps that Gentoo has a little bit different approach to storing optimized BLAS and LAPACK on the system then other Linux distros. And apparently Numpy cannot find it. Numpy documentation says that I should specify the path to ATLAS in site.cfg in numpy/distutils, but I cannot find one. Thanks for your help, Zdenek Hurak ------------------------------------------------------------------ hurak at note-zhurak ~ $ ls -l /usr/lib | grep atlas -rw-r--r-- 1 root root 5293184 led 27 15:57 libatlas.a -rwxr-xr-x 1 root root 793 led 27 15:57 libatlas.la lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so -> libatlas.so.0.0.0 lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so.0 -> libatlas.so.0.0.0 -rwxr-xr-x 1 root root 2883644 led 27 15:57 libatlas.so.0.0.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libblas.so -> blas/atlas/libblas.so lrwxrwxrwx 1 root root 23 led 27 15:57 libblas.so.0 -> blas/atlas/libblas.so.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libcblas.a -> blas/atlas/libcblas.a lrwxrwxrwx 1 root root 22 led 27 15:57 libcblas.so -> blas/atlas/libcblas.so lrwxrwxrwx 1 root root 24 led 27 15:57 libcblas.so.0 -> blas/atlas/libcblas.so.0 lrwxrwxrwx 1 root root 25 led 27 21:37 liblapack.so -> lapack/atlas/liblapack.so lrwxrwxrwx 1 root root 27 led 27 21:37 liblapack.so.0 -> lapack/atlas/liblapack.so.0 ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6/numpy/distutils $ cat ../../build/lib.linux-i686-2.4/numpy/__config__.py # This file is generated by /home/hurak/tmp/install/numpy-0.9.6/setup.py # It contains system_info results at the time of building this package. __all__ = ["get_info","show"] blas_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} lapack_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} atlas_threads_info={} blas_opt_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_blas_threads_info={} lapack_opt_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_info={} lapack_mkl_info={} blas_mkl_info={} atlas_blas_info={} mkl_info={} [...] ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6 $ ls numpy/distutils/ ccompiler.py exec_command.pyc intelccompiler.py system_info.py ccompiler.pyc extension.py lib2def.py system_info.pyc command extension.pyc line_endings.py tests conv_template.py fcompiler log.py unixccompiler.py conv_template.pyc from_template.py log.pyc unixccompiler.pyc core.py from_template.pyc mingw32ccompiler.py __version__.py core.pyc info.py misc_util.py __version__.pyc cpuinfo.py info.pyc misc_util.pyc cpuinfo.pyc __init__.py setup.py exec_command.py __init__.pyc setup.pyc ------------------------------------------------------------------ From nadavh at visionsense.com Thu Mar 16 02:57:04 2006 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu Mar 16 02:57:04 2006 Subject: [Numpy-discussion] How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) Message-ID: <07C6A61102C94148B8104D42DE95F7E8C8EF57@exchange2k.envision.co.il> On a P4 machine my site.cfg is: [DEFAULT] library_dirs = /usr/lib:/usr/local/lib include_dirs = /usr/include:/usr/local/include src_dirs = /usr/local/src:/opt/src # search static libraries (.a) in preference to shared ones (.so) search_static_first = 0 [fftw] fftw_libs = fftw3, fftw3f fftw_opt_libs = fftw3_threaded, fftw3f_threaded # if the above aren't found, look for {s,d}fftw_libs and {s,d}fftw_opt_libs [atlas] library_dirs = /usr/lib:/usr/lib/blas/atlas # for overriding the names of the atlas libraries atlas_libs = lapack, f77blas, cblas, atlas [x11] library_dirs = /usr/X11R6/lib include_dirs = /usr/X11R6/include On a amd64 I prefer to use the acml library, which preformance is about the same as ATLAS, but contains full blas/lapack compability. Nadav. -----Original Message----- From: numpy-discussion-admin at lists.sourceforge.net on behalf of Zdenek Hur?k Sent: Thu 16-Mar-06 09:44 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) Hello, is anybody here working with Numpy on Gentoo Linux system? I have troubles with compiling it with ATLAS-BLAS and ATLAS-LAPACK. The trouble is perhaps that Gentoo has a little bit different approach to storing optimized BLAS and LAPACK on the system then other Linux distros. And apparently Numpy cannot find it. Numpy documentation says that I should specify the path to ATLAS in site.cfg in numpy/distutils, but I cannot find one. Thanks for your help, Zdenek Hurak ------------------------------------------------------------------ hurak at note-zhurak ~ $ ls -l /usr/lib | grep atlas -rw-r--r-- 1 root root 5293184 led 27 15:57 libatlas.a -rwxr-xr-x 1 root root 793 led 27 15:57 libatlas.la lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so -> libatlas.so.0.0.0 lrwxrwxrwx 1 root root 17 led 27 15:57 libatlas.so.0 -> libatlas.so.0.0.0 -rwxr-xr-x 1 root root 2883644 led 27 15:57 libatlas.so.0.0.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libblas.so -> blas/atlas/libblas.so lrwxrwxrwx 1 root root 23 led 27 15:57 libblas.so.0 -> blas/atlas/libblas.so.0 lrwxrwxrwx 1 root root 21 led 27 15:57 libcblas.a -> blas/atlas/libcblas.a lrwxrwxrwx 1 root root 22 led 27 15:57 libcblas.so -> blas/atlas/libcblas.so lrwxrwxrwx 1 root root 24 led 27 15:57 libcblas.so.0 -> blas/atlas/libcblas.so.0 lrwxrwxrwx 1 root root 25 led 27 21:37 liblapack.so -> lapack/atlas/liblapack.so lrwxrwxrwx 1 root root 27 led 27 21:37 liblapack.so.0 -> lapack/atlas/liblapack.so.0 ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6/numpy/distutils $ cat ../../build/lib.linux-i686-2.4/numpy/__config__.py # This file is generated by /home/hurak/tmp/install/numpy-0.9.6/setup.py # It contains system_info results at the time of building this package. __all__ = ["get_info","show"] blas_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} lapack_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'language': 'f77'} atlas_threads_info={} blas_opt_info={'libraries': ['blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_blas_threads_info={} lapack_opt_info={'libraries': ['lapack', 'blas'], 'library_dirs': ['/usr/lib'], 'define_macros': [('NO_ATLAS_INFO', 1)], 'language': 'f77'} atlas_info={} lapack_mkl_info={} blas_mkl_info={} atlas_blas_info={} mkl_info={} [...] ------------------------------------------------------------------ hurak at note-zhurak ~/tmp/install/numpy-0.9.6 $ ls numpy/distutils/ ccompiler.py exec_command.pyc intelccompiler.py system_info.py ccompiler.pyc extension.py lib2def.py system_info.pyc command extension.pyc line_endings.py tests conv_template.py fcompiler log.py unixccompiler.py conv_template.pyc from_template.py log.pyc unixccompiler.pyc core.py from_template.pyc mingw32ccompiler.py __version__.py core.pyc info.py misc_util.py __version__.pyc cpuinfo.py info.pyc misc_util.pyc cpuinfo.pyc __init__.py setup.py exec_command.py __init__.pyc setup.pyc ------------------------------------------------------------------ ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From rudolphv at gmail.com Thu Mar 16 05:02:11 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Thu Mar 16 05:02:11 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux Message-ID: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> I followed the instructions given on http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 (linked to from main Numpy / SciPy wiki) to try and install Numpy + Scipy on a 64bit Ubuntu Linux system, but something is not working. I get the following error message during the 'build phase of Numpy (0.9.6): /usr/bin/g77 -shared build/temp.linux-x86_64-2.4/numpy/linalg/lapack_litemodule.o -L/usr/local/lib/atlas -llapack -lptf77blas -lptcblas -latlas -lg2c-pic -o build/lib.linux-x86_64-2.4/numpy/linalg/lapack_lite.so /usr/bin/ld: /usr/local/lib/atlas/liblapack.a(dlamch.o): relocation R_X86_64_32S can not be used when making a shared object; recompile with -fPIC /usr/local/lib/atlas/liblapack.a: could not read symbols: Bad value collect2: ld returned 1 exit status The main problem seems to be this part... "relocation R_X86_64_32S can not be used when making a shared object; recompile with -fPIC". This seems to be related to the ATLAS optimized LAPACK library which I built using the standard method as laid out in the ATLAS errata. The weird thing is that I did in fact specify the -fPIC compiler flag, so I'm not sure how to proceed from here. Any thoughts? Thanks R. -- Rudolph van der Merwe From matthew.brett at gmail.com Thu Mar 16 06:22:07 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu Mar 16 06:22:07 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux In-Reply-To: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> References: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> Message-ID: <1e2af89e0603160621l1164cbfbm7fe763489a99d365@mail.gmail.com> Hi, > /usr/bin/g77 -shared > build/temp.linux-x86_64-2.4/numpy/linalg/lapack_litemodule.o > -L/usr/local/lib/atlas -llapack -lptf77blas -lptcblas -latlas > -lg2c-pic -o build/lib.linux-x86_64-2.4/numpy/linalg/lapack_lite.so > /usr/bin/ld: /usr/local/lib/atlas/liblapack.a(dlamch.o): relocation > R_X86_64_32S can not be used when making a shared object; recompile > with -fPIC I wonder if you built the lapack libraries with the -fPIC flag though? I've attached my build scripts for atlas, the include make file for the lapack libraries, and a python script to combine them - I am wondering if you need the modified include make file, which should somehow end up in your LAPACK directory as make.inc (see the build_full.py script). Best, Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: atlas_lapack_build.tgz Type: application/x-gzip Size: 3017 bytes Desc: not available URL: From Norbert.Nemec.list at gmx.de Thu Mar 16 08:01:02 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu Mar 16 08:01:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: References: Message-ID: <44198B7B.6040706@gmx.de> I have a very much related problem: Not only that the idea described by Mads Ipsen does not work, but I could generally find no efficient way to do a "counting" of elements in an array, as it is needed for a histogram. The function "histogram" contained in numpy uses a rather inefficient method, involving the sorting of the data array. What would instead be needed is a function that simply gives the count of occurances of given values in a given array: >>> [4,5,2,3,2,1,4].count([0,1,2,3,4,5]) [0,1,2,1,1,2] All the solutions that I found so far involve either sorting of the data or writing a loop in Python, both of which are unacceptable for performance. Am I missing something obvious? Mads Ipsen wrote: >Hey, > >First of all, thanks for the new release. > >Here's another question regarding something I cannot quite understand: > >Suppose you want to update bins for a histogram, you might think you >could do something like: > > g = zeros(4,Int) > x = array([0.2, 0.2]) > idx = floor(x/0.1).astype(int) > g[idx] += 1 > >Here idx becomes > > array([2, 2]) > >In this case, I would naively expect g to end up like > > array([0, 0, 2, 0]) (1) > >but instead you get > > array([0, 0, 1, 0]) (2) > >Is this intended? Just being plain novice-like naive, I would expect >the slice operation g[idx] += 1 to do something like > > for i in range(len(I)): > g[ idx[i] ] += 1 > >resulting in (1) and not (2). > >// Mads > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From robert.kern at gmail.com Thu Mar 16 08:43:10 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 16 08:43:10 2006 Subject: [Numpy-discussion] Re: How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) In-Reply-To: References: Message-ID: Zden?k Hur?k wrote: > Hello, > > is anybody here working with Numpy on Gentoo Linux system? I have troubles > with compiling it with ATLAS-BLAS and ATLAS-LAPACK. The trouble is perhaps > that Gentoo has a little bit different approach to storing optimized BLAS > and LAPACK on the system then other Linux distros. And apparently Numpy > cannot find it. Numpy documentation says that I should specify the path to > ATLAS in site.cfg in numpy/distutils, but I cannot find one. Nadav gave you a sample site.cfg. The place to put it would be next to NumPy's setup.py script. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From a.h.jaffe at gmail.com Thu Mar 16 08:45:04 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu Mar 16 08:45:04 2006 Subject: [Numpy-discussion] Status of pyfits with numpy? Message-ID: <441995D3.4070309@gmail.com> Hi All, Is a numpy-compatible version of pyfits nearing stability? I see that there is a pyfits-numpy branch visible on the trac browser -- is that available and stable enough for public use? Thanks, Andrew From rmuller at sandia.gov Thu Mar 16 09:07:07 2006 From: rmuller at sandia.gov (Rick Muller) Date: Thu Mar 16 09:07:07 2006 Subject: [Numpy-discussion] Converting a large package from Numeric to numpy Message-ID: <344D571E-58B6-4DAD-8A65-BE1A9E6F109A@sandia.gov> Before I ask for help I would first like to thank everyone on the Numpy team for the hard work that they've done. A longtime user of Numeric, I'm really impressed with what I'm seeing in Numpy thus far, particularly in the configuration routines, which seem to work wonderfully so far. I maintain a Python Quantum Chemistry package called PyQuante (http:// pyquante.sourceforge.net) that is built on top of Numeric/ LinearAlgebra. I wanted to see what would be involved in converting everything over to the new packages. The big workhorse for me is the dot routine, and I ran a few tests that implied that dot ran faster in Numpy than it did in Numeric, so I was ready to go. The conversion turned out to be easy. I wrote an interface routine called NumWrap that has a flag (use_numpy) that determines whether to import from Numeric or numpy. I then modified my code to import everything from NumWrap. So far very easy. I have a test-suite that exercises many different parts of my code. I ran the test suite using Numeric and it took 351 sec. I then ran using numpy and it took 615 sec!! Almost twice as long. (I should point out that the very good news was that the test suite ran the first time I tried it with the numpy routines without any errors!) I started digging into the differences using the python profiler, and was surprised what turned up. The big difference turned out to be routines that were taking various mathematical functions (pow, exp, even simple scalar multiplication) of array elements. This routine: def vwn_xx(x,b,c): return x*x+b*x+c which is called on many elements of a very large grid, went from taking 17 sec in the test suite to taking 68 sec in the test suite. Another routine that took a log() and a pow() of grid elements, went from 15 sec to 100 sec. I looked through Travis' new documentation. I first tried the .toscalar() function, as follows: rho = 0.5*dens[i].toscalar() but this didn't work, I got AttributeError: 'float64scalar' object has no attribute 'toscalar' So now I'm using: rho = 0.5*float(dens[i]) and this fixed the problem -- it makes the timings go back to the same (or faster) than the Numeric values. So my question is has anyone else run into problems of this sort? If so, is there a "proper", sci-pythonic way of handling conversions like this? Thanks in advance for any help you can offer, and thanks again for the great new package! Rick Rick Muller rmuller at sandia.gov From oliphant.travis at ieee.org Thu Mar 16 09:13:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 16 09:13:07 2006 Subject: [Numpy-discussion] Converting a large package from Numeric to numpy In-Reply-To: <344D571E-58B6-4DAD-8A65-BE1A9E6F109A@sandia.gov> References: <344D571E-58B6-4DAD-8A65-BE1A9E6F109A@sandia.gov> Message-ID: <44199C88.3040403@ieee.org> Rick Muller wrote: > > The conversion turned out to be easy. I wrote an interface routine > called NumWrap that has a flag (use_numpy) that determines whether to > import from Numeric or numpy. I then modified my code to import > everything from NumWrap. So far very easy. Good news. Conversion from Numeric should be really easy and conversion from numarray only a little-bit harder. > > I have a test-suite that exercises many different parts of my code. I > ran the test suite using Numeric and it took 351 sec. I then ran using > numpy and it took 615 sec!! Almost twice as long. (I should point out > that the very good news was that the test suite ran the first time I > tried it with the numpy routines without any errors!) Probably scalar math again. I'm working on that right now, so hopefully it will be faster in the next release of numpy. > > I started digging into the differences using the python profiler, and > was surprised what turned up. The big difference turned out to be > routines that were taking various mathematical functions (pow, exp, > even simple scalar multiplication) of array elements. > This routine: > > def vwn_xx(x,b,c): return x*x+b*x+c > > which is called on many elements of a very large grid, went from > taking 17 sec in the test suite to taking 68 sec in the test suite. > Another routine that took a log() and a pow() of grid elements, went > from 15 sec to 100 sec. > > I looked through Travis' new documentation. I first tried the > .toscalar() function, as follows: > rho = 0.5*dens[i].toscalar() Sorry, you may have an older version. The new spelling is .item() ;-) -Travis From perry at stsci.edu Thu Mar 16 10:44:05 2006 From: perry at stsci.edu (Perry Greenfield) Date: Thu Mar 16 10:44:05 2006 Subject: [Numpy-discussion] recarray field names In-Reply-To: <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> References: <331116dc0603151220xf971e2bq1bbd64b0da4385f7@mail.gmail.com> <76fbbdec3883fc386929a698021fbf8b@stsci.edu> <331116dc0603151541x5bf4a396la3c737d66db174c2@mail.gmail.com> Message-ID: <2cde6476418820b6b798a44cc957dedb@stsci.edu> On Mar 15, 2006, at 6:41 PM, Erin Sheldon wrote: > On 3/15/06, Perry Greenfield wrote: >> You are right that this is messy. We would like to change this >> sometime. But we'd like to complete the transition to numpy first >> before doing that so it may be some months before we can (and it may >> not look quite like what you suggest). But your point is very valid. >> >> Thanks, Perry > > > Erin > P.S. If it is man power that is preventing some of the simple things > like this from being implemented, I could volunteer some time. > Ultimately, yes. But there's a little more to it. We've felt there are a number of improvements that can be made and that these might benefit from thinking more globally about how to do them rather than add a few convenience functions. Then there is the issue of whether these changes are added to the numarray version, etc. But help would be welcomed. I'd say wait a little bit until we put out an alpha and then start some discussion on how it should be improved. The astropy mailing list is a better forum for that than this one though. Thanks, Perry From pebarrett at gmail.com Thu Mar 16 10:49:00 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Thu Mar 16 10:49:00 2006 Subject: [Numpy-discussion] Status of pyfits with numpy? In-Reply-To: <441995D3.4070309@gmail.com> References: <441995D3.4070309@gmail.com> Message-ID: <40e64fa20603161048r2c08117ay916ae897011a3b6f@mail.gmail.com> On 3/16/06, Andrew Jaffe wrote: > > Hi All, > > Is a numpy-compatible version of pyfits nearing stability? I see that > there is a pyfits-numpy branch visible on the trac browser -- is that > available and stable enough for public use? > I received the following Email from Chris Hanley, who is doing the numpy port, a few days ago. -- Paul I've finished the initial pyfits numpy port. We won't be doing an alpha release until after we finish the spring STSDAS release. We are in the process of freezing this week. I thought you might want a preview. If you do, you can grab it our of subversion with, svn co http://astropy.scipy.org/svn/pyfits/branches/pyfits-numpypyfits-numpy. That version of pyfits will support both numpy and numarray. If you have both numpy and numarray installed you can make pyfits switch between them by setting an environment variable called NUMERIX. NUMERIX = 'numarray' for numarray. NUMERIX = 'numpy' for numpy. If the variable isn't set, and both packages are installed, pyfits will default to the numarray version. If only one package is installed then pyfits will use that package. Most of the differences for image support are in syntax. However, for table support I had to completely rework the inheritance structure. Be warned, I am certain there are bugs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chanley at stsci.edu Thu Mar 16 11:12:11 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Thu Mar 16 11:12:11 2006 Subject: [Numpy-discussion] Status of pyfits with numpy? In-Reply-To: <441995D3.4070309@gmail.com> References: <441995D3.4070309@gmail.com> Message-ID: <4419B86D.3050801@stsci.edu> Andrew, We are in the process of preparing an "alpha" release of a numpy-compatible version of pyfits. I have recently merged my numpy-pyfits branch with the pyfits trunk so if you do not wish to wait for a release you can grab it our of SVN with: svn co http://astropy.scipy.org/svn/pyfits/trunk pyfits This version of pyfits is not well tested but you are welcome to try it out and send me bug reports. A couple of people have already tried an earlier version and have reported some issues with the table support. We are currently working to address those issues. If you have both numpy and numarray installed you can make pyfits switch between them by setting an environment variable called NUMERIX. NUMERIX = 'numarray' for numarray. NUMERIX = 'numpy' for numpy. If the variable isn't set, and both packages are installed, pyfits will default to the numarray version. If only one package is installed then pyfits will use that package. Chris Andrew Jaffe wrote: > Hi All, > > Is a numpy-compatible version of pyfits nearing stability? I see that > there is a pyfits-numpy branch visible on the trac browser -- is that > available and stable enough for public use? > > Thanks, > > Andrew > > From oliphant at ee.byu.edu Thu Mar 16 13:04:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 16 13:04:05 2006 Subject: [Numpy-discussion] SVN version of NumPy will require rebuild of extensions Message-ID: <4419D2A1.8020504@ee.byu.edu> This is to serve notice that I altered the C-API table in the SVN version of numpy. This change will require a rebuild of extension modules. You are supposed to get notice of that when you try to import the extension module (but that didn't work for me for some reason, importing scipy just segfaulted --- ahh.. I just figured it out... the command that gets the version number is located down in the table. This location has just changed. It makes a case that this command should always be the first entry in the C-API table). The modifications (with a few new C-API calls) will allow construction of the scalar-math tables. -Travis From oliphant at ee.byu.edu Thu Mar 16 14:55:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 16 14:55:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] nd_image In-Reply-To: <1142545376.4419dbe0b1903@webmail.ster.kuleuven.ac.be> References: <1142545376.4419dbe0b1903@webmail.ster.kuleuven.ac.be> Message-ID: <4419ECB1.6050709@ee.byu.edu> joris at ster.kuleuven.ac.be wrote: > [TO]: So, what we have in SciPy is "the official" home for the numpy > [TO]: version of his package. > >Sorry, I am a bit lost here. So nd_image was a numarray library, and is now >available via scipy where it works with numpy arrays (correct?). > nd_image was a Package that built on top of numarray and was also distributed with numarray. Exactly what goes in the base numpy backage and what goes into add-on packages is a question we will wrestle with for a while. With the existence of SciPy, I don't favor growing NumPy much beyond the (general) feature-set it has now. There are already people who grumble that it includes too-much stuff that they don't need. Other functionality should be added by other packages. We are trying to make SciPy into a large collection of packages that don't all have to be installed, so that users can pick and choose what they need if they must. >At the moment, >the library is unmaintained though, regardless whether it would be part of >scipy or numpy. Correct? If so, why exactly would it be better to put it under >scipy rather than under numpy? > > Not directly accurate. The SciPy version is maintained by the SciPy developers (not by the original author). >Some people don't need scipy for their work and are happy with numpy alone. >It used to be rather easy for them to use nd_image with numarray, but if I >understand you correctly, they would now have to install an entire new package, > > Many SciPy packages can be installed separately and nd_image (now scipy.image) is one of them. So, it is very easy to just install the scipy.image package. Hopefully, more people will run with this concept as the need arises. And, scipy is not has hard to install as you might think. Especially if you go into the setup.py file and comment out all the packages you don't actually want... -Travis From oliphant at ee.byu.edu Thu Mar 16 16:51:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 16 16:51:04 2006 Subject: [Numpy-discussion] Array scalar math ideas Message-ID: <441A07C9.8050101@ee.byu.edu> I'm starting to write the remaining array scalar math tables (so that we can speed up scalar*scalar arithmetic). Right now, all operations involving scalars are converted to 0-d arrays - use the ufunc machinery - and are converted back to scalars after the operation. The scalarmathmodule.c.src file is being written to fix this and insert type-appropriate operations for each of the (number-type) array scalars. My general strategy for the binary operations is going to be the following. I wanted to bounce it off the list to see what other ideas people had: Code outline: Convert inputs so that self is the array scalar of some type and other is the other object if (other is an array scalar of the same data-type as self) arg3 = other else if (other is an array scalar of a different data-type as self) arg3 = convert_other_to_self_data_type(other) else if (other is a Python scalar) arg3 = convert_Python_scalar_to_array_scalar(other) else if (other is a 0-d array) arg3 = convert_other_to_self_data_type_from_0-d_array(other) else return (use ufunc to calculate result). return (operation using self and arg3) if an error condition is encountered, then only at that point, the proper way to handle it will be determined by looking in the local / global / builtin scope for the error-handling variable. Tthis avoids the overhead of looking up what to do to the case of an error actually occurring --- I need to change the ufuncobject.c code to also do this and save the overhead there too --- right now what to do is looked up every time, rather than waiting until an error is actually detected.) What do people think? -Travis From cjw at sympatico.ca Thu Mar 16 17:11:13 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Mar 16 17:11:13 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: <441A07C9.8050101@ee.byu.edu> References: <441A07C9.8050101@ee.byu.edu> Message-ID: <441A0C95.30902@sympatico.ca> Travis Oliphant wrote: > > I'm starting to write the remaining array scalar math tables (so that > we can speed up scalar*scalar arithmetic). > Right now, all operations involving scalars are converted to 0-d > arrays - use the ufunc machinery - and are converted back to scalars > after the operation. > > The scalarmathmodule.c.src file is being written to fix this and > insert type-appropriate operations for each of the (number-type) array > scalars. > > > My general strategy for the binary operations is going to be the > following. I wanted to bounce it off the list to see what other ideas > people had: > > Code outline: > > Convert inputs so that self is the array scalar of some type and other > is the other object > > if (other is an array scalar of the same data-type as self) > arg3 = other > else if (other is an array scalar of a different data-type as self) > arg3 = convert_other_to_self_data_type(other) --- Shouldn't this depend on whether self or other has the higher type? > <<< > else if (other is a Python scalar) > arg3 = convert_Python_scalar_to_array_scalar(other) > else if (other is a 0-d array) > arg3 = convert_other_to_self_data_type_from_0-d_array(other) > else > return (use ufunc to calculate result). > > return (operation using self and arg3) > > if an error condition is encountered, then only at that point, the > proper way to handle it will be determined by looking in the local / > global / builtin scope for the error-handling variable. > Tthis avoids the overhead of looking up what to do to the case of an > error actually occurring --- I need to change the ufuncobject.c code > to also do this and save the overhead there too --- right now what to > do is looked up every time, rather than waiting until an error is > actually detected.) > > What do people think? > > -Travis Would it make sense to reconsider the costs and benefits of dealing with dimensionless arrays? Colin W. From ndarray at mac.com Thu Mar 16 18:10:03 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 16 18:10:03 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: <441A0C95.30902@sympatico.ca> References: <441A07C9.8050101@ee.byu.edu> <441A0C95.30902@sympatico.ca> Message-ID: On 3/16/06, Colin J. Williams wrote: > Would it make sense to reconsider the costs and benefits of dealing with > dimensionless arrays? http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray http://aspn.activestate.com/ASPN/Mail/Message/2499954 http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3028210 This horse has suffered a lot beating ... From rudolphv at gmail.com Fri Mar 17 00:14:03 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Fri Mar 17 00:14:03 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux In-Reply-To: <44196EF6.9040602@mecha.uni-stuttgart.de> References: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> <44196EF6.9040602@mecha.uni-stuttgart.de> Message-ID: <97670e910603170013y527d1938q994419036cc383a8@mail.gmail.com> Nils, Thanks... that solved the problem. Much appreciated! Rudolph On 3/16/06, Nils Wagner wrote: > > Did you use > > # > # On 64 bit systems with GNU compiler > # > OPTS = -O2 -m64 -fPIC > NOOPT = -m64 -fPIC > > (two lines of make.inc) > > to compile lapack ? > > Also I have added -m64 -fPIC to some lines in the ATLAS generated makefile ( > Make.Linux_HAMMER64SSE3) > > F77 = /usr/bin/g77 > F77FLAGS = -fomit-frame-pointer -O -m64 -fPIC > FLINKER = $(F77) > FLINKFLAGS = $(F77FLAGS) > FCLINKFLAGS = $(FLINKFLAGS) > > > # --------------------------------------------------------------------------- > # Various C compilers, and the linker to be used when we are not linking in > # non-ATLAS BLAS (which usually necessitate using the Fortran linker). > # The C compilers recognized by ATLAS are: > # CC : Compiler to use to compile regular, non-generated code > # MCC : Compiler to use to compile generated, highly-optimized code > # XCC : Compiler to be used on the compile engine of a cross-compiler > # These will typically all be the same. An example of where this is not > # the case would be DEC ALPHA 21164, where you want to use gcc for MCC, > # because DEC's cc does not allow the programmer access to all 32 floating > # point registers. However, on normal C code, DEC's cc produces much faster > # code than gcc, so you CC set to cc. Of course, any system where you are > # cross-compiling, you will need to set XCC differently than CC & MCC. > # --------------------------------------------------------------------------- > CDEFS = $(L2SIZE) $(INCLUDES) $(F2CDEFS) $(ARCHDEFS) > > GOODGCC = gcc > CC = /usr/bin/gcc > CCFLAG0 = -fomit-frame-pointer -O -mfpmath=387 -m64 -fPIC > CCFLAGS = $(CDEFS) $(CCFLAG0) > MCC = /usr/bin/gcc > MMFLAGS = -fomit-frame-pointer -O -mfpmath=387 -m64 -fPIC > XCC = /usr/bin/gcc > XCCFLAGS = $(CDEFS) -fomit-frame-pointer -O -mfpmath=387 -m64 -fPIC > CLINKER = $(CC) > CLINKFLAGS = $(CCFLAGS) > BC = $(CC) > BCFLAGS = $(CCFLAGS) > ARCHIVER = ar > ARFLAGS = r > RANLIB = echo > > > Nils > > > > > -- Rudolph van der Merwe From rudolphv at gmail.com Fri Mar 17 00:15:00 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Fri Mar 17 00:15:00 2006 Subject: [Numpy-discussion] Trouble building NumPy on 64bit Ubuntu Linux In-Reply-To: <1e2af89e0603160621l1164cbfbm7fe763489a99d365@mail.gmail.com> References: <97670e910603160501y7a1bcd9ch6807ceb298ff9945@mail.gmail.com> <1e2af89e0603160621l1164cbfbm7fe763489a99d365@mail.gmail.com> Message-ID: <97670e910603170014p2a915fcdka084bbdd7f2c013c@mail.gmail.com> Matthew, I see your scripts and Makefiles are optimized for a 64 bit P4 architecture. I'm running on a 64 bit AMD Opteron system though. Thanks Rudolph On 3/16/06, Matthew Brett wrote: > Hi, > > > /usr/bin/g77 -shared > > build/temp.linux-x86_64-2.4/numpy/linalg/lapack_litemodule.o > > -L/usr/local/lib/atlas -llapack -lptf77blas -lptcblas -latlas > > -lg2c-pic -o build/lib.linux-x86_64-2.4/numpy/linalg/lapack_lite.so > > /usr/bin/ld: /usr/local/lib/atlas/liblapack.a(dlamch.o): relocation > > R_X86_64_32S can not be used when making a shared object; recompile > > with -fPIC > > I wonder if you built the lapack libraries with the -fPIC flag though? > I've attached my build scripts for atlas, the include make file for > the lapack libraries, and a python script to combine them - I am > wondering if you need the modified include make file, which should > somehow end up in your LAPACK directory as make.inc (see the > build_full.py script). > > Best, > > Matthew > > > -- Rudolph van der Merwe From arnd.baecker at web.de Fri Mar 17 00:18:04 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri Mar 17 00:18:04 2006 Subject: [Numpy-discussion] strange paths on install and some warnings Message-ID: Hi, for a current svn checkout of numpy and installing with python setup.py build python setup.py install --prefix=/tmp/NUMPY this results in an install under: $HOME/.PythonLibrary/Python2.3/ ... (("""copying build/lib.linux-i686-2.3/numpy/_import_tools.py -> /home/abaecker/.PythonLibrary/Python2.3/site-packages/numpy""")) Only f2py behaves and gets installed in /tmp/NUMPY/bin. First I thought that it has something to do with a setuptools test I did some time ago, but moving ~/.PythonLibrary to a different place before installing numpy did not help. (it was also not on the PYTHONPATH before, so it should not have made a difference anyway). Even more weirdly, everything works fine on a different machine. So it looks like some configuration issue. On the other hand I did a couple of builds before (some time ago) on this machine. Before I start looking into this in more detail, I would be interested in any ideas how the above could come about and what tests I could do. In addition to the above I get a couple of warnings/errors (see below). I am not sure about their relevance (I don't remember to have seen them before). Best, Arnd Warning: not existing path in numpy/distutils: site.cfg No module named __svn_version__ F2PY Version 2_2255 blas_opt_info: blas_mkl_info: /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=['/usr/lib/sse2/libatlas.so'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=['/usr/lib/sse2/libatlas.a'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=['/usr/lib/libatlas.so'] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_info: /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c include_dirs = ['/usr/include'] /tmp/NUMPY/numpy/numpy/distutils/system_info.py:1075: FutureWarning: hex()/oct() of negative int will return a signed string in Python 2.4 and up magic = hex(hash(repr(config))) From a.h.jaffe at gmail.com Fri Mar 17 00:31:05 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri Mar 17 00:31:05 2006 Subject: [Numpy-discussion] Re: Status of pyfits with numpy? In-Reply-To: <4419B86D.3050801@stsci.edu> References: <441995D3.4070309@gmail.com> <4419B86D.3050801@stsci.edu> Message-ID: <441A7399.3000209@gmail.com> Dear Chris, Thanks -- that's great. In a quick sprint to read the new WMAP satellite data, the table support seems to work for me at first glance. Can I ask what sort of trouble has been reported? Andrew Christopher Hanley wrote: > Andrew, > > We are in the process of preparing an "alpha" release of a > numpy-compatible version of pyfits. I have recently merged my > numpy-pyfits branch with the pyfits trunk so if you do not wish to wait > for a release you can grab it our of SVN with: > > svn co http://astropy.scipy.org/svn/pyfits/trunk pyfits > > This version of pyfits is not well tested but you are welcome to try it > out and send me bug reports. > > A couple of people have already tried an earlier version and have > reported some issues with the table support. We are currently working > to address those issues. > > If you have both numpy and numarray installed you can make pyfits switch > between them by setting an environment variable called NUMERIX. NUMERIX > = 'numarray' for numarray. NUMERIX = 'numpy' for numpy. If the > variable isn't set, and both packages are installed, pyfits will default > to the numarray version. If only one package is installed then pyfits > will use that package. > > Chris > > > > Andrew Jaffe wrote: >> Hi All, >> >> Is a numpy-compatible version of pyfits nearing stability? I see that >> there is a pyfits-numpy branch visible on the trac browser -- is that >> available and stable enough for public use? >> >> Thanks, >> >> Andrew >> >> > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 From oliphant.travis at ieee.org Fri Mar 17 00:45:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 17 00:45:05 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Computing correlations with SciPy In-Reply-To: <87wteu8ju7.fsf@peds-pc311.bsd.uchicago.edu> References: <1142524171.358429.129850@i39g2000cwa.googlegroups.com> <87wteu8ju7.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: <441A76FF.9080200@ieee.org> John Hunter wrote: > The following message is a courtesy copy of an article > that has been posted to comp.lang.python as well. > > >>>>>> "tkpmep" == tkpmep writes: >>>>>> > > tkpmep> I want to compute the correlation between two sequences X > tkpmep> and Y, and tried using SciPy to do so without success.l > tkpmep> Here's what I have, how can I correct it? > > >>>> X = [1, 2, 3, 4, 5] Y = [5, 4, 3, 2, 1] import scipy > >>>> scipy.corrcoef(X,Y) > tkpmep> Traceback (most recent call last): File " tkpmep> input>", line 1, in ? File > tkpmep> "C:\Python24\Lib\site-packages\numpy\lib\function_base.py", > tkpmep> line 671, in corrcoef d = diag(c) File > tkpmep> "C:\Python24\Lib\site-packages\numpy\lib\twodim_base.py", > tkpmep> line 80, in diag raise ValueError, "Input must be 1- or > tkpmep> 2-d." ValueError: Input must be 1- or 2-d. > >>>> > > Hmm, this may be a bug in scipy. matplotlib also defines a corrcoef > function, which you may want to use until this problem gets sorted out > > The problem is now sorted out in SVN of numpy. The problem was inherited from Numeric's MLab. I revamped the cov function (which corrcoef depended on) so it should hopefully work better than it did. Thanks for the heads up. -Travis From oliphant.travis at ieee.org Fri Mar 17 01:00:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 17 01:00:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: References: Message-ID: <441A7A5B.4010701@ieee.org> Mads Ipsen wrote: > Hey, > > First of all, thanks for the new release. > > Here's another question regarding something I cannot quite understand: > > Suppose you want to update bins for a histogram, you might think you > could do something like: > > g = zeros(4,Int) > x = array([0.2, 0.2]) > idx = floor(x/0.1).astype(int) > g[idx] += 1 > > Here idx becomes > > array([2, 2]) > > In this case, I would naively expect g to end up like > > array([0, 0, 2, 0]) (1) > > but instead you get > > array([0, 0, 1, 0]) (2) > > Is this intended? Just being plain novice-like naive, I would expect > the slice operation g[idx] += 1 to do something like > Yes, this is intended (sort of --- this particular example isn't the reason for the behavior though). The issue is that the code g[idx] +=1 is equivalent in Python to g[idx] = g[idx] + 1 Then g[idx] returns array([0,0]) as it should. This new copy of the array data then gets added to 1 resulting in array([1,1]). This array is then set into element 2 of g twice just as if you had done g[2] = 1 g[2] = 1 Then end effect is you get a 1 out of the result. Perhaps a little counter to the way you were thinking of the problem, but very much how it must be due to the way Python translates the statement g[idx] += 1 in this case. There are, of course, other ways to do what you want to do. -Travis From oliphant.travis at ieee.org Fri Mar 17 01:16:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 17 01:16:03 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <44198B7B.6040706@gmx.de> References: <44198B7B.6040706@gmx.de> Message-ID: <441A7E2A.9010903@ieee.org> Norbert Nemec wrote: > I have a very much related problem: Not only that the idea described by > Mads Ipsen does not work, but I could generally find no efficient way to > do a "counting" of elements in an array, as it is needed for a histogram. > This may be something we are lacking. It depends on what you mean by efficient, I suppose. The sorting algorithms are very fast, and the histogram routines that are scattered all over the place (including the histogram function in numpy) has been in use for a long time, and you are the first person to complain of its efficiency. That isn't to say your complaint may not be valid, it's just that for most people the speed has been sufficient. > What would instead be needed is a function that simply gives the count > of occurances of given values in a given array: > I presume you are talking of "integer" arrays, since equality-comparison of floating-point values is usually not very helpful so most histograms on floating-point values are given in terms of bins. Thus, the searchsorted function uses bins for it's "counting" operation. > >>>> [4,5,2,3,2,1,4].count([0,1,2,3,4,5]) >>>> > [0,1,2,1,1,2] > > A count function for integer arrays could certainly be written using a C-loop. But, I would first just use histogram([4,5,2,3,2,1,4], [0,1,2,3,4,5]) and make sure that's really too slow, before worrying about it too much. Also, I according to the above function, the right answer is: [0, 1, 2, 1, 2, 1] Best, -Travis From perry at stsci.edu Fri Mar 17 03:59:03 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri Mar 17 03:59:03 2006 Subject: [Numpy-discussion] Re: Status of pyfits with numpy? In-Reply-To: <441A7399.3000209@gmail.com> Message-ID: Andrew Jaffe wrote: > > Dear Chris, > > Thanks -- that's great. In a quick sprint to read the new WMAP satellite > data, the table support seems to work for me at first glance. Can I ask > what sort of trouble has been reported? > > Andrew > > It's possible to construct new tables using the Coldef mechanism that will fail because Coldef allows one to associate arrays of higher precision than the column definition allows. When this used to create a table it fails since numpy doesn't allow downcasting assignments. We are changing it so that the column constructor explicitly downcasts any arrays (and raises an exception if you try to downcast integer arrays that won't fit in the new size). If you aren't trying to construct new tables from scratch, you won't encounter this. Perry From hurak at control.felk.cvut.cz Fri Mar 17 04:08:18 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Fri Mar 17 04:08:18 2006 Subject: [Numpy-discussion] Re: How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) References: Message-ID: Robert Kern wrote: > Nadav gave you a sample site.cfg. The place to put it would be next to > NumPy's setup.py script. Thank you very much, Robert. I just wander why some default (example) site.cfg file is not present in the installation package. Sorry, I am a newcomer to Python and perhaps it is a very standard thing for all Python guys to write such files before they start their installation. Best regards from Prague, Zdenek From luszczek at cs.utk.edu Fri Mar 17 06:44:02 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 06:44:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441A7A5B.4010701@ieee.org> References: <441A7A5B.4010701@ieee.org> Message-ID: <200603170943.03095.luszczek@cs.utk.edu> On Friday 17 March 2006 03:59, Travis Oliphant wrote: > Mads Ipsen wrote: > > Hey, > > > > First of all, thanks for the new release. > > > > Here's another question regarding something I cannot quite > > understand: > > > > Suppose you want to update bins for a histogram, you might think > > you could do something like: > > > > g = zeros(4,Int) > > x = array([0.2, 0.2]) > > idx = floor(x/0.1).astype(int) > > g[idx] += 1 > > > > Here idx becomes > > > > array([2, 2]) > > > > In this case, I would naively expect g to end up like > > > > array([0, 0, 2, 0]) (1) > > > > but instead you get > > > > array([0, 0, 1, 0]) (2) > > > > Is this intended? Just being plain novice-like naive, I would > > expect the slice operation g[idx] += 1 to do something like > > Yes, this is intended (sort of --- this particular example isn't the > reason for the behavior though). > > The issue is that the code g[idx] +=1 is equivalent in Python to > > g[idx] = g[idx] + 1 This is not what I read at: http://docs.python.org/ref/numeric-types.html Quote: These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). What you describe is "lack of attempt" in which case the "fall back" behavior gets triggered. > Then g[idx] returns array([0,0]) as it should. This new copy of the > array data then gets added to 1 resulting in array([1,1]). This > array is then set into element 2 of g twice just as if you had done > > g[2] = 1 > g[2] = 1 > > Then end effect is you get a 1 out of the result. > > Perhaps a little counter to the way you were thinking of the problem, > but very much how it must be due to the way Python translates the > statement g[idx] += 1 in this case. > > There are, of course, other ways to do what you want to do. Histograms are just the trivial instantiation of the problem (the problem being unintuitive behavior at least for Mads and me). If in-place arithmetic didn't make copies it would land its nicely to sparse matrix computations. Efficient, compact and in Python. And there are also other histogram-like computations but do not use '+' but other operators. Piotr > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language that extends applications into web and mobile media. Attend > the live webcast and join the prime developer group breaking into > this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121 >642 _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From arnd.baecker at web.de Fri Mar 17 09:40:03 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri Mar 17 09:40:03 2006 Subject: [Numpy-discussion] strange paths on install and some warnings In-Reply-To: References: Message-ID: On Fri, 17 Mar 2006, Arnd Baecker wrote: > Hi, > > for a current svn checkout of numpy and installing with > python setup.py build > python setup.py install --prefix=/tmp/NUMPY > this results in an install under: > $HOME/.PythonLibrary/Python2.3/ ... > (("""copying build/lib.linux-i686-2.3/numpy/_import_tools.py -> > /home/abaecker/.PythonLibrary/Python2.3/site-packages/numpy""")) > > Only f2py behaves and gets installed in /tmp/NUMPY/bin. > > First I thought that it has something to do with a setuptools > test I did some time ago, but moving ~/.PythonLibrary > to a different place before installing numpy did not help. > (it was also not on the PYTHONPATH before, so it should not > have made a difference anyway). [...] Alright, I found the reason for this, it is related to Easy Install Stuff (still mea culpa). At some point I must have followed http://peak.telecommunity.com/DevCenter/EasyInstall#traditional-pythonpath-based-installation which lead to a file .pydistutils.cfg in my homedirectory with contents: [install] install_lib = ~/.PythonLibrary/Python$py_version_short/site-packages prefix=~/.PythonLibrary/ #[easy_install] #site-dirs=/usr/local/lib/pythonX.Y/site-packages So it seems that this wins over a ``--prefix=``. And this is ok, because: """ [...] you can use Distutils configuration files to record personal or site preferences for any Distutils options. That is, any option to any command can be stored in one of two or three (depending on your platform) configuration files, which will be consulted before the command-line is parsed. This means that configuration files will override default values, and the command-line will in turn override configuration files. """ http://docs.python.org/inst/config-syntax.html Obviously, setting up a ~/.pydistutils.cfg is a good way to cause trouble at some later time because it is easily forgotten. The errors below persist... Best, Arnd > In addition to the above I get a couple of warnings/errors (see below). > I am not sure about their relevance (I don't remember to have seen > them before). > > Best, Arnd > > > Warning: not existing path in numpy/distutils: site.cfg > No module named __svn_version__ > F2PY Version 2_2255 > blas_opt_info: > blas_mkl_info: > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['mkl', 'vml', 'guide'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > NOT AVAILABLE > > atlas_blas_threads_info: > Setting PTATLAS=ATLAS > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] > found_libs=['/usr/lib/sse2/libatlas.so'] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] > found_libs=['/usr/lib/sse2/libatlas.a'] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['ptf77blas', 'ptcblas', 'atlas'] > found_libs=['/usr/lib/libatlas.so'] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > NOT AVAILABLE > > atlas_blas_info: > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:531: UserWarning: Library > error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > FOUND: > libraries = ['f77blas', 'cblas', 'atlas'] > library_dirs = ['/usr/lib/sse2'] > language = c > include_dirs = ['/usr/include'] > > /tmp/NUMPY/numpy/numpy/distutils/system_info.py:1075: FutureWarning: > hex()/oct() of negative int will return a signed string in Python 2.4 and > up > magic = hex(hash(repr(config))) From robert.kern at gmail.com Fri Mar 17 09:46:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 17 09:46:03 2006 Subject: [Numpy-discussion] Re: How compile Numpy on Gentoo system (finding atlas-blas and atlas-lapack) In-Reply-To: References: Message-ID: Zden?k Hur?k wrote: > Robert Kern wrote: > >>Nadav gave you a sample site.cfg. The place to put it would be next to >>NumPy's setup.py script. > > Thank you very much, Robert. > > I just wander why some default (example) site.cfg file is not present in the > installation package. Sorry, I am a newcomer to Python and perhaps it is a > very standard thing for all Python guys to write such files before they > start their installation. There used to be a sample_site.cfg when numpy.distutils was scipy_distutils. It got left behind in the transition probably by accident. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Fri Mar 17 10:30:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 17 10:30:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <200603170943.03095.luszczek@cs.utk.edu> References: <441A7A5B.4010701@ieee.org> <200603170943.03095.luszczek@cs.utk.edu> Message-ID: <441AFFEE.1080508@ee.byu.edu> Piotr Luszczek wrote: >>Yes, this is intended (sort of --- this particular example isn't the >>reason for the behavior though). >> >>The issue is that the code g[idx] +=1 is equivalent in Python to >> >>g[idx] = g[idx] + 1 >> >> > >This is not what I read at: > >http://docs.python.org/ref/numeric-types.html > >Quote: > >These methods should attempt to do the operation in-place (modifying >self) and return the result (which could be, but does not have to be, >self). > >What you describe is "lack of attempt" in which case the "fall back" >behavior gets triggered. > > The problems is that this explanation is very clear when we are talking about the syntax g += 1 Then, there is a method of g that can be over-ridden to get the desired behavior. Now, what you are talking about is "indexing followed by in-place addition". i.e. at issue is how does python interpret g[idx] += 1 How does this get compiled to byte-code? There are two possibilities: 1) g[idx] creates a new object which then has 1 added to it using in-place addition. This would not produce the desired behavior as g[idx] is a copy of the data when idx is a general indexing array as it is in this case. So, you make a copy of those indices, add 1 to them and then do what with the resut? 2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 This appears to be effectively what Python actually does. Notice that there is no way for us to control this behavior because there is no __inplace_with_indexing_add__ operator to over-ride. There is no such single operation to over-ride for the object. In other words, I don't see anyay for us to even alter the object to get the behavior you want from that syntax. We can, of course, add a function or method to do that, but I we would have to extend Python to get the behavior you want here. Note, however, if idx is slice syntax, then the operation is done without making copies because, for example, g[1:10:2] returns a "view" of the data (an array that shares the same memory) as the original array. Thus, g[1:10:2] += 1 does an inplace add without data-copying. It may be possible to do what you want using zero-strided arrays, but I'll leave that for a more motivated contributor. -Travis From luszczek at cs.utk.edu Fri Mar 17 11:26:03 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 11:26:03 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441AFFEE.1080508@ee.byu.edu> References: <200603170943.03095.luszczek@cs.utk.edu> <441AFFEE.1080508@ee.byu.edu> Message-ID: <200603171424.59034.luszczek@cs.utk.edu> On Friday 17 March 2006 13:29, Travis Oliphant wrote: > Piotr Luszczek wrote: > >>Yes, this is intended (sort of --- this particular example isn't > >> the reason for the behavior though). > >> > >>The issue is that the code g[idx] +=1 is equivalent in Python to > >> > >>g[idx] = g[idx] + 1 > > > >This is not what I read at: > > > >http://docs.python.org/ref/numeric-types.html > > > >Quote: > > > >These methods should attempt to do the operation in-place (modifying > >self) and return the result (which could be, but does not have to > > be, self). > > > >What you describe is "lack of attempt" in which case the "fall back" > >behavior gets triggered. > > The problems is that this explanation is very clear when we are > talking about the syntax > > g += 1 > > Then, there is a method of g that can be over-ridden to get the > desired behavior. Now, what you are talking about is "indexing > followed by in-place addition". > > i.e. at issue is > > how does python interpret > > g[idx] += 1 > > How does this get compiled to byte-code? > > There are two possibilities: > > 1) g[idx] creates a new object which then has 1 added to it using > in-place addition. > > This would not produce the desired behavior as g[idx] is a copy > of the data when idx is a > general indexing array as it is in this case. So, you make a > copy of those indices, add 1 to them > and then do what with the resut? > > 2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > > This appears to be effectively what Python actually does. Notice > that there is no way for us to control this behavior because there is > no __inplace_with_indexing_add__ operator to over-ride. > > There is no such single operation to over-ride for the object. In > other words, I don't see anyay for us to even alter the object to get > the behavior you want from that syntax. We can, of course, add a > function or method to do that, but I we would have to extend Python > to get the behavior you want here. Hardly. At least from what I'm seeing happens on a small example. 'g[idx] += 1' becomes ('g' and 'idx' are generic objects): __getitem__(self, idx) __iadd__(1) __setitem__(result of __iadd__) By design numpy returns views from __getitem__ In this case, it would be view into 'self' and 'idx' so the __iadd__ would just use the 'idx' directly rather than a copy. Finally, __setitem__ doesn't do anything since 'self' and 'value' will be the same. Of course, this is just a quick draft. I don't know how it would work in practice and in other cases. > Note, however, if idx is slice syntax, then the operation is done > without making copies because, for example, > > g[1:10:2] > > returns a "view" of the data (an array that shares the same memory) > as the original array. Thus, > > g[1:10:2] += 1 > > does an inplace add without data-copying. > > It may be possible to do what you want using zero-strided arrays, but > I'll leave that for a more motivated contributor. > > -Travis From oliphant at ee.byu.edu Fri Mar 17 11:38:12 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 17 11:38:12 2006 Subject: [Numpy-discussion] Call for posters to gmane.comp.python.devel newsgroup Message-ID: <441B0FF8.3000606@ee.byu.edu> I'm trying to start discussions on python dev about getting a simple object into Python that at least exposes the array interface and/or has the basic C-structure of NumPy arrays. Please voice your support and comments on the newsgroup. The more people that respond, the more the python developers will see that it's not just my lonely voice asking for things to change. Perhaps it will help somebody with more time to get a PEP written up. I doubt we will make it into Python 2.5, unless somebody steps up in the next month, but it will help for Python 2.6 Thanks, -Travis From tim.hochberg at cox.net Fri Mar 17 11:44:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 11:44:04 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <200603171424.59034.luszczek@cs.utk.edu> References: <200603170943.03095.luszczek@cs.utk.edu> <441AFFEE.1080508@ee.byu.edu> <200603171424.59034.luszczek@cs.utk.edu> Message-ID: <441B114B.8010700@cox.net> Piotr Luszczek wrote: >On Friday 17 March 2006 13:29, Travis Oliphant wrote: > > >>Piotr Luszczek wrote: >> >> >>>>Yes, this is intended (sort of --- this particular example isn't >>>>the reason for the behavior though). >>>> >>>>The issue is that the code g[idx] +=1 is equivalent in Python to >>>> >>>>g[idx] = g[idx] + 1 >>>> >>>> >>>This is not what I read at: >>> >>>http://docs.python.org/ref/numeric-types.html >>> >>>Quote: >>> >>>These methods should attempt to do the operation in-place (modifying >>>self) and return the result (which could be, but does not have to >>>be, self). >>> >>>What you describe is "lack of attempt" in which case the "fall back" >>>behavior gets triggered. >>> >>> >>The problems is that this explanation is very clear when we are >>talking about the syntax >> >>g += 1 >> >>Then, there is a method of g that can be over-ridden to get the >>desired behavior. Now, what you are talking about is "indexing >>followed by in-place addition". >> >>i.e. at issue is >> >>how does python interpret >> >>g[idx] += 1 >> >>How does this get compiled to byte-code? >> >>There are two possibilities: >> >>1) g[idx] creates a new object which then has 1 added to it using >>in-place addition. >> >> This would not produce the desired behavior as g[idx] is a copy >>of the data when idx is a >> general indexing array as it is in this case. So, you make a >>copy of those indices, add 1 to them >> and then do what with the resut? >> >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 >> >> This appears to be effectively what Python actually does. Notice >>that there is no way for us to control this behavior because there is >>no __inplace_with_indexing_add__ operator to over-ride. >> >>There is no such single operation to over-ride for the object. In >>other words, I don't see anyay for us to even alter the object to get >>the behavior you want from that syntax. We can, of course, add a >>function or method to do that, but I we would have to extend Python >>to get the behavior you want here. >> >> > >Hardly. At least from what I'm seeing happens on a small example. >'g[idx] += 1' becomes ('g' and 'idx' are generic objects): >__getitem__(self, idx) >__iadd__(1) >__setitem__(result of __iadd__) > >By design numpy returns views from __getitem__ > > You are missing that idx is not a slice index. rather it is an index array (or list). In this case g[idx] does *not* return a view, it returns a copy. From this everything else follows. Conceivably g[idx] could return a psuedo-array object like flat does, but I suspect that might have bad performance characteristics. -tim >In this case, it would be view into 'self' and 'idx' so the __iadd__ >would just use the 'idx' directly rather than a copy. >Finally, __setitem__ doesn't do anything since 'self' and 'value' >will be the same. > >Of course, this is just a quick draft. I don't know how it would work >in practice and in other cases. > > > >>Note, however, if idx is slice syntax, then the operation is done >>without making copies because, for example, >> >>g[1:10:2] >> >>returns a "view" of the data (an array that shares the same memory) >>as the original array. Thus, >> >>g[1:10:2] += 1 >> >>does an inplace add without data-copying. >> >>It may be possible to do what you want using zero-strided arrays, but >>I'll leave that for a more motivated contributor. >> >>-Travis >> >> > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From robert.kern at gmail.com Fri Mar 17 11:59:05 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 17 11:59:05 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <200603171424.59034.luszczek@cs.utk.edu> References: <200603170943.03095.luszczek@cs.utk.edu> <441AFFEE.1080508@ee.byu.edu> <200603171424.59034.luszczek@cs.utk.edu> Message-ID: Piotr Luszczek wrote: > On Friday 17 March 2006 13:29, Travis Oliphant wrote: >>how does python interpret >> >>g[idx] += 1 >> >>How does this get compiled to byte-code? In [161]: c = compile('g[idx] += 1', '', 'single') In [162]: import dis In [163]: dis.dis(c) 1 0 LOAD_NAME 0 (g) 3 LOAD_NAME 1 (idx) 6 DUP_TOPX 2 9 BINARY_SUBSCR 10 LOAD_CONST 0 (1) 13 INPLACE_ADD 14 ROT_THREE 15 STORE_SUBSCR 16 LOAD_CONST 1 (None) 19 RETURN_VALUE >>There are two possibilities: >> >>1) g[idx] creates a new object which then has 1 added to it using >>in-place addition. >> >> This would not produce the desired behavior as g[idx] is a copy >>of the data when idx is a >> general indexing array as it is in this case. So, you make a >>copy of those indices, add 1 to them >> and then do what with the resut? >> >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 >> >> This appears to be effectively what Python actually does. Notice >>that there is no way for us to control this behavior because there is >>no __inplace_with_indexing_add__ operator to over-ride. >> >>There is no such single operation to over-ride for the object. In >>other words, I don't see anyay for us to even alter the object to get >>the behavior you want from that syntax. We can, of course, add a >>function or method to do that, but I we would have to extend Python >>to get the behavior you want here. > > Hardly. At least from what I'm seeing happens on a small example. > 'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > __getitem__(self, idx) > __iadd__(1) > __setitem__(result of __iadd__) > > By design numpy returns views from __getitem__ Only for slices. In [132]: a = arange(10) In [133]: idx = [2,2,3] In [134]: a[idx] Out[134]: array([2, 2, 3]) In [135]: b = a[idx] In [136]: b[-1] = 100 In [137]: a Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > In this case, it would be view into 'self' and 'idx' so the __iadd__ > would just use the 'idx' directly rather than a copy. > Finally, __setitem__ doesn't do anything since 'self' and 'value' > will be the same. No, value is the result of __iadd__ on the temporary array. 'g[idx] += 1' expands to: tmp = g.__getitem__(idx) val = tmp.__iadd__(1) g.__setitem__(idx, val) Given these class definitions: class A(object): def __getitem__(self, idx): print 'A.__getitem__(%r)' % idx return B() def __setitem__(self, idx, value): print 'A.__setitem__(%r, %r)' % (idx, value) class B(object): def __iadd__(self, x): print 'B.__iadd__(%r)' % x return self def __repr__(self): return 'B()' In [153]: a = A() In [154]: a[[0, 2, 2, 1]] += 1 A.__getitem__([0, 2, 2, 1]) B.__iadd__(1) A.__setitem__([0, 2, 2, 1], B()) > Of course, this is just a quick draft. I don't know how it would work > in practice and in other cases. Aye, there's the rub. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From luszczek at cs.utk.edu Fri Mar 17 12:14:03 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 12:14:03 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441B114B.8010700@cox.net> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> Message-ID: <200603171512.44309.luszczek@cs.utk.edu> On Friday 17 March 2006 14:43, Tim Hochberg wrote: > Piotr Luszczek wrote: > >On Friday 17 March 2006 13:29, Travis Oliphant wrote: > >>Piotr Luszczek wrote: > >>>>Yes, this is intended (sort of --- this particular example isn't > >>>>the reason for the behavior though). > >>>> > >>>>The issue is that the code g[idx] +=1 is equivalent in Python to > >>>> > >>>>g[idx] = g[idx] + 1 > >>> > >>>This is not what I read at: > >>> > >>>http://docs.python.org/ref/numeric-types.html > >>> > >>>Quote: > >>> > >>>These methods should attempt to do the operation in-place > >>> (modifying self) and return the result (which could be, but does > >>> not have to be, self). > >>> > >>>What you describe is "lack of attempt" in which case the "fall > >>> back" behavior gets triggered. > >> > >>The problems is that this explanation is very clear when we are > >>talking about the syntax > >> > >>g += 1 > >> > >>Then, there is a method of g that can be over-ridden to get the > >>desired behavior. Now, what you are talking about is "indexing > >>followed by in-place addition". > >> > >>i.e. at issue is > >> > >>how does python interpret > >> > >>g[idx] += 1 > >> > >>How does this get compiled to byte-code? > >> > >>There are two possibilities: > >> > >>1) g[idx] creates a new object which then has 1 added to it using > >>in-place addition. > >> > >> This would not produce the desired behavior as g[idx] is a > >> copy of the data when idx is a > >> general indexing array as it is in this case. So, you make a > >>copy of those indices, add 1 to them > >> and then do what with the resut? > >> > >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > >> > >> This appears to be effectively what Python actually does. > >> Notice that there is no way for us to control this behavior > >> because there is no __inplace_with_indexing_add__ operator to > >> over-ride. > >> > >>There is no such single operation to over-ride for the object. In > >>other words, I don't see anyay for us to even alter the object to > >> get the behavior you want from that syntax. We can, of course, > >> add a function or method to do that, but I we would have to extend > >> Python to get the behavior you want here. > > > >Hardly. At least from what I'm seeing happens on a small example. > >'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > >__getitem__(self, idx) > >__iadd__(1) > >__setitem__(result of __iadd__) > > > >By design numpy returns views from __getitem__ > > You are missing that idx is not a slice index. rather it is an index > array (or list). In this case g[idx] does *not* return a view, it > returns a copy. From this everything else follows. You are missing that indexing with array doesn't have to make a copy. From this everythin else follows. In other words, It's a design decision that slices give views and indirection with arrays (I don't care for lists) gives a copy. I question things on conceptual level: both objects (the array an the indexing array) are from numpy. What code would break if the operation didn't return a copy? > Conceivably g[idx] could return a psuedo-array object like flat does, > but I suspect that might have bad performance characteristics. > > -tim > > >In this case, it would be view into 'self' and 'idx' so the __iadd__ > >would just use the 'idx' directly rather than a copy. > >Finally, __setitem__ doesn't do anything since 'self' and 'value' > >will be the same. > > > >Of course, this is just a quick draft. I don't know how it would > > work in practice and in other cases. > > > >>Note, however, if idx is slice syntax, then the operation is done > >>without making copies because, for example, > >> > >>g[1:10:2] > >> > >>returns a "view" of the data (an array that shares the same memory) > >>as the original array. Thus, > >> > >>g[1:10:2] += 1 > >> > >>does an inplace add without data-copying. > >> > >>It may be possible to do what you want using zero-strided arrays, > >> but I'll leave that for a more motivated contributor. > >> > >>-Travis > > > >------------------------------------------------------- > >This SF.Net email is sponsored by xPML, a groundbreaking scripting > > language that extends applications into web and mobile media. > > Attend the live webcast and join the prime developer group breaking > > into this new coding territory! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=1 > >21642 _______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion at lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language that extends applications into web and mobile media. Attend > the live webcast and join the prime developer group breaking into > this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121 >642 _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From luszczek at cs.utk.edu Fri Mar 17 12:21:01 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 12:21:01 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: References: <200603171424.59034.luszczek@cs.utk.edu> Message-ID: <200603171519.35502.luszczek@cs.utk.edu> On Friday 17 March 2006 14:58, Robert Kern wrote: > Piotr Luszczek wrote: > > On Friday 17 March 2006 13:29, Travis Oliphant wrote: > >>how does python interpret > >> > >>g[idx] += 1 > >> > >>How does this get compiled to byte-code? > > In [161]: c = compile('g[idx] += 1', '', 'single') > > In [162]: import dis > > In [163]: dis.dis(c) > 1 0 LOAD_NAME 0 (g) > 3 LOAD_NAME 1 (idx) > 6 DUP_TOPX 2 > 9 BINARY_SUBSCR > 10 LOAD_CONST 0 (1) > 13 INPLACE_ADD > 14 ROT_THREE > 15 STORE_SUBSCR > 16 LOAD_CONST 1 (None) > 19 RETURN_VALUE This proves my point. > >>There are two possibilities: > >> > >>1) g[idx] creates a new object which then has 1 added to it using > >>in-place addition. > >> > >> This would not produce the desired behavior as g[idx] is a > >> copy of the data when idx is a > >> general indexing array as it is in this case. So, you make a > >>copy of those indices, add 1 to them > >> and then do what with the resut? > >> > >>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > >> > >> This appears to be effectively what Python actually does. > >> Notice that there is no way for us to control this behavior > >> because there is no __inplace_with_indexing_add__ operator to > >> over-ride. > >> > >>There is no such single operation to over-ride for the object. In > >>other words, I don't see anyay for us to even alter the object to > >> get the behavior you want from that syntax. We can, of course, > >> add a function or method to do that, but I we would have to extend > >> Python to get the behavior you want here. > > > > Hardly. At least from what I'm seeing happens on a small example. > > 'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > > __getitem__(self, idx) > > __iadd__(1) > > __setitem__(result of __iadd__) > > > > By design numpy returns views from __getitem__ > > Only for slices. > > In [132]: a = arange(10) > > In [133]: idx = [2,2,3] > > In [134]: a[idx] > Out[134]: array([2, 2, 3]) > > In [135]: b = a[idx] > > In [136]: b[-1] = 100 > > In [137]: a > Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) Your example uses lists as indices. This is not interesting. I'm talking solely about arrays indexing other arrays. To me it is a special and very important case. > > In this case, it would be view into 'self' and 'idx' so the > > __iadd__ would just use the 'idx' directly rather than a copy. > > Finally, __setitem__ doesn't do anything since 'self' and 'value' > > will be the same. > > No, value is the result of __iadd__ on the temporary array. > > 'g[idx] += 1' expands to: > > tmp = g.__getitem__(idx) > val = tmp.__iadd__(1) > g.__setitem__(idx, val) You're missing the point. 'tmp' can be of a very specific type so that 'g.__setitem__' doesn't have to do anything: the 'add 1' was done by '__iadd__'. > Given these class definitions: > > class A(object): > def __getitem__(self, idx): > print 'A.__getitem__(%r)' % idx > return B() > def __setitem__(self, idx, value): > print 'A.__setitem__(%r, %r)' % (idx, value) > > > class B(object): > def __iadd__(self, x): > print 'B.__iadd__(%r)' % x > return self > def __repr__(self): > return 'B()' > > In [153]: a = A() > > In [154]: a[[0, 2, 2, 1]] += 1 > A.__getitem__([0, 2, 2, 1]) > B.__iadd__(1) > A.__setitem__([0, 2, 2, 1], B()) > > > Of course, this is just a quick draft. I don't know how it would > > work in practice and in other cases. > > Aye, there's the rub. Show me a code that breaks. From robert.kern at gmail.com Fri Mar 17 13:06:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Fri Mar 17 13:06:01 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <200603171519.35502.luszczek@cs.utk.edu> References: <200603171424.59034.luszczek@cs.utk.edu> <200603171519.35502.luszczek@cs.utk.edu> Message-ID: Piotr Luszczek wrote: > On Friday 17 March 2006 14:58, Robert Kern wrote: > >>Piotr Luszczek wrote: >>>By design numpy returns views from __getitem__ >> >>Only for slices. >> >>In [132]: a = arange(10) >> >>In [133]: idx = [2,2,3] >> >>In [134]: a[idx] >>Out[134]: array([2, 2, 3]) >> >>In [135]: b = a[idx] >> >>In [136]: b[-1] = 100 >> >>In [137]: a >>Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > Your example uses lists as indices. This is not interesting. > I'm talking solely about arrays indexing other arrays. > To me it is a special and very important case. The result is exactly the same. In [164]: a = arange(10) In [165]: idx = array([2,2,3]) In [166]: b = a[idx] In [167]: b[-1] = 100 In [168]: a Out[168]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>In this case, it would be view into 'self' and 'idx' so the >>>__iadd__ would just use the 'idx' directly rather than a copy. >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' >>>will be the same. >> >>No, value is the result of __iadd__ on the temporary array. >> >>'g[idx] += 1' expands to: >> >> tmp = g.__getitem__(idx) >> val = tmp.__iadd__(1) >> g.__setitem__(idx, val) > > You're missing the point. 'tmp' can be of a very specific type > so that 'g.__setitem__' doesn't have to do anything: the 'add 1' > was done by '__iadd__'. No, I got your point just fine; I was correcting a detail. You would have to reimplement __getitem__ to return a new kind of object that represents a non-uniformly-strided array. If you want to get anywhere, go implement that object and come back. When we have something concrete to look at instead of vague assertions, then we can start tackling the issues of integrating it into the core such that 'g[idx] += 1' works like you want it to. For example, index arrays are used in more places than in-place addition. Your new type needs to be usable in all of those places since __getitem__, __iadd__ and __setitem__ don't know that they are being called in that order and in that fashion. >>Given these class definitions: >> >> class A(object): >> def __getitem__(self, idx): >> print 'A.__getitem__(%r)' % idx >> return B() >> def __setitem__(self, idx, value): >> print 'A.__setitem__(%r, %r)' % (idx, value) >> >> >> class B(object): >> def __iadd__(self, x): >> print 'B.__iadd__(%r)' % x >> return self >> def __repr__(self): >> return 'B()' >> >>In [153]: a = A() >> >>In [154]: a[[0, 2, 2, 1]] += 1 >>A.__getitem__([0, 2, 2, 1]) >>B.__iadd__(1) >>A.__setitem__([0, 2, 2, 1], B()) >> >>>Of course, this is just a quick draft. I don't know how it would >>>work in practice and in other cases. >> >>Aye, there's the rub. > > Show me a code that breaks. Show us some code that works. I'm not interested in implementing your feature request. You are. There's plenty of work that you can do that doesn't depend on anyone else agreeing with you, so you can stop arguing and start coding. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From luszczek at cs.utk.edu Fri Mar 17 13:32:02 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 13:32:02 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: References: <200603171519.35502.luszczek@cs.utk.edu> Message-ID: <200603171630.27075.luszczek@cs.utk.edu> On Friday 17 March 2006 16:04, Robert Kern wrote: > Piotr Luszczek wrote: > > On Friday 17 March 2006 14:58, Robert Kern wrote: > >>Piotr Luszczek wrote: > >>>By design numpy returns views from __getitem__ > >> > >>Only for slices. > >> > >>In [132]: a = arange(10) > >> > >>In [133]: idx = [2,2,3] > >> > >>In [134]: a[idx] > >>Out[134]: array([2, 2, 3]) > >> > >>In [135]: b = a[idx] > >> > >>In [136]: b[-1] = 100 > >> > >>In [137]: a > >>Out[137]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > > > Your example uses lists as indices. This is not interesting. > > I'm talking solely about arrays indexing other arrays. > > To me it is a special and very important case. > > The result is exactly the same. > > In [164]: a = arange(10) > > In [165]: idx = array([2,2,3]) > > In [166]: b = a[idx] > > In [167]: b[-1] = 100 > > In [168]: a > Out[168]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > >>>In this case, it would be view into 'self' and 'idx' so the > >>>__iadd__ would just use the 'idx' directly rather than a copy. > >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' > >>>will be the same. > >> > >>No, value is the result of __iadd__ on the temporary array. > >> > >>'g[idx] += 1' expands to: > >> > >> tmp = g.__getitem__(idx) > >> val = tmp.__iadd__(1) > >> g.__setitem__(idx, val) > > > > You're missing the point. 'tmp' can be of a very specific type > > so that 'g.__setitem__' doesn't have to do anything: the 'add 1' > > was done by '__iadd__'. > > No, I got your point just fine; I was correcting a detail. > > You would have to reimplement __getitem__ to return a new kind of > object that represents a non-uniformly-strided array. If you want to > get anywhere, go implement that object and come back. When we have > something concrete to look at instead of vague assertions, then we > can start tackling the issues of integrating it into the core such > that 'g[idx] += 1' works like you want it to. For example, index > arrays are used in more places than in-place addition. Your new type > needs to be usable in all of those places since __getitem__, __iadd__ > and __setitem__ don't know that they are being called in that order > and in that fashion. This is a tough requirement but perfectly reasonable. So when my day job let's me off the hook I'll give it a try. > >>Given these class definitions: > >> > >> class A(object): > >> def __getitem__(self, idx): > >> print 'A.__getitem__(%r)' % idx > >> return B() > >> def __setitem__(self, idx, value): > >> print 'A.__setitem__(%r, %r)' % (idx, value) > >> > >> > >> class B(object): > >> def __iadd__(self, x): > >> print 'B.__iadd__(%r)' % x > >> return self > >> def __repr__(self): > >> return 'B()' > >> > >>In [153]: a = A() > >> > >>In [154]: a[[0, 2, 2, 1]] += 1 > >>A.__getitem__([0, 2, 2, 1]) > >>B.__iadd__(1) > >>A.__setitem__([0, 2, 2, 1], B()) > >> > >>>Of course, this is just a quick draft. I don't know how it would > >>>work in practice and in other cases. > >> > >>Aye, there's the rub. > > > > Show me a code that breaks. > > Show us some code that works. I'm not interested in > implementing your feature request. You are. There's plenty of work > that you can do that doesn't depend on anyone else agreeing with you, > so you can stop arguing and start coding. Arguing is good to a point and I think you're right that it's time to stop. From tim.hochberg at cox.net Fri Mar 17 13:50:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 13:50:02 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <200603171512.44309.luszczek@cs.utk.edu> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> Message-ID: <441B2EF7.10006@cox.net> Piotr Luszczek wrote: >On Friday 17 March 2006 14:43, Tim Hochberg wrote: > > >>Piotr Luszczek wrote: >> >> >>>On Friday 17 March 2006 13:29, Travis Oliphant wrote: >>> >>> >>>>Piotr Luszczek wrote: >>>> >>>> >>>>>>Yes, this is intended (sort of --- this particular example isn't >>>>>>the reason for the behavior though). >>>>>> >>>>>>The issue is that the code g[idx] +=1 is equivalent in Python to >>>>>> >>>>>>g[idx] = g[idx] + 1 >>>>>> >>>>>> >>>>>This is not what I read at: >>>>> >>>>>http://docs.python.org/ref/numeric-types.html >>>>> >>>>>Quote: >>>>> >>>>>These methods should attempt to do the operation in-place >>>>>(modifying self) and return the result (which could be, but does >>>>>not have to be, self). >>>>> >>>>>What you describe is "lack of attempt" in which case the "fall >>>>>back" behavior gets triggered. >>>>> >>>>> >>>>The problems is that this explanation is very clear when we are >>>>talking about the syntax >>>> >>>>g += 1 >>>> >>>>Then, there is a method of g that can be over-ridden to get the >>>>desired behavior. Now, what you are talking about is "indexing >>>>followed by in-place addition". >>>> >>>>i.e. at issue is >>>> >>>>how does python interpret >>>> >>>>g[idx] += 1 >>>> >>>>How does this get compiled to byte-code? >>>> >>>>There are two possibilities: >>>> >>>>1) g[idx] creates a new object which then has 1 added to it using >>>>in-place addition. >>>> >>>> This would not produce the desired behavior as g[idx] is a >>>>copy of the data when idx is a >>>> general indexing array as it is in this case. So, you make a >>>>copy of those indices, add 1 to them >>>> and then do what with the resut? >>>> >>>>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 >>>> >>>> This appears to be effectively what Python actually does. >>>>Notice that there is no way for us to control this behavior >>>>because there is no __inplace_with_indexing_add__ operator to >>>>over-ride. >>>> >>>>There is no such single operation to over-ride for the object. In >>>>other words, I don't see anyay for us to even alter the object to >>>>get the behavior you want from that syntax. We can, of course, >>>>add a function or method to do that, but I we would have to extend >>>>Python to get the behavior you want here. >>>> >>>> >>>Hardly. At least from what I'm seeing happens on a small example. >>>'g[idx] += 1' becomes ('g' and 'idx' are generic objects): >>>__getitem__(self, idx) >>>__iadd__(1) >>>__setitem__(result of __iadd__) >>> >>>By design numpy returns views from __getitem__ >>> >>> >>You are missing that idx is not a slice index. rather it is an index >>array (or list). In this case g[idx] does *not* return a view, it >>returns a copy. From this everything else follows. >> >> > >You are missing that indexing with array doesn't have to make >a copy. > No I'm not. > From this everythin else follows. >In other words, It's a design decision that slices give views and >indirection with arrays (I don't care for lists) gives a copy. >I question things on conceptual level: both objects (the array an >the indexing array) are from numpy. What code would break >if the operation didn't return a copy? > > > >>Conceivably g[idx] could return a psuedo-array object like flat does, >>but I suspect that might have bad performance characteristics. >> >> As I said conceivably it could be done without making a copy. However that would have performance implications; some good and some bad. It's almost certain that some code would break too, although I don't think that's much of a concern if someone got this in before numpy 1.0. We'll probably never know how well or ill this performs unless someone steps up and trys to implement this. I'm not interested. Are you? -tim >>-tim >> >> >> >>>In this case, it would be view into 'self' and 'idx' so the __iadd__ >>>would just use the 'idx' directly rather than a copy. >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' >>>will be the same. >>> >>>Of course, this is just a quick draft. I don't know how it would >>>work in practice and in other cases. >>> >>> >>> >>>>Note, however, if idx is slice syntax, then the operation is done >>>>without making copies because, for example, >>>> >>>>g[1:10:2] >>>> >>>>returns a "view" of the data (an array that shares the same memory) >>>>as the original array. Thus, >>>> >>>>g[1:10:2] += 1 >>>> >>>>does an inplace add without data-copying. >>>> >>>>It may be possible to do what you want using zero-strided arrays, >>>>but I'll leave that for a more motivated contributor. >>>> >>>>-Travis >>>> >>>> >>>------------------------------------------------------- >>>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>>language that extends applications into web and mobile media. >>>Attend the live webcast and join the prime developer group breaking >>>into this new coding territory! >>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=1 >>>21642 _______________________________________________ >>>Numpy-discussion mailing list >>>Numpy-discussion at lists.sourceforge.net >>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>> >>> >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>language that extends applications into web and mobile media. Attend >>the live webcast and join the prime developer group breaking into >>this new coding territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121 >>642 _______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> > > > > From luszczek at cs.utk.edu Fri Mar 17 14:20:01 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Mar 17 14:20:01 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441B2EF7.10006@cox.net> References: <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> Message-ID: <200603171718.20109.luszczek@cs.utk.edu> On Friday 17 March 2006 16:49, Tim Hochberg wrote: > Piotr Luszczek wrote: > >On Friday 17 March 2006 14:43, Tim Hochberg wrote: > >>Piotr Luszczek wrote: > >>>On Friday 17 March 2006 13:29, Travis Oliphant wrote: > >>>>Piotr Luszczek wrote: > >>>>>>Yes, this is intended (sort of --- this particular example > >>>>>> isn't the reason for the behavior though). > >>>>>> > >>>>>>The issue is that the code g[idx] +=1 is equivalent in Python > >>>>>> to > >>>>>> > >>>>>>g[idx] = g[idx] + 1 > >>>>> > >>>>>This is not what I read at: > >>>>> > >>>>>http://docs.python.org/ref/numeric-types.html > >>>>> > >>>>>Quote: > >>>>> > >>>>>These methods should attempt to do the operation in-place > >>>>>(modifying self) and return the result (which could be, but does > >>>>>not have to be, self). > >>>>> > >>>>>What you describe is "lack of attempt" in which case the "fall > >>>>>back" behavior gets triggered. > >>>> > >>>>The problems is that this explanation is very clear when we are > >>>>talking about the syntax > >>>> > >>>>g += 1 > >>>> > >>>>Then, there is a method of g that can be over-ridden to get the > >>>>desired behavior. Now, what you are talking about is "indexing > >>>>followed by in-place addition". > >>>> > >>>>i.e. at issue is > >>>> > >>>>how does python interpret > >>>> > >>>>g[idx] += 1 > >>>> > >>>>How does this get compiled to byte-code? > >>>> > >>>>There are two possibilities: > >>>> > >>>>1) g[idx] creates a new object which then has 1 added to it using > >>>>in-place addition. > >>>> > >>>> This would not produce the desired behavior as g[idx] is a > >>>>copy of the data when idx is a > >>>> general indexing array as it is in this case. So, you make > >>>> a copy of those indices, add 1 to them > >>>> and then do what with the resut? > >>>> > >>>>2) g[idx] += 1 gets converted to g[idx] = g[idx] + 1 > >>>> > >>>> This appears to be effectively what Python actually does. > >>>>Notice that there is no way for us to control this behavior > >>>>because there is no __inplace_with_indexing_add__ operator to > >>>>over-ride. > >>>> > >>>>There is no such single operation to over-ride for the object. > >>>> In other words, I don't see anyay for us to even alter the > >>>> object to get the behavior you want from that syntax. We can, > >>>> of course, add a function or method to do that, but I we would > >>>> have to extend Python to get the behavior you want here. > >>> > >>>Hardly. At least from what I'm seeing happens on a small example. > >>>'g[idx] += 1' becomes ('g' and 'idx' are generic objects): > >>>__getitem__(self, idx) > >>>__iadd__(1) > >>>__setitem__(result of __iadd__) > >>> > >>>By design numpy returns views from __getitem__ > >> > >>You are missing that idx is not a slice index. rather it is an > >> index array (or list). In this case g[idx] does *not* return a > >> view, it returns a copy. From this everything else follows. > > > >You are missing that indexing with array doesn't have to make > >a copy. > > No I'm not. > > > From this everythin else follows. > >In other words, It's a design decision that slices give views and > >indirection with arrays (I don't care for lists) gives a copy. > >I question things on conceptual level: both objects (the array an > >the indexing array) are from numpy. What code would break > >if the operation didn't return a copy? > > > >>Conceivably g[idx] could return a psuedo-array object like flat > >> does, but I suspect that might have bad performance > >> characteristics. > > As I said conceivably it could be done without making a copy. However > that would have performance implications; some good and some bad. > It's almost certain that some code would break too, although I don't > think that's much of a concern if someone got this in before numpy > 1.0. We'll probably never know how well or ill this performs unless > someone steps up and trys to implement this. I'm not interested. Are > you? Yes. Very much so. To me it's a very compact and intuitive way of encoding many things I do. Histogram is trivial but useful. Sparse matrix calculations are more important to me. I've raised this issue at least once before and didn't get much response. I think it's better this time around. But ultimately I agree with Robert Kern that I need to show some code and timing data to make people pay attention. > -tim > > >>-tim > >> > >>>In this case, it would be view into 'self' and 'idx' so the > >>> __iadd__ would just use the 'idx' directly rather than a copy. > >>>Finally, __setitem__ doesn't do anything since 'self' and 'value' > >>>will be the same. > >>> > >>>Of course, this is just a quick draft. I don't know how it would > >>>work in practice and in other cases. > >>> > >>>>Note, however, if idx is slice syntax, then the operation is done > >>>>without making copies because, for example, > >>>> > >>>>g[1:10:2] > >>>> > >>>>returns a "view" of the data (an array that shares the same > >>>> memory) as the original array. Thus, > >>>> > >>>>g[1:10:2] += 1 > >>>> > >>>>does an inplace add without data-copying. > >>>> > >>>>It may be possible to do what you want using zero-strided arrays, > >>>>but I'll leave that for a more motivated contributor. > >>>> > >>>>-Travis > >>> > >>>------------------------------------------------------- > >>>This SF.Net email is sponsored by xPML, a groundbreaking scripting > >>>language that extends applications into web and mobile media. > >>>Attend the live webcast and join the prime developer group > >>> breaking into this new coding territory! > >>>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat= > >>>1 21642 _______________________________________________ > >>>Numpy-discussion mailing list > >>>Numpy-discussion at lists.sourceforge.net > >>>https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >> > >>------------------------------------------------------- > >>This SF.Net email is sponsored by xPML, a groundbreaking scripting > >>language that extends applications into web and mobile media. > >> Attend the live webcast and join the prime developer group > >> breaking into this new coding territory! > >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=1 > >>21 642 _______________________________________________ > >>Numpy-discussion mailing list > >>Numpy-discussion at lists.sourceforge.net > >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion From tim.hochberg at cox.net Fri Mar 17 14:21:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 14:21:01 2006 Subject: [Numpy-discussion] Copy vs View for array[array] (was Histograms via indirect index arrays) In-Reply-To: <441B2EF7.10006@cox.net> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> Message-ID: <441B3632.2060402@cox.net> I just figured I'd add a couple of thought outside that needlessly contentious thread. In theory I'm all for view semantics for an array indexed by an array (I'm sure we have a good name for that, but it's escaping me). Indexing in numpy can be confusing enough without some indexing operations returning views and others copies. This is orthogonal to any issues of performance. In practice, I'm a bit skeptical. The result would need to be some sort of psuedo array object (similar to array.flat). Operations on this object would necessarily have worse performance than operations on a normal array due to the added level of indirection. In some circumstances it would also hold onto a lot of memory that might otherwise be freed since it hold a reference to the data for both the original array and the index array. How much of an effect this would have on the typical user is hard to say; any effects would certainly depend a lot on usage patterns. Here's some cases and guesses as to how I think things would work out (idx is an array): a[idx] += 1 # Improved performance and more importantly result consistent with other indexing ops c = a[idx] + b[idx] # Probably neutral; Creating the psuedo arrays should be fast, then they need to copied anyway in the add. c, d = a[idx], b[idx] del a, b e = c + d f = 2*c + d # Probably bad: Need to copy the psuedo array into a contiguous buffer multiple times. # Also holds extra memory. I'd like to see someone try it, but it's not high enough on my priority list for me to dive into it (I'm blowing all my spare cycles and then some on numexpr). -tim From cjw at sympatico.ca Fri Mar 17 14:35:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 17 14:35:03 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: References: <441A07C9.8050101@ee.byu.edu> <441A0C95.30902@sympatico.ca> Message-ID: <441B3986.9060701@sympatico.ca> Sasha wrote: >On 3/16/06, Colin J. Williams wrote: > > >>Would it make sense to reconsider the costs and benefits of dealing with >>dimensionless arrays? >> >> > >http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray > >http://aspn.activestate.com/ASPN/Mail/Message/2499954 > >http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3028210 > >This horse has suffered a lot beating ... > > > Sasha, Many thanks, this sets things out clearly. On a first scan, one thing caught my eye: I think we should implement Python scalars for the other types and then eliminate rank-0 arrays completely. I could have a student do this in a few weeks if it was agreed on. -Travis Oliphant [13-Sep-02] The Wiki sems to start with the design decision to implement zero dimension arrays. It then goes on to explore the problems created by that design decision. My contribution to the second thread you quote was: It looks as though a decision has been made. I was among those who favoured abandoning rank-0 arrays, we lost. To my mind rank-0 arrays add complexity for little benefit and make explanation more difficult. I don"t spot any discussion in the PEP of the pros and cons of the nd == 0 case. Colin W. [20-Mar-05] Ralf Juengling responded: A correction! There is, in the PEP:: Questions 1) should sequence behavior (i.e. some combination of slicing, indexing, and len) be supported for 0-dim arrays? Pros: It means that len(a) always works and returns the size of the array. Slicing code and indexing code will work for any dimension (the 0-dim array is an identity element for the operation of slicing) Cons: 0-dim arrays are really scalars. They should behave like Python scalars which do not allow sequence behavior 2) should array operations that result in a 0-dim array that is the same basic type as one of the Python scalars, return the Python scalar instead? Pros: 1) Some cases when Python expects an integer (the most dramatic is when slicing and indexing a sequence: _PyEval_SliceIndex in ceval.c) it will not try to convert it to an integer first before raising an error. Therefore it is convenient to have 0-dim arrays that are integers converted for you by the array object. 2) No risk of user confusion by having two types that are nearly but not exactly the same and whose separate existence can only be explained by the history of Python and NumPy development. 3) No problems with code that does explicit typechecks (isinstance(x, float) or type(x) == types.FloatType). Although explicit typechecks are considered bad practice in general, there are a couple of valid reasons to use them. 4) No creation of a dependency on Numeric in pickle files (though this could also be done by a special case in the pickling code for arrays) Cons: It is difficult to write generic code because scalars do not have the same methods and attributes as arrays. (such as .type or .shape). Also Python scalars have different numeric behavior as well. This results in a special-case checking that is not pleasant. Fundamentally it lets the user believe that somehow multidimensional homoegeneous arrays are something like Python lists (which except for Object arrays they are not). For me and for the end user, the (2) Pros win. I agree. Incidentally, the 0-d discussion does not appear in the 12-Jan-06 version of the PEP. I don't fully understand your proposal of 18-Feb-06 although I support Francesc's suggestion that we avoid "rank" to mean the number of dimensions. For matrices, we accept the rule that a matrix has two dimensions, not more not less. What problems would be created if we had the rule that an array has not less than one dimension? The little script below produces different results, depending on the ordering of the operands. This provides another challenge for the newcomer. Colin W. # arrayStuff.py def inArrayWorld(): import numpy as _n a= _n.arange(4) return a[2] def anotherWorld(a): import math w= 2.6 * a x= a * 2 y= 2.6 * a z= math.log10(a) return z, type(z), type(y), type(x), type(w) print anotherWorld(inArrayWorld()) From rlw at stsci.edu Fri Mar 17 14:57:10 2006 From: rlw at stsci.edu (Rick White) Date: Fri Mar 17 14:57:10 2006 Subject: [Numpy-discussion] Copy vs View for array[array] (was Histograms via indirect index arrays) In-Reply-To: <441B3632.2060402@cox.net> References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> <441B3632.2060402@cox.net> Message-ID: On Fri, 17 Mar 2006, Tim Hochberg wrote: > In theory I'm all for view semantics for an array indexed by an array > (I'm sure we have a good name for that, but it's escaping me). Indexing > in numpy can be confusing enough without some indexing operations > returning views and others copies. This is orthogonal to any issues of > performance. > > In practice, I'm a bit skeptical. The result would need to be some sort > of psuedo array object (similar to array.flat). Operations on this > object would necessarily have worse performance than operations on a > normal array due to the added level of indirection. In some > circumstances it would also hold onto a lot of memory that might > otherwise be freed since it hold a reference to the data for both the > original array and the index array. Actually I think it is worse than that -- it seems to me that it actually has to make a *copy* of the index array. I don't think that we would want to keep only a reference to the index array, since if it changed then the view could respond by changing in very unexpected ways. That sounds like a nightmare side-effect to me. That's what has always made me think that this is not a good idea, even if the bookkeeping of carrying around an unevaluated array+indices could be worked out efficiently. In my applications I sometimes use very large index arrays, and I don't want to have to copy them unnecessarily. Generally I much prefer instant evaluation as in the current implementation, since that uses the minimum of memory. For what it's worth, IDL behaves exactly like the current numpy: a[idx] += 1 increments each element by 1 regardless of how many times a particular index is included in idx. From tim.hochberg at cox.net Fri Mar 17 15:16:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 15:16:06 2006 Subject: [Numpy-discussion] Copy vs View for array[array] (was Histograms via indirect index arrays) In-Reply-To: References: <200603171424.59034.luszczek@cs.utk.edu> <441B114B.8010700@cox.net> <200603171512.44309.luszczek@cs.utk.edu> <441B2EF7.10006@cox.net> <441B3632.2060402@cox.net> Message-ID: <441B4315.5040109@cox.net> Rick White wrote: >On Fri, 17 Mar 2006, Tim Hochberg wrote: > > > >>In theory I'm all for view semantics for an array indexed by an array >>(I'm sure we have a good name for that, but it's escaping me). Indexing >>in numpy can be confusing enough without some indexing operations >>returning views and others copies. This is orthogonal to any issues of >>performance. >> >>In practice, I'm a bit skeptical. The result would need to be some sort >>of psuedo array object (similar to array.flat). Operations on this >>object would necessarily have worse performance than operations on a >>normal array due to the added level of indirection. In some >>circumstances it would also hold onto a lot of memory that might >>otherwise be freed since it hold a reference to the data for both the >>original array and the index array. >> >> > >Actually I think it is worse than that -- it seems to me that it >actually has to make a *copy* of the index array. I don't think >that we would want to keep only a reference to the index array, >since if it changed then the view could respond by changing in very >unexpected ways. That sounds like a nightmare side-effect to me. > > Yeah, that would be bad. Conceivably (there's that word again) one could implement copy-on-write for the index arrays, but that would be another can of worms. Anwway, I agree that you would have to at least fake a copy somehow. >That's what has always made me think that this is not a good idea, >even if the bookkeeping of carrying around an unevaluated array+indices >could be worked out efficiently. In my applications I sometimes >use very large index arrays, and I don't want to have to copy them >unnecessarily. Generally I much prefer instant evaluation as in >the current implementation, since that uses the minimum of memory. > >For what it's worth, IDL behaves exactly like the current numpy: >a[idx] += 1 increments each element by 1 regardless of how many >times a particular index is included in idx. > > I'm not much concerned with case myself. The case that bothers me (more in the abstract than in reality since I don't use index arrays much) is the following: >>> idx1 = slice(0,None,2) >>> idx2 = [0,2,4,6,8,10] >>> idx1 = slice(0,None,2) >>> idx2 = [0,2,4,6,8] >>> a = arange(10) >>> b = arange(10) >>> ai = a[idx1] >>> bi = b[idx2] >>> ai array([0, 2, 4, 6, 8]) >>> bi array([0, 2, 4, 6, 8]) >>> ai[1] = bi[1] = 99 >>> a array([ 0, 1, 99, 3, 4, 5, 6, 7, 8, 9]) >>> b array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) The fact that 'a' and 'b' diverge here is a wart, although not one I'm sure it's worth doing anything about. -tim > > > From rmuller at sandia.gov Fri Mar 17 15:26:03 2006 From: rmuller at sandia.gov (Rick Muller) Date: Fri Mar 17 15:26:03 2006 Subject: [Numpy-discussion] Difference between eigh and Heigenvectors Message-ID: What exactly is the difference between eigh and Heigenvectors. As far as I can tell, the eigenvalues are the same, but the eigenvectors are transposes of each other. Is this a matlab compatibility thing? Thanks in advance, R. Rick Muller rmuller at sandia.gov From tim.hochberg at cox.net Fri Mar 17 15:29:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 17 15:29:06 2006 Subject: [Numpy-discussion] numexpr update Message-ID: <441B4619.6000306@cox.net> I just committed a whole slow of changes to numexpr for your enjoyment. First the good news. It now handles integers as well as floats and the framework is in place for adding complex numbers as well (it can actually add complex numbers, but nothing else yet). It should be pretty simple to add complex at this point, so that'll probably happen soon. Otherwise things are more or less the same, although I expect to modifying the interface a little bit here soon to better support typing of input values. Now the bad news: somewhere along the way I introduced some sort of memory bug and it occasionally segfaults on me. I'm not sure if it's reference counting or I'm scribbling outside my allocated block of memory somehow, but I'll continue hunting it down. More minor issues include int**int is a little bit bogus and I've turned off the aggressive power optimizations for the moment pending some improvements to the compiler. -tim From ndarray at mac.com Fri Mar 17 15:44:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 17 15:44:03 2006 Subject: [Numpy-discussion] Array scalar math ideas In-Reply-To: <441B3986.9060701@sympatico.ca> References: <441A07C9.8050101@ee.byu.edu> <441A0C95.30902@sympatico.ca> <441B3986.9060701@sympatico.ca> Message-ID: On 3/17/06, Colin J. Williams wrote: > Sasha wrote: > ... > >http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3028210 > ... > Many thanks, this sets things out clearly. On a first scan, one thing > caught my eye: > > I think we should implement Python scalars for the other types and then > eliminate rank-0 arrays completely. I could have a student do this in a > few weeks if it was agreed on. > > -Travis Oliphant [13-Sep-02] > I expressed my view when I started "A case for rank-0 arrays" thread. (See link above.) In summary, dimensionless arrays and scalars are different beasts and both have a right to live. > > The Wiki sems to start with the design decision to implement zero > dimension arrays. > It then goes on to explore the problems created by that design decision. > The wiki has a short section on "The Need for Zero-Rank Arrays." Please feel free to take over that page and add you criticism of the status quo. I think wiki is a better medium to discuss this issue than the mailing list. > For matrices, we accept the rule that a matrix has two dimensions, not more > not less. What problems would be created if we had the rule that an array > has not less than one dimension? > For me the main problem is the dificulty to write generic code both in C and in Python. Most issues stem from the fact that scalars are immutable while arrays (including dimentionless) are not. > The little script below produces different results, depending on the ordering > of the operands. This provides another challenge for the newcomer. I don't understand how this is related to dimensionless arrays. Looks like an array scalar feature: >>> type(2.0 * int_(1)) >>> type(int_(1) * 2.0) (I believe this can be fixed once scalar math is implemented.) Dimensionless arrays seem to be more consistent in this particular case: >>> type(array(1) * 2.0) >>> type(2.0*array(1)) I believe type(array(1) * 2.0) should be ndarray, but that's a different story. > > # arrayStuff.py > > def inArrayWorld(): > import numpy as _n > a= _n.arange(4) > return a[2] > > def anotherWorld(a): > import math > w= 2.6 * a > x= a * 2 > y= 2.6 * a > z= math.log10(a) > return z, type(z), type(y), type(x), type(w) > > print anotherWorld(inArrayWorld()) > > From schofield at ftw.at Sat Mar 18 08:27:03 2006 From: schofield at ftw.at (Ed Schofield) Date: Sat Mar 18 08:27:03 2006 Subject: [Numpy-discussion] NumPy 0.9.6 and win32pdh Message-ID: <441C34DD.1000901@ftw.at> Hi all, I'm trying to compile Win32 binaries for a new release of SciPy. I'm getting a warning message upon import after installing numpy-0.9.6-win32-py2.4.exe: import testing -> failed: No module named win32pdh and I'm unable to run any of NumPy's unit tests unless I manually comment out the block if os.name=='nt': ... in testing\utils.py. Has anyone else seen this? Does anyone know if the win32pdh module is supposed to be installed with the python-2.4.2.msi installer from python.org? The Python 2.3 installer, numpy-0.9.6-win32-py2.3.exe, works fine, because the corresponding if statement has an additional condition (" and sys.version[:3] > '2.3'"), so the import is never attempted. -- Ed From robert.kern at gmail.com Sat Mar 18 10:18:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Sat Mar 18 10:18:01 2006 Subject: [Numpy-discussion] Re: NumPy 0.9.6 and win32pdh In-Reply-To: <441C34DD.1000901@ftw.at> References: <441C34DD.1000901@ftw.at> Message-ID: Ed Schofield wrote: > Hi all, > > I'm trying to compile Win32 binaries for a new release of SciPy. I'm > getting a warning message upon import after installing > numpy-0.9.6-win32-py2.4.exe: > > import testing -> failed: No module named win32pdh > > and I'm unable to run any of NumPy's unit tests unless I manually > comment out the block > > if os.name=='nt': > ... > > in testing\utils.py. > > Has anyone else seen this? Does anyone know if the win32pdh module is > supposed to be installed with the python-2.4.2.msi installer from > python.org? It's from the win32all package. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dd55 at cornell.edu Sat Mar 18 12:08:01 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat Mar 18 12:08:01 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: References: <200603131943.53857.dd55@cornell.edu> Message-ID: <200603181507.00136.dd55@cornell.edu> On Monday 13 March 2006 9:05 pm, Robert Kern wrote: > Darren Dale wrote: > > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my > > home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 > > and gcc-4.1. When I try to build numpy I get warnings like "warning: > > build_ext: fcompiler=gnu is not available." I tried forcing numpy to use > > gfortran with "python setup.py config_fc --fcompiler=gnu95 build", and > > got the same warning for fcompiler=gnu95. > > Building numpy shouldn't require any kind of Fortran compiler. > > However, the problem is that the version-recognizing regex doesn't like the > version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in > numpy/distutils/fcompiler/gnu.py and the class attribute > Gnu95FCompiler.version_pattern . Thanks Robert. May I suggest the following change, from: version_pattern = r'GNU Fortran 95 \(GCC (?P[^\s*\)]+)' to: version_pattern = r'GNU Fortran 95 \(GCC?\) (?P[^\s*\)]+)' along with the following comment to go with the debian example: # Gentoo: GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) Also, I wonder if I found a bug in numpy/distutils/ccompiler.py. if status in ok_status: m = re.match(self.version_pattern,output) if m: version = m.group('version') if not m: raise ValueError("compiler version not matched (%r)" %\ (version,)) version = LooseVersion(version) self.version = version return version That ValueError will never be raised, since it has already been established that m is not NoneType. Darren From cookedm at physics.mcmaster.ca Sat Mar 18 14:48:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sat Mar 18 14:48:02 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: <200603181507.00136.dd55@cornell.edu> (Darren Dale's message of "Sat, 18 Mar 2006 15:06:59 -0500") References: <200603131943.53857.dd55@cornell.edu> <200603181507.00136.dd55@cornell.edu> Message-ID: Darren Dale writes: > On Monday 13 March 2006 9:05 pm, Robert Kern wrote: >> Darren Dale wrote: >> > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my >> > home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 >> > and gcc-4.1. When I try to build numpy I get warnings like "warning: >> > build_ext: fcompiler=gnu is not available." I tried forcing numpy to use >> > gfortran with "python setup.py config_fc --fcompiler=gnu95 build", and >> > got the same warning for fcompiler=gnu95. >> >> Building numpy shouldn't require any kind of Fortran compiler. >> >> However, the problem is that the version-recognizing regex doesn't like the >> version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in >> numpy/distutils/fcompiler/gnu.py and the class attribute >> Gnu95FCompiler.version_pattern . > > Thanks Robert. May I suggest the following change, from: > version_pattern = r'GNU Fortran 95 \(GCC (?P[^\s*\)]+)' > to: > version_pattern = r'GNU Fortran 95 \(GCC?\) (?P[^\s*\)]+)' > along with the following comment to go with the debian example: > # Gentoo: GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) I fixed that a few days ago and forgot to check my solution in :-) Now, you can define a version_match method that can do more complex matching, instead of just a regex. The GNU Fortran compilers now use this: gfortran, for instance, checks the version output to start with 'GNU Fortran 95', then something versionish (the regex [-.\d]+). > Also, I wonder if I found a bug in numpy/distutils/ccompiler.py. > > if status in ok_status: > m = re.match(self.version_pattern,output) > if m: > version = m.group('version') > if not m: > raise ValueError("compiler version not matched (%r)" %\ > (version,)) > version = LooseVersion(version) > self.version = version > return version > > That ValueError will never be raised, since it has already been established > that m is not NoneType. This is fixed too. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From dd55 at cornell.edu Sat Mar 18 15:07:09 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat Mar 18 15:07:09 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: References: <200603131943.53857.dd55@cornell.edu> <200603181507.00136.dd55@cornell.edu> Message-ID: <200603181806.29561.dd55@cornell.edu> Hi David, I cant build numpy with the changes you just commited: Running from numpy source directory. No module named __svn_version__ F2PY Version 2_2258 blas_opt_info: blas_mkl_info: /home/darren/src/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c include_dirs = ['/usr/include/atlas'] running build_src building extension "atlas_version" sources creating build creating build/src adding 'build/src/atlas_version_-0x54967df6.c' to sources. running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext building 'atlas_version' extension compiling C sources i686-pc-linux-gnu-gcc options: '-pthread -fno-strict-aliasing -DNDEBUG -fPIC' creating build/temp.linux-i686-2.4 creating build/temp.linux-i686-2.4/build creating build/temp.linux-i686-2.4/build/src compile options: '-I/usr/include/atlas -Inumpy/core/include -I/usr/include/python2.4 -c' i686-pc-linux-gnu-gcc: build/src/atlas_version_-0x54967df6.c i686-pc-linux-gnu-gcc -pthread -shared build/temp.linux-i686-2.4/build/src/atlas_version_-0x54967df6.o -L/usr/lib -llapack -lblas -lcblas -latlas -o build/temp.linux-i686-2.4/atlas_version.so FOUND: libraries = ['lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = c define_macros = [('ATLAS_INFO', '"\\"3.7.11\\""')] include_dirs = ['/usr/include/atlas'] lapack_opt_info: lapack_mkl_info: mkl_info: NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS /home/darren/src/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack_atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 include_dirs = ['/usr/include/atlas'] running build_src building extension "atlas_version" sources adding 'build/src/atlas_version_0x26de9279.c' to sources. running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext invalid syntax (gnu.py, line 225) Could not locate executable ifort Could not locate executable ifc Could not locate executable ifort Could not locate executable efort Could not locate executable efc customize IntelFCompiler customize LaheyFCompiler customize PGroupFCompiler customize AbsoftFCompiler customize NAGFCompiler invalid syntax (gnu.py, line 225) customize CompaqFCompiler customize IntelItaniumFCompiler invalid syntax (gnu.py, line 225) customize G95FCompiler invalid syntax (gnu.py, line 225) invalid syntax (gnu.py, line 225) ##### msg: invalid syntax (gnu.py, line 225) invalid syntax (gnu.py, line 225) FOUND: libraries = ['lapack', 'lapack', 'blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include/atlas'] Warning: distutils distribution has been initialized, it may be too late to add an extension lapack_lite Warning: distutils distribution has been initialized, it may be too late to add a subpackage linalg Warning: distutils distribution has been initialized, it may be too late to add an extension mtrand Warning: distutils distribution has been initialized, it may be too late to add a subpackage random Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 63, in setup_package config.add_subpackage('numpy') File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 592, in add_subpackage config_list = self.get_subpackage(subpackage_name,subpackage_path) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 582, in get_subpackage subpackage_path) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 539, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "/home/darren/src/numpy/setup.py", line 15, in configuration File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 636, in add_data_dir self.add_data_files((ds,filenames)) File "/home/darren/src/numpy/numpy/distutils/misc_util.py", line 702, in add_data_files dist.data_files.extend(data_dict.items()) AttributeError: 'NoneType' object has no attribute 'extend' On Saturday 18 March 2006 5:47 pm, David M. Cooke wrote: > Darren Dale writes: > > On Monday 13 March 2006 9:05 pm, Robert Kern wrote: > >> Darren Dale wrote: > >> > Has anyone been able to build numpy with gcc-4.1? I upgraded gcc on my > >> > home computer this weekend from glibc-2.3.6 and gcc-3.4.5 to glibc-2.4 > >> > and gcc-4.1. When I try to build numpy I get warnings like "warning: > >> > build_ext: fcompiler=gnu is not available." I tried forcing numpy to > >> > use gfortran with "python setup.py config_fc --fcompiler=gnu95 build", > >> > and got the same warning for fcompiler=gnu95. > >> > >> Building numpy shouldn't require any kind of Fortran compiler. > >> > >> However, the problem is that the version-recognizing regex doesn't like > >> the version string "GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0)". Look in > >> numpy/distutils/fcompiler/gnu.py and the class attribute > >> Gnu95FCompiler.version_pattern . > > > > Thanks Robert. May I suggest the following change, from: > > version_pattern = r'GNU Fortran 95 \(GCC (?P[^\s*\)]+)' > > to: > > version_pattern = r'GNU Fortran 95 \(GCC?\) (?P[^\s*\)]+)' > > along with the following comment to go with the debian example: > > # Gentoo: GNU Fortran 95 (GCC) 4.1.0 (Gentoo 4.1.0) > > I fixed that a few days ago and forgot to check my solution in :-) > Now, you can define a version_match method that can do more complex > matching, instead of just a regex. The GNU Fortran compilers now use > this: gfortran, for instance, checks the version output to start > with 'GNU Fortran 95', then something versionish (the regex [-.\d]+). > > > Also, I wonder if I found a bug in numpy/distutils/ccompiler.py. > > > > if status in ok_status: > > m = re.match(self.version_pattern,output) > > if m: > > version = m.group('version') > > if not m: > > raise ValueError("compiler version not matched (%r)" %\ > > (version,)) > > version = LooseVersion(version) > > self.version = version > > return version > > > > That ValueError will never be raised, since it has already been > > established that m is not NoneType. > > This is fixed too. From dd55 at cornell.edu Sat Mar 18 15:19:01 2006 From: dd55 at cornell.edu (Darren Dale) Date: Sat Mar 18 15:19:01 2006 Subject: [Numpy-discussion] Re: gcc-4.1 In-Reply-To: <200603181806.29561.dd55@cornell.edu> References: <200603131943.53857.dd55@cornell.edu> <200603181806.29561.dd55@cornell.edu> Message-ID: <200603181818.12780.dd55@cornell.edu> On Saturday 18 March 2006 6:06 pm, Darren Dale wrote: > Hi David, > > I cant build numpy with the changes you just commited: Scratch that, sorry. I deleted my local copy of gnu.py and downloaded a fresh svn version and was able to build. Thanks! From oliphant.travis at ieee.org Sat Mar 18 20:18:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Mar 18 20:18:05 2006 Subject: [Numpy-discussion] NumPy 0.9.6 and win32pdh In-Reply-To: <441C34DD.1000901@ftw.at> References: <441C34DD.1000901@ftw.at> Message-ID: <441CDB57.6010508@ieee.org> Ed Schofield wrote: > Hi all, > > I'm trying to compile Win32 binaries for a new release of SciPy. I'm > getting a warning message upon import after installing > numpy-0.9.6-win32-py2.4.exe: > > import testing -> failed: No module named win32pdh > > and I'm unable to run any of NumPy's unit tests unless I manually > comment out the block > Hmm... I guess I though that win32pdh was part of python 2.4 (I saw this error when building Pyton 2.3 packages but not Python 2.4). But, of course it was probably that I did or didn't have win32all installed. That's unfortunate. The win32pdh dependency needs to be fixed, then. It's too bad this dependency was created without appropriate checks. No doubt we'll have several who complain about this one until numpy 0.9.7 comes out... -Travis From oliphant.travis at ieee.org Sat Mar 18 20:59:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Mar 18 20:59:03 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <200603171630.27075.luszczek@cs.utk.edu> References: <200603171519.35502.luszczek@cs.utk.edu> <200603171630.27075.luszczek@cs.utk.edu> Message-ID: <441CE4DE.6040101@ieee.org> Piotr Luszczek wrote: >> Show us some code that works. I'm not interested in >> implementing your feature request. You are. There's plenty of work >> that you can do that doesn't depend on anyone else agreeing with you, >> so you can stop arguing and start coding. >> > > Arguing is good to a point and I think you're right that it's time to > stop. > All true. However, I want to thank Piotr for illustrating an important but easy to overlook point that in-fact "advanced" indexing produces copies while "simple-indexing" produces views. The reason behind this as Tim and Rick later illustrate is that no one quite knows how to produce "view" semantics for an arbitrarily indexed array. I did actually think about this for a while because I don't completely like the different behaviors which can produce warts like the one that Tim showed us. I realized that it would be possible to extend our memory model of an array to include an index array in the array structure itself that would indicate how to advance through memory (basically a pointer or its equivalent for each array element). Rick showed that because this would have to be a "copy" of the indices and so would create too much slowness. My reason for not going there was because it seemed to add too much complication for my tastes at the time to all the code. But, one could still sub-class the ndarray and implement such a thing. I think this is the best place to explore such experimental ideas. Thanks again for the discussion. There are issues like this that it is good to get out into the open. -Travis From vzkezlxdkhy at mail.medscape.com Sun Mar 19 16:51:05 2006 From: vzkezlxdkhy at mail.medscape.com (vzkezlxdkhy) Date: Sun Mar 19 16:51:05 2006 Subject: [Numpy-discussion] Fw: Message-ID: <000c01c64bb9$08d55730$726ddf94@avelazquez> ----- Original Message ----- From: Herman Augustus To: dmqztgjn at nate.com Sent: Monday, March 13, 2006 11:16 PM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bwcexnk.gif Type: image/gif Size: 9499 bytes Desc: not available URL: From hurak at control.felk.cvut.cz Mon Mar 20 02:37:01 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Mon Mar 20 02:37:01 2006 Subject: [Numpy-discussion] Rationale behind behaviour of atleast_3d in Numpy Message-ID: Hello, As a newcommer, I am trying not only to learn to use Numpy, but also to understand it. I think I need some help with function atleast_3d. When applied to 2d array, its behaviour is different then when applied to 1d array (it is a documented feature): a = array([1,2,3]) A = array([[1,2,3],[4,5,6]]) a3d = atleast_3d(a) print a3d.shape ---> (1, 3, 1) A3d = atleast_3d(A) print A3d.shape ---> (2, 3, 1) What is the rationale behind this in my opinion unintuitive inconsistency? I especially don't understand the behaviour for 1d arrays. But even with 2d arrays, I am inclined to think (with my beginner-level of mastering Python and Numpy) that a more natural result of shape function would be (1,2,3). Thank you very much for some explanation, Zdenek Hurak From chanley at stsci.edu Mon Mar 20 06:07:07 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 20 06:07:07 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 Message-ID: <441EB6E8.8030504@stsci.edu> Solaris builds began failing yesterday with the following output: Your "cron" job on basil.stsci.edu /home/chanley/bin/numpy_update.csh produced the following output: Sun Mar 19 07:20:04 EST 2006 ### NUMPY Daily Build / Build Test ### ### Setup test environment. ### /usr/ra/pyssg/Python-2.4.2/bin/python ### Checkout, build, and install new version of numpy. ### /data/basil5 A numpy/LICENSE.txt A numpy/scipy_compatibility A numpy/TODO.txt A numpy/THANKS.txt A numpy/TEST_COMMIT A numpy/setup.py A numpy/numpy A numpy/numpy/_import_tools.py A numpy/numpy/f2py A numpy/numpy/f2py/f2py.1 A numpy/numpy/f2py/diagnose.py A numpy/numpy/f2py/rules.py A numpy/numpy/f2py/info.py A numpy/numpy/f2py/capi_maps.py A numpy/numpy/f2py/auxfuncs.py A numpy/numpy/f2py/TODO.txt A numpy/numpy/f2py/cb_rules.py A numpy/numpy/f2py/__init__.py A numpy/numpy/f2py/setup.py A numpy/numpy/f2py/src A numpy/numpy/f2py/src/test A numpy/numpy/f2py/src/test/foo.f A numpy/numpy/f2py/src/test/bar.f A numpy/numpy/f2py/src/test/foo90.f90 A numpy/numpy/f2py/src/test/foomodule.c A numpy/numpy/f2py/src/test/Makefile A numpy/numpy/f2py/src/test/wrap.f A numpy/numpy/f2py/src/fortranobject.c A numpy/numpy/f2py/src/fortranobject.h A numpy/numpy/f2py/docs A numpy/numpy/f2py/docs/HISTORY.txt A numpy/numpy/f2py/docs/pytest.py A numpy/numpy/f2py/docs/usersguide A numpy/numpy/f2py/docs/usersguide/scalar_session.dat A numpy/numpy/f2py/docs/usersguide/fib2.pyf A numpy/numpy/f2py/docs/usersguide/callback.f A numpy/numpy/f2py/docs/usersguide/calculate.f A numpy/numpy/f2py/docs/usersguide/run_main_session.dat A numpy/numpy/f2py/docs/usersguide/index.txt A numpy/numpy/f2py/docs/usersguide/var.pyf A numpy/numpy/f2py/docs/usersguide/string.f A numpy/numpy/f2py/docs/usersguide/spam.pyf A numpy/numpy/f2py/docs/usersguide/ftype_session.dat A numpy/numpy/f2py/docs/usersguide/extcallback.f A numpy/numpy/f2py/docs/usersguide/setup_example.py A numpy/numpy/f2py/docs/usersguide/array.f A numpy/numpy/f2py/docs/usersguide/common_session.dat A numpy/numpy/f2py/docs/usersguide/var_session.dat A numpy/numpy/f2py/docs/usersguide/moddata.f90 A numpy/numpy/f2py/docs/usersguide/callback_session.dat A numpy/numpy/f2py/docs/usersguide/calculate_session.dat A numpy/numpy/f2py/docs/usersguide/spam_session.dat A numpy/numpy/f2py/docs/usersguide/docutils.conf A numpy/numpy/f2py/docs/usersguide/callback2.pyf A numpy/numpy/f2py/docs/usersguide/string_session.dat A numpy/numpy/f2py/docs/usersguide/moddata_session.dat A numpy/numpy/f2py/docs/usersguide/extcallback_session.dat A numpy/numpy/f2py/docs/usersguide/array_session.dat A numpy/numpy/f2py/docs/usersguide/scalar.f A numpy/numpy/f2py/docs/usersguide/allocarr.f90 A numpy/numpy/f2py/docs/usersguide/compile_session.dat A numpy/numpy/f2py/docs/usersguide/fib1.f A numpy/numpy/f2py/docs/usersguide/fib3.f A numpy/numpy/f2py/docs/usersguide/allocarr_session.dat A numpy/numpy/f2py/docs/usersguide/ftype.f A numpy/numpy/f2py/docs/usersguide/default.css A numpy/numpy/f2py/docs/usersguide/common.f A numpy/numpy/f2py/docs/usersguide/fib1.pyf A numpy/numpy/f2py/docs/pyforttest.pyf A numpy/numpy/f2py/docs/simple_session.dat A numpy/numpy/f2py/docs/FAQ.txt A numpy/numpy/f2py/docs/THANKS.txt A numpy/numpy/f2py/docs/hello.f A numpy/numpy/f2py/docs/OLDNEWS.txt A numpy/numpy/f2py/docs/docutils.conf A numpy/numpy/f2py/docs/README.txt A numpy/numpy/f2py/docs/TESTING.txt A numpy/numpy/f2py/docs/default.css A numpy/numpy/f2py/docs/simple.f A numpy/numpy/f2py/common_rules.py A numpy/numpy/f2py/NEWS.txt A numpy/numpy/f2py/use_rules.py A numpy/numpy/f2py/f2py2e.py A numpy/numpy/f2py/setup.cfg A numpy/numpy/f2py/f90mod_rules.py A numpy/numpy/f2py/func2subr.py A numpy/numpy/f2py/tests A numpy/numpy/f2py/tests/c A numpy/numpy/f2py/tests/c/return_real.py A numpy/numpy/f2py/tests/f77 A numpy/numpy/f2py/tests/f77/return_logical.py A numpy/numpy/f2py/tests/f77/return_character.py A numpy/numpy/f2py/tests/f77/callback.py A numpy/numpy/f2py/tests/f77/return_integer.py A numpy/numpy/f2py/tests/f77/return_real.py A numpy/numpy/f2py/tests/f77/return_complex.py A numpy/numpy/f2py/tests/run_all.py A numpy/numpy/f2py/tests/array_from_pyobj A numpy/numpy/f2py/tests/array_from_pyobj/wrapmodule.c A numpy/numpy/f2py/tests/array_from_pyobj/tests A numpy/numpy/f2py/tests/array_from_pyobj/tests/test_array_from_pyobj.py A numpy/numpy/f2py/tests/array_from_pyobj/__init__.py A numpy/numpy/f2py/tests/array_from_pyobj/setup.py A numpy/numpy/f2py/tests/mixed A numpy/numpy/f2py/tests/mixed/foo_fixed.f90 A numpy/numpy/f2py/tests/mixed/foo_free.f90 A numpy/numpy/f2py/tests/mixed/foo.f A numpy/numpy/f2py/tests/mixed/run.py A numpy/numpy/f2py/tests/f90 A numpy/numpy/f2py/tests/f90/return_logical.py A numpy/numpy/f2py/tests/f90/return_character.py A numpy/numpy/f2py/tests/f90/return_integer.py A numpy/numpy/f2py/tests/f90/return_real.py A numpy/numpy/f2py/tests/f90/return_complex.py A numpy/numpy/f2py/f2py_testing.py A numpy/numpy/f2py/doc A numpy/numpy/f2py/doc/notes.tex A numpy/numpy/f2py/doc/win32_notes.txt A numpy/numpy/f2py/doc/multiarray A numpy/numpy/f2py/doc/multiarray/run.pyf A numpy/numpy/f2py/doc/multiarray/array_from_pyobj.c A numpy/numpy/f2py/doc/multiarray/bar.c A numpy/numpy/f2py/doc/multiarray/fun.pyf A numpy/numpy/f2py/doc/multiarray/foo.f A numpy/numpy/f2py/doc/multiarray/transpose.txt A numpy/numpy/f2py/doc/multiarray/fortran_array_from_pyobj.txt A numpy/numpy/f2py/doc/f2python9-final A numpy/numpy/f2py/doc/f2python9-final/structure.jpg A numpy/numpy/f2py/doc/f2python9-final/mk_ps.sh A numpy/numpy/f2py/doc/f2python9-final/mk_html.sh A numpy/numpy/f2py/doc/f2python9-final/aerostructure.jpg A numpy/numpy/f2py/doc/f2python9-final/flow.jpg A numpy/numpy/f2py/doc/f2python9-final/src A numpy/numpy/f2py/doc/f2python9-final/src/examples A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1session.txt A numpy/numpy/f2py/doc/f2python9-final/src/examples/foo.pyf A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1.f A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1mess.txt A numpy/numpy/f2py/doc/f2python9-final/src/examples/foom.pyf A numpy/numpy/f2py/doc/f2python9-final/mk_pdf.sh A numpy/numpy/f2py/doc/f2python9-final/README.txt A numpy/numpy/f2py/doc/intro.tex A numpy/numpy/f2py/doc/multiarrays.txt A numpy/numpy/f2py/doc/options.tex A numpy/numpy/f2py/doc/collectinput.py A numpy/numpy/f2py/doc/ex1 A numpy/numpy/f2py/doc/ex1/foobar.f90 A numpy/numpy/f2py/doc/ex1/runme A numpy/numpy/f2py/doc/ex1/foo.f A numpy/numpy/f2py/doc/ex1/arr.f A numpy/numpy/f2py/doc/ex1/bar.f A numpy/numpy/f2py/doc/ex1/foobar-smart.f90 A numpy/numpy/f2py/doc/ex1/foobarmodule.tex A numpy/numpy/f2py/doc/bugs.tex A numpy/numpy/f2py/doc/commands.tex A numpy/numpy/f2py/doc/signaturefile.tex A numpy/numpy/f2py/doc/fortranobject.tex A numpy/numpy/f2py/doc/apps.tex A numpy/numpy/f2py/doc/Release-1.x.txt A numpy/numpy/f2py/doc/using_F_compiler.txt A numpy/numpy/f2py/doc/Release-2.x.txt A numpy/numpy/f2py/doc/Release-3.x.txt A numpy/numpy/f2py/doc/Release-4.x.txt A numpy/numpy/f2py/doc/f2py2e.tex A numpy/numpy/f2py/doc/python9.tex A numpy/numpy/f2py/doc/index.html A numpy/numpy/f2py/doc/Makefile A numpy/numpy/f2py/doc/oldnews.html A numpy/numpy/f2py/crackfortran.py A numpy/numpy/f2py/cfuncs.py A numpy/numpy/f2py/__version__.py A numpy/numpy/f2py/README.txt A numpy/numpy/f2py/Makefile A numpy/numpy/f2py/BUGS.txt A numpy/numpy/random A numpy/numpy/random/info.py A numpy/numpy/random/mtrand A numpy/numpy/random/mtrand/Python.pxi A numpy/numpy/random/mtrand/distributions.c A numpy/numpy/random/mtrand/initarray.c A numpy/numpy/random/mtrand/mtrand.pyx A numpy/numpy/random/mtrand/mtrand.c A numpy/numpy/random/mtrand/numpy.pxi A numpy/numpy/random/mtrand/distributions.h A numpy/numpy/random/mtrand/generate_mtrand_c.py A numpy/numpy/random/mtrand/initarray.h A numpy/numpy/random/mtrand/randomkit.c A numpy/numpy/random/mtrand/randomkit.h A numpy/numpy/random/__init__.py A numpy/numpy/random/setup.py A numpy/numpy/add_newdocs.py A numpy/numpy/distutils A numpy/numpy/distutils/core.py A numpy/numpy/distutils/misc_util.py A numpy/numpy/distutils/fcompiler A numpy/numpy/distutils/fcompiler/mips.py A numpy/numpy/distutils/fcompiler/gnu.py A numpy/numpy/distutils/fcompiler/intel.py A numpy/numpy/distutils/fcompiler/vast.py A numpy/numpy/distutils/fcompiler/absoft.py A numpy/numpy/distutils/fcompiler/__init__.py A numpy/numpy/distutils/fcompiler/none.py A numpy/numpy/distutils/fcompiler/compaq.py A numpy/numpy/distutils/fcompiler/lahey.py A numpy/numpy/distutils/fcompiler/g95.py A numpy/numpy/distutils/fcompiler/hpux.py A numpy/numpy/distutils/fcompiler/nag.py A numpy/numpy/distutils/fcompiler/sun.py A numpy/numpy/distutils/fcompiler/pg.py A numpy/numpy/distutils/fcompiler/ibm.py A numpy/numpy/distutils/info.py A numpy/numpy/distutils/line_endings.py A numpy/numpy/distutils/from_template.py A numpy/numpy/distutils/__init__.py A numpy/numpy/distutils/system_info.py A numpy/numpy/distutils/conv_template.py A numpy/numpy/distutils/setup.py A numpy/numpy/distutils/cpuinfo.py A numpy/numpy/distutils/lib2def.py A numpy/numpy/distutils/intelccompiler.py A numpy/numpy/distutils/tests A numpy/numpy/distutils/tests/f2py_ext A numpy/numpy/distutils/tests/f2py_ext/tests A numpy/numpy/distutils/tests/f2py_ext/tests/test_fib2.py A numpy/numpy/distutils/tests/f2py_ext/__init__.py A numpy/numpy/distutils/tests/f2py_ext/setup.py A numpy/numpy/distutils/tests/f2py_ext/src A numpy/numpy/distutils/tests/f2py_ext/src/fib2.pyf A numpy/numpy/distutils/tests/f2py_ext/src/fib1.f A numpy/numpy/distutils/tests/pyrex_ext A numpy/numpy/distutils/tests/pyrex_ext/tests A numpy/numpy/distutils/tests/pyrex_ext/tests/test_primes.py A numpy/numpy/distutils/tests/pyrex_ext/__init__.py A numpy/numpy/distutils/tests/pyrex_ext/setup.py A numpy/numpy/distutils/tests/pyrex_ext/primes.pyx A numpy/numpy/distutils/tests/setup.py A numpy/numpy/distutils/tests/gen_ext A numpy/numpy/distutils/tests/gen_ext/tests A numpy/numpy/distutils/tests/gen_ext/tests/test_fib3.py A numpy/numpy/distutils/tests/gen_ext/__init__.py A numpy/numpy/distutils/tests/gen_ext/setup.py A numpy/numpy/distutils/tests/swig_ext A numpy/numpy/distutils/tests/swig_ext/tests A numpy/numpy/distutils/tests/swig_ext/tests/test_example.py A numpy/numpy/distutils/tests/swig_ext/tests/test_example2.py A numpy/numpy/distutils/tests/swig_ext/__init__.py A numpy/numpy/distutils/tests/swig_ext/setup.py A numpy/numpy/distutils/tests/swig_ext/src A numpy/numpy/distutils/tests/swig_ext/src/example.i A numpy/numpy/distutils/tests/swig_ext/src/zoo.cc A numpy/numpy/distutils/tests/swig_ext/src/example.c A numpy/numpy/distutils/tests/swig_ext/src/zoo.h A numpy/numpy/distutils/tests/swig_ext/src/zoo.i A numpy/numpy/distutils/tests/f2py_f90_ext A numpy/numpy/distutils/tests/f2py_f90_ext/tests A numpy/numpy/distutils/tests/f2py_f90_ext/tests/test_foo.py A numpy/numpy/distutils/tests/f2py_f90_ext/include A numpy/numpy/distutils/tests/f2py_f90_ext/include/body.f90 A numpy/numpy/distutils/tests/f2py_f90_ext/__init__.py A numpy/numpy/distutils/tests/f2py_f90_ext/setup.py A numpy/numpy/distutils/tests/f2py_f90_ext/src A numpy/numpy/distutils/tests/f2py_f90_ext/src/foo_free.f90 A numpy/numpy/distutils/tests/test_misc_util.py A numpy/numpy/distutils/extension.py A numpy/numpy/distutils/ccompiler.py A numpy/numpy/distutils/log.py A numpy/numpy/distutils/__version__.py A numpy/numpy/distutils/unixccompiler.py A numpy/numpy/distutils/exec_command.py A numpy/numpy/distutils/mingw32ccompiler.py A numpy/numpy/distutils/command A numpy/numpy/distutils/command/build_clib.py A numpy/numpy/distutils/command/egg_info.py A numpy/numpy/distutils/command/__init__.py A numpy/numpy/distutils/command/build.py A numpy/numpy/distutils/command/build_ext.py A numpy/numpy/distutils/command/config_compiler.py A numpy/numpy/distutils/command/install_data.py A numpy/numpy/distutils/command/install_headers.py A numpy/numpy/distutils/command/bdist_rpm.py A numpy/numpy/distutils/command/config.py A numpy/numpy/distutils/command/build_scripts.py A numpy/numpy/distutils/command/build_src.py A numpy/numpy/distutils/command/install.py A numpy/numpy/distutils/command/sdist.py A numpy/numpy/distutils/command/build_py.py A numpy/numpy/doc A numpy/numpy/doc/records.txt A numpy/numpy/doc/ufuncs.txt A numpy/numpy/doc/DISTUTILS.txt A numpy/numpy/doc/pyrex A numpy/numpy/doc/pyrex/numpyx.pyx A numpy/numpy/doc/pyrex/numpyx.c A numpy/numpy/doc/pyrex/c_numpy.pxd A numpy/numpy/doc/pyrex/MANIFEST A numpy/numpy/doc/pyrex/setup.py A numpy/numpy/doc/pyrex/notes A numpy/numpy/doc/pyrex/run_test.py A numpy/numpy/doc/pyrex/Makefile A numpy/numpy/doc/pyrex/c_python.pxd A numpy/numpy/doc/swig A numpy/numpy/doc/swig/numpy.i A numpy/numpy/doc/swig/series.h A numpy/numpy/doc/swig/Series.i A numpy/numpy/doc/swig/testSeries.py A numpy/numpy/doc/swig/HelperFunctions.txt A numpy/numpy/doc/swig/setup.py A numpy/numpy/doc/swig/series.cxx A numpy/numpy/doc/swig/Makefile A numpy/numpy/doc/swig/README A numpy/numpy/doc/CAPI.txt A numpy/numpy/doc/README.txt A numpy/numpy/__init__.py A numpy/numpy/lib A numpy/numpy/lib/scimath.py A numpy/numpy/lib/shape_base.py A numpy/numpy/lib/machar.py A numpy/numpy/lib/info.py A numpy/numpy/lib/twodim_base.py A numpy/numpy/lib/__init__.py A numpy/numpy/lib/src A numpy/numpy/lib/src/_compiled_base.c A numpy/numpy/lib/setup.py A numpy/numpy/lib/utils.py A numpy/numpy/lib/getlimits.py A numpy/numpy/lib/tests A numpy/numpy/lib/tests/test_shape_base.py A numpy/numpy/lib/tests/test_arraysetops.py A numpy/numpy/lib/tests/test_twodim_base.py A numpy/numpy/lib/tests/test_type_check.py A numpy/numpy/lib/tests/test_function_base.py A numpy/numpy/lib/tests/test_getlimits.py A numpy/numpy/lib/tests/test_polynomial.py A numpy/numpy/lib/tests/test_ufunclike.py A numpy/numpy/lib/tests/test_index_tricks.py A numpy/numpy/lib/mlab.py A numpy/numpy/lib/convertcode.py A numpy/numpy/lib/arraysetops.py A numpy/numpy/lib/UserArray.py A numpy/numpy/lib/type_check.py A numpy/numpy/lib/function_base.py A numpy/numpy/lib/polynomial.py A numpy/numpy/lib/ufunclike.py A numpy/numpy/lib/index_tricks.py A numpy/numpy/linalg A numpy/numpy/linalg/blas_lite.c A numpy/numpy/linalg/lapack_litemodule.c A numpy/numpy/linalg/f2c.h A numpy/numpy/linalg/info.py A numpy/numpy/linalg/zlapack_lite.c A numpy/numpy/linalg/old.py A numpy/numpy/linalg/__init__.py A numpy/numpy/linalg/setup.py A numpy/numpy/linalg/f2c_lite.c A numpy/numpy/linalg/dlamch.c A numpy/numpy/linalg/dlapack_lite.c A numpy/numpy/linalg/linalg.py A numpy/numpy/setup.py A numpy/numpy/core A numpy/numpy/core/info.py A numpy/numpy/core/defchararray.py A numpy/numpy/core/arrayprint.py A numpy/numpy/core/include A numpy/numpy/core/include/numpy A numpy/numpy/core/include/numpy/arrayobject.h A numpy/numpy/core/include/numpy/arrayscalars.h A numpy/numpy/core/include/numpy/ufuncobject.h A numpy/numpy/core/include/numpy/fenv A numpy/numpy/core/include/numpy/fenv/fenv.c A numpy/numpy/core/include/numpy/fenv/fenv.h A numpy/numpy/core/ma.py A numpy/numpy/core/__init__.py A numpy/numpy/core/setup.py A numpy/numpy/core/src A numpy/numpy/core/src/_signbit.c A numpy/numpy/core/src/multiarraymodule.c A numpy/numpy/core/src/arraytypes.inc.src A numpy/numpy/core/src/_sortmodule.c.src A numpy/numpy/core/src/arraymethods.c A numpy/numpy/core/src/ucsnarrow.c A numpy/numpy/core/src/arrayobject.c A numpy/numpy/core/src/_isnan.c A numpy/numpy/core/src/scalartypes.inc.src A numpy/numpy/core/src/ufuncobject.c A numpy/numpy/core/src/umathmodule.c.src A numpy/numpy/core/src/scalarmathmodule.c.src A numpy/numpy/core/records.py A numpy/numpy/core/oldnumeric.py A numpy/numpy/core/blasdot A numpy/numpy/core/blasdot/_dotblas.c A numpy/numpy/core/blasdot/cblas.h A numpy/numpy/core/numeric.py A numpy/numpy/core/_internal.py A numpy/numpy/core/tests A numpy/numpy/core/tests/test_multiarray.py A numpy/numpy/core/tests/test_ma.py A numpy/numpy/core/tests/test_umath.py A numpy/numpy/core/tests/test_oldnumeric.py A numpy/numpy/core/tests/test_records.py A numpy/numpy/core/tests/test_numeric.py A numpy/numpy/core/tests/test_defmatrix.py A numpy/numpy/core/tests/test_unicode.py A numpy/numpy/core/tests/test_numerictypes.py A numpy/numpy/core/tests/testdata.fits A numpy/numpy/core/memmap.py A numpy/numpy/core/code_generators A numpy/numpy/core/code_generators/generate_umath.py A numpy/numpy/core/code_generators/array_api_order.txt A numpy/numpy/core/code_generators/ufunc_api_order.txt A numpy/numpy/core/code_generators/generate_array_api.py A numpy/numpy/core/code_generators/genapi.py A numpy/numpy/core/code_generators/generate_ufunc_api.py A numpy/numpy/core/code_generators/multiarray_api_order.txt A numpy/numpy/core/defmatrix.py A numpy/numpy/core/numerictypes.py A numpy/numpy/dual.py A numpy/numpy/version.py A numpy/numpy/dft A numpy/numpy/dft/fftpack.c A numpy/numpy/dft/fftpack_litemodule.c A numpy/numpy/dft/info.py A numpy/numpy/dft/tests A numpy/numpy/dft/tests/test_helper.py A numpy/numpy/dft/fftpack.h A numpy/numpy/dft/fftpack.py A numpy/numpy/dft/__init__.py A numpy/numpy/dft/helper.py A numpy/numpy/dft/setup.py A numpy/numpy/testing A numpy/numpy/testing/numpytest.py A numpy/numpy/testing/info.py A numpy/numpy/testing/__init__.py A numpy/numpy/testing/setup.py A numpy/numpy/testing/utils.py A numpy/COMPATIBILITY A numpy/DEV_README.txt A numpy/MANIFEST.in A numpy/README.txt A numpy/benchmarks A numpy/benchmarks/sorting.py U numpy Checked out revision 2259. ### Start build. ### /data/basil5/numpy build: No such file or directory Running from numpy source directory. Warning: not existing path in numpy/distutils: site.cfg /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1264: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['blas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1273: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:1276: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack_atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1183: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /data/basil5/numpy/numpy/distutils/system_info.py:1207: UserWarning: Blas (http://www.netlib.org/blas/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [blas]) or by setting the BLAS environment variable. warnings.warn(BlasNotFoundError.__doc__) /data/basil5/numpy/numpy/distutils/system_info.py:1210: UserWarning: Blas (http://www.netlib.org/blas/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [blas_src]) or by setting the BLAS_SRC environment variable. warnings.warn(BlasSrcNotFoundError.__doc__) No module named __svn_version__ F2PY Version 2_2259 blas_opt_info: blas_mkl_info: NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_info: NOT AVAILABLE blas_src_info: NOT AVAILABLE NOT AVAILABLE lapack_opt_info: lapack_mkl_info: mkl_info: NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: numpy.distutils.system_info.atlas_info NOT AVAILABLE lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/usr/local/lib'] language = f77 NOT AVAILABLE running build running config_fc running build_src building py_modules sources creating build creating build/src creating build/src/numpy creating build/src/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src/numpy/core Generating build/src/numpy/core/config.h customize SunFCompiler compiler version not matched (None) customize GnuFCompiler customize Gnu95FCompiler customize G95FCompiler customize GnuFCompiler customize Gnu95FCompiler customize SunFCompiler Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 69, in setup_package setup( **config.todict() ) File "/data/basil5/numpy/numpy/distutils/core.py", line 85, in setup return old_setup(**new_attr) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 84, in run self.build_sources() File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 99, in build_sources self.build_extension_sources(ext) File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 209, in build_extension_sources sources = self.generate_sources(sources, ext) File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line 267, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 35, in generate_config_h library_dirs = default_lib_dirs) File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/config.py", line 278, in try_run self._check_compiler() File "/data/basil5/numpy/numpy/distutils/command/config.py", line 35, in _check_compiler self.fcompiler.customize(self.distribution) File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line 281, in customize (conf,'f77flags')) File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line 511, in __get_flags var = command() File "/data/basil5/numpy/numpy/distutils/fcompiler/sun.py", line 27, in get_flags_f77 if (self.get_version() or '') >= '7': File "/data/basil5/numpy/numpy/distutils/ccompiler.py", line 256, in CCompiler_get_version raise ValueError("compiler version not matched (%r)" % (version,)) ValueError: compiler version not matched (None) From tim.hochberg at cox.net Mon Mar 20 07:58:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 20 07:58:18 2006 Subject: [Numpy-discussion] complex comparisons Message-ID: <441ED0EC.8030207@cox.net> I just noticed that complex arrays can be compared using [<, <=, > and >=]. Do we really want this? I'd prefer to follow numarray's lead here and only support [==, !=] on complex arrays. -tim From pfdubois at gmail.com Mon Mar 20 08:04:07 2006 From: pfdubois at gmail.com (Paul Dubois) Date: Mon Mar 20 08:04:07 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441ED0EC.8030207@cox.net> References: <441ED0EC.8030207@cox.net> Message-ID: Plus ca change, plus c'est la meme chose. How many times have we discussed this? I'm with Tim. In fact of course using even the equality operators is often a mistake, but at least there you can think of a use case. On 20 Mar 2006 07:58:54 -0800, Tim Hochberg wrote: > > > I just noticed that complex arrays can be compared using [<, <=, > and > >=]. Do we really want this? I'd prefer to follow numarray's lead here > and only support [==, !=] on complex arrays. > > -tim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From schofield at ftw.at Mon Mar 20 08:26:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Mon Mar 20 08:26:02 2006 Subject: [Numpy-discussion] NumPy 0.9.6-r1? Message-ID: <441ED79C.8010001@ftw.at> Hi Travis, I've re-built NumPy 0.9.6 for Win32 / Py2.4 after disabling the block in testing/utils.py that requires win32all. Let me know if you'd like me to upload it to the Wiki as NumPy 0.9.6-r1. It's 66kb smaller than your .exe file, which I don't quite understand, but otherwise seems to work fine. I've built it against ATLAS-3.6.0-PII from http://old.scipy.org. -- Ed From perry at stsci.edu Mon Mar 20 09:03:11 2006 From: perry at stsci.edu (Perry Greenfield) Date: Mon Mar 20 09:03:11 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: References: <441ED0EC.8030207@cox.net> Message-ID: Let me throw my 2 cents in as well. As Paul mentions this has been discussed before. It seems nonsensical to allow these comparisons since there is bound to be some confusion about their meaning no matter what is chosen. And if someone would like their code to be generic with regard to complex numbers and regular floats, then they should use .real, .imag, and abs() explicitly to be precise about what they are comparing. All these will work with both kinds of types. Perry On Mar 20, 2006, at 11:03 AM, Paul Dubois wrote: > Plus ca change, plus c'est la meme chose. How many times have we > discussed this? I'm with Tim. In fact of course using even the > equality operators is often a mistake, but at least there you can > think of a use case. > > On 20 Mar 2006 07:58:54 -0800, Tim Hochberg > wrote: >> I just noticed that complex arrays can be compared using [<, <=, > and >> >=]. Do we really want??this? I'd prefer to follow numarray's lead >> here >> and only support [==, !=] on complex arrays. >> From oliphant.travis at ieee.org Mon Mar 20 10:12:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 10:12:02 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441ED0EC.8030207@cox.net> References: <441ED0EC.8030207@cox.net> Message-ID: <441EF024.2000602@ieee.org> Tim Hochberg wrote: > > I just noticed that complex arrays can be compared using [<, <=, > and > >=]. Do we really want this? I'd prefer to follow numarray's lead > here and only support [==, !=] on complex arrays. > MATLAB has always provided complex comparisons by comparing the real part, so there is signifcant precedence for this. Anybody coming from MATLAB will appreciate it because while technically complex numbers are not ordered, often complex arrays are the result of calculations that result in real numbers but the data-type is still complex. But, now that .real and .imag work on *all* array data-types it is not an absolute necessity to support complex comparisons. I just think it's convenient and not especially confusing. -Travis From Fernando.Perez at colorado.edu Mon Mar 20 10:15:27 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 20 10:15:27 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441EF024.2000602@ieee.org> References: <441ED0EC.8030207@cox.net> <441EF024.2000602@ieee.org> Message-ID: <441EF0F2.8070009@colorado.edu> Travis Oliphant wrote: > Tim Hochberg wrote: > >>I just noticed that complex arrays can be compared using [<, <=, > and >> >>>=]. Do we really want this? I'd prefer to follow numarray's lead >> >>here and only support [==, !=] on complex arrays. >> > > MATLAB has always provided complex comparisons by comparing the real > part, so there is signifcant precedence for this. > > Anybody coming from MATLAB will appreciate it because while technically > complex numbers are not ordered, often complex arrays are the result of > calculations that result in real numbers but the data-type is still > complex. > > But, now that .real and .imag work on *all* array data-types it is not > an absolute necessity to support complex comparisons. I just think it's > convenient and not especially confusing. Well, I think this is one of those cases where we should strive more for uniformity with /Python/ than with matlab: In [18]: 1j<2j --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/fperez/ TypeError: cannot compare complex numbers using <, <=, >, >= Just my 1e-2j :) f From tim.hochberg at cox.net Mon Mar 20 10:24:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 20 10:24:02 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441EF024.2000602@ieee.org> References: <441ED0EC.8030207@cox.net> <441EF024.2000602@ieee.org> Message-ID: <441EF31C.1000307@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> I just noticed that complex arrays can be compared using [<, <=, > >> and >=]. Do we really want this? I'd prefer to follow numarray's >> lead here and only support [==, !=] on complex arrays. >> > MATLAB has always provided complex comparisons by comparing the real > part, so there is signifcant precedence for this. > Anybody coming from MATLAB will appreciate it because while > technically complex numbers are not ordered, often complex arrays are > the result of calculations that result in real numbers but the > data-type is still complex. > But, now that .real and .imag work on *all* array data-types it is not > an absolute necessity to support complex comparisons. I just think > it's convenient and not especially confusing. FWIW, it confused me. My first guess was that it was a bug. My second guess that it was comparing complex numbers by their magnitude. It was only after I dug into the code that I realized that it was comparing real, then imaginary. (I suppose I could have tried to look in the docs too, but it somehow seemed easier just to look in umathmodule). Although I use complex numbers all the time, I'm not going to use this, so in my case it's just a chance for errors to sneak into my code that would otherwise be caught. Much as I love MATLAB people, heck some of my best friends are MATLAB people, I don't think this is a place we should be catering to them. My $0.02. -tim From mahs at telcopartners.com Mon Mar 20 12:21:07 2006 From: mahs at telcopartners.com (Michael Spencer) Date: Mon Mar 20 12:21:07 2006 Subject: [Numpy-discussion] Re: Histograms via indirect index arrays In-Reply-To: <441CE4DE.6040101@ieee.org> References: <200603171519.35502.luszczek@cs.utk.edu> <200603171630.27075.luszczek@cs.utk.edu> <441CE4DE.6040101@ieee.org> Message-ID: Travis Oliphant wrote: > Piotr Luszczek wrote: > > The reason behind this as Tim and Rick later illustrate is that no one > quite knows how to produce "view" semantics for an arbitrarily indexed > array. I did actually think about this for a while because I don't > completely like the different behaviors which can produce warts like the > one that Tim showed us. > I realized that it would be possible to extend our memory model of an > array to include an index array in the array structure itself that would > indicate how to advance through memory (basically a pointer or its > equivalent for each array element). Rick showed that because this > would have to be a "copy" of the indices and so would create too much > slowness. My reason for not going there was because it seemed to add > too much complication for my tastes at the time to all the code. > FWIW, I implemented a pure-python ndarray, independent of numpy but with a similar interface. pyarray preserves views for arbitrary indexings, by carrying around an index array as you suggest. The reason for mentioning this here, is that looking at it might help someone think about desired semantics of views. pyarray: http://svn.brownspencer.com/pyarray/trunk/ Michael From seb at ist.utl.pt Mon Mar 20 12:47:02 2006 From: seb at ist.utl.pt (Sebastien Fabbro) Date: Mon Mar 20 12:47:02 2006 Subject: [Numpy-discussion] gentoo numpy/scipy Message-ID: <441F13CF.5070009@ist.utl.pt> Hello all, For those interested, I try to maintain recent versions of numpy, scipy and some other science related python packages for Gentoo Linux in http://gentooscience.org. All of these packages will end up in the standard Gentoo tree at some point. They are experimental to test recent features or can't wait for gentoo stabilization process. On a related subject, the numpy/scipy building process could really benefit some up-to-date, documentation package maintainer friendly, e.g. a site.cfg commented example. Regards, Sebastien From chanley at stsci.edu Mon Mar 20 13:50:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 20 13:50:02 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: <441EB6E8.8030504@stsci.edu> References: <441EB6E8.8030504@stsci.edu> Message-ID: <441F2349.1050006@stsci.edu> To provide a little more information, I do not use a custom site.cfg file for this platform. All files are deleted prior to the checkout from subversion so that this is a completely clean install. The compiler I am using is the native Sun "CC" compiler from SUNWspro-6u2. Chris Christopher Hanley wrote: > Solaris builds began failing yesterday with the following output: > > Your "cron" job on basil.stsci.edu > /home/chanley/bin/numpy_update.csh > > produced the following output: > > Sun Mar 19 07:20:04 EST 2006 > ### NUMPY Daily Build / Build Test ### > > > ### Setup test environment. ### > > /usr/ra/pyssg/Python-2.4.2/bin/python > > ### Checkout, build, and install new version of numpy. ### > > /data/basil5 > A numpy/LICENSE.txt > A numpy/scipy_compatibility > A numpy/TODO.txt > A numpy/THANKS.txt > A numpy/TEST_COMMIT > A numpy/setup.py > A numpy/numpy > A numpy/numpy/_import_tools.py > A numpy/numpy/f2py > A numpy/numpy/f2py/f2py.1 > A numpy/numpy/f2py/diagnose.py > A numpy/numpy/f2py/rules.py > A numpy/numpy/f2py/info.py > A numpy/numpy/f2py/capi_maps.py > A numpy/numpy/f2py/auxfuncs.py > A numpy/numpy/f2py/TODO.txt > A numpy/numpy/f2py/cb_rules.py > A numpy/numpy/f2py/__init__.py > A numpy/numpy/f2py/setup.py > A numpy/numpy/f2py/src > A numpy/numpy/f2py/src/test > A numpy/numpy/f2py/src/test/foo.f > A numpy/numpy/f2py/src/test/bar.f > A numpy/numpy/f2py/src/test/foo90.f90 > A numpy/numpy/f2py/src/test/foomodule.c > A numpy/numpy/f2py/src/test/Makefile > A numpy/numpy/f2py/src/test/wrap.f > A numpy/numpy/f2py/src/fortranobject.c > A numpy/numpy/f2py/src/fortranobject.h > A numpy/numpy/f2py/docs > A numpy/numpy/f2py/docs/HISTORY.txt > A numpy/numpy/f2py/docs/pytest.py > A numpy/numpy/f2py/docs/usersguide > A numpy/numpy/f2py/docs/usersguide/scalar_session.dat > A numpy/numpy/f2py/docs/usersguide/fib2.pyf > A numpy/numpy/f2py/docs/usersguide/callback.f > A numpy/numpy/f2py/docs/usersguide/calculate.f > A numpy/numpy/f2py/docs/usersguide/run_main_session.dat > A numpy/numpy/f2py/docs/usersguide/index.txt > A numpy/numpy/f2py/docs/usersguide/var.pyf > A numpy/numpy/f2py/docs/usersguide/string.f > A numpy/numpy/f2py/docs/usersguide/spam.pyf > A numpy/numpy/f2py/docs/usersguide/ftype_session.dat > A numpy/numpy/f2py/docs/usersguide/extcallback.f > A numpy/numpy/f2py/docs/usersguide/setup_example.py > A numpy/numpy/f2py/docs/usersguide/array.f > A numpy/numpy/f2py/docs/usersguide/common_session.dat > A numpy/numpy/f2py/docs/usersguide/var_session.dat > A numpy/numpy/f2py/docs/usersguide/moddata.f90 > A numpy/numpy/f2py/docs/usersguide/callback_session.dat > A numpy/numpy/f2py/docs/usersguide/calculate_session.dat > A numpy/numpy/f2py/docs/usersguide/spam_session.dat > A numpy/numpy/f2py/docs/usersguide/docutils.conf > A numpy/numpy/f2py/docs/usersguide/callback2.pyf > A numpy/numpy/f2py/docs/usersguide/string_session.dat > A numpy/numpy/f2py/docs/usersguide/moddata_session.dat > A numpy/numpy/f2py/docs/usersguide/extcallback_session.dat > A numpy/numpy/f2py/docs/usersguide/array_session.dat > A numpy/numpy/f2py/docs/usersguide/scalar.f > A numpy/numpy/f2py/docs/usersguide/allocarr.f90 > A numpy/numpy/f2py/docs/usersguide/compile_session.dat > A numpy/numpy/f2py/docs/usersguide/fib1.f > A numpy/numpy/f2py/docs/usersguide/fib3.f > A numpy/numpy/f2py/docs/usersguide/allocarr_session.dat > A numpy/numpy/f2py/docs/usersguide/ftype.f > A numpy/numpy/f2py/docs/usersguide/default.css > A numpy/numpy/f2py/docs/usersguide/common.f > A numpy/numpy/f2py/docs/usersguide/fib1.pyf > A numpy/numpy/f2py/docs/pyforttest.pyf > A numpy/numpy/f2py/docs/simple_session.dat > A numpy/numpy/f2py/docs/FAQ.txt > A numpy/numpy/f2py/docs/THANKS.txt > A numpy/numpy/f2py/docs/hello.f > A numpy/numpy/f2py/docs/OLDNEWS.txt > A numpy/numpy/f2py/docs/docutils.conf > A numpy/numpy/f2py/docs/README.txt > A numpy/numpy/f2py/docs/TESTING.txt > A numpy/numpy/f2py/docs/default.css > A numpy/numpy/f2py/docs/simple.f > A numpy/numpy/f2py/common_rules.py > A numpy/numpy/f2py/NEWS.txt > A numpy/numpy/f2py/use_rules.py > A numpy/numpy/f2py/f2py2e.py > A numpy/numpy/f2py/setup.cfg > A numpy/numpy/f2py/f90mod_rules.py > A numpy/numpy/f2py/func2subr.py > A numpy/numpy/f2py/tests > A numpy/numpy/f2py/tests/c > A numpy/numpy/f2py/tests/c/return_real.py > A numpy/numpy/f2py/tests/f77 > A numpy/numpy/f2py/tests/f77/return_logical.py > A numpy/numpy/f2py/tests/f77/return_character.py > A numpy/numpy/f2py/tests/f77/callback.py > A numpy/numpy/f2py/tests/f77/return_integer.py > A numpy/numpy/f2py/tests/f77/return_real.py > A numpy/numpy/f2py/tests/f77/return_complex.py > A numpy/numpy/f2py/tests/run_all.py > A numpy/numpy/f2py/tests/array_from_pyobj > A numpy/numpy/f2py/tests/array_from_pyobj/wrapmodule.c > A numpy/numpy/f2py/tests/array_from_pyobj/tests > A numpy/numpy/f2py/tests/array_from_pyobj/tests/test_array_from_pyobj.py > A numpy/numpy/f2py/tests/array_from_pyobj/__init__.py > A numpy/numpy/f2py/tests/array_from_pyobj/setup.py > A numpy/numpy/f2py/tests/mixed > A numpy/numpy/f2py/tests/mixed/foo_fixed.f90 > A numpy/numpy/f2py/tests/mixed/foo_free.f90 > A numpy/numpy/f2py/tests/mixed/foo.f > A numpy/numpy/f2py/tests/mixed/run.py > A numpy/numpy/f2py/tests/f90 > A numpy/numpy/f2py/tests/f90/return_logical.py > A numpy/numpy/f2py/tests/f90/return_character.py > A numpy/numpy/f2py/tests/f90/return_integer.py > A numpy/numpy/f2py/tests/f90/return_real.py > A numpy/numpy/f2py/tests/f90/return_complex.py > A numpy/numpy/f2py/f2py_testing.py > A numpy/numpy/f2py/doc > A numpy/numpy/f2py/doc/notes.tex > A numpy/numpy/f2py/doc/win32_notes.txt > A numpy/numpy/f2py/doc/multiarray > A numpy/numpy/f2py/doc/multiarray/run.pyf > A numpy/numpy/f2py/doc/multiarray/array_from_pyobj.c > A numpy/numpy/f2py/doc/multiarray/bar.c > A numpy/numpy/f2py/doc/multiarray/fun.pyf > A numpy/numpy/f2py/doc/multiarray/foo.f > A numpy/numpy/f2py/doc/multiarray/transpose.txt > A numpy/numpy/f2py/doc/multiarray/fortran_array_from_pyobj.txt > A numpy/numpy/f2py/doc/f2python9-final > A numpy/numpy/f2py/doc/f2python9-final/structure.jpg > A numpy/numpy/f2py/doc/f2python9-final/mk_ps.sh > A numpy/numpy/f2py/doc/f2python9-final/mk_html.sh > A numpy/numpy/f2py/doc/f2python9-final/aerostructure.jpg > A numpy/numpy/f2py/doc/f2python9-final/flow.jpg > A numpy/numpy/f2py/doc/f2python9-final/src > A numpy/numpy/f2py/doc/f2python9-final/src/examples > A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1session.txt > A numpy/numpy/f2py/doc/f2python9-final/src/examples/foo.pyf > A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1.f > A numpy/numpy/f2py/doc/f2python9-final/src/examples/exp1mess.txt > A numpy/numpy/f2py/doc/f2python9-final/src/examples/foom.pyf > A numpy/numpy/f2py/doc/f2python9-final/mk_pdf.sh > A numpy/numpy/f2py/doc/f2python9-final/README.txt > A numpy/numpy/f2py/doc/intro.tex > A numpy/numpy/f2py/doc/multiarrays.txt > A numpy/numpy/f2py/doc/options.tex > A numpy/numpy/f2py/doc/collectinput.py > A numpy/numpy/f2py/doc/ex1 > A numpy/numpy/f2py/doc/ex1/foobar.f90 > A numpy/numpy/f2py/doc/ex1/runme > A numpy/numpy/f2py/doc/ex1/foo.f > A numpy/numpy/f2py/doc/ex1/arr.f > A numpy/numpy/f2py/doc/ex1/bar.f > A numpy/numpy/f2py/doc/ex1/foobar-smart.f90 > A numpy/numpy/f2py/doc/ex1/foobarmodule.tex > A numpy/numpy/f2py/doc/bugs.tex > A numpy/numpy/f2py/doc/commands.tex > A numpy/numpy/f2py/doc/signaturefile.tex > A numpy/numpy/f2py/doc/fortranobject.tex > A numpy/numpy/f2py/doc/apps.tex > A numpy/numpy/f2py/doc/Release-1.x.txt > A numpy/numpy/f2py/doc/using_F_compiler.txt > A numpy/numpy/f2py/doc/Release-2.x.txt > A numpy/numpy/f2py/doc/Release-3.x.txt > A numpy/numpy/f2py/doc/Release-4.x.txt > A numpy/numpy/f2py/doc/f2py2e.tex > A numpy/numpy/f2py/doc/python9.tex > A numpy/numpy/f2py/doc/index.html > A numpy/numpy/f2py/doc/Makefile > A numpy/numpy/f2py/doc/oldnews.html > A numpy/numpy/f2py/crackfortran.py > A numpy/numpy/f2py/cfuncs.py > A numpy/numpy/f2py/__version__.py > A numpy/numpy/f2py/README.txt > A numpy/numpy/f2py/Makefile > A numpy/numpy/f2py/BUGS.txt > A numpy/numpy/random > A numpy/numpy/random/info.py > A numpy/numpy/random/mtrand > A numpy/numpy/random/mtrand/Python.pxi > A numpy/numpy/random/mtrand/distributions.c > A numpy/numpy/random/mtrand/initarray.c > A numpy/numpy/random/mtrand/mtrand.pyx > A numpy/numpy/random/mtrand/mtrand.c > A numpy/numpy/random/mtrand/numpy.pxi > A numpy/numpy/random/mtrand/distributions.h > A numpy/numpy/random/mtrand/generate_mtrand_c.py > A numpy/numpy/random/mtrand/initarray.h > A numpy/numpy/random/mtrand/randomkit.c > A numpy/numpy/random/mtrand/randomkit.h > A numpy/numpy/random/__init__.py > A numpy/numpy/random/setup.py > A numpy/numpy/add_newdocs.py > A numpy/numpy/distutils > A numpy/numpy/distutils/core.py > A numpy/numpy/distutils/misc_util.py > A numpy/numpy/distutils/fcompiler > A numpy/numpy/distutils/fcompiler/mips.py > A numpy/numpy/distutils/fcompiler/gnu.py > A numpy/numpy/distutils/fcompiler/intel.py > A numpy/numpy/distutils/fcompiler/vast.py > A numpy/numpy/distutils/fcompiler/absoft.py > A numpy/numpy/distutils/fcompiler/__init__.py > A numpy/numpy/distutils/fcompiler/none.py > A numpy/numpy/distutils/fcompiler/compaq.py > A numpy/numpy/distutils/fcompiler/lahey.py > A numpy/numpy/distutils/fcompiler/g95.py > A numpy/numpy/distutils/fcompiler/hpux.py > A numpy/numpy/distutils/fcompiler/nag.py > A numpy/numpy/distutils/fcompiler/sun.py > A numpy/numpy/distutils/fcompiler/pg.py > A numpy/numpy/distutils/fcompiler/ibm.py > A numpy/numpy/distutils/info.py > A numpy/numpy/distutils/line_endings.py > A numpy/numpy/distutils/from_template.py > A numpy/numpy/distutils/__init__.py > A numpy/numpy/distutils/system_info.py > A numpy/numpy/distutils/conv_template.py > A numpy/numpy/distutils/setup.py > A numpy/numpy/distutils/cpuinfo.py > A numpy/numpy/distutils/lib2def.py > A numpy/numpy/distutils/intelccompiler.py > A numpy/numpy/distutils/tests > A numpy/numpy/distutils/tests/f2py_ext > A numpy/numpy/distutils/tests/f2py_ext/tests > A numpy/numpy/distutils/tests/f2py_ext/tests/test_fib2.py > A numpy/numpy/distutils/tests/f2py_ext/__init__.py > A numpy/numpy/distutils/tests/f2py_ext/setup.py > A numpy/numpy/distutils/tests/f2py_ext/src > A numpy/numpy/distutils/tests/f2py_ext/src/fib2.pyf > A numpy/numpy/distutils/tests/f2py_ext/src/fib1.f > A numpy/numpy/distutils/tests/pyrex_ext > A numpy/numpy/distutils/tests/pyrex_ext/tests > A numpy/numpy/distutils/tests/pyrex_ext/tests/test_primes.py > A numpy/numpy/distutils/tests/pyrex_ext/__init__.py > A numpy/numpy/distutils/tests/pyrex_ext/setup.py > A numpy/numpy/distutils/tests/pyrex_ext/primes.pyx > A numpy/numpy/distutils/tests/setup.py > A numpy/numpy/distutils/tests/gen_ext > A numpy/numpy/distutils/tests/gen_ext/tests > A numpy/numpy/distutils/tests/gen_ext/tests/test_fib3.py > A numpy/numpy/distutils/tests/gen_ext/__init__.py > A numpy/numpy/distutils/tests/gen_ext/setup.py > A numpy/numpy/distutils/tests/swig_ext > A numpy/numpy/distutils/tests/swig_ext/tests > A numpy/numpy/distutils/tests/swig_ext/tests/test_example.py > A numpy/numpy/distutils/tests/swig_ext/tests/test_example2.py > A numpy/numpy/distutils/tests/swig_ext/__init__.py > A numpy/numpy/distutils/tests/swig_ext/setup.py > A numpy/numpy/distutils/tests/swig_ext/src > A numpy/numpy/distutils/tests/swig_ext/src/example.i > A numpy/numpy/distutils/tests/swig_ext/src/zoo.cc > A numpy/numpy/distutils/tests/swig_ext/src/example.c > A numpy/numpy/distutils/tests/swig_ext/src/zoo.h > A numpy/numpy/distutils/tests/swig_ext/src/zoo.i > A numpy/numpy/distutils/tests/f2py_f90_ext > A numpy/numpy/distutils/tests/f2py_f90_ext/tests > A numpy/numpy/distutils/tests/f2py_f90_ext/tests/test_foo.py > A numpy/numpy/distutils/tests/f2py_f90_ext/include > A numpy/numpy/distutils/tests/f2py_f90_ext/include/body.f90 > A numpy/numpy/distutils/tests/f2py_f90_ext/__init__.py > A numpy/numpy/distutils/tests/f2py_f90_ext/setup.py > A numpy/numpy/distutils/tests/f2py_f90_ext/src > A numpy/numpy/distutils/tests/f2py_f90_ext/src/foo_free.f90 > A numpy/numpy/distutils/tests/test_misc_util.py > A numpy/numpy/distutils/extension.py > A numpy/numpy/distutils/ccompiler.py > A numpy/numpy/distutils/log.py > A numpy/numpy/distutils/__version__.py > A numpy/numpy/distutils/unixccompiler.py > A numpy/numpy/distutils/exec_command.py > A numpy/numpy/distutils/mingw32ccompiler.py > A numpy/numpy/distutils/command > A numpy/numpy/distutils/command/build_clib.py > A numpy/numpy/distutils/command/egg_info.py > A numpy/numpy/distutils/command/__init__.py > A numpy/numpy/distutils/command/build.py > A numpy/numpy/distutils/command/build_ext.py > A numpy/numpy/distutils/command/config_compiler.py > A numpy/numpy/distutils/command/install_data.py > A numpy/numpy/distutils/command/install_headers.py > A numpy/numpy/distutils/command/bdist_rpm.py > A numpy/numpy/distutils/command/config.py > A numpy/numpy/distutils/command/build_scripts.py > A numpy/numpy/distutils/command/build_src.py > A numpy/numpy/distutils/command/install.py > A numpy/numpy/distutils/command/sdist.py > A numpy/numpy/distutils/command/build_py.py > A numpy/numpy/doc > A numpy/numpy/doc/records.txt > A numpy/numpy/doc/ufuncs.txt > A numpy/numpy/doc/DISTUTILS.txt > A numpy/numpy/doc/pyrex > A numpy/numpy/doc/pyrex/numpyx.pyx > A numpy/numpy/doc/pyrex/numpyx.c > A numpy/numpy/doc/pyrex/c_numpy.pxd > A numpy/numpy/doc/pyrex/MANIFEST > A numpy/numpy/doc/pyrex/setup.py > A numpy/numpy/doc/pyrex/notes > A numpy/numpy/doc/pyrex/run_test.py > A numpy/numpy/doc/pyrex/Makefile > A numpy/numpy/doc/pyrex/c_python.pxd > A numpy/numpy/doc/swig > A numpy/numpy/doc/swig/numpy.i > A numpy/numpy/doc/swig/series.h > A numpy/numpy/doc/swig/Series.i > A numpy/numpy/doc/swig/testSeries.py > A numpy/numpy/doc/swig/HelperFunctions.txt > A numpy/numpy/doc/swig/setup.py > A numpy/numpy/doc/swig/series.cxx > A numpy/numpy/doc/swig/Makefile > A numpy/numpy/doc/swig/README > A numpy/numpy/doc/CAPI.txt > A numpy/numpy/doc/README.txt > A numpy/numpy/__init__.py > A numpy/numpy/lib > A numpy/numpy/lib/scimath.py > A numpy/numpy/lib/shape_base.py > A numpy/numpy/lib/machar.py > A numpy/numpy/lib/info.py > A numpy/numpy/lib/twodim_base.py > A numpy/numpy/lib/__init__.py > A numpy/numpy/lib/src > A numpy/numpy/lib/src/_compiled_base.c > A numpy/numpy/lib/setup.py > A numpy/numpy/lib/utils.py > A numpy/numpy/lib/getlimits.py > A numpy/numpy/lib/tests > A numpy/numpy/lib/tests/test_shape_base.py > A numpy/numpy/lib/tests/test_arraysetops.py > A numpy/numpy/lib/tests/test_twodim_base.py > A numpy/numpy/lib/tests/test_type_check.py > A numpy/numpy/lib/tests/test_function_base.py > A numpy/numpy/lib/tests/test_getlimits.py > A numpy/numpy/lib/tests/test_polynomial.py > A numpy/numpy/lib/tests/test_ufunclike.py > A numpy/numpy/lib/tests/test_index_tricks.py > A numpy/numpy/lib/mlab.py > A numpy/numpy/lib/convertcode.py > A numpy/numpy/lib/arraysetops.py > A numpy/numpy/lib/UserArray.py > A numpy/numpy/lib/type_check.py > A numpy/numpy/lib/function_base.py > A numpy/numpy/lib/polynomial.py > A numpy/numpy/lib/ufunclike.py > A numpy/numpy/lib/index_tricks.py > A numpy/numpy/linalg > A numpy/numpy/linalg/blas_lite.c > A numpy/numpy/linalg/lapack_litemodule.c > A numpy/numpy/linalg/f2c.h > A numpy/numpy/linalg/info.py > A numpy/numpy/linalg/zlapack_lite.c > A numpy/numpy/linalg/old.py > A numpy/numpy/linalg/__init__.py > A numpy/numpy/linalg/setup.py > A numpy/numpy/linalg/f2c_lite.c > A numpy/numpy/linalg/dlamch.c > A numpy/numpy/linalg/dlapack_lite.c > A numpy/numpy/linalg/linalg.py > A numpy/numpy/setup.py > A numpy/numpy/core > A numpy/numpy/core/info.py > A numpy/numpy/core/defchararray.py > A numpy/numpy/core/arrayprint.py > A numpy/numpy/core/include > A numpy/numpy/core/include/numpy > A numpy/numpy/core/include/numpy/arrayobject.h > A numpy/numpy/core/include/numpy/arrayscalars.h > A numpy/numpy/core/include/numpy/ufuncobject.h > A numpy/numpy/core/include/numpy/fenv > A numpy/numpy/core/include/numpy/fenv/fenv.c > A numpy/numpy/core/include/numpy/fenv/fenv.h > A numpy/numpy/core/ma.py > A numpy/numpy/core/__init__.py > A numpy/numpy/core/setup.py > A numpy/numpy/core/src > A numpy/numpy/core/src/_signbit.c > A numpy/numpy/core/src/multiarraymodule.c > A numpy/numpy/core/src/arraytypes.inc.src > A numpy/numpy/core/src/_sortmodule.c.src > A numpy/numpy/core/src/arraymethods.c > A numpy/numpy/core/src/ucsnarrow.c > A numpy/numpy/core/src/arrayobject.c > A numpy/numpy/core/src/_isnan.c > A numpy/numpy/core/src/scalartypes.inc.src > A numpy/numpy/core/src/ufuncobject.c > A numpy/numpy/core/src/umathmodule.c.src > A numpy/numpy/core/src/scalarmathmodule.c.src > A numpy/numpy/core/records.py > A numpy/numpy/core/oldnumeric.py > A numpy/numpy/core/blasdot > A numpy/numpy/core/blasdot/_dotblas.c > A numpy/numpy/core/blasdot/cblas.h > A numpy/numpy/core/numeric.py > A numpy/numpy/core/_internal.py > A numpy/numpy/core/tests > A numpy/numpy/core/tests/test_multiarray.py > A numpy/numpy/core/tests/test_ma.py > A numpy/numpy/core/tests/test_umath.py > A numpy/numpy/core/tests/test_oldnumeric.py > A numpy/numpy/core/tests/test_records.py > A numpy/numpy/core/tests/test_numeric.py > A numpy/numpy/core/tests/test_defmatrix.py > A numpy/numpy/core/tests/test_unicode.py > A numpy/numpy/core/tests/test_numerictypes.py > A numpy/numpy/core/tests/testdata.fits > A numpy/numpy/core/memmap.py > A numpy/numpy/core/code_generators > A numpy/numpy/core/code_generators/generate_umath.py > A numpy/numpy/core/code_generators/array_api_order.txt > A numpy/numpy/core/code_generators/ufunc_api_order.txt > A numpy/numpy/core/code_generators/generate_array_api.py > A numpy/numpy/core/code_generators/genapi.py > A numpy/numpy/core/code_generators/generate_ufunc_api.py > A numpy/numpy/core/code_generators/multiarray_api_order.txt > A numpy/numpy/core/defmatrix.py > A numpy/numpy/core/numerictypes.py > A numpy/numpy/dual.py > A numpy/numpy/version.py > A numpy/numpy/dft > A numpy/numpy/dft/fftpack.c > A numpy/numpy/dft/fftpack_litemodule.c > A numpy/numpy/dft/info.py > A numpy/numpy/dft/tests > A numpy/numpy/dft/tests/test_helper.py > A numpy/numpy/dft/fftpack.h > A numpy/numpy/dft/fftpack.py > A numpy/numpy/dft/__init__.py > A numpy/numpy/dft/helper.py > A numpy/numpy/dft/setup.py > A numpy/numpy/testing > A numpy/numpy/testing/numpytest.py > A numpy/numpy/testing/info.py > A numpy/numpy/testing/__init__.py > A numpy/numpy/testing/setup.py > A numpy/numpy/testing/utils.py > A numpy/COMPATIBILITY > A numpy/DEV_README.txt > A numpy/MANIFEST.in > A numpy/README.txt > A numpy/benchmarks > A numpy/benchmarks/sorting.py > U numpy > Checked out revision 2259. > > ### Start build. ### > > /data/basil5/numpy > build: No such file or directory > Running from numpy source directory. > Warning: not existing path in numpy/distutils: site.cfg > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['ptf77blas', 'ptcblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['f77blas', 'cblas', 'atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1264: UserWarning: > Atlas (http://math-atlas.sourceforge.net/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [atlas]) or by setting > the ATLAS environment variable. > warnings.warn(AtlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['blas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1273: UserWarning: > Blas (http://www.netlib.org/blas/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [blas]) or by setting > the BLAS environment variable. > warnings.warn(BlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:1276: UserWarning: > Blas (http://www.netlib.org/blas/) sources not found. > Directories to search for the sources can be specified in the > numpy/distutils/site.cfg file (section [blas_src]) or by setting > the BLAS_SRC environment variable. > warnings.warn(BlasSrcNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['lapack_atlas'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1183: UserWarning: > Atlas (http://math-atlas.sourceforge.net/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [atlas]) or by setting > the ATLAS environment variable. > warnings.warn(AtlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:531: UserWarning: > Library error: libs=['lapack'] found_libs=[] > warnings.warn("Library error: libs=%s found_libs=%s" % \ > /data/basil5/numpy/numpy/distutils/system_info.py:1207: UserWarning: > Blas (http://www.netlib.org/blas/) libraries not found. > Directories to search for the libraries can be specified in the > numpy/distutils/site.cfg file (section [blas]) or by setting > the BLAS environment variable. > warnings.warn(BlasNotFoundError.__doc__) > /data/basil5/numpy/numpy/distutils/system_info.py:1210: UserWarning: > Blas (http://www.netlib.org/blas/) sources not found. > Directories to search for the sources can be specified in the > numpy/distutils/site.cfg file (section [blas_src]) or by setting > the BLAS_SRC environment variable. > warnings.warn(BlasSrcNotFoundError.__doc__) > No module named __svn_version__ > F2PY Version 2_2259 > blas_opt_info: > blas_mkl_info: > NOT AVAILABLE > > atlas_blas_threads_info: > Setting PTATLAS=ATLAS > NOT AVAILABLE > > atlas_blas_info: > NOT AVAILABLE > > blas_info: > NOT AVAILABLE > > blas_src_info: > NOT AVAILABLE > > NOT AVAILABLE > > lapack_opt_info: > lapack_mkl_info: > mkl_info: > NOT AVAILABLE > > NOT AVAILABLE > > atlas_threads_info: > Setting PTATLAS=ATLAS > numpy.distutils.system_info.atlas_threads_info > NOT AVAILABLE > > atlas_info: > numpy.distutils.system_info.atlas_info > NOT AVAILABLE > > lapack_info: > FOUND: > libraries = ['lapack'] > library_dirs = ['/usr/local/lib'] > language = f77 > > NOT AVAILABLE > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src > creating build/src/numpy > creating build/src/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src/numpy/core > Generating build/src/numpy/core/config.h > customize SunFCompiler > compiler version not matched (None) > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize SunFCompiler > Traceback (most recent call last): > File "setup.py", line 76, in ? > setup_package() > File "setup.py", line 69, in setup_package > setup( **config.todict() ) > File "/data/basil5/numpy/numpy/distutils/core.py", line 85, in setup > return old_setup(**new_attr) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/core.py", > line 149, in setup > dist.run_commands() > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 946, in run_commands > self.run_command(cmd) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/build.py", > line 112, in run > self.run_command(cmd_name) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/cmd.py", line > 333, in run_command > self.distribution.run_command(command) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 84, in run > self.build_sources() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 99, in build_sources > self.build_extension_sources(ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 209, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 267, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 35, in generate_config_h > library_dirs = default_lib_dirs) > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/config.py", > line 278, in try_run > self._check_compiler() > File "/data/basil5/numpy/numpy/distutils/command/config.py", line 35, > in _check_compiler > self.fcompiler.customize(self.distribution) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line > 281, in customize > (conf,'f77flags')) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", line > 511, in __get_flags > var = command() > File "/data/basil5/numpy/numpy/distutils/fcompiler/sun.py", line 27, > in get_flags_f77 > if (self.get_version() or '') >= '7': > File "/data/basil5/numpy/numpy/distutils/ccompiler.py", line 256, in > CCompiler_get_version > raise ValueError("compiler version not matched (%r)" % (version,)) > ValueError: compiler version not matched (None) > From cookedm at physics.mcmaster.ca Mon Mar 20 14:29:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 20 14:29:06 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: <441EB6E8.8030504@stsci.edu> (Christopher Hanley's message of "Mon, 20 Mar 2006 09:06:32 -0500") References: <441EB6E8.8030504@stsci.edu> Message-ID: Christopher Hanley writes: > Solaris builds began failing yesterday with the following output: Looks like it's not matching the compiler version for your fortran compiler. What does "f90 -V" give you? It's trying to match against r'(f90|f95): (Sun|Forte Developer 7|WorkShop 6 update \d+) Fortran 95 (?P[^\s]+).*' so maybe that needs work. > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src > creating build/src/numpy > creating build/src/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src/numpy/core > Generating build/src/numpy/core/config.h > customize SunFCompiler > compiler version not matched (None) > customize GnuFCompiler > customize Gnu95FCompiler > customize G95FCompiler > customize GnuFCompiler > customize Gnu95FCompiler > customize SunFCompiler > Traceback (most recent call last): > File "setup.py", line 76, in ? > setup_package() > File "setup.py", line 69, in setup_package > setup( **config.todict() ) > File "/data/basil5/numpy/numpy/distutils/core.py", line 85, in setup > return old_setup(**new_attr) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/core.py", > line 149, in setup > dist.run_commands() > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 946, in run_commands > self.run_command(cmd) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/build.py", > line 112, in run > self.run_command(cmd_name) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/cmd.py", > line 333, in run_command > self.distribution.run_command(command) > File "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/dist.py", > line 966, in run_command > cmd_obj.run() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 84, in run > self.build_sources() > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 99, in build_sources > self.build_extension_sources(ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 209, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/data/basil5/numpy/numpy/distutils/command/build_src.py", line > 267, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 35, in generate_config_h > library_dirs = default_lib_dirs) > File > "/usr/ra/pyssg/Python-2.4.2/lib/python2.4/distutils/command/config.py", > line 278, in try_run > self._check_compiler() > File "/data/basil5/numpy/numpy/distutils/command/config.py", line > 35, in _check_compiler > self.fcompiler.customize(self.distribution) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", > line 281, in customize > (conf,'f77flags')) > File "/data/basil5/numpy/numpy/distutils/fcompiler/__init__.py", > line 511, in __get_flags > var = command() > File "/data/basil5/numpy/numpy/distutils/fcompiler/sun.py", line 27, > in get_flags_f77 > if (self.get_version() or '') >= '7': > File "/data/basil5/numpy/numpy/distutils/ccompiler.py", line 256, in > CCompiler_get_version > raise ValueError("compiler version not matched (%r)" % (version,)) > ValueError: compiler version not matched (None) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Mon Mar 20 14:30:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 20 14:30:03 2006 Subject: [Numpy-discussion] complex comparisons In-Reply-To: <441ED0EC.8030207@cox.net> (Tim Hochberg's message of "Mon, 20 Mar 2006 08:57:32 -0700") References: <441ED0EC.8030207@cox.net> Message-ID: Tim Hochberg writes: > I just noticed that complex arrays can be compared using [<, <=, > and >> =]. Do we really want this? I'd prefer to follow numarray's lead >> here > and only support [==, !=] on complex arrays. +1 -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Mon Mar 20 16:16:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 20 16:16:04 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: References: <441EB6E8.8030504@stsci.edu> Message-ID: <441F4591.2050707@ee.byu.edu> David M. Cooke wrote: >Christopher Hanley writes: > > > >>Solaris builds began failing yesterday with the following output: >> >> > >Looks like it's not matching the compiler version for your fortran >compiler. What does "f90 -V" give you? > > This should fail more gracefully, especially since a fortran compiler is not even needed for numpy to get installed. This needs to be fixed in NumPy. -Travis From zpincus at stanford.edu Mon Mar 20 16:37:02 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 20 16:37:02 2006 Subject: [Numpy-discussion] linalg.svd returns fortran arrays, causes problems Message-ID: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Hi folks, I recently updated to the SVN head of numpy, and most of my data analysis code broke! Specifically, this code involved computing the SVD of flattened multidimensional data, and then reshaping the results back to the correct dimensionality. (A very common operation.) Now these reshape operations don't generate the right results. It turns out that this is due to either (a) linalg.svd is returning fortran-style arrays when it used to return c-style arrays, or (b) numpy.reshape used to convert fortran-style arrays to c-style before reshaping, and now does not. It seems to me that (a) is more likely. If so, is this a bug or a feature? I personally would argue that returning fortran-style arrays from any functions, unless specifically requested, is a serious bug, precisely because of problems like the above. Any thoughts? Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine From robert.kern at gmail.com Mon Mar 20 16:47:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 16:47:02 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: Zachary Pincus wrote: > Hi folks, > > I recently updated to the SVN head of numpy, and most of my data > analysis code broke! Specifically, this code involved computing the SVD > of flattened multidimensional data, and then reshaping the results back > to the correct dimensionality. (A very common operation.) Now these > reshape operations don't generate the right results. Can you give us some small, self-contained code that demonstrates the problem? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Mar 20 16:52:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 16:52:03 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: Zachary Pincus wrote: > Hi folks, > > I recently updated to the SVN head of numpy, and most of my data > analysis code broke! And what version of numpy *did* work for you? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Mon Mar 20 17:15:26 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 20 17:15:26 2006 Subject: [Numpy-discussion] linalg.svd returns fortran arrays, causes problems In-Reply-To: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F537D.1070705@ee.byu.edu> Zachary Pincus wrote: > Hi folks, > > I recently updated to the SVN head of numpy, and most of my data > analysis code broke! Specifically, this code involved computing the > SVD of flattened multidimensional data, and then reshaping the > results back to the correct dimensionality. (A very common > operation.) Now these reshape operations don't generate the right > results. > The Python user should not care whether the arrays are fortran-order or c-order (it's just a matter of different striding). If it does matter, then it is a bug that must be fixed. Please show us code that is not working. If you upgraded to SVN numpy, then it's more likely to be related to the change to the flags object (now it's a builtin). -Travis From zpincus at stanford.edu Mon Mar 20 17:18:15 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 20 17:18:15 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: > Can you give us some small, self-contained code that demonstrates > the problem? u, s, vt = numpy.linalg.svd([[1,2],[3,4]]) u.flags.fortran True The problem, as per my original email, is that linalg.svd seems to be now returning fortran-style arrays. I believe that it did not do this in the past, but I am not certain. It is also possible that linalg.svd always returned fortran arrays, and the behavior of numpy.reshape with regard to fortran arrays has changed. Here is an example of how numpy.reshape differs with fortran and c-style arrays. I do not know if this is a bug, or the desired behavior. numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = False), [2, 5, 2]) array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = True), [2, 5, 2]) array([[[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]], [[11, 16], [12, 17], [13, 18], [14, 19], [15, 20]]]) In any case, there are two issues at play: (1) Should any numpy functions be allowed to return fortran arrays unless specifically requested? (2) Should all structural operations like reshape (etc.) behave the same for fortran and c-style arrays? If 2 can be guaranteed, then (1) doesn't really matter, I think. But I'm not sure if 2 should be guaranteed or not. > And what version of numpy *did* work for you? I do not recall, unfortunately. It was a svn checkout not more than two weeks old. Again, what I'm reporting isn't *necessarily* a bug, but at least one (if not two) changes in semantics which are at least problematic, and at most a bug. Zach On Mar 20, 2006, at 4:45 PM, Robert Kern wrote: > Zachary Pincus wrote: >> Hi folks, >> >> I recently updated to the SVN head of numpy, and most of my data >> analysis code broke! Specifically, this code involved computing >> the SVD >> of flattened multidimensional data, and then reshaping the >> results back >> to the correct dimensionality. (A very common operation.) Now these >> reshape operations don't generate the right results. > > Can you give us some small, self-contained code that demonstrates > the problem? > > -- > Robert Kern > robert.kern at gmail.com > > "I have come to believe that the whole world is an enigma, a > harmless enigma > that is made terrible by our own mad attempt to interpret it as > though it had > an underlying truth." > -- Umberto Eco > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From robert.kern at gmail.com Mon Mar 20 17:26:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 17:26:07 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: Zachary Pincus wrote: > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = False), [2, 5, 2]) > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = True), [2, 5, 2]) > array([[[ 1, 6], > [ 2, 7], > [ 3, 8], > [ 4, 9], > [ 5, 10]], > > [[11, 16], > [12, 17], > [13, 18], > [14, 19], > [15, 20]]]) I think this is a new bug, then. In [82]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = True), (2,5,2)) Out[82]: array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) In [83]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran = False), (2,5,2)) Out[83]: array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) In [84]: numpy.__version__ Out[84]: '0.9.6.2148' -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From chanley at stsci.edu Mon Mar 20 17:35:19 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 20 17:35:19 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 Message-ID: <20060320203450.CIP50631@comet.stsci.edu> In answer to your question, f90 -V returns: basil> f90 -V f90: Sun WorkShop 6 update 2 Fortran 95 6.2 Patch 111690-10 2003/08/28 Usage: f90 [ options ] files. Use 'f90 -flags' for details However, I agree with Travis. Lack of a FORTRAN compiler should not cause the build to fail when it is not required. Thank you for your time and help. Chris >>Looks like it's not matching the compiler version for your fortran >>compiler. What does "f90 -V" give you? >> >> > >This should fail more gracefully, especially since a fortran compiler is >not even needed for numpy to get installed. This needs to be fixed in >NumPy. > >-Travis > From oliphant.travis at ieee.org Mon Mar 20 19:51:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 19:51:05 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F780F.5090602@ieee.org> Robert Kern wrote: > I think this is a new bug, then. > > In [82]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = True), (2,5,2)) > Out[82]: > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > In [83]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], > [11,12,13,14,15,16,17,18,19,20]], fortran = False), (2,5,2)) > Out[83]: > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > Robert, can you test numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], fortran = False).reshape(2,5,2) numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], fortran = False).reshape(2,5,2) please? The reshape method has not changed since your version, but the reshape wrapper did change slightly, I believe. In a manner that could have caused this difference. -Travis From oliphant.travis at ieee.org Mon Mar 20 20:12:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 20:12:02 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F7CD9.9070800@ieee.org> Zachary Pincus wrote: >> Can you give us some small, self-contained code that demonstrates the >> problem? > > u, s, vt = numpy.linalg.svd([[1,2],[3,4]]) > u.flags.fortran > True > > The problem, as per my original email, is that linalg.svd seems to be > now returning fortran-style arrays. I believe that it did not do this > in the past, but I am not certain. I think it's always done this. It is really very easy to generate fortran-order arrays (always has been --- the transpose operation has always returned a fortran-order array). a = numpy.rand(10,20).transpose() a.flags.fortran I think the issue that has raised this discussion is that the reshape function is now not running asarray if it doesn't have to --- this was done to preserve matrices... Notice that currently the asarray function will convert Fortran-order arrays to C-order arrays (and thus make a copy). Now that I'm aware it's doing this, I actually think that's a bug. This is because Fortran order is really just a special case of striding, and other non-uniformly strided arrays are not automatically converted just because asarray is called. So, I think as a side-issue that bug should be fixed. Now, the real issue that brought this to the fore-front is what is meant by the "reshape" function. It has been the case for several months that a fortran-order array has a different scan order for it's reshape *method* than any-other strided array. The reshape method is one of those functions that you can't avoid thinking about layout of the array (so my previous statement about not concerning yourself with striding was too far-reaching). We are supporting more than one view about how layout should be done. But, we should be doing it in a backward-compatible fashion so that the standard is C-contiguous unless otherwise specified. Thus, I think the bug is in the design of the reshape method in that it uses a different layout standard depending on whether or not the array is Fortran or not. This bug has been there for awhile, I think, but only exposed by the re-working of the reshape function. I propose to change the reshape method to always expect a C-contiguous perspective unless a fortran flag is set (defaults to False), in which case the perspective is Fortran-contiguous. This is true regardless of the layout of the current array. This will fix the reshape function as well but also make it possible for someone to use reshape who has a Fortran-order array-view. > > > In any case, there are two issues at play: > (1) Should any numpy functions be allowed to return fortran arrays > unless specifically requested? > (2) Should all structural operations like reshape (etc.) behave the > same for fortran and c-style arrays? > The issue is that there are only a few shape-changing functions (ravel, reshape, etc.), that the Python user needs to have any perspective at all on how the data is layed out in memory. Other routines should not depend on the striding (or if they do they need to ensure it's contiguous first). For those routines that do require memory-layout knowledge the default must continue to be C-contiguous, but we need to add a fortran keyword to allow that perspective as well should the user choose it. This is already done for the ravel method. And it should be available for reshape (and resize---though that one will take more effort and I will not do tonight---as well). Thanks for the bug-reports. -Travis From robert.kern at gmail.com Mon Mar 20 20:13:22 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 20 20:13:22 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <441F780F.5090602@ieee.org> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> <441F780F.5090602@ieee.org> Message-ID: Travis Oliphant wrote: > Robert Kern wrote: > >> I think this is a new bug, then. >> >> In [82]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], >> [11,12,13,14,15,16,17,18,19,20]], fortran = True), (2,5,2)) >> Out[82]: >> array([[[ 1, 2], >> [ 3, 4], >> [ 5, 6], >> [ 7, 8], >> [ 9, 10]], >> >> [[11, 12], >> [13, 14], >> [15, 16], >> [17, 18], >> [19, 20]]]) >> >> In [83]: reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10], >> [11,12,13,14,15,16,17,18,19,20]], fortran = False), (2,5,2)) >> Out[83]: >> array([[[ 1, 2], >> [ 3, 4], >> [ 5, 6], >> [ 7, 8], >> [ 9, 10]], >> >> [[11, 12], >> [13, 14], >> [15, 16], >> [17, 18], >> [19, 20]]]) > > Robert, can you test > > numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = False).reshape(2,5,2) > > numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = False).reshape(2,5,2) > > please? > > The reshape method has not changed since your version, but the reshape > wrapper did change slightly, I believe. In a manner that could have > caused this difference. Huh. *That* displays the bug. In [85]: numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran=False).reshape(2,5,2) Out[85]: array([[[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18], [19, 20]]]) In [86]: numpy.array([[1,2,3,4,5,6,7,8,9,10], [11,12,13,14,15,16,17,18,19,20]], fortran=True).reshape(2,5,2) Out[86]: array([[[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]], [[11, 16], [12, 17], [13, 18], [14, 19], [15, 20]]]) -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cookedm at physics.mcmaster.ca Mon Mar 20 20:15:17 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Mar 20 20:15:17 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: <20060320203450.CIP50631@comet.stsci.edu> (Christopher Hanley's message of "Mon, 20 Mar 2006 20:34:50 -0500 (EST)") References: <20060320203450.CIP50631@comet.stsci.edu> Message-ID: Christopher Hanley writes: > In answer to your question, f90 -V returns: > > basil> f90 -V > f90: Sun WorkShop 6 update 2 Fortran 95 6.2 Patch 111690-10 2003/08/28 > Usage: f90 [ options ] files. Use 'f90 -flags' for details Ok, looks like the version-matching regexp was wrong. I've replaced it with a fuzzier match. The previous version wouldn't have triggered the error you saw, but it would instead return None from .get_version(). > However, I agree with Travis. Lack of a FORTRAN compiler should not > cause the build to fail when it is not required. I've replaced the thrown error with a log.warn. Fixed in svn. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant.travis at ieee.org Mon Mar 20 21:25:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 20 21:25:03 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> Message-ID: <441F8E0C.4000901@ieee.org> Zachary Pincus wrote: >> Can you give us some small, self-contained code that demonstrates the >> problem? > > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = False), [2, 5, 2]) > array([[[ 1, 2], > [ 3, 4], > [ 5, 6], > [ 7, 8], > [ 9, 10]], > > [[11, 12], > [13, 14], > [15, 16], > [17, 18], > [19, 20]]]) > > numpy.reshape(numpy.array([[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], > fortran = True), [2, 5, 2]) > array([[[ 1, 6], > [ 2, 7], > [ 3, 8], > [ 4, 9], > [ 5, 10]], > > [[11, 16], > [12, 17], > [13, 18], > [14, 19], > [15, 20]]]) > > This should be fixed now in SVN. Thanks for pointing it out. -Travis From zpincus at stanford.edu Mon Mar 20 21:53:01 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 20 21:53:01 2006 Subject: [Numpy-discussion] Re: linalg.svd returns fortran arrays, causes problems In-Reply-To: <441F7CD9.9070800@ieee.org> References: <378FEEB0-D2C5-4391-A1D8-E97D7204C46F@stanford.edu> <441F7CD9.9070800@ieee.org> Message-ID: <7D7758DF-7744-4BBB-8AB9-F3D90A1E8C50@stanford.edu> > I propose to change the reshape method to always expect a C- > contiguous perspective unless a fortran flag is set (defaults to > False), in which case the perspective is Fortran-contiguous. This > is true regardless of the layout of the current array. This seems like an optimal solution -- thanks for looking into all of this and getting the fix in the svn so quickly! Zach From dd55 at cornell.edu Tue Mar 21 08:27:34 2006 From: dd55 at cornell.edu (Darren Dale) Date: Tue Mar 21 08:27:34 2006 Subject: [Numpy-discussion] constructing char arrays with "where" Message-ID: <200603211101.00155.dd55@cornell.edu> There is some disparity between the numpy, Numeric and numarray flavors of the "where" function, as concerns creation of char arrays. I got bit by the limitation in numarray. I found a workaround for my matplotlib code that didnt use "where", but perhaps it is worth mentioning. Darren ################################################# import Numeric, numpy, numarray In [1]: numpy.where([True, False], 'T', 'F') Out[1]: array([T, F], dtype='|S1') dtype='|S1') In [2]: numpy.where([True, False], 'True', 'False') Out[2]: array([True, False], dtype='|S5') ################################################# In [3]: Numeric.where([True, False], 'T', 'F') Out[3]: array([T, F],'c') In [4]: Numeric.where([True, False], 'True', 'False') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/darren/src/matplotlib/ /usr/lib64/python2.4/site-packages/Numeric/Numeric.py in where(condition, x, y) 603 y where condition is respectively true or false. 604 """ --> 605 return choose(not_equal(condition, 0), (y, x)) 606 607 def compress(condition, m, axis=-1): ValueError: array dimensions must agree ################################################ In [23]: numarray.where([True, False], 'T', 'F') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/darren/src/matplotlib/ /usr/lib64/python2.4/site-packages/numarray/generic.py in where(condition, x, y, out) 1066 if x is None or y is None: 1067 raise ValueError("Invalid parameter list") -> 1068 return choose(ufunc.not_equal(condition, 0), (y,x), out) 1069 1070 def clip(m, m_min, m_max): /usr/lib64/python2.4/site-packages/numarray/ufunc.py in choose(selector, population, outarr, clipmode) 1988 [0, shape[i]) result in an exception. 1989 """ -> 1990 return _choose(selector, population, outarr, clipmode) 1991 1992 def _scatteredPseudos( scattered ): /usr/lib64/python2.4/site-packages/numarray/ufunc.py in __call__(self, inarr1, inarr2, outarr, clipmode) 1894 def __call__(self, inarr1, inarr2, outarr=None, clipmode=RAISE): 1895 """The standard calling interface for UFuncs""" -> 1896 computation_mode, woutarr, cfunc, ufargs = \ 1897 self._setup(inarr1, inarr2, outarr, clipmode) 1898 /usr/lib64/python2.4/site-packages/numarray/ufunc.py in _setup(self, in1, in2, outarr, clipmode) 1910 1911 if outarr is None: -> 1912 convType = _maxPopType(in2) 1913 else: 1914 convType = outarr._type /usr/lib64/python2.4/site-packages/numarray/ufunc.py in _maxPopType(xs) 1852 t = type(x) 1853 else: -> 1854 t = _nc._maxtype(x) 1855 1856 if maxT is None: /usr/lib64/python2.4/site-packages/numarray/numarraycore.py in _maxtype(args) 129 return complex 130 else: --> 131 return PyLevel2Type[_numarray._maxtype(args)] 132 133 def _storePyValueInBuffer(buffer, Ctype, index, value): TypeError: Expecting a python numeric type, got something else. From chanley at stsci.edu Tue Mar 21 08:33:59 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Tue Mar 21 08:33:59 2006 Subject: [Numpy-discussion] Numpy build on Solaris 8 fails beginning on 3/19/06 In-Reply-To: References: <20060320203450.CIP50631@comet.stsci.edu> Message-ID: <44201C12.3070709@stsci.edu> David, Thank you for your time and help. Last night's build and tests completed without error. Chris > > I've replaced the thrown error with a log.warn. > > Fixed in svn. > From rhl at astro.princeton.edu Tue Mar 21 08:45:29 2006 From: rhl at astro.princeton.edu (Robert Lupton) Date: Tue Mar 21 08:45:29 2006 Subject: [Numpy-discussion] Multiple inheritance from ndarray In-Reply-To: <43FD32E4.10600@ieee.org> References: <5809AC56-B2DF-4403-B7BC-9AEEAAC78505@astro.princeton.edu> <43FD32E4.10600@ieee.org> Message-ID: I've finally found time to return to this problem. Travis's made the suggestion that I could use code along the lines of: > class Image(ndarray, actImage): > def __new__(subtype, *args) > act1 = actImage.__new__(actImage, *args) > actImage.__init__(act1, *args) > arr = array(act1.getArray(), 'd', copy=False) > self = arr.view(subtype) > > return self and this makes sense. Unfortunately, the arr.view(subtype) fails: TypeError: Cannot change descriptor for objectarray. [incidently, is that a typo for "object array"; the string's built by "C string" "concatenation"). Ideas? The C type "actImage" looks like: typedef struct actImage { int nrow; // number of rows int ncol; // number of columns void *base; // allocated memory, if owned by this object int baseRefCount; // reference counter for base double **rows; // Fluxes for pixels PyArrayObject *_array; // pointer to Numeric array; used by actNumPy } actImage; The _array is set in the C actImage constructor with: im->_array = (PyArrayObject *)PyArray_FromDimsAndData(2, dims, PyArray_DOUBLE, (void *)im->base); and getArray is a wrapper around: PyObject *actImageGetNumPy(const actImage *im) { Py_XINCREF(im->_array); return PyArray_Return(im->_array); } R > Robert Lupton wrote: >> I have a swig extension that defines a class that inherits from >> both a personal C-coded image struct (actImage), and also from >> Numeric's UserArray. This works very nicely, but I thought that >> it was about time to upgrade to numpy. >> >> The code looks like: >> >> from UserArray import * >> >> class Image(UserArray, actImage): >> def __init__(self, *args): >> actImage.__init__(self, *args) >> UserArray.__init__(self, self.getArray(), 'd', >> copy=False, savespace=False) >> >> I can't figure out how to convert this to use ndarray, as ndarray >> doesn't >> seem to have an __init__ method, merely a __new__. >> > > > Yes, the ndarray method doesn't have an __init__ method (so you > don't have to call it). > > What you need to do is write a __new__ method for your class. > However, with multiple-inheritance the details matter. You may > actually want to have your C-coded actImage class inherit (in C) > from the ndarray. If you would like help on that approach let me > know (I'll need to understand your actImage a bit better). > But, this can all be done in Python, too, but it is a bit of effort > to make sure things get created correctly. Perhaps it might make > sense to actually include a slightly modified form of the UserArray > in NumPy as a standard "container-class" (instead of a sub-class) > of the ndarray. In reality, a container class like UserArray and > a sub-class are different things. > > Here's an outline of what you need to do. This is, of course, > untested.... For example, I don't really know what actImage is. > > from numpy import ndarray, array > > class Image(ndarray, actImage): > def __new__(subtype, *args) > act1 = actImage.__new__(actImage, *args) > actImage.__init__(act1, *args) > arr = array(act1.getArray(), 'd', copy=False) > self = arr.view(subtype) > # you might need to copy attributes from act1 over to self > here... > return self > > > The problem here, is that apparently you are creating the array > first in actImage.__init__ and then passing it to UserArray. The > ndarray constructor wants to either create the array itself or use > a buffer-exposing object to use as the memory. > Keep us posted as your example is a good one that can help us all > learn. > > -Travis > > > > > From oliphant.travis at ieee.org Tue Mar 21 10:03:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 21 10:03:05 2006 Subject: [Numpy-discussion] Multiple inheritance from ndarray In-Reply-To: References: <5809AC56-B2DF-4403-B7BC-9AEEAAC78505@astro.princeton.edu> <43FD32E4.10600@ieee.org> Message-ID: <44203F91.7010505@ieee.org> Robert Lupton wrote: > I've finally found time to return to this problem. Travis's made the > suggestion that I could use code along the lines of: > >> class Image(ndarray, actImage): >> def __new__(subtype, *args) >> act1 = actImage.__new__(actImage, *args) >> actImage.__init__(act1, *args) >> arr = array(act1.getArray(), 'd', copy=False) >> self = arr.view(subtype) >> >> return self > > and this makes sense. Unfortunately, the arr.view(subtype) fails: > TypeError: Cannot change descriptor for objectarray. > [incidently, is that a typo for "object array"; the string's built > by "C string" "concatenation"). Yes, I think so. Hmm, it looks like you have object arrays floating around which seems strange. But, as I thought more about multiple inheritance I realized that there really is a case for a container class like UserArray that you can inherit from that only "wraps" the ndarray and does not try to inherit from it. The analogy I see with C-code is that a container class has a pointer to an array object structure, while a sub-class has the arrayobject structure itself as part of the new subclass. So, I added back the UserArray to numpy. Now, you can inherit from numpy.lib.UserArray.UserArray just like before. > Ideas? > > The C type "actImage" looks like: > typedef struct actImage { > int nrow; // number of rows > int ncol; // number of columns > void *base; // allocated memory, if owned by this > object > int baseRefCount; // reference counter for base > double **rows; // Fluxes for pixels > PyArrayObject *_array; // pointer to Numeric array; used > by actNumPy > } actImage; It looks like you have a "container" class by default here. You could, if you wanted make this a subclass by definining actImage as typedef struct actImage { PyArrayObject _array; int nrow; // redundant int ncol; // redundant void * base; // also seems redundant int baseRefCount; // perhaps not necessary either double **rows; // truly new information } actImage; Now, a pointer to actImage could be cast to a (PyArrayObject *) and used wherever arrayobjects are expected and work just fine. I'm not sure why you have a different pointer for allocated memory which could It seems to me that you should create a sub-type in C of the ndarray rather than a container class. But, a container class would also continue to work. With the re-addition of UserArray it would work in much the same way as well. > The _array is set in the C actImage constructor with: > im->_array = (PyArrayObject *)PyArray_FromDimsAndData(2, dims, > PyArray_DOUBLE, (void *)im->base); > For the subclass you would need to do a little more work in the constructor because PyArray_FromDims returns only a filled-in PyArrayObject structure. You could copy all the elements of this array over (this wouldn't copy data just pointers), and then delete the PyArrayObject structure (don't use DECREF, just tp_free). Then you could fill in the rest of the actImage structure. > and getArray is a wrapper around: > > PyObject *actImageGetNumPy(const actImage *im) > { > Py_XINCREF(im->_array); > return PyArray_Return(im->_array); > } > You could use the new tp_members structure here which allows you to get access to parts of your C-structure very easily. The easiest transition path is to just use the re-added UserArray as before. If you want to learn to sub-class, that could be done as well. But, there are no sub-classes in C that I know of yet, so although it is possible, there may be some subtle issues uncovered in the process. -Travis From rhl at astro.princeton.edu Tue Mar 21 11:10:02 2006 From: rhl at astro.princeton.edu (Robert Lupton) Date: Tue Mar 21 11:10:02 2006 Subject: [Numpy-discussion] Multiple inheritance from ndarray In-Reply-To: <44203F91.7010505@ieee.org> References: <5809AC56-B2DF-4403-B7BC-9AEEAAC78505@astro.princeton.edu> <43FD32E4.10600@ieee.org> <44203F91.7010505@ieee.org> Message-ID: Thanks Travis. >> Unfortunately, the arr.view(subtype) fails: >> TypeError: Cannot change descriptor for objectarray. >> [incidently, is that a typo for "object array"; the string's built >> by "C string" "concatenation")?] >> > Yes, I think so. Hmm, it looks like you have object arrays > floating around which seems strange. Do you know why I've acquired object arrays? I don't understand enough numpy guts to figure it out myself. If it's an interesting question, I can give you a complete tarball. > So, I added back the UserArray to numpy. Now, you can inherit from > numpy.lib.UserArray.UserArray just like before. This is in the latest svn version? R From pgmdevlist at mailcan.com Tue Mar 21 13:39:20 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 21 13:39:20 2006 Subject: [Numpy-discussion] Masked array - follow up Message-ID: <200603211553.43698.pgmdevlist@mailcan.com> Folks, First of all, many thanks to who implemented the ndim in the latest numpy.core.ma version. I ran into some other missing features with masked arrays. I updated the wiki page, and attached a suggestion for a patch: http://projects.scipy.org/scipy/numpy/wiki/MaskedArray The few modifications seem to work for me, but I guess they should be double/triple-checked. In any case, comments and ideas are more than welcome. Thanks again P. From haase at msg.ucsf.edu Tue Mar 21 15:15:04 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Mar 21 15:15:04 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" Message-ID: <200603211514.22249.haase@msg.ucsf.edu> Hi, I think I discovered another "ugly" bug: If have an array with rank 4. I thought numarray.maximum.reduce(arr, 1) would be identical to .numarray.maximum.reduce(arr, -3) But instead I get something that looks more like the transpose of the first ! I tried to check the documentation for the numarray.maximum function, but there is none at [8. Array Functions http://stsdas.stsci.edu/numarray/numarray-1.5.html/node38.html] This is a test a ran afterwards: >>> q=na.arange(8) >>> q.shape = (2,2,2) >>> q [[[0 1] [2 3]] [[4 5] [6 7]]] >>> na.maximum.reduce(q) [[4 5] [6 7]] >>> na.maximum.reduce(q,0) [[4 5] [6 7]] >>> na.maximum.reduce(q,1) [[2 3] [6 7]] >>> na.maximum.reduce(q,-1) [[1 3] [5 7]] So its not really the transpose - but in any case it's something strange... Thanks, Sebastinan Haase From pebarrett at gmail.com Wed Mar 22 05:33:09 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Wed Mar 22 05:33:09 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <200603211514.22249.haase@msg.ucsf.edu> References: <200603211514.22249.haase@msg.ucsf.edu> Message-ID: <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> On 3/21/06, Sebastian Haase wrote: > > Hi, > I think I discovered another "ugly" bug: > If have an array with rank 4. I thought > numarray.maximum.reduce(arr, 1) > would be identical to > .numarray.maximum.reduce(arr, -3) > But instead I get something that looks more like the transpose of the > first ! > I tried to check the documentation for the numarray.maximum function, > but there is none at [8. Array Functions > http://stsdas.stsci.edu/numarray/numarray-1.5.html/node38.html] > > This is a test a ran afterwards: > >>> q=na.arange(8) > >>> q.shape = (2,2,2) > >>> q > [[[0 1] > [2 3]] > [[4 5] > [6 7]]] > >>> na.maximum.reduce(q) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,0) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,1) > [[2 3] > [6 7]] > >>> na.maximum.reduce(q,-1) > [[1 3] > [5 7]] > > So its not really the transpose - but in any case it's something > strange... > The above behavior for maximum.reduce looks consistent to me. The reduce axis for examples 1 and 2 above is 0, so maximum is comparing arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. Example 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving [[2 3] [6 7]]. And the last example is comparing arrays [[0 2] [4 6]] and [[1 3] [5 7]], giving [[1 3] [5 7]]. I think it depends on how you look at the shape of the array. -- Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Mar 22 08:20:07 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 22 08:20:07 2006 Subject: [Numpy-discussion] Re: numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> References: <200603211514.22249.haase@msg.ucsf.edu> <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> Message-ID: Paul Barrett wrote: > On 3/21/06, *Sebastian Haase* > wrote: > > Hi, > I think I discovered another "ugly" bug: > If have an array with rank 4. I thought > numarray.maximum.reduce(arr, 1) > would be identical to > .numarray.maximum.reduce(arr, -3) > But instead I get something that looks more like the transpose of > the first ! > I tried to check the documentation for the numarray.maximum function, > but there is none at [8. Array Functions > http://stsdas.stsci.edu/numarray/numarray-1.5.html/node38.html > ] > > This is a test a ran afterwards: > >>> q=na.arange(8) > >>> q.shape = (2,2,2) > >>> q > [[[0 1] > [2 3]] > [[4 5] > [6 7]]] > >>> na.maximum.reduce(q) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,0) > [[4 5] > [6 7]] > >>> na.maximum.reduce(q,1) > [[2 3] > [6 7]] > >>> na.maximum.reduce(q,-1) > [[1 3] > [5 7]] > > So its not really the transpose - but in any case it's something > strange... > > The above behavior for maximum.reduce looks consistent to me. > > The reduce axis for examples 1 and 2 above is 0, so maximum is comparing > arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. > Example 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving > [[2 3] [6 7]]. And the last example is comparing arrays [[0 2] [4 6]] > and [[1 3] [5 7]], giving [[1 3] [5 7]]. > > I think it depends on how you look at the shape of the array. Well, there is at least an incompatibility with numpy and Numeric. In [6]: def test(mod): ...: q = mod.reshape(mod.arange(16), (2,2,2,2)) ...: for i in range(4): ...: mask = (mod.maximum.reduce(q, i) == mod.maximum.reduce(q, i-4)) ...: print i, mod.alltrue(mod.ravel(mask)) ...: ...: In [7]: test(numpy) 0 True 1 True 2 True 3 True In [8]: test(numarray) 0 0 1 0 2 1 3 1 In [19]: test(Numeric) 0 1 1 1 2 1 3 1 In [9]: numpy.__version__ Out[9]: '0.9.6.2148' In [10]: numarray.__version__ Out[10]: '1.5.0' In [20]: Numeric.__version__ Out[20]: '24.0' -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fonnesbeck at gmail.com Wed Mar 22 08:31:01 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Mar 22 08:31:01 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's Message-ID: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> If linalg.eigenvectors is called on a matrix that contains nan values, it hangs, quickly consuming system resources. Here is the matrix: matrix = [[ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0.89827259 0. 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0.92510948 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.93319244 nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0.89827259 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0.92510948 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0. 0.92510948 0.93319244]] I would expect an exception, but I get none. Using a recent cvs build of numpy. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From fonnesbeck at gmail.com Wed Mar 22 08:31:04 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Mar 22 08:31:04 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's Message-ID: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> If linalg.eigenvectors is called on a matrix that contains nan values, it hangs, quickly consuming system resources. Here is the matrix: matrix = [[ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0.89827259 0. 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0.92510948 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.93319244 nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0.89827259 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0.92510948 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0. 0.92510948 0.93319244]] I would expect an exception, but I get none. Using a recent cvs build of numpy. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From cimrman3 at ntc.zcu.cz Wed Mar 22 08:39:05 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed Mar 22 08:39:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <1142313130.441650aad8111@webmail.colorado.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> Message-ID: <44217D77.7030403@ntc.zcu.cz> Fernando.Perez at colorado.edu wrote: > If you have any concerns about the code, let me know and I'll be > happy to address them, but I really think that this will be much more > useful to everyone if each directory contains a complete, working > example. Thanks for the typemaps, Fernando. Do you think you could update the 'Cookbook/SWIG and NumPy' web site section to reflect their presence? For the moment there is just my simple (yet sufficient for my purposes) attempt. cheers, r. From tgrav at mac.com Wed Mar 22 09:36:06 2006 From: tgrav at mac.com (Tommy Grav) Date: Wed Mar 22 09:36:06 2006 Subject: [Numpy-discussion] Problems installing NumPy Message-ID: <4E0ECB41-EF32-4F0E-9DAE-52B55C8AE3A0@mac.com> I am new to Python and just downloaded ActivePython 2.4.2.10 on my Mac PPC with OS X 10.4. I added the numpy package (0.9.6-py2.4) and it imports fine. But when I try pydoc random I get this error [tgrav at skathi] site-packages/numpy -> pydoc random import core -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import lib -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import linalg -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import dft -> failed: dlopen(/Library/Frameworks/Python.framework/ Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, 2): Symbol not found: _PyOS_ascii_strtod Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ lib/python2.4/site-packages/numpy/core/multiarray.so Expected in: dynamic lookup import random -> failed: 'module' object has no attribute 'dtype' problem in ./random/__init__.py - ImportError: cannot import name add_newdoc Anyone know what is causing this problem? Is there something in my installation of either ActivePython or NumPy that needs to be fixed? Cheers Tommy tgrav at mac.com http://homepage.mac.com/tgrav/ "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction" -- Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Wed Mar 22 09:45:59 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 09:45:59 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Fast Gauss Transform In-Reply-To: <91cf711d0603220839g2c974524h@mail.gmail.com> References: <91cf711d0603201107k7b1f38c7g@mail.gmail.com> <91cf711d0603210652u8f93965y@mail.gmail.com> <44206894.20007@gmail.com> <1142990801.30961.11.camel@localhost.localdomain> <87ek0vs20b.fsf@peds-pc311.bsd.uchicago.edu> <91cf711d0603220839g2c974524h@mail.gmail.com> Message-ID: <44218D03.4080703@ieee.org> David Huard wrote: > While fiddling with new code to compute kde, I noticed that > gaussian_kde has an unexpected behavior for 1D arrays. > > For instance : > > x = rand(1000) > g = gaussian_kde(x) > g(z) > > returns the same value no matter what z is. The problem disappears for > multidimensional data. The dot product (self.inv_cov, diff) seems to > be causing the problem, doing the product only for the first element > of diff and returning zeros for the rest. I guess this is not the > intended behavior for dot. Yet another untested branch of the optimized dot code. Apparently it's been very hard to get this one right after re-writing the scalar-multiplication portion. Scalar multiplication is used in several branches but the parameters must be set just right. You were exposing a (1,1) x (1,N) error that uses BLAS scalar multiplication at the core but the number of multiplies was set to 1 rather than N. I added a test for this case in numpy and fixed the problem. Thanks for the report. -Travis From haase at msg.ucsf.edu Wed Mar 22 09:46:01 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Mar 22 09:46:01 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> References: <200603211514.22249.haase@msg.ucsf.edu> <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> Message-ID: <200603220945.18706.haase@msg.ucsf.edu> On Wednesday 22 March 2006 05:32, Paul Barrett wrote: > On 3/21/06, Sebastian Haase wrote: > > The above behavior for maximum.reduce looks consistent to me. > > The reduce axis for examples 1 and 2 above is 0, so maximum is comparing > arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. Example > 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving [[2 3] [6 > 7]]. And the last example is comparing arrays [[0 2] [4 6]] and [[1 3] [5 > 7]], giving [[1 3] [5 7]]. > > I think it depends on how you look at the shape of the array. > > -- Paul Sorry I did the test wrong - >>> na.maximum.reduce(q,2) [[1 3] [5 7]] >>> na.maximum.reduce(q,-1) [[1 3] [5 7]] So it obviously works as advertised. The array where I noticed the transpose behaviour was really of shape 32,15,521,512 and my command was na.maximum.reduce(arr,-3) which "looked" tranpose to what I got with na.maximum.reduce(arr,1). I using numarray for image analysis and the array might have been a memmap-ed image file. I'm just worried because I found an "accidental transpose bug" on type conversion a few years back (see mailing list)... ( I forgot to say that I running Python 2.2 and numarray version '1.4.0' ) I just reran the test with python2.4 and arr=arr.copy() [which should give a non-memmap version of arr] >>> arrM = na.maximum.reduce(arr,-3) >>> arrMt = na.transpose(arrM,(0,2,1)) >>> arrM1= na.maximum.reduce(arr,1) >>> U.mmms(arrM1-arrMt) # returns min,max,mean,stddev (0, 0, 0.0, 0.0) So it definitely transposes the last two axes ! Maybe the bug shows up on rank >=4 !? - which is the case Robert tested. Thanks, Sebastian Haase From ndarray at mac.com Wed Mar 22 12:27:00 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 12:27:00 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: <200603211553.43698.pgmdevlist@mailcan.com> References: <200603211553.43698.pgmdevlist@mailcan.com> Message-ID: Pierre, Your patch proposes to add an optional argument "flag_missing=False" to the trace method. As you explain, "`flag_missing` is True and some diagonal values are masked, returns `masked`." This feature deserves some careful discussion before it is accepted. I would think this feature is unnecessary because a.fill(0).trace() is a better way to express the desired result than a.trace(flag_missing=False). (Your implementation is equivalent to a.data.trace(), which is clearly wrong because masked elements are likely to contain garbage.) -- sasha PS: BTW, your implementation is incorrect. Currently ma assumes that "masked" is a singleton, so return MaskedArray(0, int, mask=1) is not the same as return masked. Also m = getmask(a) should probably be m = getmask(d). On 3/21/06, Pierre GM wrote: > Folks, > First of all, many thanks to who implemented the ndim in the latest > numpy.core.ma version. > I ran into some other missing features with masked arrays. I updated the wiki > page, and attached a suggestion for a patch: > http://projects.scipy.org/scipy/numpy/wiki/MaskedArray > The few modifications seem to work for me, but I guess they should be > double/triple-checked. In any case, comments and ideas are more than welcome. > Thanks again > P. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ndarray at mac.com Wed Mar 22 13:43:04 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 13:43:04 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled Message-ID: In an ideal world, any function that accepts ndarray would accept ma.array and vice versa. Moreover, if the ma.array has no masked elements and the same data as ndarray, the result should be the same. Obviously current implementation falls short of this goal, but there is one feature that seems to make this goal unachievable. This feature is the "filled" method of ma.array. Pydoc for this method reports the following: | filled(self, fill_value=None) | A numeric array with masked values filled. If fill_value is None, | use self.fill_value(). | | If mask is nomask, copy data only if not contiguous. | Result is always a contiguous, numeric array. | # Is contiguous really necessary now? That is not the best possible description ("filled" is "filled"), but the essence is that the result of a.filled(value) is a contiguous ndarray obtained from the masked array by copying non-masked elements and using value for masked values. I would like to propose to add a "filled" method to ndarray. I see several possibilities and would like to hear your opinion: 1. Make filled simply return self. 2. Make filled return a contiguous copy. 3. Make filled replace nans with the fill_value if array is of floating point type. Unfortunately, adding "filled" will result is a rather confusing situation where "fill" and "filled" both exist and have very different meanings. I would like to note that "fill" is a somewhat odd ndarray method. AFAICT, it is the only non-special method that mutates the array. It appears to be just a performance trick: the same result can be achived with "a[...] = ". From pgmdevlist at mailcan.com Wed Mar 22 13:58:02 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Wed Mar 22 13:58:02 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: References: <200603211553.43698.pgmdevlist@mailcan.com> Message-ID: <200603221654.37730.pgmdevlist@mailcan.com> Sasha, I'm still not sure the `flag_missing` option is really needed. Probably not, as you noted. I intended to use it as a reminder, just in case. I agree with you, it's confusing and should probably be discarded. that maybe useful a bit later, for cumprod for example: we could have something like: if `use_missing`: a.cumprod() = MA.masked else: a.cumprod() = a.filled(1).cumprod() > I would think this > feature is unnecessary because a.fill(0).trace() is a better way to > express the desired result than a.trace(flag_missing=False). If all the elements of the diagonal of a are masked, what should a.trace() be ? 0, MA.masked, undefined ? `a.fill(0).trace` (well, `a.filled(0).trace` actually) will always give 0. The version I implemented give you MA.masked. try it. > Your implementation is equivalent to a.data.trace(), which is clearly wrong > because masked elements are likely to contain garbage.) Well, not really: if a is a MaskedArray d = diagonal(a,offset, axis1, axis2) outputs a MaskedArray, with d.data=a.diagonal().data and d.mask=d.diagonal.mask(). Check it. The version implemented actually corresponds to: a.trace() = a.diagonal().compressed().sum() > PS: BTW, your implementation is incorrect. Currently ma assumes that > "masked" is a singleton, so return MaskedArray(0, int, mask=1) is not > the same as return masked. well, I intended to use masked, but realize `masked` wasn't defined before the last line, so I just copied its definition. > Also m = getmask(a) should probably be m = > getmask(d). Well, no, not really: m is only used to check whether the initial array is masked or not. BTW, I attach the unittest I just finished writing. That passes on my box. Let me know how it works -------------- next part -------------- A non-text attachment was scrubbed... Name: testnewma.py Type: application/x-python Size: 3431 bytes Desc: not available URL: From ndarray at mac.com Wed Mar 22 14:22:06 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 14:22:06 2006 Subject: [Numpy-discussion] Bug or feature: sum vs. add.reduce Message-ID: Is this correct? >>> numpy.array([1.,.5]).sum(dtype=None) 1.5 >>> numpy.add.reduce([1.,.5],dtype=None) 1 From ndarray at mac.com Wed Mar 22 14:43:05 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 14:43:05 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: <200603221616.30402.pgmdevlist@mailcan.com> References: <200603211553.43698.pgmdevlist@mailcan.com> <200603221616.30402.pgmdevlist@mailcan.com> Message-ID: I've aplied Pierre's patch with minor modifications. See http://projects.scipy.org/scipy/numpy/changeset/2267 . For the time being trace is defined simply as diagonal().sum(). Thanks Pierre! From ndarray at mac.com Wed Mar 22 15:14:04 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 15:14:04 2006 Subject: [Numpy-discussion] Should ma.array.clip allow masked min and max? Message-ID: Pierre has implemented the clip method for ma.array, but his implementation does not seem to do the right thing if min or max is masked: >>> from numpy.core.ma import * >>> x = ones(3) >>> max = array([0,2,0],mask=[1,1,1]) >>> min = array([0,2,0],mask=[1,1,1]) >>> x.clip(min,max) array([1 1 1]) while >>> x.clip(min.data,max.data) array([0 2 0]) I believe it would be better to mask the elements in the result for which either min or max is masked. From efiring at hawaii.edu Wed Mar 22 15:38:01 2006 From: efiring at hawaii.edu (Eric Firing) Date: Wed Mar 22 15:38:01 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: References: Message-ID: <4421DF9B.6000102@hawaii.edu> Sasha wrote: > In an ideal world, any function that accepts ndarray would accept > ma.array and vice versa. Moreover, if the ma.array has no masked > elements and the same data as ndarray, the result should be the same. This would be *very* nice. > Obviously current implementation falls short of this goal, but there > is one feature that seems to make this goal unachievable. > > This feature is the "filled" method of ma.array. Pydoc for this > method reports the following: > > | filled(self, fill_value=None) > | A numeric array with masked values filled. If fill_value is None, > | use self.fill_value(). > | > | If mask is nomask, copy data only if not contiguous. > | Result is always a contiguous, numeric array. > | # Is contiguous really necessary now? > > > That is not the best possible description ("filled" is "filled"), but > the essence is that the result of a.filled(value) is a contiguous > ndarray obtained from the masked array by copying non-masked elements > and using value for masked values. > > I would like to propose to add a "filled" method to ndarray. I see > several possibilities and would like to hear your opinion: > > 1. Make filled simply return self. > > 2. Make filled return a contiguous copy. > > 3. Make filled replace nans with the fill_value if array is of > floating point type. It seems to me that any function or method that returns an array from an array should be perfectly consistent and explicit about whether it makes a copy or not. Sometimes the filled method *needs* to return a copy; therefore it should *always* return a copy, regardless of the presence or state of masking. Hence I think the filled method of ma needs to be changed in this way also. The question for your suggestion 3 is, should a nan always be the equivalent of a masked value? One loses a little flexibility, but it has an appealing simplicity to it. I could be persuaded otherwise, but right now I would vote for it. Eric > > > Unfortunately, adding "filled" will result is a rather confusing > situation where "fill" and "filled" both exist and have very different > meanings. > > I would like to note that "fill" is a somewhat odd ndarray method. > AFAICT, it is the only non-special method that mutates the array. It > appears to be just a performance trick: the same result can be achived > with "a[...] = ". > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From pgmdevlist at mailcan.com Wed Mar 22 15:42:01 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Wed Mar 22 15:42:01 2006 Subject: [Numpy-discussion] Masked array - follow up In-Reply-To: References: <200603211553.43698.pgmdevlist@mailcan.com> <200603221616.30402.pgmdevlist@mailcan.com> Message-ID: <200603221840.51536.pgmdevlist@mailcan.com> On Wednesday 22 March 2006 17:42, Sasha wrote: > I've aplied Pierre's patch with minor modifications. See > http://projects.scipy.org/scipy/numpy/changeset/2267 . > > For the time being trace is defined simply as diagonal().sum(). Great ! Thanks a lot. I'll try to tackle some of the other missing feature very soon. From fonnesbeck at gmail.com Wed Mar 22 15:56:07 2006 From: fonnesbeck at gmail.com (Chris Fonnesbeck) Date: Wed Mar 22 15:56:07 2006 Subject: [Numpy-discussion] linalg.eigenvector hang on nan values Message-ID: <723eb6930603221555q2ff8ba7cifdbab975b4578884@mail.gmail.com> If linalg.eigenvectors is called on a matrix that contains nan values, it hangs, quickly consuming system resources. Here is the matrix: matrix = [[ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0.89827259 0. 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0.92510948 0. 0. 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. 0. 0. 0. 0. 0. ] [ 0. 0. nan nan 0. nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.93319244 nan 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0.36658624 0. 0. 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0.89827259 0. 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0.92510948 0. 0. ] [ 0. 0. 0. 0. 0. 0. 0. 0. 0.92510948 0.93319244]] I would expect an exception, but I get none. Using a recent cvs build of numpy. C. -- Chris Fonnesbeck + Atlanta, GA + http://trichech.us -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Wed Mar 22 15:59:23 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 15:59:23 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421DF9B.6000102@hawaii.edu> References: <4421DF9B.6000102@hawaii.edu> Message-ID: <4421E4A6.4020906@cox.net> Eric Firing wrote: > Sasha wrote: > >> In an ideal world, any function that accepts ndarray would accept >> ma.array and vice versa. Moreover, if the ma.array has no masked >> elements and the same data as ndarray, the result should be the same. > > > This would be *very* nice. > >> Obviously current implementation falls short of this goal, but there >> is one feature that seems to make this goal unachievable. >> >> This feature is the "filled" method of ma.array. Pydoc for this >> method reports the following: >> >> | filled(self, fill_value=None) >> | A numeric array with masked values filled. If fill_value is >> None, >> | use self.fill_value(). >> | >> | If mask is nomask, copy data only if not contiguous. >> | Result is always a contiguous, numeric array. >> | # Is contiguous really necessary now? >> >> >> That is not the best possible description ("filled" is "filled"), but >> the essence is that the result of a.filled(value) is a contiguous >> ndarray obtained from the masked array by copying non-masked elements >> and using value for masked values. >> >> I would like to propose to add a "filled" method to ndarray. I see >> several possibilities and would like to hear your opinion: >> >> 1. Make filled simply return self. >> >> 2. Make filled return a contiguous copy. >> >> 3. Make filled replace nans with the fill_value if array is of >> floating point type. > > > It seems to me that any function or method that returns an array from > an array should be perfectly consistent and explicit about whether it > makes a copy or not. Sometimes the filled method *needs* to return a > copy; therefore it should *always* return a copy, regardless of the > presence or state of masking. Hence I think the filled method of ma > needs to be changed in this way also. +1 (mumble mumble reshape mumble) > > The question for your suggestion 3 is, should a nan always be the > equivalent of a masked value? One loses a little flexibility, but it > has an appealing simplicity to it. I could be persuaded otherwise, > but right now I would vote for it. -0 (I'd be -1, but I don't really use masked arrays). While masked array should be convenient for masking. The basic ndarray object should be as flexible as possible since it's the building block for all the rest. -tim From tim.hochberg at cox.net Wed Mar 22 16:32:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 16:32:01 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 Message-ID: <4421EC3B.4020800@cox.net> I just updated from CVS and compiled using VC7. The resulting numpy segfaults immediately on import. Do not pass go, do not collect $200. It's been a couple of weeks probably since I updated from CVS, so I'm not sure where to start looking. I suppose I could start going back in time until it works again and then see what checkins broke it, but if anyone has any better ideas, let me know. -tim From zpincus at stanford.edu Wed Mar 22 16:45:00 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed Mar 22 16:45:00 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <4421EC3B.4020800@cox.net> References: <4421EC3B.4020800@cox.net> Message-ID: <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> I've seen this too, and been able to fix it by removing the build directory and rebuilding everything. I wonder if there's a bug in numpy's dependency checking that's causing certain files to not get rebuilt when they ought. I recently updated to the SVN head, and numpy segfaulted on import, just as you describe. After puttering around with GDB, it looked like some of the code was out of synch with other parts of the code, so I decided to rebuild from scratch. After I deleted the entire numpy build directory ('python setup.py clean' *did not* work) and re-built, things worked just fine. I've also had this problem with scipy, and the same solution applied. Anyhow, try removing all traces of build products from the numpy directory and rebuilding... maybe that will fix things. Zach On Mar 22, 2006, at 4:30 PM, Tim Hochberg wrote: > > I just updated from CVS and compiled using VC7. The resulting numpy > segfaults immediately on import. Do not pass go, do not collect > $200. It's been a couple of weeks probably since I updated from > CVS, so I'm not sure where to start looking. I suppose I could > start going back in time until it works again and then see what > checkins broke it, but if anyone has any better ideas, let me know. > > -tim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Fernando.Perez at colorado.edu Wed Mar 22 16:57:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 22 16:57:05 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> References: <4421EC3B.4020800@cox.net> <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> Message-ID: <4421F219.6080408@colorado.edu> Zachary Pincus wrote: > I've seen this too, and been able to fix it by removing the build > directory and rebuilding everything. > > I wonder if there's a bug in numpy's dependency checking that's > causing certain files to not get rebuilt when they ought. I recently > updated to the SVN head, and numpy segfaulted on import, just as you > describe. After puttering around with GDB, it looked like some of the > code was out of synch with other parts of the code, so I decided to > rebuild from scratch. > > After I deleted the entire numpy build directory ('python setup.py > clean' *did not* work) and re-built, things worked just fine. I've > also had this problem with scipy, and the same solution applied. > > Anyhow, try removing all traces of build products from the numpy > directory and rebuilding... maybe that will fix things. Just a hint from experience: do NOT trust distutils to do proper dependency management and know what to rebuild and what not to. Most of us have settled on some variation of: planck[scipy]> d /usr/local/installers/src/scipy total 16 -rwxr-xr-x 1 fperez 390 Mar 8 13:28 makepkg* drwxr-xr-x 6 fperez 4096 Mar 14 11:52 numpy/ drwxr-xr-x 4 fperez 4096 Nov 17 15:16 scipy/ -rwxr-xr-x 1 fperez 415 Jan 5 01:37 testpkg* planck[scipy]> cat makepkg #!/bin/sh PACKAGE=$1 PYTHON=python2.3 PYPREFIX=$HOME/tmp/local svn update ${PACKAGE} export PYTHONPATH=${PYPREFIX}/lib/${PYTHON}/site-packages:${PYTHONPATH}: # remove existing ${PACKAGE} to make sure the build doesn't pick up spurious things rm -rf $PYPREFIX/lib/${PYTHON}/site-packages/${PACKAGE} # make/install cd ${PACKAGE} rm -rf build $PYTHON setup.py install --prefix=$PYPREFIX ################ With that, I can just type ./makepkg numpy or ./makepkg scipy and I'm almost sure it will do the right thing. I wonder if we shouldn't add this to the wiki as the recommended procedure for builds (at least under *nix). I know it allows me to do fresh svn builds of numpy/scipy at any time without having to think (a process rarely successful within the confines of my skull). Cheers, f From Fernando.Perez at colorado.edu Wed Mar 22 17:06:14 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 22 17:06:14 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <44217D77.7030403@ntc.zcu.cz> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> Message-ID: <4421F455.2020003@colorado.edu> Robert Cimrman wrote: > Fernando.Perez at colorado.edu wrote: > >>If you have any concerns about the code, let me know and I'll be >>happy to address them, but I really think that this will be much more >> useful to everyone if each directory contains a complete, working >>example. > > > Thanks for the typemaps, Fernando. Do you think you could update the > 'Cookbook/SWIG and NumPy' web site section to reflect their presence? > For the moment there is just my simple (yet sufficient for my purposes) > attempt. Done, both for pyrex and swig. I added very simple notes, feel free to add further detail as necessary: http://scipy.org/Cookbook/Pyrex_and_NumPy http://scipy.org/Cookbook/SWIG_and_NumPy Cheers, f From robert.kern at gmail.com Wed Mar 22 17:19:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Mar 22 17:19:06 2006 Subject: [Numpy-discussion] Re: Current SVN segfaults under VC7 In-Reply-To: <4421F219.6080408@colorado.edu> References: <4421EC3B.4020800@cox.net> <876360C3-2B16-4252-8A31-84C6C77B8BC3@stanford.edu> <4421F219.6080408@colorado.edu> Message-ID: Fernando Perez wrote: > Zachary Pincus wrote: > >> I've seen this too, and been able to fix it by removing the build >> directory and rebuilding everything. >> >> I wonder if there's a bug in numpy's dependency checking that's >> causing certain files to not get rebuilt when they ought. I recently >> updated to the SVN head, and numpy segfaulted on import, just as you >> describe. After puttering around with GDB, it looked like some of the >> code was out of synch with other parts of the code, so I decided to >> rebuild from scratch. >> >> After I deleted the entire numpy build directory ('python setup.py >> clean' *did not* work) and re-built, things worked just fine. I've >> also had this problem with scipy, and the same solution applied. >> >> Anyhow, try removing all traces of build products from the numpy >> directory and rebuilding... maybe that will fix things. > > Just a hint from experience: do NOT trust distutils to do proper > dependency management and know what to rebuild and what not to. Generally it does an okay job, I think. numpy does quite a lot with generated code, however, and distutils *does* have a hard time with that. Unfortunately, I don't think there is much we can do about that without replacing even more of distutils. > Most of > us have settled on some variation of: ... > With that, I can just type > > ./makepkg numpy > > or > > ./makepkg scipy > > and I'm almost sure it will do the right thing. > > I wonder if we shouldn't add this to the wiki as the recommended > procedure for builds (at least under *nix). I know it allows me to do > fresh svn builds of numpy/scipy at any time without having to think (a > process rarely successful within the confines of my skull). +1. At least until 1.0 when the C API settles down. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at ee.byu.edu Wed Mar 22 17:20:19 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 22 17:20:19 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421E4A6.4020906@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> Message-ID: <4421F798.2070300@ee.byu.edu> Tim Hochberg wrote: >> >> It seems to me that any function or method that returns an array from >> an array should be perfectly consistent and explicit about whether it >> makes a copy or not. Sometimes the filled method *needs* to return a >> copy; therefore it should *always* return a copy, regardless of the >> presence or state of masking. Hence I think the filled method of ma >> needs to be changed in this way also. > > > +1 > > (mumble mumble reshape mumble) Tim makes a good point here. Should the reshape method be fixed to always return a copy? The semantics a.shape = (...) could still be used to re-shape contiguous arrays where possible. However, whether or not reshape returns a copy is consistent (but perhaps not explicitly explained). We will still have .ravel() which sometimes copies and sometimes doesn't. -Travis From oliphant at ee.byu.edu Wed Mar 22 17:26:14 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 22 17:26:14 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <4421EC3B.4020800@cox.net> References: <4421EC3B.4020800@cox.net> Message-ID: <4421F909.6050401@ee.byu.edu> Tim Hochberg wrote: > > I just updated from CVS and compiled using VC7. The resulting numpy > segfaults immediately on import. Do not pass go, do not collect $200. > It's been a couple of weeks probably since I updated from CVS, so I'm > not sure where to start looking. I suppose I could start going back in > time until it works again and then see what checkins broke it, but if > anyone has any better ideas, let me know. > You definitely need to remove your build directory. It is a build dependency issue (some needed generated code is not getting rebuilt when it should be). I'm not sure how to fix it, so I remove the build directory. It shows up when the C-API changes more significantly then just appending new C-API calls. It changed significantly recently as I moved the version checking C-API to the very front (so it won't move around in the future -- no matter what is done to the C-API), and I added the fixed Boolean array scalar True and False objects. The segfaults in scipy should disappear upon rebuild of numpy because the C-API now has the version checking at the very start and will gracefully exit with an error rather than rudely segfault. -Travis From pierregm at engr.uga.edu Wed Mar 22 17:37:03 2006 From: pierregm at engr.uga.edu (Pierre GM) Date: Wed Mar 22 17:37:03 2006 Subject: [Numpy-discussion] Re: Numpy-discussion digest, Vol 1 #1632 - 10 msgs In-Reply-To: <20060322234311.30FA78974B@sc8-sf-spam1.sourceforge.net> References: <20060322234311.30FA78974B@sc8-sf-spam1.sourceforge.net> Message-ID: <200603222035.08682.pierregm@engr.uga.edu> > Pierre has implemented the clip method for ma.array, but his > implementation does not seem to do the right thing if min or max is > masked: From core/oldnumeric.py: """clip(m, m_min, m_max) = every entry in m that is less than m_min is replaced by m_min, and every entry greater than m_max is replaced by m_max. """ My understanding was that m_min and m_max were two floats, not two arrays. I completely oversaw that. Your suggestion (masking where the limits are masked) makes complete sense. I'll work on that. From ndarray at mac.com Wed Mar 22 17:40:06 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 17:40:06 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421F798.2070300@ee.byu.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> Message-ID: On 3/22/06, Travis Oliphant wrote: > Tim makes a good point here. Should the reshape method be fixed to > always return a copy? The semantics a.shape = (...) could still be used > to re-shape contiguous arrays where possible. > Reshape never copies the data: >>> x = ones(4) >>> x.reshape((2,2)).__array_data__ == x.__array_data__ True The only inconsistency is that >>> x.reshape((2,2)) is x False >>> x.reshape((4,)) is x True I agree that this is unnecessary, but don't see much of a problem. +0 here > However, whether or not reshape returns a copy is consistent (but > perhaps not explicitly explained). > To me consistency means "is independent of the input." Whether or not reshape creates a new python object depends on the value of the argument. I would call it inconsistency. > We will still have .ravel() which sometimes copies and sometimes doesn't. Ravel should be a shortcut for x.reshape((x.size,)), so it is really the same question. +0 (to make ravel always return a new python object) From tim.hochberg at cox.net Wed Mar 22 17:57:09 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 17:57:09 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4421F798.2070300@ee.byu.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> Message-ID: <44220001.4050606@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> It seems to me that any function or method that returns an array >>> from an array should be perfectly consistent and explicit about >>> whether it makes a copy or not. Sometimes the filled method *needs* >>> to return a copy; therefore it should *always* return a copy, >>> regardless of the presence or state of masking. Hence I think the >>> filled method of ma needs to be changed in this way also. >> >> >> >> +1 >> >> (mumble mumble reshape mumble) > > > > Tim makes a good point here. And so articulately stated too ;) > Should the reshape method be fixed to always return a copy? The > semantics a.shape = (...) could still be used to re-shape contiguous > arrays where possible. > However, whether or not reshape returns a copy is consistent (but > perhaps not explicitly explained). > > We will still have .ravel() which sometimes copies and sometimes doesn't. My opinion is that all methods and functions should either: 1. Always return a copy. 2. Always return a view 3. Return a view if possible otherwise raise an exception. So, like Sasha, I'd like to see ravel changed as well. I don't really care if it's to 1 or 3 though. -tim From tim.hochberg at cox.net Wed Mar 22 18:10:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Mar 22 18:10:03 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <4421F909.6050401@ee.byu.edu> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> Message-ID: <44220366.4000004@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> I just updated from CVS and compiled using VC7. The resulting numpy >> segfaults immediately on import. Do not pass go, do not collect $200. >> It's been a couple of weeks probably since I updated from CVS, so I'm >> not sure where to start looking. I suppose I could start going back >> in time until it works again and then see what checkins broke it, but >> if anyone has any better ideas, let me know. >> > > You definitely need to remove your build directory. It is a build > dependency issue (some needed generated code is not getting rebuilt > when it should be). I'm not sure how to fix it, so I remove the build > directory. > > It shows up when the C-API changes more significantly then just > appending new C-API calls. It changed significantly recently as I > moved the version checking C-API to the very front (so it won't move > around in the future -- no matter what is done to the C-API), and I > added the fixed Boolean array scalar True and False objects. > > The segfaults in scipy should disappear upon rebuild of numpy because > the C-API now has the version checking at the very start and will > gracefully exit with an error rather than rudely segfault. This did fix things...eventually. I removed the build directory and rebuilt. Numpy still bombed the interpreter, although in a new and less exciting way. I then did a clean checkout from svn and rebuilt. Same thing. Then I manally removed numpy from site-packages and reinstalled at which points things worked and all tests passed. So in addition to needing to remove the build directory, it appears that something was hanging out in the destination directory that did not get overwritten and caused problems. Perhaps this description will help the next person who runs into this, but probably not. -tim From oliphant.travis at ieee.org Wed Mar 22 19:07:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 19:07:08 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44220B75.7030406@cox.net> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220B75.7030406@cox.net> Message-ID: <442210A7.7080802@ieee.org> Tim Hochberg wrote: > > OK, now that I've got things working, I can report a real segfault > under VC7. The following snippet of code: > > from numpy import * > a = arange(100.) > b = array(1+1j) > (a+1) ** (b+3) I can reproduce it on Linux. It looks like it's related to the power optimization stuff that was recently checked in. power(a,b+3) does not crash but >>> a**(b+3) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076248704 (LWP 1545)] 0x404162df in array_power_is_scalar (o2=0x8294200, exp=0xbfffec68) at arrayobject.c:2656 2656 arrayobject.c: No such file or directory. in arrayobject.c -Travis From ndarray at mac.com Wed Mar 22 19:20:03 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 19:20:03 2006 Subject: [Numpy-discussion] Changeset 2222 Message-ID: Changeset 2222 redefines several oldnumeric functions according to the following template: try: result = a.FUNC(...) except AttributeError: result = _wrapit(a, 'FUNC', ...) return result I believe it should be: try: FUNC = a.FUNC except AttributeError: return _wrapit(a, 'FUNC', ...) return FUNC(...) In most cases it should not matter, but some FUNCs can throw AttributeError on object arrays. From oliphant.travis at ieee.org Wed Mar 22 19:27:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 19:27:03 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44220B75.7030406@cox.net> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220B75.7030406@cox.net> Message-ID: <44221561.6060605@ieee.org> Tim Hochberg wrote: > > OK, now that I've got things working, I can report a real segfault > under VC7. The following snippet of code: > > from numpy import * > a = arange(100.) > b = array(1+1j) > (a+1) ** (b+3) > > will cause current CVS to crash. [under Window XP compiled with VC7]. > I also get a segfaults using 0.9.6 from scipy.org, which I believe is > compiled with mingw, so it must not just be a vc7 thing. It doesn't > happen if b is python scalar as opposed to a numpy scalar, FWIW. > > Can anyone reproduce this on a non-windows platform? Should be fixed in SVN, now. The problem was the use of PyArray_ISINTEGER macro on a non array object. Thanks, -Travis From oliphant.travis at ieee.org Wed Mar 22 20:09:15 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 22 20:09:15 2006 Subject: [Numpy-discussion] Changeset 2222 In-Reply-To: References: Message-ID: <44221F46.9000607@ieee.org> Sasha wrote: > Changeset 2222 redefines several oldnumeric functions according to the > following template: > > try: > result = a.FUNC(...) > except AttributeError: > result = _wrapit(a, 'FUNC', ...) > return result > > I believe it should be: > > try: > FUNC = a.FUNC > except AttributeError: > return _wrapit(a, 'FUNC', ...) > return FUNC(...) > > In most cases it should not matter, but some FUNCs can throw > AttributeError on object arrays. > Good point. Needs to be changed. -Travis From ndarray at mac.com Wed Mar 22 20:38:02 2006 From: ndarray at mac.com (Sasha) Date: Wed Mar 22 20:38:02 2006 Subject: [Numpy-discussion] Changeset 2222 In-Reply-To: <44221F46.9000607@ieee.org> References: <44221F46.9000607@ieee.org> Message-ID: On 3/22/06, Travis Oliphant wrote: > Good point. Needs to be changed. I'll do it. From Fernando.Perez at colorado.edu Wed Mar 22 22:22:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 22 22:22:02 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44220366.4000004@cox.net> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220366.4000004@cox.net> Message-ID: <44223E4A.9090208@colorado.edu> Tim Hochberg wrote: > This did fix things...eventually. I removed the build directory and > rebuilt. Numpy still bombed the interpreter, although in a new and less > exciting way. I then did a clean checkout from svn and rebuilt. Same > thing. Then I manally removed numpy from site-packages and reinstalled > at which points things worked and all tests passed. > > So in addition to needing to remove the build directory, it appears that > something was hanging out in the destination directory that did not get > overwritten and caused problems. > > Perhaps this description will help the next person who runs into this, > but probably not. Well, the script that I posted does precisely all of this in a mindless fashion, which is why I find it useful. You may want to give it a try (it's trivial, but it just ensures you don't forget these steps and end up wasting any time chasing the problem). Cheers, f From mpuqwpizl at mayerbrown.com Wed Mar 22 23:56:06 2006 From: mpuqwpizl at mayerbrown.com (mpuqwpizl) Date: Wed Mar 22 23:56:06 2006 Subject: [Numpy-discussion] Fw: mpuqwpizl Message-ID: <000901c64e4f$21d92660$ba0278c9@fantasy> ----- Original Message ----- From: Abraham Jeanie To: vqmqthe at wiley.com Sent: Tuesday, March 21, 2006 11:17 PM Subject: numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bdftlqfrty.gif Type: image/gif Size: 16338 bytes Desc: not available URL: From cimrman3 at ntc.zcu.cz Thu Mar 23 01:51:05 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Thu Mar 23 01:51:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4421F455.2020003@colorado.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> <4421F455.2020003@colorado.edu> Message-ID: <44226F75.8010401@ntc.zcu.cz> Fernando Perez wrote: > Done, both for pyrex and swig. I added very simple notes, feel free to > add further detail as necessary: > > http://scipy.org/Cookbook/Pyrex_and_NumPy > http://scipy.org/Cookbook/SWIG_and_NumPy Thanks again, I have made just a little tweak to the SWIG_and_NumPy page. r. From Fernando.Perez at colorado.edu Thu Mar 23 06:27:07 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Mar 23 06:27:07 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <44220001.4050606@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> Message-ID: <4422AFF9.5000200@colorado.edu> Tim Hochberg wrote: > My opinion is that all methods and functions should either: > > 1. Always return a copy. > 2. Always return a view > 3. Return a view if possible otherwise raise an exception. Well, but is copy/view the /only/ invariant worth guaranteeing? I think there is a valid need for functions which ensure other invariants, such as contiguity. There are applications (such as passing pointers to C/Fortran libraries which don't have striding mechanisms but will not modify their inputs) which require contiguous inputs, but where one would rather make a copy only if necessary. My take on this is that we should /document/ clearly what invariants any given function satisfies, but I think the 'always view/always copy' view excludes an important usage case. There may be others beyond contiguity, but that's the one that pops immediately to mind. Cheers, f From bsouthey at gmail.com Thu Mar 23 06:34:11 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Mar 23 06:34:11 2006 Subject: [Numpy-discussion] Re: Numpy-discussion digest, Vol 1 #1632 - 10 msgs In-Reply-To: <200603222035.08682.pierregm@engr.uga.edu> References: <20060322234311.30FA78974B@sc8-sf-spam1.sourceforge.net> <200603222035.08682.pierregm@engr.uga.edu> Message-ID: Hi, I think that there are different behaviors being obtained. One occurs when a scalar value is used and another occurs when an array is being used. In the first is it just an extension of masking the whole array twice with two different values. In the second case it seems more of an element by element masking with two different values, which requires creating min and max arrays. There is also a third case of a single dimension so min and max vectors to operate on that dimension. In anycase, the values should only be masked because it allows the values to be unmasked as necessary. In which case there needs to be an appropriate unmask function such that in the extreme one of limits could be removed. Regards Bruce On 3/22/06, Pierre GM wrote: > > Pierre has implemented the clip method for ma.array, but his > > implementation does not seem to do the right thing if min or max is > > masked: > > From core/oldnumeric.py: > """clip(m, m_min, m_max) = every entry in m that is less than m_min is > replaced by m_min, and every entry greater than m_max is replaced by > m_max. > """ > My understanding was that m_min and m_max were two floats, not two arrays. I > completely oversaw that. Your suggestion (masking where the limits are > masked) makes complete sense. I'll work on that. > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From tim.hochberg at cox.net Thu Mar 23 08:03:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 08:03:07 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44223E4A.9090208@colorado.edu> References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220366.4000004@cox.net> <44223E4A.9090208@colorado.edu> Message-ID: <4422C668.8020802@cox.net> Fernando Perez wrote: > Tim Hochberg wrote: > >> This did fix things...eventually. I removed the build directory and >> rebuilt. Numpy still bombed the interpreter, although in a new and >> less exciting way. I then did a clean checkout from svn and rebuilt. >> Same thing. Then I manally removed numpy from site-packages and >> reinstalled at which points things worked and all tests passed. >> >> So in addition to needing to remove the build directory, it appears >> that something was hanging out in the destination directory that did >> not get overwritten and caused problems. >> >> Perhaps this description will help the next person who runs into >> this, but probably not. > > > Well, the script that I posted does precisely all of this in a > mindless fashion, So it does. I should have looked at that more closely. > which is why I find it useful. You may want to give it a try (it's > trivial, but it just ensures you don't forget these steps and end up > wasting any time chasing the problem). Well, I'd probably have to rewrite it to get it work under windows. I may do that, although 'rm -rf' always scares me (even though it looks pretty safe here). Still, it would be nice if 'setup.py clean' and 'setup.py install' did this automagically. -tim > > Cheers, > > f > > From tim.hochberg at cox.net Thu Mar 23 08:12:09 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 08:12:09 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422AFF9.5000200@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> Message-ID: <4422C8A8.2050704@cox.net> Fernando Perez wrote: > Tim Hochberg wrote: > >> My opinion is that all methods and functions should either: >> >> 1. Always return a copy. >> 2. Always return a view >> 3. Return a view if possible otherwise raise an exception. > > > Well, but is copy/view the /only/ invariant worth guaranteeing? I > think there is a valid need for functions which ensure other > invariants, such as contiguity. There are applications (such as > passing pointers to C/Fortran libraries which don't have striding > mechanisms but will not modify their inputs) which require contiguous > inputs, but where one would rather make a copy only if necessary. This is a different case, I think. The result of this copy is not user visible except in terms of performance. I'm only concerned with functions that *return* copies or views depending on the input. I don't care if a function sometimes makes a copy under the covers but doesn't return it. > My take on this is that we should /document/ clearly what invariants > any given function satisfies, but I think the 'always view/always > copy' view excludes an important usage case. There may be others > beyond contiguity, but that's the one that pops immediately to mind. I don't think we're in disagreement here although I'm not sure. I will add, on the subject of continuity, that I think there should be a function 'ascontiguous' that parallels asarray, but assures that the result is contiguous. Although this sometimes returns a copy, I thinks that's OK since that's it's job. I would like to see all of the implicit copying pushed into functions like asarray and ascontiguous. This also helps efficiency. Imagine I have some calls to functions that require contiguous arrays and do copies under the covers if their args are not contiguous. In that case: a = ascontiguous(a) param1 = computeSomethingOnContiguousData(a) param2 = computeSomethingElseOnContiguousData(a) # etc. Will be much more efficient than the equivalent code without the ascontiguous when the initial a value is not discontiguous. Regards, -tim From robert.kern at gmail.com Thu Mar 23 08:25:11 2006 From: robert.kern at gmail.com (Robert Kern) Date: Thu Mar 23 08:25:11 2006 Subject: [Numpy-discussion] Re: [Numpy-user] random.randint bug ? In-Reply-To: References: Message-ID: <4422CBD1.7020606@gmail.com> Randewijk P-J wrote: > I'm having problems with randint: > > >>>>random.randint(low=1,high=10) > > 4L > >>>>random.randint(low=1,high=10) > > 7L > >>>>random.randint(low=1,high=10) > > 5L > >>>>random.randint(low=1,high=10) > > 7L > >>>>random.randint(low=1,high=10) > > 4L > >>>>random.randint(low=1,high=10) > > 0L > > The last value is wrong because it is < "low"/"1" > > The randint value should be: 1 <= value < 10 > >>>>help(random.randint) > > Help on built-in function randint: > > randint(...) > Return random integers x such that low <= x < high. > > randint(low, high=None, size=None) -> random values > > If high is None, then 0 <= x < low. > > > > Running Python 2.4.2 and NumPy 0.9.6 Fixed in SVN. Thank you. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Fernando.Perez at colorado.edu Thu Mar 23 09:02:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Mar 23 09:02:06 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422C8A8.2050704@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> Message-ID: <4422D46D.8080800@colorado.edu> Tim Hochberg wrote: > I don't think we're in disagreement here although I'm not sure. > > I will add, on the subject of continuity, that I think there should be a > function 'ascontiguous' that parallels asarray, but assures that the > result is contiguous. Although this sometimes returns a copy, I thinks > that's OK since that's it's job. I would like to see all of the implicit > copying pushed into functions like asarray and ascontiguous. I think we agree: something like ascontiguous() is precisely what I had in mind (I think that's what ravel() does today, but I'm fine if it gets a new name, as long as the functionality exists). Obviously a function like this should explicitly (docstring) say that it does NOT make any guarantees about whether its return value is a view or a copy, just that it's contiguous. Cheers, f From mfmorss at aep.com Thu Mar 23 09:05:03 2006 From: mfmorss at aep.com (mfmorss at aep.com) Date: Thu Mar 23 09:05:03 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <442210A7.7080802@ieee.org> Message-ID: Oh for the days of imperative programming! y=f(x,...) x=f(x,...) was just too easy, I guess. Mark F. Morss Principal Analyst, Market Risk American Electric Power Travis Oliphant To Sent by: numpy-discussion numpy-discussion- eforge.net cc Subject 03/22/2006 10:06 Re: [Numpy-discussion] Current SVN PM segfaults under VC7 Tim Hochberg wrote: > > OK, now that I've got things working, I can report a real segfault > under VC7. The following snippet of code: > > from numpy import * > a = arange(100.) > b = array(1+1j) > (a+1) ** (b+3) I can reproduce it on Linux. It looks like it's related to the power optimization stuff that was recently checked in. power(a,b+3) does not crash but >>> a**(b+3) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1076248704 (LWP 1545)] 0x404162df in array_power_is_scalar (o2=0x8294200, exp=0xbfffec68) at arrayobject.c:2656 2656 arrayobject.c: No such file or directory. in arrayobject.c -Travis ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From tim.hochberg at cox.net Thu Mar 23 09:37:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 09:37:04 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422D46D.8080800@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> Message-ID: <4422DC96.9030307@cox.net> Fernando Perez wrote: > Tim Hochberg wrote: > >> I don't think we're in disagreement here although I'm not sure. >> >> I will add, on the subject of continuity, that I think there should >> be a function 'ascontiguous' that parallels asarray, but assures that >> the result is contiguous. Although this sometimes returns a copy, I >> thinks that's OK since that's it's job. I would like to see all of >> the implicit copying pushed into functions like asarray and >> ascontiguous. > > > I think we agree: something like ascontiguous() is precisely what I > had in mind (I think that's what ravel() does today, but I'm fine if > it gets a new name, as long as the functionality exists). a.ravel() seems to be equivalent to reshape(a, [-1]). That is, it returns a flattened, contiguous copy. ascontiguous(a) would be slightly different in that it would preserves the shape of a. In fact I think it would look a lot like: def ascontiguous(a): """ascontiguous(a) -> contiguous representation of a. If 'a' is allready contiguous, it is returned unchanged. Otherwise, a contiguous copy is returned. """ a = asarray(a) if not a.flags['CONTIGUOUS']: a = array(a) return a > Obviously a function like this should explicitly (docstring) say that > it does NOT make any guarantees about whether its return value is a > view or a copy, just that it's contiguous. I agree. Regards, -tim From Chris.Barker at noaa.gov Thu Mar 23 09:59:04 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 09:59:04 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4421F455.2020003@colorado.edu> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> <4421F455.2020003@colorado.edu> Message-ID: <4422E1CA.70202@noaa.gov> Fernando Perez wrote: > Done, both for pyrex and swig. I added very simple notes, feel free to > add further detail as necessary: > > http://scipy.org/Cookbook/Pyrex_and_NumPy > http://scipy.org/Cookbook/SWIG_and_NumPy Thanks, this is great. Has anyone done this (for either SWIG or Pyrex) for the generic array interface, rather than explicitly for the numpy types? I'd love to see examples of how to write extensions that can take ANY nd-array. (I now that's only NumPy, Numeric and numarray now, but I'm hopeful it will become a standard) -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Fernando.Perez at colorado.edu Thu Mar 23 10:07:04 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Mar 23 10:07:04 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422DC96.9030307@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> Message-ID: <4422E397.6030206@colorado.edu> Tim Hochberg wrote: > a.ravel() seems to be equivalent to reshape(a, [-1]). That is, it > returns a flattened, contiguous copy. ascontiguous(a) would be slightly > different in that it would preserves the shape of a. In fact I think it > would look a lot like: > > def ascontiguous(a): > """ascontiguous(a) -> contiguous representation of a. > > If 'a' is allready contiguous, it is returned unchanged. Otherwise, > a contiguous copy > is returned. > > """ > a = asarray(a) > if not a.flags['CONTIGUOUS']: > a = array(a) > return a Yup, sorry: I've been thinking too much in terms of raw C (double*) pointers so to me, these days everything is 1-d :) I forgot about shape issues... But I think we agree on the basic point. Cheers, f From faltet at carabos.com Thu Mar 23 10:17:12 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Mar 23 10:17:12 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422E1CA.70202@noaa.gov> References: <4414BDEF.70001@colorado.edu> <4421F455.2020003@colorado.edu> <4422E1CA.70202@noaa.gov> Message-ID: <200603231916.41334.faltet@carabos.com> A Dijous 23 Mar? 2006 18:58, Christopher Barker va escriure: > > http://scipy.org/Cookbook/Pyrex_and_NumPy > > http://scipy.org/Cookbook/SWIG_and_NumPy > Has anyone done this (for either SWIG or Pyrex) for the generic array > interface, rather than explicitly for the numpy types? I'd love to see > examples of how to write extensions that can take ANY nd-array. (I now > that's only NumPy, Numeric and numarray now, but I'm hopeful it will > become a standard) I don't think this is feasible yet, although I don't think this is a big limitation. You can always compile your program against NumPy or numarray or Numeric and follow the previous recipes (adapted to the package of choice) to access to its data. The only thing that you must be aware of is to convert all the objects that foes to the extension to an object of the package of choice. For example, if you have chosen NumPy as your package to compile with, then you only have to convert Numeric/numarray objects into a NumPy one with: native_numpy_object = numpy.asarray(your_foreign_object) The array protocol will ensure that the conversion will be as efficient as possible. What's more, if the "foreign" objects are well-behaved (i.e. contiguous and native-endian ordered), then a data copy will not be made, so letting you to access to the original data address. PyTables follows this approach and I must say that's very handy. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant at ee.byu.edu Thu Mar 23 10:36:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 10:36:01 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422E1CA.70202@noaa.gov> References: <4414BDEF.70001@colorado.edu> <4415A968.40807@ftw.at> <4415AB45.3000702@ieee.org> <1142313130.441650aad8111@webmail.colorado.edu> <44217D77.7030403@ntc.zcu.cz> <4421F455.2020003@colorado.edu> <4422E1CA.70202@noaa.gov> Message-ID: <4422EA6E.1070401@ee.byu.edu> Christopher Barker wrote: > Fernando Perez wrote: > >> Done, both for pyrex and swig. I added very simple notes, feel free >> to add further detail as necessary: >> >> http://scipy.org/Cookbook/Pyrex_and_NumPy >> http://scipy.org/Cookbook/SWIG_and_NumPy > > > Thanks, this is great. > > Has anyone done this (for either SWIG or Pyrex) for the generic array > interface, rather than explicitly for the numpy types? I'd love to see > examples of how to write extensions that can take ANY nd-array. (I now > that's only NumPy, Numeric and numarray now, but I'm hopeful it will > become a standard) Well, there is the __array_struct__ method that should return a C-Object pointer with all the information in it about the array. To see how to use this look at the Numeric or numpy code (do a grep on __array_struct__ and you will find the relevant sections of code). I noticed that the Pyrex cookbook had an example with using the generic interface. -Travis From rowen at cesmail.net Thu Mar 23 10:37:08 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Thu Mar 23 10:37:08 2006 Subject: [Numpy-discussion] Re: Problems installing NumPy References: <4E0ECB41-EF32-4F0E-9DAE-52B55C8AE3A0@mac.com> Message-ID: In article <4E0ECB41-EF32-4F0E-9DAE-52B55C8AE3A0 at mac.com>, Tommy Grav wrote: > I am new to Python and just downloaded ActivePython 2.4.2.10 on my > Mac PPC with OS X 10.4. > I added the numpy package (0.9.6-py2.4) and it imports fine. But when > I try pydoc random I > get this error > > [tgrav at skathi] site-packages/numpy -> pydoc random > import core -> failed: dlopen(/Library/Frameworks/Python.framework/ > Versions/2.4/lib/python2.4/site-packages/numpy/core/multiarray.so, > 2): Symbol not found: _PyOS_ascii_strtod > Referenced from: /Library/Frameworks/Python.framework/Versions/2.4/ > lib/python2.4/site-packages/numpy/core/multiarray.so > Expected in: dynamic lookup >... > Anyone know what is causing this problem? Is there something in my > installation > of either ActivePython or NumPy that needs to be fixed? I'm not entirely clear on how what you're typing to get it to fail, but perhaps there is a problem in the package installer. If you suspect this, you can easily install numpy from source. -- Russell From tim.hochberg at cox.net Thu Mar 23 11:03:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 11:03:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4422E397.6030206@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> Message-ID: <4422F0A4.7010206@cox.net> I was just looking at the interface for array and asarray to see what other stuff should go in the interface of the hypothetical ascontiguous. There's 'dtype', which I knew about, and 'fortran', which I didn't, but which makes sense. However, there's also 'ndmin'. First off, it's not described in docstring for asarray, but I was able to find it in the docstring for array without a problem. Second, is it really necessary? It seems to be useful in an awfully narrow set of circumstances, particularly since when you are padding axes not everyone wants to pad to the left. It would seem to be more useful to ditch the ndmin and have some sort of paddims function that was more full featured (padding to either the left or the right at a minimum). I'm not entirely sure what the best interface to such a beast would look like, but a simple tactic would be to just provide leftpaddims and rightpaddims. If it's not allready clear by now (;), I prefer several narrow interfaces to one broad one. -tim From Chris.Barker at noaa.gov Thu Mar 23 11:26:16 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 11:26:16 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422AFF9.5000200@colorado.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> Message-ID: <4422F602.1060809@noaa.gov> Fernando Perez wrote: > Well, but is copy/view the /only/ invariant worth guaranteeing? I think > there is a valid need for functions which ensure other invariants, such > as contiguity. quite true. > My take on this is that we should /document/ clearly what invariants any > given function satisfies, but I think the 'always view/always copy' view > excludes an important usage case. There may be others beyond > contiguity, but that's the one that pops immediately to mind. there's asarray() of course. My feeling is that functions that may or may not return a copy should be functions, like asarray(), that ONLY exist to ensure a particular invariant. ascontiguous() asarray() I imagine there are others. What concerns me is functions like reshape() and ravel() that you might have all sorts of other reasons to use, but then can't ever know for sure if your method is going to be working with a copy or not. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Thu Mar 23 11:32:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 11:32:06 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: <4422F602.1060809@noaa.gov> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422F602.1060809@noaa.gov> Message-ID: <4422F79E.1040401@ee.byu.edu> There is the flatten() method which exists for precisely this reason (it *always* returns a copy). -Travis From Chris.Barker at noaa.gov Thu Mar 23 11:41:04 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 11:41:04 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <200603231916.41334.faltet@carabos.com> References: <4414BDEF.70001@colorado.edu> <4421F455.2020003@colorado.edu> <4422E1CA.70202@noaa.gov> <200603231916.41334.faltet@carabos.com> Message-ID: <4422F996.3070303@noaa.gov> Francesc Altet wrote: > A Dijous 23 Mar? 2006 18:58, Christopher Barker va escriure: >> Has anyone done this (for either SWIG or Pyrex) for the generic array >> interface, rather than explicitly for the numpy types? > I don't think this is feasible yet, um, why not? > although I don't think this is a > big limitation. You can always compile your program against NumPy or > numarray or Numeric The goal here is to be able to write code that can be compiled without the presence of numpy, or Numeric, or numarray. Indeed, no additional dependencies, but that can still efficiently use array objects passed in. My classic example is wxPython. Robin doesn't want any dependencies on other packages, so at the moment, when you pass a numpy array in, it uses python's generic sequence access to get the values, then ends up converting them to a plain old C array. This is a lot of unnecessary processing, as you can imagine. I think it can be done, and one of these days I'll figure out how, but I'd love to see an example from someone who understands these things more than myself. Travis Oliphant wrote: > Well, there is the __array_struct__ method that should return a C-Object > pointer with all the information in it about the array. > To see how to use this look at the Numeric or numpy code (do a grep on > __array_struct__ and you will find the relevant sections of code). That's exactly what I mean. Now I just need to figure out how to use it! > I noticed that the Pyrex cookbook had an example with using the generic > interface. Wonderful, I'll go look there. Do you mean the Pyrex section in the SciPy cookbook? I can't find a Pyrex cookbook. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From faltet at carabos.com Thu Mar 23 11:43:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Mar 23 11:43:06 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422EA6E.1070401@ee.byu.edu> References: <4414BDEF.70001@colorado.edu> <4422E1CA.70202@noaa.gov> <4422EA6E.1070401@ee.byu.edu> Message-ID: <200603232042.38896.faltet@carabos.com> A Dijous 23 Mar? 2006 19:35, Travis Oliphant va escriure: > Well, there is the __array_struct__ method that should return a C-Object > pointer with all the information in it about the array. > > To see how to use this look at the Numeric or numpy code (do a grep on > __array_struct__ and you will find the relevant sections of code). > > I noticed that the Pyrex cookbook had an example with using the generic > interface. I've just noticed it. That's great, but I don't understand the code quite well. In particular, if, as the introduction says, this offers "a view of the data without actually copying the data itself", then why the next line is needed?: self.data=malloc( bufsize ) Oh, well, I think I've to study more the example. Thanks! -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Thu Mar 23 11:48:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Mar 23 11:48:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <4422F996.3070303@noaa.gov> References: <4414BDEF.70001@colorado.edu> <200603231916.41334.faltet@carabos.com> <4422F996.3070303@noaa.gov> Message-ID: <200603232047.00745.faltet@carabos.com> A Dijous 23 Mar? 2006 20:40, Christopher Barker va escriure: > > I don't think this is feasible yet, > > um, why not? For no particular reason, except my ignorance ;-) > The goal here is to be able to write code that can be compiled without > the presence of numpy, or Numeric, or numarray. Indeed, no additional > dependencies, but that can still efficiently use array objects passed in. > > My classic example is wxPython. Robin doesn't want any dependencies on > other packages, so at the moment, when you pass a numpy array in, it > uses python's generic sequence access to get the values, then ends up > converting them to a plain old C array. This is a lot of unnecessary > processing, as you can imagine. > > I think it can be done, and one of these days I'll figure out how, but > I'd love to see an example from someone who understands these things > more than myself. Yeah, I understand you perfectly well :-) > Do you mean the Pyrex section in the SciPy cookbook? I can't find a > Pyrex cookbook. I think the referred page is here: http://scipy.org/Cookbook/ArrayStruct_and_Pyrex -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From strawman at astraw.com Thu Mar 23 12:06:02 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu Mar 23 12:06:02 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <200603232042.38896.faltet@carabos.com> References: <4414BDEF.70001@colorado.edu> <4422E1CA.70202@noaa.gov> <4422EA6E.1070401@ee.byu.edu> <200603232042.38896.faltet@carabos.com> Message-ID: <4422FF72.7060208@astraw.com> This example makes an __array_struct__ view of data you malloc elsewhere. In other words, it's only half of the full suite of what people may want to do. It covers elsewhere -> numpy/numarray/Numeric/__array_struct__ consumer. It does not (yet) cover numpy/__array_struct__ producer -> elsewhere. This would be a great addition, but nontrivial to cover the full spectrum of memory allocation strategies that numpy currently provides. A first step might be to support only a limited set and raise a NotImplementedError if something more complicated is encountered. I've actually been quite happy in my code doing something like allocating my own memory layouts, passing a view of them to numpy using this interface, doing any manipulation there, and then continuing on in my own code. This could, for example, be used to make a numpy view of a wx-allocated buffer... FWIW, I'm using Intel's IPP to both allocate buffers and further manipulate them, with my Python interface significantly eased by a numpy view of the data. But, like I say, I understand this is only half the solution... The wiki is open for submissions! Cheers! Andrew Francesc Altet wrote: >I've just noticed it. That's great, but I don't understand the code >quite well. In particular, if, as the introduction says, this offers >"a view of the data without actually copying the data itself", then >why the next line is needed?: > > self.data=malloc( bufsize ) > >Oh, well, I think I've to study more the example. > >Thanks! > > > From Chris.Barker at noaa.gov Thu Mar 23 12:09:05 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Mar 23 12:09:05 2006 Subject: [Numpy-discussion] Access to C/C++, typemaps, pyrex... In-Reply-To: <200603232047.00745.faltet@carabos.com> References: <4414BDEF.70001@colorado.edu> <200603231916.41334.faltet@carabos.com> <4422F996.3070303@noaa.gov> <200603232047.00745.faltet@carabos.com> Message-ID: <4423003E.9@noaa.gov> Francesc Altet wrote: > I think the referred page is here: > > http://scipy.org/Cookbook/ArrayStruct_and_Pyrex Yes, I did find that. That looks like it creates an Array_Struct, for use with other code, rather than using one that is already created. Hence the malloc. It look like there is the intent to add what I am looking for: """ Using data malloced elsewhere with a Pyrex-based extension Coming soon... """ Is it soon yet Andrew? ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jimpy at oublic.org Thu Mar 23 12:12:02 2006 From: jimpy at oublic.org (Jim Carroll) Date: Thu Mar 23 12:12:02 2006 Subject: [Numpy-discussion] Newbie: Matrix form of segment-sphere intersection? Message-ID: Hi, I'm considering using matricies for some math that determines where segments intersect spheres. I have pairs of 3d coordinates P1, P2; P3, P4; P5, P6; ... Pn-1, Pn. one is inside a sphere, and the other is outside the same sphere. I want the point where segment joining the point intersects the sphere. The math is straightforward enough done in a function (for example: http://www.siggraph.org/education/materials/HyperGraph/raytrace/rtinter1.htm) Are matricies appropriate as a way of applying this sort of computation to a large number of points? Where can I find applicable tutorials or recipies? Thanks, -Jim From oliphant at ee.byu.edu Thu Mar 23 12:50:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 12:50:07 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4422F0A4.7010206@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> Message-ID: <442309CE.1090005@ee.byu.edu> Tim Hochberg wrote: > > > I was just looking at the interface for array and asarray to see what > other stuff should go in the interface of the hypothetical > ascontiguous. There's 'dtype', which I knew about, and 'fortran', > which I didn't, but which makes sense. However, there's also 'ndmin'. > First off, it's not described in docstring for asarray, but I was able > to find it in the docstring for array without a problem. Second, is it > really necessary? It seems to be useful in an awfully narrow set of > circumstances, particularly since when you are padding axes not > everyone wants to pad to the left. Padding to the left is "default" behavior for broadcasting and so it seems appropriate. This is how all lower-dimensional arrays are interpreted as "higher" dimensional arrays throughout the code. The ndmin is very handy as atested to by the uses of atleast_1d or atleast_2d in numpy library code. It was added later as an optimization step because of the number of library routines that were using it. I've since used it several times to simplify code. I think an ascontinguous on the Python level is appropriate since such a beast exists on the C-level. On the other hand, while Tim prefers narrow interfaces, the array_from_object interface is traditionally pretty broad. Thus, in my mind, the array call should get another flag keyword that forces a contiguous result. This translates easily to the C-domain, in much the same way as the fortran keyword does. -Travis From tim.hochberg at cox.net Thu Mar 23 13:47:08 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 13:47:08 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442309CE.1090005@ee.byu.edu> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <44231741.3020308@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> >> I was just looking at the interface for array and asarray to see what >> other stuff should go in the interface of the hypothetical >> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >> which I didn't, but which makes sense. However, there's also 'ndmin'. >> First off, it's not described in docstring for asarray, but I was >> able to find it in the docstring for array without a problem. Second, >> is it really necessary? It seems to be useful in an awfully narrow >> set of circumstances, particularly since when you are padding axes >> not everyone wants to pad to the left. > > > > Padding to the left is "default" behavior for broadcasting and so it > seems appropriate. This is how all lower-dimensional arrays are > interpreted as "higher" dimensional arrays throughout the code. That makes some sense. > > The ndmin is very handy as atested to by the uses of atleast_1d or > atleast_2d in numpy library code. It was added later as an > optimization step because of the number of library routines that were > using it. I've since used it several times to simplify code. OK, I'll take your word for it. > I think an ascontinguous on the Python level is appropriate since such > a beast exists on the C-level. On the other hand, while Tim prefers > narrow interfaces, the array_from_object interface is traditionally > pretty broad. Thus, in my mind, the array call should get another flag > keyword that forces a contiguous result. This doesn't bother me since I long ago gave up any hope that the array constructor would have a narrow interface. > This translates easily to the C-domain, in much the same way as the > fortran keyword does. I'll buy that. While I accept array() needs a wide interface, I still prefer to keep as many other interfaces as possible narrow. In particular, is ndmin widely used in asarray? Or do the library routines generally use array instead. Given the choice I'd sweep as much of the dust, AKA wideness, into array90 as possible since that's irredeemably wide anyway and keep the other interfaces as narrowly focused as possible. Put another way, asarray and ascontiguous are about clarity of intent. With too much extra baggage, the intent becomes obscured. The coupling seems tight enough for dtype and fortran, but once you get to ndmin, it seems that you might as well go with the big guns and break out "array(x, copy=False, ndmin=n)". That's my $0.02 on this subject and I'll shut up about it now. -tim From travis at enthought.com Thu Mar 23 13:57:02 2006 From: travis at enthought.com (Travis N. Vaught) Date: Thu Mar 23 13:57:02 2006 Subject: [Numpy-discussion] ANN: Python Enthought Edition Version 0.9.3 Released Message-ID: <442319A6.7040108@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 0.9.3 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 0.9.3 Release Notes: -------------------- Version 0.9.3 of Python Enthought Edition includes an update to version 1.0.3 of the Enthought Tool Suite (ETS) Package-- you can look at the release notes for this ETS version here. Other major changes include: * upgrade to VTK 5.0 * addition of docutils * addition of numarray * addition of pysvn. Also, MayaVi issues should be fixed in this release. Full Release Notes are here: http://code.enthought.com/release/changelog-enthon0.9.3.shtml About Python Enthought Edition: ------------------------------- Python 2.3.5, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numeric SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com From oliphant at ee.byu.edu Thu Mar 23 13:59:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 13:59:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <44231A04.2000804@ee.byu.edu> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>Thus, in my mind, the array call should get another flag keyword that >>forces a contiguous result. >> >> > >Please don't! The fortran flag is bad enough, but has too much history >behind it. Let's not breed boolean parameters. Sooner or later >someone will use keword arguments positionally and you will end up >guessing what > > There are several boolean flags in the interface already. Adding another one won't change the current situation that you describe. There are several ways to handle this. For one, we could force the use of keyword arguments, so that the position problem does not arise. Sasha has mentioned in the past a strides array argument, but I think the default fortran and contiguous strides cases need better support then just one of many possible stridings so I wouldn't go that direction here. I'm debating whether or not the fortran flag should be used to specify both contiguous and fortran cases. Right now, the fortran argument is a three-case flag with dont-care, True, and False arguments. It seems natural to have True mean force-fortran and False mean force-contiguous with dont-care (the default) mean take an array already given (or create a C-contiguous array if we are generating a new array from another object). At any rate, if the fortran flag is there, we need to specify the contiguous case as well. So, either propose a better interface (we could change it still --- the fortran flag doesn't have that much history) to handle the situation or accept what I do ;-) -Travis From ndarray at mac.com Thu Mar 23 14:22:04 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 23 14:22:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442309CE.1090005@ee.byu.edu> References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: On 3/23/06, Travis Oliphant wrote: > Thus, in my mind, the array call should get another flag keyword that > forces a contiguous result. Please don't! The fortran flag is bad enough, but has too much history behind it. Let's not breed boolean parameters. Sooner or later someone will use keword arguments positionally and you will end up guessing what array([1,2], int8_, 1, 1, 0, 0) means. From oliphant at ee.byu.edu Thu Mar 23 14:27:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 14:27:05 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44231741.3020308@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231741.3020308@cox.net> Message-ID: <44232098.7020901@ee.byu.edu> Tim Hochberg wrote: > > That makes some sense. > >> >> The ndmin is very handy as atested to by the uses of atleast_1d or >> atleast_2d in numpy library code. It was added later as an >> optimization step because of the number of library routines that were >> using it. I've since used it several times to simplify code. > > > OK, I'll take your word for it. Not necessarily sane advice :-) --- I might be overstating things. I know that atleast_1d and atleast_2d are used all over the place in scipy. This makes sense and I can certainly understand it. I'm willing to modify things to give narrow interfaces. Right now, since the requesting both fortran and contiguous does not make sense, setting the fortran flag to false enforces C-style contiguous while setting it to True enforces fortran-style. Setting it to None (the default) specifies you don't care and the behavior will be to create C-style contiguous for new arrays and use the striding specified by the array if it's already an array. I admit that it is arguable whether or not the fortran flag should be overloaded like this or not. There are now ascontiguous and asfortran functions with fairly minimal interfaces to make it simpler. There is also a check to make sure that array is called with no more than 2 non keyword arguments. Thus, you won't be able to confuse which flag is which. -Travis From Norbert.Nemec.list at gmx.de Thu Mar 23 15:04:04 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu Mar 23 15:04:04 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's In-Reply-To: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> References: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> Message-ID: <44232949.1000606@gmx.de> The current SVN version of numpy.linalg does not contain "eigenvectors" any more (since the cleanup - patches I submitted last week), so you are obviously using something older. In any case, numpy.linalg.eig (and eigvals) do not seem to contain any check against NaNs, which probably is a bad idea. scipy.linalg.eig does contain such a check, but that module is internally very different to that in numpy. numpy.linalg certainly needs more cleanup. In that course, adding safety-checks certainly is one task. Chris Fonnesbeck wrote: > If linalg.eigenvectors is called on a matrix that contains nan values, > it hangs, quickly consuming system resources. Here is the matrix: > > matrix = [[ 0. 0. 0. 0. > 0.36658624 0. 0. > 0. 0. 0. ] > [ 0.89827259 0. 0. 0. 0. > 0. 0. > 0. 0. 0. ] > [ 0. 0.92510948 0. 0. 0. > 0. 0. > 0. 0. 0. ] > [ 0. 0. nan nan 0. > 0. 0. > 0. 0. 0. ] > [ 0. 0. nan nan 0. nan > 0. 0. 0. 0. ] > [ 0. 0. 0. 0. 0.93319244 nan > 0. 0. 0. 0. ] > [ 0. 0. 0. 0. 0.36658624 > 0. 0. > 0. 0. 0. ] > [ 0. 0. 0. 0. 0. 0. > 0.89827259 0. 0. 0. ] > [ 0. 0. 0. 0. 0. > 0. 0. > 0.92510948 0. 0. ] > [ 0. 0. 0. 0. 0. > 0. 0. > 0. 0.92510948 0.93319244]] > > I would expect an exception, but I get none. > > Using a recent cvs build of numpy. > > C. > > -- > Chris Fonnesbeck + Atlanta, GA + http://trichech.us From tim.hochberg at cox.net Thu Mar 23 15:07:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 23 15:07:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <442329E4.70007@cox.net> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>Thus, in my mind, the array call should get another flag keyword that >>forces a contiguous result. >> >> > >Please don't! The fortran flag is bad enough, but has too much history >behind it. Let's not breed boolean parameters. Sooner or later >someone will use keword arguments positionally and you will end up >guessing what > >array([1,2], int8_, 1, 1, 0, 0) > >means. > > We could always force keyword parameter by using **kwd. That would mess mess up help() and other introspection tools. Then again, if we want to discourage overuse of this sort of stuff, perhaps that's not a bad thing ;) But if we really need access to the C guts of an array, just allow a dictionary of flags to get plugged in. These would be the same as what somearray.flags puts out. The new interface for array would be: array(object, **flags) Where flags could contain: CONTIGUOUS : force a copy if object isn't contiguous if True. [default None] FORTRAN : force array to be fortran order if True, C order if False [default None] OWNDATA : force a copy if True [default True] WRITEABLE : Force a copy if object isn't writeable if Tru [default None] ALIGNED : Force a copy if object isn't aligned if True. [default None] UPDATEIFCOPY : Set the UPDATEIFCOPY flag? [default ???] With the exception of the FORTRAN, and possibly UPDATEIFCOPY, it would be an error to set any of these flags to False (Forcing an array to be discontiguous for instance, makes no sense). That's a thin interface and it ties together with the flags parameter nicely. On the downside, it's a little weird, particularly using OWNDATA for copy, although it is logical once you think about it. It also drops 'minnd' and 'subok'. I wouldn't miss them, but I expect someone would squawk. You could shoehorn them into flags, but then you lose one of the chief virtues of this scheme, which is that it makes a strong connection between the constructor and the flags parameter. 'subok' should be pretty well taken care of by 'asanyarray' and it would be easy enough to create an auxilliary function to replicate the 'minnd' functionality. With this kind of narrow interface, it might make sense to allow the flags parameter on all of the asX functions. That makes for a nice uniform, easy to remember, interface. I'm not positive I like this idea yet, but I thought it was interesting enough to throw into the ring anyway. Tangentially, I noticed that I can set ALIGNED to False. Isn't that going to break something somewhere? Should the ALIGNED flag be readonly? -tim From michael.sorich at gmail.com Thu Mar 23 15:27:11 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Thu Mar 23 15:27:11 2006 Subject: [Numpy-discussion] ndarray.fill and ma.array.filled In-Reply-To: References: Message-ID: <16761e100603231526j2a318cc8v877ff0821b76d099@mail.gmail.com> On 3/23/06, Sasha wrote: > > In an ideal world, any function that accepts ndarray would accept > ma.array and vice versa. Moreover, if the ma.array has no masked > elements and the same data as ndarray, the result should be the same. > Obviously current implementation falls short of this goal, but there > is one feature that seems to make this goal unachievable. I am just starting to use ma.array and would like to get some idea from those in the know of how close this is to reality. What percentage of functions designed for nd_arrays would work on a ma.array with no masked elements? That is if you have data with missing values, but then remove the missing values, is it necessary to convert back to a standard nd_array? The statistical language R deals with missing data fairly well. There are a number of functions for dealing with missing values (fail, omit, exclude, pass). Furthermore, there is a relatively standard way for a function to handle data with missing values, via an na.action parameter which indicates which function to call. http://spider.stat.umn.edu/R/library/stats/html/na.action.html http://spider.stat.umn.edu/R/library/stats/html/na.fail.html It would be nice to have a similar set of functions (including the fill function) for numpy. These functions could return the object without change if it is not a masked array, and if a masked array make the appropriate changes to return a nd_array or raise exception. A simple standard for indicating a function's the ability to handle masked data would be to include a mask_action parameter which holds or indicates a function for processing missing data. Also, are there any current plans to allow record type arrays to be masked? Thanks, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Thu Mar 23 15:42:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 23 15:42:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442329E4.70007@cox.net> References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <442329E4.70007@cox.net> Message-ID: <4423321A.3080808@ee.byu.edu> Tim Hochberg wrote: > > Tangentially, I noticed that I can set ALIGNED to False. Isn't that > going to break something somewhere? Should the ALIGNED flag be readonly? Responding to the tangential topic :-) The ALIGNED flag can be set False because it allows one to test those sections of code that deal with misaligned data. I don't think it would break anything, because thinking that data is misaligned when it really is aligned only costs you in copy time. You can't set it TRUE if it's really mis-aligned, though, because thinking that data is aligned when it's really mis-aligned can cause segfaults on some platforms and slow code on others. -Travis From ndarray at mac.com Thu Mar 23 17:31:16 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 23 17:31:16 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44231A04.2000804@ee.byu.edu> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> Message-ID: On 3/23/06, Travis Oliphant wrote: > At any rate, if the fortran flag is there, we need to specify the > contiguous case as well. So, either propose a better interface (we > could change it still --- the fortran flag doesn't have that much > history) to handle the situation or accept what I do ;-) Let me try. I propose to eliminate the fortran flag in favor of a more general "strides" argument. This argument can be either a sequence of integers that becomes the strides, or a callable object that takes shape and dtype arguments and return a sequence that becomes the strides. For fortran and c order functions that generate appropriate stride sequences should be predefined to enable array(..., strides=fortran, ...) and array(..., strides=contiguous). From oliphant.travis at ieee.org Thu Mar 23 21:15:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Mar 23 21:15:04 2006 Subject: [Numpy-discussion] eigenvectors() hangs on nan's In-Reply-To: <44232949.1000606@gmx.de> References: <723eb6930603220829u576bc8a5x3ffb5387358a1011@mail.gmail.com> <44232949.1000606@gmx.de> Message-ID: <4423802D.2080505@ieee.org> Norbert Nemec wrote: > The current SVN version of numpy.linalg does not contain "eigenvectors" > any more (since the cleanup - patches I submitted last week), so you are > obviously using something older. > > In any case, numpy.linalg.eig (and eigvals) do not seem to contain any > check against NaNs, which probably is a bad idea. scipy.linalg.eig does > contain such a check, but that module is internally very different to > that in numpy. > > numpy.linalg certainly needs more cleanup. In that course, adding > safety-checks certainly is one task. > Which routines don't take nans and on which platforms? It would be nice to know before checking for nans on every array. My platform (linux AMD 32-bit) seems to handle nan's fine in the linalg.eig code. Perhaps we do need to check every array for nans, but it would be nice if we could isolate it to certain platforms. -Travis From aisaac at american.edu Thu Mar 23 21:55:04 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu Mar 23 21:55:04 2006 Subject: [Numpy-discussion] Newbie: Matrix form of segment-sphere intersection? In-Reply-To: References: Message-ID: On Thu, 23 Mar 2006, (UTC) Jim Carroll apparently wrote: > Where can I find applicable tutorials or recipies? Don't know the lit, but noticed this: http://www.olympus.net/personal/7seas/contain.html fwiw, Alan Isaac From travis at enthought.com Thu Mar 23 21:57:05 2006 From: travis at enthought.com (Travis N. Vaught) Date: Thu Mar 23 21:57:05 2006 Subject: [Numpy-discussion] [wxPython-users] ANN: Python Enthought Edition Version 0.9.3 Released Message-ID: <442319A6.7040108@enthought.com> Enthought is pleased to announce the release of Python Enthought Edition Version 0.9.3 (http://code.enthought.com/enthon/) -- a python distribution for Windows. 0.9.3 Release Notes: -------------------- Version 0.9.3 of Python Enthought Edition includes an update to version 1.0.3 of the Enthought Tool Suite (ETS) Package-- you can look at the release notes for this ETS version here. Other major changes include: * upgrade to VTK 5.0 * addition of docutils * addition of numarray * addition of pysvn. Also, MayaVi issues should be fixed in this release. Full Release Notes are here: http://code.enthought.com/release/changelog-enthon0.9.3.shtml About Python Enthought Edition: ------------------------------- Python 2.3.5, Enthought Edition is a kitchen-sink-included Python distribution for Windows including the following packages out of the box: Numeric SciPy IPython Enthought Tool Suite wxPython PIL mingw f2py MayaVi Scientific Python VTK and many more... More information is available about all Open Source code written and released by Enthought, Inc. at http://code.enthought.com --------------------------------------------------------------------- To unsubscribe, e-mail: wxPython-users-unsubscribe at lists.wxwidgets.org For additional commands, e-mail: wxPython-users-help at lists.wxwidgets.org From cjw at sympatico.ca Fri Mar 24 04:38:16 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 04:38:16 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> Message-ID: <4423E7A5.3010507@sympatico.ca> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>Thus, in my mind, the array call should get another flag keyword that >>forces a contiguous result. >> >> > >Please don't! > +1 >The fortran flag is bad enough, but has too much history >behind it. Let's not breed boolean parameters. Sooner or later >someone will use keword arguments positionally and you will end up >guessing what > >array([1,2], int8_, 1, 1, 0, 0) > >means. > > The use of keyword parameters can help here. Various editors use them to provide tips and there is the help function. Colin W. From tim.hochberg at cox.net Fri Mar 24 06:33:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 06:33:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> Message-ID: <442402E5.6090901@cox.net> Sasha wrote: >On 3/23/06, Travis Oliphant wrote: > > >>At any rate, if the fortran flag is there, we need to specify the >>contiguous case as well. So, either propose a better interface (we >>could change it still --- the fortran flag doesn't have that much >>history) to handle the situation or accept what I do ;-) >> >> > >Let me try. I propose to eliminate the fortran flag in favor of a more >general "strides" argument. This argument can be either a sequence of >integers that becomes the strides, or a callable object that takes >shape and dtype arguments and return a sequence that becomes the >strides. For fortran and c order functions that generate appropriate >stride sequences should be predefined to enable array(..., >strides=fortran, ...) and array(..., strides=contiguous). > I like the idea of being able to create an array with custom strides. The applications aren't entirely clear yet, but it does seem like it could have some interesting and useful consequences. That said, I don't think this belongs in 'array'. Historically, array has been used for all sorts of array creation activities, which is why it always seems to have a wide, somewhat incoherent interface. However, most uses of array() boil down to one thing: creating a *new* array from a python object. My preference would be to focus on that functionality for array() and spin of it's other historical uses and new uses, like this custom strided array stuff, into separate factory functions. For example (and just for example, I make no great claims for either this name or interface): a = array_from_data(a_buffer_object, dtype, dims, strides) One thing that you do make clear is that contiguous and fortran should really two values of the same flag. If you combine this with one other simplification: array() always copies, we end up with a nice thin interface: # Create a new array in 'order' order. Defaults to "C" order. array(object, dtype=None, order="C"|"FORTRAN") and # Returns an array. If object is an array and order is satisfied, return object otherwise a new array. # If order is set the returned array will be contiguous and have that ordering asarray(object, dtype=None, order=None|"C"|"FORTRAN") # Just the same, but allow subtypes. asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") You could build asarray, asanyarray, etc on top of the proposed array without problems by using type(object)==ndarray and isinstance(type, ndarray) respectively. Stuff like convenience functions for minnd would also be easy to build on top of there. This looks great to me (pre-coffee). Embrace simplicity: you have nothing to lose but your clutter;) Regards, -tim From cjw at sympatico.ca Fri Mar 24 06:50:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 06:50:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4423FB59.1060403@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> Message-ID: <44240708.5090404@sympatico.ca> Tim Hochberg wrote: > Colin J. Williams wrote: > >> Tim Hochberg wrote: >> >>> >>> >>> I was just looking at the interface for array and asarray to see >>> what other stuff should go in the interface of the hypothetical >>> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >>> which I didn't, but which makes sense. However, there's also >>> 'ndmin'. First off, it's not described in docstring for asarray, but >>> I was able to find it in the docstring for array without a problem. >>> Second, is it really necessary? It seems to be useful in an awfully >>> narrow set of circumstances, particularly since when you are padding >>> axes not everyone wants to pad to the left. >>> >>> It would seem to be more useful to ditch the ndmin and have some >>> sort of paddims function that was more full featured (padding to >>> either the left or the right at a minimum). I'm not entirely sure >>> what the best interface to such a beast would look like, but a >>> simple tactic would be to just provide leftpaddims and rightpaddims. >>> >>> If it's not allready clear by now (;), I prefer several narrow >>> interfaces to one broad one. >>> >>> -tim >>> >> >> >> >> What does ascontiguous do that copy doesn't? > > > What it doesn't do is always copy the argument. Just like asarray, it > returns it unchanged if it's contiguous. Fair enough. I guess that, for some array a, "b= ascontiguous(a)" saves a few keystrokes as compared with "b= a if a.flags.contiguous else a.copy()". The intent of the latter is clearer and probably burns fewer cycles. numarray has an iscontiguous method, which is a bit simpler than concerning the Python user with flags. > >> In numarray, transpose created a discontiguous array. copy() made >> the array contiguous. Is there sufficient benefit to justify another >> method? > > > I was proposing a function, not a method. The array object has plenty > of methods already. Since the function would only operate on ArrayType instances, would it not be better as a method? Colin W. > > -tim > >> >> Colin W. >> >> > > From cjw at sympatico.ca Fri Mar 24 07:06:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 07:06:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442402E5.6090901@cox.net> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: <44240AB6.202@sympatico.ca> Tim Hochberg wrote: > Sasha wrote: > >> On 3/23/06, Travis Oliphant wrote: >> >> >>> At any rate, if the fortran flag is there, we need to specify the >>> contiguous case as well. So, either propose a better interface (we >>> could change it still --- the fortran flag doesn't have that much >>> history) to handle the situation or accept what I do ;-) >>> >> >> Contiguity is separable from fortran: [Dbg]>>> b= _n.array([[1, 2, 3], [4, 5, 6]]) [Dbg]>>> b.flags.contiguous True [Dbg]>>> c= b.transpose() [Dbg]>>> c array([[1, 4], [2, 5], [3, 6]]) [Dbg]>>> c.flags.contiguous False [Dbg]>>> >> Let me try. I propose to eliminate the fortran flag in favor of a more >> general "strides" argument. This argument can be either a sequence of >> integers that becomes the strides, or a callable object that takes >> shape and dtype arguments and return a sequence that becomes the >> strides. For fortran and c order functions that generate appropriate >> stride sequences should be predefined to enable array(..., >> strides=fortran, ...) and array(..., strides=contiguous). >> > > I like the idea of being able to create an array with custom strides. > The applications aren't entirely clear yet, but it does seem like it > could have some interesting and useful consequences. That said, I > don't think this belongs in 'array'. Historically, array has been used > for all sorts of array creation activities, which is why it always > seems to have a wide, somewhat incoherent interface. However, most > uses of array() boil down to one thing: creating a *new* array from a > python object. My preference would be to focus on that functionality > for array() and spin of it's other historical uses and new uses, like > this custom strided array stuff, into separate factory functions. For > example (and just for example, I make no great claims for either this > name or interface): > a = array_from_data(a_buffer_object, dtype, dims, strides) [***] > > One thing that you do make clear is that contiguous and fortran should > really two values of the same flag. Please see the transpose example above. > If you combine this with one other simplification: array() always > copies, we end up with a nice thin interface: > # Create a new array in 'order' order. Defaults to "C" order. > array(object, dtype=None, order="C"|"FORTRAN") I feel that [***] above is much cleaner than this. I suggest that string constants be deprecated. > and > # Returns an array. If object is an array and order is satisfied, > return object otherwise a new array. > # If order is set the returned array will be contiguous and have > that ordering > asarray(object, dtype=None, order=None|"C"|"FORTRAN") > # Just the same, but allow subtypes. > asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") > > You could build asarray, asanyarray, etc on top of the proposed array > without problems by using type(object)==ndarray and isinstance(type, > ndarray) respectively. Stuff like convenience functions for minnd > would also be easy to build on top of there. This looks great to me > (pre-coffee). > > Embrace simplicity: you have nothing to lose but your clutter;) > > Regards, > > -tim > If [***] above were adopted, it would still be helpful to adopt numarray's iscontiguous method, or better, use a property. colin W. From boyle5 at llnl.gov Fri Mar 24 07:55:05 2006 From: boyle5 at llnl.gov (James Boyle) Date: Fri Mar 24 07:55:05 2006 Subject: [Numpy-discussion] Installing numpy 0.9.6 problems on OS X Message-ID: <1efeb18f9e8baf66d0f2e02337a2a709@llnl.gov> I am running OX 10.3.9. I installed python 2.4.2 using Universal MacPython 2.4.2.dmg. When I try to install numpy 0.9.6 it fails with the message. . . gcc: _configtest.c gcc: cannot specify -o with -c or -S and multiple compilations gcc: cannot specify -o with -c or -S and multiple compilations failure. . . I am running gcc 3.3. I note that the python 2.4.2 was built using gcc 4.0.1. Could this difference in gcc versions be the problem? At present I am at a loss as to what to try next. --Jim -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 619 bytes Desc: not available URL: From tim.hochberg at cox.net Fri Mar 24 08:22:17 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 08:22:17 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44240708.5090404@sympatico.ca> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> <44240708.5090404@sympatico.ca> Message-ID: <44241C50.40206@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Colin J. Williams wrote: >> >>> Tim Hochberg wrote: >>> >>>> >>>> >>>> I was just looking at the interface for array and asarray to see >>>> what other stuff should go in the interface of the hypothetical >>>> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >>>> which I didn't, but which makes sense. However, there's also >>>> 'ndmin'. First off, it's not described in docstring for asarray, >>>> but I was able to find it in the docstring for array without a >>>> problem. Second, is it really necessary? It seems to be useful in >>>> an awfully narrow set of circumstances, particularly since when you >>>> are padding axes not everyone wants to pad to the left. >>>> >>>> It would seem to be more useful to ditch the ndmin and have some >>>> sort of paddims function that was more full featured (padding to >>>> either the left or the right at a minimum). I'm not entirely sure >>>> what the best interface to such a beast would look like, but a >>>> simple tactic would be to just provide leftpaddims and rightpaddims. >>>> >>>> If it's not allready clear by now (;), I prefer several narrow >>>> interfaces to one broad one. >>>> >>>> -tim >>>> >>> >>> >>> >>> >>> What does ascontiguous do that copy doesn't? >> >> >> >> What it doesn't do is always copy the argument. Just like asarray, it >> returns it unchanged if it's contiguous. > > > Fair enough. I guess that, for some array a, "b= ascontiguous(a)" > saves a few keystrokes as compared > with "b= a if a.flags.contiguous else a.copy()". The intent of the > latter is clearer and probably burns fewer cycles. First, the second expression is not equivalent to ascontiguous. Second, I disagree strongly that it's clearer. Third, if cycles are actually a concern, ascontiguous if implemented in C would certainly be faster. Fourth, this can't be written in any released version of Python. Let me go into a little more detail on the first and second points. The common use case for asarray and ascontiguous is to adapt objects that you don't know much about into arrays. A typical pattern is: def func(a, b, c): a = asarray(a) b = ascontiguous(b) c = asarray(c) # Code that requires arrays for a,c and a contigous array for b. For this reason, as contiguous needs to take any object and attempt to turn it into a contiguous array if it isn't one already. So "b = ascontiguous(a)" is really equivalent to "b = asarray(a); b = b if b.flags.contiguous else b.copy()". Even if we could assume all the inputs were arrays, or we separately use as array first, the pattern you propose is error prone since it violates DRY. It would be all to easy to write "b= a if c.flags.contiguous else a.copy()" or some such. In order to verify that the expressions are correct, or in order to tell what they actually do you have to actually parse through the expression. "ascontiguous", on the other hand, is pretty much bomb proof once you know what it does. > > numarray has an iscontiguous method, which is a bit simpler than > concerning the Python user with flags. > >> >>> In numarray, transpose created a discontiguous array. copy() made >>> the array contiguous. Is there sufficient benefit to justify >>> another method? >> >> >> >> I was proposing a function, not a method. The array object has plenty >> of methods already. > > > Since the function would only operate on ArrayType instances, would it > not be better as a method? Your premise is wrong. It would operate on any object, so it has to be a function. It could also be a method, but that would be superfluous. Regards, -tim From alexander.belopolsky at gmail.com Fri Mar 24 08:28:07 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Mar 24 08:28:07 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <442402E5.6090901@cox.net> References: <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: On 3/24/06, Tim Hochberg wrote: >.... For example (and just for > example, I make no great claims for either this name or interface): > a = array_from_data(a_buffer_object, dtype, dims, strides) > This looks very similar to the current ndarray "new" constructor: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) | | There are two modes of creating an array using __new__: | 1) If buffer is None, then only shape, dtype, and fortran | are used | 2) If buffer is an object exporting the buffer interface, then | all keywords are interpreted. | The dtype parameter can be any object that can be interpreted | as a numpy.dtype object. (see pydoc numpy.ndarray) I would not mind to leave array() unchanged and move discussion to streamlining ndarray.__new__ . For example, some time ago I suggested that strides should be interpreted even if buffer=None. From tim.hochberg at cox.net Fri Mar 24 08:39:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 08:39:03 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44240AB6.202@sympatico.ca> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> Message-ID: <44242065.9090307@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Sasha wrote: >> >>> On 3/23/06, Travis Oliphant wrote: >>> >>> >>>> At any rate, if the fortran flag is there, we need to specify the >>>> contiguous case as well. So, either propose a better interface (we >>>> could change it still --- the fortran flag doesn't have that much >>>> history) to handle the situation or accept what I do ;-) >>>> >>> >>> >>> > Contiguity is separable from fortran: > [Dbg]>>> b= _n.array([[1, 2, 3], [4, 5, 6]]) > [Dbg]>>> b.flags.contiguous > True > [Dbg]>>> c= b.transpose() > [Dbg]>>> c > array([[1, 4], > [2, 5], > [3, 6]]) > [Dbg]>>> c.flags.contiguous > False > [Dbg]>>> This is true, but irrelevant. To the best of my knowledge, the only reason to force an array to be in a specific order is to pass it to a C function that expects either FORTRAN- or C-ordered arrays. And, in that case, the array also needs to be contiguous. So, for the purpose of creating arrays (and for the purposes of ascontiguous), the only cases that matter are arrays that are both contiguous and the specified order. Thus, specifying continuity and order separately to the constructor needlessly complicates the interface. Or since I'm feeling jargon happy today, YAGNI. > >>> Let me try. I propose to eliminate the fortran flag in favor of a more >>> general "strides" argument. This argument can be either a sequence of >>> integers that becomes the strides, or a callable object that takes >>> shape and dtype arguments and return a sequence that becomes the >>> strides. For fortran and c order functions that generate appropriate >>> stride sequences should be predefined to enable array(..., >>> strides=fortran, ...) and array(..., strides=contiguous). >>> >> >> I like the idea of being able to create an array with custom strides. >> The applications aren't entirely clear yet, but it does seem like it >> could have some interesting and useful consequences. That said, I >> don't think this belongs in 'array'. Historically, array has been >> used for all sorts of array creation activities, which is why it >> always seems to have a wide, somewhat incoherent interface. However, >> most uses of array() boil down to one thing: creating a *new* array >> from a python object. My preference would be to focus on that >> functionality for array() and spin of it's other historical uses and >> new uses, like this custom strided array stuff, into separate factory >> functions. For example (and just for example, I make no great claims >> for either this name or interface): >> a = array_from_data(a_buffer_object, dtype, dims, strides) [***] >> >> One thing that you do make clear is that contiguous and fortran >> should really two values of the same flag. > > > Please see the transpose example above. > >> If you combine this with one other simplification: array() always >> copies, we end up with a nice thin interface: >> # Create a new array in 'order' order. Defaults to "C" order. >> array(object, dtype=None, order="C"|"FORTRAN") > > > I feel that [***] above is much cleaner than this. I suggest that > string constants be deprecated. I'm no huge fan of string constants myself, but I think you need to think this through more. First off, the interface I tossed off above doesn't cover the same ground as array, since it works off an already created buffer object. That means you'd have to go through all sorts of contortions and do at least one copy to get data into Fortran order. You could allow arbitrary, 1D, python sequences instead, but that doesn't help the common case of converting a 2D python object into a 2D array. You could allow N-D python objects, but then you have two ways of specifying the dims of the object and things become a big krufty mess. Compared to that string constants are great. >> and >> # Returns an array. If object is an array and order is satisfied, >> return object otherwise a new array. >> # If order is set the returned array will be contiguous and have >> that ordering >> asarray(object, dtype=None, order=None|"C"|"FORTRAN") >> # Just the same, but allow subtypes. >> asanyarray(object, dtype=None, order=None|"C"|"FORTRAN") >> >> You could build asarray, asanyarray, etc on top of the proposed array >> without problems by using type(object)==ndarray and isinstance(type, >> ndarray) respectively. Stuff like convenience functions for minnd >> would also be easy to build on top of there. This looks great to me >> (pre-coffee). >> >> Embrace simplicity: you have nothing to lose but your clutter;) >> >> Regards, >> >> -tim >> > If [***] above were adopted, it would still be helpful to adopt > numarray's iscontiguous method, or better, use a property. -0. In my experience, 99% of my use cases would be covered for ascontiguous and for the remaining 1% I'm happy to use a.flags.contiguous. Regards, -tim From ndarray at mac.com Fri Mar 24 08:47:10 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 24 08:47:10 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: On 3/24/06, Tim Hochberg wrote: >.... For example (and just for > example, I make no great claims for either this name or interface): > a = array_from_data(a_buffer_object, dtype, dims, strides) > This looks very similar to the current ndarray "new" constructor: | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, | offset=0, strides=None, fortran=False) | | There are two modes of creating an array using __new__: | 1) If buffer is None, then only shape, dtype, and fortran | are used | 2) If buffer is an object exporting the buffer interface, then | all keywords are interpreted. | The dtype parameter can be any object that can be interpreted | as a numpy.dtype object. (see pydoc numpy.ndarray) I would not mind to leave array() unchanged and move discussion to streamlining ndarray.__new__ . For example, some time ago I suggested that strides should be interpreted even if buffer=None. From jimpy at oublic.org Fri Mar 24 08:48:03 2006 From: jimpy at oublic.org (Jim Carroll) Date: Fri Mar 24 08:48:03 2006 Subject: [Numpy-discussion] Re: Newbie: Matrix form of segment-sphere intersection? References: Message-ID: > On Thu, 23 Mar 2006, (UTC) Jim Carroll apparently wrote: > > Where can I find applicable tutorials or recipies? > > Don't know the lit, but noticed this: > http://www.olympus.net/personal/7seas/contain.html > Interesting... but it doesn't show the how the points are formatted in the matricies, or how a more complicated algebraic function would be done... The sphere intersection 3 step algorithm defines some values up front, then finds the discriminant, then uses the discriminant to caluclate the intersection. I'm wondering if I'd be crazy if I coded the algorithm in a C ufunc: for a table where each row has x1, y1, z1, x2, y2, z2, radius if my ufunc would return a table that's got just three columns: x3, y3, z3 representing the sphere intersection. (asuming the input is translated so the sphere is at the origin.) Does anyone do this kind of things with ufuncs? Am I on the right track, but there's an easier way to do it? Or, is there a geometric math library that does this kind of thing? (I found one a year ago, but can't remember if it's this sophisticated.) Thanks, -Jim From tim.hochberg at cox.net Fri Mar 24 08:49:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 08:49:01 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: <442422C2.2010909@cox.net> Alexander Belopolsky wrote: >On 3/24/06, Tim Hochberg wrote: > > >>.... For >> >> >example (and just for > > >>example, I make no great claims for either this name or interface): >> a = array_from_data(a_buffer_object, dtype, dims, strides) >> >> >> > >This looks very similar to the current ndarray "new" constructor: > > It sure does. > | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, > | offset=0, strides=None, fortran=False) > | > | There are two modes of creating an array using __new__: > | 1) If buffer is None, then only shape, dtype, and fortran > | are used > | 2) If buffer is an object exporting the buffer interface, then > | all keywords are interpreted. > | The dtype parameter can be any object that can be interpreted > | as a numpy.dtype object. > >(see pydoc numpy.ndarray) > >I would not mind to leave array() unchanged and move discussion to >streamlining ndarray.__new__ . For example, some time ago I suggested >that strides should be interpreted even if buffer=None. > > That does look like a good place to hang any arbitrary strides creating stuff. I'm of the opinion that arguments should *never* be ignored, so I'm all for interpreting strides even when buffer is None. I'd also contend that offset should either be respected (by overallocating) or since that's probably useless, raising a ValueError when it's nonzero. Regards, Tim From fred at ucar.edu Fri Mar 24 10:25:05 2006 From: fred at ucar.edu (Fred Clare) Date: Fri Mar 24 10:25:05 2006 Subject: [Numpy-discussion] PyArray_ISSTRING Message-ID: <2a503cbe0e0f909cb2502f799d5229c4@ucar.edu> I am running NumPy 0.9.6 on Mac OS X. When I try to use the API function PyArray_ISSTRING I get: `PyArray_UCHAR' undeclared In fact I cannot see where PyArray_UCHAR is ever declared. Fred Clare From oliphant at ee.byu.edu Fri Mar 24 11:12:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 24 11:12:00 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: References: <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> Message-ID: <44244435.3050904@ee.byu.edu> Alexander Belopolsky wrote: >On 3/24/06, Tim Hochberg wrote: > > >>.... For >> >> >example (and just for > > >>example, I make no great claims for either this name or interface): >> a = array_from_data(a_buffer_object, dtype, dims, strides) >> >> >> > >This looks very similar to the current ndarray "new" constructor: > > | ndarray.__new__(subtype, shape=, dtype=int_, buffer=None, > | offset=0, strides=None, fortran=False) > | > | There are two modes of creating an array using __new__: > | 1) If buffer is None, then only shape, dtype, and fortran > | are used > | 2) If buffer is an object exporting the buffer interface, then > | all keywords are interpreted. > | The dtype parameter can be any object that can be interpreted > | as a numpy.dtype object. > >(see pydoc numpy.ndarray) > >I would not mind to leave array() unchanged and move discussion to >streamlining ndarray.__new__ . For example, some time ago I suggested >that strides should be interpreted even if buffer=None. > > > Here I'm much more happy about removing the fortran flag and just using the strides parameter as Sasha suggests in this routine, so please streamline as much as possible. -Travis From cjw at sympatico.ca Fri Mar 24 11:15:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 24 11:15:02 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44241C50.40206@cox.net> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> <44240708.5090404@sympatico.ca> <44241C50.40206@cox.net> Message-ID: <4424451B.1000008@sympatico.ca> Tim Hochberg wrote: > Colin J. Williams wrote: > >> Tim Hochberg wrote: >> >>> Colin J. Williams wrote: >>> >>>> Tim Hochberg wrote: >>>> >>>>> >>>>> >>>>> I was just looking at the interface for array and asarray to see >>>>> what other stuff should go in the interface of the hypothetical >>>>> ascontiguous. There's 'dtype', which I knew about, and 'fortran', >>>>> which I didn't, but which makes sense. However, there's also >>>>> 'ndmin'. First off, it's not described in docstring for asarray, >>>>> but I was able to find it in the docstring for array without a >>>>> problem. Second, is it really necessary? It seems to be useful in >>>>> an awfully narrow set of circumstances, particularly since when >>>>> you are padding axes not everyone wants to pad to the left. >>>>> >>>>> It would seem to be more useful to ditch the ndmin and have some >>>>> sort of paddims function that was more full featured (padding to >>>>> either the left or the right at a minimum). I'm not entirely sure >>>>> what the best interface to such a beast would look like, but a >>>>> simple tactic would be to just provide leftpaddims and rightpaddims. >>>>> >>>>> If it's not allready clear by now (;), I prefer several narrow >>>>> interfaces to one broad one. >>>>> >>>>> -tim >>>>> >>>> >>>> >>>> >>>> >>>> >>>> What does ascontiguous do that copy doesn't? >>> >>> >>> >>> >>> What it doesn't do is always copy the argument. Just like asarray, >>> it returns it unchanged if it's contiguous. >> >> >> >> Fair enough. I guess that, for some array a, "b= ascontiguous(a)" >> saves a few keystrokes as compared >> with "b= a if a.flags.contiguous else a.copy()". The intent of the >> latter is clearer and probably burns fewer cycles. > > > First, the second expression is not equivalent to ascontiguous. > Second, I disagree strongly that it's clearer. Third, if cycles are > actually a concern, ascontiguous if implemented in C would certainly > be faster. Fourth, this can't be written in any released version of > Python. > Thanks, this is illuminating. Fourth is true, but the first release of Python 2.5 is expected in eight dats. We see, in the Help: Help on built-in function array in module numpy.core.multiarray: array(...) array(object, dtype=None, copy=1, fortran=0, subok=0,ndmin=0) will return a new array formed from the given object type given. Object can be anything with an __array__ method, or any object exposing the array interface, or any (nested) sequence. If no type is given, then the type will be determined as the minimum type required to hold the objects in the sequence. If copy is zero and sequence is already an array with the right type, a reference will be returned. If the sequence is an array, type can be used only to upcast the array. For downcasting use .astype(t) method. If subok is true, then subclasses of the array may be returned. Otherwise, a base-class ndarray is returned The ndmin argument specifies how many dimensions the returned array should have as a minimum. 1's will be pre-pended to the shape as needed to meet this requirement. [Dbg]>>> help(_n.asarray) Help on function asarray in module numpy.core.numeric: asarray(a, dtype=None, fortran=False, ndmin=0) returns a as an array. Unlike array(), no copy is performed if a is already an array. Subclasses are converted to base class ndarray. It is not clear, from this, just what is acceptable, for a, by asarray. From your response, it would appear that "a" in asarray is the same as "object" in array. > Let me go into a little more detail on the first and second points. > The common use case for asarray and ascontiguous is to adapt objects > that you don't know much about into arrays. A typical pattern is: > > def func(a, b, c): > a = asarray(a) > b = ascontiguous(b) > c = asarray(c) > # Code that requires arrays for a,c and a contigous array for b. > > For this reason, as contiguous needs to take any object and attempt to > turn it into a contiguous array if it isn't one already. So "b = > ascontiguous(a)" is really equivalent to "b = asarray(a); b = b if > b.flags.contiguous else b.copy()". Even if we could assume all the > inputs were arrays, or we separately use as array first, the pattern > you propose is error prone since it violates DRY. It would be all to > easy to write "b= a if c.flags.contiguous else a.copy()" or some such. > In order to verify that the expressions are correct, or in order to > tell what they actually do you have to actually parse through the > expression. "ascontiguous", on the other hand, is pretty much bomb > proof once you know what it does. > > I would be grateful if you could clarify DRY. > >> >> numarray has an iscontiguous method, which is a bit simpler than >> concerning the Python user with flags. >> >>> >>>> In numarray, transpose created a discontiguous array. copy() made >>>> the array contiguous. Is there sufficient benefit to justify >>>> another method? >>> >>> >>> >>> >>> I was proposing a function, not a method. The array object has >>> plenty of methods already. >> >> >> >> Since the function would only operate on ArrayType instances, would >> it not be better as a method? > > > Your premise is wrong. It would operate on any object, so it has to be > a function. It could also be a method, but that would be superfluous. > > Regards, > > -tim > It would appear that you are right, ascontiguous will be permitted to operate on any object which is acceptable to array, although I can't think of a case where any object other than an array, or a sub-class could be discontiguous. Thanks for clarifying things. Colin W. From boyle5 at llnl.gov Fri Mar 24 11:17:04 2006 From: boyle5 at llnl.gov (James Boyle) Date: Fri Mar 24 11:17:04 2006 Subject: [Numpy-discussion] OS X - Universal MacPython - numpy install failure Message-ID: I am running OX 10.3.9. I installed python 2.4.2 using Universal MacPython 2.4.2.dmg. I am on a PowerPC G4. The Universal build is designed to install on the Mac PPC and Intel machines. When I run the numpy 0.9.6 install it apparently fails on the configure. The whole trace back is at the end of this posting. One thing that I noted was the line: gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' It appears that the configure routines think that I have two architectures - ppc and i386. At my low level of understanding, all I can do is speculate that the Universal install generates some confusion as to the actual cpu. Somewhere numpy is keying on information that is ambiguous in the universal install. I have installed Numeric, matplotlib and other packages without a problem. --Jim The trace back from >python setup.py build: Running from numpy source directory. Warning: not existing path in numpy/distutils: site.cfg F2PY Version 2_2236 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] running build running config_fc running build_src building py_modules sources creating build creating build/src creating build/src/numpy creating build/src/numpy/distutils building extension "numpy.core.multiarray" sources creating build/src/numpy/core Generating build/src/numpy/core/config.h customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' compile options: '-I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' gcc: _configtest.c gcc: cannot specify -o with -c or -S and multiple compilations gcc: cannot specify -o with -c or -S and multiple compilations failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 69, in setup_package setup( **config.todict() ) File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/core.py", line 85, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.4//lib/python2.4/ distutils/dist.py", line 966, in run_command cmd_obj.run() File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 84, in run self.build_sources() File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 99, in build_sources self.build_extension_sources(ext) File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 209, in build_extension_sources sources = self.generate_sources(sources, ext) File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ build_src.py", line 267, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 37, in generate_config_h raise "ERROR: Failed to test configuration" ERROR: Failed to test configuration -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 4515 bytes Desc: not available URL: From tim.hochberg at cox.net Fri Mar 24 11:36:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 11:36:06 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <4424451B.1000008@sympatico.ca> References: <4421DF9B.6000102@hawaii.edu> <4421E4A6.4020906@cox.net> <4421F798.2070300@ee.byu.edu> <44220001.4050606@cox.net> <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <4423E66C.2040905@sympatico.ca> <4423FB59.1060403@cox.net> <44240708.5090404@sympatico.ca> <44241C50.40206@cox.net> <4424451B.1000008@sympatico.ca> Message-ID: <442449FA.3070503@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Colin J. Williams wrote: >> >>> Tim Hochberg wrote: >>> >>>> Colin J. Williams wrote: >>>> >>>>> Tim Hochberg wrote: >>>>> >>>>>> >>>>>> >>>>>> I was just looking at the interface for array and asarray to see >>>>>> what other stuff should go in the interface of the hypothetical >>>>>> ascontiguous. There's 'dtype', which I knew about, and >>>>>> 'fortran', which I didn't, but which makes sense. However, >>>>>> there's also 'ndmin'. First off, it's not described in docstring >>>>>> for asarray, but I was able to find it in the docstring for array >>>>>> without a problem. Second, is it really necessary? It seems to be >>>>>> useful in an awfully narrow set of circumstances, particularly >>>>>> since when you are padding axes not everyone wants to pad to the >>>>>> left. >>>>>> >>>>>> It would seem to be more useful to ditch the ndmin and have some >>>>>> sort of paddims function that was more full featured (padding to >>>>>> either the left or the right at a minimum). I'm not entirely sure >>>>>> what the best interface to such a beast would look like, but a >>>>>> simple tactic would be to just provide leftpaddims and rightpaddims. >>>>>> >>>>>> If it's not allready clear by now (;), I prefer several narrow >>>>>> interfaces to one broad one. >>>>>> >>>>>> -tim >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> What does ascontiguous do that copy doesn't? >>>> >>>> >>>> >>>> >>>> >>>> What it doesn't do is always copy the argument. Just like asarray, >>>> it returns it unchanged if it's contiguous. >>> >>> >>> >>> >>> Fair enough. I guess that, for some array a, "b= ascontiguous(a)" >>> saves a few keystrokes as compared >>> with "b= a if a.flags.contiguous else a.copy()". The intent of the >>> latter is clearer and probably burns fewer cycles. >> >> >> >> First, the second expression is not equivalent to ascontiguous. >> Second, I disagree strongly that it's clearer. Third, if cycles are >> actually a concern, ascontiguous if implemented in C would certainly >> be faster. Fourth, this can't be written in any released version of >> Python. >> > Thanks, this is illuminating. Fourth is true, but the first release > of Python 2.5 is expected in eight dats. > > We see, in the Help: > Help on built-in function array in module numpy.core.multiarray: > > array(...) > array(object, dtype=None, copy=1, fortran=0, subok=0,ndmin=0) > will return a new array formed from the given object type given. > Object can be anything with an __array__ method, or any object > exposing the array interface, or any (nested) sequence. > If no type is given, then the type will be determined as the > minimum type required to hold the objects in the sequence. > If copy is zero and sequence is already an array with the right > type, a reference will be returned. If the sequence is an array, > type can be used only to upcast the array. For downcasting > use .astype(t) method. If subok is true, then subclasses of the > array may be returned. Otherwise, a base-class ndarray is returned > The ndmin argument specifies how many dimensions the returned > array should have as a minimum. 1's will be pre-pended to the > shape as needed to meet this requirement. > [Dbg]>>> help(_n.asarray) > Help on function asarray in module numpy.core.numeric: > > asarray(a, dtype=None, fortran=False, ndmin=0) > returns a as an array. Unlike array(), > no copy is performed if a is already an array. Subclasses are > converted > to base class ndarray. > > It is not clear, from this, just what is acceptable, for a, by asarray. That could probably be clearer, it's true. > > From your response, it would appear that "a" in asarray is the same as > "object" in array. Indeed. > >> Let me go into a little more detail on the first and second points. >> The common use case for asarray and ascontiguous is to adapt objects >> that you don't know much about into arrays. A typical pattern is: >> >> def func(a, b, c): >> a = asarray(a) >> b = ascontiguous(b) >> c = asarray(c) >> # Code that requires arrays for a,c and a contigous array for b. >> >> For this reason, as contiguous needs to take any object and attempt >> to turn it into a contiguous array if it isn't one already. So "b = >> ascontiguous(a)" is really equivalent to "b = asarray(a); b = b if >> b.flags.contiguous else b.copy()". Even if we could assume all the >> inputs were arrays, or we separately use as array first, the pattern >> you propose is error prone since it violates DRY. It would be all to >> easy to write "b= a if c.flags.contiguous else a.copy()" or some >> such. In order to verify that the expressions are correct, or in >> order to tell what they actually do you have to actually parse >> through the expression. "ascontiguous", on the other hand, is pretty >> much bomb proof once you know what it does. >> >> > I would be grateful if you could clarify DRY. Don't Repeat Yourself. Basically, it's exhortation not to write out the same expression over and over again. It was on of the main motivators for getting += into Python, since: a.foo.bar[squiggly].retrieveArrayFor("Colin") = a.foo.bar[squiggly].retrieveArrayFor("Colin") + rubber_ducky.frobinate["NOW!"] Is much easier to mess up than: a.foo.bar[squiggly].retrieveArrayFor("Colin") += rubber_ducky.frobinate["NOW!"] In the former, it's easy to miss small differences between the first and second quantities. These creep in easily when cutting and pasting. Anyway, in the case in question: b= a if a.flags.contiguous else a.copy() 'a' is repeated twice. If this is getting cut and pasted all over the place it'll frequently end up that one of the a's doesn't get changed properly. For: b = ascontiguous(a) The chances are at least half as small that one gets it wrong based on numbers of chances alone. Probably much, much better than that in practice since a isn't buried in an expression, but is nicely set off by parentheses. See also: http://en.wikipedia.org/wiki/Don%27t_repeat_yourself > >> >>> >>> numarray has an iscontiguous method, which is a bit simpler than >>> concerning the Python user with flags. >>> >>>> >>>>> In numarray, transpose created a discontiguous array. copy() made >>>>> the array contiguous. Is there sufficient benefit to justify >>>>> another method? >>>> >>>> >>>> >>>> >>>> >>>> I was proposing a function, not a method. The array object has >>>> plenty of methods already. >>> >>> >>> >>> >>> Since the function would only operate on ArrayType instances, would >>> it not be better as a method? >> >> >> >> Your premise is wrong. It would operate on any object, so it has to >> be a function. It could also be a method, but that would be superfluous. >> >> Regards, >> >> -tim >> > It would appear that you are right, ascontiguous will be permitted to > operate on any object which is acceptable to array, although I can't > think of a case where any object other than an array, or a sub-class > could be discontiguous. No, that is correct. The point is I have an object I obtained from somewhere. I need a contiguous array. ascontiguous applied to that object will either give me a contiguous array or raise an exception. Thus it can be used to adapt any thing-that-can-be-turned-into-an-array into a contiguous array. > > Thanks for clarifying things. Not a problem. Regards, Tim From oliphant at ee.byu.edu Fri Mar 24 11:50:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 24 11:50:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44242065.9090307@cox.net> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> <44242065.9090307@cox.net> Message-ID: <44244D42.5020201@ee.byu.edu> Tim Hochberg wrote: >> >> >> Please see the transpose example above. >> >>> If you combine this with one other simplification: array() always >>> copies, we end up with a nice thin interface: >>> # Create a new array in 'order' order. Defaults to "C" order. >>> array(object, dtype=None, order="C"|"FORTRAN") >> I like the order flag. It's a relatively easy switch and one we should make right now. I don't mind string constants either in this case. Removing the copy flag will break a lot of code because it's been around for a long time. This is also not an "easy thing" to add to convertcode.py though I suppose 90% of the cases could be found. We would also have to re-write asarray to be an additional C-function to make it not copy but make array copy. So, for now I'm not as enthused about that idea. -Travis From ryanlists at gmail.com Fri Mar 24 12:54:01 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Fri Mar 24 12:54:01 2006 Subject: [Numpy-discussion] Re: Newbie: Matrix form of segment-sphere intersection? In-Reply-To: References: Message-ID: I would first do this in python without ufuncs and see if it is slow before going through the extra effort to put it in a ufunc. I think if you play with array broadcasting, you will be quite happy with the results for doing calculations on lots of points. Numpy operates on an element-by-element basis by default. So, for example x=arange(0,10,0.1) y=x**2 will create a vector x and then a vector of y with the elements of x squared. This sort of math on arrays I think does what you want unless there is some math function in your algorithm that doesn't take vector inputs. In that case you might need to write your own function for that. Does this help or am I missing something important? Ryan On 3/24/06, Jim Carroll wrote: > > > On Thu, 23 Mar 2006, (UTC) Jim Carroll apparently wrote: > > > Where can I find applicable tutorials or recipies? > > > > Don't know the lit, but noticed this: > > http://www.olympus.net/personal/7seas/contain.html > > > > Interesting... but it doesn't show the how the points are formatted in the > matricies, or how a more complicated algebraic function would be done... > > The sphere intersection 3 step algorithm defines some values up front, then > finds the discriminant, then uses the discriminant to caluclate the intersection. > > I'm wondering if I'd be crazy if I coded the algorithm in a C ufunc: > for a table where each row has x1, y1, z1, x2, y2, z2, radius if my ufunc > would return a table that's got just three columns: x3, y3, z3 representing the > sphere intersection. (asuming the input is translated so the sphere is at the > origin.) > > Does anyone do this kind of things with ufuncs? Am I on the right track, but > there's an easier way to do it? Or, is there a geometric math library that does > this kind of thing? (I found one a year ago, but can't remember if it's this > sophisticated.) > > Thanks, > -Jim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cookedm at physics.mcmaster.ca Fri Mar 24 13:04:07 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Mar 24 13:04:07 2006 Subject: [Numpy-discussion] Current SVN segfaults under VC7 In-Reply-To: <44223E4A.9090208@colorado.edu> (Fernando Perez's message of "Wed, 22 Mar 2006 23:20:58 -0700") References: <4421EC3B.4020800@cox.net> <4421F909.6050401@ee.byu.edu> <44220366.4000004@cox.net> <44223E4A.9090208@colorado.edu> Message-ID: Fernando Perez writes: > Tim Hochberg wrote: > >> This did fix things...eventually. I removed the build directory and >> rebuilt. Numpy still bombed the interpreter, although in a new and >> less exciting way. I then did a clean checkout from svn and rebuilt. >> Same thing. Then I manally removed numpy from site-packages and >> reinstalled at which points things worked and all tests passed. >> So in addition to needing to remove the build directory, it appears >> that something was hanging out in the destination directory that did >> not get overwritten and caused problems. >> Perhaps this description will help the next person who runs into >> this, but probably not. > > Well, the script that I posted does precisely all of this in a > mindless fashion, which is why I find it useful. You may want to give > it a try (it's trivial, but it just ensures you don't forget these > steps and end up wasting any time chasing the problem). Me, I use eggs. One advantage is, since I use the bleeding edge for for all of my code, if there's something wrong with numpy (or scipy) and I don't have the time to fix it, I can switch to an older version. I've got a make-egg script that looks like #!/bin/sh set -e PYTHON=python2.4 ${PYTHON} setup.py build ${PYTHON} -c 'import setuptools; execfile("setup.py")' bdist_egg then easy_install2.4 dist/ -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From tim.hochberg at cox.net Fri Mar 24 13:40:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 13:40:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44244D42.5020201@ee.byu.edu> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> <44242065.9090307@cox.net> <44244D42.5020201@ee.byu.edu> Message-ID: <44246715.3030401@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> >>> Please see the transpose example above. >>> >>>> If you combine this with one other simplification: array() always >>>> copies, we end up with a nice thin interface: >>>> # Create a new array in 'order' order. Defaults to "C" order. >>>> array(object, dtype=None, order="C"|"FORTRAN") >>> >>> > I like the order flag. It's a relatively easy switch and one we > should make right now. I don't mind string constants either in this > case. Great! > Removing the copy flag will break a lot of code because it's been > around for a long time. This is also not an "easy thing" to add to > convertcode.py though I suppose 90% of the cases could be found. I kinda figured on that. But I figured I'd propose my favorite and see what came of it. > We would also have to re-write asarray to be an additional C-function > to make it not copy but make array copy. I thought so too at first, but I don't this is is so. Untested, and can could probably be cleaned up some: def asarray(obj, order=None): if type(obj) == ndarray: if order: if not obj.flags.contiguous: return array(obj, order) if order == "C" and obj.flags.fortran: return array(obj, order) if order == "FORTRAN" and not obj.flags.fortran: return array(obj, order) return obj else: if order: return array(obj, order) else: return array(obj) For asanyarray, simply replace the type test with an isinstance test. > So, for now I'm not as enthused about that idea. Yeah. Without backward compatibility constraints I'm convinced that it's the right thing to do, but I realize there is a need to balance making the transistion manageable with making things "perfect". -tim From oliphant at ee.byu.edu Fri Mar 24 14:15:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 24 14:15:04 2006 Subject: [Numpy-discussion] array, asarray as contiguous and friends In-Reply-To: <44246715.3030401@cox.net> References: <4422AFF9.5000200@colorado.edu> <4422C8A8.2050704@cox.net> <4422D46D.8080800@colorado.edu> <4422DC96.9030307@cox.net> <4422E397.6030206@colorado.edu> <4422F0A4.7010206@cox.net> <442309CE.1090005@ee.byu.edu> <44231A04.2000804@ee.byu.edu> <442402E5.6090901@cox.net> <44240AB6.202@sympatico.ca> <44242065.9090307@cox.net> <44244D42.5020201@ee.byu.edu> <44246715.3030401@cox.net> Message-ID: <44246F50.2060801@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>>> >>>> >>>> Please see the transpose example above. >>>> >>>>> If you combine this with one other simplification: array() always >>>>> copies, we end up with a nice thin interface: >>>>> # Create a new array in 'order' order. Defaults to "C" order. >>>>> array(object, dtype=None, order="C"|"FORTRAN") >>>> >>>> >>>> >> I like the order flag. It's a relatively easy switch and one we >> should make right now. I don't mind string constants either in this >> case. > > > Great! > And this is now done... So, thankfully, the fortran= keyword is gone and replaced with the more sensible order= keyword. Tests for numpy pass, but any other code that used fortran= will need changing. Sorry about that... Thanks, -Travis From faltet at carabos.com Fri Mar 24 18:10:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Mar 24 18:10:01 2006 Subject: [Numpy-discussion] A numerical agnostic extension Message-ID: <1143252541.7569.10.camel@localhost.localdomain> Hi, After seeing the inspiring work of Andrew Straw on the implementation of a __array_struct__ Pyrex interface for the array protocol, I wasn't able to relax and finally implemented another extension (heavily based on Andrew's work) that takes Numeric/numarray/NumPy objects and exposes the same interface to access to their data and metadata from both Python and Pyrex (and hence, C) spaces. All this *without* a need to compile against a specific numerical package. I've created a new recipe for this extension in: http://www.scipy.org/Cookbook/A_Pyrex_Agnostic_Class I'm pretty excited about all the possibilities that opens this relatively small class: numerical agnosticism is closer than ever :-) -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From ted.horst at earthlink.net Fri Mar 24 18:25:13 2006 From: ted.horst at earthlink.net (Ted Horst) Date: Fri Mar 24 18:25:13 2006 Subject: [Numpy-discussion] OS X - Universal MacPython - numpy install failure In-Reply-To: References: Message-ID: <8002A455-98BF-42C1-AFF3-678E13E0D9E1@earthlink.net> I could be wrong, but I don't think 10.3.9 supports universal binaries or 10.4 sdk. I think you need a different pythn install. Ted On Mar 24, 2006, at 13:16, James Boyle wrote: > I am running OX 10.3.9. I installed python 2.4.2 using Universal > MacPython 2.4.2.dmg. > I am on a PowerPC G4. The Universal build is designed to install on > the Mac PPC and Intel machines. > > When I run the numpy 0.9.6 install it apparently fails on the > configure. The whole trace back is at the end of this posting. > One thing that I noted was the line: > > gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/ > MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp- > precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' > > It appears that the configure routines think that I have two > architectures - ppc and i386. > At my low level of understanding, all I can do is speculate that > the Universal install generates some confusion as to the actual cpu. > Somewhere numpy is keying on information that is ambiguous in the > universal install. > > I have installed Numeric, matplotlib and other packages without a > problem. > > --Jim > > > The trace back from >python setup.py build: > > Running from numpy source directory. > Warning: not existing path in numpy/distutils: site.cfg > F2PY Version 2_2236 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec', '-I/System/Library/ > Frameworks/vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec'] > > running build > running config_fc > running build_src > building py_modules sources > creating build > creating build/src > creating build/src/numpy > creating build/src/numpy/distutils > building extension "numpy.core.multiarray" sources > creating build/src/numpy/core > Generating build/src/numpy/core/config.h > customize NAGFCompiler > customize AbsoftFCompiler > customize IbmFCompiler > customize GnuFCompiler > customize GnuFCompiler > customize GnuFCompiler using config > gcc options: '-arch ppc -arch i386 -isysroot /Developer/SDKs/ > MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp- > precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g' > compile options: '-I/Library/Frameworks/Python.framework/Versions/ > 2.4/include/python2.4 -Inumpy/core/src -Inumpy/core/include -I/ > Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' > gcc: _configtest.c > gcc: cannot specify -o with -c or -S and multiple compilations > gcc: cannot specify -o with -c or -S and multiple compilations > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 76, in ? > setup_package() > File "setup.py", line 69, in setup_package > setup( **config.todict() ) > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/ > core.py", line 85, in setup > return old_setup(**new_attr) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/core.py", line 149, in setup > dist.run_commands() > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/dist.py", line 946, in run_commands > self.run_command(cmd) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/command/build.py", line 112, in run > self.run_command(cmd_name) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/Library/Frameworks/Python.framework/Versions/2.4//lib/ > python2.4/distutils/dist.py", line 966, in run_command > cmd_obj.run() > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 84, in run > self.build_sources() > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 99, in build_sources > self.build_extension_sources(ext) > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 209, in build_extension_sources > sources = self.generate_sources(sources, ext) > File "/Volumes/IBM1/macPython/numpy-0.9.6/numpy/distutils/command/ > build_src.py", line 267, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 37, in generate_config_h > raise "ERROR: Failed to test configuration" > ERROR: Failed to test configuration From tim.hochberg at cox.net Fri Mar 24 18:30:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 18:30:02 2006 Subject: [Numpy-discussion] Ransom Proposals Message-ID: <4424AB16.6000007@cox.net> Now that the fortran keyword is dead, I'll take time off from rejoicing, to briefly go over other proposals that were involved in that thread lest they get lost in the shuffle. * reshape -- The origination of that whole, enormous thread was with reshape and what its return behaviour should be. Currently it returns a view if possible, otherwise it returns a copy. That's evil behaviour. I believe that it should either always return a copy or return a view if possible and otherwise raise an exception. I think I actually prefer the later, although it's probably a more disruptive change, since its more flexible when combined with asarray and friends. One possible compromise would be to have numpy.reshape always copy, while array.reshape always returns a view. * ascontiguous -- In private email Chris Barker mentioned that the name ascontiguous was confusing, or at least not to his taste and suggested "something like" ascontiguous_array. I don't like that one, but it might worth considering something that matches asarray and asanyarray. ascontigarray looks okay to me, but it's quite possible that I've staring at this too long and that's just cryptic. * ndmin -- I still think ndmin should be spun off to a separate function. It's trivial to implement a fuction, call it paddims for lack of a better name (asatleastndimensionarray seems kind of long and cryptic!). It should have minimal performance since no data gets copied, and if it makes a difference I would be willing to implement it in C in need be so that that performance impact was minimized. def paddims(a, n): "return a view of 'a' with at least 'n' dimensions" dims = a.shape b = a.view() b.shape = (1,)*(n - len(dims)) + dims return b * copy -- Yes, I understand that the copy flag is probably not going to change for backward compatibility reasons if nothing else, but there was one last point I wanted to make about the copy flag. One of the warts that results from the copy flag, and I think that this is a common problem for functions that take parameters that switch their mode of operation, is that some combinations of input become nonsensical. Consider these five possibilities: array(anarray, dtype=matchingtype, order=matchingorder, copy=True) # OK; copied array(anarray, dtype=matchingtype, order=matchingorder, copy=False) # OK; not copied array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, copy=True) # OK; copied array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, copy=False) # Ugly; copied array(nonarray, dtype=whatever, order=whatever, copy=False) # Ugly; copied [Note that I've folded nonmatchingtype and nonmatchingorder together since they have the same effect] Of these five possibilities, two have results where the arguments and the action taken become uncoupled. One way to address this would be to change the name of the copy flag to something that matches reality: force_copy. However, that seems kind of pointless, since it still introduces as the underlying problem that some of the modes the array function can operate in are kind of bogus. Compare this to the case where the two primitives are array and asarray: array(anarray, dtype=matchingtype, order=matchingorder) # copied array(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied asarray(anarray, dtype=matchingtype, order=matchingorder) # not copied asarray(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied asarray(nonarray, dtype=whatever, order=whatever, copy=False) # copied There's still five cases, so the interface hasn't narrowed any[*], but all the possible argument combinations make sense (or raise a straightforward error). And think how much easier this behaviour is to explain! Anyway that's it for now and hopefully for a while. Regards, -tim [*] In reality it does narrow the interface because we already have asarray, but now you really need as array whereas before it was really just shorthand for array(copy=False). From tim.hochberg at cox.net Fri Mar 24 20:36:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 24 20:36:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4424AB16.6000007@cox.net> References: <4424AB16.6000007@cox.net> Message-ID: <4424C889.7090706@cox.net> Hah! This was supposed to be "Random Proposals", but Ransom is pretty good too! Tim Hochberg wrote: > > Now that the fortran keyword is dead, I'll take time off from > rejoicing, to briefly go over other proposals that were involved in > that thread lest they get lost in the shuffle. > > * reshape -- The origination of that whole, enormous thread was with > reshape and what its return behaviour should be. Currently it returns > a view if possible, otherwise it returns a copy. That's evil > behaviour. I believe that it should either always return a copy or > return a view if possible and otherwise raise an exception. I think I > actually prefer the later, although it's probably a more disruptive > change, since its more flexible when combined with asarray and > friends. One possible compromise would be to have numpy.reshape > always copy, while array.reshape always returns a view. > > * ascontiguous -- In private email Chris Barker mentioned that the > name ascontiguous was confusing, or at least not to his taste and > suggested "something like" ascontiguous_array. I don't like that one, > but it might worth considering something that matches asarray and > asanyarray. ascontigarray looks okay to me, but it's quite possible > that I've staring at this too long and that's just cryptic. > > * ndmin -- I still think ndmin should be spun off to a separate > function. It's trivial to implement a fuction, call it paddims for > lack of a better name (asatleastndimensionarray seems kind of long and > cryptic!). It should have minimal performance since no data gets > copied, and if it makes a difference I would be willing to implement > it in C in need be so that that performance impact was minimized. > > def paddims(a, n): > "return a view of 'a' with at least 'n' dimensions" > dims = a.shape > b = a.view() > b.shape = (1,)*(n - len(dims)) + dims > return b > > * copy -- Yes, I understand that the copy flag is probably not going > to change for backward compatibility reasons if nothing else, but > there was one last point I wanted to make about the copy flag. One of > the warts that results from the copy flag, and I think that this is a > common problem for functions that take parameters that switch their > mode of operation, is that some combinations of input become > nonsensical. Consider these five possibilities: > > array(anarray, dtype=matchingtype, order=matchingorder, copy=True) # > OK; copied > array(anarray, dtype=matchingtype, order=matchingorder, copy=False) # > OK; not copied > array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, > copy=True) # OK; copied > array(anarray, dtype=nonmatchingtype, order=nonmatchingorder, > copy=False) # Ugly; copied > array(nonarray, dtype=whatever, order=whatever, copy=False) # Ugly; > copied > > [Note that I've folded nonmatchingtype and nonmatchingorder together > since they have the same effect] > > Of these five possibilities, two have results where the arguments and > the action taken become uncoupled. One way to address this would be to > change the name of the copy flag to something that matches reality: > force_copy. However, that seems kind of pointless, since it still > introduces as the underlying problem that some of the modes the array > function can operate in are kind of bogus. Compare this to the case > where the two primitives are array and asarray: > > array(anarray, dtype=matchingtype, order=matchingorder) # copied > array(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied > > asarray(anarray, dtype=matchingtype, order=matchingorder) # not copied > asarray(anarray, dtype=nonmatchingtype, order=nonmatchingorder) # copied > asarray(nonarray, dtype=whatever, order=whatever, copy=False) # copied > > There's still five cases, so the interface hasn't narrowed any[*], but > all the possible argument combinations make sense (or raise a > straightforward error). And think how much easier this behaviour is to > explain! > > Anyway that's it for now and hopefully for a while. > > Regards, > > -tim > > > [*] In reality it does narrow the interface because we already have > asarray, but now you really need as array whereas before it was really > just shorthand for array(copy=False). > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From ofnap at nus.edu.sg Sat Mar 25 05:00:03 2006 From: ofnap at nus.edu.sg (Ajith Prasad (OFN)) Date: Sat Mar 25 05:00:03 2006 Subject: [Numpy-discussion] USING NUMPY FOR AN ENROLLMENT PROJECTION PROBLEM Message-ID: <502D9B13021C2D42B05F32C5C3F4C1DD2875FF@MBX01.stf.nus.edu.sg> I posted this problem - arising from my work at the local university - some years back in comp.lang.python and received a solution that did not require the use of matrix algebra. The solution was: def enrollment(year, intake, survivalrate): return sum([intake[year-i]*rate for i, rate in enumerate(survivalrate)]) I would welcome help in formulating a solution using NumPy. Thanks in advance for any suggestion. The Problem: The enrolment E(n) of an institution at the beginning of year n is the sum of the intake for year n, I(n), and the survivors from the intakes of previous r years. Thus, if s(1) is the 1-year intake survival rate, s(2) is the 2-year survival rate, etc, we have: E(n)= I(n)+I(n-1)*s(1)+ I(n-2)*s(2)+...+I(n-r)*s(r) E(n+1)= I(n+1)+I(n)*s(1)+I(n-1)*s(2)+... +I(n-r-1)*s(r) . . . E(n+k)= I(n+k)+I(n+k-1)*s(1)+I(n+k-2)*s(2)+...+I(n+k-r)*s(r) Given: (a) the actual intakes for the current and previous r years, I(n), I(n-1),I(n-2),..,I(n-r), and the planned intakes for the next n+k years: I(n+1), I(n+2),..., I(n+k), we have the intake vector I = (I(n-r), I(n-r-1),...,I(n),I(n+1),..., I(n+k)); and (b) the survival rate vector, s = (1,s(1), s(2),...,s(r)) Find: The k*1 enrolment projection column vector, E = (E(n+1),E(n+2),...,E(n+k)) in terms of a k*(r+1) matrix P (derived from I) and the (r+1)*1 column vector, s. I = P*s Is there a compact Python representation of the relevant matrix P where: P = [I(n+1) I(n) I(n-1).. . I(n-r) I(n+2) I(n+1) I(n)... I(n-r-1) . . I(n+k) I(n+k-1) I(n+k-2)... I(n+k-r)] From ndarray at mac.com Sat Mar 25 09:54:01 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 09:54:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4424AB16.6000007@cox.net> References: <4424AB16.6000007@cox.net> Message-ID: On 3/24/06, Tim Hochberg wrote: > > Now that the fortran keyword is dead, I'll take time off from rejoicing, > to briefly go over other proposals that were involved in that thread > lest they get lost in the shuffle. > Many thanks for the effort. That thread actually started as "ndarray.fill and ma.array.filled", but it quickly went off the original topic and forked at your annalistic "(mumble mumble reshape mumble)." :-) > * reshape -- The origination of that whole, enormous thread was with > reshape and what its return behaviour should be. Currently it returns a > view if possible, otherwise it returns a copy. That's evil behaviour. I > believe that it should either always return a copy or return a view if > possible and otherwise raise an exception. I think I actually prefer the > later, although it's probably a more disruptive change, since its more > flexible when combined with asarray and friends. One possible > compromise would be to have numpy.reshape always copy, while > array.reshape always returns a view. > Could you, please provide some examples of the current behavior that you find undesired and how yuo would fix it? I believe the inconsistency is that x.reshape(shape) is x depends on shape. This is easy to fix. Do you also suggest that x.reshape(...).__array_data__ is x.__array_data__ should be False? > * ascontiguous -- In private email Chris Barker mentioned that the name > ascontiguous was confusing, or at least not to his taste and suggested > "something like" ascontiguous_array. I don't like that one, but it might > worth considering something that matches asarray and asanyarray. > ascontigarray looks okay to me, but it's quite possible that I've > staring at this too long and that's just cryptic. > +0 (+1 for "ascontiguousarray") > * ndmin -- I still think ndmin should be spun off to a separate > function. It's trivial to implement a fuction, call it paddims for lack > of a better name (asatleastndimensionarray seems kind of long and > cryptic!). It should have minimal performance since no data gets copied, > and if it makes a difference I would be willing to implement it in C in > need be so that that performance impact was minimized. > > def paddims(a, n): > "return a view of 'a' with at least 'n' dimensions" > dims = a.shape > b = a.view() > b.shape = (1,)*(n - len(dims)) + dims > return b +1 (Boolean parameter is almost always a sign that a function is overloaded.) > > * copy -- Yes, I understand that the copy flag is probably not going to > change for backward compatibility reasons if nothing else, but there was > one last point I wanted to make about the copy flag. I am all for eliminating boolean parameters no matter what the consequences are. For me foo(bool, ...) is always a sign that it should be refactored to foo(...) an bar(...), with shared code possibly going to _baz(...). From ndarray at mac.com Sat Mar 25 10:13:05 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 10:13:05 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] masked_array drops fill_value In-Reply-To: References: Message-ID: Please post questions about "ma" on numpy-discussion. I don't like per-array fill_value feature. I believe fill_value should be a mandatory argument to the filled method and default fill_values only used internally to avoid passing bad data to ufuncs. The behavior that you describe seem to be historical. I would like to hear from other people who rely on per-array fill values before making any change. On 3/25/06, Tim Leslie wrote: > Hi all, > > In ma.py the masked_array function the fill_value of the array passed in is > not used. The code currently looks like this: > > def masked_array (a, mask=nomask, fill_value=None): > """masked_array(a, mask=nomask) = > array(a, mask=mask, copy=0, fill_value=fill_value) > """ > return array(a, mask=mask, copy=0, fill_value=fill_value) > > > It seems to me that using the fill_value from a (if it is a MaskedArray) > would be the sane thing to do? Something like > > if fill_value == None and isinstance(a, MaskedArray): > fill_value = a.fill_value() > return array(a, mask=mask, copy=0, fill_value=fill_value) > > As it stands, all the ma functions such as transpose, reshape, etc lose the > fill_value which seems wrong to me. > > Tim > > _______________________________________________ > Scipy-dev mailing list > Scipy-dev at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-dev > > > From tim.hochberg at cox.net Sat Mar 25 15:29:13 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 25 15:29:13 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> Message-ID: <4425D222.7010301@cox.net> Sasha wrote: >On 3/24/06, Tim Hochberg wrote: > > [SNIP] > > >>* reshape -- The origination of that whole, enormous thread was with >>reshape and what its return behaviour should be. Currently it returns a >>view if possible, otherwise it returns a copy. That's evil behaviour. I >>believe that it should either always return a copy or return a view if >>possible and otherwise raise an exception. I think I actually prefer the >>later, although it's probably a more disruptive change, since its more >>flexible when combined with asarray and friends. One possible >>compromise would be to have numpy.reshape always copy, while >>array.reshape always returns a view. >> >> >> >Could you, please provide some examples of the current behavior that >you find undesired and how yuo would fix it? I believe the >inconsistency is that x.reshape(shape) is x depends on shape. This is >easy to fix. Do you also suggest that x.reshape(...).__array_data__ >is x.__array_data__ should be False? > > Hmmm. I should defer discussion of x.reshape(shape) because I've never used it . I only use the function reshape, not the method; I'm old school ;) If this thread keeps dragging on I imagine I'll have to go look into exactly what x.reshape does. As for the function, the issue I have with it is that: reshape(obj, shape) may return a view or a copy of 'obj'. Which depends at least on what 'obj' is and may depend on shape as well. I admit I don't remember -- nor should I have to. Here's a simple example. Suppose I have some image data that is stored in a flat array [R, G, B, R, G, B, ....]. Now suppose I have a function that converts this to grey scale in place by averaging the data def suck_color_from(obj): rbg = reshape(obj, [-1. 3]) grey = (rgb[:,0] + rgb[:,1] + rgb[:,2]) / 3.0 rgb[:,0] = rgb[:,1] = rgb[:,2] = grey Now, this will work find as long as reshape is returning a view. However, if reshape does not return a view, let's say obj is a list, it will fail. And not *just* fail, it will fail silently. There are obvious ways to fix this program (set obj.shape instead of using reshape, for example), but the fewer perils I need to remember the better. That's why I'd like to sweep all the view/shape indeterminacy into a few asXXXarray functions. Everything else should always return a view or always return a copy. Here's another example, in this case we are returning a new rgb object: def make_bland_picture_from(obj): obj = copy(obj) rbg = reshape(obj, [-1. 3]) grey = (rgb[:,0] + rgb[:,1] + rgb[:,2]) / 3.0 rgb[:,0] = rgb[:,1] = rgb[:,2] = grey return obj Again this works fine until someone passes in a list or other nonarray sequence. Then, again, it fails silently. If reshape either always returned a copy, or always returned a view, this kind of trap would not be available. I believe numpy.transpose also suffers from this behaviour. My preferred behaviour is that reshape and transpose always returns a view. That is, reshape(somelist) would become a type error. Always returning a copy makes it a little more tedius to write some kinds of code efficiently, however I still think it would be better than the current situation. As for x.copy(), while I don't know what the current situation is, I think it should always return a copy. Regards, Tim From tim.hochberg at cox.net Sat Mar 25 16:21:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 25 16:21:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> Message-ID: <4425DE36.60203@cox.net> Sasha wrote: >On 3/24/06, Tim Hochberg wrote: > > > [SNIP] > >>* copy -- Yes, I understand that the copy flag is probably not going to >>change for backward compatibility reasons if nothing else, but there was >>one last point I wanted to make about the copy flag. >> >> > >I am all for eliminating boolean parameters no matter what the >consequences are. For me foo(bool, ...) is always a sign that it >should be refactored to foo(...) an bar(...), with shared code >possibly going to _baz(...). > > > Regarding the backwards compatibility issues, on some reflection, they really don't seem that bad. Assuming the conversion programs I've heard about can take care of the other issues, for this one all that needs to be done is to search for "array(\(.*\), copy=0|False, \(.*\))" and replace it with "asarray(\1, \3)". [Those aren't real regex's, they're made up psuedo regex gobbeldy gook, but I'll come up with the real deal if there's any chance that'll it'd help push things along.] I'd expect the rate of false positives from this to be extremely low. Regards, -tim From ndarray at mac.com Sat Mar 25 17:45:05 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 17:45:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425D222.7010301@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> Message-ID: On 3/25/06, Tim Hochberg wrote: > ... > However, if reshape does not return a view, let's say obj is a list, it > will fail. And not *just* fail, it will fail silently. There are obvious > ways to fix this program (set obj.shape instead of using reshape, for > example), but the fewer perils I need to remember the better. Or, you can use the reshape method instead of function. I believe numpy advocates use of methods instead of functions. What you observe is just another reason for that. Since functions like reshape remain in numpy primarily for backwards compatibility, I would be against any change in semantics. From Fernando.Perez at colorado.edu Sat Mar 25 17:50:07 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sat Mar 25 17:50:07 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> Message-ID: <4425F326.5010904@colorado.edu> Sasha wrote: > On 3/25/06, Tim Hochberg wrote: > >>... >>However, if reshape does not return a view, let's say obj is a list, it >>will fail. And not *just* fail, it will fail silently. There are obvious >>ways to fix this program (set obj.shape instead of using reshape, for >>example), but the fewer perils I need to remember the better. > > > Or, you can use the reshape method instead of function. I believe > numpy advocates use of methods instead of functions. What you observe > is just another reason for that. Since functions like reshape remain > in numpy primarily for backwards compatibility, I would be against any > change in semantics. Mmh. I bet many people will continue to use the functional interface for a long time. I'd vote for uniform semantics before 1.0. Really, the whole 'reshape(foo) and foo.reshape() have different view/copy behavior' thing is horrible. WAY too easy to forget/confuse. Special cases are /almost never/ special enough to warrant this kind of extra mental overhead. At least I know /I/ will forget, get confused, and make mistakes. So I'd like to ask for as-uniform-as possible behavior. Cheers, f From strang at nmr.mgh.harvard.edu Sat Mar 25 18:01:02 2006 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Sat Mar 25 18:01:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: >> Or, you can use the reshape method instead of function. I believe >> numpy advocates use of methods instead of functions. What you observe >> is just another reason for that. Since functions like reshape remain >> in numpy primarily for backwards compatibility, I would be against any >> change in semantics. > > Mmh. I bet many people will continue to use the functional interface for a > long time. I'd vote for uniform semantics before 1.0. Really, the whole > 'reshape(foo) and foo.reshape() have different view/copy behavior' thing is > horrible. WAY too easy to forget/confuse. Special cases are /almost never/ > special enough to warrant this kind of extra mental overhead. +1 from a strongly interested user > At least I know /I/ will forget, get confused, and make mistakes. Me too. Probably at least once/week. ;-) > So I'd like to ask for as-uniform-as possible behavior. +1 -best Gary From ndarray at mac.com Sat Mar 25 18:05:01 2006 From: ndarray at mac.com (Sasha) Date: Sat Mar 25 18:05:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: On 3/25/06, Fernando Perez wrote: > At least I know /I/ will forget, get confused, and make mistakes. So I'd like > to ask for as-uniform-as possible behavior. Do I understand correctly that you and Tim propose to make reshape(obj, shape) return a view if obj is an instance of ndarray and throw an exception otherwise? If this is the case, it is probably beter to remove the reshape function alltogether. This way reshape(obj, shape) will very consistently throw a name error and will have to be replaced with obj.reshape(shape). From Fernando.Perez at colorado.edu Sat Mar 25 18:14:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sat Mar 25 18:14:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <4425F8BA.7050903@colorado.edu> Sasha wrote: > On 3/25/06, Fernando Perez wrote: > >>At least I know /I/ will forget, get confused, and make mistakes. So I'd like >>to ask for as-uniform-as possible behavior. > > > Do I understand correctly that you and Tim propose to make > reshape(obj, shape) return a view if obj is an instance of ndarray and > throw an exception otherwise? I don't propose anything in particular. I just don't want the reshape(foo) is a view on the even days of the week and a copy on the odd ones thingie :) All I'm asking is for an API that has as few special cases as possible, so that we can both learn (and teach) a small set of rules that apply everywhere. For example, the 'foo[a:b:c] for arrays is a view while a copy for a slice', while a special case, is justifiable to new users as necessary given the performance requirements of numpy (though there's been musings on python-dev of changing to such semantics in py3k, which would be great for us). But such special rules, whether regarding deviations against the core language or internally within numpy, should (IMHO) be kept to an absolute minimum. We should treat them as expensive specialties we must pay for with a very tight budget (the willingness and ability of users to keep track of the exponential rise in complexity induced by interlocking special cases). Cheers, f From Fernando.Perez at colorado.edu Sat Mar 25 18:19:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Sat Mar 25 18:19:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F8BA.7050903@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <4425F8BA.7050903@colorado.edu> Message-ID: <4425F9FC.7050208@colorado.edu> Fernando Perez wrote: > For example, the 'foo[a:b:c] for arrays is a view while a copy for a slice', ^^^^^ That was 'list', of course. f From dhjjj at 455.com Sat Mar 25 18:49:02 2006 From: dhjjj at 455.com (fjkkk) Date: Sat Mar 25 18:49:02 2006 Subject: [Numpy-discussion] =?GB2312?B?uN+8ts7Ew9jWsNK1u6/RtS3Btw==?= Message-ID: ?? ?? 0755/82129135 82128600 82128650 ??????????????????????????????????????? ??????????????? ? ? ??????? ? ?2006?4?1-2??2006?4?8-9?? ??????????? ??????????????? ??????????????????????????????????????? ? ?? ????????????????????????????????????? ??????????????????????????????????????? ??????????????????????????????????????? ?????????????????? ????????????????????????????????????? ??????????????????????? ? ???? ? ??????????????????? ? ?????????????????????????????? ? ???????????????????? ? ???????????????? ? ?????????????????????? ? ????????????????????? ? ??????????????????????????? ? ?????????????????????????? ? ???????????????????????? ??????????????????????????????????????? ???? ????????????????? ? ???????????? ? ?????????? ? ??????????? ? ?????????????? ? ??????????????? ? ??????????? ? ??????????? ? ????? ? ?????????????? ??????????? ? ???? ? ????? ? ???????????? ? ???????? ? 13??????? ? ???? ? ?????????????? ? SMART???SWOT?? ? ???????? ????????? ? ????????? ? ??????????????????? ? ???? ? ????? ? ?????????? ? ????????? ? ????????? ? ???????????????????? ? ????????????? ? ????????????? ??????????? ? ?????? ? ??????? ? ????? ? ??????? ? ?????? ? ????? ? ????????????????? ???????? ? ???????? ? ??????? ? ????? ? ???????? ? ????????? ? ????????? ????????????? ? ????????? ? ??????????? ? ????????? ? ????????????????????? ? ??????????????? ? ??????????????? ? ?????????????? ? ?????????????? ? ????????????????? ? ?????????????? ? ???????????????????? ? ?????????????????????? ?????5S??????? ? ?????????????? ? ??????? ? ????????? ? ????????????? ????????????? ? ??????? ? ??????? ??????80/20?? ?????????? ????????????????? ? ????????? ? ????????? ? ??????????? ?????? ? ?????? ? ??????????? ? ?????? ? ????????? ???????????? ? ?????????? ? ???????????? ? ??????????? ? ?????????????? ? ??????????? ? ??????? ?????????????? 1?????????? ? ???TPO?? ? ??????? ? ??????? ? ?????? ? ????? ? ?????????? ? ??????? 2? ?????? ? ??????? ? ????? ? ????? ? ???????????? ? ???????????????? 3? ?????? ? ???? ? ???? ? ????? ? ?????? 4? ???? ? ????????? ? ????????? ? ????????? ??????????????????????????????????????? ???????? ????????????????????????????????????? ??????????????????????????????????????? ???????????????????????????????????????? ??????????????????????????????????????? ???????????????????????????????? ????????????????????????????????????? ?????????????????????????????????????? ???? ???????????????????????????????????? ???????????????????????? ???????????????????????????????????? ?????????????????????????????????????? ?????????????????????????????????????? ????????????? ???????????????????????????????????????? ? ???? ?????????????????????????????????????? ??????????????????????? ????? ????2006?3?31? ????? 4?1-2?(??) 4?8-9?(??) ????? ?????? ?????? ????? 1800?/?(???????????????????) ?????? ??????? ?? ?? 0755/82129135 82128600 82128650 ?? ?? 0755/61351396 82129209 ???xiaoyu9966333 at 126.com ?? ? ?? ??? ??? ???????????????????????????????????????? From tim.hochberg at cox.net Sat Mar 25 22:35:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sat Mar 25 22:35:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> Message-ID: <442635F6.1050109@cox.net> Sasha wrote: >On 3/25/06, Tim Hochberg wrote: > > >>... >>However, if reshape does not return a view, let's say obj is a list, it >>will fail. And not *just* fail, it will fail silently. There are obvious >>ways to fix this program (set obj.shape instead of using reshape, for >>example), but the fewer perils I need to remember the better. >> >> > >Or, you can use the reshape method instead of function. I believe >numpy advocates use of methods instead of functions. What you observe >is just another reason for that. Since functions like reshape remain >in numpy primarily for backwards compatibility, I would be against any >change in semantics. > > That's not unreasonable. How would you feel about relegating reshape and transpose and whatever else is found to have this unfortunate characteristics to a backwards compatibility module so that old code can use it, but new code would not see it. That brings up another question: is the plan to keep oldnumeric around forever, or is it going away eventually? If it is going away, then the place to put these would be oldnumeric. Actually, it's OK if it sticks around as long as it doesn't end up in the numpy namespace by default. I agree with Fernando: I don't have a specific behaviour that I'm looking for in these as long as it's consistent. Just leaving it alone, but deprecating it and shunting it off to a compatibility module would be fine. Fine, as long as x.reshape behaves sensibly: does it always return a view? You (Sasha) said something about it maybe sometimes returning a copy. If so, that should be fixed. -tim From schofield at ftw.at Sun Mar 26 05:45:04 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun Mar 26 05:45:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442635F6.1050109@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: <057A5613-8FDC-49F8-B1C7-BBEBFDD9D8C0@ftw.at> On 26/03/2006, at 8:34 AM, Tim Hochberg wrote: > That brings up another question: is the plan to keep oldnumeric > around forever, or is it going away eventually? If it is going > away, then the place to put these would be oldnumeric. Actually, > it's OK if it sticks around as long as it doesn't end up in the > numpy namespace by default. I think this is a great idea. Let's remove the functions in oldnumeric.py from the numpy namespace by default. People migrating from Numeric could add "from numpy.core.oldnumeric import *" at the top of each source file for compatibility. This would send a clear message that the functions in this module are deprecated and there for compatibility reasons only, and that the methods are the preferred interface. A clear division like this between the old and the new would also simplify decisions on when to leave the existing behaviour alone and when it should be changed for the better. Then the problem Fernando sees with the reshape function and method having different copy / view behaviour would be easier to explain and understand -- the functions just do the old Numeric thing, whereas the methods are consistent with each other. I'm now a fan of method interfaces in Python. One reason is that, with .reshape(), .sum() etc. as methods, rather than functions, it's possible for me to make the behaviour of SciPy's sparse matrices highly consistent with NumPy's arrays and matrices, without NumPy needing hooks for this. But making numpy.sum etc accept either dense or sparse matrices would require extensive modification to NumPy. -- Ed From ndarray at mac.com Sun Mar 26 05:54:01 2006 From: ndarray at mac.com (Sasha) Date: Sun Mar 26 05:54:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442635F6.1050109@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: On 3/26/06, Tim Hochberg wrote: > ... > That brings up another question: is the plan to keep oldnumeric around > forever, or is it going away eventually? If it is going away, then the > place to put these would be oldnumeric. Actually, it's OK if it sticks > around as long as it doesn't end up in the numpy namespace by default. > It *is* in oldnumeric. However, I don't really know what plans for oldnumeric are. Travis? > ... > be fine. Fine, as long as x.reshape behaves sensibly: does it always > return a view? You (Sasha) said something about it maybe sometimes > returning a copy. If so, that should be fixed. I never said that. What I said was x.reshape(shape) returns x itself if shape==x.shape and a view othrwise. This should probably be fixed because it may lead to inconsistent behavior of functions that manipulate metadata after reshape. From cjw at sympatico.ca Sun Mar 26 06:16:06 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 26 06:16:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <057A5613-8FDC-49F8-B1C7-BBEBFDD9D8C0@ftw.at> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <057A5613-8FDC-49F8-B1C7-BBEBFDD9D8C0@ftw.at> Message-ID: <4426A22A.5020503@sympatico.ca> Ed Schofield wrote: > > On 26/03/2006, at 8:34 AM, Tim Hochberg wrote: > >> That brings up another question: is the plan to keep oldnumeric >> around forever, or is it going away eventually? If it is going away, >> then the place to put these would be oldnumeric. Actually, it's OK >> if it sticks around as long as it doesn't end up in the numpy >> namespace by default. > > > I think this is a great idea. Let's remove the functions in > oldnumeric.py from the numpy namespace by default. People migrating > from Numeric could add "from numpy.core.oldnumeric import *" at the > top of each source file for compatibility. This would send a clear > message that the functions in this module are deprecated and there > for compatibility reasons only, and that the methods are the > preferred interface. A clear division like this between the old and > the new would also simplify decisions on when to leave the existing > behaviour alone and when it should be changed for the better. Then > the problem Fernando sees with the reshape function and method having > different copy / view behaviour would be easier to explain and > understand -- the functions just do the old Numeric thing, whereas > the methods are consistent with each other. > +1 > I'm now a fan of method interfaces in Python. One reason is that, > with .reshape(), .sum() etc. as methods, rather than functions, it's > possible for me to make the behaviour of SciPy's sparse matrices > highly consistent with NumPy's arrays and matrices, without NumPy > needing hooks for this. But making numpy.sum etc accept either dense > or sparse matrices would require extensive modification to NumPy. > Can sparse matrices not be handled in a sub-class? Colin W. > -- Ed From cjw at sympatico.ca Sun Mar 26 06:17:03 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 26 06:17:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <442694A9.9080108@sympatico.ca> Fernando Perez wrote: > Sasha wrote: > >> On 3/25/06, Tim Hochberg wrote: >> >>> ... >>> However, if reshape does not return a view, let's say obj is a list, it >>> will fail. And not *just* fail, it will fail silently. There are >>> obvious >>> ways to fix this program (set obj.shape instead of using reshape, for >>> example), but the fewer perils I need to remember the better. >> >> >> >> Or, you can use the reshape method instead of function. I believe >> numpy advocates use of methods instead of functions. What you observe >> is just another reason for that. Since functions like reshape remain >> in numpy primarily for backwards compatibility, I would be against any >> change in semantics. > > > Mmh. I bet many people will continue to use the functional interface > for a long time. I'd vote for uniform semantics before 1.0. Really, > the whole 'reshape(foo) and foo.reshape() have different view/copy > behavior' thing is horrible. WAY too easy to forget/confuse. Special > cases are /almost never/ special enough to warrant this kind of extra > mental overhead. > > At least I know /I/ will forget, get confused, and make mistakes. So > I'd like to ask for as-uniform-as possible behavior. > +1 Perhaps most of the functions could be in a compatibility module, which the user would import as needed, with most cases defined as something like: reshape(arr, d1, d2, ...)= numpy.ndarray.reshape(arr,d1, d2, ...) Colin W. > Cheers, > > f > From bill_lang at ir.hit.edu.cn Sun Mar 26 06:17:05 2006 From: bill_lang at ir.hit.edu.cn (Bill_Lang) Date: Sun Mar 26 06:17:05 2006 Subject: [Numpy-discussion] Problem on install Numpy. Thanks~! Message-ID: <20060326141249.D949D1B005C@ir.hit.edu.cn> Hi Numpy list members, Just now, I have run numpy-0.9.6r1.win32-py2.4.exe which was download from sourceforge. Then there is a new dictionary under "C:\Python24\Lib\site-packages\". And I checked the sys.path. the numby path is included in sys.path. But when I import numeric. There is a hint that "ImportError: No module named numeric". Is there anybody could help me to solve this problem? Thanks a lot~! BTW: I am a freshman of Python. Yet, I love Python now. Best wishes; Bill_Lang -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Sun Mar 26 06:28:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Mar 26 06:28:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: <4426A4EF.4080603@sympatico.ca> Sasha wrote: >On 3/26/06, Tim Hochberg wrote: > > >>... >>That brings up another question: is the plan to keep oldnumeric around >>forever, or is it going away eventually? If it is going away, then the >>place to put these would be oldnumeric. Actually, it's OK if it sticks >>around as long as it doesn't end up in the numpy namespace by default. >> >> >> > >It *is* in oldnumeric. However, I don't really know what plans for >oldnumeric are. Travis? > > An essential part of Ed's suggestion, as I understood it, was that the names in oldnumeric would not be hauled into numpy. The user could import oldnumeric as needed. As time goes by, the need will decline. Some consideration might be given to extending this to other non-core packages. This would reduce time required to handle basic core startup i.e. the elements being proposed for inclusion in core Python 2.6. Colin W. >>... >>be fine. Fine, as long as x.reshape behaves sensibly: does it always >>return a view? You (Sasha) said something about it maybe sometimes >>returning a copy. If so, that should be fixed. >> >> > >I never said that. What I said was x.reshape(shape) returns x itself >if shape==x.shape and a view othrwise. This should probably be fixed >because it may lead to inconsistent behavior of functions that >manipulate metadata after reshape. > > > From aisaac at american.edu Sun Mar 26 07:50:01 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun Mar 26 07:50:01 2006 Subject: [Numpy-discussion] Problem on install Numpy. Thanks~! In-Reply-To: <20060326141249.D949D1B005C@ir.hit.edu.cn> References: <20060326141249.D949D1B005C@ir.hit.edu.cn> Message-ID: On Sun, 26 Mar 2006, Bill_Lang apparently wrote: > But when I import numeric. There is a hint that > "ImportError: No module named numeric". Is there anybody > could help me to solve this problem? Just looking for the most obvious possibility ... Do you import numpy ??? (That's not what you said.) Cheers, Alan Isaac From efiring at hawaii.edu Sun Mar 26 11:00:04 2006 From: efiring at hawaii.edu (Eric Firing) Date: Sun Mar 26 11:00:04 2006 Subject: [Numpy-discussion] Problem on install Numpy. Thanks~! In-Reply-To: <20060326141249.D949D1B005C@ir.hit.edu.cn> References: <20060326141249.D949D1B005C@ir.hit.edu.cn> Message-ID: <4426E475.7070601@hawaii.edu> Bill_Lang, The module is called "numpy", not "numeric", so you need to do: import numpy Eric Bill_Lang wrote: > Hi Numpy list members, > > Just now, I have run numpy-0.9.6r1.win32-py2.4.exe which was > download from sourceforge. Then there is a new dictionary under > "C:\Python24\Lib\site-packages\". And I checked the sys.path. the numby > path is included in sys.path. > > But when I import numeric. There is a hint that "ImportError: No module > named numeric". From zpincus at stanford.edu Sun Mar 26 17:11:21 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Mar 26 17:11:21 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442694A9.9080108@sympatico.ca> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <442694A9.9080108@sympatico.ca> Message-ID: <3A2F069E-5806-4822-A9C0-E32AC5F0510C@stanford.edu> Speaking of differences between numpy functions and ndarray methods, numpy.resize(...) pads out enlarged arrays with copies of the contents of that array, while ndarray.resize(...) pads out enlarged regions with zeros. Should these be reconciled as well? I'm partial to the former case, but if there was (perhaps there already is?) another simple way to achieve this result, then it wouldn't matter which semantics were chosen. To retain compatibility with Numeric, either numpy.resize would need to retain its current semantics, or a function that does the same task (expanding and padding out an array with copies of its contents) would need to be added, so that references to resize in old code could be replaced by calls to that function. Zach >> Mmh. I bet many people will continue to use the functional >> interface for a long time. I'd vote for uniform semantics before >> 1.0. Really, the whole 'reshape(foo) and foo.reshape() have >> different view/copy behavior' thing is horrible. WAY too easy to >> forget/confuse. Special cases are /almost never/ special enough >> to warrant this kind of extra mental overhead. >> >> At least I know /I/ will forget, get confused, and make mistakes. >> So I'd like to ask for as-uniform-as possible behavior. >> > +1 > Perhaps most of the functions could be in a compatibility module, > which the user would import as needed, with most cases defined as > something like: > reshape(arr, d1, d2, ...)= numpy.ndarray.reshape(arr,d1, d2, ...) From lapataia28-northern at yahoo.com Sun Mar 26 17:51:14 2006 From: lapataia28-northern at yahoo.com (Lalo) Date: Sun Mar 26 17:51:14 2006 Subject: [Numpy-discussion] Cygwin Message-ID: <20060327015015.2402.qmail@web36208.mail.mud.yahoo.com> Hi, I tried to compile numpy 0.9.6 on a standard Cygwin installation but got some errors. Do I need to modify the site.cfg defaults? Thank you for your help. Lalo ------------------------------------------------------ Running from numpy source directory. /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=['/usr/lib/liblapack.a'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=['/usr/lib/liblapack.dll.a'] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:1264: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['blas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack_atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:1183: UserWarning: Atlas (http://math-atlas.sourceforge.net/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [atlas]) or by setting the ATLAS environment variable. warnings.warn(AtlasNotFoundError.__doc__) /home/lalo/tmp/numpy-0.9.6/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ F2PY Version 2_2236 blas_opt_info: blas_mkl_info: NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS NOT AVAILABLE atlas_blas_info: NOT AVAILABLE blas_info: FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 lapack_opt_info: lapack_mkl_info: mkl_info: NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS numpy.distutils.system_info.atlas_threads_info NOT AVAILABLE atlas_info: numpy.distutils.system_info.atlas_info NOT AVAILABLE lapack_info: FOUND: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 FOUND: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] define_macros = [('NO_ATLAS_INFO', 1)] language = f77 running install running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources adding 'build/src/numpy/core/config.h' to sources. adding 'build/src/numpy/core/__multiarray_api.h' to sources. adding 'build/src/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src/numpy/core/src/scalartypes.inc', 'build/src/numpy/core/src/arraytypes.inc', 'build/src/numpy/core/config.h', 'build/src/numpy/core/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src/numpy/core/config.h' to sources. adding 'build/src/numpy/core/__ufunc_api.h' to sources. adding 'build/src/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src/numpy/core/src/scalartypes.inc', 'build/src/numpy/core/src/arraytypes.inc', 'build/src/numpy/core/config.h', 'build/src/numpy/core/__ufunc_api.h'] building extension "numpy.core._sort" sources adding 'build/src/numpy/core/config.h' to sources. adding 'build/src/numpy/core/__multiarray_api.h' to sources. numpy.core - nothing done with h_files= ['build/src/numpy/core/config.h', 'build/src/numpy/core/__multiarray_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.dft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. building extension "numpy.random.mtrand" sources Could not locate executable gfortran Could not locate executable f95 customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes' compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc: _configtest.c _configtest.c:7:2: #error No _WIN32 _configtest.c:7:2: #error No _WIN32 failure. removing: _configtest.c _configtest.o building data_files sources running build_py copying build/src/numpy/__config__.py -> build/lib.cygwin-1.5.19-i686-2.4/numpy copying build/src/numpy/distutils/__config__.py -> build/lib.cygwin-1.5.19-i686-2.4/numpy/distutils running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using build_ext building 'numpy.core.multiarray' extension compiling C sources gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes' compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.4 -c' gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.19-i686-2.4/numpy/core/src/multiarraymodule.o -L. -lpython2.4 -o build/lib.cygwin-1.5.19-i686-2.4/numpy/core/multiarray.dll /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython2.4 collect2: ld returned 1 exit status /usr/lib/gcc/i686-pc-cygwin/3.4.4/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython2.4 collect2: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.19-i686-2.4/numpy/core/src/multiarraymodule.o -L. -lpython2.4 -o build/lib.cygwin-1.5.19-i686-2.4/numpy/core/multiarray.dll" failed with exit status 1 From wbaxter at gmail.com Sun Mar 26 18:15:03 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Sun Mar 26 18:15:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: On 3/26/06, Fernando Perez wrote: > > > Or, you can use the reshape method instead of function. I believe > > numpy advocates use of methods instead of functions. What you observe > > is just another reason for that. Since functions like reshape remain > > in numpy primarily for backwards compatibility, I would be against any > > change in semantics. > > Mmh. I bet many people will continue to use the functional interface for > a > long time. And maybe they'd be right to want a functional interface, too: http://www.gotw.ca/gotw/084.htm The author argues that in general, you should make as few functions as possible be members of a class, always preferring non-member functions wherever possible. It's about C++, but the principle should apply to Python as well. It does make a certain kind of sense, although the article seems to completely ignore polymorphism, which seems like it would be another good reason to have functions be members, at least in C++. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sun Mar 26 18:53:10 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Mar 26 18:53:10 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> Message-ID: <4427535E.4060609@ieee.org> Sasha wrote: > I never said that. What I said was x.reshape(shape) returns x itself > if shape==x.shape and a view othrwise. Is this true? I just checked on it and it didn't seem to be the case. Perhaps this was "fixed" during the recent reshape bug-fix. -Travis From tim.hochberg at cox.net Sun Mar 26 19:02:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 26 19:02:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4427535E.4060609@ieee.org> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <4427535E.4060609@ieee.org> Message-ID: <44275585.6010309@cox.net> Travis Oliphant wrote: > Sasha wrote: > >> I never said that. What I said was x.reshape(shape) returns x itself >> if shape==x.shape and a view othrwise. > > Is this true? I just checked on it and it didn't seem to be the > case. Perhaps this was "fixed" during the recent reshape bug-fix. It was at least true in the fairly recent past: >>> numpy.__version__ '0.9.7.2275' >>> a = numpy.arange(9) >>> b = a.reshape(a.shape) >>> b is a True >>> c = a.reshape((3,3)) >>> c is a False I can't get through to svn.scipy.org right now to do an update, so I can't check on current SVN, but I believe that this version is less than a week old. -tim From tim.hochberg at cox.net Sun Mar 26 20:34:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun Mar 26 20:34:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <44276AF0.5070405@cox.net> Bill Baxter wrote: > > > On 3/26/06, *Fernando Perez* > wrote: > > > Or, you can use the reshape method instead of function. I believe > > numpy advocates use of methods instead of functions. What you > observe > > is just another reason for that. Since functions like reshape > remain > > in numpy primarily for backwards compatibility, I would be > against any > > change in semantics. > > Mmh. I bet many people will continue to use the functional > interface for a > long time. > > > > And maybe they'd be right to want a functional interface, too: > http://www.gotw.ca/gotw/084.htm > > > The author argues that in general, you should make as few functions as > possible be members of a class, always preferring non-member functions > wherever possible. It's about C++, but the principle should apply to > Python as well. It does make a certain kind of sense, although the > article seems to completely ignore polymorphism, which seems like it > would be another good reason to have functions be members, at least in > C++. Personally, I've always prefered the functional interface, but it appears that for historical / backwards compatibility reasons our choice is between a functional interface that is broken and a working method based interface. For that reason alone, I'm now in favor of ditching the functional interface to 'reshape' and 'transpose'. As to whether 'reshape' and 'transpose' should be functions or methods in an ideal world, I think it's worth considering them together. It's straightforward to come up with a version of 'reshape' that's fairly generic: def reshape(obj, shape): val = obj.view() val.shape = shape return val [Unlike the current version of 'reshape', this one always returns a view; that is on purpose] This will reshape any object that supports both the 'view' method and the 'shape' attribute. Writing something equivalent for 'transpose' does not appear possible. I could do something like: def transpose(obj, axes=None): if axes == None: axes = reversed(range(len(obj.shape))) else: axes = list(axes) shape = numpy.asarray(obj.shape)[axes] strides = numpy.asarray(obj.strides)[axes] val = obj.view() val.shape = shape val.strides = strides return val However, while I consider it reasonable to require all array-like objects to have 'view' and 'shape', I doubt that it's reasonable to have them need 'strides' as well. What would 'strides' mean for a sparse array for example? Therefore, 'transpose' should be a method. Since I've always considered 'reshape' and 'transpose' a pair and would find it disconcerting if one were a method while the other was a function (this may be a personal foible however), so I'm coming to the view that they should probably have been methods all along. Regards, -tim From charlesr.harris at gmail.com Sun Mar 26 20:48:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun Mar 26 20:48:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4425F326.5010904@colorado.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: On 3/25/06, Fernando Perez wrote: > > Sasha wrote: > > On 3/25/06, Tim Hochberg wrote: > > > >>... > >>However, if reshape does not return a view, let's say obj is a list, it > >>will fail. And not *just* fail, it will fail silently. There are obvious > >>ways to fix this program (set obj.shape instead of using reshape, for > >>example), but the fewer perils I need to remember the better. > > > > > > Or, you can use the reshape method instead of function. I believe > > numpy advocates use of methods instead of functions. What you observe > > is just another reason for that. Since functions like reshape remain > > in numpy primarily for backwards compatibility, I would be against any > > change in semantics. > > Mmh. I bet many people will continue to use the functional interface for > a > long time. I'd vote for uniform semantics before 1.0. Really, the whole > 'reshape(foo) and foo.reshape() have different view/copy behavior' thing > is > horrible. WAY too easy to forget/confuse. Special cases are /almost > never/ > special enough to warrant this kind of extra mental overhead. Pimpf. I just stopped using reshape. If I wanted a different shape I assigned the tuple. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gruben at bigpond.net.au Mon Mar 27 04:46:03 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Mon Mar 27 04:46:03 2006 Subject: [Numpy-discussion] (offlist) Numpy implementation of IDL & pdl's rebin? In-Reply-To: <441DF726.1080603@auckland.ac.nz> References: <4414620C.3040202@auckland.ac.nz> <441A7831.7010704@bigpond.net.au> <441DF726.1080603@auckland.ac.nz> Message-ID: <4427E9B0.3040807@bigpond.net.au> Recently a question was asked about whether numpy/scipy had a data rebin command similar to IDL's rebin. IDL's rebin can rebin data to smaller or larger arrays whose dimensions are factors or multiples of the original size. I have had a play and placed some code which takes care of the reducing case for ndarrays on the wiki here: Gary R. From cimrman3 at ntc.zcu.cz Mon Mar 27 07:16:02 2006 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Mon Mar 27 07:16:02 2006 Subject: [Numpy-discussion] scipy.sparse + umfpack + system_info Message-ID: <44280161.4030708@ntc.zcu.cz> Hi all, I have added a basic umfpack detection support to numpy.distutils.system_info() - see example site.cfg below (and replace ...). The Umfpack wrapper still resides in the sandbox, but its setup.py now uses the detection stuff. Now I would like to move the wrapper somewhere under scipy.sparse and I hit a problem: umpfack.py depends on scipy.sparse, since it uses the CSR/CSC matrix formats, but also scipy.sparse uses umfpack.py for solving linear equations.if present. How to get out of this circular dependency problem? Do you have any suggestions on how to organize the files? r. -- site.cfg: [umfpack] library_dirs = /UMFPACK/Lib:/AMD/Lib include_dirs = /UMFPACK/Include umfpack_libs = umfpack, amd From anewgene at gmail.com Mon Mar 27 08:37:04 2006 From: anewgene at gmail.com (CL) Date: Mon Mar 27 08:37:04 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array Message-ID: <44281477.9080500@hotpop.com> Hi, group, I need to get the "rank" of an 1-D array (ie. a vector). Note that "rank" here is not the value returned from "rank(a_array)". It is the order of the item in its sorted arrray. For example, I have a python function called "listrank" to return the "rank" as below: In [19]: x Out[19]: array([1, 2, 5, 3, 3, 2]) In [20]: listrank(x) Out[20]: [6, 4, 1, 2, 2, 4] Somebody suggested me to use "argsort(argsort(x))". But the problem is it does not handle ties. See the output: In [21]: argsort(argsort(x)) Out[21]: array([0, 1, 5, 3, 4, 2]) I am wondering if there is a solution in numpy/numarray/numeric to get this done nicely. Thanks. CL From cjw at sympatico.ca Mon Mar 27 08:49:12 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Mar 27 08:49:12 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44276AF0.5070405@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> Message-ID: <4428177F.4030602@sympatico.ca> Tim Hochberg wrote: > Bill Baxter wrote: > >> >> >> On 3/26/06, *Fernando Perez* > > wrote: >> >> > Or, you can use the reshape method instead of function. I believe >> > numpy advocates use of methods instead of functions. What you >> observe >> > is just another reason for that. Since functions like reshape >> remain >> > in numpy primarily for backwards compatibility, I would be >> against any >> > change in semantics. >> >> Mmh. I bet many people will continue to use the functional >> interface for a >> long time. >> >> >> And maybe they'd be right to want a functional interface, too: >> http://www.gotw.ca/gotw/084.htm >> >> >> The author argues that in general, you should make as few functions >> as possible be members of a class, always preferring non-member >> functions wherever possible. It's about C++, but the principle >> should apply to Python as well. It does make a certain kind of >> sense, although the article seems to completely ignore polymorphism, >> which seems like it would be another good reason to have functions be >> members, at least in C++. > > > > Personally, I've always prefered the functional interface, but it > appears that for historical / backwards compatibility reasons our > choice is between a functional interface that is broken and a working > method based interface. For that reason alone, I'm now in favor of > ditching the functional interface to 'reshape' and 'transpose'. The transpose function handles a list and returns an array. Of course the case could be made that things would be clearer if the user wrote the longer: a= array([[1, 2]]).transpose() in place of: a=transpose([[1, 2]]) Perhaps extension of the T property to the array would simplify things: a=array([[1, 2]]).T Colin W. > > As to whether 'reshape' and 'transpose' should be functions or methods > in an ideal world, I think it's worth considering them together. It's > straightforward to come up with a version of 'reshape' that's fairly > generic: > > def reshape(obj, shape): > val = obj.view() > val.shape = shape > return val > > [Unlike the current version of 'reshape', this one always returns a > view; that is on purpose] This will reshape any object that supports > both the 'view' method and the 'shape' attribute. Writing something > equivalent for 'transpose' does not appear possible. I could do > something like: > > def transpose(obj, axes=None): > if axes == None: > axes = reversed(range(len(obj.shape))) > else: > axes = list(axes) > shape = numpy.asarray(obj.shape)[axes] > strides = numpy.asarray(obj.strides)[axes] > val = obj.view() > val.shape = shape > val.strides = strides > return val > > However, while I consider it reasonable to require all array-like > objects to have 'view' and 'shape', I doubt that it's reasonable to > have them need 'strides' as well. What would 'strides' mean for a > sparse array for example? Therefore, 'transpose' should be a method. > Since I've always considered 'reshape' and 'transpose' a pair and > would find it disconcerting if one were a method while the other was a > function (this may be a personal foible however), so I'm coming to the > view that they should probably have been methods all along. > > Regards, > > -tim > > > > > > > > > From tim.hochberg at cox.net Mon Mar 27 09:13:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 09:13:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428177F.4030602@sympatico.ca> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> Message-ID: <44281CE3.8080800@cox.net> Colin J. Williams wrote: > Tim Hochberg wrote: > >> Bill Baxter wrote: >> >>> >>> >>> On 3/26/06, *Fernando Perez* >> > wrote: >>> >>> > Or, you can use the reshape method instead of function. I >>> believe >>> > numpy advocates use of methods instead of functions. What you >>> observe >>> > is just another reason for that. Since functions like reshape >>> remain >>> > in numpy primarily for backwards compatibility, I would be >>> against any >>> > change in semantics. >>> >>> Mmh. I bet many people will continue to use the functional >>> interface for a >>> long time. >>> >>> >>> And maybe they'd be right to want a functional interface, too: >>> http://www.gotw.ca/gotw/084.htm >>> >>> >>> The author argues that in general, you should make as few functions >>> as possible be members of a class, always preferring non-member >>> functions wherever possible. It's about C++, but the principle >>> should apply to Python as well. It does make a certain kind of >>> sense, although the article seems to completely ignore polymorphism, >>> which seems like it would be another good reason to have functions >>> be members, at least in C++. >> >> >> >> >> Personally, I've always prefered the functional interface, but it >> appears that for historical / backwards compatibility reasons our >> choice is between a functional interface that is broken and a working >> method based interface. For that reason alone, I'm now in favor of >> ditching the functional interface to 'reshape' and 'transpose'. > > > The transpose function handles a list and returns an array. Yes. And this is exactly the same property that I've been ranting about incessantly for the past several days in the case of reshape. It seems convenient, but it's evil. It leads to intermittent, hard to find bugs, since when passed a list it necessarily creates a new array object, but when passed an array, it returns a view. > Of course the case could be made that things would be clearer if the > user wrote the longer: > a= array([[1, 2]]).transpose() > in place of: > a=transpose([[1, 2]]) The best way to do deal with this is to make sure everything is an array at the boundaries of your function. def frobinate(a, b): a = asarray(a) b = asarray(b) bar = a.transpose() stool = a.reshape([1,2,3]) # etc, c = asarray(get_data_from_somewhere_that_might_not_return_an_array()) # etc. It would easy enough to automate this using decorators if the extra lines bothers people. Thus the above could become: @array_arguments def frobinate(a, b): bar = a.transpose() stool = a.reshape([1,2,3]) # etc, c = asarray(get_data_from_somewhere_that_might_not_return_an_array()) # etc. In either case, then you can use all of the array functions without worry, and with maximum efficiency, in the body of your function. > Perhaps extension of the T property to the array would simplify things: > a=array([[1, 2]]).T I tend to think that the 'T' property should stay put. 'T' is really specific to two dimensional arrays and looses much of its appeal when applied to multidimensional arrays since it takes no arguments. I think it adds more clutter than functionality, so I'm -1 on that idea. Regards, -tim From tim.hochberg at cox.net Mon Mar 27 09:51:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 09:51:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4427535E.4060609@ieee.org> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <4427535E.4060609@ieee.org> Message-ID: <442825C6.6060707@cox.net> Travis Oliphant wrote: > Sasha wrote: > >> I never said that. What I said was x.reshape(shape) returns x itself >> if shape==x.shape and a view othrwise. > > Is this true? I just checked on it and it didn't seem to be the > case. Perhaps this was "fixed" during the recent reshape bug-fix. > It looks like the bug is in PyArray_Ravel. array_reshape calls PyArray_Ravel if the newshape has dimension-1 (line 77 of array_methods.c): if (newshape.len == 1) { PyDimMem_FREE(newshape.ptr); return PyArray_Ravel(self, 0); } Ravel in turn, returns self if the array is already flat (line 187 of multiarray_module.c): if (!fortran && PyArray_ISCONTIGUOUS(a)) { if (a->nd == 1) { Py_INCREF(a); return (PyObject *)a; } return PyArray_Newshape(a, &newdim, PyArray_CORDER); } It looks like ripping out the second if statement would fix everything, but I haven't tried it yet. -tim From tim.hochberg at cox.net Mon Mar 27 10:09:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 10:09:07 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array In-Reply-To: <44281477.9080500@hotpop.com> References: <44281477.9080500@hotpop.com> Message-ID: <442829EE.7050204@cox.net> CL wrote: > Hi, group, > I need to get the "rank" of an 1-D array (ie. a vector). > Note that "rank" here is not the value returned from "rank(a_array)". > It is the order of the item in its sorted arrray. For example, I have > a python function called "listrank" to return the "rank" as below: In the future, please include the relevant function. This saves us (me anyway) time reverse engineering said function from the description you give. Is the function below equivalent to your listrank function? def listrank(v): rank = {} for i, x in enumerate(reversed(sorted(v))): if x not in rank: rank[x] = i return [rank[x]+1 for x in v] > > In [19]: x > Out[19]: array([1, 2, 5, 3, 3, 2]) > > In [20]: listrank(x) > Out[20]: [6, 4, 1, 2, 2, 4] > > Somebody suggested me to use "argsort(argsort(x))". But the problem is > it does not handle ties. See the output: > > In [21]: argsort(argsort(x)) > Out[21]: array([0, 1, 5, 3, 4, 2]) > > I am wondering if there is a solution in numpy/numarray/numeric to get > this done nicely. Unfortunately, nothing comes to mind immediately. This kind of problem, where the values at one index depend on the values at a different index is often hard to deal with in the array framework. How large of vectors are you typically dealing with? If they are not extremely large or this isn't a performance critical a python solution like above, possibly somewhat optimized, may well be sufficient. Perhaps someone else will come up with something though. Regards, -tim From oliphant at ee.byu.edu Mon Mar 27 10:46:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 10:46:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44281CE3.8080800@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> Message-ID: <4428329B.7050300@ee.byu.edu> Tim Hochberg wrote: >> >> >> The transpose function handles a list and returns an array. > > > Yes. And this is exactly the same property that I've been ranting > about incessantly for the past several days in the case of reshape. It > seems convenient, but it's evil. It leads to intermittent, hard to > find bugs, since when passed a list it necessarily creates a new array > object, but when passed an array, it returns a view. > And this is also the area where Tim is going to find people that disagree. The functional interfaces have always been there so that other objects could be manipulated as arrays. There is *a lot* of precendent behind the functional interfaces. It seems like a big change to simply get rid of them . Yes, having this ability means that you have to think about it a bit if you are going to use the functional interface and try to do in-place operations. But, I would argue that this is an "advanced" usage which is implemented to save space and time. I fully support such "advanced" usages, but believe the person using them should know what they are doing. Understanding why the functional interface may return a copy does not seem to me to be a big deal for somebody thinking about "in-place" operations. I'm -1 right now on removing (or even deprecating) the functional interfaces. I'm +1 on educating users on the advantages of using methods and that the functional interfaces are just basic wrappers around them. -Travis From oliphant at ee.byu.edu Mon Mar 27 10:48:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 10:48:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442825C6.6060707@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <442635F6.1050109@cox.net> <4427535E.4060609@ieee.org> <442825C6.6060707@cox.net> Message-ID: <44283339.8080007@ee.byu.edu> > > Ravel in turn, returns self if the array is already flat (line 187 of > multiarray_module.c): > > if (!fortran && PyArray_ISCONTIGUOUS(a)) { > if (a->nd == 1) { > Py_INCREF(a); > return (PyObject *)a; > } > return PyArray_Newshape(a, &newdim, PyArray_CORDER); > } I just eliminated this so that .ravel() always returns a new view. I can see this optimization leads to an undesirable inconsistency. I also added a special case for FORTRANORDER arrays so that is consistent with the CORDER case as well. -Travis From oliphant at ee.byu.edu Mon Mar 27 10:58:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 10:58:00 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <3A2F069E-5806-4822-A9C0-E32AC5F0510C@stanford.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <442694A9.9080108@sympatico.ca> <3A2F069E-5806-4822-A9C0-E32AC5F0510C@stanford.edu> Message-ID: <44283577.7010809@ee.byu.edu> Zachary Pincus wrote: > Speaking of differences between numpy functions and ndarray methods, > numpy.resize(...) pads out enlarged arrays with copies of the > contents of that array, while ndarray.resize(...) pads out enlarged > regions with zeros. Yep, this is a documented difference. I can see desireability of both behaviors, but I agree that using the same name for different behaviors (without a keyword flag) is not a good idea. > > Should these be reconciled as well? I'm partial to the former case, > but if there was (perhaps there already is?) another simple way to > achieve this result, then it wouldn't matter which semantics were > chosen. > > To retain compatibility with Numeric, either numpy.resize would need > to retain its current semantics, or a function that does the same > task (expanding and padding out an array with copies of its contents) > would need to be added, so that references to resize in old code > could be replaced by calls to that function. I think for consistency we should make a new function and then add it to the list of replaced functions in convertcode.py The other possibility that I'm liking more is to use a keyword argument that specifies a fill-value. The default for the method could be 0 while the default for the resize-function could be the array itself. True, different default keywords may not be convenient (but that exists already with the axis keyword for many methods and is done for backwards compatibility). All the code review is great. Please continue it. -Travis From Chris.Barker at noaa.gov Mon Mar 27 11:08:00 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Mar 27 11:08:00 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> Message-ID: <44283805.40101@noaa.gov> Sasha wrote: > If this is the case, it is probably better to remove the reshape > function altogether. +1 I'm a big fan of OO syntax -- it really cleans up name spaces. Colin J. Williams wrote: > Perhaps most of the functions could be in a compatibility module, which > the user would import as needed, with most cases defined as something like: > reshape(arr, d1, d2, ...)= numpy.ndarray.reshape(arr,d1, d2, ...) +1 Ed Schofield wrote: > I'm now a fan of method interfaces in Python. One reason is that, with > .reshape(), .sum() etc. as methods, rather than functions, it's possible > for me to make the behaviour of SciPy's sparse matrices highly > consistent with NumPy's arrays and matrices Another good reason for an OO interface! Bill Baxter wrote: > And maybe they'd be right to want a functional interface, too: > http://www.gotw.ca/gotw/084.htm I can see that this applies to sin(), cos() etc, but from that doc: """Prefer to make it a member if it needs access to internals:""" I think reshape() and friends fall into this category. I always thought there was an odd argument in the old Numeric docs: """ Choosing functions means that the same procedures can be applied to arbitrary python sequences, not just to arrays. For example, while: >> transpose([[1,2], [3,4]]) works just fine, >> [[1,2], [3,4]].transpose() can't work. """ For me, that argument would only hold if transposing a list returned a transposed list, but it doesn't: it returns an array. What's going on under the hood is something like: return asarray(TheInput).transpose() The user can not use transpose() without committing to arrays, so why not just have them make the asarray call. All you're doing is creating a less explicit interface and cluttering the name space to save some typing. As far I'm concerned, most functions that have a signature like: Array = function(Array, whatever) should probably be methods. I know there's history to consider, so I don't actually expect this to change, and I'm very happy with the new methods introduced by numpy. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From tim.hochberg at cox.net Mon Mar 27 11:08:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 11:08:03 2006 Subject: [Numpy-discussion] order flag again Message-ID: <442837E4.8020303@cox.net> I notice that some methods of the array object, notably reshape, take the order flag. I think this is a mistake. At this point, x.reshape() should always be returning a view. That means it's a cheap operation. Similarly, setting the order flag is also a cheap operation[*]. For that reason there's not a pressing efficiency reason to combine to the two operations and I feel that they should be broken apart. Thus we'd gain an operation 'setorder' or similar and lose the order flag everywhere but array and the various asXXXarray[**] functions since they need it to efficiently construct a correctly ordered array. Thus: b = a.reshape(newshape, order=neworder) would become: b = a.reshape(newshape).setorder(neworder) That doesn't look substantially better in that form, but compare: b = a.reshape(weeble, order=wobble) with: b = a.reshape(weeble).setorder(wobble) And the superiority of the second form begins to becomes apparent. The requirement that order be keyword parameter helps, but not enough IMO. The second form is orthogonal and self documenting and just all around more swell ;) Regards, -tim [*] I know this even though I haven't thought through what reshape plus the order flag is doing because reshape isn't allowed to return a copy. Thus setting the order flag in this context must amount to just rejiggering shape and strides. [**] And I suppose, maybe, ravel. However, ravel is yet another evil function, or rather method in this case, that sometimes returns a view and sometimes a copy. Ravels is really unnecessary, and I don't have the energy to wage a campaign against its evil ways (much to everyones relief I imagine), so I plan to just ignore it and recomend that no one use it . Ever. From anewgene at gmail.com Mon Mar 27 11:25:11 2006 From: anewgene at gmail.com (CL) Date: Mon Mar 27 11:25:11 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array In-Reply-To: <442829EE.7050204@cox.net> References: <44281477.9080500@hotpop.com> <442829EE.7050204@cox.net> Message-ID: <44283BEE.5090802@hotpop.com> Thanks, Tim. Your function of "listrank" is indeed equivalent with mine. My typical vector size is 150K and there are thousands of such vectors need to be processed. I think the performace would be boosted if there is any way other than pure python fucntion "listrank". Thanks again, CL Tim Hochberg wrote: > CL wrote: > >> Hi, group, >> I need to get the "rank" of an 1-D array (ie. a vector). >> Note that "rank" here is not the value returned from "rank(a_array)". >> It is the order of the item in its sorted arrray. For example, I have >> a python function called "listrank" to return the "rank" as below: > > > In the future, please include the relevant function. This saves us (me > anyway) time reverse engineering said function from the description > you give. Is the function below equivalent to your listrank function? > > def listrank(v): > rank = {} > for i, x in enumerate(reversed(sorted(v))): > if x not in rank: > rank[x] = i > return [rank[x]+1 for x in v] > >> >> In [19]: x >> Out[19]: array([1, 2, 5, 3, 3, 2]) >> >> In [20]: listrank(x) >> Out[20]: [6, 4, 1, 2, 2, 4] >> >> Somebody suggested me to use "argsort(argsort(x))". But the problem >> is it does not handle ties. See the output: >> >> In [21]: argsort(argsort(x)) >> Out[21]: array([0, 1, 5, 3, 4, 2]) >> >> I am wondering if there is a solution in numpy/numarray/numeric to >> get this done nicely. > > > Unfortunately, nothing comes to mind immediately. This kind of > problem, where the values at one index depend on the values at a > different index is often hard to deal with in the array framework. How > large of vectors are you typically dealing with? If they are not > extremely large or this isn't a performance critical a python solution > like above, possibly somewhat optimized, may well be sufficient. > > Perhaps someone else will come up with something though. > > Regards, > > -tim > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From tim.hochberg at cox.net Mon Mar 27 11:43:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 11:43:04 2006 Subject: [Numpy-discussion] A newbie question: How to get the "rank" of an 1-d array In-Reply-To: <44283BEE.5090802@hotpop.com> References: <44281477.9080500@hotpop.com> <442829EE.7050204@cox.net> <44283BEE.5090802@hotpop.com> Message-ID: <44284001.2080108@cox.net> CL wrote: > Thanks, Tim. Your function of "listrank" is indeed equivalent with > mine. My typical vector size is 150K and there are thousands of such > vectors need to be processed. Ouch! > I think the performace would be boosted if there is any way other than > pure python fucntion "listrank". OK. Well still no great ideas on this end. If you have some a priori knowledge about the vector, there might be some tricks using take. But that would require that the values in 'v; be integers that are on the order of len(v): is that the case? In any event, the following is considerably faster than the previous version: def listrank_4(v): rank = {} for i, x in enumerate(numpy.sort(v)[::-1]): if x not in rank: rank[x] = i return numpy.array([rank[x] for x in v]) + 1 The execution time here appears to be dominated by the time it takes to insert items in the dictionary. If we know enough about the items of 'v', we could potentially replace the dictionary with a vector and speed things about quite a bit more. Regards, -tim > > Thanks again, > > CL > > Tim Hochberg wrote: > >> CL wrote: >> >>> Hi, group, >>> I need to get the "rank" of an 1-D array (ie. a vector). >>> Note that "rank" here is not the value returned from >>> "rank(a_array)". It is the order of the item in its sorted arrray. >>> For example, I have a python function called "listrank" to return >>> the "rank" as below: >> >> >> >> In the future, please include the relevant function. This saves us >> (me anyway) time reverse engineering said function from the >> description you give. Is the function below equivalent to your >> listrank function? >> >> def listrank(v): >> rank = {} >> for i, x in enumerate(reversed(sorted(v))): >> if x not in rank: >> rank[x] = i >> return [rank[x]+1 for x in v] >> >>> >>> In [19]: x >>> Out[19]: array([1, 2, 5, 3, 3, 2]) >>> >>> In [20]: listrank(x) >>> Out[20]: [6, 4, 1, 2, 2, 4] >>> >>> Somebody suggested me to use "argsort(argsort(x))". But the problem >>> is it does not handle ties. See the output: >>> >>> In [21]: argsort(argsort(x)) >>> Out[21]: array([0, 1, 5, 3, 4, 2]) >>> >>> I am wondering if there is a solution in numpy/numarray/numeric to >>> get this done nicely. >> >> >> >> Unfortunately, nothing comes to mind immediately. This kind of >> problem, where the values at one index depend on the values at a >> different index is often hard to deal with in the array framework. >> How large of vectors are you typically dealing with? If they are not >> extremely large or this isn't a performance critical a python >> solution like above, possibly somewhat optimized, may well be >> sufficient. >> >> Perhaps someone else will come up with something though. >> >> Regards, >> >> -tim >> >> >> >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by xPML, a groundbreaking scripting >> language >> that extends applications into web and mobile media. Attend the live >> webcast >> and join the prime developer group breaking into this new coding >> territory! >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Mon Mar 27 11:49:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 11:49:05 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <442837E4.8020303@cox.net> References: <442837E4.8020303@cox.net> Message-ID: <4428418A.1070606@ee.byu.edu> Tim Hochberg wrote: > > I notice that some methods of the array object, notably reshape, take > the order flag. I think this is a mistake. At this point, x.reshape() > should always be returning a view. The reason for the order flag is that it is needed to interpret what you think a reshape operation should do. How should the reshape method think the data is stored. That is the intended meaning of the reshape flag. The FORTRAN flag on the array means the strides are actually equivalent to FORTRAN order The CONTIGUOUS flag on the array means the strides are actually equilvalent to C-CONTIGUOUS order. For a non-contiguous array, you may still have a sense of which dimension you think ought to vary the fastest. This kind of "sense" is not needed all the time --- only on certain operations. For these operations, the order keyword needs to be present. To do what Tim suggests would require the addition of an additional flag (call it the ORDER flag) on the array that would be carried around to tell everybody how to think about this array as a one-dimensional sequence of bytes. Such an approach is possible, of course. -Travis From zpincus at stanford.edu Mon Mar 27 12:14:56 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 27 12:14:56 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428418A.1070606@ee.byu.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> Message-ID: <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> > The reason for the order flag is that it is needed to interpret > what you think a reshape operation should do. How should the > reshape method think the data is stored. That is the intended > meaning of the reshape flag. > The FORTRAN flag on the array means the strides are actually > equivalent to FORTRAN order > The CONTIGUOUS flag on the array means the strides are actually > equilvalent to C-CONTIGUOUS order. > [...] > To do what Tim suggests would require the addition of an additional > flag (call it the ORDER flag) on the array that would be carried > around to tell everybody how to think about this array as a one- > dimensional sequence of bytes. > Such an approach is possible, of course. Does this mean that if I create a new array with FORTRAN order from the numpy.array() function, every time I reshape that array I need to tell reshape to use FORTRAN order? That seems a bit odd... Yet the text above seems to suggests that arrays will not know what order their own bytes are in, and thus would need to be told whether they should be reshaped like FORTRAN or C memory blocks. I assume I'm misinterpreting something here? Zach From tim.hochberg at cox.net Mon Mar 27 12:18:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 12:18:07 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428329B.7050300@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> Message-ID: <4428484F.2030604@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> >>> The transpose function handles a list and returns an array. >> >> >> >> Yes. And this is exactly the same property that I've been ranting >> about incessantly for the past several days in the case of reshape. >> It seems convenient, but it's evil. It leads to intermittent, hard to >> find bugs, since when passed a list it necessarily creates a new >> array object, but when passed an array, it returns a view. >> > > And this is also the area where Tim is going to find people that > disagree. The functional interfaces have always been there so that > other objects could be manipulated as arrays. There is *a lot* of > precendent behind the functional interfaces. It seems like a big > change to simply get rid of them . Yes. I certainly hope to get disagreement. While I'm certain that I'm right, this is a big change and I would feel uncomfortable if it were adopted without being thoroughly thrashed out. This way, if it does get adopted, I'll feel comfortable that it was fully vetted. And if doesn't, I get to say I told you so every time it bites someone ;) And who knows perhaps you can change my mind. > > Yes, having this ability means that you have to think about it a bit > if you are going to use the functional interface and try to do > in-place operations. But, I would argue that this is an "advanced" > usage which is implemented to save space and time. How is this true though? In what way, for instance, is: b = asarray(a).reshape(newshape) slower or less space efficient than todays: b = reshape(a) ? If the answer is just the overhead of the call to asarray, I'll rewrite it in C for you at which point the overhead will truly be negligible. [Note I don't encourage the former style. In general I think all objects you want to use as arrays should be converted at the function boundaries.] In fact, I'd argue just the opposite. The function encourage people to *not* convert objects to arrays at the boundaries of their functions. As a result, in addition to getting subtle, intermitent bugs, they tend to end up converting from objects to arrays multiple times inside a given function; wasting both space and time. Let me muddy the waters a bit by saying that the subtle bugs you can get from reshape accepting lists pale next to the issue that whether you get a view or a copy depends on what the strides of the array are. Who keeps track of strides? I'd forgotten about that until I was looking at ravel, at which point I recalled the true horror of the function that is reshape. For those of you playing along at home: >>> a = numpy.arange(9).reshape([3,3]) >>> b = a[1:,1:] >>> flat_a = a.reshape(-1) >>> flat_b = b.reshape(-1) >>> flat_a[0] = 999 >>> flat_b[0] = 999 >>> a array([[999, 1, 2], [ 3, 4, 5], [ 6, 7, 8]]) >>> b array([[4, 5], [7, 8]]) !!! > I fully support such "advanced" usages, but believe the person using > them should know what they are doing. Understanding why the > functional interface may return a copy does not seem to me to be a big > deal for somebody thinking about "in-place" operations. If I truly saw any benefit to these, other than saving the typing of a few characters, I'd agree. However, I don't see such benefit. I don't see them advanced either as they offer no benefit over the corresponding "basic" usages other than marginally less typing. And you pay for that as it encourages bad practices and results in subtle bugs. > > I'm -1 right now on removing (or even deprecating) the functional > interfaces. How about just segregating them. You can even call the submodule "advanced" if you like, although I'd prefer "danger_danger_warning", so perhaps "functions" would be a good compromise. Then I, and other like-minded numpy users, can simply ban the use of the "functions" submodule from our code and live blissful, efficient, bug free existences. Or something like that. > I'm +1 on educating users on the advantages of using methods and > that the functional interfaces are just basic wrappers around them. Let me restate that I'm not method-ist (nor a Methodist for that matter), I'd be perfectly happy with functions. However, it seems impossible for backwards compatibility and cultural reasons to have non-broken function semantics. So for that reason I'm in favor of deprecating or segregating the borked functions. I'm not actually in favor of removing them altogether, as that would unnecessarily complicate migration from Numeric, but I am in favor of booting them out of the main numpy namespace. Regards, Tim From tim.hochberg at cox.net Mon Mar 27 12:48:51 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 12:48:51 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428418A.1070606@ee.byu.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> Message-ID: <44284F3E.1080102@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> I notice that some methods of the array object, notably reshape, take >> the order flag. I think this is a mistake. At this point, x.reshape() >> should always be returning a view. > > > The reason for the order flag is that it is needed to interpret what > you think a reshape operation should do. How should the reshape > method think the data is stored. That is the intended meaning of the > reshape flag. > The FORTRAN flag on the array means the strides are actually > equivalent to FORTRAN order > The CONTIGUOUS flag on the array means the strides are actually > equilvalent to C-CONTIGUOUS order. > > For a non-contiguous array, you may still have a sense of which > dimension you think ought to vary the fastest. This kind of "sense" > is not needed all the time --- only on certain operations. For these > operations, the order keyword needs to be present. > To do what Tim suggests would require the addition of an additional > flag (call it the ORDER flag) on the array that would be carried > around to tell everybody how to think about this array as a > one-dimensional sequence of bytes. > Such an approach is possible, of course. OK, so the order flag is needed anytime you might want to copy the data, correct? What x.reshape() should do is still up in the air, but I feel strongly that it shouldn't be copying *sometimes*. I'm going to assume for the moment that my previous proposal regarding reshape and ravel gets adopted. We have three use cases: 1. Want a reshaped view: b = a.reshape(*newshape) 2. Want a reshaped copy b = a.ravel(*newshape=[-1], order=neworder) 3. Want a reshaped view if possible or a copy: b = ascontiguousarray(a, order=neworder).reshape(*newshape) Reshape doesn't need the order flag since it's not in the business of making copies. If you want a copy, use the new, revamped ravel which makes a copy and respects the order[*]. Now this third case is what I would consider an advanced usage as described in Travis' previous email. You want a contiguous array, probably flat, and you want it as efficiently as possible. However it's also a potential source of subtle bugs since you may or may not get a view. For this reason, I think that requiring ascontiguousarray to support this case is not only acceptable, it's a very good thing. As I've mentioned before, I'd like to sweep all of the operations that may return a view or a copy into a few asXXXarray functions so that these potentially problematic operations are localized and easily identifiable. Regards, -tim [*] I've never been convinced of the need for ravel myself, since it's always been possible to write it efficiently some other way. However, it's more useful under the current proposal and I have no desire to get rid of it as long as it's behaviour stops being evil! From Fernando.Perez at colorado.edu Mon Mar 27 12:57:05 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 27 12:57:05 2006 Subject: [Numpy-discussion] order flag again Message-ID: <4428516E.4070308@colorado.edu> On 01:46 PM Tim Hochberg wrote: > As I've mentioned before, I'd like to sweep all of the operations that may > return a view or a copy into a few asXXXarray functions so that these > potentially problematic operations are localized and easily identifiable. +1 on the idea that: asXXXarray: functions which take their input and return always an array, but trying to avoid copies if at all possible. This means that code calling such functions should treat their returned values strictly as /inputs/, never modifying them in place. These functions are provided by numpy strictly for performance reasons. Or something like that. I like being able to tell to users that this kind of problem is solved by a unified familiy of functions in the API, with a clear naming convention. There is a need for them, but they are also a bit dangerous, so they should be clearly labeled. Painting them all in the same bright red helps indicate their 'sharp blade' nature. Cheers, f From charlesr.harris at gmail.com Mon Mar 27 13:06:01 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 13:06:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428484F.2030604@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> Message-ID: On 3/27/06, Tim Hochberg wrote: > > Travis Oliphant wrote: > > > Tim Hochberg wrote: > > > >>> > >>> > >>> The transpose function handles a list and returns an array. > >> > >> > >> > >> Yes. And this is exactly the same property that I've been ranting > >> about incessantly for the past several days in the case of reshape. > >> It seems convenient, but it's evil. It leads to intermittent, hard to > >> find bugs, since when passed a list it necessarily creates a new > >> array object, but when passed an array, it returns a view. > >> > > > > And this is also the area where Tim is going to find people that > > disagree. The functional interfaces have always been there so that > > other objects could be manipulated as arrays. There is *a lot* of > > precendent behind the functional interfaces. It seems like a big > > change to simply get rid of them . > > Yes. I certainly hope to get disagreement. While I'm certain that I'm > right, this is a big change and I would feel uncomfortable if it were > adopted without being thoroughly thrashed out. This way, if it does get > adopted, I'll feel comfortable that it was fully vetted. And if doesn't, > I get to say I told you so every time it bites someone ;) And who knows > perhaps you can change my mind. > > > > > Yes, having this ability means that you have to think about it a bit > > if you are going to use the functional interface and try to do > > in-place operations. But, I would argue that this is an "advanced" > > usage which is implemented to save space and time. > > How is this true though? In what way, for instance, is: > > b = asarray(a).reshape(newshape) > > slower or less space efficient than todays: > > b = reshape(a) > > ? If the answer is just the overhead of the call to asarray, I'll > rewrite it in C for you at which point the overhead will truly be > negligible. [Note I don't encourage the former style. In general I think > all objects you want to use as arrays should be converted at the > function boundaries.] > > In fact, I'd argue just the opposite. The function encourage people to > *not* convert objects to arrays at the boundaries of their functions. As > a result, in addition to getting subtle, intermitent bugs, they tend to > end up converting from objects to arrays multiple times inside a given > function; wasting both space and time. > > Let me muddy the waters a bit by saying that the subtle bugs you can get > from reshape accepting lists pale next to the issue that whether you get > a view or a copy depends on what the strides of the array are. Who keeps > track of strides? I'd forgotten about that until I was looking at ravel, > at which point I recalled the true horror of the function that is > reshape. For those of you playing along at home: > > >>> a = numpy.arange(9).reshape([3,3]) > >>> b = a[1:,1:] > >>> flat_a = a.reshape(-1) > >>> flat_b = b.reshape(-1) > >>> flat_a[0] = 999 > >>> flat_b[0] = 999 > >>> a > array([[999, 1, 2], > [ 3, 4, 5], > [ 6, 7, 8]]) > >>> b > array([[4, 5], > [7, 8]]) > > !!! > > > I fully support such "advanced" usages, but believe the person using > > them should know what they are doing. Understanding why the > > functional interface may return a copy does not seem to me to be a big > > deal for somebody thinking about "in-place" operations. > > If I truly saw any benefit to these, other than saving the typing of a > few characters, I'd agree. However, I don't see such benefit. I don't > see them advanced either as they offer no benefit over the corresponding > "basic" usages other than marginally less typing. And you pay for that > as it encourages bad practices and results in subtle bugs. > > > > > I'm -1 right now on removing (or even deprecating) the functional > > interfaces. > > How about just segregating them. You can even call the submodule > "advanced" if you like, although I'd prefer "danger_danger_warning", so > perhaps "functions" would be a good compromise. Then I, and other > like-minded numpy users, can simply ban the use of the "functions" > submodule from our code and live blissful, efficient, bug free > existences. Or something like that. > > > I'm +1 on educating users on the advantages of using methods and > > that the functional interfaces are just basic wrappers around them. > > Let me restate that I'm not method-ist (nor a Methodist for that > matter), I'd be perfectly happy with functions. However, it seems > impossible for backwards compatibility and cultural reasons to have > non-broken function semantics. So for that reason I'm in favor of > deprecating or segregating the borked functions. I'm not actually in > favor of removing them altogether, as that would unnecessarily > complicate migration from Numeric, but I am in favor of booting them out > of the main numpy namespace. How about functions always return a copy unless they are carefully documented interface helper functions like asarray. That's how I generally think of them anyway. Of course, the main virtue of asarray is that it helps write functions that do precisely what Tim is arguing against: mixing list and array types and returning copies in the first case, views in the second. So if we throw out functions we should throw out asarray also and make a rule that numpy functions don't take lists. And then there are scalars... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Mar 27 13:19:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 13:19:02 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428516E.4070308@colorado.edu> References: <4428516E.4070308@colorado.edu> Message-ID: On 3/27/06, Fernando Perez wrote: > > On 01:46 PM Tim Hochberg wrote: > > > As I've mentioned before, I'd like to sweep all of the operations that > may > > return a view or a copy into a few asXXXarray functions so that these > > potentially problematic operations are localized and easily > identifiable. > > +1 on the idea that: > > asXXXarray: functions which take their input and return always an array, > but > trying to avoid copies if at all possible. This means that code calling > such > functions should treat their returned values strictly as /inputs/, never > modifying them in place. These functions are provided by numpy strictly > for > performance reasons. How about Tim's example: b = asarray(a).reshape(newshape) Or something like that. I think the problem is mixing python list types with arrays. It is convenient but we get into trouble. What purpose does ascontiguous serve? I like being able to tell to users that this kind of > problem is solved by a unified familiy of functions in the API, with a > clear > naming convention. There is a need for them, but they are also a bit > dangerous, so they should be clearly labeled. Painting them all in the > same > bright red helps indicate their 'sharp blade' nature. > > Cheers, > > f > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Mon Mar 27 13:29:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 13:29:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> Message-ID: <442858F5.6030001@cox.net> Charles R Harris wrote: [CHOP] > > How about functions always return a copy unless they are carefully > documented interface helper functions like asarray. That's how I > generally think of them anyway. I don't really care as long as they (a) stop being evil and (b) there's some way to do stuff efficiently. Functions that always return copies seem sort of useless to me, but I'd be happy to move to methods if that was the case. However, I suspect you won't make much headway with this since my impression is that many of the people who do care about functions also care passionately about efficiency as well. > Of course, the main virtue of asarray is that it helps write functions > that do precisely what Tim is arguing against: mixing list and array > types and returning copies in the first case, views in the second. So > if we throw out functions we should throw out asarray also and make a > rule that numpy functions don't take lists. This isn't right. I think it's perfectly fine, desirable in fact, for functions to operate on both list and array types. The problem only arises when you return ether a view or copy of one of the original inputs depending on what it was. In my experience, this is relatively hard to do in all but the most trivial of functions since most operations return a new array. However, it is true that I think people should be strongly discouraged from doing this. For example: def add(a, b): a = asarray(a) b = asarray(b) return a + b is fine, but: def iadd(a, b): a = asarray(a) b = asarray(b) a += b return a is not and should be rewritten. Possibly like: def iadd(a, b): if not isinstance(a, ndarray): raise TypeError("'a' must be an array") b = asarray(b) a += b return a since += is a bit squirley. (Try somelist += somearray, to see what I mean). > And then there are scalars... I'm curious what problems you forsee with scalars. Regards, -tim From Fernando.Perez at colorado.edu Mon Mar 27 13:31:12 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 27 13:31:12 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: References: <4428516E.4070308@colorado.edu> Message-ID: <44285970.5080008@colorado.edu> Charles R Harris wrote: > What purpose does ascontiguous serve? One I can think of: making sure that C code of the type double foo(double *array,long num_elements) { ... } can be called in C as foo((double *)numpy_arr->data,numpy_arr->dimensions[0]); (the dims name in C may be different, I'm not looking at code right now) and be OK with walking the given memory region as a contiguous chunk of doubles, num_elements of them. As long as foo() doesn't modify its input, this works fine and it's nice to have such guarantees: it makes writing extension code a lot easier. Cheers, f From oliphant at ee.byu.edu Mon Mar 27 13:43:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 13:43:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428484F.2030604@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> Message-ID: <44285C3D.9080802@ee.byu.edu> Tim Hochberg wrote: >> >> Yes, having this ability means that you have to think about it a bit >> if you are going to use the functional interface and try to do >> in-place operations. But, I would argue that this is an "advanced" >> usage which is implemented to save space and time. > > > How is this true though? In what way, for instance, is: > > b = asarray(a).reshape(newshape) > > slower or less space efficient than todays: > > b = reshape(a) Well, the big difference is that b=reshape(a) is actually try: reshape = a.reshape except AttributeError: return a.__array_wrap__(asarray(a).reshape(newshape)) return reshape(newshape) So, it handles more cases, more cleanly then the asarray(a).rehsape(newshape) approach does. -Travis From chanley at stsci.edu Mon Mar 27 13:48:08 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Mon Mar 27 13:48:08 2006 Subject: [Numpy-discussion] problem with ndarray attributes Message-ID: <44285D68.1060908@stsci.edu> Hi Travis, I have run across a feature in numpy which I am not sure how to handle. One of the new features of the ndarray is that named fields can be accessed as attributes. So, if a user created a recarray with fields named "address", "phone", and "names" they would be able to access all of the names in the recarray objects with "recarrayObject.names". I have found that this can cause problems when a user creates a table with a field name that happen to coincide with an existing class attribute or method. In particular, I am having a problem in pyfits where a user has a table with a column named "field". This column name is causing an attribute "field" to be created that overrides the existing class field method. It would seem to me that existing class methods and attributes should take precedence over the attributes created to correspond to column names. Is there a way to implement this behavior in ndarray? Thank you for your time and help, Chris From oliphant at ee.byu.edu Mon Mar 27 14:03:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 14:03:05 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <44285D68.1060908@stsci.edu> References: <44285D68.1060908@stsci.edu> Message-ID: <4428610B.4040307@ee.byu.edu> Christopher Hanley wrote: > Hi Travis, > > I have run across a feature in numpy which I am not sure how to > handle. One of the new features of the ndarray is that named fields > can be accessed as attributes. So, if a user created a recarray with > fields named "address", "phone", and "names" they would be able to > access all of the names in the recarray objects with > "recarrayObject.names". To be clear. This is only true for a recarray subclass. Standard ndarray's cannot look-up fields with attribute access. > > I have found that this can cause problems when a user creates a table > with a field name that happen to coincide with an existing class > attribute or method. In particular, I am having a problem in pyfits > where a user has a table with a column named "field". This column > name is causing an attribute "field" to be created that overrides the > existing class field method. This is a standard problem with using attribute access. I suppose it would be better to always lookup the attributes first on the object and then if that fails use field access. That way library code would not break if a user happened to name their fields after a method or property. Users would have to change code if new methods or properties were added (although that won't be as common). > > It would seem to me that existing class methods and attributes should > take precedence over the attributes created to correspond to column > names. Is there a way to implement this behavior in ndarray? I think this is the right thing to do. Are there any objections? -Travis From tim.hochberg at cox.net Mon Mar 27 14:09:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:09:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44285C3D.9080802@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> Message-ID: <44286266.9050503@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> >>> Yes, having this ability means that you have to think about it a bit >>> if you are going to use the functional interface and try to do >>> in-place operations. But, I would argue that this is an "advanced" >>> usage which is implemented to save space and time. >> >> >> >> How is this true though? In what way, for instance, is: >> >> b = asarray(a).reshape(newshape) >> >> slower or less space efficient than todays: >> >> b = reshape(a) > > > > Well, the big difference is that b=reshape(a) is actually > > try: > reshape = a.reshape > except AttributeError: > return a.__array_wrap__(asarray(a).reshape(newshape)) > > return reshape(newshape) > > So, it handles more cases, more cleanly then the > asarray(a).rehsape(newshape) approach does. OK. Although I should have said asanyarray(a).reshape(newshape), I still see how that this handles more cases. I do not agree that it is worth the attendant drawbacks and I would still like to see these functions spun off into a separate namespace. I need to go think about this case some more. This is not something that I've run into in practice, but I can see I'll have a stronger case if I can come up with a alternative to this in terms of safe functions. Do you have some examples of objects that do not have 'reshape', but do have '__array_wrap__'? -tim From perry at stsci.edu Mon Mar 27 14:11:05 2006 From: perry at stsci.edu (Perry Greenfield) Date: Mon Mar 27 14:11:05 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <4428610B.4040307@ee.byu.edu> References: <44285D68.1060908@stsci.edu> <4428610B.4040307@ee.byu.edu> Message-ID: <1e7e648a95a431db8c5a3844781d3211@stsci.edu> On Mar 27, 2006, at 5:02 PM, Travis Oliphant wrote: > > This is a standard problem with using attribute access. I suppose it > would be better to always lookup the attributes first on the object > and then if that fails use field access. That way library code would > not break if a user happened to name their fields after a method or > property. Users would have to change code if new methods or > properties were added (although that won't be as common). I think some mechanism must be built in to make it easy to add new attributes and methods (e.g., some method to append new object attributes that are protected?). That way subclassing record arrays wouldn't have to do their own checks on this but rather register their new methods and attributes as things that would not get overridden (or is there some more magic way of doing this that doesn't require something this explicit?) >> >> It would seem to me that existing class methods and attributes should >> take precedence over the attributes created to correspond to column >> names. Is there a way to implement this behavior in ndarray? > > > I think this is the right thing to do. Are there any objections? > Seems like the obvious thing. Otherwise any tool one builds that allows a user to use arbitrary field names can suddenly break if a user uses the wrong name. With the proposed change, there is always the field method to get around any name collisions. Perry From oliphant at ee.byu.edu Mon Mar 27 14:15:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 14:15:00 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <44285D68.1060908@stsci.edu> References: <44285D68.1060908@stsci.edu> Message-ID: <442863A7.1080803@ee.byu.edu> Christopher Hanley wrote: > Hi Travis, > > I have run across a feature in numpy which I am not sure how to > handle. One of the new features of the ndarray is that named fields > can be accessed as attributes. So, if a user created a recarray with > fields named "address", "phone", and "names" they would be able to > access all of the names in the recarray objects with > "recarrayObject.names". > > I have found that this can cause problems when a user creates a table > with a field name that happen to coincide with an existing class > attribute or method. In particular, I am having a problem in pyfits > where a user has a table with a column named "field". This column > name is causing an attribute "field" to be created that overrides the > existing class field method. > > It would seem to me that existing class methods and attributes should > take precedence over the attributes created to correspond to column > names. Is there a way to implement this behavior in ndarray? > Now that I've written that I think the class methods and attributes should come first, I should mention, however, that a library writer could write ndarray.__getattribute__(a,'field')(*args) to make sure that the method a.field(*args) always gets called. This seems like quite a burden to force on library writers, though. I'd rather have users be aware of what they are using as field names since the methods and attributes on arrays should not be changing excessively. -Travis From charlesr.harris at gmail.com Mon Mar 27 14:16:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 14:16:02 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <44285970.5080008@colorado.edu> References: <4428516E.4070308@colorado.edu> <44285970.5080008@colorado.edu> Message-ID: On 3/27/06, Fernando Perez wrote: > > Charles R Harris wrote: > > > What purpose does ascontiguous serve? > > One I can think of: making sure that C code of the type > > double foo(double *array,long num_elements) { ... } > > can be called in C as > > foo((double *)numpy_arr->data,numpy_arr->dimensions[0]); > > (the dims name in C may be different, I'm not looking at code right now) > > and be OK with walking the given memory region as a contiguous chunk of > doubles, num_elements of them. > > As long as foo() doesn't modify its input, this works fine and it's nice > to > have such guarantees: it makes writing extension code a lot easier. > OK, it's an interface. Since getting stuff back from C will require a copy if the original is not contiquous, a test involving a.iscontiguous will be needed and ascontiguous can be dispensed with. Isn't there a C version of this anyway? ISTR that numarray had one. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 14:25:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 14:25:03 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> Message-ID: <4428660A.2020607@ee.byu.edu> Zachary Pincus wrote: >> The reason for the order flag is that it is needed to interpret what >> you think a reshape operation should do. How should the reshape >> method think the data is stored. That is the intended meaning of >> the reshape flag. >> The FORTRAN flag on the array means the strides are actually >> equivalent to FORTRAN order >> The CONTIGUOUS flag on the array means the strides are actually >> equilvalent to C-CONTIGUOUS order. >> [...] >> To do what Tim suggests would require the addition of an additional >> flag (call it the ORDER flag) on the array that would be carried >> around to tell everybody how to think about this array as a one- >> dimensional sequence of bytes. >> Such an approach is possible, of course. > > > > Does this mean that if I create a new array with FORTRAN order from > the numpy.array() function, every time I reshape that array I need to > tell reshape to use FORTRAN order? That seems a bit odd... Yes, that is what it means. Doing anything else leads to a host of problems. Let me illustrate. a = rand(10,3) b = a.transpose() b.flags # b is now in Fortran order b.ravel() # do you really want this to now be interpreted in FORTRAN-order. I didn't think so. What this illustrates is that the FORTRAN and CONTIGUOUS flags on the array are just special cases of the strides (and in fact can be updated from the strides at any time). But, an ORDER flag would be an independent concept that could be set and reset at will. The ORDER flag is only necessary when somebody is interpreting the array as a linear sequence of bytes. If the ORDER flag is added, then the Array iterator would also need to be updated, to recognize and respect the order flag. Right now, the array iterator (a.flat) is always C-contiguous. > > Yet the text above seems to suggests that arrays will not know what > order their own bytes are in, and thus would need to be told whether > they should be reshaped like FORTRAN or C memory blocks. > > I assume I'm misinterpreting something here? It wasn't until a few weeks ago that I think I really understood the difference between ORDER='FORTRAN' and flags & FORTRAN == True. The order information is really an additional but related piece of information . The example above illustrates how you could have flags & FORTRAN == True but order == 'C' -Travis From tim.hochberg at cox.net Mon Mar 27 14:45:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:45:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> Message-ID: <44286AAE.3030204@cox.net> Charles R Harris wrote: > > > On 3/27/06, *Tim Hochberg* > wrote: > > Charles R Harris wrote: > [CHOP] > > > > > How about functions always return a copy unless they are carefully > > documented interface helper functions like asarray. That's how I > > generally think of them anyway. > > I don't really care as long as they (a) stop being evil and (b) > there's > some way to do stuff efficiently. Functions that always return copies > seem sort of useless to me, but I'd be happy to move to methods if > that > was the case. However, I suspect you won't make much headway with this > since my impression is that many of the people who do care about > functions also care passionately about efficiency as well. > > > Of course, the main virtue of asarray is that it helps write > functions > > that do precisely what Tim is arguing against: mixing list and array > > types and returning copies in the first case, views in the > second. So > > if we throw out functions we should throw out asarray also and > make a > > rule that numpy functions don't take lists. > > This isn't right. I think it's perfectly fine, desirable in fact, for > functions to operate on both list and array types. The problem only > arises when you return ether a view or copy of one of the original > inputs depending on what it was. In my experience, this is relatively > hard to do in all but the most trivial of functions since most > operations return a new array. However, it is true that I think people > should be strongly discouraged from doing this. For example: > > def add(a, b): > a = asarray(a) > b = asarray(b) > return a + b > > is fine, but: > > def iadd(a, b): > a = asarray(a) > b = asarray(b) > a += b > return a > > is not and should be rewritten. Possibly like: > > def iadd(a, b): > if not isinstance(a, ndarray): raise TypeError("'a' must be an > array") > b = asarray(b) > a += b > return a > > since += is a bit squirley. (Try somelist += somearray, to see > what I mean). > > > It's a syntax error, which I believe is the right thing. Really? If so, this is a version thing. This is what I get running on Python 2.4.1 and current numpy SVN (0.9.7.2286): >>> l = list(a) >>> l [999, 1, 2, 3, 4, 5, 6, 7, 8] >>> a array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>> l += a >>> l array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>> a array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > And then there are scalars... > > I'm curious what problems you forsee with scalars. > > > There was/is the whole problem of when to return scalars and if they > should be python scalars. The problem arises from trying to be > compatible with normal python stuff. Matlab got around this by making > everything 1) an array, and 2) double precision. On the other hand, > Matlab is not nearly is easy to use for nonstandard types. \ I think most of this problem will go away in Python with the introduction of the __index__ protocol, so for the moment at least I'm not worried about it. Regards, -tim From tim.hochberg at cox.net Mon Mar 27 14:51:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:51:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286AAE.3030204@cox.net> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: <44286C1C.5010605@cox.net> Tim Hochberg wrote: > Charles R Harris wrote: > >> >> >> On 3/27/06, *Tim Hochberg* > > wrote: >> >> Charles R Harris wrote: >> [CHOP] >> >> > >> > How about functions always return a copy unless they are carefully >> > documented interface helper functions like asarray. That's how I >> > generally think of them anyway. >> >> I don't really care as long as they (a) stop being evil and (b) >> there's >> some way to do stuff efficiently. Functions that always return >> copies >> seem sort of useless to me, but I'd be happy to move to methods if >> that >> was the case. However, I suspect you won't make much headway with >> this >> since my impression is that many of the people who do care about >> functions also care passionately about efficiency as well. >> >> > Of course, the main virtue of asarray is that it helps write >> functions >> > that do precisely what Tim is arguing against: mixing list and >> array >> > types and returning copies in the first case, views in the >> second. So >> > if we throw out functions we should throw out asarray also and >> make a >> > rule that numpy functions don't take lists. >> >> This isn't right. I think it's perfectly fine, desirable in fact, >> for >> functions to operate on both list and array types. The problem only >> arises when you return ether a view or copy of one of the original >> inputs depending on what it was. In my experience, this is >> relatively >> hard to do in all but the most trivial of functions since most >> operations return a new array. However, it is true that I think >> people >> should be strongly discouraged from doing this. For example: >> >> def add(a, b): >> a = asarray(a) >> b = asarray(b) >> return a + b >> >> is fine, but: >> >> def iadd(a, b): >> a = asarray(a) >> b = asarray(b) >> a += b >> return a >> >> is not and should be rewritten. Possibly like: >> >> def iadd(a, b): >> if not isinstance(a, ndarray): raise TypeError("'a' must >> be an >> array") >> b = asarray(b) >> a += b >> return a >> >> since += is a bit squirley. (Try somelist += somearray, to see >> what I mean). >> >> >> It's a syntax error, which I believe is the right thing. > > > Really? If so, this is a version thing. This is what I get running on > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > >>> l = list(a) > >>> l > [999, 1, 2, 3, 4, 5, 6, 7, 8] > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > >>> l += a > >>> l > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) Let me add that I think that this is pretty dubious, so if this is a new feature, perhaps we should revert it before it becomes entrenched. Regards, -tim > >> >> > And then there are scalars... >> >> I'm curious what problems you forsee with scalars. >> >> There was/is the whole problem of when to return scalars and if they >> should be python scalars. The problem arises from trying to be >> compatible with normal python stuff. Matlab got around this by making >> everything 1) an array, and 2) double precision. On the other hand, >> Matlab is not nearly is easy to use for nonstandard types. \ > > > I think most of this problem will go away in Python with the > introduction of the __index__ protocol, so for the moment at least I'm > not worried about it. > From tim.hochberg at cox.net Mon Mar 27 14:54:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 14:54:02 2006 Subject: [Numpy-discussion] arrays and iterators. Message-ID: <44286CCB.3030704@cox.net> Lest everybody think that I'm only about ripping things out, let me propose adding a feature. Well, sort of. I propose that we make: array(someiterator) an value error for right now instead of returning a rank-0 object array. Then, in Python 2.5, we can use the _length_hint protocol to efficiently create arrays from iterator objects without any extra copying. The exact details don't need to be worked out right now, after all we're all too tied up with reshape, but I'd like to reserve that syntax for that purpose. Regards, -tim From webb.sprague at gmail.com Mon Mar 27 15:07:10 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Mon Mar 27 15:07:10 2006 Subject: [Numpy-discussion] Bug in scipy.linalg.svd? Message-ID: Hi all, scipy.__version__: 0.3.2 numpy.__version__: 0.9.4 OS: Gentoo. Maybe I am missing something key here, but my Numerical Recipes book says: "Any M x N matrix A whose number of rows M is greater than or equal to its number of columns N, can be written as the product of of an *M x N* column orthogonal matrix U, an N x N diagonal matrix W, ... and the transpose of an N x N orthogh matrix V." So... why do I get the following behavior: (u, x, v) = scipy.linalg.svd(RandomArray.normal(0,1,[10,4]) u.shape == (10,10) By the way, if you run this from Numeric, U[:, 0:4] are the same, but Numeric doesn't generate the additional columns. I think we should write a unit test such that matrixmultipy(matrixmultiply(u,diag(x)), transpose(v)) == A. From robert.kern at gmail.com Mon Mar 27 15:20:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 27 15:20:06 2006 Subject: [Numpy-discussion] Re: Bug in scipy.linalg.svd? In-Reply-To: References: Message-ID: Webb Sprague wrote: > Hi all, > > scipy.__version__: 0.3.2 > > numpy.__version__: 0.9.4 > > OS: Gentoo. > > Maybe I am missing something key here, but my Numerical Recipes book says: > > "Any M x N matrix A whose number of rows M is greater than or equal to > its number of columns N, can be written as the product of of an *M x > N* column orthogonal matrix U, an N x N diagonal matrix W, ... and the > transpose of an N x N orthogh matrix V." This is only a convention. It is frequently called computing the "thin SVD." Another convention is to return the fully expanded U matrix. scipy.linalg.svd does the latter. -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From michael.sorich at gmail.com Mon Mar 27 15:37:12 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Mon Mar 27 15:37:12 2006 Subject: [Numpy-discussion] making a character array from numeric array Message-ID: <16761e100603271536j79da6963gc4eadf889a8d3c43@mail.gmail.com> Hi, I am trying to convert an int or float ndarray into a string ndarray. Using astype this is possible, but only if you explicity set the number of characters (see eg below). It would be nice if just setting type='S' (or N.str_) would automatically make an a character array of sufficient size to hold the string representation. Is there a way to do this? thanks, michael import numpy as N N.version.version '0.9.6' N.arange(20).astype(N.str_) array([, , , , , , , , , , , , , , , , , , , ], dtype='|S0') N.arange(20).astype('S') array([, , , , , , , , , , , , , , , , , , , ], dtype='|S0') N.arange(20).astype('S1') array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype='|S1') N.arange(20).astype('S2') array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], dtype='|S2') -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 15:41:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 15:41:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286C1C.5010605@cox.net> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> Message-ID: <442877DF.4050609@ee.byu.edu> Tim Hochberg wrote: >> Charles R Harris wrote: >> >> >>> l = list(a) >> >>> l >> [999, 1, 2, 3, 4, 5, 6, 7, 8] >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >>> l += a >> >>> l >> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > > Let me add that I think that this is pretty dubious, so if this is a > new feature, perhaps we should revert it before it becomes entrenched. I don't think it's a new feature, but it's simply the result of l += a being translated to l = l + a # lists don't have in-place add's Numeric has this behavior as well. -Travis From charlesr.harris at gmail.com Mon Mar 27 15:47:05 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 15:47:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286AAE.3030204@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: On 3/27/06, Tim Hochberg wrote: > > Charles R Harris wrote: > > > > > > > On 3/27/06, *Tim Hochberg* > > wrote: > > > > Charles R Harris wrote: > > [CHOP] > > > > > > > > How about functions always return a copy unless they are carefully > > > documented interface helper functions like asarray. That's how I > > > generally think of them anyway. > > > > I don't really care as long as they (a) stop being evil and (b) > > there's > > some way to do stuff efficiently. Functions that always return > copies > > seem sort of useless to me, but I'd be happy to move to methods if > > that > > was the case. However, I suspect you won't make much headway with > this > > since my impression is that many of the people who do care about > > functions also care passionately about efficiency as well. > > > > > Of course, the main virtue of asarray is that it helps write > > functions > > > that do precisely what Tim is arguing against: mixing list and > array > > > types and returning copies in the first case, views in the > > second. So > > > if we throw out functions we should throw out asarray also and > > make a > > > rule that numpy functions don't take lists. > > > > This isn't right. I think it's perfectly fine, desirable in fact, > for > > functions to operate on both list and array types. The problem only > > arises when you return ether a view or copy of one of the original > > inputs depending on what it was. In my experience, this is > relatively > > hard to do in all but the most trivial of functions since most > > operations return a new array. However, it is true that I think > people > > should be strongly discouraged from doing this. For example: > > > > def add(a, b): > > a = asarray(a) > > b = asarray(b) > > return a + b > > > > is fine, but: > > > > def iadd(a, b): > > a = asarray(a) > > b = asarray(b) > > a += b > > return a > > > > is not and should be rewritten. Possibly like: > > > > def iadd(a, b): > > if not isinstance(a, ndarray): raise TypeError("'a' must be > an > > array") > > b = asarray(b) > > a += b > > return a > > > > since += is a bit squirley. (Try somelist += somearray, to see > > what I mean). > > > > > > It's a syntax error, which I believe is the right thing. Really? If so, this is a version thing. This is what I get running on > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > >>> l = list(a) > >>> l > [999, 1, 2, 3, 4, 5, 6, 7, 8] > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > >>> l += a > >>> l > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) Ugh. IMHO, this should throw an error. For these sort of assignment operators I think the right hand side should be cast to the type of the left, which I think of as some sort of fancy register variable. If anything, the list should be appended to. I'll have to try the latest SVN just to appreciate this. > -tim > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 15:48:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 15:48:05 2006 Subject: [Numpy-discussion] making a character array from numeric array In-Reply-To: <16761e100603271536j79da6963gc4eadf889a8d3c43@mail.gmail.com> References: <16761e100603271536j79da6963gc4eadf889a8d3c43@mail.gmail.com> Message-ID: <44287987.90507@ee.byu.edu> Michael Sorich wrote: > Hi, > > I am trying to convert an int or float ndarray into a string ndarray. > Using astype this is possible, but only if you explicity set the > number of characters (see eg below). It would be nice if just setting > type='S' (or N.str_) would automatically make an a character array of > sufficient size to hold the string representation. Is there a way to > do this? Not with astype directly because to do what you want requires doing the conversion into variable-length strings and then finding the largest. But, you could do it this way: def convert_to_string(arr): fl = arr.flat l = [str(x) for x in fl] return array(l).reshape(arr.shape) Or as a one-liner array([str(x) for x in arr.flat], dtype='S').reshape(arr.shape) (The dtype='S' is technically unnecessary but should be a bit faster) -Travis From robert.kern at gmail.com Mon Mar 27 15:52:06 2006 From: robert.kern at gmail.com (Robert Kern) Date: Mon Mar 27 15:52:06 2006 Subject: [Numpy-discussion] Re: Ransom Proposals In-Reply-To: <442877DF.4050609@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> <442877DF.4050609@ee.byu.edu> Message-ID: Travis Oliphant wrote: > Tim Hochberg wrote: > >>> Charles R Harris wrote: >>> >>> >>> l = list(a) >>> >>> l >>> [999, 1, 2, 3, 4, 5, 6, 7, 8] >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>> >>> l += a >>> >>> l >>> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >> Let me add that I think that this is pretty dubious, so if this is a >> new feature, perhaps we should revert it before it becomes entrenched. > > I don't think it's a new feature, but it's simply the result of > > l += a being translated to > > l = l + a # lists don't have in-place add's > > Numeric has this behavior as well. Lists do have in-place adds. In [1]: a = range(10) In [2]: b = range(10, 20) In [3]: id(a) Out[3]: 92350264 In [4]: a += b In [5]: id(a) Out[5]: 92350264 In [6]: a Out[6]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Mar 27 16:00:08 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 16:00:08 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: Hmm, On 3/27/06, Charles R Harris wrote: > > > > On 3/27/06, Tim Hochberg wrote: > > > > Charles R Harris wrote: > > > > > > > > > > > On 3/27/06, *Tim Hochberg* > > > wrote: > > > > > > Charles R Harris wrote: > > > [CHOP] > > > > > > > > > > > How about functions always return a copy unless they are > > carefully > > > > documented interface helper functions like asarray. That's how I > > > > > > generally think of them anyway. > > > > > > I don't really care as long as they (a) stop being evil and (b) > > > there's > > > some way to do stuff efficiently. Functions that always return > > copies > > > seem sort of useless to me, but I'd be happy to move to methods if > > > that > > > was the case. However, I suspect you won't make much headway with > > this > > > since my impression is that many of the people who do care about > > > functions also care passionately about efficiency as well. > > > > > > > Of course, the main virtue of asarray is that it helps write > > > functions > > > > that do precisely what Tim is arguing against: mixing list and > > array > > > > types and returning copies in the first case, views in the > > > second. So > > > > if we throw out functions we should throw out asarray also and > > > make a > > > > rule that numpy functions don't take lists. > > > > > > This isn't right. I think it's perfectly fine, desirable in fact, > > for > > > functions to operate on both list and array types. The problem > > only > > > arises when you return ether a view or copy of one of the original > > > > > inputs depending on what it was. In my experience, this is > > relatively > > > hard to do in all but the most trivial of functions since most > > > operations return a new array. However, it is true that I think > > people > > > should be strongly discouraged from doing this. For example: > > > > > > def add(a, b): > > > a = asarray(a) > > > b = asarray(b) > > > return a + b > > > > > > is fine, but: > > > > > > def iadd(a, b): > > > a = asarray(a) > > > b = asarray(b) > > > a += b > > > return a > > > > > > is not and should be rewritten. Possibly like: > > > > > > def iadd(a, b): > > > if not isinstance(a, ndarray): raise TypeError("'a' must > > be an > > > array") > > > b = asarray(b) > > > a += b > > > return a > > > > > > since += is a bit squirley. (Try somelist += somearray, to see > > > what I mean). > > > > > > > > > It's a syntax error, which I believe is the right thing. > > > > > Really? If so, this is a version thing. This is what I get running on > > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > > > >>> l = list(a) > > >>> l > > [999, 1, 2, 3, 4, 5, 6, 7, 8] > > >>> a > > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > >>> l += a > > >>> l > > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > > >>> a > > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > Ugh. IMHO, this should throw an error. For these sort of assignment > operators I think the right hand side should be cast to the type of the > left, which I think of as some sort of fancy register variable. If anything, > the list should be appended to. I'll have to try the latest SVN just to > appreciate this. > > It was pilot error. On the other hand: >>> b = [1,2,3] >>> b += b >>> b [1, 2, 3, 1, 2, 3] Which makes sense from the list perspective. From what Travis says, += is actually a phony in place operator, so if [1,2,3] + array([1,2,3]) is supported and returns an array, the result is a given. This is what comes from mixing types and trying to treat them all as interconvertable. Lists aren't arrays, they are just a convenient way to specify the array entries for initialization. I think in these cases one should be forced to be explicit. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:05:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:05:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44286266.9050503@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> Message-ID: <44287D7D.8010604@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>>> >>>> Yes, having this ability means that you have to think about it a >>>> bit if you are going to use the functional interface and try to do >>>> in-place operations. But, I would argue that this is an "advanced" >>>> usage which is implemented to save space and time. >>> >>> >>> >>> >>> How is this true though? In what way, for instance, is: >>> >>> b = asarray(a).reshape(newshape) >>> >>> slower or less space efficient than todays: >>> >>> b = reshape(a) >> >> >> >> >> Well, the big difference is that b=reshape(a) is actually >> >> try: >> reshape = a.reshape >> except AttributeError: >> return a.__array_wrap__(asarray(a).reshape(newshape)) >> >> return reshape(newshape) >> >> So, it handles more cases, more cleanly then the >> asarray(a).rehsape(newshape) approach does. > > > OK. Although I should have said asanyarray(a).reshape(newshape), I > still see how that this handles more cases. Because you can define objects that aren't sub-classes of the array at all but just define the method reshape and still have the functional interface work. > > I need to go think about this case some more. This is not something > that I've run into in practice, but I can see I'll have a stronger > case if I can come up with a alternative to this in terms of safe > functions. Do you have some examples of objects that do not have > 'reshape', but do have '__array_wrap__'? Not presently, the __array_wrap__ method is new. It's a mechanism for letting functions that need arrays internally to be more polymorphic and deal with different kinds of objects. It would seem that most of Tim's complaints are directly against polymorphism. I would argue that getting rid of polymorphism is not something we should be trying to do in NumPy. Tim is pointing out that if you use polymorphic functions then you can't assume things like "in-place" have any meaning. But, then Python has this same problem, because l += a doesn't do 'in-place' for the list, but does do inplace if 'l' were an array. In my mind, this problem and Tim's concern over 'view' and 'copy' behavior is one and the same. I don't view it as a problem of function behavior as much as problem with documentation and "mismatch between what a particular user expects and what is actually done." My attitude is that to write 'in-place' functions (what seems to be the driving example that Tim brings up), you need to know some details about what kind of object you are dealing with anyway, so you can't write polymorphic in-place functions very easily. So, I guess from one perspective, Tim is arguing against things that are at the very core of Python itself. I'm very resistant to his desire to remove or significantly alter the functional behavior. I also think there is a very good reason to have a few methods that return either a view or a copy. The reason is that there are some arrays that the method can't be implemented unless a copy is made. The only other possibility --- "raising an error" --- is going to confuse a lot of new users and force them to deal with the underlying memory-layout of an array before they really need to. I think the current compromise is practical and disagree strongly that it is somehow "evil." It's only evil to somebody trying to do in-place work. If you are doing that in a polymorphic language like Python, then you need to understand the actual objects you are dealing with and should be expected to have a more mature understanding of NumPy. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:07:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:07:01 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> Message-ID: <44287DFC.6050700@ee.byu.edu> Charles R Harris wrote: > Really? If so, this is a version thing. This is what I get running on > Python 2.4.1 and current numpy SVN (0.9.7.2286): > > >>> l = list(a) > >>> l > [999, 1, 2, 3, 4, 5, 6, 7, 8] > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > >>> l += a > >>> l > array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) > >>> a > array([999, 1, 2, 3, 4, 5, 6, 7, 8]) > > > Ugh. IMHO, this should throw an error. For these sort of assignment > operators I think the right hand side should be cast to the type of > the left, which I think of as some sort of fancy register variable. If > anything, the list should be appended to. I'll have to try the latest > SVN just to appreciate this. > Again, this is a Python thing. Numeric has this behavior as well. Depending on your point of view it's either a great thing or a wart. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:10:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:10:04 2006 Subject: [Numpy-discussion] arrays and iterators. In-Reply-To: <44287C07.6010805@cox.net> References: <44286CCB.3030704@cox.net> <44287805.5040600@ee.byu.edu> <44287C07.6010805@cox.net> Message-ID: <44287ECE.80602@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>> >>> Lest everybody think that I'm only about ripping things out, let me >>> propose adding a feature. Well, sort of. I propose that we make: >>> >>> array(someiterator) >>> >>> an value error for right now instead of returning a rank-0 object >>> array. >> >> >> >> I'm O.K. with that. >> -Travis > > > OK, if no one else objects in the next day or so, I'll look into doing > it. Just to be clear. You are only going to change the case which does presently return a rank-0 object array, right? someiterator could be an array, or a list in which case array(someiterator) already does an intelligible thing. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:14:00 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:14:00 2006 Subject: [Numpy-discussion] arrays and iterators. In-Reply-To: <44287C07.6010805@cox.net> References: <44286CCB.3030704@cox.net> <44287805.5040600@ee.byu.edu> <44287C07.6010805@cox.net> Message-ID: <44287F8F.7090502@ee.byu.edu> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Tim Hochberg wrote: >> >>> >>> Lest everybody think that I'm only about ripping things out, let me >>> propose adding a feature. Well, sort of. I propose that we make: >>> >>> array(someiterator) >>> >>> an value error for right now instead of returning a rank-0 object >>> array. >> >> >> >> I'm O.K. with that. >> -Travis > > > OK, if no one else objects in the next day or so, I'll look into doing > it. You might also look at having array(generator object) actually run the generator instead of return a rank-0 object array. -Travis From michael.sorich at gmail.com Mon Mar 27 16:17:13 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Mon Mar 27 16:17:13 2006 Subject: [Numpy-discussion] precedence of '&' and '>' operators Message-ID: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> It seems that the '&' operator has a higher precedence than a comparison operator such as '>'. This does not seem so intuitive to me. Is this correct? eg >>>import numpy as N >>>N.version.version '0.9.6' >>>a = N.arange(10) >>>a>1 & a<4 Traceback (most recent call last): File "", line 1, in ? ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() >>>(a>1) & (a<4) array([False, False, True, True, False, False, False, False, False, False], dtype=bool) Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:24:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:24:02 2006 Subject: [Numpy-discussion] precedence of '&' and '>' operators In-Reply-To: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> References: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> Message-ID: <442881FD.7010005@ee.byu.edu> Michael Sorich wrote: > It seems that the '&' operator has a higher precedence than a > comparison operator such as '>'. This does not seem so intuitive to > me. Is this correct? Yes, this is correct. See http://docs.python.org/ref/summary.html The problem really is that "and" (which has lower precedence) does not allow being over-written. So, we must use the bit-wise operator. -Travis From oliphant at ee.byu.edu Mon Mar 27 16:29:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:29:06 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: References: <4428516E.4070308@colorado.edu> <44285970.5080008@colorado.edu> Message-ID: <44288339.5090507@ee.byu.edu> Charles R Harris wrote: > > > On 3/27/06, *Fernando Perez* > wrote: > > Charles R Harris wrote: > > > What purpose does ascontiguous serve? > > One I can think of: making sure that C code of the type > > double foo(double *array,long num_elements) { ... } > > can be called in C as > > foo((double *)numpy_arr->data,numpy_arr->dimensions[0]); > > (the dims name in C may be different, I'm not looking at code > right now) > > and be OK with walking the given memory region as a contiguous > chunk of > doubles, num_elements of them. > > As long as foo() doesn't modify its input, this works fine and > it's nice to > have such guarantees: it makes writing extension code a lot easier. > > > OK, it's an interface. Since getting stuff back from C will require a > copy if the original is not contiquous, a test involving > a.iscontiguous will be needed and ascontiguous can be dispensed with. > Isn't there a C version of this anyway? ISTR that numarray had one. > It think you are talking about the UPDATEIFCOPY requirements flag on the PyArray_FromAny C-API. Most aren't using this yet (and it's not directly exposed to Python). But it's how you would get a copy of the array with the correct requirements, operate on it and then once you delete your reference have the information go back into the original array (this is true whether or not the array is contiguous). The original array is also set to be read-only during the time that the copy is available so that you can't make changes to it (that are going to be lost). Perhaps what we need to do is make this scheme available to the Python user as well. -Travis From michael.sorich at gmail.com Mon Mar 27 16:31:03 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Mon Mar 27 16:31:03 2006 Subject: [Numpy-discussion] using boolean arrays as indices Message-ID: <16761e100603271630v6a46dbabq686ef548bc82aadf@mail.gmail.com> If I have a 2D array and I want to use a boolean array to select the rows. If I use a[boolarray] it works, but if I use a[boolarray, :] an exception is raised. Is there a reason for this? eg >>>N.__version__ '0.9.6' >>>a = N.arange(10).reshape((2,5)) >>>a array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) >>>a[N.array([True, False])] array([[0, 1, 2, 3, 4]]) >>>a[N.array([True, False]),:] Traceback (most recent call last): File "", line 1, in ? IndexError: arrays used as indices must be of integer type -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:37:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:37:01 2006 Subject: [Numpy-discussion] Re: Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> <442877DF.4050609@ee.byu.edu> Message-ID: <442884F3.7060807@ee.byu.edu> Robert Kern wrote: >Travis Oliphant wrote: > > >>Tim Hochberg wrote: >> >> >> >>>>Charles R Harris wrote: >>>> >>>> >>> l = list(a) >>>> >>> l >>>> [999, 1, 2, 3, 4, 5, 6, 7, 8] >>>> >>> a >>>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>>> >>> l += a >>>> >>> l >>>> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>>> >>> a >>>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>>> >>>> >>>Let me add that I think that this is pretty dubious, so if this is a >>>new feature, perhaps we should revert it before it becomes entrenched. >>> >>> >>I don't think it's a new feature, but it's simply the result of >> >>l += a being translated to >> >>l = l + a # lists don't have in-place add's >> >>Numeric has this behavior as well. >> >> > >Lists do have in-place adds. > >In [1]: a = range(10) > >In [2]: b = range(10, 20) > >In [3]: id(a) >Out[3]: 92350264 > >In [4]: a += b > >In [5]: id(a) >Out[5]: 92350264 > >In [6]: a >Out[6]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] > > > True, true. So, why this behavior? This seems like a Python issue. Compare a = range(10) b = arange(10,20) a.__iadd__(b) with a += b Shouldn't these be equivalent? -Travis From tim.leslie at gmail.com Mon Mar 27 16:46:04 2006 From: tim.leslie at gmail.com (Tim Leslie) Date: Mon Mar 27 16:46:04 2006 Subject: [Numpy-discussion] precedence of '&' and '>' operators In-Reply-To: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> References: <16761e100603271616r2d4801f2ga421056b3be9de35@mail.gmail.com> Message-ID: On 3/28/06, Michael Sorich wrote: > > It seems that the '&' operator has a higher precedence than a comparison > operator such as '>'. This does not seem so intuitive to me. Is this > correct? > Yep. http://docs.python.org/ref/summary.html Cheers, Timl eg > >>>import numpy as N > >>>N.version.version > '0.9.6' > >>>a = N.arange(10) > >>>a>1 & a<4 > Traceback (most recent call last): > File "", line 1, in ? > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() > >>>(a>1) & (a<4) > array([False, False, True, True, False, False, False, False, False, > False], dtype=bool) > > Michael > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Mar 27 16:57:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Mar 27 16:57:02 2006 Subject: [Numpy-discussion] using boolean arrays as indices In-Reply-To: <16761e100603271630v6a46dbabq686ef548bc82aadf@mail.gmail.com> References: <16761e100603271630v6a46dbabq686ef548bc82aadf@mail.gmail.com> Message-ID: <442889AB.9020508@ee.byu.edu> Michael Sorich wrote: > If I have a 2D array and I want to use a boolean array to select the > rows. If I use a[boolarray] it works, but if I use a[boolarray, :] an > exception is raised. Is there a reason for this? Yes. Currently, you can only use Boolean indexing using X[obj] which is equivalent to (but faster than) X[obj.nonzero()] There is no way currently to use Boolean indexing on a single axis. A useful specification would need to be made that is consistent with the current definition along with an implementation. You could use instead a.compress(condition, axis=0) This selects out sub-arrays along a particular axis. -Travis From zpincus at stanford.edu Mon Mar 27 17:14:18 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Mon Mar 27 17:14:18 2006 Subject: [Numpy-discussion] order flag again In-Reply-To: <4428660A.2020607@ee.byu.edu> References: <442837E4.8020303@cox.net> <4428418A.1070606@ee.byu.edu> <78F59A1C-C059-4034-AA93-F75A95283ED7@stanford.edu> <4428660A.2020607@ee.byu.edu> Message-ID: <3964FC31-4753-4E3A-B69C-9A6CD88C9202@stanford.edu> >> Does this mean that if I create a new array with FORTRAN order >> from the numpy.array() function, every time I reshape that array >> I need to tell reshape to use FORTRAN order? That seems a bit odd... > > Yes, that is what it means. Doing anything else leads to a host > of problems. Let me illustrate. > > a = rand(10,3) > b = a.transpose() > b.flags # b is now in Fortran order > b.ravel() # do you really want this to now be interpreted in > FORTRAN-order. I didn't think so. > > What this illustrates is that the FORTRAN and CONTIGUOUS flags on > the array are just special cases of the strides (and in fact can be > updated from the strides at any time). But, an ORDER flag would > be an independent concept that could be set and reset at will. The > ORDER flag is only necessary when somebody is interpreting the > array as a linear sequence of bytes. I think I understand now. Thanks. Now a question concerning a common use-case: Frequently I have either a string or a buffer protocol object that is basically a chunk of memory returned from some wrapped C library function. Sometimes this string/object is in fortran-strided memory order, and I need to tell the ndarray constructor that so that indexing works correctly. Clearly, fortran-strided memory order is just a special case of the strides, as you point out. So what I need to do in this case is provide an appropriate strides argument to the ndarray constructor. But that's harder (and less transparent for someone reading the code) than just setting a flag like Fortran=True or Order='FORTRAN'. So from this perspective it would be great if the ndarray constructor allowed something like strides='FORTRAN'. Of course, if an order flag is provided, then the ndarray constructor would have two different and orthogonal things that could/should accept a parameter of 'FORTRAN' -- Order and Strides. Now that's confusing! I guess the problem is (as you demonstrate clearly) that there are two semi-orthogonal issues around FORTRAN-ness. First is the issue of how a 1D memory region is to be indexed as a multidimensional array. (e.g. when constructing an array from a buffer.) The second is the issue of how a multidimensional array is to be treated as a 1D memory region. (e.g. when using ravel or resize [if resize is padding with the array contents]). This is very confusing, and could certainly benefit from some notational clarity. Does someone have any good ideas for what to call each of these properties? Strides and Order (respectively) seems to be what people are using in this email thread, but I'm not sure if there could be something better... Zach From tim.hochberg at cox.net Mon Mar 27 17:53:14 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 17:53:14 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442877DF.4050609@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44286C1C.5010605@cox.net> <442877DF.4050609@ee.byu.edu> Message-ID: <442896E1.8070207@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >>> Charles R Harris wrote: >>> >>> >>> l = list(a) >>> >>> l >>> [999, 1, 2, 3, 4, 5, 6, 7, 8] >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >>> >>> l += a >>> >>> l >>> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >>> >>> a >>> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >> >> >> >> Let me add that I think that this is pretty dubious, so if this is a >> new feature, perhaps we should revert it before it becomes entrenched. > > > > I don't think it's a new feature, but it's simply the result of > > l += a being translated to > > l = l + a # lists don't have in-place add's > > Numeric has this behavior as well. That's what I would expect, but Charles claimed it was resulted in an error of some sort, so I was wondering if it was new. I was also wondering if we could disable it somehow by proper finagaling of the various add and iadd ops, but I can't really see how. However, I did stumble on this odd behaviour: class reporter(object): def __init__(self, name): self.name = name def __add__(self, other): print "add called on", self.name, other.name return NotImplemented def __radd__(self, other): print "radd called on", self.name, other.name return NotImplemented def __iadd__(self, other): print "iadd called on", self.name, other.name return NotImplemented class reporter2(reporter): pass a = reporter("A") b = reporter2("B") a += b print a ==> iadd called on A B add called on A B radd called on B A iadd called on A B NotImplemented Shouldn't that raise TypeError? Yes? No? I'll go enter it as a bug, but I want to make sure I'm not missing something stupid. -tim From tim.hochberg at cox.net Mon Mar 27 18:41:02 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Mar 27 18:41:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44287D7D.8010604@ee.byu.edu> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> Message-ID: <4428A228.7040806@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> Travis Oliphant wrote: >> >>> Tim Hochberg wrote: >>> >>>>> >>>>> Yes, having this ability means that you have to think about it a >>>>> bit if you are going to use the functional interface and try to do >>>>> in-place operations. But, I would argue that this is an >>>>> "advanced" usage which is implemented to save space and time. >>>> >>>> >>>> >>>> >>>> >>>> How is this true though? In what way, for instance, is: >>>> >>>> b = asarray(a).reshape(newshape) >>>> >>>> slower or less space efficient than todays: >>>> >>>> b = reshape(a) >>> >>> >>> >>> >>> >>> Well, the big difference is that b=reshape(a) is actually >>> >>> try: >>> reshape = a.reshape >>> except AttributeError: >>> return a.__array_wrap__(asarray(a).reshape(newshape)) >>> >>> return reshape(newshape) >>> >>> So, it handles more cases, more cleanly then the >>> asarray(a).rehsape(newshape) approach does. >> >> >> >> OK. Although I should have said asanyarray(a).reshape(newshape), I >> still see how that this handles more cases. > > > Because you can define objects that aren't sub-classes of the array at > all but just define the method reshape and still have the functional > interface work. > >> >> I need to go think about this case some more. This is not something >> that I've run into in practice, but I can see I'll have a stronger >> case if I can come up with a alternative to this in terms of safe >> functions. Do you have some examples of objects that do not have >> 'reshape', but do have '__array_wrap__'? > > > > Not presently, the __array_wrap__ method is new. It's a mechanism for > letting functions that need arrays internally to be more polymorphic > and deal with different kinds of objects. > > > It would seem that most of Tim's complaints are directly against > polymorphism. I would argue that getting rid of polymorphism is not > something we should be trying to do in NumPy. I'm certainly not against polymorphism in general. However, I do believe that the intersection of polymorphism with view/copy behaviour can be dangerous in certain circumstances, and we should limit those circumstances to a few, visible functions. The cases I've been complaining about are all instances of this pattern: b = func(a, *other_args, **other_kwargs) With the added complication that b may be a view of a or it may be a new array object. This particular case is 'sharp blade' and should be treated with respect. Here's how I would probably write reshape if I were starting a Numeric like language from scratch: def reshape(a, shape): b = a.view() b.shape = shape return b That's no less polymorphic than the current reshape, but it's polymorphic in a different way. It would work on anything that has view method and a shape parameter. This as opposed to anything that has reshape or has __arrary_wrap__ and can be turned into an array. This is a nonstarter for backwards compatibility reasons however. > > Tim is pointing out that if you use polymorphic functions then you > can't assume things like "in-place" have any meaning. But, then > Python has this same problem, because > > l += a > > doesn't do 'in-place' for the list, but does do inplace if 'l' were an > array. In my mind, this problem and Tim's concern over 'view' and > 'copy' behavior is one and the same. I agree. > I don't view it as a problem of function behavior as much as problem > with documentation and "mismatch between what a particular user > expects and what is actually done." I disagree. Fernando stated this much better than I could: We should treat them as expensive specialties we must pay for with a very tight budget (the willingness and ability of users to keep track of the exponential rise in complexity induced by interlocking special cases). > > My attitude is that to write 'in-place' functions (what seems to be > the driving example that Tim brings up), you need to know some details > about what kind of object you are dealing with anyway, so you can't > write polymorphic in-place functions very easily. Correct. I doubt writing polymorphic in place functions will ever be easy. But the choice of tools we provide can influence how often they end up being correct. > > So, I guess from one perspective, Tim is arguing against things that > are at the very core of Python itself. I'm very resistant to his > desire to remove or significantly alter the functional behavior. > > I also think there is a very good reason to have a few methods that > return either a view or a copy. The reason is that there are some > arrays that the method can't be implemented unless a copy is made. > The only other possibility --- "raising an error" --- is going to > confuse a lot of new users and force them to deal with the underlying > memory-layout of an array before they really need to. I don't think this does anyone any favors myself, it just lets people get into habits early on that are going to bite them later. > I think the current compromise is practical and disagree strongly that > it is somehow "evil." It's only evil to somebody trying to do > in-place work. I think *most* of the current compromise is good. > If you are doing that in a polymorphic language like Python, then you > need to understand the actual objects you are dealing with and should > be expected to have a more mature understanding of NumPy. FWIW, I've been using Numeric and it's successors since Numeric was pre-alpha, when Jim Huginin was still around, when giants walked the earth. My understanding of numerapy [*] may be flawed, but it's definitely mature. [*] Referring to the whole Numeric/numarray/numpy family In general, when balancing safety, power and speed, numerapy has leaned heavily toward speed and power. I think that is the right choice. However, in the case at hand, there is no speed tradeoff: you'll note, for example, that I have consistently avoided advocating that reshape and friends return copies. The trade off has been between safety and power, and in this case the power is purely theoretical at this point since there aren't yet any clients to __array_wrap__ that don't define reshape. I would think that is a very narrow slice of potential object space. That being said I have a suggestion that *might* satisfy everyone. Set the WRITEABLE flag to false if reshape creates a new array: def _viewwrapit(obj, method, *args, **kwds): try: wrap = obj.__array_wrap__ except AttributeError: wrap = None if type(obj) is ndarray: writeable = True else: writeable = False obj = asarray(obj) result = getattr(obj, method)(*args, **kwds) result.flags.writeable = writeable if wrap: result = wrap(result) return result def reshape(a, newshape, order=False): try: reshape = a.reshape except AttributeError: return _viewwrapit(a, 'reshape', newshape, order=order) else: return reshape(newshape, order=order) a = arange(4) a2 = reshape(a, [2,2]) a2[0,1] = 99 print "a2 =", a2 print "a = ", a l = [0, 1, 2, 3] l2 = reshape(l, [2,2]) l2[0,1] = 99 ==> a2 = [[ 0 99] [ 2 3]] a = [ 0 99 2 3] Traceback (most recent call last): File "scratch.py", line 37, in ? l2[0,1] = 99 RuntimeError: array is not writeable Thus we get the same semantics as we have now exept in the one case that causes trouble: modifying the result of a function that returns both copies and views when it has returned a copy. The only catch is that clients of __array_wrap__ would need to be strongly encouraged to respect the writeable flag. However, since __array_wrap__ is new, this is probably not a big deal. This could also help solve the problem of how to deal with a.reshape(newshape) when the result would be a copy; simple make the copy nonwriteable. However, that case is more complex and requires more thought. No power is lost; 'advanced' users can simply set the writeable flag back to True. If we need to be able to do this in one line, setflags could be modified to return self, so that: reshape(a, newshape).setflags(write=True) is essentially equivalent to the current reshape. [A side note, it kind of looks like setflags may need some revamping; shouldn't the flag values in setflags match those in flags?] The result of reshape(a, newshape), when a new object is required can then *almost* be regarded as a view of a constant object. This is because modifying the operand and expecting it to change the result is extremely rare. Unlike modifying the result and expecting it to change the operand, which while not exactly common ,is not that rare and is how one gets into trouble. Regards, -tim From oliphant.travis at ieee.org Mon Mar 27 21:05:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 27 21:05:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428A228.7040806@cox.net> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> Message-ID: <4428C399.1080002@ieee.org> >> >> I also think there is a very good reason to have a few methods that >> return either a view or a copy. The reason is that there are some >> arrays that the method can't be implemented unless a copy is made. >> The only other possibility --- "raising an error" --- is going to >> confuse a lot of new users and force them to deal with the underlying >> memory-layout of an array before they really need to. > > I don't think this does anyone any favors myself, it just lets people > get into habits early on that are going to bite them later. > >> I think the current compromise is practical and disagree strongly >> that it is somehow "evil." It's only evil to somebody trying to do >> in-place work. > > I think *most* of the current compromise is good. > >> If you are doing that in a polymorphic language like Python, then you >> need to understand the actual objects you are dealing with and should >> be expected to have a more mature understanding of NumPy. > > FWIW, I've been using Numeric and it's successors since Numeric was > pre-alpha, when Jim Huginin was still around, when giants walked the > earth. My understanding of numerapy [*] may be flawed, but it's > definitely mature. > Weren't you involved in a port of Numeric to Jython. I seem to remember that you wrote that. So, I know you're an "old-timer" and I really appreciate that you are trying so hard to advocate your view. There are compromises all over the place and not all of them have anything remotely-approaching a "solid-reason" behind them other than that's just what got written. So, pre 1.0 I'm all for fixing things that can be fixed so we don't saddle ourselves with problems due to "muddy concepts". Of course once Python 3000 comes out (I'm guessing in 3-5 years) we'll have a chance to do this again ;-) > > In general, when balancing safety, power and speed, numerapy has > leaned heavily toward speed and power. I think that is the right > choice. However, in the case at hand, there is no speed tradeoff: > you'll note, for example, that I have consistently avoided advocating > that reshape and friends return copies. The trade off has been between > safety and power, and in this case the power is purely theoretical at > this point since there aren't yet any clients to __array_wrap__ that > don't define reshape. I would think that is a very narrow slice of > potential object space. The trade off seems to be in this case between safety and new-user-friendliness. > > That being said I have a suggestion that *might* satisfy everyone. Set > the WRITEABLE flag to false if reshape creates a new array: This is an interesting proposition. Definitely something that could be done. It would certainly serve the purpose of warning somebody that a new copy has been made. > Thus we get the same semantics as we have now exept in the one case > that causes trouble: modifying the result of a function that returns > both copies and views when it has returned a copy. The only catch is > that clients of __array_wrap__ would need to be strongly encouraged > to respect the writeable flag. However, since __array_wrap__ is new, > this is probably not a big deal. This could also help solve the > problem of how to deal with a.reshape(newshape) when the result would > be a copy; simple make the copy nonwriteable. > However, that case is more complex and requires more thought. > > No power is lost; 'advanced' users can simply set the writeable flag > back to True. If we need to be able to do this in one line, setflags > could be modified to return self, so that: > > reshape(a, newshape).setflags(write=True) Except it's the "young" user who would expect to be able to modify the array they just reshaped (and who wouldn't care that they got a copy in the first place). So, this puts more work for the "novice" user than the "advanced" user. But as you argue, having users not learn the difference between copy and view can certainly bite them. And, I can see that having somebody modify the array after reshaping is less likely --- most of the time it is just used in a later calculation. > > is essentially equivalent to the current reshape. [A side note, it > kind of looks like setflags may need some revamping; shouldn't the > flag values in setflags match those in flags?] Are you referring to keyword arguments? Then I suppose that would be true. > > The result of reshape(a, newshape), when a new object is required can > then *almost* be regarded as a view of a constant object. This is > because modifying the operand and expecting it to change the result is > extremely rare. Unlike modifying the result and expecting it to change > the operand, which while not exactly common ,is not that rare and is > how one gets into trouble. > One could try setting the writeable flag to false for copy-like behavior when both views and copies are possible and see what happens with dependent code like scipy. -Travis From charlesr.harris at gmail.com Mon Mar 27 21:40:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon Mar 27 21:40:02 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428A228.7040806@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> Message-ID: Tim, > > That being said I have a suggestion that *might* satisfy everyone. Set > the WRITEABLE flag to false if reshape creates a new array: I think this will cause problems as I myself usually want to modify the contents after a reshape. Why not in this case have a.reshape(...) return a view when possible and raise an error otherwise, and have the functional form make a copy. Reshape the method then provides a path to efficient computing, while reshape the function promotes safe computing. I like to write classes in C++ where I can do things like b.mul(c).add(d).div(e) all inplace for efficiency and the current situation strikes me as similar. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Mar 27 23:20:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Mar 27 23:20:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> Message-ID: <4428E36E.4000908@ieee.org> Charles R Harris wrote: > Tim, > > > That being said I have a suggestion that *might* satisfy everyone. Set > the WRITEABLE flag to false if reshape creates a new array: > > > I think this will cause problems as I myself usually want to modify > the contents after a reshape. Why not in this case have a.reshape(...) > return a view when possible and raise an error otherwise, and have the > functional form make a copy. Reshape the method then provides a path > to efficient computing, while reshape the function promotes safe > computing. I like to write classes in C++ where I can do things like > b.mul(c).add(d).div(e) all inplace for efficiency and the current > situation strikes me as similar. > This is the direction I could support. If we did this, then the ravel method would also have to change to be consistent. The functions would be copy-if-necessary and would *not* be deprecated. To summarize on making methods that return views raise an error when impossible and not changing and/or deprecating functions (i.e. the functions are simple wrappers around the methods). -1 on making methods that return views raise an error when impossible and changing the function to make a copy +0 -Travis From pgmdevlist at mailcan.com Tue Mar 28 00:19:06 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 28 00:19:06 2006 Subject: [Numpy-discussion] More about booleans as indices Message-ID: <200603280318.19036.pgmdevlist@mailcan.com> Folks, I was playing with the compress method and extended slicing techniques, when I ran into this problem: I want to select the records of rec.array (`arec`) for which a given tag (`tag`) has some given values, as listed in `specialtaglist`. A basic arec[arec.tag in specialtaglist] raises a ValueError exception (ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()). A workaround I found was: arec[ [arec.tag[k] in specialtaglist for k in range(len(arec))] ] I was wondering whether there was a better/faster/simpler method ? Thx in advance From pgmdevlist at mailcan.com Tue Mar 28 01:28:35 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 28 01:28:35 2006 Subject: [Numpy-discussion] New patch for MA Message-ID: <200603280427.52789.pgmdevlist@mailcan.com> Folks, You can find a new patch for MA on the wiki http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/ma-200603280900.patch along with a test suite. The 'clip' method should now work with array arguments. Were also added cumsum, cumprod, std, var and squeeze. I'll deal with flags, setflags, setfield, dump and others when I'll have a better idea of how it works -- which probably won't happen anytime soon, as I don't really have time to dig in the code for these functions. AAMOF, I'm more interested in checking/patching some other aspects of numpy for MA (eg, mlab...) Once again, please send me your comments and suggestions. Thx for everything P. From cjw at sympatico.ca Tue Mar 28 06:02:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 28 06:02:04 2006 Subject: [Numpy-discussion] problem with ndarray attributes In-Reply-To: <4428610B.4040307@ee.byu.edu> References: <44285D68.1060908@stsci.edu> <4428610B.4040307@ee.byu.edu> Message-ID: <442941B6.4050207@sympatico.ca> Travis Oliphant wrote: > Christopher Hanley wrote: > >> Hi Travis, >> >> I have run across a feature in numpy which I am not sure how to >> handle. One of the new features of the ndarray is that named fields >> can be accessed as attributes. So, if a user created a recarray with >> fields named "address", "phone", and "names" they would be able to >> access all of the names in the recarray objects with >> "recarrayObject.names". > > > > To be clear. This is only true for a recarray subclass. Standard > ndarray's cannot look-up fields with attribute access. Is any consideration being given to integrating recarray into ndarray? It seems to me that attribute access is desirable, providing that there is some check to ensure that the name does not duplicate an existing attribute. Colin W. > >> >> I have found that this can cause problems when a user creates a table >> with a field name that happen to coincide with an existing class >> attribute or method. In particular, I am having a problem in pyfits >> where a user has a table with a column named "field". This column >> name is causing an attribute "field" to be created that overrides the >> existing class field method. > > > > This is a standard problem with using attribute access. I suppose it > would be better to always lookup the attributes first on the object > and then if that fails use field access. That way library code would > not break if a user happened to name their fields after a method or > property. Users would have to change code if new methods or > properties were added (although that won't be as common). > >> >> It would seem to me that existing class methods and attributes should >> take precedence over the attributes created to correspond to column >> names. Is there a way to implement this behavior in ndarray? > > > > I think this is the right thing to do. Are there any objections? > > -Travis > From bblais at bryant.edu Tue Mar 28 06:03:14 2006 From: bblais at bryant.edu (Brian Blais) Date: Tue Mar 28 06:03:14 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? Message-ID: <4429414F.1060906@bryant.edu> Hello, I thought I had read somewhere that the values in a numpy contiguous array are stored column-wise (moving down from the upper left to the lower left first) like C as opposed to row-wise (moving right from the upper left to the upper right first). However, when I write a simply Pyrex function to access the data pointer and print it out, it seems to go across the row. Is this correct? I am using numpy 0.9.3 and Python 2.3 on Linux. This also occurs for Numeric. I post the code below, with the sample output. thanks, bb -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais #test_numpy.pyx: cdef extern from "numpy/arrayobject.h": struct PyArray_Descr: int type_num, elsize char type ctypedef class numpy.ArrayType [object PyArrayObject]: cdef char *data cdef int nd cdef int *dimensions, *strides cdef object base cdef PyArray_Descr *descr cdef int flags def test_numpy(ArrayType x): cdef int i cdef double *data cdef int length data=x.data length=x.dimensions[0]*x.dimensions[1] for i from 0 <= i < length: print data[i] #------------------------------------------------------ #testit.py: import numpy from test_numpy import * a=numpy.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12]],numpy.Float64) print a test_numpy(a) #------------------------------------------------------ #output from program: >>> execfile('testit.py') [[ 1. 2. 3.] [ 4. 5. 6.] [ 7. 8. 9.] [ 10. 11. 12.]] 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 11.0 12.0 From cjw at sympatico.ca Tue Mar 28 06:11:08 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 28 06:11:08 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <44287DFC.6050700@ee.byu.edu> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <442858F5.6030001@cox.net> <44286AAE.3030204@cox.net> <44287DFC.6050700@ee.byu.edu> Message-ID: <442943EF.8070606@sympatico.ca> Travis Oliphant wrote: > Charles R Harris wrote: > >> Really? If so, this is a version thing. This is what I get >> running on >> Python 2.4.1 and current numpy SVN (0.9.7.2286): >> >> >>> l = list(a) >> >>> l >> [999, 1, 2, 3, 4, 5, 6, 7, 8] >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >>> l += a >> >>> l >> array([1998, 2, 4, 6, 8, 10, 12, 14, 16]) >> >>> a >> array([999, 1, 2, 3, 4, 5, 6, 7, 8]) >> >> >> Ugh. IMHO, this should throw an error. For these sort of assignment >> operators I think the right hand side should be cast to the type of >> the left, which I think of as some sort of fancy register variable. >> If anything, the list should be appended to. I'll have to try the >> latest SVN just to appreciate this. >> > > Again, this is a Python thing. Numeric has this behavior as well. > Depending on your point of view it's either a great thing or a wart. A wart +1 Colin W. From faltet at carabos.com Tue Mar 28 07:19:25 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Mar 28 07:19:25 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? In-Reply-To: <4429414F.1060906@bryant.edu> References: <4429414F.1060906@bryant.edu> Message-ID: <200603281648.40265.faltet@carabos.com> A Dimarts 28 Mar? 2006 15:59, Brian Blais va escriure: > Hello, > > I thought I had read somewhere that the values in a numpy contiguous array > are stored column-wise (moving down from the upper left to the lower left > first) like C as opposed to row-wise (moving right from the upper left to > the upper right first). However, when I write a simply Pyrex function to > access the data pointer and print it out, it seems to go across the row. > Is this correct? Nope. C arrangement of values is row-wise and Fortran is column-wise. You are seeing row-wise ordering, so everything behaves as it should. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From tim.hochberg at cox.net Tue Mar 28 07:36:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 07:36:18 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428E36E.4000908@ieee.org> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> Message-ID: <442957A4.7070301@cox.net> Travis Oliphant wrote: > Charles R Harris wrote: > >> Tim, >> >> >> That being said I have a suggestion that *might* satisfy >> everyone. Set >> the WRITEABLE flag to false if reshape creates a new array: >> >> I think this will cause problems as I myself usually want to modify >> the contents after a reshape. Why not in this case have >> a.reshape(...) return a view when possible and raise an error >> otherwise, and have the functional form make a copy. Reshape the >> method then provides a path to efficient computing, while reshape the >> function promotes safe computing. I like to write classes in C++ >> where I can do things like >> b.mul(c).add(d).div(e) all inplace for efficiency and the current >> situation strikes me as similar. >> > This is the direction I could support. If we did this, then the > ravel method would also have to change to be consistent. > > The functions would be copy-if-necessary and would *not* be deprecated. > > To summarize > > on making methods that return views raise an error when impossible and > not changing and/or deprecating functions (i.e. the functions are > simple wrappers around the methods). > > -1 > > on making methods that return views raise an error when impossible and > changing the function to make a copy > > +0 +1 Assuming I understand all of the implications. * When doing automated conversion from Numeric/numarray to numpy, "reshape(obj) => asarray(obj).reshape()". Doing anything else is going to cause breakage and/or massive slowdowns. Can the conversion tool handle this? I would guess yes, but that's just a guess. * This means that "somenoncontiguousarray.reshape(newshape)" will oftern raise an error. That's a good thing, but I figured I'd get it out in the open now so no one is suprised. * Since x.reshape would no longer return copies, the order flag is unnecessary. * numpy.reshape(obj, newshape) on the other hand, always copies, so it *could* grow an order flag. I think that I'd prefer that it didn't; if you need that kind of control of the resulting arrays you should be using array to copy your matrices. * What does fixing ravel to be consistent mean? Always return a view? Or always return a copy? The latter seems more useful, as the first is already covered by both obj.reshape(-1) and obj.flat. Regards, -tim From tim.hochberg at cox.net Tue Mar 28 07:40:06 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 07:40:06 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442957A4.7070301@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> Message-ID: <44295897.9030307@cox.net> Tim Hochberg wrote: > Travis Oliphant wrote: > >> Charles R Harris wrote: >> >>> Tim, >>> >>> >>> That being said I have a suggestion that *might* satisfy >>> everyone. Set >>> the WRITEABLE flag to false if reshape creates a new array: >>> >>> I think this will cause problems as I myself usually want to modify >>> the contents after a reshape. Why not in this case have >>> a.reshape(...) return a view when possible and raise an error >>> otherwise, and have the functional form make a copy. Reshape the >>> method then provides a path to efficient computing, while reshape >>> the function promotes safe computing. I like to write classes in C++ >>> where I can do things like >>> b.mul(c).add(d).div(e) all inplace for efficiency and the current >>> situation strikes me as similar. >>> >> This is the direction I could support. If we did this, then the >> ravel method would also have to change to be consistent. >> >> The functions would be copy-if-necessary and would *not* be deprecated. >> >> To summarize >> >> on making methods that return views raise an error when impossible >> and not changing and/or deprecating functions (i.e. the functions are >> simple wrappers around the methods). >> >> -1 >> >> on making methods that return views raise an error when impossible >> and changing the function to make a copy >> >> +0 > > > +1 Assuming I understand all of the implications. > > * When doing automated conversion from Numeric/numarray to numpy, > "reshape(obj) => asarray(obj).reshape()". Doing anything else is going > to cause breakage and/or massive slowdowns. Can the conversion tool > handle this? I would guess yes, but that's just a guess. Actually, that should be: reshape(obj, newshape) => ascontiguousarray(obj).reshape(newshape) > > * This means that "somenoncontiguousarray.reshape(newshape)" will > oftern raise an error. That's a good thing, but I figured I'd get it > out in the open now so no one is suprised. > > * Since x.reshape would no longer return copies, the order flag is > unnecessary. > > * numpy.reshape(obj, newshape) on the other hand, always copies, so it > *could* grow an order flag. I think that I'd prefer that it didn't; if > you need that kind of control of the resulting arrays you should be > using array to copy your matrices. > > * What does fixing ravel to be consistent mean? Always return a view? > Or always return a copy? The latter seems more useful, as the first is > already covered by both obj.reshape(-1) and obj.flat. > > Regards, > > -tim > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From bblais at bryant.edu Tue Mar 28 08:41:04 2006 From: bblais at bryant.edu (Brian Blais) Date: Tue Mar 28 08:41:04 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? In-Reply-To: <200603281648.40265.faltet@carabos.com> References: <4429414F.1060906@bryant.edu> <200603281648.40265.faltet@carabos.com> Message-ID: <4429662A.2070307@bryant.edu> Francesc Altet wrote: > > Nope. C arrangement of values is row-wise and Fortran is column-wise. > You are seeing row-wise ordering, so everything behaves as it should. > interesting. I am coming from Matlab, where things are definitely column-wise organized. I always assumed this was the C standard, but now that I think of it the C double array [][] is row-wise. thanks, bb -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From charlesr.harris at gmail.com Tue Mar 28 09:21:10 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 09:21:10 2006 Subject: [Numpy-discussion] arrays are column-wise or row-wise? In-Reply-To: <4429662A.2070307@bryant.edu> References: <4429414F.1060906@bryant.edu> <200603281648.40265.faltet@carabos.com> <4429662A.2070307@bryant.edu> Message-ID: Hi Brian, On 3/28/06, Brian Blais wrote: > > Francesc Altet wrote: > > > > Nope. C arrangement of values is row-wise and Fortran is column-wise. > > You are seeing row-wise ordering, so everything behaves as it should. > > > > interesting. I am coming from Matlab, where things are definitely > column-wise > organized. I always assumed this was the C standard, but now that I think > of it the > C double array [][] is row-wise. Matlab was originally written in Fortran by Cleve Moler as an easy to use interface to LINPACK and EISPACK. If you google around you can probably find the original free Fortran version out on the net somewhere. Matlab was rewritten in C when John Little and Cleve Moler founded MathWorks but I imagine the new version still called many Fortran subroutines for the heavy lifting . You can find many other Fortran traces in Matlab. For instance, Matlab array indices start at 1 in the traditional Fortran way. On a side note, I believe that John Little's parents moved into the house that my family built in Lincoln, MA. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Tue Mar 28 10:45:05 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue Mar 28 10:45:05 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4428C399.1080002@ieee.org> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428C399.1080002@ieee.org> Message-ID: <4429841F.4040105@noaa.gov> I want to start by saying a am +1 on everything Tim is advocating, just so you all know it's not just him. I'm not writing any code, however. A few comments: Travis Oliphant wrote: > It would seem that most of Tim's complaints are directly against > polymorphism. I don't think so. I think Tim (and I) just want it to be really clear what the polymorphic functions are and what aren't. Also, for polymorphism to work, there always has to be some definition of what "similar" objects are: in static languages, this means they are derived from the same class. In duck-typed languages, it means they share the same behavior, so all we're doing here is deciding exactly how much behavior they need to share. > But, then Python has > this same problem, because > > l += a > > doesn't do 'in-place' for the list, but does do inplace if 'l' were an > array. Yes, it does do it for the list, but differently. However, maybe a tuple is a better example: >>> a = (1,2,3) >>> id(a) 331328 >>> a += (3,4,5) >>> a (1, 2, 3, 3, 4, 5) >>> id(a) 236712 Whereas with numpy: >>> import numpy as N >>> a = N.array((1,2,3)) >>> id(a) 6436864 >>> a += (3,4,5) >>> a array([4, 6, 8]) >>> id(a) 6436864 So there are two differences: addition is defined differently, and one is in-place, and one isn't. Personally, I've always thought that the "in place" operators should never work with immutable types: in=place should mean in-place,, not "in-place if possible, but not if not possible. and yes, this is completely analogous to the issue at hand. And then we have this (already pointed out in this thread) >>> l = [1,2,3] >>> l [1, 2, 3] >>> a array([4, 6, 8]) >>> l += a >>> l array([ 5, 8, 11]) >>> b [4, 6, 8] >>> l = [1,2,3] >>> l += b >>> l [1, 2, 3, 4, 6, 8] >>> l = [1,2,3] >>> t = (3,4,5) >>> l+=t >>> l [1, 2, 3, 3, 4, 5] How this is anything but a bug is beyond me! Lists do define the in-place operator, and really do it in-place. That being the case, I would certainly never expect += to change the type of a list! It doesn't with a tuple. Is this a numpy issue or a Python one? > I don't view it as a problem of > function behavior as much as problem with documentation and "mismatch > between what a particular user expects and what is actually done." Quite true. However, what I advocate is that to keep that mismatch as rare as possible, the ONLY functions that should have the "sometimes a view and sometimes a copy" behavior should be functions that explicitly exist to provide that behavior, such as asarray(). All it does is save typing and increase the potential for confusion and errors to built it in to functions that have other uses. Travis Oliphant wrote: > on making methods that return views raise an error when impossible and > not changing and/or deprecating functions (i.e. the functions are simple > wrappers around the methods). +1 > on making methods that return views raise an error when impossible and > changing the function to make a copy +1 : Only if the functions are moved to a separate "backwards compatible" name space. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Tue Mar 28 11:05:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 11:05:01 2006 Subject: [Numpy-discussion] Re: Ransom Proposals In-Reply-To: <4429841F.4040105@noaa.gov> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428C399.1080002@ieee.org> <4429841F.4040105@noaa.gov> Message-ID: Christopher Barker wrote: > And then we have this (already pointed out in this thread) > >>>> l = [1,2,3] >>>> l > [1, 2, 3] >>>> a > array([4, 6, 8]) >>>> l += a >>>> l > array([ 5, 8, 11]) > >>>> b > [4, 6, 8] >>>> l = [1,2,3] >>>> l += b >>>> l > [1, 2, 3, 4, 6, 8] > >>>> l = [1,2,3] >>>> t = (3,4,5) >>>> l+=t >>>> l > [1, 2, 3, 3, 4, 5] > > How this is anything but a bug is beyond me! Lists do define the > in-place operator, and really do it in-place. That being the case, I > would certainly never expect += to change the type of a list! It doesn't > with a tuple. Is this a numpy issue or a Python one? A Python one. http://mail.python.org/pipermail/python-dev/2006-March/062889.html -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at cox.net Tue Mar 28 11:22:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 11:22:07 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4429841F.4040105@noaa.gov> References: <4424AB16.6000007@cox.net> <4425D222.7010301@cox.net> <4425F326.5010904@colorado.edu> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428C399.1080002@ieee.org> <4429841F.4040105@noaa.gov> Message-ID: <44298CB6.90007@cox.net> Christopher Barker wrote: > I want to start by saying a am +1 on everything Tim is advocating, > just so you all know it's not just him. I'm not writing any code, > however. Thanks Chris. > > A few comments: > > Travis Oliphant wrote: > >> It would seem that most of Tim's complaints are directly against >> polymorphism. > > > I don't think so. I think Tim (and I) just want it to be really clear > what the polymorphic functions are and what aren't. Also, for > polymorphism to work, there always has to be some definition of what > "similar" objects are: in static languages, this means they are > derived from the same class. In duck-typed languages, it means they > share the same behavior, so all we're doing here is deciding exactly > how much behavior they need to share. > > > But, then Python has > >> this same problem, because >> >> l += a >> >> doesn't do 'in-place' for the list, but does do inplace if 'l' were >> an array. > > > Yes, it does do it for the list, but differently. However, maybe a > tuple is a better example: > > >>> a = (1,2,3) > >>> id(a) > 331328 > >>> a += (3,4,5) > >>> a > (1, 2, 3, 3, 4, 5) > >>> id(a) > 236712 > > Whereas with numpy: > > >>> import numpy as N > >>> a = N.array((1,2,3)) > >>> id(a) > 6436864 > >>> a += (3,4,5) > >>> a > array([4, 6, 8]) > >>> id(a) > 6436864 > > So there are two differences: addition is defined differently, and one > is in-place, and one isn't. Personally, I've always thought that the > "in place" operators should never work with immutable types: in=place > should mean in-place,, not "in-place if possible, but not if not > possible. and yes, this is completely analogous to the issue at hand. The problem with this is that then inplace add doesn't work with numbers; a huge use case that can't be neglected. Really though, in-place addition for immutable types works fine unless one is overusing 'is'. Issues can come up with mixing mutable and imutatble types though, or two mutable types if the left handed one doesn't define iadd. > And then we have this (already pointed out in this thread) > > >>> l = [1,2,3] > >>> l > [1, 2, 3] > >>> a > array([4, 6, 8]) > >>> l += a > >>> l > array([ 5, 8, 11]) It is a bug. Travis brought this up on PyDev and they agreed that there is a problem with PyNumber_InplaceAdd. Armin's looking at this now in the hope of getting a fix in for 2.5. The result should be a list: [1,2,3,5,8,11]. > >>> b > [4, 6, 8] > >>> l = [1,2,3] > >>> l += b > >>> l > [1, 2, 3, 4, 6, 8] > > >>> l = [1,2,3] > >>> t = (3,4,5) > >>> l+=t > >>> l > [1, 2, 3, 3, 4, 5] > > How this is anything but a bug is beyond me! Lists do define the > in-place operator, and really do it in-place. That being the case, I > would certainly never expect += to change the type of a list! It > doesn't with a tuple. Is this a numpy issue or a Python one? A Python one. However, I'll be writing more on this in a moment; the behaviour that causes the issue with list is slated to be fixed, but I think that there's some other cases to consider. > >> I don't view it as a problem of function behavior as much as problem >> with documentation and "mismatch between what a particular user >> expects and what is actually done." > > > Quite true. However, what I advocate is that to keep that mismatch as > rare as possible, the ONLY functions that should have the "sometimes a > view and sometimes a copy" behavior should be functions that > explicitly exist to provide that behavior, such as asarray(). All it > does is save typing and increase the potential for confusion and > errors to built it in to functions that have other uses. > > Travis Oliphant wrote: > >> on making methods that return views raise an error when impossible >> and not changing and/or deprecating functions (i.e. the functions are >> simple wrappers around the methods). > > > +1 > >> on making methods that return views raise an error when impossible >> and changing the function to make a copy > > > +1 : Only if the functions are moved to a separate "backwards > compatible" name space. > FWIW, I'm fine if the functions always copy and methods always returns views, since this gets rid of the view/copy dilema. I believe that Sasha previously raised objections to this on backwards compatibility grounds. I believe Sasha preferred leaving the functions alone and just not recomending their use. I was fine with that too as long as the functions got segregated. Travis didn't like that, which loops us back around to the current proposal. Regards, -tim From tim.hochberg at cox.net Tue Mar 28 12:18:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 12:18:04 2006 Subject: [Numpy-discussion] inplace add Message-ID: <442999E3.8030102@cox.net> Travis recently brought up the odd behaviour of adding an array to a list using inplace add, notably: >>> l = range(4) >>> a = arange(4) >>> l += a >>> l array([0, 2, 4, 6]) It was agreed that this was a deficiency in PyNumber_InplaceAdd and this is slated to be fixed. Hopefully for 2.5. Thanks Travis, Robert Kern and Armin Rigo for taking care of this. I had planned to write something here about, how, although this fixes the problem with lists, there is a similar problem with tuples, and since tuples don't have __iadd__, we could still expect to see this problem there. However, I now think that's not the case. The proposed resolution order will work something like: def iadd(a, b): if hasattr(a, '__iadd__'): tmp = a.__iadd__(b) if tmp is not NotImplemented: return tmp if hasattr(a, '__add__'): tmp = a.__add__(b) if tmp is not NotImplemented: return tmp if hasattr(b, '__radd__'): temp = b.__radd__ if tmp is not NotImplemented: return tmp raise TypeError("can't add %r and %r" % (type(a), type(b))) You'll notice if you try this, that "iadd(atuple, anarray)" throws an error because atuple refuses to be added to anything other than another tuple.That raises an error at "__add__" and prevents any oddities. So, we should be safe from both tuples and lists, and really they are the only types that are prevalent enough to worry about in this context. Still, I have a niggling feeling that iadd should really not be calling radd at all, but nothing concrete. -tim From oliphant.travis at ieee.org Tue Mar 28 12:41:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 12:41:03 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <442957A4.7070301@cox.net> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> Message-ID: <4429944C.9060103@ieee.org> Tim Hochberg wrote: > > +1 Assuming I understand all of the implications. > > * When doing automated conversion from Numeric/numarray to numpy, > "reshape(obj) => asarray(obj).reshape()". Doing anything else is going > to cause breakage and/or massive slowdowns. Can the conversion tool > handle this? I would guess yes, but that's just a guess. Why would we need to change this --- even to ascontiguousarray(obj).reshape()? > > * This means that "somenoncontiguousarray.reshape(newshape)" will > oftern raise an error. That's a good thing, but I figured I'd get it > out in the open now so no one is suprised. I'm still not sold that raising an error here is a good thing, which is why I think the function should work as it always as and *not* be deprecated or hard to get at even if the method is changed to raise an error if the reshape cannot be accomplished without a copy. Discontiguous arrays actually show up quite a bit. I'm not convinced that Tim's concern over "confusion" between when something returns a view and when it returns a copy is that big of a deal. I maintain it's only disconcerting for someone who wants to do something "advanced". That person will know how to figure it out. If things are "fixed" then new users will get bitten by slow code or errors that don't make sense. I'm opposed to making things unnecessarily hard for new users. Let's discuss this with a particular example in mind that is not contrived to do 'in-place' operations. I think this will exemplify the crux of the matter for those who may not quite understand what this discussion is about. a = rand(4,5,3) b = a.transpose().reshape(15,4) c = a.reshape(15,4) In this example b is now a copy of the data in a and c is a "view" of the data in a. The reason for this is because a.transpose() returns a "view" of the data which is not C-style contiguous. The reshape function needs to know how to go between a linear sequence of elements and a multidimensional array --- the default is current C-style contiguous (last-index varies the fastest). If I understand correctly, Tim want's the second line to raise an error so that the reshape method *always* returns a view. I'm not enthused about having the second line raise an error because it is an example of common code (perhaps a transpose does not precede the reshape but a slice selection --- there are lots of ways to get discontiguous arrays). As it stands, I think the only problems that arise are when the user wants to modify b and/or c in-place. Then, one is not sure whether or not b (or c) is pointing to the data of a or not. My argument is that the reshape method documentation should state that you should not modify the contents of the return value of reshape unless it doesn't matter whether or not you have a copy of the data. I would even be willing to make 'views' un-writeable by default so that there was no way to "inadvertently" modify the data in a, unless you know you want to. We had this discussion for ravel months ago and the flatten method which always returns copies was created. The ravel method is another one that returns views when it can (contiguous) and copies when it can't. I think reshape and ravel are the only methods that behave this way. I'm not convinced that changing their current, standard behavior is going to buy us anything but more confusion. > > * Since x.reshape would no longer return copies, the order flag is > unnecessary. I don't believe this. See my other posts on the difference between the array striding and how you want to interpret the array. > > * numpy.reshape(obj, newshape) on the other hand, always copies, so it > *could* grow an order flag. I think that I'd prefer that it didn't; if > you need that kind of control of the resulting arrays you should be > using array to copy your matrices. No, this is not what I understand. the reshape function would not *always* copy but would behave as it currently does. Again, you haven't sold me on the need to absolutely, all the time, return either views or copies. I think such a dogmatic view is too limiting in this case. I have not been bitten by this in over 10 years. Why is this such an issue now? > > * What does fixing ravel to be consistent mean? Always return a view? > Or always return a copy? The latter seems more useful, as the first is > already covered by both obj.reshape(-1) and obj.flat. I meant that whatever the behavior is for .reshape should be the same for .ravel since it is syntactic sugar for obj.reshape(-1) And obj.flat is not even in the same ball-park since it returns an array_iterator object and not an array. Best, -Travis From jelleferinga at gmail.com Tue Mar 28 13:21:05 2006 From: jelleferinga at gmail.com (Jelle Feringa / EZCT Architecture & Design Research) Date: Tue Mar 28 13:21:05 2006 Subject: [Numpy-discussion] scipybooksample | dtype Message-ID: <37fac3b90603281320v365e9543t55f96abc7d495e4b@mail.gmail.com> Hi all, I'm familiarizing myself with scipy, and working my way through the http://numeric.scipy.org/scipybooksample.pdf tutorial. the tutorial states: ------------------------------------- >>> a = array([[1,2,3],[4,5,6]]) >>> a.shape (2, 3) >>> a.dtype ------------------------------------- when i run this code on my machine, having numpy 0.9.6 installed, i get the following: >>> a = numpy.array([[1,2,3],[4,5,6]]) >>> a.dtype dtype('>> numpy.version.version '0.9.6' >>> >>> seems that the way dtypes are being printed changed since the scipybooksample.pdf was released, would anyone of you be able to point me out where I can read up to what the updated dtypes signify? it's a bit cryptic to the uninitiated. i do understand that at the rate scipy/numpy is progressing that it can be hard to get the documentation consist with the current version. //btw: the new ndimage module is a great piece of work, thanks so much for all your efforts! -jelle -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Mar 28 13:24:04 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 13:24:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4429944C.9060103@ieee.org> References: <4424AB16.6000007@cox.net> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> <4429944C.9060103@ieee.org> Message-ID: Travis, On 3/28/06, Travis Oliphant wrote: > > Tim Hochberg wrote: a = rand(4,5,3) > b = a.transpose().reshape(15,4) > c = a.reshape(15,4) I would do this: >> b = reshape(a.transpose(), (15,4)) or, if the functional version of transpose would always return a copy: >> b = transpose(a).reshape(15,4) As I said before, I simply stopped using reshape because of its ambiguous behaviour. There were clearer ways to get what I wanted. But in any case, the casual user will hardly notice the inefficiency of making a copy. Users concerned with speed can use methods and should know enough to understand what is going on, but they should be warned if their attemps at efficiency aren't going to pan out in a particular case. > My argument is that the reshape method documentation should state that > you should not modify the contents of the return value of reshape unless > it doesn't matter whether or not you have a copy of the data. Who reads documentation? Why should they have to rummage through the footnotes or run a bunch of test cases to discover why things aren't behaving? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Mar 28 13:38:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 13:38:02 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error Message-ID: <4429AC89.6090301@ieee.org> It look's like Tim's arguments have won out for the most part. The .reshape and .ravel methods of arrays now return views only or raise an error if that is not possible. However, both of these methods still have an order argument because I don't think it is prudent to have a.transpose().ravel() automatically column-scan --- you have to tell ravel you want a Fortran-order result. The reshape and ravel functions behave exactly as they used to (view if possible, copy otherwise). I don't think these functions should be deprecated. I don't think any of the functional interfaces should be deprecated, either. Part of the reason for the change is that it was not disruptive. None of the NumPy code broke (except for a small piece of arrayprint) after the change. None of the SciPy code broke either (as far as I can tell). Hopefully we can put this to rest and move on to more important issues... -Travis From bblais at bryant.edu Tue Mar 28 14:34:02 2006 From: bblais at bryant.edu (Brian Blais) Date: Tue Mar 28 14:34:02 2006 Subject: [Numpy-discussion] getting data pointer to numpy object in Pyrex Message-ID: <4429B89E.30100@bryant.edu> Hello, I posted this on the Pyrex list, but I figured someone here may be able to answer it too. Is here an easy way, in Pyrex, to get the data pointer to a numpy object which is *not* passed as a parameter? For example, I know I can do: #--------------- cdef extern from "numpy/arrayobject.h": struct PyArray_Descr: int type_num, elsize char type ctypedef class numpy.ArrayType [object PyArrayObject]: cdef char *data cdef int nd cdef int *dimensions, *strides cdef object base cdef PyArray_Descr *descr cdef int flags def test1(ArrayType w): # pass a numpy object cdef double *w_p # get the pointers w_p=w.data #--------------- ...but I would like to do something like... #------------------------------- def test2(d): # pass a python dict w=d['w'] # get the numpy object cdef double *w_p # get the pointers w_p=w.data #------------------------------- Is there a way to do this? thanks, Brian Blais -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From tim.hochberg at cox.net Tue Mar 28 14:40:04 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 14:40:04 2006 Subject: [Numpy-discussion] Ransom Proposals In-Reply-To: <4429944C.9060103@ieee.org> References: <4424AB16.6000007@cox.net> <44276AF0.5070405@cox.net> <4428177F.4030602@sympatico.ca> <44281CE3.8080800@cox.net> <4428329B.7050300@ee.byu.edu> <4428484F.2030604@cox.net> <44285C3D.9080802@ee.byu.edu> <44286266.9050503@cox.net> <44287D7D.8010604@ee.byu.edu> <4428A228.7040806@cox.net> <4428E36E.4000908@ieee.org> <442957A4.7070301@cox.net> <4429944C.9060103@ieee.org> Message-ID: <4429BAF4.6070801@cox.net> Travis Oliphant wrote: > Tim Hochberg wrote: > >> >> +1 Assuming I understand all of the implications. >> >> * When doing automated conversion from Numeric/numarray to numpy, >> "reshape(obj) => asarray(obj).reshape()". Doing anything else is >> going to cause breakage and/or massive slowdowns. Can the conversion >> tool handle this? I would guess yes, but that's just a guess. > > Why would we need to change this --- even to > ascontiguousarray(obj).reshape()? Because things could break or get very very slow. Perhaps I'm the only one who's ever used reshape to excess, so perhaps it won't really hurt anyone. You can see the code that originally inspired my loathing for reshape here: http://starship.python.net/~hochberg/fourier.py.html Look at fft_base2. I don't *think* that this code would break under this change, however it would certainly start running very slow if reshape started returning copies all the time. On the other hand, asarray(x).reshape would work about the same. It is true that I have nothing comparable in my active codebase, so maybe such stuff is too rare to worry about. [The code above is quite old and may not even work on current Numeric. It was quite cool though -- running within a factor of two in speed of Numeric's built in FFT for large power of two length arrays]. So in summary, some things could break. Or they could become really slow. I don't know how much of a problem, if any, that this will be in practice. I didn't find anything in my active codebase where it would be a problem, so I guess it doesn't matter to me. >> >> * This means that "somenoncontiguousarray.reshape(newshape)" will >> oftern raise an error. That's a good thing, but I figured I'd get it >> out in the open now so no one is suprised. > > I'm still not sold that raising an error here is a good thing, which > is why I think the function should work as it always as and *not* be > deprecated or hard to get at even if the method is changed to raise an > error if the reshape cannot be accomplished without a copy. I'm not sure what you are saying here. My bullet point above only relates to methods. It sounds like your on board with methods always returning views. Yet you disagree. Are we talking past each other here? > Discontiguous arrays actually show up quite a bit. I'm not convinced > that Tim's concern over "confusion" between when something returns a > view and when it returns a copy is that big of a deal. I maintain > it's only disconcerting for someone who wants to do something > "advanced". That person will know how to figure it out. If things > are "fixed" then new users will get bitten by slow code or errors that > don't make sense. I'm opposed to making things unnecessarily hard for > new users. Me too. But I have come to opposite conclusion from you here. If the functions always returned a copy, they are safe and easy to use. And the error message for the method reshape could be made quite informative: """cannot reshape a discontiguous array with the method reshape use the function reshape instead or use ascontiguousarray to get a contiguous array""" With something like that in place, I think it would be hard to get stuck to too long if one ran into an error reshaping a discontiguous array. > > Let's discuss this with a particular example in mind that is not > contrived to do 'in-place' operations. I think this will exemplify > the crux of the matter for those who may not quite understand what > this discussion is about. > > a = rand(4,5,3) > b = a.transpose().reshape(15,4) > c = a.reshape(15,4) > > In this example b is now a copy of the data in a and c is a "view" of > the data in a. The reason for this is because a.transpose() returns > a "view" of the data which is not C-style contiguous. The reshape > function needs to know how to go between a linear sequence of elements > and a multidimensional array --- the default is current C-style > contiguous (last-index varies the fastest). > If I understand correctly, Tim want's the second line to raise an > error so that the reshape method *always* returns a view. Yes. Well that's one of my many positions. I've stated my primary goal many times, which is that with a few careful exceptions, functions should always return views or always return copys. There are many ways to get closer to that goal, but each one has someone who dislikes it. This approach is one of those ways. > I'm not enthused about having the second line raise an error because > it is an example of common code (perhaps a transpose does not precede > the reshape but a slice selection --- there are lots of ways to get > discontiguous arrays). As it stands, I think the only problems that > arise are when the user wants to modify b and/or c in-place. Then, > one is not sure whether or not b (or c) is pointing to the data of a > or not. > My argument is that the reshape method documentation should state that > you should not modify the contents of the return value of reshape > unless it doesn't matter whether or not you have a copy of the data. > I would even be willing to make 'views' un-writeable by default so > that there was no way to "inadvertently" modify the data in a, unless > you know you want to. I weeble and wobble on this suggestion. It is appealing, but it is a little weird. To make it truly safe you'd need to lock both the operand and the result, which I expect would make people squawk.. > > > We had this discussion for ravel months ago and the flatten method > which always returns copies was created. The ravel method is another > one that returns views when it can (contiguous) and copies when it can't. > I think reshape and ravel are the only methods that behave this way. > I'm not convinced that changing their current, standard behavior is > going to buy us anything but more confusion. Wow! So we have: reshape(-1) ravel() flatten() flat And that's just the methods. That's not on todays agenda however. >> >> * Since x.reshape would no longer return copies, the order flag is >> unnecessary. > > I don't believe this. See my other posts on the difference between > the array striding and how you want to interpret the array. I think I see the root of our difference here. My thinking has been that the underlying data could be stored in either Fortran-order or C-order, but the numarray objects themselves were always C-order in the sense that the last axes were viewed as varying fastest. Thus you only need to know the order when you copy it since that controls the layout of the underlying data structure. On the other hand, your view is that the entire numarray object can be viewed as either C-order or Fortran-order. Thus, every reshaping operation needs to know the order of the matrix. Is that correct? I'm not sure how I feel about that, but at least I think I've got it now. And if we view the arrays that way, doesn't that mean flat is broken? >> >> * numpy.reshape(obj, newshape) on the other hand, always copies, so >> it *could* grow an order flag. I think that I'd prefer that it >> didn't; if you need that kind of control of the resulting arrays you >> should be using array to copy your matrices. > > No, this is not what I understand. the reshape function would not > *always* copy but would behave as it currently does. Again, you > haven't sold me on the need to absolutely, all the time, return either > views or copies. I think such a dogmatic view is too limiting in > this case. > > I have not been bitten by this in over 10 years. Why is this such an > issue now? Because it's probably the only chance we'll ever get to change some of these things. >> >> * What does fixing ravel to be consistent mean? Always return a view? >> Or always return a copy? The latter seems more useful, as the first >> is already covered by both obj.reshape(-1) and obj.flat. > > > I meant that whatever the behavior is for .reshape should be the same > for .ravel since it is syntactic sugar for obj.reshape(-1) > > And obj.flat is not even in the same ball-park since it returns an > array_iterator object and not an array. > Regards, -tim From robert.kern at gmail.com Tue Mar 28 15:06:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 15:06:02 2006 Subject: [Numpy-discussion] Re: getting data pointer to numpy object in Pyrex In-Reply-To: <4429B89E.30100@bryant.edu> References: <4429B89E.30100@bryant.edu> Message-ID: Brian Blais wrote: > Hello, > > I posted this on the Pyrex list, but I figured someone here may be able > to answer it too. > > Is here an easy way, in Pyrex, to get the data pointer to a numpy object > which is *not* passed as a parameter? For example, I know I can do: > > #--------------- > cdef extern from "numpy/arrayobject.h": > > struct PyArray_Descr: > int type_num, elsize > char type > > ctypedef class numpy.ArrayType [object PyArrayObject]: > cdef char *data > cdef int nd > cdef int *dimensions, *strides > cdef object base > cdef PyArray_Descr *descr > cdef int flags > > def test1(ArrayType w): # pass a numpy object > > cdef double *w_p > > # get the pointers > w_p=w.data > #--------------- > > ...but I would like to do something like... > > #------------------------------- > def test2(d): # pass a python dict > > w=d['w'] # get the numpy object > > cdef double *w_p > > # get the pointers > w_p=w.data > #------------------------------- > > > Is there a way to do this? You will probably want to use the c_numpy.pxd file in numpy/doc/pyrex/. It has exposed a bit more of the API than you have. If you have any contributions, we would like to fold them in. In general, you would just make a cast like you would for any other C type. However, you must be sure that the object actually is an array. I think the following code would work as expected, including the case where d['w'] can't be turned into an array, but I have not tested it. cimport c_numpy import numpy c_numpy.import_array() def test2(d): cdef c_numpy.ndarray w w = numpy.asarray(d['w']) # ... -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant.travis at ieee.org Tue Mar 28 15:14:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 15:14:00 2006 Subject: [Numpy-discussion] Re: [SciPy-user] multiplying sparse matrices in scipy-0.4.8? In-Reply-To: <98790998-659E-42C3-A2AD-005AB68CCA12@ftw.at> References: <87odzx3x6r.fsf@hmm.lanl.gov> <98790998-659E-42C3-A2AD-005AB68CCA12@ftw.at> Message-ID: <4429C30A.9040206@ieee.org> Ed Schofield wrote: > > > Yes, this is a bug. > > Travis, could you please take a look at this? The FORTRAN functions > dcscmucsc and dcscmucsr both seem to be returning incorrect values in > indptr. In fact, could you please explain the Python code in matmat() > that calls these functions? I'd like to understand what the "while 1" > loop is for, and in particular why we have > c, rowc, ptrc, irow, kcol, ierr = func(*args) > when c, rowc, etc are part of *args anyway. The arguments are input and output arguments because that is my understanding of f2py's preference if something is both input and ouput rather than using an in-place argument. The while True: loop is so that resizing can occur if the guess on how many non-zero elements to reserve for the output matrix is wrong. There were some problems with the re-entry that I've now fixed. The tests you added to SciPy now pass. -Travis From tim.hochberg at cox.net Tue Mar 28 15:36:01 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 15:36:01 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429AC89.6090301@ieee.org> References: <4429AC89.6090301@ieee.org> Message-ID: <4429C812.90109@cox.net> Travis Oliphant wrote: > > It look's like Tim's arguments have won out for the most part. Hooray. Now let's hope I'm right! > > The .reshape and .ravel methods of arrays now return views only or > raise an error if that is not possible. However, both of these > methods still have an order argument because I don't think it is > prudent to have a.transpose().ravel() automatically column-scan --- > you have to tell ravel you want a Fortran-order result. OK, I think I understand why you want the order flag now. But isn't flat in a weird state now? All of the other reshape methods take an order parameter, but flat is inherently C-ordered, is it not? Also, is it useful to view an array as fortran ordered? I suppose if you are working with multidimensional Fortran data you want to keep it in the correct order. Still, something seems not quite right about the current state of affairs. Let me ask a question about the order flag: does it mean anything for a noncontiguous array? As I interpret it, right now: CONTIGUOUS = True, FORTRAN = True => Strides are Fortran order CONTIGUOUS = True, FORTRAN = False=> Strides are C order CONTIGUOUS = False, FORTRAN = Any=> Matrix is Discontiguous Is that correct? Or is it possible to have strides that neither C-order not Fortran-Order in a contiguous matrix? I'm trying to find a way to shoehorn an order flag in here without breaking anything, but I don't think it'll fit. So you'd need a separate flag to indicate the reshape-order if one wanted to do that. Mostly this would make life simpler for everyone, in normal use, the reshape-order flag gets set at array creation and everything just works. flat works properly, the order flag is only needed in construction. However, since the behaviour of the array is different with respect to reshape, you'd probably want to tag the repr somehow ('farray' instead of 'array' perhaps). In practice, this would be essentially the same as having a sister class to ndarray that worked on Fortran ordered data. Everything would behave the same except for how reshape works. I'm not sure if any of that is a good idea, but I'm not entirely convinced that the order flag, although a big improvement over what it replaced, is the best approach, so I'm musing. > The reshape and ravel functions behave exactly as they used to (view > if possible, copy otherwise). I don't think these functions should be > deprecated. I don't think any of the functional interfaces should be > deprecated, either. The good thing about functions is that I can always hot patch numpy with safer versions if I like. So I'll call this good. When I get around to it, I'll probably try patching in versions of reshape and transpose that lock either views or copies and see how that works out. I'll report back on how it goes. > > Part of the reason for the change is that it was not disruptive. None > of the NumPy code broke (except for a small piece of arrayprint) after > the change. None of the SciPy code broke either (as far as I can tell). > Hopefully we can put this to rest and move on to more important issues... Hooray! -tim From charlesr.harris at gmail.com Tue Mar 28 16:58:02 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 16:58:02 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429C812.90109@cox.net> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> Message-ID: On 3/28/06, Tim Hochberg wrote: > > Travis Oliphant wrote: > > > > > It look's like Tim's arguments have won out for the most part. > > Hooray. Now let's hope I'm right! +1. Now I'm nervous. > > > > Part of the reason for the change is that it was not disruptive. None > > of the NumPy code broke (except for a small piece of arrayprint) after > > the change. None of the SciPy code broke either (as far as I can tell). > > Hopefully we can put this to rest and move on to more important > issues... But what could be more important to safe programming than having a sure grasp of when you are working with a view or a copy? O_- Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Mar 28 17:02:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 17:02:01 2006 Subject: [Numpy-discussion] NetCDF module that links against NumPy Message-ID: <4429DC3D.1010000@ieee.org> I've adapted Konrad Hinsen's Scientific.IO.NetCDF module to work with NumPy. For now it's available at http://sourceforge.net/project/showfiles.php?group_id=1315&package_id=185504&release_id=405511 I would like to put this module in SciPy but don't understand whether or not the License is compatible. It sounds GPL-ish. Would it be O.K. to place in the SciPy/sandbox? Regards, -Travis From zpincus at stanford.edu Tue Mar 28 17:36:11 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 17:36:11 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> Message-ID: <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> Uh oh, Just updated to new numpy so that dot() wouldn't be broken. What fun to play with the new order/ravel/reshape stuff. Unfortunately, the ravel/reshape methods seem to not be quite worked out properly yet. Basically (and as expected) the ravel method doesn't work anymore on fortran-strided arrays. But since fortran- strided arrays happen *all-the-time* (e.g. after a transpose operation), things get confusing fast. In fact, absolutely no reshape operation (other than the identity reshape) will work on fortran-strided arrays (see below). Also, the errors could be more helpful, and anyway ravel shouldn't fail with an error about reshape fortran-strided. This can't be the optimal situation, since it makes the method interface next-to-useless (because exceptions might be thrown at any moment) unless a *lot* of care is taken to properly sanitize the arrays that are used (in which case any speed gains are probably lost). In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.7.2291' In [3]: a = numpy.array([[1,2],[3,4],[5,6]]) In [4]: b = a.transpose() In [5]: a.ravel() Out[5]: array([1, 2, 3, 4, 5, 6]) In [6]: b.ravel() ValueError: cannot return a view from reshape. In [7]: b.shape Out[7]: (2, 3) In [8]: b.reshape((1,6)) ValueError: cannot return a view from reshape. In [9]: b.reshape((6,1)) ValueError: cannot return a view from reshape. In [10]: b.reshape((3,2)) ValueError: cannot return a view from reshape. This state of affairs seems pretty hostile to both advanced users and basic ones. Now I need to go through all of my code and eliminate the use of array methods, since about half of the time I'll be calling things like ravel on arrays that have been transposed. What a waste -- is there something better I can do or something better numpy can do? At the very least, a strong convention really needs to be established and made clear about what the methods do (never return copies) and what the functions do (sometimes return copies? hardly a strong convention), and when to choose one or the other. Zach From zpincus at stanford.edu Tue Mar 28 17:55:05 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 17:55:05 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> Message-ID: <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Let me give a more concrete example, pulled from some code I was actually using. The code is obviously a bit flawed, but it worked fine with numpy up until today. Now it breaks when a and b aren't both c-strided: def distance_squared(a, b): return numpy.sum( (a.ravel() - b.ravel())**2 ) It appears that to be safe, a function like this must be rewritten as: def distance_squared(a, b): return numpy.sum( (numpy.ravel(a) - numpy.ravel(b))**2 ) At the very least, it seems surprising (to a novice especially, but to me too, and I don't consider myself a total newb at this) that for something as simple as this, that the ravel method might not work in some cases, especially since one of the exciting new features of numpy is (finally!) having useful array methods (makes life in ipython with tab completion much easier). Zach PS. I know that the "proper" solution would look more like the below. I guess the question is how tolerant should numpy be of code that deviates from "proper". def distance_squared(a, b): return ((a - b)**2).sum() On Mar 28, 2006, at 5:35 PM, Zachary Pincus wrote: > Uh oh, > > Just updated to new numpy so that dot() wouldn't be broken. What > fun to play with the new order/ravel/reshape stuff. > > Unfortunately, the ravel/reshape methods seem to not be quite > worked out properly yet. Basically (and as expected) the ravel > method doesn't work anymore on fortran-strided arrays. But since > fortran-strided arrays happen *all-the-time* (e.g. after a > transpose operation), things get confusing fast. > > In fact, absolutely no reshape operation (other than the identity > reshape) will work on fortran-strided arrays (see below). Also, the > errors could be more helpful, and anyway ravel shouldn't fail with > an error about reshape fortran-strided. > > This can't be the optimal situation, since it makes the method > interface next-to-useless (because exceptions might be thrown at > any moment) unless a *lot* of care is taken to properly sanitize > the arrays that are used (in which case any speed gains are > probably lost). > > In [1]: import numpy > In [2]: numpy.__version__ > Out[2]: '0.9.7.2291' > In [3]: a = numpy.array([[1,2],[3,4],[5,6]]) > In [4]: b = a.transpose() > In [5]: a.ravel() > Out[5]: array([1, 2, 3, 4, 5, 6]) > In [6]: b.ravel() > ValueError: cannot return a view from reshape. > In [7]: b.shape > Out[7]: (2, 3) > In [8]: b.reshape((1,6)) > ValueError: cannot return a view from reshape. > In [9]: b.reshape((6,1)) > ValueError: cannot return a view from reshape. > In [10]: b.reshape((3,2)) > ValueError: cannot return a view from reshape. > > This state of affairs seems pretty hostile to both advanced users > and basic ones. Now I need to go through all of my code and > eliminate the use of array methods, since about half of the time > I'll be calling things like ravel on arrays that have been > transposed. What a waste -- is there something better I can do or > something better numpy can do? > > At the very least, a strong convention really needs to be > established and made clear about what the methods do (never return > copies) and what the functions do (sometimes return copies? hardly > a strong convention), and when to choose one or the other. > > Zach > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Tue Mar 28 18:20:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 18:20:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> Message-ID: <4429EEB2.7000803@ieee.org> Zachary Pincus wrote: > Uh oh, > > Just updated to new numpy so that dot() wouldn't be broken. What fun > to play with the new order/ravel/reshape stuff. > > Unfortunately, the ravel/reshape methods seem to not be quite worked > out properly yet. Basically (and as expected) the ravel method doesn't > work anymore on fortran-strided arrays. But since fortran-strided > arrays happen *all-the-time* (e.g. after a transpose operation), > things get confusing fast. > You've demonstrated my biggest concern with adopting Tim's approach. Because you can very easily get arrays that are not C-contiguous, you end up with a stituation in which the reshape and ravel methods are not very useful in my mind. > In fact, absolutely no reshape operation (other than the identity > reshape) will work on fortran-strided arrays (see below). Also, the > errors could be more helpful, and anyway ravel shouldn't fail with an > error about reshape fortran-strided. There is one more problem with what I checked in. The PyArray_Ravel() function in C is now completely different and is used in _check_axis and therefore creates a segfault. So, I've decided to compromise. .reshape maintains it's view-only behavior .ravel() goes back to copy-if-necessary behavior (If you don't like it then use .reshape(-1) ). This should fix most of Zach's concerns and also fixes the segfault you get with rand(10,3).transpose().sum() -Travis From webb.sprague at gmail.com Tue Mar 28 18:28:08 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Tue Mar 28 18:28:08 2006 Subject: [Numpy-discussion] round_() problems Message-ID: Hi all, numpy.__version__ = 0.9.4 (After my svd question, I hope to follow up with something a tad less silly. Unfortunately, this a lot vaguer.) I am getting the following backtrace when I try to round a matrix, but I am not sure why (it works in other places...) Has anyone seen this error before, any explanation? Anyway to get more information for diagnosis? The function the traceback hits only sets up an html table of rounded matrix items, and it works for some matrices. There is a possibility the matrix I am feeding it is messed up, which is why I wanted to print it.... Thanks, feel free to cc me as I read the list as a digest. Begin traceback: (import numpy as N) File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table outStr += '' + str(N.round_(item,prec)) + '' File "/usr/lib/python2.4/site-packages/numpy/core/oldnumeric.py", line 461, in round_ a = absolute(multiply(a, 10.**decimals)) TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'numpy.ndarray' From zpincus at stanford.edu Tue Mar 28 18:44:03 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 18:44:03 2006 Subject: [Numpy-discussion] segfault with min() method on fortran-strided arrays. Message-ID: Hi, I found a segfault in numpy that seems to have to do with the new 'order' stuff. Steps to reproduce and gdb backtrace below. This is new since numpy 0.9.7.2262, which is what I updated from today. Zach In [1]: import numpy In [2]: a = numpy.array([[1,2],[3,4],[5,6]]) In [3]: b = numpy.transpose(a) In [4]: a.min() Out[4]: 1 In [5]: b.min() Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x0000000c _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/core/ src/arrayobject.c:3505 3505 *axis = PyArray_NDIM(temp)-1; (gdb) backtrace #0 _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/ core/src/arrayobject.c:3505 #1 0x01170594 in array_min (self=0x6776b0, args=0x139eba0, kwds=0x0) at numpy/core/src/multiarraymodule.c:2909 #2 0x0028b984 in PyEval_EvalFrame (f=0x1868c10) at Python/ceval.c:3558 [...python stack frames snipped...] (gdb) info locals newdim = { ptr = 0xbfffeaa8, len = 1 } val = {-1} temp = (PyObject *) 0x6776b0 n = 2 (gdb) info args arr = (PyArrayObject *) 0x76696577 axis = (int *) 0xbfffeb0c flags = 0 From zpincus at stanford.edu Tue Mar 28 19:06:05 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Tue Mar 28 19:06:05 2006 Subject: [Numpy-discussion] segfault with min() method on fortran-strided arrays. In-Reply-To: References: Message-ID: <8B8C9DFE-430F-41CD-B170-0545DE57A4F4@stanford.edu> Already fixed in svn -- thanks Travis. Zach On Mar 28, 2006, at 6:43 PM, Zachary Pincus wrote: > Hi, > > I found a segfault in numpy that seems to have to do with the new > 'order' stuff. Steps to reproduce and gdb backtrace below. This is > new since numpy 0.9.7.2262, which is what I updated from today. > > Zach > > > In [1]: import numpy > In [2]: a = numpy.array([[1,2],[3,4],[5,6]]) > In [3]: b = numpy.transpose(a) > In [4]: a.min() > Out[4]: 1 > In [5]: b.min() > > Program received signal EXC_BAD_ACCESS, Could not access memory. > Reason: KERN_PROTECTION_FAILURE at address: 0x0000000c > _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/ > core/src/arrayobject.c:3505 > 3505 *axis = PyArray_NDIM(temp)-1; > (gdb) backtrace > #0 _check_axis (arr=0x76696577, axis=0xbfffeb0c, flags=0) at numpy/ > core/src/arrayobject.c:3505 > #1 0x01170594 in array_min (self=0x6776b0, args=0x139eba0, > kwds=0x0) at numpy/core/src/multiarraymodule.c:2909 > #2 0x0028b984 in PyEval_EvalFrame (f=0x1868c10) at Python/ceval.c: > 3558 > [...python stack frames snipped...] > (gdb) info locals > newdim = { > ptr = 0xbfffeaa8, > len = 1 > } > val = {-1} > temp = (PyObject *) 0x6776b0 > n = 2 > (gdb) info args > arr = (PyArrayObject *) 0x76696577 > axis = (int *) 0xbfffeb0c > flags = 0 > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From pgmdevlist at mailcan.com Tue Mar 28 19:31:09 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Tue Mar 28 19:31:09 2006 Subject: [Numpy-discussion] round_() problems In-Reply-To: References: Message-ID: <200603282230.32210.pgmdevlist@mailcan.com> > File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table > outStr += '' + str(N.round_(item,prec)) + '' Have you tried to use around instead of round ? That way, I think you wouldn't be using oldnumeric and stick to ndarrays... From efiring at hawaii.edu Tue Mar 28 20:44:05 2006 From: efiring at hawaii.edu (Eric Firing) Date: Tue Mar 28 20:44:05 2006 Subject: [Numpy-discussion] NetCDF module that links against NumPy In-Reply-To: <4429DC3D.1010000@ieee.org> References: <4429DC3D.1010000@ieee.org> Message-ID: <442A1075.1030506@hawaii.edu> Travis, Thanks--that is an important addition. In case you or others are not already aware of it, below is the address of Jeff Whitaker's NetCDF4 module for numpy, with NetCDF3 read/write capability. It requires HDF5 as well as NetCDF4 so it is not the easiest module to install, but it is nice to know that NetCDF4 support is already in place for numpy, before NetCDF4 has even been released for general use! http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/python/netCDF4.html Eric Travis Oliphant wrote: > > I've adapted Konrad Hinsen's Scientific.IO.NetCDF module to work with > NumPy. > > For now it's available at > > http://sourceforge.net/project/showfiles.php?group_id=1315&package_id=185504&release_id=405511 > > > I would like to put this module in SciPy but don't understand whether or > not the License is compatible. It sounds GPL-ish. > > Would it be O.K. to place in the SciPy/sandbox? > > > Regards, > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From webb.sprague at gmail.com Tue Mar 28 21:47:03 2006 From: webb.sprague at gmail.com (Webb Sprague) Date: Tue Mar 28 21:47:03 2006 Subject: [Numpy-discussion] round_() problems Message-ID: > > > File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table > > outStr += '' + str(N.round_(item,prec)) + '' > > > Have you tried to use around instead of round ? That way, I think you wouldn't > be using oldnumeric and stick to ndarrays... Yes. Same result. From robert.kern at gmail.com Tue Mar 28 21:51:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 21:51:02 2006 Subject: [Numpy-discussion] Re: round_() problems In-Reply-To: References: Message-ID: Webb Sprague wrote: > Hi all, > > numpy.__version__ = 0.9.4 > > (After my svd question, I hope to follow up with something a tad less > silly. Unfortunately, this a lot vaguer.) > > I am getting the following backtrace when I try to round a matrix, but > I am not sure why (it works in other places...) Has anyone seen this > error before, any explanation? Anyway to get more information for > diagnosis? > > The function the traceback hits only sets up an html table of rounded > matrix items, and it works for some matrices. There is a possibility > the matrix I am feeding it is messed up, which is why I wanted to > print it.... > > Thanks, feel free to cc me as I read the list as a digest. > > Begin traceback: > > (import numpy as N) > > File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table > outStr += '' + str(N.round_(item,prec)) + '' > > File "/usr/lib/python2.4/site-packages/numpy/core/oldnumeric.py", > line 461, in round_ > a = absolute(multiply(a, 10.**decimals)) > > TypeError: unsupported operand type(s) for ** or pow(): 'float' and > 'numpy.ndarray' Well, I haven't found an input that reproduces the bug. Can you try to find a (small) input that does fail and post that? -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From baecker at physik.tu-dresden.de Tue Mar 28 21:54:02 2006 From: baecker at physik.tu-dresden.de (Arnd Baecker) Date: Tue Mar 28 21:54:02 2006 Subject: [Numpy-discussion] 64bit check_manyways segfault. Message-ID: Hi, with current svn I get a segfault for numpy.test(10, 10): check_manyways (numpy.lib.tests.test_arraysetops.test_aso)Segmentation fault There are no "incompatible pointer type" during the build of numpy. The backtrace is below. I hope this has not been reported during the last two days (I hardly manage to keep up with the traffic during the week ;-). Best, Arnd Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 46912507335168 (LWP 12055)] 0x00002aaaabaa4513 in PyArray_NewFromDescr (subtype=0x2aaaabbef320, descr=0x2aaaabbeba80, nd=1, dims=0x0, strides=0x0, data=0x0, flags=0, obj=0x0) at arrayobject.c:3969 3969 if (dims[i] < 0) { (gdb) bt #0 0x00002aaaabaa4513 in PyArray_NewFromDescr (subtype=0x2aaaabbef320, descr=0x2aaaabbeba80, nd=1, dims=0x0, strides=0x0, data=0x0, flags=0, obj=0x0) at arrayobject.c:3969 #1 0x00002aaaabaa6b2e in PyArray_Zeros (nd=1, dims=0x0, type=0x2aaaabad4c40, fortran_=0) at multiarraymodule.c:4679 #2 0x00002aaaabacebd5 in array_zeros (ignored=0x2aaaabbef320, args=0x2aaaabbef320, kwds=0x2aaaabad4c40) at multiarraymodule.c:4723 #3 0x0000000000478cfb in PyEval_EvalFrame (f=0x95dd30) at ceval.c:3558 #4 0x0000000000479fb1 in PyEval_EvalFrame (f=0x701c80) at ceval.c:3640 #5 0x0000000000479fb1 in PyEval_EvalFrame (f=0x716970) at ceval.c:3640 #6 0x0000000000479fb1 in PyEval_EvalFrame (f=0x7178c0) at ceval.c:3640 #7 0x0000000000479fb1 in PyEval_EvalFrame (f=0x6e4740) at ceval.c:3640 #8 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab948d50, globals=0x2aaaabad4c40, locals=0x0, args=0x6e4740, argcount=2, kws=0x91bad0, kwcount=0, defs=0x2aaaab950ce8, defcount=1, closure=0x0) at ceval.c:2736 #9 0x00000000004c6099 in function_call (func=0x2aaaab95e758, arg=0x2aaaae026758, kw=0x91b1e0) at funcobject.c:548 #10 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #11 0x00000000004772ea in PyEval_EvalFrame (f=0x6e7bb0) at ceval.c:3835 #12 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab948dc0, globals=0x2aaaabad4c40, locals=0x0, args=0x6e7bb0, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #13 0x00000000004c6099 in function_call (func=0x2aaaab95e7d0, arg=0x2aaaae0265f0, kw=0x0) at funcobject.c:548 #14 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #15 0x0000000000420ee0 in instancemethod_call (func=0x2aaaabbef320, arg=0x2aaaae0265f0, kw=0x0) at classobject.c:2447 #16 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #17 0x00000000004777d9 in PyEval_EvalFrame (f=0x80c070) at ceval.c:3766 #18 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab93a340, globals=0x2aaaabad4c40, locals=0x0, args=0x80c070, argcount=2, kws=0x0, kwcount=0, defs=0x2aaaab9607a8, defcount=1, closure=0x0) at ceval.c:2736 #19 0x00000000004c6099 in function_call (func=0x2aaaab961cf8, arg=0x2aaaae026878, kw=0x0) at funcobject.c:548 #20 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #21 0x0000000000420ee0 in instancemethod_call (func=0x2aaaabbef320, arg=0x2aaaae026878, kw=0x0) at classobject.c:2447 #22 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #23 0x000000000044fd80 in slot_tp_call (self=0x2aaaad97d890, args=0x2aaaae04e450, kwds=0x0) at typeobject.c:4536 #24 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #25 0x00000000004777d9 in PyEval_EvalFrame (f=0x739bd0) at ceval.c:3766 #26 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab94c810, globals=0x2aaaabad4c40, locals=0x0, args=0x739bd0, argcount=2, kws=0x8af160, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #27 0x00000000004c6099 in function_call (func=0x2aaaab95f050, arg=0x2aaaae029248, kw=0x91afa0) at funcobject.c:548 #28 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #29 0x00000000004772ea in PyEval_EvalFrame (f=0x7301d0) at ceval.c:3835 #30 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab94c880, globals=0x2aaaabad4c40, locals=0x0, args=0x7301d0, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #31 0x00000000004c6099 in function_call (func=0x2aaaab95f0c8, arg=0x2aaaae026d40, kw=0x0) at funcobject.c:548 #32 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #33 0x0000000000420ee0 in instancemethod_call (func=0x2aaaabbef320, arg=0x2aaaae026d40, kw=0x0) at classobject.c:2447 #34 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #35 0x000000000044fd80 in slot_tp_call (self=0x2aaaaca7acd0, args=0x2aaaae022c50, kwds=0x0) at typeobject.c:4536 #36 0x0000000000417700 in PyObject_Call (func=0x2aaaabbef320, arg=0x2aaaabad4c40, kw=0x0) at abstract.c:1756 #37 0x00000000004777d9 in PyEval_EvalFrame (f=0x7760a0) at ceval.c:3766 #38 0x0000000000479fb1 in PyEval_EvalFrame (f=0x6ec260) at ceval.c:3640 #39 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab93adc0, globals=0x2aaaabad4c40, locals=0x0, args=0x6ec260, argcount=3, kws=0x6d26d0, kwcount=0, defs=0x2aaaab963578, defcount=2, closure=0x0) at ceval.c:2736 #40 0x00000000004788f7 in PyEval_EvalFrame (f=0x6d2520) at ceval.c:3650 #41 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaaab23dc0, globals=0x2aaaabad4c40, locals=0x0, args=0x6d2520, argcount=2, kws=0x66c900, kwcount=0, defs=0x2aaaaca84e30, defcount=2, closure=0x0) at ceval.c:2736 #42 0x00000000004788f7 in PyEval_EvalFrame (f=0x66c760) at ceval.c:3650 ---Type to continue, or q to quit--- #43 0x000000000047ad2f in PyEval_EvalCodeEx (co=0x2aaaab948030, globals=0x2aaaabad4c40, locals=0x0, args=0x66c760, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ceval.c:2736 #44 0x000000000047af72 in PyEval_EvalCode (co=0x2aaaabbef320, globals=0x2aaaabad4c40, locals=0x0) at ceval.c:484 #45 0x00000000004a1c72 in PyRun_InteractiveOneFlags (fp=0x2aaaaab132f0, filename=0x4cbf24 "", flags=0x7fffff82fcfc) at pythonrun.c:1265 #46 0x00000000004a1e04 in PyRun_InteractiveLoopFlags (fp=0x2aaaab556b00, filename=0x4cbf24 "", flags=0x7fffff82fcfc) at pythonrun.c:695 #47 0x00000000004a2350 in PyRun_AnyFileExFlags (fp=0x2aaaab556b00, filename=0x2aaaabad4c40 "number of dimensions must be >=0", closeit=0, flags=0x7fffff82fcfc) at pythonrun.c:658 #48 0x0000000000410788 in Py_Main (argc=0, argv=0x7fffff832721) at main.c:484 #49 0x00002aaaab34d5aa in __libc_start_main () from /lib64/tls/libc.so.6 #50 0x000000000040fdfa in _start () at start.S:113 #51 0x00007fffff82fdf8 in ?? () #52 0x00002aaaaabc19c0 in rtld_errno () from /lib64/ld-linux-x86-64.so.2 #53 0x0000000000000001 in ?? () From charlesr.harris at gmail.com Tue Mar 28 22:01:03 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 22:01:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: On 3/28/06, Zachary Pincus wrote: > > Let me give a more concrete example, pulled from some code I was > actually using. > > The code is obviously a bit flawed, but it worked fine with numpy up > until today. Now it breaks when a and b aren't both c-strided: > def distance_squared(a, b): > return numpy.sum( (a.ravel() - b.ravel())**2 ) def distance_squared(a,b) return ((a - b)**2).sum() The function should probably throw an error if a and b don't have the same shape and order and now it does. I note that your original function was written for the old functional sum and attempts to be efficient. The new Numpy sum method make it much easier to do what you want. Thanks, Travis. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Tue Mar 28 22:03:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Mar 28 22:03:02 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <7F869A35-1EC7-41F5-BA60-FBF65F6937D5@stanford.edu> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> <7F869A35-1EC7-41F5-BA60-FBF65F6937D5@stanford.edu> Message-ID: <442A22F0.7080506@ieee.org> Zachary Pincus wrote: >>> Unfortunately, the ravel/reshape methods seem to not be quite worked >>> out properly yet. Basically (and as expected) the ravel method >>> doesn't work anymore on fortran-strided arrays. But since >>> fortran-strided arrays happen *all-the-time* (e.g. after a transpose >>> operation), things get confusing fast. >>> > > > One question: was the issue in my example code where fortran-strided > arrays couldn't be reshaped at all a feature or a bug in the reshape > method? It's now a feature. Reshape will only work if it can produce a view of the array. Actually, I can't claim that I've thoroughly analyzed every possibility to determine if a view is possible. I'm sure there are cases where a view is technically feasible. The simple rules are: 1) If you are only adding or deleting ones from the shape, then it always succeeds. 2) If you aren't actually changing the shape it always succeeds (a -1 present means you are changing the shape) Other than that it only succeeds if you are reshaping a C-contiguous array using C-ordering or you are reshaping a Fortran-contiguous array using Fortran-ordering. -Travis From robert.kern at gmail.com Tue Mar 28 22:11:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Mar 28 22:11:01 2006 Subject: [Numpy-discussion] Re: reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: Charles R Harris wrote: > > > On 3/28/06, *Zachary Pincus* > wrote: > > Let me give a more concrete example, pulled from some code I was > actually using. > > The code is obviously a bit flawed, but it worked fine with numpy up > until today. Now it breaks when a and b aren't both c-strided: > def distance_squared(a, b): > return numpy.sum( (a.ravel() - b.ravel())**2 ) > > def distance_squared(a,b) > return ((a - b)**2).sum() > > The function should probably throw an error if a and b don't have the > same shape and order and now it does. Does it? In [55]: def distance_squared(a, b): ....: return ((a - b) ** 2).sum() ....: In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) Out[56]: 55 -- Robert Kern robert.kern at gmail.com "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tim.hochberg at cox.net Tue Mar 28 22:27:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Mar 28 22:27:05 2006 Subject: [Numpy-discussion] Re: reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: <442A2880.90409@cox.net> Robert Kern wrote: >Charles R Harris wrote: > > >>On 3/28/06, *Zachary Pincus* >> wrote: >> >> Let me give a more concrete example, pulled from some code I was >> actually using. >> >> The code is obviously a bit flawed, but it worked fine with numpy up >> until today. Now it breaks when a and b aren't both c-strided: >> def distance_squared(a, b): >> return numpy.sum( (a.ravel() - b.ravel())**2 ) >> >>def distance_squared(a,b) >> return ((a - b)**2).sum() >> >>The function should probably throw an error if a and b don't have the >>same shape and order and now it does. >> >> > >Does it? > >In [55]: def distance_squared(a, b): > ....: return ((a - b) ** 2).sum() > ....: > >In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) >Out[56]: 55 > > Right. It will work with any two arrays that can be broadcast to the same shape. Given numpy's rules, that's arguably the right thing to do in general although whether it's the right thing to do here depends on the specific application, which I don't know. Still, this is almost certainly better than the original case, which will blithely compute the distance_squared for any two arrays with the same total size. Then again, it's possible that's what the author really wanted. Unlikely, I think, but possible. Regards, -tim From charlesr.harris at gmail.com Tue Mar 28 22:34:03 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 22:34:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429EEB2.7000803@ieee.org> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> Message-ID: On 3/28/06, Travis Oliphant wrote: > > > rand(10,3).transpose().sum() Shouldn't sum be fixed instead? I admit that a quick look at the code leaves me no more enlightened. It is going to take more than a little time to figure out how numpy is put together. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnd.baecker at web.de Tue Mar 28 22:44:06 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Tue Mar 28 22:44:06 2006 Subject: [Numpy-discussion] 64bit check_manyways segfault. In-Reply-To: <442A26BF.7080008@ieee.org> References: <442A26BF.7080008@ieee.org> Message-ID: On Tue, 28 Mar 2006, Travis Oliphant wrote: > Arnd Baecker wrote: > > Hi, > > > > with current svn I get a segfault for numpy.test(10, 10): > > > > check_manyways (numpy.lib.tests.test_arraysetops.test_aso)Segmentation > > fault > > > Thanks for the test. Nobody had caught that one yet. > > Should be fixed now in SVN. That was quick, indeed! Works fine now. So presently the only problems on 64 Bit are the ndimage ones for scipy - I still have found no time (and won't during the next two weeks at least) to look into that in a bit more detail - sorry... Best, Arnd From michael.sorich at gmail.com Tue Mar 28 22:47:04 2006 From: michael.sorich at gmail.com (Michael Sorich) Date: Tue Mar 28 22:47:04 2006 Subject: [Numpy-discussion] stringscalar comparisons with a chararray Message-ID: <16761e100603282240h51c9774ax8e51eafe9fc02058@mail.gmail.com> As shown in the example below, chararray comparisons do not seem to work well when stringscalars are used. In the example below I am also curious as to whether the the length of c[0] should be 2 instead of 5. numpy.__version__ 0.9.6 >>> from numpy import * >>> c =char.array(['on','two','three']) >>> c chararray([on, two, three], dtype='|S5') >>> c=='on' array([True, False, False], dtype=bool) >>> c==c[0] array([False, False, False], dtype=bool) >>> c==str(c[0]) array([True, False, False], dtype=bool) >>> c[0] 'on' >>> len(c[0]) 5 >>> c.dtype dtype('|S5') >>> c==c[2] array([False, False, True], dtype=bool) -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Mar 28 22:49:05 2006 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue Mar 28 22:49:05 2006 Subject: [Numpy-discussion] Re: reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <47D9A4B7-947E-4878-8675-DB9675840ED8@stanford.edu> Message-ID: Robert, On 3/28/06, Robert Kern wrote: > > Charles R Harris wrote: > > > > > > On 3/28/06, *Zachary Pincus* > > wrote: > > > > Let me give a more concrete example, pulled from some code I was > > actually using. > > > > The code is obviously a bit flawed, but it worked fine with numpy up > > until today. Now it breaks when a and b aren't both c-strided: > > def distance_squared(a, b): > > return numpy.sum( (a.ravel() - b.ravel())**2 ) > > > > def distance_squared(a,b) > > return ((a - b)**2).sum() > > > > The function should probably throw an error if a and b don't have the > > same shape and order and now it does. > > Does it? > > In [55]: def distance_squared(a, b): > ....: return ((a - b) ** 2).sum() > ....: > > In [56]: distance_squared(arange(5), arange(3)[:,newaxis]) > Out[56]: 55 Oops! Guess there should be a test for shape equality in there somewhere. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Wed Mar 29 00:01:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Mar 29 00:01:08 2006 Subject: [Numpy-discussion] getting data pointer to numpy object in Pyrex In-Reply-To: <4429B89E.30100@bryant.edu> References: <4429B89E.30100@bryant.edu> Message-ID: <200603291000.10625.faltet@carabos.com> A Dimecres 29 Mar? 2006 00:28, Brian Blais va escriure: > ...but I would like to do something like... > > #------------------------------- > def test2(d): # pass a python dict > > w=d['w'] # get the numpy object > > cdef double *w_p > > # get the pointers > w_p=w.data > #------------------------------- > > > Is there a way to do this? The next works for any numerical package that supports the array protocol: data_address = int(w.__array_data__[0], 16) w_p = PyInt_AsLong(data_address) and declare PyInt_AsLong as: cdef extern from "Python.h": long PyInt_AsLong(object) It is important to note that the above solution does not require to compile against the numpy (or any other numerical package) headers at all. See "A Pyrex Agnostic Class" recipe in the SciPy cookbook (http://www.scipy.org/Cookbook) for a more complete example. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From georgi at molgen.mpg.de Wed Mar 29 01:51:17 2006 From: georgi at molgen.mpg.de (Benjamin Georgi) Date: Wed Mar 29 01:51:17 2006 Subject: [Numpy-discussion] verbosity control for warnings Message-ID: Hi, my application frequently requires to take logarithms of zero, i.e. >>> import numarray >>> numarray.log(0.0) Warning: Encountered divide by zero(s) in log -inf Now, while the result of -inf works just fine, any direct application of numarray.log produces tons of warnings. Right now I'm masking the input arrays to avoid the warnings, which is rather inefficient. So, the question is whether there is some kind of verbosity control I could use to deactivate these warnings. If there is no such functionality, I would like to suggest looking into the build-in Python logging package for that purpose. Best, Benjamin From bblais at bryant.edu Wed Mar 29 05:44:29 2006 From: bblais at bryant.edu (Brian Blais) Date: Wed Mar 29 05:44:29 2006 Subject: [Numpy-discussion] Re: getting data pointer to numpy object in Pyrex In-Reply-To: References: <4429B89E.30100@bryant.edu> Message-ID: <442A8E61.7090605@bryant.edu> Robert Kern wrote: > Brian Blais wrote: >> Is here an easy way, in Pyrex, to get the data pointer to a numpy object >> which is *not* passed as a parameter? For example, I know I can do: >> > You will probably want to use the c_numpy.pxd file in numpy/doc/pyrex/. It has > exposed a bit more of the API than you have. If you have any contributions, we > would like to fold them in. > > In general, you would just make a cast like you would for any other C type. > However, you must be sure that the object actually is an array. I think the > following code would work as expected, including the case where d['w'] can't be > turned into an array, but I have not tested it. > > > > cimport c_numpy > import numpy > > c_numpy.import_array() > > def test2(d): > cdef c_numpy.ndarray w > > w = numpy.asarray(d['w']) > # ... > > > I tried this, and got some warnings, and the code didn't work. I get the compile-time warning of: warning: use of cast expressions as lvalues is deprecated (along with all of the other, normal, Pyrex compile warnings) I am including the code here: #test3.pyx cimport c_numpy import numpy c_numpy.import_array() cdef extern from "numpy/arrayobject.h": struct PyArray_Descr: int type_num, elsize char type ctypedef class numpy.ArrayType [object PyArrayObject]: cdef char *data cdef int nd cdef int *dimensions, *strides cdef object base cdef PyArray_Descr *descr cdef int flags def fun1(d): cdef ArrayType W cdef double *w cdef int i,l W=d['w'] w=W.data l=W.dimensions[0]*W.dimensions[1] for i from 0 <= i < l: print "%.4e" % w[i] def fun2(d): cdef c_numpy.ndarray W cdef double *w cdef int i,l W = numpy.asarray(d['w']) w=W.data l=W.dimensions[0]*W.dimensions[1] for i from 0 <= i < l: print "%.4e" % w[i] #================================================= #run_test3.py import numpy import test3 w=numpy.arange(0,2,.5) d={"w":w} test3.fun1(d) print "===================" test3.fun2(d) #=================================================== #output from program: >>> execfile('run_test3.py') 0.0000e+00 5.0000e-01 1.0000e+00 1.5000e+00 8.3394e-312 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 1.9354e+01 5.3216e-315 0.0000e+00 2.1127e+01 5.3428e-315 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 -5.1923e+00 2.5548e+02 =================== 0.0000e+00 5.0000e-01 1.0000e+00 1.5000e+00 8.3394e-312 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 1.9354e+01 5.3216e-315 0.0000e+00 2.1127e+01 5.3428e-315 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 0.0000e+00 -5.1923e+00 2.5548e+02 -- ----------------- bblais at bryant.edu http://web.bryant.edu/~bblais From perry at stsci.edu Wed Mar 29 06:01:10 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Mar 29 06:01:10 2006 Subject: [Numpy-discussion] verbosity control for warnings In-Reply-To: References: Message-ID: <00d31387aa3cf09970ef1d4517dcb52d@stsci.edu> On Mar 29, 2006, at 4:50 AM, Benjamin Georgi wrote: > > Hi, > > my application frequently requires to take logarithms of zero, i.e. > >>>> import numarray >>>> numarray.log(0.0) > Warning: Encountered divide by zero(s) in log > -inf > > Now, while the result of -inf works just fine, any direct application > of numarray.log produces tons of warnings. Right now I'm masking the > input arrays to avoid the warnings, which is rather inefficient. > > So, the question is whether there is some kind of verbosity control I > could use to deactivate these warnings. If there is no such > functionality, I would like to suggest looking into the build-in > Python logging package for that purpose. > Yes, look at section 6.2 of the numarray manual. Perry Greenfield From benjamin at decideur.info Wed Mar 29 06:20:23 2006 From: benjamin at decideur.info (Benjamin Thyreau) Date: Wed Mar 29 06:20:23 2006 Subject: [Numpy-discussion] distutils.misc_util Configuration error Message-ID: <200603291418.k2TEILOG018907@decideur.info> Hi, when trying to create an distutils.misc_util.Configuration instance, since numpy 0.9.5, i got an Python 2.4.1 (#1, May 2 2005, 15:06:50) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy.distutils.misc_util >>> numpy.distutils.misc_util.Configuration() Traceback (most recent call last): File "", line 1, in ? File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 450, in __init__ f = get_frame(i) File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 359, in get_frame return sys._getframe(level+1) ValueError: call stack is not deep enough Error. Strangely enough (or not), when running from ipython, it seems to work. On jan 31th 2006, pearu commited: for i in range(1,3): f = get_frame(i) try: caller_instance = eval('self',f.f_globals,f.f_locals) break except NameError: caller_instance = None if isinstance(caller_instance, self.__class__): if caller_instance.options['delegate_options_to_subpackages']: self.set_options(**caller_instance.options) on misc_util.py, at the end of the Configuration constructor. I don't exactly understand what that code do, but it prevent my setup.py to run since then. Thanks for your help -- Benjamin Thyreau decideur.info From pearu at scipy.org Wed Mar 29 06:33:05 2006 From: pearu at scipy.org (Pearu Peterson) Date: Wed Mar 29 06:33:05 2006 Subject: [Numpy-discussion] distutils.misc_util Configuration error In-Reply-To: <200603291418.k2TEILOG018907@decideur.info> References: <200603291418.k2TEILOG018907@decideur.info> Message-ID: On Wed, 29 Mar 2006, Benjamin Thyreau wrote: > Hi, > when trying to create an distutils.misc_util.Configuration instance, since > numpy 0.9.5, i got an > > Python 2.4.1 (#1, May 2 2005, 15:06:50) > [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import numpy.distutils.misc_util >>>> numpy.distutils.misc_util.Configuration() > Traceback (most recent call last): > File "", line 1, in ? > File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 450, in __init__ > f = get_frame(i) > File "/somewhere/numpy-0.9.6/build/lib.linux-i686-2.4/numpy/distutils/misc_util.py", line 359, in get_frame > return sys._getframe(level+1) > ValueError: call stack is not deep enough Fixed in numpy svn. Thanks, Pearu From cjw at sympatico.ca Wed Mar 29 06:49:10 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 29 06:49:10 2006 Subject: [Numpy-discussion] round_() problems and arange incompatibility In-Reply-To: <200603282230.32210.pgmdevlist@mailcan.com> References: <200603282230.32210.pgmdevlist@mailcan.com> Message-ID: <442A9E4C.8060403@sympatico.ca> Pierre GM wrote: >> File "/var/www/localhost/htdocs/larry/LcUtil.py", line 120, in mat2table >> outStr += '' + str(N.round_(item,prec)) + '' >> >> > > >Have you tried to use around instead of round ? That way, I think you wouldn't >be using oldnumeric and stick to ndarrays... > > > "round" seems a better word to use than "around". Would it not be better to have some sort of compatibility switch for those who wish to use names with their old meanings, rather than prepending "a" to indicate the new way? Incidentally, particularly for testing, I found the shape parameter in arange to be useful: [Dbg]>>> import numarray as _na [Dbg]>>> help(_na.arange) Help on function arange in module numarray.numarraycore: arange(a1, a2=None, stride=1, type=None, shape=None, typecode=None, dtype=None) Returns an array of numbers in sequence over the specified range. With numpy, we have: [Dbg]>>> a= _n.arange(6, shape=(3, 2), dtype= _n.ArrayType) Traceback (most recent call last): File "", line 1, in ? TypeError: 'shape' is an invalid keyword argument for this function [Dbg]>>> help(_n.arange) Help on built-in function arange in module numpy.core.multiarray: arange(...) arange([start,] stop[, step,], dtype=None) For integer arguments, just like range() except it returns an array whose type can be specified by the keyword argument dtype. If dtype is not specified, the type of the result is deduced from the type of the arguments. For floating point arguments, the length of the result is ceil((stop - start)/step). This rule may result in the last element of the result be greater than stop. Colin W. From zingale at gmail.com Wed Mar 29 07:02:07 2006 From: zingale at gmail.com (Mike Zingale) Date: Wed Mar 29 07:02:07 2006 Subject: [Numpy-discussion] numarray 1.5.1 self test failure Message-ID: <885402e30603290700s2b85e98aucc16eb5f9034111@mail.gmail.com> Hi, I just installed numarray 1.5.1 on my Fedora Core 4 box (python version 2.4.1). Right afterwards, I did: import numarray.testall as testall testall.test() at the python prompt, and I got 1 failure. The full output is below. Is this a known failure? My machine is a dual Xeon, and I am running the 32bit version of Fedora. Mike >>> import numarray.testall as testall >>> testall.test() Testing numarray 1.5.1 on normal Python (2, 4, 1, 'final', 0) on platform linux2 with builtin linear algebra support numarray.numtest: 2.41 ((0, 1208), (0, 1208)) numarray.ieeespecial: 0.06 (0, 86) numarray.records: 0.05 (0, 48) numarray.strings: 0.16 (0, 189) numarray.memmap: 0.07 (0, 82) numarray.objects: 0.12 (0, 105) numarray.memorytest: 0.01 (0, 16) numarray.examples.convolve: 0.09 ((0, 20), (0, 20), (0, 20), (0, 20)) numarray.convolve: 0.07 (0, 45) numarray.fft: 0.13 (0, 75) numarray.linear_algebra: 1.55 ((0, 46), (0, 51)) numarray.image: 0.05 (0, 37) numarray.nd_image: 2.76 (0, 397) numarray.random_array: 0.10 (0, 53) numarray.ma: 1.42 (0, 674) numarray.matrix: 0.03 (0, 11) ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 32, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na, copy=1) num2 = Numeric.array([1,2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 49, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na, copy=0) num2 = Numeric.array([1,2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 66, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[::2], copy=0) num2 = Numeric.array([1,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 83, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[::2], copy=1) num2 = Numeric.array([1,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 100, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[1:], copy=1) num2 = Numeric.array([2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 117, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: na = numarray.array([1,2,3], typecode) num = Numeric.zeros(shape=2, typecode=typecode) num = Numeric.array(na[1:], copy=0) num2 = Numeric.array([2,3], typecode) print typecode, num == num2, int(num.typecode() == num2.typecode()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 208, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num, copy=1) nb = numarray.array([1,2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 i [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 i [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 226, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num, copy=0) nb = numarray.array([1,2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1 1] 1 1 [1 1 1] 1 s [1 1 1] 1 l [1 1 1] 1 f [1 1 1] 1 d [1 1 1] 1 F [1 1 1] 1 D [1 1 1] 1 i [1 1 1] 1 Got: b [1 1 1] 0 1 [1 1 1] 0 s [1 1 1] 0 l [1 1 1] 1 f [1 1 1] 0 d [1 1 1] 1 F [1 1 1] 0 D [1 1 1] 1 i [1 1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 244, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[::2], copy=1) nb = numarray.array([1,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 262, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[::2], copy=0) nb = numarray.array([1,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 280, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[1:], copy=1) nb = numarray.array([2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", line 298, in numarray.array_protocol.test_Numeric Failed example: for typecode in typecodes: num = Numeric.array([1,2,3], typecode) na = numarray.zeros(shape=2, typecode=typecode) na = numarray.array(num[1:], copy=0) nb = numarray.array([2,3], typecode) print typecode, na == nb, int(na.type() == nb.type()) Expected: b [1 1] 1 1 [1 1] 1 s [1 1] 1 l [1 1] 1 f [1 1] 1 d [1 1] 1 F [1 1] 1 D [1 1] 1 i [1 1] 1 Got: b [1 1] 0 1 [1 1] 0 s [1 1] 0 l [1 1] 1 f [1 1] 0 d [1 1] 1 F [1 1] 0 D [1 1] 1 i [1 1] 1 ********************************************************************** 1 items had failures: 12 of 18 in numarray.array_protocol.test_Numeric ***Test Failed*** 12 failures. numarray.array_protocol: 0.09 (12, 18) numarray.dtype: 0.02 (0, 15) Total time: 9.27 > From jmiller at stsci.edu Wed Mar 29 07:11:10 2006 From: jmiller at stsci.edu (Todd Miller) Date: Wed Mar 29 07:11:10 2006 Subject: [Numpy-discussion] numarray 1.5.1 self test failure In-Reply-To: <885402e30603290700s2b85e98aucc16eb5f9034111@mail.gmail.com> References: <885402e30603290700s2b85e98aucc16eb5f9034111@mail.gmail.com> Message-ID: <442AA36D.3060505@stsci.edu> Hi Mike, The portion of the numarray self-test that is failing requires Numeric-24.2 to succeed because it is testing the new "array interface" which did not exist until 24.2. Without 24.2, numarray sees Numeric arrays as more general Python sequences and therefore casts to the default array type for that Python scalar kind. This results in type mismatches. Regards, Todd Mike Zingale wrote: > Hi, I just installed numarray 1.5.1 on my Fedora Core 4 box (python > version 2.4.1). Right afterwards, I did: > > import numarray.testall as testall > testall.test() > > at the python prompt, and I got 1 failure. The full output is below. > > Is this a known failure? My machine is a dual Xeon, and I am running > the 32bit version of Fedora. > > Mike > > > >>>> import numarray.testall as testall >>>> testall.test() >>>> > Testing numarray 1.5.1 on normal Python (2, 4, 1, 'final', 0) on > platform linux2 with builtin linear algebra support > numarray.numtest: 2.41 ((0, 1208), (0, 1208)) > numarray.ieeespecial: 0.06 (0, 86) > numarray.records: 0.05 (0, 48) > numarray.strings: 0.16 (0, 189) > numarray.memmap: 0.07 (0, 82) > numarray.objects: 0.12 (0, 105) > numarray.memorytest: 0.01 (0, 16) > numarray.examples.convolve: 0.09 ((0, 20), (0, 20), > (0, 20), (0, 20)) > numarray.convolve: 0.07 (0, 45) > numarray.fft: 0.13 (0, 75) > numarray.linear_algebra: 1.55 ((0, 46), (0, 51)) > numarray.image: 0.05 (0, 37) > numarray.nd_image: 2.76 (0, 397) > numarray.random_array: 0.10 (0, 53) > numarray.ma: 1.42 (0, 674) > numarray.matrix: 0.03 (0, 11) > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 32, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na, copy=1) > num2 = Numeric.array([1,2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 49, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na, copy=0) > num2 = Numeric.array([1,2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 66, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[::2], copy=0) > num2 = Numeric.array([1,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 83, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[::2], copy=1) > num2 = Numeric.array([1,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 100, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[1:], copy=1) > num2 = Numeric.array([2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 117, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > na = numarray.array([1,2,3], typecode) > num = Numeric.zeros(shape=2, typecode=typecode) > num = Numeric.array(na[1:], copy=0) > num2 = Numeric.array([2,3], typecode) > print typecode, num == num2, int(num.typecode() == num2.typecode()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 208, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num, copy=1) > nb = numarray.array([1,2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > i [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > i [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 226, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num, copy=0) > nb = numarray.array([1,2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1 1] 1 > 1 [1 1 1] 1 > s [1 1 1] 1 > l [1 1 1] 1 > f [1 1 1] 1 > d [1 1 1] 1 > F [1 1 1] 1 > D [1 1 1] 1 > i [1 1 1] 1 > Got: > b [1 1 1] 0 > 1 [1 1 1] 0 > s [1 1 1] 0 > l [1 1 1] 1 > f [1 1 1] 0 > d [1 1 1] 1 > F [1 1 1] 0 > D [1 1 1] 1 > i [1 1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 244, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[::2], copy=1) > nb = numarray.array([1,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 262, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[::2], copy=0) > nb = numarray.array([1,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 280, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[1:], copy=1) > nb = numarray.array([2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > File "/usr/lib/python2.4/site-packages/numarray/array_protocol.py", > line 298, in numarray.array_protocol.test_Numeric > Failed example: > for typecode in typecodes: > num = Numeric.array([1,2,3], typecode) > na = numarray.zeros(shape=2, typecode=typecode) > na = numarray.array(num[1:], copy=0) > nb = numarray.array([2,3], typecode) > print typecode, na == nb, int(na.type() == nb.type()) > Expected: > b [1 1] 1 > 1 [1 1] 1 > s [1 1] 1 > l [1 1] 1 > f [1 1] 1 > d [1 1] 1 > F [1 1] 1 > D [1 1] 1 > i [1 1] 1 > Got: > b [1 1] 0 > 1 [1 1] 0 > s [1 1] 0 > l [1 1] 1 > f [1 1] 0 > d [1 1] 1 > F [1 1] 0 > D [1 1] 1 > i [1 1] 1 > ********************************************************************** > 1 items had failures: > 12 of 18 in numarray.array_protocol.test_Numeric > ***Test Failed*** 12 failures. > numarray.array_protocol: 0.09 (12, 18) > numarray.dtype: 0.02 (0, 15) > Total time: 9.27 > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=k&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Wed Mar 29 10:11:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 29 10:11:07 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <442A2572.9000800@cox.net> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> <442A2572.9000800@cox.net> Message-ID: <442ACD95.60101@ieee.org> Tim Hochberg wrote: > > Hi Travis, > > Ouch! > > In a sense, the problems that Zachary is having illustrates both of > our points perfectly: he's now put in a position of going through a > bunch of needless pain. On the other hand, the implicit copying has > resulted in a bunch of code that does a bunch of uneeded copies, > probably without the original author knowing it, resulting in both > speed an memory inefficiency. Yeah. I really see your point. I'm just uncomfortable breaking people's code too much. A sub-class is a good idea. It's a good place to experiment with new ideas. It's one of the reasons we wanted the array to be sub-classable. Making a sub-class a super-class is also possible once it's working. Best, -Travis From oliphant.travis at ieee.org Wed Mar 29 10:12:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Mar 29 10:12:11 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> Message-ID: <442ACDD1.7060501@ieee.org> Charles R Harris wrote: > > > On 3/28/06, *Travis Oliphant* > wrote: > > > rand(10,3).transpose().sum() > > > Shouldn't sum be fixed instead? I admit that a quick look at the code > leaves me no more enlightened. It is going to take more than a little > time to figure out how numpy is put together. > FWIW: I also fixed the _check_axis code (it was ignoring errors and shouldn't have been). -Travis From Chris.Barker at noaa.gov Wed Mar 29 12:11:12 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed Mar 29 12:11:12 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <4429EEB2.7000803@ieee.org> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> Message-ID: <442AE9AA.2090007@noaa.gov> >> Since fortran-strided >> arrays happen *all-the-time* (e.g. after a transpose operation), I have to wonder if it's really a good idea to do this. Why not just make a re-ordered copy on transpose, rather than making it Fortran strided? Is the optimization really that important? I had always thought of Fortran strided arrays as being a special case exception (for working with external Fortran codes), rather than something that happens all the time. I suppose if all these issues are worked out, it'd be transparent to me, but it strikes me as more complication that necessary. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Wed Mar 29 13:14:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Mar 29 13:14:03 2006 Subject: [Numpy-discussion] reshape and ravel methods of arrays now return views or raise error In-Reply-To: <442AE9AA.2090007@noaa.gov> References: <4429AC89.6090301@ieee.org> <4429C812.90109@cox.net> <332D7D11-CB23-4075-8881-DE29EDD8D21E@stanford.edu> <4429EEB2.7000803@ieee.org> <442AE9AA.2090007@noaa.gov> Message-ID: <442AF853.7050509@ee.byu.edu> Christopher Barker wrote: >>> Since fortran-strided arrays happen *all-the-time* (e.g. after a >>> transpose operation), >> > > I have to wonder if it's really a good idea to do this. Why not just > make a re-ordered copy on transpose, rather than making it Fortran > strided? Is the optimization really that important? And optimization is important. The _dotblas.c function for example now works much faster with very common transpose operations because the underlying blas knows how to deal with striding so it was a complete wasted to always use C-ordering and get a copy of a Fortran-strided array. On large arrays, avoiding the unnecessary copy can be very significant. -Travis From david.huard at gmail.com Wed Mar 29 14:21:21 2006 From: david.huard at gmail.com (David Huard) Date: Wed Mar 29 14:21:21 2006 Subject: [Numpy-discussion] Freezing numpy code Message-ID: <91cf711d0603291420g7068c84y623b43d36a83c850@mail.gmail.com> Hi, I would like to freeze some code in order to run it on machines without installing all the modules I use. I compiled Python 2.4.3 from source and rebuilt numpy. When I try to freeze the following code : #test.py from numpy import mean, rand x = rand(1000) print mean(x) with python /usr/share/doc/python2.4/examples/Tools/freeze/freeze.py test.py make I get the following error message : No scipy-style subpackage 'testing' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'core' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'lib' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'linalg' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'dft' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'random' found in n:u:m:p:y. Ignoring. No scipy-style subpackage 'f2py' found in n:u:m:p:y. Ignoring. Traceback (most recent call last): File "test.py", line 1, in ? from numpy import mean, rand File "/usr/lib/python2.4/site-packages/numpy/__init__.py", line 44, in ? __doc__ += pkgload.get_pkgdocs() File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 320, in get_pkgdocs retstr = self._format_titles(titles) +\ File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 283, in _format_titles max_length = max(lengths) ValueError: max() arg is an empty sequence Does anybody know what I'm missing ? I'm using the latest svn. Thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From zpincus at stanford.edu Wed Mar 29 16:04:05 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Wed Mar 29 16:04:05 2006 Subject: [Numpy-discussion] numpy.linalg: eig versus eigh? Message-ID: <5CFF089A-5B17-4B4F-83BA-2B75CC2E7BDF@stanford.edu> Can anyone tell me what the difference between numpy.linalg.eig and numpy.linalg.eigh is? (Ditto for eigvals and eigvalsh.) Thanks, Zach From aeronautic at hrvatskamail.com Wed Mar 29 19:22:07 2006 From: aeronautic at hrvatskamail.com (=?windows-1251?B?NS02IODv8OXr/w==?=) Date: Wed Mar 29 19:22:07 2006 Subject: [Numpy-discussion] =?windows-1251?B?ysDE0M7CzsUgxMXLzs/QzsjHws7E0dLCzg==?= Message-ID: <8de501c652a1$0761d935$c92b9c86@hrvatskamail.com> ?????????? ??? ? ????? ?????? ??????? ??????? ? ???????????? ?????????: 5-6 ?????? ???????? ???????????????? ? ??????????? ???????????:???????????? ? ?????? ? ????. 6 ?????? - ??????????? ? ????????????? ? ?? ??????????? ??????? ???? . ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. 232-2133 ???????? ???????????????? ? ??????????? ???????????:???????????? ? ?????? ? ????. 5-6 ?????? 2006 ???? 1. ?????? ???????????? ??????????? ???????? ?????? ???????? ?????? ?????????????. ????? ?? ???????? ???? ??????? 2. ???????? ????????????????, ??? ??????????? ??????????. ?????????, ???????????? ??? ??????? ? ???????????? ?????????? ?? ??????????????-???????? ?????: ?????? ???????? ???????????? ??????????. ???? ????????? ?? ???????, ??? ??? ????? ???????? ??? ????? ????????? ???? ??????????? ??????????????? ????????. ??? ????? ???????? ?? ?????????? ?????????? 3. ??????????? ???????????????? ? ????. ?????? ???????? ?????????? ??????? ????????? ???????????????? ? ????????. ?????????? ??????????????? ???????: ???????????? ???????? ????????. 4. ?????????? ?????????? ????????? ?????, ???????????? ?????? ????????. ??????? ??????????? ????????? ??????????, ??????????? ??????????, ????????? ? ?????? ???????????? ??????, ?????????? ?? ?????? ????? ? ??. ???????????? ?????????? ? ??????????? ?????????? ???????????? ?????? 5. ??? ????????? ??????? ?????? ? ???????? ???????????? ????? ???????, ????? ??? ??? ??????????? ???????? ?????????? ? ?????? ???? ????????????? ???????? ???????. ??????? ???????. ??????? ???????. ???????????????? ????????????? ?????????????? ?????. 6. ???????? ?????????? ???????? ?????????? ??? ???????? ?????????. ?????????? ?????????? ??? ??????, ????????, ???????, ??????????. ?????????? ?????????????? ???????????? ? ??. ????????????? ??????? ??????????????? ????. ???????????????? ?????????????? ????????? ?????????. 7. ??????????? ?????????? ??????. ???? ??????????? ??? ????? ??????? ?????????????, ? ????????? ?? ??????, ??? ???? ???????? ?????-?? ?????????, ?? ???, ????? ?????? ?????? ???????????? 8. ?????????????? ???????????????? ? ?????? ?? ?????????? ??????? ??????????. ????? ?????????? ???????? ? 10.00 ?? 17.00. ? ????????? ??????: ????, ???????????? ?????????, + 3 ??? ???????????????? ?????????. ????????? 9500 ???. + ???. ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. 232-2133 ?????????? ? ?????????????? ?? ??????????? ??????? ????6 ?????? 2006 ???? 1. ????? ???????????????? ? ????? ??????????? ??????????? ??????? ????. 2. ??????????????? ???????????? ?? ????????? ???????????? ????????????????. ???????????????? ??????????????? ??????????? ???????????????? ??????????????? ??????????? ??? ???????????????? ??????????????? ?????????? ??? ?????? ??????????? ????????? 3. ???????? ????????? ??????????? ?????????? ? ?? ??????? ??????-?????? ??????????? ???? ? ?????????? ??????????????? ??????????? ????????? ???????????????? ?? ??????????? (????? ???????? ???????, ?????? ?????, ????????????, ?????? ?????, ?????????? ???????????, ???????????????) ??????????? ??????????? ?? ?????? ??????? ??? ????? ?? ?????? ??????? ?????? - ?? ????????? ????????????? ??????????-???????? ????????? ? ??????????? ?????? ? ?????. 4. ??????????? ?????????????, ???????????? ???? ??????????? ????????? ?????????? ?? ??????????? ??????????? ??????? ???? ?????????? ????????????? ?? ????? ?????? ?????????, ??????? ?? ????????? ?????????? ?? ??????????, ?? ????????????? ???????? ???????????? ????????????? ???????????????? ?.?????? - ???????? ????????? 5. ?????????????? ?????? ? ?? ??????????? ??????????????? ????? ??????????? ???????????? ?????? ??? ?.?????? ???? ?.?????? ?????????? ?? ????? ???????? ??????????????? ????????? ????? ? ?.?????? 6. ?????????????? ???????????? ????????? ???????: 5800 ???. + ??? ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. 232-2133 -------------- next part -------------- An HTML attachment was scrubbed... URL: From luszczek at cs.utk.edu Wed Mar 29 19:26:03 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Wed Mar 29 19:26:03 2006 Subject: [Numpy-discussion] numpy.linalg: eig versus eigh? In-Reply-To: <5CFF089A-5B17-4B4F-83BA-2B75CC2E7BDF@stanford.edu> References: <5CFF089A-5B17-4B4F-83BA-2B75CC2E7BDF@stanford.edu> Message-ID: <200603292224.09131.luszczek@cs.utk.edu> On Wednesday 29 March 2006 19:03, Zachary Pincus wrote: > Can anyone tell me what the difference between numpy.linalg.eig and > numpy.linalg.eigh is? (Ditto for eigvals and eigvalsh.) > > Thanks, > > Zach eig() is for nonsymmetric matrices and eigh() is for symmetric (or hermitian matrices). The former most likely will return complex eigen values. The latter always returns real eigen values. eig() is slower and numerically less reliable (it calls LAPACK's DGEEV and ZGEEV). Even for symmetric matrices it can produce complex eigen pairs due to numerical round-off. eigh() can be called with an optional parameter UPLO that tells the routine which triangle of the matrix it should operate on. That triangle is destroyed and the other triangle is left untouched. Piotr From haase at msg.ucsf.edu Wed Mar 29 20:54:01 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Mar 29 20:54:01 2006 Subject: [Numpy-discussion] numarray-BUG in arr.maximum.reduce: negative axis returns "transpose" In-Reply-To: <200603220945.18706.haase@msg.ucsf.edu> References: <200603211514.22249.haase@msg.ucsf.edu> <40e64fa20603220532k231afe66g3a607d3094520085@mail.gmail.com> <200603220945.18706.haase@msg.ucsf.edu> Message-ID: <442B6460.5030909@msg.ucsf.edu> Hi, Please consider this a bug report (I have never filed a bug with SF...) I just don't want this falling through the cracks... Again: na.maximum.reduce(arr,-3) returned the transpose of what it should have given ( na.maximum.reduce(arr,1) in my case - see below for details ) Thanks, Sebastian haase Sebastian Haase wrote: > On Wednesday 22 March 2006 05:32, Paul Barrett wrote: >> On 3/21/06, Sebastian Haase wrote: > >> The above behavior for maximum.reduce looks consistent to me. >> >> The reduce axis for examples 1 and 2 above is 0, so maximum is comparing >> arrays [[0 1] [2 3]] and [[4 5] [6 7]], which gives [[4 5] [6 7]]. Example >> 3 is comparing arrays [[0 1] [4 5]] and [[2 3] [6 7]], giving [[2 3] [6 >> 7]]. And the last example is comparing arrays [[0 2] [4 6]] and [[1 3] [5 >> 7]], giving [[1 3] [5 7]]. >> >> I think it depends on how you look at the shape of the array. >> >> -- Paul > Sorry > I did the test wrong - >>>> na.maximum.reduce(q,2) > [[1 3] > [5 7]] >>>> na.maximum.reduce(q,-1) > [[1 3] > [5 7]] > So it obviously works as advertised. > > The array where I noticed the transpose behaviour was really of shape > 32,15,521,512 and my command was na.maximum.reduce(arr,-3) which "looked" > tranpose to what I got with na.maximum.reduce(arr,1). > I using numarray for image analysis and the array might have been a memmap-ed > image file. I'm just worried because I found an "accidental transpose bug" > on type conversion a few years back (see mailing list)... > ( I forgot to say that I running Python 2.2 and numarray version '1.4.0' ) > > I just reran the test with python2.4 and arr=arr.copy() [which should give a > non-memmap version of arr] > >>>> arrM = na.maximum.reduce(arr,-3) >>>> arrMt = na.transpose(arrM,(0,2,1)) >>>> arrM1= na.maximum.reduce(arr,1) >>>> U.mmms(arrM1-arrMt) # returns min,max,mean,stddev > (0, 0, 0.0, 0.0) > > So it definitely transposes the last two axes ! Maybe the bug shows up on > rank >=4 !? - which is the case Robert tested. > > Thanks, > Sebastian Haase > From pearu at scipy.org Thu Mar 30 00:38:01 2006 From: pearu at scipy.org (Pearu Peterson) Date: Thu Mar 30 00:38:01 2006 Subject: [Numpy-discussion] Freezing numpy code In-Reply-To: <91cf711d0603291420g7068c84y623b43d36a83c850@mail.gmail.com> References: <91cf711d0603291420g7068c84y623b43d36a83c850@mail.gmail.com> Message-ID: On Wed, 29 Mar 2006, David Huard wrote: > I would like to freeze some code in order to run it on machines without > installing all the modules I use. I compiled Python 2.4.3 from source and > rebuilt numpy. When I try to freeze the following code : > > #test.py > from numpy import mean, rand > x = rand(1000) > print mean(x) > > with > > python /usr/share/doc/python2.4/examples/Tools/freeze/freeze.py test.py > make > > I get the following error message : > > No scipy-style subpackage 'testing' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'core' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'lib' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'linalg' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'dft' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'random' found in n:u:m:p:y. Ignoring. > No scipy-style subpackage 'f2py' found in n:u:m:p:y. Ignoring. > Traceback (most recent call last): > File "test.py", line 1, in ? > from numpy import mean, rand > File "/usr/lib/python2.4/site-packages/numpy/__init__.py", line 44, in ? > __doc__ += pkgload.get_pkgdocs() > File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 320, > in get_pkgdocs > retstr = self._format_titles(titles) +\ > File "/usr/lib/python2.4/site-packages/numpy/_import_tools.py", line 283, > in _format_titles > max_length = max(lengths) > ValueError: max() arg is an empty sequence I have fixed these errors in svn. Now I get File "/usr/local/lib/python2.4/site-packages/numpy/core/numerictypes.py", line 85, in ? from multiarray import typeinfo, ndarray, array, empty ImportError: cannot import name typeinfo I checked that given multiarray comes from Numeric. It looks like freeze is not able to freeze extension modules that are not in path. Pearu From vmas at carabos.com Thu Mar 30 03:46:06 2006 From: vmas at carabos.com (Vicent Mas (V+)) Date: Thu Mar 30 03:46:06 2006 Subject: [Numpy-discussion] USING NUMPY FOR AN ENROLLMENT PROJECTION PROBLEM In-Reply-To: <502D9B13021C2D42B05F32C5C3F4C1DD2875FF@MBX01.stf.nus.edu.sg> References: <502D9B13021C2D42B05F32C5C3F4C1DD2875FF@MBX01.stf.nus.edu.sg> Message-ID: <200603301345.39880.vmas@carabos.com> El S?bado 25 Marzo 2006 13:59, Ajith Prasad (OFN) escribi?: > I posted this problem - arising from my work at the local university > - some years back in comp.lang.python and received a solution that > did not require the use of matrix algebra. The solution was: > > def enrollment(year, intake, survivalrate): > return sum([intake[year-i]*rate > for i, rate in enumerate(survivalrate)]) > > > I would welcome help in formulating a solution using NumPy. Thanks in > advance for any suggestion. > > > The Problem: > > > The enrolment E(n) of an institution at the beginning of year n is > the sum of the intake for year n, I(n), and the survivors from the > intakes of previous r years. Thus, if s(1) is the 1-year intake > survival rate, s(2) is the 2-year survival rate, etc, we have: > > > E(n)= I(n)+I(n-1)*s(1)+ I(n-2)*s(2)+...+I(n-r)*s(r) > E(n+1)= I(n+1)+I(n)*s(1)+I(n-1)*s(2)+... +I(n-r-1)*s(r) > . > . > . > E(n+k)= I(n+k)+I(n+k-1)*s(1)+I(n+k-2)*s(2)+...+I(n+k-r)*s(r) > > > Given: > (a) the actual intakes for the current and previous r years, I(n), > I(n-1),I(n-2),..,I(n-r), and the planned intakes for the next n+k > years: I(n+1), I(n+2),..., I(n+k), we have the intake vector I = > (I(n-r), I(n-r-1),...,I(n),I(n+1),..., I(n+k)); and > (b) the survival rate vector, s = (1,s(1), s(2),...,s(r)) > Find: > The k*1 enrolment projection column vector, E = > (E(n+1),E(n+2),...,E(n+k)) in terms of a k*(r+1) matrix P (derived > from I) and the (r+1)*1 column vector, s. > > > I = P*s > > > Is there a compact Python representation of the relevant matrix P > where: > > > P = [I(n+1) I(n) I(n-1).. . I(n-r) > I(n+2) I(n+1) I(n)... I(n-r-1) > . > . > I(n+k) I(n+k-1) I(n+k-2)... I(n+k-r)] > > > If I've understood properly you simply want to multiply two matrices. Just store your intakes in an array, your survival rates in another one and call matrixmultiply (or dot). The following interactive session shows it: In [1]: import numarray In [2]: p_matrix = numarray.array([1,2,3,4,5,10,1,2,3,4,20,10,1,2,3,30,20,10,1,2], ...: shape=(4, 5), type="Int8") In [3]: sr_array = numarray.array([1, 0.2, 0.15, 0.25, 0.3]) In [4]: enrolment_array = numarray.matrixmultiply(p_matrix, sr_array) In [5]: print p_matrix [[ 1 2 3 4 5] [10 1 2 3 4] [20 10 1 2 3] [30 20 10 1 2]] In [6]: print sr_array [ 1. 0.2 0.15 0.25 0.3 ] In [7]: print enrolment_array [ 4.35 12.45 23.55 36.35] Regards -- :: \ / Vicent Mas http://www.carabos.com 0;0 / \ C?rabos Coop. Enjoy Data V V " " -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From Norbert.Nemec.list at gmx.de Thu Mar 30 06:14:17 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Thu Mar 30 06:14:17 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <441A7E2A.9010903@ieee.org> References: <44198B7B.6040706@gmx.de> <441A7E2A.9010903@ieee.org> Message-ID: <442BE77D.7090001@gmx.de> Sorry, Travis, I missed your answer for a week... I confess, my efficiency concerns about the histogram routine are to some extent, of theoretical nature. However, given the widespread use of the function and considering the simplicity of the solution, I think it is still worth thinking about it: The current histogram routine is based on a "sort" of the incoming data. Quicksort is usually O(N log N), worst case O(N**2). Doing the histogram by simply counting is a simple O(N) procedure. The absolute cost of the quicksort algorithm is low, but the simple counting should be even lower, if done efficiently in C. What probably weighs heavier though, is the memory usage: sorting can either be done in place, destroying the original data, or requires a copy. Counting could be done even on an iterator without storing all the data. Lacking a comparable implementation of the counting algorithm, I have not done any actual speed comparisons. Why nobody ever found this inefficiency, I cannot say. Probably nobody ever had a real comparison how fast the histogram could be. Unless you are dealing with really huge data, the difference will not show up. Anyway, as I said, a solution should be simple to code once it is decided how to do it. Simply recoding the histogram routine itself in C would be the minimal solution. Implementing a more flexible counting routine that can be used in other contexts as well would certainly be more desirable, but I have no clear vision what that should look like. Greetings, Norbert Travis Oliphant wrote: > Norbert Nemec wrote: > >> I have a very much related problem: Not only that the idea described by >> Mads Ipsen does not work, but I could generally find no efficient way to >> do a "counting" of elements in an array, as it is needed for a >> histogram. >> > > This may be something we are lacking. > > It depends on what you mean by efficient, I suppose. The sorting > algorithms are very fast, and the histogram routines that are > scattered all over the place (including the histogram function in > numpy) has been in use for a long time, and you are the first person > to complain of its efficiency. That isn't to say your complaint may > not be valid, it's just that for most people the speed has been > sufficient. > >> What would instead be needed is a function that simply gives the count >> of occurances of given values in a given array: >> > > I presume you are talking of "integer" arrays, since > equality-comparison of floating-point values is usually not very > helpful so most histograms on floating-point values are given in terms > of bins. Thus, the searchsorted function uses bins for it's > "counting" operation. > >> >> >>>>> [4,5,2,3,2,1,4].count([0,1,2,3,4,5]) >>>>> >>>> >> [0,1,2,1,1,2] >> >> > > > A count function for integer arrays could certainly be written using a > C-loop. But, I would first just use histogram([4,5,2,3,2,1,4], > [0,1,2,3,4,5]) and make sure that's really too slow, before worrying > about it too much. > > Also, I according to the above function, the right answer is: > > [0, 1, 2, 1, 2, 1] > > > Best, > > > -Travis > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From tim.hochberg at cox.net Thu Mar 30 08:53:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 30 08:53:05 2006 Subject: [Numpy-discussion] Histograms via indirect index arrays In-Reply-To: <442BE77D.7090001@gmx.de> References: <44198B7B.6040706@gmx.de> <441A7E2A.9010903@ieee.org> <442BE77D.7090001@gmx.de> Message-ID: <442C0CD6.3030105@cox.net> Norbert Nemec wrote: >Sorry, Travis, I missed your answer for a week... > >I confess, my efficiency concerns about the histogram routine are to >some extent, of theoretical nature. However, given the widespread use of >the function and considering the simplicity of the solution, I think it >is still worth thinking about it: > >The current histogram routine is based on a "sort" of the incoming data. >Quicksort is usually O(N log N), worst case O(N**2). Doing the histogram >by simply counting is a simple O(N) procedure. The absolute cost of the >quicksort algorithm is low, but the simple counting should be even >lower, if done efficiently in C. > > I'm not so sure about this analysis: histogram allows aribtrarily spaced bins, so I think the actual comparison is O(N log N) versus O(N log M) where M is the number of bins. The number of bins will typically be much lower than N it's true, but the log will tend to wash that out some. Take, for example, 1024 items into 32 bins. Then log N is 10 versus log M being 5. This difference is small enough that probably the differences in constant factors will dominate. >What probably weighs heavier though, is the memory usage: sorting can >either be done in place, destroying the original data, or requires a >copy. > I wouldn't worry about the memory usage itself so much. We all have oodles of RAM and virtual memory these days anyway. In addition, you'll probably end up copying discontiguous data anyway, which one ends up with rather frequently. However, the counting algorithm should be more cache friendly than the sort based on. I find that operations on matrices that don't fit in the cache are dominated by memory access time, not CPU time. On my machine that happens somewhere between 10k and 100k elements. For arrays with 100k or so elements, the counting algorithm might be a big win. >Counting could be done even on an iterator without storing all the >data. > > This is true. However, if you go through the iterator interface, it will be much slower than directly addressing the array memory. Such a thing might be cool to have somewhere (itertools or something), but probably isn't appropriate for numpy. >Lacking a comparable implementation of the counting algorithm, I have >not done any actual speed comparisons. Why nobody ever found this >inefficiency, I cannot say. Probably nobody ever had a real comparison >how fast the histogram could be. Unless you are dealing with really huge >data, the difference will not show up. > > This is the crux of the issue I think. The implementaion of histogram looks fine for what it does. For reasonably large arrays, the overhead of it being implemented in Python will be negligible. Without someone who actually needs something different, a new implementation is liable to wander off into the realm of pointless microoptimization. So, my advice is to leave it alone till it actually causes you problems. >Anyway, as I said, a solution should be simple to code once it is >decided how to do it. Simply recoding the histogram routine itself in C >would be the minimal solution. Implementing a more flexible counting >routine that can be used in other contexts as well would certainly be >more desirable, but I have no clear vision what that should look like. > > Well if you were going to do this. I would code a drop in replacement for the current histogram. No iterators (they belong in a different library). Variable spaces bins allowed, etc. It doesn't sound like we need a different histogram function and the more overlapping functions that there are, the less each individual one gets used and tested and the more likely that bugs and bitrot set in. What I'd acutally do is implent something that behaved like this: def histogram_core(data, bins): # I hope I grabbed the right stuff out histogram_core # bins is an actual list of bins edges as per histogram # which could really us a docstring n = sort(a).searchsorted(bins) n = concatenate([n, [len(a)]]) n = n[1:]-n[:-1] return n Except written in C and using a counting algorithm. Then the histogram can be just a thin wrapper over histogram_core. Then test it with a bunch of different array sizes and bin counts and see how it does. My guess is that the sort based algoritm will do better in some cases (arrays with 10k or fewer elements and lots of bins) while the counting algorithm does better in others (few bins or 100k or more elements). But we'll see. Or maybe we won't. Regards, -tim >Greetings, >Norbert > > > >Travis Oliphant wrote: > > > >>Norbert Nemec wrote: >> >> >> >>>I have a very much related problem: Not only that the idea described by >>>Mads Ipsen does not work, but I could generally find no efficient way to >>>do a "counting" of elements in an array, as it is needed for a >>>histogram. >>> >>> >>> >>This may be something we are lacking. >> >>It depends on what you mean by efficient, I suppose. The sorting >>algorithms are very fast, and the histogram routines that are >>scattered all over the place (including the histogram function in >>numpy) has been in use for a long time, and you are the first person >>to complain of its efficiency. That isn't to say your complaint may >>not be valid, it's just that for most people the speed has been >>sufficient. >> >> >> >>>What would instead be needed is a function that simply gives the count >>>of occurances of given values in a given array: >>> >>> >>> >>I presume you are talking of "integer" arrays, since >>equality-comparison of floating-point values is usually not very >>helpful so most histograms on floating-point values are given in terms >>of bins. Thus, the searchsorted function uses bins for it's >>"counting" operation. >> >> >> >>> >>> >>> >>> >>>>>>[4,5,2,3,2,1,4].count([0,1,2,3,4,5]) >>>>>> >>>>>> >>>>>> >>>[0,1,2,1,1,2] >>> >>> >>> >>> >>A count function for integer arrays could certainly be written using a >>C-loop. But, I would first just use histogram([4,5,2,3,2,1,4], >>[0,1,2,3,4,5]) and make sure that's really too slow, before worrying >>about it too much. >> >>Also, I according to the above function, the right answer is: >> >>[0, 1, 2, 1, 2, 1] >> >> >>Best, >> >> >>-Travis >> >> >> >> >>------------------------------------------------------- >>This SF.Net email is sponsored by xPML, a groundbreaking scripting >>language >>that extends applications into web and mobile media. Attend the live >>webcast >>and join the prime developer group breaking into this new coding >>territory! >>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> >> >> > > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > From david.huard at gmail.com Thu Mar 30 09:10:06 2006 From: david.huard at gmail.com (David Huard) Date: Thu Mar 30 09:10:06 2006 Subject: [Numpy-discussion] Re: Freezing numpy code Message-ID: <91cf711d0603300844n1fdc7227ud5bc14533ee69ed4@mail.gmail.com> Thanks Pearu, >From the documentation in freeze.py, it is possible to tell freeze the explicit path to modules. freeze [options...] script [module] where module is an absolute path. I'll try this out. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Thu Mar 30 11:22:11 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 30 11:22:11 2006 Subject: [Numpy-discussion] Problem with _wrapit Message-ID: <442C2FBA.5080705@cox.net> There's a problem with the _wrapit function. If the function that it wraps, returns a scalar, it blows up. Here's an example: import numpy class narray(numpy.ndarray): def get_argmax(self): raise AttributeError argmax = property(get_argmax) a = narray([3], int, numpy.arange(3)) print type(a), isinstance(a, numpy.ndarray) print a print numpy.argmax(a) ==> True [0 1 2] Traceback (most recent call last): File "find_bug.py", line 13, in ? print numpy.argmax(a) File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", line 278, in argmax return _wrapit(a, 'argmax', axis) File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", line 170, in _wrapit result = wrap(result) TypeError: can only be called with ndarray object A possible fix is to do another isinstance check on the way out: def _wrapit(obj, method, *args, **kwds): try: wrap = obj.__array_wrap__ except AttributeError: wrap = None result = getattr(asarray(obj),method)(*args, **kwds) if wrap and isinstance(result, mu.ndarray): result = wrap(result) return result That fixes this problem, and I think it should be OK in general, but I'd like a second opinion before comitting it. regards, -tim From simon at arrowtheory.com Thu Mar 30 14:50:01 2006 From: simon at arrowtheory.com (Simon Burton) Date: Thu Mar 30 14:50:01 2006 Subject: [Numpy-discussion] ufunc broadcast for object arrays Message-ID: <20060331094857.241d9485.simon@arrowtheory.com> >>> a=numpy.array(list('foabcdeef'),dtype=numpy.dtype("O")) >>> numpy.equal(a,'f') NotImplemented So where do I start implementing this ? core/src/ufuncobject.c ? Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From ndarray at mac.com Thu Mar 30 19:55:01 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 30 19:55:01 2006 Subject: [Numpy-discussion] ufunc broadcast for object arrays In-Reply-To: <20060331094857.241d9485.simon@arrowtheory.com> References: <20060331094857.241d9485.simon@arrowtheory.com> Message-ID: On 3/30/06, Simon Burton wrote: > > >>> a=numpy.array(list('foabcdeef'),dtype=numpy.dtype("O")) > >>> numpy.equal(a,'f') > NotImplemented > > So where do I start implementing this ? > core/src/ufuncobject.c ? > Yes, 879 if (PyTypeNum_ISFLEXIBLE(arg_types[i])) { 880 loop->notimplemented = 1; 881 return nargs; 882 } (line numbers from r2310) From oliphant at ee.byu.edu Thu Mar 30 19:55:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Mar 30 19:55:03 2006 Subject: [Numpy-discussion] ufunc broadcast for object arrays In-Reply-To: <20060331094857.241d9485.simon@arrowtheory.com> References: <20060331094857.241d9485.simon@arrowtheory.com> Message-ID: <442C6F98.3060601@ee.byu.edu> Simon Burton wrote: >>>>a=numpy.array(list('foabcdeef'),dtype=numpy.dtype("O")) >>>>numpy.equal(a,'f') >>>> >>>> A couple of things: 1) dtype=numpy.dtype("O") is equivalent to dtype="O" 2) Because you have an object array this should work. You can work around the bug using numpy.equal(a,array('f','O')) -Travis From pgmdevlist at mailcan.com Thu Mar 30 19:55:05 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Thu Mar 30 19:55:05 2006 Subject: [Numpy-discussion] Dumping record arrays Message-ID: <200603302127.24231.pgmdevlist@mailcan.com> Folks, I'd like to dump/pickle some record arrays. The pickling works, the unpickling raises a ValueError (on my version of numpy 0.9.6). (cf below). Is this already corrected in the svn version ? Thx ########################################################################### # x1 = array([21,32,14]) x2 = array(['my','first','name']) x3 = array([3.1, 4.5, 6.2]) r = rec.fromarrays([x1,x2,x3], names='id, word, number') r.dump('dumper') rb=load('dumper') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/backtopop/Work/workspace-python/pyflows/src/ /usr/lib64/python2.4/site-packages/numpy/core/numeric.py in load(file) 331 if isinstance(file, type("")): 332 file = _file(file,"rb") --> 333 return _cload(file) 334 335 # These are all essentially abbreviations /usr/lib64/python2.4/site-packages/numpy/core/_internal.py in _reconstruct(subtype, shape, dtype) 251 252 def _reconstruct(subtype, shape, dtype): --> 253 return ndarray.__new__(subtype, shape, dtype) 254 255 ValueError: ('data-type with unspecified variable length', , (, (0,), 'V')) From tim.hochberg at cox.net Thu Mar 30 19:55:13 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Mar 30 19:55:13 2006 Subject: [Numpy-discussion] Segfault! Message-ID: <442C670A.1070209@cox.net> The following snippet utilizing a rank-0 array will segfault numpy: import numpy a = numpy.array(4) b = numpy.arange(10) a *= b (Build '0.9.7.2301' using VC7 on windows). I'll file a better bug report tomorrow, assuming this hasn't been fixed, but I'm on my way out the door. -tim From ndarray at mac.com Thu Mar 30 21:20:02 2006 From: ndarray at mac.com (Sasha) Date: Thu Mar 30 21:20:02 2006 Subject: [Numpy-discussion] Segfault! In-Reply-To: <442C670A.1070209@cox.net> References: <442C670A.1070209@cox.net> Message-ID: Fixed. On 3/30/06, Tim Hochberg wrote: > > The following snippet utilizing a rank-0 array will segfault numpy: > > import numpy > a = numpy.array(4) > b = numpy.arange(10) > a *= b > > (Build '0.9.7.2301' using VC7 on windows). > > I'll file a better bug report tomorrow, assuming this hasn't been fixed, > but I'm on my way out the door. > > -tim > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From aberrant at takas.lt Fri Mar 31 01:40:29 2006 From: aberrant at takas.lt (=?windows-1251?B?wu3o7ODt6OUg08PQzsfA?=) Date: Fri Mar 31 01:40:29 2006 Subject: [Numpy-discussion] =?windows-1251?B?we7w/OHgIPEg6u7w7+Dw4PLo4u377CDs7vjl7e3o9+Xx8uLu7A==?= Message-ID: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 10 13 ?????? 2006 ???? ???? ????????????? ?????????, ???????? ?????????? ???????????, ???????? ??????????? ? ?????? ??????? ???????????? ????????? ???? ?????????????? ?????????? ? ????????, ????? ?? ???????????? ???????? ???????? ?????????? ????????. ??? ??????? ????? ????????????? ? ?????????? ?????????? ??????????? ?? ???????????? ??????? ????????? ??? ???? ? ????? ? ????????? ???????. ?????? ?? ? ??? ???????? ????? ???????? ?????????? ?????? ??????????? ????????, ??????? ???????? ??????????? ??????? ???????????? ?????, ????????? ??????????????????? ? ???????????????? ?????? ? ?????? ? ????????? ????????????? ????????? ?????????? ???????, ???????? ??? ??????????????, ?????????? ? ????????????????? ?? ??????? ???????????. ??????? ????????? ?? ????????????? ?????????? ??????????? ???????, ?????? ??????? ?????????? ? ??????????? ????????, ? ????? ????????????? ?????????? ?????. ?????????? ??????????, ??????????? ? ???????????? ?????????? ? ????????? ??????????????? ??????????????????? ????????-??????????: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 10 13 ?????? 2006 ???? ???? ?????: ??????????? ??????????? ??????? ? ?????????? ??????????? ??????????? ??????????? ? ????????? ? ??????????? ???????? ??????????? ????????, ??????? ???????? ?????????? ????? ?????????????? ??????? ????? ???????? ?????? ??????????????? ???????????? ? ?????????????? ?????????. ????????? ?? ??? ?????????????? ????????????, ????? ???????? ???????? ???????? ???????? ???????? ? ??????? ??????????? ? ????? ??????? ???????? ? ??????????? ????????????? ????? ? ???????? ???????? ?????????? ???????? ?? ???????? ?????????? ? ?????????? ????????. ?? ???????? ????? ???????? ??????-????? ????????????? ??????????? ??????????? ???????? ??? ?????? ?? ????????????? ???????? ?????????????????? ? ?????????????? ???????, ? ????? ??????????? ? ?????? ?????????? ???????? ?? ???????????? ???????? ????????. ????? ??????? ??????? ? ????? ??? ???????? ????????? ??????????, ????????? ??? ?? ?????????: (495) 98O-65-16, 98?-65-?9 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fullung at gmail.com Fri Mar 31 04:24:05 2006 From: fullung at gmail.com (Albert Strasheim) Date: Fri Mar 31 04:24:05 2006 Subject: [Numpy-discussion] Segfault! In-Reply-To: Message-ID: <009b01c654bd$dbc746a0$0a84a8c0@dsp.sun.ac.za> Hello all I've noticed a lot of fixes going into NumPy during the past couple of weeks and I'm very excited about the progress that is being made. However, it seems many changes are being made without the addition of test cases to make sure that things don't break again in the future. The code snippet Tim provided to reproduce the bug mentioned below would be a perfect test, but it doesn't seem as if a test for this bug was added in the changeset where the bug was fixed or any changeset since. Any comments? Regards, Albert > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net [mailto:numpy- > discussion-admin at lists.sourceforge.net] On Behalf Of Sasha > Sent: 31 March 2006 07:20 > To: tim.hochberg at ieee.org > Cc: numpy-discussion > Subject: Re: [Numpy-discussion] Segfault! > > Fixed. > > On 3/30/06, Tim Hochberg wrote: > > > > The following snippet utilizing a rank-0 array will segfault numpy: > > > > import numpy > > a = numpy.array(4) > > b = numpy.arange(10) > > a *= b > > > > (Build '0.9.7.2301' using VC7 on windows). > > > > I'll file a better bug report tomorrow, assuming this hasn't been fixed, > > but I'm on my way out the door. > > > > -tim From ndarray at mac.com Fri Mar 31 06:26:04 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 31 06:26:04 2006 Subject: [Numpy-discussion] Segfault! In-Reply-To: <009b01c654bd$dbc746a0$0a84a8c0@dsp.sun.ac.za> References: <009b01c654bd$dbc746a0$0a84a8c0@dsp.sun.ac.za> Message-ID: Point taken. On 3/31/06, Albert Strasheim wrote: > ... > However, it seems many changes are being made without the addition of test > cases to make sure that things don't break again in the future. The code > snippet Tim provided to reproduce the bug mentioned below would be a perfect > test, but it doesn't seem as if a test for this bug was added in the > changeset where the bug was fixed or any changeset since. From pebarrett at gmail.com Fri Mar 31 06:43:04 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri Mar 31 06:43:04 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX Message-ID: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> I'm having trouble build Numpy on HP-UX. I get the following error message when using the native 'cc' compiler: From ndarray at mac.com Fri Mar 31 06:48:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Mar 31 06:48:03 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX In-Reply-To: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> References: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> Message-ID: Try to remove the "build" directory. On 3/31/06, Paul Barrett wrote: > I'm having trouble build Numpy on HP-UX. > > I get the following error message when using the native 'cc' compiler: > > cc: numpy/core/src/multiarraymodule.c > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: Illegal rede > claration for identifier "ushort". > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: Illegal rede > claration for identifier "uint". > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: Redeclaration of "Py > Array_SetNumericOps" with a different storage class specifier: "PyArray_SetNumer > icOps" will have internal linkage. > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: Illegal rede > claration for identifier "ushort". > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: Illegal rede > claration for identifier "uint". > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: Redeclaration of "Py > Array_SetNumericOps" with a different storage class specifier: "PyArray_SetNumer > icOps" will have internal linkage. > error: Command "cc -Ae -DNDEBUG -O +z -Ibuild/src/numpy/core/src -Inumpy/core/in > clude -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/ > include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.hp-ux-B.11. > 11-9000/785-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1 > > And this error message when trying to use 'gcc' instead: > > error: don't know how to compile C/C++ code on platform 'posix' with 'gcc' comp > iler > > Has this been fixed in the current SVN or should can someone suggest a fix? > > -- Paul > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting language > that extends applications into web and mobile media. Attend the live webcast > and join the prime developer group breaking into this new coding territory! > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From pebarrett at gmail.com Fri Mar 31 06:51:01 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri Mar 31 06:51:01 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX In-Reply-To: References: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> Message-ID: <40e64fa20603310649v61b9f9d8wd32f4aa818753ea6@mail.gmail.com> Yes, I've done this numerous times without success. It looks like a configuration problem. -- Paul On 3/31/06, Sasha wrote: > > Try to remove the "build" directory. > > On 3/31/06, Paul Barrett wrote: > > I'm having trouble build Numpy on HP-UX. > > > > I get the following error message when using the native 'cc' compiler: > > > > cc: numpy/core/src/multiarraymodule.c > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > error: Command "cc -Ae -DNDEBUG -O +z -Ibuild/src/numpy/core/src > -Inumpy/core/in > > clude -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include > -I/usr/local/ > > include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.hp- > ux-B.11. > > 11-9000/785-2.4/numpy/core/src/multiarraymodule.o" failed with exit > status 1 > > > > And this error message when trying to use 'gcc' instead: > > > > error: don't know how to compile C/C++ code on platform 'posix' with > 'gcc' comp > > iler > > > > Has this been fixed in the current SVN or should can someone suggest a > fix? > > > > -- Paul > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > > that extends applications into web and mobile media. Attend the live > webcast > > and join the prime developer group breaking into this new coding > territory! > > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pebarrett at gmail.com Fri Mar 31 06:51:05 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Fri Mar 31 06:51:05 2006 Subject: [Numpy-discussion] Trouble building Numpy on HP-UX In-Reply-To: References: <40e64fa20603310642wad57aaai2437811b2972ebc5@mail.gmail.com> Message-ID: <40e64fa20603310649v61b9f9d8wd32f4aa818753ea6@mail.gmail.com> Yes, I've done this numerous times without success. It looks like a configuration problem. -- Paul On 3/31/06, Sasha wrote: > > Try to remove the "build" directory. > > On 3/31/06, Paul Barrett wrote: > > I'm having trouble build Numpy on HP-UX. > > > > I get the following error message when using the native 'cc' compiler: > > > > cc: numpy/core/src/multiarraymodule.c > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > cc: "numpy/core/include/numpy/arrayobject.h", line 139: error 1713: > Illegal rede > > claration for identifier "ushort". > > cc: "numpy/core/include/numpy/arrayobject.h", line 140: error 1713: > Illegal rede > > claration for identifier "uint". > > cc: "numpy/core/src/arrayobject.c", line 2420: warning 562: > Redeclaration of "Py > > Array_SetNumericOps" with a different storage class specifier: > "PyArray_SetNumer > > icOps" will have internal linkage. > > error: Command "cc -Ae -DNDEBUG -O +z -Ibuild/src/numpy/core/src > -Inumpy/core/in > > clude -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include > -I/usr/local/ > > include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.hp- > ux-B.11. > > 11-9000/785-2.4/numpy/core/src/multiarraymodule.o" failed with exit > status 1 > > > > And this error message when trying to use 'gcc' instead: > > > > error: don't know how to compile C/C++ code on platform 'posix' with > 'gcc' comp > > iler > > > > Has this been fixed in the current SVN or should can someone suggest a > fix? > > > > -- Paul > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > > that extends applications into web and mobile media. Attend the live > webcast > > and join the prime developer group breaking into this new coding > territory! > > http://sel.as-us.falkag.net/sel?cmdlnk&kid0944&bid$1720&dat1642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Fri Mar 31 08:32:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 08:32:03 2006 Subject: [Numpy-discussion] Re: Problem with _wrapit In-Reply-To: <442C2FBA.5080705@cox.net> References: <442C2FBA.5080705@cox.net> Message-ID: <442D5949.9010907@cox.net> Well, I got no negative opinions (no opinions at all for that matter), so I went ahead and commited this change (with test cases -- thanks for the push Albert). -tim Tim Hochberg wrote: > > There's a problem with the _wrapit function. If the function that it > wraps, returns a scalar, it blows up. Here's an example: > > import numpy > > class narray(numpy.ndarray): > def get_argmax(self): > raise AttributeError > argmax = property(get_argmax) > a = narray([3], int, numpy.arange(3)) > print type(a), isinstance(a, numpy.ndarray) > > print a > print numpy.argmax(a) > > ==> > > True > [0 1 2] > Traceback (most recent call last): > File "find_bug.py", line 13, in ? > print numpy.argmax(a) > File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", > line 278, in argmax > return _wrapit(a, 'argmax', axis) > File "C:\python24\lib\site-packages\numpy\core\oldnumeric.py", > line 170, in _wrapit > result = wrap(result) > TypeError: can only be called with ndarray object > > > A possible fix is to do another isinstance check on the way out: > > def _wrapit(obj, method, *args, **kwds): > try: > wrap = obj.__array_wrap__ > except AttributeError: > wrap = None > result = getattr(asarray(obj),method)(*args, **kwds) > if wrap and isinstance(result, mu.ndarray): > result = wrap(result) > return result > > That fixes this problem, and I think it should be OK in general, but > I'd like a second opinion before comitting it. > > regards, > > -tim > > > > > > From filip at ftv.pl Fri Mar 31 10:13:05 2006 From: filip at ftv.pl (Filip Wasilewski) Date: Fri Mar 31 10:13:05 2006 Subject: [Numpy-discussion] Confused with Py_ssize_t and Py_intptr_t (intp) Message-ID: <274382036.20060331201149@ftv.pl> Hello, I'm trying to figure out what is the proper type for indexing arrays. Is it safe to use Py_ssize_t although PyArrayObject dimensions are of Py_intptr_t (intp) type or use intp instead of Py_ssize_t? Is there any platform on which these two types has different size or range? References: [1] http://www.python.org/dev/peps/pep-0353/#why-not-py-intptr-t [2] numpy 0.9.6, arrayobject.c: static _int_or_ssize_t array_length(PyArrayObject *self) { if (self->nd != 0) { return self->dimensions[0]; } else { PyErr_SetString(PyExc_TypeError, "len() of unsized object"); return -1; } } // self->dimensions is of intp* type [3] http://svn.python.org/projects/python/branches/ssize_t/Include/abstract.h PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *obj, const void **buffer, Py_ssize_t *buffer_len); -- best, fw From haase at msg.ucsf.edu Fri Mar 31 12:30:02 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Mar 31 12:30:02 2006 Subject: [Numpy-discussion] first impressions with numpy Message-ID: <442D9124.5020905@msg.ucsf.edu> Hi, I'm a long time user of numarray. Now I downloaded numpy for the first time - and am quite excited to maybe soon being able to use things like weave ! Thanks for all the good work ! 1) I was following the discussion about dtype with interest, and so it was one of the first things to check... >>> repr(aa.dtype) 'dtype('>> str(aa.dtype) ''>> aa.dtype.name 'int32' Would it be possible to change str(aa.dtype) to aa.dtype.name ?? I think it would be much more readable ! I'm mainly thinking of (some "less involved") people that just want to know that it's a '32bit integer' ; ' References: <442D9124.5020905@msg.ucsf.edu> Message-ID: <442D9AC8.9090400@stsci.edu> Sebastian Haase wrote: > Hi, > I'm a long time user of numarray. Now I downloaded numpy for the first > time - and am quite excited to maybe soon being able to use things > like weave ! > Thanks for all the good work ! > > 1) > I was following the discussion about dtype with interest, and so it > was one of the first things to check... > >>> repr(aa.dtype) > 'dtype(' >>> str(aa.dtype) > '' >>> aa.dtype.name > 'int32' > > Would it be possible to change str(aa.dtype) to aa.dtype.name ?? > I think it would be much more readable ! > I'm mainly thinking of (some "less involved") people that just want to > know that it's a '32bit integer' ; ' > > 2) > This is probably more numarray related: > Why does numarray.asarray( numpy.array([1]) ) return a numpy array, > not a numarray ??? This is even true for numarray.array( > numpy.array([1]) ) !! This is what I get for numarray-CVS and numpy-CVS: >>> a = numarray.array(numpy.array([1])) >>> type(a) >>> a = numarray.asarray(numpy.array([1])) >>> type(a) So... I don't see a problem. Also: I logged the numarray.maximum.reduce axis=-3 transpose problem on SF. It should get fixed for numarray-1.5.2 but that may be a while. Todd > > I'll keep exploring... > Thanks, > Sebastian Haase > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant at ee.byu.edu Fri Mar 31 14:24:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Mar 31 14:24:03 2006 Subject: [Numpy-discussion] first impressions with numpy In-Reply-To: <442D9124.5020905@msg.ucsf.edu> References: <442D9124.5020905@msg.ucsf.edu> Message-ID: <442DABA8.20003@ee.byu.edu> Sebastian Haase wrote: > Hi, > I'm a long time user of numarray. Now I downloaded numpy for the first > time - and am quite excited to maybe soon being able to use things > like weave ! > Thanks for all the good work ! > > 1) > I was following the discussion about dtype with interest, and so it > was one of the first things to check... > >>> repr(aa.dtype) > 'dtype(' >>> str(aa.dtype) > '' >>> aa.dtype.name > 'int32' > > Would it be possible to change str(aa.dtype) to aa.dtype.name ?? It's definitely possible and probably a good idea for some of the builtin types. -Travis From pgmdevlist at mailcan.com Fri Mar 31 15:37:03 2006 From: pgmdevlist at mailcan.com (Pierre GM) Date: Fri Mar 31 15:37:03 2006 Subject: [Numpy-discussion] Pb with vectorizing Message-ID: <200603311824.38683.pgmdevlist@mailcan.com> Folks, I'm trying to use `vectorize` on the following lambda function, taking a scalar as input, and output a 10-char string fnc = lambda x: datetime.date.fromordinal(x).strftime("%Y-%m-%d") I can't find a way to get an actual 'S10' chararray as output. OIn top of that, it seems that vectorize still uses the oldnumeric typecode: is this on purpose (and then which one), or just because it has been updated yet ? Here's what I tried so far. Any idea/comment (even a 'that can't work, forget it) more than welcome, as usual Thx P. ---------------------------------------------------------- x = array([719893,719894, 719895]) fnc(x[0]) >>> '1972-01-01' fnc(x[:2]) >>> TypeError: only length-1 arrays can be converted to Python scalars (OK, that wasn't suppose to work) vectorize(fnc)(x[:2]) >>> array([, ], dtype='|S0') (not quite) vectorize(fnc,otypes="S")(x[:2]) >>> array([, ], dtype='|S0') (ditto) vectorize(fnc,otypes="|S10")(x[:2]) >>> ValueError: invalid typecode specified asarray(vectorize(fnc)(newddatesordinal[:2]),dtype='|S10') >>> array([?6z??*, ?6z??*], dtype='|S10') From tim.hochberg at cox.net Fri Mar 31 17:25:10 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 17:25:10 2006 Subject: [Numpy-discussion] first impressions with numpy In-Reply-To: <442DB91F.9030103@msg.ucsf.edu> References: <442D9124.5020905@msg.ucsf.edu> <442D9695.2050900@cox.net> <442DB655.2050203@cox.net> <442DB91F.9030103@msg.ucsf.edu> Message-ID: <442DD638.60706@cox.net> Sebastian Haase wrote: > Thanks Tim, > that's OK - I got the idea... > BTW, is there a (policy) reason that you sent the first email just to > me and not the mailing list !? No. Just clumsy fingers. Probably the same reason the functions got all garbled! > > I would really be more interested in comments to my first point ;-) > I think it's important that numpy will not be to cryptic and only for > "hackers", but nice to look at ... (hope you get what I mean ;-) Well, I think it's probably a good idea and it sounds like Travis like the idea " for some of the builtin types". I suspect that's code for "not types for which it doesn't make sense, like recarrays". > I'm have developed an "image analysis algorithm development" platform > (based on wxPython + PyShell) that more and more people in our lab are > using (instead of Matlab !) and I changed the default sys.displayhook > to print str(...) instead of repr(...) > mainly to get .3 instead of .2999999999998 > but seing int32 instead of Thanks for your great work .. Oh, I'm not doing much in the way of great work. I'm mostly just causing Travis headaches. -tim > Sebastian > > > Tim Hochberg wrote: > >> Tim Hochberg wrote: >> >>> Sebastian Haase wrote: >>> >>>> Hi, >>>> I'm a long time user of numarray. Now I downloaded numpy for the >>>> first time - and am quite excited to maybe soon being able to use >>>> things like weave ! >>>> Thanks for all the good work ! >>>> >>> [SNIP] >>> >>>> >>>> >>>> 2) >>>> This is probably more numarray related: >>>> Why does numarray.asarray( numpy.array([1]) ) return a numpy array, >>>> not a numarray ??? This is even true for numarray.array( >>>> numpy.array([1]) ) !! >>>> >>> I expect that numarray is grabbing the output of __array__() and >>> using that for asarray. What's happening in the second case is hard >>> to know, but I bet it's some snafu with assuming that anything that >>> isn't a numarray, but implements __array__ must be returning a new >>> numarray. That's just a guess. FWIW, I'm using the following two >>> functions to go back and forth between numarray and numpy. So far >>> they seem to work and they're faster (or actually work in the case >>> of asnumarray). You may want to rename them to suit your personal >>> preferences. If you use them, please let me know if you find any >>> problems. >>> >>> Regards, >>> >> I see this got thoroughly mangled somehow. If you want a copy, let me >> know and I'll just send you it as a file. >> >> -tim >> > > > From tim.hochberg at cox.net Fri Mar 31 18:38:07 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 18:38:07 2006 Subject: [Numpy-discussion] numpy error handling Message-ID: <442DE773.4060104@cox.net> I've just been looking at how numpy handles changing the behaviour that is triggered when there are numeric error conditions (overflow, underflow, etc.). If I understand it correctly, and that's a big if, I don't think I like it nearly as much as the what numarray has in place. It appears that numpy uses the two functions, seterr and geterr, to set and query the error handling. These set/read a secret variable stored in the local scope. I assume that the various ufuncs then examine that value to determine how to handle errors. The secret variable approach is a little clunky, but that's not what concerns me. What concerns me is that this approach is *only* useful for built in numpy functions and falls down if we call any user defined functions. Suppose we want to be warned on underflow. Setting this is as simple as: def func(*args): numpy.seterr(under='warn') # do stuff with args return result Since seterr is local to the function, we don't have to reset the error handling at the end, which is convenient. And, this works fine if all we are doing is calling numpy functions and methods. However, if we are calling a function of our own devising we're out of luck since the called function will not inherit the error settings that we have set. Thus we have no way to influence the error settings of functions downstream from us. Compare this to the numarray approach, where you write code like the following: def func(*args) numarray.Error.pushMode(underflow='warn') try: # do stuff with args return result finally: numarray.Error.popMode() At first glance this looks a lot more clunky. However, it's also a lot more useful. In this case, pushMode pushes a new error mode value onto a global[1] stack of mode values. Thus, any functions that func calls, will inherit it's error settings, unless they explicitly override them. As for the clunkiness factor, that's partly just the choice of camelCase versus smashwords, but also the try-finally. The second, however, will be addressed in Python 2.5 with the 'with' statement. At that point, we could have a numpy equivalent to the above numarray code that looked something like: def func(*args) with numpy.errors(underflow='warn'): # do stuff with args return result I believe we should rework the error handling to follow the numarray model before things get set in stone at 1.0. The 'with' statement support can be worked in a bit later if need be, although I suspect that will be easy. I also would prefer more verbose keys ala numarray (underflow, overflow, dicidebyzero and invalid) than those currently used by numpy (under, over, divide and invalid). And (will he never stop) I like numarrays defaults better here too: overflow='warn', underflow='ignore', dividebyzero='warn', invalid='warn'. Currently, numpy defaults to ignore for all cases. These last points are relatively minor though. Regards, -tim [1] I believe it's actually thread local, and if it's not it should be, but that detail is not important for our discussion here. From zpincus at stanford.edu Fri Mar 31 18:52:04 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri Mar 31 18:52:04 2006 Subject: [Numpy-discussion] array_arguments decorator Message-ID: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> Hi folks - I had seen some talk on this list about the utility of a decorator for functions that need to convert their arguments to numpy arrays. This would help eliminate boilerplate calls to 'asarray' like: def distance_squared(a, b): a = numpy.asarray(a) b = numpy.asarray(b) return ((a - b)**2).sum() Here is a trivial decorator I was thinking of adding to the wiki -- does this cover enough cases to be useful? In a bigger sense, would it be worthwhile to add some decorators like this to numpy itself? (I'm not sure I'm in favor of this, since I kind of like smaller APIs over bigger ones.) def array_arguments(f): """Wrap a function such that any positional arguments are converted into numpy arrays before the function is called.""" def convert_arg_wrapper(*args, **kwargs): array_args = [numpy.asarray(a) for a in args] return f(*array_args, **kwargs) return convert_arg_wrapper now distance_squared can look like: @array_arguments def distance_squared(a, b): return ((a - b)**2).sum() if using python 2.4, or if not so using: def distance_squared(a, b): return ((a - b)**2).sum() distance_squared = array_arguments(distance_squared) Zach From tim.hochberg at cox.net Fri Mar 31 19:49:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri Mar 31 19:49:03 2006 Subject: [Numpy-discussion] array_arguments decorator In-Reply-To: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> References: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> Message-ID: <442DF814.8020806@cox.net> Zachary Pincus wrote: > Hi folks - > > I had seen some talk on this list about the utility of a decorator > for functions that need to convert their arguments to numpy arrays. > This would help eliminate boilerplate calls to 'asarray' like: > > def distance_squared(a, b): > a = numpy.asarray(a) > b = numpy.asarray(b) > return ((a - b)**2).sum() > > Here is a trivial decorator I was thinking of adding to the wiki -- > does this cover enough cases to be useful? In a bigger sense, would > it be worthwhile to add some decorators like this to numpy itself? > (I'm not sure I'm in favor of this, since I kind of like smaller APIs > over bigger ones.) > > def array_arguments(f): > """Wrap a function such that any positional arguments are > converted into numpy arrays before the function is called.""" > def convert_arg_wrapper(*args, **kwargs): > array_args = [numpy.asarray(a) for a in args] > return f(*array_args, **kwargs) > return convert_arg_wrapper > > now distance_squared can look like: > @array_arguments > def distance_squared(a, b): > return ((a - b)**2).sum() > > if using python 2.4, or if not so using: > def distance_squared(a, b): > return ((a - b)**2).sum() > distance_squared = array_arguments(distance_squared) Great minds think alike. Or at least our minds think alike ;) I also wrote up a decorator for this same purpose. Then I got distracted and forgot to post it. Mine has more features at the expense of being more complicated. The main extra feature is that it allows you to decide both which args get checked and what there types should be. It also preserves the signature of the original function. Some of this stuff is accomplished using the decorator decorator, which you can find here: http://www.phyast.pitt.edu/~micheles/python/ The upshot of all of this is that you can do stuff like: @array_function(a=float, b=None, c=complex) def foo(a, b, c=1, d=None): print repr(a), repr(b), repr(c), repr(d) And it will convert 'a' and 'c' to float and complex arrays respectively and convert 'b' to some type of array. Arguments not mentioned don't get touched, unless you specify no arguments, in which case all of the positional arguments get converted (*args and **kwargs are not touched in this case). I'm not certain that this is the world's best interface, nor am I certain that the extra complexity is worth it -- yours is certainly easier to understand. However, it was fun to write. I am unlikely to find the time to do anything with it anytime soon, so should you desire to revamp and place it on the wiki, feel free. Or, if you want to ignore it, feel free to do that as well. The code is below: Regards, -tim import inspect, decorator, numpy def array_function(**signature): def deco(func): regargs, varargs, varkwargs, defaults = inspect.getargspec(func) if not signature: signature.update((name, None) for x in regargs) def caller(func, *args, **kwargs): args = list(args) for i, (name, value) in enumerate(zip(regargs, args)): if name in signature: args[i] = numpy.asarray(value, signature[name]) for name, value in kwargs.items(): if name in signature: kwargs[name] = numpy.asarray(value, signature[name]) return func(*args, **kwargs) return decorator._decorate(func, caller) return deco From zpincus at stanford.edu Fri Mar 31 23:41:06 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Fri Mar 31 23:41:06 2006 Subject: [Numpy-discussion] array_arguments decorator In-Reply-To: <442DF814.8020806@cox.net> References: <07C746D4-DD08-4CC2-BA26-4CE220790089@stanford.edu> <442DF814.8020806@cox.net> Message-ID: <0B289EAA-C75B-4A3C-96C8-59C98EBA37E5@stanford.edu> Hi Tim - Thanks for posting your featureful decorator. I didn't know that some of that stuff was possible -- very cool. Hopefully I'll find some time to put these up on the wiki. Zach On Mar 31, 2006, at 7:48 PM, Tim Hochberg wrote: > Zachary Pincus wrote: > >> Hi folks - >> >> I had seen some talk on this list about the utility of a >> decorator for functions that need to convert their arguments to >> numpy arrays. This would help eliminate boilerplate calls to >> 'asarray' like: >> >> def distance_squared(a, b): >> a = numpy.asarray(a) >> b = numpy.asarray(b) >> return ((a - b)**2).sum() >> >> Here is a trivial decorator I was thinking of adding to the wiki >> -- does this cover enough cases to be useful? In a bigger sense, >> would it be worthwhile to add some decorators like this to numpy >> itself? (I'm not sure I'm in favor of this, since I kind of like >> smaller APIs over bigger ones.) >> >> def array_arguments(f): >> """Wrap a function such that any positional arguments are >> converted into numpy arrays before the function is called.""" >> def convert_arg_wrapper(*args, **kwargs): >> array_args = [numpy.asarray(a) for a in args] >> return f(*array_args, **kwargs) >> return convert_arg_wrapper >> >> now distance_squared can look like: >> @array_arguments >> def distance_squared(a, b): >> return ((a - b)**2).sum() >> >> if using python 2.4, or if not so using: >> def distance_squared(a, b): >> return ((a - b)**2).sum() >> distance_squared = array_arguments(distance_squared) > > Great minds think alike. Or at least our minds think alike ;) I > also wrote up a decorator for this same purpose. Then I got > distracted and forgot to post it. Mine has more features at the > expense of being more complicated. The main extra feature is that > it allows you to decide both which args get checked and what there > types should be. It also preserves the signature of the original > function. Some of this stuff is accomplished using the decorator > decorator, which you can find here: > > http://www.phyast.pitt.edu/~micheles/python/ > > The upshot of all of this is that you can do stuff like: > > @array_function(a=float, b=None, c=complex) > def foo(a, b, c=1, d=None): > print repr(a), repr(b), repr(c), repr(d) > > And it will convert 'a' and 'c' to float and complex arrays > respectively and convert 'b' to some type of array. Arguments not > mentioned don't get touched, unless you specify no arguments, in > which case all of the positional arguments get converted (*args and > **kwargs are not touched in this case). > > I'm not certain that this is the world's best interface, nor am I > certain that the extra complexity is worth it -- yours is certainly > easier to understand. However, it was fun to write. I am unlikely > to find the time to do anything with it anytime soon, so should you > desire to revamp and place it on the wiki, feel free. Or, if you > want to ignore it, feel free to do that as well. > > The code is below: > > Regards, > > -tim > > > > import inspect, decorator, numpy > > def array_function(**signature): > def deco(func): > regargs, varargs, varkwargs, defaults = inspect.getargspec > (func) > if not signature: > signature.update((name, None) for x in regargs) > def caller(func, *args, **kwargs): > args = list(args) > for i, (name, value) in enumerate(zip(regargs, args)): > if name in signature: > args[i] = numpy.asarray(value, signature[name]) > for name, value in kwargs.items(): > if name in signature: > kwargs[name] = numpy.asarray(value, signature > [name]) > return func(*args, **kwargs) > return decorator._decorate(func, caller) > return deco > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the > live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Fri Mar 31 23:44:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Mar 31 23:44:01 2006 Subject: [Numpy-discussion] numpy error handling In-Reply-To: <442DE773.4060104@cox.net> References: <442DE773.4060104@cox.net> Message-ID: <442E2F05.5080809@ieee.org> Tim Hochberg wrote: > > I've just been looking at how numpy handles changing the behaviour > that is triggered when there are numeric error conditions (overflow, > underflow, etc.). If I understand it correctly, and that's a big if, I > don't think I like it nearly as much as the what numarray has in place. > > It appears that numpy uses the two functions, seterr and geterr, to > set and query the error handling. These set/read a secret variable > stored in the local scope. This approach was decided on after discussions with Guido who didn't like the idea of pushing and popping from a global stack. I'm not sure I'm completely in love with it my self, but it is actually more flexible then the numarray approach. You can get the numarray approach back simply by setting the error in the builtin scope (instead of in the local scope which is done by default. Then, at the end of the function, you can restore it. If it was felt useful to create a stack to handle this on the builtin level then that is easily done as well. > I assume that the various ufuncs then examine that value to determine > how to handle errors. The secret variable approach is a little clunky, > but that's not what concerns me. What concerns me is that this > approach is *only* useful for built in numpy functions and falls down > if we call any user defined functions. > > Suppose we want to be warned on underflow. Setting this is as simple as: > > def func(*args): > numpy.seterr(under='warn') > # do stuff with args > return result > > Since seterr is local to the function, we don't have to reset the > error handling at the end, which is convenient. And, this works fine > if all we are doing is calling numpy functions and methods. However, > if we are calling a function of our own devising we're out of luck > since the called function will not inherit the error settings that we > have set. Again, you have control over where you set the "secret" variable (local, global (module), and builtin). I also don't see how that's anymore clunky then a "secret" stack. You may set the error in the builtin scope --- in fact it would probably be trivial to implement a stack based on this and implement the pushMode popMode interface of numarray. But, I think this question does deserve a bit of debate. I don't think there has been a serious discussion over the method. To help Tim and others understand what happens: When a ufunc is called, a specific variable name is searched for in the following name-spaces in the following order: 1) local 2) global 3) builtin (There is a bit of an optimization in that when the error mode is the default mode --- do nothing, a global flag is set which by-passes the search for the name). The first time the variable name is found, the error mode is read from that variable. This error mode is placed as part of the ufunc loop object. At the end of each 1-d loop the IEEE error mode flags are checked (depending on the state of the error mode) and appropriate action taken. By the way, it would not be too difficult to change how the error mode is set (probably an hour's worth of work). So, concern over implementation changes should not be a factor right now. Currently the error mode is read from a variable using standard scoping rules. It would save the (not insignificant) name-space lookup time to instead use a global stack (i.e. a Python list) and just get the error mode from the top of that stack. > Thus we have no way to influence the error settings of functions > downstream from us. Of course, there is a way to do this by setting the variable in the global or builtin scope as I've described above. What's really the argument here, is whether having the flexibility at the local and global name-spaces really worth the extra name-lookups for each ufunc. I've argued that the numarray behavior can result from using the builtin namespace for the error control. (perhaps with better Python-side support for setting and retrieving it). What numpy has is control at the global and local namespace level as well which can override the builtin name-space behavior. So, we should at least frame the discussion in terms of what is actually possible. > > I also would prefer more verbose keys ala numarray (underflow, > overflow, dicidebyzero and invalid) than those currently used by numpy > (under, over, divide and invalid). In my mind, verbose keys are just extra baggage unless they are really self documenting. You just need reminders and clues. It seems to be a preference thing. I guess I hate typing long strings when only the first few letters clue me in to what is being talked about. > And (will he never stop) I like numarrays defaults better here too: > overflow='warn', underflow='ignore', dividebyzero='warn', > invalid='warn'. Currently, numpy defaults to ignore for all cases. > These last points are relatively minor though. This has optimization issues the way the code is written now. The defaults are there to produce the fastest loops. So, I'm hesitant to change them based only on ambiguous preferences. Good feedback. Thanks again for taking the time to look at this and offer review. -Travis > > Regards, > > -tim > > > [1] I believe it's actually thread local, and if it's not it should > be, but that detail is not important for our discussion here. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion