Mailman 3 April 2010 - NumPy-Discussion

proposing a "beware of [as]matrix()" warning
by David Warde-Farley 04 May '10

04 May '10

Trying to debug code written by an undergrad working for a colleague of mine who ported code over from MATLAB, I am seeing an ugly melange of matrix objects and ndarrays that are interacting poorly with each other and various functions in SciPy/other libraries. In particular there was a custom minimizer function that contained a line "a * b", that was receiving an Nx1 "matrix" and a N-length array and computing an outer product. Hence the unexpected 6 GB of memory usage and weird results... We've had this discussion before and it seems that the matrix class isn't going anywhere (I *really* wish it would at least be banished from the top-level namespace), but it has its adherents for pedagogical reasons. Could we at least consider putting a gigantic warning on all the functions for creating matrix objects (matrix, mat, asmatrix, etc.) that they may not behave quite so predictably in some situations and should be avoided when writing nontrivial code? There are already such warnings scattered about on SciPy.org but the situation is so bad, in my opinion (bad from a programming perspective and bad from a new user perspective, asking "why doesn't this work? why doesn't that work? why is this language/library/etc. so stupid, inconsistent, etc.?") that the situation warrants steering people still further away from the matrix object. I apologize for ranting, but it pains me when people give up on Python/NumPy because they can't figure out inconsistencies that aren't really there for a good reason. IMHO, of course. David David

16 32

Poll: Semantics for % in Cython
by Dag Sverre Seljebotn 04 May '10

04 May '10

(First off, is it OK to continue polling the NumPy list now and then on Cython language decisions? Or should I expect that any interested Cython users follow the Cython list?) In Python, if I write "-1 % 5", I get 4. However, in C if I write "-1 % 5" I get -1. The question is, what should I get in Cython if I write (a % b) where a and b are cdef ints? Should I [ ] Get 4, because it should behave just like in Python, avoiding surprises when adding types to existing algorithms (this will require extra logic and be a bit slower) [ ] Get -1, because they're C ints, and besides one isn't using Cython if one doesn't care about performance Whatever we do, this also affects the division operator, so that one in any case will have a==(a//b)*b+a%b. (Orthogonal to this, we can introduce compiler directives to change the meaning of the operator from the default in a code blocks, and/or make special functions for the semantics that are not chosen as default.) -- Dag Sverre

10 16

ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ?
by Sebastian Haase 03 May '10

03 May '10

Hi, I wanted to write some C code to accept labels as they come from ndimage.label. For some reason ndimage.label produces its output as an int32 array - even on my 64bit system . BTW, could this be considered a bug ? Now, if I use the typemaps of numpy.i I can choose between NPY_LONG and NPY_INT. But those are sometimes 32 sometimes 64 bit, depending on the system. Any ideas ... ? Thanks, Sebastian Haase

6 10

How to make scipy.interpolate give a an extrapolated result beyond the input range?
by Salim, Fadhley (CA-CIB) 30 Apr '10

30 Apr '10

I'm trying to port a program which currently uses a hand-rolled C++ interpolator (developed by a mathematician colleage) over to use the interpolators provided by scipy. I'd like to use or wrap the scipy interpolator so that it's behavior is as close as possible behavior to our old interpolator. A key difference between the two functions is that in our original interpolator - if the input value is above or below the input range, our original interpolator will extrapolate the result. If you try this with the scipy interpolator it raises a ValueError. Consider this program as an example: # EXAMPLE import numpy as np from scipy import interpolate x = np.arange(0,10) y = np.exp(-x/3.0) f = interpolate.interp1d(x, y) print f(9) print f(11) ##### Causes ValueError, because it's greater than max(x) # END OF EXAMPLE In the example above, I'd like the last line not to raise a ValueError, but to return a value calculated from the gradient of the line between f(x[-2]) and f(x[-1]). Is there a sensible way to make it so that instead of crashing, the final line will simply do a linear extrapolate, continuing the gradients defined by the first and last pairs of input data-points to infinity? I know that this is a simple enough function to write myself, however I'd rather not re-invent the wheel, especially as if I wanted to introduce new basic math functions into our library they would need to be validated by a number of gate-keepers before they were permitted into our library! I'm on Python 2.4, scipy 0.7 on Windows XP, 32bit Incidentally, I have seen this tutorial which has a "left" and "right" argument on the interpolator. This does not seem to exist on any version of the interp1d function which I can use on Windows Python 2.4 - can anybody speculate which version of Scipy this tutorial is intended for? http://projects.scipy.org/scipy/browser/branches/Interpolate1D/docs/tuto rial.rst?rev=4591 Sal This email does not create a legal relationship between any member of the Crédit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Crédit Agricole Corporate and Investment Bank is authorised by the Comité des Etablissements de Crédit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Crédit Agricole Corporate and Investment Bank is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA.

2 1

Fwd: [SciPy-Dev] numpy 1.4.1 f2py on Snow Leopard.
by Charles سمير Doutriaux 30 Apr '10

30 Apr '10

I'm forwarding this to the numpy list as well since it seems to be f2py related therefore belonging here really. C. Begin forwarded message: > From: Charles سمير Doutriaux <doutriaux1(a)llnl.gov> > Date: April 30, 2010 9:10:08 AM PDT > To: SciPy Developers List <scipy-dev(a)scipy.org> > Subject: Re: [SciPy-Dev] numpy 1.4.1 f2py on Snow Leopard. > > Ok, > > It seems to be totally unrelated. Even using your suggested > gfortran. On a 32bit machine. I still get the same error. > > Since scipy is HUGE I'm sending a REALLY simple example that fails > for me. > > omar /git/cdat >gfortran --version > GNU Fortran (GCC) 4.2.3 > Copyright (C) 2007 Free Software Foundation, Inc. > > GNU Fortran comes with NO WARRANTY, to the extent permitted by law. > You may redistribute copies of GNU Fortran > under the terms of the GNU General Public License. > For more information about these matters, see the file named COPYING > > omar /git/cdat >gcc --version > i686-apple-darwin9-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5574) > Copyright (C) 2007 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There > is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > PURPOSE. > > omar /git/cdat >uname -a > Darwin omar 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 > PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 > > > > On Apr 29, 2010, at 4:52 PM, Jonathan Guyer wrote: > >> On Apr 29, 2010, at 12:17 PM, Charles سمير Doutriaux wrote: >> >>> I jsut built numpy 1.4.1 on my mac (64bit only no universal) >>> Now f2py fails building any fortran code... >> >> >>> meryem /svn/cdat/trunk >gfortran --version >>> GNU Fortran (GCC) 4.5.0 20100107 (experimental) >>> Copyright (C) 2009 Free Software Foundation, Inc. >> >> I've recently been writing up instructions for building FiPy's >> prerequisites on Snow Leopard http://*matforge.org/fipy/wiki/ >> SnowLeopardSourceBuild, largely based on this blog posting: http:// >> *blog.hyperjeff.net/?p=160 >> >> There seem to be issues with gfortran 4.5.0, whereas gfortran 4.2.3 >> has worked for me and others. >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev(a)scipy.org >> http://*mail.scipy.org/mailman/listinfo/scipy-dev >

1 0

Numpy compilation error
by Pradeep Jha 30 Apr '10

30 Apr '10

Hi, I am trying to install a software called Cantera which uses numpy. When I run the make command, during the installation it gives me the following error: */bin/rm -f _build (CXX="g++"; export CXX; CC="g++"; export CC; CFLAGS="-O3 -Wall -fPIC"; export CFLAGS; PURIFY=""; export PURIFY; /usr/bin/python2 setup.py build) Traceback (most recent call last): File "setup.py", line 3, in ? from distutils.core import setup, Extension File "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/__init__.py", line 6, in ? import ccompiler File "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/ccompiler.py", line 7, in ? from distutils import ccompiler ImportError: cannot import name ccompiler make[1]: *** [_build] Error 1 make[1]: Leaving directory `/nfs/carv/d1/people/pradeep/cantera/Cantera/python' make: *** [python] Error 2* My numpy is stored in the following directory */nfs/carv/d1/people/pradeep/src/numpy-1.3.0/ *I am running Linux (Fedora). When I do a sys.path in command line Python, the output is: *['', '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0', '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy', '/nfs/carv/d1/people/pradeep/cantera', '/nfs/carv/d1/people/pradeep/cantera/bin', '/nfs/carv/d1/people/pradeep/cantera/build/bin/i686-pc-linux-gnu', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/build/lib.linux-i686-2.4/Cantera', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/Cantera', '/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages', '/usr/lib/python2.4/site-packages/Numeric', '/usr/lib/python2.4/site-packages/gtk-2.0'] * I have a lot of unwanted directories as I was trying out different things. Any clues whats going wrong? Thanks

5 21

What should be the value of nansum of nan's?
by Charles R Harris 29 Apr '10

29 Apr '10

Hi All, We need to make a decision for ticket #1123<http://projects.scipy.org/numpy/ticket/1123#comment:11>regarding what nansum should return when all values are nan. At some earlier point it was zero, but currently it is nan, in fact it is nan whatever the operation is. That is consistent, simple and serves to mark the array or axis as containing all nans. I would like to close the ticket and am a bit inclined to go with the current behaviour although there is an argument to be made for returning 0 for the nansum case. Thoughts? Chuck

6 8

numpy 1.4.1 f2py on Snow Leopard.
by Charles سمير Doutriaux 29 Apr '10

29 Apr '10

Hello, I jsut built numpy 1.4.1 on my mac (64bit only no universal) Now f2py fails building any fortran code... my python: 2.6.5 meryem /svn/cdat/trunk >gcc --version i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646) (dot 1) Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. meryem /svn/cdat/trunk >gfortran --version GNU Fortran (GCC) 4.5.0 20100107 (experimental) Copyright (C) 2009 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING I'm also attaching bzipped logs of my numpy and scipy (as an example that would be familiar to you guys) installs. scipy obviously fails because the f2py doesn't work. Thx, C.

1 0

Recommended way to add Cython extension using numpy.distutils?
by Matthew Brett 28 Apr '10

28 Apr '10

Hi, We (neuroimaging.scipy.org) are using numpy.distutils, and we have .pyx files that we build with Cython. I wanted to add these in our current setup.py scripts, with something like: def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration config = Configuration('statistics', parent_package, top_path) config.add_extension('intvol', ['intvol.pyx'], include_dirs = [np.get_include()]) return config but of course numpy only knows about Pyrex, and returns: error: Pyrex required for compiling 'nipy/algorithms/statistics/intvol.pyx' but notavailable Is there a recommended way to plumb Cython into the numpy build machinery? Should I try and patch numpy distutils to use Cython if present? Best, Matthew

3 3

Fwd: Advice for grouping recarrays
by Tom Denniston 28 Apr '10

28 Apr '10

Someone inquired about this one today and I wanted to clarify there is now a better way to do this that I didn't know about when I posted the original: >>> ind = numpy.array([0,0,0,0,1,1,1,2,2,2,]) >>> data = numpy.arange(10) >>> borders = numpy.arange(len(ind)).compress(numpy.hstack([[1], ind[1:]!=ind[:-1]])) >>> numpy.add.reduceat(data, borders) array([ 6, 15, 24]) On Tue, Jul 18, 2006 at 8:49 AM, Tom Denniston <tom.denniston(a)alum.dartmouth.org> wrote: > I suggest > > lexsort > itertools.groupby of the indices > take > > I think it would be really great if numpy had the first two as a > function or something like that. It is really useful to be able to > take an array and bucket it and apply further numpy operations like > accumulation functions. > > On 7/18/06, Stephen Simmons <mail(a)stevesimmons.com> wrote: >> >> Hi, >> >> Does anyone have any suggestions for summarising data in numpy? >> >> The quick description is that I want to do something like the SQL >> statement: >> SELECT sum(field1), sum(field2) FROM table GROUP BY field3; >> >> The more accurate description is that my data is stored in PyTables HDF >> format, with 24 monthly files, each with 4m records describing how >> customers performed that month. Each record looks something like this: >> ('200604', 651404500000L, '800', 'K', 12L, 162.0, 2000.0, 0.054581, 0.0, >> 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, >> 8.80, 0.86, 7.80 17.46, 0.0, 70.0, 0.0, 70.0, -142.93, 0.0, 2000.0, >> 2063.93, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -9.71, 7.75, >> 87.46, 77.75, -3.45, 0.22, -0.45, -0.57, 73.95) >> The first 5 fields are status fields (month_string, account_number, >> product_code, account_status, months_since_customer_joined). The >> remaining 48 fields represent different aspects of the customer's >> performance during that month. I read 100,000 of these records at a time >> and turn them into a numpy recarray with: >> dat = hdf_table.read(start=pos, stop=pos+block_size) >> dat = numpy.asarray(dat._flatArray, dtype=dat.array_descr) >> >> I'd like to reduce these 96m records x 53 fields down to monthly >> averages for each tuple (month_string, months_since_customer_joined) >> which in the case above is ('200604', 12L). This will let me compare the >> performance of newly acquired customers at the same point in their >> lifecycle as customers acquired 1 or 2 years ago. >> >> The end result should be a dataset something like >> res[month_index, months_since_customer_joined] >> = array([ num_records, sum_field_5, sum_field_6, sum_field_7, ... >> sum_field_52 ]) >> with a shape of (24, 24, 49). >> >> I've played around with lexsort(), take(), sum(), etc, but get very >> confused and end up feeling that I'm making things more complicated than >> they need to be. So any advice from numpy veterans on how best to >> proceed would be very welcome! >> >> Cheers >> >> Stephen >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share >> your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >

1 0