NumPy-Discussion April 2008

numpy-discussion@python.org

127 participants
190 discussions

Forcing the use of -lgfortran
by Andreas Klöckner 06 Apr '08

06 Apr '08

Hi all, I'm having trouble getting numpy to compile something usable on a cluster I'm using, in particular I see 8< ----------------------------------------------------- ImportError: /users/kloeckner/mach/x86_64/pool/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: _gfortran_st_write 8< ----------------------------------------------------- Key point: I need -lgfortran on the link command line, or else I get unresolved symbols stemming from my LAPACK library. But even if I add this: 8< ----------------------------------------------------- [blas_opt] libraries = f77blas, cblas, atlas,gfortran library_dirs = /users/kloeckner/mach/x86_64/pool/lib include_dirs = /users/kloeckner/mach/x86_64/pool/include # [lapack_opt] libraries = lapack, f77blas, cblas, atlas,gfortran library_dirs = /users/kloeckner/mach/x86_64/pool/lib 8< ----------------------------------------------------- to site.cfg, numpy seemingly ignores this request and uses ATLAS's ptblas instead (which I positively do *not* want it to use). How can I fix this? This is what I get for __config__.py: 8< ----------------------------------------------------- blas_opt_info={'libraries': ['ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'language': 'c', 'define_macros': [('ATLAS_INFO', '"\\"3.8.1\\""')], 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} atlas_blas_threads_info={'libraries': ['ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'langu age': 'c', 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} lapack_opt_info={'libraries': ['lapack', 'ptf77blas', 'ptcblas', 'atlas'], 'library_dirs': ['/users/kloeckner/mach/x86_64/pool/lib'], 'lan guage': 'f77', 'define_macros': [('ATLAS_INFO', '"\\"3.8.1\\""')], 'include_dirs': ['/users/kloeckner/mach/x86_64/pool/include']} 8< ----------------------------------------------------- Thanks, Andreas

1 1

Re: [Numpy-discussion] Simple financial functions for NumPy
by Neil Crighton 05 Apr '08

05 Apr '08

I'm just a numpy user, but for what it's worth, I would much prefer to have a single numpy namespace with a small as possible number of objects inside that namespace. To me, 'as small as possible' means that it only includes the array and associated array manipulation functions (searchsorted, where, and the record array functions), and the various functions that operate on arrays (exp, log, sin, cos, abs, any, etc). Having a small number of objects in a single namespace means that numpy is much easier to learn for beginners, as it's easier to find the appropriate thing for what you want to do (this is also helped by reducing duplication *shakes fist at .compress* and good documentation). It's also much easier to explore with dir() to jog your memory as to what function you need for a task. If I felt I contributed enough to this list to have a '1', I would be -1 on adding financial functions to numpy. > On Fri, Apr 4, 2008 at 3:31 PM, Anne Archibald <peridot.faceted(a)gmail.com> > wrote: > > > On 04/04/2008, Alan G Isaac <aisaac(a)american.edu> wrote: > > > > It seems to me that there are two separate issues people are talking > > about when they talk about packaging: > > > > * How should functions be arranged in the namespaces? numpy.foo(), > > scipy.foo(), numpy.lib.financial.foo(), scikits.foo(), > > numkitfull.foo()? > > > > * Which code should be distributed together? Should scipy require > > separate downloading and compilation from numpy? > > > > The two questions are not completely independent - it would be > > horribly confusing to have the set of functions available in a given > > namespace depend on which packages you had installed - but for the > > most part it's not a problem to have several toplevel namespaces in > > one package (python's library is the biggest example of this I know > > of). > > > > For the first question, there's definitely a question about how much > > should be done with namespaces and how much with documentation. The > > second is a different story. > > > > Personally, I would prefer if numpy and scipy were distributed > > together, preferably with matplotlib. Then anybody who used numpy > > would have available all the scpy tools and all the plotting tools; I > > think it would cut down on wheel reinvention and make application > > development easier. Teachers would not need to restrict themselves to > > using only functions built into numpy for fear that their students > > might not have scipy installed - how many students have learned to > > save their arrays in unportable binary formats because their teacher > > didn't want them to have to install scipy? > > > > I realize that this poses technical problems. For me installing scipy > > is just a matter of clicking on a checkbox and installing a 30 MB > > package, but I realize that some platforms make this much more > > difficult. If we can't just bundle the two, fine. But I think it is > > mad to consider subdividing further if we don't have to. > > > If these were tightly tied together, for instance in one big dll , this > would be unpleasant for me. I still have people downloading stuff over 56k > modems and adding an extra 30 MB to the already somewhat bloated numpy > distribution would make there lives more tedious than they already are. > > I think python's success is due in part to its "batteries included" > > > library. The fact that you can just write a short python script with > > no extra dependencies that can download files from the Web, parse XML, > > manage subprocesses, and save persistent objects makes development > > much faster than if you had to forever decide between adding > > dependencies and reinventing the wheel. I think numpy and scipy should > > follow the same principle, of coming "batteries included". > > > One thing they try to do in Python proper is think a lot more before adding > stuff to the standard library. Generally packages need to exist separately > for some period of time to prove there general utility and to stabilize > before they get accepted. Particularly in the core, but in the library as > well, they make an effort to chose a compact set of primitive operations > without a lot of duplication (the old "There should be one-- and preferably > only one --obvious way to do it."). The numpy community has, particularly of > late, been rather quick to add things that seem like they *might *be useful. > > One of the advantages of having multiple namespaces would have been to > enforce a certain amount of discipline on what went into numpy, since it > would've been easier to look at and evaluate a few dozen functions that > might have comprised some subpackage rather than, let's say, five hundred or > so. > > I suspect it's too late now; numpy has chosen the path of matlab and the > other array packages and is slowly accumulating nearly everything in one big > flat namespace. I don't like it, but it seems pointless to fight it at this > point. > > > So in this specific case, yes, I think the financial functions should > > absolutely be included; whether they should be included in scipy or > > numpy is less important to me because I think everyone should install > > both packages. > > > > Personally I think it's a bad idea to add stuff that, as far as I can tell, > no has even asked for yet. Put them in the sandbox. Advertise them. If > people use them, figure out what needs to be changed. Then add them to SciPy > once they've stabilized, if they actually get used. > >

1 0

Simple financial functions for NumPy
by Travis E. Oliphant 05 Apr '08

05 Apr '08

Hi all, Last night I put together some simple financial functions based on the basic ones available in Excel (and on a financial calculator). It seems to me that NumPy ought to have these basic functions. There may be some disagreement about what to call them and what the interface should be. I've stuck with the Excel standard names and functionality because most of the people that will use these in the future, I expect will have seen them from Excel and it minimizes the impedance mismatch to have them be similarly named. The interface is also similar to the IMSL libraries. However, if clearly better interfaces can be discovered, then we could change it. For now, the functions are not imported into the numpy namespace but live in numpy.lib.financial I could see a future scipy module containing much, much more. Comments and improvement suggestions welcome. We are a week away from release of NumPy 1.0.5, and hopefully we can agree before then. Best regards, -Travis O.

10 12

Problem with numpy.linalg.eig?
by Peter Creasey 05 Apr '08

05 Apr '08

Hi all, The following code calling numpy v1.0.4 fails to terminate on my machine, which was not the case with v1.0.3.1 from numpy import arange, float64 from numpy.linalg import eig a = arange(13*13, dtype = float64) a.shape = (13,13) a = a%17 eig(a) Regards, Peter

6 19

Efficient reading of binary data
by Nicolas Bigaouette 04 Apr '08

04 Apr '08

Hi, I have a C program which outputs large (~GB) files. It is a simple binary dump of an array of structure containing 9 doubles. You can see this as a double 1D array of size 9*Stot (Stot being the allocated size of the array of structure). The 1D array represents a 3D array (Sx * Sy * Sz = Stot) containing 9 values per cell. I want to read these files in the most efficient way possible, and I would like to have your insight on this. Right now, the fastest way I found was: imzeros = zeros((Sy,Sz),dtype=float64,order='C') imex = imshow(imzeros) f = open(filename, 'rb') data = numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot) mask_Ex = numpy.arange(6,9*Stot,9) Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() imex.set_array(squeeze(Ex3D[:,:,z])) The arrays will be big, so everything should be well optimized. I have multiple questions: 1) Should I change this: Ex = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() imex.set_array(squeeze(Ex3D[:,:,z])) to: imex.set_array(squeeze(data[mask].reshape((Sz,Sy,Sx), order='C').transpose()[:,:,z])) I mean, is I don't use a temporary variable, will it be faster or less memory hungry? 2) If not, is the operation "Ex = " update the variable data or create another one? Ideally I would like to only update it. Maybe this would be better: Ex[:,:,:] = data[mask].reshape((Sz,Sy,Sx), order='C').transpose() Would it? 3) The machine where the code will be run might be big-endian. Is there a way for python to read the big-endian file and "translate" it automatically to little-endian? Something like "numpy.fromfile(file=f, dtype=numpy.float64, count=9*Stot, endianness='big')"? Thanx a lot! ;) Nicolas

5 8

sorting ndarray
by harryos 04 Apr '08

04 Apr '08

i have a 1 dim numpy array D=array( [[ 3. , 2. , 1. , 4. , 5. , 1.5, 2.2]] ) i need to get this sorted in descending order and then access the elements . D.sort() will make D as [[ 1. 1.5 2. 2.2 3. 4. 5. ]] how will i reverse it? or is there a simpler way? harry

3 2

numpy unittest failure on Mac PPC and Solaris 10
by Christopher Hanley 04 Apr '08

04 Apr '08

Hi, We are seeing the following error on both Solaris and Mac PPC when running the numpy unittests: .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F....................................................................................................................................................................................................................................................................... ====================================================================== FAIL: test_record (numpy.lib.tests.test_io.Testloadtxt) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/stsci/pyssgdev/2.5.1/numpy/lib/tests/test_io.py", line 42, in test_record assert_array_equal(x, a) File "/usr/stsci/pyssgdev/2.5.1/numpy/testing/utils.py", line 225, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/usr/stsci/pyssgdev/2.5.1/numpy/testing/utils.py", line 217, in assert_array_compare assert cond, msg AssertionError: Arrays are not equal (mismatch 100.0%) x: array([(1, 2), (3, 4)], dtype=[('x', '>i4'), ('y', '>i4')]) y: array([(1, 2), (3, 4)], dtype=[('x', '<i4'), ('y', '<i4')]) ---------------------------------------------------------------------- Ran 837 tests in 4.281s FAILED (failures=1) Given the platforms this error occurs on I am guessing this is a big vs. little-endian problem. Chris -- Christopher Hanley Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338

1 0

loading data with gaps
by Tim Michelsen 04 Apr '08

04 Apr '08

Hello! How can I load a data file (e.g. CSV, DAT) in ASCII which has some gaps? The file has been saved with from a spreadsheet program which leaves cells with not data empty: 1,23. 2,13. 3, 4,34. Would this code be correct: ### test_loadtxt.py ### import numpy import maskedarray # load data which has empty 'cells' as beeing saved from spreadsheet: # 1,23. # 2,13. # 3, # 4,34. data = numpy.loadtxt('./loadtxt_test.csv',dtype=str,delimiter=',') # create a masked array with all no data ('', empty cells from CSV) masked my_masked_array = maskedarray.masked_equal(data,'') ###### * How can I change the data type of my maskedarray (my_masked_array) to a type that allows me to perform calulations? * Would you do this task differently or more efficient? * What possibilities do I have to estimate/interpolate the masked values? A example would be nice. * How to I convert maskedarray (my_masked_array) to a array without masked values? Thanks in advance for your help, Tim Michelsenholg

2 1

Re: [Numpy-discussion] Simple financial functions for NumPy
by Joe Harrington 04 Apr '08

04 Apr '08

+1 for simple financial functions in numpy, and congrats that it's on OLPC! If we have an FFT in numpy, we should have an internal rate of return. Anyone with investments needs that, and that's more people than those needing an FFT. I agree that Excel will bring in the most familiarity, but their names are not always rational. Please don't propagate irrational names. Consider looking at what they're called in Matlab and IDL, as code conversion/familiarity from those communities counts as well. Maybe for each function take the most rational name and arg order from those three sources, with strong preference for Excel unless there is a clear better way to do it. --jh--

2 1

multiply array
by wilson 04 Apr '08

04 Apr '08

hello i have two arrays #of shape (1,6) eval=array([[3.,3.2,1.,1.1,5.,0.5]]) #of shape (6,8) egimgs=array([ [3.,2.,1.,4.,5.,1.5,2.5,1.1], [1.1,3.,.5,.2,.1,4.3,3.2,1.2], [4.,3.,2.,6.,1.,4.,5.1,2.4], [3.2,1.3,2.2,4.4,1.1,2.1,3.3,2.4], [.4,.2,.1,.5,.3,.2,1.2,4.2], [5.2,2.1,4.3,5.5,2.2,1.1,1.4,3.2] ] ) i need to multiply the first row of egimgs with the first element of eval , second row of egimgs with second element of eval ..etc ...and so on..finally i should get a result array of same shape as egimgs. how should i proceed ? using loops seems inefficient.can someone suggest a better way? thanks W

3 3

Jump to page:

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion April 2008