Mailman 3 December 2008 - NumPy-Discussion

numpy.random and multiprocessing
by Gael Varoquaux 12 Dec '08

12 Dec '08

Hi there, I have been using the multiprocessing module a lot to do statistical tests such as Monte Carlo or resampling, and I have just discovered something that makes me wonder if I haven't been accumulating false results. Given two files: === test.py === from test_helper import task from multiprocessing import Pool p = Pool(4) jobs = list() for i in range(4): jobs.append(p.apply_async(task, (4, ))) print [j.get() for j in jobs] p.close() p.join() === test_helper.py === import numpy as np def task(x): return np.random.random(x) ======= If I run test.py, I get: [array([ 0.35773964, 0.63945684, 0.50855196, 0.08631373]), array([ 0.35773964, 0.63945684, 0.50855196, 0.08631373]), array([ 0.35773964, 0.63945684, 0.50855196, 0.08631373]), array([ 0.65357725, 0.35649382, 0.02203999, 0.7591353 ])] In other words, the 4 processes give me the same exact results. Now I understand why this is the case: the different instances of the random number generator where created by forking from the same process, so they are exactly the very same object. This is howver a fairly bad trap. I guess other people will fall into it. The take home message is: **call 'numpy.random.seed()' when you are using multiprocessing** I wonder if we can find a way to make this more user friendly? Would be easy, in the C code, to check if the PID has changed, and if so reseed the random number generator? I can open up a ticket for this if people think this is desirable (I think so). On a side note, there are a score of functions in numpy.random with __module__ to None. It makes it inconvenient to use it with multiprocessing (for instance it forced the creation of the 'test_helper' file here). Gaël

10 30

Save a class
by Ross Williamson 12 Dec '08

12 Dec '08

Dear all I have a class that contains various data arrays and constants Is there a way of using numpy.save() to save the class so that when I reload it back in I have access to all the member arrays? Thanks Ross

2 1

Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64)
by George Goussard 12 Dec '08

12 Dec '08

Hello David. I am using the Intel MKL BLAS/LAPACK. I have replaced this with AMD's ACML library. Now there is no exception raised due to a "Singular matrix" while trying to move the legend(wiggling the graph). So, the graph is updated and the interaction is fine(you can wiggle the graph and it updates, minimize, maximeie etc.). But ... the legend is now only drawn sometimes and the graphs are drawn with an intermittent line, as if the - - - pattern was specified. Something is still not right. I just can't seem to put my finger on it since there are some many parties involved(numpy,matplotlib,python, ctypes etc.) I also ran the numpy.test() with NUmpy that I compiled with AMD's ACML. The results are included: Running unit tests for numpy NumPy version 1.2.1 Results of numpy.test() NumPy is installed in C:\Development\Python\2_5_2\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Dec 12 2008, 08:38:07) [MSC v.1400 64 bit (AMD64)] nose version 0.10.4 Forcing DISTUTILS_USE_SDK=1 ............................................................................................................................................ .........................................................................................................K..K............................... ............................................................................................................................................ .......................K.................................................................................Ignoring "MSVCCompiler instance has no attribute '_MSVCCompiler__root'" (I think it is msvccompiler.py bug) ...........................S................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ....................................................................................... ---------------------------------------------------------------------- Ran 1592 tests in 10.704s OK (KNOWNFAIL=3, SKIP=1) <nose.result.TextTestResult run=1592 errors=0 failures=0> Thanks. George. Message: 3 Date: Tue, 9 Dec 2008 02:43:25 +0900 From: "David Cournapeau" <cournape(a)gmail.com> Subject: Re: [Numpy-discussion] Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64) To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Message-ID: <5b8d13220812080943g69d4c670jabd6aef66d336e29(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Tue, Dec 9, 2008 at 12:50 AM, George Goussard <gwg(a)emss.co.za> wrote: > Hello. > > > > I have been battling with the following error for the past week. The output > from the terminal is: > What does numpy.test() says ? Did you use an external blas/lapack when you built numpy for AMD64 David ------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion End of Numpy-discussion Digest, Vol 27, Issue 22 ************************************************

1 0

building numpy trunk on WindowsXP
by josef.pktd＠gmail.com 12 Dec '08

12 Dec '08

I just tried to build numpy for the first time, on Windows XP SP2, sse2, single CPU with MingW 3.45, Python25 I used `setup.py bdist` and copied extracted archive into sitepackages, (so I can delete it again) Are the errors and failures below expected, or did my build not work correctly? Josef Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.3.0.dev6139 NumPy is installed in C:\Programs\Python25\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] nose version 0.10.4 ................................................................................ ................................................................................ ................................................................................ .......FF....................................................................... ................................................................................ ...........................................K.................................... ...............................................................Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s hould fix me in fcompiler/compaq.py) ................................................................................ ................................................................................ ................................................................................ ................................................................................ ..................E..F.....E.................................................... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ......S......................................................................... ................................................................................ ................................................................................ ................................................................................ ............................................ ====================================================================== ERROR: test_mmap (test_io.TestSaveLoad) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 64, in test_mmap self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'}) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 72, in roundtrip RoundtripTest.roundtrip(self, np.save, *args, **kwargs) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 40, in roundtrip arr_reloaded = np.load(load_file, **load_kwds) File "C:\Programs\Python25\lib\site-packages\numpy\lib\io.py", line 137, in lo ad fid = _file(file,"rb") IOError: [Errno 13] Permission denied: 'c:\\docume~1\\carrasco\\locals~1\\temp\\ tmp2zygyo' ====================================================================== ERROR: test_mmap (test_io.TestSavezLoad) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 64, in test_mmap self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'}) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 77, in roundtrip RoundtripTest.roundtrip(self, np.savez, *args, **kwargs) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 40, in roundtrip arr_reloaded = np.load(load_file, **load_kwds) File "C:\Programs\Python25\lib\site-packages\numpy\lib\io.py", line 137, in lo ad fid = _file(file,"rb") IOError: [Errno 13] Permission denied: 'c:\\docume~1\\carrasco\\locals~1\\temp\\ tmpedpthb' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\core\tests\test_print.py", line 28, in test_complex_types assert_equal(str(t(x)), str(complex(x))) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 183 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '(0+5.9287877500949585e-323j)' DESIRED: '(1+0j)' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\core\tests\test_print.py", line 16, in test_float_types assert_equal(str(t(x)), str(float(x))) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 183 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '0.0' DESIRED: '1.0' ====================================================================== FAIL: test_array (test_io.TestSaveTxt) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 105, in test_array '3.000000000000000000e+00 4.000000000000000000e+00\n']) AssertionError ---------------------------------------------------------------------- Ran 1627 tests in 10.640s FAILED (KNOWNFAIL=1, SKIP=1, errors=2, failures=3) <nose.result.TextTestResult run=1627 errors=2 failures=3>

3 4

On the quality of the numpy.random.normal() distribution
by Michael Gilbert 11 Dec '08

11 Dec '08

Hello, I have been reading that there may be potential issues with the Box-Muller transform, which is used by the numpy.random.normal() function. Supposedly, since f*x1 and f*x2 are not independent variables, then the individual elements (corresponding to f*x1 and f*x2 ) of the distribution also won't be independent. For example, see "Stochastic Simulation" by Ripley, pages 54-59, where the random values end up distributed on a spiral. Note that they mention that they only looked at "congruential generators." Is the random number generator used by numpy congruential? I have tried to generate plots that demonstrate this problem, but have come up short. For example: import numpy , pylab nsamples = 10**6 n = numpy.random.normal( 0.0 , 1.0 , nsamples ) pylab.scatter( n[0:-1:2] , n[1:-1:2] , 0.1 ) pylab.show() I can zoom in and out, and the scatter still looks random (white noise -- almost like tv static). Does this prove that there is no problem? And if so, why does numpy do a better job than as demonstrated by Ripley? Regards, Mike Gilbert

4 4

Failing to build numpy properly on Ubuntu Hardy
by Gael Varoquaux 11 Dec '08

11 Dec '08

Hi all, Looks like I am following the long line of people failing to build numpy :). I must admit I am clueless with building problems. Numpy builds alright, but I get: ImportError: /usr/lib/sse2/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done On import. This used to work a while ago. I am not sure what I changed, but it sure does fail. I really don't understand where the gfortran comes in. I tried building numpy with or without gfortran. From what I gather it is the numpy is being built by a different compiler than the atlas libraries (hurray for ABI compatibility), but I don't really understand how this is possible. How can I debug this? Cheers, Gaël

3 6

Excluding index in numpy like negative index in R?
by Bab Tei 10 Dec '08

10 Dec '08

Keith Goodman <kwgoodman <at> gmail.com> writes: > > On Tue, Dec 9, 2008 at 12:25 PM, Bab Tei <babaktei <at> yahoo.com> wrote: > > > I can exclude a list of items by using negative index in R (R-project) ie myarray[-excludeindex]. As > negative indexing in numpy (And python) behave differently ,how can I exclude a list of item in numpy? > > Here's a painful way to do it: > > >> x = np.array([0,1,2,3,4]) > >> excludeindex = [1,3] > >> idx = list(set(range(4)) - set(excludeindex)) > >> x[idx] > array([0, 2]) > > To make it more painful, you might want to sort idx. > > But if excludeindex is True/False, then just use ~excludeindex. > Thank you. However it seems I have to create a full list at first and then exclude items. It is somehow painful as I have some very large sparse matrices and creating a full index eats a lot of memory. Maybe adding this functionality to numpy saves memory and makes the syntax more clear ie a syntax like x[~excludeindex] which smartly distinguish between excludeindex as a list of numerical indexes and a mask (list of true/false indexes). Regards

1 0

Support for sparse matrix in Distance function (and clustering)?
by Bab Tei 10 Dec '08

10 Dec '08

Damian Eads <eads <at> soe.ucsc.edu> writes: > > Hi, > > Can you be more specific? Do you need sparse matrices to represent > observation vectors because they are sparse? Or do you need sparse > matrices to represent distance matrices because most vectors you are > clustering are similar while a few are dissimilar? > Damian > > On Tue, Dec 9, 2008 at 1:28 PM, Bab Tei <babaktei <at> yahoo.com> wrote: > > Hi > > Does the distance function in spatial package support sparse matrix? > > regards Hi I need sparse matrices to represent observation vectors because they are sparse. I have a large sparse matrix. I also use kmeans (Besides hierarchical clustering) which can directly work with very large data. Teimourpour

1 0

rollaxis and reshape
by Elfnor 10 Dec '08

10 Dec '08

Hi I'm trying to split an array into two pieces and have the two pieces in a new dimension. Here it is in code, because that's hard to explain in words. >>>data.shape (4, 50, 3) >>>new_data = numpy.zeros((2, 4, 25, 3)) >>>new_data[0,...] = data[:,:25,:] >>>new_data[1,...] = data[:,25:,:] >>>new_data.shape (2, 4, 25, 3) That works but when I try it with reshape the elements get in the wrong place. I've tried various combinations of rollaxis before the reshape, but can't get it right. Thanks Eleanor -- View this message in context: http://www.nabble.com/rollaxis-and-reshape-tp20943690p20943690.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

1 0

Numscons issues: numpy.core.umath_tests not built, built-in ld detection, MAIN__ not being set-up
by Peter Norton 10 Dec '08

10 Dec '08

I've got a few issues that I hope won't be overwhelming on one message: (1) Because of some issues in the past in building numpy with numscons, the numpy.core.umath_tests don't get built with numpy+numscons (at least not as of svn version 6128). $ python -c 'import numpy; print numpy.__version__; import numpy.core.umath_tests' 1.3.0.dev6139 Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named umath_tests What needs to be done to get this module incorporated into the numscons build? (2) I've found that in numscons-0.9.4, the detection of the correct linker assumes that if gcc is in use, the linker is gnu ld. However, on solaris this isn't the recommended toolchain, so it's typical to build gcc with gnu as and the solaris /usr/ccs/bin/ld under the hood. What this means is that when setting a run_path in the binary (which we need to do) the linker flags are set to "-Wl,-rpath=<library>". However, this isn't valid for the solaris ld. It needs -R<libname>, or -Wl,-R<libname>. I'm pretty sure that on Solaris trying to link a library with -Wl,-rpath= and looking for an error should be enough to determine the correct format for the linker. (3) Numscons tries to check for the need for a MAIN__ function when linking with gfortran. However, any libraries built with numscons come out with an unsatisfied dependency on MAIN__. The log looks like this in build/scons/numpy/linalg/config.log looks like this: scons: Configure: Checking if gfortran needs dummy main - scons: Configure: "build/scons/numpy/linalg/sconf/conftest_0.c" is up to date. scons: Configure: The original builder output was: |build/scons/numpy/linalg/sconf/conftest_0.c <- | | | |int dummy() { return 0; } | | | scons: Configure: "build/scons/numpy/linalg/sconf/conftest_0.o" is up to date. scons: Configure: The original builder output was: |gcc -o build/scons/numpy/linalg/sconf/conftest_0.o -c -O3 -m64 -g -fPIC -DPIC build/scons/numpy/linalg/sconf/conftest_0.c | scons: Configure: Building "build/scons/numpy/linalg/sconf/conftest_0" failed in a previous run and all its sources are up to date. scons: Configure: The original builder output was: |gfortran -o build/scons/numpy/linalg/sconf/conftest_0 -O3 -g -L/usr/local/lib/gcc-4.3.1/amd64 -Wl,-R/usr/local/lib/gcc-4.3.1/amd64 -L/usr/local/amd64/python/lib -Wl,-R/usr/local/amd64/python/lib -L. -lgcc_s build/scons/numpy/linalg/sconf/conftest_0.o | It then goes on to discover that it needs main: scons: Configure: "build/scons/numpy/linalg/sconf/conftest_1" is up to date. scons: Configure: The original builder output was: |gfortran -o build/scons/numpy/linalg/sconf/conftest_1 -O3 -g -L/usr/local/lib/gcc-4.3.1/amd64 -Wl,-R/usr/local/lib/gcc-4.3.1/amd64 -L/usr/local/amd64/python/lib -Wl,-R/usr/local/amd64/python/lib -L. -lgcc_s build/scons/numpy/linalg/sconf/conftest_1.o | scons: Configure: (cached) MAIN__. Doesn't this clearly indicate that a dummy main is needed? I'm working around this with a silly library that just has the MAIN__ symbol in it, but I'd love to do without that. Thanks, Peter

3 3

Importance of order when summing values in an array
by Hanni Ali 10 Dec '08

10 Dec '08

Hi All, I have encountered a puzzling issue and I am not certain if this is a mistake of my own doing or not. Would someone kindly just look over this issue to make sure I'm not doing something very silly. So, why would the sum of an array have a different value depending on the order I select the indices of the array? >>> vector[[39, 46, 49, 50, 6, 9, 12, 14, 15, 17, 21]].sum() 8933281.8757099733 >>> vector[[6, 9, 12, 14, 15, 17, 21, 39, 46, 49, 50]].sum() 8933281.8757099714 >>> sum(vector[[39, 46, 49, 50, 6, 9, 12, 14, 15, 17, 21]]) 8933281.8757099733 >>> sum(vector[[6, 9, 12, 14, 15, 17, 21, 39, 46, 49, 50]]) 8933281.8757099714 Any thoughts? Cheers, Hanni

7 17

Find index of repeated numbers in array
by Ross Williamson 10 Dec '08

10 Dec '08

Hi Everyone I think I'm missing something really obvious but what I would like to do is extract the indexes from an array where a number matches - For example data = [0,1,2,960,5,6,960,7] I would like to know, for example the indices which match 960 - i.e. it would return 3 and 6 I could do this with a loop but I was wondering if there was a built in numpy function to do this? BTW if anyone is interested I'm converting some idl code to numpy and trying to mmic the IDL function where Cheers Ross

4 4

Python2.4 support
by Pierre GM 10 Dec '08

10 Dec '08

All, * What versions of Python should be supported by what version of numpy ? Are we to expect users to rely on Python2.5 for the upcoming 1.3.x ? Could we have some kind of timeline on the trac site or elsewhere (and if such a timeline exists already, can I get the link?) ? * Talking about 1.3.x, what's the timeline? Are we still shooting for a release in 2008 or could we wait till mid Jan. 2009 ? Thx a lot in advance

11 18

access ndarray in C++
by Thomas Hrabe 10 Dec '08

10 Dec '08

Hi all! I am currently developing a python module in C/C++ which is supposed to access nd arrays as defined by the following command in python a = numpy.array([1,1,1]) I want to access the array the following way and use the nd array data for further processing in C. mymod.doSthg(a) The example code on http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm#pgfId-36721 (!PyArg_ParseTuple(args, "O!",&PyArray_Type, &array)) does not work for nd arrays. I always get TypeError: argument 1 must be array, not numpy.ndarray I assume the error is the constant provided as the third parameter, saying that the imput is of PyArray_Type and no nd array. So here are my questions: 1. is there any good tutorial / example code for acessing nd arrays in C? 2. what is the difference between both (arrays and nd arrays? - I am new to python and heard there were different approaches to arrays and that nd arrays work better for multi dimensional applications. Is this right?) 3. which one of both will be used in the future? Thank you in advance for your help, Thomas

15 31

Superclassing numpy.matrix: got an unexpected keyword argument 'dtype'
by Robert.Conde＠sungard.com 10 Dec '08

10 Dec '08

Hello, I'm using numpy-1.1.1 for Python 2.3. I'm trying to create a class that acts just like the numpy.matrix class with my own added methods and attributes. I want to pass my class a list of custom "instrument" objects and do some math based on these objects to set the matrix. To this end I've done the following: from numpy import matrix class rcMatrix(matrix): def __init__(self,instruments): """Do some calculations and set the values of the matrix.""" self[0,0] = 100 # Just an example self[0,1] = 100 # The real init method self[1,0] = 200 # Does some math based on the input objects self[1,1] = 300 # def __new__(cls,instruments): """When creating a new instance begin by creating an NxN matrix of zeroes.""" len_ = len(instruments) return matrix.__new__(cls,[[0.0]*len_]*len_) It works great and I can, for example, multiply two of my custom matrices seamlessly. I can also get the transpose. However, when I try to get the inverse I get an error: > rcm = rcMatrix(['instrument1','instrument2']) > print rcm [[ 100. 100.] [ 200. 300.]] > print rcm.T [[ 100. 200.] [ 100. 300.]] > print [5,10] * rcm [[ 2500. 3500.]] > print rcm.I Traceback (most recent call last): File "[Standard]/deleteme", line 29, in ? File "C:\Python23\Lib\site-packages\numpy\core\defmatrix.py", line 492, in getI return asmatrix(func(self)) File "C:\Python23\Lib\site-packages\numpy\core\defmatrix.py", line 52, in asmatrix return matrix(data, dtype=dtype, copy=False) TypeError: __init__() got an unexpected keyword argument 'dtype' I've had to overwrite the getI function in order for things to work out: def getI(self): return matrix(self.tolist()).I I = property(getI, None, doc="inverse") Is this the correct way to achieve my goals? Please let me know if anything is unclear. Thanks, Robert Conde

2 1

Little vectorization help...
by Andrea Gavana 10 Dec '08

10 Dec '08

Hi All, I am tryin to "vectorize" 3 nested for loops but I am not having much success. Here is the code I use: import numpy import numpy.ma as masked grid = numpy.zeros((nx, ny), dtype=numpy.float32) xOut = numpy.zeros((nx, ny), dtype=numpy.float32) yOut = numpy.zeros((nx, ny), dtype=numpy.float32) z = GetCentroids() # Some vector z values prop = GetValue() # Some other vector values NaN = numpy.NaN for I in xrange(1, nx+1): for J in xrange(1, ny+1): theSum = [] for K in xrange(1, nz+1): cellPos = I-1 + nx*(J-1) + nx*ny*(K-1) centroid = z[cellPos] if low <= centroid <= high and actnum[cellPos] > 0: theSum.append(prop[cellPos]) if theSum: grid[I-1, J-1] = sum(theSum)/len(theSum) else: grid[I-1, J-1] = NaN xOut[I-1, J-1], yOut[I-1, J-1] = x[cellPos], y[cellPos] grid = masked.masked_where(numpy.isnan(grid), grid) Some explanation: 1) "z" is a vector of nx*ny*nz components, where nx = 100, ny = 73, nz = 23, which represents 3D hexahedron cell centroids; 2) "prop" is a vector like z, with the same shape, with some floating point values in it; 3) "actnum" is a vector of integers (0 or 1) with the same shape as z, and indicates if a cell should be considered in the loop or not; 4) low and high are 2 floating point values with low < high: if the cell centroid fall between low and high and the cell is active (as stated in "actnum"), then I take the value of "prop" in that cell and I append it to the "theSum" list; 5) At the end of the K loop, I just take an arithmetic mean of the values in "theSum" list. I think I may be able to figure out how to vectorize the part regarding the "grid" variable, but I have no idea on what to do for the xOut and yOut variables, and I need them because I use them in a later call to matplotlib.contourf. If you could drop some hint on how to proceed, ot would be very appreciated. Thank you for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/

1 0

Little vectorization help...
by Andrea Gavana 10 Dec '08

10 Dec '08

Hi All, I am tryin to "vectorize" 3 nested for loops but I am not having much success. Here is the code I use: import numpy import numpy.ma as masked grid = numpy.zeros((nx, ny), dtype=numpy.float32) xOut = numpy.zeros((nx, ny), dtype=numpy.float32) yOut = numpy.zeros((nx, ny), dtype=numpy.float32) z = GetCentroids() # Some vector z values prop = GetValue() # Some other vector values NaN = numpy.NaN for I in xrange(1, nx+1): for J in xrange(1, ny+1): theSum = [] for K in xrange(1, nz+1): cellPos = I-1 + nx*(J-1) + nx*ny*(K-1) centroid = z[cellPos] if low <= centroid <= high and actnum[cellPos] > 0: theSum.append(prop[cellPos]) if theSum: grid[I-1, J-1] = sum(theSum)/len(theSum) else: grid[I-1, J-1] = NaN xOut[I-1, J-1], yOut[I-1, J-1] = x[cellPos], y[cellPos] grid = masked.masked_where(numpy.isnan(grid), grid) Some explanation: 1) "z" is a vector of nx*ny*nz components, where nx = 100, ny = 73, nz = 23, which represents 3D hexahedron cell centroids; 2) "prop" is a vector like z, with the same shape, with some floating point values in it; 3) "actnum" is a vector of integers (0 or 1) with the same shape as z, and indicates if a cell should be considered in the loop or not; 4) low and high are 2 floating point values with low < high: if the cell centroid fall between low and high and the cell is active (as stated in "actnum"), then I take the value of "prop" in that cell and I append it to the "theSum" list; 5) At the end of the K loop, I just take an arithmetic mean of the values in "theSum" list. I think I may be able to figure out how to vectorize the part regarding the "grid" variable, but I have no idea on what to do for the xOut and yOut variables, and I need them because I use them in a later call to matplotlib.contourf. If you could drop some hint on how to proceed, ot would be very appreciated. Thank you for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/

1 0

Some numpy statistics
by Charles R Harris 10 Dec '08

10 Dec '08

Hi All, I bumped into this while searching for something else: http://www.ohloh.net/p/numpy/analyses/latest Chuck

4 4

How to unitize a array in numpy?
by Grissiom 10 Dec '08

10 Dec '08

Hi all, Nice to neet you all. I am a newbie in numpy. Is there any function that could unitize a array? Thanks in advance. -- Cheers, Grissiom

2 5

Support for sparse matrix in Distance function (and clustering)?
by Bab Tei 09 Dec '08

09 Dec '08

Hi Does the distance function in spatial package support sparse matrix? regards

2 1

Excluding index in numpy like negative index in R?
by Bab Tei 09 Dec '08

09 Dec '08

Hi I can exclude a list of items by using negative index in R (R-project) ie myarray[-excludeindex]. As negative indexing in numpy (And python) behave differently ,how can I exclude a list of item in numpy? Regards, Teimourpour

3 2

numpy build error on Solaris, No module named _md5
by Gong, Shawn (Contractor) 09 Dec '08

09 Dec '08

hi list, I tried to build numpy 1.2.1 on Solaris 9 with gcc 3.4.6 when I typed "python setup.py build", I got error from hashlib.py File "/home/sgong/dev181/dist/lib/python2.5/hashlib.py", line 133, in <module> md5 = __get_builtin_constructor('md5') File "/home/sgong/dev181/dist/lib/python2.5/hashlib.py", line 60, in __get_builtin_constructor import _md5 ImportError: No module named _md5 I then tried python 2.6.1 instead of 2.5.2, but got the same error. I did not get the error while building on Linux. But I performed steps on Linux: 1) copy *.a Atlas libraries to my local_install/atlas/ 2) ranlib *.a 3) created a site.cfg Do I need to do the same on Solaris? Any help is appreciated. thanks, Shawn

4 5

Line of best fit!
by James 09 Dec '08

09 Dec '08

Hi, I am trying to plot a line of best fit for some data i have, is there a simple way of doing it? Cheers

8 13

Please help prepare the SciPy 0.7 release notes
by Jarrod Millman 09 Dec '08

09 Dec '08

We are almost ready for SciPy 0.7.0rc1 (we just need to sort out the Numerical Recipes issues and I haven't had time to look though them yet). So I wanted to ask once more for help with preparing the release notes: http://projects.scipy.org/scipy/scipy/browser/trunk/doc/release/0.7.0-notes… There have been numerous improvements and changes. As always I would appreciate any feedback about mistakes or omissions. It would also be nice to know how many tests were in the last release and how many are there now. Highlighting major bug fixes or pointing out know issues would be very useful. I would also like to ask if anyone would be interested in stepping forward to work on something like Andrew Kuchling's "What's New in Python ....": http://docs.python.org/whatsnew/2.6.html This would be a great area to contribute. The release notes provide visibility for our developers' immense contributions of time and effort. They help provide an atmosphere of momentum, maturity, and excitement to a project. It is also a great service to users who haven't been following the trunk closely as well as other developer's who have missed what is happening in other areas of the code. It is also becomes a nice historical artifact for the future. It would be great if someone wanted to contribute in this way. Ideally, I would like to have someone who be interested in doing this for several releases of scipy and numpy. Such a person could develop a standard template for this and write some scripts to gather specific statistics (e.g., how many lines of code have changed, how many unit tests were added, what is the test coverage, what is the docstring coverage, who were the top contributors, who has increased their code contributions the most, how many new developers, etc.) Just a thought. Figure it won't happen, if I don't ask. Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

1 0

when will osx linker option -bundle be reflected in distutils
by Garry Willgoose 09 Dec '08

09 Dec '08

I was just wondering what plans there were to reflect the different linker options (i.e. -bundle instead of -shared) that are required on OSX in the fcompiler files within distutils. While its a minor thing it always catches the users of my software when they either install fresh or update numpy ... and sometimes on a bad day it even catches me ;-) ==================================================================== Prof Garry Willgoose, Australian Professorial Fellow in Environmental Engineering, Director, Centre for Climate Impact Management (C2IM), School of Engineering, The University of Newcastle, Callaghan, 2308 Australia. Centre webpage: www.c2im.org.au Phone: (International) +61 2 4921 6050 (Tues-Fri AM); +61 2 6545 9574 (Fri PM-Mon) FAX: (International) +61 2 4921 6991 (Uni); +61 2 6545 9574 (personal and Telluric) Env. Engg. Secretary: (International) +61 2 4921 6042 email: garry.willgoose(a)newcastle.edu.au; g.willgoose(a)telluricresearch.com email-for-life: garry.willgoose(a)alum.mit.edu personal webpage: www.telluricresearch.com/garry ==================================================================== "Do not go where the path may lead, go instead where there is no path and leave a trail" Ralph Waldo Emerson ====================================================================

2 1

checksum on numpy float array
by Brennan Williams 08 Dec '08

08 Dec '08

My app reads in one or more float arrays from a binary file. Sometimes due to network timeouts etc the array is not read correctly. What would be the best way of checking the validity of the data? Would some sort of checksum approach be a good idea? Would that work with an array of floating point values? Or are checksums more for int,byte,string type data?

5 14

ANNOUNCE: EPD with Py2.5 version 4.0.30002 RC2 available for testing
by Travis E. Oliphant 06 Dec '08

06 Dec '08

Hello, We've recently posted the beta1 build of EPD (the Enthought Python Distribution) with Python 2.5 version 4.1.30001 to the EPD website. You may download the beta from here: http://www.enthought.com/products/epdearlyaccess.php You can check out the release notes here: https://svn.enthought.com/epd/wiki/Python2.5.2/4.1.300/Beta1 Please help us test it out and provide feedback on the EPD Trac instance: https://svn.enthought.com/epd or via e-mail to epd-support(a)enthought.com. If everything goes well, we are planning a final release for December. About EPD --------- The Enthought Python Distribution (EPD) is a "kitchen-sink-included" distribution of the Python™ Programming Language, including over 60 additional tools and libraries. The EPD bundle includes NumPy, SciPy, IPython, 2D and 3D visualization, database adapters, GUI building libraries, and a lot of other tools right out of the box. http://www.enthought.com/products/epd.php It is currently available as a single-click installer for Windows XP (x86), Mac OS X (a universal binary for OS X 10.4 and above), and RedHat 3 and 4 (x86 and amd64). EPD is free for academic use. An annual subscription and installation support are available for individual commercial use. Enterprise subscriptions with support for particular deployment environments are also available for commercial purchase. Enthought Build Team

5 8

subclassing ndarray
by Steve Schmerler 05 Dec '08

05 Dec '08

Hi all I'm subclassing ndarray following [1] and I'd like to know if i'm doing it right. My goals are - ndarray subclass MyArray with additional methods - replacement for np.array, np.asarray on module level returning MyArray instances - expose new methods as functions on module level import numpy as np class MyArray(np.ndarray): def __new__(cls, arr, **kwargs): return np.asarray(arr, **kwargs).view(dtype=arr.dtype, type=cls) # define new methods here ... def print_shape(self): print self.shape # replace np.array() def array(*args, **kwargs): return MyArray(np.array(*args, **kwargs)) # replace np.asarray() def asarray(*args, **kwargs): return MyArray(*args, **kwargs) # expose array method as function def ps(a): asarray(a).print_shape() Would that work? PS: I found a little error in [1]: In section "__new__ and __init__", the class def should read class C(object): def __new__(cls, *args): + print 'cls is:", cls print 'Args in __new__:', args return object.__new__(cls, *args) def __init__(self, *args): + print 'self is:", self print 'Args in __init__:', args [1] http://docs.scipy.org/doc/numpy/user/basics.subclassing.html best, steve

1 0

genloadtxt: second serving
by Pierre GM 05 Dec '08

05 Dec '08

All, Here's the second round of genloadtxt. That's a tad cleaner version than the previous one, where I tried to take into account the different comments and suggestions that were posted. So, tabs should be supported and explicit whitespaces are not collapsed. FYI, in the __main__ section, you'll find 2 hotshot tests and a timeit comparison: same input, no missing data, one with genloadtxt, one with np.loadtxt and a last one with matplotlib.mlab.csv2rec. As you'll see, genloadtxt is roughly twice slower than np.loadtxt, but twice faster than csv2rec. One of the explanation for the slowness is indeed the use of classes for splitting lines and converting values. Instead of a basic function, we use the __call__ method of the class, which itself calls another function depending on the attribute values. I'd like to reduce this overhead, any suggestion is more than welcome, as usual. Anyhow: as we do need speed, I suggest we put genloadtxt somewhere in numpy.ma, with an alias recfromcsv for John, using his defaults. Unless somebody comes with a brilliant optimization. Let me know how it goes, Cheers, P.

4 5

Py3k and numpy
by Erik Tollerud 05 Dec '08

05 Dec '08

I noticed that the Python 3000 final was released today... is there any sense of how long it will take to get numpy working under 3k? I would imagine it'll be a lot to adapt given the low-level change, but is the work already in progress?

5 5

PyArray_EMPTY and Cython
by Gabriel Gellner 04 Dec '08

04 Dec '08

After some discussion on the Cython lists I thought I would try my hand at writing some Cython accelerators for empty and zeros. This will involve using PyArray_EMPTY, I have a simple prototype I would like to get working, but currently it segfaults. Any tips on what I might be missing? import numpy as np cimport numpy as np cdef extern from "numpy/arrayobject.h": PyArray_EMPTY(int ndims, np.npy_intp* dims, int type, bint fortran) cdef np.ndarray empty(np.npy_intp length): cdef np.ndarray[np.double_t, ndim=1] ret cdef int type = np.NPY_DOUBLE cdef int ndims = 1 cdef np.npy_intp* dims dims = &length print dims[0] print type ret = PyArray_EMPTY(ndims, dims, type, False) return ret def test(): cdef np.ndarray[np.double_t, ndim=1] y = empty(10) return y The code seems to print out the correct dims and type info but segfaults when the PyArray_EMPTY call is made. Thanks, Gabriel

2 1

Apply a function to an array elementwise
by Elfnor 04 Dec '08

04 Dec '08

Hi I want to apply a function (myfunc which takes and returns a scalar) to each element in a multi-dimensioned array (data): I can do this: newdata = numpy.array([myfunc(d) for d in data.flat]).reshape(data.shape) But I'm wondering if there's a faster more numpy way. I've looked at the vectorize function but can't work it out. thanks Eleanor -- View this message in context: http://www.nabble.com/Apply-a-function-to-an-array-elementwise-tp20823768p2… Sent from the Numpy-discussion mailing list archive at Nabble.com.

3 2

in(np.nan) on python 2.6
by Pierre GM 04 Dec '08

04 Dec '08

All, Sorry to bump my own post, and I was kinda threadjacking anyway: Some functions of numy.ma (eg, ma.max, ma.min...) accept explicit outputs that may not be MaskedArrays. When such an explicit output is not a MaskedArray, a value that should have been masked is transformed into np.nan. That worked great in 2.5, with np.nan automatically transformed to 0 when the explicit output had a int dtype. With Python 2.6, a ValueError is raised instead, as np.nan can no longer be casted to int. What should be the recommended behavior in this case ? Raise a ValueError or some other exception, to follow the new Python2.6 convention, or silently replace np.nan by some value acceptable by int dtype (0, or something else) ? Thanks for any suggestion, P.

4 5

Re: [Numpy-discussion] Py3k and numpy
by Tommy Grav 04 Dec '08

04 Dec '08

On Dec 4, 2008, at 2:03 PM, Robert Kern wrote: > It does. What problems are people seeing? Is it just the Windows build > that causes people to say "numpy doesn't work with Python 2.6"? There is currently no official Mac OSX binary for numpy for python 2.6, but you can build it from source. Is there any time table for generating a 2.6 Mac OS X binary? Cheers Tommy

2 1

Broadcasting question
by Olivier Grisel 04 Dec '08

04 Dec '08

Hi list, Suppose I have array a with dimensions (d1, d3) and array b with dimensions (d2, d3). I want to compute array c with dimensions (d1, d2) holding the squared euclidian norms of vectors in a and b with size d3. My first take was to use a python level loop: >>> from numpy import * >>> c = array([sum((a_i - b) ** 2, axis=1) for a_i in a]) But this is too slow and allocate a useless temporary list of python references. To avoid the python level loop I then tried to use broadcasting as follows: >>> c = sum((a[:,newaxis,:] - b) ** 2, axis=2) But this build a useless and huge (d1, d2, d3) temporary array that does not fit in memory for large values of d1, d2 and d3... Do you have any better idea? I would like to simulate a runtime behavior similar to: >>> c = dot(a, b.T) but for for squared euclidian norms instead of dotproducts. I can always write a the code in C and wrap it with ctypes but I wondered whether this is possible only with numpy. -- Olivier

3 4

Compiler options for mingw?
by Zachary Pincus 04 Dec '08

04 Dec '08

Hi all, I'm curious about how to control compiler options for mingw builds of numpy on windows... Specifically, I want to build binaries without SSE support, so that they can run on older hardware. Setting a CFLAGS variable on the command-line doesn't appear to do anything, but perhaps appearances are deceiving. Thanks for any suggestions -- I've googled fruitlessly for a while. Zach

3 5

genloadtxt: second serving (tests)
by Pierre GM 04 Dec '08

04 Dec '08

And now for the tests:

1 0

Re: [Numpy-discussion] np.loadtxt : yet a new implementation...
by Pierre GM 03 Dec '08

03 Dec '08

Well, looks like the attachment is too big, so here's the implementation. The tests will come in another message.

7 21

trouble subclassing ndarray
by Sébastien Barthélemy 03 Dec '08

03 Dec '08

Hello, I'm trying to write a small library of differential geometry, and I have some trouble subclassing ndarray. I'd like an HomogeneousMatrix class that subclasse ndarray and overloads some methods, such as inv(). Here is my first try, the inv() function and the inv_v1() method work as expected, but the inv_v2() and inv_v3() methods do not change the object at all. Can somebody explain me what is happening here ? import numpy as np def inv(H): """ inverse of an homogeneous matrix """ R = H[0:3,0:3] p = H[0:3,3:4] return np.vstack( (np.hstack((R.T,-np.dot(R.T,p))), [0,0,0,1])) class HomogeneousMatrix(np.ndarray): def __new__(subtype, data=np.eye(4)): subarr = np.array(data) if htr.ishomogeneousmatrix(subarr): return subarr.view(subtype) else: raise ValueError def inv_v1(self): self[0:4,0:4] = htr.inv(self) def inv_v2(self): data = htr.inv(self) self = HomogeneousMatrix(data) def inv_v3(self): self = htr.inv(self) Thank you ! -- Sébastien

3 4

[F2PY] Fortran call fails in IDLE / PyScripter
by Christophe Chappet 03 Dec '08

03 Dec '08

>What version of gfortran are you using (i.e. exactly which binary did >you download)? GNU Fortran (GCC) 4.4.0 20080603 (experimental) [trunk revision 136333] >Is this a write to standard output "write (*,*) szText" ? yes, it is. I forgot to say that it also works with pydev in Eclipse but I'm looking for a simple interactive python shell that can execute some programs. IPython does the job but is less friendly than IDLE in term of program editing. Anyway, I think I will use it for now. Thanks for your reply. Regards, Christophe On Tue, Dec 2, 2008 at 08:26, Christophe Chappet <Christophe.Chappet(a)onera.fr <http://projects.scipy.org/mailman/listinfo/numpy-discussion>> wrote: >/ Hi all, />/ I compile the followinq code using "f2py -c --fcompiler=gnu95 />/ --compiler=mingw32" -m hello />/ subroutine AfficheMessage(szText) />/ character szText*100 />/ write (*,*) szText />/ return />/ end />/ />/ Using python console : />>>>/import hello />>>>/hello.affichemessage(" />/ Hello") />/ works fine ! />/ />/ I do the same in the program window of IDLE and : />/ - no message is displayed. />/ - the shell restart (or IDLE crah if launched with -n) />/ />/ Same problem with PyScripter IDE. (crash)./

1 0

ANN: HDF5 for Python 1.0
by Andrew Collette 02 Dec '08

02 Dec '08

===================================== Announcing HDF5 for Python (h5py) 1.0 ===================================== What is h5py? ------------- HDF5 for Python (h5py) is a general-purpose Python interface to the Hierarchical Data Format library, version 5. HDF5 is a versatile, mature scientific software library designed for the fast, flexible storage of enormous amounts of data. >From a Python programmer's perspective, HDF5 provides a robust way to store data, organized by name in a tree-like fashion. You can create datasets (arrays on disk) hundreds of gigabytes in size, and perform random-access I/O on desired sections. Datasets are organized in a filesystem-like hierarchy using containers called "groups", and accesed using the tradional POSIX /path/to/resource syntax. This is the fourth major release of h5py, and represents the end of the "unstable" (0.X.X) design phase. Why should I use it? -------------------- H5py provides a simple, robust read/write interface to HDF5 data from Python. Existing Python and NumPy concepts are used for the interface; for example, datasets on disk are represented by a proxy class that supports slicing, and has dtype and shape attributes. HDF5 groups are are presented using a dictionary metaphor, indexed by name. A major design goal of h5py is interoperability; you can read your existing data in HDF5 format, and create new files that any HDF5- aware program can understand. No Python-specific extensions are used; you're free to implement whatever file structure your application desires. Almost all HDF5 features are available from Python, including things like compound datatypes (as used with NumPy recarray types), HDF5 attributes, hyperslab and point-based I/O, and more recent features in HDF 1.8 like resizable datasets and recursive iteration over entire files. The foundation of h5py is a near-complete wrapping of the HDF5 C API. HDF5 identifiers are first-class objects which participate in Python reference counting, and expose the C API via methods. This low-level interface is also made available to Python programmers, and is exhaustively documented. See the Quick-Start Guide for a longer introduction with code examples: http://h5py.alfven.org/docs/guide/quick.html Where to get it --------------- * Main website, documentation: http://h5py.alfven.org * Downloads, bug tracker: http://h5py.googlecode.com * The HDF group website also contains a good introduction: http://www.hdfgroup.org/HDF5/doc/H5.intro.html Requires -------- * UNIX-like platform (Linux or Mac OS-X); Windows version is in progress. * Python 2.5 or 2.6 * NumPy 1.0.3 or later (1.1.0 or later recommended) * HDF5 1.6.5 or later, including 1.8. Some features only available when compiled against HDF5 1.8. * Optionally, Cython (see cython.org) if you want to use custom install options. You'll need version 0.9.8.1.1 or later. About this version ------------------ Version 1.0 follows version 0.3.1 as the latest public release. The major design phase (which began in May of 2008) is now over; the design of the high-level API will be supported as-is for the rest of the 1.X series, with minor enhancements. This is the first version to support Python 2.6, and the first to use Cython for the low-level interface. The license remains 3-clause BSD. ** This project is NOT affiliated with The HDF Group. ** Thanks ------ Thanks to D. Dale, E. Lawrence and other for their continued support and comments. Also thanks to the PyTables project, for inspiration and generously providing their code to the community, and to everyone at the HDF Group for creating such a useful piece of software.

3 4

[F2PY] Fortran call fails in IDLE / PyScripter
by Christophe Chappet 02 Dec '08

02 Dec '08

Hi all, I compile the followinq code using "f2py -c --fcompiler=gnu95 --compiler=mingw32" -m hello subroutine AfficheMessage(szText) character szText*100 write (*,*) szText return end Using python console : >>>import hello >>>hello.affichemessage(" Hello") works fine ! I do the same in the program window of IDLE and : - no message is displayed. - the shell restart (or IDLE crah if launched with -n) Same problem with PyScripter IDE. (crash). Any suggestion ? Regards, Christophe

3 2

bug in ma.masked_all()?
by Eric Firing 02 Dec '08

02 Dec '08

Pierre, ma.masked_all does not seem to work with fancy dtypes and more then one dimension: In [1]:import numpy as np In [2]:dt = np.dtype({'names': ['a', 'b'], 'formats': ['f', 'f']}) In [3]:x = np.ma.masked_all((2,), dtype=dt) In [4]:x Out[4]: masked_array(data = [(--, --) (--, --)], mask = [(True, True) (True, True)], fill_value=(1.0000000200408773e+20, 1.0000000200408773e+20)) In [5]:x = np.ma.masked_all((2,2), dtype=dt) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/efiring/<ipython console> in <module>() /usr/local/lib/python2.5/site-packages/numpy/ma/extras.pyc in masked_all(shape, dtype) 78 """ 79 a = masked_array(np.empty(shape, dtype), ---> 80 mask=np.ones(shape, bool)) 81 return a 82 /usr/local/lib/python2.5/site-packages/numpy/ma/core.pyc in __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, hard_mask, flag, shrink, **options) 1304 except TypeError: 1305 mask = np.array([tuple([m]*len(mdtype)) for m in mask], -> 1306 dtype=mdtype) 1307 # Make sure the mask and the data have the same shape 1308 if mask.shape != _data.shape: TypeError: expected a readable buffer object ----------------- Eric

2 7

np.loadtxt : yet a new implementation...
by Pierre GM 02 Dec '08

02 Dec '08

All, Please find attached to this message another implementation of np.loadtxt, which focuses on missing values. It's basically a combination of John Hunter's et al mlab.csv2rec, Ryan May's patches and pieces of code I'd been working on over the last few weeks. Besides some helper classes (StringConverter to convert a string into something else, NameValidator to check names..._), you'll find 3 functions: * `genloadtxt` is the base function that makes all the work. It outputs 2 arrays, one for the data (missing values being substituted by the appropriate default) and one for the mask. It would go in np.lib.io * `loadtxt` would replace the current np.loadtxt. It outputs a ndarray, where missing data being filled. It would also go in np.lib.io * `mloadtxt` would go into np.ma.io (to be created) and renamed `loadtxt`. Right now, I needed a different name to avoid conflicts. It combines the outputs of `genloadtxt` into a single masked array. You'll also several series of tests, that you can use as examples. Please give it a try and send me some feedback (bugs, wishes, suggestions). I'd like it to make the 1.3.0 release (I need some of the functionalities to improve the corresponding function in scikits.timeseries, currently fubar...) P.

9 17

fromiter typo?
by Neal Becker 01 Dec '08

01 Dec '08

Says it takes a default dtype arg, but doesn't act like it's an optional arg: fromiter (iterator or generator, dtype=None) Construct an array from an iterator or a generator. Only handles 1-dimensional cases. By default the data-type is determined from the objects returned from the iterator. ---> 20 z = fromiter (y) TypeError: function takes at least 2 arguments (1 given)

2 1

Fwd: np.loadtxt : yet a new implementation...
by Pierre GM 01 Dec '08

01 Dec '08

(Sorry about that, I pressed "Reply" instead of "Reply all". Not my day for emails...) > On Dec 1, 2008, at 1:54 PM, John Hunter wrote: >> >> It looks like I am doing something wrong -- trying to parse a CSV >> file >> with dates formatted like '2008-10-14', with:: >> >> import datetime, sys >> import dateutil.parser >> StringConverter.upgrade_mapper(dateutil.parser.parse, >> default=datetime.date(1900,1,1)) >> r = loadtxt(sys.argv[1], delimiter=',', names=True) > > John, > The problem you have is that the default dtype is 'float' (for > backwards compatibility w/ the original np.loadtxt). What you want > is to automatically change the dtype according to the content of > your file: you should use dtype=None > > r = loadtxt(sys.argv[1], delimiter=',', names=True, dtype=None) > > As you'll want a recarray, we could make a np.records.loadtxt > function where dtype=None would be the default...

2 2

Re: [Numpy-discussion] np.loadtxt : yet a new implementation...
by Pierre GM 01 Dec '08

01 Dec '08

And now for the tests:

1 0

memmap & dtype issue
by Wim Bakker 01 Dec '08

01 Dec '08

For a long time now, numpy's memmap has me puzzled by its behavior. When I use memmap straightforward on a file it seems to work fine, but whenever I try to do a memmap using a dtype it seems to gobble up the whole file into memory. This, of course, makes the use of memmap futile. I would expect that the result of such an operation would give me a true memmap and that the data would be converted to dtype on the fly. I've seen this behavior in version version 1.04, 1.1.1 and still in 1.2.1. I'm working on Windows haven't tried it on Linux. Am I doing something wrong? Are my expectations wrong? Or is this an issue somewhere deeper in numpy? I looked at the memmap.py and it seems to me that most of the work is delegated to numpy.ndarray.__new__. Something wrong there maybe? Can somebody help please? Thanks! Regards, Wim Bakker

2 1

optimising single value functions for array calculations
by Timmie 01 Dec '08

01 Dec '08

Hello, I am developing a module which bases its calculations on another specialised module. My module uses numpy arrays a lot. The problem is that the other module I am building upon, does not work with (whole) arrays but with single values. Therefore, I am currently forces to loop over the array: ### a = numpy.arange(100) b = numpy.arange(100,200) for i in range(0,a.size): a[i] = myfunc(a[i])* b[i] ### The results come out well. But the problem is that this way of calculation is very ineffiecent and takes time. May anyone give me a hint on how I can improve my code without having to modify the package I am building upon. I do not want to change it a lot because I would always have to run behind the chnages in the other package. To summarise: How to I make a calculation function array-aware? Thanks in advance, Timmie

5 5