Mailman 3 December 2008 - NumPy-Discussion

numpy.random and multiprocessing
by Gael Varoquaux Dec. 12, 2008

Dec. 12, 2008

Hi there, I have been using the multiprocessing module a lot to do statistical tests such as Monte Carlo or resampling, and I have just discovered something that makes me wonder if I haven't been accumulating false results. Given two files: === test.py === from test_helper import task from multiprocessing import Pool p = Pool(4) jobs = list() for i in range(4): jobs.append(p.apply_async(task, (4, ))) print [j.get() for j in jobs] p.close() p.join() === test_helper.py === import numpy as np def task(x): return np.random.random(x) ======= If I run test.py, I get: [array([ 0.35773964, 0.63945684, 0.50855196, 0.08631373]), array([ 0.35773964, 0.63945684, 0.50855196, 0.08631373]), array([ 0.35773964, 0.63945684, 0.50855196, 0.08631373]), array([ 0.65357725, 0.35649382, 0.02203999, 0.7591353 ])] In other words, the 4 processes give me the same exact results. Now I understand why this is the case: the different instances of the random number generator where created by forking from the same process, so they are exactly the very same object. This is howver a fairly bad trap. I guess other people will fall into it. The take home message is: **call 'numpy.random.seed()' when you are using multiprocessing** I wonder if we can find a way to make this more user friendly? Would be easy, in the C code, to check if the PID has changed, and if so reseed the random number generator? I can open up a ticket for this if people think this is desirable (I think so). On a side note, there are a score of functions in numpy.random with __module__ to None. It makes it inconvenient to use it with multiprocessing (for instance it forced the creation of the 'test_helper' file here). Gaël

10 30

Save a class
by Ross Williamson Dec. 12, 2008

Dec. 12, 2008

Dear all I have a class that contains various data arrays and constants Is there a way of using numpy.save() to save the class so that when I reload it back in I have access to all the member arrays? Thanks Ross

2 1

Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64)
by George Goussard Dec. 12, 2008

Dec. 12, 2008

Hello David. I am using the Intel MKL BLAS/LAPACK. I have replaced this with AMD's ACML library. Now there is no exception raised due to a "Singular matrix" while trying to move the legend(wiggling the graph). So, the graph is updated and the interaction is fine(you can wiggle the graph and it updates, minimize, maximeie etc.). But ... the legend is now only drawn sometimes and the graphs are drawn with an intermittent line, as if the - - - pattern was specified. Something is still not right. I just can't seem to put my finger on it since there are some many parties involved(numpy,matplotlib,python, ctypes etc.) I also ran the numpy.test() with NUmpy that I compiled with AMD's ACML. The results are included: Running unit tests for numpy NumPy version 1.2.1 Results of numpy.test() NumPy is installed in C:\Development\Python\2_5_2\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Dec 12 2008, 08:38:07) [MSC v.1400 64 bit (AMD64)] nose version 0.10.4 Forcing DISTUTILS_USE_SDK=1 ............................................................................................................................................ .........................................................................................................K..K............................... ............................................................................................................................................ .......................K.................................................................................Ignoring "MSVCCompiler instance has no attribute '_MSVCCompiler__root'" (I think it is msvccompiler.py bug) ...........................S................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ............................................................................................................................................ ....................................................................................... ---------------------------------------------------------------------- Ran 1592 tests in 10.704s OK (KNOWNFAIL=3, SKIP=1) <nose.result.TextTestResult run=1592 errors=0 failures=0> Thanks. George. Message: 3 Date: Tue, 9 Dec 2008 02:43:25 +0900 From: "David Cournapeau" <cournape(a)gmail.com> Subject: Re: [Numpy-discussion] Singular Matrix problem with Matplitlib in Numpy (Windows - AMD64) To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Message-ID: <5b8d13220812080943g69d4c670jabd6aef66d336e29(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Tue, Dec 9, 2008 at 12:50 AM, George Goussard <gwg(a)emss.co.za> wrote: > Hello. > > > > I have been battling with the following error for the past week. The output > from the terminal is: > What does numpy.test() says ? Did you use an external blas/lapack when you built numpy for AMD64 David ------------------------------ _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion End of Numpy-discussion Digest, Vol 27, Issue 22 ************************************************

1 0

building numpy trunk on WindowsXP
by josef.pktd＠gmail.com Dec. 12, 2008

Dec. 12, 2008

I just tried to build numpy for the first time, on Windows XP SP2, sse2, single CPU with MingW 3.45, Python25 I used `setup.py bdist` and copied extracted archive into sitepackages, (so I can delete it again) Are the errors and failures below expected, or did my build not work correctly? Josef Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.3.0.dev6139 NumPy is installed in C:\Programs\Python25\lib\site-packages\numpy Python version 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Int el)] nose version 0.10.4 ................................................................................ ................................................................................ ................................................................................ .......FF....................................................................... ................................................................................ ...........................................K.................................... ...............................................................Ignoring "Python was built with Visual Studio 2003; extensions must be built with a compiler than can generate compatible binaries. Visual Studio 2003 was not found on this system. If you have Cygwin installed, you can try compiling with MingW32, by passing "-c mingw32" to setup.py." (one s hould fix me in fcompiler/compaq.py) ................................................................................ ................................................................................ ................................................................................ ................................................................................ ..................E..F.....E.................................................... ................................................................................ ................................................................................ ................................................................................ ................................................................................ ......S......................................................................... ................................................................................ ................................................................................ ................................................................................ ............................................ ====================================================================== ERROR: test_mmap (test_io.TestSaveLoad) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 64, in test_mmap self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'}) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 72, in roundtrip RoundtripTest.roundtrip(self, np.save, *args, **kwargs) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 40, in roundtrip arr_reloaded = np.load(load_file, **load_kwds) File "C:\Programs\Python25\lib\site-packages\numpy\lib\io.py", line 137, in lo ad fid = _file(file,"rb") IOError: [Errno 13] Permission denied: 'c:\\docume~1\\carrasco\\locals~1\\temp\\ tmp2zygyo' ====================================================================== ERROR: test_mmap (test_io.TestSavezLoad) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 64, in test_mmap self.roundtrip(a, file_on_disk=True, load_kwds={'mmap_mode': 'r'}) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 77, in roundtrip RoundtripTest.roundtrip(self, np.savez, *args, **kwargs) File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 40, in roundtrip arr_reloaded = np.load(load_file, **load_kwds) File "C:\Programs\Python25\lib\site-packages\numpy\lib\io.py", line 137, in lo ad fid = _file(file,"rb") IOError: [Errno 13] Permission denied: 'c:\\docume~1\\carrasco\\locals~1\\temp\\ tmpedpthb' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\core\tests\test_print.py", line 28, in test_complex_types assert_equal(str(t(x)), str(complex(x))) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 183 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '(0+5.9287877500949585e-323j)' DESIRED: '(1+0j)' ====================================================================== FAIL: Check formatting. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\core\tests\test_print.py", line 16, in test_float_types assert_equal(str(t(x)), str(float(x))) File "C:\Programs\Python25\lib\site-packages\numpy\testing\utils.py", line 183 , in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '0.0' DESIRED: '1.0' ====================================================================== FAIL: test_array (test_io.TestSaveTxt) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Programs\Python25\lib\site-packages\numpy\lib\tests\test_io.py", line 105, in test_array '3.000000000000000000e+00 4.000000000000000000e+00\n']) AssertionError ---------------------------------------------------------------------- Ran 1627 tests in 10.640s FAILED (KNOWNFAIL=1, SKIP=1, errors=2, failures=3) <nose.result.TextTestResult run=1627 errors=2 failures=3>

3 4

On the quality of the numpy.random.normal() distribution
by Michael Gilbert Dec. 11, 2008

Dec. 11, 2008

Hello, I have been reading that there may be potential issues with the Box-Muller transform, which is used by the numpy.random.normal() function. Supposedly, since f*x1 and f*x2 are not independent variables, then the individual elements (corresponding to f*x1 and f*x2 ) of the distribution also won't be independent. For example, see "Stochastic Simulation" by Ripley, pages 54-59, where the random values end up distributed on a spiral. Note that they mention that they only looked at "congruential generators." Is the random number generator used by numpy congruential? I have tried to generate plots that demonstrate this problem, but have come up short. For example: import numpy , pylab nsamples = 10**6 n = numpy.random.normal( 0.0 , 1.0 , nsamples ) pylab.scatter( n[0:-1:2] , n[1:-1:2] , 0.1 ) pylab.show() I can zoom in and out, and the scatter still looks random (white noise -- almost like tv static). Does this prove that there is no problem? And if so, why does numpy do a better job than as demonstrated by Ripley? Regards, Mike Gilbert

4 4

Failing to build numpy properly on Ubuntu Hardy
by Gael Varoquaux Dec. 11, 2008

Dec. 11, 2008

Hi all, Looks like I am following the long line of people failing to build numpy :). I must admit I am clueless with building problems. Numpy builds alright, but I get: ImportError: /usr/lib/sse2/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done On import. This used to work a while ago. I am not sure what I changed, but it sure does fail. I really don't understand where the gfortran comes in. I tried building numpy with or without gfortran. From what I gather it is the numpy is being built by a different compiler than the atlas libraries (hurray for ABI compatibility), but I don't really understand how this is possible. How can I debug this? Cheers, Gaël

3 6

Excluding index in numpy like negative index in R?
by Bab Tei Dec. 10, 2008

Dec. 10, 2008

Keith Goodman <kwgoodman <at> gmail.com> writes: > > On Tue, Dec 9, 2008 at 12:25 PM, Bab Tei <babaktei <at> yahoo.com> wrote: > > > I can exclude a list of items by using negative index in R (R-project) ie myarray[-excludeindex]. As > negative indexing in numpy (And python) behave differently ,how can I exclude a list of item in numpy? > > Here's a painful way to do it: > > >> x = np.array([0,1,2,3,4]) > >> excludeindex = [1,3] > >> idx = list(set(range(4)) - set(excludeindex)) > >> x[idx] > array([0, 2]) > > To make it more painful, you might want to sort idx. > > But if excludeindex is True/False, then just use ~excludeindex. > Thank you. However it seems I have to create a full list at first and then exclude items. It is somehow painful as I have some very large sparse matrices and creating a full index eats a lot of memory. Maybe adding this functionality to numpy saves memory and makes the syntax more clear ie a syntax like x[~excludeindex] which smartly distinguish between excludeindex as a list of numerical indexes and a mask (list of true/false indexes). Regards

1 0

Support for sparse matrix in Distance function (and clustering)?
by Bab Tei Dec. 10, 2008

Dec. 10, 2008

Damian Eads <eads <at> soe.ucsc.edu> writes: > > Hi, > > Can you be more specific? Do you need sparse matrices to represent > observation vectors because they are sparse? Or do you need sparse > matrices to represent distance matrices because most vectors you are > clustering are similar while a few are dissimilar? > Damian > > On Tue, Dec 9, 2008 at 1:28 PM, Bab Tei <babaktei <at> yahoo.com> wrote: > > Hi > > Does the distance function in spatial package support sparse matrix? > > regards Hi I need sparse matrices to represent observation vectors because they are sparse. I have a large sparse matrix. I also use kmeans (Besides hierarchical clustering) which can directly work with very large data. Teimourpour

1 0

rollaxis and reshape
by Elfnor Dec. 10, 2008

Dec. 10, 2008

Hi I'm trying to split an array into two pieces and have the two pieces in a new dimension. Here it is in code, because that's hard to explain in words. >>>data.shape (4, 50, 3) >>>new_data = numpy.zeros((2, 4, 25, 3)) >>>new_data[0,...] = data[:,:25,:] >>>new_data[1,...] = data[:,25:,:] >>>new_data.shape (2, 4, 25, 3) That works but when I try it with reshape the elements get in the wrong place. I've tried various combinations of rollaxis before the reshape, but can't get it right. Thanks Eleanor -- View this message in context: http://www.nabble.com/rollaxis-and-reshape-tp20943690p20943690.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

1 0

Numscons issues: numpy.core.umath_tests not built, built-in ld detection, MAIN__ not being set-up
by Peter Norton Dec. 10, 2008

Dec. 10, 2008

I've got a few issues that I hope won't be overwhelming on one message: (1) Because of some issues in the past in building numpy with numscons, the numpy.core.umath_tests don't get built with numpy+numscons (at least not as of svn version 6128). $ python -c 'import numpy; print numpy.__version__; import numpy.core.umath_tests' 1.3.0.dev6139 Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named umath_tests What needs to be done to get this module incorporated into the numscons build? (2) I've found that in numscons-0.9.4, the detection of the correct linker assumes that if gcc is in use, the linker is gnu ld. However, on solaris this isn't the recommended toolchain, so it's typical to build gcc with gnu as and the solaris /usr/ccs/bin/ld under the hood. What this means is that when setting a run_path in the binary (which we need to do) the linker flags are set to "-Wl,-rpath=<library>". However, this isn't valid for the solaris ld. It needs -R<libname>, or -Wl,-R<libname>. I'm pretty sure that on Solaris trying to link a library with -Wl,-rpath= and looking for an error should be enough to determine the correct format for the linker. (3) Numscons tries to check for the need for a MAIN__ function when linking with gfortran. However, any libraries built with numscons come out with an unsatisfied dependency on MAIN__. The log looks like this in build/scons/numpy/linalg/config.log looks like this: scons: Configure: Checking if gfortran needs dummy main - scons: Configure: "build/scons/numpy/linalg/sconf/conftest_0.c" is up to date. scons: Configure: The original builder output was: |build/scons/numpy/linalg/sconf/conftest_0.c <- | | | |int dummy() { return 0; } | | | scons: Configure: "build/scons/numpy/linalg/sconf/conftest_0.o" is up to date. scons: Configure: The original builder output was: |gcc -o build/scons/numpy/linalg/sconf/conftest_0.o -c -O3 -m64 -g -fPIC -DPIC build/scons/numpy/linalg/sconf/conftest_0.c | scons: Configure: Building "build/scons/numpy/linalg/sconf/conftest_0" failed in a previous run and all its sources are up to date. scons: Configure: The original builder output was: |gfortran -o build/scons/numpy/linalg/sconf/conftest_0 -O3 -g -L/usr/local/lib/gcc-4.3.1/amd64 -Wl,-R/usr/local/lib/gcc-4.3.1/amd64 -L/usr/local/amd64/python/lib -Wl,-R/usr/local/amd64/python/lib -L. -lgcc_s build/scons/numpy/linalg/sconf/conftest_0.o | It then goes on to discover that it needs main: scons: Configure: "build/scons/numpy/linalg/sconf/conftest_1" is up to date. scons: Configure: The original builder output was: |gfortran -o build/scons/numpy/linalg/sconf/conftest_1 -O3 -g -L/usr/local/lib/gcc-4.3.1/amd64 -Wl,-R/usr/local/lib/gcc-4.3.1/amd64 -L/usr/local/amd64/python/lib -Wl,-R/usr/local/amd64/python/lib -L. -lgcc_s build/scons/numpy/linalg/sconf/conftest_1.o | scons: Configure: (cached) MAIN__. Doesn't this clearly indicate that a dummy main is needed? I'm working around this with a silly library that just has the MAIN__ symbol in it, but I'd love to do without that. Thanks, Peter

3 3