Mailman 3 November 2012 - NumPy-Discussion

Is apply_over_axes working for np.ma function?
by Chao YUE 22 Nov '12

22 Nov '12

Dear all, I tried the np.apply_over_axes and np.ma.apply_over_axes, it seems that they are not working for the masked array? I searched the wiki and there are two tickets (1480,8417) related with this, it seems that it's a solved issue? the example is below: In [67]: a = np.arange(60.).reshape(3,4,5) In [68]: tempm = np.tile(np.array([True,True,False,False,False]),(3,4,1)) In [69]: b = np.ma.masked_array(a,mask=tempm) In [70]: np.apply_over_axes(np.ma.sum,a,[1,2]) Out[70]: array([[[ 190.]], [[ 590.]], [[ 990.]]]) In [71]: np.ma.sum(np.ma.sum(a,axis=1),axis=1) Out[71]: array([ 190., 590., 990.]) In [72]: np.ma.sum(np.ma.sum(b,axis=1),axis=1) Out[72]: masked_array(data = [126.0 366.0 606.0], mask = [False False False], fill_value = 1e+20) In [73]: np.apply_over_axes(np.ma.sum,b,[1,2]) Out[73]: array([[[ 190.]], [[ 590.]], [[ 990.]]]) In [74]: np.apply_over_axes(np.sum,b,[1,2]) Out[74]: array([[[ 190.]], [[ 590.]], [[ 990.]]]) In [75]: np.ma.apply_over_axes(np.sum,b,[1,2]) Out[75]: array([[[ 190.]], [[ 590.]], [[ 990.]]]) In [76]: np.ma.apply_over_axes(np.ma.sum,b,[1,2]) Out[76]: array([[[ 190.]], [[ 590.]], [[ 990.]]]) thanks, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

1 1

the mean, var, std of empty arrays
by Charles R Harris 22 Nov '12

22 Nov '12

What should be the value of the mean, var, and std of empty arrays? Currently In [12]: a Out[12]: array([], dtype=int64) In [13]: a.mean() Out[13]: nan In [14]: a.std() Out[14]: nan In [15]: a.var() Out[15]: nan I think the nan comes from 0/0. All of these also raise warnings the first time they are called. Chuck

6 12

Use OpenBLAS for the binary releases?
by Sturla Molden 21 Nov '12

21 Nov '12

I think NumPy and SciPy should consider to use OpenBLAS (a fork of GotoBLAS2) instead of ATLAS or f2c'd Netlib BLAS for the binary releases. Here are its virtues: * Very easy to build: Just a makefile, no configuration script or special build tools. * Building ATLAS can be a PITA. So why bother? * Faster than ATLAS, sometimes faster than MKL. * Multithreaded BLAS kernels: OpenMP on Unix, Windows threads on Windows. * The quality of its ancestor GotoBLAS is undisputed. I was the BLAS implementation of choice for major HPC projects around the World. * Free as in BSD licensed. * Funded and developed for use in major Chinese HPC projects. Actively maintained. (GotoBLAS2 is abandonware.) * Open source. The C sources are a pleasure to read, and very easy to verify. * No OpenMP on Windows means no dependency on pthreads-win32 (an LGPL library) when building with MinGW. * Builds on Windows with MinGW and MSYS, and perhaps even without MSYS. * Cygwin is not needed on Windows (this is just BS from the GotoBLAS documentation). Thus, 64-buit builds are possible (I've built it using TDM-GCC for Win64 and 32-bit MSYS). Sturla

8 26

FOSS for scientists devroom at FOSDEM 2013
by Christos Siopis 21 Nov '12

21 Nov '12

Dear members of the Numerical Python ecosystem (with apologies for cross-postings), A day-long session ("devroom") on Free/Libre and Open Source Software (FLOSS) for scientists will be held during the next FOSDEM conference, Brussels, 2-3 February 2013 (http://fosdem.org/2013). We aim at having a dozen or two short talks introducing projects, advertising brand new features of established tools, discussing issues relevant to the development of software for scientific computing, and touching on the interdependence of FLOSS and open science. You can find more info on the call for talks at: http://slayoo.github.com/fosdem2013/ The deadline for sending talk proposals is December 16th 2012. Please send your submissions or comments to: foss4scientists-devroom(a)lists.fosdem.org Please do forward this message to anyone potentially interested. Please also let us know if you have any suggestions for what would you like to hear about in the devroom. Looking forward to meeting you in Brussels. Thanks in advance. The conveners, Sylwester Arabas, Juan Antonio Añel, Christos Siopis P.S. There are open calls for main-track talks, lightning talks, and stands at FOSDEM as well, see: http://fosdem.org/2013/ -------------- I would like to add to the above general announcement, that it would be great if a main track talk were to be given at FOSDEM about the importance of scientific open source software in science and engineering today. Main track talks last 50 minutes, and are addressed to all FOSDEM participants, something that would add to the visibility of scientific software. As an extra bonus, main track speakers have their travel and hotel expenses covered by FOSDEM. I think that the numerical python "ecosystem" could serve as an excellent "case study" of the data processing and visualisation workflow, while adding an interesting historical dimension, being one of the oldest projects of its sort. If you decide to respond to the call for main track speakers, you should start here: https://fosdem.org/2013/call_for_main_speakers/ Please note the December 1 deadline. I urge you to let us (the science software devroom conveners) know about your proposed talk, so that we may send a word of recommendation to the FOSDEM committee who will make the ultimate selection. We thank you in advance for your expressions of interest and participation! _______________________________________________ foss4scientists-devroom mailing list foss4scientists-devroom(a)lists.fosdem.org https://lists.fosdem.org/listinfo/foss4scientists-devroom

1 0

Crash using "reshape"...
by Terry J. Ligocki 21 Nov '12

21 Nov '12

I am having a problem with "reshape" crashing: > python Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.6.2' >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b')) >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]] >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F') Segmentation fault If I change the "5000" to a "4000", everything is fine. I'm not running out of memory - my system had 48 GB of memory and nothing else is using a significant portion of this memory. Note: 700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 2450000000 > 2^31. I suspect somewhere in the underlying code there is a signed 32-bit integer being used for an index/pointer offset (this is running on a 64-bit machine). I did some searching of the archives and didn't find a match for this problem. Thank you for any and all help! Terry J. (Ligocki, tjligocki(a)lbl.gov) ------------------------------------------------------------------------ *Wishes * /When you wish upon a falling star, your dreams can come true. Unless it's really a meteorite hurtling to the Earth which will destroy all life. Then you're pretty much hosed no matter what you wish for. Unless it's death by meteor. / (Despair, Inc.)

3 4

Building numpy with OpenBLAS using bento
by George Nurser 21 Nov '12

21 Nov '12

Hi, I've tried to build numpy with OpenBLAS following on from the blog http://cournape.wordpress.com/2012/10/10/notes-on-building-numpyscipy-with-… I've cloned and installed an up to date version of bento from git. Problem is that the bentomaker command doesn't seem to know about the --with-blas-lapack-libdir option. The only place this option is defined in the code seems to be in the blas_lapack.py routine .... but I can't see how to use this routine -- it isn't imported into any other file. (I've tried sending this smessage to the bento list, but it doesn't seem to be appearing on it as far as I can see.) Best regards, George Nurser

2 1

confused about univaritespline
by Neal Becker 20 Nov '12

20 Nov '12

I don't understand why the plot of the spline continues on a negative slope at the end, but the plot of the integral of it flattens. ----------------------------------------------------- import numpy as np import matplotlib.pyplot as plt ibo = np.array ((12, 14, 16, 18, 20, 22, 24, 26, 28, 29, 29.8, 30.2)) gain_deriv = np.array ((0, 0, 0, 0, 0, 0, .2, .4, .5, .5, 0, -2)) import scipy.interpolate s = scipy.interpolate.UnivariateSpline(ibo, gain_deriv, s=0.1) xs = linspace(12, 31, 100) gain = np.vectorize (lambda x: s.integral (12, x)) (xs) plot (xs, s(xs)) plot (xs, gain)

3 3

Confused with qr decomposition function
by Virgil Stokes 20 Nov '12

20 Nov '12

I am using the latest versions of numpy (from numpy-1.7.0b2-win32-superpack-python2.7.exe) and scipy (from scipy-0.11.0-win32-superpack-python2.7.exe ) on a windows 7 (32-bit) platform. I have used import numpy as np q,r = np.linalg.qr(A) and compared the results obtained from MATLAB (R2010B) [q,r] = qr(A) The q,r returned from numpy are both the negative of theq,r returned from MATLAB for the same matrix A. I believe that the q,r returned from MATLAB are correct. Why am I getting their negative from numpy? Note, I have tried this on several different matrices --- numpy always gives the negative of MATLAB's q,r values. [I mistakenly have already sent a similar email to the scipy list --- please excuse this mistake.]

3 5

the fast way to loop over ndarray elements?
by Chao YUE 20 Nov '12

20 Nov '12

Dear all, I need to make a linear contrast of the 2D numpy array "data" from an interval to another, the approach is: I have another two list: "base" & "target", then I check for each ndarray element "data[i,j]", if base[m] =< data[i,j] <= base[m+1], then it will be linearly converted to be in the interval of (target[m], target[m+1]), using another function called "lintrans". #The way I do is to loop each row and column of the 2D array, and finally loop the intervals constituted by base list: for row in range(data.shape[0]): for col in range(data.shape[1]): for i in range(len(base)-1): if data[row,col]>=base[i] and data[row,col]<=base[i+1]: data[row,col]=lintrans(data[row,col],(base[i],base[i+1]),(target[i],target[i+1])) break #use break to jump out of loop as the data have to be ONLY transferred ONCE. Now the profiling result shows that most of the time has been used in this loop over the array ("plot_array_transg"), and less time in calling the linear transformation fuction "lintrans": ncalls tottime percall cumtime percall filename:lineno(function) 18047 0.110 0.000 0.110 0.000 mathex.py:132(lintrans) 1 12.495 12.495 19.061 19.061 mathex.py:196(plot_array_transg) so is there anyway I can speed up this loop? Thanks for any suggestions!! best, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

4 5

Compiling NumPy on Windows for Python 3.3 with MSVC 2010
by Peter Cock 20 Nov '12

20 Nov '12

Changing title to reflect the fact this thread is now about using the Microsoft compiler rather than mingw32 as in the old thread. On Sat, Nov 10, 2012 at 11:04 PM, Peter Cock <p.j.a.cock(a)googlemail.com> wrote: > On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: >> >> >> >> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <p.j.a.cock(a)googlemail.com> >> wrote: >>> >>> Dear all, >>> >>> Since the NumPy 1.7.0b2 release didn't include a Windows >>> (32 bit) installer for Python 3.3, I am considering compiling it >>> myself for local testing. What compiler is recommended? >> >> >> Either MSVC or MinGW 3.4.5. For the latter see >> https://github.com/certik/numpy-vendor > > Thanks Ralf, > > I was trying with MSVC 9.0 installed, but got this cryptic error: > > C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build > ... > error: Unable to find vcvarsall.bat > > After sprinkling distutils with debug statements, I found it was > looking for MSVC v10 (not numpy's fault but the error is most > unhelpful). > > Presumably "Microsoft Visual C++ 2010 Express Edition" is > the appropriate thing to download? > http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express > I would have tried this earlier, but it required Windows XP SP3 and there was something amiss with the permissions on my work Windows machine preventing that update. Solved now, and this file now exists: C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat I've tried my branch (focussed on mingw32 fixes), and numpy-1.7.0b2 but both fail at the same point. I did remove the old build directory first. C:\Downloads\numpy-1.7.0b2>c:\python33\python setup.py build Converting to Python3 via 2to3... <snip> Could not locate executable efc don't know how to compile Fortran code on platform 'nt' C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Inumpy\core\src\private -Inumpy\core\src -Inumpy\core -Inumpy\ core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include -Ic:\python33\include /Tc_configtest.c /Fo_configtest.obj Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /nologo /INCREMENTAL:NO _configtest.obj /OUT:_configtest.exe /MANIFESTFILE:_configtest.exe.manifest Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe mt.exe -nologo -manifest _configtest.exe.manifest -outputresource:_configtest.exe;1 Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe _configtest.exe.manifest : general error c1010070: Failed to load and parse the manifest. The system cannot find the file specified. failure. removing: _configtest.c _configtest.obj Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\core.py", line 186 , in setup return old_setup(**new_attr) File "c:\python33\lib\distutils\core.py", line 148, in setup dist.run_commands() File "c:\python33\lib\distutils\dist.py", line 917, in run_commands self.run_command(cmd) File "c:\python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "c:\python33\lib\distutils\command\build.py", line 126, in run self.run_command(cmd_name) File "c:\python33\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "c:\python33\lib\distutils\dist.py", line 936, in run_command cmd_obj.run() File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src. py", line 152, in run self.build_sources() File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src. py", line 163, in build_sources self.build_library_sources(*libname_info) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src. py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src. py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 648, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program It appears a similar issue was raised before: http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html Any tips? Peter

2 6

NumPy-Discussion November 2012