Mailman 3 April 2009 - NumPy-Discussion

Bug in dstack?
by Charles R Harris 03 Apr '09

03 Apr '09

Note: In [133]: l = [[1,0,0],[1,1,0],[1,1,1]] In [134]: dstack(l) Out[134]: array([[[1, 1, 1], [0, 1, 1], [0, 0, 1]]]) In [135]: dstack(l).shape Out[135]: (1, 3, 3) Shouldn't the shape be (3,3)? Also, for generalized ufuncs and broadcasting I think a function that stacked along the first axis instead of the last would be useful. Maybe gstack or astack? Chuck

3 3

[Announce] Numpy 1.3.0 rc1
by David Cournapeau 02 Apr '09

02 Apr '09

Hi, I am pleased to announce the release of the rc1 for numpy 1.3.0. You can find source tarballs and installers for both Mac OS X and Windows on the sourceforge page: https://sourceforge.net/projects/numpy/ <https://sourceforge.net/projects/numpy/> The release note for the 1.3.0 release are below, The Numpy developers ========================= NumPy 1.3.0 Release Notes ========================= This minor includes numerous bug fixes, official python 2.6 support, and several new features such as generalized ufuncs. Highlights ========== Python 2.6 support ~~~~~~~~~~~~~~~~~~ Python 2.6 is now supported on all previously supported platforms, including windows. http://www.python.org/dev/peps/pep-0361/ Generalized ufuncs ~~~~~~~~~~~~~~~~~~ There is a general need for looping over not only functions on scalars but also over functions on vectors (or arrays), as explained on http://scipy.org/scipy/numpy/wiki/GeneralLoopingFunctions. We propose to realize this concept by generalizing the universal functions (ufuncs), and provide a C implementation that adds ~500 lines to the numpy code base. In current (specialized) ufuncs, the elementary function is limited to element-by-element operations, whereas the generalized version supports "sub-array" by "sub-array" operations. The Perl vector library PDL provides a similar functionality and its terms are re-used in the following. Each generalized ufunc has information associated with it that states what the "core" dimensionality of the inputs is, as well as the corresponding dimensionality of the outputs (the element-wise ufuncs have zero core dimensions). The list of the core dimensions for all arguments is called the "signature" of a ufunc. For example, the ufunc numpy.add has signature "(),()->()" defining two scalar inputs and one scalar output. Another example is (see the GeneralLoopingFunctions page) the function inner1d(a,b) with a signature of "(i),(i)->()". This applies the inner product along the last axis of each input, but keeps the remaining indices intact. For example, where a is of shape (3,5,N) and b is of shape (5,N), this will return an output of shape (3,5). The underlying elementary function is called 3*5 times. In the signature, we specify one core dimension "(i)" for each input and zero core dimensions "()" for the output, since it takes two 1-d arrays and returns a scalar. By using the same name "i", we specify that the two corresponding dimensions should be of the same size (or one of them is of size 1 and will be broadcasted). The dimensions beyond the core dimensions are called "loop" dimensions. In the above example, this corresponds to (3,5). The usual numpy "broadcasting" rules apply, where the signature determines how the dimensions of each input/output object are split into core and loop dimensions: While an input array has a smaller dimensionality than the corresponding number of core dimensions, 1's are pre-pended to its shape. The core dimensions are removed from all inputs and the remaining dimensions are broadcasted; defining the loop dimensions. The output is given by the loop dimensions plus the output core dimensions. Experimental Windows 64 bits support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now be built on windows 64 bits (amd64 only, not IA64), with both MS compilers and mingw-w64 compilers: This is *highly experimental*: DO NOT USE FOR PRODUCTION USE. See INSTALL.txt, Windows 64 bits section for more information on limitations and how to build it by yourself. New features ============ Formatting issues ~~~~~~~~~~~~~~~~~ Float formatting is now handled by numpy instead of the C runtime: this enables locale independent formatting, more robust fromstring and related methods. Special values (inf and nan) are also more consistent across platforms (nan vs IND/NaN, etc...), and more consistent with recent python formatting work (in 2.6 and later). Nan handling in max/min ~~~~~~~~~~~~~~~~~~~~~~~ The maximum/minimum ufuncs now reliably propagate nans. If one of the arguments is a nan, then nan is retured. This affects np.min/np.max, amin/amax and the array methods max/min. New ufuncs fmax and fmin have been added to deal with non-propagating nans. Nan handling in sign ~~~~~~~~~~~~~~~~~~~~ The ufunc sign now returns nan for the sign of anan. New ufuncs ~~~~~~~~~~ #. fmax - same as maximum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. fmin - same as minimum for integer types and non-nan floats. Returns the non-nan argument if one argument is nan and returns nan if both arguments are nan. #. deg2rad - converts degrees to radians, same as the radians ufunc. #. rad2deg - converts radians to degrees, same as the degrees ufunc. #. log2 - base 2 logarithm. #. exp2 - base 2 exponential. #. trunc - truncate floats to nearest integer towards zero. #. logaddexp - add numbers stored as logarithms and return the logarithm of the result. #. logaddexp2 - add numbers stored as base 2 logarithms and return the base 2 logarithm of the result result. Masked arrays ~~~~~~~~~~~~~ Several new features and bug fixes, including: * structured arrays should now be fully supported by MaskedArray (r6463, r6324, r6305, r6300, r6294...) * Minor bug fixes (r6356, r6352, r6335, r6299, r6298) * Improved support for __iter__ (r6326) * made baseclass, sharedmask and hardmask accesible to the user (but read-only) * doc update gfortran support on windows ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Gfortran can now be used as a fortran compiler for numpy on windows, even when the C compiler is Visual Studio (VS 2005 and above; VS 2003 will NOT work). Gfortran + Visual studio does not work on windows 64 bits (but gcc + gfortran does). It is unclear whether it will be possible to use gfortran and visual studio at all on x64. Arch option for windows binary ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Automatic arch detection can now be bypassed from the command line for the superpack installed: numpy-1.3.0-superpack-win32.exe /arch=nosse will install a numpy which works on any x86, even if the running computer supports SSE set. Deprecated features =================== Histogram ~~~~~~~~~ The semantics of histogram has been modified to fix long-standing issues with outliers handling. The main changes concern #. the definition of the bin edges, now including the rightmost edge, and #. the handling of upper outliers, now ignored rather than tallied in the rightmost bin. The previous behavior is still accessible using `new=False`, but this is deprecated, and will be removed entirely in 1.4.0. Documentation changes ===================== A lot of documentation has been added. Both user guide and references can be built from sphinx. New C API ========= Multiarray API ~~~~~~~~~~~~~~ The following functions have been added to the multiarray C API: * PyArray_GetEndianness: to get runtime endianness Ufunc API ~~~~~~~~~~~~~~ The following functions have been added to the ufunc API: * PyUFunc_FromFuncAndDataAndSignature: to declare a more general ufunc (generalized ufunc). New defines ~~~~~~~~~~~ New public C defines are available for ARCH specific code through numpy/npy_cpu.h: * NPY_CPU_X86: x86 arch (32 bits) * NPY_CPU_AMD64: amd64 arch (x86_64, NOT Itanium) * NPY_CPU_PPC: 32 bits ppc * NPY_CPU_PPC64: 64 bits ppc * NPY_CPU_SPARC: 32 bits sparc * NPY_CPU_SPARC64: 64 bits sparc * NPY_CPU_S390: S390 * NPY_CPU_IA64: ia64 * NPY_CPU_PARISC: PARISC New macros for CPU endianness has been added as well (see internal changes below for details): * NPY_BYTE_ORDER: integer * NPY_LITTLE_ENDIAN/NPY_BIG_ENDIAN defines Those provide portable alternatives to glibc endian.h macros for platforms without it. Portable NAN, INFINITY, etc... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ npy_math.h now makes available several portable macro to get NAN, INFINITY: * NPY_NAN: equivalent to NAN, which is a GNU extension * NPY_INFINITY: equivalent to C99 INFINITY * NPY_PZERO, NPY_NZERO: positive and negative zero respectively Corresponding single and extended precision macros are available as well. All references to NAN, or home-grown computation of NAN on the fly have been removed for consistency. Internal changes ================ numpy.core math configuration revamp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This should make the porting to new platforms easier, and more robust. In particular, the configuration stage does not need to execute any code on the target platform, which is a first step toward cross-compilation. http://projects.scipy.org/numpy/browser/trunk/doc/neps/math_config_clean.txt umath refactor ~~~~~~~~~~~~~~ A lot of code cleanup for umath/ufunc code (charris). Improvements to build warnings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Numpy can now build with -W -Wall without warnings http://projects.scipy.org/numpy/browser/trunk/doc/neps/warnfix.txt Separate core math library ~~~~~~~~~~~~~~~~~~~~~~~~~~ The core math functions (sin, cos, etc... for basic C types) have been put into a separate library; it acts as a compatibility layer, to support most C99 maths functions (real only for now). The library includes platform-specific fixes for various maths functions, such as using those versions should be more robust than using your platform functions directly. The API for existing functions is exactly the same as the C99 math functions API; the only difference is the npy prefix (npy_cos vs cos). The core library will be made available to any extension in 1.4.0. CPU arch detection ~~~~~~~~~~~~~~~~~~ npy_cpu.h defines numpy specific CPU defines, such as NPY_CPU_X86, etc... Those are portable across OS and toolchains, and set up when the header is parsed, so that they can be safely used even in the case of cross-compilation (the values is not set when numpy is built), or for multi-arch binaries (e.g. fat binaries on Max OS X). npy_endian.h defines numpy specific endianness defines, modeled on the glibc endian.h. NPY_BYTE_ORDER is equivalent to BYTE_ORDER, and one of NPY_LITTLE_ENDIAN or NPY_BIG_ENDIAN is defined. As for CPU archs, those are set when the header is parsed by the compiler, and as such can be used for cross-compilation and multi-arch binaries. Checksums ========= 5c6b2f02d0846317c6e7bffa39f6f828 release/installers/numpy-1.3.0rc1.zip 20cdddd69594420b0f8556bbc4a27a5a release/installers/numpy-1.3.0rc1.tar.gz f85231c4a27b39f7cb713ef22926931e release/installers/numpy-1.3.0rc1-py2.5-macosx10.5.dmg b24bb536492502611ea797d9410bb7c2 release/installers/numpy-1.3.0rc1-win32-superpack-python2.5.exe

12 37

Out-of-RAM FFTs
by Greg Novak 02 Apr '09

02 Apr '09

Hello, I'd like to do an FFT of a moderately large 3D cube, 1024^3. Looking at the run-time of smaller arrays, this is not a problem in terms of compute time, but the array doesn't fit in memory. So, several questions: 1) Numerical Recipes has an out-of-memory FFT algorithm, but looking through the numpy and scipy docs and modules, I didn't find a function that does the same thing. Did I miss it? Should I get to work typing it in? 2) I had high hopes for just memory-mapping the large array and passing it to the standard fft function. However, the memory-mapped region must fit into the address space, and I don't seem to be able to use more than 2 GB at a time. So memory mapping doesn't seem to help me at all. This last issue leads to another series of things that puzzle me. I have an iMac running OS X 10.5 with an Intel Core 2 duo processor and 4 GB of memory. As far as I've learned, the processor is 64 bit, the operating system is 64 bit, so I should be able to happily memory-map my entire disk if I want. However, Python seems to run out of steam when it's used 2 GB. This is true of both 2.5 and 2.6. What gives? Is this a Python issue? Thanks, Greg

7 6

GSoC Proposals Due on the 3rd
by Stéfan van der Walt 02 Apr '09

02 Apr '09

Hi all Students interested in working on NumPy or SciPy for GSoC2009, please note that the deadline for proposals is the 3rd of April. http://socghop.appspot.com/document/show/program/google/gsoc2009/userguide#… Let's keep those applications coming! Cheers Stéfan

1 0

1.3.0 rc1 MATHLIB env variable / bad compiler flags
by Mark Sienkiewicz 01 Apr '09

01 Apr '09

I have this configuration: numpy 1.3.0 rc1 Solaris 10 Python 2.5.4 compiled as a 64 bit executable When I try to install numpy, it says: C compiler: cc -DNDEBUG -O -xarch=native64 -xcode=pic32 compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/stsci/Python-2.5.4/include/python2.5 -c' cc: _configtest.c cc _configtest.o -lm -o _configtest ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: File processing errors. No output written to _configtest ld: fatal: file _configtest.o: wrong ELF class: ELFCLASS64 ld: fatal: File processing errors. No output written to _configtest failure. ... mathlibs = check_mathlib(config_cmd) File "numpy/core/setup.py", line 253, in check_mathlib raise EnvironmentError("math library missing; rerun " EnvironmentError: math library missing; rerun setup.py after setting the MATHLIB env variable Of course, the problem is that it is using the wrong compiler flags during the link phase, so nothing I set MATHLIB to can possibly work. I found that I can get it to compile by creating these shell scripts: % cat cc #!/bin/sh /opt/SUNWspro-6u2/bin/cc -xarch=native64 -xcode=pic32 $* % cat f90 #!/bin/sh /opt/SUNWspro-6u2/bin/f90 -xarch=native64 -xcode=pic32 $* I think this looks like a bug. I thought I might try to make a patch (since this is all about installing, so, in principle, you don't need to know much about numpy), but I did not get very far in figuring out how the install works. The good news is that once you get it to build, it seems to work. (IIRC, rc1 fails the same test that it had problems with in my other email. The fix on the trunk also works in 64 bit solaris.) Mark S.

1 0

bad line in setup.py
by Darren Dale 01 Apr '09

01 Apr '09

In setup.py, svn_revision(), there is a line: log.warn("unrecognized .svn/entries format; skipping %s", base) log is not defined in setup.py. I'm using svn-1.6. Darren

2 1

Numpy Positional Array
by Ian Mallett 01 Apr '09

01 Apr '09

Hello, I'm trying to make an array of size n*n*2. It should be of the form: [[[0,0],[1,0],[2,0],[3,0],[4,0], ... ,[n,0]], [[0,1],[1,1],[2,1],[3,1],[4,1], ... ,[n,1]], [[0,2],[1,2],[2,2],[3,2],[4,2], ... ,[n,2]], [[0,3],[1,3],[2,3],[3,3],[4,3], ... ,[n,3]], [[0,4],[1,4],[2,4],[3,4],[4,4], ... ,[n,4]], ... ... ... ... ... ... ... [[0,n],[1,n],[2,n],[3,n],[4,n], ... ,[n,n]]] Each vec2 represents the x,y position of the vec in the array itself. I'm completely stuck on implementing this setup in numpy. Any pointers? Thanks, Ian

3 8