Mailman 3 January 2009 - NumPy-Discussion

A buildbot farm with shell access - for free ?
by David Cournapeau 30 Jan '09

30 Jan '09

Hi, Just saw that on one ML: http://www.snakebite.org/ http://mail.python.org/pipermail/python-committers/2009-January/000331.html Bottom line: it looks like there is a set of machines which were donated to the PSF for buildbot *with shell access* so that people can fix problems appearing on some platforms. If you look at the email, there are some 'exotic' machines that mere mortals cannot have access to (like True64 on Itanium: to quote the email "massive quad Itanium 2 RX-5670s, chock full of 73GB 15k disks and no less than 78GB of RAM between the two servers; 32GB in one and 46GB in the other"). There are also windows machines available. It is said in the email that this is reserved to the python project, and prominent python projects like Twisted and Django. Would it be ok to try to be qualified as a prominent python project as well ? cheers, David

7 14

NumPy distutils limitation?
by Francesc Alted 30 Jan '09

30 Jan '09

Hi, Gregor and I are trying to give support for Intel's VML (Vector Mathematical Library) in numexpr. For this, we are trying to make use of the weaponery in NumPy's distutils so as to be able to discover where the MKL (the package that contains VML) is located. The libraries that we need to link with are: mkl_gf_lp64, mkl_gnu_thread, mkl_core *and* iomp5 (Intel's OpenMP library). The problem is that I have installed MKL as part as the Intel compiler for Unix. In this setup, most of the libraries are in one place, namely: /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ However, the OpenMP library is in another directory: /opt/intel/Compiler/11.0/074/lib/intel64 So, I need to specify *two* directories to get the complete set of libraries. My first attempt was setting a site.cfg like: [DEFAULT] #libraries = gfortran [mkl] library_dirs= /opt/intel/Compiler/11.0/074/mkl/lib/em64t/:/opt/intel/Compiler/11.0/074/lib/intel64 include_dirs = /opt/intel/Compiler/11.0/074/mkl/include/ mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core, iomp5 Unfortunately, distutils complains and says that it cannot find the complete set of libraries: mkl_info: libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found in /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found in /opt/intel/Compiler/11.0/074/lib/intel64 NOT AVAILABLE After some debugging of the problem, it seems that distutils needs to find *all* the required libraries in *one* single directory. As iomp5 is on a different directory, distutils thinks that the requisites are not fulfilled. I've solved this by requering the iomp5 to be find in the DEFAULT section. Something like: [DEFAULT] library_dirs = /opt/intel/Compiler/11.0/074/lib/intel64 #libraries = gfortran, iomp5 libraries = iomp5 [mkl] library_dirs = /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ include_dirs = /opt/intel/Compiler/11.0/074/mkl/include/ mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core However, in case one would need to specify several other directories in the DEFAULT section for finding other hypothetical necessary libraries (like gfortran or others), we may run into the same problem than above. My question is, is there an elegant way to handle this problem, or it is a limitation of the current distutils? If the later, it would be nice it that could be solved in a future version, and several libraries can be found in *several* directories. Thanks, -- Francesc Alted

3 2

numpy.distutils and shared libraries
by Brian Granger 30 Jan '09

30 Jan '09

Hi, 2 ?'s about numpy.distutils: 1. I am using config.add_library to build a c++ library that I will link into some Cython extensions. This is working fine and generating a .a library for me. However, I need a shared library instead. Is this possible with numpy.distutils or will I need something like numscons? 2. When calling add_library, what is the difference between the depends and headers arguments? Thanks! Brian

2 2

Data file format choice.
by Gary Pajer 30 Jan '09

30 Jan '09

It's time for me to select a data format. My data are (more or less) spectra ( a couple of thousand samples), six channels, each channel running around 10 Hz, collecting for a minute or so. Plus all the settings on the instrument. I don't see any significant differences between netCDF4 and HDF5. Similarly, I don't see significant differences between pytables and h5py. Does one play better with numpy? What are the best numpy solutions for netCDF4? Can anyone provide thoughts, pros and cons, etc, that I can mull over? -gary

3 2

puzzle: generate index with many ranges
by Raik Gruenberg 30 Jan '09

30 Jan '09

Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array "a" with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. In reality, a and b have 10000+ elements and the arrays to be "sliced" are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik

3 6

concatenate trouble
by Neal Becker 30 Jan '09

30 Jan '09

What's the problem here? print np.concatenate (np.ones (10, dtype=complex), np.zeros (10, dtype=complex)) TypeError: only length-1 arrays can be converted to Python scalars

2 1

Building on WinXP 64-bit, Intel Compilers
by Michael Colonno 30 Jan '09

30 Jan '09

Hi ~ I'm trying to build numpy (hopefully eventually scipy with the same setup) with the Intel compilers (and Intel MKL) on the WinXP 64-bit platform. Finding / linking to the Intel MKL seems to be successful (see below) but I have an issue with the settings defined somewhere in the various setup scripts (can't find where). Per the output below, the Intel compilers on Windows are looking for ".obj" object files rather than the Linux-style ".o" files. I'd also like to get rid of the /L and -L flags (no longer supported in Intel C++ v. 11.0 it seems) but this just throws a warning and does not seem to cause any problems. Can anyone point me to the python file(s) I need to edit to modify the .o object file setting to .obj? (The file "_configtest.obj" is created.) Once operational, I'll pass along all of my config info for anyone else building in this environment. Thanks! ~Mike C. Output from build: >>python setup.py config --compiler=intel --fcompiler=intelem install Running from numpy source directory. Forcing DISTUTILS_USE_SDK=1 F2PY Version 2_5972 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_em64t', 'mkl_dll'] library_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\ 061\\cpp\\m kl\\em64t\\lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\include'] FOUND: libraries = ['mkl_em64t', 'mkl_dll'] library_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\em64t\\lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_em64t', 'mkl_dll'] library_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\em64t\\lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\include'] FOUND: libraries = ['mkl_lapack', 'mkl_em64t', 'mkl_dll'] library_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\em64t\\lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\include'] FOUND: libraries = ['mkl_lapack', 'mkl_em64t', 'mkl_dll'] library_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\em64t\\lib'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['C:\\Program Files (x86)\\Intel\\Compiler\\11.0\\061\\cpp\\m kl\\include'] running config running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src building py_modules sources creating build creating build\src.win32-2.5 creating build\src.win32-2.5\numpy creating build\src.win32-2.5\numpy\distutils building extension "numpy.core.multiarray" sources creating build\src.win32-2.5\numpy\core Generating build\src.win32-2.5\numpy\core\include/numpy\config.h Could not locate executable icc Could not locate executable ecc Ignoring "MSVCCompiler instance has no attribute '_MSVCCompiler__root'" (I think it is msvccompiler.py bug) customize IntelEM64TFCompiler Found executable C:\Program Files (x86)\Intel\Compiler\11.0\061\fortran\Bin\inte l64\ifort.exe Found executable C:\Program Files (x86)\Intel\Compiler\11.0\061\fortran\Bin\inte l64\ifort.exe C compiler: icl compile options: '-IC:\Python25\include -Inumpy\core\src -Inumpy\core\include -I C:\Python25\include -IC:\Python25\PC -c' icl: _configtest.c Found executable C:\Program Files (x86)\Intel\Compiler\11.0\061\cpp\Bin\intel64\ icl.exe icl _configtest.o -LC:\Python25\lib -LC:\ -LC:\Python25\libs -o _configtest Intel(R) C++ Intel(R) 64 Compiler Professional for applications running on Intel (R) 64, Version 11.0 Build 20080930 Package ID: w_cproc_p_11.0.061 Copyright (C) 1985-2008 Intel Corporation. All rights reserved. icl: command line warning #10161: unrecognized source type '_configtest.o'; obje ct file assumed icl: command line warning #10006: ignoring unknown option '/LC:\Python25\lib' icl: command line warning #10006: ignoring unknown option '/LC:\' icl: command line warning #10006: ignoring unknown option '/LC:\Python25\libs' ipo: warning #11009: file format not recognized for _configtest.o Microsoft (R) Incremental Linker Version 9.00.30729.01 Copyright (C) Microsoft Corporation. All rights reserved. -out:_configtest.exe _configtest.o LINK : fatal error LNK1181: cannot open input file '_configtest.o' failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 96, in <module> setup_package() File "setup.py", line 89, in setup_package configuration=configuration ) File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\core.py", line 184, in setup return old_setup(**new_attr) File "C:\Python25\lib\distutils\core.py", line 151, in setup dist.run_commands() File "C:\Python25\lib\distutils\dist.py", line 974, in run_commands self.run_command(cmd) File "C:\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\command\install.py", line 49, in run r = old_install.run(self) File "C:\Python25\lib\distutils\command\install.py", line 506, in run self.run_command('build') File "C:\Python25\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\command\build.py", line 37, in run old_build.run(self) File "C:\Python25\lib\distutils\command\build.py", line 112, in run self.run_command(cmd_name) File "C:\Python25\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python25\lib\distutils\dist.py", line 994, in run_command cmd_obj.run() File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\command\build_src.py", line 130, in run self.build_sources() File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\command\build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\command\build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "C:\Documents and Settings\mike\Desktop\Software\Python\numpy-1.2.1\numpy \distutils\command\build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 87, in generate_config_h raise SystemError,"Failed to test configuration. "\ SystemError: Failed to test configuration. See previous error messages for more information.

5 16

Ironclad 0.8 released
by William Reade 29 Jan '09

29 Jan '09

Hi all I'm delighted to announce the release of Ironclad v0.8 -- the all-singing, all-dancing CPython API compatibility layer for IronPython -- available now from http://code.google.com/p/ironclad/ . Notable improvements over the last release include: * Ironclad is now a neatly self-contained package -- just copy to your site-packages and 'import ironclad'. * No more silly requirement to call ironclad.shutdown() when you're finished. * A few performance improvements. * Over 900 NumPy tests now pass: in fact, almost all the tests from the core, fft, lib, linalg, ma, oldnumeric and random subpackages. * Over half the .pyds distributed with CPython 2.5 now import cleanly; some of them appear to actually work, including _hashlib and _elementtree. Ironclad grows more stable and mature with every release, and I urge IronPython users to try it out and share their impressions: feedback, whether positive or negative, is always welcomed. Cheers William

2 3

convolution axis
by Gideon Simpson 29 Jan '09

29 Jan '09

Is there an easy way to perform convolutions along a particular axis of an array? -gideon

2 2

Documentation: objects.inv ?
by Pierre GM 29 Jan '09

29 Jan '09

All, Is there an objects.inv lying around for the numpy reference guide, or should I start one from scratch ? Thx a lot in advance P.

3 3