Mailman 3 March 2011 - NumPy-Discussion

Problems building NumPy with GotoBLAS
by ashford＠whisperpc.com March 28, 2011

March 28, 2011

I'm having problems getting the GotoBLAS library (Nehalem optimized BLAS - "http://www.tacc.utexas.edu/tacc-projects/gotoblas2/") working properly under the Python NumPy package ("http://numpy.scipy.org/") on a quad-core Nehalem under FC10. The command used to build the library is: make BINARY=64 USE_THREAD=1 MAX_CPU_NUMBER=4 I'm limiting this to four cores, as I believe HyperThreading will slow it down (I've seen this happen with other scientific code). I'll benchmark later to see whether or not HyperThreading helps. I built the library (it uses -fPIC), then installed it under /usr/local/lib64, and created the appropriate links: # cp libgoto2_nehalemp-r1.13.a /usr/local/lib64 # cp libgoto2_nehalemp-r1.13.so /usr/local/lib64 # cd /usr/local/lib64 # ln -s libgoto2_nehalemp-r1.13.a libgoto2.a # ln -s libgoto2_nehalemp-r1.13.so libgoto2.so # ln -s libgoto2_nehalemp-r1.13.a libblas.a # ln -s libgoto2_nehalemp-r1.13.so libblas.so Without the libblas links, the NumPy configuration used the system default BLAS library (single-threaded NetLib under FC10); it's set up for the NetLib and Atlas BLAS libraries. I used the latest release of NumPy, with no site.cfg file, and no NumPy directory ("rm -rf /usr/local/python2.6/lib/python2.6/site-packages/numpy"). The configuration step ("python setup.py config") appears to run OK, as do the build ("python setup.py build") and install ("python setup.py install") steps. The problem is that the benchmark takes 8.5 seconds, which is what it took before I changed the library. python -c "import numpy as N; a=N.random.randn(1000, 1000); N.dot(a, a)" I expect I'm missing something really simple here, but I've spent >10 hours on it, and I have no idea as to what it could be. I've tried various permutations on the site.cfg file, all to no avail. I've also tried different names on the library, and different locations. I've even tried a set of symbolic links in /usr/local/lib64 for liblapack that point to libgoto. Could someone offer some suggestions? Thank you for your time. Peter Ashford

6 11

Compiling numpy using icc gets missing library error
by Hoyt Koepke March 27, 2011

March 27, 2011

Hello, I've been trying out intel's compiler with python and so far things have been going pretty well. I managed to compile python 2.7.1 with icc (with help from http://software.intel.com/en-us/forums/showthread.php?t=56652), so distutils automatically defaults to the icc compiler. So far a number of packages work great, so the overall system seems to work fine. However, I'm running into a few problems compiling numpy. I'm using the latest development build from git (rev ac3cba). The first is that some typing for long double isn't getting recognized -- building numpy breaks with the following error (I'll also post a bug report on this if requested). Traceback (most recent call last): File "setup.py", line 201, in <module> setup_package() File "setup.py", line 194, in setup_package configuration=configuration ) File "/home/hoytak/sysroot/src/numpy/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/home/hoytak/sysroot/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/home/hoytak/sysroot/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/home/hoytak/sysroot/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/home/hoytak/sysroot/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/home/hoytak/sysroot/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/home/hoytak/sysroot/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 169, in build_sources self.build_extension_sources(ext) File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 328, in build_extension_sources sources = self.generate_sources(sources, ext) File "/home/hoytak/sysroot/src/numpy/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 442, in generate_config_h rep = check_long_double_representation(config_cmd) File "numpy/core/setup_common.py", line 136, in check_long_double_representation type = long_double_representation(pyod(object)) File "numpy/core/setup_common.py", line 280, in long_double_representation raise ValueError("Could not lock sequences (%s)" % saw) ValueError: Could not lock sequences (None) This happens regardless of whether I use the --compiler=intel option. I'm using one of the new sandy bridge processors (i7-2600k), which may be part of the reason. Looking at the code, it seems this comes from a failure to detect the type info from a particular string. I've attached a dump of the lines variable passed to the long_double_representation function. As a workaround, I set it manually to return 'INTEL_EXTENDED_12_BYTES_LE' with the idea that I could test this using the unit tests later on. When I did that, everything else compiled fine and numpy installed. However, attempting to import numpy gives me this: >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/__init__.py", line 137, in <module> import add_newdocs File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in <module> from numpy.lib import add_newdoc File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/lib/__init__.py", line 4, in <module> from type_check import * File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/lib/type_check.py", line 8, in <module> import numpy.core.numeric as _nx File "/home/hoytak/sysroot/lib/python2.7/site-packages/numpy/core/__init__.py", line 5, in <module> import multiarray ImportError: /home/hoytak/sysroot/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: npy_half_to_float In case it's relevant: $ ldd /home/hoytak/sysroot/lib/python2.7/site-packages/numpy/core/multiarray.so linux-vdso.so.1 => (0x00007fff60fff000) libpython2.7.so.1.0 => /home/hoytak/sysroot/lib/libpython2.7.so.1.0 (0x00007f5916dc3000) libimf.so => /usr/intel/composerxe-2011.2.137/composerxe-2011.2.137/compiler/lib/intel64/libimf.so (0x00007f59169df000) libsvml.so => /usr/intel/composerxe-2011.2.137/composerxe-2011.2.137/compiler/lib/intel64/libsvml.so (0x00007f5916337000) libm.so.6 => /lib/libm.so.6 (0x00007f5916093000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f5915e7c000) libintlc.so.5 => /usr/intel/composerxe-2011.2.137/composerxe-2011.2.137/compiler/lib/intel64/libintlc.so.5 (0x00007f5915d2d000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f5915b10000) libc.so.6 => /lib/libc.so.6 (0x00007f591578c000) libdl.so.2 => /lib/libdl.so.2 (0x00007f5915588000) libutil.so.1 => /lib/libutil.so.1 (0x00007f5915385000) /lib64/ld-linux-x86-64.so.2 (0x00007f591770c000) I also attached the build log here (which is the same for the first error up to the crash). Any suggestions of where to look or where something could have gone wrong? Thanks! -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak(a)gmail.com ++++++++++++++++++++++++++++++++++++++++++

3 6

Please help me
by dileep kunjaai March 27, 2011

March 27, 2011

Dear sir, Can we take a numpy array excluding particular columns Example: A=[[ 20. 71. 57.5] [ 21. 71. 56. ] [ 22. 71. 74.3] [ 23. 71. 66. ] [ 24. 71. 96.5] [ 25. 71. 61.7] [ 20. 72. 72.9] (Here i am give up the last column) The answer should be [[ 20. 71. ] [ 21. 71. ] [ 22. 71. ] [ 23. 71. ] [ 24. 71. ] [ 25. 71. ] [ 20. 72. ] Please help me ............................... -- DILEEPKUMAR. R J R F, IIT DELHI

2 2

ValueError: Unknown format code 'g' for object of type 'str'
by Nils Wagner March 26, 2011

March 26, 2011

>>> numpy.__version__ '2.0.0.dev-10db259' ====================================================================== ERROR: Test the str.format method with NumPy scalar types ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/case.py", line 183, in runTest self.test(*self.arg) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/core/tests/test_print.py", line 223, in test_scalar_format assert_equal(fmat.format(val), fmat.format(valtype(val)), ValueError: Unknown format code 'g' for object of type 'str'

2 1

ANN: TreeDict 0.12, a lightweight library for hierarchical parameter/data management
by Hoyt Koepke March 25, 2011

March 25, 2011

Hello, I thought the numpy/scipy community might be particularly interested in this library, so here goes: I'm pleased to announce a release of the next version of TreeDict, a dictionary-like, hierarcical python container to simplify the bookkeeping surrounding parameters, variables and data. It aims to be fast, lightweight, intuitive, feature-rich and stable. While intended for general python coding, it includes a number of features particularly useful for scientific programming. It is similar in basic functionality to MATLAB structures in terms of concise syntax and implicit branch creation, which, I've found, makes organizing parameters/data much easier than with other methods. In addition, though, TreeDict implements all the methods of regular dictionaries, pickling, fast non-intersecting hashing for efficient cache lookups, manipulations on the tree structure, a system for forward referencing branches to make lists of parameters more readable, BSD license, and a full suite of unit tests. More info can be found at http://www.stat.washington.edu/~hoytak/code/treedict/index.html. Questions or comments or feedback are welcome. Thanks! --Hoyt P.S. One thought regarding scipy -- since this library is fairly similar to matlab structs, it may be easy to integrate this with the current matlab io routines to translate things between the two more easily. In this direction, I've added in options to two methods, fromdict and convertTo, to make conversion to/from the existing python representations of the matlab objects. Discussion/comments on this point are welcome. ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak(a)gmail.com ++++++++++++++++++++++++++++++++++++++++++

1 0

Ticket closing policy
by David March 25, 2011

March 25, 2011

Hi Mark, hi all, I noticed you did a lot of cleaning in the bug trackers, thank you for helping there, this is sorely needed. However, I noticed quite a few tickets were closed as wontfix even though they are valid. I understand the concern of getting many languishing tickets, but one should not close valid tickets either. Also, in my experience, issues specific to win32 should not be closed because they work on Linux - I remember the fastputmask issue was still there not so long ago (but could not understand what was going on, unfortunately). cheers, David

3 2

HP-UX users, please test distutils changes
by Ralf Gommers March 25, 2011

March 25, 2011

Hi, Here are some updated compile flags for HP-UX that were submitted on the bug tracker. It would be nice to have someone confirm this works: https://github.com/rgommers/numpy/tree/hp Thanks, Ralf

1 0

So...
by Sturla Molden March 25, 2011

March 25, 2011

FFTs should keep the GIL locked, because sharing the interpreter is not funny. Array indices should be sizeof(void*), because following C standard and Python C API is lame. Median should be calculated in O(n log n) instead of O(n) time, because redundant sorting is gymnatics for the CPU. But at least something happened to the tickets, after a year or two :-) Sturla

2 3

help me
by dileep kunjaai March 25, 2011

March 25, 2011

Dear sir, I have a numpy array,,. in which i have to choose only different values only, For example Let A= [ 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25. 20. 21. 22. 23. 24. 25.] then the answer is [20, 21. 22. 23. 24. 25.] ie choose only different values Please help me. -- DILEEPKUMAR. R J R F, IIT DELHI

2 2

when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs
by Dmitrey March 25, 2011

March 25, 2011

hi, when numpy in Linux apt will be updated? It's still 1.3.0 with many bugs I tried to install numpy from PYPI where 1.5.1 seesm to be present, but somehow it involves 1.3.0 instead: $ sudo easy_install numpy install_dir /usr/local/lib/python2.6/dist-packages/ Searching for numpy Best match: numpy 1.3.0 Adding numpy 1.3.0 to easy-install.pth file only after aptitude remove python-numpy version 1.5.1. is involved by easy_install, but it fails: $ sudo easy_install numpy .... Adding numpy 1.5.1 to easy-install.pth file Installing f2py script to /usr/local/bin Installed /usr/local/lib/python2.6/dist-packages/numpy-1.5.1-py2.6-linux-x86_64.egg Processing dependencies for numpy Finished processing dependencies for numpy /tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py:251: RuntimeWarning: Parent module 'numpy.distutils' not found while handling absolute import Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", line 251, in clean_up_temporary_directory ImportError: No module named numpy.distutils Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/tmp/easy_install-QF6uJM/numpy-1.5.1/numpy/distutils/misc_util.py", line 251, in clean_up_temporary_directory ImportError: No module named numpy.distutils I have Linux KUBUNTU 10.10 D.

4 5