Mailman 3 August 2013 - NumPy-Discussion

Correct way to query NumPy for linktime BLAS and LAPACK
by Aron Ahmadia 08 Aug '13

08 Aug '13

Dear NumPy Developers, In the Clawpack/* repositories [1], we use a mixture of Fortran and Python source, currently glued together using f2py. Occasionally, we'll need to link the Fortran code directly against LAPACK. In particular, we're using dgeev and dgesv to solve several different Riemann problems [2,3]. In the past, we've relied on either the operating system or the user to provide these link commands for us, but it would be ideal in the future if we could query NumPy for how it is linked. Currently, the only information I can find is in the hidden __config__ module of NumPy's distutils module: numpy.distutils.__config__.blas_opt_info['extra_link_args'] numpy.distutils.__config__.lapack_opt_info['extra_link_args'] This seems to suggest that we shouldn't be relying on this information being available in future versions of NumPy (or at least, not in this location). That said, we'd still probably like to use this to avoid the possibility of multiple BLAS/LAPACK libraries being linked in to our builds. Any comments? Thanks, Aron [1] https://github.com/clawpack/clawpack [2] https://github.com/clawpack/riemann/blob/master/src/rp1_layered_shallow_wat… [3] https://github.com/clawpack/riemann/blob/master/src/rpn2_layered_shallow_wa…

2 2

Bug in gufuncs affecting umath_linalg
by Jaime Fernández del Río 06 Aug '13

06 Aug '13

Hi, I think I have found an "undocumented feature" of the gufuncs machinery. I have filed a bug report: https://github.com/numpy/numpy/issues/3582 Some more background on what i am seeing... I have coded a gufunc with signature '(r,c,p),(g,g,g,q)->(r,c,q)'. It is a color map, i.e. a transformation of a 3-dimensional array of p channels (with p=3, an RGB image of r rows and c columns), into a 3-dimensional array of q channels (with q=4, a CMYK image of the same size), via a p-dimensional look-up-table (LUT). For all practical purposes, the LUT always has the first three dimensions identical, hence the repeated g's in the signature. The function registered with this signature, receives the expected values in the dimensions argument: 'n', 'r', 'c', 'p', 'g', 'q', with 'n' being the length of the gufunc loop. But there is a problem with the steps argument. As expected I get a 13 item long array: 3 main loop strides, 3 strides (r, c, p) for the first argument, 4 strides (g, g, g, q) for the second, and 3 strides (r, c, q) for the return. Everything is OK except for the strides for the repeating 'g's: instead of getting three different stride values, the first two are the same as the last. This does not happen if I modify the signature to be '(r,c,p),(i,j,k,q)->(r,c,q)', which is the workaround I am implementing in my code for the time being. I have also managed to repeat the behavior in repeated dimensions on other arguments, e.g. '(r,r,p),(i,j,k,q)->(r,c,p)' shows the same issue for the strides of the first argument. I have seen that the gufunc version of umath_linalg makes use of a similar, repeated index scheme, e.g. in the 'solve' and 'det' gufuncs. At least for these two, the tests in place do not catch the error. For solve, the tests run these two cases (the results below come from the traditional linalg, as I am running numpy 1.7.1): >>> np.linalg.solve([[1, 2], [3, 4]], [[4, 3], [2, 1]]) array([[-6., -5.], [ 5., 4.]]) >>> np.linalg.solve([[1+2j,2+3j], [3+4j,4+5j]], [[4+3j,3+2j], [2+1j,1+0j]]) array([[-6. +0.00000000e+00j, -5. +0.00000000e+00j], [ 5. -3.46944695e-16j, 4. -3.46944695e-16j]]) But because of their highly strucutred nature, these particular test cases give the same result if you get the strides wrong (!!!): >>> np.linalg.solve([[1, 2], [2, 3]], [[4, 3], [3, 2]]) array([[-6., -5.], [ 5., 4.]]) >>> np.linalg.solve([[1+2j,2+3j], [2+3j,3+4j]], [[4+3j,3+2j], [3+2j,2+1j]]) array([[-6. -1.09314267e-15j, -5. -1.09314267e-15j], [ 5. +1.08246745e-15j, 4. +1.08246745e-15j]]) As for the determinant, no abolute check of the return value is performed: the return of 'det' is compared to the product of the return of 'eigvals', which also has the '(m, m)' signature, and interprets the data equally wrong. For my particular issue, I am simply going to register the gufunc with non-repeating dimensions, check for equality in a Python wrapper, and discard the repeated values in my C code. Not sure what is the best way of going about umath_linalg. Probably better to fix the issue in the gufunc machinery than to patch umath_lnalg. If there's any way, other than reporting it, in which I can help getting this fixed, I'll be more than happy to do it. But for this job I am clearly unqualified labor, and would need to work under someone else's command. Regards, Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0

Problems building from source
by Raul Cota 05 Aug '13

05 Aug '13

Hello, I had not updated my code for a few months. I updated today to the latest source and I cannot build anymore, (Windows, Python 2.6) When I do, python setup.py build I get, """ Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 192, in <module> setup_package() File "setup.py", line 169, in setup_package from numpy.distutils.core import setup File "C:\Users\Raul\Documents\GitHub\numpy\numpy\distutils\__init__.py", line 37, in <module> from numpy.testing import Tester File "C:\Users\Raul\Documents\GitHub\numpy\numpy\testing\__init__.py", line 13 , in <module> from .utils import * File "C:\Users\Raul\Documents\GitHub\numpy\numpy\testing\utils.py", line 13, i n <module> from numpy.core import float32, empty, arange File "C:\Users\Raul\Documents\GitHub\numpy\numpy\core\__init__.py", line 6, in <module> from . import multiarray ImportError: cannot import name multiarray """ I can see numpy\numpy\core\__init__.py changed the imports to be 'relative' based from when it used to work. Any hints or reference to a related discussion ? All I found was pr 3178 "2to3 apply import fixer" Raul

2 2

Questions on GUFUNCS
by Jaime Fernández del Río 05 Aug '13

05 Aug '13

Hi! I have spent the last couple of weeks playing around with GUFUNCS, and am literally blown away by the power that a C compiler and NumPy put at the tip of my fingers! I still have many questions, but the following ones are the most pressing in my current state of amazed ignorance: 1. **The data argument to the GUFUNC loop function** Where in the source code can I find example of it's use? I remember reading that many UFUNCS are just the same looping, with a pointer to a different function passed in this argument. I think I understand the idea, but would like to see it implemented. 2. **Is there a place for initializations?** I have a GUFNC in mind that will need, in each iteration, a small dynamically allocated array, which can be reused by other iterations. Rather than malloc-ing it for every loop, I am thinking of wrapping the GUFUNC in a Python interface that creates a numpy array for this, and passes it down as another parameter. Is there an easy way to keep this all bundled in the C code, e.g. by defining some initialization code to be run before doing any looping? 3. **What happens with missing types?** Say I only register a function that takes unsigned ints. What happens if I try to call it with ints? I figure it will complain, but not if it is the other way around and the type can be safely casted. Is that really so? Can this behavior be in any way configured? Is it documented anywhere? 4. **.src files** The prime example of GUFUNC implementation I have found is "umath_linalg.c.src" Correct me if I am wrong, but this is simply C code plus the special syntax to generate multiple versions of the same functions changing only a few types and names, using the @tag@ syntax. While the way it works seems clear, I had never seen this done like this before. Is this standard or just a numpy thing? How do you parse and expand this code to its full glory before compiling? Thanks! Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

2 1

distutils seems to pick the wrong fortran compiler flags
by Hugo Gagnon 05 Aug '13

05 Aug '13

Hi, I'm using f2py shipping with EPD 32 bit on 64 bit Mac OS 10.8. The command "f2py -c -m plot3d --fcompiler=gnu95 plot3d.f90" compiles the objects files in 32 bit, which is good, but fails at the linkage step with a "file was built for i386 which is not the architecture being linked (x86_64)" warning. See this post for a full output: http://stackoverflow.com/questions/18003399/wrapping-32-bit-libraries-with-… My solution was to hack numpy/distutils/fcompiler/gnu.py by explicitly adding the "-m32" flag to the "compiler_f77", "compiler_f90" and "linker_so" keys of the "executables" dictionary of the "Gnu95FCompiler" class (line 251). I'm sure there's a nicer way to do that though... -- Hugo Gagnon

1 0

Installation bug on Python 3.3?
by Jaakko Luttinen 05 Aug '13

05 Aug '13

Hi, I'm trying to install NumPy 1.7.1 for Python 3.3 using pip install numpy However, I get the following error after a while: error: numpy.egg-info/dependency_links.txt: Operation not supported Is this a bug or am I doing something wrong? If it matters, I'm using virtualenv as I do not have root permission on this computer. Thanks for any help! Jaakko PS. The end of the error log: ---------------------------------------- Command /home/jluttine/.virtualenvs/bayespy-3.3/bin/python3.3 -c "import setuptools;__file__='/home/jluttine/.virtualenvs/bayespy-3.3/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-309wd8-record/install-record.txt --single-version-externally-managed --install-headers /home/jluttine/.virtualenvs/bayespy-3.3/include/site/python3.3 failed with error code 1 in /home/jluttine/.virtualenvs/bayespy-3.3/build/numpy Exception information: Traceback (most recent call last): File "/home/jluttine/.virtualenvs/bayespy-3.3/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg/pip/basecommand.py", line 107, in main status = self.run(options, args) File "/home/jluttine/.virtualenvs/bayespy-3.3/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg/pip/commands/install.py", line 261, in run requirement_set.install(install_options, global_options) File "/home/jluttine/.virtualenvs/bayespy-3.3/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg/pip/req.py", line 1166, in install requirement.install(install_options, global_options) File "/home/jluttine/.virtualenvs/bayespy-3.3/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg/pip/req.py", line 589, in install cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False) File "/home/jluttine/.virtualenvs/bayespy-3.3/lib/python3.3/site-packages/pip-1.2.1-py3.3.egg/pip/util.py", line 612, in call_subprocess % (command_desc, proc.returncode, cwd)) pip.exceptions.InstallationError: Command /home/jluttine/.virtualenvs/bayespy-3.3/bin/python3.3 -c "import setuptools;__file__='/home/jluttine/.virtualenvs/bayespy-3.3/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-309wd8-record/install-record.txt --single-version-externally-managed --install-headers /home/jluttine/.virtualenvs/bayespy-3.3/include/site/python3.3 failed with error code 1 in /home/jluttine/.virtualenvs/bayespy-3.3/build/numpy

1 1

Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 83, Issue 3
by Graeme B. Bell 05 Aug '13

05 Aug '13

Pauli, Thanks very much for investigating this and fixing it. Your patch works perfectly for me. :) Runtests.py now automatically uses "NumPy version 1.8.0.dev-af12c09". The only comment I have is that the line number patched was different on my copy (line 192). Perhaps we can add "run 'python runtest.py'" to the development workflow page, to guide new contributors? http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html Graeme. > Thanks, the issue seems to be due to some patch Fedora applied to its > Python version. The fix seems to be to change on line 234 in runtests.py > the line > > site_dir = get_python_lib(prefix=dst_dir) > > to > > site_dir = get_python_lib(prefix=dst_dir, plat_specific=True)

1 0

Re: [Numpy-discussion] unit tests / developing numpy (Pauli Virtanen)
by Graeme B. Bell 02 Aug '13

02 Aug '13

Hi Pauli, Thanks for looking into this. Apologies for mangling the email subject line in my previous reply. Answers are below, inline: > That is interesting, as it has worked for me on all configurations. > You can check under 'build/testenv/' --- does your Python version by > chance install it to a `lib64` directory instead of `lib`? $ ls build/testenv/ bin lib64 > (i) > > What does > > import os > from distutils.sysconfig import get_python_lib > get_python_lib(prefix=os.path.abspath('build/testenv')) > > report? Is there a 'numpy' directory below the reported directory after > running runtests.py? >>> import os >>> from distutils.sysconfig import get_python_lib >>> get_python_lib(prefix=os.path.abspath('build/testenv')) '/ssd-space/home/X/github/numpy/build/testenv/lib/python2.7/site-packages' I'm running this in a fresh python client in the directory 'github/numpy'. > (ii) > > Start a fresh Python interpreter and run > > import sys > print sys.modules.get('numpy') > > (Note: no "import numpy" above). Does it print `None` or something else? >>> import sys >>> print sys.modules.get('numpy') None Thanks, Pauli and Stefan. Graeme.

2 1

Announcing tinyarray
by Christoph Groth 01 Aug '13

01 Aug '13

Hello, Almost two years ago a discussion on this list took place under the subject "speeding up operations on small vectors". The issue was the rather high constant time and space overhead of small NumPy arrays. I'm aware that in general one should combine small arrays into large ones, but there are cases where this is not feasible. So, today, I'm pleased to announce a Python package that provides an alternative for those who would like or have to use many small arrays of numbers in Python: Tinyarray http://kwant-project.org/tinyarray/ https://pypi.python.org/pypi/tinyarray/ """ Tinyarray is a numerical array module for Python. The multi-dimensional arrays it provides are best thought of as (possibly nested) tuples of numbers that, unlike Python's built-in tuples, support mathematical operations. Like tuples, tinyarrays are hashable and immutable and thus can be used as dictionary keys. The module's interface is a subset of that of NumPy and hence should be familiar to many Python programmers. Tinyarray has been heavily optimized for small arrays: For example, common operations on 1-d arrays of length 3 run up to 35 times faster than with NumPy. When storing many small arrays, memory consumption is reduced by a factor of 3. In summary, Tinyarray is a more efficient alternative to NumPy when many separate small numerical arrays are to be used. """ I will be happy to hear comments and suggestions, and I will gladly accept useful patches. As I will be traveling the next days, I may not be very responsive. Best, Christoph

1 0

ANN: matplotlib 1.3.0 released
by Michael Droettboom 01 Aug '13

01 Aug '13

On behalf of a veritable army of super coders, I'm pleased to announce the release of matplotlib 1.3.0. Downloads Downloads are available here: <http://matplotlib.org/downloads.html>http://matplotlib.org/downloads.html as well as through |pip|. Check with your distro for when matplotlib 1.3.0 will become packaged for your environment. (Note: Mac .dmg installers are still forthcoming due to some issues with the new installation approach.) Important known issues matplotlib no longer ships with its Python dependencies, including dateutil, pytz, pyparsing and six. When installing from source or |pip|, |pip| will install these for you automatically. When installing from packages (on Linux distributions, MacPorts, homebrew etc.) these dependencies should also be handled automatically. The Windows binary installers do not include or install these dependencies. You may need to remove any old matplotlib installations before installing 1.3.0 to ensure matplotlib has access to the latest versions of these dependencies. The following backends have been removed: QtAgg (Qt version 3.x only), FlktAgg and Emf. For a complete list of removed features, see <http://matplotlib.org/api/api_changes.html#changes-in-1-3>http://matplotlib.org/api/api_changes.html#changes-in-1-3 What's new * xkcd-style sketch plotting * webagg backend for displaying and interacting with plots in a web browser * event plots * triangular grid interpolation * control of baselines in stackplot * many improvements to text and color handling For a complete list of what's new, see <http://matplotlib.org/users/whats_new.html#new-in-matplotlib-1-3>http://matplotlib.org/users/whats_new.html#new-in-matplotlib-1-3 Have fun, and enjoy matplotlib! Michael Droettboom

2 1