Mailman 3 November 2015 - NumPy-Discussion

Failed numpy.test() with numpy-1.10.1 on RHEL 6
by Lintula Nov. 9, 2015

Nov. 9, 2015

Hello, I'm setting up numpy 1.10.1 on RHEL6 (python 2.6.6, atlas-3.8.4, lapack-3.2.1, gcc-4.4.7), and this test fails for me. I notice that someone else has had the same at https://github.com/numpy/numpy/issues/6063 in July. Is this harmless or is it of concern? ====================================================================== FAIL: test_umath.TestComplexFunctions.test_branch_cuts(<ufunc 'arccosh'>, [-1, 0.5], [1j, 1j], 1, -1, True) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/usr/lib64/python2.6/site-packages/numpy/core/tests/test_umath.py", line 1748, in _check_branch_cut assert_(np.all(np.absolute(y0.imag - yp.imag) < atol), (y0, yp)) File "/usr/lib64/python2.6/site-packages/numpy/testing/utils.py", line 53, in assert_ raise AssertionError(smsg) AssertionError: (array([ 0.00000000e+00+3.14159265j, 1.11022302e-16-1.04719755j]), array([ 4.71216091e-07+3.14159218j, 1.28119737e-13+1.04719755j])) ---------------------------------------------------------------------- Ran 5955 tests in 64.284s FAILED (KNOWNFAIL=3, SKIP=2, failures=1) <nose.result.TextTestResult run=5955 errors=0 failures=1>

2 1

Compilation problems npy_float64
by Johan Nov. 7, 2015

Nov. 7, 2015

Hello, I searched the forum, but couldn't find a post related to my problem. I am installing scipy via pip in cygwin environment pip install scipy Note: numpy version 1.10.1 was installed with pip install -U numpy /usr/bin/gfortran -Wall -g -Wall -g -shared -Wl,-gc-sections -Wl,-s build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/geom2.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/geom.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/global.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/io.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/libqhull.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/mem.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/merge.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/poly2.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/poly.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/qset.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/random.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/rboxlib.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/stat.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/user.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/usermem.o build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/qhull/src/userprintf.o build/temp.cygwin-2.2.1-x86_64- 2.7/scipy/spatial/qhull/src/userprintf_rbox.o -L/usr/lib - L/usr/lib/gcc/x86_64-pc-cygwin/4.9.3 -L/usr/lib/python2.7/config - L/usr/lib -Lbuild/temp.cygwin-2.2.1-x86_64-2.7 -llapack -lblas - lpython2.7 -lgfortran -o build/lib.cygwin-2.2.1-x86_64- 2.7/scipy/spatial/qhull.dll building 'scipy.spatial.ckdtree' extension compiling C++ sources C compiler: g++ -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit- function-declaration -fdebug-prefix-map=/usr/src/ports/python/python- 2.7.10-1.x86_64/build=/usr/src/debug/python-2.7.10-1 -fdebug-prefix- map=/usr/src/ports/python/python-2.7.10-1.x86_64/src/Python- 2.7.10=/usr/src/debug/python-2.7.10-1 -DNDEBUG -g -fwrapv -O3 -Wall creating build/temp.cygwin-2.2.1-x86_64-2.7/scipy/spatial/ckdtree creating build/temp.cygwin-2.2.1-x86_64- 2.7/scipy/spatial/ckdtree/src compile options: '-I/usr/include/python2.7 - I/usr/lib/python2.7/site-packages/numpy/core/include - Iscipy/spatial/ckdtree/src -I/usr/lib/python2.7/site- packages/numpy/core/include -I/usr/include/python2.7 -c' g++: scipy/spatial/ckdtree/src/ckdtree_cpp_exc.cxx cc1plus: warning: command line option ‘-Wimplicit-function- declaration’ is valid for C/ObjC but not for C++ g++: scipy/spatial/ckdtree/src/ckdtree_query.cxx cc1plus: warning: command line option ‘-Wimplicit-function- declaration’ is valid for C/ObjC but not for C++ In file included from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/ndarraytypes.h:1781:0, from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/ndarrayobject.h:18, from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/arrayobject.h:4, from scipy/spatial/ckdtree/src/ckdtree_query.cxx:15: /usr/lib/python2.7/site- packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it by " \ ^ In file included from scipy/spatial/ckdtree/src/ckdtree_query.cxx:31:0: scipy/spatial/ckdtree/src/ckdtree_cpp_methods.h:12:20: error: ‘npy_float64 infinity’ redeclared as different kind of symbol extern npy_float64 infinity; ^ In file included from /usr/include/python2.7/pyport.h:325:0, from /usr/include/python2.7/Python.h:58, from scipy/spatial/ckdtree/src/ckdtree_query.cxx:14: /usr/include/math.h:263:15: note: previous declaration ‘double infinity()’ extern double infinity _PARAMS((void)); ^ In file included from scipy/spatial/ckdtree/src/ckdtree_query.cxx:31:0: scipy/spatial/ckdtree/src/ckdtree_cpp_methods.h: In function ‘npy_float64 _distance_p(const npy_float64*, const npy_float64*, npy_float64, npy_intp, npy_float64)’: scipy/spatial/ckdtree/src/ckdtree_cpp_methods.h:139:17: error: invalid operands of types ‘const npy_float64 {aka const double}’ and ‘double()’ to binary ‘operator==’ else if (p==infinity) { ^ scipy/spatial/ckdtree/src/ckdtree_query.cxx: In function ‘PyObject* query_knn(const ckdtree*, npy_float64*, npy_intp*, const npy_float64*, npy_intp, npy_intp, npy_float64, npy_float64, npy_float64)’: scipy/spatial/ckdtree/src/ckdtree_query.cxx:431:111: error: cannot convert ‘double (*)()’ to ‘npy_float64 {aka double}’ for argument ‘9’ to ‘void __query_single_point(const ckdtree*, npy_float64*, npy_intp*, const npy_float64*, npy_intp, npy_float64, npy_float64, npy_float64, npy_float64)’ __query_single_point(self, dd_row, ii_row, xx_row, k, eps, p, distance_upper_bound, ::infinity); ^ In file included from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/ndarrayobject.h:27:0, from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/arrayobject.h:4, from scipy/spatial/ckdtree/src/ckdtree_query.cxx:15: /usr/lib/python2.7/site- packages/numpy/core/include/numpy/__multiarray_api.h: At global scope: /usr/lib/python2.7/site- packages/numpy/core/include/numpy/__multiarray_api.h:1634:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function] _import_array(void) ^ cc1plus: warning: command line option ‘-Wimplicit-function- declaration’ is valid for C/ObjC but not for C++ In file included from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/ndarraytypes.h:1781:0, from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/ndarrayobject.h:18, from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/arrayobject.h:4, from scipy/spatial/ckdtree/src/ckdtree_query.cxx:15: /usr/lib/python2.7/site- packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it by " \ ^ In file included from scipy/spatial/ckdtree/src/ckdtree_query.cxx:31:0: scipy/spatial/ckdtree/src/ckdtree_cpp_methods.h:12:20: error: ‘npy_float64 infinity’ redeclared as different kind of symbol extern npy_float64 infinity; ^ In file included from /usr/include/python2.7/pyport.h:325:0, from /usr/include/python2.7/Python.h:58, from scipy/spatial/ckdtree/src/ckdtree_query.cxx:14: /usr/include/math.h:263:15: note: previous declaration ‘double infinity()’ extern double infinity _PARAMS((void)); ^ In file included from scipy/spatial/ckdtree/src/ckdtree_query.cxx:31:0: scipy/spatial/ckdtree/src/ckdtree_cpp_methods.h: In function ‘npy_float64 _distance_p(const npy_float64*, const npy_float64*, npy_float64, npy_intp, npy_float64)’: scipy/spatial/ckdtree/src/ckdtree_cpp_methods.h:139:17: error: invalid operands of types ‘const npy_float64 {aka const double}’ and ‘double()’ to binary ‘operator==’ else if (p==infinity) { ^ scipy/spatial/ckdtree/src/ckdtree_query.cxx: In function ‘PyObject* query_knn(const ckdtree*, npy_float64*, npy_intp*, const npy_float64*, npy_intp, npy_intp, npy_float64, npy_float64, npy_float64)’: scipy/spatial/ckdtree/src/ckdtree_query.cxx:431:111: error: cannot convert ‘double (*)()’ to ‘npy_float64 {aka double}’ for argument ‘9’ to ‘void __query_single_point(const ckdtree*, npy_float64*, npy_intp*, const npy_float64*, npy_intp, npy_float64, npy_float64, npy_float64, npy_float64)’ __query_single_point(self, dd_row, ii_row, xx_row, k, eps, p, distance_upper_bound, ::infinity); ^ In file included from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/ndarrayobject.h:27:0, from /usr/lib/python2.7/site- packages/numpy/core/include/numpy/arrayobject.h:4, from scipy/spatial/ckdtree/src/ckdtree_query.cxx:15: /usr/lib/python2.7/site- packages/numpy/core/include/numpy/__multiarray_api.h: At global scope: /usr/lib/python2.7/site- packages/numpy/core/include/numpy/__multiarray_api.h:1634:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function] _import_array(void) ^ error: Command "g++ -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit- function-declaration -fdebug-prefix-map=/usr/src/ports/python/python- 2.7.10-1.x86_64/build=/usr/src/debug/python-2.7.10-1 -fdebug-prefix- map=/usr/src/ports/python/python-2.7.10-1.x86_64/src/Python- 2.7.10=/usr/src/debug/python-2.7.10-1 -DNDEBUG -g -fwrapv -O3 -Wall - I/usr/include/python2.7 -I/usr/lib/python2.7/site- packages/numpy/core/include -Iscipy/spatial/ckdtree/src - I/usr/lib/python2.7/site-packages/numpy/core/include - I/usr/include/python2.7 -c scipy/spatial/ckdtree/src/ckdtree_query.cxx - o build/temp.cygwin-2.2.1-x86_64- 2.7/scipy/spatial/ckdtree/src/ckdtree_query.o" failed with exit status 1 ---------------------------------------- Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build- vAliRx/scipy/setup.py';exec(compile(getattr(tokenize, 'open', open) (__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install -- record /tmp/pip-gxrCbK-record/install-record.txt --single-version- externally-managed --compile" failed with error code 1 in /tmp/pip- build-vAliRx/scipy

3 2

Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
by Nathaniel Smith Nov. 6, 2015

Nov. 6, 2015

Hi all, Apparently it is not well known that if you have a Python project source tree (e.g., a numpy checkout), then the correct way to install it is NOT to type python setup.py install # bad and broken! but rather to type pip install . (I.e., pip install isn't just for packages on pypi -- you can also pass it the path to an arbitrary source directory or the URL of a source tarball and it will do its thing. In this case "install ." means "install the project in the current directory".) These don't quite have identical results -- the main difference is that the latter makes sure that proper metadata gets installed so that later on it will be possible to upgrade or uninstall correctly. If you call setup.py directly, and then later you try to upgrade your package, then it's entirely possible to end up with a mixture of old and new versions of the package installed in your PYTHONPATH. (One common effect is in numpy's case is that we get people sending us mysterious bug reports about failing tests in files don't even exist (!) -- because nose is finding tests in files from one version of numpy and running them against a different version of numpy.) But this isn't the only issue -- using pip also avoids a bunch of weird corner cases in distutils/setuptools. E.g., if setup.py uses plain distutils, then it turns out this will mangle numpy version numbers in ways that cause weird horribleness -- see [1] for a bug report of the form "matplotlib doesn't build anymore" which turned out to be because of using 'setup.py install' to install numpy. OTOH if setup.py uses setuptools then you get different weirdnesses, like you can easily end up with multiple versions of the same library installed simultaneously. And finally, an advantage of getting used to using 'pip install .' now is that you'll be prepared for the glorious future when we kill distutils and get rid of setup.py entirely in favor of something less terrible [2]. So a proposal that came out of the discussion in [1] is that we modify numpy's setup.py now so that if you try running python setup.py install you get Error: Calling 'setup.py install' directly is NOT SUPPORTED! Instead, do: pip install . Alternatively, if you want to proceed at your own risk, you can try 'setup.py install --force-raw-setup.py' For more information see http://... (Other setup.py commands would continue to work as normal.) I believe that this would also break both 'easy_install numpy', and attempts to install numpy via the setup_requires= argument to setuptools.setup (because setup_requires= implicitly calls easy_install). install_requires= would *not* be affected, and setup_requires= would still be fine in cases where numpy was already installed. This would hopefully cut down on the amount of time everyone spends trying to track down these stupid weird bugs, but it will also require some adjustment in people's workflows, so... objections? concerns? -n [1] https://github.com/numpy/numpy/issues/6551 [2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html -- Nathaniel J. Smith -- http://vorpus.org

15 40

Help wanted: implementation of 3D medial axis skeletonization
by Stefan van der Walt Nov. 6, 2015

Nov. 6, 2015

Hi all, I have been approached by a group that is interested in sponsoring the development of 3D skeletonization in scikit-image. One potential starting place would be: http://www.insight-journal.org/browse/publication/181 Is anyone interested in working on this? Please get in touch either on the scikit-image mailing list or by mailing me directly. Thanks! Stéfan

1 0

New behavior of allclose
by Charles R Harris Nov. 5, 2015

Nov. 5, 2015

Hi All, This is to open a discussion of a change of behavior of `np.allclose`. That function uses `isclose` in numpy 1.10 with the result that array subtypes are preserved whereas before they were not. In particular, memmaps are returned when at least one of the inputs is a memmap. By and large I think this is a good thing, OTOH, it is a change in behavior. It is easy to fix, just run `np.array(result, copy=False)` on the current `result`, but I thought I'd raise the topic on the list in case there is a good argument to change things. Chuck

4 9

querying backend information
by Stefan Seefeld Nov. 5, 2015

Nov. 5, 2015

Hello, is there a way to query Numpy for information about backends (BLAS, LAPACK, etc.) that it was compiled against, including compiler / linker flags that were used ? Consider the use-case where instead of calling a function such as numpy.dot() I may want to call the appropriate backend directly using the C API as an optimization technique. Is there a straight-forward way to do that ? In a somewhat related line of thought: Is there a way to see what backends are available during Numpy compile-time ? I'm looking for a list of flags to pick ATLAS/OpenBLAS/LAPACK/MKL or any other backend that might be available, combined with variables (compiler and linker flags, notably) I might have to set. Is that information available at all ? Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

4 3

deprecate fromstring() for text reading?
by Chris Barker Nov. 4, 2015

Nov. 4, 2015

There was just a question about a bug/issue with scipy.fromstring (which is numpy.fromstring) when used to read integers from a text file. https://mail.scipy.org/pipermail/scipy-user/2015-October/036746.html fromstring() is bugging and inflexible for reading text files -- and it is a very, very ugly mess of code. I dug into it a while back, and gave up -- just to much of a mess! So we really should completely re-implement it, or deprecate it. I doubt anyone is going to do a big refactor, so that means deprecating it. Also -- if we do want a fast read numbers from text files function (which would be nice, actually), it really should get a new name anyway. (and the hopefully coming new dtype system would make it easier to write cleanly) I'm not sure what deprecating something means, though -- have it raise a deprecation warning in the next version? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

8 12

Re: [Numpy-discussion] [Distutils] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
by Chris Barker - NOAA Federal Nov. 3, 2015

Nov. 3, 2015

>> I'm not talking about in place installs, I'm talking about e.g. building a >> wheel and then tweaking one file and rebuilding -- traditionally build >> systems go to some effort to keep track of intermediate artifacts and reuse >> them across builds when possible, but if you always copy the source tree >> into a temporary directory before building then there's not much the build >> system can do. This strikes me as an optimization -- is it an important one? If I'm doing a lot of tweaking and re-running, I'm usually in develop mode. I can see that when you build a wheel, you may build it, test it, discover an wheel-specific error, and then need to repeat the cycle -- but is that a major use-case? That being said, I have been pretty frustrated debugging conda-build scripts -- there is a lot of overhead setting up the build environment each time you do a build... But with wheel building there is much less overhead, and far fewer complications requiring the edit-build cycle. And couldn't make-style this-has-already-been-done checking happen with a copy anyway? CHB > Ah yes. So I don't think pip should do what it does. It a violation of > the abstractions we all want to see within it. However its not me you > need to convince ;). > > -Rob > > -- > Robert Collins <rbtcollins(a)hp.com> > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig

1 0

histogram gives meaningless results with non-finite range
by Luke Zoltan Kelley Nov. 3, 2015

Nov. 3, 2015

This came up in [a matplotlib issue](https://github.com/matplotlib/matplotlib/issues/5221): >>> np.histogram(np.arange(10), range=(0.0, np.inf)) (array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), array([ nan, inf, inf, inf, inf, inf, inf, inf, inf, inf, inf])) >>> np.histogram(np.arange(10), range=(0.0, np.nan)) (array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), array([ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])) Clearly the behavior is undefined for those arguments, but perhaps there should be an assertion that the given range must be finite? Happy to make a PR for this. Luke

1 0

Re: [Numpy-discussion] [Distutils] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
by Nathaniel Smith Nov. 2, 2015

Nov. 2, 2015

On Nov 2, 2015 6:51 PM, "Robert Collins" <robertc(a)robertcollins.net> wrote: > > On 3 November 2015 at 14:57, Nathaniel Smith <njs(a)pobox.com> wrote: > > [Adding distutils-sig to the CC as a heads-up. The context is that > > numpy is looking at deprecating the use of 'python setup.py install' > > and enforcing the use of 'pip install .' instead, and running into > > some issues that will probably need to be addressed if 'pip install .' > > is going to become the standard interface to work with source trees.] > > > > On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > > [...] > >> Hmm, after some more testing I'm going to have to bring up a few concerns > >> myself: > >> > >> 1. ``pip install .`` still has a clear bug; it starts by copying everything > >> (including .git/ !) to a tempdir with shutil, which is very slow. And the > >> fix for that will go via ``setup.py sdist``, which is still slow. > > > > Ugh. If 'pip (install/wheel) .' is supposed to become the standard way > > to build things, then it should probably build in-place by default. > > Working in a temp dir makes perfect sense for 'pip install > > <requirement>' or 'pip install <url>', but if the user supplies an > > actual named on-disk directory then presumably the user is expecting > > this directory to be used, and to be able to take advantage of > > incremental rebuilds etc., no? > > Thats what 'pip install -e .' does. 'setup.py develop' -> 'pip install -e .' I'm not talking about in place installs, I'm talking about e.g. building a wheel and then tweaking one file and rebuilding -- traditionally build systems go to some effort to keep track of intermediate artifacts and reuse them across builds when possible, but if you always copy the source tree into a temporary directory before building then there's not much the build system can do. > >> 3. ``pip install .`` refuses to upgrade an already installed development > >> version. For released versions that makes sense, but if I'm in a git tree > >> then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal > >> to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1). > > > > Ugh, this is clearly just a bug -- `pip install .` should always > > unconditionally install, IMO. (Did you file a bug yet?) At least the > > workaround is just 'pip uninstall numpy; pip install .', which is > > still better the running 'setup.py install' and having it blithely > > overwrite some files and not others. > > There is a bug open. https://github.com/pypa/pip/issues/536 Thanks! -n

1 0