Mailman 3 January 2016 - NumPy-Discussion

maintenance/1.11.x branched
by Charles R Harris Jan. 20, 2016

Jan. 20, 2016

Hi All, I've branched 1.11.x. There is still work to be done, but the branch draws the line for PRs to be included in 1.11. Experience teaches that fixups will needed, but new work/enhancements will be in 1.12.0-dev0. Please run tests on this branch if you have the time. Travis CI has begun failing for Python 3.2 due to a bug in pip. I don't know if this is on our account or something on the Travis end. Thoughts on that are welcome. Chuck

1 0

building with setuptools
by Ralf Gommers Jan. 18, 2016

Jan. 18, 2016

Hi all, We've been able to hold out for a long time, but this is to let you all know that we have now had to switch the numpy build to rely on setuptools: https://github.com/numpy/numpy/pull/6895. Main reason: plain distutils doesn't support PEP 440 naming. See https://github.com/numpy/numpy/issues/6257 and https://github.com/numpy/numpy/issues/6551 for details. So if you see different build weirdness on master, it's likely due to this change. Cheers, Ralf

1 0

Software Capabilities of NumPy in Our Tensor Survey Paper
by Li Jiajia Jan. 15, 2016

Jan. 15, 2016

Hi all, I’m a PhD student in Georgia Tech. Recently, we’re working on a survey paper about tensor algorithms: basic tensor operations, tensor decomposition and some tensor applications. We are making a table to compare the capabilities of different software and planning to include NumPy. We’d like to make sure these parameters are correct to make a fair compare. Although we have looked into the related documents, please help us to confirm these. Besides, if you think there are more features of your software and a more preferred citation, please let us know. We’ll consider to update them. We want to show NumPy supports tensors, and we also include "scikit-tensor” in our survey, which is based on NumPy. Please let me know any confusion or any advice! Thanks a lot! :-) Notice: 1. “YES/NO” to show whether or not the software supports the operation or has the feature. 2. “?” means we’re not sure of the feature, and please help us out. 3. “Tensor order” means the maximum number of tensor dimensions that users can do with this software. 4. For computational cores, 1) "Element-wise Tensor Operation (A * B)” includes element-wise add/minus/multiply/divide, also Kronecker, outer and Katri-Rao products. If the software contains one of them, we mark “YES”. 2) “TTM” means tensor-times-matrix multiplication. We distinguish TTM from tensor contraction. If the software includes tensor contraction, it can also support TTM. 3) For “MTTKRP”, we know most software can realize it through the above two operations. We mark it “YES”, only if an specified optimization for the whole operation. <> <>Software Name <> NumPy Computational Cores Element-wise Tensor Operation (A * B) YES Tensor Contraction (A Xmn B) NO TTM ( A Xn B) NO Matriced Tensor Times Khatri-Rao Product (MTTKRP) NO Tensor Decomposition CP NO Tucker NO Hierarchical Tucker (HT) NO Tensor Train (TT) NO Tensor Features Tensor Order Arbitrary Dense Tensors YES Sparse Tensors NO ? Parallelized NO ? Software Information Application Domain General Programming Environment Python Latest Version 1.10.4 Release Date 2016 Citation: 1. AN DER WALT, S., COLBERT, S., AND VAROQUAUX, G. The NumPy array: A structure for efficient numerical computation. Computing in Science Engineering 13, 2 (March 2011), 22–30. 2. OLIPHANT, T. E. Python for scientific computing. Computing in Science Engineering 9, 3 (May 2007), 10–20. 3. NumPy (Version1.10.4).Available from http://www.numpy.org, Jan <http://www.numpy.org, Jan> 2016. Best regards! Jiajia Li ------------------------------------------ E-mail: jiajiali(a)gatech.edu Tel: +1 (404)9404603 Computational Science & Engineering Georgia Institute of Technology

6 5

inconsistency in np.isclose
by Andrew Nelson Jan. 15, 2016

Jan. 15, 2016

Hi all, I think there is an inconsistency with np.isclose when I compare two numbers: >>> np.isclose(0, np.inf) array([False], dtype=bool) >>> np.isclose(0, 1) False The first comparison returns a bool array, the second returns a bool. Shouldn't they both return the same result? -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 1

Fast Access to Container of Numpy Arrays on Disk?
by Ryan R. Rosario Jan. 14, 2016

Jan. 14, 2016

Hi, I have a very large dictionary that must be shared across processes and does not fit in RAM. I need access to this object to be fast. The key is an integer ID and the value is a list containing two elements, both of them numpy arrays (one has ints, the other has floats). The key is sequential, starts at 0, and there are no gaps, so the “outer” layer of this data structure could really just be a list with the key actually being the index. The lengths of each pair of arrays may differ across keys. For a visual: { key=0: [ numpy.array([1,8,15,…, 16000]), numpy.array([0.1,0.1,0.1,…,0.1]) ], key=1: [ numpy.array([5,6]), numpy.array([0.5,0.5]) ], … } I’ve tried: - manager proxy objects, but the object was so big that low-level code threw an exception due to format and monkey-patching wasn’t successful. - Redis, which was far too slow due to setting up connections and data conversion etc. - Numpy rec arrays + memory mapping, but there is a restriction that the numpy arrays in each “column” must be of fixed and same size. - I looked at PyTables, which may be a solution, but seems to have a very steep learning curve. - I haven’t tried SQLite3, but I am worried about the time it takes to query the DB for a sequential ID, and then translate byte arrays. Any ideas? I greatly appreciate any guidance you can provide. Thanks, Ryan

8 9

Get rid of special scalar arithmetic
by Charles R Harris Jan. 13, 2016

Jan. 13, 2016

Hi All, I've opened issue #7002 <https://github.com/numpy/numpy/issues/7002>, reproduced below, for discussion. > Numpy umath has a file scalarmath.c.src that implements scalar arithmetic > using special functions that are about 10x faster than the equivalent > ufuncs. > > In [1]: a = np.float64(1) > > In [2]: timeit a*a > 10000000 loops, best of 3: 69.5 ns per loop > > In [3]: timeit np.multiply(a, a) > 1000000 loops, best of 3: 722 ns per loop > > I contend that in large programs this improvement in execution time is not > worth the complexity and maintenance overhead; it is unlikely that > scalar-scalar arithmetic is a significant part of their execution time. > Therefore I propose to use ufuncs for all of the scalar-scalar arithmetic. > This would also bring the benefits of __numpy_ufunc__ to scalars with > minimal effort. > Thoughts? Chuck

5 4

1.11 branch this weekend
by Charles R Harris Jan. 12, 2016

Jan. 12, 2016

Hi All, Just a heads up. Note that we are trying to move to a faster release cycle and that changes the flavor of the releases. Instead of a list of goals to meet before each release, releases will more resemble snapshots of master at branch time. The main thing is that they be regression and bug free to the best of our ability when the branch is made. Chuck

1 0

Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]
by Robert McGibbon Jan. 12, 2016

Jan. 12, 2016

Hi all, I went ahead and tried to collect a list of all of the libraries that could be considered to constitute the "base" system for linux-64. The strategy I used was to leverage off the work done by the folks at Continuum by searching through their pre-compiled binaries from https://repo.continuum.io/pkgs/free/linux-64/ to find shared libraries that were dependened on (according to ldd) that were not accounted for by the declared dependencies that each package made known to the conda package manager. The full list of these system libraries, sorted in from most-commonly-depend-on to rarest, is below. There are 158 of them. ['linux-vdso.so.1', 'libc.so.6', 'libpthread.so.0', 'libm.so.6', 'libdl.so.2', 'libutil.so.1', 'libgcc_s.so.1', 'libstdc++.so.6', 'libexpat.so.1', 'librt.so.1', 'libpng12.so.0', 'libcrypt.so.1', 'libffi.so.6', 'libresolv.so.2', 'libkeyutils.so.1', 'libcom_err.so.2', 'libp11-kit.so.0', 'libkrb5.so.26', 'libheimntlm.so.0', 'libtasn1.so.6', 'libheimbase.so.1', 'libgssapi.so.3', 'libroken.so.18', 'libhcrypto.so.4', 'libhogweed.so.4', 'libnettle.so.6', 'libhx509.so.5', 'libwind.so.0', 'libgnutls-deb0.so.28', 'libasn1.so.8', 'libgmp.so.10', 'libsasl2.so.2', 'libidn.so.11', 'librtmp.so.1', 'liblber-2.4.so.2', 'libldap_r-2.4.so.2', 'libXdmcp.so.6', 'libX11.so.6', 'libXau.so.6', 'libxcb.so.1', 'libgssapi_krb5.so.2', 'libkrb5.so.3', 'libk5crypto.so.3', 'libkrb5support.so.0', 'libicudata.so.55', 'libicuuc.so.55', 'libhdf5_serial.so.10', 'libcurl-gnutls.so.4', 'libhdf5_serial_hl.so.10', 'libtinfo.so.5', 'libgcrypt.so.20', 'libgpg-error.so.0', 'libnsl.so.1', 'libXext.so.6', 'libncursesw.so.5', 'libpanelw.so.5', 'libXrender.so.1', 'libjbig.so.0', 'libpcre.so.3', 'libglib-2.0.so.0', 'libnvidia-tls.so.352.41', 'libnvidia-glcore.so.352.41', 'libGL.so.1', 'libuuid.so.1', 'libSM.so.6', 'libICE.so.6', 'libgobject-2.0.so.0', 'libgfortran.so.1', 'liblzma.so.5', 'libXt.so.6', 'libgmodule-2.0.so.0', 'libXi.so.6', 'libgstpbutils-1.0.so.0', 'liborc-0.4.so.0', 'libgstreamer-1.0.so.0', 'libgsttag-1.0.so.0', 'libgstvideo-1.0.so.0', 'libxslt.so.1', 'libaudio.so.2', 'libjpeg.so.8', 'libgstaudio-1.0.so.0', 'libgstbase-1.0.so.0', 'libgstapp-1.0.so.0', 'libz.so.1', 'libgthread-2.0.so.0', 'libfreetype.so.6', 'libfontconfig.so.1', 'libdbus-1.so.3', 'libsystemd.so.0', 'libltdl.so.7', 'libGLU.so.1', 'libsqlite3.so.0', 'libpgm-5.1.so.0', 'libgomp.so.1', 'libxcb-render.so.0', 'libxcb-shm.so.0', 'libncurses.so.5', 'libxml2.so.2', 'libXss.so.1', 'libXft.so.2', 'libtk.so', 'libtcl.so', 'libasound.so.2', 'libharfbuzz.so.0', 'libpixman-1.so.0', 'libgio-2.0.so.0', 'libXinerama.so.1', 'libselinux.so.1', 'libXcomposite.so.1', 'libthai.so.0', 'libXdamage.so.1', 'libgdk-x11-2.0.so.0', 'libpangoft2-1.0.so.0', 'libcairo.so.2', 'libpangocairo-1.0.so.0', 'libdatrie.so.1', 'libatk-1.0.so.0', 'libXcursor.so.1', 'libXfixes.so.3', 'libgraphite2.so.3', 'libgdk_pixbuf-2.0.so.0', 'libgtk-x11-2.0.so.0', 'libquadmath.so.0', 'libpango-1.0.so.0', 'libXrandr.so.2', 'libgfortran.so.3', 'libjson-c.so.2', 'libshiboken-python2.7.so.1.1', 'libogg.so.0', 'libvorbis.so.0', 'libatlas.so.3', 'libcurl.so.4', 'libhdf5.so.9', 'libodbcinst.so.1', 'libpcap.so.0.9', 'libnetcdf.so.7', 'libblas.so.3', 'libpulse.so.0', 'libcaca.so.0', 'libgstreamer-0.10.so.0', 'libXxf86vm.so.1', 'libhdf5_hl.so.9', 'libpulse-simple.so.0', 'libasyncns.so.0', 'libwrap.so.0', 'libvorbisenc.so.2', 'libmagic.so.1', 'libssl.so.1.0.0', 'libFLAC.so.8', 'libSDL-1.2.so.0', 'libsndfile.so.1', 'libslang.so.2', 'libglapi.so.0', 'libaio.so.1', 'libgstinterfaces-0.10.so.0', 'libpulsecommon-6.0.so', 'libjpeg.so.62', 'libcrypto.so.1.0.0'] This list actually contains a fair number of false positives, so it would need to be pruned manually. If you stare at it a little while, you might see some libraries in there that you recognize that shouldn't be part of the base system, like libatlas.so.3. This gist https://gist.github.com/rmcgibbo/a13e7623c38ec54fcc93 contains some more detailed data -- for each of libraries in the list above, it gives a list of names of the packages that depend on this library. For example, for libatlas.so.3, the there is only a single package which depends on it, ["scikit-learn-0.11-np16py27_ce0"]. So, probably a bug. "libgfortran.so.1" is also in the list. It's depended on by ["cvxopt-1.1.6-py27_0", "cvxopt-1.1.7-py27_0", "cvxopt-1.1.7-py34_0", "cvxopt-1.1.7-py35_0", "numpy-1.5.1-py27_1", "numpy-1.5.1-py27_3", "numpy-1.5.1-py27_4", "numpy-1.5.1-py27_ce0", "numpy-1.6.2-py27_1", "numpy-1.6.2-py27_3", "numpy-1.6.2-py27_4", "numpy-1.6.2-py27_ce0", "numpy-1.7.0-py27_0", "numpy-1.7.0b2-py27_ce0", "numpy-1.7.0rc1-py27_0", "numpy-1.7.1-py27_0", "numpy-1.7.1-py27_2", "numpy-1.8.0-py27_0", "numpy-1.8.1-py27_0", "numpy-1.8.1-py34_0", "numpy-1.8.2-py27_0", "numpy-1.8.2-py34_0", "numpy-1.9.0-py27_0", "numpy-1.9.0-py34_0", "numpy-1.9.1-py27_0", "numpy-1.9.1-py34_0", "numpy-1.9.2-py27_0", "numpy-1.9.2-py34_0"]. Note that this list of numpy versions doesn't include the latest ones -- all of the numpy-1.10 binaries made by Continuum pick up libgfortan from a conda package and don't depend on it being provided by the system. Also, the final '_0' or '_1' segment of many of these package names is the build number, which is to make a new release of the same release of a package, usually because of a packaging problem. So many of these packages were probably built incorrectly and superseded by new builds with a higher build number. So it's not perfect. But it might be a useful starting place. -Robert

4 8

Re: [Numpy-discussion] Windows build/distribute plan & MingwPy funding
by Erik Bray Jan. 9, 2016

Jan. 9, 2016

On Sat, Jan 2, 2016 at 3:20 AM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > Hi all, > > You probably know that building Numpy, Scipy and the rest of the Scipy Stack > on Windows is problematic. And that there are plans to adopt the static > MinGW-w64 based toolchain that Carl Kleffner has done a lot of work on for > the last two years to fix that situation. > > The good news is: this has become a lot more concrete just now, with this > proposal for funding: http://mingwpy.github.io/proposal_december2015.html > > Funding for phases 1 and 2 is already confirmed; the phase 3 part has been > submitted to the PSF. Phase 1 (of $1000) is funded by donations made to > Numpy and Scipy (through NumFOCUS), and phase 2 (of $4000) by NumFOCUS > directly. So a big thank you to everyone who made a donation to Numpy, Scipy > and NumFOCUS! > > I hope that that proposal gives a clear idea of the work that's going to be > done over the next months. Note that the http://mingwpy.github.io contains a > lot more background info, description of technical issues, etc. > > Feedback & ideas very welcome of course! > > Cheers, > Ralf Hi Ralph, I've seen you drop hints about this recently, and am interested to follow this work. I've been hired as part of the OpenDreamKit project to work, in large part, on developing a sensible toolchain for building and distributing Sage on Windows. I know you've been following the thread on that too. Although the primary goal there is "whatever works", I'm personally inclined to focus on the mingwpy / mingw-w64 approach, due in large part with my past success with the MinGW 32-bit toolchain. (I personally have a desire to improve support for building with MSVC as well, but that's a less important goal as far as the funding is concerned.) So anyways, please keep me in the loop about this, as I will also be putting effort into this over the next year as well. Has there been any discussion about setting up a mailing list specifically for this project? Thanks, Erik

3 2

Generalize hstack/vstack --> stack; Block matrices like in matlab
by Stefan Otte Jan. 9, 2016

Jan. 9, 2016

Hey, quite often I work with block matrices. Matlab offers the convenient notation [ a b; c d ] to stack matrices. The numpy equivalent is kinda clumsy: vstack([hstack([a,b]), hstack([c,d])]) I wrote the little function `stack` that does exactly that: stack([[a, b], [c, d]]) In my case `stack` replaced `hstack` and `vstack` almost completely. If you're interested in including it in numpy I created a pull request [1]. I'm looking forward to getting some feedback! Best, Stefan [1] https://github.com/numpy/numpy/pull/5057

12 24