Mailman 3 February 2016 - NumPy-Discussion

Fwd: Multi-distribution Linux wheels - please test
by Freddy Rietdijk 09 Feb '16

09 Feb '16

On Nix we also had trouble with OpenBLAS 0.2.15. Version 0.2.14 did not cause any segmentation faults so we reverted to that version. https://github.com/scipy/scipy/issues/5620 (hopefully this time the e-mail gets through) On Tue, Feb 9, 2016 at 10:21 AM, Daπid <davidmenhur(a)gmail.com> wrote: > On 8 February 2016 at 20:25, Matthew Brett <matthew.brett(a)gmail.com> > wrote: > >> >> I used the latest release, v0.2.15: >> >> https://github.com/matthew-brett/manylinux-builds/blob/master/build_openbla… >> >> Is there a later version that we should try? >> >> Cheers, >> > > That is the one in the Fedora repos that is working for me. How are you > compiling it? > > Mine is compiled with GCC 5 with the options seen in the source rpm: > http://koji.fedoraproject.org/koji/packageinfo?packageID=15277 > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > >

1 0

Multi-distribution Linux wheels - please test
by Matthew Brett 09 Feb '16

09 Feb '16

Hi, As some of you may have seen, Robert McGibbon and Nathaniel have just guided a PEP for multi-distribution Linux wheels past the approval process over on distutils-sig: https://www.python.org/dev/peps/pep-0513/ The PEP includes a docker image on which y'all can build wheels which match the PEP: https://quay.io/repository/manylinux/manylinux Now we're at the stage where we need stress-testing of the built wheels to find any problems we hadn't thought of. I've built numpy and scipy wheels here: https://nipy.bic.berkeley.edu/manylinux/ So, if you have a Linux distribution handy, we would love to hear from you about the results of testing these guys, maybe on the lines of: pip install -f https://nipy.bic.berkeley.edu/manylinux numpy scipy python -c 'import numpy; numpy.test()' python -c 'import scipy; scipy.test()' These manylinux wheels should soon be available on pypi, and soon after, installable with latest pip, so we would like to fix as many problems as possible before going live. Cheers, Matthew

7 24

resizeable arrays using shared memory?
by Elliot Hallmark 09 Feb '16

09 Feb '16

Hi all, I have a program that uses resize-able arrays. I already over-provision the arrays and use slices, but every now and then the data outgrows that array and it needs to be resized. Now, I would like to have these arrays shared between processes spawned via multiprocessing (for fast interprocess communication purposes, not for parallelizing work on an array). I don't care about mapping to a file on disk, and I don't want disk I/O happening. I don't care (really) about data being copied in memory on resize. I *do* want the array to be resized "in place", so that the child processes can still access the arrays from the object they were initialized with. I can share arrays easily using arrays that are backed by memmap. Ie: ``` #Source: http://github.com/rainwoodman/sharedmem class anonymousmemmap(numpy.memmap): def __new__(subtype, shape, dtype=numpy.uint8, order='C'): descr = numpy.dtype(dtype) _dbytes = descr.itemsize shape = numpy.atleast_1d(shape) size = 1 for k in shape: size *= k bytes = int(size*_dbytes) if bytes > 0: mm = mmap.mmap(-1,bytes) else: mm = numpy.empty(0, dtype=descr) self = numpy.ndarray.__new__(subtype, shape, dtype=descr, buffer=mm, order=order) self._mmap = mm return self def __array_wrap__(self, outarr, context=None): return numpy.ndarray.__array_wrap__(self.view(numpy.ndarray), outarr, context) ``` This cannot be resized because it does not own it's own data (ValueError: cannot resize this array: it does not own its data). (numpy.memmap has this same issue [0], even if I set refcheck to False and even though the docs say otherwise [1]). arr._mmap.resize(x) fails because it is annonymous (error: [Errno 9] Bad file descriptor). If I create a file and use that fileno to create the memmap, then I can resize `arr._mmap` but the array itself is not resized. Is there a way to accomplish what I want? Or, do I just need to figure out a way to communicate new arrays to the child processes? Thanks, Elliot [0] https://github.com/numpy/numpy/issues/4198. [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.resize.html

4 4

Re: [Numpy-discussion] [SciPy-Dev] Multi-distribution Linux wheels - please test
by Nathaniel Smith 08 Feb '16

08 Feb '16

On Feb 7, 2016 15:27, "Charles R Harris" <charlesr.harris(a)gmail.com> wrote: > > > > On Sun, Feb 7, 2016 at 2:16 PM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> On Sun, Feb 7, 2016 at 9:49 AM, Charles R Harris >> <charlesr.harris(a)gmail.com> wrote: >> > >> > >> > On Sun, Feb 7, 2016 at 3:40 AM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> >> >> On Feb 6, 2016 12:27 PM, "Matthew Brett" <matthew.brett(a)gmail.com> wrote: >> >> > >> >> > Hi, >> >> > >> >> > As some of you may have seen, Robert McGibbon and Nathaniel have just >> >> > guided a PEP for multi-distribution Linux wheels past the approval >> >> > process over on distutils-sig: >> >> > >> >> > https://www.python.org/dev/peps/pep-0513/ >> >> > >> >> > The PEP includes a docker image on which y'all can build wheels which >> >> > match the PEP: >> >> > >> >> > https://quay.io/repository/manylinux/manylinux >> >> >> >> This is the wrong repository :-) It moved, and there are two now: >> >> >> >> quay.io/pypa/manylinux1_x86_64 >> >> quay.io/pypa/manylinux1_i686 >> > >> > >> > I'm going to put out 1.11.0b3 today. What would be the best thing to do for >> > testing? >> >> I'd say, don't worry about building linux wheels as part of the >> release cycle yet -- it'll still be a bit before they're allowed on >> pypi or pip will recognize the new special tag. So for now you can >> leave it to Matthew or someone to build test images and stick them up >> on a server somewhere, same as before :-) > > > Should I try putting the sources up on pypi? +1 -n

1 0

ANN: numexpr 2.5
by Francesc Alted 06 Feb '16

06 Feb '16

========================= Announcing Numexpr 2.5 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== In this version, a lock has been added so that numexpr can be called from multithreaded apps. Mind that this does not prevent numexpr to use multiple cores internally. Also, a new min() and max() functions have been added. Thanks to contributors! In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

Re: [Numpy-discussion] Numpy 1.11.0b2 released
by josef.pktd＠gmail.com 05 Feb '16

05 Feb '16

On Fri, Feb 5, 2016 at 3:24 PM, <josef.pktd(a)gmail.com> wrote: > > > On Fri, Feb 5, 2016 at 1:14 PM, Pauli Virtanen <pav(a)iki.fi> wrote: > >> 05.02.2016, 19:55, Nathaniel Smith kirjoitti: >> > On Feb 5, 2016 8:28 AM, "Chris Barker - NOAA Federal" < >> chris.barker(a)noaa.gov> >> > wrote: >> >> >> >>> An extra ~2 hours of tests / 6-way parallelism is not that big a deal >> >>> in the grand scheme of things (and I guess it's probably less than >> >>> that if we can take advantage of existing binary builds) >> >> >> >> If we set up a numpy-testing conda channel, it could be used to cache >> >> binary builds for all he versions of everything we want to test >> >> against. >> >> >> >> Conda-build-all could make it manageable to maintain that channel. >> > >> > What would be the advantage of maintaining that channel ourselves >> instead >> > of using someone else's binary builds that already exist (e.g. >> Anaconda's, >> > or official project wheels)? >> >> ABI compatibility. However, as I understand it, not many backward ABI >> incompatible changes in Numpy are not expected in future. >> >> If they were, I note that if you work in the same environment, you can >> push repeated compilation times to zero compared to the time it takes to >> run tests in a way that requires less configuration, by enabling >> ccache/f90cache. >> > > > control of fortran compiler and libraries > > I was just looking at some new test errors on TravisCI in unchanged code > of statsmodels, and it looks like conda switched from openblas to mkl > yesterday. > > (statsmodels doesn't care when compiling which BLAS/LAPACK is used as long > as they work because we don't have Fortran code.) > > Josef > > (sending again, delivery refused) > > >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >

1 0

[OT] Interpolation of an unevently sampled bandwidth limited signal
by Nadav Horesh 04 Feb '16

04 Feb '16

I have several cases of hand digitized spectra that I'd like to resample these spectra at even spacings. My problem is that cubic or RBF splines often result in an unacceptible over-shooting. Is there a python module that provides something similar to sinc interpolation on unevenly space sampled signal? Nadav.

3 4

Re: [Numpy-discussion] [OT] Interpolation of an unevently sampled, bandwidth limited signal
by Jonathan Stickel 04 Feb '16

04 Feb '16

On 2/4/16 02:42 , numpy-discussion-request(a)scipy.org wrote: > Date: Thu, 4 Feb 2016 09:32:36 +0000 > From: Nadav Horesh<nadavh(a)visionsense.com> > To: numpy-discussion<numpy-discussion(a)scipy.org> > Subject: [Numpy-discussion] [OT] Interpolation of an unevently sampled > bandwidth limited signal > Message-ID: > <SN1PR08MB1758DB00A8EE38386BCA3D88CBD10(a)SN1PR08MB1758.namprd08.prod.outlook.com> > > Content-Type: text/plain; charset="iso-8859-1" > > I have several cases of hand digitized spectra that I'd like to resample these spectra at even spacings. My problem is that cubic or RBF splines often result in an unacceptible over-shooting. Is there a python module that provides something similar to sinc interpolation on unevenly space sampled signal? > > > Nadav. You might try scikit-datasmooth: https://pypi.python.org/pypi/scikits.datasmooth BTW, this wouldn't be offtopic on the scipy-user list. Regards, Jonathan

1 0

julia - Multidimensional algorithms and iteration
by Neal Becker 03 Feb '16

03 Feb '16

http://julialang.org/blog/2016/02/iteration/

2 1

Numpy 1.11.0b1 is out
by Charles R Harris 02 Feb '16

02 Feb '16

Hi All, I'm pleased to announce that Numpy 1.11.0b1 <http://sourceforge.net/projects/numpy/files/NumPy/1.11.0b1/> is now available on sourceforge. This is a source release as the mingw32 toolchain is broken. Please test it out and report any errors that you discover. Hopefully we can do better with 1.11.0 than we did with 1.10.0 ;) Chuck

5 12