From ralf.gommers at gmail.com Sun Jun 1 03:49:05 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 1 Jun 2014 09:49:05 +0200 Subject: [Numpy-discussion] Renaming OSX wheels on pypi to make them more general In-Reply-To: References: Message-ID: On Sun, Jun 1, 2014 at 4:49 AM, Matthew Brett wrote: > Hi, > > On Fri, May 30, 2014 at 2:17 PM, Matthew Brett > wrote: > > Hi, > > > > This is actually for both of numpy and scipy. > > > > I would like to rename the current OSX wheels on pypi so that they > > will be installed by default on system python, homebrew, macports, as > > well as Python.org Python. > > > > At the moment, they will only be found and installed by default by > > Python.org Python. > > > > For reasons explained here: > > > > https://github.com/MacPython/wiki/wiki/Spinning-wheels > > > > and confirmed with testing here: > > > > > https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/25131865 > > > > - OSX wheels built for Python.org python do in fact work correctly for > > the homebrew, macports and system python. > > > > In fact, future versions of pip will very likely offer the Python.org > > OSX wheels for installation on these other systems by default: > > > > https://github.com/pypa/pip/pull/1465 > > > > Renaming the wheels just adds the 'platform tag' for these other > > versions of Python to the wheel name, so pip sees they are compatible. > > > > For example, I propose to rename the current numpy wheel from: > > > > numpy-1.8.1-cp27-none-macosx_10_6_intel.whl > > > > to: > > > > > numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl > > > > I think this is only an improvement to the current situation, in that > > users of pip on these other OSX systems will get a fast binary install > > rather than a slow compiled install. > > > > Any comments? > > OK - I propose to go ahead with this on Monday unless there are any > objections. > +1 I don't see any problem with this. Ralf Here's the latest test grid showing tests passing on Python.org / > system python / homebrew / macports using numpy / scipy / matplotlib > wheels with the proposed naming scheme: > > https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/26482436 > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Mon Jun 2 07:27:25 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 02 Jun 2014 07:27:25 -0400 Subject: [Numpy-discussion] fftw supported? Message-ID: I just d/l numpy-1.8.1 and try to build. I uncomment: [fftw] libraries = fftw3 This is fedora 20. fftw3 (and devel) is installed as fftw. I see nothing written to stderr during the build that has any reference to fftw. From sebastian at sipsolutions.net Mon Jun 2 07:36:28 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Mon, 02 Jun 2014 13:36:28 +0200 Subject: [Numpy-discussion] fftw supported? In-Reply-To: References: Message-ID: <1401708988.6437.0.camel@sebastian-t440> On Mo, 2014-06-02 at 07:27 -0400, Neal Becker wrote: > I just d/l numpy-1.8.1 and try to build. I uncomment: > > [fftw] > libraries = fftw3 > > This is fedora 20. fftw3 (and devel) is installed as fftw. > > I see nothing written to stderr during the build that has any reference to fftw. > I don't know the details, but this is not supported currently. It did work for some old versions of numpy I think. - Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From ndbecker2 at gmail.com Mon Jun 2 07:41:52 2014 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 02 Jun 2014 07:41:52 -0400 Subject: [Numpy-discussion] fftw supported? References: <1401708988.6437.0.camel@sebastian-t440> Message-ID: Sebastian Berg wrote: > On Mo, 2014-06-02 at 07:27 -0400, Neal Becker wrote: >> I just d/l numpy-1.8.1 and try to build. I uncomment: >> >> [fftw] >> libraries = fftw3 >> >> This is fedora 20. fftw3 (and devel) is installed as fftw. >> >> I see nothing written to stderr during the build that has any reference to >> fftw. >> > > I don't know the details, but this is not supported currently. It did > work for some old versions of numpy I think. > > - Sebastian > If fftw is not supported anymore, then the comments should be removed from site.cfg.example From cournape at gmail.com Mon Jun 2 08:26:58 2014 From: cournape at gmail.com (David Cournapeau) Date: Mon, 2 Jun 2014 13:26:58 +0100 Subject: [Numpy-discussion] fftw supported? In-Reply-To: References: Message-ID: FFTW is not used anymore in neither numpy or scipy (has not been for several years). If you want to use fftw with numpy, there are 3rd party extensions to do it, like pyfftw On Mon, Jun 2, 2014 at 12:27 PM, Neal Becker wrote: > I just d/l numpy-1.8.1 and try to build. I uncomment: > > [fftw] > libraries = fftw3 > > This is fedora 20. fftw3 (and devel) is installed as fftw. > > I see nothing written to stderr during the build that has any reference to > fftw. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaoyuejoy at gmail.com Mon Jun 2 09:58:42 2014 From: chaoyuejoy at gmail.com (Chao YUE) Date: Mon, 2 Jun 2014 15:58:42 +0200 Subject: [Numpy-discussion] percentile function for masked array? Message-ID: Dear all, It seems that there is not a percentile function for masked array in numpy or scipy? I checked numpy.percentile and scipy.percentile, it seems not support only nonmasked array? And there is no percentile function in scipy.stats.mstats, so I have to use np.percentile(arr.compressed()) I guess. Thanks for any comments. Best, Chao -- please visit: http://www.globalcarbonatlas.org/ *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Mon Jun 2 10:07:10 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 2 Jun 2014 16:07:10 +0200 Subject: [Numpy-discussion] fftw supported? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 2:26 PM, David Cournapeau wrote: > FFTW is not used anymore in neither numpy or scipy (has not been for several > years). If you want to use fftw with numpy, there are 3rd party extensions > to do it, like pyfftw > > If it was once supported in numpy, does someone remember the reason why it was removed? Was it the license or a technical reason? In the long term I would like to add fftw support to numpy, at least for the source distribution where the licensing is no concern. Some testing I did a while ago showed fftpack and fftw have different numerical behaviour and depending on the size fftpack can be faster due to fftws overheads when not using pregenerated plans, so to avoid breaking user code the implementation should probably be selectable at runtime by the user instead of hardwiring the implementation at compile time. But I don't know if I'll ever find the time to do so :( From robert.kern at gmail.com Mon Jun 2 10:10:24 2014 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 2 Jun 2014 15:10:24 +0100 Subject: [Numpy-discussion] fftw supported? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 3:07 PM, Julian Taylor wrote: > On Mon, Jun 2, 2014 at 2:26 PM, David Cournapeau wrote: >> FFTW is not used anymore in neither numpy or scipy (has not been for several >> years). If you want to use fftw with numpy, there are 3rd party extensions >> to do it, like pyfftw > > If it was once supported in numpy, does someone remember the reason > why it was removed? Was it the license or a technical reason? It was never supported in numpy. It used to be supported in scipy, but was removed because the maintenance burden outstripped the performance benefits. -- Robert Kern From jtaylor.debian at googlemail.com Mon Jun 2 11:29:32 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 2 Jun 2014 17:29:32 +0200 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 3:58 PM, Chao YUE wrote: > Dear all, > > It seems that there is not a percentile function for masked array in numpy > or scipy? > I checked numpy.percentile and scipy.percentile, it seems not support only > nonmasked array? And there is no percentile function in scipy.stats.mstats, > so I have to use np.percentile(arr.compressed()) I guess. > > Thanks for any comments. > there is currently no ma.percentile, but numpy 1.9 which we will hopefully release as a first beta very soon, will contain np.nanpercentile which can be used to emulate ma.percentile for floating point data: r = np.nanpercentile(maskedarray.filled(np.nan), (5, 95), axis=(0,1)) r = ma.masked_array(r, np.isnan(r)) for 1 dimensional arrays np.percentile(arr.compressed(), (5, 95), overwrite_input=True) works fine and is also the fastest possible way. a generic masked percentile would be useful, patches for that are welcome. Ideally a patch should also take care of the poor performance of multitimensional nanpercentile along small axes similar to how this PR fixes it for ma.median/nanmedian: https://github.com/numpy/numpy/pull/4760 From ndarray at mac.com Mon Jun 2 11:30:12 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Mon, 2 Jun 2014 11:30:12 -0400 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: > > It seems that there is not a percentile function for masked array in numpy > or scipy? > Percentile is not the only function missing in ma. See for example https://github.com/numpy/numpy/issues/4356 https://github.com/numpy/numpy/issues/4355 It seems to me that ma was treated on par with np.matrix in the recent years while several attempts were made to replace it with something better. I don't think any better alternative have materialized, so it is probably time to declare that ma in the supported mechanism to deal with missing values in numpy and make an effort to keep the np and ma interfaces in sync. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 2 11:48:43 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 2 Jun 2014 09:48:43 -0600 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 9:30 AM, Alexander Belopolsky wrote: > It seems that there is not a percentile function for masked array in numpy >> or scipy? >> > > Percentile is not the only function missing in ma. See for example > > https://github.com/numpy/numpy/issues/4356 > https://github.com/numpy/numpy/issues/4355 > > It seems to me that ma was treated on par with np.matrix in the recent > years while several attempts were made to replace it with something better. > > I don't think any better alternative have materialized, so it is probably > time to declare that ma in the supported mechanism to deal with missing > values in numpy and make an effort to keep the np and ma interfaces in > sync. > Masked arrays have no maintainer, and haven't for several years, nor do I see anyone coming along to take it up. It would be good to have methods for dealing with missing values in mainline as they would get more maintenance. Perhaps it is time to reopen that discussion and find a way forward. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Jun 2 12:15:29 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Mon, 2 Jun 2014 12:15:29 -0400 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 11:48 AM, Charles R Harris wrote: > Masked arrays have no maintainer, and haven't for several years, nor do I > see anyone coming along to take it up. I was effectively a maintainer of ma after Numeric -> numpy transition and before it was rewritten to use inheritance from ndarray. I cannot commit to implementing new features myself, but I will review the patches that come along. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 2 12:25:00 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 2 Jun 2014 10:25:00 -0600 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 10:15 AM, Alexander Belopolsky wrote: > > On Mon, Jun 2, 2014 at 11:48 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> Masked arrays have no maintainer, and haven't for several years, nor do I >> see anyone coming along to take it up. > > > I was effectively a maintainer of ma after Numeric -> numpy transition and > before it was rewritten to use inheritance from ndarray. > > I cannot commit to implementing new features myself, but I will review the > patches that come along. > Most recent ma patches are coming from the astropy folks who want masked arrays to work better for numpy subclasses. We've put off committing them until 1.10 development begins, but there are several in the queue. I think the masked array code is also due a cleanup/rationalization. Any comments you have along that line are welcome. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Jun 2 12:50:06 2014 From: ndarray at mac.com (Alexander Belopolsky) Date: Mon, 2 Jun 2014 12:50:06 -0400 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: On Mon, Jun 2, 2014 at 12:25 PM, Charles R Harris wrote: > I think the masked array code is also due a cleanup/rationalization. Any > comments you have along that line are welcome. Here are a few thoughts: 1. Please avoid another major rewrite. 2. Stop pretending that instances of ma.MaskedArray and ndarray have "is a" relationship. Use of inheritance should become an implementation detail and any method that is not explicitly overridden should raise an exception. 3. Add a mechanism to keep numpy and numpy.ma APIs in sync. At a minimum - add a test comparing public functions and methods and for pure python functions compare signatures. 4. Consider deprecating the ma.masked scalar. 5. Support duck-typing in MaskedArray constructors. If supplied data object has mask attribute it should be used as mask. This will allow interoperability with alternative missing values implementations. (ndarray may itself grow mask attribute one day which will be equivalent to isnan. Bit views, anyone?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Jun 2 13:35:00 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 2 Jun 2014 10:35:00 -0700 Subject: [Numpy-discussion] Distutils - way to check validity of compiler flag? In-Reply-To: References: Message-ID: Hi, On Mon, May 12, 2014 at 12:50 PM, Robert McGibbon wrote: > In a couple of my projects, we check for flags by compiling little test > files -- autotools style -- to check for SSE, OpenMP, etc. See e.g. > https://github.com/rmcgibbo/mdtraj/blob/master/setup.py#L215 > > If anyone has a better solution, I'm all ears. > > -Robert > > > On Mon, May 12, 2014 at 12:24 PM, Matthew Brett > wrote: >> >> Hi, >> >> I'm sorry to ask this, I guess I should know - but is there any way in >> disutils or numpy distutils to check whether a compiler flag is valid >> before doing extension building? >> >> I'm thinking of something like this, to check whether the compiler can >> handle '-fopenmp': >> >> have_openmp = check_compiler_flag('-fopenmp') >> flags = ['-fopenmp'] if have_openmp else [] >> >> ext = Extension('myext', ['myext.c'], >> extra_compile_args = flags, >> extra_link_args = flags]) >> >> I guess this would have to go somewhere in the main setup() call in >> order to pick up custom compilers on the command line and such? Thanks a lot for the reply. Your code is nice and clean, but I think that it won't pick up custom compilers such as python setup.py build --compiler=mingw32 or similar? I ended up going for an approach suggested by Min RK of IPython fame - code here: https://github.com/nipy/dipy/pull/371/files The basic idea is to wait until the last distutils moment after the compiler has been configured, and then test with that exact compiler setup. It's a little ugly and not too general as it stands, but I'm posting in case someone runs into the same thing. Cheers, Matthew From kyle.mandli at gmail.com Tue Jun 3 19:08:55 2014 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Tue, 3 Jun 2014 18:08:55 -0500 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation Message-ID: Hello everyone, As one of the co-chairs in charge of organizing the birds-of-a-feather sesssions at the SciPy conference this year, I wanted to solicit through the NumPy list to see if we could get enough interest to hold a NumPy centered BoF this year. The BoF format would be up to those who would lead the discussion, a couple of ideas used in the past include picking out a few of the lead devs to be on a panel and have a Q&A type of session or an open Q&A with perhaps audience guided list of topics. I can help facilitate organization of something but we would really like to get something organized this year (last year NumPy was the only major project that was not really represented in the BoF sessions). Thanks! Kyle Manldi (and via proxy Matt McCormick) -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Jun 3 19:33:21 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 3 Jun 2014 17:33:21 -0600 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: Message-ID: On Tue, Jun 3, 2014 at 5:08 PM, Kyle Mandli wrote: > Hello everyone, > > As one of the co-chairs in charge of organizing the birds-of-a-feather > sesssions at the SciPy conference this year, I wanted to solicit through > the NumPy list to see if we could get enough interest to hold a NumPy > centered BoF this year. The BoF format would be up to those who would lead > the discussion, a couple of ideas used in the past include picking out a > few of the lead devs to be on a panel and have a Q&A type of session or an > open Q&A with perhaps audience guided list of topics. I can help > facilitate organization of something but we would really like to get > something organized this year (last year NumPy was the only major project > that was not really represented in the BoF sessions). > > I'll be at the conference, but I don't know who else will be there. I feel that NumPy has matured to the point where most of the current work is cleaning stuff up, making it run faster, and fixing bugs. A topic that I'd like to see discussed is where do we go from here. One option to look at is Blaze, which looks to have matured a lot in the last year. The problem with making it a NumPy replacement is that NumPy has become quite widespread, with downloads from PyPi running at about 3 million per year. With that much penetration it may be difficult for a new core like Blaze to gain traction. So I'd like to also discuss ways to bring the two branches of development together at some point and explore what NumPy can do to pave the way. Mind, there are definitely things that would be nice to add to NumPy, a better type system, missing values, etc., but doing that is difficult given the current design. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Jun 3 21:26:46 2014 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Jun 2014 02:26:46 +0100 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: Message-ID: On Wed, Jun 4, 2014 at 12:33 AM, Charles R Harris wrote: > On Tue, Jun 3, 2014 at 5:08 PM, Kyle Mandli wrote: >> >> Hello everyone, >> >> As one of the co-chairs in charge of organizing the birds-of-a-feather >> sesssions at the SciPy conference this year, I wanted to solicit through the >> NumPy list to see if we could get enough interest to hold a NumPy centered >> BoF this year. The BoF format would be up to those who would lead the >> discussion, a couple of ideas used in the past include picking out a few of >> the lead devs to be on a panel and have a Q&A type of session or an open Q&A >> with perhaps audience guided list of topics. I can help facilitate >> organization of something but we would really like to get something >> organized this year (last year NumPy was the only major project that was not >> really represented in the BoF sessions). > > I'll be at the conference, but I don't know who else will be there. I feel > that NumPy has matured to the point where most of the current work is > cleaning stuff up, making it run faster, and fixing bugs. A topic that I'd > like to see discussed is where do we go from here. One option to look at is > Blaze, which looks to have matured a lot in the last year. The problem with > making it a NumPy replacement is that NumPy has become quite widespread, > with downloads from PyPi running at about 3 million per year. With that much > penetration it may be difficult for a new core like Blaze to gain traction. > So I'd like to also discuss ways to bring the two branches of development > together at some point and explore what NumPy can do to pave the way. Mind, > there are definitely things that would be nice to add to NumPy, a better > type system, missing values, etc., but doing that is difficult given the > current design. I won't be at the conference unfortunately (I'm on the wrong continent and have family commitments then anyway), but I think there's lots of exciting stuff that can be done in numpy-land. We absolutely could rewrite the dtype system, and this would straightforwardly give us excellent support for missing values, units, categorical data, automatic differentiation, better datetimes, etc. etc. -- and make numpy much more friendly in general to third-party extensions. I'd like to see the ufunc system revisited in the light of all the things we know now, to make gufuncs more first-class, provide better support for user-defined types, more flexible loop selection (e.g. make it possible to implement np.add.reduce(a, type="kahan")), etc.; one goal would be to convert a lot of ufunc-like functions (np.mean etc.) into being real ufuncs, and then they'd automatically benefit from __numpy_ufunc__, which would also massively improve interoperability with alternative array types like blaze. I'd like to see support for extensible label-based indexing, like pandas. Internally, I'd like to see internal migrating out of C and into Cython -- we have hundreds of lines of code that could be replaced with a few lines of Cython and no-one would notice. (Combining this with a cffi cython backend and pypy would be pretty interesting too...) I'd like to see sparse ndarrays, with integration into the ufunc looping machinery so all ufuncs just work. Or even better, I'd like to see the right hooks added so that anyone can write a sparse ndarray package using only public APIs, and have all ufuncs just work. (I was going to put down deferred/loop-fused/out-of-core computation as a wishlist item too, but if we do it right then this too could be implemented by anyone without needing to be baked into numpy proper.) All of these things would take some work and care, but I think they could all be done incrementally and without breaking backwards compatibility. Compare to ipython, which -- as Fernando likes to point out :-) -- went from a little console program to its current distributed-notebook-skynet-whatever-it-is by merging one working PR at a time. Certainly these changes would much easier and less disruptive than any plan that involves throwing out numpy and starting over. But they also do help smooth the way for an incremental transition to a world where numpy is regularly used alongside other libraries. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From travis at continuum.io Wed Jun 4 02:18:20 2014 From: travis at continuum.io (Travis Oliphant) Date: Wed, 4 Jun 2014 01:18:20 -0500 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: Message-ID: I will be at the conference as will Mark Wiebe for at least part of the time. Others from the Blaze team like Andy Terrel and Matthew Rocklin will also be available at least part of the time (so it depends on when the BoF is). I'm sure they will all have opinions about this. I would be happy to be involved with a discussion around the future of NumPy as it is one of the things I've been thinking about for quite a while. Obviously, what happens will be more a function of what people have resources to do than just what is discussed, but it is helpful to get people from multiple projects discussing what they are working on and how it relates or could relate to a possible NumPy 2.0 effort. I'm happy to participate. My bias is that I do not believe it is going to be possible practically to simply modify NumPy itself directly. This was the original direction we considered when we started Continuum -- and spent some time and money in that direction --- but it's a difficult problem that would require a lot of time and patience and testing from multiple people. I'm not sure IPython is the right project to compare against here as it's user-story is quite different. NumPy is already a hybrid that evolved from Numeric. Of course it is likely *technically* feasible. We could replace every implementation detail with something different --- but not without likely impact on users and more cost than it would be just to re-write sections. However, the challenge is more about the user-base (especially the silent but large user-base), the semantic expectations of that user base, and the challenge that exists in really creating a test suite that covers the entire surface area of actual NumPy use. Even relatively simple changes can have significant impact at this point. Nathaniel has laid out a fantastic list of great features. These are the kind of features I have been eager to see as well. This is why I have been working to fund and help explore these ideas in the Numba array object as well as in Blaze. Gnumpy, Theano, Pandas, and other projects also have useful tales to tell regarding a potential NumPy 2.0. Ultimately, I do think it is time to talk seriously about NumPy 2.0, and what it might look like. I personally think it looks a lot more like a re-write, than a continuation of the modifications of Numeric that became NumPy 1.0. Right out of the gate, for example, I would make sure that NumPy 2.0 objects somehow used PyObject_VAR_HEAD so that they were variable-sized objects where the strides and dimension information was stored directly in the object structure itself instead of allocated separately (thus requiring additional loads and stores from memory). This would be a relatively simple change. But, it can't be done and preserve ABI compatibility. It may also, at this point, have impact on Cython code, or other code that is deeply-aware of the NumPy code-structure. Some of the changes that should be made will ultimately require a porting exercise for new code --- at which point why not just use a new project. Dynd (which is a separate but related project from Blaze) is actually a pretty good start to a NumPy 2.0 already: https://github.com/ContinuumIO/dynd-python and https://github.com/ContinuumIO/libdynd (C++ library). It can be provided with a backwards-compatible API without too much difficulty so that extension modules built for NumPy 1.X would still work. Numba can support Dynd and Numba's array object provides a useful, deferred-expression evaluation mechanism along with JIT compilation when desired that can support the GPU. I would make the case that by the end of the year this combination of Dynd plus Numba (and it's array object) could easily provide much of the functionality needed for a solid NumPy++. Separate from that, Blaze provides a pluggable mechanism so that array-oriented computations can be done on a large-variety of backends (including distributed systems). I agree that users of NumPy should not have to see a big API change in 2.0 --- but any modification of indexing or calculations would present slightly different semantics in certain corner cases --- which I think will be unavoidable in NumPy 2.0 regardless of how it is created. I also think NumPy 2.0 should take the opportunity to look hard at the API and what can be simplified (do we have the right collection of methods?). I'm also a big fan of introducing a common "array of structure" object that has a smaller API footprint than Pandas but has indexing and group-by functionality. Fortunately, with the buffer protocol in Python, multiple array objects can easily co-exist in the Python ecosystem with no memory copies. I think that is where we are headed and I don't see it as a bad thing. I think agreeing on how to describe types would be very beneficial (it's an under-developed part of the buffer protocol). This is exactly why we have made datashape an independent project that other projects can use as a data-type-description mini-language: https://github.com/ContinuumIO/datashape I think that a really good project for an enterprising young graduate student, post-doc, or professor (who is willing to delay their PhD or risk their tenure) would be to re-write the ufunc system using more modern techniques and put generalized ufuncs front and center as Nathaniel described. It sounds like many agree that we can improve the ufunc object implementation. A new ufunc system is an entirely achievable goal and could even be shipped as an "add-on" project external from NumPy for several years before being adopted fully. I know at least 4 people with demo-ware versions of a new ufunc-object that could easily replace current NumPy ufuncs eventually. If you are interested in that, I would love to share what I know with you. After spending quite a bit of time thinking about this over the past 2 years, interacting with many in the user community outside of this list, and working with people as they explore a few options --- I do have a fair set of opinions. But, there are also a lot of possibilities and many opportunities. I'm looking forward to seeing what emerges in the coming months and years and cooperating where possible with others having overlapping interests. Best, -Travis On Tue, Jun 3, 2014 at 6:08 PM, Kyle Mandli wrote: > Hello everyone, > > As one of the co-chairs in charge of organizing the birds-of-a-feather > sesssions at the SciPy conference this year, I wanted to solicit through > the NumPy list to see if we could get enough interest to hold a NumPy > centered BoF this year. The BoF format would be up to those who would lead > the discussion, a couple of ideas used in the past include picking out a > few of the lead devs to be on a panel and have a Q&A type of session or an > open Q&A with perhaps audience guided list of topics. I can help > facilitate organization of something but we would really like to get > something organized this year (last year NumPy was the only major project > that was not really represented in the BoF sessions). > > Thanks! > > Kyle Manldi (and via proxy Matt McCormick) > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Travis Oliphant CEO Continuum Analytics, Inc. http://www.continuum.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From chaoyuejoy at gmail.com Wed Jun 4 04:02:06 2014 From: chaoyuejoy at gmail.com (Chao YUE) Date: Wed, 4 Jun 2014 10:02:06 +0200 Subject: [Numpy-discussion] percentile function for masked array? In-Reply-To: References: Message-ID: Dear all, Thank you for this information. I will return to this issue later and probably make patch (as temporary solution) for this. Because I never tried before, so it may take me some time. For the other overall masked array constructing issues, it might be left further for more discussion. Best, Chao On Mon, Jun 2, 2014 at 6:50 PM, Alexander Belopolsky wrote: > > On Mon, Jun 2, 2014 at 12:25 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> I think the masked array code is also due a cleanup/rationalization. Any >> comments you have along that line are welcome. > > > Here are a few thoughts: > > 1. Please avoid another major rewrite. > 2. Stop pretending that instances of ma.MaskedArray and ndarray have "is > a" relationship. Use of inheritance should become an implementation detail > and any method that is not explicitly overridden should raise an exception. > 3. Add a mechanism to keep numpy and numpy.ma APIs in sync. At a minimum > - add a test comparing public functions and methods and for pure python > functions compare signatures. > 4. Consider deprecating the ma.masked scalar. > 5. Support duck-typing in MaskedArray constructors. If supplied data > object has mask attribute it should be used as mask. This will allow > interoperability with alternative missing values implementations. (ndarray > may itself grow mask attribute one day which will be equivalent to isnan. > Bit views, anyone?) > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- please visit: http://www.globalcarbonatlas.org/ *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Wed Jun 4 04:58:49 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Wed, 04 Jun 2014 10:58:49 +0200 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: Message-ID: <1401872329.18622.3.camel@sebastian-t440> On Mi, 2014-06-04 at 02:26 +0100, Nathaniel Smith wrote: > On Wed, Jun 4, 2014 at 12:33 AM, Charles R Harris > wrote: > > On Tue, Jun 3, 2014 at 5:08 PM, Kyle Mandli wrote: > >> > >> Hello everyone, > >> > >> As one of the co-chairs in charge of organizing the birds-of-a-feather > >> sesssions at the SciPy conference this year, I wanted to solicit through the > >> NumPy list to see if we could get enough interest to hold a NumPy centered > >> BoF this year. The BoF format would be up to those who would lead the > >> discussion, a couple of ideas used in the past include picking out a few of > >> the lead devs to be on a panel and have a Q&A type of session or an open Q&A > >> with perhaps audience guided list of topics. I can help facilitate > >> organization of something but we would really like to get something > >> organized this year (last year NumPy was the only major project that was not > >> really represented in the BoF sessions). > > > > I'll be at the conference, but I don't know who else will be there. I feel > > that NumPy has matured to the point where most of the current work is > > cleaning stuff up, making it run faster, and fixing bugs. A topic that I'd > > like to see discussed is where do we go from here. One option to look at is > > Blaze, which looks to have matured a lot in the last year. The problem with > > making it a NumPy replacement is that NumPy has become quite widespread, > > with downloads from PyPi running at about 3 million per year. With that much > > penetration it may be difficult for a new core like Blaze to gain traction. > > So I'd like to also discuss ways to bring the two branches of development > > together at some point and explore what NumPy can do to pave the way. Mind, > > there are definitely things that would be nice to add to NumPy, a better > > type system, missing values, etc., but doing that is difficult given the > > current design. > > I won't be at the conference unfortunately (I'm on the wrong continent > and have family commitments then anyway), but I think there's lots of > exciting stuff that can be done in numpy-land. > I wouldn't like to come, but to be honest have not planned to yet and it doesn't fit too well with the stuff I work on mostly right now. So will have to see. - Sebastian > We absolutely could rewrite the dtype system, and this would > straightforwardly give us excellent support for missing values, units, > categorical data, automatic differentiation, better datetimes, etc. > etc. -- and make numpy much more friendly in general to third-party > extensions. > > I'd like to see the ufunc system revisited in the light of all the > things we know now, to make gufuncs more first-class, provide better > support for user-defined types, more flexible loop selection (e.g. > make it possible to implement np.add.reduce(a, type="kahan")), etc.; > one goal would be to convert a lot of ufunc-like functions (np.mean > etc.) into being real ufuncs, and then they'd automatically benefit > from __numpy_ufunc__, which would also massively improve > interoperability with alternative array types like blaze. > > I'd like to see support for extensible label-based indexing, like pandas. > > Internally, I'd like to see internal migrating out of C and into > Cython -- we have hundreds of lines of code that could be replaced > with a few lines of Cython and no-one would notice. (Combining this > with a cffi cython backend and pypy would be pretty interesting > too...) > > I'd like to see sparse ndarrays, with integration into the ufunc > looping machinery so all ufuncs just work. Or even better, I'd like to > see the right hooks added so that anyone can write a sparse ndarray > package using only public APIs, and have all ufuncs just work. (I was > going to put down deferred/loop-fused/out-of-core computation as a > wishlist item too, but if we do it right then this too could be > implemented by anyone without needing to be baked into numpy proper.) > > All of these things would take some work and care, but I think they > could all be done incrementally and without breaking backwards > compatibility. Compare to ipython, which -- as Fernando likes to point > out :-) -- went from a little console program to its current > distributed-notebook-skynet-whatever-it-is by merging one working PR > at a time. Certainly these changes would much easier and less > disruptive than any plan that involves throwing out numpy and starting > over. But they also do help smooth the way for an incremental > transition to a world where numpy is regularly used alongside other > libraries. > > -n > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From mwojc at p.lodz.pl Wed Jun 4 05:30:21 2014 From: mwojc at p.lodz.pl (Marek Wojciechowski) Date: Wed, 04 Jun 2014 11:30:21 +0200 Subject: [Numpy-discussion] "multi meshgrid" Message-ID: <5944183.x2cGEYB0X3@think> Hi! Maybe there's someone who can answer the following question at stackoverflow: http://stackoverflow.com/q/24019099/1606022?sem=2 It is about extending meshgrid functionality to 2D input arrays... Regards, -- Marek From cournape at gmail.com Wed Jun 4 06:09:33 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 4 Jun 2014 11:09:33 +0100 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: <1401872329.18622.3.camel@sebastian-t440> References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: I won't be able to make it at scipy this year sadly. I concur with Nathaniel that we can do a lot of things without a full rewrite -- it is all too easy to see what is gained with a rewrite and lose sight of what is lost. I have yet to see a really strong argument for a full rewrite. It may be easier to do a rewrite for a core when you have a few full-time people, but that's a different story for a community effort like numpy. The main issue preventing new features in numpy is the lack of internal architecture at the C level, but nothing that could not be done by refactoring. Using cython to move away from the python C api would be great, though we need to talk with the cython people so that we can share common code between multiple extensions using cython, to avoid binary size explosion. There are things that may require some backward incompatible changes in the C API, but that's much more acceptable than a significant break at the python level. David On Wed, Jun 4, 2014 at 9:58 AM, Sebastian Berg wrote: > On Mi, 2014-06-04 at 02:26 +0100, Nathaniel Smith wrote: > > On Wed, Jun 4, 2014 at 12:33 AM, Charles R Harris > > wrote: > > > On Tue, Jun 3, 2014 at 5:08 PM, Kyle Mandli > wrote: > > >> > > >> Hello everyone, > > >> > > >> As one of the co-chairs in charge of organizing the birds-of-a-feather > > >> sesssions at the SciPy conference this year, I wanted to solicit > through the > > >> NumPy list to see if we could get enough interest to hold a NumPy > centered > > >> BoF this year. The BoF format would be up to those who would lead the > > >> discussion, a couple of ideas used in the past include picking out a > few of > > >> the lead devs to be on a panel and have a Q&A type of session or an > open Q&A > > >> with perhaps audience guided list of topics. I can help facilitate > > >> organization of something but we would really like to get something > > >> organized this year (last year NumPy was the only major project that > was not > > >> really represented in the BoF sessions). > > > > > > I'll be at the conference, but I don't know who else will be there. I > feel > > > that NumPy has matured to the point where most of the current work is > > > cleaning stuff up, making it run faster, and fixing bugs. A topic that > I'd > > > like to see discussed is where do we go from here. One option to look > at is > > > Blaze, which looks to have matured a lot in the last year. The problem > with > > > making it a NumPy replacement is that NumPy has become quite > widespread, > > > with downloads from PyPi running at about 3 million per year. With > that much > > > penetration it may be difficult for a new core like Blaze to gain > traction. > > > So I'd like to also discuss ways to bring the two branches of > development > > > together at some point and explore what NumPy can do to pave the way. > Mind, > > > there are definitely things that would be nice to add to NumPy, a > better > > > type system, missing values, etc., but doing that is difficult given > the > > > current design. > > > > I won't be at the conference unfortunately (I'm on the wrong continent > > and have family commitments then anyway), but I think there's lots of > > exciting stuff that can be done in numpy-land. > > > > I wouldn't like to come, but to be honest have not planned to yet and it > doesn't fit too well with the stuff I work on mostly right now. So will > have to see. > > - Sebastian > > > We absolutely could rewrite the dtype system, and this would > > straightforwardly give us excellent support for missing values, units, > > categorical data, automatic differentiation, better datetimes, etc. > > etc. -- and make numpy much more friendly in general to third-party > > extensions. > > > > I'd like to see the ufunc system revisited in the light of all the > > things we know now, to make gufuncs more first-class, provide better > > support for user-defined types, more flexible loop selection (e.g. > > make it possible to implement np.add.reduce(a, type="kahan")), etc.; > > one goal would be to convert a lot of ufunc-like functions (np.mean > > etc.) into being real ufuncs, and then they'd automatically benefit > > from __numpy_ufunc__, which would also massively improve > > interoperability with alternative array types like blaze. > > > > I'd like to see support for extensible label-based indexing, like pandas. > > > > Internally, I'd like to see internal migrating out of C and into > > Cython -- we have hundreds of lines of code that could be replaced > > with a few lines of Cython and no-one would notice. (Combining this > > with a cffi cython backend and pypy would be pretty interesting > > too...) > > > > I'd like to see sparse ndarrays, with integration into the ufunc > > looping machinery so all ufuncs just work. Or even better, I'd like to > > see the right hooks added so that anyone can write a sparse ndarray > > package using only public APIs, and have all ufuncs just work. (I was > > going to put down deferred/loop-fused/out-of-core computation as a > > wishlist item too, but if we do it right then this too could be > > implemented by anyone without needing to be baked into numpy proper.) > > > > All of these things would take some work and care, but I think they > > could all be done incrementally and without breaking backwards > > compatibility. Compare to ipython, which -- as Fernando likes to point > > out :-) -- went from a little console program to its current > > distributed-notebook-skynet-whatever-it-is by merging one working PR > > at a time. Certainly these changes would much easier and less > > disruptive than any plan that involves throwing out numpy and starting > > over. But they also do help smooth the way for an incremental > > transition to a world where numpy is regularly used alongside other > > libraries. > > > > -n > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.eberspaecher at gmail.com Wed Jun 4 17:34:24 2014 From: alex.eberspaecher at gmail.com (=?ISO-8859-15?Q?Alexander_Ebersp=E4cher?=) Date: Wed, 04 Jun 2014 23:34:24 +0200 Subject: [Numpy-discussion] fftw supported? In-Reply-To: References: Message-ID: <538F90E0.3080300@gmail.com> On 02.06.2014 14:26, David Cournapeau wrote: > FFTW is not used anymore in neither numpy or scipy (has not been for > several years). If you want to use fftw with numpy, there are 3rd party > extensions to do it, like pyfftw If you feel pyfftw bothers you with too many FFTW details, you may try something like https://github.com/aeberspaecher/transparent_pyfftw (be careful, it's a hack that has seen only little testing). Alex From njs at pobox.com Wed Jun 4 20:56:51 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Jun 2014 01:56:51 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) Message-ID: On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant wrote: > Even relatively simple changes can have significant impact at this point. > Nathaniel has laid out a fantastic list of great features. These are the > kind of features I have been eager to see as well. This is why I have been > working to fund and help explore these ideas in the Numba array object as > well as in Blaze. Gnumpy, Theano, Pandas, and other projects also have > useful tales to tell regarding a potential NumPy 2.0. I think this is somewhat missing the main point of my message :-). I was specifically laying out a list of features that we could start working on *right now*, *without* waiting for the mythical "numpy 2.0". > Ultimately, I do think it is time to talk seriously about NumPy 2.0, and > what it might look like. I personally think it looks a lot more like a > re-write, than a continuation of the modifications of Numeric that became > NumPy 1.0. Right out of the gate, for example, I would make sure that > NumPy 2.0 objects somehow used PyObject_VAR_HEAD so that they were > variable-sized objects where the strides and dimension information was > stored directly in the object structure itself instead of allocated > separately (thus requiring additional loads and stores from memory). This > would be a relatively simple change. But, it can't be done and preserve ABI > compatibility. It may also, at this point, have impact on Cython code, or > other code that is deeply-aware of the NumPy code-structure. Some of the > changes that should be made will ultimately require a porting exercise for > new code --- at which point why not just use a new project. I'm not aware of any obstacles to packing strides/dimension/data into the ndarray object right now, tomorrow if you like -- we've even discussed doing this recently in the tracker. PyObject_VAR_HEAD in particular seems... irrelevant? All it is is syntactic sugar for adding an integer field called "ob_size" to a Python object struct, plus a few macros for working with this field. We don't need or want such a field anyway (for shape/strides it would be redundant with ndim), and even if we did want such a field we could add it any time without breaking ABI. And if someday we do discover some compelling advantage to breaking ABI by rearranging the ndarray struct, then we can do this with a bit of planning by using #ifdef's to make the rearrangement coincide with a new Python release. E.g., people building against python 3.5 get the new struct layout, people building against 3.4 get the old, and in a few years we drop support for the old. No compatibility breaks needed, never mind rewrites. More generally: I wouldn't rule out "numpy 2.0" entirely, but we need to remember the immense costs that a rewrite-and-replace strategy will incur. Writing a new library is very expensive, so that's one cost. But that cost is nothing compared to the costs of getting that new library to the same level of maturity that numpy has already reached. And those costs, in turn, are absolutely dwarfed by the transition costs of moving the whole ecosystem from one foundation to a different, incompatible one. And probably even these costs are small compared to the opportunity costs -- all the progress that *doesn't* get made in the mean time because fragmented ecosystems suck and make writing code hard, and the best hackers are busy porting code instead of writing awesome new stuff. I'm sure dynd is great, but we have to be realistic: the hard truth is that even if it's production-ready today, that only brings us a fraction of a fraction of a percent closer to making it a real replacement for numpy. Consider the python 2 to python 3 transition: Python 3 itself was an immense amount of work for a large number of people, with intense community scrutiny of the design. It came out in 2008. 6 years and many many improvements later, it's maybe sort-of starting to look like a plurality of users might start transitioning soonish? It'll be years yet before portable libraries can start taking advantage of python 3's new awesomeness. And in the mean time, the progress of the whole Python ecosystem has been seriously disrupted: think of how much awesome stuff we'd have if all the time that's been spent porting and testing different packages had been spent on moving them forward instead. We also have experience closer to home -- did anyone enjoy the numeric/numarray->numpy transition so much they want to do it again? And numpy will be much harder to replace than numeric -- numeric wasn't the most-imported package in the pythonverse ;-). And my biggest worry is that if anyone even tries to convince everyone to make this kind of transition, then if they're successful at all then they'll create a substantial period where the ecosystem is a big incompatible mess (and they might still eventually fail, providing no long-term benefit to make up for the immediate costs). This scenario is a nightmare for end-users all around. By comparison, if we improve numpy incrementally, then we can in most cases preserve compatibility totally, and in the rare cases where it's necessary to break something we can do it mindfully, minimally, and with a managed transition. (Downstream packages are already used to handling a few limited API changes at a time, it's not that hard to support both APIs during the transition period, etc., so this way we bring the ecosystem with us.) Every incremental improvement to numpy immediately benefits its immense user base, and gets feedback and testing from that immense user base. And if we incrementally improve interoperability between numpy and other libraries like dynd, then instead of creating fragmentation, it will let downstream packages use both in a complementary way, switching back and forth depending on which provides more utility on a case-by-case basis. If this means that numpy eventually withers away because users vote with their feet, then great, that'd be compelling evidence that whatever they were migrating to really is better, which I trust a lot more than any guesses we make on a mailing list. The gradual approach does require that we be grown-ups and hold our noses while refactoring out legacy spaghetti and writing unaesthetic compatibility hacks. But if you compare this to the alternative... the benefits of incrementalism are, IMO, overwhelming. The only exception is when two specific criteria are met: (1) there are changes that are absolutely necessary for the ecosystem's long term health (e.g., py3's unicode-for-mere-mortals and true division), AND (2) it's absolutely impossible to make these changes incrementally (unicode and true division first entered Python in 2000 and 2001, respectively, and immense effort went into finding the smoothest transition, so it's pretty clear that as painful as py3 has been, there isn't really anything better.). What features could meet these two criteria in numpy's case? If I were the numpy ecosystem and you tried to convince me to suffer through a big-bang transition for the sake of PyObject_VAR_HEAD then I think I'd be kinda unconvinced. And it only took me a few minutes to rattle off a whole list of incremental changes that haven't even been tried yet. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From travis at continuum.io Wed Jun 4 21:29:13 2014 From: travis at continuum.io (Travis Oliphant) Date: Wed, 4 Jun 2014 20:29:13 -0500 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: Believe me, I'm all for incremental changes if it is actually possible and doesn't actually cost more. It's also why I've been silent until now about anything we are doing being a candidate for a NumPy 2.0. I understand the challenges of getting people to change. But, features and solid improvements *will* get people to change --- especially if their new library can be used along with the old library and the transition can be done gradually. Python 3's struggle is the lack of features. At some point there *will* be a NumPy 2.0. What features go into NumPy 2.0, how much backward compatibility is provided, and how much porting is needed to move your code from NumPy 1.X to NumPy 2.X is the real user question --- not whether it is characterized as "incremental" change or "re-write". What I call a re-write and what you call an "incremental-change" are two points on a spectrum and likely overlap signficantly if we really compared what we are thinking about. One huge benefit that came out of the numeric / numarray / numpy transition that we mustn't forget about was actually the extended buffer protocol and memory view objects. This really does allow multiple array objects to co-exist and libraries to use the object that they prefer in a way that did not exist when Numarray / numeric / numpy came out. So, we shouldn't be afraid of that world. The existence of easy package managers to update environments to try out new features and have applications on a single system that use multiple versions of the same library is also something that didn't exist before and that will make any transition easier for users. One thing I regret about my working on NumPy originally is that I didn't have the foresight, skill, and understanding to work more on a more extended and better designed multiple-dispatch system so that multiple array objects could participate together in an expression flow. The __numpy_ufunc__ mechanism gives enough capability in that direction that it may be better now. Ultimately, I don't disagree that NumPy can continue to exist in "incremental" change mode ( though if you are swapping out whole swaths of C-code for Cython code --- it sounds a lot like a "re-write") as long as there are people willing to put the effort into changing it. I think this is actually benefited by the existence of other array objects that are pushing the feature envelope without the constraints --- in much the same way that the Python standard library is benefitted by many versions of different capabilities being tried out before moving into the standard library. I remain optimistic that things will continue to improve in multiple ways --- if a little "messier" than any of us would conceive individually. It *is* great to see all the PR's coming from multiple people on NumPy and all the new energy around improving things whether great or small. Best, -Travis ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Jun 4 22:36:39 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 4 Jun 2014 20:36:39 -0600 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant wrote: > Believe me, I'm all for incremental changes if it is actually possible and > doesn't actually cost more. It's also why I've been silent until now about > anything we are doing being a candidate for a NumPy 2.0. I understand the > challenges of getting people to change. But, features and solid > improvements *will* get people to change --- especially if their new > library can be used along with the old library and the transition can be > done gradually. Python 3's struggle is the lack of features. > > At some point there *will* be a NumPy 2.0. What features go into NumPy > 2.0, how much backward compatibility is provided, and how much porting is > needed to move your code from NumPy 1.X to NumPy 2.X is the real user > question --- not whether it is characterized as "incremental" change or > "re-write". What I call a re-write and what you call an > "incremental-change" are two points on a spectrum and likely overlap > signficantly if we really compared what we are thinking about. > > One huge benefit that came out of the numeric / numarray / numpy > transition that we mustn't forget about was actually the extended buffer > protocol and memory view objects. This really does allow multiple array > objects to co-exist and libraries to use the object that they prefer in a > way that did not exist when Numarray / numeric / numpy came out. So, we > shouldn't be afraid of that world. The existence of easy package managers > to update environments to try out new features and have applications on a > single system that use multiple versions of the same library is also > something that didn't exist before and that will make any transition easier > for users. > > One thing I regret about my working on NumPy originally is that I didn't > have the foresight, skill, and understanding to work more on a more > extended and better designed multiple-dispatch system so that multiple > array objects could participate together in an expression flow. The > __numpy_ufunc__ mechanism gives enough capability in that direction that it > may be better now. > > Ultimately, I don't disagree that NumPy can continue to exist in > "incremental" change mode ( though if you are swapping out whole swaths of > C-code for Cython code --- it sounds a lot like a "re-write") as long as > there are people willing to put the effort into changing it. I think this > is actually benefited by the existence of other array objects that are > pushing the feature envelope without the constraints --- in much the same > way that the Python standard library is benefitted by many versions of > different capabilities being tried out before moving into the standard > library. > > I remain optimistic that things will continue to improve in multiple ways > --- if a little "messier" than any of us would conceive individually. It > *is* great to see all the PR's coming from multiple people on NumPy and all > the new energy around improving things whether great or small. > @nathaniel IIRC, one of the objections to the missing values work was that it changed the underlying array object by adding a couple of variables to the structure. I'm willing to do that sort of thing, but it would be good to have general agreement that that is acceptable. As to blaze/dynd, I'd like to steal bits here and there, and maybe in the long term base numpy on top of it with a compatibility layer. There is a lot of thought and effort that has gone into those projects and we should use what we can. As is, I think numpy is good for another five to ten years and will probably hang on for fifteen, but it will be outdated by the end of that period. Like great whites, we need to keep swimming just to have oxygen. Software projects tend to be obligate ram ventilators. The Python 3 experience is definitely something we want to avoid. And while blaze does big data and offers some nice features, I don't know that it offers compelling reasons to upgrade to the more ordinary user at this time, so I'd like to sort of slip it into numpy if possible. If we do start moving numpy forward in more radical steps, we should try to have some agreement beforehand as to what sort of changes are acceptable. For instance, to maintain backward compatibility, is it sufficient that a recompile will do the job, or do we require forward compatibility for extensions compiled against earlier releases? Do we stay with C or should we support C++ code with its advantages of smart pointers, exception handling, and templates? We will need a certain amount of flexibility going forward and we should decide, or at least discuss, such issues up front. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Jun 5 04:44:16 2014 From: toddrjen at gmail.com (Todd) Date: Thu, 5 Jun 2014 10:44:16 +0200 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On 5 Jun 2014 02:57, "Nathaniel Smith" wrote: > > On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant wrote: > And numpy will be much harder to replace than numeric -- > numeric wasn't the most-imported package in the pythonverse ;-). If numpy is really such a core part of python ecosystem, does it really make sense to keep it as a stand-alone package? Rather than thinking about a numpy 2, might it be better to be focusing on getting ndarray and dtype to a level of quality where acceptance upstream might be plausible? Matlab and python are no longer the only games in town for scientific computing anymore. I worry that the lack of a multidimensional array literals, not to mention the lack of built-in multidimensional arrays at all, can only hurt python's attractiveness compared to languages like Julia long-term. For people who already know and love python, this doesn't bother us much if at all. But thinking of attracting new users long-term, I worry that it will be harder to convince outsiders that python is really a first-class scientific computing language when it lacks the key data type for scientific computing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidmenhur at gmail.com Thu Jun 5 05:13:08 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Thu, 5 Jun 2014 11:13:08 +0200 Subject: [Numpy-discussion] fftw supported? In-Reply-To: <538F90E0.3080300@gmail.com> References: <538F90E0.3080300@gmail.com> Message-ID: On 4 June 2014 23:34, Alexander Ebersp?cher wrote: > If you feel pyfftw bothers you with too many FFTW details, you may try > something like https://github.com/aeberspaecher/transparent_pyfftw > (be careful, it's a hack that has seen only little testing). > pyFFTW provides a drop-in replacement for Numpy and Scipy's fftw: https://hgomersall.github.io/pyFFTW/pyfftw/interfaces/interfaces.html You can still set the number of threads and other advanced parameters, but you can just ignore them and view it as a very simple library. Does your wrapper set these to reasonable values? If not, I am missing completely the point. I am starting to use pyFFTW, and maybe I can help you test tfftw. /David. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Jun 5 08:28:14 2014 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Jun 2014 13:28:14 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 9:44 AM, Todd wrote: > > On 5 Jun 2014 02:57, "Nathaniel Smith" wrote: > > > > On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant > wrote: > > And numpy will be much harder to replace than numeric -- > > numeric wasn't the most-imported package in the pythonverse ;-). > > If numpy is really such a core part of python ecosystem, does it really > make sense to keep it as a stand-alone package? Rather than thinking about > a numpy 2, might it be better to be focusing on getting ndarray and dtype > to a level of quality where acceptance upstream might be plausible? > There has been discussions about integrating numpy a long time ago (can't find a reference right now), and the consensus was that this was possible in its current shape nor advisable. The situation has not changed. Putting something in the stdlib means it basically cannot change anymore: API compatibility requirements would be stronger than what we provide even now. NumPy is also a large codebase which would need some major clean up to be accepted, etc... David -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Jun 5 08:40:02 2014 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Jun 2014 13:40:02 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris wrote: > > > > On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant > wrote: > >> Believe me, I'm all for incremental changes if it is actually possible >> and doesn't actually cost more. It's also why I've been silent until now >> about anything we are doing being a candidate for a NumPy 2.0. I >> understand the challenges of getting people to change. But, features and >> solid improvements *will* get people to change --- especially if their new >> library can be used along with the old library and the transition can be >> done gradually. Python 3's struggle is the lack of features. >> >> At some point there *will* be a NumPy 2.0. What features go into NumPy >> 2.0, how much backward compatibility is provided, and how much porting is >> needed to move your code from NumPy 1.X to NumPy 2.X is the real user >> question --- not whether it is characterized as "incremental" change or >> "re-write". What I call a re-write and what you call an >> "incremental-change" are two points on a spectrum and likely overlap >> signficantly if we really compared what we are thinking about. >> >> One huge benefit that came out of the numeric / numarray / numpy >> transition that we mustn't forget about was actually the extended buffer >> protocol and memory view objects. This really does allow multiple array >> objects to co-exist and libraries to use the object that they prefer in a >> way that did not exist when Numarray / numeric / numpy came out. So, we >> shouldn't be afraid of that world. The existence of easy package managers >> to update environments to try out new features and have applications on a >> single system that use multiple versions of the same library is also >> something that didn't exist before and that will make any transition easier >> for users. >> >> One thing I regret about my working on NumPy originally is that I didn't >> have the foresight, skill, and understanding to work more on a more >> extended and better designed multiple-dispatch system so that multiple >> array objects could participate together in an expression flow. The >> __numpy_ufunc__ mechanism gives enough capability in that direction that it >> may be better now. >> >> Ultimately, I don't disagree that NumPy can continue to exist in >> "incremental" change mode ( though if you are swapping out whole swaths of >> C-code for Cython code --- it sounds a lot like a "re-write") as long as >> there are people willing to put the effort into changing it. I think this >> is actually benefited by the existence of other array objects that are >> pushing the feature envelope without the constraints --- in much the same >> way that the Python standard library is benefitted by many versions of >> different capabilities being tried out before moving into the standard >> library. >> >> I remain optimistic that things will continue to improve in multiple ways >> --- if a little "messier" than any of us would conceive individually. It >> *is* great to see all the PR's coming from multiple people on NumPy and all >> the new energy around improving things whether great or small. >> > > @nathaniel IIRC, one of the objections to the missing values work was that > it changed the underlying array object by adding a couple of variables to > the structure. I'm willing to do that sort of thing, but it would be good > to have general agreement that that is acceptable. > I think changing the ABI for some versions of numpy (2.0 , whatever) is acceptable. There is little doubt that the ABI will need to change to accommodate a better and more flexible architecture. Changing the C API is more tricky: I am not up to date to the changes from the last 2-3 years, but at that time, most things could have been changed internally without breaking much, though I did not go far enough to estimate what the performance impact could be (if any). > As to blaze/dynd, I'd like to steal bits here and there, and maybe in the > long term base numpy on top of it with a compatibility layer. There is a > lot of thought and effort that has gone into those projects and we should > use what we can. As is, I think numpy is good for another five to ten years > and will probably hang on for fifteen, but it will be outdated by the end > of that period. Like great whites, we need to keep swimming just to have > oxygen. Software projects tend to be obligate ram ventilators. > > The Python 3 experience is definitely something we want to avoid. And > while blaze does big data and offers some nice features, I don't know that > it offers compelling reasons to upgrade to the more ordinary user at this > time, so I'd like to sort of slip it into numpy if possible. > > If we do start moving numpy forward in more radical steps, we should try > to have some agreement beforehand as to what sort of changes are > acceptable. For instance, to maintain backward compatibility, is it > sufficient that a recompile will do the job, or do we require forward > compatibility for extensions compiled against earlier releases? Do we stay > with C or should we support C++ code with its advantages of smart pointers, > exception handling, and templates? We will need a certain amount of > flexibility going forward and we should decide, or at least discuss, such > issues up front. > Last time the C++ discussion was brought up, no consensus could be made. I think quite a few radical changes can be made without that consensus already, though other may disagree there. IMO, what is needed the most is refactoring the internal to extract the Python C API low level from the rest of the code, as I think that's the main bottleneck to get more contributors (or get new core features more quickly). David -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Jun 5 08:58:04 2014 From: toddrjen at gmail.com (Todd) Date: Thu, 5 Jun 2014 14:58:04 +0200 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On 5 Jun 2014 14:28, "David Cournapeau" wrote: > > > > > On Thu, Jun 5, 2014 at 9:44 AM, Todd wrote: >> >> >> On 5 Jun 2014 02:57, "Nathaniel Smith" wrote: >> > >> > On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant wrote: >> > And numpy will be much harder to replace than numeric -- >> > numeric wasn't the most-imported package in the pythonverse ;-). >> >> If numpy is really such a core part of python ecosystem, does it really make sense to keep it as a stand-alone package? Rather than thinking about a numpy 2, might it be better to be focusing on getting ndarray and dtype to a level of quality where acceptance upstream might be plausible? > > > There has been discussions about integrating numpy a long time ago (can't find a reference right now), and the consensus was that this was possible in its current shape nor advisable. The situation has not changed. > > Putting something in the stdlib means it basically cannot change anymore: API compatibility requirements would be stronger than what we provide even now. NumPy is also a large codebase which would need some major clean up to be accepted, etc... > > David I am not suggesting merging all of numpy, only ndarray and dtype (which I know is a huge job itself). And perhaps not even all of what us currently included in those, some methods could be split out to their own functions. And any numpy 2.0 would also imply a major code cleanup. So although ndarray and dtype are certainly not ready for such a thing right now, if you are talking about numpy 2.0 already, perhaps part of that discussion could involve a plan to get the code into a state where such a move might be plausible. Even if the merge doesn't actually happen, having the code at that quality level would still be a good thing. I agree that the relationship between numpy and python has not changed very much in the last few years, but I think the scientific computing landscape is changing. The latter issue is where my primary concern lies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Jun 5 09:00:34 2014 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 5 Jun 2014 14:00:34 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 1:58 PM, Todd wrote: > > On 5 Jun 2014 14:28, "David Cournapeau" wrote: >> >> There has been discussions about integrating numpy a long time ago (can't >> find a reference right now), and the consensus was that this was possible in >> its current shape nor advisable. The situation has not changed. > > I am not suggesting merging all of numpy, only ndarray and dtype (which I > know is a huge job itself). And perhaps not even all of what us currently > included in those, some methods could be split out to their own functions. That is what was discussed and rejected in favor of putting the enhanced buffer protocol into the language. -- Robert Kern From josef.pktd at gmail.com Thu Jun 5 09:07:36 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Jun 2014 09:07:36 -0400 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 8:58 AM, Todd wrote: > > On 5 Jun 2014 14:28, "David Cournapeau" wrote: > > > > > > > > > > On Thu, Jun 5, 2014 at 9:44 AM, Todd wrote: > >> > >> > >> On 5 Jun 2014 02:57, "Nathaniel Smith" wrote: > >> > > >> > On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant > wrote: > >> > And numpy will be much harder to replace than numeric -- > >> > numeric wasn't the most-imported package in the pythonverse ;-). > >> > >> If numpy is really such a core part of python ecosystem, does it > really make sense to keep it as a stand-alone package? Rather than > thinking about a numpy 2, might it be better to be focusing on getting > ndarray and dtype to a level of quality where acceptance upstream might be > plausible? > > > > > > There has been discussions about integrating numpy a long time ago > (can't find a reference right now), and the consensus was that this was > possible in its current shape nor advisable. The situation has not changed. > > > > Putting something in the stdlib means it basically cannot change > anymore: API compatibility requirements would be stronger than what we > provide even now. NumPy is also a large codebase which would need some > major clean up to be accepted, etc... > > > > David > > I am not suggesting merging all of numpy, only ndarray and dtype (which I > know is a huge job itself). And perhaps not even all of what us currently > included in those, some methods could be split out to their own functions. > > And any numpy 2.0 would also imply a major code cleanup. So although > ndarray and dtype are certainly not ready for such a thing right now, if > you are talking about numpy 2.0 already, perhaps part of that discussion > could involve a plan to get the code into a state where such a move might > be plausible. Even if the merge doesn't actually happen, having the code > at that quality level would still be a good thing. > > I agree that the relationship between numpy and python has not changed > very much in the last few years, but I think the scientific computing > landscape is changing. The latter issue is where my primary concern lies. > I don't think it would have any effect on scientific computing users. It might be useful for other users that occasionally want to do a bit of array processing. Scientific users need the extended SciPy stack and not a part of numpy that can be imported from the standard library. For example in "Data Science", where I pay more attention and where Python is getting pretty popular, the usual recommended list requires numpy scipy and 5 to 10 more python libraries. Should pandas also go into the python standard library? Python 3.4 got a statistics library, but I don't think it has any effect on the potential statsmodels user base. Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Jun 5 09:40:03 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 5 Jun 2014 09:40:03 -0400 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 8:40 AM, David Cournapeau wrote: > > > > On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant >> wrote: >> >>> Believe me, I'm all for incremental changes if it is actually possible >>> and doesn't actually cost more. It's also why I've been silent until now >>> about anything we are doing being a candidate for a NumPy 2.0. I >>> understand the challenges of getting people to change. But, features and >>> solid improvements *will* get people to change --- especially if their new >>> library can be used along with the old library and the transition can be >>> done gradually. Python 3's struggle is the lack of features. >>> >>> At some point there *will* be a NumPy 2.0. What features go into NumPy >>> 2.0, how much backward compatibility is provided, and how much porting is >>> needed to move your code from NumPy 1.X to NumPy 2.X is the real user >>> question --- not whether it is characterized as "incremental" change or >>> "re-write". What I call a re-write and what you call an >>> "incremental-change" are two points on a spectrum and likely overlap >>> signficantly if we really compared what we are thinking about. >>> >>> One huge benefit that came out of the numeric / numarray / numpy >>> transition that we mustn't forget about was actually the extended buffer >>> protocol and memory view objects. This really does allow multiple array >>> objects to co-exist and libraries to use the object that they prefer in a >>> way that did not exist when Numarray / numeric / numpy came out. So, we >>> shouldn't be afraid of that world. The existence of easy package managers >>> to update environments to try out new features and have applications on a >>> single system that use multiple versions of the same library is also >>> something that didn't exist before and that will make any transition easier >>> for users. >>> >>> One thing I regret about my working on NumPy originally is that I didn't >>> have the foresight, skill, and understanding to work more on a more >>> extended and better designed multiple-dispatch system so that multiple >>> array objects could participate together in an expression flow. The >>> __numpy_ufunc__ mechanism gives enough capability in that direction that it >>> may be better now. >>> >>> Ultimately, I don't disagree that NumPy can continue to exist in >>> "incremental" change mode ( though if you are swapping out whole swaths of >>> C-code for Cython code --- it sounds a lot like a "re-write") as long as >>> there are people willing to put the effort into changing it. I think this >>> is actually benefited by the existence of other array objects that are >>> pushing the feature envelope without the constraints --- in much the same >>> way that the Python standard library is benefitted by many versions of >>> different capabilities being tried out before moving into the standard >>> library. >>> >>> I remain optimistic that things will continue to improve in multiple >>> ways --- if a little "messier" than any of us would conceive individually. >>> It *is* great to see all the PR's coming from multiple people on NumPy >>> and all the new energy around improving things whether great or small. >>> >> >> @nathaniel IIRC, one of the objections to the missing values work was >> that it changed the underlying array object by adding a couple of variables >> to the structure. I'm willing to do that sort of thing, but it would be >> good to have general agreement that that is acceptable. >> > > > I think changing the ABI for some versions of numpy (2.0 , whatever) is > acceptable. There is little doubt that the ABI will need to change to > accommodate a better and more flexible architecture. > > Changing the C API is more tricky: I am not up to date to the changes from > the last 2-3 years, but at that time, most things could have been changed > internally without breaking much, though I did not go far enough to > estimate what the performance impact could be (if any). > My impression is that you can do it once (in a while) so that no more than two incompatible versions of numpy are alive at the same time. It doesn't look worse to me than supporting a new python version, but doubles the number of binaries and wheels. (Supporting python 3.4 for cython based projects was hoping or helping that cython takes care of it. And cython developers took care of it. ) Josef > > > >> As to blaze/dynd, I'd like to steal bits here and there, and maybe in the >> long term base numpy on top of it with a compatibility layer. There is a >> lot of thought and effort that has gone into those projects and we should >> use what we can. As is, I think numpy is good for another five to ten years >> and will probably hang on for fifteen, but it will be outdated by the end >> of that period. Like great whites, we need to keep swimming just to have >> oxygen. Software projects tend to be obligate ram ventilators. >> >> The Python 3 experience is definitely something we want to avoid. And >> while blaze does big data and offers some nice features, I don't know that >> it offers compelling reasons to upgrade to the more ordinary user at this >> time, so I'd like to sort of slip it into numpy if possible. >> >> If we do start moving numpy forward in more radical steps, we should try >> to have some agreement beforehand as to what sort of changes are >> acceptable. For instance, to maintain backward compatibility, is it >> sufficient that a recompile will do the job, or do we require forward >> compatibility for extensions compiled against earlier releases? Do we stay >> with C or should we support C++ code with its advantages of smart pointers, >> exception handling, and templates? We will need a certain amount of >> flexibility going forward and we should decide, or at least discuss, such >> issues up front. >> > > Last time the C++ discussion was brought up, no consensus could be made. I > think quite a few radical changes can be made without that consensus > already, though other may disagree there. > > IMO, what is needed the most is refactoring the internal to extract the > Python C API low level from the rest of the code, as I think that's the > main bottleneck to get more contributors (or get new core features more > quickly). > > David > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Jun 5 09:51:56 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 5 Jun 2014 07:51:56 -0600 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 6:40 AM, David Cournapeau wrote: > > > > On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant >> wrote: >> >>> Believe me, I'm all for incremental changes if it is actually possible >>> and doesn't actually cost more. It's also why I've been silent until now >>> about anything we are doing being a candidate for a NumPy 2.0. I >>> understand the challenges of getting people to change. But, features and >>> solid improvements *will* get people to change --- especially if their new >>> library can be used along with the old library and the transition can be >>> done gradually. Python 3's struggle is the lack of features. >>> >>> At some point there *will* be a NumPy 2.0. What features go into NumPy >>> 2.0, how much backward compatibility is provided, and how much porting is >>> needed to move your code from NumPy 1.X to NumPy 2.X is the real user >>> question --- not whether it is characterized as "incremental" change or >>> "re-write". What I call a re-write and what you call an >>> "incremental-change" are two points on a spectrum and likely overlap >>> signficantly if we really compared what we are thinking about. >>> >>> One huge benefit that came out of the numeric / numarray / numpy >>> transition that we mustn't forget about was actually the extended buffer >>> protocol and memory view objects. This really does allow multiple array >>> objects to co-exist and libraries to use the object that they prefer in a >>> way that did not exist when Numarray / numeric / numpy came out. So, we >>> shouldn't be afraid of that world. The existence of easy package managers >>> to update environments to try out new features and have applications on a >>> single system that use multiple versions of the same library is also >>> something that didn't exist before and that will make any transition easier >>> for users. >>> >>> One thing I regret about my working on NumPy originally is that I didn't >>> have the foresight, skill, and understanding to work more on a more >>> extended and better designed multiple-dispatch system so that multiple >>> array objects could participate together in an expression flow. The >>> __numpy_ufunc__ mechanism gives enough capability in that direction that it >>> may be better now. >>> >>> Ultimately, I don't disagree that NumPy can continue to exist in >>> "incremental" change mode ( though if you are swapping out whole swaths of >>> C-code for Cython code --- it sounds a lot like a "re-write") as long as >>> there are people willing to put the effort into changing it. I think this >>> is actually benefited by the existence of other array objects that are >>> pushing the feature envelope without the constraints --- in much the same >>> way that the Python standard library is benefitted by many versions of >>> different capabilities being tried out before moving into the standard >>> library. >>> >>> I remain optimistic that things will continue to improve in multiple >>> ways --- if a little "messier" than any of us would conceive individually. >>> It *is* great to see all the PR's coming from multiple people on NumPy >>> and all the new energy around improving things whether great or small. >>> >> >> @nathaniel IIRC, one of the objections to the missing values work was >> that it changed the underlying array object by adding a couple of variables >> to the structure. I'm willing to do that sort of thing, but it would be >> good to have general agreement that that is acceptable. >> > > > I think changing the ABI for some versions of numpy (2.0 , whatever) is > acceptable. There is little doubt that the ABI will need to change to > accommodate a better and more flexible architecture. > > Changing the C API is more tricky: I am not up to date to the changes from > the last 2-3 years, but at that time, most things could have been changed > internally without breaking much, though I did not go far enough to > estimate what the performance impact could be (if any). > > > >> As to blaze/dynd, I'd like to steal bits here and there, and maybe in the >> long term base numpy on top of it with a compatibility layer. There is a >> lot of thought and effort that has gone into those projects and we should >> use what we can. As is, I think numpy is good for another five to ten years >> and will probably hang on for fifteen, but it will be outdated by the end >> of that period. Like great whites, we need to keep swimming just to have >> oxygen. Software projects tend to be obligate ram ventilators. >> >> The Python 3 experience is definitely something we want to avoid. And >> while blaze does big data and offers some nice features, I don't know that >> it offers compelling reasons to upgrade to the more ordinary user at this >> time, so I'd like to sort of slip it into numpy if possible. >> >> If we do start moving numpy forward in more radical steps, we should try >> to have some agreement beforehand as to what sort of changes are >> acceptable. For instance, to maintain backward compatibility, is it >> sufficient that a recompile will do the job, or do we require forward >> compatibility for extensions compiled against earlier releases? Do we stay >> with C or should we support C++ code with its advantages of smart pointers, >> exception handling, and templates? We will need a certain amount of >> flexibility going forward and we should decide, or at least discuss, such >> issues up front. >> > > Last time the C++ discussion was brought up, no consensus could be made. I > think quite a few radical changes can be made without that consensus > already, though other may disagree there. > > IMO, what is needed the most is refactoring the internal to extract the > Python C API low level from the rest of the code, as I think that's the > main bottleneck to get more contributors (or get new core features more > quickly). > > What do you mean by "extract the Python C API"? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jun 5 10:04:42 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Jun 2014 15:04:42 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 2:29 AM, Travis Oliphant wrote: > At some point there *will* be a NumPy 2.0. What features go into NumPy > 2.0, how much backward compatibility is provided, and how much porting is > needed to move your code from NumPy 1.X to NumPy 2.X is the real user > question --- not whether it is characterized as "incremental" change or > "re-write". There may or may not ever be a numpy 2.0. Maybe there will be a numpy 1.20 instead. Obviously there will be changes, and I think we generally agree on the end goal, but the question is how we get from here to there. > What I call a re-write and what you call an > "incremental-change" are two points on a spectrum and likely overlap > signficantly if we really compared what we are thinking about. [...] > Ultimately, I don't disagree that NumPy can continue to exist in > "incremental" change mode ( though if you are swapping out whole swaths of > C-code for Cython code --- it sounds a lot like a "re-write") as long as > there are people willing to put the effort into changing it. This is why I'm trying to emphasize the a contrast between big-bang versus incremental, rather than rewrite-versus-not-rewrite. If Theseus goes through replacing every timber in his ship, and does it one at a time, then the boat still floats. If he tries to do it all at once, then the end goal may be the same but the actual results are rather different. And perception matters. If we set out to design "numpy 2.0" then that conversation will go one way. If we set out to design "numpy 1.20", then the conversation will be different. I want to convince people that the numpy 1.20 approach is a worthwhile place to put our efforts. > I think this > is actually benefited by the existence of other array objects that are > pushing the feature envelope without the constraints --- in much the same > way that the Python standard library is benefitted by many versions of > different capabilities being tried out before moving into the standard > library. Indeed! -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From cournape at gmail.com Thu Jun 5 10:24:31 2014 From: cournape at gmail.com (David Cournapeau) Date: Thu, 5 Jun 2014 15:24:31 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 2:51 PM, Charles R Harris wrote: > > > > On Thu, Jun 5, 2014 at 6:40 AM, David Cournapeau > wrote: > >> >> >> >> On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> >>> On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant >>> wrote: >>> >>>> Believe me, I'm all for incremental changes if it is actually possible >>>> and doesn't actually cost more. It's also why I've been silent until now >>>> about anything we are doing being a candidate for a NumPy 2.0. I >>>> understand the challenges of getting people to change. But, features and >>>> solid improvements *will* get people to change --- especially if their new >>>> library can be used along with the old library and the transition can be >>>> done gradually. Python 3's struggle is the lack of features. >>>> >>>> At some point there *will* be a NumPy 2.0. What features go into >>>> NumPy 2.0, how much backward compatibility is provided, and how much >>>> porting is needed to move your code from NumPy 1.X to NumPy 2.X is the real >>>> user question --- not whether it is characterized as "incremental" change >>>> or "re-write". What I call a re-write and what you call an >>>> "incremental-change" are two points on a spectrum and likely overlap >>>> signficantly if we really compared what we are thinking about. >>>> >>>> One huge benefit that came out of the numeric / numarray / numpy >>>> transition that we mustn't forget about was actually the extended buffer >>>> protocol and memory view objects. This really does allow multiple array >>>> objects to co-exist and libraries to use the object that they prefer in a >>>> way that did not exist when Numarray / numeric / numpy came out. So, we >>>> shouldn't be afraid of that world. The existence of easy package managers >>>> to update environments to try out new features and have applications on a >>>> single system that use multiple versions of the same library is also >>>> something that didn't exist before and that will make any transition easier >>>> for users. >>>> >>>> One thing I regret about my working on NumPy originally is that I >>>> didn't have the foresight, skill, and understanding to work more on a more >>>> extended and better designed multiple-dispatch system so that multiple >>>> array objects could participate together in an expression flow. The >>>> __numpy_ufunc__ mechanism gives enough capability in that direction that it >>>> may be better now. >>>> >>>> Ultimately, I don't disagree that NumPy can continue to exist in >>>> "incremental" change mode ( though if you are swapping out whole swaths of >>>> C-code for Cython code --- it sounds a lot like a "re-write") as long as >>>> there are people willing to put the effort into changing it. I think this >>>> is actually benefited by the existence of other array objects that are >>>> pushing the feature envelope without the constraints --- in much the same >>>> way that the Python standard library is benefitted by many versions of >>>> different capabilities being tried out before moving into the standard >>>> library. >>>> >>>> I remain optimistic that things will continue to improve in multiple >>>> ways --- if a little "messier" than any of us would conceive individually. >>>> It *is* great to see all the PR's coming from multiple people on NumPy >>>> and all the new energy around improving things whether great or small. >>>> >>> >>> @nathaniel IIRC, one of the objections to the missing values work was >>> that it changed the underlying array object by adding a couple of variables >>> to the structure. I'm willing to do that sort of thing, but it would be >>> good to have general agreement that that is acceptable. >>> >> >> >> I think changing the ABI for some versions of numpy (2.0 , whatever) is >> acceptable. There is little doubt that the ABI will need to change to >> accommodate a better and more flexible architecture. >> >> Changing the C API is more tricky: I am not up to date to the changes >> from the last 2-3 years, but at that time, most things could have been >> changed internally without breaking much, though I did not go far enough to >> estimate what the performance impact could be (if any). >> >> >> >>> As to blaze/dynd, I'd like to steal bits here and there, and maybe in >>> the long term base numpy on top of it with a compatibility layer. There is >>> a lot of thought and effort that has gone into those projects and we should >>> use what we can. As is, I think numpy is good for another five to ten years >>> and will probably hang on for fifteen, but it will be outdated by the end >>> of that period. Like great whites, we need to keep swimming just to have >>> oxygen. Software projects tend to be obligate ram ventilators. >>> >>> The Python 3 experience is definitely something we want to avoid. And >>> while blaze does big data and offers some nice features, I don't know that >>> it offers compelling reasons to upgrade to the more ordinary user at this >>> time, so I'd like to sort of slip it into numpy if possible. >>> >>> If we do start moving numpy forward in more radical steps, we should try >>> to have some agreement beforehand as to what sort of changes are >>> acceptable. For instance, to maintain backward compatibility, is it >>> sufficient that a recompile will do the job, or do we require forward >>> compatibility for extensions compiled against earlier releases? Do we stay >>> with C or should we support C++ code with its advantages of smart pointers, >>> exception handling, and templates? We will need a certain amount of >>> flexibility going forward and we should decide, or at least discuss, such >>> issues up front. >>> >> >> Last time the C++ discussion was brought up, no consensus could be made. >> I think quite a few radical changes can be made without that consensus >> already, though other may disagree there. >> >> IMO, what is needed the most is refactoring the internal to extract the >> Python C API low level from the rest of the code, as I think that's the >> main bottleneck to get more contributors (or get new core features more >> quickly). >> >> > What do you mean by "extract the Python C API"? > Poor choice of words: I meant extracting the lower level part of array/ufunc/etc... from its wrapping into the python C API (with the idea that the latter could be done in Cython, modulo improvements in cython to manage the binary/code size explosion). IOW, split numpy into core and core-py (I think dynd benefits a lots from that, on top of its feature set). David > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kyle.mandli at gmail.com Thu Jun 5 16:32:24 2014 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Thu, 5 Jun 2014 15:32:24 -0500 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: It sounds like there is a lot to discuss come July and I am sure there will be others "willing" to voice their opinions as well. The primary goal in all of this would be to have a constructive discussion concerning the future of NumPy, do you guys have a feeling for what might be the most effective way to do this? A panel comes to mind but then people for the panel would have to be chosen. In the past I know that we have simply gathered in a circle and discussed which works as well. Whatever the case, if someone could volunteer to "lead" the discussion and also submit it via the SciPy conference website (you have to sign into the dashboard and submit a new proposal) to help us keep track of everything I would be very appreciative. Kyle On Wed, Jun 4, 2014 at 5:09 AM, David Cournapeau wrote: > I won't be able to make it at scipy this year sadly. > > I concur with Nathaniel that we can do a lot of things without a full > rewrite -- it is all too easy to see what is gained with a rewrite and lose > sight of what is lost. I have yet to see a really strong argument for a > full rewrite. It may be easier to do a rewrite for a core when you have a > few full-time people, but that's a different story for a community effort > like numpy. > > The main issue preventing new features in numpy is the lack of internal > architecture at the C level, but nothing that could not be done by > refactoring. Using cython to move away from the python C api would be > great, though we need to talk with the cython people so that we can share > common code between multiple extensions using cython, to avoid binary size > explosion. > > There are things that may require some backward incompatible changes in > the C API, but that's much more acceptable than a significant break at the > python level. > > David > > > On Wed, Jun 4, 2014 at 9:58 AM, Sebastian Berg > wrote: > >> On Mi, 2014-06-04 at 02:26 +0100, Nathaniel Smith wrote: >> > On Wed, Jun 4, 2014 at 12:33 AM, Charles R Harris >> > wrote: >> > > On Tue, Jun 3, 2014 at 5:08 PM, Kyle Mandli >> wrote: >> > >> >> > >> Hello everyone, >> > >> >> > >> As one of the co-chairs in charge of organizing the >> birds-of-a-feather >> > >> sesssions at the SciPy conference this year, I wanted to solicit >> through the >> > >> NumPy list to see if we could get enough interest to hold a NumPy >> centered >> > >> BoF this year. The BoF format would be up to those who would lead >> the >> > >> discussion, a couple of ideas used in the past include picking out a >> few of >> > >> the lead devs to be on a panel and have a Q&A type of session or an >> open Q&A >> > >> with perhaps audience guided list of topics. I can help facilitate >> > >> organization of something but we would really like to get something >> > >> organized this year (last year NumPy was the only major project that >> was not >> > >> really represented in the BoF sessions). >> > > >> > > I'll be at the conference, but I don't know who else will be there. I >> feel >> > > that NumPy has matured to the point where most of the current work is >> > > cleaning stuff up, making it run faster, and fixing bugs. A topic >> that I'd >> > > like to see discussed is where do we go from here. One option to look >> at is >> > > Blaze, which looks to have matured a lot in the last year. The >> problem with >> > > making it a NumPy replacement is that NumPy has become quite >> widespread, >> > > with downloads from PyPi running at about 3 million per year. With >> that much >> > > penetration it may be difficult for a new core like Blaze to gain >> traction. >> > > So I'd like to also discuss ways to bring the two branches of >> development >> > > together at some point and explore what NumPy can do to pave the way. >> Mind, >> > > there are definitely things that would be nice to add to NumPy, a >> better >> > > type system, missing values, etc., but doing that is difficult given >> the >> > > current design. >> > >> > I won't be at the conference unfortunately (I'm on the wrong continent >> > and have family commitments then anyway), but I think there's lots of >> > exciting stuff that can be done in numpy-land. >> > >> >> I wouldn't like to come, but to be honest have not planned to yet and it >> doesn't fit too well with the stuff I work on mostly right now. So will >> have to see. >> >> - Sebastian >> >> > We absolutely could rewrite the dtype system, and this would >> > straightforwardly give us excellent support for missing values, units, >> > categorical data, automatic differentiation, better datetimes, etc. >> > etc. -- and make numpy much more friendly in general to third-party >> > extensions. >> > >> > I'd like to see the ufunc system revisited in the light of all the >> > things we know now, to make gufuncs more first-class, provide better >> > support for user-defined types, more flexible loop selection (e.g. >> > make it possible to implement np.add.reduce(a, type="kahan")), etc.; >> > one goal would be to convert a lot of ufunc-like functions (np.mean >> > etc.) into being real ufuncs, and then they'd automatically benefit >> > from __numpy_ufunc__, which would also massively improve >> > interoperability with alternative array types like blaze. >> > >> > I'd like to see support for extensible label-based indexing, like >> pandas. >> > >> > Internally, I'd like to see internal migrating out of C and into >> > Cython -- we have hundreds of lines of code that could be replaced >> > with a few lines of Cython and no-one would notice. (Combining this >> > with a cffi cython backend and pypy would be pretty interesting >> > too...) >> > >> > I'd like to see sparse ndarrays, with integration into the ufunc >> > looping machinery so all ufuncs just work. Or even better, I'd like to >> > see the right hooks added so that anyone can write a sparse ndarray >> > package using only public APIs, and have all ufuncs just work. (I was >> > going to put down deferred/loop-fused/out-of-core computation as a >> > wishlist item too, but if we do it right then this too could be >> > implemented by anyone without needing to be baked into numpy proper.) >> > >> > All of these things would take some work and care, but I think they >> > could all be done incrementally and without breaking backwards >> > compatibility. Compare to ipython, which -- as Fernando likes to point >> > out :-) -- went from a little console program to its current >> > distributed-notebook-skynet-whatever-it-is by merging one working PR >> > at a time. Certainly these changes would much easier and less >> > disruptive than any plan that involves throwing out numpy and starting >> > over. But they also do help smooth the way for an incremental >> > transition to a world where numpy is regularly used alongside other >> > libraries. >> > >> > -n >> > >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Thu Jun 5 16:45:09 2014 From: stefan at sun.ac.za (=?utf-8?Q?St=C3=A9fan?= van der Walt) Date: Thu, 05 Jun 2014 22:45:09 +0200 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: Message-ID: <87y4xbf5m2.fsf@sun.ac.za> Hi Kyle Kyle Mandli writes: > The BoF format would be up to those who would lead > the discussion, a couple of ideas used in the past include picking out a > few of the lead devs to be on a panel and have a Q&A type of session or an > open Q&A with perhaps audience guided list of topics. Unfortunately I won't be at the conference this year, but if I were I'd have enjoyed seeing a couple of short presentations, drawn from, e.g., some of the people involved in this discussion (Nathan can perhaps join in via Google Hangout), about possible future directions. That way one can sketch out the playing field to seed the discussion. In addition, I those sketches would provide a useful update to all those watching the conference remotely via video. Regards St?fan From alex.eberspaecher at gmail.com Thu Jun 5 17:11:59 2014 From: alex.eberspaecher at gmail.com (=?UTF-8?B?QWxleGFuZGVyIEViZXJzcMOkY2hlcg==?=) Date: Thu, 05 Jun 2014 23:11:59 +0200 Subject: [Numpy-discussion] fftw supported? In-Reply-To: References: <538F90E0.3080300@gmail.com> Message-ID: <5390DD1F.9030703@gmail.com> On 05.06.2014 11:13, Da?id wrote: > pyFFTW provides a drop-in replacement for Numpy and Scipy's fftw: > > https://hgomersall.github.io/pyFFTW/pyfftw/interfaces/interfaces.html Sure. But if you want use multi-threading and the wisdom mechanisms, you have to take care of it yourself. You didn't have to with anfft. > You can still set the number of threads and other advanced parameters, > but you can just ignore them and view it as a very simple library. Sure. > Does your wrapper set these to reasonable values? If not, I am missing > completely the point. You have to decide on the number of threads when you configure tfftw. Anyway it is well possible that tfftw is completely useless for you. > I am starting to use pyFFTW, and maybe I can help you test tfftw. Contributions or bug reports are welcome! Alex From njs at pobox.com Thu Jun 5 18:41:33 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Jun 2014 23:41:33 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris wrote: > @nathaniel IIRC, one of the objections to the missing values work was that > it changed the underlying array object by adding a couple of variables to > the structure. I'm willing to do that sort of thing, but it would be good to > have general agreement that that is acceptable. I can't think of reason why adding new variables to the structure *per se* would be objectionable to anyone? IIRC the objection you're thinking of wasn't to the existence of new variables, but to their effect on compatibility: their semantics meant that every piece of legacy C code that worked with ndarrays had to be updated to check for the new variables before it could correctly interpret the ->data field, and if it wasn't updated it would just return incorrect results. And there wasn't really a clear story for how we were going to detect and fix all this legacy code. This specific kind of compatibility break does seem pretty objectionable, but that's because of the details of its behaviour, not because variables in general are problematic, I think. > As to blaze/dynd, I'd like to steal bits here and there, and maybe in the > long term base numpy on top of it with a compatibility layer. There is a lot > of thought and effort that has gone into those projects and we should use > what we can. As is, I think numpy is good for another five to ten years and > will probably hang on for fifteen, but it will be outdated by the end of > that period. Like great whites, we need to keep swimming just to have > oxygen. Software projects tend to be obligate ram ventilators. I worry a bit that this could become a self-fulfilling prophecy. Plenty of software survives longer than that; the Linux kernel hasn't had a "real" major number increase [1] since 2.6.0, more than 10 years ago, and it's still an extraordinarily vital project. Partly this is because they have resources we don't etc., but partly it's just because they've decided that incremental change is how they're going to do things, and approached each new feature with that in mind. And in ten years they haven't yet found any features that required a serious compatibility break. This is a pretty minor worry though -- we don't have to agree about what will happen in 10 years to agree about what to do now :-). [1] http://www.pcmag.com/article2/0,2817,2388926,00.asp > The Python 3 experience is definitely something we want to avoid. And while > blaze does big data and offers some nice features, I don't know that it > offers compelling reasons to upgrade to the more ordinary user at this time, > so I'd like to sort of slip it into numpy if possible. > > If we do start moving numpy forward in more radical steps, we should try to > have some agreement beforehand as to what sort of changes are acceptable. > For instance, to maintain backward compatibility, is it sufficient that a > recompile will do the job, or do we require forward compatibility for > extensions compiled against earlier releases? I find it hard to discuss these things in general, since specific compatibility issues usually involve complicated trade-offs -- will every package have to recompile or just some of them, if they don't will it be a nice error message or a segfault, is there some way we can issue warnings ahead of time for the offending behaviour, etc. etc. That said, my vote is that if there's a change that (a) can't be done some other way, (b) requires a recompile, (c) doesn't cause segfaults but rather produces some sensible error message like "ABI mismatch please recompile", (d) is a change that's worth the bother (this determination to include at least canvassing the list to check that users in general agree that it's worth it), then yeah we should do it. I don't anticipate that this will happen very often given how far we've gotten without it, but yeah. It's possible we should be making a fuss now on distutils-sig about handling these cases in the brave new world of wheels, so that the relevant features have some chance of existing by the time we need them (e.g., 'pip upgrade numpy' should become smart enough to detect when this necessitates an upgrade of scipy). > Do we stay with C or should we > support C++ code with its advantages of smart pointers, exception handling, > and templates? We will need a certain amount of flexibility going forward > and we should decide, or at least discuss, such issues up front. This is an easier question, since it doesn't affect end-users at all (at least, so long as they have a decent toolchain available, but scipy already requires C++). Personally I'd like to see a more concrete plan for how exactly C++ would be used and why it's better than alternatives (as mentioned I have the vague idea that Cython would be even better), but I can't see why we should rule it out up front either. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From njs at pobox.com Thu Jun 5 18:48:07 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 5 Jun 2014 23:48:07 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 3:24 PM, David Cournapeau wrote: > On Thu, Jun 5, 2014 at 2:51 PM, Charles R Harris > wrote: >> On Thu, Jun 5, 2014 at 6:40 AM, David Cournapeau >> wrote: >>> IMO, what is needed the most is refactoring the internal to extract the >>> Python C API low level from the rest of the code, as I think that's the main >>> bottleneck to get more contributors (or get new core features more quickly). >>> >> >> What do you mean by "extract the Python C API"? > > Poor choice of words: I meant extracting the lower level part of > array/ufunc/etc... from its wrapping into the python C API (with the idea > that the latter could be done in Cython, modulo improvements in cython to > manage the binary/code size explosion). > > IOW, split numpy into core and core-py (I think dynd benefits a lots from > that, on top of its feature set). Can you give some examples of these benefits? I'm kinda wary of refactoring-for-the-sake-of-it -- IME usually it's easier, more valuable, and more fun to refactor in the process of making some concrete improvement. Also, it's very much pie-in-the-sky at the moment, but if the pypy or numba or pyston compilers gained the ability to grok cython code directly, then having everything in cython instead of C could potentially allow for a single numpy code base to be shared between cpython and jitted-python, with the former working as it does now and the latter doing JIT loop fusion etc. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Fri Jun 6 01:17:05 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 5 Jun 2014 22:17:05 -0700 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: On Thu, Jun 5, 2014 at 1:32 PM, Kyle Mandli wrote: > In the past I know that we have simply gathered in a circle and discussed > which works as well. Whatever the case, if someone could volunteer to > "lead" the discussion > It's my experience that a really good facilitator could make all the difference in how productive this kind of discussion is. I have no idea how to find such a facilitator (it's a pretty rare skill), but it would be nice to try, rather than taking whoever is willing to do the bureaucratic part.... and also submit it via the SciPy conference website (you have to sign into > the dashboard and submit a new proposal) to help us keep track of > everything I would be very appreciative. > someone could still take on the organizer role while trying to find a facilitator... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Jun 6 04:29:24 2014 From: cournape at gmail.com (David Cournapeau) Date: Fri, 6 Jun 2014 09:29:24 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 11:41 PM, Nathaniel Smith wrote: > On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris > wrote: > > @nathaniel IIRC, one of the objections to the missing values work was > that > > it changed the underlying array object by adding a couple of variables to > > the structure. I'm willing to do that sort of thing, but it would be > good to > > have general agreement that that is acceptable. > > I can't think of reason why adding new variables to the structure *per > se* would be objectionable to anyone? IIRC the objection you're > thinking of wasn't to the existence of new variables, but to their > effect on compatibility: their semantics meant that every piece of > legacy C code that worked with ndarrays had to be updated to check for > the new variables before it could correctly interpret the ->data > field, and if it wasn't updated it would just return incorrect > results. And there wasn't really a clear story for how we were going > to detect and fix all this legacy code. This specific kind of > compatibility break does seem pretty objectionable, but that's because > of the details of its behaviour, not because variables in general are > problematic, I think. > > > As to blaze/dynd, I'd like to steal bits here and there, and maybe in the > > long term base numpy on top of it with a compatibility layer. There is a > lot > > of thought and effort that has gone into those projects and we should use > > what we can. As is, I think numpy is good for another five to ten years > and > > will probably hang on for fifteen, but it will be outdated by the end of > > that period. Like great whites, we need to keep swimming just to have > > oxygen. Software projects tend to be obligate ram ventilators. > > I worry a bit that this could become a self-fulfilling prophecy. > Plenty of software survives longer than that; the Linux kernel hasn't > had a "real" major number increase [1] since 2.6.0, more than 10 years > ago, and it's still an extraordinarily vital project. Partly this is > because they have resources we don't etc., but partly it's just > because they've decided that incremental change is how they're going > to do things, and approached each new feature with that in mind. And > in ten years they haven't yet found any features that required a > serious compatibility break. > > This is a pretty minor worry though -- we don't have to agree about > what will happen in 10 years to agree about what to do now :-). > > [1] http://www.pcmag.com/article2/0,2817,2388926,00.asp > > > The Python 3 experience is definitely something we want to avoid. And > while > > blaze does big data and offers some nice features, I don't know that it > > offers compelling reasons to upgrade to the more ordinary user at this > time, > > so I'd like to sort of slip it into numpy if possible. > > > > If we do start moving numpy forward in more radical steps, we should try > to > > have some agreement beforehand as to what sort of changes are acceptable. > > For instance, to maintain backward compatibility, is it sufficient that a > > recompile will do the job, or do we require forward compatibility for > > extensions compiled against earlier releases? > > I find it hard to discuss these things in general, since specific > compatibility issues usually involve complicated trade-offs -- will > every package have to recompile or just some of them, if they don't > will it be a nice error message or a segfault, is there some way we > can issue warnings ahead of time for the offending behaviour, etc. > etc. > > That said, my vote is that if there's a change that (a) can't be done > some other way, (b) requires a recompile, (c) doesn't cause segfaults > but rather produces some sensible error message like "ABI mismatch > please recompile", (d) is a change that's worth the bother (this > determination to include at least canvassing the list to check that > users in general agree that it's worth it), then yeah we should do it. > I don't anticipate that this will happen very often given how far > we've gotten without it, but yeah. > Changing the ABI 'safely' (i.e. raise a python exception if changed) is already handled in numpy. We can always increase the ABI version if we think it is worth it David -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Jun 6 04:48:53 2014 From: cournape at gmail.com (David Cournapeau) Date: Fri, 6 Jun 2014 09:48:53 +0100 Subject: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation) In-Reply-To: References: Message-ID: On Thu, Jun 5, 2014 at 11:48 PM, Nathaniel Smith wrote: > On Thu, Jun 5, 2014 at 3:24 PM, David Cournapeau > wrote: > > On Thu, Jun 5, 2014 at 2:51 PM, Charles R Harris < > charlesr.harris at gmail.com> > > wrote: > >> On Thu, Jun 5, 2014 at 6:40 AM, David Cournapeau > >> wrote: > >>> IMO, what is needed the most is refactoring the internal to extract the > >>> Python C API low level from the rest of the code, as I think that's > the main > >>> bottleneck to get more contributors (or get new core features more > quickly). > >>> > >> > >> What do you mean by "extract the Python C API"? > > > > Poor choice of words: I meant extracting the lower level part of > > array/ufunc/etc... from its wrapping into the python C API (with the idea > > that the latter could be done in Cython, modulo improvements in cython to > > manage the binary/code size explosion). > > > > IOW, split numpy into core and core-py (I think dynd benefits a lots from > > that, on top of its feature set). > > Can you give some examples of these benefits? numpy.core is difficult to approach as a codebase: it is big, and quite entangled. Why the concepts are sound, there is not much internal architecture. I would love for numpy to have a proper internal C API. I'd like to think my effort of splitting multiarray giant .c files into multiple files somehow made everybody's like easier. A lot of the current code is python C API, which nobody cares about, and could be handled by e.g. cython (although again the feasibility of that needs to be discussed with the cython team, as cython cannot realisticallybe used pervasively for numpy ATM, see e.g. http://mail.scipy.org/pipermail/scipy-dev/2012-July/017717.html). Such a separation would also be helpful for the pie-in-the-sky projects you mentioned. I think Wes Mc Kinney and the pandas team experience about using cython for the core vs just for the C<->Python integration would be useful there as well. I'm kinda wary of > refactoring-for-the-sake-of-it -- IME usually it's easier, more > valuable, and more fun to refactor in the process of making some > concrete improvement. > Sure, I am just suggesting there should be a conscious effort to not just add features but also think about the internal consistency. One concrete example is dtype and its pluggability: I would love to see things like datetime in a separate extension. It would keep us honest to allow people creating custom dtype (last time I've checked, there were some hardcoding that made it hard to do). David -------------- next part -------------- An HTML attachment was scrubbed... URL: From godlion at web.de Fri Jun 6 07:49:46 2014 From: godlion at web.de (Vertigo) Date: Fri, 6 Jun 2014 04:49:46 -0700 (PDT) Subject: [Numpy-discussion] minimize Algorithmen Problem with boundarys Message-ID: <1402055385902-37709.post@n7.nabble.com> Hello everybody, i have a Problem, with three minimize Algorithmen in scipy. I need Algorithmen to support a minimize Algorithmen with boundarys. My favorits are the 'l-bfgs-b', 'slsqp' and 'tnc' but all of them give a failure meanwhile solve my problem. By call the methode SLSQP it give me failure back out of methode in file slsqp.py in the line 397 of code: "failed in converting 8th argument 'g' of _slsqp.slsqp to C/Fortran". my Jacobi Matrix ist a differentiate from the jccurve(in the Code below) formula and i code this in Lambda formalism. The Variable "x" are the optimize Parameter in this case also x[0]=a,x[1]=b,x[2]=n,x[3]=c In my case sigma and epsilon is the flow curve of metal and i optimize the Johnson-Cook curve with this Code. That mean sigma ist a array with 128 elements also epsilon is a array with 128 elements. The Paramete a,b,n,c are number in this case. The Solver l-bfgs-b give me a failure back out of the methode in the file lbfgsb.py in the line 264 in the function g=fac(x,*args). That function is a referenz to my Jacobi-matrix "jcjac" and it give me a TypeError "'builtin_function_or_method' object has no attribute '__getitem__'" I don?t what that mean. This failure also in the Solver from TNC. I hope i could explain my Problem and everbody can help me. Here begin my extract from code: jcjac = lambda x,sig,eps,peps : array[(1+x[3]*log(peps/1)), ((eps**x[2])+(eps**x[2])*log(peps/1)+ x[0]*log(peps/1)), ( x[0]*log(peps/1)), (x[1]*(eps**x[2])*log(eps)+(eps**x[2])*log(eps)*log(peps/1))] def __jccurve__(self,a,b,n,c,epsilon,pointepsilon): sig_jc=(a+b*epsilon**n)*(1+c*numpy.log(pointepsilon/1)) return sig_jc def __residuals__(self,const,sig,epsilon,pointepsilon): return sig-self.__jccurve__(const[0],const[1],const[2],const[3],epsilon,pointepsilon) p_guess=a1,b1,n1,c max_a=a1+(a1*0.9) min_a=a1-(a1*0.9) min_a=0 max_b=b1+(b1*0.9) min_b=b1-(b1*0.9) min_b=0 max_n=n1+(n1*10) min_n=n1-(n1*10) min_n=0 max_c=c+(c*100) min_c=c-(c*100) min_c=0 optimize.minimize(self.__residuals__,p_guess,args=(sigma,epsilon,pointepsilon),method='slsqp',jac=jcjac, hess=None,hessp=None,bounds=((min_a,max_a),(min_b,max_b),(min_n,max_n),(min_c,max_c))) best greets Lars -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/minimize-Algorithmen-Problem-with-boundarys-tp37709.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From jtaylor.debian at googlemail.com Sun Jun 8 16:34:51 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 08 Jun 2014 22:34:51 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release Message-ID: <5394C8EB.3020404@googlemail.com> Hello, I'm happy to announce the fist beta release of Numpy 1.9.0. 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 - 3.4. Due to low demand windows binaries for the beta are only available for Python 2.7, 3.3 and 3.4. Please try it and report any issues to the numpy-discussion mailing list or on github. The 1.9 release will consists of mainly of many small improvements and bugfixes. The highlights are: * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray subclasses. Please note that there are still some known issues with this mechanism which we hope to resolve before the final release (e.g. #4753) * Numerous performance improvements in various areas, most notably indexing and operations on small arrays are significantly faster. Indexing operations now also release the GIL. * Addition of nanmedian and nanpercentile rounds out the nanfunction set. The changes involve a lot of small changes that might affect some applications, please read the release notes for the full details on all changes: https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-notes.rst Please also take special note of the future changes section which will apply to the following release 1.10.0 and make sure to check if your applications would be affected by them. Source tarballs, windows installers and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 Cheers, Julian Taylor -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From charlesr.harris at gmail.com Sun Jun 8 16:43:42 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sun, 8 Jun 2014 14:43:42 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <5394C8EB.3020404@googlemail.com> References: <5394C8EB.3020404@googlemail.com> Message-ID: On Sun, Jun 8, 2014 at 2:34 PM, Julian Taylor wrote: > Hello, > > I'm happy to announce the fist beta release of Numpy 1.9.0. > 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 > - 3.4. > Due to low demand windows binaries for the beta are only available for > Python 2.7, 3.3 and 3.4. > Please try it and report any issues to the numpy-discussion mailing list > or on github. > > The 1.9 release will consists of mainly of many small improvements and > bugfixes. The highlights are: > > * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray > subclasses. Please note that there are still some known issues with this > mechanism which we hope to resolve before the final release (e.g. #4753) > * Numerous performance improvements in various areas, most notably > indexing and operations on small arrays are significantly faster. > Indexing operations now also release the GIL. > * Addition of nanmedian and nanpercentile rounds out the nanfunction set. > > The changes involve a lot of small changes that might affect some > applications, please read the release notes for the full details on all > changes: > > https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-notes.rst > Please also take special note of the future changes section which will > apply to the following release 1.10.0 and make sure to check if your > applications would be affected by them. > > Source tarballs, windows installers and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 > > Thanks for this, good job. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sun Jun 8 19:54:51 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 8 Jun 2014 23:54:51 +0000 (UTC) Subject: [Numpy-discussion] Only debug builds with 64-bit MinGW? Message-ID: <1401818064423964337.258744sturla.molden-gmail.com@news.gmane.org> Why does 64-bit MinGW require -O0 and -DDEBUG? https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler.py#L120 Is this a bug or intentional? Sturla From jtaylor.debian at googlemail.com Sun Jun 8 20:00:41 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 09 Jun 2014 02:00:41 +0200 Subject: [Numpy-discussion] Only debug builds with 64-bit MinGW? In-Reply-To: <1401818064423964337.258744sturla.molden-gmail.com@news.gmane.org> References: <1401818064423964337.258744sturla.molden-gmail.com@news.gmane.org> Message-ID: <5394F929.1070601@googlemail.com> On 09.06.2014 01:54, Sturla Molden wrote: > Why does 64-bit MinGW require -O0 and -DDEBUG? > > https://github.com/numpy/numpy/blob/master/numpy/distutils/mingw32ccompiler.py#L120 > > Is this a bug or intentional? > probably a temporary commit that was never removed, it originates from 2008: commit 55446eef45da7a66ee300e68cd99411467efdb9c Author: David Cournapeau Date: Sat Dec 20 17:31:48 2008 +0000 Remove optimization flags for now, to speed up builds. --- numpy/distutils/mingw32ccompiler.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) From sturla.molden at gmail.com Sun Jun 8 20:36:57 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 9 Jun 2014 00:36:57 +0000 (UTC) Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) Message-ID: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> NumPy seems to define BLAS and LAPACK functions with gfortran ABI: https://github.com/numpy/numpy/blob/master/numpy/linalg/umath_linalg.c.src#L289 extern float FNAME(sdot)(int *n, float *sx, int *incx, float *sy, int *incy); What happens on OS X where Accelerate Framework uses f2c ABI? SciPy has C wrappers for Accelerate to deal with this. Should NumPy adopt those as well? And while we are at it, why are LAPACK subroutines declared to return int? I thought a Fortran subroutine should map to a void function in C. From the same file: extern int FNAME(dgesv)(int *n, int *nrhs, double a[], int *lda, int ipiv[], double b[], int *ldb, int *info); Sturla From sturla.molden at gmail.com Sun Jun 8 20:42:13 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 9 Jun 2014 00:42:13 +0000 (UTC) Subject: [Numpy-discussion] Only debug builds with 64-bit MinGW? References: <1401818064423964337.258744sturla.molden-gmail.com@news.gmane.org> <5394F929.1070601@googlemail.com> Message-ID: <1731545220423967066.834203sturla.molden-gmail.com@news.gmane.org> Julian Taylor wrote: > probably a temporary commit that was never removed, it originates from 2008: Should we get rid of it for NumPy 1.9? Sturla From rmcgibbo at gmail.com Sun Jun 8 21:31:08 2014 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Sun, 8 Jun 2014 18:31:08 -0700 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: Yeah, that's definitely not the right signature for sdot when linked against the Accelerate framework. But what's the purpose of those wrappers in _umath_linalg? The one for sdot, for example, doesn't appear to be used internally, and it isn't available from the python side. While scipy has all of those function pointers accessible in scipy.linalg.blas (e.g. scipy.linalg.blas.sdot._cpointer), those function pointers point to the underlying blas function and expose the ABI differences between platforms, leading to, for example the utter nightmare of using the scipy blas function pointers from cython in a cross platform way: https://gist.github.com/anonymous/6659007. -Robert On Sun, Jun 8, 2014 at 5:36 PM, Sturla Molden wrote: > NumPy seems to define BLAS and LAPACK functions with gfortran ABI: > > > https://github.com/numpy/numpy/blob/master/numpy/linalg/umath_linalg.c.src#L289 > > extern float > FNAME(sdot)(int *n, > float *sx, int *incx, > float *sy, int *incy); > > What happens on OS X where Accelerate Framework uses f2c ABI? SciPy has C > wrappers for Accelerate to deal with this. Should NumPy adopt those as > well? > > And while we are at it, why are LAPACK subroutines declared to return int? > I thought a Fortran subroutine should map to a void function in C. From the > same file: > > extern int > FNAME(dgesv)(int *n, int *nrhs, > double a[], int *lda, > int ipiv[], > double b[], int *ldb, > int *info); > > > Sturla > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sun Jun 8 23:33:23 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 9 Jun 2014 04:33:23 +0100 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Mon, Jun 9, 2014 at 2:31 AM, Robert McGibbon wrote: > Yeah, that's definitely not the right signature for sdot when linked against > the Accelerate framework. > > But what's the purpose of those wrappers in _umath_linalg? The one for sdot, > for example, doesn't appear to be used internally, and it isn't available > from the python side. While scipy has all of those function pointers > accessible in scipy.linalg.blas (e.g. scipy.linalg.blas.sdot._cpointer), > those function pointers point to the underlying blas function and expose the > ABI differences between platforms, leading to, for example the utter > nightmare of using the scipy blas function pointers from cython in a cross > platform way: https://gist.github.com/anonymous/6659007. Is it possible this is related to: https://github.com/numpy/numpy/issues/4007 or even: https://github.com/numpy/numpy/issues/4776 ? Best, Matthew From rmcgibbo at gmail.com Mon Jun 9 00:07:51 2014 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Sun, 8 Jun 2014 21:07:51 -0700 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: #4776 is definitely a separate issue. #4007 could be related similar f2c ABI issues, but I don't think so. The difference between the gfotran and g77/f2c ABIs, to my knowledge, is that (1) fortran REAL (single precision floating point) return values are actually returned as a c double, not a c float. (2) when return values are complex, the function is actually implemented with an ABI that looks like a c function with an extra pass-by-reference argument at the front of the function where the return value is placed. Despite the differences in the handling of the return values, I don't think there is any difference in how float32 parameters are passed *in*, so I'm not sure that these issues are related to subroutines like SGEMV which is presumably behind #4007. -Robert On Sun, Jun 8, 2014 at 8:33 PM, Matthew Brett wrote: > Hi, > > On Mon, Jun 9, 2014 at 2:31 AM, Robert McGibbon > wrote: > > Yeah, that's definitely not the right signature for sdot when linked > against > > the Accelerate framework. > > > > But what's the purpose of those wrappers in _umath_linalg? The one for > sdot, > > for example, doesn't appear to be used internally, and it isn't available > > from the python side. While scipy has all of those function pointers > > accessible in scipy.linalg.blas (e.g. scipy.linalg.blas.sdot._cpointer), > > those function pointers point to the underlying blas function and expose > the > > ABI differences between platforms, leading to, for example the utter > > nightmare of using the scipy blas function pointers from cython in a > cross > > platform way: https://gist.github.com/anonymous/6659007. > > Is it possible this is related to: > > https://github.com/numpy/numpy/issues/4007 > > or even: > > https://github.com/numpy/numpy/issues/4776 > > ? > > Best, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmcgibbo at gmail.com Mon Jun 9 00:08:17 2014 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Sun, 8 Jun 2014 21:08:17 -0700 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: But I can't reproduce #4007 on my machine, so I'm not totally sure. -Robert On Sun, Jun 8, 2014 at 9:07 PM, Robert McGibbon wrote: > #4776 is definitely a separate issue. #4007 could be related similar f2c > ABI issues, but I don't think so. The difference between the gfotran and > g77/f2c ABIs, to my knowledge, is that (1) fortran REAL (single precision > floating point) return values are actually returned as a c double, not a c > float. (2) when return values are complex, the function is actually > implemented with an ABI that looks like a c function with an extra > pass-by-reference argument at the front of the function where the return > value is placed. > > Despite the differences in the handling of the return values, I don't > think there is any difference in how float32 parameters are passed *in*, so > I'm not sure that these issues are related to subroutines like SGEMV which > is presumably behind #4007. > > -Robert > > > On Sun, Jun 8, 2014 at 8:33 PM, Matthew Brett > wrote: > >> Hi, >> >> On Mon, Jun 9, 2014 at 2:31 AM, Robert McGibbon >> wrote: >> > Yeah, that's definitely not the right signature for sdot when linked >> against >> > the Accelerate framework. >> > >> > But what's the purpose of those wrappers in _umath_linalg? The one for >> sdot, >> > for example, doesn't appear to be used internally, and it isn't >> available >> > from the python side. While scipy has all of those function pointers >> > accessible in scipy.linalg.blas (e.g. scipy.linalg.blas.sdot._cpointer), >> > those function pointers point to the underlying blas function and >> expose the >> > ABI differences between platforms, leading to, for example the utter >> > nightmare of using the scipy blas function pointers from cython in a >> cross >> > platform way: https://gist.github.com/anonymous/6659007. >> >> Is it possible this is related to: >> >> https://github.com/numpy/numpy/issues/4007 >> >> or even: >> >> https://github.com/numpy/numpy/issues/4776 >> >> ? >> >> Best, >> >> Matthew >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Jun 9 00:25:47 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 9 Jun 2014 05:25:47 +0100 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Jun 9, 2014 at 5:08 AM, Robert McGibbon wrote: > But I can't reproduce #4007 on my machine, so I'm not totally sure. > Are you using the current OSX wheel from pypi to install? You may need to run the test code a few times to get the issue. Cheers, Matthew From rmcgibbo at gmail.com Mon Jun 9 00:47:57 2014 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Sun, 8 Jun 2014 21:47:57 -0700 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: > You may need to run the test code a few times to get the issue. Okay, yeah, I can reproduce the #4007 issue now. Both with the 3.4 dmg from python.org and the OSX wheel as well as 2.7 with numpy compiled from source. I need to run the test [1] a couple times to get it. But I think Sturla's observation about the _umath_linalg ABI issues is separate. -Robert [1] https://github.com/numpy/numpy/issues/4007#issuecomment-27708193 On Sun, Jun 8, 2014 at 9:25 PM, Matthew Brett wrote: > On Mon, Jun 9, 2014 at 5:08 AM, Robert McGibbon > wrote: > > But I can't reproduce #4007 on my machine, so I'm not totally sure. > > > > Are you using the current OSX wheel from pypi to install? > > You may need to run the test code a few times to get the issue. > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Jun 9 07:40:55 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 9 Jun 2014 11:40:55 +0000 (UTC) Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> Message-ID: <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Is it possible this is related to: > > https://github.com/numpy/numpy/issues/4007 Possibly, but it seems to be in _dotblas which uses cblas. NumPy has its own cblas.h. Perhaps it conflicts with Apple's? Apple's documentation says that cblas_sdot returns float, but knowing Apple this might not be accurate. OK, time to locate cblas.h on my MacBook... > or even: > > https://github.com/numpy/numpy/issues/4776 No. This happens because Accelerate and GCD are not designed to be fork-safe. One can argue that this bug is actually in multiprocessing and not in NumPy or Accelerate, since BLAS is not a part of POSIX. Only POSIX functions are required to be fork safe. Python 3.4 has a fix for it by allowing spawn (fork+exec) to be used instead of fork. It will never be fixed on Python 2.7 unless someone backports multiprocessing from Python 3.4. BTW: GotoBLAS2 is also affected by this problem, not just Accelerate. MKL and Atlas are safe, and so is OpenBLAS from master at GitHub. Sturla From sturla.molden at gmail.com Mon Jun 9 08:37:02 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 09 Jun 2014 14:37:02 +0200 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> Message-ID: On 09/06/14 13:40, Sturla Molden wrote: >> https://github.com/numpy/numpy/issues/4007 > > Possibly, but it seems to be in _dotblas which uses cblas. NumPy has its > own cblas.h. Perhaps it conflicts with Apple's? > > Apple's documentation says that cblas_sdot returns float, but knowing Apple > this might not be accurate. OK, time to locate cblas.h on my MacBook... Hm, no, the declarations of cblas_sdot seems to be the same. NumPy: float cblas_sdot(const int N, const float *X, const int incX, const float *Y, const int incY); Apple: float cblas_sdot(const int N, const float *X, const int incX, const float *Y, const int incY) __OSX_AVAILABLE_STARTING(__MAC_10_2,__IPHONE_4_0); Sturla From sturla.molden at gmail.com Mon Jun 9 08:53:13 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Mon, 09 Jun 2014 14:53:13 +0200 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> Message-ID: On 09/06/14 14:37, Sturla Molden wrote: > Hm, no, the declarations of cblas_sdot seems to be the same. Oh, stupid me, matrix * vector ... Then it must be cblas_sgemv. But those are the same too. Anyway, Enthought Canopy (linked with MKL) is *NOT* affected >>> m = np.random.rand(200).astype(np.float32) >>> s = np.random.rand(10000, 200).astype(np.float32) >>> np.dot(s,m) array([ 52.27999878, 55.47311401, 63.61336517, ..., 56.2276535 , 56.59591293, 55.27639008], dtype=float32) I see an Anaconda user reports Anaconda is affected, but Anaconda is linked with MKL as well (or used to be?) Sturla From olivier.grisel at ensta.org Mon Jun 9 08:59:27 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Mon, 9 Jun 2014 14:59:27 +0200 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> Message-ID: 2014-06-09 14:53 GMT+02:00 Sturla Molden : > > I see an Anaconda user reports Anaconda is affected, but Anaconda is > linked with MKL as well (or used to be?) Not necessarily. Only if you buy the MKL optimization package: https://store.continuum.io/cshop/mkl-optimizations/ There is time limited evaluation package though. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From cmkleffner at gmail.com Mon Jun 9 09:51:17 2014 From: cmkleffner at gmail.com (Carl Kleffner) Date: Mon, 9 Jun 2014 15:51:17 +0200 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> Message-ID: The free windows conda packages are linked against MKL statically, similar to C. Gohlke's packges. My guess: the MKL optimization package supports multithreading and SVML, the free packages only a serial interface to MKL. Carl 2014-06-09 14:59 GMT+02:00 Olivier Grisel : > 2014-06-09 14:53 GMT+02:00 Sturla Molden : > > > > I see an Anaconda user reports Anaconda is affected, but Anaconda is > > linked with MKL as well (or used to be?) > > Not necessarily. Only if you buy the MKL optimization package: > > https://store.continuum.io/cshop/mkl-optimizations/ > > There is time limited evaluation package though. > > -- > Olivier > http://twitter.com/ogrisel - http://github.com/ogrisel > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Mon Jun 9 10:51:37 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Mon, 9 Jun 2014 16:51:37 +0200 Subject: [Numpy-discussion] BLAS and LAPACK ABI questions (mostly OSX related) In-Reply-To: References: <736985433423966337.381819sturla.molden-gmail.com@news.gmane.org> <893154093424004298.464749sturla.molden-gmail.com@news.gmane.org> Message-ID: 2014-06-09 15:51 GMT+02:00 Carl Kleffner : > The free windows conda packages are linked against MKL statically, similar > to C. Gohlke's packges. > > My guess: the MKL optimization package supports multithreading and SVML, the > free packages only a serial interface to MKL. That would make sense, indeed. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From jaime.frio at gmail.com Mon Jun 9 10:56:48 2014 From: jaime.frio at gmail.com (=?UTF-8?Q?Jaime_Fern=C3=A1ndez_del_R=C3=ADo?=) Date: Mon, 9 Jun 2014 07:56:48 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> Message-ID: On Sun, Jun 8, 2014 at 1:43 PM, Charles R Harris wrote: > > > > On Sun, Jun 8, 2014 at 2:34 PM, Julian Taylor < > jtaylor.debian at googlemail.com> wrote: > >> Hello, >> >> I'm happy to announce the fist beta release of Numpy 1.9.0. >> 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 >> - 3.4. >> Due to low demand windows binaries for the beta are only available for >> Python 2.7, 3.3 and 3.4. >> Please try it and report any issues to the numpy-discussion mailing list >> or on github. >> >> The 1.9 release will consists of mainly of many small improvements and >> bugfixes. The highlights are: >> >> * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray >> subclasses. Please note that there are still some known issues with this >> mechanism which we hope to resolve before the final release (e.g. #4753) >> * Numerous performance improvements in various areas, most notably >> indexing and operations on small arrays are significantly faster. >> Indexing operations now also release the GIL. >> * Addition of nanmedian and nanpercentile rounds out the nanfunction set. >> >> The changes involve a lot of small changes that might affect some >> applications, please read the release notes for the full details on all >> changes: >> >> https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-notes.rst >> Please also take special note of the future changes section which will >> apply to the following release 1.10.0 and make sure to check if your >> applications would be affected by them. >> >> Source tarballs, windows installers and release notes can be found at >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 >> >> > Thanks for this, good job. > > Chuck > This is the first release of numpy I have had any involvement with, and I am truly amazed at the amount of talent and dedication you guys put into it. Big, big thank you to everyone! Jaime > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ay?dale en sus planes de dominaci?n mundial. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Jun 9 12:10:07 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 9 Jun 2014 09:10:07 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <5394C8EB.3020404@googlemail.com> References: <5394C8EB.3020404@googlemail.com> Message-ID: I take nothing ever happened to clean up the datetime64 timezone mess? sigh. -Chris On Sun, Jun 8, 2014 at 1:34 PM, Julian Taylor wrote: > Hello, > > I'm happy to announce the fist beta release of Numpy 1.9.0. > 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 > - 3.4. > Due to low demand windows binaries for the beta are only available for > Python 2.7, 3.3 and 3.4. > Please try it and report any issues to the numpy-discussion mailing list > or on github. > > The 1.9 release will consists of mainly of many small improvements and > bugfixes. The highlights are: > > * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray > subclasses. Please note that there are still some known issues with this > mechanism which we hope to resolve before the final release (e.g. #4753) > * Numerous performance improvements in various areas, most notably > indexing and operations on small arrays are significantly faster. > Indexing operations now also release the GIL. > * Addition of nanmedian and nanpercentile rounds out the nanfunction set. > > The changes involve a lot of small changes that might affect some > applications, please read the release notes for the full details on all > changes: > > https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-notes.rst > Please also take special note of the future changes section which will > apply to the following release 1.10.0 and make sure to check if your > applications would be affected by them. > > Source tarballs, windows installers and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 > > Cheers, > Julian Taylor > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cgohlke at uci.edu Mon Jun 9 19:21:57 2014 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 09 Jun 2014 16:21:57 -0700 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <5394C8EB.3020404@googlemail.com> References: <5394C8EB.3020404@googlemail.com> Message-ID: <53964195.2090605@uci.edu> On 6/8/2014 1:34 PM, Julian Taylor wrote: > Hello, > > I'm happy to announce the fist beta release of Numpy 1.9.0. > 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 > - 3.4. > Due to low demand windows binaries for the beta are only available for > Python 2.7, 3.3 and 3.4. > Please try it and report any issues to the numpy-discussion mailing list > or on github. > > The 1.9 release will consists of mainly of many small improvements and > bugfixes. The highlights are: > > * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray > subclasses. Please note that there are still some known issues with this > mechanism which we hope to resolve before the final release (e.g. #4753) > * Numerous performance improvements in various areas, most notably > indexing and operations on small arrays are significantly faster. > Indexing operations now also release the GIL. > * Addition of nanmedian and nanpercentile rounds out the nanfunction set. > > The changes involve a lot of small changes that might affect some > applications, please read the release notes for the full details on all > changes: > https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-notes.rst > Please also take special note of the future changes section which will > apply to the following release 1.10.0 and make sure to check if your > applications would be affected by them. > > Source tarballs, windows installers and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 > > Cheers, > Julian Taylor > Hello, I tested numpy-MKL-1.9.0b1 (msvc9, Intel MKL build) on win-amd64-py2.7 against a few other packages that were built against numpy-MKL-1.8.x. While numpy and scipy pass all tests, some other packages (matplotlib, statsmodels, skimage, pandas, pytables, sklearn...) show a few new test failures (compared to testing with numpy-MKL-1.8.1). Many test errors are of kind: ValueError: shape mismatch: value array of shape (24,) could not be broadcast to indexing result of shape (8,3) I have attached a list of failing tests. The full test results are at (compare to ) I have not investigated any further... Christoph -------------- next part -------------- matplotlib 1.3.1 ================ ====================================================================== ERROR: test suite for ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\suite.py", line 208, in run self.setUp() File "X:\Python27-x64\lib\site-packages\nose\suite.py", line 291, in setUp self.setupContext(ancestor) File "X:\Python27-x64\lib\site-packages\nose\suite.py", line 314, in setupContext try_run(context, names) File "X:\Python27-x64\lib\site-packages\nose\util.py", line 470, in try_run return func() File "X:\Python27-x64\lib\site-packages\matplotlib\testing\decorators.py", line 102, in setup_class cls._func() File "X:\Python27-x64\lib\site-packages\matplotlib\tests\test_triangulation.py", line 715, in test_tri_smooth_contouring tri_refi, z_test_refi = refiner.refine_field(z0, subdiv=4) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\trirefine.py", line 179, in refine_field subdiv=subdiv, return_tri_index=True) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\trirefine.py", line 125, in refine_triangulation ] = np.repeat(ancestors[ancestor_mask], 3) ValueError: shape mismatch: value array of shape (13824,) could not be broadcast to indexing result of shape (4608,3) ====================================================================== ERROR: test suite for ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\suite.py", line 208, in run self.setUp() File "X:\Python27-x64\lib\site-packages\nose\suite.py", line 291, in setUp self.setupContext(ancestor) File "X:\Python27-x64\lib\site-packages\nose\suite.py", line 314, in setupContext try_run(context, names) File "X:\Python27-x64\lib\site-packages\nose\util.py", line 470, in try_run return func() File "X:\Python27-x64\lib\site-packages\matplotlib\testing\decorators.py", line 102, in setup_class cls._func() File "X:\Python27-x64\lib\site-packages\matplotlib\tests\test_triangulation.py", line 752, in test_tri_smooth_gradient tri_refi, z_test_refi = refiner.refine_field(V, subdiv=3) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\trirefine.py", line 179, in refine_field subdiv=subdiv, return_tri_index=True) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\trirefine.py", line 125, in refine_triangulation ] = np.repeat(ancestors[ancestor_mask], 3) ValueError: shape mismatch: value array of shape (5376,) could not be broadcast to indexing result of shape (1792,3) ====================================================================== ERROR: matplotlib.tests.test_triangulation.test_triinterp ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\matplotlib\tests\test_triangulation.py", line 306, in test_triinterp (interp_dzsdx, interp_dzsdy) = cubic_user.gradient(x, y) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\triinterpolate.py", line 435, in gradient return_keys=('dzdx', 'dzdy')) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\triinterpolate.py", line 208, in _interpolate_multikeys return_key, valid_tri_index, valid_x, valid_y) * scale TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions ====================================================================== ERROR: matplotlib.tests.test_triangulation.test_triinterpcubic_C1_continuity ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\matplotlib\tests\test_triangulation.py", line 405, in test_triinterpcubic_C1_continuity check_continuity(interp, (ax, ay), values[:, 0]) File "X:\Python27-x64\lib\site-packages\matplotlib\tests\test_triangulation.py", line 366, in check_continuity (dzx, dzy) = interpolator.gradient([loc_x], [loc_y]) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\triinterpolate.py", line 435, in gradient return_keys=('dzdx', 'dzdy')) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\triinterpolate.py", line 208, in _interpolate_multikeys return_key, valid_tri_index, valid_x, valid_y) * scale TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions ====================================================================== ERROR: matplotlib.tests.test_triangulation.test_trirefine ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\matplotlib\tests\test_triangulation.py", line 880, in test_trirefine refined_triang, refined_z = refiner.refine_field(z, subdiv=1) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\trirefine.py", line 179, in refine_field subdiv=subdiv, return_tri_index=True) File "X:\Python27-x64\lib\site-packages\matplotlib\tri\trirefine.py", line 116, in refine_triangulation found_index[refi_triangles] = np.repeat(ancestors, 3) ValueError: shape mismatch: value array of shape (24,) could not be broadcast to indexing result of shape (8,3) pandas 0.14.0 ============= ====================================================================== ERROR: test_interp_regression (pandas.tests.test_generic.TestSeries) ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\pandas\tests\test_generic.py", line 501, in test_interp_regression interp_s = ser.reindex(new_index).interpolate(method='pchip') File "X:\Python27-x64\lib\site-packages\pandas\core\generic.py", line 2582, in interpolate **kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\internals.py", line 2197, in interpolate return self.apply('interpolate', **kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\internals.py", line 2164, in apply applied = getattr(b, f)(**kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\internals.py", line 667, in interpolate **kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\internals.py", line 733, in _interpolate interp_values = np.apply_along_axis(func, axis, data) File "D:\Build\Test\numpy-build\numpy\lib\shape_base.py", line 86, in apply_along_axis res = func1d(arr[tuple(i.tolist())], *args, **kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\internals.py", line 730, in func bounds_error=False, **kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\common.py", line 1489, in interpolate_1d bounds_error=bounds_error, **kwargs) File "X:\Python27-x64\lib\site-packages\pandas\core\common.py", line 1541, in _interpolate_scipy_wrapper new_y = method(x, y, new_x) File "X:\Python27-x64\lib\site-packages\scipy\interpolate\_monotone.py", line 221, in pchip_interpolate return P(x) File "X:\Python27-x64\lib\site-packages\scipy\interpolate\_monotone.py", line 98, in __call__ out = self._bpoly(x, der, extrapolate) File "X:\Python27-x64\lib\site-packages\scipy\interpolate\interpolate.py", line 673, in __call__ self._evaluate(x, nu, extrapolate, out) File "X:\Python27-x64\lib\site-packages\scipy\interpolate\interpolate.py", line 1071, in _evaluate self.x, x, nu, bool(extrapolate), out, self.c.dtype) File "_ppoly.pyx", line 846, in scipy.interpolate._ppoly.evaluate_bernstein (scipy\interpolate\_ppoly.c:15014) File "stringsource", line 622, in View.MemoryView.memoryview_cwrapper (scipy\interpolate\_ppoly.c:23370) File "stringsource", line 327, in View.MemoryView.memoryview.__cinit__ (scipy\interpolate\_ppoly.c:19922) ValueError: buffer source array is read-only statsmodels 0.5.0 ================= ====================================================================== ERROR: statsmodels.emplike.tests.test_aft.Test_AFTModel.test_beta_vect ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\tests\test_aft.py", line 34, in test_beta_vect assert_almost_equal(self.res1.test_beta([3.5, -.035], [0, 1]), File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\aft_el.py", line 481, in test_beta llr, pval, new_weights = reg_model.el_test(b0_vals, param_nums, return_weights=True) # Needs to be changed File "X:\Python27-x64\lib\site-packages\statsmodels\regression\linear_model.py", line 1519, in el_test stochastic_exog=stochastic_exog) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\elregress.py", line 58, in _opt_nuis_regress params[nuis_param_index] = nuisance_params ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (0,) ====================================================================== ERROR: statsmodels.emplike.tests.test_origin.TestOrigin.test_ci_beta ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\tests\test_origin.py", line 35, in test_ci_beta ci = self.res1.conf_int_el(1) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\originregress.py", line 256, in conf_int_el lowerl = optimize.brentq(f, lower_bound, self.params[param_num]) File "X:\Python27-x64\lib\site-packages\scipy\optimize\zeros.py", line 415, in brentq r = _zeros._brentq(f,a,b,xtol,rtol,maxiter,args,full_output,disp) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\originregress.py", line 255, in stochastic_exog=stochastic_exog)[0] - r0 File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\originregress.py", line 202, in el_test return_weights=return_weights) File "X:\Python27-x64\lib\site-packages\statsmodels\regression\linear_model.py", line 1519, in el_test stochastic_exog=stochastic_exog) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\elregress.py", line 58, in _opt_nuis_regress params[nuis_param_index] = nuisance_params ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (0,) ====================================================================== ERROR: statsmodels.emplike.tests.test_origin.TestOrigin.test_hypothesis_beta1 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\tests\test_origin.py", line 31, in test_hypothesis_beta1 assert_almost_equal(self.res1.el_test([.0034],[1])[0], File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\originregress.py", line 202, in el_test return_weights=return_weights) File "X:\Python27-x64\lib\site-packages\statsmodels\regression\linear_model.py", line 1519, in el_test stochastic_exog=stochastic_exog) File "X:\Python27-x64\lib\site-packages\statsmodels\emplike\elregress.py", line 58, in _opt_nuis_regress params[nuis_param_index] = nuisance_params ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (0,) sklearn 0.14.1 ============== ====================================================================== ERROR: sklearn.tests.test_common.test_regressors_train ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\sklearn\tests\test_common.py", line 798, in test_regressors_train assert_raises(ValueError, regressor.fit, X, y[:-1]) File "X:\Python27-x64\lib\unittest\case.py", line 473, in assertRaises callableObj(*args, **kwargs) File "X:\Python27-x64\lib\site-packages\sklearn\linear_model\least_angle.py", line 937, in fit for train, test in cv) File "X:\Python27-x64\lib\site-packages\sklearn\externals\joblib\parallel.py", line 516, in __call__ for function, args, kwargs in iterable: File "X:\Python27-x64\lib\site-packages\sklearn\linear_model\least_angle.py", line 937, in for train, test in cv) IndexError: index 199 is out of bounds for axis 1 with size 199 ====================================================================== FAIL: sklearn.utils.tests.test_extmath.test_random_weights ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\sklearn\utils\tests\test_extmath.py", line 68, in test_random_weights assert_true(np.all(score.ravel() == w[:, :5].sum(1))) AssertionError: False is not true skimage 0.10.0 ============== ====================================================================== ERROR: test_join.test_relabel_sequential_offset1 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\tests\test_join.py", line 30, in test_relabel_sequential_offset1 ar_relab, fw, inv = relabel_sequential(ar) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\_join.py", line 127, in relabel_sequential forward_map[labels0] = np.arange(offset, offset + len(labels0) + 1) ValueError: shape mismatch: value array of shape (6,) could not be broadcast to indexing result of shape (5,) ====================================================================== ERROR: test_join.test_relabel_sequential_offset5 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\tests\test_join.py", line 42, in test_relabel_sequential_offset5 ar_relab, fw, inv = relabel_sequential(ar, offset=5) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\_join.py", line 127, in relabel_sequential forward_map[labels0] = np.arange(offset, offset + len(labels0) + 1) ValueError: shape mismatch: value array of shape (6,) could not be broadcast to indexing result of shape (5,) ====================================================================== ERROR: test_join.test_relabel_sequential_offset5_with0 ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\tests\test_join.py", line 54, in test_relabel_sequential_offset5_with0 ar_relab, fw, inv = relabel_sequential(ar, offset=5) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\_join.py", line 127, in relabel_sequential forward_map[labels0] = np.arange(offset, offset + len(labels0) + 1) ValueError: shape mismatch: value array of shape (6,) could not be broadcast to indexing result of shape (5,) ====================================================================== ERROR: test_join.test_relabel_sequential_dtype ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\tests\test_join.py", line 66, in test_relabel_sequential_dtype ar_relab, fw, inv = relabel_sequential(ar, offset=5) File "X:\Python27-x64\lib\site-packages\skimage\segmentation\_join.py", line 127, in relabel_sequential forward_map[labels0] = np.arange(offset, offset + len(labels0) + 1) ValueError: shape mismatch: value array of shape (6,) could not be broadcast to indexing result of shape (5,) tables 3.1.1 ============ ====================================================================== ERROR: test05b_modifyColumns (tables.tests.test_nestedtypes.WriteNoReopen) Modifying two nested columns (modify_columns). ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\tables\tests\common.py", line 144, in newmethod return oldmethod(self, *args, **kwargs) File "X:\Python27-x64\lib\site-packages\tables\tests\test_nestedtypes.py", line 527, in test05b_modifyColumns dtype=raCols.dtype) File "D:\Build\Test\numpy-build\numpy\core\records.py", line 519, in fromarrays arrayList = [sb.asarray(x) for x in arrayList] File "D:\Build\Test\numpy-build\numpy\core\numeric.py", line 461, in asarray return array(a, dtype, copy=False, order=order) File "X:\Python27-x64\lib\site-packages\tables\table.py", line 3549, in __iter__ out=buf_slice) File "X:\Python27-x64\lib\site-packages\tables\table.py", line 1975, in read arr = self._read(start, stop, step, field, out) File "X:\Python27-x64\lib\site-packages\tables\table.py", line 1879, in _read bytes_required)) ValueError: output array size invalid, got 8 bytes, need 16 bytes ====================================================================== ERROR: test05b_modifyColumns (tables.tests.test_nestedtypes.WriteReopen) Modifying two nested columns (modify_columns). ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\tables\tests\common.py", line 144, in newmethod return oldmethod(self, *args, **kwargs) File "X:\Python27-x64\lib\site-packages\tables\tests\test_nestedtypes.py", line 527, in test05b_modifyColumns dtype=raCols.dtype) File "D:\Build\Test\numpy-build\numpy\core\records.py", line 519, in fromarrays arrayList = [sb.asarray(x) for x in arrayList] File "D:\Build\Test\numpy-build\numpy\core\numeric.py", line 461, in asarray return array(a, dtype, copy=False, order=order) File "X:\Python27-x64\lib\site-packages\tables\table.py", line 3549, in __iter__ out=buf_slice) File "X:\Python27-x64\lib\site-packages\tables\table.py", line 1975, in read arr = self._read(start, stop, step, field, out) File "X:\Python27-x64\lib\site-packages\tables\table.py", line 1879, in _read bytes_required)) ValueError: output array size invalid, got 8 bytes, need 16 bytes bottleneck 0.8.0 ================ ====================================================================== FAIL: Test nansum. ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\bottleneck\tests\func_test.py", line 80, in unit_maker assert_array_equal(actual, desired, err_msg) File "D:\Build\Test\numpy-build\numpy\testing\utils.py", line 734, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "D:\Build\Test\numpy-build\numpy\testing\utils.py", line 623, in assert_array_compare chk_same_position(x_isnan, y_isnan, hasval='nan') File "D:\Build\Test\numpy-build\numpy\testing\utils.py", line 603, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal func nansum | input a24 (float32) | shape (0L,) | axis -1 Input array: [] x and y nan location mismatch: x: array(nan, dtype=float32) y: array(0.0, dtype=float32) ====================================================================== FAIL: Test move_nansum. ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\bottleneck\tests\move_test.py", line 63, in unit_maker err_msg) File "D:\Build\Test\numpy-build\numpy\testing\utils.py", line 836, in assert_array_almost_equal precision=decimal) File "D:\Build\Test\numpy-build\numpy\testing\utils.py", line 623, in assert_array_compare chk_same_position(x_isnan, y_isnan, hasval='nan') File "D:\Build\Test\numpy-build\numpy\testing\utils.py", line 603, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not almost equal to 5 decimals func move_nansum | window 1 | input a6 (float32) | shape (4L,) | axis -1 Input array: [ nan 1. 2. 3.] x and y nan location mismatch: x: array([ nan, 1., 2., 3.], dtype=float32) y: array([ 0., 1., 2., 3.], dtype=float32) pyfits 3.2.4 ============ ====================================================================== ERROR: pyfits.tests.test_table.TestTableFunctions.test_mask_array ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\pyfits\tests\test_table.py", line 962, in test_mask_array hdu.writeto(self.temp('newtable.fits')) File "X:\Python27-x64\lib\site-packages\pyfits\hdu\base.py", line 1646, in writeto checksum=checksum) File "X:\Python27-x64\lib\site-packages\pyfits\hdu\hdulist.py", line 644, in writeto hdu._prewriteto(checksum=checksum) File "X:\Python27-x64\lib\site-packages\pyfits\hdu\table.py", line 384, in _prewriteto self.data._scale_back() File "X:\Python27-x64\lib\site-packages\pyfits\fitsrec.py", line 1015, in _scale_back dummy[idx] = val + (pad * (itemsize - len(val))) ValueError: assignment destination is read-only milk 0.5.3 ========== ====================================================================== FAIL: milk.tests.test_pdist.test_pdist ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\milk\tests\test_pdist.py", line 11, in test_pdist assert np.allclose(Dxx[i,j], np.sum((X[i]-X[j])**2)) AssertionError ====================================================================== FAIL: milk.tests.test_pdist.test_plike ---------------------------------------------------------------------- Traceback (most recent call last): File "X:\Python27-x64\lib\site-packages\nose\case.py", line 197, in runTest self.test(*self.arg) File "X:\Python27-x64\lib\site-packages\milk\tests\test_pdist.py", line 27, in test_plike assert Lxx[0,0] == Lxx2[0,0] AssertionError From charlesr.harris at gmail.com Mon Jun 9 19:29:30 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Jun 2014 17:29:30 -0600 Subject: [Numpy-discussion] Reporting a bug to Apple. Message-ID: Hi All, Julian has tracked down a bug in the Accelerate library in Maverick, details here . Is there a registered Apple Developer here who can report the bug to Apple? TIA, Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 9 20:10:43 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Jun 2014 18:10:43 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <53964195.2090605@uci.edu> References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 5:21 PM, Christoph Gohlke wrote: > On 6/8/2014 1:34 PM, Julian Taylor wrote: > >> Hello, >> >> I'm happy to announce the fist beta release of Numpy 1.9.0. >> 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 >> - 3.4. >> Due to low demand windows binaries for the beta are only available for >> Python 2.7, 3.3 and 3.4. >> Please try it and report any issues to the numpy-discussion mailing list >> or on github. >> >> The 1.9 release will consists of mainly of many small improvements and >> bugfixes. The highlights are: >> >> * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray >> subclasses. Please note that there are still some known issues with this >> mechanism which we hope to resolve before the final release (e.g. #4753) >> * Numerous performance improvements in various areas, most notably >> indexing and operations on small arrays are significantly faster. >> Indexing operations now also release the GIL. >> * Addition of nanmedian and nanpercentile rounds out the nanfunction set. >> >> The changes involve a lot of small changes that might affect some >> applications, please read the release notes for the full details on all >> changes: >> https://github.com/numpy/numpy/blob/maintenance/1.9.x/ >> doc/release/1.9.0-notes.rst >> Please also take special note of the future changes section which will >> apply to the following release 1.10.0 and make sure to check if your >> applications would be affected by them. >> >> Source tarballs, windows installers and release notes can be found at >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 >> >> Cheers, >> Julian Taylor >> >> > Hello, > > I tested numpy-MKL-1.9.0b1 (msvc9, Intel MKL build) on win-amd64-py2.7 > against a few other packages that were built against numpy-MKL-1.8.x. > > While numpy and scipy pass all tests, some other packages (matplotlib, > statsmodels, skimage, pandas, pytables, sklearn...) show a few new test > failures (compared to testing with numpy-MKL-1.8.1). Many test errors are > of kind: > > ValueError: shape mismatch: value array of shape (24,) could not be > broadcast to indexing result of shape (8,3) > > I have attached a list of failing tests. The full test results are at < > http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/20140609-win-amd64-py2.7- > numpy-1.9.0b1/> (compare to gohlke/pythonlibs/tests/20140609-win-amd64-py2.7/>) > > I have not investigated any further... > One of the matplotlib failures, and I suspect the others, comes from the assignment found_index[refi_triangles[ancestor_mask, :] ] = np.repeat(ancestors[ancestor_mask], 3) This fails with the error ValueError: shape mismatch: value array of shape (13824,) could not be broadcast to indexing result of shape (4608,3) I confess I find the construction odd, but the error probably results from stricter indexing rules. Indeed, (13824,) does not broadcast to (4608,3). Apart from considerations of backward compatibility, should it? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeffreback at gmail.com Mon Jun 9 20:21:00 2014 From: jeffreback at gmail.com (Jeff Reback) Date: Mon, 9 Jun 2014 20:21:00 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <53964195.2090605@uci.edu> References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: The one pandas test failure that is valid: ERROR: test_interp_regression (pandas.tests.test_generic.TestSeries) has been fixed in pandas master / 0.14.1 (prob releasing in 1 month). (the other test failures are for clipboard / network issues) On Mon, Jun 9, 2014 at 7:21 PM, Christoph Gohlke wrote: > On 6/8/2014 1:34 PM, Julian Taylor wrote: > >> Hello, >> >> I'm happy to announce the fist beta release of Numpy 1.9.0. >> 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 >> - 3.4. >> Due to low demand windows binaries for the beta are only available for >> Python 2.7, 3.3 and 3.4. >> Please try it and report any issues to the numpy-discussion mailing list >> or on github. >> >> The 1.9 release will consists of mainly of many small improvements and >> bugfixes. The highlights are: >> >> * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray >> subclasses. Please note that there are still some known issues with this >> mechanism which we hope to resolve before the final release (e.g. #4753) >> * Numerous performance improvements in various areas, most notably >> indexing and operations on small arrays are significantly faster. >> Indexing operations now also release the GIL. >> * Addition of nanmedian and nanpercentile rounds out the nanfunction set. >> >> The changes involve a lot of small changes that might affect some >> applications, please read the release notes for the full details on all >> changes: >> https://github.com/numpy/numpy/blob/maintenance/1.9.x/ >> doc/release/1.9.0-notes.rst >> Please also take special note of the future changes section which will >> apply to the following release 1.10.0 and make sure to check if your >> applications would be affected by them. >> >> Source tarballs, windows installers and release notes can be found at >> https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 >> >> Cheers, >> Julian Taylor >> >> > Hello, > > I tested numpy-MKL-1.9.0b1 (msvc9, Intel MKL build) on win-amd64-py2.7 > against a few other packages that were built against numpy-MKL-1.8.x. > > While numpy and scipy pass all tests, some other packages (matplotlib, > statsmodels, skimage, pandas, pytables, sklearn...) show a few new test > failures (compared to testing with numpy-MKL-1.8.1). Many test errors are > of kind: > > ValueError: shape mismatch: value array of shape (24,) could not be > broadcast to indexing result of shape (8,3) > > I have attached a list of failing tests. The full test results are at < > http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/20140609-win-amd64-py2.7- > numpy-1.9.0b1/> (compare to gohlke/pythonlibs/tests/20140609-win-amd64-py2.7/>) > > I have not investigated any further... > > Christoph > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 9 20:21:59 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Jun 2014 18:21:59 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 6:10 PM, Charles R Harris wrote: > > > > On Mon, Jun 9, 2014 at 5:21 PM, Christoph Gohlke wrote: > >> On 6/8/2014 1:34 PM, Julian Taylor wrote: >> >>> Hello, >>> >>> I'm happy to announce the fist beta release of Numpy 1.9.0. >>> 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 >>> - 3.4. >>> Due to low demand windows binaries for the beta are only available for >>> Python 2.7, 3.3 and 3.4. >>> Please try it and report any issues to the numpy-discussion mailing list >>> or on github. >>> >>> The 1.9 release will consists of mainly of many small improvements and >>> bugfixes. The highlights are: >>> >>> * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray >>> subclasses. Please note that there are still some known issues with this >>> mechanism which we hope to resolve before the final release (e.g. #4753) >>> * Numerous performance improvements in various areas, most notably >>> indexing and operations on small arrays are significantly faster. >>> Indexing operations now also release the GIL. >>> * Addition of nanmedian and nanpercentile rounds out the nanfunction set. >>> >>> The changes involve a lot of small changes that might affect some >>> applications, please read the release notes for the full details on all >>> changes: >>> https://github.com/numpy/numpy/blob/maintenance/1.9.x/ >>> doc/release/1.9.0-notes.rst >>> Please also take special note of the future changes section which will >>> apply to the following release 1.10.0 and make sure to check if your >>> applications would be affected by them. >>> >>> Source tarballs, windows installers and release notes can be found at >>> https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 >>> >>> Cheers, >>> Julian Taylor >>> >>> >> Hello, >> >> I tested numpy-MKL-1.9.0b1 (msvc9, Intel MKL build) on win-amd64-py2.7 >> against a few other packages that were built against numpy-MKL-1.8.x. >> >> While numpy and scipy pass all tests, some other packages (matplotlib, >> statsmodels, skimage, pandas, pytables, sklearn...) show a few new test >> failures (compared to testing with numpy-MKL-1.8.1). Many test errors are >> of kind: >> >> ValueError: shape mismatch: value array of shape (24,) could not be >> broadcast to indexing result of shape (8,3) >> >> I have attached a list of failing tests. The full test results are at < >> http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/20140609-win-amd64-py2.7- >> numpy-1.9.0b1/> (compare to > gohlke/pythonlibs/tests/20140609-win-amd64-py2.7/>) >> >> I have not investigated any further... >> > > One of the matplotlib failures, and I suspect the others, comes from the > assignment > > found_index[refi_triangles[ancestor_mask, :] > ] = np.repeat(ancestors[ancestor_mask], 3) > > This fails with the error > > ValueError: shape mismatch: value array of shape (13824,) > > could not be broadcast to indexing result of shape (4608,3) > > I confess I find the construction odd, but the error probably results from > stricter indexing rules. Indeed, (13824,) does not broadcast to (4608,3). > Apart from considerations of backward compatibility, should it? > > Other errors are of the type: TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensionalinput, input has 2 dimensions This one looks to arise is from stricter rules for boolean indexing. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 9 20:23:46 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Jun 2014 18:23:46 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 6:21 PM, Jeff Reback wrote: > The one pandas test failure that is valid: ERROR: test_interp_regression (pandas.tests.test_generic.TestSeries) > > has been fixed in pandas master / 0.14.1 (prob releasing in 1 month). > > (the other test failures are for clipboard / network issues) > > > > Thanks for the feedback. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From nouiz at nouiz.org Mon Jun 9 20:27:43 2014 From: nouiz at nouiz.org (=?UTF-8?B?RnLDqWTDqXJpYyBCYXN0aWVu?=) Date: Mon, 9 Jun 2014 20:27:43 -0400 Subject: [Numpy-discussion] Reporting a bug to Apple. In-Reply-To: References: Message-ID: Hi, We already did a bug report to Apple, but they didn't acted on this yet. A process call the kernel and it loop infinitly in the kernel. The only way to "kill" the process is to reboot. Arnaud, how did you report it? Good luck, and if they act on this, I would be happy to know how you did. Fr?d?ric Bastien On Mon, Jun 9, 2014 at 7:29 PM, Charles R Harris wrote: > Hi All, > > Julian has tracked down a bug in the Accelerate library in Maverick, > details here > . Is > there a registered Apple Developer here who can report the bug to Apple? > > TIA, > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Jun 9 20:37:33 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Jun 2014 00:37:33 +0000 (UTC) Subject: [Numpy-discussion] NumPy 1.9.0 beta release References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: <359412406424053347.417575sturla.molden-gmail.com@news.gmane.org> Charles R Harris wrote: > I confess I find the construction odd, but the error probably results from > stricter indexing rules. Indeed, (13824,) does not broadcast to (4608,3). > Apart from considerations of backward compatibility, should it? Probably not. (But breaking matplotlib is bad.) From charlesr.harris at gmail.com Mon Jun 9 20:47:56 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Jun 2014 18:47:56 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 6:23 PM, Charles R Harris wrote: > > > > On Mon, Jun 9, 2014 at 6:21 PM, Jeff Reback wrote: > >> The one pandas test failure that is valid: ERROR: test_interp_regression (pandas.tests.test_generic.TestSeries) >> >> has been fixed in pandas master / 0.14.1 (prob releasing in 1 month). >> >> (the other test failures are for clipboard / network issues) >> >> >> >> > Thanks for the feedback. > > > > Looks to me like a fair number of the other test failures are actually bugs in the tests, or could be argued to be so. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Jun 9 20:49:57 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 9 Jun 2014 18:49:57 -0600 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 6:47 PM, Charles R Harris wrote: > > > > On Mon, Jun 9, 2014 at 6:23 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Mon, Jun 9, 2014 at 6:21 PM, Jeff Reback wrote: >> >>> The one pandas test failure that is valid: ERROR: test_interp_regression (pandas.tests.test_generic.TestSeries) >>> >>> has been fixed in pandas master / 0.14.1 (prob releasing in 1 month). >>> >>> (the other test failures are for clipboard / network issues) >>> >>> >>> >>> >> Thanks for the feedback. >> >> >> >> > Looks to me like a fair number of the other test failures are actually > bugs in the tests, or could be argued to be so. > Or rather, actual bugs, but not in numpy. I can see this will take a while to settle ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Mon Jun 9 20:52:21 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Jun 2014 00:52:21 +0000 (UTC) Subject: [Numpy-discussion] Reporting a bug to Apple. References: Message-ID: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Fr?d?ric Bastien wrote: OSes > Good luck, and if they act on this, I would be happy to know how you did. > If we report a segfault Apple will probably think we are to blame. 99.9 % of bug reports come from idiots, and those who screen bug reports are basically payed to hit the ignore button. Indeed, good luck on reporting it. I think in the short run we will be better off re-implementing cblas_sgemv with cblas_sgemm on Apple OSes. Luckily the error in cblas_sgemv does not silently produce garbage data but segfaults the process, so anyone affected will see it. Sturla From josef.pktd at gmail.com Mon Jun 9 23:10:52 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Jun 2014 23:10:52 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 8:49 PM, Charles R Harris wrote: > > > > On Mon, Jun 9, 2014 at 6:47 PM, Charles R Harris > wrote: >> >> >> >> >> On Mon, Jun 9, 2014 at 6:23 PM, Charles R Harris >> wrote: >>> >>> >>> >>> >>> On Mon, Jun 9, 2014 at 6:21 PM, Jeff Reback wrote: >>>> >>>> The one pandas test failure that is valid: ERROR: test_interp_regression >>>> (pandas.tests.test_generic.TestSeries) >>>> >>>> has been fixed in pandas master / 0.14.1 (prob releasing in 1 month). >>>> >>>> (the other test failures are for clipboard / network issues) >>>> >>>> >>>> >>> >>> Thanks for the feedback. >>> >>> >>> >> >> Looks to me like a fair number of the other test failures are actually >> bugs in the tests, or could be argued to be so. > > > Or rather, actual bugs, but not in numpy. I can see this will take a while > to settle ;) not really a bug on our side given the old numpy behavior I was surprised about the failure because it still produces the correct result. params[nuis_param_index] = nuisance_params ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (0,) As far as I have figured out based on Christoph's test failures `nuis_param_index` is array([], dtype=int32) and because numpy didn't assign anything, `nuisance_params` was picked essentially arbitrary along this code path >>> x = np.arange(5.) >>> we_dont_care_what_this_is = np.random.randn(10) >>> x[[]] = we_dont_care_what_this_is >>> x array([ 0., 1., 2., 3., 4.]) Does x[[]] = [] work with numpy 1.9? Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Mon Jun 9 23:13:41 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 9 Jun 2014 23:13:41 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: On Mon, Jun 9, 2014 at 11:10 PM, wrote: > On Mon, Jun 9, 2014 at 8:49 PM, Charles R Harris > wrote: >> >> >> >> On Mon, Jun 9, 2014 at 6:47 PM, Charles R Harris >> wrote: >>> >>> >>> >>> >>> On Mon, Jun 9, 2014 at 6:23 PM, Charles R Harris >>> wrote: >>>> >>>> >>>> >>>> >>>> On Mon, Jun 9, 2014 at 6:21 PM, Jeff Reback wrote: >>>>> >>>>> The one pandas test failure that is valid: ERROR: test_interp_regression >>>>> (pandas.tests.test_generic.TestSeries) >>>>> >>>>> has been fixed in pandas master / 0.14.1 (prob releasing in 1 month). >>>>> >>>>> (the other test failures are for clipboard / network issues) >>>>> >>>>> >>>>> >>>> >>>> Thanks for the feedback. >>>> >>>> >>>> >>> >>> Looks to me like a fair number of the other test failures are actually >>> bugs in the tests, or could be argued to be so. >> >> >> Or rather, actual bugs, but not in numpy. I can see this will take a while >> to settle ;) > > not really a bug on our side given the old numpy behavior forgot to define: our side = statsmodels > I was surprised about the failure because it still produces the correct result. > > params[nuis_param_index] = nuisance_params > ValueError: shape mismatch: value array of shape (2,) could not be > broadcast to indexing result of shape (0,) > > As far as I have figured out based on Christoph's test failures > > `nuis_param_index` is array([], dtype=int32) > and because numpy didn't assign anything, `nuisance_params` was picked > essentially arbitrary along this code path > >>>> x = np.arange(5.) >>>> we_dont_care_what_this_is = np.random.randn(10) >>>> x[[]] = we_dont_care_what_this_is >>>> x > array([ 0., 1., 2., 3., 4.]) > > Does x[[]] = [] work with numpy 1.9? > > Josef > >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> From sebastian at sipsolutions.net Tue Jun 10 04:43:44 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 10 Jun 2014 10:43:44 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: <1402389824.4487.3.camel@sebastian-t440> On Mo, 2014-06-09 at 18:21 -0600, Charles R Harris wrote: > > > > On Mon, Jun 9, 2014 at 6:10 PM, Charles R Harris > wrote: > > > > > > Other errors are of the type: > TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensionalinput, input has 2 dimensions > This one looks to arise is from stricter rules for boolean indexing. > Hmmmm, I may have removed an "if size of the boolean index matches the size of the output array ignore the shape (dimensions)" special cased, I guess we can put that in again. The other error looks a bit different because of the nonzero logic, but probably is the same, i.e. also boolean indexing. The last one is the change that `arr[[1,2,3,4]] = [1,2]` does not work anymore. A workaround (maybe also for the rest possibly) is `arr.flat[[1,2,3,4]] = [1,2]`, but I guess workarounds are not an option with matplotlib, so have to think about it. - Sebastian > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From sebastian at sipsolutions.net Tue Jun 10 04:55:32 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 10 Jun 2014 10:55:32 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <1402389824.4487.3.camel@sebastian-t440> References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> <1402389824.4487.3.camel@sebastian-t440> Message-ID: <1402390532.4487.5.camel@sebastian-t440> On Di, 2014-06-10 at 10:43 +0200, Sebastian Berg wrote: > On Mo, 2014-06-09 at 18:21 -0600, Charles R Harris wrote: > > > > > > > > On Mon, Jun 9, 2014 at 6:10 PM, Charles R Harris > > wrote: > > > > > > > > > > > > > > Other errors are of the type: > > TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensionalinput, input has 2 dimensions > > This one looks to arise is from stricter rules for boolean indexing. > > > > Hmmmm, I may have removed an "if size of the boolean index matches the > size of the output array ignore the shape (dimensions)" special cased, I > guess we can put that in again. > > The other error looks a bit different because of the nonzero logic, but > probably is the same, i.e. also boolean indexing. The last one is the > change that `arr[[1,2,3,4]] = [1,2]` does not work anymore. A workaround > (maybe also for the rest possibly) is `arr.flat[[1,2,3,4]] = [1,2]`, but > I guess workarounds are not an option with matplotlib, so have to think > about it. Correction: I think all (indexing) errors are the second case since the boolean special case is not taken. > > - Sebastian > > > Chuck > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From njs at pobox.com Tue Jun 10 05:50:17 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 10 Jun 2014 10:50:17 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <1402389824.4487.3.camel@sebastian-t440> References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> <1402389824.4487.3.camel@sebastian-t440> Message-ID: On 10 Jun 2014 09:44, "Sebastian Berg" wrote: > The other error looks a bit different because of the nonzero logic, but > probably is the same, i.e. also boolean indexing. The last one is the > change that `arr[[1,2,3,4]] = [1,2]` does not work anymore. A workaround > (maybe also for the rest possibly) is `arr.flat[[1,2,3,4]] = [1,2]`, but > I guess workarounds are not an option with matplotlib, so have to think > about it. If the beta is breaking code then let's put the change off to 1.10 or so and raise a deprecation warning in 1.9. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jun 10 06:15:24 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 10 Jun 2014 12:15:24 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> <1402389824.4487.3.camel@sebastian-t440> Message-ID: <1402395324.5333.0.camel@sebastian-t440> On Di, 2014-06-10 at 10:50 +0100, Nathaniel Smith wrote: > On 10 Jun 2014 09:44, "Sebastian Berg" > wrote: > > The other error looks a bit different because of the nonzero logic, > but > > probably is the same, i.e. also boolean indexing. The last one is > the > > change that `arr[[1,2,3,4]] = [1,2]` does not work anymore. A > workaround > > (maybe also for the rest possibly) is `arr.flat[[1,2,3,4]] = [1,2]`, > but > > I guess workarounds are not an option with matplotlib, so have to > think > > about it. > > If the beta is breaking code then let's put the change off to 1.10 or > so and raise a deprecation warning in 1.9. > Yes, unfortunately it is a bit more complicating than that. > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From njs at pobox.com Tue Jun 10 06:24:41 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 10 Jun 2014 11:24:41 +0100 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <1402395324.5333.0.camel@sebastian-t440> References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> <1402389824.4487.3.camel@sebastian-t440> <1402395324.5333.0.camel@sebastian-t440> Message-ID: On 10 Jun 2014 11:15, "Sebastian Berg" wrote: > > On Di, 2014-06-10 at 10:50 +0100, Nathaniel Smith wrote: > > On 10 Jun 2014 09:44, "Sebastian Berg" > > wrote: > > > The other error looks a bit different because of the nonzero logic, > > but > > > probably is the same, i.e. also boolean indexing. The last one is > > the > > > change that `arr[[1,2,3,4]] = [1,2]` does not work anymore. A > > workaround > > > (maybe also for the rest possibly) is `arr.flat[[1,2,3,4]] = [1,2]`, > > but > > > I guess workarounds are not an option with matplotlib, so have to > > think > > > about it. > > > > If the beta is breaking code then let's put the change off to 1.10 or > > so and raise a deprecation warning in 1.9. > > > > Yes, unfortunately it is a bit more complicating than that. Is it impossible to emulate the old arr[[1, 2, 3, 4]] = [1, 2] behavior for some reason? Or what do you mean? (I'm not suggesting we literally go back to the 1.8 indexing code.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Tue Jun 10 07:04:34 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Tue, 10 Jun 2014 13:04:34 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> <1402389824.4487.3.camel@sebastian-t440> <1402395324.5333.0.camel@sebastian-t440> Message-ID: <1402398274.5333.2.camel@sebastian-t440> On Di, 2014-06-10 at 11:24 +0100, Nathaniel Smith wrote: > On 10 Jun 2014 11:15, "Sebastian Berg" > wrote: > > > > On Di, 2014-06-10 at 10:50 +0100, Nathaniel Smith wrote: > > > On 10 Jun 2014 09:44, "Sebastian Berg" > > > > wrote: > > > > The other error looks a bit different because of the nonzero > logic, > > > but > > > > probably is the same, i.e. also boolean indexing. The last one > is > > > the > > > > change that `arr[[1,2,3,4]] = [1,2]` does not work anymore. A > > > workaround > > > > (maybe also for the rest possibly) is `arr.flat[[1,2,3,4]] = > [1,2]`, > > > but > > > > I guess workarounds are not an option with matplotlib, so have > to > > > think > > > > about it. > > > > > > If the beta is breaking code then let's put the change off to 1.10 > or > > > so and raise a deprecation warning in 1.9. > > > > > > > Yes, unfortunately it is a bit more complicating than that. > > Is it impossible to emulate the old arr[[1, 2, 3, 4]] = [1, 2] > behavior for some reason? Or what do you mean? (I'm not suggesting we > literally go back to the 1.8 indexing code.) > Yeah, just have to check carefully. Maybe easiest is to just try/except it in C-code, throw a warning, and (if applicable) try calling the `arr.flat[...] = ...` code (which still exists). - Sebastian > -n > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From smudkavi at uwaterloo.ca Tue Jun 10 08:38:37 2014 From: smudkavi at uwaterloo.ca (Sankarshan Mudkavi) Date: Tue, 10 Jun 2014 08:38:37 -0400 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: References: <5394C8EB.3020404@googlemail.com> Message-ID: <99681E1F-7421-4A2F-9728-B1800A671DCA@uwaterloo.ca> Still working on it, it's unfortunately taking much more time than I'd anticipated. Cheers, Sankarshan Mudkavi On Jun 9, 2014, at 12:10 PM, Chris Barker wrote: > I take nothing ever happened to clean up the datetime64 timezone mess? > > sigh. > > -Chris > > > > On Sun, Jun 8, 2014 at 1:34 PM, Julian Taylor wrote: > Hello, > > I'm happy to announce the fist beta release of Numpy 1.9.0. > 1.9.0 will be a new feature release supporting Python 2.6 - 2.7 and 3.2 > - 3.4. > Due to low demand windows binaries for the beta are only available for > Python 2.7, 3.3 and 3.4. > Please try it and report any issues to the numpy-discussion mailing list > or on github. > > The 1.9 release will consists of mainly of many small improvements and > bugfixes. The highlights are: > > * Addition of __numpy_ufunc__ to allow overriding ufuncs in ndarray > subclasses. Please note that there are still some known issues with this > mechanism which we hope to resolve before the final release (e.g. #4753) > * Numerous performance improvements in various areas, most notably > indexing and operations on small arrays are significantly faster. > Indexing operations now also release the GIL. > * Addition of nanmedian and nanpercentile rounds out the nanfunction set. > > The changes involve a lot of small changes that might affect some > applications, please read the release notes for the full details on all > changes: > https://github.com/numpy/numpy/blob/maintenance/1.9.x/doc/release/1.9.0-notes.rst > Please also take special note of the future changes section which will > apply to the following release 1.10.0 and make sure to check if your > applications would be affected by them. > > Source tarballs, windows installers and release notes can be found at > https://sourceforge.net/projects/numpy/files/NumPy/1.9.0b1 > > Cheers, > Julian Taylor > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sankarshan Mudkavi Undergraduate in Physics, University of Waterloo www.smudkavi.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From matthew.brett at gmail.com Tue Jun 10 08:57:54 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 10 Jun 2014 13:57:54 +0100 Subject: [Numpy-discussion] Reporting a bug to Apple. In-Reply-To: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Message-ID: Hi, On Tue, Jun 10, 2014 at 1:52 AM, Sturla Molden wrote: > Fr?d?ric Bastien wrote: > OSes >> Good luck, and if they act on this, I would be happy to know how you did. >> > > If we report a segfault Apple will probably think we are to blame. 99.9 % > of bug reports come from idiots, and those who screen bug reports are > basically payed to hit the ignore button. Indeed, good luck on reporting > it. > > I think in the short run we will be better off re-implementing cblas_sgemv > with cblas_sgemm on Apple OSes. Luckily the error in cblas_sgemv does not > silently produce garbage data but segfaults the process, so anyone affected > will see it. Would you consider doing a PR for that? I believe I can build a 32 / 64 bit numpy binary against ATLAS, but it's certainly much less straightforward. Cheers, Matthew From sturla.molden at gmail.com Tue Jun 10 09:33:07 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Jun 2014 13:33:07 +0000 (UTC) Subject: [Numpy-discussion] Reporting a bug to Apple. References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Message-ID: <602413533424099434.774170sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Would you consider doing a PR for that? Yes, I started on it yesterday. ;-) Sturla From aron at ahmadia.net Tue Jun 10 09:46:38 2014 From: aron at ahmadia.net (Aron Ahmadia) Date: Tue, 10 Jun 2014 09:46:38 -0400 Subject: [Numpy-discussion] Reporting a bug to Apple. In-Reply-To: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Message-ID: On Mon, Jun 9, 2014 at 8:52 PM, Sturla Molden wrote: > > If we report a segfault Apple will probably think we are to blame. 99.9 % > of bug reports come from idiots, and those who screen bug reports are > basically payed to hit the ignore button. Indeed, good luck on reporting > it. I've reported bugs to Apple and seen them fixed in major releases of the operating system (though not necessarily minor releases). Unless things have changed considerably since my last bug report, I don't think it's fair to characterize their quality assurance team this way, particularly without evidence to back up your point. A -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Tue Jun 10 13:02:01 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Tue, 10 Jun 2014 19:02:01 +0200 Subject: [Numpy-discussion] Reporting a bug to Apple. In-Reply-To: References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Message-ID: <53973A09.5030202@googlemail.com> On 10.06.2014 15:46, Aron Ahmadia wrote: > > On Mon, Jun 9, 2014 at 8:52 PM, Sturla Molden > wrote: > > > If we report a segfault Apple will probably think we are to blame. > 99.9 % > of bug reports come from idiots, and those who screen bug reports are > basically payed to hit the ignore button. Indeed, good luck on reporting > it. > > > I've reported bugs to Apple and seen them fixed in major releases of the > operating system (though not necessarily minor releases). Unless things > have changed considerably since my last bug report, I don't think it's > fair to characterize their quality assurance team this way, particularly > without evidence to back up your point. > I'm also optimistic they'll fix it. E.g. they also fixed the -mno-fused-madd screwup they did with their python installation in 10.9 in the minor release 10.9.3. So with a little luck 10.9.4 will already have the fix. The bug has now been filed by Matthew, thanks for that. From sturla.molden at gmail.com Tue Jun 10 17:48:08 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 10 Jun 2014 23:48:08 +0200 Subject: [Numpy-discussion] Patch for accelerate cblas_sgemv (Re: Reporting a bug to Apple) In-Reply-To: References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Message-ID: On 10/06/14 14:57, Matthew Brett wrote: > Would you consider doing a PR for that? Here is a patch you can try before I post a PR. It can also be build independently of NumPy, so you don't need to rebuild NumPy just for testing it. (The change to numpy is in a different folder.) I decided against using cblas_sgemm. Instead it just enforces alignment to 32 byte boundaries. Because cblas_sgemm would require a copy if the vector is strided, it didn't matter. I have tested with Accelerate, OpenBLAS and MKL, clang and icc. From what I can tell it works correctly and does not segfault on misalignment. Sturla -------------- next part -------------- A non-text attachment was scrubbed... Name: sgemv_patch.zip Type: application/zip Size: 54470 bytes Desc: not available URL: From sturla.molden at gmail.com Tue Jun 10 20:57:37 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 11 Jun 2014 02:57:37 +0200 Subject: [Numpy-discussion] Patch for accelerate cblas_sgemv (Re: Reporting a bug to Apple) In-Reply-To: References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> Message-ID: On 10/06/14 23:48, Sturla Molden wrote: > Here is a patch you can try before I post a PR. A slightly cleaned up version. Sturla -------------- next part -------------- A non-text attachment was scrubbed... Name: sgemv_patch.zip Type: application/zip Size: 54389 bytes Desc: not available URL: From kyle.mandli at gmail.com Wed Jun 11 15:41:22 2014 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Wed, 11 Jun 2014 14:41:22 -0500 Subject: [Numpy-discussion] Reporting a bug to Apple. In-Reply-To: <53973A09.5030202@googlemail.com> References: <840115269424053607.630244sturla.molden-gmail.com@news.gmane.org> <53973A09.5030202@googlemail.com> Message-ID: I also have filed a bug report with apple in the past to fix an accelerate bug. The key I think was to clearly demonstrate and isolate the problem like any good bug report should have. In my case, they got back to me within a week and the bug was fixed a week after that. The release schedule is a bit difficult however but we should file one in any case, especially with Yosemite being under active testing and 10.9.4 in the testing stages. Kyle On Tue, Jun 10, 2014 at 12:02 PM, Julian Taylor < jtaylor.debian at googlemail.com> wrote: > On 10.06.2014 15:46, Aron Ahmadia wrote: > > > > On Mon, Jun 9, 2014 at 8:52 PM, Sturla Molden > > wrote: > > > > > > If we report a segfault Apple will probably think we are to blame. > > 99.9 % > > of bug reports come from idiots, and those who screen bug reports are > > basically payed to hit the ignore button. Indeed, good luck on > reporting > > it. > > > > > > I've reported bugs to Apple and seen them fixed in major releases of the > > operating system (though not necessarily minor releases). Unless things > > have changed considerably since my last bug report, I don't think it's > > fair to characterize their quality assurance team this way, particularly > > without evidence to back up your point. > > > > > I'm also optimistic they'll fix it. E.g. they also fixed the > -mno-fused-madd screwup they did with their python installation in 10.9 > in the minor release 10.9.3. So with a little luck 10.9.4 will already > have the fix. > > The bug has now been filed by Matthew, thanks for that. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian at sipsolutions.net Thu Jun 12 04:28:51 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Thu, 12 Jun 2014 10:28:51 +0200 Subject: [Numpy-discussion] ANN: NumPy 1.9.0 beta release In-Reply-To: <53964195.2090605@uci.edu> References: <5394C8EB.3020404@googlemail.com> <53964195.2090605@uci.edu> Message-ID: <1402561731.15767.3.camel@sebastian-t440> On Mo, 2014-06-09 at 16:21 -0700, Christoph Gohlke wrote: > On 6/8/2014 1:34 PM, Julian Taylor wrote: > > Hello, > > > > Hello, > > I tested numpy-MKL-1.9.0b1 (msvc9, Intel MKL build) on win-amd64-py2.7 > against a few other packages that were built against numpy-MKL-1.8.x. > > While numpy and scipy pass all tests, some other packages (matplotlib, > statsmodels, skimage, pandas, pytables, sklearn...) show a few new test > failures (compared to testing with numpy-MKL-1.8.1). Many test errors > are of kind: > > ValueError: shape mismatch: value array of shape (24,) could not be > broadcast to indexing result of shape (8,3) > > I have attached a list of failing tests. The full test results are at > > (compare to > ) > > I have not investigated any further... > I have put up a sketch for the indexing fix at https://github.com/numpy/numpy/pull/4804 not sure when I will have time to finish it off, so if someone has, go ahead ;). It should fix all those indexing errors (though numpy has harmless test failures with it currently). Deprecationwarnings (and not there yet FutureWarnings for the error type change) should be given then. - Sebastian > Christoph > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From matthew.brett at gmail.com Fri Jun 13 08:07:12 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 13 Jun 2014 13:07:12 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels Message-ID: Hi, Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework. We've run into some trouble with a segfault in recent OSX Accelerate: https://github.com/numpy/numpy/issues/4007 and Accelerate also doesn't play well with multiprocessing. https://github.com/numpy/numpy/issues/4776 Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas: https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 The build process needs some automating, but it's recorded here: https://github.com/matthew-brett/numpy-atlas-binaries It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that. Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing. I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that? Cheers, Matthew [1] http://mail.scipy.org/pipermail/numpy-discussion/2014-June/070333.html From sturla.molden at gmail.com Fri Jun 13 09:03:12 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 13 Jun 2014 13:03:12 +0000 (UTC) Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels References: Message-ID: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Matthew Brett wrote: > Meanwhile Sturla kindly worked up a patch to numpy to work round the > Accelerate segfault [1]. I haven't tested that, but given I'd already > built the wheels, I prefer the ATLAS builds because they work with > multiprocessing. It is an ugly hack. If it is used it would be better to keep it in a separate branch rather than in NumPy master. I am not able to post a PR because my fork of NumPy is unavailable on GitHub. > I propose uploading these wheels as the default for numpy and scipy. > Does anyone have any objection or comments before I go ahead and do > that? ATLAS should be fine. Performance will not be anything like the alternatives (Accelerate and OpenBLAS), but correctness is more important. I am not sure why ATLAS is preferred over OpenBLAS, though, but I don't really care. Perhaps we could ask Intel permission to use MKL in the binary wheels? The Accelerate vs. multiptocessing issue will never be fixed for Python 2.7 and poissibly Python 3.3 (it is fixed in Python 3.4). The error is in multiprocessing, so Apple will not patch Accelerate. New features will not be added to Python 2.7 standard lib, so there is no chance of getting the required fix backported there either. Stalemate. Sturla From ralf.gommers at gmail.com Fri Jun 13 11:18:10 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 13 Jun 2014 17:18:10 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> References: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Jun 13, 2014 at 3:03 PM, Sturla Molden wrote: > > Perhaps we could ask Intel permission to use MKL in the binary wheels? We have already asked and obtained that permission, under the condition that we put some attribution to Intel MKL on our website (which we already have at http://scipy.org/scipylib/donations.html). I would not be in favor of distributing only MKL binaries, but distributing those in addition to ATLAS/Accelerate ones would be fine. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Jun 13 11:53:48 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 13 Jun 2014 08:53:48 -0700 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Jun 13, 2014 at 8:18 AM, Ralf Gommers wrote: > We have already asked and obtained that permission, under the condition > that we put some attribution to Intel MKL on our website (which we already > have at http://scipy.org/scipylib/donations.html). I would not be in > favor of distributing only MKL binaries, but distributing those in addition > to ATLAS/Accelerate ones would be fine. > I'm curious, why not? But in any case, if the numpy project itself is distributing more than one binary type, then we still need to decide what goes on PyPy. I'm confused about the whole pip optional features thing, but could you put up both and have folks access it with, for example: pip install numpy[mkl] ? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Jun 13 11:55:35 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 13 Jun 2014 17:55:35 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett wrote: > Hi, > > Summary : I'm planning to upload OSX wheels for numpy and scipy using > the ATLAS blas / lapack library instead of the default OSX Accelerate > framework. > > We've run into some trouble with a segfault in recent OSX Accelerate: > > https://github.com/numpy/numpy/issues/4007 > > and Accelerate also doesn't play well with multiprocessing. > > https://github.com/numpy/numpy/issues/4776 > > Because there's nothing I love more than half-day compilation runs on > my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy > and scipy to them to make OSX wheels. These pass all tests in i386 > and x86_64 mode, including numpy, scipy, matplotlib, pandas: > > https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 > > The build process needs some automating, but it's recorded here: > > https://github.com/matthew-brett/numpy-atlas-binaries > > It's possible to get travis-ci to build these guys from a bare machine > and then upload them somewhere, but I haven't tried to do that. > > Meanwhile Sturla kindly worked up a patch to numpy to work round the > Accelerate segfault [1]. I haven't tested that, but given I'd already > built the wheels, I prefer the ATLAS builds because they work with > multiprocessing. > > I propose uploading these wheels as the default for numpy and scipy. > Does anyone have any objection or comments before I go ahead and do > that? > >From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right? Ralf $ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE ====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so ... <42 more errors> ... > Cheers, > > Matthew > > [1] http://mail.scipy.org/pipermail/numpy-discussion/2014-June/070333.html > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Jun 13 12:00:52 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 13 Jun 2014 18:00:52 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Message-ID: On Fri, Jun 13, 2014 at 5:53 PM, Chris Barker wrote: > On Fri, Jun 13, 2014 at 8:18 AM, Ralf Gommers > wrote: > >> We have already asked and obtained that permission, under the condition >> that we put some attribution to Intel MKL on our website (which we already >> have at http://scipy.org/scipylib/donations.html). I would not be in >> favor of distributing only MKL binaries, but distributing those in addition >> to ATLAS/Accelerate ones would be fine. >> > > I'm curious, why not? > Because I think that we should provide a set of binaries that people can reproduce without needing to buy/obtain licenses. Good for debugging, rotating release manager duties, etc. > But in any case, if the numpy project itself is distributing more than one > binary type, then we still need to decide what goes on PyPy. I'm confused > about the whole pip optional features thing, but could you put up both and > have folks access it with, for example: > > pip install numpy[mkl] > ? > I think that's possible. Ralf > > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Fri Jun 13 12:12:11 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 13 Jun 2014 16:12:11 +0000 (UTC) Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels References: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Message-ID: <560576211424367984.909724sturla.molden-gmail.com@news.gmane.org> Ralf Gommers wrote: > We have already asked and obtained that permission, under the condition > that we put some attribution to Intel MKL on our website (which we already > have at href="http://scipy.org/scipylib/donations.html">http://scipy.org/scipylib/donations.html). > I would not be in favor > of distributing only MKL binaries, but distributing those in addition to > ATLAS/Accelerate ones would be fine. That is great news :) If we can distribute MKL and ATLAS binaries it will make everyone happy. I suppose this applies to Windows as well? Sturla From sturla.molden at gmail.com Fri Jun 13 12:36:01 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Fri, 13 Jun 2014 16:36:01 +0000 (UTC) Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels References: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Message-ID: <1663008177424369530.865409sturla.molden-gmail.com@news.gmane.org> Chris Barker wrote: > I'm curious, why not? Because an MKL license is required to redistribute MKL. If someone wants to include the binaries in their product they must acquire a license. An MKL-based binary wheel would be for end-users that wants to install and use NumPy. It would not be for those who redistribute NumPy as a component of a larger application. ATLAS and OpenBLAS are free, and Accelerate Framework is a part of Apple's operating systems. Those binaries can be redistributed by third parties without Intel's licensisng restrictions. But if you are an end-user installing NumPy for your own work you'd want the MKL binary wheel instead. Sturla From davidmenhur at gmail.com Fri Jun 13 12:55:07 2014 From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=) Date: Fri, 13 Jun 2014 18:55:07 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: <1499611128424356329.772711sturla.molden-gmail.com@news.gmane.org> Message-ID: On 13 June 2014 18:00, Ralf Gommers wrote: > pip install numpy[mkl] >> ? >> > > I think that's possible. > MKL are fairly famous, but perhaps it would be legally safer to use [mkl-nonfree] (or something of the sort) to signal the licence. But maybe I am bikeshedding here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtaylor.debian at googlemail.com Fri Jun 13 13:45:52 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Fri, 13 Jun 2014 19:45:52 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: <539B38D0.3010004@googlemail.com> On 13.06.2014 14:07, Matthew Brett wrote: > Hi, > > Summary : I'm planning to upload OSX wheels for numpy and scipy using > the ATLAS blas / lapack library instead of the default OSX Accelerate > framework. > hi, thanks for doing this. Have you built a generic atlas binary? atlas tunes it self to the capabilities of the machine its building on. So if the machine used to create be binary is newer than the oldest mac machine we want to support it can produce errors due to the use of illegal instructions (avx/sse4 etc.) So far I know the unknown machine type will produce a somewhat generic binary (but not for ARM). From nadavh at visionsense.com Sat Jun 14 02:11:39 2014 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 14 Jun 2014 06:11:39 +0000 Subject: [Numpy-discussion] numpy 1.9b1 bug in pad function? Message-ID: <1402726075014.19216@visionsense.com> In [1]: import numpy as np In [2]: a = np.arange(4) In [3]: np.pad(a,2) --------------------------------------------------------------------------- ValueError??????????????????????????????? Traceback (most recent call last) in () ----> 1 np.pad(a,2) /usr/lib64/python3.3/site-packages/numpy/lib/arraypad.py in pad(array, pad_width, mode, **kwargs) ?? 1331???? elif mode is None: ?? 1332???????? raise ValueError('Keyword "mode" must be a function or one of %s.' % -> 1333????????????????????????? (list(allowedkwargs.keys()),)) ?? 1334???? else: ?? 1335???????? # Drop back to old, slower np.apply_along_axis mode for user-supplied ValueError: Keyword "mode" must be a function or one of ['edge', 'constant', 'wrap', 'reflect', 'median', 'maximum', 'minimum', 'symmetric', 'linear_ramp', 'mean']. In [4]: np.__version__ Out[4]: '1.9.0b1' The documentation specify that the mode parameter is optional I am getting the same for both python 2.7 and 3.3 OS: Gentoo linux Nadav From matthew.brett at gmail.com Sat Jun 14 05:27:24 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Jun 2014 10:27:24 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: <539B38D0.3010004@googlemail.com> References: <539B38D0.3010004@googlemail.com> Message-ID: On Friday, June 13, 2014, Julian Taylor wrote: > On 13.06.2014 14:07, Matthew Brett wrote: > > Hi, > > > > Summary : I'm planning to upload OSX wheels for numpy and scipy using > > the ATLAS blas / lapack library instead of the default OSX Accelerate > > framework. > > > > hi, > thanks for doing this. > > Have you built a generic atlas binary? > atlas tunes it self to the capabilities of the machine its building on. > So if the machine used to create be binary is newer than the oldest mac > machine we want to support it can produce errors due to the use of > illegal instructions (avx/sse4 etc.) > So far I know the unknown machine type will produce a somewhat generic > binary (but not for ARM). > _______________________________________ > Sorry am traveling and dont have the reference to hand, but all intel macs have sse2. So I built the atlas binaries with sse2 only... Cheers, Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Sat Jun 14 05:56:38 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 14 Jun 2014 10:56:38 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: On Friday, June 13, 2014, Ralf Gommers wrote: > > > > On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett > wrote: > >> Hi, >> >> Summary : I'm planning to upload OSX wheels for numpy and scipy using >> the ATLAS blas / lapack library instead of the default OSX Accelerate >> framework. >> >> We've run into some trouble with a segfault in recent OSX Accelerate: >> >> https://github.com/numpy/numpy/issues/4007 >> >> and Accelerate also doesn't play well with multiprocessing. >> >> https://github.com/numpy/numpy/issues/4776 >> >> Because there's nothing I love more than half-day compilation runs on >> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >> and scipy to them to make OSX wheels. These pass all tests in i386 >> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >> >> >> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >> >> The build process needs some automating, but it's recorded here: >> >> https://github.com/matthew-brett/numpy-atlas-binaries >> >> It's possible to get travis-ci to build these guys from a bare machine >> and then upload them somewhere, but I haven't tried to do that. >> >> Meanwhile Sturla kindly worked up a patch to numpy to work round the >> Accelerate segfault [1]. I haven't tested that, but given I'd already >> built the wheels, I prefer the ATLAS builds because they work with >> multiprocessing. >> >> I propose uploading these wheels as the default for numpy and scipy. >> Does anyone have any objection or comments before I go ahead and do >> that? >> > > From your README and wscript I don't see what numpy version you're using > to compile scipy against. I got the impression that you used 1.8.1, but it > should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. > > I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see > below). Your wheels should work with all common Python installs (mine is > homebrew) right? > > Ralf > > > $ python2.7 -c "import scipy; scipy.test()" > Running unit tests for scipy > NumPy version 1.9.0.dev-056ab73 > NumPy is installed in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy > SciPy version 0.14.0 > SciPy is installed in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy > Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 > Compatible Apple LLVM 4.2 (clang-425.0.28)] > nose version 1.3.0 > E...............EEEEEE............EEEEEEEEEE > ====================================================================== > ERROR: Failure: ImportError > (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, > 2): Symbol not found: _PyModule_Create2 > Referenced from: > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so > Expected in: flat namespace > in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", > line 413, in loadTestsFromName > addr.filename, addr.module) > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", > line 27, in > from . import vq, hierarchy > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", > line 175, in > from . import _hierarchy_wrap > ImportError: > dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, > 2): Symbol not found: _PyModule_Create2 > Referenced from: > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so > Expected in: flat namespace > in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so > > ... > <42 more errors> > That is strange homebrew is one of tests in the grid and the installation path looks strange. Can you try downloading the wheel from the url and installing from the local file? Cheers, Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at sun.ac.za Sat Jun 14 06:39:12 2014 From: stefan at sun.ac.za (=?UTF-8?Q?St=C3=A9fan_van_der_Walt?=) Date: Sat, 14 Jun 2014 12:39:12 +0200 Subject: [Numpy-discussion] numpy 1.9b1 bug in pad function? In-Reply-To: <1402726075014.19216@visionsense.com> References: <1402726075014.19216@visionsense.com> Message-ID: Hi Nadav On Sat, Jun 14, 2014 at 8:11 AM, Nadav Horesh wrote: > In [4]: np.__version__ > Out[4]: '1.9.0b1' > > The documentation specify that the mode parameter is optional I don't see the optional specification in the docstring. Perhaps because mode=None in the signature? The reason is that then, if you do not specify the signature (as in your case), you get the following helpful message: ValueError: Keyword "mode" must be a function or one of ['reflect', 'linear_ramp', 'edge', 'constant', 'minimum', 'wrap', 'symmetric', 'median', 'maximum', 'mean']. Instead of pad() takes exactly 3 arguments (2 given) Regards St?fan From nadavh at visionsense.com Sat Jun 14 08:40:29 2014 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 14 Jun 2014 12:40:29 +0000 Subject: [Numpy-discussion] numpy 1.9b1 bug in pad function? In-Reply-To: References: <1402726075014.19216@visionsense.com>, Message-ID: <1402749405315.94236@visionsense.com> This is most likely a documentation error since: In [7]: np.pad(a) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 np.pad(a) TypeError: pad() missing 1 required positional argument: 'pad_width' Nadav ________________________________________ From: numpy-discussion-bounces at scipy.org on behalf of St?fan van der Walt Sent: 14 June 2014 13:39 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy 1.9b1 bug in pad function? Hi Nadav On Sat, Jun 14, 2014 at 8:11 AM, Nadav Horesh wrote: > In [4]: np.__version__ > Out[4]: '1.9.0b1' > > The documentation specify that the mode parameter is optional I don't see the optional specification in the docstring. Perhaps because mode=None in the signature? The reason is that then, if you do not specify the signature (as in your case), you get the following helpful message: ValueError: Keyword "mode" must be a function or one of ['reflect', 'linear_ramp', 'edge', 'constant', 'minimum', 'wrap', 'symmetric', 'median', 'maximum', 'mean']. Instead of pad() takes exactly 3 arguments (2 given) Regards St?fan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion From stefan at sun.ac.za Sat Jun 14 08:55:14 2014 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 14 Jun 2014 14:55:14 +0200 Subject: [Numpy-discussion] numpy 1.9b1 bug in pad function? In-Reply-To: <1402749405315.94236@visionsense.com> References: <1402726075014.19216@visionsense.com> <1402749405315.94236@visionsense.com> Message-ID: <8738f7zm4t.fsf@sun.ac.za> On 2014-06-14 14:40:29, Nadav Horesh wrote: > This is most likely a documentation error since: > > In [7]: np.pad(a) > --------------------------------------------------------------------------- > TypeError Traceback (most recent call last) > in () > ----> 1 np.pad(a) > > TypeError: pad() missing 1 required positional argument: 'pad_width' That is because the signature is pad(array, pad_width, mode=None, ...) But mode *does* need to be specified. I see why this can be confusing, though, so perhaps we should simply make mode a positional argument too. St?fan From stefan at sun.ac.za Sat Jun 14 09:06:14 2014 From: stefan at sun.ac.za (Stefan van der Walt) Date: Sat, 14 Jun 2014 15:06:14 +0200 Subject: [Numpy-discussion] numpy 1.9b1 bug in pad function? In-Reply-To: <1402749405315.94236@visionsense.com> References: <1402726075014.19216@visionsense.com> <1402749405315.94236@visionsense.com> Message-ID: <871turzlmh.fsf@sun.ac.za> On 2014-06-14 14:40:29, Nadav Horesh wrote: > TypeError: pad() missing 1 required positional argument: 'pad_width' I've added a PR here for further discussion: https://github.com/numpy/numpy/pull/4808 St?fan From tmp50 at ukr.net Sun Jun 15 08:47:24 2014 From: tmp50 at ukr.net (Dmitrey) Date: Sun, 15 Jun 2014 15:47:24 +0300 Subject: [Numpy-discussion] new OpenOpt Suite release 0.54 Message-ID: <1402836389.898416638.pbuknjzx@frv44.fwdcdn.com> I'm glad to inform you about new OpenOpt Suite release 0.54: ? ? * Some changes for PyPy compatibility ? ? * FuncDesigner translator() can handle sparse derivatives from automatic differentiation ? ? * New interalg parameter rTol (relative tolerance, default 10^-8) ? ? * Bugfix and improvements for categorical variables? ??? * Some more changes and improvements Regards, D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Sun Jun 15 17:51:57 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 15 Jun 2014 23:51:57 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett wrote: > > > On Friday, June 13, 2014, Ralf Gommers wrote: > >> >> >> >> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett >> wrote: >> >>> Hi, >>> >>> Summary : I'm planning to upload OSX wheels for numpy and scipy using >>> the ATLAS blas / lapack library instead of the default OSX Accelerate >>> framework. >>> >>> We've run into some trouble with a segfault in recent OSX Accelerate: >>> >>> https://github.com/numpy/numpy/issues/4007 >>> >>> and Accelerate also doesn't play well with multiprocessing. >>> >>> https://github.com/numpy/numpy/issues/4776 >>> >>> Because there's nothing I love more than half-day compilation runs on >>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >>> and scipy to them to make OSX wheels. These pass all tests in i386 >>> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >>> >>> >>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >>> >>> The build process needs some automating, but it's recorded here: >>> >>> https://github.com/matthew-brett/numpy-atlas-binaries >>> >>> It's possible to get travis-ci to build these guys from a bare machine >>> and then upload them somewhere, but I haven't tried to do that. >>> >>> Meanwhile Sturla kindly worked up a patch to numpy to work round the >>> Accelerate segfault [1]. I haven't tested that, but given I'd already >>> built the wheels, I prefer the ATLAS builds because they work with >>> multiprocessing. >>> >>> I propose uploading these wheels as the default for numpy and scipy. >>> Does anyone have any objection or comments before I go ahead and do >>> that? >>> >> >> From your README and wscript I don't see what numpy version you're using >> to compile scipy against. I got the impression that you used 1.8.1, but it >> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. >> >> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see >> below). Your wheels should work with all common Python installs (mine is >> homebrew) right? >> >> Ralf >> >> >> $ python2.7 -c "import scipy; scipy.test()" >> Running unit tests for scipy >> NumPy version 1.9.0.dev-056ab73 >> NumPy is installed in >> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy >> SciPy version 0.14.0 >> SciPy is installed in >> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy >> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 >> Compatible Apple LLVM 4.2 (clang-425.0.28)] >> nose version 1.3.0 >> E...............EEEEEE............EEEEEEEEEE >> ====================================================================== >> ERROR: Failure: ImportError >> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >> 2): Symbol not found: _PyModule_Create2 >> Referenced from: >> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >> Expected in: flat namespace >> in >> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File >> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", >> line 413, in loadTestsFromName >> addr.filename, addr.module) >> File >> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >> line 47, in importFromPath >> return self.importFromDir(dir_path, fqname) >> File >> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >> line 94, in importFromDir >> mod = load_module(part_fqname, fh, filename, desc) >> File >> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", >> line 27, in >> from . import vq, hierarchy >> File >> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", >> line 175, in >> from . import _hierarchy_wrap >> ImportError: >> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >> 2): Symbol not found: _PyModule_Create2 >> Referenced from: >> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >> Expected in: flat namespace >> in >> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >> >> ... >> <42 more errors> >> > > That is strange homebrew is one of tests in the grid and the installation > path looks strange. > > Can you try downloading the wheel from the url and installing from the > local file? > That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though. Ralf > > Cheers, > > Matthew > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiping.lqp at alibaba-inc.com Mon Jun 16 06:00:35 2014 From: qiping.lqp at alibaba-inc.com (HongQi) Date: Mon, 16 Jun 2014 03:00:35 -0700 (PDT) Subject: [Numpy-discussion] Fail to build rpm using setup.py Message-ID: <1402912835265-37777.post@n7.nabble.com> Using `python setup.py bdist_rpm`, we can build a rpm for a python package. But when using this command to build rpm for numpy, we get the the following error message: ``` Installed (but unpackaged) file(s) found: /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_api.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_api.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_arrayprint.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_arrayprint.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_defchararray.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_defchararray.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_dtype.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_dtype.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_einsum.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_einsum.pyo /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_errstate.pyc /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_errstate.pyo ... ``` and ``` File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_clib.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_clib.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_clib.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_data.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_data.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_data.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_headers.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_headers.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_headers.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/scons.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/scons.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/scons.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/sdist.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/sdist.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/command/sdist.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/compat.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/compat.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/compat.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/conv_template.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/conv_template.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/conv_template.pyo File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/core.py File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/core.pyc File listed twice: /usr/local/lib/python2.7/site-packages/numpy/distutils/core.pyo ... ``` we tested building numpy-1.4.1 and numpy-1.7.1 on Redhat 5.7 and building numpy-1.7.1 and numpy-1.8.1 on Fedora 20, none of these builds succeeded, But we can build other python packages(Scrapy) on these system. Any suggestions for this problem? Thanks for your help. Regards, Qiping -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Fail-to-build-rpm-using-setup-py-tp37777.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From matthew.brett at gmail.com Mon Jun 16 07:10:47 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 16 Jun 2014 12:10:47 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: Hi, On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers wrote: > > > > On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett > wrote: >> >> >> >> On Friday, June 13, 2014, Ralf Gommers wrote: >>> >>> >>> >>> >>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett >>> wrote: >>>> >>>> Hi, >>>> >>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using >>>> the ATLAS blas / lapack library instead of the default OSX Accelerate >>>> framework. >>>> >>>> We've run into some trouble with a segfault in recent OSX Accelerate: >>>> >>>> https://github.com/numpy/numpy/issues/4007 >>>> >>>> and Accelerate also doesn't play well with multiprocessing. >>>> >>>> https://github.com/numpy/numpy/issues/4776 >>>> >>>> Because there's nothing I love more than half-day compilation runs on >>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >>>> and scipy to them to make OSX wheels. These pass all tests in i386 >>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >>>> >>>> >>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >>>> >>>> The build process needs some automating, but it's recorded here: >>>> >>>> https://github.com/matthew-brett/numpy-atlas-binaries >>>> >>>> It's possible to get travis-ci to build these guys from a bare machine >>>> and then upload them somewhere, but I haven't tried to do that. >>>> >>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the >>>> Accelerate segfault [1]. I haven't tested that, but given I'd already >>>> built the wheels, I prefer the ATLAS builds because they work with >>>> multiprocessing. >>>> >>>> I propose uploading these wheels as the default for numpy and scipy. >>>> Does anyone have any objection or comments before I go ahead and do >>>> that? >>> >>> >>> From your README and wscript I don't see what numpy version you're using >>> to compile scipy against. I got the impression that you used 1.8.1, but it >>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. >>> >>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see >>> below). Your wheels should work with all common Python installs (mine is >>> homebrew) right? >>> >>> Ralf >>> >>> >>> $ python2.7 -c "import scipy; scipy.test()" >>> Running unit tests for scipy >>> NumPy version 1.9.0.dev-056ab73 >>> NumPy is installed in >>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy >>> SciPy version 0.14.0 >>> SciPy is installed in >>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy >>> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 >>> Compatible Apple LLVM 4.2 (clang-425.0.28)] >>> nose version 1.3.0 >>> E...............EEEEEE............EEEEEEEEEE >>> >>> ====================================================================== >>> ERROR: Failure: ImportError >>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>> 2): Symbol not found: _PyModule_Create2 >>> Referenced from: >>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>> Expected in: flat namespace >>> in >>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) >>> >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File >>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", >>> line 413, in loadTestsFromName >>> addr.filename, addr.module) >>> File >>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>> line 47, in importFromPath >>> return self.importFromDir(dir_path, fqname) >>> File >>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>> line 94, in importFromDir >>> mod = load_module(part_fqname, fh, filename, desc) >>> File >>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", >>> line 27, in >>> from . import vq, hierarchy >>> File >>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", >>> line 175, in >>> from . import _hierarchy_wrap >>> ImportError: >>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>> 2): Symbol not found: _PyModule_Create2 >>> Referenced from: >>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>> Expected in: flat namespace >>> in >>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>> >>> ... >>> <42 more errors> >> >> >> That is strange homebrew is one of tests in the grid and the installation >> path looks strange. >> >> Can you try downloading the wheel from the url and installing from the >> local file? > > > That's what I did (easier than remembering the magic pip incantation). The > install path looks fine to me, maybe homebrew changed it recently? I can try > to update my install, will take a few days though. Yes, sorry - that does look like the normal homebrew install path, I didn't realize it had the exotic framework parts to it. I just ran these commands on my machine: SPI=https://nipy.bic.berkeley.edu/scipy_installers BREW_BIN=/usr/local/Cellar/python/2.7.6/bin curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/python -c "import scipy; scipy.test()" and the scipy tests passed. I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies binary compatible? What difference would I expect to see if I'd built scipy against numpy 1.5.1 or 1.7 ? Cheers, Matthew From josef.pktd at gmail.com Mon Jun 16 07:51:37 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 16 Jun 2014 07:51:37 -0400 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett wrote: > Hi, > > On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers wrote: >> >> >> >> On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett >> wrote: >>> >>> >>> >>> On Friday, June 13, 2014, Ralf Gommers wrote: >>>> >>>> >>>> >>>> >>>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using >>>>> the ATLAS blas / lapack library instead of the default OSX Accelerate >>>>> framework. >>>>> >>>>> We've run into some trouble with a segfault in recent OSX Accelerate: >>>>> >>>>> https://github.com/numpy/numpy/issues/4007 >>>>> >>>>> and Accelerate also doesn't play well with multiprocessing. >>>>> >>>>> https://github.com/numpy/numpy/issues/4776 >>>>> >>>>> Because there's nothing I love more than half-day compilation runs on >>>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >>>>> and scipy to them to make OSX wheels. These pass all tests in i386 >>>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >>>>> >>>>> >>>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >>>>> >>>>> The build process needs some automating, but it's recorded here: >>>>> >>>>> https://github.com/matthew-brett/numpy-atlas-binaries >>>>> >>>>> It's possible to get travis-ci to build these guys from a bare machine >>>>> and then upload them somewhere, but I haven't tried to do that. >>>>> >>>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the >>>>> Accelerate segfault [1]. I haven't tested that, but given I'd already >>>>> built the wheels, I prefer the ATLAS builds because they work with >>>>> multiprocessing. >>>>> >>>>> I propose uploading these wheels as the default for numpy and scipy. >>>>> Does anyone have any objection or comments before I go ahead and do >>>>> that? >>>> >>>> >>>> From your README and wscript I don't see what numpy version you're using >>>> to compile scipy against. I got the impression that you used 1.8.1, but it >>>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. >>>> >>>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see >>>> below). Your wheels should work with all common Python installs (mine is >>>> homebrew) right? >>>> >>>> Ralf >>>> >>>> >>>> $ python2.7 -c "import scipy; scipy.test()" >>>> Running unit tests for scipy >>>> NumPy version 1.9.0.dev-056ab73 >>>> NumPy is installed in >>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy >>>> SciPy version 0.14.0 >>>> SciPy is installed in >>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy >>>> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 >>>> Compatible Apple LLVM 4.2 (clang-425.0.28)] >>>> nose version 1.3.0 >>>> E...............EEEEEE............EEEEEEEEEE >>>> >>>> ====================================================================== >>>> ERROR: Failure: ImportError >>>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>> 2): Symbol not found: _PyModule_Create2 >>>> Referenced from: >>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>> Expected in: flat namespace >>>> in >>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) >>>> >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> File >>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", >>>> line 413, in loadTestsFromName >>>> addr.filename, addr.module) >>>> File >>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>> line 47, in importFromPath >>>> return self.importFromDir(dir_path, fqname) >>>> File >>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>> line 94, in importFromDir >>>> mod = load_module(part_fqname, fh, filename, desc) >>>> File >>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", >>>> line 27, in >>>> from . import vq, hierarchy >>>> File >>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", >>>> line 175, in >>>> from . import _hierarchy_wrap >>>> ImportError: >>>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>> 2): Symbol not found: _PyModule_Create2 >>>> Referenced from: >>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>> Expected in: flat namespace >>>> in >>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>> >>>> ... >>>> <42 more errors> >>> >>> >>> That is strange homebrew is one of tests in the grid and the installation >>> path looks strange. >>> >>> Can you try downloading the wheel from the url and installing from the >>> local file? >> >> >> That's what I did (easier than remembering the magic pip incantation). The >> install path looks fine to me, maybe homebrew changed it recently? I can try >> to update my install, will take a few days though. > > Yes, sorry - that does look like the normal homebrew install path, I > didn't realize it had the exotic framework parts to it. > > I just ran these commands on my machine: > > SPI=https://nipy.bic.berkeley.edu/scipy_installers > BREW_BIN=/usr/local/Cellar/python/2.7.6/bin > curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl > curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl > $BREW_BIN/pip install > numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl > $BREW_BIN/pip install > scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl > $BREW_BIN/python -c "import scipy; scipy.test()" > > and the scipy tests passed. > > I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies > binary compatible? What difference would I expect to see if I'd built > scipy against numpy 1.5.1 or 1.7 ? I summarized here what David explained a while ago about the difference between forward and backwards binary compatibility http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312 Josef > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From matthew.brett at gmail.com Mon Jun 16 08:00:53 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 16 Jun 2014 13:00:53 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: Hi, On Mon, Jun 16, 2014 at 12:51 PM, wrote: > On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett wrote: >> Hi, >> >> On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers wrote: >>> >>> >>> >>> On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett >>> wrote: >>>> >>>> >>>> >>>> On Friday, June 13, 2014, Ralf Gommers wrote: >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using >>>>>> the ATLAS blas / lapack library instead of the default OSX Accelerate >>>>>> framework. >>>>>> >>>>>> We've run into some trouble with a segfault in recent OSX Accelerate: >>>>>> >>>>>> https://github.com/numpy/numpy/issues/4007 >>>>>> >>>>>> and Accelerate also doesn't play well with multiprocessing. >>>>>> >>>>>> https://github.com/numpy/numpy/issues/4776 >>>>>> >>>>>> Because there's nothing I love more than half-day compilation runs on >>>>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >>>>>> and scipy to them to make OSX wheels. These pass all tests in i386 >>>>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >>>>>> >>>>>> >>>>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >>>>>> >>>>>> The build process needs some automating, but it's recorded here: >>>>>> >>>>>> https://github.com/matthew-brett/numpy-atlas-binaries >>>>>> >>>>>> It's possible to get travis-ci to build these guys from a bare machine >>>>>> and then upload them somewhere, but I haven't tried to do that. >>>>>> >>>>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the >>>>>> Accelerate segfault [1]. I haven't tested that, but given I'd already >>>>>> built the wheels, I prefer the ATLAS builds because they work with >>>>>> multiprocessing. >>>>>> >>>>>> I propose uploading these wheels as the default for numpy and scipy. >>>>>> Does anyone have any objection or comments before I go ahead and do >>>>>> that? >>>>> >>>>> >>>>> From your README and wscript I don't see what numpy version you're using >>>>> to compile scipy against. I got the impression that you used 1.8.1, but it >>>>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. >>>>> >>>>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see >>>>> below). Your wheels should work with all common Python installs (mine is >>>>> homebrew) right? >>>>> >>>>> Ralf >>>>> >>>>> >>>>> $ python2.7 -c "import scipy; scipy.test()" >>>>> Running unit tests for scipy >>>>> NumPy version 1.9.0.dev-056ab73 >>>>> NumPy is installed in >>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy >>>>> SciPy version 0.14.0 >>>>> SciPy is installed in >>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy >>>>> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 >>>>> Compatible Apple LLVM 4.2 (clang-425.0.28)] >>>>> nose version 1.3.0 >>>>> E...............EEEEEE............EEEEEEEEEE >>>>> >>>>> ====================================================================== >>>>> ERROR: Failure: ImportError >>>>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>>> 2): Symbol not found: _PyModule_Create2 >>>>> Referenced from: >>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>> Expected in: flat namespace >>>>> in >>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) >>>>> >>>>> ---------------------------------------------------------------------- >>>>> Traceback (most recent call last): >>>>> File >>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", >>>>> line 413, in loadTestsFromName >>>>> addr.filename, addr.module) >>>>> File >>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>>> line 47, in importFromPath >>>>> return self.importFromDir(dir_path, fqname) >>>>> File >>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>>> line 94, in importFromDir >>>>> mod = load_module(part_fqname, fh, filename, desc) >>>>> File >>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", >>>>> line 27, in >>>>> from . import vq, hierarchy >>>>> File >>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", >>>>> line 175, in >>>>> from . import _hierarchy_wrap >>>>> ImportError: >>>>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>>> 2): Symbol not found: _PyModule_Create2 >>>>> Referenced from: >>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>> Expected in: flat namespace >>>>> in >>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>> >>>>> ... >>>>> <42 more errors> >>>> >>>> >>>> That is strange homebrew is one of tests in the grid and the installation >>>> path looks strange. >>>> >>>> Can you try downloading the wheel from the url and installing from the >>>> local file? >>> >>> >>> That's what I did (easier than remembering the magic pip incantation). The >>> install path looks fine to me, maybe homebrew changed it recently? I can try >>> to update my install, will take a few days though. >> >> Yes, sorry - that does look like the normal homebrew install path, I >> didn't realize it had the exotic framework parts to it. >> >> I just ran these commands on my machine: >> >> SPI=https://nipy.bic.berkeley.edu/scipy_installers >> BREW_BIN=/usr/local/Cellar/python/2.7.6/bin >> curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >> curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >> $BREW_BIN/pip install >> numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >> $BREW_BIN/pip install >> scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >> $BREW_BIN/python -c "import scipy; scipy.test()" >> >> and the scipy tests passed. >> >> I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies >> binary compatible? What difference would I expect to see if I'd built >> scipy against numpy 1.5.1 or 1.7 ? > > I summarized here what David explained a while ago about the > difference between forward and backwards binary compatibility > > http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312 > I see - thanks for the summary. I will recompile. Cheers, Matthew From evanwormer at ucdavis.edu Mon Jun 16 13:38:31 2014 From: evanwormer at ucdavis.edu (Liz VanWormer) Date: Mon, 16 Jun 2014 12:38:31 -0500 Subject: [Numpy-discussion] numpy.random selection from a loglogistic distribution? Message-ID: Dear all, I'm a novice Python user, and I need to draw random variables from a loglogistic distribution. I've used the numpy.random command in the past to select variables from supported distributions (beta, normal, lognormal, etc). Is there a way to add in a distribution to be used with the numpy.random command? Thank you for your insight, Liz -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Jun 16 14:05:00 2014 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 16 Jun 2014 14:05:00 -0400 Subject: [Numpy-discussion] numpy.random selection from a loglogistic distribution? In-Reply-To: References: Message-ID: On Mon, Jun 16, 2014 at 1:38 PM, Liz VanWormer wrote: > Dear all, > > I'm a novice Python user, and I need to draw random variables from a > loglogistic distribution. I've used the numpy.random command in the past to > select variables from supported distributions (beta, normal, lognormal, > etc). Is there a way to add in a distribution to be used with the > numpy.random command? You or someone can add new distribution for numpy.random. However, scipy has the fisk distribution which according to wikipedia is the same as log-logistic. fisk is implemented as a special case of burr and has an explicit inverse cdf (ppf) So using fisk.rvs should be reasonably fast for vectorized calls, since the overhead is much larger than for numpy.random functions. http://en.wikipedia.org/wiki/Log-logistic_distribution to make distribution names more fun: Wikipedia:"generalized log-logistic distribution" "These include the Burr Type XII distribution (also known as the Singh-Maddala distribution) " Josef > > Thank you for your insight, > Liz > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From evanwormer at ucdavis.edu Mon Jun 16 16:23:40 2014 From: evanwormer at ucdavis.edu (Liz VanWormer) Date: Mon, 16 Jun 2014 15:23:40 -0500 Subject: [Numpy-discussion] numpy.random selection from a loglogistic distribution? In-Reply-To: References: Message-ID: Dear Josef, Thank you for the excellent suggestion! Using fisk.rvs fixed the problem. Best wishes, Liz On Mon, Jun 16, 2014 at 1:05 PM, wrote: > On Mon, Jun 16, 2014 at 1:38 PM, Liz VanWormer > wrote: > > Dear all, > > > > I'm a novice Python user, and I need to draw random variables from a > > loglogistic distribution. I've used the numpy.random command in the past > to > > select variables from supported distributions (beta, normal, lognormal, > > etc). Is there a way to add in a distribution to be used with the > > numpy.random command? > > You or someone can add new distribution for numpy.random. > > However, scipy has the fisk distribution which according to wikipedia > is the same as log-logistic. > fisk is implemented as a special case of burr and has an explicit > inverse cdf (ppf) > > So using fisk.rvs should be reasonably fast for vectorized calls, > since the overhead is much larger than for numpy.random functions. > > http://en.wikipedia.org/wiki/Log-logistic_distribution > > to make distribution names more fun: > Wikipedia:"generalized log-logistic distribution" "These include the > Burr Type XII distribution (also known as the Singh-Maddala > distribution) " > > Josef > > > > > Thank you for your insight, > > Liz > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kyle.mandli at gmail.com Tue Jun 17 16:40:51 2014 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Tue, 17 Jun 2014 15:40:51 -0500 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: Hi Everyone, Fernando Perez has volunteered to act as a facilitator for a BoF on NumPy if there is still interest in having this discussion. I can make sure that it is not scheduled in conjunction with some of the other planned BoFs that may interest this crowd. If someone could take the lead on organizing, submitting something to the conference website and I helping to gather interested parties I would be much obliged. Kyle On Fri, Jun 6, 2014 at 12:17 AM, Chris Barker wrote: > On Thu, Jun 5, 2014 at 1:32 PM, Kyle Mandli wrote: > >> In the past I know that we have simply gathered in a circle and >> discussed which works as well. Whatever the case, if someone could >> volunteer to "lead" the discussion >> > > It's my experience that a really good facilitator could make all the > difference in how productive this kind of discussion is. I have no idea how > to find such a facilitator (it's a pretty rare skill), but it would be nice > to try, rather than taking whoever is willing to do the bureaucratic > part.... > > and also submit it via the SciPy conference website (you have to sign into >> the dashboard and submit a new proposal) to help us keep track of >> everything I would be very appreciative. >> > > someone could still take on the organizer role while trying to find a > facilitator... > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Tue Jun 17 18:23:50 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 17 Jun 2014 16:23:50 -0600 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: On Tue, Jun 17, 2014 at 2:40 PM, Kyle Mandli wrote: > Hi Everyone, > > Fernando Perez has volunteered to act as a facilitator for a BoF on NumPy > if there is still interest in having this discussion. I can make sure that > it is not scheduled in conjunction with some of the other planned BoFs that > may interest this crowd. If someone could take the lead on organizing, > submitting something to the conference website and I helping to gather > interested parties I would be much obliged. > I can submit something to the conference website next week when I get back in town. We need to make sure it fits with the schedule of the Continuum folks. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Jun 18 14:54:50 2014 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 18 Jun 2014 20:54:50 +0200 Subject: [Numpy-discussion] Fail to build rpm using setup.py In-Reply-To: <1402912835265-37777.post@n7.nabble.com> References: <1402912835265-37777.post@n7.nabble.com> Message-ID: On Mon, Jun 16, 2014 at 12:00 PM, HongQi wrote: > Using `python setup.py bdist_rpm`, we can build a rpm for a python package. > But when using this command to build rpm for numpy, we get the the > following > error message: > > ``` > Installed (but unpackaged) file(s) found: > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_api.pyc > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_api.pyo > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_arrayprint.pyc > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_arrayprint.pyo > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.pyc > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_blasdot.pyo > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyo > > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_defchararray.pyc > > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_defchararray.pyo > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_dtype.pyc > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_dtype.pyo > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_einsum.pyc > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_einsum.pyo > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_errstate.pyc > > /usr/local/lib/python2.7/site-packages/numpy/core/tests/test_errstate.pyo > ... > ``` > > and > > ``` > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_clib.py > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_clib.pyc > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_clib.pyo > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_data.py > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_data.pyc > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_data.pyo > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_headers.py > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_headers.pyc > File listed twice: > > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/install_headers.pyo > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/scons.py > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/scons.pyc > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/scons.pyo > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/sdist.py > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/sdist.pyc > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/command/sdist.pyo > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/compat.py > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/compat.pyc > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/compat.pyo > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/conv_template.py > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/conv_template.pyc > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/conv_template.pyo > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/core.py > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/core.pyc > File listed twice: > /usr/local/lib/python2.7/site-packages/numpy/distutils/core.pyo > ... > ``` > > we tested building numpy-1.4.1 and numpy-1.7.1 on Redhat 5.7 and building > numpy-1.7.1 and numpy-1.8.1 on Fedora 20, none of these builds succeeded, > But we can build other python packages(Scrapy) on these system. Any > suggestions for this problem? > Looks a lot like http://bugs.python.org/issue1533164. Both the discussion there and the linked Fedora bug report say that this isn't solved in all cases. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jun 19 06:39:57 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 19 Jun 2014 11:39:57 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: Hi, On Mon, Jun 16, 2014 at 1:00 PM, Matthew Brett wrote: > Hi, > > On Mon, Jun 16, 2014 at 12:51 PM, wrote: >> On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett wrote: >>> Hi, >>> >>> On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers wrote: >>>> >>>> >>>> >>>> On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett >>>> wrote: >>>>> >>>>> >>>>> >>>>> On Friday, June 13, 2014, Ralf Gommers wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett >>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using >>>>>>> the ATLAS blas / lapack library instead of the default OSX Accelerate >>>>>>> framework. >>>>>>> >>>>>>> We've run into some trouble with a segfault in recent OSX Accelerate: >>>>>>> >>>>>>> https://github.com/numpy/numpy/issues/4007 >>>>>>> >>>>>>> and Accelerate also doesn't play well with multiprocessing. >>>>>>> >>>>>>> https://github.com/numpy/numpy/issues/4776 >>>>>>> >>>>>>> Because there's nothing I love more than half-day compilation runs on >>>>>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >>>>>>> and scipy to them to make OSX wheels. These pass all tests in i386 >>>>>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >>>>>>> >>>>>>> >>>>>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >>>>>>> >>>>>>> The build process needs some automating, but it's recorded here: >>>>>>> >>>>>>> https://github.com/matthew-brett/numpy-atlas-binaries >>>>>>> >>>>>>> It's possible to get travis-ci to build these guys from a bare machine >>>>>>> and then upload them somewhere, but I haven't tried to do that. >>>>>>> >>>>>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the >>>>>>> Accelerate segfault [1]. I haven't tested that, but given I'd already >>>>>>> built the wheels, I prefer the ATLAS builds because they work with >>>>>>> multiprocessing. >>>>>>> >>>>>>> I propose uploading these wheels as the default for numpy and scipy. >>>>>>> Does anyone have any objection or comments before I go ahead and do >>>>>>> that? >>>>>> >>>>>> >>>>>> From your README and wscript I don't see what numpy version you're using >>>>>> to compile scipy against. I got the impression that you used 1.8.1, but it >>>>>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. >>>>>> >>>>>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see >>>>>> below). Your wheels should work with all common Python installs (mine is >>>>>> homebrew) right? >>>>>> >>>>>> Ralf >>>>>> >>>>>> >>>>>> $ python2.7 -c "import scipy; scipy.test()" >>>>>> Running unit tests for scipy >>>>>> NumPy version 1.9.0.dev-056ab73 >>>>>> NumPy is installed in >>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy >>>>>> SciPy version 0.14.0 >>>>>> SciPy is installed in >>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy >>>>>> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 >>>>>> Compatible Apple LLVM 4.2 (clang-425.0.28)] >>>>>> nose version 1.3.0 >>>>>> E...............EEEEEE............EEEEEEEEEE >>>>>> >>>>>> ====================================================================== >>>>>> ERROR: Failure: ImportError >>>>>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>>>> 2): Symbol not found: _PyModule_Create2 >>>>>> Referenced from: >>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>>> Expected in: flat namespace >>>>>> in >>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) >>>>>> >>>>>> ---------------------------------------------------------------------- >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", >>>>>> line 413, in loadTestsFromName >>>>>> addr.filename, addr.module) >>>>>> File >>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>>>> line 47, in importFromPath >>>>>> return self.importFromDir(dir_path, fqname) >>>>>> File >>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>>>> line 94, in importFromDir >>>>>> mod = load_module(part_fqname, fh, filename, desc) >>>>>> File >>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", >>>>>> line 27, in >>>>>> from . import vq, hierarchy >>>>>> File >>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", >>>>>> line 175, in >>>>>> from . import _hierarchy_wrap >>>>>> ImportError: >>>>>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>>>> 2): Symbol not found: _PyModule_Create2 >>>>>> Referenced from: >>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>>> Expected in: flat namespace >>>>>> in >>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>>> >>>>>> ... >>>>>> <42 more errors> >>>>> >>>>> >>>>> That is strange homebrew is one of tests in the grid and the installation >>>>> path looks strange. >>>>> >>>>> Can you try downloading the wheel from the url and installing from the >>>>> local file? >>>> >>>> >>>> That's what I did (easier than remembering the magic pip incantation). The >>>> install path looks fine to me, maybe homebrew changed it recently? I can try >>>> to update my install, will take a few days though. >>> >>> Yes, sorry - that does look like the normal homebrew install path, I >>> didn't realize it had the exotic framework parts to it. >>> >>> I just ran these commands on my machine: >>> >>> SPI=https://nipy.bic.berkeley.edu/scipy_installers >>> BREW_BIN=/usr/local/Cellar/python/2.7.6/bin >>> curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>> curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>> $BREW_BIN/pip install >>> numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>> $BREW_BIN/pip install >>> scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>> $BREW_BIN/python -c "import scipy; scipy.test()" >>> >>> and the scipy tests passed. >>> >>> I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies >>> binary compatible? What difference would I expect to see if I'd built >>> scipy against numpy 1.5.1 or 1.7 ? >> >> I summarized here what David explained a while ago about the >> difference between forward and backwards binary compatibility >> >> http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312 >> > > I see - thanks for the summary. I will recompile. Last call - binaries recompiled, tested, more comments here: https://github.com/numpy/numpy/issues/4007 I'm planning to upload these binaries to pypi later on today, Cheers, Matthew From manolo at austrohungaro.com Thu Jun 19 09:20:02 2014 From: manolo at austrohungaro.com (Manolo =?iso-8859-1?Q?Mart=EDnez?=) Date: Thu, 19 Jun 2014 15:20:02 +0200 Subject: [Numpy-discussion] numpy.sort() failing mysteriously Message-ID: <20140619132001.GA4789@ManoloMartinez.localdomain> Dear all, I have a couple of structured arrays that, as far as I can see, only differ in irrelevant ways; yet numpy.sort chokes on one, not the other: This structured array *cannot* be sorted with np.sort(array, order='weight')... array([([0, 0, 0], 0.0), ([0, 0, 1], 0.0), ([0, 0, 2], 0.0), ([0, 1, 0], 0.0), ([0, 1, 1], 0.0), ([0, 1, 2], 0.0), ([0, 2, 0], 0.0), ([0, 2, 1], 0.0), ([0, 2, 2], 0.0), ([1, 0, 0], 0.0), ([1, 0, 1], 0.0), ([1, 0, 2], 0.0), ([1, 1, 0], 0.0), ([1, 1, 1], 0.0), ([1, 1, 2], 0.0), ([1, 2, 0], 0.0), ([1, 2, 1], 0.0), ([1, 2, 2], 0.0), ([2, 0, 0], 4.08179211371555e-289), ([2, 0, 1], 0.0), ([2, 0, 2], 0.0), ([2, 1, 0], 1.0), ([2, 1, 1], 6.939504595227983e-225), ([2, 1, 2], 1.0626819127933375e-224), ([2, 2, 0], 1.1209583874093894e-260), ([2, 2, 1], 0.0), ([2, 2, 2], 0.0)], dtype=[('strat', 'O'), ('weight', ' in () ----> 1 np.sort(aa, order=['weight', 'strat']) /usr/lib/python3.4/site-packages/numpy/core/fromnumeric.py in sort(a, axis, kind, order) 786 else: 787 a = asanyarray(a).copy() --> 788 a.sort(axis, kind, order) 789 return a 790 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() I should say that I asked this very question earlier today over at #scipy, and jtaylor kindly run the offending array, and could not reproduce the problem. I'm a bit stumped. python --version: 3.4.1 numpy.__version__: 1.8.1 ipython --version: 2.1.0 Thanks for any advice. Cheers, Manolo From matthew.brett at gmail.com Thu Jun 19 13:16:27 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 19 Jun 2014 18:16:27 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: On Thu, Jun 19, 2014 at 11:39 AM, Matthew Brett wrote: > Hi, > > On Mon, Jun 16, 2014 at 1:00 PM, Matthew Brett wrote: >> Hi, >> >> On Mon, Jun 16, 2014 at 12:51 PM, wrote: >>> On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett wrote: >>>> Hi, >>>> >>>> On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers wrote: >>>>> >>>>> >>>>> >>>>> On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett >>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Friday, June 13, 2014, Ralf Gommers wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett >>>>>>> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using >>>>>>>> the ATLAS blas / lapack library instead of the default OSX Accelerate >>>>>>>> framework. >>>>>>>> >>>>>>>> We've run into some trouble with a segfault in recent OSX Accelerate: >>>>>>>> >>>>>>>> https://github.com/numpy/numpy/issues/4007 >>>>>>>> >>>>>>>> and Accelerate also doesn't play well with multiprocessing. >>>>>>>> >>>>>>>> https://github.com/numpy/numpy/issues/4776 >>>>>>>> >>>>>>>> Because there's nothing I love more than half-day compilation runs on >>>>>>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >>>>>>>> and scipy to them to make OSX wheels. These pass all tests in i386 >>>>>>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >>>>>>>> >>>>>>>> >>>>>>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >>>>>>>> >>>>>>>> The build process needs some automating, but it's recorded here: >>>>>>>> >>>>>>>> https://github.com/matthew-brett/numpy-atlas-binaries >>>>>>>> >>>>>>>> It's possible to get travis-ci to build these guys from a bare machine >>>>>>>> and then upload them somewhere, but I haven't tried to do that. >>>>>>>> >>>>>>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the >>>>>>>> Accelerate segfault [1]. I haven't tested that, but given I'd already >>>>>>>> built the wheels, I prefer the ATLAS builds because they work with >>>>>>>> multiprocessing. >>>>>>>> >>>>>>>> I propose uploading these wheels as the default for numpy and scipy. >>>>>>>> Does anyone have any objection or comments before I go ahead and do >>>>>>>> that? >>>>>>> >>>>>>> >>>>>>> From your README and wscript I don't see what numpy version you're using >>>>>>> to compile scipy against. I got the impression that you used 1.8.1, but it >>>>>>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. >>>>>>> >>>>>>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see >>>>>>> below). Your wheels should work with all common Python installs (mine is >>>>>>> homebrew) right? >>>>>>> >>>>>>> Ralf >>>>>>> >>>>>>> >>>>>>> $ python2.7 -c "import scipy; scipy.test()" >>>>>>> Running unit tests for scipy >>>>>>> NumPy version 1.9.0.dev-056ab73 >>>>>>> NumPy is installed in >>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy >>>>>>> SciPy version 0.14.0 >>>>>>> SciPy is installed in >>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy >>>>>>> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 >>>>>>> Compatible Apple LLVM 4.2 (clang-425.0.28)] >>>>>>> nose version 1.3.0 >>>>>>> E...............EEEEEE............EEEEEEEEEE >>>>>>> >>>>>>> ====================================================================== >>>>>>> ERROR: Failure: ImportError >>>>>>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>>>>> 2): Symbol not found: _PyModule_Create2 >>>>>>> Referenced from: >>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>>>> Expected in: flat namespace >>>>>>> in >>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) >>>>>>> >>>>>>> ---------------------------------------------------------------------- >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", >>>>>>> line 413, in loadTestsFromName >>>>>>> addr.filename, addr.module) >>>>>>> File >>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>>>>> line 47, in importFromPath >>>>>>> return self.importFromDir(dir_path, fqname) >>>>>>> File >>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", >>>>>>> line 94, in importFromDir >>>>>>> mod = load_module(part_fqname, fh, filename, desc) >>>>>>> File >>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", >>>>>>> line 27, in >>>>>>> from . import vq, hierarchy >>>>>>> File >>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", >>>>>>> line 175, in >>>>>>> from . import _hierarchy_wrap >>>>>>> ImportError: >>>>>>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, >>>>>>> 2): Symbol not found: _PyModule_Create2 >>>>>>> Referenced from: >>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>>>> Expected in: flat namespace >>>>>>> in >>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so >>>>>>> >>>>>>> ... >>>>>>> <42 more errors> >>>>>> >>>>>> >>>>>> That is strange homebrew is one of tests in the grid and the installation >>>>>> path looks strange. >>>>>> >>>>>> Can you try downloading the wheel from the url and installing from the >>>>>> local file? >>>>> >>>>> >>>>> That's what I did (easier than remembering the magic pip incantation). The >>>>> install path looks fine to me, maybe homebrew changed it recently? I can try >>>>> to update my install, will take a few days though. >>>> >>>> Yes, sorry - that does look like the normal homebrew install path, I >>>> didn't realize it had the exotic framework parts to it. >>>> >>>> I just ran these commands on my machine: >>>> >>>> SPI=https://nipy.bic.berkeley.edu/scipy_installers >>>> BREW_BIN=/usr/local/Cellar/python/2.7.6/bin >>>> curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>>> curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>>> $BREW_BIN/pip install >>>> numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>>> $BREW_BIN/pip install >>>> scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl >>>> $BREW_BIN/python -c "import scipy; scipy.test()" >>>> >>>> and the scipy tests passed. >>>> >>>> I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies >>>> binary compatible? What difference would I expect to see if I'd built >>>> scipy against numpy 1.5.1 or 1.7 ? >>> >>> I summarized here what David explained a while ago about the >>> difference between forward and backwards binary compatibility >>> >>> http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312 >>> >> >> I see - thanks for the summary. I will recompile. > > Last call - binaries recompiled, tested, more comments here: > > https://github.com/numpy/numpy/issues/4007 > > I'm planning to upload these binaries to pypi later on today, Done - please test and let me know of any problems, Cheers, Matthew From olivier.grisel at ensta.org Thu Jun 19 13:31:31 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Thu, 19 Jun 2014 19:31:31 +0200 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: Just successfully tested on Python 3.4 from python.org / OSX 10.9 and all sklearn tests pass, including a tests that involves multiprocessing and that used to crash with Accelerate. Thanks very much! -- Olivier From matthew.brett at gmail.com Thu Jun 19 13:35:34 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 19 Jun 2014 18:35:34 +0100 Subject: [Numpy-discussion] Switch to using ATLAS for OSX binary wheels In-Reply-To: References: Message-ID: On Thu, Jun 19, 2014 at 6:31 PM, Olivier Grisel wrote: > Just successfully tested on Python 3.4 from python.org / OSX 10.9 and > all sklearn tests pass, including a tests that involves > multiprocessing and that used to crash with Accelerate. > > Thanks very much! De rien - thanks for your help with the Rackspace account, of which more in another email, Cheers, matthew From matthew.brett at gmail.com Thu Jun 19 14:23:22 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 19 Jun 2014 19:23:22 +0100 Subject: [Numpy-discussion] Automated test build machinery on travis - move to numpy or scipy org? Message-ID: Hi, As some of you might have seen, I automated the build of numpy / scipy against ATLAS using recent gcc here: https://github.com/matthew-brett/numpy-atlas-binaries with testing and uploading to Rackspace here: https://travis-ci.org/matthew-brett/numpy-atlas-binaries (thanks to Matt Terry for the original code for travis-ci build / testing [1] and Olivier Grisel for setting me up for Rackspace upload). Maybe it would be a good idea to move numpy-atlas-build to numpy or scipy organization and maintain it as part of the release process? Cheers, Matthew [1] https://github.com/matplotlib/mpl_mac_testing From jjhelmus at gmail.com Fri Jun 20 11:43:44 2014 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Fri, 20 Jun 2014 10:43:44 -0500 Subject: [Numpy-discussion] Conditionally compiling Fortran 90 extensions using numpy.distutils Message-ID: <53A456B0.30302@gmail.com> I have been working on creating a setup.py file for a sub-module with the aim to conditionally compile a Fortran 90 extension using F2PY depending on the availability of an appropriate compiler on the system. Basically if gfortran or another Fortran 90 compiler is available on the system, the extension should be built. If no compiler is available, the extension should not be built and the build should continue without error. After a bit of work I was able to get a working script which uses numpy.distutils: from os.path import join def configuration(parent_package='', top_path=None): global config from numpy.distutils.misc_util import Configuration config = Configuration('retrieve', parent_package, top_path) config.add_data_dir('tests') # Conditionally add Steiner echo classifier extension. config.add_extension('echo_steiner',sources=[steiner_echo_gen_source]) return config def steiner_echo_gen_source(ext, build_dir): try: config.have_f90c() return [join(config.local_path, 'echo_steiner.pyf'), join(config.local_path, 'src', 'echo_steiner.f90')] except: return None if __name__ == '__main__': from numpy.distutils.core import setup setup(**configuration(top_path='').todict()) Is there a better way of accomplishing this conditional compiling, perhaps without using a global variable or a try/except block? Additionally is the expected behaviour of the have_90c function to raise an exception when a Fortran 90 compiler is not available or is this a bug? From the documentation [1] it was unclear what the results should be. I should mention that the above code snippet was aided greatly by information in the NumPy Packinging documentation [2], NumPy Distutils - Users Guide [3], and code from the f2py utils unit tests [4]. Thanks, - Jonathan Helmus nmrglue.com/jhelmus [1] http://docs.scipy.org/doc/numpy/reference/distutils.html#numpy.distutils.misc_util.Configuration.have_f90c [2] http://docs.scipy.org/doc/numpy/reference/distutils.html [3] http://wiki.scipy.org/Wiki/Documentation/numpy_distutils [4] https://github.com/numpy/numpy/blob/master/numpy/f2py/tests/util.py From Nicolas.Rougier at inria.fr Sun Jun 22 04:22:19 2014 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Sun, 22 Jun 2014 10:22:19 +0200 Subject: [Numpy-discussion] Find n closest values Message-ID: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> Hi, I have an array L with regular spaced values between 0 and width. I have a (sorted) array I with irregular spaced values between 0 and width. I would like to find the closest value in I for any value in L. Currently, I'm using the following script but I wonder if I missed an obvious (and faster) solution: import numpy as np def find_closest(A, target): idx = A.searchsorted(target) idx = np.clip(idx, 1, len(A) - 1) left = A[idx - 1] right = A[idx] idx -= target - left < right - target return idx n, width = 256, 100.0 # 10 random sorted values in [0,width] I = np.sort(np.random.randint(0,width,10)) # n regular spaced values in [0,width] L = np.linspace(0, width, n) print I[find_closest(I,L)] Nicolas From hoogendoorn.eelco at gmail.com Sun Jun 22 04:30:12 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sun, 22 Jun 2014 10:30:12 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> Message-ID: Perhaps you could simplify some statements, but at least the algorithmic complexity is fine, and everything is vectorized, so I doubt you will get huge gains. You could take a look at the functions in scipy.spatial, and see how they perform for your problem parameters. On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier < Nicolas.Rougier at inria.fr> wrote: > > > Hi, > > I have an array L with regular spaced values between 0 and width. > I have a (sorted) array I with irregular spaced values between 0 and width. > > I would like to find the closest value in I for any value in L. > > Currently, I'm using the following script but I wonder if I missed an > obvious (and faster) solution: > > > import numpy as np > > def find_closest(A, target): > idx = A.searchsorted(target) > idx = np.clip(idx, 1, len(A) - 1) > left = A[idx - 1] > right = A[idx] > idx -= target - left < right - target > return idx > > n, width = 256, 100.0 > > # 10 random sorted values in [0,width] > I = np.sort(np.random.randint(0,width,10)) > > # n regular spaced values in [0,width] > L = np.linspace(0, width, n) > > print I[find_closest(I,L)] > > > > Nicolas > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Sun Jun 22 11:16:18 2014 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Sun, 22 Jun 2014 17:16:18 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> Message-ID: <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> Thanks for the answer. I was secretly hoping for some kind of hardly-known numpy function that would make things faster auto-magically... Nicolas On 22 Jun 2014, at 10:30, Eelco Hoogendoorn wrote: > Perhaps you could simplify some statements, but at least the algorithmic complexity is fine, and everything is vectorized, so I doubt you will get huge gains. > > You could take a look at the functions in scipy.spatial, and see how they perform for your problem parameters. > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier wrote: > > > Hi, > > I have an array L with regular spaced values between 0 and width. > I have a (sorted) array I with irregular spaced values between 0 and width. > > I would like to find the closest value in I for any value in L. > > Currently, I'm using the following script but I wonder if I missed an obvious (and faster) solution: > > > import numpy as np > > def find_closest(A, target): > idx = A.searchsorted(target) > idx = np.clip(idx, 1, len(A) - 1) > left = A[idx - 1] > right = A[idx] > idx -= target - left < right - target > return idx > > n, width = 256, 100.0 > > # 10 random sorted values in [0,width] > I = np.sort(np.random.randint(0,width,10)) > > # n regular spaced values in [0,width] > L = np.linspace(0, width, n) > > print I[find_closest(I,L)] > > > > Nicolas > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Nicolas.Rougier at inria.fr Sun Jun 22 11:27:14 2014 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Sun, 22 Jun 2014 17:27:14 +0200 Subject: [Numpy-discussion] 100 numpy exercises, now in Julia Message-ID: Hi all, Michiaki Ariga has started conversion of 100 numpy exercises in Julia. I know this might be a bit off-topic but I thought it was interesting enough. Github repository is at https://github.com/chezou/julia-100-exercises Nicolas From sebastian at sipsolutions.net Sun Jun 22 12:24:55 2014 From: sebastian at sipsolutions.net (Sebastian Berg) Date: Sun, 22 Jun 2014 18:24:55 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> Message-ID: <1403454295.525.0.camel@sebastian-t440> On So, 2014-06-22 at 17:16 +0200, Nicolas P. Rougier wrote: > Thanks for the answer. > I was secretly hoping for some kind of hardly-known numpy function that would make things faster auto-magically... > I doubt it is faster, but if you got scipy anyway, using KDTree may be pretty idiomatic. - Sebastian > > Nicolas > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn wrote: > > > Perhaps you could simplify some statements, but at least the algorithmic complexity is fine, and everything is vectorized, so I doubt you will get huge gains. > > > > You could take a look at the functions in scipy.spatial, and see how they perform for your problem parameters. > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier wrote: > > > > > > Hi, > > > > I have an array L with regular spaced values between 0 and width. > > I have a (sorted) array I with irregular spaced values between 0 and width. > > > > I would like to find the closest value in I for any value in L. > > > > Currently, I'm using the following script but I wonder if I missed an obvious (and faster) solution: > > > > > > import numpy as np > > > > def find_closest(A, target): > > idx = A.searchsorted(target) > > idx = np.clip(idx, 1, len(A) - 1) > > left = A[idx - 1] > > right = A[idx] > > idx -= target - left < right - target > > return idx > > > > n, width = 256, 100.0 > > > > # 10 random sorted values in [0,width] > > I = np.sort(np.random.randint(0,width,10)) > > > > # n regular spaced values in [0,width] > > L = np.linspace(0, width, n) > > > > print I[find_closest(I,L)] > > > > > > > > Nicolas > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From ben.root at ou.edu Sun Jun 22 12:35:51 2014 From: ben.root at ou.edu (Benjamin Root) Date: Sun, 22 Jun 2014 12:35:51 -0400 Subject: [Numpy-discussion] Find n closest values In-Reply-To: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> Message-ID: I would echo KDTree, but one way I think you can simplify the existing code you have is to shift the values of L by half the spacing, then you shouldn't need the check for left and right values. Cheers! Ben Root On Sun, Jun 22, 2014 at 4:22 AM, Nicolas P. Rougier < Nicolas.Rougier at inria.fr> wrote: > > > Hi, > > I have an array L with regular spaced values between 0 and width. > I have a (sorted) array I with irregular spaced values between 0 and width. > > I would like to find the closest value in I for any value in L. > > Currently, I'm using the following script but I wonder if I missed an > obvious (and faster) solution: > > > import numpy as np > > def find_closest(A, target): > idx = A.searchsorted(target) > idx = np.clip(idx, 1, len(A) - 1) > left = A[idx - 1] > right = A[idx] > idx -= target - left < right - target > return idx > > n, width = 256, 100.0 > > # 10 random sorted values in [0,width] > I = np.sort(np.random.randint(0,width,10)) > > # n regular spaced values in [0,width] > L = np.linspace(0, width, n) > > print I[find_closest(I,L)] > > > > Nicolas > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Sun Jun 22 13:05:35 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sun, 22 Jun 2014 19:05:35 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> Message-ID: Well, if the spacing is truly uniform, then of course you don't really need the search, and you can do away with the extra log-n, and there is a purely linear solution: def find_closest_direct(start, end, count, A): Q = (A-start)/(end-start)*count mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) boundary = np.zeros(count, np.int) boundary[mid] = 1 return np.add.accumulate(boundary) I expect this to be a bit faster, but nothing dramatic, unless your datasets are huge. It isn't really more or less elegant either, id say. Note that the output isn't 100% identical; youd need to do a little tinkering to figure out the correct/desired rounding behavior. On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier < Nicolas.Rougier at inria.fr> wrote: > > Thanks for the answer. > I was secretly hoping for some kind of hardly-known numpy function that > would make things faster auto-magically... > > > Nicolas > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn > wrote: > > > Perhaps you could simplify some statements, but at least the algorithmic > complexity is fine, and everything is vectorized, so I doubt you will get > huge gains. > > > > You could take a look at the functions in scipy.spatial, and see how > they perform for your problem parameters. > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > > > > > > Hi, > > > > I have an array L with regular spaced values between 0 and width. > > I have a (sorted) array I with irregular spaced values between 0 and > width. > > > > I would like to find the closest value in I for any value in L. > > > > Currently, I'm using the following script but I wonder if I missed an > obvious (and faster) solution: > > > > > > import numpy as np > > > > def find_closest(A, target): > > idx = A.searchsorted(target) > > idx = np.clip(idx, 1, len(A) - 1) > > left = A[idx - 1] > > right = A[idx] > > idx -= target - left < right - target > > return idx > > > > n, width = 256, 100.0 > > > > # 10 random sorted values in [0,width] > > I = np.sort(np.random.randint(0,width,10)) > > > > # n regular spaced values in [0,width] > > L = np.linspace(0, width, n) > > > > print I[find_closest(I,L)] > > > > > > > > Nicolas > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Sun Jun 22 13:06:22 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sun, 22 Jun 2014 19:06:22 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> Message-ID: Also, if you use scipy.spatial.KDTree, make sure to use cKDTree; the native python kdtree is sure to be slow as hell. On Sun, Jun 22, 2014 at 7:05 PM, Eelco Hoogendoorn < hoogendoorn.eelco at gmail.com> wrote: > Well, if the spacing is truly uniform, then of course you don't really > need the search, and you can do away with the extra log-n, and there is a > purely linear solution: > > def find_closest_direct(start, end, count, A): > Q = (A-start)/(end-start)*count > mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) > boundary = np.zeros(count, np.int) > boundary[mid] = 1 > return np.add.accumulate(boundary) > > I expect this to be a bit faster, but nothing dramatic, unless your > datasets are huge. It isn't really more or less elegant either, id say. > Note that the output isn't 100% identical; youd need to do a little > tinkering to figure out the correct/desired rounding behavior. > > > On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > >> >> Thanks for the answer. >> I was secretly hoping for some kind of hardly-known numpy function that >> would make things faster auto-magically... >> >> >> Nicolas >> >> >> On 22 Jun 2014, at 10:30, Eelco Hoogendoorn >> wrote: >> >> > Perhaps you could simplify some statements, but at least the >> algorithmic complexity is fine, and everything is vectorized, so I doubt >> you will get huge gains. >> > >> > You could take a look at the functions in scipy.spatial, and see how >> they perform for your problem parameters. >> > >> > >> > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier < >> Nicolas.Rougier at inria.fr> wrote: >> > >> > >> > Hi, >> > >> > I have an array L with regular spaced values between 0 and width. >> > I have a (sorted) array I with irregular spaced values between 0 and >> width. >> > >> > I would like to find the closest value in I for any value in L. >> > >> > Currently, I'm using the following script but I wonder if I missed an >> obvious (and faster) solution: >> > >> > >> > import numpy as np >> > >> > def find_closest(A, target): >> > idx = A.searchsorted(target) >> > idx = np.clip(idx, 1, len(A) - 1) >> > left = A[idx - 1] >> > right = A[idx] >> > idx -= target - left < right - target >> > return idx >> > >> > n, width = 256, 100.0 >> > >> > # 10 random sorted values in [0,width] >> > I = np.sort(np.random.randint(0,width,10)) >> > >> > # n regular spaced values in [0,width] >> > L = np.linspace(0, width, n) >> > >> > print I[find_closest(I,L)] >> > >> > >> > >> > Nicolas >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Sun Jun 22 14:30:58 2014 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Sun, 22 Jun 2014 20:30:58 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> Message-ID: <9FA36BA4-9B34-4AAC-BA04-B2177DBB6DBC@inria.fr> Thanks, I'll try your solution. Data (L) is not so big actually, it represents pixels on screen and (I) represents line position (for grids). I need to compute this quantity everytime the user zoom in or out. Nicolas On 22 Jun 2014, at 19:05, Eelco Hoogendoorn wrote: > Well, if the spacing is truly uniform, then of course you don't really need the search, and you can do away with the extra log-n, and there is a purely linear solution: > > def find_closest_direct(start, end, count, A): > Q = (A-start)/(end-start)*count > mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) > boundary = np.zeros(count, np.int) > boundary[mid] = 1 > return np.add.accumulate(boundary) > > I expect this to be a bit faster, but nothing dramatic, unless your datasets are huge. It isn't really more or less elegant either, id say. Note that the output isn't 100% identical; youd need to do a little tinkering to figure out the correct/desired rounding behavior. > > > On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier wrote: > > Thanks for the answer. > I was secretly hoping for some kind of hardly-known numpy function that would make things faster auto-magically... > > > Nicolas > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn wrote: > > > Perhaps you could simplify some statements, but at least the algorithmic complexity is fine, and everything is vectorized, so I doubt you will get huge gains. > > > > You could take a look at the functions in scipy.spatial, and see how they perform for your problem parameters. > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier wrote: > > > > > > Hi, > > > > I have an array L with regular spaced values between 0 and width. > > I have a (sorted) array I with irregular spaced values between 0 and width. > > > > I would like to find the closest value in I for any value in L. > > > > Currently, I'm using the following script but I wonder if I missed an obvious (and faster) solution: > > > > > > import numpy as np > > > > def find_closest(A, target): > > idx = A.searchsorted(target) > > idx = np.clip(idx, 1, len(A) - 1) > > left = A[idx - 1] > > right = A[idx] > > idx -= target - left < right - target > > return idx > > > > n, width = 256, 100.0 > > > > # 10 random sorted values in [0,width] > > I = np.sort(np.random.randint(0,width,10)) > > > > # n regular spaced values in [0,width] > > L = np.linspace(0, width, n) > > > > print I[find_closest(I,L)] > > > > > > > > Nicolas > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From hoogendoorn.eelco at gmail.com Sun Jun 22 15:14:41 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sun, 22 Jun 2014 21:14:41 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: <9FA36BA4-9B34-4AAC-BA04-B2177DBB6DBC@inria.fr> References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> <9FA36BA4-9B34-4AAC-BA04-B2177DBB6DBC@inria.fr> Message-ID: Protip: if you are writing your own rasterization code in python, be prepared to forget about performance altogether. Something like numba or other c-like extension will be necessary unless you are willing to leave big gobs of performance on the table; and even with pure C you will get nowhere close to the performance of super-duper optimized library code you are used to. But before you go down that rabbit hole, its probably worth thinking about whether you can get an existing rendering framework to do what you want to do. On Sun, Jun 22, 2014 at 8:30 PM, Nicolas P. Rougier < Nicolas.Rougier at inria.fr> wrote: > > Thanks, I'll try your solution. > > Data (L) is not so big actually, it represents pixels on screen and (I) > represents line position (for grids). I need to compute this quantity > everytime the user zoom in or out. > > > Nicolas > > > On 22 Jun 2014, at 19:05, Eelco Hoogendoorn > wrote: > > > Well, if the spacing is truly uniform, then of course you don't really > need the search, and you can do away with the extra log-n, and there is a > purely linear solution: > > > > def find_closest_direct(start, end, count, A): > > Q = (A-start)/(end-start)*count > > mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) > > boundary = np.zeros(count, np.int) > > boundary[mid] = 1 > > return np.add.accumulate(boundary) > > > > I expect this to be a bit faster, but nothing dramatic, unless your > datasets are huge. It isn't really more or less elegant either, id say. > Note that the output isn't 100% identical; youd need to do a little > tinkering to figure out the correct/desired rounding behavior. > > > > > > On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > > > > Thanks for the answer. > > I was secretly hoping for some kind of hardly-known numpy function that > would make things faster auto-magically... > > > > > > Nicolas > > > > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn > wrote: > > > > > Perhaps you could simplify some statements, but at least the > algorithmic complexity is fine, and everything is vectorized, so I doubt > you will get huge gains. > > > > > > You could take a look at the functions in scipy.spatial, and see how > they perform for your problem parameters. > > > > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > > > > > > > > > Hi, > > > > > > I have an array L with regular spaced values between 0 and width. > > > I have a (sorted) array I with irregular spaced values between 0 and > width. > > > > > > I would like to find the closest value in I for any value in L. > > > > > > Currently, I'm using the following script but I wonder if I missed an > obvious (and faster) solution: > > > > > > > > > import numpy as np > > > > > > def find_closest(A, target): > > > idx = A.searchsorted(target) > > > idx = np.clip(idx, 1, len(A) - 1) > > > left = A[idx - 1] > > > right = A[idx] > > > idx -= target - left < right - target > > > return idx > > > > > > n, width = 256, 100.0 > > > > > > # 10 random sorted values in [0,width] > > > I = np.sort(np.random.randint(0,width,10)) > > > > > > # n regular spaced values in [0,width] > > > L = np.linspace(0, width, n) > > > > > > print I[find_closest(I,L)] > > > > > > > > > > > > Nicolas > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Sun Jun 22 15:51:31 2014 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Sun, 22 Jun 2014 21:51:31 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> <9FA36BA4-9B34-4AAC-BA04-B2177DBB6DBC@inria.fr> Message-ID: <66820533-090E-466E-88B7-E5272CC1583D@inria.fr> Actually, it's already working pretty well but it slows down when you're doing a lot of zoom in/out. The trick is that rendering is done using shader (OpenGL) and this computation is used to give information to the shader to where to draw antialiased lines. In the end, this shader is able to draw any amiunt of grids/ticks (as in matplotlib). Some old example are available from here: https://github.com/rougier/gl-agg I tested your solution and it is faster by only a tiny amount but the way you wrote it might open the door for other improvements. Thanks. Nicolas On 22 Jun 2014, at 21:14, Eelco Hoogendoorn wrote: > Protip: if you are writing your own rasterization code in python, be prepared to forget about performance altogether. > > Something like numba or other c-like extension will be necessary unless you are willing to leave big gobs of performance on the table; and even with pure C you will get nowhere close to the performance of super-duper optimized library code you are used to. > > But before you go down that rabbit hole, its probably worth thinking about whether you can get an existing rendering framework to do what you want to do. > > > On Sun, Jun 22, 2014 at 8:30 PM, Nicolas P. Rougier wrote: > > Thanks, I'll try your solution. > > Data (L) is not so big actually, it represents pixels on screen and (I) represents line position (for grids). I need to compute this quantity everytime the user zoom in or out. > > > Nicolas > > > On 22 Jun 2014, at 19:05, Eelco Hoogendoorn wrote: > > > Well, if the spacing is truly uniform, then of course you don't really need the search, and you can do away with the extra log-n, and there is a purely linear solution: > > > > def find_closest_direct(start, end, count, A): > > Q = (A-start)/(end-start)*count > > mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) > > boundary = np.zeros(count, np.int) > > boundary[mid] = 1 > > return np.add.accumulate(boundary) > > > > I expect this to be a bit faster, but nothing dramatic, unless your datasets are huge. It isn't really more or less elegant either, id say. Note that the output isn't 100% identical; youd need to do a little tinkering to figure out the correct/desired rounding behavior. > > > > > > On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier wrote: > > > > Thanks for the answer. > > I was secretly hoping for some kind of hardly-known numpy function that would make things faster auto-magically... > > > > > > Nicolas > > > > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn wrote: > > > > > Perhaps you could simplify some statements, but at least the algorithmic complexity is fine, and everything is vectorized, so I doubt you will get huge gains. > > > > > > You could take a look at the functions in scipy.spatial, and see how they perform for your problem parameters. > > > > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier wrote: > > > > > > > > > Hi, > > > > > > I have an array L with regular spaced values between 0 and width. > > > I have a (sorted) array I with irregular spaced values between 0 and width. > > > > > > I would like to find the closest value in I for any value in L. > > > > > > Currently, I'm using the following script but I wonder if I missed an obvious (and faster) solution: > > > > > > > > > import numpy as np > > > > > > def find_closest(A, target): > > > idx = A.searchsorted(target) > > > idx = np.clip(idx, 1, len(A) - 1) > > > left = A[idx - 1] > > > right = A[idx] > > > idx -= target - left < right - target > > > return idx > > > > > > n, width = 256, 100.0 > > > > > > # 10 random sorted values in [0,width] > > > I = np.sort(np.random.randint(0,width,10)) > > > > > > # n regular spaced values in [0,width] > > > L = np.linspace(0, width, n) > > > > > > print I[find_closest(I,L)] > > > > > > > > > > > > Nicolas > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From pi8l4vion at outlook.fr Sun Jun 22 16:01:43 2014 From: pi8l4vion at outlook.fr (pi8L4vion) Date: Sun, 22 Jun 2014 22:01:43 +0200 Subject: [Numpy-discussion] =?iso-8859-1?q?NumPy_en_fran=E7ais?= Message-ID: Bonsoir ? tous, Est-ce qu?il y a des francophones qui pourraient ?changer avec moi ? Merci. Pi8L4vion -------------- next part -------------- An HTML attachment was scrubbed... URL: From hoogendoorn.eelco at gmail.com Sun Jun 22 16:52:17 2014 From: hoogendoorn.eelco at gmail.com (Eelco Hoogendoorn) Date: Sun, 22 Jun 2014 22:52:17 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: <66820533-090E-466E-88B7-E5272CC1583D@inria.fr> References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> <9FA36BA4-9B34-4AAC-BA04-B2177DBB6DBC@inria.fr> <66820533-090E-466E-88B7-E5272CC1583D@inria.fr> Message-ID: That's pretty cool; and it makes sense that way. Still, couldn't you fold this kind of computation into a shader? Have you looked at vispy btw? I think its a really nice initiative; having a high quality vector graphics module in there would make it even better. Would be nice if those projects could be merged. On Sun, Jun 22, 2014 at 9:51 PM, Nicolas P. Rougier < Nicolas.Rougier at inria.fr> wrote: > > Actually, it's already working pretty well but it slows down when you're > doing a lot of zoom in/out. > > The trick is that rendering is done using shader (OpenGL) and this > computation is used to give information to the shader to where to draw > antialiased lines. In the end, this shader is able to draw any amiunt of > grids/ticks (as in matplotlib). Some old example are available from here: > https://github.com/rougier/gl-agg > > I tested your solution and it is faster by only a tiny amount but the way > you wrote it might open the door for other improvements. Thanks. > > > Nicolas > > On 22 Jun 2014, at 21:14, Eelco Hoogendoorn > wrote: > > > Protip: if you are writing your own rasterization code in python, be > prepared to forget about performance altogether. > > > > Something like numba or other c-like extension will be necessary unless > you are willing to leave big gobs of performance on the table; and even > with pure C you will get nowhere close to the performance of super-duper > optimized library code you are used to. > > > > But before you go down that rabbit hole, its probably worth thinking > about whether you can get an existing rendering framework to do what you > want to do. > > > > > > On Sun, Jun 22, 2014 at 8:30 PM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > > > > Thanks, I'll try your solution. > > > > Data (L) is not so big actually, it represents pixels on screen and (I) > represents line position (for grids). I need to compute this quantity > everytime the user zoom in or out. > > > > > > Nicolas > > > > > > On 22 Jun 2014, at 19:05, Eelco Hoogendoorn > wrote: > > > > > Well, if the spacing is truly uniform, then of course you don't really > need the search, and you can do away with the extra log-n, and there is a > purely linear solution: > > > > > > def find_closest_direct(start, end, count, A): > > > Q = (A-start)/(end-start)*count > > > mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) > > > boundary = np.zeros(count, np.int) > > > boundary[mid] = 1 > > > return np.add.accumulate(boundary) > > > > > > I expect this to be a bit faster, but nothing dramatic, unless your > datasets are huge. It isn't really more or less elegant either, id say. > Note that the output isn't 100% identical; youd need to do a little > tinkering to figure out the correct/desired rounding behavior. > > > > > > > > > On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > > > > > > Thanks for the answer. > > > I was secretly hoping for some kind of hardly-known numpy function > that would make things faster auto-magically... > > > > > > > > > Nicolas > > > > > > > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn < > hoogendoorn.eelco at gmail.com> wrote: > > > > > > > Perhaps you could simplify some statements, but at least the > algorithmic complexity is fine, and everything is vectorized, so I doubt > you will get huge gains. > > > > > > > > You could take a look at the functions in scipy.spatial, and see how > they perform for your problem parameters. > > > > > > > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier < > Nicolas.Rougier at inria.fr> wrote: > > > > > > > > > > > > Hi, > > > > > > > > I have an array L with regular spaced values between 0 and width. > > > > I have a (sorted) array I with irregular spaced values between 0 and > width. > > > > > > > > I would like to find the closest value in I for any value in L. > > > > > > > > Currently, I'm using the following script but I wonder if I missed > an obvious (and faster) solution: > > > > > > > > > > > > import numpy as np > > > > > > > > def find_closest(A, target): > > > > idx = A.searchsorted(target) > > > > idx = np.clip(idx, 1, len(A) - 1) > > > > left = A[idx - 1] > > > > right = A[idx] > > > > idx -= target - left < right - target > > > > return idx > > > > > > > > n, width = 256, 100.0 > > > > > > > > # 10 random sorted values in [0,width] > > > > I = np.sort(np.random.randint(0,width,10)) > > > > > > > > # n regular spaced values in [0,width] > > > > L = np.linspace(0, width, n) > > > > > > > > print I[find_closest(I,L)] > > > > > > > > > > > > > > > > Nicolas > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Rougier at inria.fr Mon Jun 23 00:46:20 2014 From: Nicolas.Rougier at inria.fr (Nicolas P. Rougier) Date: Mon, 23 Jun 2014 06:46:20 +0200 Subject: [Numpy-discussion] Find n closest values In-Reply-To: References: <08864F9F-FDBA-4A08-AA02-1FED6D65BFE6@inria.fr> <26939F1F-5498-48E3-91AB-B77ECFA9B7FD@inria.fr> <9FA36BA4-9B34-4AAC-BA04-B2177DBB6DBC@inria.fr> <66820533-090E-466E-88B7-E5272CC1583D@inria.fr> Message-ID: On 22 Jun 2014, at 22:52, Eelco Hoogendoorn wrote: > That's pretty cool; and it makes sense that way. Still, couldn't you fold this kind of computation into a shader? > > Have you looked at vispy btw? I think its a really nice initiative; having a high quality vector graphics module in there would make it even better. Would be nice if those projects could be merged. > I'm part of vispy actually, those are side experiments for this project. Nicolas > > On Sun, Jun 22, 2014 at 9:51 PM, Nicolas P. Rougier wrote: > > Actually, it's already working pretty well but it slows down when you're doing a lot of zoom in/out. > > The trick is that rendering is done using shader (OpenGL) and this computation is used to give information to the shader to where to draw antialiased lines. In the end, this shader is able to draw any amiunt of grids/ticks (as in matplotlib). Some old example are available from here: https://github.com/rougier/gl-agg > > I tested your solution and it is faster by only a tiny amount but the way you wrote it might open the door for other improvements. Thanks. > > > Nicolas > > On 22 Jun 2014, at 21:14, Eelco Hoogendoorn wrote: > > > Protip: if you are writing your own rasterization code in python, be prepared to forget about performance altogether. > > > > Something like numba or other c-like extension will be necessary unless you are willing to leave big gobs of performance on the table; and even with pure C you will get nowhere close to the performance of super-duper optimized library code you are used to. > > > > But before you go down that rabbit hole, its probably worth thinking about whether you can get an existing rendering framework to do what you want to do. > > > > > > On Sun, Jun 22, 2014 at 8:30 PM, Nicolas P. Rougier wrote: > > > > Thanks, I'll try your solution. > > > > Data (L) is not so big actually, it represents pixels on screen and (I) represents line position (for grids). I need to compute this quantity everytime the user zoom in or out. > > > > > > Nicolas > > > > > > On 22 Jun 2014, at 19:05, Eelco Hoogendoorn wrote: > > > > > Well, if the spacing is truly uniform, then of course you don't really need the search, and you can do away with the extra log-n, and there is a purely linear solution: > > > > > > def find_closest_direct(start, end, count, A): > > > Q = (A-start)/(end-start)*count > > > mid = ((Q[1:]+Q[:-1]+1)/2).astype(np.int) > > > boundary = np.zeros(count, np.int) > > > boundary[mid] = 1 > > > return np.add.accumulate(boundary) > > > > > > I expect this to be a bit faster, but nothing dramatic, unless your datasets are huge. It isn't really more or less elegant either, id say. Note that the output isn't 100% identical; youd need to do a little tinkering to figure out the correct/desired rounding behavior. > > > > > > > > > On Sun, Jun 22, 2014 at 5:16 PM, Nicolas P. Rougier wrote: > > > > > > Thanks for the answer. > > > I was secretly hoping for some kind of hardly-known numpy function that would make things faster auto-magically... > > > > > > > > > Nicolas > > > > > > > > > On 22 Jun 2014, at 10:30, Eelco Hoogendoorn wrote: > > > > > > > Perhaps you could simplify some statements, but at least the algorithmic complexity is fine, and everything is vectorized, so I doubt you will get huge gains. > > > > > > > > You could take a look at the functions in scipy.spatial, and see how they perform for your problem parameters. > > > > > > > > > > > > On Sun, Jun 22, 2014 at 10:22 AM, Nicolas P. Rougier wrote: > > > > > > > > > > > > Hi, > > > > > > > > I have an array L with regular spaced values between 0 and width. > > > > I have a (sorted) array I with irregular spaced values between 0 and width. > > > > > > > > I would like to find the closest value in I for any value in L. > > > > > > > > Currently, I'm using the following script but I wonder if I missed an obvious (and faster) solution: > > > > > > > > > > > > import numpy as np > > > > > > > > def find_closest(A, target): > > > > idx = A.searchsorted(target) > > > > idx = np.clip(idx, 1, len(A) - 1) > > > > left = A[idx - 1] > > > > right = A[idx] > > > > idx -= target - left < right - target > > > > return idx > > > > > > > > n, width = 256, 100.0 > > > > > > > > # 10 random sorted values in [0,width] > > > > I = np.sort(np.random.randint(0,width,10)) > > > > > > > > # n regular spaced values in [0,width] > > > > L = np.linspace(0, width, n) > > > > > > > > print I[find_closest(I,L)] > > > > > > > > > > > > > > > > Nicolas > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion at scipy.org > > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > _______________________________________________ > > > NumPy-Discussion mailing list > > > NumPy-Discussion at scipy.org > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From Jerome.Kieffer at esrf.fr Mon Jun 23 03:25:32 2014 From: Jerome.Kieffer at esrf.fr (Jerome Kieffer) Date: Mon, 23 Jun 2014 09:25:32 +0200 Subject: [Numpy-discussion] =?utf-8?q?NumPy_en_fran=C3=A7ais?= In-Reply-To: References: Message-ID: <20140623092532.237c767d.Jerome.Kieffer@esrf.fr> Hi, As the request is is French, I will answer in French... On Sun, 22 Jun 2014 22:01:43 +0200 pi8L4vion wrote: > Bonsoir ? tous, > > Est-ce qu?il y a des francophones qui pourraient ?changer avec moi ? Je suis Fran?ais ... par contre je ne connais pas de liste de discussion sur numpy ni sur python scientifique en Fran?ais. Je me pose d'ailleurs la question de la pertinence d'une telle liste puisque toute la documentation est en anglais et que la communaut? est d?j? assez limit?, ce n'est pas en la fragmentant que cela va aider. Mon conseil serait donc de communiquer en anglais (quitte ? s'excuser de la mauvaise qualit? de l'expression anglaise: la plupart des gens ici ne sont pas anglophones et comprendrons) Bien cordialement -- J?r?me Kieffer From charlesr.harris at gmail.com Fri Jun 27 23:44:45 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 27 Jun 2014 21:44:45 -0600 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: Hi Kyle, On Tue, Jun 17, 2014 at 2:40 PM, Kyle Mandli wrote: > Hi Everyone, > > Fernando Perez has volunteered to act as a facilitator for a BoF on NumPy > if there is still interest in having this discussion. I can make sure that > it is not scheduled in conjunction with some of the other planned BoFs that > may interest this crowd. If someone could take the lead on organizing, > submitting something to the conference website and I helping to gather > interested parties I would be much obliged. > I see you have reserved a slot on July 10 at 1:30 PM in room 204. That looks good to me. Along with discussing the future development of NumPy, I'd like to propose adopting something like the Blaze standard for datetime, which is similar to the current Pandas treatment, but with a different time base. Hmm... there seems to be a proliferation of time implementations, we should try to pick just one. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Jun 28 14:25:40 2014 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sat, 28 Jun 2014 11:25:40 -0700 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> Message-ID: <-3348772324412470165@unknownmsgid> On Jun 27, 2014, at 8:44 PM, Charles R Harris wrote: > > Hi Kyle, > >> On Tue, Jun 17, 2014 at 2:40 I'd like to propose adopting something like the Blaze standard for datetime, +1 for some focused discussion of datetime. This has been lingering far too long. -Chris From fperez.net at gmail.com Sat Jun 28 22:29:56 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 28 Jun 2014 19:29:56 -0700 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: <-3348772324412470165@unknownmsgid> References: <1401872329.18622.3.camel@sebastian-t440> <-3348772324412470165@unknownmsgid> Message-ID: Hi folks, I've just created a page on the numpy wiki: https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 I'd appreciate it if people could put down on that page the title of topics, plus a brief summary (with links to this or other relevant threads), they'd like to see discussed. BoFs can be very useful but they can also easily devolve into random digressions. I'll do my best to fulfill my role as discussion facilitator by pushing for an agenda for the discussion that can fit realistically in the time we have, and trying to keep us on track. It would help a lot if people put their ideas in advance, so we can sort/refine/prioritize a little bit. Hopefully this will lead to a more productive conversation. Cheers f On Sat, Jun 28, 2014 at 11:25 AM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > On Jun 27, 2014, at 8:44 PM, Charles R Harris > wrote: > > > > Hi Kyle, > > > >> On Tue, Jun 17, 2014 at 2:40 I'd like to propose adopting something > like the Blaze standard for datetime, > > +1 for some focused discussion of datetime. This has been lingering > far too long. > > -Chris > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jun 28 23:19:25 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Jun 2014 21:19:25 -0600 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> <-3348772324412470165@unknownmsgid> Message-ID: On Sat, Jun 28, 2014 at 8:29 PM, Fernando Perez wrote: > Hi folks, > > I've just created a page on the numpy wiki: > > https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 > > I'd appreciate it if people could put down on that page the title of > topics, plus a brief summary (with links to this or other relevant > threads), they'd like to see discussed. > > BoFs can be very useful but they can also easily devolve into random > digressions. I'll do my best to fulfill my role as discussion facilitator > by pushing for an agenda for the discussion that can fit realistically in > the time we have, and trying to keep us on track. > > It would help a lot if people put their ideas in advance, so we can > sort/refine/prioritize a little bit. Hopefully this will lead to a more > productive conversation. > > Cheers > > I've been thinking more along the lines of a planning session than a panel discussion, something like the round table discussions that preceded implementation of python3 support and the move to github. Perhaps we could arrange something like that as a follow up on the BOF meetup. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Sat Jun 28 23:40:07 2014 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 28 Jun 2014 21:40:07 -0600 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> <-3348772324412470165@unknownmsgid> Message-ID: On Sat, Jun 28, 2014 at 9:19 PM, Charles R Harris wrote: > > > > On Sat, Jun 28, 2014 at 8:29 PM, Fernando Perez > wrote: > >> Hi folks, >> >> I've just created a page on the numpy wiki: >> >> https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 >> >> I'd appreciate it if people could put down on that page the title of >> topics, plus a brief summary (with links to this or other relevant >> threads), they'd like to see discussed. >> >> BoFs can be very useful but they can also easily devolve into random >> digressions. I'll do my best to fulfill my role as discussion facilitator >> by pushing for an agenda for the discussion that can fit realistically in >> the time we have, and trying to keep us on track. >> >> It would help a lot if people put their ideas in advance, so we can >> sort/refine/prioritize a little bit. Hopefully this will lead to a more >> productive conversation. >> >> Cheers >> >> > I've been thinking more along the lines of a planning session than a panel > discussion, something like the round table discussions that preceded > implementation of python3 support and the move to github. Perhaps we could > arrange something like that as a follow up on the BOF meetup. > > I've added a preliminary list of topics. I'm sure there is more to be added. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat Jun 28 23:44:39 2014 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 28 Jun 2014 20:44:39 -0700 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> <-3348772324412470165@unknownmsgid> Message-ID: Great, thanks! And we can certainly try to either move into planning at the end, or plan for such afterwards. Best f On Sat, Jun 28, 2014 at 8:40 PM, Charles R Harris wrote: > > > > On Sat, Jun 28, 2014 at 9:19 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Sat, Jun 28, 2014 at 8:29 PM, Fernando Perez >> wrote: >> >>> Hi folks, >>> >>> I've just created a page on the numpy wiki: >>> >>> https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 >>> >>> I'd appreciate it if people could put down on that page the title of >>> topics, plus a brief summary (with links to this or other relevant >>> threads), they'd like to see discussed. >>> >>> BoFs can be very useful but they can also easily devolve into random >>> digressions. I'll do my best to fulfill my role as discussion facilitator >>> by pushing for an agenda for the discussion that can fit realistically in >>> the time we have, and trying to keep us on track. >>> >>> It would help a lot if people put their ideas in advance, so we can >>> sort/refine/prioritize a little bit. Hopefully this will lead to a more >>> productive conversation. >>> >>> Cheers >>> >>> >> I've been thinking more along the lines of a planning session than a >> panel discussion, something like the round table discussions that preceded >> implementation of python3 support and the move to github. Perhaps we could >> arrange something like that as a follow up on the BOF meetup. >> >> > I've added a preliminary list of topics. I'm sure there is more to be > added. > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail -------------- next part -------------- An HTML attachment was scrubbed... URL: From kyle.mandli at gmail.com Sun Jun 29 15:47:21 2014 From: kyle.mandli at gmail.com (Kyle Mandli) Date: Sun, 29 Jun 2014 14:47:21 -0500 Subject: [Numpy-discussion] SciPy 2014 BoF NumPy Participation In-Reply-To: References: <1401872329.18622.3.camel@sebastian-t440> <-3348772324412470165@unknownmsgid> Message-ID: I am really excited to see that we have a great agenda for the BoF, I hope that the discussion will be fruitful! Kyle On Sat, Jun 28, 2014 at 10:44 PM, Fernando Perez wrote: > Great, thanks! And we can certainly try to either move into planning at > the end, or plan for such afterwards. > > Best > > f > > > On Sat, Jun 28, 2014 at 8:40 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> >> On Sat, Jun 28, 2014 at 9:19 PM, Charles R Harris < >> charlesr.harris at gmail.com> wrote: >> >>> >>> >>> >>> On Sat, Jun 28, 2014 at 8:29 PM, Fernando Perez >>> wrote: >>> >>>> Hi folks, >>>> >>>> I've just created a page on the numpy wiki: >>>> >>>> https://github.com/numpy/numpy/wiki/Numpy-BoF-at-Scipy-2014 >>>> >>>> I'd appreciate it if people could put down on that page the title of >>>> topics, plus a brief summary (with links to this or other relevant >>>> threads), they'd like to see discussed. >>>> >>>> BoFs can be very useful but they can also easily devolve into random >>>> digressions. I'll do my best to fulfill my role as discussion facilitator >>>> by pushing for an agenda for the discussion that can fit realistically in >>>> the time we have, and trying to keep us on track. >>>> >>>> It would help a lot if people put their ideas in advance, so we can >>>> sort/refine/prioritize a little bit. Hopefully this will lead to a more >>>> productive conversation. >>>> >>>> Cheers >>>> >>>> >>> I've been thinking more along the lines of a planning session than a >>> panel discussion, something like the round table discussions that preceded >>> implementation of python3 support and the move to github. Perhaps we could >>> arrange something like that as a follow up on the BOF meetup. >>> >>> >> I've added a preliminary list of topics. I'm sure there is more to be >> added. >> >> Chuck >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > Fernando Perez (@fperez_org; http://fperez.org) > fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) > fernando.perez-at-berkeley: contact me here for any direct mail > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Mon Jun 30 04:31:03 2014 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Mon, 30 Jun 2014 10:31:03 +0200 Subject: [Numpy-discussion] genfromtxt universal newline support Message-ID: Hi all, I was just having a new look into the mess that is, imo, the support for automatic line ending recognition in genfromtxt, and more generally, the Python file openers. I am glad at least reading gzip files is no longer entirely broken in Python3, but actually detecting in particular ?old Mac? style CR line endings currently only work for uncompressed and bzip2 files under 2.6/2.7. This is largely because genfromtxt wants to open everything in binary mode, which arguably makes no sense for ASCII text files with numbers. I think the only reason this works in 2.x is that the ?U? reading mode overrides the ?b?. So on the Python side what actually works for automatic line ending detection is: Python 2.6 2.7 3.2 3.3/3.4 uncompressed: U U t t gzip: E N E t bzip2: U U E t* lzma: - - - t* U - works with mode ?rU? E - mode ?rU? raises an error N - mode ?rU? is accepted, but does not detect CR (?\r?) line endings (actually I think ?U? is simply internally discarded by gzip.open() in 2.7.4+) t - works with mode ?rt? (default with plain open()) - * means requires the '.open()' rather than the '.XXXFile()' method of bz2/lzma Therefore I?d propose the changes in https://github.com/dhomeier/numpy/commit/995ec93 to extend universal newline recognition as far as possible with the above openers. There are some potential issues with this: 1. Switching to ?rt? mode for Python3.x means that np.lib._datasource.open() does not return byte strings by itself, so genfromtxt has to use asbytes() on the returned lines. Since this occurs only in two places, I don?t see a major problem with this. 2. In the tests I had to work around the lack of fileobj support in bz2.BZ2File by using os.system(?bzip2 ??) on the temporary file, which might not work on all systems. In particular I?d expect it to fail under Windows, but it?s not clear to me how far the entire mkstemp thing works under Windows... As a final note, http://bugs.python.org/issue13989#msg153127 suggests a workaround that might make this work with gzip.open() (and perhaps bz2?) on 3.2 as well. I am not sure how high 3.2 support is ranking for the near future; for the moment I am not strongly inclined to implement it? Grateful for comments or tests (especially under Windows!) of the commit(s) above - Derek From jtaylor.debian at googlemail.com Mon Jun 30 07:33:50 2014 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Mon, 30 Jun 2014 13:33:50 +0200 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: Message-ID: genfromtxt and loadtxt need an almost full rewrite to fix the botched python3 conversion of these functions. There are a couple threads about this on this list already. There are numerous PRs fixing stuff in these functions which I currently all -1'd because we need to fix the underlying unicode issues first. I have a PR were I started this for loadtxt but it is incredibly annoying to try to support all the broken use cases the function accidentally supported. 1.9 beta still uses the broken functions because I had no time to get this done correctly. But we should probably put a big fat future warning into the release notes that genfromtxt and loadtxt may stop working for your binary streams. That will probably allow us to start fixing these functions. From njs at pobox.com Mon Jun 30 10:39:35 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 30 Jun 2014 15:39:35 +0100 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: Message-ID: On Mon, Jun 30, 2014 at 12:33 PM, Julian Taylor wrote: > genfromtxt and loadtxt need an almost full rewrite to fix the botched > python3 conversion of these functions. There are a couple threads > about this on this list already. > There are numerous PRs fixing stuff in these functions which I > currently all -1'd because we need to fix the underlying unicode > issues first. > I have a PR were I started this for loadtxt but it is incredibly > annoying to try to support all the broken use cases the function > accidentally supported. > > 1.9 beta still uses the broken functions because I had no time to get > this done correctly. > But we should probably put a big fat future warning into the release > notes that genfromtxt and loadtxt may stop working for your binary > streams. > That will probably allow us to start fixing these functions. +1 to doing the proper fix instead of piling up buggy hacks. Do we understand the difference between the current code and the "proper" code well enough to detect cases where they differ and issue warnings in those cases specifically? -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From derek at astro.physik.uni-goettingen.de Mon Jun 30 10:47:56 2014 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Mon, 30 Jun 2014 16:47:56 +0200 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: Message-ID: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> On 30 Jun 2014, at 04:39 pm, Nathaniel Smith wrote: > On Mon, Jun 30, 2014 at 12:33 PM, Julian Taylor > wrote: >> genfromtxt and loadtxt need an almost full rewrite to fix the botched >> python3 conversion of these functions. There are a couple threads >> about this on this list already. >> There are numerous PRs fixing stuff in these functions which I >> currently all -1'd because we need to fix the underlying unicode >> issues first. >> I have a PR were I started this for loadtxt but it is incredibly >> annoying to try to support all the broken use cases the function >> accidentally supported. >> >> 1.9 beta still uses the broken functions because I had no time to get >> this done correctly. >> But we should probably put a big fat future warning into the release >> notes that genfromtxt and loadtxt may stop working for your binary >> streams. What binary streams? >> That will probably allow us to start fixing these functions. > > +1 to doing the proper fix instead of piling up buggy hacks. Do we > understand the difference between the current code and the "proper" > code well enough to detect cases where they differ and issue warnings > in those cases specifically? Does it make sense to keep maintaing both functions at all? IIRC the idea that loadtxt would be the faster version of the two has been discarded long ago, thus it seems there is very little, if anything, loadtxt can do that cannot be done just as well by genfromtxt. Main compatibility issue is probably different default behaviour and interface of the two, but perhaps that might be best solved by replacing loadtxt with another genfromtxt wrapper? A real need, which had also been discussed at length, is a truly performant text IO function (i.e. one using a compiled ASCII number parser, and optimally also a more memory-efficient one), but unfortunately all people interested in implementing this seem to have drifted away (not excluding myself from this)? Cheers, Derek From njs at pobox.com Mon Jun 30 10:56:22 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 30 Jun 2014 15:56:22 +0100 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> Message-ID: On Mon, Jun 30, 2014 at 3:47 PM, Derek Homeier wrote: > Does it make sense to keep maintaing both functions at all? IIRC the idea that > loadtxt would be the faster version of the two has been discarded long ago, > thus it seems there is very little, if anything, loadtxt can do that cannot be done > just as well by genfromtxt. Main compatibility issue is probably different default > behaviour and interface of the two, but perhaps that might be best solved by > replacing loadtxt with another genfromtxt wrapper? > A real need, which had also been discussed at length, is a truly performant text IO > function (i.e. one using a compiled ASCII number parser, and optimally also a more > memory-efficient one), but unfortunately all people interested in implementing this > seem to have drifted away (not excluding myself from this)? It's possible we could steal some code from Pandas for this. IIRC they have C/Cython text parsing routines. (It's also an interesting question whether they've fixed the unicode/binary issues, might be worth checking before rewriting from scratch...) -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From chris.barker at noaa.gov Mon Jun 30 12:04:31 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 30 Jun 2014 09:04:31 -0700 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> Message-ID: > > It's also an interesting > question whether they've fixed the unicode/binary issues, Which brings up the "how do we handle text/strings in numpy? issue. We had a good thread going here about what the 'S' data type should be , what with py3 and all, but I don't think we ever really resolved that. IIRC, the key issue was whether we should have a "proper" one-byte-per-character text type -- after all, ASCI/ANSI text is pretty common in scientific data sets, and 4 bytes per char is a fair bit of overhead. Anyway, this all ties in with the text file parsing issues... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jun 30 12:31:23 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 30 Jun 2014 17:31:23 +0100 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> Message-ID: On 30 Jun 2014 17:05, "Chris Barker" wrote: >> >> It's also an interesting >> question whether they've fixed the unicode/binary issues, > > > Which brings up the "how do we handle text/strings in numpy? issue. We had a good thread going here about what the 'S' data type should be , what with py3 and all, but I don't think we ever really resolved that. > > IIRC, the key issue was whether we should have a "proper" one-byte-per-character text type -- after all, ASCI/ANSI text is pretty common in scientific data sets, and 4 bytes per char is a fair bit of overhead. > > Anyway, this all ties in with the text file parsing issues... Only tangentially though :-) maybe start another thread if you want to talk about that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Jun 30 16:46:52 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 30 Jun 2014 13:46:52 -0700 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> Message-ID: On Mon, Jun 30, 2014 at 9:31 AM, Nathaniel Smith wrote: > On 30 Jun 2014 17:05, "Chris Barker" wrote: > > > Anyway, this all ties in with the text file parsing issues... > > Only tangentially though :-) > well, a fast text parser (and "text mode") input file will either need to deal with Unicode properly or not. But your point is well taken. We did have a good thread about his a few months back, which resulted in the usual thing of kind of withering away with no decision or action. But I've added it to the list to talk about at SciPy... -Chris > maybe start another thread if you want to talk about that? > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Mon Jun 30 16:58:37 2014 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Mon, 30 Jun 2014 22:58:37 +0200 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> Message-ID: <61D4ADBB-9619-482E-A338-1DFAB405C24C@astro.physik.uni-goettingen.de> On 30 Jun 2014, at 04:56 pm, Nathaniel Smith wrote: >> A real need, which had also been discussed at length, is a truly performant text IO >> function (i.e. one using a compiled ASCII number parser, and optimally also a more >> memory-efficient one), but unfortunately all people interested in implementing this >> seem to have drifted away (not excluding myself from this)? > > It's possible we could steal some code from Pandas for this. IIRC they > have C/Cython text parsing routines. (It's also an interesting > question whether they've fixed the unicode/binary issues, might be > worth checking before rewriting from scratch...) Good point, last time I was playing with Pandas it was not any faster, but now a 10x speedup speaks for itself. Their C engine does not support generic whitespace separators, but that could probably be addressed in a numpy implementation. Derek From jeffreback at gmail.com Mon Jun 30 17:10:56 2014 From: jeffreback at gmail.com (Jeff Reback) Date: Mon, 30 Jun 2014 17:10:56 -0400 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: <61D4ADBB-9619-482E-A338-1DFAB405C24C@astro.physik.uni-goettingen.de> References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> <61D4ADBB-9619-482E-A338-1DFAB405C24C@astro.physik.uni-goettingen.de> Message-ID: In pandas 0.14.0, generic whitespace IS parsed via the c-parser, e.g. specifying '\s+' as a separator. Not sure when you were playing last with pandas, but the c-parser has been in place since late 2012. (version 0.8.0) http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#text-parsing-api-changes > On Jun 30, 2014, at 4:58 PM, Derek Homeier wrote: > > On 30 Jun 2014, at 04:56 pm, Nathaniel Smith wrote: > >>> A real need, which had also been discussed at length, is a truly performant text IO >>> function (i.e. one using a compiled ASCII number parser, and optimally also a more >>> memory-efficient one), but unfortunately all people interested in implementing this >>> seem to have drifted away (not excluding myself from this)? >> >> It's possible we could steal some code from Pandas for this. IIRC they >> have C/Cython text parsing routines. (It's also an interesting >> question whether they've fixed the unicode/binary issues, might be >> worth checking before rewriting from scratch...) > > Good point, last time I was playing with Pandas it was not any faster, but now a 10x > speedup speaks for itself. Their C engine does not support generic whitespace separators, > but that could probably be addressed in a numpy implementation. > > Derek > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From derek at astro.physik.uni-goettingen.de Mon Jun 30 19:06:32 2014 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 1 Jul 2014 01:06:32 +0200 Subject: [Numpy-discussion] genfromtxt universal newline support In-Reply-To: References: <61FC3DA1-30F9-4AF8-8CE9-745822578A61@astro.physik.uni-goettingen.de> <61D4ADBB-9619-482E-A338-1DFAB405C24C@astro.physik.uni-goettingen.de> Message-ID: On 30.06.2014, at 23:10, Jeff Reback wrote: > In pandas 0.14.0, generic whitespace IS parsed via the c-parser, e.g. specifying '\s+' as a separator. Not sure when you were playing last with pandas, but the c-parser has been in place since late 2012. (version 0.8.0) > > http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#text-parsing-api-changes Ah, I did not see the '\s' syntax in the documentation and thought ' *' would be the only option. Thanks, Derek