Mailman 3 April 2015 - NumPy-Discussion

EuroScipy 2015 : Call for talks, posters and tutorials [Reminder]
by Nicolas P. Rougier April 17, 2015

April 17, 2015

[Apology for cross-posting] Dear all, EuroScipy 2015, the annual conference on Python in science will take place in Cambridge, UK on 26-30 August 2015. The conference features two days of tutorials followed by two days of scientific talks & posters and an extra day dedicated to developer sprints. It is the major event in Europe in the field of technical/scientific computing within the Python ecosystem. Data scientists, analysts, quants, PhD's, scientists and students from more than 20 countries attended the conference last year. The topics presented at EuroSciPy are very diverse, with a focus on advanced software engineering and original uses of Python and its scientific libraries, either in theoretical or experimental research, from both academia and the industry. Submissions for posters, talks & tutorials (beginner and advanced) are welcome on our website at http://www.euroscipy.org/2015/ Sprint proposals should be addressed directly to the organisation at euroscipy-org(a)python.org Important dates Mar 24, 2015 Call for talks, posters & tutorials Apr 30, 2015 Talk and tutorials submission deadline May 1, 2015 Registration opens May 30, 2015 Final program announced Jun 15, 2015 Early-bird registration ends Aug 26-27, 2015 Tutorials Aug 28-29, 2015 Main conference Aug 30, 2015 Sprints We look forward to an exciting conference and hope to see you in Cambridge The EuroSciPy 2015 Team - http://www.euroscipy.org/2015/

1 0

Re: [Numpy-discussion] IDE's for numpy development?
by Suzen, Mehmet April 15, 2015

April 15, 2015

> Spyder supports C. Thanks for correcting this. I wasn't aware of it. How was your experience with it? Best, -m

3 2

[ANN] python-blosc v1.2.5
by Valentin Haenel April 15, 2015

April 15, 2015

============================= Announcing python-blosc 1.2.5 ============================= What is new? ============ This release contains support for Blosc v1.5.4 including changes to how the GIL is kept. This was required because Blosc was refactored in the v1.5.x line to remove global variables and to use context objects instead. As such, it became necessary to keep the GIL while calling Blosc from Python code that uses the multiprocessing module. In addition, is now possible to change the blocksize used by Blosc using ``set_blocksize``. When using this however, bear in mind that the blocksize has been finely tuned to be a good default value and that randomly messing with this value may have unforeseen and unpredictable consequences on the performance of Blosc. Additionally, we can now compile on Posix architectures, thanks again to Andreas Schwab for that one. For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc/wiki/Release-notes More docs and examples are available in the documentation site: http://python-blosc.blosc.org What is it? =========== Blosc (http://www.blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc is the first compressor that is meant not only to reduce the size of large datasets on-disk or in-memory, but also to accelerate object manipulations that are memory-bound (http://www.blosc.org/docs/StarvingCPUs.pdf). See http://www.blosc.org/synthetic-benchmarks.html for some benchmarks on how much speed it can achieve in some datasets. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for the Blosc compression library. There is also a handy tool built on Blosc called Bloscpack (https://github.com/Blosc/bloscpack). It features a commmand line interface that allows you to compress large binary datafiles on-disk. It also comes with a Python API that has built-in support for serializing and deserializing Numpy arrays both on-disk and in-memory at speeds that are competitive with regular Pickle/cPickle machinery. Installing ========== python-blosc is in PyPI repository, so installing it is easy: $ pip install -U blosc # yes, you should omit the python- prefix Download sources ================ The sources are managed through github services at: http://github.com/Blosc/python-blosc Documentation ============= There is Sphinx-based documentation site at: http://python-blosc.blosc.org/ Mailing list ============ There is an official mailing list for Blosc at: blosc(a)googlegroups.com http://groups.google.es/group/blosc Licenses ======== Both Blosc and its Python wrapper are distributed using the MIT license. See: https://github.com/Blosc/python-blosc/blob/master/LICENSES for more details. ---- **Enjoy data!**

1 0

ANN: numexpr 2.4.1 released
by Francesc Alted April 14, 2015

April 14, 2015

========================= Announcing Numexpr 2.4.1 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== In this version there is improved support for newer MKL library as well as other minor improvements. This version is meant for production. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

Numpy compilation error
by Peter Kerpedjiev April 12, 2015

April 12, 2015

Dear all, Upon trying to install numpy using 'pip install numpy' in a virtualenv, I get the following error messages: creating build/temp.linux-x86_64-2.7/numpy/random/mtrand compile options: '-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.7/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -Ibuild/src.linux-x86_64-2.7/numpy/core/src/private -c' gcc: numpy/random/mtrand/distributions.c numpy/random/mtrand/distributions.c: In function ‘loggam’: numpy/random/mtrand/distributions.c:892:1: internal compiler error: Illegal instruction } ^ Please submit a full bug report, with preprocessed source if appropriate. See <http://bugzilla.redhat.com/bugzilla> for instructions. Preprocessed source stored into /tmp/ccjkBSd2.out file, please attach this to your bugreport. This leads to the compilation process failing with this error: Cleaning up... Command /home/mescalin/pkerp/.virtualenvs/notebooks/bin/python -c "import setuptools;__file__='/home/mescalin/pkerp/.virtualenvs/notebooks/build/numpy/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file __, 'exec'))" install --record /tmp/pip-c_Cd7B-record/install-record.txt --single-version-externally-managed --install-headers /home/mescalin/pkerp/.virtualenvs/notebooks/include/site/python2.7 failed with error code 1 in /ho me/mescalin/pkerp/.virtualenvs/notebooks/build/numpy Traceback (most recent call last): File "/home/mescalin/pkerp/.virtualenvs/notebooks/bin/pip", line 9, in <module> load_entry_point('pip==1.4.1', 'console_scripts', 'pip')() File "/home/mescalin/pkerp/.virtualenvs/notebooks/lib/python2.7/site-packages/pip/__init__.py", line 148, in main return command.main(args[1:], options) File "/home/mescalin/pkerp/.virtualenvs/notebooks/lib/python2.7/site-packages/pip/basecommand.py", line 169, in main text = '\n'.join(complete_log) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128) Have any of you encountered a similar problem before? Thanks in advance, -Peter ================================================ The gcc version is: [pkerp@fluidspace ~]$ gcc --version gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-7) Copyright (C) 2013 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. PS:

2 1

Non-meta indexing improvements discussion
by Sebastian Berg April 10, 2015

April 10, 2015

Hi all, Let me take a shot at summing up some suggestions to make the indexing less surprising, and maybe we can gather some more in a more concentrated way now. 1. Implement something like `arr.fancy_index[...]` and `arr.ortho_index[...]` (i.e. Jaimes PR is the start for trying this) 2. Add warnings for non-consecutive advanced indexing (i.e. the original example `arr[0, :, index_array]`). 3. I do not know if it possible or useful, but I could imagine a module wide switch (similar to __future__ imports) to change the default indexing behaviour. One more thing, implementing this (especially the "new" indexing) is non-trivial, so as always help beyond just a discussion is appreciated and in my opinion the best way to push an actual change to happen sooner rather then in some far off future. I do not have time for concentrating much on an implementation for a while myself for a while at least. - Sebastian

3 3

Trouble subclassing ndarray
by Elliot April 10, 2015

April 10, 2015

Hi all, Sorry if this is the wrong forum for a question like this. I'm trying to create an object with multiple inheritance, one of which is from numpy.ndarray. The other gives it cacheable properties and defines a __getattr__ to deliver the cached properties. The initial instantiation is successful, but ufuncs and slices cause an infinite recursion where the __getattr__ function is used but the _cacheable attribute is not set (ie: from __init__ ) I am using docs.scipy.org/doc/numpy/user/basics.subclassing.html as a reference. Here is code that shows my problem (python 2.7, numpy 1.8.2) ===================== from __future__ import print_function import numpy as np class Cacheable(object): def __init__(self,*args,**kwargs): self._cacheable = {} def __getattr__(self,key): print("getting %s"%key) if key in self._cacheable: print(" found it") self._cacheable[key]() return self.__dict__[key] #if chache function does't update # data you're going to have a bad time else: raise AttributeError def _clear(self): '''clears derived properties''' for key in self._cacheable: if key in self.__dict__: del self.__dict__[key] class BaseGeometry(np.ndarray,Cacheable): '''Numpy array with extra attributes that are cacheable arrays''' def __new__(cls,input_array,dtype=np.float64,*args,**kwargs): # Input array is an already formed ndarray instance # We first cast to be our class type obj = np.asarray(input_array,dtype=dtype).view(cls) # Finally, we must return the newly created object: return obj def __init__(self,dims=None,dtype=np.float64,readonly=True): #TODO: sort through args and kwargs to make better self.readonly=readonly if readonly: self.flags.writeable=False self.dims=dims self._dtype = dtype Cacheable.__init__(self) def writeable_copy(self): ret = np.copy(self) ret.flags.writeable = True return ret def __array_finalize_(self,obj): #New object, will be created in __new__ print("array_finalize") if obj is None: return # created from slice or template print("finalizing slice") self._cacheable = getattr(obj, '_cacheable', None) if __name__ == "__main__": n = 5 test = BaseGeometry(np.random.randint(-25,25,(n,2))) print("this works:",test._cacheable) broken = test[1:4] #interestingly, no problem here print(broken) #infinite recursion =================== array_finalize is never called. output is: this works: {} getting _cacheable getting _cacheable getting _cacheable [and on and on] getting _cacheable getting _cacheable <repr(<__main__.BaseGeometry at 0x7fa42e2921b8>) failed: RuntimeError: maximum recursion depth exceeded while calling a Python object> -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Trouble-subclassing-ndarray-tp4… Sent from the Numpy-discussion mailing list archive at Nabble.com.

4 4

On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
by Nathaniel Smith April 10, 2015

April 10, 2015

On Apr 5, 2015 7:04 AM, "Robert Kern" <robert.kern(a)gmail.com> wrote: > > On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > > > > On Apr 4, 2015 4:12 AM, "Todd" <toddrjen(a)gmail.com> wrote: > > > > > > There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. > > > > I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. > > It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. Ok, let me try to make my point another way. I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of "you" that includes everyone in the discussion, not picking on Robert in particular :-).) My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the "majority of benefits of such a change" is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one is going to sneak anything by anyone. Plus, I dunno, our current approach to discussions just seems to make things hostile and shouty and unpleasant. If a grad student or junior colleague comes to you with an idea where you see some potentially critical flaw, do you yell THAT WILL NEVER WORK and kick them out of your office? Or, do you maybe ask a few leading questions and see where they go? I think things will work better if the next time something like this comes up, *one* person just says "hmm, interesting idea, but the backcompat issues seem pretty severe; do you have any ideas about how to mitigate that?", and then we let that point be taken as having been made and see where the discussion goes. Maybe we can all give it a try? -n

12 27

ANN: HDF5 for Python 2.5.0
by Andrew Collette April 9, 2015

April 9, 2015

Announcing HDF5 for Python (h5py) 2.5.0 ======================================== The h5py team is happy to announce the availability of h5py 2.5.0. This release introduces experimental support for the highly-anticipated "Single Writer Multiple Reader" (SWMR) feature in the upcoming HDF5 1.10 release. SWMR allows sharing of a single HDF5 file between multiple processes without the complexity of MPI or multiprocessing-based solutions. This is an experimental feature that should NOT be used in production code. We are interested in getting feedback from the broader community with respect to performance and the API design. For more details, check out the h5py user guide: http://docs.h5py.org/en/latest/swmr.html SWMR support was contributed by Ulrik Pedersen. What's h5py? ------------ The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Documentation is at: http://docs.h5py.org Changes ------- * Experimental SWMR support * Group and AttributeManager classes now inherit from the appropriate ABCs * Fixed an issue with 64-bit float VLENS * Cython warning cleanups related to "const" * Entire code base ported to "six"; 2to3 removed from setup.py Acknowledgements --------------- This release incorporates changes from, among others: * Ulrik Pedersen * James Tocknell * Will Parkin * Antony Lee * Peter H. Li * Peter Colberg * Ghislain Antony Vaillant Where to get it --------------- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org

4 5

what files to include with compiled fortran extension on Mac
by Mark Bakker April 9, 2015

April 9, 2015

Hello list, I want to send somebody my compiled fortran extension on a Mac (compiled with f2py and gfortran). Problem is that it doesn't work on other Macs unless they also instal xcode (2 GB, yikes!) and gfortran. So apparently there are some additional files missing when I just send the compiled extension. Does anybody know what other files to include or (better) how to compile a fortran extension without needing to send any additional files? Thanks, Mark

1 0