NumPy-Discussion September 2015

numpy-discussion@python.org

69 participants
66 discussions

Looking for a developer who will work with me for at least 6 months to fix NumPy's dtype system.
by Travis Oliphant 12 Sep '15

12 Sep '15

Hi all, Apologies for cross-posting, but I need to get the word out and twitter doesn't provide enough explanation. I've been working on a second edition of my "Guide to NumPy" book. It's been a time-pressured activity, but it's helped me put more meat around my ideas for how to fix NumPy's dtype system -- which I've been contemplating off an on for 8 years. I'm pretty sure I know exactly how to do it --- in a way that fits more cleanly into Python. It will take 3-6 months and will have residual efforts needed that will last another 6 months --- making more types available with NumPy, improving calculations etc. This work will be done completely in public view and allow for public comment. It will not solve *all* of NumPy's problems, but it will put NumPy's dtype system on the footing it in retrospect should have been put on in the first place (if I had known then what I know now). It won't be a grandiose rewrite. It will be a pretty surgical fix to a few key places in the code. However, it will break the ABI and require recompilation of NumPy extensions (and so would need to be called NumPy 2.0). This is unavoidable, but I don't see any problem with breaking the ABI today given how easy it is to get distributions of Python these days from a variety of sources (including using conda --- but not only using conda). For those that remember what happened in Python dev land, the changes will be similar to when Guido changed Python 1.5.2 to Python 2.0. I can mentor and work closely with someone who will work on this and we will invite full participation and feedback from whomever in the community also wants to participate --- but I can't do it myself full time (and it needs someone full time+). Fortunately, I can pay someone to do it if they are willing to commit at least 6 months (it is not required to work at Continuum for this, but you can have a job at Continuum if you want one). I'm only looking for people who have enough experience with C or preferably the Python C-API. You also have to *want* to work on this. You need to be willing to work with me on the project directly and work to have a mind-meld with my ideas which will undoubtedly give rise to additional perspectives and ideas for later work for you. When I wrote NumPy 1.0, I put in 80+ hour weeks for about 6 months or more and then 60+ weeks for another year. I was pretty obsessed with it. This won't need quite that effort, but it will need something like it. Being able to move to Austin is a plus but not required. I can sponsor a visa for the right candidate as well (though it's not guaranteed you will get one with the immigration policies what they are). This is a labor of love for so many of us and my desire to help the dtype situation in NumPy comes from the same space that my desire to work on NumPy in the first place came. I will be interviewing people to work on this as not everyone who may want to will really be qualified to do it --- especially with so many people writing Cython these days instead of good-ole C-API code :-) Feel free to spread the news to anyone you can. I won't say more until I've found someone to work with me on this --- because I won't have the time to follow-up with any questions or comments. Even if I can't find someone I will publish the ideas --- but that also takes time and effort that is in short supply for me right now. If there is someone willing to fund this work, please let me know as well -- that could free up more of my time. Best, -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io

1 0

review of #6247
by pizza＠netspace.net.au 12 Sep '15

12 Sep '15

Hi all, first attempt at this: small doco patch to address https://github.com/numpy/numpy/issues/6247 https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=u…

2 3

review of #6191
by jason 12 Sep '15

12 Sep '15

doco patch to address https://github.com/numpy/numpy/issues/6191 https://github.com/numpy/numpy/compare/master...pizzathief:issue6191

2 1

feature request - increment counter on write check
by Daniel Manson 11 Sep '15

11 Sep '15

Originally posted as issue 6301 <https://github.com/numpy/numpy/issues/6301> on github. Presumably any block of code that modifies an ndarray's buffer is wrapped in a (thread safe?) check of the writable flag. Would it be possible to hold a counter rather than a simple bool flag and then increment the counter whenever you test the flag? Hopefully this would introduce only a tiny additional overhead, but would permit pseudo-hashing to test whether a mutable array has changed since you last encountered it. Ideally this should be exposed a bit like python's __hash__ <https://docs.python.org/2/reference/datamodel.html#object.__hash__>method, lets say __mutablehash__, meaning a hash is returned but be warned that the object is mutable. For an ndarray, X, containing objects that themselves have a __mutablehash__ method (e.g. other ndarrays, or some user object), the X.__mutablehash__ method will need to do the full check over all constituent objects, or simply return None. Defining and API of this sort would make it possible to - for example - let pandas DataFrames also implement this interface. In terms of usage cases, the one I was motivated by was imagining improvements to the "variable explorer" in Spyder - roughly speaking, this widget's job is to always display an up-to-date summary of variables in current scope, e.g. currently it can show max/min and shape, but you could imagine also showing graphical summaries of the contents of an ndarray. If the widget could cache summaries and check which arrays have really changed it should be much faster/offer more features/be simpler internally. Note that pandas DataFrames are relevant here as an example of complex objects, containing ndarrays, which would benefit from being able to have their summaries cached. A more common/general usage case would be as a check in some kind of memoization process... #simple example... @memoize_please def hasnans(x): return np.any(np.isnan(x)) # more complex example... def convolve_fft(a,b, _cache={}): a_hash = mutablehash(a) b_hash = mutablehash(b) if a_hash not in _cache: _cache[a_hash] = fft(a) if b_hash not in _cache: _cache[b_hash] = fft(b) return ifft(_cache[a_hash] * _cache[b_hash]) A quick though on an implementation detail... I'm not sure exactly how to deal with the counter overflowing: perhaps if you treated counter==0 to mean not-writable (i.e. that would be the new version of the old write flag) then you might get some uint-wraparound checking for free (because when it wraps back around to zero the buffer ends up becoming locked)? Alternatively you could just say that no guarantee is given of wraparound being caught..though that might seriously impact on the range of possible uses. In summary... Hopefully the stuff needed to make __mutablehash__ work could be implemented simply by adding a single extra operation to the write-check (and maybe changing the footprint of the ndarray slightly to accomodate a counter). But I suspect someone will tell me that life is never that simple!

5 7

ANN: pandas v0.17.0rc1 - RELEASE CANDIDATE
by Jeff Reback 11 Sep '15

11 Sep '15

Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.17.0. Please try this RC and report any issues here: Pandas Issues <https://github.com/pydata/pandas/issues/10848> We will be releasing officially in 1-2 weeks or so. **RELEASE CANDIDATE 1** This is a major release from 0.16.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Highlights include: - Release the Global Interpreter Lock (GIL) on some cython operations, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Plotting methods are now available as attributes of the .plot accessor, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The sorting API has been revamped to remove some long-time inconsistencies, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Support for a datetime64[ns] with timezones as a first-class dtype, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The default for to_datetime will now be to raise when presented with unparseable formats, previously this would return the original input, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The default for dropna in HDFStore has changed to False, to store by default all rows even if they are all NaN, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Support for Series.dt.strftime to generate formatted strings for datetime-likes, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Development installed versions of pandas will now have PEP440 compliant version strings GH9518 <https://github.com/pydata/pandas/issues/9518> - Development support for benchmarking with the Air Speed Velocity library GH8316 <https://github.com/pydata/pandas/pull/8316> - Support for reading SAS xport files, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Removal of the automatic TimeSeries broadcasting, deprecated since 0.8.0, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> See the Whatsnew <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html> for much more information. Best way to get this is to install via conda <http://pandas-docs.github.io/pandas-docs-travis/install.html#installing-pan…> from our development channel. Builds for osx-64,linux-64,win-64 for Python 2.7 and Python 3.4 are all available. conda install pandas -c pandas Thanks to all who made this release happen. It is a very large release! Jeff

1 0

cross compile numpy
by chenchao＠inhand.com.cn 11 Sep '15

11 Sep '15

hi: I have to use numpy package. My python runs on my embedded arm device. So, how do i cross compile numpy? ps: my embedded arm device supports floating point operation.

1 0

Commit rights for Stephan Hoyer
by Charles R Harris 10 Sep '15

10 Sep '15

Hi All, Stephan Hoyer now has commit rights to Numpy to compliment all the other projects of which he is a member. Congratutlations Stephan. Chuck

2 1

ANN: bcolz 0.11.0 released
by Francesc Alted 09 Sep '15

09 Sep '15

======================= Announcing bcolz 0.11.0 ======================= What's new ========== Although this is mostly a maintenance release that fixes some bugs, the setup.py is entirely based now in setuptools and has been greatly modernized to use a new versioning system. Just this deserves a bump in the minor version. Thanks to Gabi Davar (@mindw) for such a nice improvement. Also, many improvements to the Continuous Integration part (and hence not directly visible to users) and others have been made by Francesc Elies (@FrancescElies). Thanks for his quiet but effective work. And last but not least, I would like to announce that Valentin Haenel (@esc) just stepped down as release manager. Thanks Valentin for all the hard work that you put in making bcolz a better piece of software! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted

1 0

Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
by Matthew Brett 06 Sep '15

06 Sep '15

Hi, Splitting this one off too because it's a rather different discussion, although related. On Tue, Aug 25, 2015 at 11:03 AM, Nathaniel Smith <njs(a)pobox.com> wrote: [snip] > Formalizing our governance/decision making > ========================================== > > This was a major focus of discussion. At a high level, the consensus > was to steal IPython's governance document ("IPEP 29") and modify it > to remove its use of a BDFL as a "backstop" to normal community > consensus-based decision, and replace it with a new "backstop" based > on Apache-project-style consensus voting amongst the core team. Here's a plea to avoid a 'core' structure if at all possible. Historically it seems to have some severe risks, and experienced people have blamed this structure for the decline of various projects including NetBSD and Xfree86, summaries here: http://asterisk.dynevor.org/melting-core.html http://asterisk.dynevor.org/xfree-forked.html In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate. There's also evidence from the NetBSD / OpenBSD split [1] and the XFree86 / X.org split [2] - that the core structure can lead to bad decisions being taken in private that no or few members of the core group are prepared to defend. I guess what is happening is that distributed responsibility leads to poor accountability, and therefore poor decisions. So, I hope very much we can avoid that trap in our own governance. Best, Matthew [1] http://mail-index.netbsd.org/netbsd-users/1994/12/23/0000.html [2] http://www.xfree86.org/pipermail/forum/2003-March/001997.html

11 45

meshgrid dtype casting
by josef.pktd＠gmail.com 04 Sep '15

04 Sep '15

I'm trying to build a meshgrid with small nonnegative integers default is int32 >>> np.meshgrid([0,1,2], [0,1])[0].dtype dtype('int32') If I use uint, then the arrays are upcast to int64 - Why? >>> np.meshgrid(np.array([0,1,2], np.uint), np.array([0,1], np.uint))[0].dtype dtype('int64') broadcast_arrays preserves dtype >>> np.broadcast_arrays(np.array([0,1,2], np.uint)[:,None], np.array([0,1], np.uint)[None, :]) [array([[0, 0], [1, 1], [2, 2]], dtype=uint32), array([[0, 1], [0, 1], [0, 1]], dtype=uint32)] with uint8 >>> np.broadcast_arrays(np.array([0,1,2], np.uint8)[:,None], np.array([0,1], np.uint8)[None, :]) [array([[0, 0], [1, 1], [2, 2]], dtype=uint8), array([[0, 1], [0, 1], [0, 1]], dtype=uint8)] >>> np.meshgrid(np.array([0,1,2], np.uint8), np.array([0,1], np.uint8))[0].dtype dtype('int32') >>> Winpython 64 bit >>> np.__version__ '1.9.2rc1' Josef

1 0

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion September 2015