Mailman 3 September 2015 - NumPy-Discussion

Re: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure?
by Nathaniel Smith Sept. 13, 2015

Sept. 13, 2015

[This is getting fairly off-topic for python-ideas (since AFAICT there is no particular reason right now to add a new deterministic generator to the stdlib), so CC'ing numpy-discussion and I'd suggest followups be directed to there alone.] On Thu, Sep 10, 2015 at 10:41 AM, Robert Kern <robert.kern(a)gmail.com> wrote: > On 2015-09-10 04:56, Nathaniel Smith wrote: >> >> On Wed, Sep 9, 2015 at 8:35 PM, Tim Peters <tim.peters(a)gmail.com> wrote: >>> >>> … [View More]There are some clean and easy approaches to this based on >>> crypto-inspired schemes, but giving up crypto strength for speed. If >>> you haven't read it, this paper is delightful: >>> >>> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf >> >> >> It really is! As AES acceleration instructions become more common >> (they're now standard IIUC on x86, x86-64, and even recent ARM?), even >> just using AES in CTR mode becomes pretty compelling -- it's fast, >> deterministic, provably equidistributed, *and* cryptographically >> secure enough for many purposes. > > > I'll also recommend the PCG paper (and algorithm) as the author's > cross-PRNGs comparisons have been bandied about in this thread already. The > paper lays out a lot of the relevant issues and balances the various > qualities that are important: statistical quality, speed, and security (of > various flavors). > > http://www.pcg-random.org/paper.html > > I'm actually not that impressed with Random123. The core idea is nice and > clean, but the implementation is hideously complex. I'm curious what you mean by this? In terms of the code, or...? I haven't looked at the code, but the simplest generator they recommend in the paper is literally just def rng_stream(seed): counter = 0 while True: # AES128 takes a 128 bit key and 128 bits of data and returns 128 bits of encrypted data val = AES128(key=seed, data=counter) yield low_64_bits(val) yield high_64_bits(val) counter += 1 which gives a 64-bit generator with a period of 2^129. They benchmark it as faster than the Mersenne Twister on modern CPUs (<2 cycles-per-byte on recent x86, x86-64, ARMv8), it requires less scratch space, is incredibly simple to work with -- you can parallelize it, get independent random streams, etc., in a totally trivial way -- and has a *way* stronger guarantee of random-looking-ness than merely passing TestU01. The downsides are that it does still require 176 bytes of read-only scratch storage (used to cache an expanded version of the "key"), the need for a modern CPU to get that speed, and that it doesn't play well with GPUs. So they also provide a set of three more ad hoc generators designed to solve these problems. I'm not as convinced about these, but hey, they pass TestU01... The PCG paper does a much better job of all the other stuff *around* making a good RNG -- it has the nice website, clear comparisons, nice code, etc. -- which is definitely important. But to me the increase in speed and reduction in memory use doesn't seem worth it given how fast these generators are to start with + the nice properties of counter mode + and cryptographic guarantees of randomness that you get from AES, for code that's generally targeting non-embedded non-GPU systems. -n -- Nathaniel J. Smith -- http://vorpus.org [View Less]

1 0

Just to this list --- more details of the approach
by Travis Oliphant Sept. 12, 2015

Sept. 12, 2015

Hey all, To the NumPy list only, I'll at least give the highlights of the surgical approach I would like to get someone to work on -- I can help mentor and guide. These are just the highlights, but it should give someone familiar with the code the general gist. There are some details to work out, of course, but it could be done. It may be very similar to what Nathaniel is contemplating --- except I think breaking the ABI is the only way to really do this --- could be wrong but I'm not … [View More]

1 0

Looking for a developer who will work with me for at least 6 months to fix NumPy's dtype system.
by Travis Oliphant Sept. 12, 2015

Sept. 12, 2015

Hi all, Apologies for cross-posting, but I need to get the word out and twitter doesn't provide enough explanation. I've been working on a second edition of my "Guide to NumPy" book. It's been a time-pressured activity, but it's helped me put more meat around my ideas for how to fix NumPy's dtype system -- which I've been contemplating off an on for 8 years. I'm pretty sure I know exactly how to do it --- in a way that fits more cleanly into Python. It will take 3-6 months and will have … [View More]residual efforts needed that will last another 6 months --- making more types available with NumPy, improving calculations etc. This work will be done completely in public view and allow for public comment. It will not solve *all* of NumPy's problems, but it will put NumPy's dtype system on the footing it in retrospect should have been put on in the first place (if I had known then what I know now). It won't be a grandiose rewrite. It will be a pretty surgical fix to a few key places in the code. However, it will break the ABI and require recompilation of NumPy extensions (and so would need to be called NumPy 2.0). This is unavoidable, but I don't see any problem with breaking the ABI today given how easy it is to get distributions of Python these days from a variety of sources (including using conda --- but not only using conda). For those that remember what happened in Python dev land, the changes will be similar to when Guido changed Python 1.5.2 to Python 2.0. I can mentor and work closely with someone who will work on this and we will invite full participation and feedback from whomever in the community also wants to participate --- but I can't do it myself full time (and it needs someone full time+). Fortunately, I can pay someone to do it if they are willing to commit at least 6 months (it is not required to work at Continuum for this, but you can have a job at Continuum if you want one). I'm only looking for people who have enough experience with C or preferably the Python C-API. You also have to *want* to work on this. You need to be willing to work with me on the project directly and work to have a mind-meld with my ideas which will undoubtedly give rise to additional perspectives and ideas for later work for you. When I wrote NumPy 1.0, I put in 80+ hour weeks for about 6 months or more and then 60+ weeks for another year. I was pretty obsessed with it. This won't need quite that effort, but it will need something like it. Being able to move to Austin is a plus but not required. I can sponsor a visa for the right candidate as well (though it's not guaranteed you will get one with the immigration policies what they are). This is a labor of love for so many of us and my desire to help the dtype situation in NumPy comes from the same space that my desire to work on NumPy in the first place came. I will be interviewing people to work on this as not everyone who may want to will really be qualified to do it --- especially with so many people writing Cython these days instead of good-ole C-API code :-) Feel free to spread the news to anyone you can. I won't say more until I've found someone to work with me on this --- because I won't have the time to follow-up with any questions or comments. Even if I can't find someone I will publish the ideas --- but that also takes time and effort that is in short supply for me right now. If there is someone willing to fund this work, please let me know as well -- that could free up more of my time. Best, -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io [View Less]

1 0

review of #6247
by pizza＠netspace.net.au Sept. 12, 2015

Sept. 12, 2015

Hi all, first attempt at this: small doco patch to address https://github.com/numpy/numpy/issues/6247 https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=u…

2 3

review of #6191
by jason Sept. 12, 2015

Sept. 12, 2015

doco patch to address https://github.com/numpy/numpy/issues/6191 https://github.com/numpy/numpy/compare/master...pizzathief:issue6191

2 1

feature request - increment counter on write check
by Daniel Manson Sept. 11, 2015

Sept. 11, 2015

Originally posted as issue 6301 <https://github.com/numpy/numpy/issues/6301> on github. Presumably any block of code that modifies an ndarray's buffer is wrapped in a (thread safe?) check of the writable flag. Would it be possible to hold a counter rather than a simple bool flag and then increment the counter whenever you test the flag? Hopefully this would introduce only a tiny additional overhead, but would permit pseudo-hashing to test whether a mutable array has changed since you … [View More]last encountered it. Ideally this should be exposed a bit like python's __hash__ <https://docs.python.org/2/reference/datamodel.html#object.__hash__>method, lets say __mutablehash__, meaning a hash is returned but be warned that the object is mutable. For an ndarray, X, containing objects that themselves have a __mutablehash__ method (e.g. other ndarrays, or some user object), the X.__mutablehash__ method will need to do the full check over all constituent objects, or simply return None. Defining and API of this sort would make it possible to - for example - let pandas DataFrames also implement this interface. In terms of usage cases, the one I was motivated by was imagining improvements to the "variable explorer" in Spyder - roughly speaking, this widget's job is to always display an up-to-date summary of variables in current scope, e.g. currently it can show max/min and shape, but you could imagine also showing graphical summaries of the contents of an ndarray. If the widget could cache summaries and check which arrays have really changed it should be much faster/offer more features/be simpler internally. Note that pandas DataFrames are relevant here as an example of complex objects, containing ndarrays, which would benefit from being able to have their summaries cached. A more common/general usage case would be as a check in some kind of memoization process... #simple example... @memoize_please def hasnans(x): return np.any(np.isnan(x)) # more complex example... def convolve_fft(a,b, _cache={}): a_hash = mutablehash(a) b_hash = mutablehash(b) if a_hash not in _cache: _cache[a_hash] = fft(a) if b_hash not in _cache: _cache[b_hash] = fft(b) return ifft(_cache[a_hash] * _cache[b_hash]) A quick though on an implementation detail... I'm not sure exactly how to deal with the counter overflowing: perhaps if you treated counter==0 to mean not-writable (i.e. that would be the new version of the old write flag) then you might get some uint-wraparound checking for free (because when it wraps back around to zero the buffer ends up becoming locked)? Alternatively you could just say that no guarantee is given of wraparound being caught..though that might seriously impact on the range of possible uses. In summary... Hopefully the stuff needed to make __mutablehash__ work could be implemented simply by adding a single extra operation to the write-check (and maybe changing the footprint of the ndarray slightly to accomodate a counter). But I suspect someone will tell me that life is never that simple! [View Less]

5 7

ANN: pandas v0.17.0rc1 - RELEASE CANDIDATE
by Jeff Reback Sept. 11, 2015

Sept. 11, 2015

Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.17.0. Please try this RC and report any issues here: Pandas Issues <https://github.com/pydata/pandas/issues/10848> We will be releasing officially in 1-2 weeks or so. **RELEASE CANDIDATE 1** This is a major release from 0.16.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all … [View More]

1 0

cross compile numpy
by chenchao＠inhand.com.cn Sept. 10, 2015

Sept. 10, 2015

hi: I have to use numpy package. My python runs on my embedded arm device. So, how do i cross compile numpy? ps: my embedded arm device supports floating point operation.

1 0

Commit rights for Stephan Hoyer
by Charles R Harris Sept. 9, 2015

Sept. 9, 2015

Hi All, Stephan Hoyer now has commit rights to Numpy to compliment all the other projects of which he is a member. Congratutlations Stephan. Chuck

2 1

ANN: bcolz 0.11.0 released
by Francesc Alted Sept. 9, 2015

Sept. 9, 2015

======================= Announcing bcolz 0.11.0 ======================= What's new ========== Although this is mostly a maintenance release that fixes some bugs, the setup.py is entirely based now in setuptools and has been greatly modernized to use a new versioning system. Just this deserves a bump in the minor version. Thanks to Gabi Davar (@mindw) for such a nice improvement. Also, many improvements to the Continuous Integration part (and hence not directly visible to users) and others … [View More]have been made by Francesc Elies (@FrancescElies). Thanks for his quiet but effective work. And last but not least, I would like to announce that Valentin Haenel (@esc) just stepped down as release manager. Thanks Valentin for all the hard work that you put in making bcolz a better piece of software! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted [View Less]

1 0

Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
by Matthew Brett Sept. 6, 2015

Sept. 6, 2015

Hi, Splitting this one off too because it's a rather different discussion, although related. On Tue, Aug 25, 2015 at 11:03 AM, Nathaniel Smith <njs(a)pobox.com> wrote: [snip] > Formalizing our governance/decision making > ========================================== > > This was a major focus of discussion. At a high level, the consensus > was to steal IPython's governance document ("IPEP 29") and modify it > to remove its use of a BDFL as a "backstop" to normal … [View More]

11 45

meshgrid dtype casting
by josef.pktd＠gmail.com Sept. 4, 2015

Sept. 4, 2015

I'm trying to build a meshgrid with small nonnegative integers default is int32 >>> np.meshgrid([0,1,2], [0,1])[0].dtype dtype('int32') If I use uint, then the arrays are upcast to int64 - Why? >>> np.meshgrid(np.array([0,1,2], np.uint), np.array([0,1], np.uint))[0].dtype dtype('int64') broadcast_arrays preserves dtype >>> np.broadcast_arrays(np.array([0,1,2], np.uint)[:,None], np.array([0,1], np.uint)[None, :]) [array([[0, 0], [1, 1], [2, 2]], … [View More]

1 0

Notes from the numpy dev meeting at scipy 2015
by Nathaniel Smith Sept. 3, 2015

Sept. 3, 2015

Hi all, These are the notes from the NumPy dev meeting held July 7, 2015, at the SciPy conference in Austin, presented here so the list can keep up with what happens, and so you can give feedback. Please do give feedback, none of this is final! (Also, if anyone who was there notices anything I left out or mischaracterized, please speak up -- these are a lot of notes I'm trying to gather together, so I could easily have missed something!) Thanks to Jill Cowan and the rest of the SciPy … [View More]organizers for donating space and organizing logistics for us, and to the Berkeley Institute for Data Science for funding travel for Jaime, Nathaniel, and Sebastian. Attendees ========= Present in the room for all or part: Daniel Allan, Chris Barker, Sebastian Berg, Thomas Caswell, Jeff Reback, Jaime Fernández del Río, Chuck Harris, Nathaniel Smith, Stéfan van der Walt. (Note: I'm pretty sure this list is incomplete) Joining remotely for all or part: Stephan Hoyer, Julian Taylor. Formalizing our governance/decision making ========================================== This was a major focus of discussion. At a high level, the consensus was to steal IPython's governance document ("IPEP 29") and modify it to remove its use of a BDFL as a "backstop" to normal community consensus-based decision, and replace it with a new "backstop" based on Apache-project-style consensus voting amongst the core team. I'll send out a proper draft of this shortly for further discussion. Development roadmap =================== General consensus: Let's assume NumPy is going to remain important indefinitely, and try to make it better, instead of waiting for something better to come along. (This is unlikely to be wasted effort even if something better does come along, and it's hardly a sure thing that that will happen anyway.) Let's focus on evolving numpy as far as we can without major break-the-world changes (no "numpy 2.0", at least in the foreseeable future). And, as a target for that evolution, let's change our focus from numpy as "NumPy is the library that gives you the np.ndarray object (plus some attached infrastructure)", to "NumPy provides the standard framework for working with arrays and array-like objects in Python" This means, creating defined interfaces between array-like objects / ufunc objects / dtype objects, so that it becomes possible for third parties to add their own and mix-and-match. Right now ufuncs are pretty good at this, but if you want a new array class or dtype then in most cases you pretty much have to modify numpy itself. Vision: instead of everyone who wants a new container type having to reimplement all of numpy, Alice can implement an array class using (sparse / distributed / compressed / tiled / gpu / out-of-core / delayed / ...) storage, pass it to code that was written using direct calls to np.* functions, and it just works. (Instead of np.sin being "the way you calculate the sine of an ndarray", it's "the way you calculate the sine of any array-like container object".) Vision: Darryl can implement a new dtype for (categorical data / astronomical dates / integers-with-missing-values / ...) without having to touch the numpy core. Vision: Chandni can then come along and combine them by doing a = alice_array([...], dtype=darryl_dtype) and it just works. Vision: no-one is tempted to subclass ndarray, because anything you can do with an ndarray subclass you can also easily do by defining your own new class that implements the "array protocol". Supporting third-party array types ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sub-goals: - Get __numpy_ufunc__ done, which will cover a good chunk of numpy's API right there. - Go through the rest of the stuff in numpy, and figure out some story for how to let it handle third-party array classes: - ufunc ALL the things: Some things can be converted directly into (g)ufuncs and then use __numpy_ufunc__ (e.g., np.std); some things could be converted into (g)ufuncs if we extended the (g)ufunc interface a bit (e.g. np.sort, np.matmul). - Some things probably need their own __numpy_ufunc__-like extensions (__numpy_concatenate__?) - Provide tools to make it easier to implement the more complicated parts of an array object (e.g. the bazillion different methods, many of which are ufuncs in disguise, or indexing) - Longer-run interesting research project: __numpy_ufunc__ requires that one or the other object have explicit knowledge of how to handle the other, so to handle binary ufuncs with N array types you need something like N**2 __numpy_ufunc__ code paths. As an alternative, if there were some interface that an object could export that provided the operations nditer needs to efficiently iterate over (chunks of) it, then you would only need N implementations of this interface to handle all N**2 operations. This would solve a lot of problems for projects like: - blosc - dask - distarray - numpy.ma - pandas - scipy.sparse - xray Supporting third-party dtypes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We already have something like a C level "dtype protocol". Conceptually, the way you define a new dtype is by defining a new class whose instances have data attributes defining the parameters of the dtype (what fields are in *this* record dtype, how many characters are in *this* string dtype, what units are used for *this* datetime64, etc.), and you define a bunch of methods to do things like convert an object from a Python object to your dtype or vice-versa, to copy an array of your dtype from one place to another, to cast to and from your new dtype, etc. This part is great. The problem is, in the current implementation, we don't actually use the Python object system to define these classes / attributes / methods. Instead, all possible dtypes are jammed into a single Python-level class, whose struct has fields for the union of all possible dtype's attributes, and instead of Python-style method slots there's just a big table of function pointers attached to each object. So the main proposal is that we keep the basic design, but switch it so that the float64 dtype, the int64 dtype, etc. actually literally are subclasses of np.dtype, each implementing their own fields and Python-style methods. Some of the pieces involved in doing this: - The current dtype methods should be cleaned up -- e.g. 'dot' and 'less_than' are both dtype methods, when conceptually they're much more like ufuncs. - The ufunc inner-loop interface currently does not get a reference to the dtype object, so they can't see its attributes and this is a big obstacle to many interesting dtypes (e.g., it's hard to implement np.equal for categoricals if you don't know what categories each has). So we need to add new arguments to the core ufunc loop signature. (Fortunately this can be done in a backwards-compatible way.) - We need to figure out what exactly the dtype methods should be, and add them to the dtype class (possibly with backwards compatibility shims for anyone who is accessing PyArray_ArrFuncs directly). - Casting will be possibly the trickiest thing to work out, though the basic idea of using dunder-dispatch-like __cast__ and __rcast__ methods seems workable. (Encouragingly, this is also exactly what dynd also does, though unfortunately dynd does not yet support user-defined dtypes even to the extent that numpy does, so there isn't much else we can steal from them.) - We may also want to rethink the casting rules while we're at it, since they have some very weird corners right now (e.g. see [https://github.com/numpy/numpy/issues/6240]) - We need to migrate the current dtypes over to the new system, which can be done in stages: - First stick them all in a single "legacy dtype" class whose methods just dispatch to the PyArray_ArrFuncs per-object "method table" - Then move each of them into their own classes - We should provide a Python-level wrapper for the protocol, so that you can call dtype methods from Python - And vice-versa, it should be possible to subclass dtype at the Python level - etc. Fortunately, AFAICT pretty much all of this can be done while maintaining backwards compatibility (though we may want to break some obscure cases to avoid expending *too* much effort with weird backcompat contortions that will only help a vanishingly small proportion of the userbase), and a lot of the above changes can be done as semi-independent mini-projects, so there's no need for some branch to go off and spend a year rewriting the world. Obviously there are still a lot of details to work out, though. But overall, there was widespread agreement that this is one of the #1 pain points for our users (e.g. it's the single main request from pandas), and fixing it is very high priority. Some features that would become straightforward to implement (e.g. even in third-party libraries) if this were fixed: - missing value support - physical unit tracking (meters / seconds -> array of velocity; meters + seconds -> error) - better and more diverse datetime representations (e.g. datetimes with attached timezones, or using funky geophysical or astronomical calendars) - categorical data - variable length strings - strings-with-encodings (e.g. latin1) - forward mode automatic differentiation (write a function that computes f(x) where x is an array of float64; pass that function an array with a special dtype and get out both f(x) and f'(x)) - probably others I'm forgetting right now I should also note that there was one substantial objection to this plan, from Travis Oliphant (in discussions later in the conference). I'm not confident I understand his objections well enough to reproduce them here, though -- perhaps he'll elaborate. Money ===== There was an extensive discussion on the topic of: "if we had money, what would we do with it?" This is partially motivated by the realization that there are a number of sources that we could probably get money from, if we had a good story for what we wanted to do, so it's not just an idle question. Points of general agreement: - Doing the in-person meeting was a good thing. We should plan do that again, at least once a year. So one thing to spend money on is travel subsidies to make sure that happens and is productive. - While it's tempting to imagine hiring junior people for the more frustrating/boring work like maintaining buildbots, release infrastructure, updating docs, etc., this seems difficult to do realistically with our current resources -- how do we hire for this, who would manage them, etc.? - On the other hand, the general feeling was that if we found the money to hire a few more senior people who could take care of themselves more, then that would be good and we could realistically absorb that extra work without totally unbalancing the project. - A major open question is how we would recruit someone for a position like this, since apparently all the obvious candidates who are already active on the NumPy team already have other things going on. [For calibration on how hard this can be: NYU has apparently had an open position for a year with the job description of "come work at NYU full-time with a private-industry-competitive-salary on whatever your personal open-source scientific project is" (!) and still is having an extremely difficult time filling it: [http://cds.nyu.edu/research-engineer/]] - General consensus though was that there isn't much to be done about this though, except try it and see. - (By the way, if you're someone who's reading this and potentially interested in like a postdoc or better working on numpy, then let's talk...) More specific changes to numpy that had general consensus, but don't really fit into a high-level roadmap ========================================================================================================= - Resolved: we should merge multiarray.so and umath.so into a single extension module, so that they can share utility code without the current awkward contortions. - Resolved: we should start hiding new fields in the ufunc and dtype structs as soon as possible going forward. (I.e. they would not be present in the version of the structs that are exposed through the C API, but internally we would use a more detailed struct.) - Mayyyyyybe we should even go ahead and hide the subset of the existing fields that are really internal details that no-one should be using. If we did this without changing anything else then it would preserve ABI (the fields would still be where existing compiled extensions expect them to be, if any such extensions exist) while breaking API (trying to compile such extensions would give a clear error), so would be a smoother ramp if we think we need to eventually break those fields for real. (As discussed above, there are a bunch of fields in the dtype base class that only make sense for specific dtype subclasses, e.g. only record dtypes need a list of field names, but right now all dtypes have one anyway. So it would be nice to remove these from the base class entirely, but that is potentially ABI-breaking.) - Resolved: np.array should never return an object array unless explicitly requested (e.g. with dtype=object); it just causes too many surprising problems. - First step: add a deprecation warning - Eventually: make it an error. - The matrix class - Resolved: We won't add warnings yet, but we will prominently document that it is deprecated and should be avoided where-ever possible. - Stéfan van der Walt volunteers to do this. - We'd all like to deprecate it properly, but the feeling was that the precondition for this is for scipy.sparse to provide sparse "arrays" that don't return np.matrix objects on ordinary operatoins. Until that happens we can't reasonably tell people that using np.matrix is a bug. - Resolved: we should add a similar prominent note to the "subclassing ndarray" documentation, warning people that this is painful and barely works and please don't do it if you have any alternatives. - Resolved: we want more, smaller releases -- every 6 months at least, aiming to go even faster (every 4 months?) - On the question of using Cython inside numpy core: - Everyone agrees that there are places where this would be an improvement (e.g., Python<->C interfaces, and places "when you want to do computer science", e.g. complicated algorithmic stuff like graph traversals) - Chuck wanted it to be clear though that he doesn't think it would be a good goal to try and rewrite all of numpy in Cython -- there also exist places where Cython ends up being "an uglier version of C". No-one disagreed. - Our text reader is apparently not very functional on Python 3, and generally slow and hard to work with. - Resolved: We should extract Pandas's awesome text reader/parser and convert it into its own package, that could then become a new backend for both pandas and numpy.loadtxt. - Jeff thinks this is a great idea - Thomas Caswell volunteers to do the extraction. - We should work on improving our tools for evolving the ABI, so that we will eventually be less constrained by decisions made decades ago. - One idea that had a lot of support was to switch from our current append-only C-API to a "sliding window" API based on explicit versions. So a downstream package might say #define NUMPY_API_VERSION 4 and they'd get the functions and behaviour provided in "version 4" of the numpy C api. If they wanted to get access to new stuff that was added in version 5, then they'd need to switch that #define, and at the same time clean up any usage of stuff that was removed or changed in version 5. And to provide a smooth migration path, one version of numpy would support multiple versions at once, gradually deprecating and dropping old versions. - If anyone wants to help bring pip up to scratch WRT tracking ABI dependencies (e.g., 'pip install numpy==<version with new ABI>' -> triggers rebuild of scipy against the new ABI), then that would be an extremely useful thing. Policies that should be documented ================================== ...together with some notes about what the contents of the document should be: How we manage bugs in the bug tracker. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Github "milestones" should *only* be assigned to release-blocker bugs (which mostly means "regression from the last release"). In particular, if you're tempted to push a bug forward to the next release... then it's clearly not a blocker, so don't set it to the next release's milestone, just remove the milestone entirely. (Obvious exception to this: deprecation followup bugs where we decide that we want to keep the deprecation around a bit longer are a case where a bug actually does switch from being a blocker to release 1.x to being a blocker for release 1.(x+1).) - Don't hesitate to close an issue if there's no way forward -- e.g. a PR where the author has disappeared. Just post a link to this policy and close, with a polite note that we need to keep our tracker useful as a todo list, but they're welcome to re-open if things change. Deprecations and breakage policy: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - How long do we need to keep DeprecationWarnings around before we break things? This is tricky because on the one hand an aggressive (short) deprecation period lets us deliver new features and important cleanups more quickly, but on the other hand a too-aggressive deprecation period is difficult for our more conservative downstream users. - Idea that had the most support: pick a somewhat-aggressive warning period as our default, and make a rule that if someone asks for an extension during the beta cycle for the release that removes it, then we put it back for another release or two worth of grace period. (While also possibly upgrading the warning to be more visible during the grace period.) This gives us deprecation periods that are more adaptive on a case-by-case basis. - Lament: it would be really nice if we could get more people to test our beta releases, because in practice right now 1.x.0 ends up being where we actually the discover all the bugs, and 1.x.1 is where it actually becomes usable. Which sucks, and makes it difficult to have a solid policy about what counts as a regression, etc. Is there anything we can do about this? - ABI breakage: we distinguish between an ABI break that breaks everything (e.g., "import scipy" segfaults), versus an ABI break that breaks an occasional rare case (e.g., only apps that poke around in some obscure corner of some struct are affected). - The "break-the-world" type remains off-limit for now: the pain is still too large (conda helps, but there are lots of people who don't use conda!), and there aren't really any compelling improvements that this would enable anyway. - For the "break-0.1%-of-users" type, it is *not* ruled out by fiat, though we remain conservative: we should treat it like other API breaks in principle, and do a careful case-by-case analysis of the details of the situation, taking into account what kind of code would be broken, how common these cases are, how important the benefits are, whether there are any specific mitigation strategies we can use, etc. -- with this process of course taking into account that a segfault is nastier than a Python exception. Other points that were discussed ================================ - There was inconclusive discussion of what we should do with dot() in the places where it disagrees with the PEP 465 matmul semantics (specifically this is when both arguments have ndim >= 3, or one argument has ndim == 0). - The concern is that the current behavior is not very useful, and as far as we can tell no-one is using it; but, as people get used to the more-useful PEP 465 behavior, they will increasingly try to use it on the assumption that np.dot will work the same way, and this will create pain for lots of people. So Nathaniel argued that we should start at least issuing a visible warning when people invoke the corner-case behavior. - But OTOH, np.dot is such a core piece of infrastructure, and there's such a large landscape of code out there using numpy that we can't see, that others were reasonably wary of making any change. - For now: document prominently, but no change in behavior. Links to raw notes ================== Main page: [https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting] Notes from the meeting proper: [https://docs.google.com/document/d/1IJcYdsHtk8MVAM4AZqFDBSf_nVG-mrB4Tv2bh9u…] Slides from the followup BoF: [https://gist.github.com/njsmith/eb42762054c88e810786/raw/b74f978ce10a972831…] Notes from the followup BoF: [https://docs.google.com/document/d/11AuTPms5dIPo04JaBOWEoebXfk-tUzEZ-CvFnLI…] -n -- Nathaniel J. Smith -- http://vorpus.org [View Less]

15 26

np.sign and object comparisons
by Jaime Fernández del Río Sept. 2, 2015

Sept. 2, 2015

There's been some work going on recently on Py2 vs Py3 object comparisons. If you want all the background, see gh-6265 <https://github.com/numpy/numpy/issues/6265> and follow the links there. There is a half baked PR in the works, gh-6269 <https://github.com/numpy/numpy/pull/6269>, that tries to unify behavior and fix some bugs along the way, by replacing all 2.x uses of PyObject_Compare with several calls to PyObject_RichCompareBool, which is available on 2.6, the oldest Python … [View More]

6 8

Numpy helper function for __getitem__?
by Fabien Sept. 2, 2015

Sept. 2, 2015

Folks, My search engine was not able to help me on this one, possibly because I don't know exactly *what* I am looking for. I need to override __getitem__ for a class that wrapps a numpy array. I know the dimensions of my array (which can be variable from instance to instance), and I know what I want to do: for one preselected dimension, I need to select another slice than requested by the user, do something with the data, and return the variable. I am looking for a function that helps … [View More]

5 7

UTC-based datetime64
by Francesc Alted Sept. 1, 2015

Sept. 1, 2015

Hi, We've found that NumPy uses the local TZ for printing datetime64 timestamps: In [22]: t = datetime.utcnow() In [23]: print t 2015-08-26 11:52:10.662745 In [24]: np.array([t], dtype="datetime64[s]") Out[24]: array(['2015-08-26T13:52:10+0200'], dtype='datetime64[s]') Googling for a way to print UTC out of the box, the best thing I could find is: In [40]: [str(i.item()) for i in np.array([t], dtype="datetime64[s]")] Out[40]: ['2015-08-26 11:52:10'] Now, is there a better way to specify … [View More]

3 2

Numpty FFT.FFT slow with certain samples
by Joseph Codadeen Sept. 1, 2015

Sept. 1, 2015

Hi, I am a numpy newbie. I have two wav files, one that numpy takes a long time to process the FFT. They was created within audacity using white noise and silence for gaps. my_1_minute_noise_with_gaps.wavmy_1_minute_noise_with_gaps_truncated.wav The files are very similar in the following way; 1. is white noise with silence gaps on every 15 second interval.2. is 1. but slightly shorter, i.e. I trimmed some ms off the end but it still has the last gap at 60s. The code I am using processes the … [View More]

10 17

Cythonizing some of NumPy
by David Cournapeau Sept. 1, 2015

Sept. 1, 2015

Hi there, Reading Nathaniel summary from the numpy dev meeting, it looks like there is a consensus on using cython in numpy for the Python-C interfaces. This has been on my radar for a long time: that was one of my rationale for splitting multiarray into multiple "independent" .c files half a decade ago. I took the opportunity of EuroScipy sprints to look back into this, but before looking more into it, I'd like to make sure I am not going astray: 1. The transition has to be gradual 2. The … [View More]

2 2