Mailman 3 September 2015 - NumPy-Discussion

Re: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure?
by Nathaniel Smith 14 Sep '15

14 Sep '15

[This is getting fairly off-topic for python-ideas (since AFAICT there is no particular reason right now to add a new deterministic generator to the stdlib), so CC'ing numpy-discussion and I'd suggest followups be directed to there alone.] On Thu, Sep 10, 2015 at 10:41 AM, Robert Kern <robert.kern(a)gmail.com> wrote: > On 2015-09-10 04:56, Nathaniel Smith wrote: >> >> On Wed, Sep 9, 2015 at 8:35 PM, Tim Peters <tim.peters(a)gmail.com> wrote: >>> >>> There are some clean and easy approaches to this based on >>> crypto-inspired schemes, but giving up crypto strength for speed. If >>> you haven't read it, this paper is delightful: >>> >>> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf >> >> >> It really is! As AES acceleration instructions become more common >> (they're now standard IIUC on x86, x86-64, and even recent ARM?), even >> just using AES in CTR mode becomes pretty compelling -- it's fast, >> deterministic, provably equidistributed, *and* cryptographically >> secure enough for many purposes. > > > I'll also recommend the PCG paper (and algorithm) as the author's > cross-PRNGs comparisons have been bandied about in this thread already. The > paper lays out a lot of the relevant issues and balances the various > qualities that are important: statistical quality, speed, and security (of > various flavors). > > http://www.pcg-random.org/paper.html > > I'm actually not that impressed with Random123. The core idea is nice and > clean, but the implementation is hideously complex. I'm curious what you mean by this? In terms of the code, or...? I haven't looked at the code, but the simplest generator they recommend in the paper is literally just def rng_stream(seed): counter = 0 while True: # AES128 takes a 128 bit key and 128 bits of data and returns 128 bits of encrypted data val = AES128(key=seed, data=counter) yield low_64_bits(val) yield high_64_bits(val) counter += 1 which gives a 64-bit generator with a period of 2^129. They benchmark it as faster than the Mersenne Twister on modern CPUs (<2 cycles-per-byte on recent x86, x86-64, ARMv8), it requires less scratch space, is incredibly simple to work with -- you can parallelize it, get independent random streams, etc., in a totally trivial way -- and has a *way* stronger guarantee of random-looking-ness than merely passing TestU01. The downsides are that it does still require 176 bytes of read-only scratch storage (used to cache an expanded version of the "key"), the need for a modern CPU to get that speed, and that it doesn't play well with GPUs. So they also provide a set of three more ad hoc generators designed to solve these problems. I'm not as convinced about these, but hey, they pass TestU01... The PCG paper does a much better job of all the other stuff *around* making a good RNG -- it has the nice website, clear comparisons, nice code, etc. -- which is definitely important. But to me the increase in speed and reduction in memory use doesn't seem worth it given how fast these generators are to start with + the nice properties of counter mode + and cryptographic guarantees of randomness that you get from AES, for code that's generally targeting non-embedded non-GPU systems. -n -- Nathaniel J. Smith -- http://vorpus.org

1 0

Just to this list --- more details of the approach
by Travis Oliphant 13 Sep '15

13 Sep '15

Hey all, To the NumPy list only, I'll at least give the highlights of the surgical approach I would like to get someone to work on -- I can help mentor and guide. These are just the highlights, but it should give someone familiar with the code the general gist. There are some details to work out, of course, but it could be done. It may be very similar to what Nathaniel is contemplating --- except I think breaking the ABI is the only way to really do this --- could be wrong but I'm not wiling to risk *not* just breaking the ABI. 1) Create a new meta-type in C (call it dtype) 2) Create Python Classes (in C) that are instances of this meta-type for each "kind" of data-type 3) Make PyArray_Descr * be a reference to one of these new objects (which can be built either in C or Python) and should be published outside NumPy as well. 4) Remove most of the "per-type function calls" in PyArray_ArrFuncs --- instead replacing those with the Generalized Ufunc equivalents and expand the capability of Generalized Ufuncs 5) Keep the Array Scalar Types but change them so that they also use the dtype meta-type as their foundation and mixin an array-methods type. Also, have these be in a separate project from NumPy itself. 6) The current void* would be replaced with real Python classes instead of structured arrays being shoved through a single data-type. 7) The documented ways to spell a dtype would be reduced --- but backwards compatibility would be preserved. 8) Make sure Numba can create these Descriptor objects with Ahead of Time Compilation and start to move code of NumPy to Numba 9) Ensure the Generalized Ufunc framework can take the data-type as an argument so that *all* data-types can participate in the general multi-method approach. There is more to it, but that is the basic idea. Please forgive me if I can't respond to any feedback from the list in a timely way. I will as I can. -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io

1 0

Looking for a developer who will work with me for at least 6 months to fix NumPy's dtype system.
by Travis Oliphant 13 Sep '15

13 Sep '15

Hi all, Apologies for cross-posting, but I need to get the word out and twitter doesn't provide enough explanation. I've been working on a second edition of my "Guide to NumPy" book. It's been a time-pressured activity, but it's helped me put more meat around my ideas for how to fix NumPy's dtype system -- which I've been contemplating off an on for 8 years. I'm pretty sure I know exactly how to do it --- in a way that fits more cleanly into Python. It will take 3-6 months and will have residual efforts needed that will last another 6 months --- making more types available with NumPy, improving calculations etc. This work will be done completely in public view and allow for public comment. It will not solve *all* of NumPy's problems, but it will put NumPy's dtype system on the footing it in retrospect should have been put on in the first place (if I had known then what I know now). It won't be a grandiose rewrite. It will be a pretty surgical fix to a few key places in the code. However, it will break the ABI and require recompilation of NumPy extensions (and so would need to be called NumPy 2.0). This is unavoidable, but I don't see any problem with breaking the ABI today given how easy it is to get distributions of Python these days from a variety of sources (including using conda --- but not only using conda). For those that remember what happened in Python dev land, the changes will be similar to when Guido changed Python 1.5.2 to Python 2.0. I can mentor and work closely with someone who will work on this and we will invite full participation and feedback from whomever in the community also wants to participate --- but I can't do it myself full time (and it needs someone full time+). Fortunately, I can pay someone to do it if they are willing to commit at least 6 months (it is not required to work at Continuum for this, but you can have a job at Continuum if you want one). I'm only looking for people who have enough experience with C or preferably the Python C-API. You also have to *want* to work on this. You need to be willing to work with me on the project directly and work to have a mind-meld with my ideas which will undoubtedly give rise to additional perspectives and ideas for later work for you. When I wrote NumPy 1.0, I put in 80+ hour weeks for about 6 months or more and then 60+ weeks for another year. I was pretty obsessed with it. This won't need quite that effort, but it will need something like it. Being able to move to Austin is a plus but not required. I can sponsor a visa for the right candidate as well (though it's not guaranteed you will get one with the immigration policies what they are). This is a labor of love for so many of us and my desire to help the dtype situation in NumPy comes from the same space that my desire to work on NumPy in the first place came. I will be interviewing people to work on this as not everyone who may want to will really be qualified to do it --- especially with so many people writing Cython these days instead of good-ole C-API code :-) Feel free to spread the news to anyone you can. I won't say more until I've found someone to work with me on this --- because I won't have the time to follow-up with any questions or comments. Even if I can't find someone I will publish the ideas --- but that also takes time and effort that is in short supply for me right now. If there is someone willing to fund this work, please let me know as well -- that could free up more of my time. Best, -Travis -- *Travis Oliphant* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io

1 0

review of #6247
by pizza＠netspace.net.au 12 Sep '15

12 Sep '15

Hi all, first attempt at this: small doco patch to address https://github.com/numpy/numpy/issues/6247 https://github.com/numpy/numpy/compare/master...pizzathief:issue6247?diff=u…

2 3

review of #6191
by jason 12 Sep '15

12 Sep '15

doco patch to address https://github.com/numpy/numpy/issues/6191 https://github.com/numpy/numpy/compare/master...pizzathief:issue6191

2 1

feature request - increment counter on write check
by Daniel Manson 11 Sep '15

11 Sep '15

Originally posted as issue 6301 <https://github.com/numpy/numpy/issues/6301> on github. Presumably any block of code that modifies an ndarray's buffer is wrapped in a (thread safe?) check of the writable flag. Would it be possible to hold a counter rather than a simple bool flag and then increment the counter whenever you test the flag? Hopefully this would introduce only a tiny additional overhead, but would permit pseudo-hashing to test whether a mutable array has changed since you last encountered it. Ideally this should be exposed a bit like python's __hash__ <https://docs.python.org/2/reference/datamodel.html#object.__hash__>method, lets say __mutablehash__, meaning a hash is returned but be warned that the object is mutable. For an ndarray, X, containing objects that themselves have a __mutablehash__ method (e.g. other ndarrays, or some user object), the X.__mutablehash__ method will need to do the full check over all constituent objects, or simply return None. Defining and API of this sort would make it possible to - for example - let pandas DataFrames also implement this interface. In terms of usage cases, the one I was motivated by was imagining improvements to the "variable explorer" in Spyder - roughly speaking, this widget's job is to always display an up-to-date summary of variables in current scope, e.g. currently it can show max/min and shape, but you could imagine also showing graphical summaries of the contents of an ndarray. If the widget could cache summaries and check which arrays have really changed it should be much faster/offer more features/be simpler internally. Note that pandas DataFrames are relevant here as an example of complex objects, containing ndarrays, which would benefit from being able to have their summaries cached. A more common/general usage case would be as a check in some kind of memoization process... #simple example... @memoize_please def hasnans(x): return np.any(np.isnan(x)) # more complex example... def convolve_fft(a,b, _cache={}): a_hash = mutablehash(a) b_hash = mutablehash(b) if a_hash not in _cache: _cache[a_hash] = fft(a) if b_hash not in _cache: _cache[b_hash] = fft(b) return ifft(_cache[a_hash] * _cache[b_hash]) A quick though on an implementation detail... I'm not sure exactly how to deal with the counter overflowing: perhaps if you treated counter==0 to mean not-writable (i.e. that would be the new version of the old write flag) then you might get some uint-wraparound checking for free (because when it wraps back around to zero the buffer ends up becoming locked)? Alternatively you could just say that no guarantee is given of wraparound being caught..though that might seriously impact on the range of possible uses. In summary... Hopefully the stuff needed to make __mutablehash__ work could be implemented simply by adding a single extra operation to the write-check (and maybe changing the footprint of the ndarray slightly to accomodate a counter). But I suspect someone will tell me that life is never that simple!

5 7

ANN: pandas v0.17.0rc1 - RELEASE CANDIDATE
by Jeff Reback 11 Sep '15

11 Sep '15

Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.17.0. Please try this RC and report any issues here: Pandas Issues <https://github.com/pydata/pandas/issues/10848> We will be releasing officially in 1-2 weeks or so. **RELEASE CANDIDATE 1** This is a major release from 0.16.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. Highlights include: - Release the Global Interpreter Lock (GIL) on some cython operations, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Plotting methods are now available as attributes of the .plot accessor, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The sorting API has been revamped to remove some long-time inconsistencies, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Support for a datetime64[ns] with timezones as a first-class dtype, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The default for to_datetime will now be to raise when presented with unparseable formats, previously this would return the original input, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - The default for dropna in HDFStore has changed to False, to store by default all rows even if they are all NaN, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Support for Series.dt.strftime to generate formatted strings for datetime-likes, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Development installed versions of pandas will now have PEP440 compliant version strings GH9518 <https://github.com/pydata/pandas/issues/9518> - Development support for benchmarking with the Air Speed Velocity library GH8316 <https://github.com/pydata/pandas/pull/8316> - Support for reading SAS xport files, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> - Removal of the automatic TimeSeries broadcasting, deprecated since 0.8.0, see here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0170…> See the Whatsnew <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html> for much more information. Best way to get this is to install via conda <http://pandas-docs.github.io/pandas-docs-travis/install.html#installing-pan…> from our development channel. Builds for osx-64,linux-64,win-64 for Python 2.7 and Python 3.4 are all available. conda install pandas -c pandas Thanks to all who made this release happen. It is a very large release! Jeff

1 0

cross compile numpy
by chenchao＠inhand.com.cn 11 Sep '15

11 Sep '15

hi: I have to use numpy package. My python runs on my embedded arm device. So, how do i cross compile numpy? ps: my embedded arm device supports floating point operation.

1 0

Commit rights for Stephan Hoyer
by Charles R Harris 10 Sep '15

10 Sep '15

Hi All, Stephan Hoyer now has commit rights to Numpy to compliment all the other projects of which he is a member. Congratutlations Stephan. Chuck

2 1

ANN: bcolz 0.11.0 released
by Francesc Alted 09 Sep '15

09 Sep '15

======================= Announcing bcolz 0.11.0 ======================= What's new ========== Although this is mostly a maintenance release that fixes some bugs, the setup.py is entirely based now in setuptools and has been greatly modernized to use a new versioning system. Just this deserves a bump in the minor version. Thanks to Gabi Davar (@mindw) for such a nice improvement. Also, many improvements to the Continuous Integration part (and hence not directly visible to users) and others have been made by Francesc Elies (@FrancescElies). Thanks for his quiet but effective work. And last but not least, I would like to announce that Valentin Haenel (@esc) just stepped down as release manager. Thanks Valentin for all the hard work that you put in making bcolz a better piece of software! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) the Blaze project (http://blaze.pydata.org/), Quantopian (https://www.quantopian.com/) and Scikit-Allel (https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Blaze: * Notebooks showing Blaze + Pandas + BColz interaction: * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-csv.ipynb * http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bcolz.ip… * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/bcolz.html Installing ========== bcolz is in the PyPI repository, so installing it is easy:: $ pip install -U bcolz Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted

1 0