I just wanted to clarify, that I am very excited about a few ideas I have
--- but I don't have time myself to engage in the community process to get
these changes into NumPy. However, those are real processes --- I've
been coaching a few people in those processes for the past several years
So, rather than do nothing, what I'm looking to do is to work with a few
people who I can share my ideas with, get excited about the ideas, and then
who will work with the community to get them implemented. That's what I
was announcing and talking about yesterday --- looking for interested
people who want to work on NumPy *with* the NumPy community.
In my enthusiasm, I realize that some may have mis-understood my
intention. There is no 'imminent' fork, nor am I planning on doing some
crazy amount of work that I then try to force on other developers of NumPy.
What I'm planning to do is find people to train on NumPy code base (people
to increase the diversity of the developers would be ideal -- but hard to
accomplish). I plan to train them on NumPy based on my experience, and on
what I think should be done --- and then have *them* work through the
community process and engage with others to get consensus (hopefully not
losing too much in translation in the process --- but instead getting even
During that process I will engage as a member of the community and help
write NEPs and other documents and help clarify where it makes sense as I
can. I will be filtering for people that actually want to see NumPy get
better. Until I identify the people and work with them, it will be hard
to tell how this will best work. So, stay tuned.
If all goes well, what you should see in a few weeks time are specific
proposals, a branch or two, and the beginnings of some pull requests. If
you don't see that, then I will not have found the right people to help me,
and we will all continue to go back to searching.
While I'm expecting the best, in the worst case, we get additional people
who know the NumPy code base and can help squash bugs as well as implement
changes that are desired. Three things are needed if you want to
participate in this: 1) A willingness to work with the open source
community, 2) a deep knowledge of C and in-particular CPython's brand of C,
and 3) a willingness to engage with me, do a mind-meld and dump around the
NumPy code base, and then improve on what is in my head with the rest of
I'm just building numpy 1.9.2 for Python 3.5 (just released).
In order to get the tests to pass on Python 3.5, I need to cherry pick
commit 7d6aa8c onto the 1.9.2 tag position.
Does anyone object to me uploading a wheel built from this patched
version to pypi as 1.9.2 for Python 3.5 on OSX? It would help to get
the ball rolling for Python 3.5 binary wheels.
Announcing python-blosc 1.2.8
What is new?
This is a maintenance release. Internal C-Blosc has been upgraded to
1.7.0 (although new bitshuffle support has not been made public, as it
seems not ready for production yet).
Also, there is support for bytes-like objects that support the buffer
interface as input to ``compress`` and ``decompress``. On Python 2.x
this includes unicode, on Python 3.x it doesn't. Thanks to Valentin
Finally, a memory leak in ``decompress```has been hunted and fixed. And
new tests have been added to catch possible leaks in the future. Thanks
to Santi Villalba.
For more info, you can have a look at the release notes in:
More docs and examples are available in the documentation site:
What is it?
Blosc (http://www.blosc.org) is a high performance compressor
optimized for binary data. It has been designed to transmit data to
the processor cache faster than the traditional, non-compressed,
direct memory fetch approach via a memcpy() OS call.
Blosc is the first compressor that is meant not only to reduce the size
of large datasets on-disk or in-memory, but also to accelerate object
manipulations that are memory-bound
http://www.blosc.org/synthetic-benchmarks.html for some benchmarks on
how much speed it can achieve in some datasets.
Blosc works well for compressing numerical arrays that contains data
with relatively low entropy, like sparse data, time series, grids with
regular-spaced values, etc.
python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for
the Blosc compression library.
There is also a handy tool built on Blosc called Bloscpack
(https://github.com/Blosc/bloscpack). It features a commmand line
interface that allows you to compress large binary datafiles on-disk.
It also comes with a Python API that has built-in support for
serializing and deserializing Numpy arrays both on-disk and in-memory at
speeds that are competitive with regular Pickle/cPickle machinery.
python-blosc is in PyPI repository, so installing it is easy:
$ pip install -U blosc # yes, you must omit the 'python-' prefix
The sources are managed through github services at:
There is Sphinx-based documentation site at:
There is an official mailing list for Blosc at:
Both Blosc and its Python wrapper are distributed using the MIT license.
for more details.
I've just submitted a PR which overhauls the implementation of ufuncs
for object types.
The motivation for this is that many ufuncs (eg all transcendental
functions) can't handle objects. This PR will also make object arrays
more customizable, as all ufuncs can now be overridden by the user by
defining methods on the object.
I'd like to see if there's enough interest/agreement for me to finish
it, and also to have more eyes look over the pure-python ufunc
implementations since I am sure I have missed some edge cases. I would
love suggestions, and I need help for a few cases I couldn't figure out
(eg np.nextafter, np.spacing, np.cbrt. Also, np.expm1 np.signbit and
others are a bit suspect).
Announcing Numexpr 2.4.4
Numexpr is a fast numerical expression evaluator for NumPy. With it,
expressions that operate on arrays (like "3*a+4*b") are accelerated
and use less memory than doing the same calculation in Python.
It wears multi-threaded capabilities, as well as support for Intel's
MKL (Math Kernel Library), which allows an extremely fast evaluation
of transcendental functions (sin, cos, tan, exp, log...) while
squeezing the last drop of performance out of your multi-core
processors. Look here for a some benchmarks of numexpr using MKL:
Its only dependency is NumPy (MKL is optional), so it works well as an
easy-to-deploy, easy-to-use, computational engine for projects that
don't want to adopt other solutions requiring more heavy dependencies.
This is a maintenance release which contains several bug fixes, like
better testing on Python3 platform and some harmless data race. Among
the enhancements, AppVeyor support is here and OMP_NUM_THREADS is
honored as a fallback in case NUMEXPR_NUM_THREADS is not set.
In case you want to know more in detail what has changed in this
Where I can find Numexpr?
The project is hosted at GitHub in:
You can get the packages from PyPI as well (but not for RC releases):
Share your experience
Let us know of any bugs, suggestions, gripes, kudos, etc. you may
[This is getting fairly off-topic for python-ideas (since AFAICT there
is no particular reason right now to add a new deterministic generator
to the stdlib), so CC'ing numpy-discussion and I'd suggest followups
be directed to there alone.]
On Thu, Sep 10, 2015 at 10:41 AM, Robert Kern <robert.kern(a)gmail.com> wrote:
> On 2015-09-10 04:56, Nathaniel Smith wrote:
>> On Wed, Sep 9, 2015 at 8:35 PM, Tim Peters <tim.peters(a)gmail.com> wrote:
>>> There are some clean and easy approaches to this based on
>>> crypto-inspired schemes, but giving up crypto strength for speed. If
>>> you haven't read it, this paper is delightful:
>> It really is! As AES acceleration instructions become more common
>> (they're now standard IIUC on x86, x86-64, and even recent ARM?), even
>> just using AES in CTR mode becomes pretty compelling -- it's fast,
>> deterministic, provably equidistributed, *and* cryptographically
>> secure enough for many purposes.
> I'll also recommend the PCG paper (and algorithm) as the author's
> cross-PRNGs comparisons have been bandied about in this thread already. The
> paper lays out a lot of the relevant issues and balances the various
> qualities that are important: statistical quality, speed, and security (of
> various flavors).
> I'm actually not that impressed with Random123. The core idea is nice and
> clean, but the implementation is hideously complex.
I'm curious what you mean by this? In terms of the code, or...?
I haven't looked at the code, but the simplest generator they
recommend in the paper is literally just
counter = 0
# AES128 takes a 128 bit key and 128 bits of data and returns
128 bits of encrypted data
val = AES128(key=seed, data=counter)
counter += 1
which gives a 64-bit generator with a period of 2^129. They benchmark
it as faster than the Mersenne Twister on modern CPUs (<2
cycles-per-byte on recent x86, x86-64, ARMv8), it requires less
scratch space, is incredibly simple to work with -- you can
parallelize it, get independent random streams, etc., in a totally
trivial way -- and has a *way* stronger guarantee of
random-looking-ness than merely passing TestU01.
The downsides are that it does still require 176 bytes of read-only
scratch storage (used to cache an expanded version of the "key"), the
need for a modern CPU to get that speed, and that it doesn't play well
with GPUs. So they also provide a set of three more ad hoc generators
designed to solve these problems. I'm not as convinced about these,
but hey, they pass TestU01...
The PCG paper does a much better job of all the other stuff *around*
making a good RNG -- it has the nice website, clear comparisons, nice
code, etc. -- which is definitely important. But to me the increase in
speed and reduction in memory use doesn't seem worth it given how fast
these generators are to start with + the nice properties of counter
mode + and cryptographic guarantees of randomness that you get from
AES, for code that's generally targeting non-embedded non-GPU systems.
Nathaniel J. Smith -- http://vorpus.org
To the NumPy list only, I'll at least give the highlights of the surgical
approach I would like to get someone to work on -- I can help mentor and
guide. These are just the highlights, but it should give someone familiar
with the code the general gist. There are some details to work out, of
course, but it could be done.
It may be very similar to what Nathaniel is contemplating --- except I
think breaking the ABI is the only way to really do this --- could be wrong
but I'm not wiling to risk *not* just breaking the ABI.
1) Create a new meta-type in C (call it dtype)
2) Create Python Classes (in C) that are instances of this meta-type for
each "kind" of data-type
3) Make PyArray_Descr * be a reference to one of these new objects (which
can be built either in C or Python) and should be published outside NumPy
4) Remove most of the "per-type function calls" in PyArray_ArrFuncs ---
instead replacing those with the Generalized Ufunc equivalents and expand
the capability of Generalized Ufuncs
5) Keep the Array Scalar Types but change them so that they also use the
dtype meta-type as their foundation and mixin an array-methods type.
Also, have these be in a separate project from NumPy itself.
6) The current void* would be replaced with real Python classes instead of
structured arrays being shoved through a single data-type.
7) The documented ways to spell a dtype would be reduced --- but backwards
compatibility would be preserved.
8) Make sure Numba can create these Descriptor objects with Ahead of Time
Compilation and start to move code of NumPy to Numba
9) Ensure the Generalized Ufunc framework can take the data-type as an
argument so that *all* data-types can participate in the general
There is more to it, but that is the basic idea. Please forgive me if I
can't respond to any feedback from the list in a timely way. I will as I
*Co-founder and CEO*