Mailman 3 November 2013 - NumPy-Discussion

ANN: NumPy 1.7.2rc1 release
by Julian Taylor Nov. 15, 2013

Nov. 15, 2013

Hi all, I'm happy to announce the release candidate of Numpy 1.7.2. This is a bugfix only release supporting Python 2.4 - 2.7 and 3.1 - 3.3. More than 37 issues were fixed, the most important issues are listed in the release notes: https://github.com/numpy/numpy/blob/v1.7.2rc1/doc/release/1.7.2-notes.rst It is supposed to not break any existing code, so please test the releases and report any issues you find. Source tarballs and release notes can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.2rc1/. Currently only Windows installers are available. OS X installer will follow soon. Concerning OS X currently only a single person can create the binary installers which is not a good situation. If you have a suitable machine [0] and want to help out please contact us. Cheers, Julian Taylor [0] we currently base the releases on macos 10.6 using python.org python versions

8 14

segv PyArray_Check
by Burlen Loring Nov. 14, 2013

Nov. 14, 2013

Hi, I'd like to add numpy support to an existing code that uses swig. I've changed the source file that has code to convert python lists into native data from c to c++ so I can use templates to handle data conversions. The problem I'm having is a segfault on PyArray_Check called from my c++ source. Looking at things in the debugger the argument passed to it is indeed an intialized python list, my swig file has import_array() in it's init, and I've verified that it is getting called. adding a function in my c++ source to call import_array() a second time prevents the segfault. Could anyone explain why I need the import_array() in both places? If that's the correct way to handle it? Thanks Burlen

3 4

savetxt fmt argument fails when using a unicode string
by Pierre Haessig Nov. 14, 2013

Nov. 14, 2013

Hi, I just noticed (with numpy 1.7.1) that the following code import numpy as np np.savetxt('a.csv', [1], fmt=u'%.3f') fails with: 1045 else: 1046 for row in X: -> 1047 fh.write(asbytes(format % tuple(row) + newline)) 1048 if len(footer) > 0: 1049 footer = footer.replace('\n', '\n' + comments) UnboundLocalError: local variable 'format' referenced before assignment (of course, the simple solution is to remove the "u" prefix, but in my real code, I have a "from __future__ import unicode_literals") Since parts of the savetxt function were changed since that, I don't know if this bug is still applicable or not. I'm guessing that since Warren Weckesser commit https://github.com/numpy/numpy/commit/0d284764a855cae11699228ff1b81e6d18f38… the problem would be caught earlier with a much nicer ValueError. However, I still see isinstance(fmt, str): (on line 1035 https://github.com/numpy/numpy/blob/master/numpy/lib/npyio.py#L1035 ) How can I make the fmt argument work with unicode_literals ? best, Pierre

2 3

mingw-w64 and openblas test
by Carl Kleffner Nov. 13, 2013

Nov. 13, 2013

Hi list, I created a repository at google code https://code.google.com/p/mingw-w64-static with some downloads as well as my last numpy setp.py log. Regards Carl

2 9

[ANN, x-post] Creating a space for scientific open source at Berkeley (with UW and NYU)
by Fernando Perez Nov. 13, 2013

Nov. 13, 2013

Hi folks, forgive me for the x-post to a few lists and the semi off-topic nature of this post, but I think it's worth mentioning this to our broader community. To keep the SNR of each list high, I'd prefer any replies to happen on the numfocus list. Yesterday, during an event at the White House OSTP, an announcement was made about a 5-year, $37.8M initative funded by the Moore and Sloan foundations to create a collaboration between UC Berkeley, the University of Washington and NYU on Data Science environments: - Press release: http://www.moore.org/newsroom/press-releases/2013/11/12/%20bold_new_partner… - Project description: http://www.moore.org/programs/science/data-driven-discovery/data-science-en… We worked in private on this for a year, so it's great to be able to finally engage the community in an open fashion. I've provided some additional detail in my blog: http://blog.fperez.org/2013/11/an-ambitious-experiment-in-data-science.html At Berkeley, we are using this as an opportunity to create the new Berkeley Institute for Data Science (BIDS): http://newscenter.berkeley.edu/2013/11/13/new-data-science-institute-to-hel… and from the very start, open source and the scientific Python ecosystem have been at the center of our thinking. In the team of co-PIs we have, in addition to me, a bunch of Python supporters: - Josh Bloom leads our Python bootcamps and graduate seminar) - Cathryn Carson founded the DLab (dlab.berkeley.edu), which runs python.berkeley.edu. - Philip Stark: Stats Chair, teaches reproducible research with Python tools. - Kimmen Sjolander: comp. biologist whose tools are all open source Python. - Mike Franklin and Ion Stoica: co-directors of AMPLab, whose Spark framework has Python support. - Dave Culler: chair of CS, which now uses Python for its undergraduate intro courses. We will be working very hard to basically make BIDS "a place for people like us" (and by that I mean open source scientific computing, not just Python: Juila, R, etc. are equally welcome). This is a community that has a significant portion of academic scientists who struggle with all the issues I list in my post, and solving that problem is an explicit goal of this initiative (in fact, it was the key point identified by the foundations when they announced the competition for this grant). Beyond that, we want to create a space where the best of academia, the power of a university like Berkeley, and the best of our open source communities, can come together. We are just barely getting off the ground, deep in more mundane issues like building renovations, but over the next few months we'll be clarifying our scientific programs, starting to have open positions, etc. Very importantly, I want to thank everyone who, for the last decade+, has been working like mad to make all of this possible. It's absolutely clear to me that the often unrewarded work of many of you was essential in this process, shaping the very existence of "data science" and the recognition that it should be done in an open, collaborative, reproducible fashion. Consider this event an important victory along the way, and hopefully a starting point for much more work in slightly better conditions. Here are some additional resources for anyone interested: http://bitly.com/bundles/fperezorg/1 -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail

1 0

Precision/value change moving from C to Python
by Bart Baker Nov. 13, 2013

Nov. 13, 2013

Hi all, First time posting. I've been working on a class for solving a group of affine models and decided to drop down to C for a portion of the code for improved performance. There are two questions that I've had come up from this. 1) I've written the section of code, which involved a series of fully populated masked arrays as arguments. There are a few array operations such as dot products and differences that are taken on the arrays to generate the final value. I've written versions in both Python/NumPy and C and now have a verification issue. The issue is that there are some minor (10^-16) differences in the values when I do the calculation in C vs Python. The way that I test these differences is by passing the NumPy arrays to the C extension and testing for equality in gdb. I'm not relying on any of the NumPy API for the array operations such as dot product, addition, and subtraction. All of the arrays I'm treating as two-dimensional in C right now. Once I figure out this bug, I'm planning on rewriting the computations in terms of one dimensional arrays. All of the values that I'm dealing with in C right now are double precision. 2) Beyond this issue, I also noticed that the values that are equal in gdb are not eqaul once I return to Python! I pass the same arrays back from C to Python, test eqaulity by: npy_array == c_array and find that almost the entire arrays are non-equal, even though equality tests in gdb had been true. Is there some obvious reason why this would be. Is there some sort of implicit value conversion performed when moving from C back to Python? I can post some code if need be, but I wanted to get anyone's first insight if there is anything obvious that I'm not doing right. I appreciate your help. Thanks, Bart

2 3

Numpy 1.9 release date
by Charles R Harris Nov. 12, 2013

Nov. 12, 2013

Hi All, The question has come up as to how much effort we should spend backporting fixes to 1.8.x. An alternative would be to tag 1.9.0 early next year, aiming for a release around April. I think there is almost enough in 1.9-devel to justify a release. There is Sebastian's index work, Julian's continuing work on speedups, the removal of oldnumeric and numarray support, and various other deprecations and cleanups that add up to a significant number of changes. I've tended to think of 1.9 as a cleanup and consolidation release and think that the main thing missing at this point is fixing the datetime problems. Thoughts? Chuck

5 4

numpy.lib functions as gufuncs
by Jaime Fernández del Río Nov. 12, 2013

Nov. 12, 2013

Hi, Inspired by the great rewrite of numpy.linalg in 1.8, I've spent the last couple of days coding a couple of the functions in numpy.lib as gufuncs, namely np.interp and np.bincount. I want to do something along the same lines to np.digitize, but haven't started on it just yet. I'm currently keeping the code in a separate repository ( https://github.com/jaimefrio/new_gufuncs), because I have a lot of problems compiling numpy, and this makes my life much easier while developing. If there's interest in adding these code to numpy, I´ll be more than happy to figure my compilation issues and turn this into a PR. If you run python setup.py install, you should then be able to `import new_gufuncs as ng`. Basically, there are 4 gufuncs (ng._interp, ng._minmax, ng._bincount, ng._bincount_wt) and 2 Python wrappers to them (ng.interp, ng.bincount) that reproduce the interface of their numpy counterparts. Some obvious advantages: 1. Added support for more types, including complex, in both the interp data points and the bincount weights. 2. You can use ng.bincount to do things like clustering, or ng.interp to interpolate vector functions of a single variable. 3. It makes the code much more flexible, I'm sure people will come up with new uses. Some things I'm concerned about: 1. For very small datasets, performance is several times worse than the current functions. For larger datasets the gufunc versions perform better than the current functions, but the gufunc dispatch mechanism seems to take for ever to get going. 2. There's what feels like a very nasty hack in the bincount gufuncs: since the shape of the output depends on the contents of the input arrays, the signature cannot be determined beforehand. So the gufunc has signature '(m)->(n)', which means that the wrapper function has to determine what the shape of the output is, instantiate it, and pass it in with the 'out' keyword argument to the gufunc for things to work properly. IT works beautifully, but it just doesn't seem right. 3. The broadcasting syntax doesn't seem to be the most convenient for the obvious applications: 3.1. Say you have a function R->R^3. The obvious way of describing it as piecewise linear would be to have xp of shape (n,) and yp of shape (n, 3). To run ng.interp on a set of points x of shape (m,), and get back an output of shape (m, 3) you would have to do ng.interp(x, xp, yp.T).T. 3.2. Say you have counts of terms in documents. The typical way of storing this would be in an array `dataset` of shape (docs, terms). If you then run clustering on that data, you would typically get a `clusters` array of shape (docs,). To cluster the counts you would do something like ng.bincount(clusters, dataset.T).T Those transpositions in these two cases seem a little annoying, but then again "special cases aren't special enough to break the rules," or is it "although practicality beats purity" that applies here? Any feedback on any of the above points (or others) is more than welcome. If you think this is a worthy addition to numpy, I'll work on some tests and performance benchmarks. Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 0

Can I use numpy-MKL for commercial purposes without an Intel MKL license?
by Brunick, Gerard:(Constellation) Nov. 11, 2013

Nov. 11, 2013

I am looking for a 64-bit, Windows version of Numpy and I have found a couple pre-built versions that are linked against the Intel MKL library. I would like to use this library for in-house commercial use. In particular, I do not intend to redistribute my code or link against the Intel MKL libraries. Does this require an MKL license? One of the FAQ questions from http://software.intel.com/en-us/articles/intel-math-kernel-library-licensin… reads: "Can I redistribute the Intel Math Kernel Library with my application? Yes. When you purchase Intel MKL, you receive rights to redistribute computational portions of Intel MKL with your application. The evaluation versions of Intel MKL do not include redistribution rights. The list of files that can be redistributed is provided in redist.txt included in the Intel MKL distribution with product license." This suggests that I can use a binary that has been compiled by a license holder without a license, but numpy is itself a library, in some sense, so it is not entirely clear to me. I suppose that I would also need to confirm that the individual who built the version of numpy that I am using holds a MKL product license? Thanks in advance for any (non-legally binding) insight that you can share, Gerard This e-mail and any attachments are confidential, may contain legal, professional or other privileged information, and are intended solely for the addressee. If you are not the intended recipient, do not use the information in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP

2 1

Error in docstring np.random.pareto
by Conny Kühne Nov. 11, 2013

Nov. 11, 2013

I think the docstring of np.random.pareto (Version 1.8.0) is erroneous. In a nutshell np.random.pareto draws samples from a Lomax distribution with shape a and location m'=1. To convert those samples to a classical Pareto distribution with shape a and location m you have to add 1 and multiply by m, instead of adding m as stated in the docstring. More specifically, the docstring says: "The Lomax or Pareto II distribution is a shifted Pareto distribution. The classical Pareto distribution can be obtained from the Lomax distribution by adding the location parameter m, see below." Instead, it should read "[..] by adding 1 and multiplying my m, see below." The example at the bottom therefore should read: >>> a, m = 3., 1. # shape and mode >>> s = (np.random.pareto(a, 1000) + 1) * m Maybe an example with m=10 makes it clearer >>> a, m = 3., 10. # shape and mode >>> s = (np.random.pareto(a, 1000) + 1) * m Additionally, calling m the location parameter could be misleading. For simple Pareto (Type I) distributions it is usually referred to as the x_min oder mode parameter of the distribution. When discussing generalized Pareto distributions m is usually called the scale parameter (a constant factor) while mu is the location (an additive term) [1]. I assume the misleading naming could have caused some confusion and lead to the errors described above. Last but not least I think it might also cause confusion to call a function 'pareto' while its meaning is 'shifted by -1 pareto'. :) [1] http://en.wikipedia.org/wiki/Generalized_Pareto_distribution -- Conny Kuehne

1 0