Mailman 3 May 2012 - NumPy-Discussion

Extension types repository
by Charles R Harris May 7, 2012

May 7, 2012

All, Tom Aldcroft volunteered to bring quaternions into numpy. The proposal is to set up a separate repository under the numpy name on github, npydtypes or some such, and bring in Martin Ling's quaternion extension dtype as a start. Other extension types that would reside in the repository would be the simple rational type, and perhaps some specialized astronomical time types. So here is the proposal. 1. Make Tom a member of the numpy organization on github. 2. Set up an extension dtypes repository in github.com/numpy Other proposals for the name are welcome. Chuck

4 7

Quaternion data type
by Ilan Schnell May 7, 2012

May 7, 2012

Hello everyone, what is the plan for Quaternion data types in numpy? I saw that during last years SciPy spring https://github.com/martinling/numpy_quaternion was started, but not updated or released since then. Thanks Ilan

8 15

numpy_quaternion and gcc 4.1.2
by Tom Aldcroft May 6, 2012

May 6, 2012

I ran into a problem trying to build and import the numpy_quaternion extension on CentOS-5 x86_64: $ python setup.py build <SNIP> C compiler: gcc -pthread -fno-strict-aliasing -fPIC -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-I/data/cosmos2/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/numpy/core/include -I/data/cosmos2/ska/arch/x86_64-linux_CentOS-5/include/python2.7 -c' gcc: quaternion.c quaternion.c: In function \u2018quaternion_isfinite\u2019: quaternion.c:55: warning: implicit declaration of function \u2018isfinite\u2019 gcc: numpy_quaternion.c gcc -pthread -shared build/temp.linux-x86_64-2.7/quaternion.o build/temp.linux-x86_64-2.7/numpy_quaternion.o -o build/lib.linux-x86_64-2.7/quaternion/numpy_quaternion.so running scons There was a subsequent import error with "numpy_quaternion.so: undefined symbol: isfinite". This problem does not occur for Ubuntu 11.10 and I presume it is due to CentOS-5 gcc (4.1.2) defaulting to -c89. I fixed this in setup.py by adding "extra_compile_args['-std=c99']" to the add_extension() call. Is there a more general way in numpy to deal with issues like this? Thanks, Tom

2 1

ANN Scikit-learn 0.11-beta
by Gael Varoquaux May 6, 2012

May 6, 2012

On behalf of our release manager, Andreas Mueller, and all the scikit-learn contributors, I am happy to announce the 0.11 beta. We are doing a quick beta and will hopefuly be releasing the final version tomorrow. The purpose of this beta is to get feedback on any release-critical bugs such as build issues. You can download the zip files of the beta on: https://github.com/scikit-learn/scikit-learn/zipball/0.11-beta You can also retrieve the latest code on https://github.com/scikit-learn/scikit-learn/zipball/master or using 'git clone git@github.com:scikit-learn/scikit-learn.git' Any feedback is more than welcome, Cheers, Gaël

1 0

sparse array data
by Wolfgang Kerzendorf May 5, 2012

May 5, 2012

Hi all, I'm currently writing a code that needs three dimensional data (for the physicists it's dimensions are atom, ion, level). The problem is that not all combinations do exist (a sparse array). Sparse matrices in scipy only deal with two dimensions. The operations that I need to do on those are running functions like exp(item/constant) on all of the items. I also want to sum them up in the last dimension. What's the best way to make a class that takes this kind of data and does the required operations fast. Maybe some phycisists have implemented these things already. Any thoughts? Cheers Wolfgang

8 18

Is NumpyDotNet (aka numpy-refactor) likely to be merged into the mainline?
by Seth Nickell May 5, 2012

May 5, 2012

With a little work, I think numpy/scipy could be very useful to those of us who have to program on .NET for one reason or another, but 64-bit is currently not supported (at least, not as released). I'm considering working out 64-bit support, but it appears to me like the numpy-refactor repository isn't on a path to merging with the mainline, and is likely to bit-rot (if it hasn't already). Is anyone working on this, or is NumpyDotNet 'resting' at the moment, so to speak? Its sort of pointless to work on a dead-end branch ;-) thanks! -seth

3 2

Fwd: [SIAM-CSE] pOSKI - Autotuner for parallel sparse-matrix-vector multiplication
by Fernando Perez May 5, 2012

May 5, 2012

Interesting, given the recent discussion on sparsity... ---------- Forwarded message ---------- From: James Demmel <demmel(a)eecs.berkeley.edu> Date: Fri, May 4, 2012 at 12:03 PM Subject: [SIAM-CSE] pOSKI - Autotuner for parallel sparse-matrix-vector multiplication To: siam-cse(a)siam.org We are pleased to announce the first public release of pOSKI, the Parallel Optimized Sparse Kernel Interface. pOSKI automatically produces high performance implementations of Sparse-Matrix-Vector multiplication (SpMV) and related operations. pOSKI targets current multicore CPU platforms, and is an extension of prior work on OSKI, which targeted single core processors. Both pOSKI and OSKI use autotuning, which means searching a design space of sparse matrix data structures and algorithms that depend on the matrix properties (eg sparsity pattern and symmetry), the operation to be performed (A*x, transpose(A)*x, etc.), and the target platform. For example, on an Intel Jaketown, with 6 cores and 12 threads, pOSKI gets a speedup of 4.8x over 1 core, and is 1.6x faster than Intel MKL's dcsrmv, both on the matrix wikipedia-2007 from the University of Florida collection. pOSKI is freely available under a BSD license at bebop.cs.berkeley.edu/poski which also includes all the documentation, and a mailing list for asking questions. pOSKI is based on contributions from many people, and was produced by BEBOP, the Berkeley Benchmarking and OPtimization Group (bebop.cs.berkeley.edu). Regards, Jim Demmel and Jong-Ho Byun UC Berkeley _______________________________________________ SIAM-CSE mailing list To post messages to the list please send them to: SIAM-CSE(a)siam.org http://lists.siam.org/mailman/listinfo/siam-cse

1 0

Fwd: [SIAM-CSE] pOSKI - Autotuner for parallel sparse-matrix-vector
by Fernando Perez May 5, 2012

May 5, 2012

Interesting, given the recent discussion on sparsity... ---------- Forwarded message ---------- From: James Demmel <demmel(a)eecs.berkeley.edu> Date: Fri, May 4, 2012 at 12:03 PM Subject: [SIAM-CSE] pOSKI - Autotuner for parallel sparse-matrix-vector multiplication To: siam-cse(a)siam.org We are pleased to announce the first public release of pOSKI, the Parallel Optimized Sparse Kernel Interface. pOSKI automatically produces high performance implementations of Sparse-Matrix-Vector multiplication (SpMV) and related operations. pOSKI targets current multicore CPU platforms, and is an extension of prior work on OSKI, which targeted single core processors. Both pOSKI and OSKI use autotuning, which means searching a design space of sparse matrix data structures and algorithms that depend on the matrix properties (eg sparsity pattern and symmetry), the operation to be performed (A*x, transpose(A)*x, etc.), and the target platform. For example, on an Intel Jaketown, with 6 cores and 12 threads, pOSKI gets a speedup of 4.8x over 1 core, and is 1.6x faster than Intel MKL's dcsrmv, both on the matrix wikipedia-2007 from the University of Florida collection. pOSKI is freely available under a BSD license at bebop.cs.berkeley.edu/poski which also includes all the documentation, and a mailing list for asking questions. pOSKI is based on contributions from many people, and was produced by BEBOP, the Berkeley Benchmarking and OPtimization Group (bebop.cs.berkeley.edu). Regards, Jim Demmel and Jong-Ho Byun UC Berkeley _______________________________________________ SIAM-CSE mailing list To post messages to the list please send them to: SIAM-CSE(a)siam.org http://lists.siam.org/mailman/listinfo/siam-cse

1 0

numpy doc for percentile function
by Pierre Haessig May 5, 2012

May 5, 2012

Hi, I'm looking for the entry point in Numpy doc for the percentile function. I'm assuming it should sit in routines.statistics but do not see it : http://docs.scipy.org/doc/numpy/reference/routines.statistics.html Am I missing something ? If indeed the percentile entry should be added, do you agree it could be added to the "Histogram" section ? (and "Histogram" would become "Histograms and percentiles") Also, as Frédéric Bastien pointed out, I feel that the current doc build is broken (especially the links :-( ) Best, Pierre

3 2

What is consensus anyway
by Nathaniel Smith May 4, 2012

May 4, 2012

If you hang around big FOSS projects, you'll see the word "consensus" come up a lot. For example, the glibc steering committee recently dissolved itself in favor of governance "directly by the consensus of the people active in glibc development"[1]. It's the governing rule of the IETF, which defines many of the most important internet standards[2]. It is the "primary way decisions are made on Wikipedia"[3]. It's "one of the fundamental aspects of accomplishing things within the Apache framework"[4]. [1] https://lwn.net/Articles/488778/ [2] https://www.ietf.org/tao.html#getting.things.done [3] https://en.wikipedia.org/wiki/Wikipedia:Consensus [4] https://www.apache.org/foundation/voting.html But it turns out that this "consensus" thing is actually somewhat mysterious, and one that most programmers immersed in this culture pick it up by osmosis. And numpy in particular has a lot of developers who are not coming from a classic FOSS programmer background! So this is my personal attempt to articulate what it is, and why requiring consensus is probably the best possible approach to project decision making. So what is "consensus"? Like, voting or something? ----------------------------------------------------- This is surprisingly subtle and specific. "Consensus" means something like, "everyone who cares is satisfied with the result". It does *not* mean * Every opinion counts equally * We vote on anything * Every solution must be perfect and flawless * Every solution must leave everyone overjoyed * Everyone must sign off on every solution. It *does* mean * We invite people to speak up * We generally trust individuals to decide how important their opinion is * We generally trust individuals to decide whether or not they can live with some outcome * If they can't, then we take the time to find something better. One simple way of stating this is, everyone has a veto. In practice, such vetoes are almost never used, so this rule is not particularly illuminating on its own. Hence, the rest of this document. What a waste of time! That all sounds very pretty on paper, but we have stuff to get done. ----------------------------------------------------------------------------------- First, I'll note that this seemingly utopian scheme has a track record of producing such impractical systems as TCP/IP, SMTP, DNS, Apache, GCC, Linux, Samba, Python, ... But mere empirical results are often less convincing than a good story, so I will give you two. Why does a requirement for consensus work? Reason 1 (for optimists): *All of us are smarter than any of us.* For a complex project with many users, it's extraordinarily difficult for any one person to understand the full ramifications of any decision, particularly the sort of far-reaching architectural decisions that are most important. It's even more difficult to understand all the possibilities of all the different possible solutions. In fact, it's *extremely* common that the correct solution to a problem is the one that no-one thinks of until after a month of annoying debate. Spending a month to avoid an architectural problem that will haunt us for years is an *excellent* trade-off, even if it feels interminable at the time. Even two months. Usually disagreements are an indication that a better solution is possible, even when it's not clear what that would be. Reason 2 (for pessimists): *You **will** reach consensus sooner or later; it's less painful to do up front.* Example: NA handling. There are two schemes that people use for this right now -- numpy.ma and ugly NaN kluges (see e.g. nanmean). These are generally agreed to be suboptimal. Recently, two new contenders have shown up: the NEP masked-NA support currently in master, and the unrelated NA support in pandas (which as a library is attracting a *lot* of the statistical analysis folk who care about missing data, kudos to Wes). I think that right now, the most likely future is that a few years from now, many people will still be using the old solutions, and others will have switched to the new (incompatible) solutions, and we will have *4* suboptimal schemes in concurrent use. If (when) this happens, we will have to re-open this discussion yet again, but now with a heck of a mess to clean up. This is FOSS -- if people aren't convinced by your solution, they will just ignore it and do their own thing. So a policy that allows changes to be made without consensus is a recipe for entrenching disagreements and splitting the community. Okay, great, but even if it's the best thing ever, we *can't* hold a vote on every change! What are you actually suggesting we do? ---------------------------------------------------------------------------- Right, that's not the idea. Most changes are pretty obviously uncontroversial, and in fact we usually have the opposite problem -- it's hard to get people to do code review! So having consensus on every change is an ideal, and in practice, just following the reasonable person principle lets us get pretty close to that ideal. If no-one objects to a change, then it's probably fine. (And it's not like anyone *wants* that segfault to remain unfixed! That's obvious.) OTOH, this isn't an excuse to try gaming the system -- if you have a change that might affect people adversely, then it can be worthwhile to send them a ping, even if they didn't object. They might have just missed seeing it go by, and if it's going to be a problem, better to find out now! In fact, one of the nice things about having a consistent culture of consensus-building is that people learn to trust that if they do have a problem, it will be taken seriously. And that, in turn, makes it okay to make judgement calls about whether the participants in some discussion basically agree, or whether the apparent disagreement is just bikeshedding[1], or whatever. If you make the wrong judgement call, then someone will tell you, and no harm is done. If you do not have such a culture, then people may (will) despair of being taken seriously, and go do something more pleasant and productive, like fire-walking or coding in Matlab. [1] http://producingoss.com/en/common-pitfalls.html#bikeshed So mainly what I'm saying we should do is: 1. Make it as easy as possible for people to see what's going on and join the discussion. All decisions and reasoning behind decisions take place in public. (On this note, it would be *really* good if pull request notifications went to the list.) 2. If someone raises a substantive objection, take that seriously. 3. If someone says "no, this is just not going to work for me, because... <something substantive here>", then it can't go in. It turns out that since we are all human, it's much easier to take people's concerns seriously when you know that they can veto your code! The result is that in practice, disagreements get resolved at point (2) there, and no-one feels the need to take such extreme measures as vetoing anything. I'm as lazy as anyone else; I produce better solutions when I'm forced to keep looking for them. So if we just follow this consensus stuff, everything will be perfect? --------------------------------------------------- Ha ha ha no. It can and does work better than any other options I'm aware of, but it takes practice and there are certainly still failure modes. Here's some that come to mind: * Bikeshedding: Mentioned above. There's nothing *wrong* with everyone speaking up to voice their opinion, but it shouldn't be an obstacle to getting work done when in fact everyone will be satisfied regardless. * False compromise: if all you're trying to do is make everybody happy, then it's easy to end up with a "compromise" that takes one bit from each proposal (or simply takes the union of them all). This is usually worse than any of the original proposals. Good designs take work; skipping that work is tempting; resist it. You also can end up with some surprising solutions, e.g.: * The group reaches consensus that the problem is well understood and there simply is no perfect solution, but something is better than nothing. So everyone agrees to, in effect, flip a coin. (Example: the Python ternary operator) * The group reaches consensus that while other points of view may be valid, this project only has room for one of them, sorry, anyone who disagrees will have to start their own project. (Example: GPL versus BSD debates) * The group reaches consensus that at least one piece of a larger proposal is okay to start, which puts off the show-down over the rest of it until another day. (At which point more data more be available, opinions may have changed, or the whole issue may have become irrelevant.) Remember, the goal is always to find some way forward that we can collectively live with. Sometimes you can successfully convince everyone of the intrinsic awesomeness of your original idea through argument along... but clever outside-the-box proposals often pay-off too. But what about obstructive people abusing their veto power? ---------------------------------------------------------------- This concern makes perfect sense, but it turns out to just not be as much of a problem as often as you'd think. Most people have more interesting things to do with their lives than to gum up the mailing list for some random software library. It's a fair assumption that if someone cares enough to speak up, it's because they have some legitimate interest in numpy's future. And again, energy spent on trying to sniff out obstructionists can usually be more profitably spent on finding better solutions. That said, yes, sometimes people may be obstructive or act in bad faith. Here's some good experience-based advice on this: http://producingoss.com/en/difficult-people.html Notice that everything they say is still oriented around consensus -- "You may not persuade the person in question, but that's okay as long as you persuade everyone else", "a perfect example of how to build a strong case on neutral, quantitative data", the "masterful strategy" is to build consensus before acting. The consensus ideal is perfectly compatible with dealing with difficult people. Other documents --------------------- Like I said at the start, this is just my attempt to distill some abstract principles from my own experience. I can't take credit for most of these insights, and no doubt I've articulated some of them poorly. Fortunately, we don't all have to agree on every detail to get things done :-). But if you want to read more about this topic, and from other perspectives, here are some decent documents: http://producingoss.com/en/consensus-democracy.html https://www.ietf.org/tao.html#getting.things.done https://en.wikipedia.org/wiki/Consensus_decision-making And, of course, I would love to hear feedback on this document! - N

21 87