Mailman 3 April 2012 - NumPy-Discussion

numpy doc for percentile function
by Pierre Haessig 05 May '12

05 May '12

Hi, I'm looking for the entry point in Numpy doc for the percentile function. I'm assuming it should sit in routines.statistics but do not see it : http://docs.scipy.org/doc/numpy/reference/routines.statistics.html Am I missing something ? If indeed the percentile entry should be added, do you agree it could be added to the "Histogram" section ? (and "Histogram" would become "Histograms and percentiles") Also, as Frédéric Bastien pointed out, I feel that the current doc build is broken (especially the links :-( ) Best, Pierre

3 2

What is consensus anyway
by Nathaniel Smith 04 May '12

04 May '12

If you hang around big FOSS projects, you'll see the word "consensus" come up a lot. For example, the glibc steering committee recently dissolved itself in favor of governance "directly by the consensus of the people active in glibc development"[1]. It's the governing rule of the IETF, which defines many of the most important internet standards[2]. It is the "primary way decisions are made on Wikipedia"[3]. It's "one of the fundamental aspects of accomplishing things within the Apache framework"[4]. [1] https://lwn.net/Articles/488778/ [2] https://www.ietf.org/tao.html#getting.things.done [3] https://en.wikipedia.org/wiki/Wikipedia:Consensus [4] https://www.apache.org/foundation/voting.html But it turns out that this "consensus" thing is actually somewhat mysterious, and one that most programmers immersed in this culture pick it up by osmosis. And numpy in particular has a lot of developers who are not coming from a classic FOSS programmer background! So this is my personal attempt to articulate what it is, and why requiring consensus is probably the best possible approach to project decision making. So what is "consensus"? Like, voting or something? ----------------------------------------------------- This is surprisingly subtle and specific. "Consensus" means something like, "everyone who cares is satisfied with the result". It does *not* mean * Every opinion counts equally * We vote on anything * Every solution must be perfect and flawless * Every solution must leave everyone overjoyed * Everyone must sign off on every solution. It *does* mean * We invite people to speak up * We generally trust individuals to decide how important their opinion is * We generally trust individuals to decide whether or not they can live with some outcome * If they can't, then we take the time to find something better. One simple way of stating this is, everyone has a veto. In practice, such vetoes are almost never used, so this rule is not particularly illuminating on its own. Hence, the rest of this document. What a waste of time! That all sounds very pretty on paper, but we have stuff to get done. ----------------------------------------------------------------------------------- First, I'll note that this seemingly utopian scheme has a track record of producing such impractical systems as TCP/IP, SMTP, DNS, Apache, GCC, Linux, Samba, Python, ... But mere empirical results are often less convincing than a good story, so I will give you two. Why does a requirement for consensus work? Reason 1 (for optimists): *All of us are smarter than any of us.* For a complex project with many users, it's extraordinarily difficult for any one person to understand the full ramifications of any decision, particularly the sort of far-reaching architectural decisions that are most important. It's even more difficult to understand all the possibilities of all the different possible solutions. In fact, it's *extremely* common that the correct solution to a problem is the one that no-one thinks of until after a month of annoying debate. Spending a month to avoid an architectural problem that will haunt us for years is an *excellent* trade-off, even if it feels interminable at the time. Even two months. Usually disagreements are an indication that a better solution is possible, even when it's not clear what that would be. Reason 2 (for pessimists): *You **will** reach consensus sooner or later; it's less painful to do up front.* Example: NA handling. There are two schemes that people use for this right now -- numpy.ma and ugly NaN kluges (see e.g. nanmean). These are generally agreed to be suboptimal. Recently, two new contenders have shown up: the NEP masked-NA support currently in master, and the unrelated NA support in pandas (which as a library is attracting a *lot* of the statistical analysis folk who care about missing data, kudos to Wes). I think that right now, the most likely future is that a few years from now, many people will still be using the old solutions, and others will have switched to the new (incompatible) solutions, and we will have *4* suboptimal schemes in concurrent use. If (when) this happens, we will have to re-open this discussion yet again, but now with a heck of a mess to clean up. This is FOSS -- if people aren't convinced by your solution, they will just ignore it and do their own thing. So a policy that allows changes to be made without consensus is a recipe for entrenching disagreements and splitting the community. Okay, great, but even if it's the best thing ever, we *can't* hold a vote on every change! What are you actually suggesting we do? ---------------------------------------------------------------------------- Right, that's not the idea. Most changes are pretty obviously uncontroversial, and in fact we usually have the opposite problem -- it's hard to get people to do code review! So having consensus on every change is an ideal, and in practice, just following the reasonable person principle lets us get pretty close to that ideal. If no-one objects to a change, then it's probably fine. (And it's not like anyone *wants* that segfault to remain unfixed! That's obvious.) OTOH, this isn't an excuse to try gaming the system -- if you have a change that might affect people adversely, then it can be worthwhile to send them a ping, even if they didn't object. They might have just missed seeing it go by, and if it's going to be a problem, better to find out now! In fact, one of the nice things about having a consistent culture of consensus-building is that people learn to trust that if they do have a problem, it will be taken seriously. And that, in turn, makes it okay to make judgement calls about whether the participants in some discussion basically agree, or whether the apparent disagreement is just bikeshedding[1], or whatever. If you make the wrong judgement call, then someone will tell you, and no harm is done. If you do not have such a culture, then people may (will) despair of being taken seriously, and go do something more pleasant and productive, like fire-walking or coding in Matlab. [1] http://producingoss.com/en/common-pitfalls.html#bikeshed So mainly what I'm saying we should do is: 1. Make it as easy as possible for people to see what's going on and join the discussion. All decisions and reasoning behind decisions take place in public. (On this note, it would be *really* good if pull request notifications went to the list.) 2. If someone raises a substantive objection, take that seriously. 3. If someone says "no, this is just not going to work for me, because... <something substantive here>", then it can't go in. It turns out that since we are all human, it's much easier to take people's concerns seriously when you know that they can veto your code! The result is that in practice, disagreements get resolved at point (2) there, and no-one feels the need to take such extreme measures as vetoing anything. I'm as lazy as anyone else; I produce better solutions when I'm forced to keep looking for them. So if we just follow this consensus stuff, everything will be perfect? --------------------------------------------------- Ha ha ha no. It can and does work better than any other options I'm aware of, but it takes practice and there are certainly still failure modes. Here's some that come to mind: * Bikeshedding: Mentioned above. There's nothing *wrong* with everyone speaking up to voice their opinion, but it shouldn't be an obstacle to getting work done when in fact everyone will be satisfied regardless. * False compromise: if all you're trying to do is make everybody happy, then it's easy to end up with a "compromise" that takes one bit from each proposal (or simply takes the union of them all). This is usually worse than any of the original proposals. Good designs take work; skipping that work is tempting; resist it. You also can end up with some surprising solutions, e.g.: * The group reaches consensus that the problem is well understood and there simply is no perfect solution, but something is better than nothing. So everyone agrees to, in effect, flip a coin. (Example: the Python ternary operator) * The group reaches consensus that while other points of view may be valid, this project only has room for one of them, sorry, anyone who disagrees will have to start their own project. (Example: GPL versus BSD debates) * The group reaches consensus that at least one piece of a larger proposal is okay to start, which puts off the show-down over the rest of it until another day. (At which point more data more be available, opinions may have changed, or the whole issue may have become irrelevant.) Remember, the goal is always to find some way forward that we can collectively live with. Sometimes you can successfully convince everyone of the intrinsic awesomeness of your original idea through argument along... but clever outside-the-box proposals often pay-off too. But what about obstructive people abusing their veto power? ---------------------------------------------------------------- This concern makes perfect sense, but it turns out to just not be as much of a problem as often as you'd think. Most people have more interesting things to do with their lives than to gum up the mailing list for some random software library. It's a fair assumption that if someone cares enough to speak up, it's because they have some legitimate interest in numpy's future. And again, energy spent on trying to sniff out obstructionists can usually be more profitably spent on finding better solutions. That said, yes, sometimes people may be obstructive or act in bad faith. Here's some good experience-based advice on this: http://producingoss.com/en/difficult-people.html Notice that everything they say is still oriented around consensus -- "You may not persuade the person in question, but that's okay as long as you persuade everyone else", "a perfect example of how to build a strong case on neutral, quantitative data", the "masterful strategy" is to build consensus before acting. The consensus ideal is perfectly compatible with dealing with difficult people. Other documents --------------------- Like I said at the start, this is just my attempt to distill some abstract principles from my own experience. I can't take credit for most of these insights, and no doubt I've articulated some of them poorly. Fortunately, we don't all have to agree on every detail to get things done :-). But if you want to read more about this topic, and from other perspectives, here are some decent documents: http://producingoss.com/en/consensus-democracy.html https://www.ietf.org/tao.html#getting.things.done https://en.wikipedia.org/wiki/Consensus_decision-making And, of course, I would love to hear feedback on this document! - N

21 87

Test failures - which dependencies am I missing?
by Chris Ball 02 May '12

02 May '12

Hi, When I build NumPy and then run the tests on Ubuntu (10.04 LTS) and Debian (6.1), I always seem to get several failures. I guess most of these failures come from not having some dependencies installed, but I can't figure out which ones by reading e.g. http://docs.scipy.org/doc/numpy/user/install.html. It would be great if someone could tell me what I've likely missed! I remember Gael Varoquaux posted a few weeks back with some of the same errors (http://thread.gmane.org/gmane.comp.python.numeric.general/49032/). He was also using Ubuntu (though a newer version). Anyway, on Ubuntu here are the errors - other than known failures - after "python setup.py build_ext -i" (or python setup.py build_ext -i -- fcompiler=gnu") followed by nosetests: ====================================================================== ERROR: Failure: ImportError (cannot import name fib2) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/scratch/ceball/numpy/numpy/distutils/tests/f2py_ext/tests/ test_fib2.py", line 3, in <module> from f2py_ext import fib2 ImportError: cannot import name fib2 ====================================================================== ERROR: Failure: ImportError (cannot import name foo) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/scratch/ceball/numpy/numpy/distutils/tests/f2py_f90_ext/tests/ test_foo.py", line 3, in <module> from f2py_f90_ext import foo ImportError: cannot import name foo ====================================================================== ERROR: Failure: ImportError (cannot import name fib3) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/scratch/ceball/numpy/numpy/distutils/tests/gen_ext/tests/ test_fib3.py", line 3, in <module> from gen_ext import fib3 ImportError: cannot import name fib3 ====================================================================== ERROR: Failure: ImportError (No module named primes) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/scratch/ceball/numpy/numpy/distutils/tests/pyrex_ext/tests/ test_primes.py", line 3, in <module> from pyrex_ext.primes import primes ImportError: No module named primes ====================================================================== ERROR: Failure: ImportError (cannot import name example) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/scratch/ceball/numpy/numpy/distutils/tests/swig_ext/tests/ test_example.py", line 3, in <module> from swig_ext import example ImportError: cannot import name example ====================================================================== ERROR: Failure: ImportError (cannot import name example2) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/scratch/ceball/numpy/numpy/distutils/tests/swig_ext/tests/ test_example2.py", line 3, in <module> from swig_ext import example2 ImportError: cannot import name example2 ---------------------------------------------------------------------- I see something similar on Debian 6 (x86_64). You can see the full output for that at https://jenkins.shiningpanda.com/scipy/job/NumPy/ PLATFORM=debian6,PYTHON=CPython-2.7/19/consoleFull, in case it helps. Thanks, Chris

3 4

Continuous Integration
by Travis Oliphant 02 May '12

02 May '12

Hello all, NumFOCUS has been working with Continuum Analytics and multiple people in the community on Continuous Integration services for NumPy. Right now the tools we are using are: TeamCity ShiningPandas One great thing about Continuous Integration is that you don't have to make a single decision as long as you have a place to report everything. Continuum Analytics is hosting a TeamCity site and I believe some ShiningPandas sites have been set up. We have some work to do to get the build agents up and running and a single point that communicates the status of all the agents. This email is to just let people know that we are still working on this. Maggie Mari and Bryan van de Ven have been doing the work on the TeamCity side. There are several others who have been involved world wide in setting up build agents and investigating ShiningPandas. Continuum Analytics is still looking for someone who can devote at least 50% of their time to working on these issues as part of their full time job. You can be located anywhere in the world for this. If you are interested send a note to jobs(a)continuum.io If you have particular reasons why we should choose a particular CI service, please speak up and let your voice be heard. There is still time to make a difference in what we are setting up. Best regards, -Travis

5 8

1.6.2 release - backports and MSVC testing help
by Ralf Gommers 01 May '12

01 May '12

Hi all, Charles has done a great job of backporting a lot of bug fixes to 1.6.2, see PRs 260, 261, 262 and 263. For those who are interested, please have a look at those PRs to see and comment on what's proposed to go into 1.6.2. I also have a request for help with testing: can someone who uses MSVC test (preferably with a 2.x and a 3.x version)? I have a branch with all four PRs merged at https://github.com/rgommers/numpy/tree/bports Thanks, Ralf

4 9

Python3, genfromtxt and unicode
by Antony Lee 01 May '12

01 May '12

With bytes fields, genfromtxt(dtype=None) sets the sizes of the fields to the largest number of chars (npyio.py line 1596), but it doesn't do the same for unicode fields, which is a pity. See example below. I tried to change npyio.py around line 1600 to add that but it didn't work; from my limited understanding the problem comes earlier, in the way StringBuilder is defined(?). Antony Lee import io, numpy as np s = io.BytesIO() s.write(b"abc 1\ndef 2") s.seek(0) t = np.genfromtxt(s, dtype=None) # (or converters={0: bytes}) print(t, t.dtype) # -> [(b'a', 1) (b'b', 2)] [('f0', '|S1'), ('f1', '<i8')] s.seek(0) t = np.genfromtxt(s, dtype=None, converters={0: lambda s: s.decode("utf-8")}) print(t, t.dtype) # -> [('', 1) ('', 2)] [('f0', '<U0'), ('f1', '<i8')]

2 2

datetime dtype possible regression
by Travis Vaught 30 Apr '12

30 Apr '12

With NumPy 1.6.1 (from EPD 7.2-2) I get this behavior: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In [1]: import numpy as np In [2]: schema = np.dtype({'names':['symbol', 'date', 'open', 'high', 'low', ...: 'close', 'volume', 'adjclose'], ...: 'formats':['S8', 'M8', float, float, float, float, ...: float, float]}) In [3]: data = [("AAPL", "2012-04-12", 600.0, 605.0, 598.0, 602.0, 50000000, 602.0),] In [4]: recdata = np.array(data, dtype=schema) In [5]: recdata Out[5]: array([ ('AAPL', datetime.datetime(2012, 4, 12, 0, 0), 600.0, 605.0, 598.0, 602.0, 50000000.0, 602.0)], dtype=[('symbol', '|S8'), ('date', ('<M8[us]', {})), ('open', '<f8'), ('high', '<f8'), ('low', '<f8'), ('close', '<f8'), ('volume', '<f8'), ('adjclose', '<f8')]) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With numpy-1.7.0.dev_3cb783e I get this: >>> import numpy as np >>> schema = np.dtype({'names':['symbol','data','open','high','low','close','volume','adjclose'], 'formats':['S8','M8',float,float,float,float,float,float]}) >>> data = [("AAPL", "2012-04-12", 600.0, 605.0, 598.0, 602.0, 50000000, 602.0),] >>> recdata = np.array(data, dtype=schema) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot create a NumPy datetime other than NaT with generic units >>> np.version.version '1.7.0.dev-3cb783e' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Any hints about a regression I can check for? Or perhaps I missed an api change for specifying datetime dtypes? Best, Travis

5 8

Euroscipy 2012 deadline extension: May 7th
by Emmanuelle Gouillart 30 Apr '12

30 Apr '12

The committee of the Euroscipy 2012 conference has extended the deadline for abstract submission to **Monday May 7th, midnight** (Brussels time). Up to then, new abstracts may be submitted on http://www.euroscipy.org/conference/euroscipy2012, and already-submitted abstracts can be modified. We are very much looking forward to your submissions to the conference. Euroscipy 2012 is the annual European conference for scientists using Python. It will be held August 23-27 2012 in Brussels, Belgium. It is also still possible to propose sprints that will take place after the conference, please write to Berkin Malkoc (malkocb(a)itu.edu.tr) for practical organization (rooms, ...). Any other questions should be addressed exclusively to org-team(a)lists.euroscipy.org -- Emmanuelle

1 0

Alternative to R phyper
by Bruno Santos 30 Apr '12

30 Apr '12

Hello everyone, I have a bit of code where I am using rpy2 to import R phyper so I can perform an hypergeometric test. Unfortunately our cluster does not have a functional installation of rpy2 working. So I am wondering if I could translate to scipy which would make the code completly independent of R. The python code I have is as following: def lphyper(self,q,m,n,k): """ self.phyper(self,q,m,n,k) Calculate p-value using R function phyper from rpy2 low-level interface. "R Documentation phyper(q, m, n, k, lower.tail = TRUE, log.p = FALSE) q: vector of quantiles representing the number of white balls drawn without replacement from an urn which contains both black and white balls. m: the number of white balls in the urn. n: the number of black balls in the urn. k: the number of balls drawn from the urn. log.p: logical; if TRUE, probabilities p are given as log(p). lower.tail: logical; if TRUE (default), probabilities are P[X <= x], otherwise, P[X > x]. """ phyper_q = SexpVector([q,], rinterface.INTSXP) phyper_m = SexpVector([m,], rinterface.INTSXP) phyper_n = SexpVector([n,], rinterface.INTSXP) phyper_k = SexpVector([k,], rinterface.INTSXP) return phyper(phyper_q,phyper_m,phyper_n,phyper_k,**myparams)[0] I have looked to scipy.stats.hypergeom but it is giving me a different result which is also negative. > 1-phyper(45, 92, 7518, 1329) [1] 6.92113e-13 In [24]: stats.hypergeom.sf(45,(92+7518),92,1329) Out[24]: -8.4343643180773142e-12 This was supposed to be an error with an older version of scipy but I am using more recent versions of it which should not contain the error anymore: In [26]: numpy.__version__ Out[26]: '1.5.1' In [27]: scipy.__version__ Out[27]: '0.9.0' thank you very much in advance for any help. Best, Bruno

3 2

A crazy masked-array thought
by Richard Hattersley 29 Apr '12

29 Apr '12

The masked array discussions have brought up all sorts of interesting topics - too many to usefully list here - but there's one aspect I haven't spotted yet. Perhaps that's because it's flat out wrong, or crazy, or just too awkward to be helpful. But ... Shouldn't masked arrays (MA) be a superclass of the plain-old-array (POA)? In the library I'm working on, the introduction of MAs (via numpy.ma) required us to sweep through the library and make a fair few changes. That's not the sort of thing one would normally expect from the introduction of a subclass. Putting aside the ABI issue, would it help downstream API compatibility if the POA was a subclass of the MA? Code that's expecting/casting-to a POA might continue to work and, where appropriate, could be upgraded in their own time to accept MAs. Richard Hattersley

7 13