[Numpy-discussion] Future of numpy (was: DARPA funding for Blaze and passing the NumPy torch)
matthew.brett at gmail.com
Mon Dec 24 04:24:38 EST 2012
On Sun, Dec 23, 2012 at 7:00 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Sat, Dec 22, 2012 at 5:36 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> On Thu, Dec 20, 2012 at 5:39 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>> On Thu, Dec 20, 2012 at 11:46 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>> Travis - I think you are suggesting that there should be no one
>>>> person in charge of numpy, and I think this is very unlikely to work
>>>> well. Perhaps there are good examples of well-led projects where
>>>> there is not a clear leader, but I can't think of any myself at the
>>>> moment. My worry would be that, without a clear leader, it will be
>>>> unclear how decisions are made, and that will make it very hard to
>>>> take strategic decisions.
>>> Curious; my feeling is the opposite, that among mature and successful
>>> FOSS projects, having a clear leader is the uncommon case. GCC
>>> doesn't, Glibc not only has no leader but they recently decided to get
>>> rid of their formal steering committee, I'm pretty sure git doesn't,
>>> Apache certainly doesn't, Samba doesn't really, etc. As usual Karl
>>> Fogel has sensible comments on this:
>> Ah yes - that is curious. My - er - speculation was based on:
>> Numpy - Travis golden age in which we still bask
>> Sympy - Ondrej, then Aaron - evolving into group decision making AFAICT
>> IPython - Fernando, evolving into group decision making, AFAICT
>> Cython - Robert Bradshaw - evolving into ... - you get the idea.
>> and then reading about businesses particularly Good to Great, Built to
>> Last, the disaster at HP when they didn't take care about succession.
>> In general, that reading gave me the impression that successful
>> organizations take enormous care about succession. I can't think of
>> any case in the business literature I've read where a successful
>> leader handed over to a group of three.
> I think this is just a case of different organizational styles working
> differently. If organizations were optimisation algorithms, good
> businesses would be Newton's method, and good FOSS projects would be
> simulated annealing, or maybe GAs. Slower and less focused, but more
> robust against noise and local minima, and less susceptible to
> perturbations. They depend much less on the focused attention of
> visionary leaders.
You seem to be implying that any management organization for numpy
would have the same effect as any other as far as we know, and if that
were true, it would certainly not be worth discussing in any detail.
But I doubt very much that is true, and, following the optimization
strategy logic, there may be a reason that having 3 people lead an
organization has not been a common and visible option, and that is
that it doesn't work very well.
> (Also I wouldn't consider numpy to have a formal "group of three
> leaders" now just because Travis mentioned three names in his email.
> Leadership is something people do, not something people are, so it's a
> fuzzy category in the first place.)
In Travis' email I saw the three of you and a 2 to 1 voting
suggestion. There doesn't seem to be much appetite for discussing
alternatives and neither Chuck nor Ralf have joined this discussion,
so that seems to be the only option on the table. Is there another
one? Or are you thinking that something will gradually evolve from -
essentially - no prescription of how things work. Again, I doubt very
much that "no prescription" has been a successful option in the past,
even for FOSS. It just never happens in businesses or successful
governments - does it?
The problem is obviously going to be what to do when we get problems,
as we have in the past. When there is no or little structure to use,
then decisions don't get made, or get made by default, and the debate
deteriorates because there is no-one to moderate it. That seems to me
the situation numpy is in, and doing nothing or having poorly defined
management can only prolong that or even make it worse.
>>> In practice the main job of a successful FOSS leader is to refuse to
>>> make decisions, nudge people to work things out, and then if they
>>> refuse to work things out tell them to go away until they do:
>>> and what actually gives people influence in a project is the respect
>>> of the other members. The former stuff is stuff anyone can do, and the
>>> latter isn't something you can confer or take away with a vote.
>> Right. My impression is - I'm happy to be corrected with better
>> information - that the leader of a to-be-successful organization is
>> very good at encouraging a spirit of free and vigorous debate, strong
>> opinion, and reasoned decisions - and that may be the main gift they
>> give to the organization. At that point, usually under that leader's
>> supervision, the decision making starts diffusing over the group, as
>> they learn to discuss and make decisions together.
>> As I was teaching my niece and nephew to say to their parents in the
>> car - Daddy - are we there yet?
>> If we are not already there, how are we going to get there?
For example - you didn't answer this one. What are your thoughts?
>>> Nor do we necessarily have a great track record for executive
>>> decisions actually working things out.
>> No, I agree, the right leader will help form the group well for making
>> good group decisions. I think.
>> In the mean-time - now that there is a change - could I ask - where do
>> you three see Numpy going in the next five years? What do you see
>> as the challenges to solve? What are the big risks? What are the big
> Personally I'd like to see NA support and sparse ndarrays in numpy
> proper, but I'm not going to have the time to write them myself in the
> forseeable future...
> In the long run of course everyone wants a version of numpy+python
> that can do automatic loop fusion (since that's the core feature for
> achieving throughput on modern CPUs) without giving up the ability to
> interface with C code and CPython compatibility. In my dreams the PyPy
> people will get their act together WRT interfacing with C code, the
> Cython people will take advantage of this to write a Cython-to-RPython
> compiler that lets the PyPy optimizer see the internals of
> Cython-written code, and then we port numpy to Cython and get a single
> compatible code-base that can run fast on both CPython and PyPy. But
> who knows what will actually make sense, if anything; as they say,
> it's very hard to make predictions, especially about the future.
> And of course the actual long-term strategic plan is "review PRs,
> merge the good ones".
The contrast between these last two paragraphs is strong. The first
is a vision for how numpy might be - to coin a phrase - "the next
generation of numpy". It seems exciting and interesting, but you
don't hold out much hope of getting there, and having
not-much-management makes it less likely we will have any real new
direction to the project. That's your second paragraph - keep on
keeping on. In effect it condemns numpy to be a slow-moving
development effort waiting for something more interesting to overtake
it. Is that really necessary? Can we not hope for better? Is there
any real chance of attracting a force of new developers in that
More information about the NumPy-Discussion