[Numpy-discussion] Future of numpy (was: DARPA funding for Blaze and passing the NumPy torch)

Nathaniel Smith njs at pobox.com
Sun Dec 23 14:00:10 EST 2012

On Sat, Dec 22, 2012 at 5:36 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
> On Thu, Dec 20, 2012 at 5:39 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Thu, Dec 20, 2012 at 11:46 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> Travis - I think you are suggesting that there should be no  one
>>> person in charge of numpy, and I think this is very unlikely to work
>>> well.   Perhaps there are good examples of well-led projects where
>>> there is not a clear leader, but I can't think of any myself at the
>>> moment.  My worry would be that, without a clear leader, it will be
>>> unclear how decisions are made, and that will make it very hard to
>>> take strategic decisions.
>> Curious; my feeling is the opposite, that among mature and successful
>> FOSS projects, having a clear leader is the uncommon case. GCC
>> doesn't, Glibc not only has no leader but they recently decided to get
>> rid of their formal steering committee, I'm pretty sure git doesn't,
>> Apache certainly doesn't, Samba doesn't really, etc. As usual Karl
>> Fogel has sensible comments on this:
>>   http://producingoss.com/en/consensus-democracy.html
> Ah yes - that is curious.  My - er - speculation was based on:
> Numpy - Travis golden age in which we still bask
> Sympy - Ondrej, then Aaron - evolving into group decision making AFAICT
> IPython - Fernando, evolving into group decision making, AFAICT
> Cython - Robert Bradshaw - evolving into ... - you get the idea.
> and then reading about businesses particularly Good to Great, Built to
> Last, the disaster at HP when they didn't take care about succession.
> In general, that reading gave me the impression that  successful
> organizations take enormous care about succession.  I can't think of
> any case in the business literature I've read where a successful
> leader handed over to a group of three.

I think this is just a case of different organizational styles working
differently. If organizations were optimisation algorithms, good
businesses would be Newton's method, and good FOSS projects would be
simulated annealing, or maybe GAs. Slower and less focused, but more
robust against noise and local minima, and less susceptible to
perturbations. They depend much less on the focused attention of
visionary leaders.

(Also I wouldn't consider numpy to have a formal "group of three
leaders" now just because Travis mentioned three names in his email.
Leadership is something people do, not something people are, so it's a
fuzzy category in the first place.)

>> In practice the main job of a successful FOSS leader is to refuse to
>> make decisions, nudge people to work things out, and then if they
>> refuse to work things out tell them to go away until they do:
>>   https://lwn.net/Articles/105375/
>> and what actually gives people influence in a project is the respect
>> of the other members. The former stuff is stuff anyone can do, and the
>> latter isn't something you can confer or take away with a vote.
> Right.  My impression is - I'm happy to be corrected with better
> information - that the leader of a to-be-successful organization is
> very good at encouraging a spirit of free and vigorous debate, strong
> opinion, and reasoned decisions - and that may be the main gift they
> give to the organization. At that point, usually under that leader's
> supervision, the decision making starts diffusing over the group, as
> they learn to discuss and make decisions together.
> As I was teaching my niece and nephew to say to their parents in the
> car - Daddy - are we there yet?
> If we are not already there, how are we going to get there?
>> Nor do we necessarily have a great track record for executive
>> decisions actually working things out.
> No, I agree, the right leader will help form the group well for making
> good group decisions.  I think.
> In the mean-time - now that there is a change - could I ask - where do
> you three see Numpy going in the next five years?    What do you see
> as the challenges to solve?  What are the big risks?  What are the big
> possibilities?

Personally I'd like to see NA support and sparse ndarrays in numpy
proper, but I'm not going to have the time to write them myself in the
forseeable future...

In the long run of course everyone wants a version of numpy+python
that can do automatic loop fusion (since that's the core feature for
achieving throughput on modern CPUs) without giving up the ability to
interface with C code and CPython compatibility. In my dreams the PyPy
people will get their act together WRT interfacing with C code, the
Cython people will take advantage of this to write a Cython-to-RPython
compiler that lets the PyPy optimizer see the internals of
Cython-written code, and then we port numpy to Cython and get a single
compatible code-base that can run fast on both CPython and PyPy. But
who knows what will actually make sense, if anything; as they say,
it's very hard to make predictions, especially about the future.

And of course the actual long-term strategic plan is "review PRs,
merge the good ones".


More information about the NumPy-Discussion mailing list