Comments on governance proposal (was: Notes from the numpy dev meeting at scipy 2015)
Hi,
Splitting this one off too because it's a rather different discussion, although related.
On Tue, Aug 25, 2015 at 11:03 AM, Nathaniel Smith njs@pobox.com wrote:
[snip]
Formalizing our governance/decision making
This was a major focus of discussion. At a high level, the consensus was to steal IPython's governance document ("IPEP 29") and modify it to remove its use of a BDFL as a "backstop" to normal community consensus-based decision, and replace it with a new "backstop" based on Apache-project-style consensus voting amongst the core team.
Here's a plea to avoid a 'core' structure if at all possible. Historically it seems to have some severe risks, and experienced people have blamed this structure for the decline of various projects including NetBSD and Xfree86, summaries here:
http://asterisk.dynevor.org/melting-core.html http://asterisk.dynevor.org/xfree-forked.html
In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate.
There's also evidence from the NetBSD / OpenBSD split [1] and the XFree86 / X.org split [2] - that the core structure can lead to bad decisions being taken in private that no or few members of the core group are prepared to defend. I guess what is happening is that distributed responsibility leads to poor accountability, and therefore poor decisions.
So, I hope very much we can avoid that trap in our own governance.
Best,
Matthew
[1] http://mail-index.netbsd.org/netbsd-users/1994/12/23/0000.html [2] http://www.xfree86.org/pipermail/forum/2003-March/001997.html
Hi Matthew
On 2015-08-26 10:50:47, Matthew Brett matthew.brett@gmail.com wrote:
In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate.
Can you describe how a democratic governance structure would look? It's not clear from the discussions linked where successful examples are to be found.
Thanks Stéfan
Hi,
On Wed, Aug 26, 2015 at 11:46 PM, Stefan van der Walt stefanv@berkeley.edu wrote:
Hi Matthew
On 2015-08-26 10:50:47, Matthew Brett matthew.brett@gmail.com wrote:
In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate.
Can you describe how a democratic governance structure would look? It's not clear from the discussions linked where successful examples are to be found.
Ah yes - as I was writing at the top of the xfree86 summary, it's difficult to assess governance models, because you cannot tell if a project that has a particular governance model would have been more successful with another model. For example, would clang be competing so successfully with gcc, if gcc had had a different governance model? Would Apache be further ahead of the many competitors in the web-server space with different management? Difficult to know.
The advantage of studying forks is that they usually arise from disagreements about how a project is managed. All other things being equal, we might expect a fork to fail, given the general aversion to forks and the considerable new work that has to be done to get one going. So, if a fork succeeds in the long term, that is probably an indication that the governance / management of the fork is indeed an improvement on the previous model.
So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision.
Cheers,
Matthew
On Aug 27, 2015, at 9:36 AM, Matthew Brett matthew.brett@gmail.com wrote:
So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision.
Anecdotes about two projects is not compelling evidence of anything unless you can also point to a comparison of the corresponding success rate. Two failures out of three is suggestive. Two failures out of three hundred is significantly less interesting. More useful would be actual details of an alternative proposal or pointers to examples of alternative arrangements that could be modeled.
Bryan
Hi,
On Thu, Aug 27, 2015 at 10:15 AM, Bryan Van de Ven bryanv@continuum.io wrote:
On Aug 27, 2015, at 9:36 AM, Matthew Brett matthew.brett@gmail.com wrote:
So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision.
Anecdotes about two projects is not compelling evidence of anything unless you can also point to a comparison of the corresponding success rate. Two failures out of three is suggestive. Two failures out of three hundred is significantly less interesting. More useful would be actual details of an alternative proposal or pointers to examples of alternative arrangements that could be modeled.
Unfortunately, I don't think we have much choice but to do our best in sifting through the anecdotal evidence we have available, weak and contradictory as it is. Successful forks in large projects are pretty rare, and as I was arguing before, they are particularly useful as evidence about governance models.
In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """
http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html
Cheers,
Matthew
On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.brett@gmail.com wrote:
In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """
Who goes on to propose:
7) The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the "core" group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the "core" group worked during most of the project's growth period.)
Which, AFAICT, is exactly in line with the Numpy proposal:
""" During the everyday project activities, council members participate in all discussions, code review and other project activities as peers with all other Contributors and the Community. In these everyday activities, Council Members do not have any special power or privilege through their membership on the Council. ... However, the Council's primary responsibility is to facilitate the ordinary community-based decision making procedure described above. If we ever have to step in and formally override the community for the health of the Project, then we will do so, but we will consider reaching this point to indicate a failure in our leadership. """
Bryan
Hi,
On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bryanv@continuum.io wrote:
On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.brett@gmail.com wrote:
In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """
Who goes on to propose:
- The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the "core" group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the "core" group worked during most of the project's growth period.)
Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble.
It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback.
It would be great, and very convenient, if the only management we needed was getting out of the way, but I doubt very much that that is the case.
Cheers,
Matthew
On Do, 2015-08-27 at 10:45 +0100, Matthew Brett wrote:
Hi,
On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bryanv@continuum.io wrote:
On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.brett@gmail.com wrote:
In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """
Who goes on to propose:
- The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the "core" group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the "core" group worked during most of the project's growth period.)
Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble.
It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback.
Maybe so. I do not know much at all about these models, but I am not sure how much applies here to numpy. Isn't at least FreeBSD a magnitude larger then numpy? We do need to have some formality about how to give out commit rights, and do final decision when all else fails.
One thing I do not know is how a "community vote" could work at all, considering I do not even know how to count its members. Votes and presidents make sense to me for large projects with hundrets of developers on different corners (think of the gnome foundation, debian probably) [1].
One thing I could imagine adding is that the community should be encouraged to ask for/propose new members for the "core" team.
Nobody is particularly in love with this model, but maybe out of our own ignorance, we do not see many alternatives after ruling out a BDFL. Yes, it is a lot fixing a status quo, but we have to fixate something. Any alternative suggestions are welcome and would be even after deciding on this. Though maybe that takes away some momentum.
- Sebastian
[1] If we were to have a central governance for the SciPy stack the story would seem very different to me.
It would be great, and very convenient, if the only management we needed was getting out of the way, but I doubt very much that that is the case.
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
On Thu, Aug 27, 2015 at 12:11 PM, Sebastian Berg sebastian@sipsolutions.net wrote:
On Do, 2015-08-27 at 10:45 +0100, Matthew Brett wrote:
Hi,
On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bryanv@continuum.io wrote:
On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.brett@gmail.com wrote:
In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """
Who goes on to propose:
- The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the "core" group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the "core" group worked during most of the project's growth period.)
Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble.
It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback.
Maybe so. I do not know much at all about these models, but I am not sure how much applies here to numpy. Isn't at least FreeBSD a magnitude larger then numpy?
It seems to me that numpy suffers from the same risks of poor accountability, stagnation and conservatism that larger projects do. Is there a reason that would not be the case?
We do need to have some formality about how to give out commit rights, and do final decision when all else fails.
Yes, sure, something formal is probably but not certainly better than nothing, depending on what the 'something formal' is.
One thing I do not know is how a "community vote" could work at all, considering I do not even know how to count its members. Votes and presidents make sense to me for large projects with hundrets of developers on different corners (think of the gnome foundation, debian probably) [1].
The 'president' idea is to get at the problem of lack of accountability, along with selection for leadership skill rather than coding ability. It's trying to get at the advantages of the BDFL model in our situation where there is no obvious BDFL. For the me the problem is that, at the moment, if the formal or informal governing body makes a bad decision, then no member will feel responsible for that decision or its consequences. That tends to lead to an atmosphere of - "oh well, what could we do, X wouldn't agree to A and Y wouldn't agree to B so we're stuck". It seems to me we need a system such that whoever is in charge feels so strongly that it is their job to make numpy as good as possible, that they will take whatever difficult or sensitive decisions are necessary to make that happen. On the other hand the 'core' system seems to function on a model of mutual deference and personal loyalty that I believe is destructive of good management.
Cheers,
Matthew
On Thu, Aug 27, 2015 at 8:57 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 12:11 PM, Sebastian Berg sebastian@sipsolutions.net wrote:
On Do, 2015-08-27 at 10:45 +0100, Matthew Brett wrote:
Hi,
On Thu, Aug 27, 2015 at 10:35 AM, Bryan Van de Ven bryanv@continuum.io
wrote:
On Aug 27, 2015, at 10:22 AM, Matthew Brett matthew.brett@gmail.com
wrote:
In the case of the 'core' model, we have some compelling testimony from someone with a great deal of experience:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being
in
wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and
XFree86,
for example, have both forked successor projects (Dragonfly and
X.org)
for very similar reasons. """
Who goes on to propose:
- The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept
feedback,
and make good decisions. More to the point, though, the "core"
group
must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the
community
from developing better solutions. (This is how the "core" group worked during most of the project's growth period.)
Sure. I think it's reasonable to give high weight to Hannum's assessment of the failure of the core group, but less weight to his proposal for a replacement, because at the time, I don't believe he was in a good position to assess whether his (apparent) alternative would run into the same trouble.
It's always tempting to blame the people rather than the system, but in this case, I strongly suspect that it was the system that was fundamentally flawed, therefore either promoting the wrong people or putting otherwise competent people into situations where they are no longer getting useful feedback.
Maybe so. I do not know much at all about these models, but I am not sure how much applies here to numpy. Isn't at least FreeBSD a magnitude larger then numpy?
It seems to me that numpy suffers from the same risks of poor accountability, stagnation and conservatism that larger projects do. Is there a reason that would not be the case?
We do need to have some formality about how to give out commit rights, and do final decision when all else fails.
Yes, sure, something formal is probably but not certainly better than nothing, depending on what the 'something formal' is.
One thing I do not know is how a "community vote" could work at all, considering I do not even know how to count its members. Votes and presidents make sense to me for large projects with hundrets of developers on different corners (think of the gnome foundation, debian probably) [1].
The 'president' idea is to get at the problem of lack of accountability, along with selection for leadership skill rather than coding ability. It's trying to get at the advantages of the BDFL model in our situation where there is no obvious BDFL. For the me the problem is that, at the moment, if the formal or informal governing body makes a bad decision, then no member will feel responsible for that decision or its consequences. That tends to lead to an atmosphere of - "oh well, what could we do, X wouldn't agree to A and Y wouldn't agree to B so we're stuck". It seems to me we need a system such that whoever is in charge feels so strongly that it is their job to make numpy as good as possible, that they will take whatever difficult or sensitive decisions are necessary to make that happen. On the other hand the 'core' system seems to function on a model of mutual deference and personal loyalty that I believe is destructive of good management.
I don't really see a problem with "codifying" the status quo.
It might become necessary to have something like an administrative director if numpy becomes a more formal organization with funding, but for the development of the project I don't see any need for a president. If there is no obvious BDFL, then I guess there is also no obvious president. (I would vote for Ralf as president of everything, but I don't think he's available.)
As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president.
Given the current pattern all critical issues end up in a public debate on the mailing list. What numpy (and scipy) need is to have someone as a tie breaker to make any final decisions if there is no clear consensus, if there is no BDFL for it, then having the "core" group making those decisions looks appropriate to me.
"On the other hand the 'core' system seems to function on a
model of mutual deference and personal loyalty that I believe is destructive of good management."
That sounds actually like a good basis for team work to me. Also that has "mutual" in it instead of just deferring and being loyal to a president.
Since I know scipy development much better:
scipy has made a huge progress in the last 5 or 6 years since I've been following it, both in terms of code, in terms of development workflow, and in the number of developers. (When I started, I was essentially alone in scipy.stats, now there are 3 to 5 "core" developers that at least partially work on it, everything goes through PRs with public discussion and with critical issues additionally raised on the mailing list.)
Ralf and Pauli are the defacto BDFLs for scipy overall, but all decisions in recent years have been without a fight, but not without lots of arguments, and, given the size and breadth of scipy, there are field experts (although not enough of those) to help in the discussion.
There are still stalled PRs, blocked proposals, and decisions that I didn't like, but I think that's unavoidable.
One feature of the current "core" system is that it is relatively open, almost all discussions are public, and it is relatively (?) easy to get into it for new developers. I currently don't worry that a closed clique is taking over the numpy or scipy "core" group:.
Josef
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
* what numpy is doing well; * what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Given the current pattern all critical issues end up in a public debate on the mailing list. What numpy (and scipy) need is to have someone as a tie breaker to make any final decisions if there is no clear consensus, if there is no BDFL for it, then having the "core" group making those decisions looks appropriate to me.
"On the other hand the 'core' system seems to function on a
model of mutual deference and personal loyalty that I believe is destructive of good management."
That sounds actually like a good basis for team work to me. Also that has "mutual" in it instead of just deferring and being loyal to a president.
Since I know scipy development much better:
scipy has made a huge progress in the last 5 or 6 years since I've been following it, both in terms of code, in terms of development workflow, and in the number of developers. (When I started, I was essentially alone in scipy.stats, now there are 3 to 5 "core" developers that at least partially work on it, everything goes through PRs with public discussion and with critical issues additionally raised on the mailing list.)
Ralf and Pauli are the defacto BDFLs for scipy overall, but all decisions in recent years have been without a fight, but not without lots of arguments, and, given the size and breadth of scipy, there are field experts (although not enough of those) to help in the discussion.
I agree entirely, I think scipy is a good example where stability and clarity of leadership has made a huge difference to the health of the project.
Cheers,
Matthew
[1] https://hbr.org/2005/07/level-5-leadership-the-triumph-of-humility-and-fierc...
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public discussion
about
the direction of the project without having to delegate providing a
vision
to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons) to me, and cannot be pushed of to an official president.
Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion.
Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO.
(Given the importance of numpy itself, there should be enough remedies if the "core" group ever gets `out of touch` with the very large user base.)
Josef
Given the current pattern all critical issues end up in a public debate
on
the mailing list. What numpy (and scipy) need is to have someone as a tie breaker to make any final decisions if there is no clear consensus, if
there
is no BDFL for it, then having the "core" group making those decisions
looks
appropriate to me.
"On the other hand the 'core' system seems to function on a
model of mutual deference and personal loyalty that I believe is destructive of good management."
That sounds actually like a good basis for team work to me. Also that has "mutual" in it instead of just deferring and being loyal to a president.
Since I know scipy development much better:
scipy has made a huge progress in the last 5 or 6 years since I've been following it, both in terms of code, in terms of development workflow,
and
in the number of developers. (When I started, I was essentially alone in scipy.stats, now there are 3 to 5 "core" developers that at least
partially
work on it, everything goes through PRs with public discussion and with critical issues additionally raised on the mailing list.)
Ralf and Pauli are the defacto BDFLs for scipy overall, but all
decisions in
recent years have been without a fight, but not without lots of
arguments,
and, given the size and breadth of scipy, there are field experts
(although
not enough of those) to help in the discussion.
I agree entirely, I think scipy is a good example where stability and clarity of leadership has made a huge difference to the health of the project.
Cheers,
Matthew
[1] https://hbr.org/2005/07/level-5-leadership-the-triumph-of-humility-and-fierc... _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons) to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion.
Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO.
Yes, if the roles were honorary and administrative, they would not be useful.
Cheers,
Matthew
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett <matthew.brett@gmail.com
wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons)
to
me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
First, it's several individuals taking joint decisions, jointly agree or not object (LGTM) to merging PRs, it's still a joint decision and accountability is not exclusive. The PR review process makes decisions much more into a joint decision process than it was with isolated SVN commits. (*)
Second, if there are separated decisions, then it could also lead to excess change. All these enthusiastic new developers bringing in whatever they (and the local chief) like, and nobody to stop them.
In either case, the developer, or local chief, has to deal with the consequences. You merged this PR, now fix it. or Why are you holding up this PR? I can merge it.
(*) Even though I'm not a scipy developer anymore, I still feel partially responsible for it, I'm still reviewing some PRs and comment on them, sometimes as cheerleader in favor of something or sometimes pointing out problems, or just checking that it makes sense, and always with an eye on what downstream impact it might have.
Another thought: Having an accountable president might actually reduce the feeling of accountability and responsibility of individual developers, so the neteffect is negative. "The president is responsible for this (even though he doesn't have enough time), so I can skip part of this review."
Josef
Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a
large
number of contributors. (stylized roles) and noisy mailing list for feedback and discussion.
Given the size of the numpy development group, numpy needs individuals
for
the vision and to push things not a president, vice-presidents and
assistant
vice-presidents, IMO.
Yes, if the roles were honorary and administrative, they would not be useful.
I'm not sure what you mean here. Given that it's all volunteer work, any president wouldn't have any hard tools.
Josef
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons) to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
Cheers,
Matthew
On 2015-08-27 11:06:10, Matthew Brett matthew.brett@gmail.com wrote:
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
It looks to me as though the team is doing an excellent job of maintaining NumPy. The growth of the project has stagnated somewhat for numerous reasons---and a lack of ideas on the table is not one of them, rather whether / how to take them forward.
The question, and I think what you also highlighted in the earlier part of this discussion, is: how to decide on which vision to adopt, and who takes responsibility for making that happen?
Are the two models proposed thus far so different, or can they be merged in a way that makes sense? E.g., can we work as a community to rally behind a vision as set out by one person, and then repeat that process to focus on another a year later? Think of it as the iterative development equivalent of governance.
This may just be another way of phrasing a precedency, but with a strong emphasis on its temporary nature, as well as a focus on a group-decided outcome. Alternatively, see it as a community governance model with a strong emphasis on responsibility.
Stéfan
On Do, 2015-08-27 at 12:34 -0700, Stefan van der Walt wrote:
On 2015-08-27 11:06:10, Matthew Brett matthew.brett@gmail.com wrote:
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
It looks to me as though the team is doing an excellent job of maintaining NumPy. The growth of the project has stagnated somewhat for numerous reasons---and a lack of ideas on the table is not one of them, rather whether / how to take them forward.
The question, and I think what you also highlighted in the earlier part of this discussion, is: how to decide on which vision to adopt, and who takes responsibility for making that happen?
Are the two models proposed thus far so different, or can they be merged in a way that makes sense? E.g., can we work as a community to rally behind a vision as set out by one person, and then repeat that process to focus on another a year later? Think of it as the iterative development equivalent of governance.
This may just be another way of phrasing a precedency, but with a strong emphasis on its temporary nature, as well as a focus on a group-decided outcome. Alternatively, see it as a community governance model with a strong emphasis on responsibility.
Agreed. Are not PEP's/NEP's just that (and could possibly be formalized more, not sure how much they are in the current proposal) in some sense? Since they have a sponsor/author who can be said to be assigned to it/responsible once accepted.
I will add one more thing which I think is important:
The governance has to be create as little hassle as possible and it should be simple/short enough to quickly understand.
- Sebastian
Stéfan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi Sebastian
On 2015-08-27 14:45:50, Sebastian Berg sebastian@sipsolutions.net wrote:
Agreed. Are not PEP's/NEP's just that (and could possibly be formalized more, not sure how much they are in the current proposal) in some sense? Since they have a sponsor/author who can be said to be assigned to it/responsible once accepted.
I would consider a collection of NEPs for the following year to be such a thing. When implementing a bigger plan, it does help to have one person who owns the entire vision at the helm, pushing it forward. I think of it like a symphony orchestra: we agree to play the same music, and then let the director oversee the execution of the piece as a whole.
The governance has to be create as little hassle as possible and it should be simple/short enough to quickly understand.
I completely agree; and I think bikeshedding (everyone gets to argue their point of view, regardless of the stakes) can be the antithesis to productive focus.
Stéfan
On Thu, Aug 27, 2015 at 2:06 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett <matthew.brett@gmail.com
wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current
situation
is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public
discussion
about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for
the
performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to
persons)
to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
If you mean the XFree and NetBSD cases, then I don't see any similarity to the numpy or scipy development pattern. If I would draw any conclusion, then maybe that NetBSD hat too much of formal governance structures and not enough informal governance. It would be difficult to take over the government if there is no government.
just one aside "*No desire to recruit new users" *
We are on a mission to take over the world (*). And forks of numpy like pandas turn out to be mostly complementary and increase the userbase of numpy.
(R and Python are in friendly, or sometimes unfriendly, competition, but, AFAICS, we are both gaining users because of the others' presence. It's not a zero sum game in this case.)
(*) But that's not in the "mission statement".
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
I thought that was implicit or explicit in the other thread.
Josef
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett <matthew.brett@gmail.com
wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current
situation
is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public
discussion
about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for
the
performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to
persons)
to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
One thing we are doing very badly is leveraging resources outside of contributions of work and time from individuals. Getting sponsors to finance work on what is the cornerstone of just about any Python package that has to add two numbers together shouldn't be too hard, especially seeing success stories like Jupyter's, who I believe has several paid developers working full time. That requires formalizing governance, because apparently sponsors are a little wary of giving money to "people on the internet". ;-) Fernando Pérez was extremely emphatic about the size of the opportunity NumPy was letting slip by not formalizing *any* governance model. And it is a necessary first step so that e.g. we have the money to, say a year from now, get the right people together for a couple of days to figure out a better governance model. I'd argue that money would be better spent financing a talented developer to advance e.g. Nathaniel's new dtype system to end all dtype systems, but that's a different story.
Largely because of the above, even if Nathaniel's document involved tossing a coin to resolve disputes, I'd rather have that now than something much better never. Because there really is no alternative to Nathaniel's write-up of the status quo, other than the status quo without a write-up: it has taken him two months to put this draft together, **after** we agreed over several hours of face to face discussion on what the model should be. And I'm sure he has hated every minute he has had to put into it. So if we keep going around this in circles, after a few days we will all grow tired and go back to fighting over whether indexing should transpose subspaces or not, and all that other cool stuff we really enjoy. And a year from now we will be in the same place we are now, only a year older and deeper in (technical) debt.
Jaime
+10 Very well written down ideas Jaime.
2015-08-28 6:59 GMT+02:00 Jaime Fernández del Río jaime.frio@gmail.com:
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett <
matthew.brett@gmail.com>
wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip] > I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current
situation
is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
> As the current debate shows it's possible to have a public
discussion
> about > the direction of the project without having to delegate providing
a
> vision > to a president.
The idea of a president that I had in mind, was not someone who
makes
all decisions, but the person who holds themselves responsible for
the
performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If
you
don't know it already, I highly recommend Jim Collins' work on
'level
5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to
persons)
to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
One thing we are doing very badly is leveraging resources outside of contributions of work and time from individuals. Getting sponsors to finance work on what is the cornerstone of just about any Python package that has to add two numbers together shouldn't be too hard, especially seeing success stories like Jupyter's, who I believe has several paid developers working full time. That requires formalizing governance, because apparently sponsors are a little wary of giving money to "people on the internet". ;-) Fernando Pérez was extremely emphatic about the size of the opportunity NumPy was letting slip by not formalizing *any* governance model. And it is a necessary first step so that e.g. we have the money to, say a year from now, get the right people together for a couple of days to figure out a better governance model. I'd argue that money would be better spent financing a talented developer to advance e.g. Nathaniel's new dtype system to end all dtype systems, but that's a different story.
Largely because of the above, even if Nathaniel's document involved tossing a coin to resolve disputes, I'd rather have that now than something much better never. Because there really is no alternative to Nathaniel's write-up of the status quo, other than the status quo without a write-up: it has taken him two months to put this draft together, **after** we agreed over several hours of face to face discussion on what the model should be. And I'm sure he has hated every minute he has had to put into it. So if we keep going around this in circles, after a few days we will all grow tired and go back to fighting over whether indexing should transpose subspaces or not, and all that other cool stuff we really enjoy. And a year from now we will be in the same place we are now, only a year older and deeper in (technical) debt.
Jaime
-- (__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip] > I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
> As the current debate shows it's possible to have a public > discussion > about > the direction of the project without having to delegate providing > a > vision > to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons) to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
One reason might be that the specific form of governance can have no influence on the long-term health of the project.
I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project.
If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance.
Cheers,
Matthew
On Fr, 2015-08-28 at 09:46 +0100, Matthew Brett wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote: > > Hi, > > On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: > [snip] > > I don't really see a problem with "codifying" the status quo. > > That's an excellent point. If we believe that the current > situation > is the best possible, both now and in the future, then codifying the > status quo is an excellent idea. > > So, we should probably first start by asking ourselves: > > * what numpy is doing well; > * what numpy could do better; > > and then ask, is there some way we could make it more likely we will > improve over time. > > [snip] > > > As the current debate shows it's possible to have a public > > discussion > > about > > the direction of the project without having to delegate providing > > a > > vision > > to a president. > > The idea of a president that I had in mind, was not someone who > makes > all decisions, but the person who holds themselves responsible for > the > performance of the project. If the project has a coherent vision > already, the president has no need to provide one, but it's the > president's job to worry about whether we have vision or not, and do > what they need to, to make sure we don't lose track of that. If > you > don't know it already, I highly recommend Jim Collins' work on > 'level > 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons) to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
Oh, it is true. I think we (those in the room in Austin) just have thought about it a bit already, so now we have to be a bit patient with everyone who just saw the plans the first time. But I hope we can agree that we should decide on some form of governance in the next few weeks, even if it may not be perfect.
My personal problem with your ideas is not that I do not care for the warnings, but having already spend some time trying to put together this (and this is nothing weird, this is very common practice in open source), I personally do not want to spend time inventing something completely new.
We must discuss improvements to the document, and even whole different approaches. But for me at least, I need something a little more specific. Maybe I am daft, but I hear "this is a bad idea" without also providing another approach (that seems doable). And I do not buy that it is *that* bad, it is a very common governance structure for open source. The presidency suggestions may be another approach and certainly something we can pick up ideas from, but to me it is so vague that I cannot even start comprehending what it would mean for the actual governance structure specifically for numpy (considering the size of the project, etc.).
But by all means, I like proposals/learning from your ideas (i.e. maybe you can propose changes to the NEP sections), I personally would just like to see a bit more clearly where it goes.
- Sebastian
One reason might be that the specific form of governance can have no influence on the long-term health of the project.
I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project.
If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance.
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Aug 28, 2015 at 2:40 AM, Sebastian Berg sebastian@sipsolutions.net wrote:
On Fr, 2015-08-28 at 09:46 +0100, Matthew Brett wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett <
matthew.brett@gmail.com>
wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote: > > > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett > matthew.brett@gmail.com > wrote: >> >> Hi, >> >> On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: >> [snip] >> > I don't really see a problem with "codifying" the status quo. >> >> That's an excellent point. If we believe that the current >> situation >> is the best possible, both now and in the future, then
codifying the
>> status quo is an excellent idea. >> >> So, we should probably first start by asking ourselves: >> >> * what numpy is doing well; >> * what numpy could do better; >> >> and then ask, is there some way we could make it more likely we
will
>> improve over time. >> >> [snip] >> >> > As the current debate shows it's possible to have a public >> > discussion >> > about >> > the direction of the project without having to delegate
providing
>> > a >> > vision >> > to a president. >> >> The idea of a president that I had in mind, was not someone who >> makes >> all decisions, but the person who holds themselves responsible
for
>> the >> performance of the project. If the project has a coherent
vision
>> already, the president has no need to provide one, but it's the >> president's job to worry about whether we have vision or not,
and do
>> what they need to, to make sure we don't lose track of that.
If
>> you >> don't know it already, I highly recommend Jim Collins' work on >> 'level >> 5 leadership' [1] > > > Still doesn't sound like the need for a president to me > > " the person who holds themselves responsible for the > performance of the project" > > sounds more like the role of the "core" group (adding plural to > persons) > to > me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions
has
led to the situation where no-one feels themselves accountable for
the
result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going anywhere useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
<joke> Are you trying to prove the point that consensus doesn't work by
making it
impossible to reach a consensus on this? ;-)
</joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
Oh, it is true. I think we (those in the room in Austin) just have thought about it a bit already, so now we have to be a bit patient with everyone who just saw the plans the first time. But I hope we can agree that we should decide on some form of governance in the next few weeks, even if it may not be perfect.
My personal problem with your ideas is not that I do not care for the warnings, but having already spend some time trying to put together this (and this is nothing weird, this is very common practice in open source), I personally do not want to spend time inventing something completely new.
We must discuss improvements to the document, and even whole different approaches. But for me at least, I need something a little more specific. Maybe I am daft, but I hear "this is a bad idea" without also providing another approach (that seems doable). And I do not buy that it is *that* bad, it is a very common governance structure for open source. The presidency suggestions may be another approach and certainly something we can pick up ideas from, but to me it is so vague that I cannot even start comprehending what it would mean for the actual governance structure specifically for numpy (considering the size of the project, etc.).
But by all means, I like proposals/learning from your ideas (i.e. maybe you can propose changes to the NEP sections), I personally would just like to see a bit more clearly where it goes.
Perhaps we could add a paragraph to the document, stating that we understand the risks and will keep an eye open for the dilution of responsibility and lack of direction and ownership that may come from consensus based decision making. And make it part of our governance model that we will review the model yearly, to identify and correct issues. That wouldn't require any substantial change right now, but wouldn't crystallize a potentially harmful organization either.
Jaime
P.S. At some point during the discussion in Austin, the idea going around was that the NUMFocus committee, which at the time was going to have three members only, would also be vested with ultimate decision power. Just imagine, we could have had a proper triumvirate: Chuck, Nathaniel and Ralf, wearing togas and feasting around a triclinium while they decided the fate of NumPy!
On Fri, Aug 28, 2015 at 3:36 PM, Jaime Fernández del Río < jaime.frio@gmail.com> wrote:
On Fri, Aug 28, 2015 at 2:40 AM, Sebastian Berg < sebastian@sipsolutions.net> wrote:
On Fr, 2015-08-28 at 09:46 +0100, Matthew Brett wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:06 AM, Matthew Brett <
matthew.brett@gmail.com>
wrote:
Hi,
On Thu, Aug 27, 2015 at 6:23 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 12:22 PM, Matthew Brett matthew.brett@gmail.com wrote: > > Hi > > On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote: > > > > > > On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett > > matthew.brett@gmail.com > > wrote: > >> > >> Hi, > >> > >> On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com
wrote:
> >> [snip] > >> > I don't really see a problem with "codifying" the status quo. > >> > >> That's an excellent point. If we believe that the current > >> situation > >> is the best possible, both now and in the future, then
codifying the
> >> status quo is an excellent idea. > >> > >> So, we should probably first start by asking ourselves: > >> > >> * what numpy is doing well; > >> * what numpy could do better; > >> > >> and then ask, is there some way we could make it more likely
we will
> >> improve over time. > >> > >> [snip] > >> > >> > As the current debate shows it's possible to have a public > >> > discussion > >> > about > >> > the direction of the project without having to delegate
providing
> >> > a > >> > vision > >> > to a president. > >> > >> The idea of a president that I had in mind, was not someone who > >> makes > >> all decisions, but the person who holds themselves responsible
for
> >> the > >> performance of the project. If the project has a coherent
vision
> >> already, the president has no need to provide one, but it's the > >> president's job to worry about whether we have vision or not,
and do
> >> what they need to, to make sure we don't lose track of that.
If
> >> you > >> don't know it already, I highly recommend Jim Collins' work on > >> 'level > >> 5 leadership' [1] > > > > > > Still doesn't sound like the need for a president to me > > > > " the person who holds themselves responsible for the > > performance of the project" > > > > sounds more like the role of the "core" group (adding plural to > > persons) > > to > > me, and cannot be pushed of to an official president. > > Except that, in the past, having multiple people taking decisions
has
> led to the situation where no-one feels themselves accountable
for the
> result, hence this situation tends to lead to stagnation.
Is there any evidence for this?
Oh - dear - that's the key point, but I'm obviously not making it clearly enough. Yes there is, and that was the evidence I was pointing to before.
But anyway - Sebastian is right - this discussion isn't going
anywhere
useful.
So - let's step back.
In thinking about governance, we first need to ask what we want to achieve. This includes considering the risks ahead for the project.
So, in the spirit of fruitful discussion, can I ask what y'all consider to be the current problems with working on numpy (other than the technical ones). What is numpy doing well, and what is it doing badly? What risks do we have to plan for in the future?
<joke> Are you trying to prove the point that consensus doesn't work by
making it
impossible to reach a consensus on this? ;-)
</joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
Oh, it is true. I think we (those in the room in Austin) just have thought about it a bit already, so now we have to be a bit patient with everyone who just saw the plans the first time. But I hope we can agree that we should decide on some form of governance in the next few weeks, even if it may not be perfect.
My personal problem with your ideas is not that I do not care for the warnings, but having already spend some time trying to put together this (and this is nothing weird, this is very common practice in open source), I personally do not want to spend time inventing something completely new.
We must discuss improvements to the document, and even whole different approaches. But for me at least, I need something a little more specific. Maybe I am daft, but I hear "this is a bad idea" without also providing another approach (that seems doable). And I do not buy that it is *that* bad, it is a very common governance structure for open source. The presidency suggestions may be another approach and certainly something we can pick up ideas from, but to me it is so vague that I cannot even start comprehending what it would mean for the actual governance structure specifically for numpy (considering the size of the project, etc.).
But by all means, I like proposals/learning from your ideas (i.e. maybe you can propose changes to the NEP sections), I personally would just like to see a bit more clearly where it goes.
Perhaps we could add a paragraph to the document, stating that we understand the risks and will keep an eye open for the dilution of responsibility and lack of direction and ownership that may come from consensus based decision making. And make it part of our governance model that we will review the model yearly, to identify and correct issues. That wouldn't require any substantial change right now, but wouldn't crystallize a potentially harmful organization either.
Jaime
P.S. At some point during the discussion in Austin, the idea going around was that the NUMFocus committee, which at the time was going to have three members only, would also be vested with ultimate decision power. Just imagine, we could have had a proper triumvirate: Chuck, Nathaniel and Ralf, wearing togas and feasting around a triclinium while they decided the fate of NumPy!
The idea is appealing, but I don't think anyone should have to see me in a toga.
Chuck
1) I very much agree that governance can make or break a project. However, the actual governance approach often ends up making less difference than the people involved.
2) While the FreeBSD and XFree examples do point to some real problems with the "core" model it seems that there are many other projects that are using it quite successfully.
3) at least in the XFree case, maybe the biggest issue was not the core (or the board), but the fact that discussions and decision making process were kept secret (and restriction to CVS, EVEN read-only) -- this does not seem to be inherent to the "core" model at all, and not how nupy will even operate.
4) the biggest issue numpy has faced in years is a lack of people that can/will/do actually contribute code itself. there simply isn't a big group f folks waiting to step up and be the president. And the "leader", president of BDFL, etc, while primarily being a management role, needs to be fully technically competent -- there are very few people qualified at this point, and I suspect none of them want that job. -- in theory, one can manage without the technical competency, but does anyone have a single example of a successful open-source software project run by a good manager that isn't a top technical expert? All the BDFLs I know of are absolutely top of the heap technically.
5) Sadly, at the end of the day, democracy is often a way to make very poor decisions.
So -- it seems there is consensus that we need to formalize the governance of numpy. And honestly, I don't think there are any other options that would work at this point with the current community.
And the biggest point: while forking may well demonstrate a complete failure of the governance of a project -- it also represents a success of open-source. In fact, when I read Mathew's history of X11 -- it seems the biggest impediment at each stage were the efforts to make it less that truly open-source.
-Chris
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov wrote:
- I very much agree that governance can make or break a project. However,
the actual governance approach often ends up making less difference than the people involved.
- While the FreeBSD and XFree examples do point to some real problems with
the "core" model it seems that there are many other projects that are using it quite successfully.
- at least in the XFree case, maybe the biggest issue was not the core (or
the board), but the fact that discussions and decision making process were kept secret (and restriction to CVS, EVEN read-only) -- this does not seem to be inherent to the "core" model at all, and not how nupy will even operate.
This was an issue, but not the central one. I would argue that the central problem was that XFree86 had grown up slowly and organically, to find itself at the heart of the Linux ecosystem. At this point the private developers-club aspect of XFree86 became a massive hindrance to the huge pressure from outside to develop new stuff.
- the biggest issue numpy has faced in years is a lack of people that
can/will/do actually contribute code itself. there simply isn't a big group f folks waiting to step up and be the president. And the "leader", president of BDFL, etc, while primarily being a management role, needs to be fully technically competent -- there are very few people qualified at this point, and I suspect none of them want that job. -- in theory, one can manage without the technical competency, but does anyone have a single example of a successful open-source software project run by a good manager that isn't a top technical expert? All the BDFLs I know of are absolutely top of the heap technically.
I can't easily imagine someone without reasonable technical ability wanting to lead the project - although it appears that technical ability is overrated for technical management [1].
- Sadly, at the end of the day, democracy is often a way to make very poor
decisions.
I don't think the question here is whether to have democracy or not, but what kind of democracy. In fact the current proposal is a sort of consensus democracy. My interest is whether this really is a good model for numpy. For example, will this model lead to a lack of accountability and coherent direction? Will it make it harder for other projects to interact with numpy and negotiate about numpy direction?
So -- it seems there is consensus that we need to formalize the governance of numpy. And honestly, I don't think there are any other options that would work at this point with the current community.
It is true, that this discussion needs some commitment. That either has to come from {Chuck, Nathaniel, Ralf} or very strongly from the rest of us, and we don't have either of those conditions, so I think we are going to fall back to the status quo by default.
Cheers,
Matthew
[1] http://www.nytimes.com/2011/03/13/business/13hire.html?_r=0
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov wrote:
- I very much agree that governance can make or break a project. However,
the actual governance approach often ends up making less difference than the people involved.
- While the FreeBSD and XFree examples do point to some real problems with
the "core" model it seems that there are many other projects that are using it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Cheers,
Matthew
[1] http://julipedia.meroh.net/2013/06/self-interview-after-leaving-netbsd.html [2] http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html
On 2015/09/04 10:53 AM, Matthew Brett wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov wrote:
- I very much agree that governance can make or break a project. However,
the actual governance approach often ends up making less difference than the people involved.
- While the FreeBSD and XFree examples do point to some real problems with
the "core" model it seems that there are many other projects that are using it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
This is consistent with Chris's first point.
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person?
Eric
Cheers,
Matthew
[1] http://julipedia.meroh.net/2013/06/self-interview-after-leaving-netbsd.html [2] http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing efiring@hawaii.edu wrote:
On 2015/09/04 10:53 AM, Matthew Brett wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov wrote:
- I very much agree that governance can make or break a project. However,
the actual governance approach often ends up making less difference than the people involved.
- While the FreeBSD and XFree examples do point to some real problems with
the "core" model it seems that there are many other projects that are using it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
This is consistent with Chris's first point.
Do you mean Chris' point that "I very much agree that governance can make or break a project"? Charles Hannum's complaints about NetBSD are very specific in blaming the model rather than the people. I think the XFree86 story supports the same conclusion - that the governance model caused a sense of diffused responsibility that lead to bad decisions and lack of direction.
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person?
That's a good question, and the answer is that in the current situation (zero interest in this discussion from the three current members of the numpy leadership team) - no reasonable person would be interested in that job. That's the situation we're in, and so we have to accept that nothing is going to change, with the consequences that implies. If the situation were different, and we had the interest or commitment to explore this problem, then I guess we could discuss other options including the one I suggested further up the thread.
Cheers,
Matthew
On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing efiring@hawaii.edu wrote:
On 2015/09/04 10:53 AM, Matthew Brett wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com
wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov
wrote:
- I very much agree that governance can make or break a project.
However,
the actual governance approach often ends up making less difference
than the
people involved.
- While the FreeBSD and XFree examples do point to some real
problems with
the "core" model it seems that there are many other projects that are
using
it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
This is consistent with Chris's first point.
Do you mean Chris' point that "I very much agree that governance can make or break a project"? Charles Hannum's complaints about NetBSD are very specific in blaming the model rather than the people. I think the XFree86 story supports the same conclusion - that the governance model caused a sense of diffused responsibility that lead to bad decisions and lack of direction.
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person?
That's a good question, and the answer is that in the current situation (zero interest in this discussion from the three current members of the numpy leadership team) - no reasonable person would be interested in that job. That's the situation we're in, and so we have to accept that nothing is going to change, with the consequences that implies. If the situation were different, and we had the interest or commitment to explore this problem, then I guess we could discuss other options including the one I suggested further up the thread.
"
Today, the project is run by a different cabal. This is the result of a coup that took place in 2000-2001, in which The NetBSD Foundation was taken over by a fraudulent change of the board of directors. (Note: It's probably too late for me to pursue any legal remedy for this, unfortunately.) Although "The NetBSD Project" and "The NetBSD Foundation" were intended from the start to be separate entities -- the latter supplying support infrastructure for the former -- this distinction has been actively blurred since, so that the current "board" of TNF has rather tight control over many aspects of TNP.
"
"
The existing NetBSD Foundation must be disbanded, and replaced with an organization that fulfills its original purpose: to merely handle administrative issues, and not to manage day-to-day affairs.
"
It doesn't sound to me like a developer and community driven governance structure to me.
Cheers
Josef https://jeb2016.com/?lang=es
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sat, Sep 5, 2015 at 12:04 AM, josef.pktd@gmail.com wrote:
On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing efiring@hawaii.edu wrote:
On 2015/09/04 10:53 AM, Matthew Brett wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov wrote:
- I very much agree that governance can make or break a project.
However, the actual governance approach often ends up making less difference than the people involved.
- While the FreeBSD and XFree examples do point to some real
problems with the "core" model it seems that there are many other projects that are using it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
This is consistent with Chris's first point.
Do you mean Chris' point that "I very much agree that governance can make or break a project"? Charles Hannum's complaints about NetBSD are very specific in blaming the model rather than the people. I think the XFree86 story supports the same conclusion - that the governance model caused a sense of diffused responsibility that lead to bad decisions and lack of direction.
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person?
That's a good question, and the answer is that in the current situation (zero interest in this discussion from the three current members of the numpy leadership team) - no reasonable person would be interested in that job. That's the situation we're in, and so we have to accept that nothing is going to change, with the consequences that implies. If the situation were different, and we had the interest or commitment to explore this problem, then I guess we could discuss other options including the one I suggested further up the thread.
"
Today, the project is run by a different cabal. This is the result of a coup that took place in 2000-2001, in which The NetBSD Foundation was taken over by a fraudulent change of the board of directors. (Note: It's probably too late for me to pursue any legal remedy for this, unfortunately.) Although "The NetBSD Project" and "The NetBSD Foundation" were intended from the start to be separate entities -- the latter supplying support infrastructure for the former -- this distinction has been actively blurred since, so that the current "board" of TNF has rather tight control over many aspects of TNP.
"
"
The existing NetBSD Foundation must be disbanded, and replaced with an organization that fulfills its original purpose: to merely handle administrative issues, and not to manage day-to-day affairs.
"
It doesn't sound to me like a developer and community driven governance structure to me.
I think that's a separate issue - the distinction between the 'board' and the 'core'. It would be great if the 'core' concept was fine as long as there is no 'board' but I think that's a hard argument to make.
Cheers,
Matthew
On Fri, Sep 4, 2015 at 7:19 PM, Matthew Brett matthew.brett@gmail.com wrote:
On Sat, Sep 5, 2015 at 12:04 AM, josef.pktd@gmail.com wrote:
On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing efiring@hawaii.edu
wrote:
On 2015/09/04 10:53 AM, Matthew Brett wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett <
matthew.brett@gmail.com>
wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker <chris.barker@noaa.gov
wrote: > 1) I very much agree that governance can make or break a project. > However, > the actual governance approach often ends up making less difference > than the > people involved. > > 2) While the FreeBSD and XFree examples do point to some real > problems with > the "core" model it seems that there are many other projects that
are
> using > it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional
because
they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the
problems
with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
This is consistent with Chris's first point.
Do you mean Chris' point that "I very much agree that governance can make or break a project"? Charles Hannum's complaints about NetBSD are very specific in blaming the model rather than the people. I think the XFree86 story supports the same conclusion - that the governance model caused a sense of diffused responsibility that lead to bad decisions and lack of direction.
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person?
That's a good question, and the answer is that in the current situation (zero interest in this discussion from the three current members of the numpy leadership team) - no reasonable person would be interested in that job. That's the situation we're in, and so we have to accept that nothing is going to change, with the consequences that implies. If the situation were different, and we had the interest or commitment to explore this problem, then I guess we could discuss other options including the one I suggested further up the thread.
"
Today, the project is run by a different cabal. This is the result of a coup that took place in 2000-2001, in which The NetBSD Foundation was taken over by a fraudulent change of the board of directors. (Note: It's probably too late for me to pursue any legal remedy for this, unfortunately.) Although "The NetBSD Project" and "The NetBSD Foundation" were intended from the start to be separate entities -- the latter supplying support infrastructure for the former -- this distinction has been actively blurred since, so that the current "board" of TNF has rather tight control over many aspects of TNP.
"
"
The existing NetBSD Foundation must be disbanded, and replaced with an organization that fulfills its original purpose: to merely handle administrative issues, and not to manage day-to-day affairs.
"
It doesn't sound to me like a developer and community driven governance structure to me.
I think that's a separate issue - the distinction between the 'board' and the 'core'. It would be great if the 'core' concept was fine as long as there is no 'board' but I think that's a hard argument to make.
there is an "esprit de corps" pronounced "esprit de core" but not an "esprit de board"
I trust the core developers, but not ...
But maybe I don't understand some definitions
"
The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions.
"
I thought that's what the "core" group is.
Josef
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sat, Sep 5, 2015 at 12:47 AM, josef.pktd@gmail.com wrote:
On Fri, Sep 4, 2015 at 7:19 PM, Matthew Brett matthew.brett@gmail.com wrote:
On Sat, Sep 5, 2015 at 12:04 AM, josef.pktd@gmail.com wrote:
On Fri, Sep 4, 2015 at 5:55 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Sep 4, 2015 at 10:22 PM, Eric Firing efiring@hawaii.edu wrote:
On 2015/09/04 10:53 AM, Matthew Brett wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com wrote: > Hi, > > On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker > chris.barker@noaa.gov > wrote: >> 1) I very much agree that governance can make or break a project. >> However, >> the actual governance approach often ends up making less >> difference >> than the >> people involved. >> >> 2) While the FreeBSD and XFree examples do point to some real >> problems with >> the "core" model it seems that there are many other projects that >> are >> using >> it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
This is consistent with Chris's first point.
Do you mean Chris' point that "I very much agree that governance can make or break a project"? Charles Hannum's complaints about NetBSD are very specific in blaming the model rather than the people. I think the XFree86 story supports the same conclusion - that the governance model caused a sense of diffused responsibility that lead to bad decisions and lack of direction.
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
Do you have suggestions as to who would make a good numpy president or BDFL and potentially has the time and inclination to do it, or how to identify and recruit such a person?
That's a good question, and the answer is that in the current situation (zero interest in this discussion from the three current members of the numpy leadership team) - no reasonable person would be interested in that job. That's the situation we're in, and so we have to accept that nothing is going to change, with the consequences that implies. If the situation were different, and we had the interest or commitment to explore this problem, then I guess we could discuss other options including the one I suggested further up the thread.
"
Today, the project is run by a different cabal. This is the result of a coup that took place in 2000-2001, in which The NetBSD Foundation was taken over by a fraudulent change of the board of directors. (Note: It's probably too late for me to pursue any legal remedy for this, unfortunately.) Although "The NetBSD Project" and "The NetBSD Foundation" were intended from the start to be separate entities -- the latter supplying support infrastructure for the former -- this distinction has been actively blurred since, so that the current "board" of TNF has rather tight control over many aspects of TNP.
"
"
The existing NetBSD Foundation must be disbanded, and replaced with an organization that fulfills its original purpose: to merely handle administrative issues, and not to manage day-to-day affairs.
"
It doesn't sound to me like a developer and community driven governance structure to me.
I think that's a separate issue - the distinction between the 'board' and the 'core'. It would be great if the 'core' concept was fine as long as there is no 'board' but I think that's a hard argument to make.
there is an "esprit de corps" pronounced "esprit de core" but not an "esprit de board"
I trust the core developers, but not ...
But maybe I don't understand some definitions
"
The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions.
"
I thought that's what the "core" group is.
To avoid repeating myself any more than I have already - see further up this thread : http://mail.scipy.org/pipermail/numpy-discussion/2015-August/073452.html
Matthew
On Fri, Sep 4, 2015 at 1:53 PM, Matthew Brett matthew.brett@gmail.com wrote:
On Fri, Sep 4, 2015 at 2:33 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Wed, Sep 2, 2015 at 5:41 PM, Chris Barker chris.barker@noaa.gov wrote:
- I very much agree that governance can make or break a project. However,
the actual governance approach often ends up making less difference than the people involved.
- While the FreeBSD and XFree examples do point to some real problems with
the "core" model it seems that there are many other projects that are using it quite successfully.
I was just rereading the complaints about the 'core' structure from high-level NetBSD project leaders:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)." [1]
"There is no high-level direction; if you ask "what about the problems with threads" or "will there be a flash-friendly file system", the best you'll get is "we'd love to have both" -- but no work is done to recruit people to code these things, or encourage existing developers to work on them." [2]
I imagine we will have to reconcile ourselves to similar problems, if we adopt the same structures.
I guess I just don't see how you think we can legislate ourselves into having a vision. Like... we'll elect a volunteer to produce a roadmap, and then if they don't we'll fire them (accountability!), leaving us with no volunteer and no roadmap? How will that help?
The Jupyter/IPython project probably has one of the the most developed roadmaps in all of open-source, since their grant-funded development model requires them to actually commit to plans ahead of time. And AFAIK the way they accomplish this has nothing to do with Fernando sitting down and having visions; it involves dragging the "core" team in front of a whiteboard 2x a year for a week, and seeing what they can come up with. And I guess we have some tentative evidence from the other thread that this strategy may work for us too...
I'm not sure how useful this focus on NetBSD is, given the elephant in the room: the fundamental challenge for NetBSD is that NetBSD has no compelling reason to exist; even its old distinctive feature of being the most portable free OS (which is already an intrinsically niche appeal) has been taken over by Linux. (And Linux, notably, has always explicitly refused to have a roadmap [1][2]... it seems to be doing okay.) Nonetheless, looking at the Hannum [3] post in particular, I'm struck by how much it *doesn't* apply to us. In particular, his main two prescriptions aside from "have leadership" are #6 and #7. #6 is that the legal Foundation part of the project should restrict itself to administrative activities and get out of technical decision making; the draft governance document says "[The NumFOCUS] Subcommittee shall NOT make decisions about the direction, scope or technical direction of the Project.". #7 is:
The "core" group must be replaced with people who are actually competent and dedicated enough to review proposals, accept feedback, and make good decisions. More to the point, though, the "core" group must only act when *needed* -- most technical decisions should be left to the community to hash out; it must not preempt the community from developing better solutions. (This is how the "core" group worked during most of the project's growth period.)
So he has nothing at all against the idea of a "core group", he just thinks it should be competent and... basically follow the rules that we attempted to codify in the draft governance document....?
-n
[1] http://www.cnn.com/TECH/computing/9906/21/linus.idg/ [2] https://en.wikipedia.org/wiki/Linux_kernel#Development [3] http://mail-index.netbsd.org/netbsd-users/2006/08/30/0016.html
On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
[...]
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
One reason might be that the specific form of governance can have no influence on the long-term health of the project.
I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project.
If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance.
That seems fair, and I also think that even if we end up not changing anything then there's still some utility in having the discussion. Certainly one of my secret fears in writing all this up is that it would end up being just some words that only I cared about and everyone else ignored, so seeing other people engaging with it is heartening :-).
I'm not feeling inspired this Friday evening to write a full white paper reviewing the universe of possible governance, but I'll summarize a bit what the reasoning was that led us to what we ended up with (at least as I remember it and from my perspective).
Some of the sources that were mentioned for reference and positive or negative inspiration include Karl Fogel's chapter on governance, the Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc ditto, subversion, Jupyter/ipython, the Linux kernel, python, matplotlib, the Apache foundation's processes, the gnome foundation, nodejs, ... I'm certainly forgetting some.
The basic observation that we started from was that all the successful projects seemed to fit a similar template: (1) basically all actual decision making is done by informal and maximally inclusive consensus-based processes, (2) there are some sort of more formal "backstop" rules that are called into play if informal processes break down (which hopefully never even happens). These projects don't tend to look that similar on paper, because people tend to only write down the "backstop" rules, and those vary a lot.
So e.g., for some projects like Python or IPython, the formal rule is "the BDFL decides". But then if you look at what successful BDFLs do with this power, they mostly enforce rule #1 above. (Karl discusses this: http://producingoss.com/en/producingoss.html#benevolent-dictator). E.g., if you watch Guido, I don't think I've ever seen him say "I know people don't like X but we're doing it anyway". He mostly sits back to watch the debate, steps in occasionally to prune off unproductive lines of conversation (e.g. in the PEP 465 debate where he watched for a while, stepped in to tell people to knock it off with the pointless bikeshedding about which character to use, and then went back to watching) or encourage people to keep thinking about something, is always very careful not to actually make any strong statements, etc. Or, e.g., the bit in the draft governance about how the steering committee's secondary job is to make decisions, and their primary job is to prevent things reaching the point where they have to? That's a straight paraphrase of Fernando talking about his BDFL philosophy.
Or on the other end of things, you have e.g. Subversion, which had an elaborate defined governance system with different levels of "core-ness", a voting system, etc. -- and they were 6 years into the project before they had their first vote. (The vote was on the crucial technical decision of whether to write function calls like "f ()" or "f()".) Or e.g. the massive and famously fractious Debian has their CTTE, which theoretically is the final arbiter of technical decisions in Debian and has immense powers... but they issue like ~5 rulings a year on average, and have gone years without issuing a ruling at all.
So, our thought process was: for our purposes, since we don't have a BDFL, we can't depend on them to be the unwritten mechanism for enforcing rule #1, so first we should write down rule #1. And then we should attach some kind of workable backstop procedure. "Consensus among a core group" seemed like a good bet. It does mean you need to have a formal rule for defining who is in the "core group", because the point of having a formal backstop rule is that it be... formal... but otherwise it's about as simple as you can get: it requires minimal voting logistics (it's trivial to tell whether you have consensus, because if someone disagrees, there's your answer), it's minimally gameable (no temptation to form factions), and because in practice doing anything more complicated than consensus would first require consensus to agree on the more complicated system, so it's a safe procedure to agree on now and try to improve later if we have problems. But we don't really expect to have problems, since our goal is to never even use this system, so spending lots of effort imagining hypothetical scenarios didn't seem like a great use of time.
And then we took the Jupyter/IPython governance document, ripped out the BDFL-dependent parts, replaced them with the two ideas above, and called it a day.
This is an interesting article I came across today, about how "merely" documenting the status quo provides non-trivial benefits: https://kateheddleston.com/blog/the-null-process
(Not that I would necessarily say that the draft I sent is just formalizing the status quo -- IMO the biggest problem with the status quo is that no-one is quite sure what it even is! Which leads to things like that time people panicked because I said we should consider all our options before making a decision [1], or the frustrating and unproductive tendency of contentious technical debates to switch over into becoming ad-hoc one-off debates about decision making processes...)
-n
[1] http://mail.scipy.org/pipermail/numpy-discussion/2015-April/072629.html
Hi,
Thanks for replying, that is helpful.
I'm going to write carefully, because I suspect we've only got this one shot at getting it right.
I first wanted to come back to an earlier email of yours:
On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith njs@pobox.com wrote:
At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama, (b) when some amount of interpersonal drama does inevitably arise anyway, provide some regulated channel for it, hopefully one that leads to a drama sink.
I think this is the wrong way to think of the problem. There is no drama when:
* everyone agrees (problem is trivial); * no one cares (problem is not important); * everyone who does care has come to the conclusion that it is not worth arguing.
The opposite of drama is not efficiency, but boredom and frustration.
As evidence, vigorous, substantial and bad-tempered debates are characteristic of successful companies:
""" "They were chaos. We would stay there for hours, ironing out the issues, until we came to something ... At times the meetings would get so violent that people almost went across the table at each other.... People yelled. They waved their arms around and pounded on tables. Faces would get red and veins bulged out". """ [1]
I'm sure you have hung out with the IPython team as I have. You will then hear their description of how much the team disagree with each other, and often, how important that disagreement is, to finding the right answer.
On the other hand, I don't think it is our culture in numpy to value that kind of debate. That brings us to my next comment:
On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith njs@pobox.com wrote:
On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
[...]
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
One reason might be that the specific form of governance can have no influence on the long-term health of the project.
I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project.
If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance.
That seems fair, and I also think that even if we end up not changing anything then there's still some utility in having the discussion. Certainly one of my secret fears in writing all this up is that it would end up being just some words that only I cared about and everyone else ignored, so seeing other people engaging with it is heartening :-).
I'm not feeling inspired this Friday evening to write a full white paper reviewing the universe of possible governance, but I'll summarize a bit what the reasoning was that led us to what we ended up with (at least as I remember it and from my perspective).
Some of the sources that were mentioned for reference and positive or negative inspiration include Karl Fogel's chapter on governance, the Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc ditto, subversion, Jupyter/ipython, the Linux kernel, python, matplotlib, the Apache foundation's processes, the gnome foundation, nodejs, ... I'm certainly forgetting some.
The basic observation that we started from was that all the successful projects seemed to fit a similar template: (1) basically all actual decision making is done by informal and maximally inclusive consensus-based processes, (2) there are some sort of more formal "backstop" rules that are called into play if informal processes break down (which hopefully never even happens). These projects don't tend to look that similar on paper, because people tend to only write down the "backstop" rules, and those vary a lot.
So e.g., for some projects like Python or IPython, the formal rule is "the BDFL decides". But then if you look at what successful BDFLs do with this power, they mostly enforce rule #1 above. (Karl discusses this: http://producingoss.com/en/producingoss.html#benevolent-dictator). E.g., if you watch Guido, I don't think I've ever seen him say "I know people don't like X but we're doing it anyway". He mostly sits back to watch the debate, steps in occasionally to prune off unproductive lines of conversation (e.g. in the PEP 465 debate where he watched for a while, stepped in to tell people to knock it off with the pointless bikeshedding about which character to use, and then went back to watching) or encourage people to keep thinking about something, is always very careful not to actually make any strong statements, etc. Or, e.g., the bit in the draft governance about how the steering committee's secondary job is to make decisions, and their primary job is to prevent things reaching the point where they have to? That's a straight paraphrase of Fernando talking about his BDFL philosophy.
Or on the other end of things, you have e.g. Subversion, which had an elaborate defined governance system with different levels of "core-ness", a voting system, etc. -- and they were 6 years into the project before they had their first vote. (The vote was on the crucial technical decision of whether to write function calls like "f ()" or "f()".) Or e.g. the massive and famously fractious Debian has their CTTE, which theoretically is the final arbiter of technical decisions in Debian and has immense powers... but they issue like ~5 rulings a year on average, and have gone years without issuing a ruling at all.
So, our thought process was: for our purposes, since we don't have a BDFL, we can't depend on them to be the unwritten mechanism for enforcing rule #1, so first we should write down rule #1.
What you are describing in the previous paragraph seems to be a well-known feature of successful leaders. The CEOs of highly successful companies described in [2] differed from less-successful comparators in encouraging debate and acting as the final arbiter for the decisions of the group.
I don't think you are right that you can create that effect without having such a leader. The point about the leader is that they are a trusted referee and mentor to the group. They provide stability and space for the debate. For example, in my little church, there is an elected position called the 'moderator' with that purpose. Most groups recognize that someone has to do that job, and that the group cannot do that themselves.
On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith njs@pobox.com wrote:
I guess I just don't see how you think we can legislate ourselves into having a vision. Like... we'll elect a volunteer to produce a roadmap, and then if they don't we'll fire them (accountability!), leaving us with no volunteer and no roadmap? How will that help?
I agree it is not sensible for the president or moderator to produce a roadmap, but a roadmap is different from a vision.
We often run into trouble on the list, discussing whether to make backwards-incompatible changes. The reason for the trouble is that different people have different ideas about what numpy is. Some people think numpy is a legacy library who's job it is to tread water while waiting for a better alternative. For them, backwards compatibility is of overriding importance. Others think that numpy should be gradually refactored into a better and more extensible version that will continue to be the backbone of scientific computing into the fairly distant future. These are two different visions that are not compatible one with another. Trying to keep both groups happy is likely to lead to dissatisfaction and frustration from both sides.
On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith njs@pobox.com wrote:
I'm not sure how useful this focus on NetBSD is, given the elephant in the room: the fundamental challenge for NetBSD is that NetBSD has no compelling reason to exist;
That is now true, but I would argue (I think Hannum would argue) that this was the direct effect of very poor management, with no vision for where the project was going. I imagine you'd agree that OpenBSD does have a purpose, and OpenBSD started as a one-person-in-charge fork of NetBSD, where the fork was the result of very bad management by the 'core'.
I hate to quote these again but:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)."
"As things are today, the project leaders will almost always settle for either the conservative solution or a solution that compromises on every detail so as to not annoy anyone. " [3]
A organization that is governed like this, just has to fail, in the long term.
Cheers,
Matthew
[1] 'Exhibit E' in http://nipyworld.blogspot.co.uk/2012/02/what-is-meritocracy-in-open-source.h... [2] https://en.wikipedia.org/wiki/Good_to_Great [3] http://asterisk.dynevor.org/melting-core.html
On Sat, Sep 5, 2015 at 4:20 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
Thanks for replying, that is helpful.
I'm going to write carefully, because I suspect we've only got this one shot at getting it right.
Not sure why? "Change the governance document" is explicitly listed as one of the steering council's powers, so I don't see what we'd lose by adopting what we have now.
I first wanted to come back to an earlier email of yours:
On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith njs@pobox.com wrote:
At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama, (b) when some amount of interpersonal drama does inevitably arise anyway, provide some regulated channel for it, hopefully one that leads to a drama sink.
I think this is the wrong way to think of the problem. There is no drama when:
- everyone agrees (problem is trivial);
- no one cares (problem is not important);
- everyone who does care has come to the conclusion that it is not
worth arguing.
The opposite of drama is not efficiency, but boredom and frustration.
As evidence, vigorous, substantial and bad-tempered debates are characteristic of successful companies:
""" "They were chaos. We would stay there for hours, ironing out the issues, until we came to something ... At times the meetings would get so violent that people almost went across the table at each other.... People yelled. They waved their arms around and pounded on tables. Faces would get red and veins bulged out". """ [1]
No, by drama I mean situations where technical issues become personal, and where people end up feeling emotionally hurt and disrespected and so forth. I agree that vigorous debate is important, but strongly disagree that veins bulging and violence are necessary or helpful to success. (This is tangential to your main point, but I wanted to point it out.) Since I happened to be reading Kate Heddleston's blog yesterday, here's another relevant link... https://www.kateheddleston.com/blog/argument-cultures-and-unregulated-aggres...
I'm sure you have hung out with the IPython team as I have. You will then hear their description of how much the team disagree with each other, and often, how important that disagreement is, to finding the right answer.
Indeed, that's... why we have a document that is designed to provide a framework for exactly that sort of debate? I don't really understand the criticism here.
My objection to elections is that they create a whole *extra* source of drama that runs the risk of compromising technical debates. Suddenly people have to campaign and have fraught arguments over who gets to vote and how to run the election and it's just borrowing trouble IMO. (Are you by chance on the PSF members mailing list? If not then ask someone how things have been going there vis-a-vis elections -- I'm not myself, but I've heard stories...) Obviously it's a fallacy to think that we can entirely ignore social issues and focus only on technical problems, but still -- I'd rather we spend our difficult debates on technical issues as much as possible.
Even worse, elections mean that someone might think "Hmm, should I give in on this point, even though I know I'm right, so as to avoid annoying this person who might otherwise retaliate by telling their friends not to vote for me next year?" Or "hmm, did they give in on that point because they actually agree or because the person they work for wants the person they're agreeing with to win the next election?" Or etc. Hopefully no-one would actually start playing politics to that degree, but IMHO it's poisonous to even set up a situation where people are tempted to perform such calculations.
On the other hand, I don't think it is our culture in numpy to value that kind of debate. That brings us to my next comment:
On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith njs@pobox.com wrote:
On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
[...]
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
One reason might be that the specific form of governance can have no influence on the long-term health of the project.
I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project.
If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance.
That seems fair, and I also think that even if we end up not changing anything then there's still some utility in having the discussion. Certainly one of my secret fears in writing all this up is that it would end up being just some words that only I cared about and everyone else ignored, so seeing other people engaging with it is heartening :-).
I'm not feeling inspired this Friday evening to write a full white paper reviewing the universe of possible governance, but I'll summarize a bit what the reasoning was that led us to what we ended up with (at least as I remember it and from my perspective).
Some of the sources that were mentioned for reference and positive or negative inspiration include Karl Fogel's chapter on governance, the Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc ditto, subversion, Jupyter/ipython, the Linux kernel, python, matplotlib, the Apache foundation's processes, the gnome foundation, nodejs, ... I'm certainly forgetting some.
The basic observation that we started from was that all the successful projects seemed to fit a similar template: (1) basically all actual decision making is done by informal and maximally inclusive consensus-based processes, (2) there are some sort of more formal "backstop" rules that are called into play if informal processes break down (which hopefully never even happens). These projects don't tend to look that similar on paper, because people tend to only write down the "backstop" rules, and those vary a lot.
So e.g., for some projects like Python or IPython, the formal rule is "the BDFL decides". But then if you look at what successful BDFLs do with this power, they mostly enforce rule #1 above. (Karl discusses this: http://producingoss.com/en/producingoss.html#benevolent-dictator). E.g., if you watch Guido, I don't think I've ever seen him say "I know people don't like X but we're doing it anyway". He mostly sits back to watch the debate, steps in occasionally to prune off unproductive lines of conversation (e.g. in the PEP 465 debate where he watched for a while, stepped in to tell people to knock it off with the pointless bikeshedding about which character to use, and then went back to watching) or encourage people to keep thinking about something, is always very careful not to actually make any strong statements, etc. Or, e.g., the bit in the draft governance about how the steering committee's secondary job is to make decisions, and their primary job is to prevent things reaching the point where they have to? That's a straight paraphrase of Fernando talking about his BDFL philosophy.
Or on the other end of things, you have e.g. Subversion, which had an elaborate defined governance system with different levels of "core-ness", a voting system, etc. -- and they were 6 years into the project before they had their first vote. (The vote was on the crucial technical decision of whether to write function calls like "f ()" or "f()".) Or e.g. the massive and famously fractious Debian has their CTTE, which theoretically is the final arbiter of technical decisions in Debian and has immense powers... but they issue like ~5 rulings a year on average, and have gone years without issuing a ruling at all.
So, our thought process was: for our purposes, since we don't have a BDFL, we can't depend on them to be the unwritten mechanism for enforcing rule #1, so first we should write down rule #1.
What you are describing in the previous paragraph seems to be a well-known feature of successful leaders. The CEOs of highly successful companies described in [2] differed from less-successful comparators in encouraging debate and acting as the final arbiter for the decisions of the group.
I don't think you are right that you can create that effect without having such a leader. The point about the leader is that they are a trusted referee and mentor to the group. They provide stability and space for the debate. For example, in my little church, there is an elected position called the 'moderator' with that purpose. Most groups recognize that someone has to do that job, and that the group cannot do that themselves.
Having a good moderator is certainly a wonderful thing if you can manage it, but it's just not true that "Most groups recognize [...] the group cannot do that themselves". FOSS projects with a formal leader are in the minority. And there's no particular need for a specific person to be designated to always do the job: in a consensus-based framework, pretty much anyone has the power to step up to act as an arbitrator or to protect a minority opinion.
The main requirement for someone to be a successful arbitrator is that the participants trust and respect their judgement as a neutral party. If you don't have such a person, then an election won't create that that trust and respect. If you do have such a person -- or multiple such people! -- then an election is unnecessary.
On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith njs@pobox.com wrote:
I guess I just don't see how you think we can legislate ourselves into having a vision. Like... we'll elect a volunteer to produce a roadmap, and then if they don't we'll fire them (accountability!), leaving us with no volunteer and no roadmap? How will that help?
I agree it is not sensible for the president or moderator to produce a roadmap, but a roadmap is different from a vision.
We often run into trouble on the list, discussing whether to make backwards-incompatible changes. The reason for the trouble is that different people have different ideas about what numpy is. Some people think numpy is a legacy library who's job it is to tread water while waiting for a better alternative. For them, backwards compatibility is of overriding importance. Others think that numpy should be gradually refactored into a better and more extensible version that will continue to be the backbone of scientific computing into the fairly distant future. These are two different visions that are not compatible one with another. Trying to keep both groups happy is likely to lead to dissatisfaction and frustration from both sides.
Yep. And like you said above re: IPython and so forth, the solution is to confront that problem head-on and have the debate. Having a dictator come in and dictate the outcome wouldn't help anything.
On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith njs@pobox.com wrote:
I'm not sure how useful this focus on NetBSD is, given the elephant in the room: the fundamental challenge for NetBSD is that NetBSD has no compelling reason to exist;
That is now true, but I would argue (I think Hannum would argue) that this was the direct effect of very poor management, with no vision for where the project was going. I imagine you'd agree that OpenBSD does have a purpose, and OpenBSD started as a one-person-in-charge fork of NetBSD, where the fork was the result of very bad management by the 'core'.
I think you might be trying too hard to make history line up with your conclusions here :-/. Theo de Raadt didn't quit; he got kicked out in 1994 because they felt his "long history of rudeness towards and abuse of users and developers" finally outweighed his "long history of positive contributions":
https://mail-index.netbsd.org/netbsd-users/1994/12/23/0000.html
Obviously I don't know the whole sordid backstory, but "productive + rude + abusive" still pretty much characterizes his reputation today. And in general making this kind of difficult decision is exactly what I would look for as a sign of *good* management. And isn't 1994 well within the period where you and Hannum are both arguing that the NetBSD 'core' was still functional and healthy?
Anyway, who knows why NetBSD failed. It's surely some complex and overdetermined combination of their governance structure, the license (Linux's use of the GPL has often been cited as a crucial factor in its becoming "the standard free kernel", b/c it created a level playing field for otherwise competing companies to collaborate and this set off the feedback cycle where it's the obvious one to use -> so people use it and contribute back -> so it's even more obviously the one to use -> ...), the particular individuals who happened to attach themselves to NetBSD vs. the other BSDs vs. Linux, the different approaches to handling userspace (developed alongside the kernel vs. developed separately), and so on and so forth. I just don't think these cherrypicked messages about one project are terribly compelling as a data point.
I hate to quote these again but:
"[the "core" and "board of directors"] teams are dysfunctional because they do not provide leadership: all they do is act reactively to requests from users and/or to resolve internal disputes. In other words: there is no initiative nor vision emerging from these teams (and, for that matter, from anybody)."
"As things are today, the project leaders will almost always settle for either the conservative solution or a solution that compromises on every detail so as to not annoy anyone. " [3]
A organization that is governed like this, just has to fail, in the long term.
Sure, but I don't understand why you conclude that these are inevitable consequences of any organization where more developers are "more central" than others without a single strong leader. You're quoting Hannum, but Hannum doesn't think that - his prescription is to get better a better core team. Linux itself -- the project that ate NetBSD's lunch -- doesn't have a core leader providing vision and initiative, and it seems to be doing okay. (They have Linus obviously, but he mostly acts as a moderator -- he definitely does not tell anyone that they need to go write a better filesystem or go invent cgroups or perf or whatever. That's all bottom-up.)
Leadership is something that people do, not something people are. You can't legislate a vision into existence any more than you can legislate trust into existence. You can try to create the conditions for it to happen, like by getting people into the same room and providing a clear path for newcomers to propose ideas and so forth, but to me elections seem like exactly the wrong approach.
Or instead of debating how to create a vision in the abstract, we could shift our energy over to the other thread that's debating what the vision should actually be :-).
-n
On Sat, Sep 5, 2015 at 7:14 PM, Nathaniel Smith njs@pobox.com wrote:
On Sat, Sep 5, 2015 at 4:20 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
Thanks for replying, that is helpful.
I'm going to write carefully, because I suspect we've only got this one shot at getting it right.
Not sure why? "Change the governance document" is explicitly listed as one of the steering council's powers, so I don't see what we'd lose by adopting what we have now.
In practice, I doubt that there will be any proposal to change the governance in the medium term, especially if the governance does turn out to be risk averse and conservative.
I first wanted to come back to an earlier email of yours:
On Thu, Aug 27, 2015 at 3:05 AM, Nathaniel Smith njs@pobox.com wrote:
At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama, (b) when some amount of interpersonal drama does inevitably arise anyway, provide some regulated channel for it, hopefully one that leads to a drama sink.
I think this is the wrong way to think of the problem. There is no drama when:
- everyone agrees (problem is trivial);
- no one cares (problem is not important);
- everyone who does care has come to the conclusion that it is not
worth arguing.
The opposite of drama is not efficiency, but boredom and frustration.
As evidence, vigorous, substantial and bad-tempered debates are characteristic of successful companies:
""" "They were chaos. We would stay there for hours, ironing out the issues, until we came to something ... At times the meetings would get so violent that people almost went across the table at each other.... People yelled. They waved their arms around and pounded on tables. Faces would get red and veins bulged out". """ [1]
No, by drama I mean situations where technical issues become personal, and where people end up feeling emotionally hurt and disrespected and so forth. I agree that vigorous debate is important, but strongly disagree that veins bulging and violence are necessary or helpful to success. (This is tangential to your main point, but I wanted to point it out.) Since I happened to be reading Kate Heddleston's blog yesterday, here's another relevant link... https://www.kateheddleston.com/blog/argument-cultures-and-unregulated-aggres...
The distinction is between a culture that welcomes expressions of disagreement with one that does not. In practice, if a group is easily alarmed by expressions of strong feeling, most people will be careful about expressing disagreement. This is of course a rather subtle point. It is easy to say that we do welcome disagreement, when in practice what happens is that the person disagreeing will in fact be labeled as being impolite.
then hear their description of how much the team disagree with each other, and often, how important that disagreement is, to finding the right answer.
Indeed, that's... why we have a document that is designed to provide a framework for exactly that sort of debate? I don't really understand the criticism here.
Er - I was arguing that, by attempting to formalize a system that doesn't have a leader, that will lead to a culture where debate is more difficult and less effective.
My objection to elections is that they create a whole *extra* source of drama that runs the risk of compromising technical debates. Suddenly people have to campaign and have fraught arguments over who gets to vote and how to run the election and it's just borrowing trouble IMO. (Are you by chance on the PSF members mailing list? If not then ask someone how things have been going there vis-a-vis elections -- I'm not myself, but I've heard stories...) Obviously it's a fallacy to think that we can entirely ignore social issues and focus only on technical problems, but still -- I'd rather we spend our difficult debates on technical issues as much as possible.
Even worse, elections mean that someone might think "Hmm, should I give in on this point, even though I know I'm right, so as to avoid annoying this person who might otherwise retaliate by telling their friends not to vote for me next year?" Or "hmm, did they give in on that point because they actually agree or because the person they work for wants the person they're agreeing with to win the next election?" Or etc. Hopefully no-one would actually start playing politics to that degree, but IMHO it's poisonous to even set up a situation where people are tempted to perform such calculations.
I suppose these things might happen. The benefit of elections is that it forces us to be explicit about what the issues are, and what is at stake. It seems to me that is something that numpy has to do, and soon, whatever the mechanism.
On the other hand, I don't think it is our culture in numpy to value that kind of debate. That brings us to my next comment:
On Fri, Sep 4, 2015 at 8:35 PM, Nathaniel Smith njs@pobox.com wrote:
On Fri, Aug 28, 2015 at 1:46 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Fri, Aug 28, 2015 at 5:59 AM, Jaime Fernández del Río jaime.frio@gmail.com wrote:
[...]
<joke> Are you trying to prove the point that consensus doesn't work by making it impossible to reach a consensus on this? ;-) </joke>
Forgive me if I use this joke to see if I can get us any further.
If this was code, I think this joke would not be funny, because we wouldn't expect to reach consensus without considering all the options, and discussing their pros and cons.
Why would that not be useful in the case of forms of governance?
One reason might be that the specific form of governance can have no influence on the long-term health of the project.
I am convinced that that is wrong - that the form of governance has a large influence on the long-term health of a project.
If there is some possibility that this is true, then it seems to me that we would be foolish not to try and come to some reasoned choice about the form of governance.
That seems fair, and I also think that even if we end up not changing anything then there's still some utility in having the discussion. Certainly one of my secret fears in writing all this up is that it would end up being just some words that only I cared about and everyone else ignored, so seeing other people engaging with it is heartening :-).
I'm not feeling inspired this Friday evening to write a full white paper reviewing the universe of possible governance, but I'll summarize a bit what the reasoning was that led us to what we ended up with (at least as I remember it and from my perspective).
Some of the sources that were mentioned for reference and positive or negative inspiration include Karl Fogel's chapter on governance, the Debian constitution, the gcc/egcs fork+reverse-fork, glibc/eglibc ditto, subversion, Jupyter/ipython, the Linux kernel, python, matplotlib, the Apache foundation's processes, the gnome foundation, nodejs, ... I'm certainly forgetting some.
The basic observation that we started from was that all the successful projects seemed to fit a similar template: (1) basically all actual decision making is done by informal and maximally inclusive consensus-based processes, (2) there are some sort of more formal "backstop" rules that are called into play if informal processes break down (which hopefully never even happens). These projects don't tend to look that similar on paper, because people tend to only write down the "backstop" rules, and those vary a lot.
So e.g., for some projects like Python or IPython, the formal rule is "the BDFL decides". But then if you look at what successful BDFLs do with this power, they mostly enforce rule #1 above. (Karl discusses this: http://producingoss.com/en/producingoss.html#benevolent-dictator). E.g., if you watch Guido, I don't think I've ever seen him say "I know people don't like X but we're doing it anyway". He mostly sits back to watch the debate, steps in occasionally to prune off unproductive lines of conversation (e.g. in the PEP 465 debate where he watched for a while, stepped in to tell people to knock it off with the pointless bikeshedding about which character to use, and then went back to watching) or encourage people to keep thinking about something, is always very careful not to actually make any strong statements, etc. Or, e.g., the bit in the draft governance about how the steering committee's secondary job is to make decisions, and their primary job is to prevent things reaching the point where they have to? That's a straight paraphrase of Fernando talking about his BDFL philosophy.
Or on the other end of things, you have e.g. Subversion, which had an elaborate defined governance system with different levels of "core-ness", a voting system, etc. -- and they were 6 years into the project before they had their first vote. (The vote was on the crucial technical decision of whether to write function calls like "f ()" or "f()".) Or e.g. the massive and famously fractious Debian has their CTTE, which theoretically is the final arbiter of technical decisions in Debian and has immense powers... but they issue like ~5 rulings a year on average, and have gone years without issuing a ruling at all.
So, our thought process was: for our purposes, since we don't have a BDFL, we can't depend on them to be the unwritten mechanism for enforcing rule #1, so first we should write down rule #1.
What you are describing in the previous paragraph seems to be a well-known feature of successful leaders. The CEOs of highly successful companies described in [2] differed from less-successful comparators in encouraging debate and acting as the final arbiter for the decisions of the group.
I don't think you are right that you can create that effect without having such a leader. The point about the leader is that they are a trusted referee and mentor to the group. They provide stability and space for the debate. For example, in my little church, there is an elected position called the 'moderator' with that purpose. Most groups recognize that someone has to do that job, and that the group cannot do that themselves.
Having a good moderator is certainly a wonderful thing if you can manage it, but it's just not true that "Most groups recognize [...] the group cannot do that themselves". FOSS projects with a formal leader are in the minority.
I believe you - I was referring to groups like businesses, universities, governments, churches and so on.
And there's no particular need for a specific person to be designated to always do the job: in a consensus-based framework, pretty much anyone has the power to step up to act as an arbitrator or to protect a minority opinion.
This is of course the ideal, but I think that just doesn't work in practice. When tempers fray and people feel strongly, they will tend not to accept arbitration from any of the other active participants, or a self-designated outsider.
On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith njs@pobox.com wrote:
I guess I just don't see how you think we can legislate ourselves into having a vision. Like... we'll elect a volunteer to produce a roadmap, and then if they don't we'll fire them (accountability!), leaving us with no volunteer and no roadmap? How will that help?
I agree it is not sensible for the president or moderator to produce a roadmap, but a roadmap is different from a vision.
We often run into trouble on the list, discussing whether to make backwards-incompatible changes. The reason for the trouble is that different people have different ideas about what numpy is. Some people think numpy is a legacy library who's job it is to tread water while waiting for a better alternative. For them, backwards compatibility is of overriding importance. Others think that numpy should be gradually refactored into a better and more extensible version that will continue to be the backbone of scientific computing into the fairly distant future. These are two different visions that are not compatible one with another. Trying to keep both groups happy is likely to lead to dissatisfaction and frustration from both sides.
Yep. And like you said above re: IPython and so forth, the solution is to confront that problem head-on and have the debate. Having a dictator come in and dictate the outcome wouldn't help anything.
Yes, I see I am not being clear. I was claiming that it is difficult to have this debate without someone who will make the final call. That person doesn't 'dictate' but they have the final say when all the arguments are in. If I am right, then, with the no-leader consensus model you propose, it will be hard to have that debate because it is harder to get to the final decision.
On Sat, Sep 5, 2015 at 12:46 AM, Nathaniel Smith njs@pobox.com wrote:
I'm not sure how useful this focus on NetBSD is, given the elephant in the room: the fundamental challenge for NetBSD is that NetBSD has no compelling reason to exist;
That is now true, but I would argue (I think Hannum would argue) that this was the direct effect of very poor management, with no vision for where the project was going. I imagine you'd agree that OpenBSD does have a purpose, and OpenBSD started as a one-person-in-charge fork of NetBSD, where the fork was the result of very bad management by the 'core'.
I think you might be trying too hard to make history line up with your conclusions here :-/. Theo de Raadt didn't quit; he got kicked out in 1994 because they felt his "long history of rudeness towards and abuse of users and developers" finally outweighed his "long history of positive contributions":
I know that history very well, sorry that I did not unpack what I meant. I believe that the 'core' group made a terrible social mess of trying to deal with TdR's interaction style, in a characteristic 'core' way, which was that they ended up throwing him out with an officious email where no one member of the core prepared to accept the responsibility for doing that, and with no way back once they had done that. I would also argue that the 'core' way tends to a rather conservative and joyless social model that made it particularly difficult for them to deal with TdR.
But anyway - that was not my point. My point was that, in 1994, as NetBSD was losing direction, it was perfectly possible to fork it and take it in a new and fruitful direction, with the right governance and leader. Despite his later prescription, Hannum is arguing that the core structure, specifically, was at fault for preventing that happening in NetBSD and in other projects:
""" Much of this early structure (CVS, web site, cabal ["core" group], etc.) was copied verbatim by other open source (this term not being in wide use yet) projects -- even the form of the project name and the term "core". This later became a kind of standard template for starting up an open source project. [...] I'm sorry to say that I helped create this problem, and that most of the projects which modeled themselves after NetBSD (probably due to its high popularity in 1993 and 1994) have suffered similar problems. FreeBSD and XFree86, for example, have both forked successor projects (Dragonfly and X.org) for very similar reasons. """ [1[
Cheers,
Matthew
On Sat, Sep 5, 2015 at 8:08 PM, Matthew Brett matthew.brett@gmail.com wrote:
On Sat, Sep 5, 2015 at 7:14 PM, Nathaniel Smith njs@pobox.com wrote:
On Sat, Sep 5, 2015 at 4:20 PM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
Thanks for replying, that is helpful.
I'm going to write carefully, because I suspect we've only got this one shot at getting it right.
Not sure why? "Change the governance document" is explicitly listed as one of the steering council's powers, so I don't see what we'd lose by adopting what we have now.
In practice, I doubt that there will be any proposal to change the governance in the medium term, especially if the governance does turn out to be risk averse and conservative.
Whether there will be a proposal depends entirely on whether someone (perhaps you) makes one :-).
Whether the proposal will be successful depends on many things, but I'm guessing you'd have *more* luck getting traction in this hypothetical future where you can point to how the project is suffering from risk aversion and over-conservative decision making. Right now everyone seems to be getting excited about getting stuff done and trying out this proposed model, which seems like the worst possible time to try and convince them that they're doomed to stagnation.
[...]
No, by drama I mean situations where technical issues become personal, and where people end up feeling emotionally hurt and disrespected and so forth. I agree that vigorous debate is important, but strongly disagree that veins bulging and violence are necessary or helpful to success. (This is tangential to your main point, but I wanted to point it out.) Since I happened to be reading Kate Heddleston's blog yesterday, here's another relevant link... https://www.kateheddleston.com/blog/argument-cultures-and-unregulated-aggres...
The distinction is between a culture that welcomes expressions of disagreement with one that does not. In practice, if a group is easily alarmed by expressions of strong feeling, most people will be careful about expressing disagreement. This is of course a rather subtle point. It is easy to say that we do welcome disagreement, when in practice what happens is that the person disagreeing will in fact be labeled as being impolite.
I worry about this too. But this is a culture thing, not something that seems amenable to quick fixes; it requires buy-in from everyone individually. In particular I don't see what it has to do with whether we have a named leader or not -- being named "project leader" doesn't make someone's emails particularly more or less likely to change how anyone in particular thinks about things.
[...]
Yep. And like you said above re: IPython and so forth, the solution is to confront that problem head-on and have the debate. Having a dictator come in and dictate the outcome wouldn't help anything.
Yes, I see I am not being clear. I was claiming that it is difficult to have this debate without someone who will make the final call. That person doesn't 'dictate' but they have the final say when all the arguments are in. If I am right, then, with the no-leader consensus model you propose, it will be hard to have that debate because it is harder to get to the final decision.
I suggest we try it and see :-)
-n
On Do, 2015-08-27 at 17:22 +0100, Matthew Brett wrote:
Hi
On Thu, Aug 27, 2015 at 5:11 PM, josef.pktd@gmail.com wrote:
On Thu, Aug 27, 2015 at 11:04 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, Aug 27, 2015 at 3:34 PM, josef.pktd@gmail.com wrote: [snip]
I don't really see a problem with "codifying" the status quo.
That's an excellent point. If we believe that the current situation is the best possible, both now and in the future, then codifying the status quo is an excellent idea.
So, we should probably first start by asking ourselves:
- what numpy is doing well;
- what numpy could do better;
and then ask, is there some way we could make it more likely we will improve over time.
[snip]
As the current debate shows it's possible to have a public discussion about the direction of the project without having to delegate providing a vision to a president.
The idea of a president that I had in mind, was not someone who makes all decisions, but the person who holds themselves responsible for the performance of the project. If the project has a coherent vision already, the president has no need to provide one, but it's the president's job to worry about whether we have vision or not, and do what they need to, to make sure we don't lose track of that. If you don't know it already, I highly recommend Jim Collins' work on 'level 5 leadership' [1]
Still doesn't sound like the need for a president to me
" the person who holds themselves responsible for the performance of the project"
sounds more like the role of the "core" group (adding plural to persons) to me, and cannot be pushed of to an official president.
Except that, in the past, having multiple people taking decisions has led to the situation where no-one feels themselves accountable for the result, hence this situation tends to lead to stagnation.
Frankly, I am failing to see the direction of these arguments. One thing to remember, that a "core" group is much like a BDFL/president with multiple personalities ;), and a "core" group is not a fixed Oligarchy. Anyone able and willing should be in it and the governance document is clear about that I think (of course nothing is perfect, but we can try).
There is the thing of "how". I simply fail to see how the president can even be defined considering the size of the numpy development team (say 10, most of whom are busy with other things most of the time).
Also, I fail to see how the president would be any more useful then the agreement of some tasks being handled by some people who are enthusiastic about them (note those do not even have to be in the "core" group for starters, though they should become part of it quickly).
This is a community effort and I am starting to feel that the ideas you are giving are from a different management/company context. The goal of the governance is to show how and hopefully make it easy for *anyone* to provide vision. We do not need a manager who decides how to focus allocate resources, instead we must tell everyone that we are happy about any help we can get, and that anyone can pick up a topic they are enthusiastic about and drive numpy ahead.
And considering accountability, that help may well amount in saying: "Do NOT do this."
A "president" willing to run for such an election, should have a specific vision? Why should they be special to implement it? Note this is also the case in BDFL organizations. If you have a vision to improve python, it does not really matter if you happen to be Guido.
You write a PEP and, if people like it, it will be implemented.
At the same time we *must* have a well defined form of governance also for organizational things. Right now we cannot even decide on putting someone in charge of overseeing our NumFOCUS donations. NumPy could not even spend its own money!
Sorry, getting way too long :(....
- Sebastian
Nathaniel to push and organize the discussion, Chuck for continuity, and several core developers for detailed ideas and implementation, and a large number of contributors. (stylized roles) and noisy mailing list for feedback and discussion.
Given the size of the numpy development group, numpy needs individuals for the vision and to push things not a president, vice-presidents and assistant vice-presidents, IMO.
Yes, if the roles were honorary and administrative, they would not be useful.
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Aug 27, 2015, at 1:57 PM, Matthew Brett matthew.brett@gmail.com wrote:
The 'president' idea
...seems to be predicated on a steady stream of people who: actually want job, don't mind campaigning, are willing to accept any and all blame, and have the technical experience to make "final decisions". As others have pointed out the active developer community for NumPy is not measured in the hundreds (or even the tens, really). So: what is your proposed recourse if you hold an election and no-one shows up to run?
Bryan
Hi,
On Thu, Aug 27, 2015 at 3:43 PM, Bryan Van de Ven bryanv@continuum.io wrote:
On Aug 27, 2015, at 1:57 PM, Matthew Brett matthew.brett@gmail.com wrote:
The 'president' idea
...seems to be predicated on a steady stream of people who: actually want job, don't mind campaigning, are willing to accept any and all blame, and have the technical experience to make "final decisions". As others have pointed out the active developer community for NumPy is not measured in the hundreds (or even the tens, really). So: what is your proposed recourse if you hold an election and no-one shows up to run?
That seems to me a soluble problem, if there's agreement that the president idea is a sensible one. One very simple idea would be to revert to a 'core' system for the term for which there were no candidates.
On the other hand, I suspect that there are people who care enough about numpy that they are prepared to step up and take the blame if things go wrong.
Cheers,
Matthew
On Thu, Aug 27, 2015 at 9:36 AM, Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Wed, Aug 26, 2015 at 11:46 PM, Stefan van der Walt stefanv@berkeley.edu wrote:
Hi Matthew
On 2015-08-26 10:50:47, Matthew Brett matthew.brett@gmail.com wrote:
In short, the core structure seems to be characteristically associated with a conservatism and lack of vision that causes the project to stagnate.
Can you describe how a democratic governance structure would look? It's not clear from the discussions linked where successful examples are to be found.
Ah yes - as I was writing at the top of the xfree86 summary, it's difficult to assess governance models, because you cannot tell if a project that has a particular governance model would have been more successful with another model. For example, would clang be competing so successfully with gcc, if gcc had had a different governance model? Would Apache be further ahead of the many competitors in the web-server space with different management? Difficult to know.
The advantage of studying forks is that they usually arise from disagreements about how a project is managed. All other things being equal, we might expect a fork to fail, given the general aversion to forks and the considerable new work that has to be done to get one going. So, if a fork succeeds in the long term, that is probably an indication that the governance / management of the fork is indeed an improvement on the previous model.
So, in answer to your question, it's difficult to know if a particular governance model is successful. It isn't enough that a project has lasted, or is still active, because there are so many factors in play. On the other hand, I think it is possible to point to models that have a tendency to fail in particular ways, and the by-invitation meritocratic 'core' group (I think this is close to the 'steering committee' in our current draft) is the model that failed for NetBSD and XFree86, with a particular pattern of poor or absent accountability and lack of project vision.
Sorry to follow up on my own email, but:
I'm just speculating here, without data, but I suspect one the key elements that led to the decline and fall of NetBSD and XFree86 was the perception that there was no way for the community to depose the government. It seems these projects managed to combine aspects of the dictatorship model, with lots of emphasis on personal loyalty and expected gratitude, with a dysfunctional oligarchy, in which no-one felt able or willing to change the project direction, when the project was failing.
The other problem with the meritocracy / invitation model, is that some people are terrible managers. In the XFree86 project, for example, I think David Dawes did a terrible job of guiding the project when it ran into trouble. He was in the position he was in because of his huge commitment and contributions to the project, but I think he was not the right person to manage the project.
The standard 'core' model, doesn't take that into account. For example, I suspect that, if we had a David Dawes, no matter how terrible we thought they were at managing, we would feel obliged to put them onto the steering committee. It is much easier to count or review commits than it is to assess someone for their qualities as a leader or manager. We most of us would hate to be the person to make that assessment, and it's very tempting to negotiate ourselves into a world-view where this assessment is not necessary.
So, I speculate, that a good governance model would have:
* one 'president' who has to take final responsibility for all decisions; * this president might well have a fixed term, maybe with limits on the number of terms they can serve. * the president would be chosen by community vote and explicitly on the basis that they were good managers as well as coders; * for the presidential election, the candidates should set out what their vision for the project is, and how they plan to achieve that vision;
The point about these features is that we explicitly emphasize accountability, vision and management ability. Instead of a small number of people being in the position of assessing their peers for their ability to manage, the whole community (somehow defined) takes responsibility for that assessment, therefore making it easier to think about without distracting issues of personal loyalty or implied obligation.
See you,
Matthew
On Thu, Aug 27, 2015 at 2:16 AM, Matthew Brett matthew.brett@gmail.com wrote:
So, I speculate, that a good governance model would have:
- one 'president' who has to take final responsibility for all decisions;
- this president might well have a fixed term, maybe with limits on
the number of terms they can serve.
- the president would be chosen by community vote and explicitly on
the basis that they were good managers as well as coders;
- for the presidential election, the candidates should set out what
their vision for the project is, and how they plan to achieve that vision;
We actually discussed some variants on this kind of idea at the meeting, and I think the general sense of those present was we didn't want to go there (for whatever that's worth).
At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama, (b) when some amount of interpersonal drama does inevitably arise anyway, provide some regulated channel for it, hopefully one that leads to a drama sink. But elections are a huge massive drama source. No-one wants to spend time campaigning or wondering how some technical proposal will effect their re-election chances, we want to get this sorted out so that we can stop thinking about it and go back to solving actually interesting problems...
As for evidence... there are obviously projects that have had serious problems with some variant of core team model, but there are also many many successful projects that are also using variants of this model, and the document I sent around attempts to incorporate the lessons that have been learned in the process. OTOH after wracking my brain I think the only project I'm familiar with that has elections at all like this is Fedora, which elects... a "core team" (FESCo). Given that we don't have the problem of trying to manage thousands of contributors, I'm not sure their experience is really relevant. Or I guess Debian's use of General Resolutions as a decision-making procedure of last resort is kinda relevant, but... pretty different. (They also elect the project leader, which is more similar to what you're describing, but the project leader has no technical authority; in Debian the final authority short of a GR is the CTTE, which is explicitly designed as a classic beholden-to-nobody institution -- and even overriding the CTTE requires a supermajority.)
I kinda feel like... as a rule of thumb, if your description of your governance model starts with the words "I speculate that...", then NumPy is probably not a good project to use for your experiment?
-n
Hi,
On Thu, Aug 27, 2015 at 11:05 AM, Nathaniel Smith njs@pobox.com wrote:
On Thu, Aug 27, 2015 at 2:16 AM, Matthew Brett matthew.brett@gmail.com wrote:
So, I speculate, that a good governance model would have:
- one 'president' who has to take final responsibility for all decisions;
- this president might well have a fixed term, maybe with limits on
the number of terms they can serve.
- the president would be chosen by community vote and explicitly on
the basis that they were good managers as well as coders;
- for the presidential election, the candidates should set out what
their vision for the project is, and how they plan to achieve that vision;
We actually discussed some variants on this kind of idea at the meeting, and I think the general sense of those present was we didn't want to go there (for whatever that's worth).
At least personally, I have to admit that the idea of a governance model involving elections fills me with creeping horror. The reason is that whole point of having a governance model (IMHO) is to (a) minimize the rise of interpersonal drama
Right - I think this is key to the problem in this model. It is designed not to cause any trouble, and to keep things running as they are without controversy. It works OK on average as long as 'no change' is the desired outcome. In general the core group know each other fairly well, and feel a sense of shared loyalty to the group. This loyalty is exercised when some outside or inside force challenges the direction of the project. This was what made it so hard for the XFree86 core group to pull back from the course they had set.
The question is - is avoiding the potential controversy important enough to force us into a model that has (in my opinion) a high risk of tending to conservatism and stagnation?
As for evidence... there are obviously projects that have had serious problems with some variant of core team model, but there are also many many successful projects that are also using variants of this model, and the document I sent around attempts to incorporate the lessons that have been learned in the process. OTOH after wracking my brain I think the only project I'm familiar with that has elections at all like this is Fedora, which elects... a "core team" (FESCo). Given that we don't have the problem of trying to manage thousands of contributors, I'm not sure their experience is really relevant. Or I guess Debian's use of General Resolutions as a decision-making procedure of last resort is kinda relevant, but... pretty different. (They also elect the project leader, which is more similar to what you're describing, but the project leader has no technical authority; in Debian the final authority short of a GR is the CTTE, which is explicitly designed as a classic beholden-to-nobody institution -- and even overriding the CTTE requires a supermajority.)
I kinda feel like... as a rule of thumb, if your description of your governance model starts with the words "I speculate that...", then NumPy is probably not a good project to use for your experiment?
So my argument would be that our current data on success (lots of projects use this model and many of them are OK) is much less useful than the data from successful forks.
I suppose my question ends up being - do you agree that the core model does have these risks? Do they worry you? What do you think we can do to guard against them?
Cheers,
Matthew
participants (11)
-
Bryan Van de Ven -
Charles R Harris -
Chris Barker -
Eric Firing -
Francesc Alted -
Jaime Fernández del Río -
josef.pktd＠gmail.com -
Matthew Brett -
Nathaniel Smith -
Sebastian Berg -
Stefan van der Walt