With Regards to Dustin's Proposal
I’ve been kind of on and off again in my time lately, so I haven’t had much time to push this. However I would like to bring back the discussion here to Dustin’s proposal.
Besides the specific process itself, here is the basic gist of it:
- Individual projects still control themselves. - The PyPA handles cross project decisions - Specific topics get a Interest Area Authority delegated to them, to act as the final decision maker (e.g. BDFL-Delegate). - Topics without an IAA, or ones where the IAA recuse themselves for whatever reason (time, conflict, whatever) fall back to voting by the PyPA committers. - No longer using the PEP process except for actual changes to Python itself (e.g. ensurepip would still have been a PEP, PEP 440 would not).
Basically I think this is overall a good thing for us. The basic structure is largely similar to what we had setup under the PEP process, except instead of authority being delegated by the BDFL, it’s delegated by us, the PyPA. Ultimately I think this is better, because we’re the folks investing the most time and effort into this space. Particularly with python-dev grappling with what the future decision making process is for python-dev (which is going to change the PEP process).
I think it would be confusing for us to have a process strictly for governance stuff, and then another process for non PEP-worthy changes, but that still are cross team, and then the PEP process itself. It’s a lot easier and simpler if we just use the same process for it all. Obviously we’ll still have the PEP process for changes to the interpreter or standard library, but that’s a pretty easy clear line to draw between “third party packaging stuff via the PyPA” and “First party standard library / interpreter”.
I know that Paul was concerned about his role, and ultimately I think that with the changes Dustin has made, that role is preserved as an Interest Area Authority (assuming the PyPA votes to keep him in that role! Which I see no reason why we wouldn’t). However it gives us a better answer for what to do if Paul steps down, or feels the need to recuse himself for one reason or another (or the extremely unlikely situation that an IAA starts making harmful decisions).
I also know that Nick has a few concerns with accessibility to the wider community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need. So we can adjust it as we go along to what makes the most sense to us, without having to go to python-dev and get them to change their process to fit our needs.
All in all, I’m a big fan of the proposal, and I think it is the right direction for us to move in.
I found that drawing some pictures helped clarify my thinking. Maybe they'll be useful for others too:
https://docs.google.com/document/d/10mAf7eWPt_fjevECE7GOytxoFmTIueprpzk59JQs...
On Thu, Aug 30, 2018 at 5:52 PM, Donald Stufft donald@stufft.io wrote:
I’ve been kind of on and off again in my time lately, so I haven’t had much time to push this. However I would like to bring back the discussion here to Dustin’s proposal.
Besides the specific process itself, here is the basic gist of it:
- Individual projects still control themselves.
- The PyPA handles cross project decisions
- Specific topics get a Interest Area Authority delegated to them, to act as the final decision maker (e.g. BDFL-Delegate).
- Topics without an IAA, or ones where the IAA recuse themselves for whatever reason (time, conflict, whatever) fall back to voting by the PyPA committers.
- No longer using the PEP process except for actual changes to Python itself (e.g. ensurepip would still have been a PEP, PEP 440 would not).
Basically I think this is overall a good thing for us. The basic structure is largely similar to what we had setup under the PEP process, except instead of authority being delegated by the BDFL, it’s delegated by us, the PyPA. Ultimately I think this is better, because we’re the folks investing the most time and effort into this space. Particularly with python-dev grappling with what the future decision making process is for python-dev (which is going to change the PEP process).
I think it would be confusing for us to have a process strictly for governance stuff, and then another process for non PEP-worthy changes, but that still are cross team, and then the PEP process itself. It’s a lot easier and simpler if we just use the same process for it all. Obviously we’ll still have the PEP process for changes to the interpreter or standard library, but that’s a pretty easy clear line to draw between “third party packaging stuff via the PyPA” and “First party standard library / interpreter”.
I know that Paul was concerned about his role, and ultimately I think that with the changes Dustin has made, that role is preserved as an Interest Area Authority (assuming the PyPA votes to keep him in that role! Which I see no reason why we wouldn’t). However it gives us a better answer for what to do if Paul steps down, or feels the need to recuse himself for one reason or another (or the extremely unlikely situation that an IAA starts making harmful decisions).
I also know that Nick has a few concerns with accessibility to the wider community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need. So we can adjust it as we go along to what makes the most sense to us, without having to go to python-dev and get them to change their process to fit our needs.
All in all, I’m a big fan of the proposal, and I think it is the right direction for us to move in. _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mm3/mailman3/lists/pypa-committers.python.org/
On Fri, 31 Aug 2018 at 01:52, Donald Stufft donald@stufft.io wrote:
I also know that Nick has a few concerns with accessibility to the wider community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need. So we can adjust it as we go along to what makes the most sense to us, without having to go to python-dev and get them to change their process to fit our needs.
I don't think the wider community necessarily participates in discussions, but nevertheless I feel that the *ability* to do so is essential. There's already an undercurrent of feeling that the PyPA make decisions without considering the community[1], which I think we need to address. It's important to me that anyone in the community can comment on proposals and be heard (and that's not just "technically has the ability to", but rather it's "knows how to, and has the information they need to do so"). Currently, the PEP process is well known, and "everyone" knows how to participate (more accurately, they know they can, and that if they want to they can find out how to do so). Any new PyPA process *won't* be well known, by definition, and when decisions made using that process impact the public, there is naturally going to be a feeling that those decisions were made by an exclusive group. One way we can mitigate that is by pointing at a well-publicised, dirt simple process for people to contribute, plus a clear process of keeping "the community" up to date with progress of ongoing discussions. That's more than the current PEP process provides, but I think we *need* to provide more, simply to overcome the barrier of being an unfamiliar new process.
I do agree that having a process that suits us, and isn't tied to Python-dev's different needs, is probably beneficial in the longer term. But we need to make sure that in the short term, we maintain the trust of the community that the new process is at least as open and accessible as the PEP process - and ideally more so (because it's better if the new process has benefits for both the community and ourselves, rather than just benefiting us).
I need to review the proposal, because I can't recall right now what it says on the matter of a replacement for the PEP process, so maybe this is already covered. If not, it's something we should consider.
[1] I'm not giving examples at the moment - if people want me to, I can, but if we are all aware of that sentiment, I don't think focusing on details is that important. Also, to give examples I'd have to choose between giving pip-specific ones, which don't feel like they make the point that this is a PyPA issue, or making comments about other projects' decisions, which I don't want to do. My examples aren't necessarily "PyPA governance" level issues, but in this context I don't think the community distinguishes between the PyPA and the individual projects (nor do I think they should - we should hold our individual projects to a similar standard of behaviour as we hold ourselves as a group).
All in all, I’m a big fan of the proposal, and I think it is the right direction for us to move in.
Overall, I think it's reasonable, and the change to have separate authorities over particular areas addresses my main concern over process. But I do think (as noted above) we need to take care over the question of openness.
Paul
On Aug 31, 2018, at 4:07 AM, Paul Moore p.f.moore@gmail.com wrote:
On Fri, 31 Aug 2018 at 01:52, Donald Stufft donald@stufft.io wrote:
I also know that Nick has a few concerns with accessibility to the wider community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need. So we can adjust it as we go along to what makes the most sense to us, without having to go to python-dev and get them to change their process to fit our needs.
I don't think the wider community necessarily participates in discussions, but nevertheless I feel that the *ability* to do so is essential. There's already an undercurrent of feeling that the PyPA make decisions without considering the community[1], which I think we need to address. It's important to me that anyone in the community can comment on proposals and be heard (and that's not just "technically has the ability to", but rather it's "knows how to, and has the information they need to do so"). Currently, the PEP process is well known, and "everyone" knows how to participate (more accurately, they know they can, and that if they want to they can find out how to do so). Any new PyPA process *won't* be well known, by definition, and when decisions made using that process impact the public, there is naturally going to be a feeling that those decisions were made by an exclusive group. One way we can mitigate that is by pointing at a well-publicised, dirt simple process for people to contribute, plus a clear process of keeping "the community" up to date with progress of ongoing discussions. That's more than the current PEP process provides, but I think we *need* to provide more, simply to overcome the barrier of being an unfamiliar new process.
Yea I agree the ability to do so is essential, an early draft of this proposal had all discussion happening on pypa-committers IIRC, and I pushed for everything but the actual vote to happen on the PR, precisely so that literally anyone with a Github account could participate.
I would say that the fact that the *current* process involves participating in distutils-sig actually makes it less likely that someone *does* participate in that discussion. I know that more than one time I’ve suggested that someone bring something up on distutils-sig, and they balked either because it was “another mailing list to subscribe to” [1] or because of the past history of distutils-sig and catalog-sig being very toxic places— even though I don’t think that’s the case *now*, the history is still very much in people’s minds, and they avoid it because of it still.
We have the pypi-announce mailing list, and we have @ThePyPA and @PyPI on twitter, so one potential here is for accepted RFCs to be published using those communications mechanisms. Although that’s late in the process for someone who might want to participate in that discussion.
One potential option I could think of, and I wouldn’t want to actually bake this into the RFC process itself but as something we could do, is setup a simple bot that anytime a new PR was opened on the RFCs repository (we could make it a bit smart, and have it only work on new RFCs) it would trigger a notification email to distutils-sig. That effectively gives us a similar level of visibility for distutils-sig subscribers, and prods people that hey, this new thing exists where we’re having discussions.
Another option is if we do adopt this process/model is that we can write a short PEP, accepted by both yourself and myself (as the BDFL-Delegate for packaging standards and PyPI), formally transitioning authority over those areas to the new RFC process. That would be somewhat abusing the PEP process (since at that point it wouldn’t *really* be up for discussion) but would stand as a way for someone who knows/follows only PEPs to know that there is a new process for these specific types. It would also clear up any potential confusion about what process “owns” these things, since there’s a clear marker of “not me!” recorded as a PEP.
I do think it is important that if we do things like notify distutils-sig or something like that, that those notifications are simple notifications, and not forking the discussion into multiple places. The discussion should live in a single place, and currently the proposal has that single place being on the PR proposing a new RFC.
I think we should also make sure that we document how to write a RFC in packaging.python.org http://packaging.python.org/ somewhere, since that site acts as a really nice central hub for someone who isn’t quite sure where to go for a specific packaging thing.
I don’t think a lot of these things need to be, or should be in the actual proposal itself unless there are changes to the actual process that we think should be added or changed to help this. However, I think we should do these extras if we do adopt it, to help with the transition.
[1] With the move to MM3 it’s no longer actually required to subscribe to it to comment on an issue, but if you don’t subscribe you don’t get notification of responses. So you still have to subscribe to get notified of updates, otherwise you have to explicitly pull only, vs something that allows a user to selectively subscribe to threads they care about.
On Fri, 31 Aug 2018 at 12:49, Donald Stufft donald@stufft.io wrote:
Yea I agree the ability to do so is essential, an early draft of this proposal had all discussion happening on pypa-committers IIRC, and I pushed for everything but the actual vote to happen on the PR, precisely so that literally anyone with a Github account could participate.
There *are* some people who argue that they won't participate because they specifically don't want to have a github account. But I think at this point, that's their choice to exclude themselves, and we don't need to worry too much about that. Just thought I'd mention it so it's explicit.
I do think it is important that if we do things like notify distutils-sig or something like that, that those notifications are simple notifications, and not forking the discussion into multiple places. The discussion should live in a single place, and currently the proposal has that single place being on the PR proposing a new RFC.
I still have to think about this. Personally, I've found that I'm much *less* likely to get involved in debates on RFCs when they are on PR conversations than when they are on a mailing list. That reflects my personal workflow and how I choose to allocate my volunteer time, so again it's something of a "my choice" issue, but I will admit that it makes me feel excluded from decisions (I'm basing this on the odd Python PEP, and some packaging discussions, that I didn't get involved in, so the sample size is small, but it feels significant to me).
I do feel it's harder to have subthreads, side debates and digressions on a github issue. And it's more or less impossible to start a second related but independent thread. There are plusses and minuses to this of course - rambling discussions can be the death of a proposal, but so can missing peripheral but important points. Also, I find the differences between github and email etiquette (specifically, the level of quoting of relevant context) makes complex debates much harder to follow on github.
I'm not arguing *against* github, as such. More arguing that different tools have different strengths and weaknesses, and excluding certain tools can harm the process. Of course, so can having to monitor too many tools - but that argument works both ways (what's "yet another mailing list" to some people is "yet another github PR to track" to others).
I think we should also make sure that we document how to write a RFC in packaging.python.org somewhere, since that site acts as a really nice central hub for someone who isn’t quite sure where to go for a specific packaging thing.
Agreed - but equally if the process is complex enough that people can't get it mostly right just by guessing, we've over-engineered it :-)
I don’t think a lot of these things need to be, or should be in the actual proposal itself unless there are changes to the actual process that we think should be added or changed to help this. However, I think we should do these extras if we do adopt it, to help with the transition.
Again agreed, but if they aren't in the overall governance proposal, then they should probably be in another separate proposal. I *don't* think it's OK for us to change our standards definition process without that change having the support and involvement of the community (even if it's only "we've decided on this new process, the discussions are over here in this public archive, before it's signed off shout if you have any objections" - at some point things get too recursive and we end up agreeing on how we agree on the process for agreement :-))
Paul
Hey all,
Not much to add that hasn’t been said... I think Nathaniel’s picture would be great explanatory material to accompany any announcements.
With regard to everything but the vote happening on github I find myself mostly with Paul, although I think if we had a broader solution for internal and intraproject communication (the first thing we had discussed) this would be a bit easier. Github has advantages that I acknowledge and per Paul’s points which I’m sure at least in part refer to the process of moving pipenv into the pypa, putting those kinds of conversations on github would at least give the community a more obvious place to participate.
From my experience so far, though, even though those discussions were public, people who didn’t participate still feel angry at us or at the pypa in general for ‘hiding the process’ or ‘recommending’ anything that doesn’t work all the time. Packaging is a core, central use— when someone sits down to begin doing some python thing, they start with <insert pypa tool> as a builder or installer or whatever. When those things don’t work, it’s not even possible to begin your python, so while we can move to an RFC process and such, I expect this is going to be seen as an implementation detail where people’s frustration is probably more about oversight and collaboration (why does pipenv reimplement pip things and sometimes not fix them whenever pip fixes them etc, per the recent discussion on distutuls-sig).
Dan Ryan // pipenv maintainer gh: @techalchemy
On Aug 31, 2018, at 8:45 AM, Paul Moore p.f.moore@gmail.com wrote:
On Fri, 31 Aug 2018 at 12:49, Donald Stufft donald@stufft.io wrote:
Yea I agree the ability to do so is essential, an early draft of this proposal had all discussion happening on pypa-committers IIRC, and I pushed for everything but the actual vote to happen on the PR, precisely so that literally anyone with a Github account could participate.
There *are* some people who argue that they won't participate because they specifically don't want to have a github account. But I think at this point, that's their choice to exclude themselves, and we don't need to worry too much about that. Just thought I'd mention it so it's explicit.
I do think it is important that if we do things like notify distutils-sig or something like that, that those notifications are simple notifications, and not forking the discussion into multiple places. The discussion should live in a single place, and currently the proposal has that single place being on the PR proposing a new RFC.
I still have to think about this. Personally, I've found that I'm much *less* likely to get involved in debates on RFCs when they are on PR conversations than when they are on a mailing list. That reflects my personal workflow and how I choose to allocate my volunteer time, so again it's something of a "my choice" issue, but I will admit that it makes me feel excluded from decisions (I'm basing this on the odd Python PEP, and some packaging discussions, that I didn't get involved in, so the sample size is small, but it feels significant to me).
I do feel it's harder to have subthreads, side debates and digressions on a github issue. And it's more or less impossible to start a second related but independent thread. There are plusses and minuses to this of course - rambling discussions can be the death of a proposal, but so can missing peripheral but important points. Also, I find the differences between github and email etiquette (specifically, the level of quoting of relevant context) makes complex debates much harder to follow on github.
I'm not arguing *against* github, as such. More arguing that different tools have different strengths and weaknesses, and excluding certain tools can harm the process. Of course, so can having to monitor too many tools - but that argument works both ways (what's "yet another mailing list" to some people is "yet another github PR to track" to others).
I think we should also make sure that we document how to write a RFC in packaging.python.org somewhere, since that site acts as a really nice central hub for someone who isn’t quite sure where to go for a specific packaging thing.
Agreed - but equally if the process is complex enough that people can't get it mostly right just by guessing, we've over-engineered it :-)
I don’t think a lot of these things need to be, or should be in the actual proposal itself unless there are changes to the actual process that we think should be added or changed to help this. However, I think we should do these extras if we do adopt it, to help with the transition.
Again agreed, but if they aren't in the overall governance proposal, then they should probably be in another separate proposal. I *don't* think it's OK for us to change our standards definition process without that change having the support and involvement of the community (even if it's only "we've decided on this new process, the discussions are over here in this public archive, before it's signed off shout if you have any objections" - at some point things get too recursive and we end up agreeing on how we agree on the process for agreement :-))
Paul _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mm3/mailman3/lists/pypa-committers.python.org/
On Fri, Aug 31, 2018, 01:08 Paul Moore p.f.moore@gmail.com wrote:
On Fri, 31 Aug 2018 at 01:52, Donald Stufft donald@stufft.io wrote:
I also know that Nick has a few concerns with accessibility to the wider
community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need. So we can adjust it as we go along to what makes the most sense to us, without having to go to python-dev and get them to change their process to fit our needs.
I don't think the wider community necessarily participates in discussions, but nevertheless I feel that the *ability* to do so is essential. There's already an undercurrent of feeling that the PyPA make decisions without considering the community[1], which I think we need to address. It's important to me that anyone in the community can comment on proposals and be heard (and that's not just "technically has the ability to", but rather it's "knows how to, and has the information they need to do so"). Currently, the PEP process is well known, and "everyone" knows how to participate (more accurately, they know they can, and that if they want to they can find out how to do so).
A rule of thumb I learned from writers: outside critique is great at identifying *that* there's a problem, but lousy at identifying *what* the problem is.
The complaints about PyPA being arbitrary and autocratic always seem weird on their face, because we all know we're a bunch of disorganized volunteers who would love more participation.
I think the problem is actually that we're not autocratic enough :-). Or at least, not organized enough. I actually don't know how the PyPA makes decisions. People make claims about what "the PyPA" has done, and no one has the authority to even say whether they're true or not. Most python users have absolutely no idea where PEPs come from, and then even if they do, the packaging PEP process is its own weird thing with confusing and poorly documented exceptions to the regular PEP process. We have so many partially overlapping mailing lists that it's effectively impossible to follow everything. This all creates the impression of opacity, insularity, arbitrariness.
I think the main thing that will make a difference here is having a clear, simple picture of what the PyPA is and isn't, what the communication channels are, how decisions get made, how people can get involved, etc. And then actually writing that down and putting it somewhere prominent.
-n
On Aug 31, 2018, at 2:26 PM, Nathaniel Smith njs@pobox.com wrote:
I think the main thing that will make a difference here is having a clear, simple picture of what the PyPA is and isn't, what the communication channels are, how decisions get made, how people can get involved, etc. And then actually writing that down and putting it somewhere prominent.
This is basically my hope :) Step 1 is, in my opinion, coming up with how we decide the answers to those things.
On Fri, 31 Aug 2018 at 10:52, Donald Stufft donald@stufft.io wrote:
I also know that Nick has a few concerns with accessibility to the wider community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need.
I think the wider community cares more about "Is it in a PEP?" than you believe, as we had folks outright refusing to accept Description-Content-Type as legitimate when it was only defined in https://packaging.python.org/specifications/core-metadata/#description-conte..., with no PEP to back it up as being an officially endorsed interoperability standard.
Attempting to build legitimacy and credibility for a random collection of text files in a GitHub repo rather than continuing to use https://www.python.org/dev/peps/ feels like a pointless time-wasting distraction to me. It's not like PEP-level proposals come up all that often - even with the 5 year delay induced by the PEP 426 detour, PEP 566 still only had a handful of major changes in it (Description-Content-Type, Provides-Extra, the canonical conversion to a JSON-compatible data structure, allowing the Description to be in the metadata body, and explicitly referencing PEPs 440 and 508).
Cheers, Nick.
On Sep 2, 2018, at 12:32 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On Fri, 31 Aug 2018 at 10:52, Donald Stufft donald@stufft.io wrote:
I also know that Nick has a few concerns with accessibility to the wider community, which I believe I addressed in the other email thread, but to reiterate, I don’t think that the wider community cares one way or another, and I think the biggest benefit comes from being able to tailor this process to what we need.
I think the wider community cares more about "Is it in a PEP?" than you believe, as we had folks outright refusing to accept Description-Content-Type as legitimate when it was only defined in https://packaging.python.org/specifications/core-metadata/#description-conte..., with no PEP to back it up as being an officially endorsed interoperability standard.
Who refused to accept it? Is there any indication that they will only implement PEPs, or that they just wanted it defined as a standard before supporting it, and they don’t care how we define it as a standard.
Attempting to build legitimacy and credibility for a random collection of text files in a GitHub repo rather than continuing to use https://www.python.org/dev/peps/ feels like a pointless time-wasting distraction to me. It's not like PEP-level proposals come up all that often - even with the 5 year delay induced by the PEP 426 detour, PEP 566 still only had a handful of major changes in it (Description-Content-Type, Provides-Extra, the canonical conversion to a JSON-compatible data structure, allowing the Description to be in the metadata body, and explicitly referencing PEPs 440 and 508).
How much of “PEP-level proposals don’t come up that often” is because people don’t want to engage the PEP process? I know personally I’ve suggested more than one person write a PEP, and they balked at the idea. It’s entirely possible they’d also balk at this idea too, it’s impossible to know for sure, but for better or worse, the PEP process has a perception amongst some people as being a heavy weight bike shedding exercise. Some people also refuse to join distutils-sig to engage the PEP process, because of the history of toxicity of those mailing lists.
I know of *several* “PEP-level” changes I’ve got on my personal back burner that I’ve avoided because the current process aligns itself well with bike shedding and arguments. The PEP process itself encourages the behavior that burnt out Guido, because it uses archaic tools that incentivize people to repeat the same tired arguments over and over again. To comment on old versions of a proposal and straw man the hell out of them. I’m not exactly the thinnest skinned person who participates in the PEP process, and every time I think about writing a PEP, I feel like I have to mentally prepare myself. Some of that is the nature of “politics”, and no system will ever solve it. Some of it is due to the process and tools we use, and we can’t ever fix that for ourselves without breaking away from whatever python-dev does.
On Sun, Sep 2, 2018 at 9:32 AM, Nick Coghlan ncoghlan@gmail.com wrote:
I think the wider community cares more about "Is it in a PEP?" than you believe, as we had folks outright refusing to accept Description-Content-Type as legitimate when it was only defined in https://packaging.python.org/specifications/core-metadata/#description-conte..., with no PEP to back it up as being an officially endorsed interoperability standard.
Attempting to build legitimacy and credibility for a random collection of text files in a GitHub repo rather than continuing to use https://www.python.org/dev/peps/ feels like a pointless time-wasting distraction to me.
I think it's important to have a clear and unambiguous story for what counts as an official spec and what doesn't, but I doubt anyone cares whether it specifically involves PEPs or not. But if you're worried, we could write a PEP that says "python-dev hereby hands off packaging authority to the PyPA <link>, from now own their process is authoritative".
-n
participants (5)
-
Dan Ryan -
Donald Stufft -
Nathaniel Smith -
Nick Coghlan -
Paul Moore