On Mon, Apr 20, 2020 at 6:21 PM Eric Snow <ericsnowcurrently@gmail.com> wrote:
Nathaniel,
Your tone and approach to this conversation concern me. I appreciate that you have strong feelings here and readily recognize I have my own biases, but it's becoming increasingly hard to draw any constructive insight from what tend to be very longs posts from you. It ends up being a large commitment of time for small gains. And honestly, it's also becoming hard to not counter some of your more elaborate statements with my own unhelpful prose. In the interest of making things better, please take it all down a notch or two.
I'm sorry it's landing that way on you. I am frustrated, and I think that's a reasonable reaction. But I know we're all here because we want to make Python better. So let me try again to explain my position, to maybe reboot the conversation in a more productive way. All engineering decisions come down to costs vs. benefits. My frustration is about how you're approaching the costs, and how you're approaching the benefits. **Costs** I think you've been downplaying the impact of subinterpreter support on the existing extension ecosystem. All features have a cost, which is why PEPs always require substantial rationales and undergo intense scrutiny. But subinterpreters are especially expensive. Most features only affect a small group of modules (e.g. async/await affected twisted and tornado, but 99% of existing libraries didn't care); OTOH subinterpreters require updates to every C extension module. And if we start telling users that subinterpreters are a supported way to run arbitrary Python code, then we've effectively limited extension authors options to "update to support subinterpreters" or "explain to users why they aren't writing a proper Python module", which is an intense amount of pressure; for most features maintainers have the option of saying "well, that isn't relevant to me", but with subinterpreter support that option's been removed. (You object to my calling this an API break, but you're literally saying that old code that worked fine is being redefined to be incorrect, and that all maintainers need to learn new techniques. That's the definition of an API break!) And until everything is updated, you're creating a schism in the ecosystem, between modules that support subinterpreters and those that don't. I did just read your reply to Sebastian, and it sounds like you're starting to appreciate this impact more, which I'm glad to see. None of this means that subinterpreters are necessarily a bad idea. For example, the Python 2 -> Python 3 transition was very similar, in terms of maintainers being forced to go along and creating a temporary schism in the ecosystem, and that was justified by the deep, unfixable problems with Python 2. But it does mean that subinterpreters need an even stronger rationale than most features. And IMO, the point where PEP 554 is accepted and we start adding new public APIs for subinterpreters is the point where most of these costs kick in, because that's when we start sending the message that this is a real thing and start forcing third-party maintainers to update their code. So that's when we need the rationale. **Benefits** In talks and informal conversations, you paint a beautiful picture of all the wonderful things subinterpreters will do. Lots of people are excited by these wonderful things. I tried really hard to be excited too. (In fact I spent a few weeks trying to work out a subinterpreter-style proposal myself way back before you started working on this!) But the problem is, whenever I look more closely at the exciting benefits, I end up convincing myself that they're a mirage, and either they don't work at all (e.g. quickly sharing arbitrary objects between interpreters), or else end up being effectively a more complex, fragile version of things that already exist. I've been in lots of groups before where everyone (including me!) got excited about a cool plan, focused exclusively on the positives, and ignored critical flaws until it was too late. See also: "groupthink", "confirmation bias", etc. The whole subinterpreter discussion feels very familiar that way. I'm worried that that's what's happening. Now, I might be right, or I might be wrong, I dunno; subinterpreters are a complex topic. Generally the way we sort these things out is to write down the arguments for and against and figure out the technical merits. That's one of the purposes of writing a PEP. But: you've been *systematically refusing to do this.* Every time I've raised a concern about one rationale, then instead of discussing the technical substance of my concern, you switch to a different rationale, or say "oh well, that rationale isn't the important one right now". And the actual text in PEP 554 is *super* vague, like it's so vague it's kind of an insult to the PEP process. From your responses in this thread, I think your core position now is that the rationale is irrelevant, because some half-baked subinterpreter support was merged back in the Python 1.x days, and therefore we *must* do whatever is necessary to make subinterpreters work now. Even if subinterpreters are completely useless and require breaking the C API, it doesn't matter, we have to do it. But this is a terrible way to run a language. (In fact of course you don't think this; you think that we should support subinterpreters because they *are* useful. But if you keep refusing to give a rationale then how can we know?) **Conclusion** Either one of these issues would be concerning on its own, but the combination is what's got me *so* frustrated. Put together, I feel like you've ended up saying "I have the power to force all the world's package maintainers to jump through hoops to support my vanity project and they can't stop me, so I don't have to bother justifying it". I know that's not what you intend, but like... intentions don't really change anything. All I want is for you to write down the rationale for subinterpreters properly, like every other PEP author has to, so we can have a proper technical discussion and your plan can stand or fall on its merits. If you're right and subinterpreters are worth the cost, then you have no reason to fear scrutiny. And if you're wrong... well, you still have nothing to fear :-), because it's better to figure that out now, *before* we start making commitments in our public API. idk, maybe the most productive way to move this forward would be for me to write up a PEP for removing subinterpreter support, so we can all look at the different options and make an informed decision? -n -- Nathaniel J. Smith -- https://vorpus.org