On Fri, Apr 17, 2020 at 1:38 PM Brett Cannon <brett@python.org> wrote:

Eric Snow wrote:

...

1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 (they get parallelism for free in 3.10) 2. like 1, but mark the module as provisional until per-interpreter GIL lands 3. do not merge PEP 554 until per-interpreter GIL is merged 4. like 3, but publish a 3.9-only module to PyPI in the meantime

My vote is for #2. No need to rush it and it gives you feedback on the API from the public before it gets locked in for backwards-compatibility.

Yeah, I'd be fine with that. Actually, making the module provisional is already part of the PEP. :)

And you do know you could have done this as a poll on Discourse, right?

Yeah, I thought of that after the fact. Then again, discourse isn't the tool I already had open. :) -eric

On Fri, Apr 17, 2020 at 1:39 PM Victor Stinner <vstinner@python.org> wrote:

I have concerns about shared data.

The current implementation of CPython and subinterpreters still shares singletons. Moreover, I understand that the PEP proposes to first directly share PyObject between subinterpreters, like bytes objects.

Hmm, the PEP explicitly says: Note that objects are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects' data is passed between interpreters. See the Shared data section for more details about sharing between interpreters.

Isolating subinterpreters is a big task and so all issues cannot be solved at once. So I am not with making some compromises for a first milestone.

+1 To me, one of the most important things is to get the functionality out there as soon as possible, even if it is limited (but not useless).

The problem is to define which semantics can be exposed in the public API for the first milestone.

[snip]

What is the plan for Python 3.9? What semantics would be exposed in Python 3.9?

Should we wait until these questions are answered before making the API public?

The PEP proposes minimal functionality. We can relax that in future versions of Python on a case-by-case basis. If you are specifically referring to object sharing, the PEP already says there is none. -eric

On Fri, 17 Apr 2020 13:59:46 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:

...

For bytes or buffer objects, I understand that you propose to share the exact same PyObject objects between interpreters, at least in the first implementation.

It may be better to have one proxy object in each interpreter which would control which interpreters can read and which interpreters can write into the object. It's a similar but more complex issue than singletons.

It isn't the same Python object. It is the buffer that gets shared (for objects that support the buffer protocol).

When such a buffer dies, in which interpreter is the original object's destructor called? Regards Antoine.

On Sat, 18 Apr 2020 10:08:27 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:

On Sat, Apr 18, 2020, 01:16 Antoine Pitrou <solipsis@pitrou.net> wrote:

...

On Fri, 17 Apr 2020 13:59:46 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:

...

...

For bytes or buffer objects, I understand that you propose to share the exact same PyObject objects between interpreters, at least in the first implementation.

It may be better to have one proxy object in each interpreter which would control which interpreters can read and which interpreters can write into the object. It's a similar but more complex issue than singletons.

It isn't the same Python object. It is the buffer that gets shared (for objects that support the buffer protocol).

When such a buffer dies, in which interpreter is the original object's destructor called?

In the original interpreter (via a "pending call").

Great, thank you! Regards Antoine.

On Fri, Apr 17, 2020 at 1:54 PM Victor Stinner <vstinner@python.org> wrote:

Honestly, I don't think that we are far from being able to move the GIL lock per interpreter. But this change alone is useless, since we still have too many C extensions which are not ready for that:

Agreed. This is a big part of why I don't think we can make the GIL per-interpreter for 3.9. However, there are some things we could do to mitigate the problem if we want to aim for 3.9: * only allow the main interpreter to import extension modules that do not implement PEP 489 support + in subinterpreters it would be an ImportError * allow subinterpreters to be created either isolated (not sharing the GIL, etc.) or not isolated (sharing the GIL, etc.) + the default is "not isolated" (but only while the "interpreters" module is provisional) and users must opt in to isolated mode + this would allow extension authors to more easily check their modules for subinterpreter support, without causing excessive bug reports from users trying out subinterpreters I also am not convinced that in the next month we can work out all the possible corner cases problems that I expect will pop up once we have a per-interpreter GIL. Then again, arguably that could be addressed after the feature freeze if we think those possible issues could be sorted out in a month or two. :)

I would prefer PEP 554 to be restricted to the communication between interpreters.

Isolating subinterpreters should be out of the scope of this PEP, since it's a different problem and we cannot solve all problems in a single PEP.

Isolating interpreters isn't part of PEP 554, other than that objects are not shared between interpreters.

Not sure if a PEP is needed to isolate subinterpreters. There are already many issues and https://github.com/ericsnowcurrently/multi-core-python/ to track the work.

Agreed. I don't know that anyone had or has any intention of adding a PEP related to isolation (i.e. moving the GIL to PyInterpreterState). -eric

On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith <njs@pobox.com> wrote:

I think some perspective might be useful here :-).

The last time we merged a new concurrency model in the stdlib, it was asyncio.

[snip]

OTOH, AFAICT the new concurrency model in PEP 554 has never actually been used, and it isn't even clear whether it's useful at all.

Perhaps I didn't word things quite right. PEP 554 doesn't provide a new concurrency model so much as it provides functionality that could probably be used as the foundation for one. Ultimately the module proposed in the PEP does the following: * exposes the existing subinterpreters functionality almost as-is * provides a minimal way to pass information between subinterpreters (which you don't need in C but do in Python code) * adds a few minor conveniences like propagating exceptions and making it easier to share buffers safely So the comparison with asyncio isn't all that fair.

Designing useful concurrency models is *stupidly* hard. And on top of that, it requires major reworks of the interpreter internals +

Nearly all of the "rework" is worth doing for other good reasons. Furthermore, I'd call at most a few of the reworks "major".

disrupts the existing C extension module ecosystem -- which is very different from asyncio, where folks who didn't use it could just ignore it.

The concern is users opening issues saying "your extension won't work in subinterpreters", right? (You brought up the possible burden on extension authors in discussions several years ago, for which I'm still grateful.) How is that different from any other feature, new or not? "Your library doesn't provide an awaitable API." "Your library doesn't support pickling." "Your library doesn't implement the buffer protocol." PEP 554 doesn't introduce some new kind of impact. So (unless I've misunderstood), by your reasoning we wouldn't add any new features for which library authors might have to change something. Are you suggesting that the burden from PEP 554 will be larger than saying "then don't try to use our extension in subinterpreters"? Are you concerned about users reporting bugs that surface when an incompatible extension is used in a subinterpreter? That shouldn't be a problem if we raise ImportError if an extension that does not support PEP 489 is imported in a subinterpreter. FWIW, the impact to extension authors is the one thing about which I still have any meaningful uncertainty and worry. Various people have explained to me how it won't be a big problem, but I'm still nervous about it. I just don't think my worry is large than the actual risk (and possible cost).

So to me, it's kind of shocking that you'd even bring up the possibility of merging PEP 554 as-is, without even a provisional marker.

Unsurprisingly it isn't shocking to me. :) From my point of view it seems okay. However, I'll be the first to recognize how hard it can be to see things from a different perspective. Hence I started this thread. :)

And if it's possible for it to live on PyPI, then why would we even consider putting it into the stdlib?

Given that we're exposing functionality of the CPython runtime I don't see the point in keeping this out of the stdlib. Furthermore, there are use cases to explore for subinterpreters in our test suite that we can address only if the "interpreters" module is part of the CPython repo. So why keep it hidden away and then publish the exact same thing on PyPI?

Personally, I'm still leaning towards thinking that the whole subinterpreter project is fundamentally flawed, and that on net we'd be better off removing support for them entirely.

By which I imagine you mean drop the subinterpreters API and not actually get rid of all the architecture related to PyInterpreterState (which is valuable from a code health perspective).

But that's a more complex and nuanced question that I'm not 100% certain of, while the idea of merging it for 3.9 seems like a glaringly obvious bad idea.

Yeah, I remember your position from previous conversations (and still appreciate your feedback). :)

I know you want folks to consider PEP 554 on its own merits, ignoring the GIL-splitting work, but let's be realistic: purely as a concurrency framework, there's at least a dozen more mature/featureful/compelling options in the stdlib and on PyPI, and as an isolation mechanism, subinterpreters have been around for >20 years and in that time they've found 3 users and no previous champions. Obviously the GIL stuff is the only reason PEP 554 might be worth accepting.

Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 exposes existing functionality that hasn't been all that popular (until recently for some reason <wink>) mostly because it is old, was never publicized (until recently), and involved using the C-API. As soon as folks learn about it they want it, for various reasons including (relative) isolation and reduced resource usage in large-scale deployment scenarios. It becomes even more attractive if you say subinterpreters allow you to work around the GIL in a single process, but that isn't the only reason.

Or if PEP 554 is really a good idea on its own merits, purely as a new concurrency API, then why not build that concurrency API on top of multiprocessing and put it on PyPI and let real users try it out?

As I said, the aim of PEP 554 isn't to provide a full concurrency model, though it could facilitate something like CSP. FWIW, there are CSP libraries on PyPI already, but they are limited due to their reliance on threads or multiprocessing.

[snip]

Normally, when people reference this story they focus on the bikeshed, hence the term "bikeshedding". But for PEP 554, you're building a nuclear power plant :-).

:)

The whole conglomeration of a new concurrency API,

As noted, rather than a new concurrency API it provides the basic functionality could be useful to create such a new API. The distinction might be subtle, but it is significant and rests on the expectations set for users.

new subinterpreter support in the interpreter,

There is nothing new here. Subinterpreters have been around for over 20 years and almost nothing has changed with that functionality for many years.

GIL splitting,

The work related to a per-interpreter GIL is almost entirely things that are getting done anyway for other reasons. There are only a few things (like the GIL) that we would not have made per-interpreter anyway.

etc. is stupendously complex,

The project involves lots of little pieces, each supremely tractable. So if by "stupendously complex" you mean "stupendously tedious/boring" then I agree. :) It isn't something that requires a big brain so much as a willingness to stick with it.

and I feel like this means that each piece has gotten way less scrutiny than it would have it if it had to stand on its own.

I apologize if I haven't communicated the nature of this project clearly. There really aren't any significant architectural changes involved. Rather we're moving a lot of little things around in a uniform way. Probably the biggest thing is updating modules for PEPs 3121 and 489, and the several other PEPs (like 573) that have come about as part of that. As I've said, most of things I need done for my project are things others need done for other projects, which is a happy situation for all of us. :) In fact, I'm sure I have actually done relatively little of the work (in case that was a source of concern for you <wink> -- most of the code I've touched is new as part of the PEP 554 implementation). So most of this stuff already does "stand on its own" and has gotten just as much scrutiny as any other work we do all the time in CPython (e.g. PR reviews, BPO discussion).

This seems like a bad outcome, since you'd think that if something is more complex, it should get *more* scrutiny, not less. But it's very hard to effectively reason about the whole conglomeration and formulate feedback.

Do you still have concerns after my explanation above? -eric

It is not clear to me (as an end-user of Python) what is the value of exposing subinterpreters at the Python level except for multi core concurrency, but that requires one GIL per subinterpreter. Paul Moore mentions “alternative code structuring” as a benefit, but I don’t know what that means in concrete terms. If Paul or anyone else would make a case for the subinterpreters sharing a single GIL, that would be great. I understand many PEPs have to do with reducing technical debt in CPython internals, which is great. But PEP 554 is about exposing a feature to Python end users, so without use cases for them I believe this PEP should be put in the back burner until such cases are articulated more clearly for the wider Python community. Thanks, Luciano On Sat, Apr 18, 2020 at 07:09 Paul Moore <p.f.moore@gmail.com> wrote:

On Sat, 18 Apr 2020 at 00:03, Eric Snow <ericsnowcurrently@gmail.com> wrote:

...

On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith <njs@pobox.com> wrote:

...

...

I know you want folks to consider PEP 554 on its own merits, ignoring the GIL-splitting work, but let's be realistic: purely as a concurrency framework, there's at least a dozen more mature/featureful/compelling options in the stdlib and on PyPI, and as an isolation mechanism, subinterpreters have been around for >20 years and in that time they've found 3 users and no previous champions. Obviously the GIL stuff is the only reason PEP 554 might be worth accepting.

Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 exposes existing functionality that hasn't been all that popular (until recently for some reason <wink>) mostly because it is old, was never publicized (until recently), and involved using the C-API. As soon as folks learn about it they want it, for various reasons including (relative) isolation and reduced resource usage in large-scale deployment scenarios. It becomes even more attractive if you say subinterpreters allow you to work around the GIL in a single process, but that isn't the only reason.

As some context in contrast to Nathaniel's view, I remember back around 15 years ago, when I first discovered the subinterpreter support in the C API, I was tremendously interested in it, and very frustrated that it wasn't exposed in Python. (To be fair, I was still using Perl at the time, and it reminded me of Perl's implementation of fork on Windows using subinterpreters, so maybe I was primed to see use cases for it). I never did enough extension coding to use it in C, but I remained interested in it for some time.

This PEP is very exciting for me, precisely because it gives me a chance to explore subinterpreters from Python, which I've been interested in for all those years. I've not really thought of it in terms of concurrency, so the GIL isn't key to me, but more as an alternative code structuring option. I don't know if I'll use it much in production code, but being able to experiment and assess the options is something I'm looking forward to. So I for one am grateful to you for doing this work, and I'm in favour of it being in 3.9, just so I can get my hands on it at last.

I do think that making it initially provisional may be a sensible option, as it gives us the option to refine the Python API while people are still learning how to use the new model.

Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LIGC4BUY... Code of Conduct: http://python.org/psf/codeofconduct/

-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg

On 4/20/2020 6:30 PM, Nathaniel Smith wrote:

We already have robust support for threads for low-isolation and subprocesses for high-isolation. Can you name some use cases where neither of these are appropriate and you instead want an in-between isolation – like subprocesses, but more fragile and with odd edge cases where state leaks between them? I don't know if this has been mentioned before or not, but I'll bring it up now: massively concurrent networking code on Windows.  Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL (provided the GIL actually becomes per-interpreter).  On *nix you can fork, this would give CPython on Windows similar capabilities.

On 4/20/2020 7:33 PM, Nathaniel Smith wrote:

On Mon, Apr 20, 2020 at 4:26 PM Edwin Zimmerman <edwin@211mainstreet.net> wrote:

...

On 4/20/2020 6:30 PM, Nathaniel Smith wrote:

...

We already have robust support for threads for low-isolation and subprocesses for high-isolation. Can you name some use cases where neither of these are appropriate and you instead want an in-between isolation – like subprocesses, but more fragile and with odd edge cases where state leaks between them? I don't know if this has been mentioned before or not, but I'll bring it up now: massively concurrent networking code on Windows. Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL (provided the GIL actually becomes per-interpreter). On *nix you can fork, this would give CPython on Windows similar capabilities. Both Windows and Unix have APIs for passing sockets between related or unrelated processes -- no fork needed. On Windows, it's exposed as the socket.share method: https://docs.python.org/3/library/socket.html#socket.socket.share

The APIs for managing and communicating between processes are definitely not the most obvious or simplest to use, but they're very mature and powerful, and it's a lot easier to wrap them up in a high-level API than it is to effectively reimplement process separation from scratch inside CPython.

-n +1 on not being most obvious or simplest to use.  Not only that, but to use it you have to write Windows-specific code.  PEP 554 would provide a uniform, cross-platform capability that I would choose any day over a random pile of os-specific hacks. --Edwin

On 21/04/20 11:24 am, Edwin Zimmerman wrote:

Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL

I would need convincing that the processing involved things that are truly held up by the GIL, rather than things like blocking I/O calls that can release the GIL. -- Greg

Nathaniel, Your tone and approach to this conversation concern me. I appreciate that you have strong feelings here and readily recognize I have my own biases, but it's becoming increasingly hard to draw any constructive insight from what tend to be very longs posts from you. It ends up being a large commitment of time for small gains. And honestly, it's also becoming hard to not counter some of your more elaborate statements with my own unhelpful prose. In the interest of making things better, please take it all down a notch or two. I apologize if I sound frustrated. I am frustrated, which is only more frustrating because I respect you a lot and feel like your feedback should be more helpful. I'm trying to moderate my responses, but I expect some of my emotion may slip through. :/ On Mon, Apr 20, 2020 at 4:30 PM Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Apr 17, 2020 at 3:57 PM Eric Snow <ericsnowcurrently@gmail.com> wrote: That makes it worse, right? If I wrote a PEP saying "here's some features that could possibly someday be used to make a new concurrency model", that wouldn't make it past the first review.

Clearly, tying this to "concurrency models" is confusing here. So let's just say, as Paul Moore put it, the PEP allows us to "organize" our code in a new way (effectively along the lines of isolated threads with message passing).

I guess your perspective is, subinterpreters are already a CPython feature, so we're not adding anything, and we don't really need to talk about whether CPython should support subinterpreters.

But this simply isn't true. Yes, there's some APIs for subinterpreters added back in the 1.x days, but they were never really thought through, and have never actually worked.

The C-API was thought through more than sufficiently. Subinterpreters are conceptually and practically a very light wrapper around the fundamental architecture of CPython's runtime. The API exposes exactly that, no more, no less. What is missing or broken? They also work fine in most cases. Mostly they have problems with extension modules that have unsafe process-global state and break in some less common cases due to bugs in CPython (which have not been fixed because no one cared enough).

There are exactly 3 users, and all have serious issues, and a strategy for avoiding subinterpreters because of the brokenness. In practice, the existing ecosystem of C extensions has never supported subinterpreters.

Catch-22: why would they ever bother if no one is using them.

This is clearly not a great state of affairs – we should either support them or not support them. Shipping a broken feature doesn't help anyone. But the current status isn't terribly harmful, because the general consensus across the ecosystem is that they don't work and aren't used.

If we start exposing them in the stdlib and encouraging people to use them, though, that's a *huge* change.

You are arguing that this is effectively a new feature. As you noted earlier, I am saying it isn't.

Our users trust us. If we tell them that subinterpreters are a real thing now, then they'll spend lots of effort on trying to support them.

What is "lots"? We've yet to see clear evidence of possible severe impact. On the contrary, I've gotten feedback from folks highly involved in the ecosystem that it will not be a big problem. It won't take care of itself, but it won't require a massive effort.

Since subinterpreters are confusing, and break the C API/ABI

How are they confusing and how do they break either the C-API or C-ABI? This sort of misinformation (or perhaps just miscommunication) is not helpful at all to your argument.

, this means that every C extension author will have to spend a substantial amount of time figuring out what subinterpreters are, how they work, squinting at PEP 489, asking questions, auditing their code, etc.

You make it sounds like tons of work, but I'm unconvinced, as noted earlier. Consider that we regularly have new features for which extensions must provide support. How is this different?

This will take years, and in the mean time, users will expect subinterpreters to work, be confused at why they break, yell at random third-party maintainers, spend days trying to track down mysterious problems that turn out to be caused by subinterpreters, etc. There will be many many blog posts trying to explain subinterpreters and understand when they're useful (if ever), arguments about whether to support them. Twitter threads. Production experiments. If you consider that we have thousands of existing C extensions and millions of users, accepting PEP 554 means forcing people you don't know to collectively spend many person-years on subinterpreters.

Again you're painting a hopeless picture, but so far it's no more than a picture that contrasts with other less negative feedback I've gotten. So yours comes off as unhelpful here.

Random story time: NumPy deprecated some C APIs some years ago, a little bit before I got involved. Unfortunately, it wasn't fully thought through; the new APIs were a bit nicer-looking, but didn't enable any new features, didn't provide any path to getting rid of the old APIs, and in fact it turned out that there were some critical use cases that still required the old API. So in practice, the deprecation was never going anywhere; the old APIs work just as well and are never going to get removed, so spending time migrating to the new APIs was, unfortunately, a completely pointless waste of time that provided zero value to anyone.

Nonetheless, our users trusted us, so lots and lots of projects spend substantial effort on migrating to the new API: figuring out how it worked, making PRs, reviewing them, writing shims to work across the old and new API, having big discussions about how to make the new API work with Cython, debating what to do about the cases where the new APIs were inadequate, etc. None of this served any purpose: they just did it because they trusted us, and we misled them. It's pretty shameful, honestly. Everyone meant well, but in retrospect it was a terrible betrayal of our users' trust.

Now, that only affected projects that were using the NumPy C API, and even then, only developers who were diligent and trying to follow the latest updates; there were no runtime warnings, nothing visible to end-users, etc. Your proposal has something like 100x-1000x more impact, because you want to make all C extensions in Python get updated or at least audited, and projects that aren't updated will produce mysterious crashes, incorrect output, or loud error messages that cause users to come after the developers and demand fixes.

So if the comparison is fair, you are saying: * extension authors will feel tremendous pressure to support subinterpreters * it will be years of work * users won't get much benefit out of subinterpreters I counter that the opposite of all 3 is true.

Now maybe that's worth it. I think on net the Py3 transition was worth it, and that was even more difficult. But Py3 had an incredible amount of scrutiny and rationale. Here you're talking about breaking the C API,

Again no C-API is getting broken.

and your rationales so far are, I'm sorry, completely half-assed. You've never even tried to address the most difficult objections, the rationales you have written down are completely hand-wave-y, and AFAICT none of them stand up to any serious scrutiny.

That's fine. I don't feel like my proposal needs more than what I've written in the PEP. Based on feedback so far (other than yours), my feeling is correct.

(For one random example: have you even measured how much subinterpreters might improve startup time on Windows versus subprocesses? I did, and AFAICT in any realistic scenario it's completely irrelevant – the majority of startup cost is importing modules, not spawning a subprocess, and anyway in any case where subinterpreters make sense to use, startup costs are only a tiny fraction of total runtime. Maybe I'm testing the wrong scenario, and you can come up with a better one. But how are you at the point of asking for PEP acceptance without any test results at all?!)

Performance is not the objective of the PEP nor does it ever suggest that it is. The point is to expose an existing functionality to a broader audience. If performance were a factor then I would agree that demonstration of improvement would be important.

Yes, subinterpreters are a neat idea, and a beautiful dream. But on its own, that's not enough to justify burning up many person-years of our users' lives. You can do better than this, and you need to.

Again, I can just as easily say "it won't cost anyone anything, so it would be irresponsible not to do it".

...

* provides a minimal way to pass information between subinterpreters (which you don't need in C but do in Python code) * adds a few minor conveniences like propagating exceptions and making it easier to share buffers safely

These are a new API, and the current draft does seem like, well, a draft. Probably there's not much point in talking about it until the points above are resolved. But even if CPython should support subinterpreters, it would still be better to evolve the API outside the stdlib until it's more mature. Or at least have some users! Every API sucks in its first draft, that's just how API design works.

The proposed API has gone through at least 5 rounds on python-dev, as well as a lot of careful thought, research, and practical use (by me). So a "first draft" it is not.

...

Are you concerned about users reporting bugs that surface when an incompatible extension is used in a subinterpreter? That shouldn't be a problem if we raise ImportError if an extension that does not support PEP 489 is imported in a subinterpreter.

Making subinterpreter support opt-in would definitely be better than making it opt-out. When C extensions break with subinterpreters, it's often in super-obscure ways where it's not at all clear that subinterpreters are involved.

But notice that this means that no-one can use subinterpreters at all, until all of their C extensions have had significant reworks to use the new API, which will take years and tons of work -- it's similar to the Python 3 transition. Many libraries will never make the jump.

Again, that is a grand statement that makes things sound much worse than they really are. I expect very very few extensions will need "significant reworks". Adding PEP 489 support will not take much effort, on the order of minutes. Dealing with process-global state will depend on how much, if any. Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi?

And why should anyone bother to wait?

...

Saying it's "obviously" the "only" reason is a bit much. :) PEP 554 exposes existing functionality that hasn't been all that popular (until recently for some reason <wink>) mostly because it is old, was never publicized (until recently), and involved using the C-API. As soon as folks learn about it they want it, for various reasons including (relative) isolation and reduced resource usage in large-scale deployment scenarios. It becomes even more attractive if you say subinterpreters allow you to work around the GIL in a single process, but that isn't the only reason.

I'm worried that you might be too close to this, and convincing yourself that there's some pent-up demand that doesn't actually exist. Subinterpreters have always been documented in the C API docs, and they've had decades for folks to try them out and/or improve support if it was useful. CPython has seen *huge* changes in that time, with massive investments on many fronts. But no serious work happened on subinterpreters until you started advocating for the GIL splitting idea.

False. Much of the work that is going on is motivated by desire to improve runtime startup/finalization, to improve embed-ability, and to address the demands of certain large-scale deployments. Most of that work began either before my project started or independently of it.

But anyway, you say here that it's useful for "(relative) isolation and reduced resource usage". That's great, I'm asking for rationale and there's some rationale! Can you expand that into something that's detailed enough to actually evaluate?

What are you after here, exactly? It sounds like you are saying every new feature in Python should go through an exhaustive trial to quantitatively prove its worth before inclusion. That isn't how the PEP process works, for practical reasons. Usually the analysis is much more subjective, under the careful scrutiny of the BDFL-delegate. Are you saying that PEP 554 is exceptional in that regard? Thus far I haven't been able to discern from your feedback any concrete justification for such a qualification.

We already have robust support for threads for low-isolation and subprocesses for high-isolation. Can you name some use cases where neither of these are appropriate and you instead want an in-between isolation – like subprocesses, but more fragile and with odd edge cases where state leaks between them?

Why do you believe that subinterpreters will have reduced resource usage? I assume you're comparing them to subprocesses here. Subinterpreters are "shared-nothing"; all code, data, etc. has to be duplicated, except for static C code ... which is exactly the same as how subprocesses work. So I don't see any theoretical reason why they should have reduced resource usage.

The PEP does not talk about resource usage at all. I do not think that such side effects should influence the decision of accepting or rejecting the PEP. As to actual improvements to resource usage, there are only a few that folks might see currently when using subinterpreters. This includes not running into host limits on resources like #pids, which matters for large-scale uses of Python. Otherwise any other benefits are mostly hypothetical (hence not mentioned in the PEP). Some of the improvements should not require much effort, but mostly require that we already have the base functionality provided by PEP 554.

And theory aside, have you measured the resource usage? Can you share your measurements?

...

...

Or if PEP 554 is really a good idea on its own merits, purely as a new concurrency API, then why not build that concurrency API on top of multiprocessing and put it on PyPI and let real users try it out?

As I said, the aim of PEP 554 isn't to provide a full concurrency model, though it could facilitate something like CSP. FWIW, there are CSP libraries on PyPI already, but they are limited due to their reliance on threads or multiprocessing.

What are these limitations? Can you name some?

Sorry, I don't have any more time to spend looking this up. One lib to look at is https://python-csp.readthedocs.io/en/latest/.

...

...

etc. is stupendously complex,

The project involves lots of little pieces, each supremely tractable. So if by "stupendously complex" you mean "stupendously tedious/boring" then I agree. :) It isn't something that requires a big brain so much as a willingness to stick with it.

I think you're being over-optimistic here :-/.

The two of us have had a number of conversations about this project over the last few years. And as I remember it, I've repeatedly pointed out that there were several fundamental unanswered questions, any one of which could easily sink the whole project, and also a giant pile of boring straightforward work, and I encouraged you to start with the high-risk parts to prove out the idea before investing all that time in the tedious parts. And you've explicitly told me that no, you wanted to work on the easy parts first, and defer the big questions until later.

I'm sure I called them the "trickiest" parts. There are no "fundamental unanswered questions" at this point, just a lot of little things to be done. Mostly the tricky parts entail making the allocators and GIL per-interpreter. Neither will require a lot of direct work and we have a high-level of confidence that it can be done. They are also blocked by just about everything else, so doing them first isn't really an option. Another important thing that needs effort is in helping extension authors support subinterpreters. Feedback on that would be helpful, especially given your experience in certain parts of that ecosystem.

So, well... you're asking for a PEP to be accepted. I think that means it's "later".

None of that applies to the PEP. I continue to argue that the PEP should stand on its own, aside from the work related to the GIL.

And I feel like a bit of a jerk raising these difficult questions, after all the work you and others have poured into this, but... that's kind of what you explicitly decided to set yourself up for? I'm not sure what you were expecting.

Honestly, I appreciate the interest and that you are willing to take an unpopular contrary position. I'm just disappointed about inaccuracies and frustrated by the difficulty in getting more helpful feedback from you. From my perspective you have a lot of helpful insight to offer.

tl;dr: accepting PEP 554 is effectively a C API break, and will force many thousands of people worldwide to spend many hours wrangling with subinterpreter support. And I've spent a ton of time thinking about it, talking to folks about it, etc., over the last few years, and I still just can't see any rationale that stands up to scrutiny. So I think accepting PEP 554 now would be a betrayal of our users' trust, harm our reputation, and lead to a situation where a few years down the road we all look back and think "why did we waste so much energy on that?"

As noted throughout this thread, some of that is not correct and some of it is overstating the challenges. I would love more feedback on how we could mitigate the problems you foresee, rather than a consistent push to abandon all hope. -eric

On 21/04/20 8:34 pm, Ronald Oussoren via Python-Dev wrote:

As far as I understand proper support for subinterpreters also requires moving away from static type definition to avoid sharing objects > between interpreters (that is, use the PyType_FromSpec to build types).

Which means updating every extension module that defines its own Python types and was written before PyType_FromSpec existed. I expect there is a huge number of those. -- Greg

Thanks for explaining that, Ronald. It sounds like a lot of the effort would relate to making classes work. I have some comments in-line below. -eric On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:

...

On 21 Apr 2020, at 03:21, Eric Snow <ericsnowcurrently@gmail.com> wrote: Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi?

Fully supporting sub-interpreters in PyObjC will likely be a lot of work, mostly due to being able to subclass Objective-C classes from Python. With sub- interpreters a Python script in an interpreter could see an Objective-C class in a different sub-interpreter. The current PyObjC architecture assumes that there’s exactly one (sub-)interpreter, that’s probably fixable but is far from trivial.

Are the Objective-C classes immutable? Are the wrappers stateful at all? Without context I'm not clear on how you would be impacted by operation under subinterpreters (i.e. PEP 554), but it sounds like the classes do have global state that is in fact interpreter-specific. I expect you would also be impacted by subinterpreters not sharing the GIL but that is a separate matter (see below). Regardless, I expect there are others in a similar situation. It would be good to understand your use case and help with a solution. Is there a specific example you can point to of code that would be problematic under subinterpreters?

With the current API it might not even be possible to add sub-interpreter support

What additional API would be needed?

(although I write this without having read the PEP).

Currently PEP 554 does not talk about how to make extension modules compatible with subinterpreters. That may be worth doing, though it would definitely have to happen in the docs and (to an extent) the 3.9 "What's New" page. There is already some discussion on what should be in those docs (see https://github.com/ericsnowcurrently/multi-core-python/issues/53). Note that, until the GIL becomes per-interpreter, sharing objects isn't a problem. We were not planning on having a PEP for the stop-sharing-the-GIL effort, but I'm starting to think that it may be worth it, to cover the impact on extension modules (e.g. mitigations). So if you leave out the complications due to not sharing the GIL, the main problem extension authors face with subinterpreters (exposed by PEP 554) is when their module has process-global state that breaks under subinterpreters. From your description above, it sounds like you may be in that situation.

As far as I understand proper support for subinterpreters also requires moving away from static type definitions to avoid sharing objects between interpreters (that is, use the PyType_FromSpec to build types).

Correct, though that is not technically a problem until we stop sharing the GIL.

At first glance this API does not support everything I do in PyObjC (fun with metaclasses, in C code).

What specific additions/changes would you need?

On Wed, Apr 22, 2020 at 2:43 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:

My mail left out some important information, sorry about that.

No worries. :)

PyObjC is a two-way bridge between Python and Objective-C. One half of this is that is bridging Objective-C classes (and instances) to Python. This is fairly straightforward, although the proxy objects are not static and can have methods defined in Python (helper methods that make the Objective-C classes nicer to use from Python, for example to define methods that make it possible to use an NSDictionary as if it were a regular Python dict).

Cool. (also fairly straightforward!)

The other half is that it is possible to implement Objective-C classes in Python:

class MyClass (Cocoa.NSObject): def anAction_(self, sender): …

This defines a Python classes named “MyClass”, but also an Objective-C class of the same name that forwards Objective-C calls to Python.

Even cooler! :)

The implementation for this uses PyGILState_Ensure, which AFAIK is not yet useable with sub-interpreters.

That is correct. It is one of the few major subinterpreter bugs/"bugs" remaining to be addressed in the CPython code. IIRC, there were several proposed solutions (between 2 BPO issues) that would fix it but we got distracted before the matter was settled.

PyObjC also has Objective-C proxy classes for generic Python objects, making it possible to pass a normal Python dictionary to an Objective-C API that expects an NSDictionary instance.

Also super cool. How similar is this to Jython and IronPython?

Things get interesting when combining the two with sub-interpreters: With the current implementation the Objective-C world would be a channel for passing “live” Python objects between sub-interpreters.

+1

The translation tables for looking up existing proxies (mapping from Python to Objective-C and vice versa) are currently singletons.

This is probably fixable with another level of administration, by keeping track of the sub-interpreter that owns a Python object I could ensure that Python objects owned by a different sub-interpreter are proxied like any other Objective-C object which would close this loophole. That would require significant changes to a code base that’s already fairly complex, but should be fairly straightforward.

Do you think there are any additions we could make to the C-API (more than have been done recently, e.g. PEP 573) that would make this easier. From what I understand, this pattern of a cache/table of global Python objects is a relatively common one. So anything we can do to help transition these to per-interpreter would be broadly beneficial. Ideally it would be done in the least intrusive way possible, reducing churn and touch points. (e.g. a macro to convert existing tables,etc. + an init func to call during module init.) Also, FWIW, I've been thinking about possible approaches where the first/main interpreter uses the existing static types, etc. and further subinterpreters use a heap type (etc.) derived mostly automatically from the static one. It's been on my mind because this is one of the last major hurdles to clear in the CPython code before we can make the GIL per-interpreter.

...

What additional API would be needed?

See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of time thinking about this though, I might find other issues when I try to support sub-interpreters.

Any feedback on this would be useful.

...

...

As far as I understand proper support for subinterpreters also requires moving away from static type definitions to avoid sharing objects between interpreters (that is, use the PyType_FromSpec to build types).

Correct, though that is not technically a problem until we stop sharing the GIL.

Right. But a major selling point of sub-interpreters is that this provide a way forward towards having multiple Python threads that don’t share a GIL.

IMHO it would be better to first work out what’s needed to get there, and in particular what changes are needed in extensions. Otherwise extensions may have to be changed multiple times.

Yeah, I see what you're saying. It's been a hard balance to strike. There are really 2 things that have to be done: move all global state to per-interpreter and deal with static types (etc.). Do you think both will require significant work in the community? My instinct, partly informed by my work in CPython along these lines, is that the former is more work and sometimes trickier. The latter is fairly straightforward and much more of an opportunity for automatic approaches.

...

...

At first glance this API does not support everything I do in PyObjC (fun with metaclasses, in C code).

What specific additions/changes would you need?

At least:

- A variant of PyGILState_Ensure that supports sub-interpreters - Defining subclasses of built-in types using PyType_FromSpec, in particular a subclass of “type”.

Thanks!

BTW. In my first mail I mentioned I don’t have a use cases for subinterpreters. I might have a limited use case in the PyObjC domain: implementing plugin bundles for Objective-C applications in Python. These currently share the same interpreter, which can cause problems. Subinterpreters could be helpful there to isolate code, but that would require having an API that conditionally initialises the Python runtime (similar to PyGILState_Ensure, but for the runtime itself).

Ah, interesting.

This wouldn’t fix all problems because you can’t have two different python versions in one proces, but would be better than the status quo.

FWIW, I've been involved in past discussions about the idea of supporting multiple Python runtimes (including difference versions) in the same process. :) On top of that, I know someone who demonstrated to me how to use dlmopen() to do something like this right now. :) -eric

Thanks for the great insights into PyObjC! On Wed, Apr 29, 2020 at 9:02 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:

I don’t know how much the move of global state to per-interpreter state affects extensions, other than references to singletons and static types.

That's the million dollar question. :) FYI, one additional challenge is when an extension module depends on a third-party C library which itself keeps global state which might leak between subinterpreters. The Cryptography project ran into this several years ago with OpenSSL and they were understandably grumpy about it.

But with some macro trickery that could be made source compatible for extensions.

Yeah, that's one approach that we've discussed in the past (e.g. at the last core sprint). -eric

On Tue, Apr 21, 2020 at 3:10 AM Antoine Pitrou <solipsis@pitrou.net> wrote:

On Mon, 20 Apr 2020 19:21:21 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:

...

Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi?

A slightly tricky question is what happens if a PyObject survives longer than the subinterpreter that created it.

For example, in PyArrow, we allow passing a Python buffer-like object as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept around by C++ code for some time. When the C++ buffer is destroyed, Py_DECREF() is called on the Python object (I figure that we would have to switch to the future interpreter-aware PyGILState API -- when will it be available?). But what happens if the interpreter which created the Python object is already destroyed at this point?

Good question (and example). I'm not familiar enough with the GC machinery to have a good answer. I do know that GC is per-interpreter now, but the allocators are still global. I'm not sure how object lifetime is *currently* tied to interpreter lifetime. I'd expect that, at the point we have per-interpreter allocators, that all objects in that interpreter would be destroyed when the interpreter is destroyed. -eric

On Tue, 21 Apr 2020 18:46:04 +0200 Petr Viktorin <encukou@gmail.com> wrote:

On 2020-04-21 11:01, Antoine Pitrou wrote:

...

On Mon, 20 Apr 2020 19:21:21 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:

...

Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi?

A slightly tricky question is what happens if a PyObject survives longer than the subinterpreter that created it.

For example, in PyArrow, we allow passing a Python buffer-like object as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept around by C++ code for some time. When the C++ buffer is destroyed, Py_DECREF() is called on the Python object (I figure that we would have to switch to the future interpreter-aware PyGILState API -- when will it be available?).

...

But what happens if the interpreter which created the Python object is already destroyed at this point?

That's a good question. What happens today if someone calls Py_Finalize while the buffer is still around?

I don't think you need to treat *sub*interpreters specially.

The difference is that subinterpreters may have a shorter lifetime than the main interpreter. Also, we're currently using a global memory allocator, so simple operations may work until the process dies. Regards Antoine.

On Tue, 21 Apr 2020 12:05:28 -0700 "Gregory P. Smith" <greg@krypto.org> wrote:

On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou <solipsis@pitrou.net> wrote:

...

On Tue, 21 Apr 2020 18:46:04 +0200 Petr Viktorin <encukou@gmail.com> wrote:

...

On 2020-04-21 11:01, Antoine Pitrou wrote:

...

On Mon, 20 Apr 2020 19:21:21 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:

...

Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi?

A slightly tricky question is what happens if a PyObject survives longer than the subinterpreter that created it.

For example, in PyArrow, we allow passing a Python buffer-like object as a C++ buffer to C++ APIs. The C++ buffer could conceivably be kept around by C++ code for some time. When the C++ buffer is destroyed, Py_DECREF() is called on the Python object (I figure that we would have to switch to the future interpreter-aware PyGILState API -- when will it be available?).

In this scenario given that the PyObject owning the buffer passed to C++ by PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release anything who's refcount never made it to 0. Which also implies leaving enough of the interpreter state around so that Py_DECREF could be called and trigger the free().

That's assuming a trivial destructor. Of course, that would cater for most concrete use cases. Regards Antoine.

On Mon, 20 Apr 2020 15:30:37 -0700 Nathaniel Smith <njs@pobox.com> wrote:

tl;dr: accepting PEP 554 is effectively a C API break, and will force many thousands of people worldwide to spend many hours wrangling with subinterpreter support.

For the record, that's not my reading of the PEP. PEP 554 doesn't mandate that C extensions be rewritten to be subinterpreter-compliant. It makes that an ideal, not an obligation. Regards Antoine.

On Tue, 21 Apr 2020 at 15:31, Eric Snow <ericsnowcurrently@gmail.com> wrote:

Here are the options for handling non-compliant extension modules:

1. do nothing (extensions may break in ways that are hard for users to deal with) 2. raise ImportError if an extension without PEP 489 support is imported in a subinterpreter 3. add a per-interpreter isolated mode (default) that provides the behavior of #2, where non-isolated gives #1

At this point my plan was go with #2, to avoid giving extension authors the pain that comes with #1. If we went with #3, at least during the provisional phase, it would allow extension authors to try out their extensions in subinterpreters.

This seems sensible to me - #2 ensures that we don't get weird bugs from users trying things that we know they shouldn't be doing. I'm not sure how much advantage #3 would have, presumably extension authors wanting to "try out" subinterpreters would at some point have to mark them as importable as per #2, so why not just do that much and then see what else needs changing? But I don't know the C API for subinterpreters, so maybe there are complexities I don't understand, in which case having #3 probably makes more sense in that context. Paul

Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith <njs@pobox.com> a écrit :

Why do you believe that subinterpreters will have reduced resource usage? I assume you're comparing them to subprocesses here. Subinterpreters are "shared-nothing"; all code, data, etc. has to be duplicated, except for static C code ... which is exactly the same as how subprocesses work. So I don't see any theoretical reason why they should have reduced resource usage.

As you, I would like to see benchmarks of subinterpreters running in parallel before saying that "yeah, that approach works to speed up this specific use case". But I don't think that it should hold the PEP 554. Eric is transparent about his intent and limitations of the current implementation. There is no silver bullet for parallelism and concurrency. For example, multiprocessing uses threads in addition to processes, and asyncio also uses threads internally to execute blocking code like DNS resolution. IMHO it's worth it to explore the subinterpreters approach. I don't think that it's going to be a major maintenance burden: the code is already written and tests. The PEP is only about adding a public Python API.

tl;dr: accepting PEP 554 is effectively a C API break, and will force many thousands of people worldwide to spend many hours wrangling with subinterpreter support.

I fail to follow your logic. When the asyncio PEP was approved, I don't recall that suddenly the whole Python community started to rewrite all projects to use coroutines everywhere. I tried hard to replace eventlet with asyncio in OpenStack and I failed because such migration was a very large project with dubious benefits (people impacted by eventlet issues were the minority). When asyncio landed in Python 3.4, a few people started to experiment it. Some had a bad experience. Some others were excited and put a few applications in production. Even today, asyncio didn't replace threads, multiprocessing, concurrent.futures, etc. There are even competitor projects like Twisted, trio and curio! (Also eventlet and gevent based on greenlet which is a different approach). I only started to see very recently project like httpx which supports both blocking and asynchronous API. I see a slow adoption of asyncio because asyncio solves very specific use cases. And that's fine! I don't expect that everyone will suddenly spend months of work to rewrite their C code and Python code to be more efficient or fix issues with subinterpreters, until a critical mass of users proved that subinterpreters are amazing and way more efficient! Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Tue, 2020-04-21 at 11:21 -0500, Sebastian Berg wrote:

...

Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith <njs@pobox.com> a écrit : <snip> <snip> As far as I can tell, nobody can or _should_ expect subinterpreters to actually run most general python code for many years. Yes, its a chicken-and-egg problem, unless users start to use subinterpreters successfully, C-extensions should probably not even worry to

On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote: transition. This PEP wants to break the chicken-and-egg problem to have a start, but as of now, as far as I can tell, it *must not* promise that it will ever work out.

So, I cannot judge the sentiment or subinterpreters. But it may be good to make it *painfully* clear what you expect from a project like NumPy

Maybe one of the frustrating points about this criticism is that it does not belong in this PEP. And that is actually true! I wholeheartedly agree that it doesn't really belong in this PEP itself. *But* the existence of a document detailing the "state and vision for subinterpreters" that includes these points is probably a prerequisite for this PEP. And this document must be linked prominently from the PEP. So the suggestion should maybe not be to discuss it in the PEP, but to to write it either in the documentation on subinterpreters or as an informational PEP. Maybe such document already exists, but then it is not linked prominently enough probably. - Sebastian

in the next few years. Alternatively, make it painfully clear that you possibly even discourage us from spending time on it now, if its not straight forward. Those using this module are on their own for many years, probably even after success is proven.

Best,

Sebastian

[1] As of now, the way I see it is that I could not even make NumPy (and probably many C extensions) work, because I doubt that the limited API has been exercised enough [2] and I am pretty sure it has holes. Also the PEP about passing module state around to store globals efficiently seems necessary, and is not in yet? (Again, trust: I have to trust you that e.g. what you do to make argument parsing not have overhead in argument clinic will be something that I can use for similar purposes within NumPy)

[2] I hope that we will do (many) these changes for other reasons within a year or so, but they go deep into code barely touched in a decade. Realistically, even after the straight forward changes (such as using the new PEPs for module initialization), these may take up an additional few months of dev time (sure, get someone very good or does nothing else, they can do it much quicker maybe). So yes, from the perspective of a complex C-extension, this is probably very comparable to the 2to3 change (it happened largely before my time though).

[3] E.g. I think I want an ExtensionMetaClass, a bit similar as an ABC, but I would prefer to store the data in a true C-slot fashion. The limited API cannot do MetaClasses correctly as far as I could tell and IIRC is likely even a bit buggy. Are ExtensionMetaClasses crazy? Maybe, but PySide does it too (and as far as I can tell, they basically get away with it by a bit of hacking and relying on Python implementation details.

...

When asyncio landed in Python 3.4, a few people started to experiment it. Some had a bad experience. Some others were excited and put a few applications in production.

Even today, asyncio didn't replace threads, multiprocessing, concurrent.futures, etc. There are even competitor projects like Twisted, trio and curio! (Also eventlet and gevent based on greenlet which is a different approach). I only started to see very recently project like httpx which supports both blocking and asynchronous API.

I see a slow adoption of asyncio because asyncio solves very specific use cases. And that's fine!

I don't expect that everyone will suddenly spend months of work to rewrite their C code and Python code to be more efficient or fix issues with subinterpreters, until a critical mass of users proved that subinterpreters are amazing and way more efficient!

Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3MK2NANM... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ONEVR72K... Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 2020-04-28 at 19:20 -0600, Eric Snow wrote:

On Tue, Apr 21, 2020 at 11:17 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:

...

Maybe one of the frustrating points about this criticism is that it does not belong in this PEP. And that is actually true! I wholeheartedly agree that it doesn't really belong in this PEP itself.

*But* the existence of a document detailing the "state and vision for subinterpreters" that includes these points is probably a prerequisite for this PEP. And this document must be linked prominently from the PEP.

So the suggestion should maybe not be to discuss it in the PEP, but to to write it either in the documentation on subinterpreters or as an informational PEP. Maybe such document already exists, but then it is not linked prominently enough probably.

That is an excellent point. It would definitely help to have more clarity about the feature (subinterpreters). I'll look into what makes the most sense. I've sure Victor has already effectively written something like this. :)

I will note one more time that I want to back up almost all that Nathaniel said (I simply cannot judge the technical side though). While I still think it is probably not part of PEP 554 as such, I guess it needs a full blown PEP on its own. Saying that Python should implement subinterpreters. (I am saying "implement" because I believe you must consider subinterpreters basically a non-feature at this time. It has neither users nor reasonable ecosystem support.) In many ways I assume that a lot of the ground work for subinterpreters was useful on its own. But please do not underestimate how much effort it will take to make subinterpreters first class citizen in the language! Take PyPy for example, it took years for PyPy support in NumPy (and the PyPy people did pretty much _all_ the work). And PyPy provides a compatibility layer that makes the support orders of magnitude simpler than supporting subinterpreters. And yet, I am sure there are many many C-Extensions out there that will fail on PyPy. So unless the potential subinterpreter userbase is magnitudes larger than PyPy's the situation will be much worse. With the added frustration because PyPy users probably expect incompatibilities, but Python users may get angry if they think subinterpreters are a language feature. There have been points made about e.g. just erroring on import for modules which do not choose to support subinterpreters. And maybe in the sum of saying: * We warn that most C-extensions won't work -> If you error, at least it won't crash silently (some mitigation) * Nobody must expect any C-extension to work until subinterpreters have proven useful *and* a large userbase! * In times, we are taking here about, what? - Maybe 3 years until proven useful and a potentially large userbase? - Some uncertain amount longer until the user-base actually grows - Maybe 5 years until fairly widespread support for some central libraries after that? (I am sure Python 2 to 3 took that long) * Prototyping the first few years (such as an external package, or even a fork!) are not really very good, because... ? Or alternatively the warnings will be so penetrant that prototyping within cpython is acceptable. Maybe you have to use: python --subinterpreters-i-know-this-is-only-a-potential-feature-which-may-be-removed-in-future-python-versions myscript.py This is not in the same position as most "experimental" APIs, which are almost settled but you want to be careful. This one has a real chance of getting ripped out entirely!? is good enough, maybe it is not. Once it is written down I am confident the Python devs and steering council will make the right call. Again, I do not want to hinder the effort. It takes courage and a good champion to go down such a long and windy road. But Nathaniel is right that putting in the effort puts you into the trap of thinking that now that we are 90% there from a Python perspective, we should go 100%. 100% is nice, but you may have to reach 1000++% (i.e. C-extension modules/ecysystem support) to actually have a fully functional feature. You should get the chance to prove them useful, there seems enough positivity around the idea. But the question is within what framework that can reasonably happen. Simply pushing in PEP 554 with a small warning in the documentation is not the right framework. I hope you can find the right framework to push this on. But unfortunately it is the difficult job of the features champion. And Nathaniel, etc. are actually trying to help you with it (but in the end are not champions for it, so you cannot expect too much). E.g. by asking if it cannot be developed outside of cpython and pointing out how other similar project approached the same impassible mountain of work. Believe me, I have been there and its tough to write these documents and then get feedback which you are not immediately sure what to make of. Thus, I hope those supporting the idea of subinterpreters will help you out and formulate a better framework and clarify PEP 554 when it comes to the fuzzy long term user-impact side of the PEP. - Sebastian PS: Sorry for the long post...

-eric _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2W2GOKCU... Code of Conduct: http://python.org/psf/codeofconduct/

On 22/04/20 4:21 am, Sebastian Berg wrote:

Sure, but this is very different. You can still use NumPy in a project using asyncio. You are _not_ able to use NumPy in a project using subinterpreters.

To put it another way, the moment you start using subinterpreters, the set of extension modules you are able to use will shrink *enormously*. And if I understand correctly, you won't get any nice "This module does not support subinterpreters" exception if you import an incompatible module -- just an obscure crash, probably of the core-dumping variety. To me this would feel like programming in the middle of a lethal minefield. -- Greg

On 2020-04-22 08:05, Glenn Linderman wrote:

On 4/21/2020 10:26 PM, Greg Ewing wrote:

...

And if I understand correctly, you won't get any nice "This module does not support subinterpreters" exception if you import an incompatible module -- just an obscure crash, probably of the core-dumping variety.

This sounds fixable: modules that support subinterpreters should set a flag saying so, and the either the load of a non-flagged module when subinterpreters are in use, or the initiation of a subinterpreter when a non-flagged module has been loaded, should raise.

There was talk about making a orthogonal flag just for opting into subinterpreter support. But what you use now as such a flag is multi-phase initialization: https://docs.python.org/3/c-api/module.html?highlight=pymodule_fromdefandspe... (Though I like the PEP 489 wording, "expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly", better than what ended up in the docs.) The simplest strategy to support subinterpreters correctly is to refuse to create the extension module more than once per process (with an appropriate error, of course).

On Tue, Apr 21, 2020 at 10:24 AM Sebastian Berg <sebastian@sipsolutions.net> wrote:

On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote:

...

I fail to follow your logic. When the asyncio PEP was approved, I don't recall that suddenly the whole Python community started to rewrite all projects to use coroutines everywhere. I tried hard to replace eventlet with asyncio in OpenStack and I failed because such migration was a very large project with dubious benefits (people impacted by eventlet issues were the minority).

Sure, but this is very different. You can still use NumPy in a project using asyncio. You are _not_ able to use NumPy in a project using subinterpreters.

True. Is that a short-term problem? I don't know. A long-term problem? Definitely. So it will have to be addressed at some point. The biggest concern here is what is the resulting burden on extension authors and what can we do to help mitigate that. The first step is to understand what that burden might entail.

Right now, I have to say as soon as the first bug report asking for this is opened and tells me: But see PEP 554 you should support it! I would be tempted to put on the NumPy Roadmap/Vision that no current core dev will put serious efforts into subinterpreters. Someone is bound to be mad.

Yeah. And I don't want to put folks in the position that they get fussed at for something like this. This isn't a ploy to force projects like numpy to fix their subinterpreter support. My (honest) question is, how many folks using subinterpreters are going to want to use numpy (or module X) enough to get mad about it before the extension supports subinterpreters? What will user expectations be when it comes to subinterpreters? We will make the docs as clear as we can, but there are plenty of users out there that will not pay enough attention to know that most extension modules will not support subinterpreters at first. Is there anything we can do to mitigate this impact? How much would it help if the ImportError for incompatible modules give a clear (though lengthier) explanation of the situation?

Basically, if someone wants it in NumPy, I personally may expect them to be prepared to invest a year worth of good dev time [1]. Maybe that is pessimistic, but your guess is as good as mine. At normal dev-pace it will be at least a few years of incremental changes before NumPy might be ready (how long did it take Python?)?

The PEP links to NumPy bugs, I am not sure that we ever fixed a single one. Even if, the remaining ones are much larger and deeper. As of now, the NumPy public API has to be changed to even start supporting subinterpreters as far as I aware [2]. This is because right now we sometimes need to grab the GIL (raise errors) in functions that are not passed GIL state.

What do you expect to have to change? It might not be as bad as you think...or I suppose it could be. :) Keep in mind that subinterpreter support means making sure all of the module's global state is per-interpreter. I'm hearing about things like passing around GIL state and using the limited C-API. None of that should be a factor.

This all is not to say that this PEP itself doesn't seem harmless. But the _expectation_ that subinterpreters should be first class citizens will be a real and severe transition burden. And if it does not, the current text of the PEP gives me, as someone naive about subinterpreters, very few reasons why I should put in that effort or reasons to make me believe that it actually is not as bad a transition as it seems.

Yeah, the PEP is very light on useful information extension module maintainers. What information do you think would be most helpful?

Right now, I would simply refuse to spend time on it. But as Nathaniel said, it may be worse if I did not refuse and in the end only a handful of users get anything out of my work: The time is much better spend elsewhere. And you, i.e. CPython will spend your "please fix your C- extension" chips on subinterpreters. Maybe that is the only thing on the agenda, but if it is not, it could push other things away.

Good point.

Reading the PEP, it is fuzzy on the promises (the most concrete I remember is that it may be good for security relevant reasons), which is fine, because the goal is "experimentation" more than use?

The PEP is definitely lacking clear details on how folks might use subinterpreters (via the proposed module). There are a variety of reasons. I originally wrote the PEP mostly as "let's expose existing functionality more broadly", with the goal of getting it into folks' hands sooner rather than later. My focus was mostly on the API. I didn't see a strong need to convince anyone that the feature itself was worth it (since it already existed). In many ways the PEP is a side effect of my efforts to achieve a good multi-core Python story (via a per-interpreter GIL). All the relevant parties in that effort saw PEP 554 as worth it for that, so there wasn't much pressure to elaborate. At the same time, in the PEP I tried to avoid the GIL connection because I personally think subinterpreters offer a meaningful value already, even while they share the GIL, and that the PEP should stand on its own merits. I just didn't develop that aspect very far. On top of that, ultimately a PEP is a vehicle to aid a BDFL-delegate in making a decision, so I usually write PEPs with a specific audience. Put all that together, along with the limited time I have, and you get a lackluster explanation of the benefits from subinterpreters. At this point I'm torn on spending more time on fleshing that out. I want folks to get inspired, but at the same time I have to ration my open-source time carefully.

So if its more about "experimentation", then I have to ask, whether:

1. The PEP can state that more obviously, it wants to be provisionally/experimentally accept? So maybe it should even say that that extension modules are not (really) encouraged to transition unless they feel a significant portion of their users will gain.

That sounds fair. Then extension authors can point concerned users to the docs. The docs would say something like "Support for subinterpreters in extension modules should not be expected while this module is provisional." That isn't the only thing we should do to set user expectations properly, but it would help. (FWIW, for extension maintainers we would likely also have a link on the "interpreters" module docs pointing to the how-to-support-subinterpreters-in-an-extension-module page.)

2. The point about developing it outside of the Python standard lib should be considered more seriously. I do not know if that can be done, but C-API additions/changes/tweaks seem a bit orthogonal to the python exposure? So maybe it actually is possible?

It is possible. The low-level implementation is an extension module that uses the public C-API exclusively. Using the internal C-API would be a little dishonest. That said, the module is very closely tied to the CPython runtime, so it makes sense to keep it in the CPython repo. Furthermore, I don't see much value in keeping it out of the stdlib. Finally, a few of us have plans for using subinterpreters in CPython's test suite. I suppose we could keep the module's code and tests in the CPython repo, while releasing it separately, but I don't see the point.

As far as I can tell, nobody can or _should_ expect subinterpreters to actually run most general python code for many years.

Subinterpreters run all Python code right now. I'm guessing by "general python code" you are talking about the code folks are writing plus their dependencies. In that case, it's only with extension modules that we run into a problem, and we still don't know with how many of those it's a problem where it will take a lot of work. However, I *am* convinced that there is a non-trivial amount of work there and that it impacts large extension modules more than others. The question is, what can we do to mitigate the amount of work there?

Yes, its a chicken-and-egg problem, unless users start to use subinterpreters successfully, C-extensions should probably not even worry to transition. This PEP wants to break the chicken-and-egg problem to have a start, but as of now, as far as I can tell, it *must not* promise that it will ever work out.

I see where you're coming from but think it's highly unlikely at this point in CPython's life that subinterpreters (as a public feature) will ever go away. I'm definitely biased :) but it would be hard to justify removing a feature that has been publicly available for most of Python's existence. We know of a small number of public projects that already use subinterpreters through the C-API and expect there is a not insignificant number of private projects that rely on the feature. (That doesn't even consider the projects that tried to use subinterpreters but couldn't move past the limitations (which are mostly due to lack of use).)

So, I cannot judge the sentiment or subinterpreters. But it may be good to make it *painfully* clear what you expect from a project like NumPy in the next few years. Alternatively, make it painfully clear that you possibly even discourage us from spending time on it now, if its not straight forward. Those using this module are on their own for many years, probably even after success is proven.

Thanks for the feedback!

[1] As of now, the way I see it is that I could not even make NumPy (and probably many C extensions) work, because I doubt that the limited API has been exercised enough [2] and I am pretty sure it has holes. Also the PEP about passing module state around to store globals efficiently seems necessary, and is not in yet? (Again, trust: I have to trust you that e.g. what you do to make argument parsing not have overhead in argument clinic will be something that I can use for similar purposes within NumPy)

The relevant API is on its way. We'd be glad to pursue C-API improvements to facilitate efficient support for subinterpreters in extension modules. FWIW, often such improvements provide other benefits that are desirable on their own.

[2] I hope that we will do (many) these changes for other reasons within a year or so,

That's good to know. What specific changes are you planning for? Are there documents or issue #s you could point us at? This sort of thing may be helpful to identify how we can assist from the CPython side.

but they go deep into code barely touched in a decade. Realistically, even after the straight forward changes (such as using the new PEPs for module initialization), these may take up an additional few months of dev time (sure, get someone very good or does nothing else, they can do it much quicker maybe). So yes, from the perspective of a complex C-extension, this is probably very comparable to the 2to3 change (it happened largely before my time though).

Also good to know. I'm hopeful that nothing will ever be as disruptive as the 2/3 transition. :) I can see what you mean on a project-by-project basis though.

[3] E.g. I think I want an ExtensionMetaClass, a bit similar as an ABC, but I would prefer to store the data in a true C-slot fashion. The limited API cannot do MetaClasses correctly as far as I could tell and IIRC is likely even a bit buggy. Are ExtensionMetaClasses crazy? Maybe, but PySide does it too (and as far as I can tell, they basically get away with it by a bit of hacking and relying on Python implementation details.

That's an interesting idea. How much would that help with subinterpreter support? Regardless, you should consider bringing this up separately on the capi-sig mailing list. Thanks again! -eric

On Wed, Apr 29, 2020 at 1:52 AM Paul Moore <p.f.moore@gmail.com> wrote:

One thing that isn't at all clear to me here is that when you say "Subinterpreters run all Python code", do you *just* mean the core language? Or the core language plus all builtins? Or the core language, builtins and the standard library? Because I think that the vast majority of users would expect a core/stdlib function like subinterpreters to support the full core+stdlib language.

Agreed.

So my question would be, do all of the stdlib C extension modules support subinterpreters[1]? If they don't, then I think it's very reasonable to expect that to be fixed, in the spirit of "eating our own dogfood" - if we aren't willing or able to make the stdlib support subinterpreters, it's not exactly reasonable or fair to expect 3rd party extensions to do so.

That is definitely the right question. :) Honestly I had not thought of it that way (nor checked of course). While many stdlib modules have been updated to use heap types (see PEP 384) and support PEP 489 (Multi-phase Extension Module Initialization), there are still a few stragglers. Furthermore, I expect that there are few modules that would give us trouble (maybe ssl, cdecimal). It's all about global state that gets shared inadvertently between subinterpreters. Probably the best way to find out is to run the entire test suite in a subinterpreter. I'll do that as soon as I can.

If, on the other hand, the stdlib *is* supported, then I think that "all of Python and the stdlib, plus all 3rd party pure Python packages" is a significant base of functionality, and an entirely reasonable starting point for the feature.

Yep, that's what I meant. I just need to identify modules where we need fixes. Thanks for bringing this up!

It certainly still excludes big parts of the Python ecosystem (notably scientific / data science users) but that seems fine to me - big extension users like those can be expected to have additional limitations. It's not really that different from the situation around C extension support in PyPy.

Agreed. -eric

On Wed, Apr 29, 2020 at 5:40 AM Julien Salort <listes@salort.eu> wrote:

Le 29/04/2020 à 03:18, Eric Snow a écrit :

...

My (honest) question is, how many folks using subinterpreters are going to want to use numpy (or module X) enough to get mad about it before the extension supports subinterpreters? What will user expectations be when it comes to subinterpreters?

We will make the docs as clear as we can, but there are plenty of users out there that will not pay enough attention to know that most extension modules will not support subinterpreters at first. Is there anything we can do to mitigate this impact? How much would it help if the ImportError for incompatible modules give a clear (though lengthier) explanation of the situation?

For what it's worth, I can give you the feedback of a simple user. It happens that I tried some time ago to use Numpy in a Flask project which was deployed with mod_wsgi on an Apache server. Basically, the page was dynamically generating some plots. And I got weird unreliable behaviour, which took me some time to debug.

I had to look it up on the internet to figure out the problem was that Numpy cannot reliably work with mod_wsgi. I originally thought that I had made a mistake somewhere in my code instead. So, I rewrote the code to remove the dependency on Numpy. I had used Numpy in the first place, because, as a physicist, this is what I am used to, but it was clearly very possible to rewrite this particular code without Numpy.

If your proposal leads to an intelligible actual error, and a clear warning in the documentation, instead of a silent crash, this sounds like progress, even for those packages which won't work on subinterpreters anytime soon...

+10 to this, the mysterious failures of today are way worse than a clear "this module doesn't support this execution environment" ImportError. I'm not worried at all about someone going and filing an issue in a project saying "please support this execution environment". Someone may eventually come along and decide they want to be the one to make that happen and do the work because they have a reason. -gps

On Tue, Apr 21, 2020 at 7:33 AM Victor Stinner <vstinner@python.org> wrote:

IMHO it's worth it to explore the subinterpreters approach. I don't think that it's going to be a major maintenance burden: the code is already written and tests. The PEP is only about adding a public Python API.

While this PEP may not create a maintenance burden for CPython, it does have the effect of raising the complexity bar for an alternative Python implementation.

Even today, asyncio didn't replace threads, multiprocessing, concurrent.futures, etc. There are even competitor projects like Twisted, trio and curio! (Also eventlet and gevent based on greenlet which is a different approach). I only started to see very recently project like httpx which supports both blocking and asynchronous API.

Because asyncio had been implemented as a library, the up-take of asyncio could have been lower because the demand was fulfilled, at least in part, by those third-party libraries? A thought that may have already been mentioned elsewhere: perhaps the PEP could be more made more acceptable by de-scoping it to expose a minimal set of C-API hooks to enable third-party libraries for the sub-interpreter feature rather than providing that feature in the standard library?

Hi Nathaniel, Thanks for your very interesting analysis :-) Le ven. 17 avr. 2020 à 23:12, Nathaniel Smith <njs@pobox.com> a écrit :

- The asyncio designers (esp. Guido) did a very extensive analysis of these libraries' design choices, spoke to the maintainers about what they'd learned from hard experience, etc.

asyncio and Twisted could perfectly live outside CPython code base. asyncio even started as a 3rd party project named Tulip for Python 3.3! Subinterpreters are different: the implementation rely a lot on CPython internals. Developing it outside CPython would be way harder if not impossible. For example, Eric had to add PyThreadState.id and PyInterpreterState.id members. He made many changes in Py_NewInterpreter() and Py_EndInterpreter() which are part of Python internals. Moreover, there are many large refactoring changes which are required to properly implement subinterpreters, especially to better isolate each others: _PyRuntimeState structure to better identify shared globals, PyConfig API (PEP 587), move more structures per-interpreter (GC state, warnings state, pending calls, etc.).

- Even today, the limitations imposed by the stdlib release cycle still add substantial difficulty to maintaining asyncio

From what I understood, asyncio has a very large API since it has tons of features (subprocesses, transports and protocols, streams, futures and tasks, etc.) and requires a lot of glue code to provide the same API on all platforms which is really hard. I understood that the subinterpreters module is very light: just a few functions to provide "channels" and that's it (in short). $ ./python Python 3.9.0a5+ (heads/master:6a9e80a931, Apr 21 2020, 02:28:15)

...

...

import _xxsubinterpreters len(dir(_xxsubinterpreters)) 30

...

...

import asyncio len(dir(asyncio)) 122

...

...

dir(_xxsubinterpreters) ['ChannelClosedError', 'ChannelEmptyError', 'ChannelError', 'ChannelID', 'ChannelNotEmptyError', 'ChannelNotFoundError', 'InterpreterID', 'RunFailedError', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_channel_id', 'channel_close', 'channel_create', 'channel_destroy', 'channel_list_all', 'channel_recv', 'channel_release', 'channel_send', 'create', 'destroy', 'get_current', 'get_main', 'is_running', 'is_shareable', 'list_all', 'run_string']

OTOH, AFAICT the new concurrency model in PEP 554 has never actually been used, and it isn't even clear whether it's useful at all.

Antoine (BDFL-delegate of the PEP) wants to mark the PEP as provisional: it means that we *can* still remove the whole module later if we decide that it's not worth it. The same was done with asyncio. By the way, Antoine was also the BDFL-delegate of the heavy PEP 3156 (asyncio) ;-)

Designing useful concurrency models is *stupidly* hard. And on top of that, it requires major reworks of the interpreter internals + disrupts the existing C extension module ecosystem -- which is very different from asyncio, where folks who didn't use it could just ignore it.

A lot of work done to isolate subinterpreters also helps to "fix" CPython internals. For example, converting C extension modules to multiphase initialization (PEP 489) and add a module state helps to destroy all module objects at exit and also helps to break reference cycles in the garbage collector. It helps to fix this issue created in 2007: "Py_Finalize() doesn't clear all Python objects at exit" https://bugs.python.org/issue1635741 It also enhances the implementation of the "embed Python" use case: "leak" less objects at exit, better cleanup things, etc. That's why I'm supporter of the overall project: even if subinterpreters never take off, the implementation will make CPython better ;-) FYI it even helps indirectly my project to clean the C API ;-) PEP 554 implementation looks quite small: 2 800 lines of C code (Modules/_xxsubinterpretersmodule.c and Objects/interpreteridobject.c) and 2 100 lines of tests (Lib/test/test__xxsubinterpreters.py). Compare it to asyncio: 13 000 lines (Lib/asyncio/*.py) of Python and 20 800 lines of tests (Lib/test/test_asyncio/*.py). Victor -- Night gathers, and now my watch begins. It shall not end until my death.