Mailman 3 April 2020 - Python-Dev

Virtual machine bleeds into generator implementation?
by Skip Montanaro 30 Apr '20

30 Apr '20

This is more an observation and question than anything else, but perhaps it will stimulate some ideas from the experts. Consider this trivial generator function: def gen(a): yield a When the YIELD_VALUE instruction is executed, it executes (in the non-async case): retval = POP(); f->f_stacktop = stack_pointer; goto exiting; This is fine as far as it goes. However, execution eventually leads to Objects/genobject.c where we hit this code (I think after falling off the YIELD_VALUE instruction, but perhaps virtual machine execution reaches RETURN_VALUE): /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ if (result && f->f_stacktop == NULL) { There are several other references to f->f_stacktop in genobject.c. I've not yet investigated all of them. As I'm working on a register-based virtual machine implementation, I don't fiddle with the stack at all, so it's a bit problematic that the generator implementation is so intimate with the stack. As this is an area of the core which is completely new to me, I wonder if someone can suggest alternate ways of achieving the same effect without relying on the state of the stack. It seems to me that from within PyEval_EvalFrameDefault the implementations of relevant instructions could reference the generator object via f->f_gen and call some (new?) private API on generators which toggles the relevant bit of state in the generator. I think it's worse that this though, as it seems that in gen_send_ex() it actually pushes a value onto the stack. That can't be solved by simply adding a state attribute to the generator object struct. Skip

3 4

PEP 554 comments
by Antoine Pitrou 30 Apr '20

30 Apr '20

Hello, First, I would like to say that I have no fondamental problem with this PEP. While I agree with Nathaniel that the rationale given about the CSP concurrency model seems a bit weak, the author is obviously expressing his opinion there and I won't object to that. However, I think the PEP is desirable for other reasons. Mostly, I hope that by making the subinterpreters functionality available to pure Python programmers (while it was formally an advanced and arcane part of the C API), we will spur of bunch of interesting third-party experimentations, including possibilities that we on python-dev have not thought about. The appeal of the PEP for experimentations is multiple: 1) ability to concurrently run independent execution environments without spawning child processes (which on some platforms and in some situations may not be very desirable: for example on Windows where the cost of spawning is rather high; also, child processes may crash, and sometimes it is not easy for the parent to recover, especially if a synchronization primitive is left in an unexpected state) 2) the potential for parallelizing CPU-bound pure Python code in a single process, if a per-interpreter GIL is finally implemented 3) easier support for sharing large data between separate execution environments, without the hassle of setting up shared memory or the fragility of relying on fork() semantics (and as I said, I hope people find other applications) As for the argument that we already have asyncio and several other packages, I actually think that combining these different concurrency mechanisms would be interesting complex applications (such as distributed systems). For that, however, I think the PEP as currently written is a bit lacking, see below. Now for the detailed comments. * I think the module should indeed be provisional. Experimentation may discover warts that call for a change in the API or semantics. Let's not prevent ourselves from fixing those issues. * The "association" timing seems quirky and potentially annoying: an interpreter only becomes associated with a channel the first time it calls recv() or send(). How about, instead, associating an interpreter with a channel as soon as that channel is given to it through `Interpreter.run(..., channels=...)` (or received through `recv()`)? * How hard would it be, in the current implementation, to add buffering to channels? It doesn't have to be infinite: you can choose a fixed buffer size (or make it configurable in the create() function, which allows passing 0 for unbuffered). Like Nathaniel, I think unbuffered channels will quickly be annoying to work with (yes, you can create a helper thread... now you have one additional thread per channel, which isn't pretty -- especially with the GIL). * In the same vein, I think channels should allow adding readiness callbacks (that are called whenever a channel becomes ready for sending or receiving, respectively). This would make it easy to plug them into an event loop or other concurrency systems (such as Future-based concurrency). Note that each interpreter "associated" with a channel should be able to set its own readiness callback: so one callback per Python object representing the channel, but potentially multiple callbacks for the underlying channel primitive. (how would the callback be scheduled for execution in the right interpreter? perhaps using `_PyEval_AddPendingCall()` or a similar mechanism?) * I think either `interpreters.get_main()` or `interpreters.is_main()` is desirable. Inevitable, the slight differences between main and non-main interpreters will surface in non-trivial applications (finalization issues in distributed systems can really be hairy). It seems this should be mostly costless to provide, so let's do it. * I do think a minimal synchronization primitive would be nice. Either a Lock (in the Python sense) or a Semaphore: both should be relatively easy to provide, by wrapping an OS-level synchronization primitive. Then you can recreate all high-level synchronization primitives, like the threading and multiprocessing modules do (using a Lock or a Semaphore, respectively). (note you should be able to emulate a semaphore using blocking send() and recv() calls, but that's probably not very efficient, and efficiency is important) Of course, I hope these are all actionable before beta1 :-) If not, here is my preferential priority list: * High priority: fix association timing * High priority: either buffering /or/ readiness callbacks * Middle priority: get_main() /or/ is_main() * Middle / low priority: a simple synchronization primitive But I would stress the more of these we provide, the more we encourage people to experiment without pulling too much of their hair. (also, of course, I hope other people read the PEP and emit feedback) Best regards Antoine.

7 29

[RELEASE] Python 3.8.3rc1 is now ready for testing
by Łukasz Langa 30 Apr '20

30 Apr '20

Python 3.8.3rc1 is the release candidate of the third maintenance release of Python 3.8. Go get it here: https://www.python.org/downloads/release/python-383rc1/ <https://www.python.org/downloads/release/python-383rc1/> Assuming no critical problems are found prior to 2020-05-11, the scheduled release date for 3.8.3, no code changes are planned between this release candidate and the final release. That being said, please keep in mind that this is a pre-release and as such its main purpose is testing. Maintenance releases for the 3.8 series will continue at regular bi-monthly intervals, with 3.8.4 planned for mid-July 2020. What’s new? The Python 3.8 series is the newest feature release of the Python language, and it contains many new features and optimizations. See the “What’s New in Python 3.8 <https://docs.python.org/3.8/whatsnew/3.8.html>” document for more information about features included in the 3.8 series. Detailed information about all changes made in version 3.8.3 specifically can be found in its change log <https://docs.python.org/release/3.8.3rc1/whatsnew/changelog.html#python-3-8…>. We hope you enjoy Python 3.8! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv>

1 0

PEP 554 for 3.9 or 3.10?
by Eric Snow 30 Apr '20

30 Apr '20

Hi all, It was nice to speak with many of you at the online language summit this week. It was the next best thing to seeing you all in person! :) With the 3.9 feature freeze coming up I'm considering options for PEP 554. I'm hopeful to have a pronouncement from Antoine in the near future. If that comes in time for 3.9, we will have the implementation ready to go. It is already nearly complete and the remaining work should be done in the next week or so. I'm looking for feedback on a nagging worry I have with merging PEP 554 into 3.9 (if it is accepted) without having the per-interpreter GIL. See below. I'll update the PEP later with relevant info from this discussion. wishing-you-were-all-here-ly yours, -eric +++++++++++++++++++++++++++++++++++++++++++++++++ (++++++++++ Dilemma ============ Many folks have conflated PEP 554 with having a per-interpreter GIL. In fact, I was careful to avoid any mention of parallelism or the GIL in the PEP. Nonetheless some are expecting that when PEP 554 lands we will reach multi-core nirvana. While PEP 554 might be accepted and the implementation ready in time for 3.9, the separate effort toward a per-interpreter GIL is unlikely to be sufficiently done in time. That will likely happen in the next couple months (for 3.10). So...would it be sufficiently problematic for users if we land PEP 554 in 3.9 without per-interpreter GIL? Options ============ Here are the options as I see them (if the PEP is accepted in time for 3.9): 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 (they get parallelism for free in 3.10) 2. like 1, but mark the module as provisional until per-interpreter GIL lands 3. do not merge PEP 554 until per-interpreter GIL is merged 4. like 3, but publish a 3.9-only module to PyPI in the meantime Context ============ Like I said, PEP 554 purposefully does not talk about parallelism or the GIL. Instead it focuses strictly on the following (with an emphasis on minimalism): * exposing the existing C-API * supporting a new concurrency model (CSP) Regarding that first one, the value is in exposing the existing functionality to a broader group of people. keep in mind that subinterpreters currently have various other limitations (aside from sharing the GIL): * nothing about them has been optimized (e.g. interpreter startup, data sharing, data passing) * extension modules with process-global state may break * some bugs still remain in the corners Currently, there is a chicken-and-egg problem. It is unlikely that there will be sufficient motivation to address those limitations until the community starts really using subinterpreters. Hence PEP 554. My Position ============ At this point I think we should go with option #1 (or possibly #2). IMHO, we would be fine merging PEP 554 without per-interpreter GIL. Here's why: * the two objectives of the PEP have value for the community (as explained in the PEP) * the sooner we can get the functionality of subinterpreters out to the broader world, the better * I don't think the lack of parallelism in 3.9 will trip anyone up My only concern is that folks try them out in 3.9, get frustrated by the limitations, and then there is a mental barrier to trying them again in the future when the limitations have been reduced or eliminated. However, I don't think that will be a problem for people that would benefit from serious use of subinterpreters. Furthermore, if we're clear that "for now" subinterpreters still share the GIL in 3.9 then I'm not so worried about folks getting confused. In fact, as long as we're clear about all the current limitations then there isn't much downside to option #1. Marking the module as "provisional" may help communicate that there are limitations that impact use.

16 60

Comments on PEP 554 (Multiple Interpreters in the Stdlib)
by Mark Shannon 30 Apr '20

30 Apr '20

Hi, I'm generally in favour of PEP 554, but I don't think it is ready to be accepted in its current form. My main objection is that without per-subinterpeter GILs (SILs?) PEP 554 provides no value over threading or multi-processing. Multi-processing provides true parallelism and threads provide shared memory concurrency. If per-subinterpeter GILs are possible then, and only then, sub-interpreters will provide true parallelism and (limited) shared memory concurrency. The problem is that we don't know whether we can implement per-subinterpeter GILs without too large a negative performance impact. I think we can, but we can't say so for certain. So, IMO, we should not accept PEP 554 until we know for sure that per-subinterpeter GILs can be implemented efficiently. Detailed critique ----------------- I don't see how `list_all()` can be both safe and accurate. The Java equivalent makes no guarantees of accuracy. Attempting to lock the list is likely to lead to deadlock and not locking it will lead to races; potentially dangerous ones. I think it would be best to drop this. `list_all_channels()`. See `list_all()` above. `.destroy()` is either misleading or unsafe. What does this do? >>> is.destroy() >>> is.run() If `run()` raises an exception then the interpreter must exist. Rename to `close()` perhaps? `Channel.interpreters` see `list_all()` and `list_all_channels()` above. How does `is_shareable()` work? Are you proposing some mechanism to transfer an object from one sub-interpreter to another? How would that work? If objects are not shared but serialized, why not use marshal or pickle instead of inventing a third serialization protocol? It would be clearer if channels only dealt with simple, contiguous binary data. As it stands the PEP doesn't state what form the received object will take. Once channels supporting the transfer of bytes objects work, then it is simple to pass more complex objects using pickle or marshal. Channels need a more detailed description of their lifespan. Ideally a state machine. For example: How does an interpreter detach from the receiving end of a channel that is never empty? What happens if an interpreter deletes the last reference to a non-empty channel? On the receiving end, or on the sending end? Isn't the lack of buffering in channels a recipe for deadlocks? What is the mechanism for reliably copying exceptions from one sub-interpreter to another in the `run()` method? If `run()` can raise an exception, why not let it return values? Cheers, Mark.

5 8

[RELEASE] Python 3.9.0a6 is now available for testing
by Łukasz Langa 29 Apr '20

29 Apr '20

On behalf of the entire Python development community, and the currently serving Python release team in particular, I’m pleased to announce the release of Python 3.9.0a6. Get it here: https://www.python.org/downloads/release/python-390a6/ <https://www.python.org/downloads/release/python-390a6/> This is an early developer preview of Python 3.9 Python 3.9 is still in development. This release, 3.9.0a6, is the last out of six planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2020-05-18) and, if necessary, may be modified or deleted up until the release candidate phase (2020-08-10). Please keep in mind that this is a preview release and its use is not recommended for production environments. Major new features of the 3.9 series, compared to 3.8 Many new features for Python 3.9 are still being planned and written. Among the new major new features and changes so far: PEP 584 <https://www.python.org/dev/peps/pep-0584/>, Union Operators in dict PEP 585 <https://www.python.org/dev/peps/pep-0585/>, Type Hinting Generics In Standard Collections PEP 593 <https://www.python.org/dev/peps/pep-0593/>, Flexible function and variable annotations PEP 602 <https://www.python.org/dev/peps/pep-0602/>, Python adopts a stable annual release cadence PEP 616 <https://www.python.org/dev/peps/pep-0616/>, String methods to remove prefixes and suffixes PEP 617 <https://www.python.org/dev/peps/pep-0617/>, New PEG parser for CPython BPO 38379 <https://bugs.python.org/issue38379>, garbage collection does not block on resurrected objects; BPO 38692 <https://bugs.python.org/issue38692>, os.pidfd_open added that allows process management without races and signals; BPO 39926 <https://bugs.python.org/issue39926>, Unicode support updated to version 13.0.0 BPO 1635741 <https://bugs.python.org/issue1635741>, when Python is initialized multiple times in the same process, it does not leak memory anymore A number of Python builtins (range, tuple, set, frozenset, list) are now sped up using PEP 590 <https://www.python.org/dev/peps/pep-0590> vectorcall A number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined by PEP 384 <https://www.python.org/dev/peps/pep-0384/>. (Hey, fellow core developer, if a feature you find important is missing from this list, let Łukasz know <mailto:lukasz@python.org>.) The next pre-release, the first beta release of Python 3.9, will be 3.9.0b1. It is currently scheduled for 2020-05-18. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv>

5 5

killing static types (for sub-interpreters?)
by Jim J. Jewett 29 Apr '20

29 Apr '20

Why do sub-interpreters require (separate and) heap-allocated types? It seems types that are statically allocated are a pretty good use for immortal objects, where you never change the refcount ... and then I don't see why you need more than one copy.

4 5

Announcement: pip 20.1b1 beta release
by Sumana Harihareswara 29 Apr '20

29 Apr '20

On behalf of the PyPA, I am pleased to announce that a beta release of pip, pip 20.1b1, has been released. I hope you will help us by testing this and checking for new bugs in how pip deals with complicated projects and dependencies. The highlights for this release are: * Significant speedups when building local directories, by changing behavior to perform in-place builds, instead of copying to temporary directories. * Significant speedups in `pip list --outdated`, by parallelizing network access. This is the first instance of parallel code within pip's codebase. * A new `pip cache` command, which makes it possible to introspect and manage pip's cache directory. * Better `pip freeze` for packages installed from direct URLs, enabled by the implementation of PEP 610. We would be grateful for all the testing that users could do to ensure that, when pip 20.1 is released, it's as solid as we can make it. You can upgrade to this beta with `python -m pip install -U --pre pip`. This release also contains an alpha version of pip's next generation resolver. It is **off by default** because it is **unstable and not ready for everyday use**. If you're curious about this, please visit this GitHub issue about the resolver, what doesn't work yet, and what kind of testing would help us out https://github.com/pypa/pip/issues/8099 . The full changelog is available. https://pip.pypa.io/en/latest/news/ As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Specific thanks go to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for their support that enabled the work on the new resolver. -- Sumana Harihareswara pip project manager under contract with Python Software Foundation Changeset Consulting https://changeset.nyc

3 6

Improvement to SimpleNamespace
by Raymond Hettinger 29 Apr '20

29 Apr '20

SimpleNamespace() is really good at giving attribute style-access. I would like to make that functionality available to the JSON module (or just about anything else that accepts a custom dict) by adding the magic methods for mappings so that this works: catalog = json.load(f, object_hook=SimpleNamespace) print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity']) # currently possible with dict() print(catalog.clothing.mens.shoes.extra_wide.quantity]) # proposed with SimpleNamespace() print(catalog.clothing.boys['3t'].tops.quantity # would also be supported I've already seen something like this in production; however, people are having to write custom subclasses to do it. This is kind of bummer because the custom subclasses are a pain to write, are non-standard, and are generally somewhat slow. I would like to see a high-quality version this made more broadly available. The core idea is keep the simple attribute access but make it easier to load data programmatically: >>> ns = SimpleNamespace(roses='red', violets='blue') >>> thing = input() sugar >>> quality = input() sweet >>> setattr(ns, thing, quality) # current >>> ns['sugar'] = 'sweet' # proposed If the PEP 584 __ior__ method were supported, updating a SimpleNamespace would be much cleaner: ns |= some_dict I posted an issue on the tracker: https://bugs.python.org/issue40284 . There was a suggestion to create a different type for this, but I don't see the point in substantially duplicating everything SimpleNamespace already does just so we can add some supporting dunder methods. Please add more commentary so we can figure-out the best way to offer this powerful functionality. Raymond

14 19

Request For Review: Add support for CAN_J1939 sockets (bpo-40291)
by Karl Ding 28 Apr '20

28 Apr '20

Hi all, Could someone take a look at the following PR to add support for CAN_J1939 to the socket module? I'd like to try landing this for 3.9. This enhancement would be useful for anyone working in automotive and/or dealing with the SAE J1939 CAN protocol. This feature is available on Linux 5.4+ (Ubuntu 20.04 LTS ships with a compatible kernel). PR Link: https://github.com/python/cpython/pull/19538 BPO Link: https://bugs.python.org/issue40291 You may find the following links useful if you want to find out more about how the kernel J1939 implementation is used. J1939 Kernel Docs: https://www.kernel.org/doc/html/latest/networking/j1939.html J1939 header: https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/can/j1939… Thanks! -- Karl Ding

2 1