Mailman 3 August 2016 - Python-Dev

PEP 526 ready for review: Syntax for Variable and Attribute Annotations
by Guido van Rossum 05 Sep '16

05 Sep '16

I'm happy to present PEP 526 for your collective review: https://www.python.org/dev/peps/pep-0526/ (HTML) https://github.com/python/peps/blob/master/pep-0526.txt (source) There's also an implementation ready: https://github.com/ilevkivskyi/cpython/tree/pep-526 I don't want to post the full text here but I encourage feedback on the high-order ideas, including but not limited to - Whether (given PEP 484's relative success) it's worth adding syntax for variable/attribute annotations. - Whether the keyword-free syntax idea proposed here is best: NAME: TYPE TARGET: TYPE = VALUE Note that there's an extensive list of rejected ideas in the PEP; please be so kind to read it before posting here: https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-lef… -- --Guido van Rossum (python.org/~guido)

11 34

Patch reviews
by Christian Heimes 04 Sep '16

04 Sep '16

Hi, I have 7 patches for 3.6 ready for merging. The new features were discussed on Security-SIG and reviewed by Victor or GPS. The patches just need one final review and an ACK. The first three patches should land in 2.7, 3.4 and 3.5, too. http://bugs.python.org/issue26470 Make OpenSSL module compatible with OpenSSL 1.1.0 https://bugs.python.org/issue27850 Remove 3DES from cipher list (sweet32 CVE-2016-2183) Also adds ChaCha20 Poly1305 http://bugs.python.org/issue27691 X509 cert with GEN_RID subject alt name causes SytemError http://bugs.python.org/issue27866 ssl: get list of enabled ciphers https://bugs.python.org/issue27744 Add AF_ALG (Linux Kernel crypto) to socket module http://bugs.python.org/issue16113 Add SHA-3 and SHAKE (Keccak) support http://bugs.python.org/issue26798 add BLAKE2 to hashlib Christian

3 4

Update on PEP 523 and adding a co_extra field to code objects
by Brett Cannon 03 Sep '16

03 Sep '16

For quick background for those that don't remember, part of PEP 523 proposed adding a co_extra field to code objects along with making the frame evaluation function pluggable ( https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea was that things like JITs and debuggers could use the field as a scratch space of sorts to store data related to the code object. People who objected to the new field did either for memory ("it adds another pointer to the struct that won't be typically used"), or for conceptual reasons ("the code object is immutable and you're proposing a mutable field"). The latter is addressed by not exposing the field in Python and clearly stating that code should never expect the field to be filled. For the former issue of whether the memory was worth it, Dino has been testing whether the field is necessary for performance from a JIT perspective. Well, Dino found the time to test Pyjion without the co_extra field and it isn't pretty. With the field, Pyjion is faster than stock Python in 15 benchmarks ( https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the co_extra field and using an unordered_map from the C++ STL drops that number to 2. Performance is even worse if we try and use a Python dictionary instead. That means we still want to find a solution to attach arbitrary data to code objects without sacrificing performance. One proposal is what's in PEP 523 for the extra field. Another option is to make the memory allocator for code objects pluggable and introduce a new flag that signals that the object was created using a non-default allocator. Obviously we prefer the former solution due to its simplicity. :) Anyway, we would like to get this settled this week so that I can get whatever solution we agree to (if any) in next week in time for Python 3.6b1 feature freeze that would be greatly appreciated.

13 35

Supported versions of OpenSSL
by Christian Heimes 31 Aug '16

31 Aug '16

Hi, we need to talk about OpenSSL and LibreSSL before the next release of Python. I'm working on a PEP. Most likely it won't be ready before the feature freeze. But first let me start with some good news. OpenSSL 1.1 was released a couple of days ago. It changed a lot of aspects of its internal API, e.g. all structs are opaque and must be allocated / freed with OpenSSL API calls. Since I have been tracking changes in OpenSSL for the last half year and have submitted a couple of patches to OpenSSL, we are in a good shape. My patch https://bugs.python.org/issue26470 makes Python 2 and 3 compatible with OpenSSL 0.9.8 to 1.1.0 and with LibreSSL, too. It needs to go through review, though. I have asked Alex to verify my patch. Now to the bad news. The SSL module is a mess. It looks like a junk room owned by collector of ancient OpenSSL versions. For example it contains version checks for OpenSSL 0.9.5 -- which was decommissioned in 2000! That pre-dates new style classes! I like to reduce the maintenance burden and list of supported OpenSSL versions ASAP. OpenSSL has deprecated 0.9.8 and 1.0.0 last year. 1.0.1 will reach EOL by the end of this year, https://www.openssl.org/policies/releasestrat.html . However OpenSSL 0.9.8 is still required for some platforms (OSX). Here is the deal for 2.7 to 3.5: 1) All versions older than 0.9.8 are completely out-of-scope and no longer supported. 2) 0.9.8 is semi-support. Python will still compile and work with 0.9.8. However we do NOT promise that is secure to run 0.9.8. We also require a recent version. Patch level 0.9.8zc from October 2014 is reasonable because it comes with SCSV fallback (CVE-2014-3566). 3) 1.0.0 is irrelevant. Users are either stuck on 0.9.8 or are able to upgrade to 1.0.1+. Let's not support it. 4) 1.0.1 is discouraged but still supported until its EOL. 5) 1.0.2 is the recommend version. 6) 1.1 support will be added by #26470 soon. 7) LibreSSL 2.3 is supported but with a slightly limited feature set. LibreSSL removed some features like SSL_CERT_FILE and OPENSSL_CONF env vars. For upcoming 3.6 I would like to limit support to 1.0.2+ and require 1.0.2 features for 3.7. What is the status of Python.org's OSX builds? Is it possible to drop 0.9.8? Christian

16 50

Re: [Python-Dev] File system path encoding on Windows
by Victor Stinner 30 Aug '16

30 Aug '16

2016-08-30 16:31 GMT+02:00 Steve Dower <steve.dower(a)python.org>: > It's the > random user on Windows who installed their library that has the problem. > They don't know the fix, and may not know how to apply it (e.g. if it's > their Jupyter notebook that won't find one of their files - no obvious > command line options here). There is already a DeprecationWarning. Sadly, it's hidden by default: you need a debug build of Python or more simply to pass -Wd command line option. Maybe we should make this warning (Deprecation warning on bytes paths) visible by default, or add a new warning suggesting to enable -X utf8 the first time a Python function gets a byte string (like a filename)? > Any system that requires communication between two different versions of > Python must have install instructions (if it's public) or someone who > maintains it. It won't magically break without an upgrade, and it should not > get an upgrade without testing. The environment variable is available for > this kind of scenario, though I'd hope the testing occurs during beta and it > gets fixed by the time we release. I disagree that breaking backward compatibility is worth it. Most users don't care of Unicode since their application already "just works well" for their use case. Having to set an env var to "repair" their app to be able to upgrade Python is not really convenient. Victor

4 10

What do we do about bad slicing and possible crashes (issue 27867)
by Terry Reedy 30 Aug '16

30 Aug '16

Slicing can be made to malfunction and even crash with an 'evil' __index__ method. https://bugs.python.org/issue27867 The crux of the problem is this: PySlice_GetIndicesEx receives a slice object and a sequence length. Calling __index__ on the start, stop, and step components can mutate the sequence and invalidate the length. Adjusting the int values of start and stop according to an invalid length (in particular, one that is too long) will result in invalid results or a crash. Possible actions -- very briefly. For more see end of https://bugs.python.org/issue27867?@ok_message=msg 273801 0. Do nothing. 1. Detect length change and raise. 2. Retrieve length after any possible changes and proceed as normal. Possible implementation strategies for 1. and 2. A. Change all functions that call PySlice_GetIndicesEx. B. Add PySlice_GetIndicesEx2 (or ExEx?), which would receive *collection instead of length, so the length could be retrieved after the __index__ calls. Change calls. Deprecate PySlice_GetIndicesEx. Which of the 4 possible patches, if any, would be best? I personally prefer 2B. And what versions should be patched? -- Terry Jan Reedy

7 10

Lib/http/client.py: could it return an OSError with the current response?
by Ivo Bellin Salarin 30 Aug '16

30 Aug '16

Hi everybody, Sorry for bothering you, this is my first post to the python-dev ML. While using requests to tunnel a request via a proxy requiring user authentication, I have seen that httplib ( https://hg.python.org/cpython/file/3.5/Lib/http/client.py#l831) raises the message returned by the proxy, along with its status code (407) without including the proxy response. This one could be very interesting to the consumer, since it could contain some useful headers (like the supported authentication schemes). Would it be possible to change the http/client.py behavior in order to raise an exception including the whole response? If you don't see any problem with my proposal, how can I propose a pull request? :-) Thanks in advance, Ivo

3 2

File system path encoding on Windows
by Steve Dower 30 Aug '16

30 Aug '16

Hi python-dev About a week ago I proposed on python-ideas making some changes to how Python deals with encodings on Windows, specifically in relation to how Python interacts with the operating system. Changes to the console were uncontroversial, and I have posted patches at http://bugs.python.org/issue1602 and http://bugs.python.org/issue17620 to enable the full range of Unicode input to be used at interactive stdin/stdout. However, changes to sys.getfilesystemencoding(), which determines how the os module (and most filesystem functions in general) interpret bytes parameters, were more heatedly discussed. I've summarised the discussion in this email I'll declare up front that my preferred change is to treat bytes as utf-8 in Python 3.6, and I've posted a patch to do that at http://bugs.python.org/issue27781. Hopefully I haven't been too biased in my presentation of the alternatives, but this is so you at least know which way I'm biased. I'm looking for some agreement on the answers to the questions I pose in the summary. There is much more detail about them presented after that, as there are a number of non-obvious issues at play here. I suspect this will eventually become a PEP, but it's presented here as a summary of a discussion and not a PEP. Cheers, Steve Summary ======= Representing file system paths on Windows as bytes may result in data loss due to the way Windows encodes/decodes strings via its bytes API. We can mitigate this by only using Window's Unicode API and doing our own encoding and decoding (i.e. within posixmodule.c's path converter). Invalid characters could cause encoding exceptions rather than data loss. We can go further to fix this by declaring the encoding of bytes paths on Windows must be utf-8, which would also prevent encoding exceptions, as utf-8 can fully represent all paths on Windows (natively utf-16-le). Even though using bytes for paths on Windows has been deprecated for three releases, this is not widely known and it may be too soon to change the behaviour. Questions: * should we always use Window's Unicode APIs instead of switching between bytes/Unicode based on parameter type? * should we allow users to pass bytes and interpret them as utf-8 rather than letting Windows do the decoding? * should we do it in 3.6, 3.7 or 3.8? Background ========== File system paths are almost universally represented as text in some encoding determined by the file system. In Python, we expose these paths via a number of interfaces, such as the os and io modules. Paths may be passed either direction across these interfaces, that is, from the filesystem to the application (for example, os.listdir()), or from the application to the filesystem (for example, os.unlink()). When paths are passed between the filesystem and the application, they are either passed through as a bytes blob or converted to/from str using sys.getfilesystemencoding(). The result of encoding a string with sys.getfilesystemencoding() is a blob of bytes in the native format for the default file system. On Windows, the native format for the filesystem is utf-16-le. The recommended platform APIs for accessing the filesystem all accept and return text encoded in this format. However, prior to Windows NT (and possibly further back), the native format was a configurable machine option and a separate set of APIs existed to accept this format. The option (the "active code page") and these APIs (the "*A functions") still exist in recent versions of Windows for backwards compatibility, though new functionality often only has a utf-16-le API (the "*W functions"). In Python, we recommend using str as the default format because (with the surrogateescape handling on POSIX), it can correctly round-trip all characters used in paths. On Windows this is strongly recommended because the legacy OS support for bytes cannot round-trip all characters used in paths. Our support for bytes explicitly uses the *A functions and hence the encoding for the bytes is "whatever the active code page is". Since the active code page cannot represent all Unicode characters, the conversion of a path into bytes can lose information without warning (and we can't get a warning from the OS here - more on this later). As a demonstration of this: >>> open('test\uAB00.txt', 'wb').close() >>> import glob >>> glob.glob('test*') ['test\uab00.txt'] >>> glob.glob(b'test*') [b'test?.txt'] The Unicode character in the second call to glob has been replaced by a '?', which means passing the path back into the filesystem will result in a FileNotFoundError (though ironically, passing it back into glob() will find the file again, since '?' is a single-character wildcard). You can observe the same results in os.listdir() or any function that matches the return type to the parameter type. Why is this a problem? ====================== While the obvious and correct answer is to just use str everywhere, in general on POSIX systems there is no possibility of confusion when using bytes exclusively. Even if the encoding is "incorrect" by some standard, the file system can still map the bytes back to the file. Making use of this avoids the cost of decoding and reencoding, such that (theoretically, and only on POSIX), code like below is faster because of the use of `b'.'`: >>> for f in os.listdir(b'.'): ... os.stat(f) ... On Windows, if a filename exists that cannot be encoding with the active code page, you will receive an error from the above code. These errors are why in Python 3.3 the use of bytes paths on Windows was deprecated (listed in the What's New, but not clearly obvious in the documentation - more on this later). The above code produces multiple deprecation warnings in 3.3, 3.4 and 3.5 on Windows. However, we still keep seeing libraries use bytes paths, which can cause unexpected issues on Windows (well, all platforms, but less and less common on POSIX as systems move to utf-8 - Windows long ago decided to move to utf-16 for the same reason, but Python's bytes interface did not keep up). Given the current approach of not-very-aggressively recommending that library developers either write their code twice (once for bytes and once for str) or use str exclusively are not working, we should consider alternative mitigations. Proposals ========= There are two dimensions here - the fix and the timing. We can basically choose any fix and any timing. The main differences between the fixes are the balance between incorrect behaviour and backwards-incompatible behaviour. The main issue with respect to timing is whether or not we believe using bytes as paths on Windows was correctly deprecated in 3.3 and sufficiently advertised since to allow us to change the behaviour in 3.6. Fixes ----- Fix #1: Change sys.getfilesystemencoding() to utf-8 on Windows Currently the default filesystem encoding is 'mbcs', which is a meta-encoder that uses the active code page. However, when bytes are passed to the filesystem they go through the *A APIs and the operating system handles encoding. In this case, paths are always encoded using the equivalent of 'mbcs:replace' - we have no ability to change this (though there is a user/machine configuration option to change the encoding from CP_ACP to CP_OEM, so it won't necessarily always match mbcs...) This proposal would remove all use of the *A APIs and only ever call the *W APIs. When Windows returns paths to Python as str, they will be decoded from utf-16-le and returned as text. When paths are to be returned as bytes, we would decode from utf-16-le to utf-8 using surrogatepass (as Windows does not validate surrogate pairs, so it is possible to have invalid surrogates in filenames). Equally, when paths are provided as bytes, they are decoded from utf-8 into utf-16-le and passed to the *W APIs. The use of utf-8 will not be configurable, with the possible exception of a "legacy mode" environment variable or Xflag. surrogateescape does not apply here, as we are not concerned about keeping arbitrary bytes in the path. Any bytes path returned from the operating system will be valid; any bytes path created by the user may raise a decoding error (currently it would raise a file not found or similar OSError). The choice of utf-8 (as opposed to returning utf-16-le bytes) is to ensure the ability to round-trip, while also allowing basic manipulation of paths - essentially just slicing and concatenating at '\' characters. Applications doing this have to ensure that their encoding matches sys.getfilesystemencoding(), or just use str everywhere. It is debated, but I believe this is not a backwards compatibility issue because: * byte paths in Python are specified as being encoded by sys.getfilesystemencoding() * byte paths on Windows have been deprecated for three versions Unfortunately, the deprecation is not explicitly called out anywhere in the docs apart from the What's New page, so there is an argument that it shouldn't be counted despite the warnings in the interpreter. However, this is more directly addressed in the discussion of timing below. Equally, sys.getfilesystemencoding() documents the specific return values for various platforms, as well as that it is part of the protocol for using bytes to represent filesystem strings. I believe both of these arguments are invalid, that the only code that will break as a result of this change is relying on deprecated functionality and incorrect encoding, and that the (probably noisy) breakage that will occur is less bad than the silent breakage that currently exists. As far as implementation goes, there is already a patch for this at http://bugs.python.org/issue27781. In short, we update the path converter to decode bytes (path->narrow) to Unicode (path->wide) and remove all the code that would call *A APIs. In my patch I've changed path->narrow to a flag that indicates whether to convert back to bytes on return, and also to prevent compilation of code that tries to use ->narrow as a string on Windows (maybe that will get too annoying for contributors? good discussion for the tracker IMHO). Fix #2: Do the mbcs decoding ourselves This is essentially the same as fix #1, but instead of changing to utf-8 we keep mbcs as the encoding. This approach will allow us to utilise new functionality that is only available as *W APIs, and also lets us be more strict about encoding/decoding to bytes. For example, rather than silently replacing Unicode characters with '?', we could warn or fail the operation, potentially modifying that behaviour with an environment variable or flag. Compared to fix #1, this will enable some new functionality but will not fix any of the problems immediately. New runtime errors may cause some problems to be more obvious and lead to fixes, provided library maintainers are interested in supporting Windows and adding a separate code path to treat filesystem paths as strings. This is a middle-ground proposal. On the positive side, it significantly reduces the code we have to maintain in CPython (e.g. posixmodule.c), as we won't require separate code paths to call the *A APIs. However, it doesn't really improve things for users apart from giving more exceptions, which are likely unexpected (people probably handle OSError but not UnicodeDecodeError when accessing the file system). Fix #3: Make bytes paths on Windows an error By preventing the use of bytes paths on Windows completely we prevent users from hitting encoding issues. However, we do this at the expense of usability. Obviously the deprecation concerns also play a big role in whether this is feasible. I don't have numbers of libraries that will simply fail on Windows if this "fix" is made, but given I've already had people directly email me and tell me about their problems we can safely assume it's non-zero. I'm really not a fan of this fix, because it doesn't actually make things better in a practical way, despite being more "pure". Timing #1: Change it in 3.6 This timing assumes that we believe the deprecation of using bytes for paths in Python 3.3 was sufficiently well advertised that we can freely make changes in 3.6. A typical deprecation cycle would be two versions before removal (though we also often leave things in forever when they aren't fundamentally broken), so we have passed that point and theoretically can remove or change the functionality without breaking it. In this case, we would announce in 3.6 that using bytes as paths on Windows is no longer deprecated, and that the encoding used is whatever is returned by sys.getfilesystemencoding(). Timing #2: Change it in 3.7 This timing assumes that the deprecation in 3.3 was valid, but acknowledges that it was not well publicised. For 3.6, we aggressively make it known that only strings should be used to represent paths on Windows and bytes are invalid and going to change in 3.7. (It has been suggested that I could use a keynote at PyCon to publicise this, and while I'd totally accept a keynote, I'd hate to subject a crowd to just this issue for an hour :) ). My concern with this approach is that there is no benefit to the change at all. If we aggressively publicise the fact that libraries that don't handle Unicode paths on Windows properly are using deprecated functionality and need to be fixed by 3.7 in order to avoid breaking (more precisely - continuing to be broken, but with a different error message), then we will alienate non-Windows developers further from the platform (net loss for the ecosystem) and convince some to switch to str everywhere (net gain for the ecosystem). It doesn't For those who listen and change to str, it removes the need to make any change in 3.7 at all, so we would really just be making noise about something that some people may not have noticed without necessarily going in and fixing anything. For those who don't listen, the change in 3.7 is going to break them just as much as if we made the change in 3.6. Timing #3: Change it in 3.8 This timing assumes that the deprecation in 3.3 was not sufficient and we need to start a new deprecation cycle. This is strengthened by the fact that the deprecation announcement does not explicitly include the io module or the builtin open() function, and so some developers may believe that using bytes for paths with these is okay despite the os module being deprecated. The one upside to this approach is that it would also allow us to change locale.getpreferredencoding() to utf-8 on Windows (to affect the default behaviour of open(..., 'r') ), which I don't believe is going to be possible without a new deprecation cycle. There is a strong argument that the following code should also round-trip regardless of platform: >>> with open('list.txt', 'w') as f: ... for i in os.listdir('.'): ... print(i, file=f) ... >>> with open('list.txt', 'r') as f: ... files = list(f) ... Currently, the default encoding for open() cannot represent all filenames that may be returned from listdir(). This may affect makefiles and configuration files that contain paths. Currently they will work correctly for paths that can be represented in the machine's active code page (though it should be noted that the *A APIs may be changed in a process by user/machine configuration to use the OEM code page rather than the active code page, which would potentially lead to encoding issues even for CP_ACP compatible names). Possibly resolving both issues simultaneously is worth waiting for two more releases? I'm not convinced the change to getfilesystemencoding() needs to wait for getpreferredencoding() to also change, or that they necessarily need to match, but it would not be hugely surprising to see the changes bundled together. I'll also note that there has been limited discussion about changing getpreferredencoding() so far, though there have been a number of "+1" votes alongside some "+1 with significant concerns" votes. Changing the default encoding of the contents of data files is pretty scary, so I'm not in any rush to force it in. On the other hand, changing the encoding for paths without changing the default encoding for text files may break "bytes in, bytes through, bytes out" for some files (especially makefiles and .ini files). Arguably this idea was already deprecated with Python 3's bytes/text separation anyway. Acknowledgements ================ Thanks to Stephen Turnbull, Eryk Sun, Victor Stinner and Random832 for their significant contributions and willingness to engage, and to everyone else on python-ideas for contributing to the discussion.

7 27

Summary of Python tracker Issues
by Python tracker 26 Aug '16

26 Aug '16

ACTIVITY SUMMARY (2016-08-19 - 2016-08-26) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5606 (+12) closed 34018 (+53) total 39624 (+65) Open issues with patches: 2436 Issues opened (44) ================== #12713: argparse: allow abbreviation of sub commands by users https://bugs.python.org/issue12713 reopened by zach.ware #27803: ctypes automatic byref failing on custom classes attributes https://bugs.python.org/issue27803 opened by lepaperwan #27805: In Python 3, open('/dev/stdout', 'a') raises OSError with errn https://bugs.python.org/issue27805 opened by hathawsh #27806: 2.7 32-bit builds fail on future releases of OS X due to depen https://bugs.python.org/issue27806 opened by ned.deily #27807: Prevent site-packages .pth files from causing test_site failur https://bugs.python.org/issue27807 opened by Rosuav #27808: os.waitpid does not return (0,0) when child has not exited (Fr https://bugs.python.org/issue27808 opened by Casey Lucas #27809: _PyObject_FastCall(): add support for keyword arguments https://bugs.python.org/issue27809 opened by haypo #27810: Add METH_FASTCALL: new calling convention for C functions https://bugs.python.org/issue27810 opened by haypo #27811: _PyGen_Finalize() should not fail with an exception https://bugs.python.org/issue27811 opened by arigo #27812: PyFrameObject.f_gen can be left pointing to a dangling generat https://bugs.python.org/issue27812 opened by arigo #27815: Make SSL suppress_ragged_eofs default more secure https://bugs.python.org/issue27815 opened by martin.panter #27816: sock.proto does not reflect actual protocol https://bugs.python.org/issue27816 opened by christian.heimes #27817: tkinter string variable misinterpreted as boolean https://bugs.python.org/issue27817 opened by Andreas Bolsch #27818: Speed up number format spec parsing https://bugs.python.org/issue27818 opened by scoder #27820: Possible bug in smtplib when initial_response_ok=False https://bugs.python.org/issue27820 opened by Dario D'Amico #27822: Fail to create _SelectorTransport with unbound socket https://bugs.python.org/issue27822 opened by Paul McGuire #27823: Change bare AttributeError messages to be more informative https://bugs.python.org/issue27823 opened by ebarry #27824: update ConfigParser docs regarding in-line comments https://bugs.python.org/issue27824 opened by jahschwa #27825: Make the documentation for statistics' data argument clearer. https://bugs.python.org/issue27825 opened by nedbat #27827: pathlib is_reserved fails for some reserved paths on Windows https://bugs.python.org/issue27827 opened by eryksun #27829: test.regrtest: changed environment variables are not logged https://bugs.python.org/issue27829 opened by haypo #27830: Add _PyObject_FastCallKeywords(): avoid the creation of a temp https://bugs.python.org/issue27830 opened by haypo #27831: Python 3.4.5 leaks memory when attempting to load non-existent https://bugs.python.org/issue27831 opened by geeknik #27833: Process is locked when try to execute Queue.put() inside https://bugs.python.org/issue27833 opened by Guni #27835: SystemExit in setUpClass etc. terminates test run https://bugs.python.org/issue27835 opened by alexander.sturm #27838: test_os.test_chown() random failure on "AMD64 FreeBSD CURRENT https://bugs.python.org/issue27838 opened by haypo #27839: "Python [...] calls sys.displayhook() on unhandled exceptions" https://bugs.python.org/issue27839 opened by jwilk #27842: Order CSV header fields https://bugs.python.org/issue27842 opened by holdenweb #27844: Python-3.6a4 build messages to stderr (on AIX and xlc compiler https://bugs.python.org/issue27844 opened by Michael.Felt #27845: Optimize update_keyword_args() function https://bugs.python.org/issue27845 opened by haypo #27846: Base64 expansion factor is 4 to 3, not 6 to 4 https://bugs.python.org/issue27846 opened by Jens.Jørgen.Mortensen #27847: os.set_inheritable() looks to be broken on OpenIndiana, regres https://bugs.python.org/issue27847 opened by haypo #27850: Remove 3DES from cipher list (sweet32 CVE-2016-2183) https://bugs.python.org/issue27850 opened by christian.heimes #27853: Add title to examples in importlib docs https://bugs.python.org/issue27853 opened by brett.cannon #27854: Installed 2.7: IDLE Help disabled because help.html is missing https://bugs.python.org/issue27854 opened by terry.reedy #27855: 2to3: Wrong code output w/ has_key without trailing space https://bugs.python.org/issue27855 opened by rippey.e #27859: argparse - subparsers does not retain namespace https://bugs.python.org/issue27859 opened by nickpapior #27860: Improvements to ipaddress module https://bugs.python.org/issue27860 opened by moritzs #27861: sqlite3 type confusion and multiple frees https://bugs.python.org/issue27861 opened by tehybel #27863: multiple issues in _elementtree module https://bugs.python.org/issue27863 opened by tehybel #27864: test_socket: unknown thread blocks forever on "AMD64 FreeBSD 9 https://bugs.python.org/issue27864 opened by haypo #27865: WeakMethod does not support builtin methods https://bugs.python.org/issue27865 opened by Antony.Lee #27866: ssl: get list of enabled ciphers https://bugs.python.org/issue27866 opened by christian.heimes #27867: various issues due to misuse of PySlice_GetIndicesEx https://bugs.python.org/issue27867 opened by tehybel Most recent 15 issues with no replies (15) ========================================== #27867: various issues due to misuse of PySlice_GetIndicesEx https://bugs.python.org/issue27867 #27866: ssl: get list of enabled ciphers https://bugs.python.org/issue27866 #27865: WeakMethod does not support builtin methods https://bugs.python.org/issue27865 #27864: test_socket: unknown thread blocks forever on "AMD64 FreeBSD 9 https://bugs.python.org/issue27864 #27860: Improvements to ipaddress module https://bugs.python.org/issue27860 #27859: argparse - subparsers does not retain namespace https://bugs.python.org/issue27859 #27854: Installed 2.7: IDLE Help disabled because help.html is missing https://bugs.python.org/issue27854 #27853: Add title to examples in importlib docs https://bugs.python.org/issue27853 #27844: Python-3.6a4 build messages to stderr (on AIX and xlc compiler https://bugs.python.org/issue27844 #27839: "Python [...] calls sys.displayhook() on unhandled exceptions" https://bugs.python.org/issue27839 #27835: SystemExit in setUpClass etc. terminates test run https://bugs.python.org/issue27835 #27827: pathlib is_reserved fails for some reserved paths on Windows https://bugs.python.org/issue27827 #27825: Make the documentation for statistics' data argument clearer. https://bugs.python.org/issue27825 #27815: Make SSL suppress_ragged_eofs default more secure https://bugs.python.org/issue27815 #27812: PyFrameObject.f_gen can be left pointing to a dangling generat https://bugs.python.org/issue27812 Most recent 15 issues waiting for review (15) ============================================= #27861: sqlite3 type confusion and multiple frees https://bugs.python.org/issue27861 #27860: Improvements to ipaddress module https://bugs.python.org/issue27860 #27850: Remove 3DES from cipher list (sweet32 CVE-2016-2183) https://bugs.python.org/issue27850 #27842: Order CSV header fields https://bugs.python.org/issue27842 #27835: SystemExit in setUpClass etc. terminates test run https://bugs.python.org/issue27835 #27830: Add _PyObject_FastCallKeywords(): avoid the creation of a temp https://bugs.python.org/issue27830 #27823: Change bare AttributeError messages to be more informative https://bugs.python.org/issue27823 #27822: Fail to create _SelectorTransport with unbound socket https://bugs.python.org/issue27822 #27818: Speed up number format spec parsing https://bugs.python.org/issue27818 #27812: PyFrameObject.f_gen can be left pointing to a dangling generat https://bugs.python.org/issue27812 #27811: _PyGen_Finalize() should not fail with an exception https://bugs.python.org/issue27811 #27809: _PyObject_FastCall(): add support for keyword arguments https://bugs.python.org/issue27809 #27807: Prevent site-packages .pth files from causing test_site failur https://bugs.python.org/issue27807 #27799: Fix base64-codec and bz2-codec incremental decoders https://bugs.python.org/issue27799 #27781: Change sys.getfilesystemencoding() on Windows to UTF-8 https://bugs.python.org/issue27781 Top 10 most discussed issues (10) ================================= #27809: _PyObject_FastCall(): add support for keyword arguments https://bugs.python.org/issue27809 17 msgs #27850: Remove 3DES from cipher list (sweet32 CVE-2016-2183) https://bugs.python.org/issue27850 14 msgs #27128: Add _PyObject_FastCall() https://bugs.python.org/issue27128 12 msgs #27643: test_ctypes fails on AIX with xlc https://bugs.python.org/issue27643 9 msgs #27768: ssl: get CPU cap flags for AESNI and PCLMULQDQ https://bugs.python.org/issue27768 9 msgs #27805: In Python 3, open('/dev/stdout', 'a') raises OSError with errn https://bugs.python.org/issue27805 9 msgs #12319: [http.client] HTTPConnection.request not support "chunked" Tra https://bugs.python.org/issue12319 8 msgs #25658: PyThread assumes pthread_key_t is an integer, which is against https://bugs.python.org/issue25658 7 msgs #12713: argparse: allow abbreviation of sub commands by users https://bugs.python.org/issue12713 6 msgs #25532: infinite loop when running inspect.unwrap over unittest.mock.c https://bugs.python.org/issue25532 6 msgs Issues closed (50) ================== #2466: os.path.ismount doesn't work for mounts the user doesn't have https://bugs.python.org/issue2466 closed by r.david.murray #6057: sqlite3 error classes should be documented https://bugs.python.org/issue6057 closed by berker.peksag #12982: Document that importing .pyo files needs python -O https://bugs.python.org/issue12982 closed by berker.peksag #19504: Change "customise" to "customize". https://bugs.python.org/issue19504 closed by rhettinger #20124: The documentation for the atTime parameter of TimedRotatimeFil https://bugs.python.org/issue20124 closed by python-dev #21718: sqlite3 cursor.description seems to rely on incomplete stateme https://bugs.python.org/issue21718 closed by berker.peksag #23746: sysconfg.is_python_build() is buggy https://bugs.python.org/issue23746 closed by python-dev #25564: Document that IDLE -python difference for `del __builtins__` https://bugs.python.org/issue25564 closed by terry.reedy #25604: [Minor] bug in integer true division algorithm https://bugs.python.org/issue25604 closed by mark.dickinson #25916: wave module readframes now returns bytes not str https://bugs.python.org/issue25916 closed by r.david.murray #26266: add classattribute to enum to handle non-Enum attributes https://bugs.python.org/issue26266 closed by ethan.furman #26488: hashlib command line interface https://bugs.python.org/issue26488 closed by rhettinger #26907: Add missing getsockopt constants https://bugs.python.org/issue26907 closed by r.david.murray #26984: int() can return not exact int instance https://bugs.python.org/issue26984 closed by serhiy.storchaka #27323: ncurses putwin() fails in test_module_funcs https://bugs.python.org/issue27323 closed by serhiy.storchaka #27487: -m switch regression in Python 3.5.2 (under rare circumstances https://bugs.python.org/issue27487 closed by ncoghlan #27539: negative Fraction ** negative int not normalized https://bugs.python.org/issue27539 closed by mark.dickinson #27587: Issues, reported by PVS-Studio static analyzer https://bugs.python.org/issue27587 closed by berker.peksag #27595: Document PEP 495 (Local Time Disambiguation) features https://bugs.python.org/issue27595 closed by python-dev #27598: Add Collection to collections.abc and typing https://bugs.python.org/issue27598 closed by gvanrossum #27614: Race in test_docxmlrpc.py https://bugs.python.org/issue27614 closed by martin.panter #27662: Check against PY_SSIZE_T_MAX instead of PY_SIZE_MAX in List_Ne https://bugs.python.org/issue27662 closed by mark.dickinson #27692: Clean serveral unnecessary NULL checks in exceptions.c https://bugs.python.org/issue27692 closed by serhiy.storchaka #27713: Spurious "platform dependent libraries" warnings when running https://bugs.python.org/issue27713 closed by ned.deily #27714: some test_idle tests are not re-runnable, producing false fail https://bugs.python.org/issue27714 closed by terry.reedy #27782: Multi-phase extension module initialization, inconsistent exce https://bugs.python.org/issue27782 closed by ncoghlan #27787: Avoid test_main() in test_httplib; gc.collect() dangling threa https://bugs.python.org/issue27787 closed by martin.panter #27802: Add __eq__ and __ne__ to collections.abc.Sequence. https://bugs.python.org/issue27802 closed by gvanrossum #27804: IDLE 3.5.2 crashes when typing ^ on keyboard https://bugs.python.org/issue27804 closed by ned.deily #27813: When I assign for a item which list of single item https://bugs.python.org/issue27813 closed by martin.panter #27814: contextlib.suppress: Add whitelist argument https://bugs.python.org/issue27814 closed by ncoghlan #27819: Always distribute sdist packages as gztar. https://bugs.python.org/issue27819 closed by jason.coombs #27821: IDLE custom keymaps don't work anymore (Windows, Python 3.6.0a https://bugs.python.org/issue27821 closed by terry.reedy #27826: Null-pointer dereference in tuplehash() function https://bugs.python.org/issue27826 closed by rhettinger #27828: Regression in http.cookies parsing with single key without val https://bugs.python.org/issue27828 closed by SilentGhost #27832: fractions.Fraction with 3 arguments: error passes silently https://bugs.python.org/issue27832 closed by mark.dickinson #27834: test_datetime fails on "x86 Gentoo Non-Debug with X 3.x" build https://bugs.python.org/issue27834 closed by belopolsky #27836: VCRUNTIME140.DLL not found https://bugs.python.org/issue27836 closed by steve.dower #27837: Logging Cookbook Improvement https://bugs.python.org/issue27837 closed by vinay.sajip #27840: functools.partial: don't copy keywoard arguments in partial_ca https://bugs.python.org/issue27840 closed by ned.deily #27841: Use fast call in method_call() and slot_tp_new() https://bugs.python.org/issue27841 closed by haypo #27843: Spaces in filenames in setuptools https://bugs.python.org/issue27843 closed by brett.cannon #27848: C function calls: use Py_ssize_t rather than C int for number https://bugs.python.org/issue27848 closed by haypo #27849: Spam https://bugs.python.org/issue27849 closed by berker.peksag #27851: re.sub() bug with IGNORECASE https://bugs.python.org/issue27851 closed by r.david.murray #27852: itertools -> flatten_all() https://bugs.python.org/issue27852 closed by rhettinger #27856: Issue while passing the parameter https://bugs.python.org/issue27856 closed by berker.peksag #27857: n identical objects divide into m different groups https://bugs.python.org/issue27857 closed by terry.reedy #27858: Add identity function to functools https://bugs.python.org/issue27858 closed by rhettinger #27862: Make urllib.parse runnable as a script https://bugs.python.org/issue27862 closed by SilentGhost

1 0

PEP 525
by Yury Selivanov 25 Aug '16

25 Aug '16

Hi, I think it's time to discuss PEP 525 on python-dev (pasted below). There were no changes in the PEP since I posted it to python-ideas a couple of weeks ago. One really critical thing that will block PEP acceptance is asynchronous generators (AG) finalization. The problem is to provide a reliable way to correctly close all AGs on program shutdown. To recap: PEP 492 requires an event loop or a coroutine runner to run coroutines. PEP 525 defines AGs using asynchronous iteration protocol, also defined in PEP 492. AGs require an 'async for' statement to iterate over them, which in turn can only be used in a coroutine. Therefore, AGs also require an event loop or a coroutine runner to operate. The finalization problem is related to partial iteration. For instance, let's look at an ordinary synchronous generator: def gen(): try: while True: yield 42 finally: print("done") g = gen() next(g) next(g) del g In the above example, when 'g' is GCed, the interpreter will try to close the generator. It will do that by throwing a GeneratorExit exception into 'g', which would trigger the 'finally' statement. For AGs we have a similar problem. Except that they can have `await` expressions in their `finally` statements, which means that the interpreter can't close them on its own. An event loop is required to run an AG, and an event loop is required to close it correctly. To enable correct AGs finalization, PEP 525 proposes to add a `sys.set_asyncgen_finalizer` API. The idea is to have a finalizer callback assigned to each AG, and when it's time to close the AG, that callback will be called. The callback will be installed by the event loop (or coroutine runner), and should schedule a correct asynchronous finalization of the AG (remember, AGs can have 'await' expressions in their finally statements). The problem with 'set_asyncgen_finalizer' is that the event loop doesn't know about AGs until they are GCed. This can be a problem if we want to write a program that gracefully closes all AGs when the event loop is being closed. There is an alternative approach to finalization of AGs: instead of assigning a finalizer callback to an AG, we can add an API to intercept AG first iteration. That would allow event loops to have weak references to all AGs running under their control: 1. that would make it possible to intercept AGs garbage collection similarly to the currently proposed set_asyncgen_finalizer 2. it would also allow us to implement 'loop.shutdown' coroutine, which would try to asynchronously close all open AGs. The second approach gives event loops more control and allows to implement APIs to collect open resources gracefully. The only downside is that it's a bit harder for event loops to work with. Let's discuss. PEP: 525 Title: Asynchronous Generators Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Discussions-To: <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Jul-2016 Python-Version: 3.6 Post-History: 02-Aug-2016 Abstract ======== PEP 492 introduced support for native coroutines and ``async``/``await`` syntax to Python 3.5. It is proposed here to extend Python's asynchronous capabilities by adding support for *asynchronous generators*. Rationale and Goals =================== Regular generators (introduced in PEP 255) enabled an elegant way of writing complex *data producers* and have them behave like an iterator. However, currently there is no equivalent concept for the *asynchronous iteration protocol* (``async for``). This makes writing asynchronous data producers unnecessarily complex, as one must define a class that implements ``__aiter__`` and ``__anext__`` to be able to use it in an ``async for`` statement. Essentially, the goals and rationale for PEP 255, applied to the asynchronous execution case, hold true for this proposal as well. Performance is an additional point for this proposal: in our testing of the reference implementation, asynchronous generators are **2x** faster than an equivalent implemented as an asynchronous iterator. As an illustration of the code quality improvement, consider the following class that prints numbers with a given delay once iterated:: class Ticker: """Yield numbers from 0 to `to` every `delay` seconds.""" def __init__(self, delay, to): self.delay = delay self.i = 0 self.to = to def __aiter__(self): return self async def __anext__(self): i = self.i if i >= self.to: raise StopAsyncIteration self.i += 1 if i: await asyncio.sleep(self.delay) return i The same can be implemented as a much simpler asynchronous generator:: async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) Specification ============= This proposal introduces the concept of *asynchronous generators* to Python. This specification presumes knowledge of the implementation of generators and coroutines in Python (PEP 342, PEP 380 and PEP 492). Asynchronous Generators ----------------------- A Python *generator* is any function containing one or more ``yield`` expressions:: def func(): # a function return def genfunc(): # a generator function yield We propose to use the same approach to define *asynchronous generators*:: async def coro(): # a coroutine function await smth() async def asyncgen(): # an asynchronous generator function await smth() yield 42 The result of calling an *asynchronous generator function* is an *asynchronous generator object*, which implements the asynchronous iteration protocol defined in PEP 492. It is a ``SyntaxError`` to have a non-empty ``return`` statement in an asynchronous generator. Support for Asynchronous Iteration Protocol ------------------------------------------- The protocol requires two special methods to be implemented: 1. An ``__aiter__`` method returning an *asynchronous iterator*. 2. An ``__anext__`` method returning an *awaitable* object, which uses ``StopIteration`` exception to "yield" values, and ``StopAsyncIteration`` exception to signal the end of the iteration. Asynchronous generators define both of these methods. Let's manually iterate over a simple asynchronous generator:: async def genfunc(): yield 1 yield 2 gen = genfunc() assert gen.__aiter__() is gen assert await gen.__anext__() == 1 assert await gen.__anext__() == 2 await gen.__anext__() # This line will raise StopAsyncIteration. Finalization ------------ PEP 492 requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have ``try..finally`` blocks, as well as ``async with``. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example:: async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 1000): if i == 100: break The above code defines an asynchronous generator that uses ``async with`` to iterate over a database cursor in a transaction. The generator is then iterated over with ``async for``, which interrupts the iteration at some point. The ``square_series()`` generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: 1. Implement an ``aclose`` method on asynchronous generators returning a special *awaitable*. When awaited it throws a ``GeneratorExit`` into the suspended generator and iterates over it until either a ``GeneratorExit`` or a ``StopAsyncIteration`` occur. This is very similar to what the ``close()`` method does to regular Python generators, except that an event loop is required to execute ``aclose()``. 2. Raise a ``RuntimeError``, when an asynchronous generator executes a ``yield`` expression in its ``finally`` block (using ``await`` is fine, though):: async def gen(): try: yield finally: await asyncio.sleep(1) # Can use 'await'. yield # Cannot use 'yield', # this line will trigger a # RuntimeError. 3. Add two new methods to the ``sys`` module: ``set_asyncgen_finalizer()`` and ``get_asyncgen_finalizer()``. The idea behind ``sys.set_asyncgen_finalizer()`` is to allow event loops to handle generators finalization, so that the end user does not need to care about the finalization problem, and it just works. When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. If there is none, a ``RuntimeError`` is raised. This provides a strong guarantee that every asynchronous generator object will always have a finalizer installed by the correct event loop. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an ``aclose()`` call with the loop that was active when the iteration started. For instance, here is how asyncio is modified to allow safe finalization of asynchronous generators:: # asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_finalizer(old_finalizer) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) ``sys.set_asyncgen_finalizer()`` is thread-specific, so several event loops running in parallel threads can use it safely. Asynchronous Generator Object ----------------------------- The object is modeled after the standard Python generator object. Essentially, the behaviour of asynchronous generators is designed to replicate the behaviour of synchronous generators, with the only difference in that the API is asynchronous. The following methods and properties are defined: 1. ``agen.__aiter__()``: Returns ``agen``. 2. ``agen.__anext__()``: Returns an *awaitable*, that performs one asynchronous generator iteration when awaited. 3. ``agen.asend(val)``: Returns an *awaitable*, that pushes the ``val`` object in the ``agen`` generator. When the ``agen`` has not yet been iterated, ``val`` must be ``None``. Example:: async def gen(): await asyncio.sleep(0.1) v = yield 42 print(v) await asyncio.sleep(0.2) g = gen() await g.asend(None) # Will return 42 after sleeping # for 0.1 seconds. await g.asend('hello') # Will print 'hello' and # raise StopAsyncIteration # (after sleeping for 0.2 seconds.) 4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that throws an exception into the ``agen`` generator. Example:: async def gen(): try: await asyncio.sleep(0.1) yield 'hello' except ZeroDivisionError: await asyncio.sleep(0.2) yield 'world' g = gen() v = await g.asend(None) print(v) # Will print 'hello' after # sleeping for 0.1 seconds. v = await g.athrow(ZeroDivisionError) print(v) # Will print 'world' after $ sleeping 0.2 seconds. 5. ``agen.aclose()``: Returns an *awaitable*, that throws a ``GeneratorExit`` exception into the generator. The *awaitable* can either return a yielded value, if ``agen`` handled the exception, or ``agen`` will be closed and the exception will propagate back to the caller. 6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable name and qualified name attributes. 7. ``agen.ag_await``: The object that ``agen`` is currently *awaiting* on, or ``None``. This is similar to the currently available ``gi_yieldfrom`` for generators and ``cr_await`` for coroutines. 8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``: defined in the same way as similar attributes of standard generators. ``StopIteration`` and ``StopAsyncIteration`` are not propagated out of asynchronous generators, and are replaced with a ``RuntimeError``. Implementation Details ---------------------- Asynchronous generator object (``PyAsyncGenObject``) shares the struct layout with ``PyGenObject``. In addition to that, the reference implementation introduces three new objects: 1. ``PyAsyncGenASend``: the awaitable object that implements ``__anext__`` and ``asend()`` methods. 2. ``PyAsyncGenAThrow``: the awaitable object that implements ``athrow()`` and ``aclose()`` methods. 3. ``_PyAsyncGenWrappedValue``: every directly yielded object from an asynchronous generator is implicitly boxed into this structure. This is how the generator implementation can separate objects that are yielded using regular iteration protocol from objects that are yielded using asynchronous iteration protocol. ``PyAsyncGenASend`` and ``PyAsyncGenAThrow`` are awaitables (they have ``__await__`` methods returning ``self``) and are coroutine-like objects (implementing ``__iter__``, ``__next__``, ``send()`` and ``throw()`` methods). Essentially, they control how asynchronous generators are iterated: .. image:: pep-0525-1.png :align: center :width: 80% PyAsyncGenASend and PyAsyncGenAThrow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``PyAsyncGenASend`` is a coroutine-like object that drives ``__anext__`` and ``asend()`` methods and implements the asynchronous iteration protocol. ``agen.asend(val)`` and ``agen.__anext__()`` return instances of ``PyAsyncGenASend`` (which hold references back to the parent ``agen`` object.) The data flow is defined as follows: 1. When ``PyAsyncGenASend.send(val)`` is called for the first time, ``val`` is pushed to the parent ``agen`` object (using existing facilities of ``PyGenObject``.) Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 2. When ``PyAsyncGenASend.throw(*exc)`` is called for the first time, ``*exc`` is throwed into the parent ``agen`` object. Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 3. ``return`` statements in asynchronous generators raise ``StopAsyncIteration`` exception, which is propagated through ``PyAsyncGenASend.send()`` and ``PyAsyncGenASend.throw()`` methods. ``PyAsyncGenAThrow`` is very similar to ``PyAsyncGenASend``. The only difference is that ``PyAsyncGenAThrow.send()``, when called first time, throws an exception into the parent ``agen`` object (instead of pushing a value into it.) New Standard Library Functions and Types ---------------------------------------- 1. ``types.AsyncGeneratorType`` -- type of asynchronous generator object. 2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()`` methods to set up asynchronous generators finalizers in event loops. 3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()`` introspection functions. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. In Python 3.5 it is a ``SyntaxError`` to define an ``async def`` function with a ``yield`` expression inside, therefore it's safe to introduce asynchronous generators in 3.6. Performance =========== Regular Generators ------------------ There is no performance degradation for regular generators. The following micro benchmark runs at the same speed on CPython with and without asynchronous generators:: def gen(): i = 0 while i < 100000000: yield i i += 1 list(gen()) Improvements over asynchronous iterators ---------------------------------------- The following micro-benchmark shows that asynchronous generators are about **2.3x faster** than asynchronous iterators implemented in pure Python:: N = 10 ** 7 async def agen(): for i in range(N): yield i class AIter: def __init__(self): self.i = 0 def __aiter__(self): return self async def __anext__(self): i = self.i if i >= N: raise StopAsyncIteration self.i += 1 return i Design Considerations ===================== ``aiter()`` and ``anext()`` builtins ------------------------------------ Originally, PEP 492 defined ``__aiter__`` as a method that should return an *awaitable* object, resulting in an asynchronous iterator. However, in CPython 3.5.2, ``__aiter__`` was redefined to return asynchronous iterators directly. To avoid breaking backwards compatibility, it was decided that Python 3.6 will support both ways: ``__aiter__`` can still return an *awaitable* with a ``DeprecationWarning`` being issued. Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot add a synchronous implementation of ``aiter()`` built-in. Therefore, it is proposed to wait until Python 3.7. Asynchronous list/dict/set comprehensions ----------------------------------------- Syntax for asynchronous comprehensions is unrelated to the asynchronous generators machinery, and should be considered in a separate PEP. Asynchronous ``yield from`` --------------------------- While it is theoretically possible to implement ``yield from`` support for asynchronous generators, it would require a serious redesign of the generators implementation. ``yield from`` is also less critical for asynchronous generators, since there is no need provide a mechanism of implementing another coroutines protocol on top of coroutines. And to compose asynchronous generators a simple ``async for`` loop can be used:: async def g1(): yield 1 yield 2 async def g2(): async for v in g1(): yield v Why the ``asend()`` and ``athrow()`` methods are necessary ---------------------------------------------------------- They make it possible to implement concepts similar to ``contextlib.contextmanager`` using asynchronous generators. For instance, with the proposed design, it is possible to implement the following pattern:: @async_context_manager async def ctx(): await open() try: yield finally: await close() async with ctx(): await ... Another reason is that it is possible to push data and throw exceptions into asynchronous generators using the object returned from ``__anext__`` object, but it is hard to do that correctly. Adding explicit ``asend()`` and ``athrow()`` will pave a safe way to accomplish that. In terms of implementation, ``asend()`` is a slightly more generic version of ``__anext__``, and ``athrow()`` is very similar to ``aclose()``. Therefore having these methods defined for asynchronous generators does not add any extra complexity. Example ======= A working example with the current reference implementation (will print numbers from 0 to 9 with one second delay):: async def ticker(delay, to): for i in range(to): yield i await asyncio.sleep(delay) async def run(): async for i in ticker(1, 10): print(i) import asyncio loop = asyncio.get_event_loop() try: loop.run_until_complete(run()) finally: loop.close() Implementation ============== The complete reference implementation is available at [1]_. References ========== .. [1] https://github.com/1st1/cpython/tree/async_gen Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

5 9