Mailman 3 September 2016 - Python-Dev

PEP 525, fourth update
by Yury Selivanov Sept. 7, 2016

Sept. 7, 2016

Hi, I've updated PEP 525 with a new section about asyncio changes. Essentially, asyncio event loop will get a new "shutdown_asyncgens" method that allows to close the loop and all associated AGs with it reliably. Only the updated section is pasted below: asyncio ------- The asyncio event loop will use ``sys.set_asyncgen_hooks()`` API to maintain a weak set of all scheduled asynchronous generators, and to schedule their ``aclose()`` coroutine methods when it is time for generators to be GCed. To make sure that asyncio programs can finalize all scheduled asynchronous generators reliably, we propose to add a new event loop method ``loop.shutdown_asyncgens(*, timeout=30)``. The method will schedule all currently open asynchronous generators to close with an ``aclose()`` call. After calling the ``loop.shutdown_asyncgens()`` method, the event loop will issue a warning whenever a new asynchronous generator is iterated for the first time. The idea is that after requesting all asynchronous generators to be shutdown, the program should not execute code that iterates over new asynchronous generators. An example of how ``shutdown_asyncgens`` should be used:: try: loop.run_forever() # or loop.run_until_complete(...) finally: loop.shutdown_asyncgens() loop.close() - Yury

2 3

What's the status of PEP 515?
by Eric V. Smith Sept. 7, 2016

Sept. 7, 2016

The implementation of '_' in numeric literals is here: http://bugs.python.org/issue26331 And to add '_' in int.__format__ is here: http://bugs.python.org/issue27080 But I don't want to add support in int.__format__ unless numeric literal support is added. So, Georg and Serhiy: is issue 26331 going to get committed? If so, I'll commit 27080 (or you can). I just don't want the second part of PEP 515 to not make the deadline if the first part makes it in at the last minute. Thanks! Eric.

3 2

Do PEP 526 type declarations define the types of variables or not?
by Mark Shannon Sept. 7, 2016

Sept. 7, 2016

Hi, PEP 526 states that "This PEP aims at adding syntax to Python for annotating the types of variables" and Guido seems quite insistent that the declarations are for the types of variables. However, I get the impression that most (all) of the authors and proponents of PEP 526 are quite keen to emphasise that the PEP in no way limits type checkers from doing what they want. This is rather contradictory. The behaviour of a typechecker is defined by the typesystem that it implements. Whether a type annotation determines the type of a variable or an expression alters changes what typesystems are feasible. So, stating that annotations define the type of variables *does* limit what a typechecker can or cannot do. Unless of course, others may have a different idea of what the "type of a variable" means. To me, it means it means that for all assignments `var = expr` the type of `expr` must be a subtype of the variable, and for all uses of var, the type of the use is the same as the type of the variable. In this example: def bar()->Optional[int]: ... def foo()->int: x:Optional[int] = bar() if x is None: return -1 return x According to PEP 526 the annotation `x:Optional[int]` means that the *variable* `x` has the type `Optional[int]`. So what is the type of `x` in `return x`? If it is `Optional[int]`, then a type checker is obliged to reject this code. If it is `int` then what does "type of a variable" actually mean, and why aren't the other uses of `x` int as well? Cheers, Mark.

12 15

The Amazing Unreferenced Weakref
by Larry Hastings Sept. 7, 2016

Sept. 7, 2016

This is all about current (3.6) trunk. In Objects/weakrefobject.c, we have the function PyObject_ClearWeakRefs(). This is called when a generic object that supports weakrefs is destroyed; this is the code that calls the callbacks. Here's a little paragraph of code from the center: for (i = 0; i < count; ++i) { PyWeakReference *next = current->wr_next; if (((PyObject *)current)->ob_refcnt > 0) { Py_INCREF(current); PyTuple_SET_ITEM(tuple, i * 2, (PyObject *) current); PyTuple_SET_ITEM(tuple, i * 2 + 1, current->wr_callback); } else { Py_DECREF(current->wr_callback); } current->wr_callback = NULL; clear_weakref(current); current = next; } "current" is the doubly-linked list of PyWeakReference objects stored inside the object that's getting destroyed. My question: under what circumstances would ob_refcnt ever be 0? The tp_dealloc handler for PyWeakReference * objects removes it from this list and frees the memory. How could the reference count reach 0 without tp_dealloc being called and it being removed from the list? Scratching my head like crazy, //arry/ p.s. If you're thinking "why does he care?", understanding this would maybe help with the Gilectomy. So yes there's a point to this question.

2 1

PEP 447: Add __getdescriptor__ to metaclasses
by Ronald Oussoren Sept. 7, 2016

Sept. 7, 2016

Hi, It’s getting a tradition for me to work on PEP 447 during the EuroPython sprints and disappear afterwards. Hopefully I can manage to avoid the latter step this year… Last year the conclusion appeared to be that this is an acceptable PEP, but Mark Shannon had a concern about a default implementation for __getdescriptor__ on type in <https://mail.python.org/pipermail/python-dev/2015-July/140938.html> (follow the link for more context): > "__getdescriptor__" is fundamentally different from "__getattribute__" in that > is defined in terms of itself. > > object.__getattribute__ is defined in terms of type.__getattribute__, but > type.__getattribute__ just does > dictionary lookups. However defining type.__getattribute__ in terms of > __descriptor__ causes a circularity as > __descriptor__ has to be looked up on a type. > > So, not only must the cycle be broken by special casing "type", but that > "__getdescriptor__" can be defined > not only by a subclass, but also a metaclass that uses "__getdescriptor__" to > define "__getdescriptor__" on the class. > (and so on for meta-meta classes, etc.) My reaction that year is in <https://mail.python.org/pipermail/python-dev/2015-August/141114.html>. As I wrote there I did not fully understand the concerns Mark has, probably because I’m focussed too much on the implementation in CPython. If removing type.__getdescriptor__ and leaving this special method as an optional hook for subclasses of type fixes the conceptual concerns then that’s fine by me. I used type.__getdescriptor__ as the default implementation both because it appears to be cleaner to me and because this gives subclasses an easy way to access the default implementation. The implementation of the PEP in issue 18181 <http://bugs.python.org/issue18181> does special-case type.__getdescriptor__ but only as an optimisation, the code would work just as well without that special casing because the normal attribute lookup machinery is not used when accessing special methods written in C. That is, the implementation of object.__getattribute__ directly accesses fields of the type struct at the C level. Some magic behavior appears to be necessary even without the addition of __getdescriptor__ (type is a subclass of itself, object.__getattribute__ has direct access to dict.__getitem__, …). I’m currently working on getting the patch in 18181 up-to-date w.r.t. the current trunk, the patch in the issue no longer applies cleanly. After that I’ll try to think up some tests that seriously try to break the new behaviour, and I want to update a patch I have for PyObjC to make use of the new functionality to make sure that the PEP actually fixes the issues I had w.r.t. builtin.super’s behavior. What is the best way forward after that? As before this is a change in behavior that, unsurprisingly, few core devs appear to be comfortable with evaluating, combined with new functionality that will likely see little use beyond PyObjC (although my opinions of that shouldn’t carry much weight, I thought that decorators would have limited appeal when those where introduced and couldn’t have been more wrong about that). Ronald P.S. The PEP itself: <https://www.python.org/dev/peps/pep-0447>

4 6

Requesting on python directories
by RAMU V Sept. 6, 2016

Sept. 6, 2016

Sir I learn the basics of python from online and I am pretty much confident with my basics and I want to show them to the outside world by publishing them as packages in python. There were set of instructions for that process but I could not figure out the exact process so please help me with this because I learnt python out of my interest but not based on my syllabus or for job. Its like a refreshing enjoyment for my brain. Sent from Mail for Windows 10

3 2

Re: [Python-Dev] Please reject or postpone PEP 526
by Mark Shannon Sept. 6, 2016

Sept. 6, 2016

On 02/09/16 19:19, Steven D'Aprano wrote: > On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote: >> "I'm not seeing what distinction you think you are making here. What >> distinction do you see between: >> >> x: int = func(value) >> >> and >> >> x = func(value) #type: int" >> >> Not sure whether I agree with Mark on this particular point, but the >> difference I see here is that the first describes what types x may >> ever contain, while the latter describes what type of being assigned >> to x right here. So one is a variable annotation while the other is an >> expression annotation. > > Ultimately Python is a dynamically typed language, and that's not > changing. This means types are fundamentally associated with *values*, > not *variables* (names). But in practice, you can go a long way by > pretending that it is the variable that carries the type. That's the > point of the static type checker: if you see that x holds an int here, > then assume (unless told differently) that x should always be an int. > Because in practice, most exceptions to that are due to bugs, or at > least sloppy code. > > Of course, it is up to the type checker to decide how strict it wants to > be, whether to treat violations as a warning or a error, whether to > offer the user a flag to set the behaviour, etc. None of this is > relevant to the PEP. The PEP only specifies the syntax, leaving > enforcement or non-enforcement to the checker, and it says: > > PEP 484 introduced type hints, a.k.a. type annotations. While its > main focus was function annotations, it also introduced the notion > of type comments to annotate VARIABLES [emphasis added] If I recall, Guido and I agreed to differ on that point. We still do, it seems. We did manage to agree on the syntax though. > > not expressions. And: > > This PEP aims at adding syntax to Python for annotating the types > of variables and attributes, instead of expressing them through > comments > > which to me obviously implies that the two ways (type comment, and > variable type hint) are intended to be absolutely identical in > semantics, at least as far as the type-checker is concerned. The key difference is in placement. PEP 484 style variable = value # annotation Which reads to me as if the annotation refers to the value. PEP 526 variable: annotation = value Which reads very much as if the annotation refers to the variable. That is a change in terms of semantics and a change for the worse, in terms of expressibility. Cheers, Mark.

10 41

Where are the list and array.array implementations in CPython source?
by Jonathan Goble Sept. 6, 2016

Sept. 6, 2016

I'd like to study the CPython implementations of lists and array.array instances for a personal project of mine, but I've very unfamiliar with the Python source code as it pertains to internals like this. Which files would I need to look at to do this, and are there a few particular functions/structures I should pay attention to? I'm just looking for a brief pointer in the right direction here, not a full explanation of how it works -- I'll get that from studying the source code. :-) (Sorry if this isn't the best place to post, but I felt that a question about CPython's internals fit slightly better on python-dev rather than python-list, since this is where those familiar with that code are more likely to see the post.)

3 3

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8
by Paul Moore Sept. 6, 2016

Sept. 6, 2016

On 5 September 2016 at 14:36, Steve Dower <steve.dower(a)python.org> wrote: > The best fix is to use a buffered reader, which will read all the available > bytes and then let you .read(1), even if it happens to be an incomplete > character. But this is sys.stdin.buffer.raw, we're talking about. People can't really layer anything on top of that, it's precisely because they are trying to *bypass* the existing layering (that doesn't work the way that they need it to, because it blocks) that is the problem here. > We could theoretically add buffering to the raw reader to handle one character, > which would allow very small reads from raw, but that severely complicates > things and the advice to use a buffered reader is good advice anyway. Can you provide an example of how I'd rewrite the code that I quoted previously to follow this advice? Note - this is not theoretical, I expect to have to provide a PR to fix exactly this code should this change go in. At the moment I can't find a way that doesn't impact the (currently working and not expected to need any change) Unix version of the code, most likely I'll have to add buffering of 4-byte reads (which as you say is complex). The problem I have is that we're forcing application code to do the buffering to cater for Windows (where you're proposing that the raw IO layer doesn't handle it and will potentially fail reads of <4 bytes). Code written for POSIX doesn't need to do that, and the additional maintenance overhead is potentially large enough to put POSIX developers off adding the necessary code - this is in direct contrast to the proposal to make fsencoding UTF-8 to make it easier for POSIX-compatible code to "just work" on Windows. If the goals are to handle Unicode correctly for stdin, and to work in a way that POSIX-compatible code works without special effort on Windows, then as far as I can see we have to handle the buffering of partial reads of UTF-8 code sequences (because POSIX does so). If, on the other hand, we just want Unicode to work on Windows, and we're not looking for POSIX code to work without change, then the proposed behaviour is OK (although I still maintain it needs to be flagged, as it's very close to being a compatibility break in practice, even if it's technically within the rules). Paul PS I'm not 100% sure that under POSIX read() will return partial UTF-8 byte sequences. I think it must, because otherwise a lot of code I've seen would be broken, but if a POSIX expert can confirm or deny my assumption, that would be great.

3 6

TLS handshake performance boost
by Christian Heimes Sept. 6, 2016

Sept. 6, 2016

Hi, I have yet another patch for the ssl module, http://bugs.python.org/issue19500 . The patch adds support for SSL session resumption on the client side. A SSLContext automatically handles server-side session. SSL sessions speed up successive TLS connections to the same host considerable. My naïve benchmark shows about 15 to 20% performance improvements for short-lived connections to PyPI. In real-life applications with keep-alive, the speed-up will be a bit smaller. Cory expects that requests is going to be about 5% faster for subsequent requests. https://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html has more information on the topic. Why is session handling different on the client side? OpenSSL does not re-use sessions on the client side automatically. To use session resumptions a SSL_SESSION must be copied from an established SSLSocket to a new SSLSocket before the handshake. OpenSSL has further restrictions, e.g. both sockets must use the same SSLContext. Session cannot be shared between SSLContext. My patch takes care of these details. The basic features are pretty much done and tested. But I won't be able to write all documentation by the end of the week or to write a high-level mechanism to auto-reuse sessions. I still like to get the feature in before Monday. What do you think? Are you fine with low-level session feature and reduced documentation for the beta release? Christian

2 1