Mailman 3 September 2016 - Python-Dev

Push PEP 528 (use utf8 on Windows) right now, but revert before 3.6 if needed
by Victor Stinner 05 Sep '16

05 Sep '16

Hi, I just spoke with Steve Dower (thanks for the current sprint!) about the PEP 528. We somehow agreed that we need to push his implementation of the PEP right now to get enough time to test as much applications as possible on Windows to have a wide view of possible all regressions. The hope is that enough users will test the first Python 3.6 beta 1 (feature freeze!) with their app on Windows. If we find blocker points before Python 3.6 final, we still have time to revert code to restore the Python 3.5 behaviour (use ANSI code page for bytes). What do you think? Note: First I was strongly opposed to any kind of change related to encodings on Windows, but then I made my own tests and had to confess that my proposed changes break the world. Steve's approach makes more sense and is more realistic. Victor

2 1

PEP 526 ready for review: Syntax for Variable and Attribute Annotations
by Guido van Rossum 05 Sep '16

05 Sep '16

I'm happy to present PEP 526 for your collective review: https://www.python.org/dev/peps/pep-0526/ (HTML) https://github.com/python/peps/blob/master/pep-0526.txt (source) There's also an implementation ready: https://github.com/ilevkivskyi/cpython/tree/pep-526 I don't want to post the full text here but I encourage feedback on the high-order ideas, including but not limited to - Whether (given PEP 484's relative success) it's worth adding syntax for variable/attribute annotations. - Whether the keyword-free syntax idea proposed here is best: NAME: TYPE TARGET: TYPE = VALUE Note that there's an extensive list of rejected ideas in the PEP; please be so kind to read it before posting here: https://www.python.org/dev/peps/pep-0526/#rejected-proposals-and-things-lef… -- --Guido van Rossum (python.org/~guido)

11 34

Please reject or postpone PEP 526
by Mark Shannon 04 Sep '16

04 Sep '16

Hi everyone, I think we should reject, or at least postpone PEP 526. PEP 526 represents a major change to the language, however there are, I believe, a number of technical flaws with the PEP. It is probable that with significant revisions it can be a worthwhile addition to the language, but that cannot happen in time for 3.6 beta 1 (in 11 days). PEP 526 claims to be an extension of PEP 484, but I don't think that is entirely correct. PEP 484 was primarily about defining the semantics of pre-existing syntax. PEP 526 is about adding new syntax. Historically the bar for adding new syntax has been set very high. I don't think that PEP 526, in its current form, reaches that bar. Below is a list of the issues I have with the PEP as it stands. In many cases it makes it more effort than type comments ======================================================== Type hints should be as easy to use as possible, and that means pushing as much work as possible onto the checker, and not burdening the programmer. Attaching type hints to variables, rather than expressions, reduces the potential for inference. This makes it harder for programmer, but easier for the checker, which is the wrong way around. For example,, given a function: def spam(x: Optional[List[int]])->None: ... With type comments, this is intuitively correct and should type check: def eggs(cond:bool): if cond: x = None else: x = [] # type: List[int] spam(x) # Here we can infer the type of x With PEP 526 we loose the ability to infer types. def eggs(cond:bool): if cond: x = None # Not legal due to type declaration below else: x: List[int] = [] spam(x) So we need to use a more complex type def eggs(cond:bool): x: Optional[List[int]] if cond: x = None # Now legal else: x: = [] spam(x) I don't think this improves readability. Whether this is an acceptable change is debatable, but it does need some debate. It limits the use of variables ============================== In Python a name (variable) is just a binding that refers to an object. A name only exists in a meaningful sense once an object has been assigned to it. Any attempt to use that name, without an object bound to it, will result in a NameError. PEP 526 makes variables more than just bindings, as any rebinding must conform to the given type. This looses us some of the dynamism for which we all love Python. Quoting from the PEP: ``` a: int a: str # Static type checker will warn about this. ``` In other words, it is illegal for a checker to split up the variable, even though it is straightforward to do so. However, without the type declarations, ``` a = 1 a = "Hi" ``` is just fine. Useless, but fine. We should be free to add extra variables, whenever we choose, for clarity. For example, total = foo() - bar() should not be treated differently from: revenue = foo() tax = bar() total = revenue - tax If types are inferred, there is no problem. However, if they must be declared, then the use of meaningfully named variables is discouraged. [A note about type-inference: Type inference is not a universal panacea, but it can make life a lot easier for programmers in statically type languages. Languages like C# use local type inference extensively and it means that many variables often do not need their type declared. We should take care not to limit the ability of checkers to infer values and types and make programmers' lives easier. Within a function, type inference is near perfect, failing only occasionally for some generic types. One place where type inference definitely breaks down is across calls, which is why PEP 484 is necessary. ] It is premature =============== There are still plenty of issues to iron out w.r.t. PEP 484 types. I don't think we should be adding more syntax, until we have a *precise* idea of what is required. PEP 484 states: "If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version." Has it proved useful in general? I don't think it has. Maybe it will in future, but it hasn't yet. It seems confused about class attributes and instance attributes ================================================================ The PEP also includes a section of how to define class attributes and instance attributes. It seems that everything needs to be defined in the class scope, even it is not an attribute of the class, but of its instances. This seems confusing, both to human reader and machine analyser. Example from PEP 526: class Starship: captain: str = 'Picard' damage: int stats: ClassVar[Dict[str, int]] = {} def __init__(self, damage: int, captain: str = None): self.damage = damage if captain: self.captain = captain # Else keep the default With type hints as they currently exist, the same code is shorter and doesn't contaminate the class namespace with the 'damage' attribute. class Starship: captain = 'Picard' stats = {} # type: Dict[str, int] def __init__(self, damage: int, captain: str = None): self.damage = damage # Can infer type as int if captain: self.captain = captain # Can infer type as str This isn't an argument against adding type syntax for attributes in general, just that the form suggested in PEP 526 doesn't seem to follow Python semantics. One could imagine applying minimal PEP 526 style hints, with standard Python semantics and relying on type inference, as follows: class Starship: captain = 'Picard' stats: Dict[str, int] = {} def __init__(self, damage: int, captain: str = None): self.damage = damage if captain: self.captain = captain The PEP overstates the existing use of static typing in Python ============================================================== Finally, in the rejected proposal section, under "Should we introduce variable annotations at all?" it states that "Variable annotations have already been around for almost two years in the form of type comments, sanctioned by PEP 484." I don't think that this is entirely true. PEP 484 was about the syntax for types, declaring parameter and return types, and declaring custom types to be generic. PEP 484 does include a description of type comments, but they are always annotations on assignment statements and were primarily intended for use in stub files. Please don't turn Python into some sort of inferior Java. There is potential in this PEP, but in its current form I think it should be rejected. Cheers, Mark.

10 15

Patch reviews
by Christian Heimes 04 Sep '16

04 Sep '16

Hi, I have 7 patches for 3.6 ready for merging. The new features were discussed on Security-SIG and reviewed by Victor or GPS. The patches just need one final review and an ACK. The first three patches should land in 2.7, 3.4 and 3.5, too. http://bugs.python.org/issue26470 Make OpenSSL module compatible with OpenSSL 1.1.0 https://bugs.python.org/issue27850 Remove 3DES from cipher list (sweet32 CVE-2016-2183) Also adds ChaCha20 Poly1305 http://bugs.python.org/issue27691 X509 cert with GEN_RID subject alt name causes SytemError http://bugs.python.org/issue27866 ssl: get list of enabled ciphers https://bugs.python.org/issue27744 Add AF_ALG (Linux Kernel crypto) to socket module http://bugs.python.org/issue16113 Add SHA-3 and SHAKE (Keccak) support http://bugs.python.org/issue26798 add BLAKE2 to hashlib Christian

3 4

Tweak to PEP 523 for storing a tuple in co_extra
by Brett Cannon 04 Sep '16

04 Sep '16

Below is the `co_extra` section of PEP 523 with the update saying that users are expected to put a tuple in the field for easier simultaneous use of the field. Since the `co_extra` discussions do not affect CPython itself I'm planning on landing the changes stemming from the PEP probably on Monday. ---------- Expanding ``PyCodeObject`` -------------------------- One field is to be added to the ``PyCodeObject`` struct [#pycodeobject]_:: typedef struct { ... PyObject *co_extra; /* "Scratch space" for the code object. */ } PyCodeObject; The ``co_extra`` will be ``NULL`` by default and will not be used by CPython itself. Third-party code is free to use the field as desired. Values stored in the field are expected to not be required in order for the code object to function, allowing the loss of the data of the field to be acceptable. The field will be freed like all other fields on ``PyCodeObject`` during deallocation using ``Py_XDECREF()``. Code using the field is expected to always store a tuple in the field. This allows for multiple users of the field to not trample over each other while being as performant as possible. Typical usage of the field is expected to roughly follow the following pseudo-code:: if co_extra is None: data = DataClass() co_extra = (data,) else: assert isinstance(co_extra, tuple) for x in co_extra: if isinstance(x, DataClass): data = x break else: data = DataClass() co_extra += (data,) Using a list was considered but was found to be less performant, and with a key use-case being JIT usage the performance consideration it was deemed more important to use a tuple than a list. A tuple also makes more sense semantically as the objects stored in the tuple will be heterogeneous. A dict was also considered, but once again performance was more important. While a dict will have constant overhead in looking up data, the overhead for the common case of a single object being stored in the data structure leads to a tuple having better performance characteristics (i.e. iterating a tuple of length 1 is faster than the overhead of hashing and looking up an object in a dict).

5 14

PEP 525, third round, better finalization
by Yury Selivanov 04 Sep '16

04 Sep '16

Hi, I've spent quite a while thinking and experimenting with PEP 525 trying to figure out how to make asynchronous generators (AG) finalization reliable. I've tried to replace the callback for GCed with a callback to intercept first iteration of AGs. Turns out it's very hard to work with weak-refs and make asyncio event loop to reliably track and shutdown all open AGs. My new approach is to replace the "sys.set_asyncgen_finalizer(finalizer)" function with "sys.set_asyncgen_hooks(firstiter=None, finalizer=None)". This design allows us to: 1. intercept first iteration of an AG. That makes it possible for event loops to keep a weak set of all "open" AGs, and to implement a "shutdown" method to close the loop and close all AGs *reliably*. 2. intercept AGs GC. That makes it possible to call "aclose" on GCed AGs to guarantee that 'finally' and 'async with' statements are properly closed. 3. in later Python versions we can add more hooks, although I can't think of anything else we need to add right now. I'm posting below the only updated PEP section. The latest PEP revision should also be available on python.org shortly. All new proposed changes are available to play with in my fork of CPython here: https://github.com/1st1/cpython/tree/async_gen Finalization ------------ PEP 492 requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have ``try..finally`` blocks, as well as ``async with``. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example:: async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 1000): if i == 100: break The above code defines an asynchronous generator that uses ``async with`` to iterate over a database cursor in a transaction. The generator is then iterated over with ``async for``, which interrupts the iteration at some point. The ``square_series()`` generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: 1. Implement an ``aclose`` method on asynchronous generators returning a special *awaitable*. When awaited it throws a ``GeneratorExit`` into the suspended generator and iterates over it until either a ``GeneratorExit`` or a ``StopAsyncIteration`` occur. This is very similar to what the ``close()`` method does to regular Python generators, except that an event loop is required to execute ``aclose()``. 2. Raise a ``RuntimeError``, when an asynchronous generator executes a ``yield`` expression in its ``finally`` block (using ``await`` is fine, though):: async def gen(): try: yield finally: await asyncio.sleep(1) # Can use 'await'. yield # Cannot use 'yield', # this line will trigger a # RuntimeError. 3. Add two new methods to the ``sys`` module: ``set_asyncgen_hooks()`` and ``get_asyncgen_hooks()``. The idea behind ``sys.set_asyncgen_hooks()`` is to allow event loops to intercept asynchronous generators iteration and finalization, so that the end user does not need to care about the finalization problem, and everything just works. ``sys.set_asyncgen_hooks()`` accepts two arguments: * ``firstiter``: a callable which will be called when an asynchronous generator is iterated for the first time. * ``finalizer``: a callable which will be called when an asynchronous generator is about to be GCed. When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. If there is none, a ``RuntimeError`` is raised. This provides a strong guarantee that every asynchronous generator object will always have a finalizer installed by the correct event loop. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an ``aclose()`` call with the loop that was active when the iteration started. For instance, here is how asyncio is modified to allow safe finalization of asynchronous generators:: # asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_hooks = sys.get_asyncgen_hooks() sys.set_asyncgen_hooks(finalizer=self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_hooks(*old_hooks) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) The second argument, ``firstiter``, allows event loops to maintain a weak set of asynchronous generators instantiated under their control. This makes it possible to implement "shutdown" mechanisms to safely finalize all open generators and close the event loop. ``sys.set_asyncgen_hooks()`` is thread-specific, so several event loops running in parallel threads can use it safely. ``sys.get_asyncgen_hooks()`` returns a namedtuple-like structure with ``firstiter`` and ``finalizer`` fields.

5 7

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8
by Adam Bartoš 03 Sep '16

03 Sep '16

Nick Coghlan (ncoghlan at gmail.com) on Sat Sep 3 12:27:44 EDT 2016 wrote: > After also reading the Windows console encoding PEP, I realised > there's a couple of missing discussions here regarding the impacts on > sys.argv, os.environ, and os.environb. > > The reason that's relevant is that "sys.getfilesystemencoding" is a > bit of a misnomer, as it's also used to determine the assumed encoding > of command line arguments and environment variables. > > Regarding sys.argv, AFAIK Unicode arguments work well on Python 3. Even non-BMP characters are transferred correctly. Adam Bartoš

1 0

Update on PEP 523 and adding a co_extra field to code objects
by Brett Cannon 03 Sep '16

03 Sep '16

For quick background for those that don't remember, part of PEP 523 proposed adding a co_extra field to code objects along with making the frame evaluation function pluggable ( https://www.python.org/dev/peps/pep-0523/#expanding-pycodeobject). The idea was that things like JITs and debuggers could use the field as a scratch space of sorts to store data related to the code object. People who objected to the new field did either for memory ("it adds another pointer to the struct that won't be typically used"), or for conceptual reasons ("the code object is immutable and you're proposing a mutable field"). The latter is addressed by not exposing the field in Python and clearly stating that code should never expect the field to be filled. For the former issue of whether the memory was worth it, Dino has been testing whether the field is necessary for performance from a JIT perspective. Well, Dino found the time to test Pyjion without the co_extra field and it isn't pretty. With the field, Pyjion is faster than stock Python in 15 benchmarks ( https://github.com/Microsoft/Pyjion/tree/master/Perf). Removing the co_extra field and using an unordered_map from the C++ STL drops that number to 2. Performance is even worse if we try and use a Python dictionary instead. That means we still want to find a solution to attach arbitrary data to code objects without sacrificing performance. One proposal is what's in PEP 523 for the extra field. Another option is to make the memory allocator for code objects pluggable and introduce a new flag that signals that the object was created using a non-default allocator. Obviously we prefer the former solution due to its simplicity. :) Anyway, we would like to get this settled this week so that I can get whatever solution we agree to (if any) in next week in time for Python 3.6b1 feature freeze that would be greatly appreciated.

13 35

What should a good type checker do? (was: Please reject or postpone PEP 526)
by Koos Zevenhoven 03 Sep '16

03 Sep '16

On Fri, Sep 2, 2016 at 9:04 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote: > >> A good checker should be able to infer that x is a union type at the >> point that it's passed to spam, even without the type annotation. For >> example: >> >> def eggs(cond:bool): >> if cond: >> x = 1 >> else: >> x = 1.5 >> spam(x) # a good type checker infers that x is of type Union[int, float] > > Oh I really hope not. I wouldn't call that a *good* type checker. I > would call that a type checker that is overly permissive. I guess it's perfectly fine if we disagree about type checking ideals, and I can imagine the justification for you thinking that way. There can also be different type checkers, and which can have different modes. But assume (a) that the above function is perfectly working code, and spam(...) accepts Union[int, float]. Why would I want the type checker to complain? Then again, (b) instead of that being working code, it might be an error and spam only takes float. No problem, the type checker will catch that. In case of (b), to get the behavior you want (but in my hypothetical semantics), this could be annotated as def eggs(cond:bool): x : float if cond: x = 1 # type checker says error else: x = 1.5 spam(x) So here the programmer thinks the type of x should be more constrained than what spam(...) accepts. Or you might have something like this def eggs(cond:bool): if cond: x = 1 else: x = 1.5 # type checker has inferred x to be Union[int, float] x : float # type checker finds an error spam(x) Here, the same error is found, but at a different location. > Maybe you think that it's okay because ints and floats are somewhat > compatible. But suppose I wrote: > > if cond: > x = HTTPServer(*args) > else: > x = 1.5 It might be clear by now, but no, that's not why I wrote that. That was just a slightly more "realistic" example than this HTTP & 1.5 one. [...] > Do you have a better idea for variable > syntax? I had one but it turned out it was worse. -- Koos > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com -- + Koos Zevenhoven + http://twitter.com/k7hoven +

6 6

Re: [Python-Dev] [New-bugs-announce] [issue27948] f-strings: allow backslashes only in the string parts, not in the expression parts
by Eric V. Smith 03 Sep '16

03 Sep '16

I'm aware of the buildbot failures due to this commit. I'm working on it. Sorry about that: tests passed on my machine. Eric. On 09/03/2016 09:24 AM, Eric V. Smith wrote: > > New submission from Eric V. Smith: > > See issue 27921. > > Currently (and for 3.6 beta 1), backslashes are not allowed anywhere in f-strings. This needs to be changed to allow them in the string parts, but not in the expression parts. > > Also, require that the start and end of an expression be literal '{' and '}, not escapes like '\0x7b' and '\u007d'. > > ---------- > assignee: eric.smith > components: Interpreter Core > messages: 274294 > nosy: eric.smith > priority: normal > severity: normal > stage: needs patch > status: open > title: f-strings: allow backslashes only in the string parts, not in the expression parts > type: behavior > versions: Python 3.6 > > _______________________________________ > Python tracker <report(a)bugs.python.org> > <http://bugs.python.org/issue27948> > _______________________________________ > _______________________________________________ > New-bugs-announce mailing list > New-bugs-announce(a)python.org > https://mail.python.org/mailman/listinfo/new-bugs-announce >

1 0