Python-Dev April 2018

python-dev@python.org

103 participants
90 discussions

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

by Jeroen Demeyer

Hello, I just saw this PEP. There is a bit of overlap between PEP 573 and PEP 575 since these both change the calling convention for built-in methods. In particular, PEP 575 also proposes to add a "defining class" member (for different reasons). In PEP 575, this is added to the PyCFunction struct itself instead of a separate struct PyCMethod. It would be nice to justify whether you really need a new class (PyCMethod_Type) to support METH_METHOD. It looks strange to me that the class of some object depends on an implementation detail like whether METH_METHOD is specified. The current PEP 573 implies that backwards compatibility concerns would arise every time that METH_METHOD is added to an existing method. People have asked questions on PEP 575 about that: it would break code depending on "types.BuiltinFunctionType" for example. You could instead just change PyCFunctionObject to add that field (that's what I did in PEP 575). For practical reasons, it would be nice to implement PEP 573 and PEP 575 together as they affect the same code (assuming that both PEPs are accepted of course). Jeroen.

2 years, 10 months

Introducing python.zulipchat.com

by Brett Cannon

As an experiment we have gotten an instance of Zulip running for Python's development at https://python.zulipchat.com (IOW this is for discussing the development *of* Python only*)*. As Guido has put it you can view Zulip like "hyper-interactive email" as we have streams corresponding to equivalent mailing lists and all messages fall under a topic so conversations are on-topic. The instance is currently rather open at the suggestion of Zulip, so people can create new streams, add bots, etc. There are already bots I have added for commit notifications, Travis failures, and deployments of Bedevere and The Knights Who Say Ni. The invite message when you sign up mentions all this, the CoC, etc., so do give it a read. Otherwise feel free to join and help us decide if this is useful enough to make a permanent thing for Python's development!

2 years, 10 months

https://bugs.python.org/issue33127 breaks pip / easy_install / pipenv etc in corporate networks on MS Windows using self-signed certificates

by Oleg Sivokon

It is common practice in corporate networks that connect MS Windows machines to redirect all (encrypted included) traffic through company's router. For this purpose routers are usually configured to act as a CA. However, the certificate issued by such "CA" will of course not be found in the certificates distributed with LibreSSL (how would they even know?). MS Windows networking, however, has a way to configure these policies. Prior to this issue, Python relied on the OS libraries to implement TLS protocol, so the overall setup worked transparently for users. Since 3.6.5, however, this is no longer possible (requires alteration of certificates distributed with Python). I'm asking that this be made configurable / possible to disable using simple means, perhaps an environment variable / registry key or similar. PS. I still cannot register to the bug tracker (never received a confirmation email), this is why you are reading this email. - Best. Oleg This communication and all information contained in or attached to it is confidential, intended solely for the addressee, may be legally privileged and is the intellectual property of one of the companies of NEX Group plc ("NEX") or third parties. If you are not the intended addressee or receive this message in error, please immediately delete all copies of it and notify the sender. We have taken precautions to minimise the risk of transmitting software viruses, but we advise you to carry out your own virus checks on any attachments. We do not accept liability for any loss or damage caused by software viruses. NEX reserves the right to monitor all communications. We do not accept any legal responsibility for the content of communications, and no communication shall be considered legally binding. Furthermore, if the content of this communication is personal or unconnected with our business, we accept no liability or responsibility for it. NEX Group plc is a public limited company registered in England and Wales under number 10013770 and certain of its affiliates are authorised and regulated by regulatory authorities. For further regulatory information please see www.NEX.com.

2 years, 10 months

PEP 561 implemented and minor clarification

by Ethan Smith

Hello, I've updated PEP 561 to clarify that any installed stub package should supersede an installed inline package. In other words if there is: /global/site-packages/pkg/ /user/site-packages/pkg-stubs/ Even if pkg in the global site packages is found first and marks that it supports types, the stub package should supersede it. I also point to mypy's docs on its implementation of the PEP (which can be read about here: https://mypy.readthedocs.io/en/latest/installed_packages.html). The implementation has been merged into master and will be available in the 0.590 release. As always, the PEP text is replicated below. Cheers! Ethan ============== PEP: 561 Title: Distributing and Packaging Type Information Author: Ethan Smith <ethan(a)ethanhs.me> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017 Abstract ======== PEP 484 introduced type hinting to Python, with goals of making typing gradual and easy to adopt. Currently, typing information must be distributed manually. This PEP provides a standardized means to leverage existing tooling to package and distribute type information with minimal work and an ordering for type checkers to resolve modules and collect this information for type checking. Rationale ========= Currently, package authors wish to distribute code that has inline type information. Additionally, maintainers would like to distribute stub files to keep Python 2 compatibility while using newer annotation syntax. However, there is no standard method to distribute packages with type information. Also, if one wished to ship stub files privately the only method available would be via setting ``MYPYPATH`` or the equivalent to manually point to stubs. If the package can be released publicly, it can be added to typeshed [1]_. However, this does not scale and becomes a burden on the maintainers of typeshed. In addition, it ties bug fixes in stubs to releases of the tool using typeshed. PEP 484 has a brief section on distributing typing information. In this section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for shipping stub files. However, manually adding a path to stub files for each third party library does not scale. The simplest approach people have taken is to add ``site-packages`` to their ``MYPYPATH``, but this causes type checkers to fail on packages that are highly dynamic (e.g. sqlalchemy and Django). Definition of Terms =================== The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are to be interpreted as described in RFC 2119. "inline" - the types are part of the runtime code using PEP 526 and 3107 syntax. "stubs" - files containing only type information, empty of runtime code. "Distributions" are the packaged files which are used to publish and distribute a release. [3]_ "Module" a file containing Python runtime code or stubbed type information. "Package" a directory or directories that namespace Python modules. Specification ============= There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that users of typing wish to create: 1. The package maintainer would like to add type information inline. 2. The package maintainer would like to add type information via stubs. 3. A third party or package maintainer would like to share stub files for a package, but the maintainer does not want to include them in the source of the package. This PEP aims to support these scenarios and make them simple to add to packaging and deployment. The two major parts of this specification are the packaging specifications and the resolution order for resolving module type information. The type checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_. New third party stub libraries SHOULD distribute stubs via the third party packaging methods proposed in this PEP in place of being added to typeshed. Typeshed will remain in use, but if maintainers are found, third party stubs in typeshed MAY be split into their own package. Packaging Type Information -------------------------- In order to make packaging and distributing type information as simple and easy as possible, packaging and distribution is done through existing frameworks. Package maintainers who wish to support type checking of their code MUST add a marker file named ``py.typed`` to their package supporting typing. This marker applies recursively: if a top-level package includes it, all its sub-packages MUST support type checking as well. To have this file installed with the package, maintainers can use existing packaging options such as ``package_data`` in distutils, shown below. Distutils option example:: setup( ..., package_data = { 'foopkg': ['py.typed'], }, ..., ) For namespace packages, the ``py.typed`` file should be in the submodules of the namespace, to avoid conflicts and for clarity. This PEP does not support distributing typing information as part of module-only distributions. The code should be refactored into a package-based distribution and indicate that the package supports typing as described above. Stub Only Packages '''''''''''''''''' For package maintainers wishing to ship stub files containing all of their type information, it is preferred that the ``*.pyi`` stubs are alongside the corresponding ``*.py`` files. However, the stubs can also be put in a separate package and distributed separately. Third parties can also find this method useful if they wish to distribute stub files. The name of the stub package MUST follow the scheme ``foopkg-stubs`` for type stubs for the package named ``foopkg``. Note that for stub only packages adding a py.typed marker is not needed since the name ``*-stubs`` is enough to indicate it is a source of typing information. Third parties seeking to distribute stub files are encouraged to contact the maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a third party stub only package can be created. In addition, stub-only distributions SHOULD indicate which version(s) of the runtime package are supported by indicating the runtime distribution's version(s) through normal dependency data. For example, the stub package ``flyingcircus-stubs`` can indicate the versions of the runtime ``flyingcircus`` distribution it supports through ``install_requires`` in distutils-based tools, or the equivalent in other packaging tools. Note that in pip 9.0, if you update ``flyingcircus-stubs``, it will update ``flyingcircus``. In pip 9.0, you can use the ``--upgrade-strategy=only-if-needed`` flag. In pip 10.0 this is the default behavior. Type Checker Module Resolution Order ------------------------------------ The following is the order in which type checkers supporting this PEP SHOULD resolve modules containing type information: 1. User code - the files the type checker is running on. 2. Stubs or Python source manually put in the beginning of the path. Type checkers SHOULD provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages. 3. Stub packages - these packages SHOULD supersede any installed inline package. They can be found at ``foopkg-stubs`` for package ``foopkg``. 4. Inline packages - if there is nothing overriding the installed package, and it opts into type checking, inline types SHOULD be used. 5. Typeshed (if used) - Provides the stdlib types and several third party libraries. Type checkers that check a different Python version than the version they run on MUST find the type information in the ``site-packages``/``dist-packages`` of that Python version. This can be queried e.g. ``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. The normal resolution order of checking ``*.pyi`` before ``*.py`` will be maintained. Implementation ============== The proposed scheme of indicating support for typing is completely backwards compatible, and requires no modification to package tooling. A sample package with inline types is available [typed_pkg]_, as well as a sample package checker [pkg_checker]_ which reads the metadata of installed packages and reports on their status as either not typed, inline typed, or a stub package. The mypy type checker has an implementation of PEP 561 searching which can be read about in the mypy docs [4]_. Acknowledgements ================ This PEP would not have been possible without the ideas, feedback, and support of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, Nathaniel Smith, and Guido van Rossum. Version History =============== * 2018-04-09 * Add reference to mypy implementation * Clarify stub package priority. * 2018-02-02 * Change stub only package suffix to be -stubs not _stubs. * Note that py.typed is not needed for stub only packages. * Add note about pip and upgrading stub packages. * 2017-11-12 * Rewritten to use existing tooling only * No need to indicate kind of type information in metadata * Name of marker file changed from ``.typeinfo`` to ``py.typed`` * 2017-11-10 * Specification re-written to use package metadata instead of distribution metadata. * Removed stub only packages and merged into third party packages spec. * Removed suggestion for typecheckers to consider checking runtime versions * Implementations updated to reflect PEP changes. * 2017-10-26 * Added implementation references. * Added acknowledgements and version history. * 2017-10-06 * Rewritten to use .distinfo/METADATA over a distutils specific command. * Clarify versioning of third party stub packages. * 2017-09-11 * Added information about current solutions and typeshed. * Clarify rationale. References ========== .. [1] Typeshed (https://github.com/python/typeshed) .. [2] PEP 484, Storing and Distributing Stub Files (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-fil…) .. [3] PEP 426 definitions (https://www.python.org/dev/peps/pep-0426/) .. [4] Example implementation in a type checker (https://mypy.readthedocs.io/en/latest/installed_packages.html) .. [typed_pkg] Sample typed package (https://github.com/ethanhs/sample-typed-package) .. [pkg_checker] Sample package checker (https://github.com/ethanhs/check_typedpkg) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

2 years, 10 months

Re: [Python-Dev] PEP 572: Assignment Expressions

by Ivan Levkivskyi

Replying also to the list. On 22 April 2018 at 09:14, Ivan Levkivskyi <levkivskyi(a)gmail.com> wrote: > On 20 April 2018 at 21:59, Guido van Rossum <guido(a)python.org> wrote: > >> Does the PEP currently propose to *allow* that horrible example? I >> thought Tim Peters successfully pleaded to *only* allow a single "NAME := >> <expr>". You don't have to implement this restriction -- we know it's >> possible to implement, and if specifying this alone were to pull enough >> people from -1 to +0 there's a lot of hope! >> >> > * FWIW I an -1 on anything but a simple name. > > * Also Tim proposed a good idea to call these "binding expressions". > Because in contrasts the different purposes. Binding expressions would be > probably typically used to (temporarily) name an expression, while > assignment statements are actually creating "variables" -- long living > names intended to be accessed externally to a class/module. The latter > access can be programmed to trigger arbitrary complex code (see properties, > __getattr__/__setattr__, etc). > > * Re implementing restrictions: there is a CST -> AST step that will allow > to easily prohibit unwanted forms (FWIW this is how unpacking an chaining > is prohibited for annotated assignments). > > * Re using plain "=": Although I am still using this in C quite often, I > was bitten badly by this several times when I was younger, I don't want a > similar experience when _learning_ Python. > > Modulo these points I would be +0 on the PEP. > > -- > Ivan > > >

2 years, 10 months

PEP 572: Now with 25% less reference implementation!

by Chris Angelico

Working on the reference implementation for PEP 572 is turning out to be a massive time sink, both on my personal schedule and on the PEP's discussion. I can't just hold off all discussion on a topic until I figure out whether something is possible or not, because that could take me several days, even a week or more. And considering the massive backlash against the proposal, it seems like a poor use of my time to try to prove that something's impossible, find that I don't know enough about grammar editing to be able to say anything more than "well, I couldn't do it, but someone else might be able to", and then try to resume the discussion with no more certainty than we had before. So here's the PEP again, simplified. I'm fairly sure it's just going to be another on a growing list of rejected PEPs to my name, and I'm done with trying to argue some of these points. Either the rules get simplified, or they don't. Trying to simplify the rules and maintain perfect backward compatibility is just making the rules even more complicated. PEP 572, if accepted, *will* change the behaviour of certain constructs inside comprehensions, mainly due to interactions with class scope that make no sense unless you know how they're implemented internally. The official tutorial pretends that comprehensions are "equivalent to" longhand: https://docs.python.org/3/tutorial/datastructures.html?highlight=equivalent… https://docs.python.org/3/howto/functional.html?highlight=equivalent#genera… and this is an inaccuracy for the sake of simplicity. PEP 572 will make this far more accurate; the only difference is that the comprehension is inside a function. Current semantics are far more bizarre than that. Do you want absolutely 100% backward compatibility? Then reject this PEP. Or better still, keep using Python 3.7, and don't upgrade to 3.8, in case something breaks. Do you want list comprehensions that make better sense? Then accept that some code will need to change, if it tried to use the same name in multiple scopes, or tried to use ancient Python 2 semantics with a yield expression in the outermost iterable. I'm pretty much ready for pronouncement. https://www.python.org/dev/peps/pep-0572/ ChrisA PEP: 572 Title: Assignment Expressions Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018, 17-Apr-2018 Abstract ======== This is a proposal for creating a way to assign to variables within an expression. Additionally, the precise scope of comprehensions is adjusted, to maintain consistency and follow expectations. Rationale ========= Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts. Merely introducing a way to assign as an expression would create bizarre edge cases around comprehensions, though, and to avoid the worst of the confusions, we change the definition of comprehensions, causing some edge cases to be interpreted differently, but maintaining the existing behaviour in the majority of situations. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a **named expression** can appear. This is of the form ``target := expr`` where ``expr`` is any valid Python expression, and ``target`` is any valid assignment target. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:: # Handle a matched regex if (match := pattern.search(data)) is not None: ... # A more explicit alternative to the 2-arg form of iter() invocation while (value := read_next_item()) is not None: ... # Share a subexpression between a comprehension filter clause and its output filtered_data = [y for x in data if (y := f(x)) is not None] Differences from regular assignment statements ---------------------------------------------- Most importantly, since ``:=`` is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions. An assignment statement can assign to multiple targets, left-to-right:: x = y = z = 0 The equivalent assignment expression is parsed as separate binary operators, and is therefore processed right-to-left, as if it were spelled thus:: assert 0 == (x := (y := (z := 0))) Augmented assignment is not supported in expression form:: >>> x +:= 1 File "<stdin>", line 1 x +:= 1 ^ SyntaxError: invalid syntax Otherwise, the semantics of assignment are identical in statement and expression forms. Alterations to comprehensions ----------------------------- The current behaviour of list/set/dict comprehensions and generator expressions has some edge cases that would behave strangely if an assignment expression were to be used. Therefore the proposed semantics are changed, removing the current edge cases, and instead altering their behaviour *only* in a class scope. As of Python 3.7, the outermost iterable of any comprehension is evaluated in the surrounding context, and then passed as an argument to the implicit function that evaluates the comprehension. Under this proposal, the entire body of the comprehension is evaluated in its implicit function. Names not assigned to within the comprehension are located in the surrounding scopes, as with normal lookups. As one special case, a comprehension at class scope will **eagerly bind** any name which is already defined in the class scope. A list comprehension can be unrolled into an equivalent function. With Python 3.7 semantics:: numbers = [x + y for x in range(3) for y in range(4)] # Is approximately equivalent to def <listcomp>(iterator): result = [] for x in iterator: for y in range(4): result.append(x + y) return result numbers = <listcomp>(iter(range(3))) Under the new semantics, this would instead be equivalent to:: def <listcomp>(): result = [] for x in range(3): for y in range(4): result.append(x + y) return result numbers = <listcomp>() When a class scope is involved, a naive transformation into a function would prevent name lookups (as the function would behave like a method):: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " prefixed_names = [prefix + name for name in names] With Python 3.7 semantics, this will evaluate the outermost iterable at class scope, which will succeed; but it will evaluate everything else in a function:: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " def <listcomp>(iterator): result = [] for name in iterator: result.append(prefix + name) return result prefixed_names = <listcomp>(iter(names)) The name ``prefix`` is thus searched for at global scope, ignoring the class name. Under the proposed semantics, this name will be eagerly bound; and the same early binding then handles the outermost iterable as well. The list comprehension is thus approximately equivalent to:: class X: names = ["Fred", "Barney", "Joe"] prefix = "> " def <listcomp>(names=names, prefix=prefix): result = [] for name in names: result.append(prefix + name) return result prefixed_names = <listcomp>() With list comprehensions, this is unlikely to cause any confusion. With generator expressions, this has the potential to affect behaviour, as the eager binding means that the name could be rebound between the creation of the genexp and the first call to ``next()``. It is, however, more closely aligned to normal expectations. The effect is ONLY seen with names that are looked up from class scope; global names (eg ``range()``) will still be late-bound as usual. One consequence of this change is that certain bugs in genexps will not be detected until the first call to ``next()``, where today they would be caught upon creation of the generator. Recommended use-cases ===================== Simplifying list comprehensions ------------------------------- A list comprehension can map and filter efficiently by capturing the condition:: results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0] Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:: stuff = [[y := f(x), x/y] for x in range(5)] # There are a number of less obvious ways to spell this in current # versions of Python, such as: # Inline helper function stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] # Extra 'for' loop - potentially could be optimized internally stuff = [[y, x/y] for x in range(5) for y in [f(x)]] # Using a mutable cache object (various forms possible) c = {} stuff = [[c.update(y=f(x)) or c['y'], x/c['y']] for x in range(5)] In all cases, the name is local to the comprehension; like iteration variables, it cannot leak out into the surrounding context. Capturing condition values -------------------------- Assignment expressions can be used to good effect in the header of an ``if`` or ``while`` statement:: # Proposed syntax while (command := input("> ")) != "quit": print("You entered:", command) # Capturing regular expression match objects # See, for instance, Lib/pydoc.py, which uses a multiline spelling # of this effect if match := re.search(pat, text): print("Found:", match.group(0)) # Reading socket data until an empty string is returned while data := sock.read(): print("Received data:", data) # Equivalent in current Python, not caring about function return value while input("> ") != "quit": print("You entered a command.") # To capture the return value in current Python demands a four-line # loop header. while True: command = input("> "); if command == "quit": break print("You entered:", command) Particularly with the ``while`` loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Rejected alternative proposals ============================== Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. Alternative spellings --------------------- Broadly the same semantics as the current proposal, but spelled differently. 1. ``EXPR as NAME``:: stuff = [[f(x) as y, x/y] for x in range(5)] Since ``EXPR as NAME`` already has meaning in ``except`` and ``with`` statements (with different semantics), this would create unnecessary confusion or require special-casing (eg to forbid assignment within the headers of these statements). 2. ``EXPR -> NAME``:: stuff = [[f(x) -> y, x/y] for x in range(5)] This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow ``y <- f(x)`` is not possible in Python, as it would be interpreted as less-than and unary minus.) This syntax has a slight advantage over 'as' in that it does not conflict with ``with`` and ``except`` statements, but otherwise is equivalent. 3. Adorning statement-local names with a leading dot:: stuff = [[(f(x) as .y), x/.y] for x in range(5)] # with "as" stuff = [[(.y := f(x)), x/.y] for x in range(5)] # with ":=" This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable's scope is encoded into its name, making refactoring harder. 4. Adding a ``where:`` to any statement to create local name bindings:: value = x**2 + 2*x where: x = spam(1, 4, 7, q) Execution order is inverted (the indented body is performed first, followed by the "header"). This requires a new keyword, unless an existing keyword is repurposed (most likely ``with:``). See PEP 3150 for prior discussion on this subject (with the proposed keyword being ``given:``). 5. ``TARGET from EXPR``:: stuff = [[y from f(x), x/y] for x in range(5)] This syntax has fewer conflicts than ``as`` does (conflicting only with the ``raise Exc from Exc`` notation), but is otherwise comparable to it. Instead of paralleling ``with expr as target:`` (which can be useful but can also be confusing), this has no parallels, but is evocative. Special-casing conditional statements ------------------------------------- One of the most popular use-cases is ``if`` and ``while`` statements. Instead of a more general solution, this proposal enhances the syntax of these two statements to add a means of capturing the compared value:: if re.search(pat, text) as match: print("Found:", match.group(0)) This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return `''` when done), and completely useless in more complicated cases (eg where the condition is ``f(x) < 0`` and you want to capture the value of ``f(x)``). It also has no benefit to list comprehensions. Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction of possible use-cases, even in ``if``/``while`` statements. Special-casing comprehensions ----------------------------- Another common use-case is comprehensions (list/set/dict, and genexps). As above, proposals have been made for comprehension-specific solutions. 1. ``where``, ``let``, or ``given``:: stuff = [(y, x/y) where y = f(x) for x in range(5)] stuff = [(y, x/y) let y = f(x) for x in range(5)] stuff = [(y, x/y) given y = f(x) for x in range(5)] This brings the subexpression to a location in between the 'for' loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, ``where`` reads the most cleanly, but also has the greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` methods, as does ``tkinter.dnd.Icon`` in the standard library). 2. ``with NAME = EXPR``:: stuff = [(y, x/y) with y = f(x) for x in range(5)] As above, but reusing the `with` keyword. Doesn't read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into "longhand" for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why "with NAME = EXPR:" cannot be used as a statement on its own. 3. ``with EXPR as NAME``:: stuff = [(y, x/y) with f(x) as y for x in range(5)] As per option 2, but using ``as`` rather than an equals sign. Aligns syntactically with other uses of ``as`` for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of ``with`` inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax. Regardless of the spelling chosen, this introduces a stark difference between comprehensions and the equivalent unrolled long-hand form of the loop. It is no longer possible to unwrap the loop into statement form without reworking any name bindings. The only keyword that can be repurposed to this task is ``with``, thus giving it sneakily different semantics in a comprehension than in a statement; alternatively, a new keyword is needed, with all the costs therein. Lowering operator precedence ---------------------------- There are two logical precedences for the ``:=`` operator. Either it should bind as loosely as possible, as does statement-assignment; or it should bind more tightly than comparison operators. Placing its precedence between the comparison and arithmetic operators (to be precise: just lower than bitwise OR) allows most uses inside ``while`` and ``if`` conditions to be spelled without parentheses, as it is most likely that you wish to capture the value of something, then perform a comparison on it:: pos = -1 while pos := buffer.find(search_term, pos + 1) >= 0: ... Once find() returns -1, the loop terminates. If ``:=`` binds as loosely as ``=`` does, this would capture the result of the comparison (generally either ``True`` or ``False``), which is less useful. While this behaviour would be convenient in many situations, it is also harder to explain than "the := operator behaves just like the assignment statement", and as such, the precedence for ``:=`` has been made as close as possible to that of ``=``. Migration path ============== The semantic changes to list/set/dict comprehensions, and more so to generator expressions, may potentially require migration of code. In many cases, the changes simply make legal what used to raise an exception, but there are some edge cases that were previously legal and now are not, and a few corner cases with altered semantics. The Outermost Iterable ---------------------- As of Python 3.7, the outermost iterable in a comprehension is special: it is evaluated in the surrounding context, instead of inside the comprehension. Thus it is permitted to contain a ``yield`` expression, to use a name also used elsewhere, and to reference names from class scope. Also, in a genexp, the outermost iterable is pre-evaluated, but the rest of the code is not touched until the genexp is first iterated over. Class scope is now handled more generally (see above), but if other changes require the old behaviour, the iterable must be explicitly elevated from the comprehension:: # Python 3.7 def f(x): return [x for x in x if x] def g(): return [x for x in [(yield 1)]] # With PEP 572 def f(x): return [y for y in x if y] def g(): sent_item = (yield 1) return [x for x in [sent_item]] This more clearly shows that it is g(), not the comprehension, which is able to yield values (and is thus a generator function). The entire comprehension is consistently in a single scope. The following expressions would, in Python 3.7, raise exceptions immediately. With the removal of the outermost iterable's special casing, they are now equivalent to the most obvious longhand form:: gen = (x for x in rage(10)) # NameError gen = (x for x in 10) # TypeError (not iterable) gen = (x for x in range(1/0)) # ZeroDivisionError def <genexp>(): for x in rage(10): yield x gen = <genexp>() # No exception yet tng = next(gen) # NameError Open questions ============== Importing names into comprehensions ----------------------------------- A list comprehension can use and update local names, and they will retain their values from one iteration to another. It would be convenient to use this feature to create rolling or self-effecting data streams:: progressive_sums = [total := total + value for value in data] This will fail with UnboundLocalError due to ``total`` not being initalized. Simply initializing it outside of the comprehension is insufficient - unless the comprehension is in class scope:: class X: total = 0 progressive_sums = [total := total + value for value in data] At other scopes, it may be beneficial to have a way to fetch a value from the surrounding scope. Should this be automatic? Should it be controlled with a keyword? Hypothetically (and using no new keywords), this could be written:: total = 0 progressive_sums = [total := total + value import nonlocal total for value in data] Translated into longhand, this would become:: total = 0 def <listcomp>(total=total): result = [] for value in data: result.append(total := total + value) return result progressive_sums = <listcomp>() ie utilizing the same early-binding technique that is used at class scope. Frequently Raised Objections ============================ Why not just turn existing assignment into an expression? --------------------------------------------------------- C and its derivatives define the ``=`` operator as an expression, rather than a statement as is Python's way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between ``if (x == y)`` and ``if (x = y)`` belies their drastically different semantics. Thus this proposal uses ``:=`` to clarify the distinction. This could be used to create ugly code! --------------------------------------- So can anything else. This is a tool, and it is up to the programmer to use it where it makes sense, and not use it where superior constructs can be used. With assignment expressions, why bother with assignment statements? ------------------------------------------------------------------- The two forms have different flexibilities. The ``:=`` operator can be used inside a larger expression; the ``=`` statement can be augmented to ``+=`` and its friends. The assignment statement is a clear declaration of intent: this value is to be assigned to this target, and that's it. Why not use a sublocal scope and prevent namespace pollution? ------------------------------------------------------------- Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from ``for`` loops or other constructs, and can be solved the same way: ``del`` the name once it is no longer needed, or prefix it with an underscore. Names bound within a comprehension are local to that comprehension, even in the outermost iterable, and can thus be used freely without polluting the surrounding namespace. (The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2]_) Style guide recommendations =========================== As this adds another way to spell some of the same effects as can already be done, it is worth noting a few broad recommendations. These could be included in PEP 8 and/or other style guides. 1. If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent. 2. If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead. Acknowledgements ================ The author wishes to thank Guido van Rossum and Nick Coghlan for their considerable contributions to this proposal, and to members of the core-mentorship mailing list for assistance with implementation. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/assignment-expressions) .. [2] Pivotal post regarding inline assignment semantics (https://mail.python.org/pipermail/python-ideas/2018-March/049409.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

2 years, 10 months

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2018-04-13 - 2018-04-20) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6587 (+15) closed 38484 (+32) total 45071 (+47) Open issues with patches: 2573 Issues opened (30) ================== #33274: minidom removeAttributeNode returns None https://bugs.python.org/issue33274 opened by iter #33275: glob.glob should explicitly note that results aren't sorted https://bugs.python.org/issue33275 opened by Ben FrantzDale #33276: Clarify that __path__ can't be set to just anything https://bugs.python.org/issue33276 opened by brett.cannon #33277: Deprecate __loader__, __package__, __file__, and __cached__ on https://bugs.python.org/issue33277 opened by brett.cannon #33278: libexpat uses HAVE_SYSCALL_GETRANDOM instead of HAVE_GETRANDOM https://bugs.python.org/issue33278 opened by lemburg #33280: Update link to Tcl/Tk 8.6 man pages in tkinter.rst https://bugs.python.org/issue33280 opened by adelfino #33281: ctypes.util.find_library not working on macOS https://bugs.python.org/issue33281 opened by Ian Burgwin #33282: Subprocess Popen communicate hung if stdin not given to Popen, https://bugs.python.org/issue33282 opened by JatinGoel #33284: Increase test coverage for numbers.py https://bugs.python.org/issue33284 opened by Barry Devlin #33286: Conflict between tqdm and multiprocessing on windows https://bugs.python.org/issue33286 opened by schwemro #33289: askcolor is returning floats for r,g,b values instead of ints https://bugs.python.org/issue33289 opened by Bryan.Oakley #33294: Support complex expressions for py-print command. https://bugs.python.org/issue33294 opened by Martin Liška #33295: ERROR: test_sites_no_connection_close (test.test_urllib2net.Ot https://bugs.python.org/issue33295 opened by inada.naoki #33296: datetime.datetime.utcfromtimestamp call decimal causes precisi https://bugs.python.org/issue33296 opened by anglister #33297: Mention Pillow package on tkinter.rst to work with more image https://bugs.python.org/issue33297 opened by adelfino #33300: Bad usage example in id() DocString https://bugs.python.org/issue33300 opened by gneff #33301: Add __contains__ to pathlib https://bugs.python.org/issue33301 opened by Alok Singh #33302: The search for pyvenv.cfg doesn't match PEP 405 https://bugs.python.org/issue33302 opened by mattheww #33303: ElementTree Comment text isn't escaped https://bugs.python.org/issue33303 opened by johnburnett #33305: Improve syntax error for numbers with leading zero https://bugs.python.org/issue33305 opened by steven.daprano #33306: Improving SyntaxError for unmatched parentheses https://bugs.python.org/issue33306 opened by serhiy.storchaka #33309: Unittest Mock objects do not freeze arguments they are called https://bugs.python.org/issue33309 opened by slacknate #33311: cgitb: remove parentheses when the error is in module https://bugs.python.org/issue33311 opened by sblondon #33312: ubsan undefined behavior sanitizer flags struct _dictkeysobjec https://bugs.python.org/issue33312 opened by gregory.p.smith #33314: Bad rendering in the documentation for the os module https://bugs.python.org/issue33314 opened by pablogsal #33315: Allow queue.Queue to be used in type annotations https://bugs.python.org/issue33315 opened by sproshev #33316: PyThread_release_lock always fails https://bugs.python.org/issue33316 opened by Ivan.Pozdeev #33317: `repr()` of string in NFC and NFD forms does not differ https://bugs.python.org/issue33317 opened by pekka.klarck #33318: Move folding tuples of constants into compiler.c from peephole https://bugs.python.org/issue33318 opened by serhiy.storchaka #33319: `subprocess.run` documentation doesn't tell is using `stdout=P https://bugs.python.org/issue33319 opened by pekka.klarck Most recent 15 issues with no replies (15) ========================================== #33319: `subprocess.run` documentation doesn't tell is using `stdout=P https://bugs.python.org/issue33319 #33318: Move folding tuples of constants into compiler.c from peephole https://bugs.python.org/issue33318 #33316: PyThread_release_lock always fails https://bugs.python.org/issue33316 #33315: Allow queue.Queue to be used in type annotations https://bugs.python.org/issue33315 #33314: Bad rendering in the documentation for the os module https://bugs.python.org/issue33314 #33311: cgitb: remove parentheses when the error is in module https://bugs.python.org/issue33311 #33309: Unittest Mock objects do not freeze arguments they are called https://bugs.python.org/issue33309 #33306: Improving SyntaxError for unmatched parentheses https://bugs.python.org/issue33306 #33303: ElementTree Comment text isn't escaped https://bugs.python.org/issue33303 #33302: The search for pyvenv.cfg doesn't match PEP 405 https://bugs.python.org/issue33302 #33301: Add __contains__ to pathlib https://bugs.python.org/issue33301 #33297: Mention Pillow package on tkinter.rst to work with more image https://bugs.python.org/issue33297 #33294: Support complex expressions for py-print command. https://bugs.python.org/issue33294 #33289: askcolor is returning floats for r,g,b values instead of ints https://bugs.python.org/issue33289 #33284: Increase test coverage for numbers.py https://bugs.python.org/issue33284 Most recent 15 issues waiting for review (15) ============================================= #33318: Move folding tuples of constants into compiler.c from peephole https://bugs.python.org/issue33318 #33316: PyThread_release_lock always fails https://bugs.python.org/issue33316 #33314: Bad rendering in the documentation for the os module https://bugs.python.org/issue33314 #33312: ubsan undefined behavior sanitizer flags struct _dictkeysobjec https://bugs.python.org/issue33312 #33311: cgitb: remove parentheses when the error is in module https://bugs.python.org/issue33311 #33306: Improving SyntaxError for unmatched parentheses https://bugs.python.org/issue33306 #33305: Improve syntax error for numbers with leading zero https://bugs.python.org/issue33305 #33297: Mention Pillow package on tkinter.rst to work with more image https://bugs.python.org/issue33297 #33295: ERROR: test_sites_no_connection_close (test.test_urllib2net.Ot https://bugs.python.org/issue33295 #33294: Support complex expressions for py-print command. https://bugs.python.org/issue33294 #33284: Increase test coverage for numbers.py https://bugs.python.org/issue33284 #33280: Update link to Tcl/Tk 8.6 man pages in tkinter.rst https://bugs.python.org/issue33280 #33266: 2to3 doesn't parse all valid string literals https://bugs.python.org/issue33266 #33263: Asyncio server enters an invalid state after a request with SO https://bugs.python.org/issue33263 #33262: Deprecate shlex.split(None) to read from stdin. https://bugs.python.org/issue33262 Top 10 most discussed issues (10) ================================= #33312: ubsan undefined behavior sanitizer flags struct _dictkeysobjec https://bugs.python.org/issue33312 11 msgs #32232: building extensions as builtins is broken in 3.7 https://bugs.python.org/issue32232 8 msgs #33261: inspect.isgeneratorfunction fails on hand-created methods https://bugs.python.org/issue33261 8 msgs #24882: ThreadPoolExecutor doesn't reuse threads until #threads == max https://bugs.python.org/issue24882 6 msgs #33295: ERROR: test_sites_no_connection_close (test.test_urllib2net.Ot https://bugs.python.org/issue33295 6 msgs #11594: 2to3 does not preserve line endings https://bugs.python.org/issue11594 5 msgs #33257: Race conditions in Tkinter with non-threaded Tcl https://bugs.python.org/issue33257 5 msgs #14573: json iterencode can not handle general iterators https://bugs.python.org/issue14573 4 msgs #33131: Upgrade to pip 10 for Python 3.7 https://bugs.python.org/issue33131 4 msgs #33280: Update link to Tcl/Tk 8.6 man pages in tkinter.rst https://bugs.python.org/issue33280 4 msgs Issues closed (31) ================== #11485: Default SDK value on MacOSX needs changing https://bugs.python.org/issue11485 closed by ned.deily #14499: Extension module builds fail with Xcode 4.3 on OS X 10.7 due t https://bugs.python.org/issue14499 closed by ned.deily #19019: Investigate using Apple clang for building OS X installers https://bugs.python.org/issue19019 closed by ned.deily #24414: MACOSX_DEPLOYMENT_TARGET set incorrectly by configure in Pytho https://bugs.python.org/issue24414 closed by ned.deily #32726: macOS installer and framework enhancements and changes for 3.7 https://bugs.python.org/issue32726 closed by ned.deily #33176: Allow memoryview.toreadonly() https://bugs.python.org/issue33176 closed by pitrou #33184: Update OpenSSL to 1.1.0h / 1.0.2o https://bugs.python.org/issue33184 closed by steve.dower #33185: Python 3.7.0b3 fails in pydoc where b2 did not. https://bugs.python.org/issue33185 closed by ncoghlan #33189: pygettext doesn't work with f-strings https://bugs.python.org/issue33189 closed by serhiy.storchaka #33205: GROWTH_RATE prevents dict shrinking https://bugs.python.org/issue33205 closed by inada.naoki #33264: Remove to-be-deprecated urllib.request.urlretrieve function re https://bugs.python.org/issue33264 closed by orsenthil #33269: InteractiveConsole behaves differently on terminal, within scr https://bugs.python.org/issue33269 closed by terry.reedy #33270: tags for anonymous code objects should be interned https://bugs.python.org/issue33270 closed by serhiy.storchaka #33271: Exception handling should match subclasses, not subtypes https://bugs.python.org/issue33271 closed by serhiy.storchaka #33279: Py_BuildValue is causing crashes with the "L" and "K" format c https://bugs.python.org/issue33279 closed by serhiy.storchaka #33283: Mention PNG as a supported image format by Tcl/Tk https://bugs.python.org/issue33283 closed by serhiy.storchaka #33285: pip upgrade runtime crash https://bugs.python.org/issue33285 closed by paul.moore #33287: "pip iinstall packageName.py" fails https://bugs.python.org/issue33287 closed by berker.peksag #33288: Spam https://bugs.python.org/issue33288 closed by zach.ware #33290: Python.org macOS pkg installs pip3 as pip https://bugs.python.org/issue33290 closed by ned.deily #33291: issue in the default setup.py which comes with the Python 3.6. https://bugs.python.org/issue33291 closed by brett.cannon #33292: Fix secrets.randbelow docstring https://bugs.python.org/issue33292 closed by serhiy.storchaka #33293: Using datetime.datetime.utcnow().timestamp() in Python3.6.0 ca https://bugs.python.org/issue33293 closed by ned.deily #33298: Wrap only constants with _PyCode_ConstantKey() in the compiler https://bugs.python.org/issue33298 closed by serhiy.storchaka #33299: Return an object itself for some types in _PyCode_ConstantKey( https://bugs.python.org/issue33299 closed by serhiy.storchaka #33304: Syntax Error on leading 0 in integer tokens https://bugs.python.org/issue33304 closed by steven.daprano #33307: socket.send() fails to send large amount of bytes https://bugs.python.org/issue33307 closed by christian.heimes #33308: parser.st2list(..., col_info=True) triggers a SystemError https://bugs.python.org/issue33308 closed by serhiy.storchaka #33310: Update references to PIL-style arrays https://bugs.python.org/issue33310 closed by skrah #33313: pwritev, preadv and posix_spawn are missing from "What's new i https://bugs.python.org/issue33313 closed by serhiy.storchaka #33320: Clarify who can add the "skip issue" label on GitHub https://bugs.python.org/issue33320 closed by adelfino

2 years, 10 months

Re: [Python-Dev] PEP 572: Assignment Expressions

by Eric Snow

On Tue, Apr 17, 2018 at 7:55 AM, Steve Dower <steve.dower(a)python.org> wrote: > Agree with Paul. The PEP is well thought out and well presented, but I > really don’t think we need this in Python (and I say this as someone who > uses it regularly in C/C#). > > -1 on the idea; no disrespect intended toward to people who did a lot of > work on it. Same here. I'm more interested in having anonymous blocks (i.e. scopes), along the lines of PEP 3150, though it's currently #2 on my wish list. :) -eric

2 years, 10 months

Timing for removing legacy Unicode APIs deprecated by PEP 393

by INADA Naoki

Hi, PEP 393 [1] deprecates some Unicode APIs relating to Py_UNICODE. The PEP doesn't provide schedule for removing them. But the APIs are marked "will be removed in 4.0" in the document. When removing them, we can reduce `wchar_t *` member of unicode object. It takes 8 bytes on 64bit platform. [1]: "Flexible String Representation" https://www.python.org/dev/peps/pep-0393/ I thought Python 4.0 is the next version of 3.9. But Guido has different idea. He said following at Zulip chat (we're trying it for now). > No, 4.0 is not just what comes after 3.9 -- the major number change would indicate some kind of major change somewhere (like possibly the Gilectomy, which changes a lot of the C APIs). If we have more than 10 3.x versions, we'll just live with 3.10, 3.11 etc. And he said about these APIs: >> Unicode objects has some "Deprecated since version 3.3, will be removed in version 4.0" APIs (pep-393). >> When removing them, we can reduce PyUnicode size about 8~12byte. > > We should be able to deprecate these sooner by updating the docs. Then, I want to reschedule the removal of these APIs. Can we remove them in 3.8? 3.9? or 3.10? I prefer sooner as possible. --- Slightly off topic, there are 4bytes alignment gap in the unicode object, on 64bit platform. typedef struct { .... struct { unsigned int interned:2; unsigned int kind:3; unsigned int compact:1; unsigned int ascii:1; unsigned int ready:1; unsigned int :24; } state; // 4 bytes // implicit 4 bytes gap here. wchar_t *wstr; // 8 bytes } PyASCIIObject; So, I think we can reduce 12 bytes instead of 8 bytes when removing wstr. Or we can reduce 4 bytes soon by moving `wstr` before `state`. Off course, it needs siphash support 4byte aligned data instead of 8byte. Regards, -- INADA Naoki <songofacandy(a)gmail.com>

2 years, 10 months

Re: [Python-Dev] PEP 572: Assignment Expressions

by Antoine Pitrou

Agreed with Paul and Steve. A lot of work seems to have gone into the PEP (congratulations for that), but still the feature brings little to no additional power to the language while making it more complex. -1 from me. Regards Antoine. On Tue, 17 Apr 2018 06:55:58 -0700 Steve Dower <steve.dower(a)python.org> wrote: > Agree with Paul. The PEP is well thought out and well presented, but I really don’t think we need this in Python (and I say this as someone who uses it regularly in C/C#). > > -1 on the idea; no disrespect intended toward to people who did a lot of work on it. > > Top-posted from my Windows phone > > From: Paul Moore > Sent: Tuesday, April 17, 2018 6:31 > To: David Mertz > Cc: Nick Coghlan; Python-Dev > Subject: Re: [Python-Dev] PEP 572: Assignment Expressions > > On 17 April 2018 at 14:07, Paul Moore <p.f.moore(a)gmail.com> wrote: > > On 17 April 2018 at 14:01, David Mertz <mertz(a)gnosis.cx> wrote: > >> Strongly agree with Nick that only simple name targets should be permitted > >> (at least initially). NONE of the motivating cases use more complex targets, > >> and allowing them encourages obscurity and code golf. > > > > I also agree. Originally I would have said why not allow them, it's a > > potentially useful generalisation. But Nick's examples pretty clearly > > demonstrate that there are a lot of unclear edge cases involved, and > > even though "prevent people writing ugly code" is explicitly stated as > > a non-goal in the PEP, that doesn't mean it's OK to allow an obvious > > bug magnet with no clear use cases. > > I should also point out that I remain -0 on this proposal (I'd already > said this on python-ideas, but I should probably repeat it here). For > me, the use cases are mostly marginal, and the major disadvantage is > in having two forms of assignment. Explaining to a beginner why we use > a := b in an expression, but a = b in a statement is going to be a > challenge. > > The fact that the PEP needs a section covering all the style guide > warnings we feel are needed seems like it's a warning bell, too. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > >

2 years, 10 months

← Newer
1
2
3
4
5
6
7
8
9
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev April 2018