Python-ideas January 2019

python-ideas@python.org

67 participants
38 discussions

kwargs for return
by Thomas Güttler Lists 28 Jan '19

28 Jan '19

I know this is going to get rejected, but I want to speak out the idea nevertheless: I love kwargs and named arguments from the beginning (roughly 18 years now) I guess you came across this several times before: In the first version of the API one variable gets returned: Example: status = backend.transmit_data() But later you want to add something to the API. For the input part of a method this is solved. You can add an optional kwarg. But for the output part of a method, there you can't change the interface easily up to now. Use case: you want to add an optional list of messages which could get returned. You want to change to status, messages = backend.transmit_data() If you have 10 different backend implementations, then you need to change all of them. This is difficult, if the backends reside in different repos and maybe you even don't own some of these repos. Current draw-back: you need to change all of them at once. Of course you could work around it be using this status_messages = backend.transmit_data() And then do some fancy guessing if the variable contains only the status or a tuple containing status and messages. Some days ago I head the idea that kwargs for return would help here. This should handle both cases: Case1: The old backend returns only the status, and the caller wants both status and messages. Somehow the default for messages needs to be defined. In my case it would be the empty list. Case2: The new backends returning the status and messages. The old caller just gets the status. The messages get discarded. Above is the use case. How could kwargs for return look like? Maybe like this: ..... Sorry, I could not find a nice, clean and simple syntax for this up to now. Maybe someone else is more creative than I am. What do you think about this? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines

12 25

Single line single expression try/except syntax
by Alex Shafer 28 Jan '19

28 Jan '19

Hello, I'd like to discuss an idea I had to shorten the syntax for the common case of having a try/except/finally/else block where all of the following conditions are met: * There is only one except block, no finally or else * The exception is not captured in the except block, i.e. `except KeyError:` not `except KeyError as e:` * The contents of the except block is only a single expression * Perhaps, the expression starts with a control word such as pass, break, continue, return, raise. As it happens, everything useful I can think to do with this right now currently uses these. Unclear to me if this should be a requirement. The syntax I envisioned would be something like the following: try on ValueError pass: �� some_list.remove('value') I'm not at all attached to the `on` token specifically, but I think something is necessary there. Other examples: def func(): �� try on KeyError, ValueError return False: �� dict_of_lists['key'].remove('value') key = 'foo' try on KeyError raise MyApplicationError(f'{key} not found'): �� a_dict[key] for i in range(100): �� try on TypeError continue: �� a_list[i] += 1 �� etc() I realize this could be accomplished with context managers, but that seems like overkill to simply throw away the exception, and would increase the overall required code length. Thanks for your input! Alex

5 5

Backtick expression: similar to a shorter lambda syntax
by James Lu 23 Jan '19

23 Jan '19

Backtick expressions work exactly like lambdas, except that they are bound to the instance they are created in every time that class is used to create one. To illustrate, this “percent” property is bound to the instance, not to the class. class Example: percent = property(`self.v*self.v2/100`) And a few more examples for clarity. def example(): locals()['a'] = 1 expr = `a+1` return expr() # error: one variable is required Any variable names that exist when the backtick expression is created are bound to the expression, and the reference to the expression is stored within the expression. Names that do not exist when the expresssion is created must be passed in as parameters. Such names can also be passed in as keyword arguments. Backtick expressions are created when their scope is created. Variable names that are declared but have not been assigned to will be considered to exist for the purposes of the backtick expression. Directly calling a backtick expression as soon as it’s created is forbidden: `v+1`(a) But this is technically allowed but discouraged, like how := works: (`v+1`)(a) Use Cases This can be used anywhere a lambda would feel “heavy” or long. Here are a few use cases where using a backtick expression would allow code to be significantly mote readable: If/else chains that would be switch statements. Creating decorators. Passing in logging hooks. Writing design-by-contract contracts. (See icontract on GitHub for an example of what DBC looks like in Python.) Tests and assertions. Additionally, the instance binding enables: A shorthand way to create a class that wraps an API to a better or more uniform code interface. Previously you’d need to make defs and @property, now each wrapped property and method is a single, readable line of code. Appendix I propose syntax highlighters show a backtick expression on a different background color, a lighter shade of black for a dark theme; dirty white for a light thing. I also propose the following attributes on the backtick expression. __str__(): the string [parameter names separated by commas] => [the string of the backtick expression] __repr__(): the original string of the backtick expression, surrounded by backticks. I secondarily propose that backtick expressions are only bound to their instances when defined within a class when the following syntax is used: def a = <expression> — Now, let’s bikeshed.

10 18

Discussion: Duck typing with “concepts”
by James Lu 23 Jan '19

23 Jan '19

So here’s an interesting idea, not a proposal yet. In C++20, a Concept is a list of Boolean expressions with a name that can be used in place of a type in a templated (ie type-generic) function. from typing import Concept Iterator = Concept(lambda o: hasattr(o, "__iter__", lambda o: iter(o) != NotImplemented) # Concept inheritance Iterable = Concept(lambda o: hasattr(o, "__next__"), Iterator) You could use concepts to define many practical “real-world” duck types. A concept is like an opt-in duck typing type assertion. Since it’s a part of type annotation syntax, assertions can be generated automatically by looking at assertions. You would use a Concept like any other type in type annotations. Marko, how do you think Concepts might integrate with icontract? (I’m imagining overriding an import hook to automatically add contracts to functions with concepts.) How frequently do you use duck typing at Parquery? How might Concepts affect how often you used duck typing?

3 2

Fwd: Potential PEP: with/except
by Paul Ferrell 22 Jan '19

22 Jan '19

> > The thing that concerns me is that any such problem and solution seems > to apply equally to any other kind of block. Why not allow excepts on fo > loops, for example? > Very good point. I think 'with' is special in that it typically contains the entirety of the use of an object, and the type of objects one tends to use in a 'with' are prone to throwing exceptions. Other statements like it don't intrinsically encapsulate the usage of an object. On Tue, Jan 22, 2019 at 1:23 PM Calvin Spealman <cspealma(a)redhat.com> wrote: > > > On Tue, Jan 22, 2019 at 3:11 PM Paul Ferrell <pflarr(a)gmail.com> wrote: > >> I've found that almost any time I'm writing a 'with' block, it's doing >> something that could throw an exception. As a result, each of those >> 'with' blocks needs to be nested within a 'try' block. Due to the >> nature of 'with', it is rarely (if ever) the case that the try block >> contains anything other than the with block itself. >> >> As a result, I would like to propose that the syntax for 'with' blocks >> be changed such that they can be accompanied by 'except', 'finally', >> and/or 'else' blocks as per a standard 'try' block. These would handle >> exceptions that occur in the 'with' block, including the execution of >> the applicable __enter__ and __exit__ methods. >> >> Example: >> >> try: >> with open(path) as myfile: >> ... # Do stuff with file >> except (OSError, IOError) as err: >> logger.error("Failed to read/open file {}: {}".format(path, err) >> >> The above would turn into simply: >> >> with open(path) as myfile: >> ... # Do stuff with file >> except (OSError, IOError) as err: >> logger.error(...) >> >> > It definitely makes sense, both the problem and the proposed solution. > > The thing that concerns me is that any such problem and solution seems > to apply equally to any other kind of block. Why not allow excepts on fo > loops, for example? > > >> >> I think this is rather straightforward in meaning and easy to read, >> and simplifies some unnecessary nesting. I see this as the natural >> evolution of what 'with' >> is all about - replacing necessary try-finally blocks with something >> more elegant. We just didn't include the 'except' portion. >> >> I'm a bit hesitant to put this out there. I'm not worried about it >> getting shot down - that's kind of the point here. I'm just pretty >> strongly against to unnecessary syntactical additions to the language. >> This though, I think I can except. It introduces no new concepts and >> requires no special knowledge to use. There's no question about what >> is going on when you read it. >> >> -- >> Paul Ferrell >> pflarr(a)gmail.com >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > > CALVIN SPEALMAN > > SENIOR QUALITY ENGINEER > > cspealma(a)redhat.com M: +1.336.210.5107 > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > -- Paul Ferrell pflarr(a)gmail.com -- Paul Ferrell pflarr(a)gmail.com

1 0

Add support for external annotations in the typing module
by Till 18 Jan '19

18 Jan '19

We started a discussion in https://github.com/python/typing/issues/600 about adding support for extra annotations in the typing module. Since this is probably going to turn into a PEP I'm transferring the discussion here to have more visibility. The document below has been modified a bit from the one in GH to reflect the feedback I got: + Added a small blurb about how ``Annotated`` should support being used as an alias Things that were raised but are not reflected in this document: + The dataclass example is confusing. I kept it for now because dataclasses often come up in conversations about why we might want to support annotations in the typing module. Maybe I should rework the section. + `...` as a valid parameter for the first argument (if you want to add an annotation but use the type inferred by your type checker). This is an interesting idea, it's probably worth adding support for it if and only if we decide to support in other places. (c.f.: https://github.com/python/typing/issues/276) Thanks, Add support for external annotations in the typing module ========================================================== We propose adding an ``Annotated`` type to the typing module to decorate existing types with context-specific metadata. Specifically, a type ``T`` can be annotated with metadata ``x`` via the typehint ``Annotated[T, x]``. This metadata can be used for either static analysis or at runtime. If a library (or tool) encounters a typehint ``Annotated[T, x]`` and has no special logic for metadata ``x``, it should ignore it and simply treat the type as ``T``. Unlike the `no_type_check` functionality that current exists in the ``typing`` module which completely disables typechecking annotations on a function or a class, the ``Annotated`` type allows for both static typechecking of ``T`` (e.g., via MyPy or Pyre, which can safely ignore ``x``) together with runtime access to ``x`` within a specific application. We believe that the introduction of this type would address a diverse set of use cases of interest to the broader Python community. Motivating examples: ~~~~~~~~~~~~~~~~~~~~ reading binary data +++++++++++++++++++ The ``struct`` module provides a way to read and write C structs directly from their byte representation. It currently relies on a string representation of the C type to read in values:: record = b'raymond \x32\x12\x08\x01\x08' name, serialnum, school, gradelevel = unpack('<10sHHb', record) The struct documentation [struct-examples]_ suggests using a named tuple to unpack the values and make this a bit more tractable:: from collections import namedtuple Student = namedtuple('Student', 'name serialnum school gradelevel') Student._make(unpack('<10sHHb', record)) # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) However, this recommendation is somewhat problematic; as we add more fields, it's going to get increasingly tedious to match the properties in the named tuple with the arguments in ``unpack``. Instead, annotations can provide better interoperability with a type checker or an IDE without adding any special logic outside of the ``struct`` module:: from typing import NamedTuple UnsignedShort = Annotated[int, struct.ctype('H')] SignedChar = Annotated[int, struct.ctype('b')] @struct.packed class Student(NamedTuple): # MyPy typechecks 'name' field as 'str' name: Annotated[str, struct.ctype("<10s")] serialnum: UnsignedShort school: SignedChar gradelevel: SignedChar # 'unpack' only uses the metadata within the type annotations Student.unpack(record)) # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) dataclasses ++++++++++++ Here's an example with dataclasses [dataclass]_ that is a problematic from the typechecking standpoint:: from dataclasses import dataclass, field @dataclass class C: myint: int = 0 # the field tells the @dataclass decorator that the default action in the # constructor of this class is to set "self.mylist = list()" mylist: List[int] = field(default_factory=list) Even though one might expect that ``mylist`` is a class attribute accessible via ``C.mylist`` (like ``C.myint`` is) due to the assignment syntax, that is not the case. Instead, the ``@dataclass`` decorator strips out the assignment to this attribute, leading to an ``AttributeError`` upon access:: C.myint # Ok: 0 C.mylist # AttributeError: type object 'C' has no attribute 'mylist' This can lead to confusion for newcomers to the library who may not expect this behavior. Furthermore, the typechecker needs to understand the semantics of dataclasses and know to not treat the above example as an assignment operation in (which translates to additional complexity). It makes more sense to move the information contained in ``field`` to an annotation:: @dataclass class C: myint: int = 0 mylist: Annotated[List[int], field(default_factory=list)] # now, the AttributeError is more intuitive because there is no assignment operator C.mylist # AttributeError # the constructor knows how to use the annotations to set the 'mylist' attribute c = C() c.mylist # [] The main benefit of writing annotations like this is that it provides a way for clients to gracefully degrade when they don't know what to do with the extra annotations (by just ignoring them). If you used a typechecker that didn't have any special handling for dataclasses and the ``field`` annotation, you would still be able to run checks as though the type were simply:: class C: myint: int = 0 mylist: List[int] lowering barriers to developing new types +++++++++++++++++++++++++++++++++++++++++ Typically when adding a new type, we need to upstream that type to the typing module and change MyPy [MyPy]_, PyCharm [PyCharm]_, Pyre [Pyre]_, pytype [pytype]_, etc. This is particularly important when working on open-source code that makes use of our new types, seeing as the code would not be immediately transportable to other developers' tools without additional logic (this is a limitation of MyPy plugins [MyPy-plugins]_), which allow for extending MyPy but would require a consumer of new typehints to be using MyPy and have the same plugin installed). As a result, there is a high cost to developing and trying out new types in a codebase. Ideally, we should be able to introduce new types in a manner that allows for graceful degradation when clients do not have a custom MyPy plugin, which would lower the barrier to development and ensure some degree of backward compatibility. For example, suppose that we wanted to add support for tagged unions [tagged-unions]_ to Python. One way to accomplish would be to annotate ``TypedDict`` in Python such that only one field is allowed to be set:: Currency = Annotated( TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False), TaggedUnion, ) This is a somewhat cumbersome syntax but it allows us to iterate on this proof-of-concept and have people with non-patched IDEs work in a codebase with tagged unions. We could easily test this proposal and iron out the kinks before trying to upstream tagged union to `typing`, MyPy, etc. Moreover, tools that do not have support for parsing the ``TaggedUnion`` annotation would still be able able to treat `Currency` as a ``TypedDict``, which is still a close approximation (slightly less strict). Details of proposed changes to ``typing`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ syntax ++++++ ``Annotated`` is parameterized with a type and an arbitrary list of Python values that represent the annotations. Here are the specific details of the syntax: * The first argument to ``Annotated`` must be a valid ``typing`` type or ``...`` (to use the infered type). * Multiple type annotations are supported (Annotated supports variadic arguments): ``Annotated[int, ValueRange(3, 10), ctype("char")]`` * ``Annotated`` must be called with at least two arguments (``Annotated[int]`` is not valid) * The order of the annotations is preserved and matters for equality checks:: Annotated[int, ValueRange(3, 10), ctype("char")] != \ Annotated[int, ctype("char"), ValueRange(3, 10)] * Nested ``Annotated`` types are flattened, with metadata ordered starting with the innermost annotation:: Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] ==\ Annotated[int, ValueRange(3, 10), ctype("char")] * Duplicated annotations are not removed: ``Annotated[int, ValueRange(3, 10)] != Annotated[int, ValueRange(3, 10), ValueRange(3, 10)]`` * ``Annotation`` can be used a higher order aliases:: Typevar T = ... Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] # Vec[int] == `Annotated[List[Tuple[int, int]], MaxLen(10)] consuming annotations ++++++++++++++++++++++ Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the `Annotated` type. A tool or library encountering an `Annotated` type can scan through the annotations to determine if they are of interest (e.g., using `isinstance`). **Unknown annotations** When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. For example, if we were to add an annotation that is not an instance of `struct.ctype` to the annotation for name (e.g., `Annotated[str, 'foo', struct.ctype("<10s")]`), the unpack method should ignore it. **Namespacing annotations** We do not need namespaces for annotations since the class used by the annotations acts as a namespace. **Multiple annotations** It's up to the tool consuming the annotations to decide whether the client is allowed to have several annotations on one type and how to merge those annotations. Since the ``Annotated`` type allows you to put several annotations of the same (or different) type(s) on any node, the tools or libraries consuming those annotations are in charge of dealing with potential duplicates. For example, if you are doing value range analysis you might allow this:: T1 = Annotated[int, ValueRange(-10, 5)] T2 = Annotated[T1, ValueRange(-20, 3)] Flattening nested annotations, this translates to:: T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)] An application consuming this type might choose to reduce these annotations via an intersection of the ranges, in which case ``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-10, 3)]``. An alternative application might reduce these via a union, in which case ``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-20, 5)]``. Other applications may decide to not support multiple annotations and throw an exception. References =========== .. [struct-examples] https://docs.python.org/3/library/struct.html#examples .. [dataclass] https://docs.python.org/3/library/dataclasses.html .. [MyPy] https://github.com/python/mypy .. [MyPy-plugins] https://mypy.readthedocs.io/en/latest/extending_mypy.html#extending-mypy-us… .. [PyCharm] https://www.jetbrains.com/pycharm/ .. [Pyre] https://pyre-check.org/ .. [pytype] https://github.com/google/pytype .. [tagged-unions] https://en.wikipedia.org/wiki/Tagged_union

2 2

tkinter: time for round buttons?
by Abdur-Rahmaan Janhangeer 17 Jan '19

17 Jan '19

without starting a should we ban tkinter discussion, i'd like to propose that we add rounded corners buttons. that might make the aesthetic level go up a bit more poor me, if only py had some really nice native gui Abdur-Rahmaan Janhangeer http://www.pythonmembers.club Mauritius

8 11

Fwd: NAN handling in the statistics module
by David Mertz 10 Jan '19

10 Jan '19

> > One possible argument for making PASS the default, even if that means >> implementation-dependent behaviour with NANs, is that in the absense of a >> clear preference for FAIL or RETURN, at least PASS is backwards compatible. >> >> You might shoot yourself in the foot, but at least you know its the same >> foot you shot yourself in using the previous version *wink* >> > I've lost attribution chain. I think this is Steven, but it doesn't really matter. This statement is untrue, or at least only accidentally true at most. The behavior of sorted() against partially ordered collections is unspecified. The author of Timsort says exactly this. If stastics.median() keeps the same implementation—or keeps it with a PASS argument—it may or may not produce the same result in a later Python versions. Timsort is great, but even that has been tweaked sightly over time. I guess the statement is true if "same foot" means "meaningless answer" not some specific value. But that hardly feels like a defense of the behavior. >

2 1

NAN handling in the statistics module
by Steven D'Aprano 10 Jan '19

10 Jan '19

Bug #33084 reports that the statistics library calculates median and other stats wrongly if the data contains NANs. Worse, the result depends on the initial placement of the NAN: py> from statistics import median py> NAN = float('nan') py> median([NAN, 1, 2, 3, 4]) 2 py> median([1, 2, 3, 4, NAN]) 3 See the bug report for more detail: https://bugs.python.org/issue33084 The caller can always filter NANs out of their own data, but following the lead of some other stats packages, I propose a standard way for the statistics module to do so. I hope this will be uncontroversial (he says, optimistically...) but just in case, here is some prior art: (1) Nearly all R stats functions take a "na.rm" argument which defaults to False; if True, NA and NAN values will be stripped. (2) The scipy.stats.ttest_ind function takes a "nan_policy" argument which specifies what to do if a NAN is seen in the data. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.… (3) At least some Matlab functions, such as mean(), take an optional flag that determines whether to ignore NANs or include them. https://au.mathworks.com/help/matlab/ref/mean.html#bt5b82t-1-nanflag I propose adding a "nan_policy" keyword-only parameter to the relevant statistics functions (mean, median, variance etc), and defining the following policies: IGNORE: quietly ignore all NANs FAIL: raise an exception if any NAN is seen in the data PASS: pass NANs through unchanged (the default) RETURN: return a NAN if any NAN is seen in the data WARN: ignore all NANs but raise a warning if one is seen PASS is equivalent to saying that you, the caller, have taken full responsibility for filtering out NANs and there's no need for the function to slow down processing by doing so again. Either that, or you want the current implementation-dependent behaviour. FAIL is equivalent to treating all NANs as "signalling NANs". The presence of a NAN is an error. RETURN is equivalent to "NAN poisoning" -- the presence of a NAN in a calculation causes it to return a NAN, allowing NANs to propogate through multiple calculations. IGNORE and WARN are the same, except IGNORE is silent and WARN raises a warning. Questions: - does anyone have an serious objections to this? - what do you think of the names for the policies? - are there any additional policies that you would like to see? (if so, please give use-cases) - are you happy with the default? Bike-shed away! -- Steve

8 12

Re: [Python-ideas] NAN handling in the statistics module
by Steven D'Aprano 09 Jan '19

09 Jan '19

On Sun, Jan 06, 2019 at 10:52:47PM -0500, David Mertz wrote: > Playing with Tim's examples, this suggests that statistics.median() is > simply outright WRONG. I can think of absolutely no way to characterize > these as reasonable results: > > Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) > In [4]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4, 5]) > Out[4]: 1 > In [5]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4]) > Out[5]: nan The second is possibly correct if one thinks that the median of a list containing NAN should return NAN -- but its only correct by accident, not design. As I wrote on the bug tracker: "I agree that the current implementation-dependent behaviour when there are NANs in the data is troublesome." The only reason why I don't call it a bug is that median() makes no promises about NANs at all, any more than it makes promises about the median of a list of sets or any other values which don't define a total order. help(median) says: Return the median (middle value) of numeric data. By definition, data containing Not A Number values isn't numeric :-) I'm not opposed to documenting this better. Patches welcome :-) There are at least three correct behaviours in the face of data containing NANs: propogate a NAN result, fail fast with an exception, or treat NANs as missing data that can be ignored. Only the caller can decide which is the right policy for their data set. Aside: the IEEE-754 standard provides both signalling and quiet NANs. It is hard and unreliable to generate signalling float NANs in Python, but we can do it with Decimal: py> from statistics import median py> from decimal import Decimal py> median([1, 3, 4, Decimal("sNAN"), 2]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.5/statistics.py", line 349, in median data = sorted(data) decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>] In principle, one ought to be able to construct float signalling NANs too, but unfortunately that's platform dependent: https://mail.python.org/pipermail/python-dev/2018-November/155713.html Back to the topic on hand: I agree that median() does "the wrong thing" when NANs are involved, but there is no one "right thing" that we can do in its place. People disagree as to whether NANs should propogate, or raise, or be treated as missing data, and I see good arguments for all three. -- Steve

6 16

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas January 2019