Mailman 3 January 2019 - Python-ideas

Add support for external annotations in the typing module
by Till Jan. 18, 2019

Jan. 18, 2019

We started a discussion in https://github.com/python/typing/issues/600 about adding support for extra annotations in the typing module. Since this is probably going to turn into a PEP I'm transferring the discussion here to have more visibility. The document below has been modified a bit from the one in GH to reflect the feedback I got: + Added a small blurb about how ``Annotated`` should support being used as an alias Things that were raised but are not reflected in this document: + The … [View More]dataclass example is confusing. I kept it for now because dataclasses often come up in conversations about why we might want to support annotations in the typing module. Maybe I should rework the section. + `...` as a valid parameter for the first argument (if you want to add an annotation but use the type inferred by your type checker). This is an interesting idea, it's probably worth adding support for it if and only if we decide to support in other places. (c.f.: https://github.com/python/typing/issues/276) Thanks, Add support for external annotations in the typing module ========================================================== We propose adding an ``Annotated`` type to the typing module to decorate existing types with context-specific metadata. Specifically, a type ``T`` can be annotated with metadata ``x`` via the typehint ``Annotated[T, x]``. This metadata can be used for either static analysis or at runtime. If a library (or tool) encounters a typehint ``Annotated[T, x]`` and has no special logic for metadata ``x``, it should ignore it and simply treat the type as ``T``. Unlike the `no_type_check` functionality that current exists in the ``typing`` module which completely disables typechecking annotations on a function or a class, the ``Annotated`` type allows for both static typechecking of ``T`` (e.g., via MyPy or Pyre, which can safely ignore ``x``) together with runtime access to ``x`` within a specific application. We believe that the introduction of this type would address a diverse set of use cases of interest to the broader Python community. Motivating examples: ~~~~~~~~~~~~~~~~~~~~ reading binary data +++++++++++++++++++ The ``struct`` module provides a way to read and write C structs directly from their byte representation. It currently relies on a string representation of the C type to read in values:: record = b'raymond \x32\x12\x08\x01\x08' name, serialnum, school, gradelevel = unpack('<10sHHb', record) The struct documentation [struct-examples]_ suggests using a named tuple to unpack the values and make this a bit more tractable:: from collections import namedtuple Student = namedtuple('Student', 'name serialnum school gradelevel') Student._make(unpack('<10sHHb', record)) # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) However, this recommendation is somewhat problematic; as we add more fields, it's going to get increasingly tedious to match the properties in the named tuple with the arguments in ``unpack``. Instead, annotations can provide better interoperability with a type checker or an IDE without adding any special logic outside of the ``struct`` module:: from typing import NamedTuple UnsignedShort = Annotated[int, struct.ctype('H')] SignedChar = Annotated[int, struct.ctype('b')] @struct.packed class Student(NamedTuple): # MyPy typechecks 'name' field as 'str' name: Annotated[str, struct.ctype("<10s")] serialnum: UnsignedShort school: SignedChar gradelevel: SignedChar # 'unpack' only uses the metadata within the type annotations Student.unpack(record)) # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) dataclasses ++++++++++++ Here's an example with dataclasses [dataclass]_ that is a problematic from the typechecking standpoint:: from dataclasses import dataclass, field @dataclass class C: myint: int = 0 # the field tells the @dataclass decorator that the default action in the # constructor of this class is to set "self.mylist = list()" mylist: List[int] = field(default_factory=list) Even though one might expect that ``mylist`` is a class attribute accessible via ``C.mylist`` (like ``C.myint`` is) due to the assignment syntax, that is not the case. Instead, the ``@dataclass`` decorator strips out the assignment to this attribute, leading to an ``AttributeError`` upon access:: C.myint # Ok: 0 C.mylist # AttributeError: type object 'C' has no attribute 'mylist' This can lead to confusion for newcomers to the library who may not expect this behavior. Furthermore, the typechecker needs to understand the semantics of dataclasses and know to not treat the above example as an assignment operation in (which translates to additional complexity). It makes more sense to move the information contained in ``field`` to an annotation:: @dataclass class C: myint: int = 0 mylist: Annotated[List[int], field(default_factory=list)] # now, the AttributeError is more intuitive because there is no assignment operator C.mylist # AttributeError # the constructor knows how to use the annotations to set the 'mylist' attribute c = C() c.mylist # [] The main benefit of writing annotations like this is that it provides a way for clients to gracefully degrade when they don't know what to do with the extra annotations (by just ignoring them). If you used a typechecker that didn't have any special handling for dataclasses and the ``field`` annotation, you would still be able to run checks as though the type were simply:: class C: myint: int = 0 mylist: List[int] lowering barriers to developing new types +++++++++++++++++++++++++++++++++++++++++ Typically when adding a new type, we need to upstream that type to the typing module and change MyPy [MyPy]_, PyCharm [PyCharm]_, Pyre [Pyre]_, pytype [pytype]_, etc. This is particularly important when working on open-source code that makes use of our new types, seeing as the code would not be immediately transportable to other developers' tools without additional logic (this is a limitation of MyPy plugins [MyPy-plugins]_), which allow for extending MyPy but would require a consumer of new typehints to be using MyPy and have the same plugin installed). As a result, there is a high cost to developing and trying out new types in a codebase. Ideally, we should be able to introduce new types in a manner that allows for graceful degradation when clients do not have a custom MyPy plugin, which would lower the barrier to development and ensure some degree of backward compatibility. For example, suppose that we wanted to add support for tagged unions [tagged-unions]_ to Python. One way to accomplish would be to annotate ``TypedDict`` in Python such that only one field is allowed to be set:: Currency = Annotated( TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False), TaggedUnion, ) This is a somewhat cumbersome syntax but it allows us to iterate on this proof-of-concept and have people with non-patched IDEs work in a codebase with tagged unions. We could easily test this proposal and iron out the kinks before trying to upstream tagged union to `typing`, MyPy, etc. Moreover, tools that do not have support for parsing the ``TaggedUnion`` annotation would still be able able to treat `Currency` as a ``TypedDict``, which is still a close approximation (slightly less strict). Details of proposed changes to ``typing`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ syntax ++++++ ``Annotated`` is parameterized with a type and an arbitrary list of Python values that represent the annotations. Here are the specific details of the syntax: * The first argument to ``Annotated`` must be a valid ``typing`` type or ``...`` (to use the infered type). * Multiple type annotations are supported (Annotated supports variadic arguments): ``Annotated[int, ValueRange(3, 10), ctype("char")]`` * ``Annotated`` must be called with at least two arguments (``Annotated[int]`` is not valid) * The order of the annotations is preserved and matters for equality checks:: Annotated[int, ValueRange(3, 10), ctype("char")] != \ Annotated[int, ctype("char"), ValueRange(3, 10)] * Nested ``Annotated`` types are flattened, with metadata ordered starting with the innermost annotation:: Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] ==\ Annotated[int, ValueRange(3, 10), ctype("char")] * Duplicated annotations are not removed: ``Annotated[int, ValueRange(3, 10)] != Annotated[int, ValueRange(3, 10), ValueRange(3, 10)]`` * ``Annotation`` can be used a higher order aliases:: Typevar T = ... Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] # Vec[int] == `Annotated[List[Tuple[int, int]], MaxLen(10)] consuming annotations ++++++++++++++++++++++ Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the `Annotated` type. A tool or library encountering an `Annotated` type can scan through the annotations to determine if they are of interest (e.g., using `isinstance`). **Unknown annotations** When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. For example, if we were to add an annotation that is not an instance of `struct.ctype` to the annotation for name (e.g., `Annotated[str, 'foo', struct.ctype("<10s")]`), the unpack method should ignore it. **Namespacing annotations** We do not need namespaces for annotations since the class used by the annotations acts as a namespace. **Multiple annotations** It's up to the tool consuming the annotations to decide whether the client is allowed to have several annotations on one type and how to merge those annotations. Since the ``Annotated`` type allows you to put several annotations of the same (or different) type(s) on any node, the tools or libraries consuming those annotations are in charge of dealing with potential duplicates. For example, if you are doing value range analysis you might allow this:: T1 = Annotated[int, ValueRange(-10, 5)] T2 = Annotated[T1, ValueRange(-20, 3)] Flattening nested annotations, this translates to:: T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)] An application consuming this type might choose to reduce these annotations via an intersection of the ranges, in which case ``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-10, 3)]``. An alternative application might reduce these via a union, in which case ``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-20, 5)]``. Other applications may decide to not support multiple annotations and throw an exception. References =========== .. [struct-examples] https://docs.python.org/3/library/struct.html#examples .. [dataclass] https://docs.python.org/3/library/dataclasses.html .. [MyPy] https://github.com/python/mypy .. [MyPy-plugins] https://mypy.readthedocs.io/en/latest/extending_mypy.html#extending-mypy-us… .. [PyCharm] https://www.jetbrains.com/pycharm/ .. [Pyre] https://pyre-check.org/ .. [pytype] https://github.com/google/pytype .. [tagged-unions] https://en.wikipedia.org/wiki/Tagged_union [View Less]

2 2

tkinter: time for round buttons?
by Abdur-Rahmaan Janhangeer Jan. 16, 2019

Jan. 16, 2019

without starting a should we ban tkinter discussion, i'd like to propose that we add rounded corners buttons. that might make the aesthetic level go up a bit more poor me, if only py had some really nice native gui Abdur-Rahmaan Janhangeer http://www.pythonmembers.club Mauritius

8 11

Fwd: NAN handling in the statistics module
by David Mertz Jan. 10, 2019

Jan. 10, 2019

> > One possible argument for making PASS the default, even if that means >> implementation-dependent behaviour with NANs, is that in the absense of a >> clear preference for FAIL or RETURN, at least PASS is backwards compatible. >> >> You might shoot yourself in the foot, but at least you know its the same >> foot you shot yourself in using the previous version *wink* >> > I've lost attribution chain. I think this is Steven, but it doesn't really … [View More]

2 1

NAN handling in the statistics module
by Steven D'Aprano Jan. 10, 2019

Jan. 10, 2019

Bug #33084 reports that the statistics library calculates median and other stats wrongly if the data contains NANs. Worse, the result depends on the initial placement of the NAN: py> from statistics import median py> NAN = float('nan') py> median([NAN, 1, 2, 3, 4]) 2 py> median([1, 2, 3, 4, NAN]) 3 See the bug report for more detail: https://bugs.python.org/issue33084 The caller can always filter NANs out of their own data, but following the lead of some other stats packages,… [View More] I propose a standard way for the statistics module to do so. I hope this will be uncontroversial (he says, optimistically...) but just in case, here is some prior art: (1) Nearly all R stats functions take a "na.rm" argument which defaults to False; if True, NA and NAN values will be stripped. (2) The scipy.stats.ttest_ind function takes a "nan_policy" argument which specifies what to do if a NAN is seen in the data. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.… (3) At least some Matlab functions, such as mean(), take an optional flag that determines whether to ignore NANs or include them. https://au.mathworks.com/help/matlab/ref/mean.html#bt5b82t-1-nanflag I propose adding a "nan_policy" keyword-only parameter to the relevant statistics functions (mean, median, variance etc), and defining the following policies: IGNORE: quietly ignore all NANs FAIL: raise an exception if any NAN is seen in the data PASS: pass NANs through unchanged (the default) RETURN: return a NAN if any NAN is seen in the data WARN: ignore all NANs but raise a warning if one is seen PASS is equivalent to saying that you, the caller, have taken full responsibility for filtering out NANs and there's no need for the function to slow down processing by doing so again. Either that, or you want the current implementation-dependent behaviour. FAIL is equivalent to treating all NANs as "signalling NANs". The presence of a NAN is an error. RETURN is equivalent to "NAN poisoning" -- the presence of a NAN in a calculation causes it to return a NAN, allowing NANs to propogate through multiple calculations. IGNORE and WARN are the same, except IGNORE is silent and WARN raises a warning. Questions: - does anyone have an serious objections to this? - what do you think of the names for the policies? - are there any additional policies that you would like to see? (if so, please give use-cases) - are you happy with the default? Bike-shed away! -- Steve [View Less]

8 12

Re: [Python-ideas] NAN handling in the statistics module
by Steven D'Aprano Jan. 9, 2019

Jan. 9, 2019

On Sun, Jan 06, 2019 at 10:52:47PM -0500, David Mertz wrote: > Playing with Tim's examples, this suggests that statistics.median() is > simply outright WRONG. I can think of absolutely no way to characterize > these as reasonable results: > > Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) > In [4]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4, 5]) > Out[4]: 1 > In [5]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4]) > Out[5]: nan The second … [View More]is possibly correct if one thinks that the median of a list containing NAN should return NAN -- but its only correct by accident, not design. As I wrote on the bug tracker: "I agree that the current implementation-dependent behaviour when there are NANs in the data is troublesome." The only reason why I don't call it a bug is that median() makes no promises about NANs at all, any more than it makes promises about the median of a list of sets or any other values which don't define a total order. help(median) says: Return the median (middle value) of numeric data. By definition, data containing Not A Number values isn't numeric :-) I'm not opposed to documenting this better. Patches welcome :-) There are at least three correct behaviours in the face of data containing NANs: propogate a NAN result, fail fast with an exception, or treat NANs as missing data that can be ignored. Only the caller can decide which is the right policy for their data set. Aside: the IEEE-754 standard provides both signalling and quiet NANs. It is hard and unreliable to generate signalling float NANs in Python, but we can do it with Decimal: py> from statistics import median py> from decimal import Decimal py> median([1, 3, 4, Decimal("sNAN"), 2]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.5/statistics.py", line 349, in median data = sorted(data) decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>] In principle, one ought to be able to construct float signalling NANs too, but unfortunately that's platform dependent: https://mail.python.org/pipermail/python-dev/2018-November/155713.html Back to the topic on hand: I agree that median() does "the wrong thing" when NANs are involved, but there is no one "right thing" that we can do in its place. People disagree as to whether NANs should propogate, or raise, or be treated as missing data, and I see good arguments for all three. -- Steve [View Less]

6 16

Re: [Python-ideas] NAN handling in the statistics module
by Steven D'Aprano Jan. 8, 2019

Jan. 8, 2019

On Tue, Jan 08, 2019 at 04:25:17PM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > By definition, data containing Not A Number values isn't numeric :-) > > Unfortunately, that's just a joke, because in fact numeric functions > produce NaNs. I'm not sure if you're agreeing with me or disagreeing, so I'll assume you're agreeing and move on :-) > I agree that this can easily be resolved by documenting that it is the > caller's responsibility to … [View More]

1 0

Possible PEP regarding the use of the continue keyword in try/except blocks
by Simon Jan. 7, 2019

Jan. 7, 2019

I was writing some python code earlier, and I noticed that in a code that looks somwhat like this one : try: i = int("string") print("continued on") j = int(9.0) except ValueError as e: print(e) >>> "invalid literal for int() with base 10: 'string'" this code will handle the exception, but the code in the try block will not continue. I propose to be able to use the continue keyword to continue the execution of the try block even when an error … [View More]

13 16

Re: [Python-ideas] NAN handling in the statistics module
by Tim Peters Jan. 6, 2019

Jan. 6, 2019

[David Mertz <david.mertz(a)gmail.com>] > Thanks Tim for clarifying. Is it even the case that sorts are STABLE in > the face of non-total orderings under __lt__? A couple quick examples > don't refute that, but what I tried was not very thorough, nor did I > think much about TimSort itself. I'm not clear on what "stable" could mean in the absence of a total ordering. Not only does sort not assume __lt__ is a total ordering, it doesn't assume it's transitive, or even … [View More]

3 4

Re: [Python-ideas] NAN handling in the statistics module
by David Mertz Jan. 6, 2019

Jan. 6, 2019

[... apologies if this is dup, got a bounce ...] > [David Mertz <mertz(a)gnosis.cx>] >> I have to say though that the existing behavior of `statistics.median[_low|_high|]` >> is SURPRISING if not outright wrong. It is the behavior in existing Python, >> but it is very strange. >> >> The implementation simply does whatever `sorted()` does, which is an >> implementation detail. In particular, NaN's being neither less than nor >> greater than … [View More]

1 0

Fixed point format for numbers with locale based separators
by Stefan Krah Jan. 6, 2019

Jan. 6, 2019

Eric V. Smith wrote: > If the locale character is "*", use locale-aware formatting for the > given "type", with LC_NUMERIC. So, "*g" would be equivalent to the > existing "n", and "*f" would give you the current "f" formatting, except > using LC_NUMERIC for the decimal point. If the locale character is "$" > use locale-aware LC_MONETARY. So then we could use "$g", "$f", etc. > These locale characters would also work with int, so "*d" would make "n" > obsolete (but I'm … [View More]

1 0

回复：Python-ideas Digest, Vol 146, Issue 13
by Moon丶sun Jan. 5, 2019

Jan. 5, 2019

Thanks for your reply.But the answer is not I except, I will show you some examples to explain what result I except: @contextmanagerdef cm(): print('open file') yield print('close file')with cm(): 1/0 If I use a contextmanager ,I except it can help me to close the file anytime,even raise an error,but if I define a function with @contextmanager like the example which I have showed for you, it will never print('close file') I can only modify it like this:@contextmanagerdef cm(): … [View More]try: print('open file') yield except Exception as e: print('Error',e) finally: print('close file') It is not friendly for us to use it, so I modify the contextlib to fix it,you can catch it from the e-mail attachment.It's in the line 79 and line 97---------------------------------------------------------------------- 发件人：python-ideas-request(a)python.org 收件人：python-ideas(a)python.org 主题：Python-ideas Digest, Vol 146, Issue 13 日期：2019年01月06日 01点05分 Send Python-ideas mailing list submissions to python-ideas(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-ideas or, via email, send a message with subject or body 'help' to python-ideas-request(a)python.org You can reach the person managing the list at python-ideas-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-ideas digest..." Today's Topics: 1. Re: Make the @contextmanager of contextlib to be a real contextmanager (Serhiy Storchaka) 2. Re: Fixed point format for numbers with locale based separators (?ukasz Stelmach) ---------------------------------------------------------------------- Message: 1 Date: Sat, 5 Jan 2019 16:45:34 +0200 From: Serhiy Storchaka <storchaka(a)gmail.com> To: python-ideas(a)python.org Subject: Re: [Python-ideas] Make the @contextmanager of contextlib to be a real contextmanager Message-ID: <q0qfqd$q1q$1(a)blaine.gmane.org> Content-Type: text/plain; charset=UTF-8; format=flowed 05.01.19 14:52, Moon?sun ????: > As we know,when we import the module--'contextlib',we can use the > decorator '@contextmanager' and keyword ?yield? to make a 'instance' of > Class '_GeneratorContextManager' in 'contextlib' module,then we can use > it like: > with 'instance' as 'xx': > ? ? 'code block' > ? ? pass > But there is a little bug,when the code block raise a error,the instance > cannot run the code which after the keyword 'yield'. This is not a bug. Consider the following example: @contextmanager def cm(): try: yield except BaseException as err: print('Fail:', err) raise else: print('Success') with cm(): 1/0 What result would you expect? Test it with the stdlib implementation and with your implementation. ------------------------------ Message: 2 Date: Sat, 05 Jan 2019 16:41:20 +0100 From: ?ukasz Stelmach <steelman(a)post.pl> To: Steven D'Aprano <steve(a)pearwood.info> Cc: python-ideas(a)python.org Subject: Re: [Python-ideas] Fixed point format for numbers with locale based separators Message-ID: <87y37z9knz.fsf%steelman(a)post.pl> Content-Type: text/plain; charset="utf-8" Steven D'Aprano <steve(a)pearwood.info> writes: > On Fri, Jan 04, 2019 at 03:57:53PM +0100, ?ukasz Stelmach wrote: >> Hi, >> >> I would like to present two pull requests[1][2] implementing fixed point >> presentation of numbers and ask for comments. The first is mine. I >> learnt about the second after publishing mine. > > Before I look at the implementation, can you explain the functional > requirements please? As I stated in the original message below the table: >> In the application I want to create I am going to present users numbers >> ranging up to 3 orders of magnitude and I (my users) want them to be >> presented consistently with regards to number of decimal digits AND I >> want to conform to rules of languages of my users. And I would like to >> avoid the exponent notation by all means. The pint[1] library I use, implements formatting of physical quantities using the format()/__format__ code. As far as I can tell my patch for Python is shorter and more straightforward than a patch for pint to use locale.format(). Because the "g" based "n" formatter has been present since the advanced string formatting was described in PEP-3101, I think it is necessary to add the "m" formatter based on "f". The advanced string formatting facility in Python is very convenient and programmers shouldn't forced to use locale.format() like this "The total length of {} sticks is {} meters.".format(n_sticks, locale.format(".2f", l_sticks)) instead of "The total length of {} sticks is {:.2f} meters.".format(n_sticks, l_sticks) > In other words, what is the new feature you hope to have excepted? > Explain the intention and the API (the interface). The implementation is > the least important part :-) I wish to add a new formatter "m" for float/complex/decimal numbers, which behaves like the existing "f", but uses the decimal separator from the locale database. There is "n" formmatter which behaves like "g" but it does not fit my needs. > [...] >> Formatting 1.23456789 * n (LC_ALL=3Dpl_PL.UTF-8) >> | n | ".2f" | ".3n" | >> |---+----------+----------| >> | 1 | 1.23 | 1,23 | >> | 2 | 12.35 | 12,3 | >> | 3 | 123.46 | 123 | >> | 4 | 1234.57 | 1,23e+03 | > > I'm afraid I cannot work out what that table means. You say "Formatting > 1.23... * n" (multiplying by n) but the results shown aren't multiplied > by n=2, n=3, n=4 as the table suggests. > > Can you show what Python code you expect will produce the expected > output? for n in range(1,5): print("| {} | {:8.2f} | {:8.3n} |".format(n,1.23456789 * 10**n, 1.23456789 * 10**n)) [1] http://pint.readthedocs.io/ -- By?o mi bardzo mi?o. --- Rurku. --- ... >?ukasz< --- To dobrze, ?e mnie s?uchasz. [View Less]

2 1

Make the @contextmanager of contextlib to be a real contextmanager
by Moon丶sun Jan. 5, 2019

Jan. 5, 2019

As we know,when we import the module--'contextlib',we can use the decorator '@contextmanager' and keyword ‘yield’ to make a 'instance' of Class '_GeneratorContextManager' in 'contextlib' module,then we can use it like:with 'instance' as 'xx': 'code block' passBut there is a little bug,when the code block raise a error,the instance cannot run the code which after the keyword 'yield'.So I try to repair this bug, I add a method '_next()' in the Class '_GeneratorContextManager',and then … [View More]

2 1

Add regex pattern literal p""
by Ma Lin Jan. 1, 2019

Jan. 1, 2019

We can use this literal to represent a compiled pattern, for example: >>> p"(?i)[a-z]".findall("a1B2c3") ['a', 'B', 'c'] >>> compiled = p"(?<=abc)def" >>> m = compiled.search('abcdef') >>> m.group(0) 'def' >>> rp'\W+'.split('Words, words, words.') ['Words', 'words', 'words', ''] This allows peephole optimizer to store compiled pattern in .pyc file, we can get performance optimization like replacing constant set by frozenset in .pyc file.… [View More]

14 27