Mailman 3 January 2019 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow 18 Feb '22

18 Feb '22

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino 26 Mar '21

26 Mar '21

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman 12 Mar '21

12 Mar '21

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman

12 35

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Add the imath module
by Serhiy Storchaka 22 Mar '20

22 Mar '20

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math module. * as_integer_ration(x) Equivalents to: def as_integer_ration(x): if hasattr(x, 'as_integer_ration'): return x.as_integer_ration() else: return (x.numerator, x.denominator) * binom(n, k) Returns factorial(n) // (factorial(k) * factorial(n-k)), but uses more efficient algorithm. * sqrt(n) Returns the largest integer r such that r**2 <= n and (r+1)**2 > n. * isprime(n) Tests if n is a prime number. * primes() Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,... Are there more ideas?

14 38

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar 07 Mar '20

07 Mar '20

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to allow > > item = object[1, *some_tuple, 2] > item = object[1, *some_tuple, :] > > Was this overlooked in the original proposal, or deliberately rejected? > > Eric >

2 2

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Fixed point format for numbers with locale based separators
by Łukasz Stelmach 20 Dec '19

20 Dec '19

Hi, I would like to present two pull requests[1][2] implementing fixed point presentation of numbers and ask for comments. The first is mine. I learnt about the second after publishing mine. The only format using decimal separator from locale data for float/complex/decimal numbers at the moment is "n" which behaves like "g". The drawback of these formats, I would like to overcome, is the inability to print numbers ranging more than one order of magnitude with the same number of decimal digits without "manually" (with some additional custom code) adjusting precission. The other option is to "manually" replace "." as printed by "f" with a local decimal separator. Neither of these option is appealing to my. Formatting 1.23456789 * n (LC_ALL=3Dpl_PL.UTF-8) | n | ".2f" | ".3n" | |---+----------+----------| | 1 | 1.23 | 1,23 | | 2 | 12.35 | 12,3 | | 3 | 123.46 | 123 | | 4 | 1234.57 | 1,23e+03 | In the application I want to create I am going to present users numbers ranging up to 3 orders of magnitude and I (my users) want them to be presented consistently with regards to number of decimal digits AND I want to conform to rules of languages of my users. And I would like to avoid the exponent notation by all means. I can't say much about James Emerton's implementation or his intentions, but please take a look at our patches and give your comments so either of us or together we can implement this feature. PS. In theory both implementations could be merged because James chose to use "l" to use LC_MONETARY category and I chose "m" to use LC_NUMERIC. [1] https://github.com/python/cpython/pull/11405 [2] https://github.com/python/cpython/pull/8612 -- Miłego dnia, Łukasz Stelmach

5 9

Please consider adding of functions file system operations to pathlib
by George Fischhof 16 Nov '19

16 Nov '19

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger 23 Oct '19

23 Oct '19

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel

17 49

support toml for pyproject support
by Jimmy Girardet 29 Jul '19

29 Jul '19

Hi, I don't know if this was already debated but I don't know how to search in the whole archive of the list. For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library. Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency. Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to not have the configparser module. I know it's complicated to add more and more thing to the stdlib but I really think it is necessary for python packaging being more consistent. Maybe we could thought to a readonly lib to limit the added code. If it's conceivable, I'd be happy to help in it. Nice Day guys and girls. Jimmy

13 15

Re: [Python-ideas] Trigonometry in degrees
by Hugh Fisher 08 May '19

08 May '19

> Date: Thu, 7 Jun 2018 12:33:29 +0000 > From: Robert Vanden Eynde <robertvandeneynde(a)hotmail.com> > To: python-ideas <python-ideas(a)python.org> > Subject: [Python-ideas] Trigonometry in degrees > Message-ID: > > > I suggest adding degrees version of the trigonometric functions in the math module. > > - Useful in Teaching and replacing calculators by python, importing something is seen by the young students much more easy than to define a function. I agree that degrees are useful for teaching. They are also very useful for graphics programming, especially with my favourite OpenGL API. But I think that the use of radians in programming language APIs is more prevalent, so the initial advantage of easy learning will be outweighed by the long term inconvenience of adjusting to what everyone else is doing. Writing degrees(x) and radians(x) is a little inconvenient, but it does make it clear what units are being used. And even if your proposal is adopted, there is still going to be a lot of code around that uses the older math routines. With the current API it is a least safe to assume that angles are radians unless stated otherwise. > - Special values could be treated, aka when the angle is a multiple of 90, young students are often surprise to see that cos(pi/2) != 0 > > Testing for a special value Isn't very costly (x % 90 == 0) but it could be pointed out that there is a small overhead using the "degrees" equivalent of trig function because of the radians to degrees conversion And the special values testing. Not just young students :-) I agree with this, but I would prefer the check to be in the implementation of the existing functions as well. Any sin/cos very close to 0 becomes 0, any close to 1 becomes 1. > - Standard names will be chosen so that everyone will use the same name convention. I suggest adding a "d" like sind, cosd, tand, acosd, asind, atand, atan2d. Not "d". In the OpenGL 3D API, and many 3D languages/APIs since, appending "d" means "double precision". It's even sort of implied by the C math library which has sinf and friends for single precision. > > Creating a new package like 'from math.degrees import cos' however I would not recommend that because "cos" in the source code would mean to lookup the import to know if it's in degrees or radians (and that leads to very filthy bugs). Also "degrees" is already so the name would have to change the name of the package. Agree, not a good idea. -- cheers, Hugh Fisher

16 29

f-string "debug" conversion
by Eric V. Smith 03 May '19

03 May '19

This idea was proposed to me at the core sprints last month by Larry Hastings. I've discussed it with a few people, who seem generally positive about it, and we've tweaked it a little bit. I've spent some time implementing it, and I think it's doable. I thought I'd post it here for any additional feedback. Here’s the idea: for f-strings, we add a !d conversion operator, which is superficially similar to !s, !r, and !a. The meaning of !d is: produce the text of the expression (not its value!), followed by an equal sign, followed by the repr of the value of the expression. So: value = 10 s = 'a string!' print(f'{value!d}') print(f'next: {value+1!d}') print(f'{s!d}') produces: value=10 next: value+1=11 s='a string!' I’m not proposing this for str.format(). It would only really make sense for named arguments, and I don’t think print('{value!d}'.format(value=value) is much of a win. The result is a string, so if you really wanted to, you could use a string formatting spec. So: print(f'*{value!d:^20}*' would produce: * value=10 * Although I don’t think that would be very useful in general. The mnemonic is !d for “debugging”. I’d wanted to use !=, because there’s an equal sign involved in the result, but = is the one character that can’t be used after ! (it’s “not equal” in expressions, and f-strings look specifically for that case). I also mentioned !!, but I think I prefer !d as being less confusing. This would be used in debugging print statements, that currently end up looking like: print(f'value={value!r}') and would now be: print(f'{value!d}') There have been discussions about ways to specify str() vs. repr(), using characters other than '=', adding spaces, etc. But they all end up over-complicating what should be a simple tool, not a Swiss Army knife. Thoughts? Eric

12 36

Return for assignment blocks
by Calvin Spealman 20 Mar '19

20 Mar '19

I'd like to suggest what I think would be a simple addition to `def` and `class` blocks. I don't know if calling those "Assignment Blocks" is accurate, but I just mean to refer to block syntaxes that assign to a name. Anyway, I propose a combined return-def structure, and optionally also allowing a return-class version. Omitting the name would be allowable, as well. This would only apply to a `def` or `class` statement made as the last part of the function body, of course. def ignore_exc(exc_type): return def (func): @wraps(func) return def (*args, **kwargs): try: return func(*args, **kwargs) except exc_type: pass Thanks for considering and for any comments, thoughts, or feedback on the idea!

14 33

Support localization of unicode descriptions
by Pander 25 Feb '19

25 Feb '19

Dear all, Who might also be interested in setting up a project that supports localization for Unicode block description and character description. Translations are available from https://github.com/unicode-table/unicode-table-data/tree/master/loc <https://github.com/unicode-table/unicode-table-data/tree/master/loc> If possible, use a gettext approach similar to https://pypi.org/project/pycountry/ <https://pypi.org/project/pycountry/> Implementing this feature will allow users to read Unicode descriptions in their own language, other than English. For example, now is possible only in English: from unicodedata import name print(name('ß')) LATIN SMALL LETTER SHARP S So unicodedata could provide a way to translate LATIN SMALL LETTER SHARP S to e.g. German with: from unicodedata import name from unicodedata_l10n import LOCALED_DIR from gettext import translation german = translation('UnicodeData' LOCALED_DIR, languages=['de']) german.install() print(_(name('ß'))) LATEINISCHER KLEINBUCHSTABE SCHARFES S and something similar for unicodedata.category Best, Pander

5 8

Add list.join() please
by Jamesie Pic 11 Feb '19

11 Feb '19

Hello, During the last 10 years, Python has made steady progress in convenience to assemble strings. However, it seems to me that joining is still, when possible, the cleanest way to code string assembly. However, I'm still sometimes confused between the different syntaxes used by join methods: 0. os.path.join takes *args 1. str.join takes a list argument, this inconsistence make it easy to mistake with the os.path.join signature Also, I still think that: '_'.join(['cancel', name]) Would be more readable as such: ['cancel', name].join('_') Not only this would fix both of my issues with the current status-quo, but this would also be completely backward compatible, and probably not very hard to implement: just add a join method to list. Thanks in advance for your reply Have a great day -- ∞

31 160

Dart like multi line strings identation
by Marius Räsener 08 Feb '19

08 Feb '19

Hey List, this is my very first approach to suggest a Python improvement I'd think worth discussing. At some point, maybe with Dart 2.0 or a little earlier, Dart is now supporting multiline strings with "proper" identation (tried, but I can't find the according docs at the moment. probably due to the rather large changes related to dart 2.0 and outdated docs.) What I have in mind is probably best described with an Example: print(""" I am a multiline String. """) the closing quote defines the "margin indentation" - so in this example all lines would get reduces by their leading 4 spaces, resulting in a "clean" and unintended string. anyways, if dart or not, doesn't matter - I like the Idea and I think python3.x could benefit from it. If that's possible at all :) I could also imagine that this "indentation cleanup" only is applied if the last quotes are on their own line? Might be too complicated though, I can't estimated or understand this... thx for reading, Marius

23 47

AMEND PEP-8 TO DISCOURAGE ALL CAPS
by Abe Dillon 03 Feb '19

03 Feb '19

I keep coming back to this great video <https://vimeo.com/74316116> about coding style, and one point in particular rings true to me: ALL_CAPS_IS_OBNOXIOUS It destroys the visual flow of code and for what? To signify a global, constant, or Enum? Is that really so important? I don't think so. I think the all caps style has out-lived its usefulness and needs to go the way of the dodo. The last time I saw all caps used appropriately was in a YouTube comment where some guy was ranting about the communist Jewish banker conspiracy to control the world. In that case, all caps clearly communicated to me that the person was a frothing lunatic (thought find the idea of communist bankers intriguing). Currently PEP-8 prescribes all caps for constants <https://www.python.org/dev/peps/pep-0008/#constants> and uses the all cap variable "FILES" as an example in a different section. <https://www.python.org/dev/peps/pep-0008/#when-to-use-trailing-commas> It also appears to be the defacto-standard for enums (based on the documentation <https://docs.python.org/3/library/enum.html#creating-an-enum> ) I don't think it's necessary to make any breaking changes. Just pep-8 and (of less importance) spurious documentation examples.

29 79

Potential PEP: with/except
by Paul Ferrell 31 Jan '19

31 Jan '19

I've found that almost any time I'm writing a 'with' block, it's doing something that could throw an exception. As a result, each of those 'with' blocks needs to be nested within a 'try' block. Due to the nature of 'with', it is rarely (if ever) the case that the try block contains anything other than the with block itself. As a result, I would like to propose that the syntax for 'with' blocks be changed such that they can be accompanied by 'except', 'finally', and/or 'else' blocks as per a standard 'try' block. These would handle exceptions that occur in the 'with' block, including the execution of the applicable __enter__ and __exit__ methods. Example: try: with open(path) as myfile: ... # Do stuff with file except (OSError, IOError) as err: logger.error("Failed to read/open file {}: {}".format(path, err) The above would turn into simply: with open(path) as myfile: ... # Do stuff with file except (OSError, IOError) as err: logger.error(...) I think this is rather straightforward in meaning and easy to read, and simplifies some unnecessary nesting. I see this as the natural evolution of what 'with' is all about - replacing necessary try-finally blocks with something more elegant. We just didn't include the 'except' portion. I'm a bit hesitant to put this out there. I'm not worried about it getting shot down - that's kind of the point here. I'm just pretty strongly against to unnecessary syntactical additions to the language. This though, I think I can except. It introduces no new concepts and requires no special knowledge to use. There's no question about what is going on when you read it. -- Paul Ferrell pflarr(a)gmail.com

11 18

Stack traces ought to flag when a module has been changed on disk
by Steven D'Aprano 30 Jan '19

30 Jan '19

This thought is motivated by this bug report: https://bugs.python.org/issue35857 If you import a module, then edit the .py file that goes with it, and then an exception occurs, the stack trace can show the wrong line. It doesn't happen very often, but when it does happen, it can be very perplexing. Here's a proposal: When a stack trace is printed, before printing each line, the interpreter checks whether the file's modification time is later than the time recorded in the .pyc file. If the times are different, the stack trace can flag the line and print an addition line stating that the file may have changed and the stack trace may not be accurate. Something like this perhaps? Traceback (most recent call last): File "spam.py", line 99, in <spam> eggs.foo() File "eggs.py", line 123, in <eggs> ? for obj in sequence: File "cheese.py", line 456, in <cheese> n = len(x) *** one or more files may have changed *** lines starting with ? may be inaccurate TypeError: object of type 'NoneType' has no len() I don't think performance will matter. Generating stack traces are rarely performance critical, so I don't think that a few extra file system checks will make any meaningful difference. Thoughts? -- Steve

7 7

kwargs for return
by Thomas Güttler Lists 28 Jan '19

28 Jan '19

I know this is going to get rejected, but I want to speak out the idea nevertheless: I love kwargs and named arguments from the beginning (roughly 18 years now) I guess you came across this several times before: In the first version of the API one variable gets returned: Example: status = backend.transmit_data() But later you want to add something to the API. For the input part of a method this is solved. You can add an optional kwarg. But for the output part of a method, there you can't change the interface easily up to now. Use case: you want to add an optional list of messages which could get returned. You want to change to status, messages = backend.transmit_data() If you have 10 different backend implementations, then you need to change all of them. This is difficult, if the backends reside in different repos and maybe you even don't own some of these repos. Current draw-back: you need to change all of them at once. Of course you could work around it be using this status_messages = backend.transmit_data() And then do some fancy guessing if the variable contains only the status or a tuple containing status and messages. Some days ago I head the idea that kwargs for return would help here. This should handle both cases: Case1: The old backend returns only the status, and the caller wants both status and messages. Somehow the default for messages needs to be defined. In my case it would be the empty list. Case2: The new backends returning the status and messages. The old caller just gets the status. The messages get discarded. Above is the use case. How could kwargs for return look like? Maybe like this: ..... Sorry, I could not find a nice, clean and simple syntax for this up to now. Maybe someone else is more creative than I am. What do you think about this? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines

12 25

Single line single expression try/except syntax
by Alex Shafer 28 Jan '19

28 Jan '19

Hello, I'd like to discuss an idea I had to shorten the syntax for the common case of having a try/except/finally/else block where all of the following conditions are met: * There is only one except block, no finally or else * The exception is not captured in the except block, i.e. `except KeyError:` not `except KeyError as e:` * The contents of the except block is only a single expression * Perhaps, the expression starts with a control word such as pass, break, continue, return, raise. As it happens, everything useful I can think to do with this right now currently uses these. Unclear to me if this should be a requirement. The syntax I envisioned would be something like the following: try on ValueError pass: �� some_list.remove('value') I'm not at all attached to the `on` token specifically, but I think something is necessary there. Other examples: def func(): �� try on KeyError, ValueError return False: �� dict_of_lists['key'].remove('value') key = 'foo' try on KeyError raise MyApplicationError(f'{key} not found'): �� a_dict[key] for i in range(100): �� try on TypeError continue: �� a_list[i] += 1 �� etc() I realize this could be accomplished with context managers, but that seems like overkill to simply throw away the exception, and would increase the overall required code length. Thanks for your input! Alex

5 5

Backtick expression: similar to a shorter lambda syntax
by James Lu 23 Jan '19

23 Jan '19

Backtick expressions work exactly like lambdas, except that they are bound to the instance they are created in every time that class is used to create one. To illustrate, this “percent” property is bound to the instance, not to the class. class Example: percent = property(`self.v*self.v2/100`) And a few more examples for clarity. def example(): locals()['a'] = 1 expr = `a+1` return expr() # error: one variable is required Any variable names that exist when the backtick expression is created are bound to the expression, and the reference to the expression is stored within the expression. Names that do not exist when the expresssion is created must be passed in as parameters. Such names can also be passed in as keyword arguments. Backtick expressions are created when their scope is created. Variable names that are declared but have not been assigned to will be considered to exist for the purposes of the backtick expression. Directly calling a backtick expression as soon as it’s created is forbidden: `v+1`(a) But this is technically allowed but discouraged, like how := works: (`v+1`)(a) Use Cases This can be used anywhere a lambda would feel “heavy” or long. Here are a few use cases where using a backtick expression would allow code to be significantly mote readable: If/else chains that would be switch statements. Creating decorators. Passing in logging hooks. Writing design-by-contract contracts. (See icontract on GitHub for an example of what DBC looks like in Python.) Tests and assertions. Additionally, the instance binding enables: A shorthand way to create a class that wraps an API to a better or more uniform code interface. Previously you’d need to make defs and @property, now each wrapped property and method is a single, readable line of code. Appendix I propose syntax highlighters show a backtick expression on a different background color, a lighter shade of black for a dark theme; dirty white for a light thing. I also propose the following attributes on the backtick expression. __str__(): the string [parameter names separated by commas] => [the string of the backtick expression] __repr__(): the original string of the backtick expression, surrounded by backticks. I secondarily propose that backtick expressions are only bound to their instances when defined within a class when the following syntax is used: def a = <expression> — Now, let’s bikeshed.

10 18

Discussion: Duck typing with “concepts”
by James Lu 23 Jan '19

23 Jan '19

So here’s an interesting idea, not a proposal yet. In C++20, a Concept is a list of Boolean expressions with a name that can be used in place of a type in a templated (ie type-generic) function. from typing import Concept Iterator = Concept(lambda o: hasattr(o, "__iter__", lambda o: iter(o) != NotImplemented) # Concept inheritance Iterable = Concept(lambda o: hasattr(o, "__next__"), Iterator) You could use concepts to define many practical “real-world” duck types. A concept is like an opt-in duck typing type assertion. Since it’s a part of type annotation syntax, assertions can be generated automatically by looking at assertions. You would use a Concept like any other type in type annotations. Marko, how do you think Concepts might integrate with icontract? (I’m imagining overriding an import hook to automatically add contracts to functions with concepts.) How frequently do you use duck typing at Parquery? How might Concepts affect how often you used duck typing?

3 2

Fwd: Potential PEP: with/except
by Paul Ferrell 22 Jan '19

22 Jan '19

> > The thing that concerns me is that any such problem and solution seems > to apply equally to any other kind of block. Why not allow excepts on fo > loops, for example? > Very good point. I think 'with' is special in that it typically contains the entirety of the use of an object, and the type of objects one tends to use in a 'with' are prone to throwing exceptions. Other statements like it don't intrinsically encapsulate the usage of an object. On Tue, Jan 22, 2019 at 1:23 PM Calvin Spealman <cspealma(a)redhat.com> wrote: > > > On Tue, Jan 22, 2019 at 3:11 PM Paul Ferrell <pflarr(a)gmail.com> wrote: > >> I've found that almost any time I'm writing a 'with' block, it's doing >> something that could throw an exception. As a result, each of those >> 'with' blocks needs to be nested within a 'try' block. Due to the >> nature of 'with', it is rarely (if ever) the case that the try block >> contains anything other than the with block itself. >> >> As a result, I would like to propose that the syntax for 'with' blocks >> be changed such that they can be accompanied by 'except', 'finally', >> and/or 'else' blocks as per a standard 'try' block. These would handle >> exceptions that occur in the 'with' block, including the execution of >> the applicable __enter__ and __exit__ methods. >> >> Example: >> >> try: >> with open(path) as myfile: >> ... # Do stuff with file >> except (OSError, IOError) as err: >> logger.error("Failed to read/open file {}: {}".format(path, err) >> >> The above would turn into simply: >> >> with open(path) as myfile: >> ... # Do stuff with file >> except (OSError, IOError) as err: >> logger.error(...) >> >> > It definitely makes sense, both the problem and the proposed solution. > > The thing that concerns me is that any such problem and solution seems > to apply equally to any other kind of block. Why not allow excepts on fo > loops, for example? > > >> >> I think this is rather straightforward in meaning and easy to read, >> and simplifies some unnecessary nesting. I see this as the natural >> evolution of what 'with' >> is all about - replacing necessary try-finally blocks with something >> more elegant. We just didn't include the 'except' portion. >> >> I'm a bit hesitant to put this out there. I'm not worried about it >> getting shot down - that's kind of the point here. I'm just pretty >> strongly against to unnecessary syntactical additions to the language. >> This though, I think I can except. It introduces no new concepts and >> requires no special knowledge to use. There's no question about what >> is going on when you read it. >> >> -- >> Paul Ferrell >> pflarr(a)gmail.com >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > > CALVIN SPEALMAN > > SENIOR QUALITY ENGINEER > > cspealma(a)redhat.com M: +1.336.210.5107 > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > -- Paul Ferrell pflarr(a)gmail.com -- Paul Ferrell pflarr(a)gmail.com

1 0

Add support for external annotations in the typing module
by Till 18 Jan '19

18 Jan '19

We started a discussion in https://github.com/python/typing/issues/600 about adding support for extra annotations in the typing module. Since this is probably going to turn into a PEP I'm transferring the discussion here to have more visibility. The document below has been modified a bit from the one in GH to reflect the feedback I got: + Added a small blurb about how ``Annotated`` should support being used as an alias Things that were raised but are not reflected in this document: + The dataclass example is confusing. I kept it for now because dataclasses often come up in conversations about why we might want to support annotations in the typing module. Maybe I should rework the section. + `...` as a valid parameter for the first argument (if you want to add an annotation but use the type inferred by your type checker). This is an interesting idea, it's probably worth adding support for it if and only if we decide to support in other places. (c.f.: https://github.com/python/typing/issues/276) Thanks, Add support for external annotations in the typing module ========================================================== We propose adding an ``Annotated`` type to the typing module to decorate existing types with context-specific metadata. Specifically, a type ``T`` can be annotated with metadata ``x`` via the typehint ``Annotated[T, x]``. This metadata can be used for either static analysis or at runtime. If a library (or tool) encounters a typehint ``Annotated[T, x]`` and has no special logic for metadata ``x``, it should ignore it and simply treat the type as ``T``. Unlike the `no_type_check` functionality that current exists in the ``typing`` module which completely disables typechecking annotations on a function or a class, the ``Annotated`` type allows for both static typechecking of ``T`` (e.g., via MyPy or Pyre, which can safely ignore ``x``) together with runtime access to ``x`` within a specific application. We believe that the introduction of this type would address a diverse set of use cases of interest to the broader Python community. Motivating examples: ~~~~~~~~~~~~~~~~~~~~ reading binary data +++++++++++++++++++ The ``struct`` module provides a way to read and write C structs directly from their byte representation. It currently relies on a string representation of the C type to read in values:: record = b'raymond \x32\x12\x08\x01\x08' name, serialnum, school, gradelevel = unpack('<10sHHb', record) The struct documentation [struct-examples]_ suggests using a named tuple to unpack the values and make this a bit more tractable:: from collections import namedtuple Student = namedtuple('Student', 'name serialnum school gradelevel') Student._make(unpack('<10sHHb', record)) # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) However, this recommendation is somewhat problematic; as we add more fields, it's going to get increasingly tedious to match the properties in the named tuple with the arguments in ``unpack``. Instead, annotations can provide better interoperability with a type checker or an IDE without adding any special logic outside of the ``struct`` module:: from typing import NamedTuple UnsignedShort = Annotated[int, struct.ctype('H')] SignedChar = Annotated[int, struct.ctype('b')] @struct.packed class Student(NamedTuple): # MyPy typechecks 'name' field as 'str' name: Annotated[str, struct.ctype("<10s")] serialnum: UnsignedShort school: SignedChar gradelevel: SignedChar # 'unpack' only uses the metadata within the type annotations Student.unpack(record)) # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) dataclasses ++++++++++++ Here's an example with dataclasses [dataclass]_ that is a problematic from the typechecking standpoint:: from dataclasses import dataclass, field @dataclass class C: myint: int = 0 # the field tells the @dataclass decorator that the default action in the # constructor of this class is to set "self.mylist = list()" mylist: List[int] = field(default_factory=list) Even though one might expect that ``mylist`` is a class attribute accessible via ``C.mylist`` (like ``C.myint`` is) due to the assignment syntax, that is not the case. Instead, the ``@dataclass`` decorator strips out the assignment to this attribute, leading to an ``AttributeError`` upon access:: C.myint # Ok: 0 C.mylist # AttributeError: type object 'C' has no attribute 'mylist' This can lead to confusion for newcomers to the library who may not expect this behavior. Furthermore, the typechecker needs to understand the semantics of dataclasses and know to not treat the above example as an assignment operation in (which translates to additional complexity). It makes more sense to move the information contained in ``field`` to an annotation:: @dataclass class C: myint: int = 0 mylist: Annotated[List[int], field(default_factory=list)] # now, the AttributeError is more intuitive because there is no assignment operator C.mylist # AttributeError # the constructor knows how to use the annotations to set the 'mylist' attribute c = C() c.mylist # [] The main benefit of writing annotations like this is that it provides a way for clients to gracefully degrade when they don't know what to do with the extra annotations (by just ignoring them). If you used a typechecker that didn't have any special handling for dataclasses and the ``field`` annotation, you would still be able to run checks as though the type were simply:: class C: myint: int = 0 mylist: List[int] lowering barriers to developing new types +++++++++++++++++++++++++++++++++++++++++ Typically when adding a new type, we need to upstream that type to the typing module and change MyPy [MyPy]_, PyCharm [PyCharm]_, Pyre [Pyre]_, pytype [pytype]_, etc. This is particularly important when working on open-source code that makes use of our new types, seeing as the code would not be immediately transportable to other developers' tools without additional logic (this is a limitation of MyPy plugins [MyPy-plugins]_), which allow for extending MyPy but would require a consumer of new typehints to be using MyPy and have the same plugin installed). As a result, there is a high cost to developing and trying out new types in a codebase. Ideally, we should be able to introduce new types in a manner that allows for graceful degradation when clients do not have a custom MyPy plugin, which would lower the barrier to development and ensure some degree of backward compatibility. For example, suppose that we wanted to add support for tagged unions [tagged-unions]_ to Python. One way to accomplish would be to annotate ``TypedDict`` in Python such that only one field is allowed to be set:: Currency = Annotated( TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False), TaggedUnion, ) This is a somewhat cumbersome syntax but it allows us to iterate on this proof-of-concept and have people with non-patched IDEs work in a codebase with tagged unions. We could easily test this proposal and iron out the kinks before trying to upstream tagged union to `typing`, MyPy, etc. Moreover, tools that do not have support for parsing the ``TaggedUnion`` annotation would still be able able to treat `Currency` as a ``TypedDict``, which is still a close approximation (slightly less strict). Details of proposed changes to ``typing`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ syntax ++++++ ``Annotated`` is parameterized with a type and an arbitrary list of Python values that represent the annotations. Here are the specific details of the syntax: * The first argument to ``Annotated`` must be a valid ``typing`` type or ``...`` (to use the infered type). * Multiple type annotations are supported (Annotated supports variadic arguments): ``Annotated[int, ValueRange(3, 10), ctype("char")]`` * ``Annotated`` must be called with at least two arguments (``Annotated[int]`` is not valid) * The order of the annotations is preserved and matters for equality checks:: Annotated[int, ValueRange(3, 10), ctype("char")] != \ Annotated[int, ctype("char"), ValueRange(3, 10)] * Nested ``Annotated`` types are flattened, with metadata ordered starting with the innermost annotation:: Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] ==\ Annotated[int, ValueRange(3, 10), ctype("char")] * Duplicated annotations are not removed: ``Annotated[int, ValueRange(3, 10)] != Annotated[int, ValueRange(3, 10), ValueRange(3, 10)]`` * ``Annotation`` can be used a higher order aliases:: Typevar T = ... Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] # Vec[int] == `Annotated[List[Tuple[int, int]], MaxLen(10)] consuming annotations ++++++++++++++++++++++ Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the `Annotated` type. A tool or library encountering an `Annotated` type can scan through the annotations to determine if they are of interest (e.g., using `isinstance`). **Unknown annotations** When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. For example, if we were to add an annotation that is not an instance of `struct.ctype` to the annotation for name (e.g., `Annotated[str, 'foo', struct.ctype("<10s")]`), the unpack method should ignore it. **Namespacing annotations** We do not need namespaces for annotations since the class used by the annotations acts as a namespace. **Multiple annotations** It's up to the tool consuming the annotations to decide whether the client is allowed to have several annotations on one type and how to merge those annotations. Since the ``Annotated`` type allows you to put several annotations of the same (or different) type(s) on any node, the tools or libraries consuming those annotations are in charge of dealing with potential duplicates. For example, if you are doing value range analysis you might allow this:: T1 = Annotated[int, ValueRange(-10, 5)] T2 = Annotated[T1, ValueRange(-20, 3)] Flattening nested annotations, this translates to:: T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)] An application consuming this type might choose to reduce these annotations via an intersection of the ranges, in which case ``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-10, 3)]``. An alternative application might reduce these via a union, in which case ``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-20, 5)]``. Other applications may decide to not support multiple annotations and throw an exception. References =========== .. [struct-examples] https://docs.python.org/3/library/struct.html#examples .. [dataclass] https://docs.python.org/3/library/dataclasses.html .. [MyPy] https://github.com/python/mypy .. [MyPy-plugins] https://mypy.readthedocs.io/en/latest/extending_mypy.html#extending-mypy-us… .. [PyCharm] https://www.jetbrains.com/pycharm/ .. [Pyre] https://pyre-check.org/ .. [pytype] https://github.com/google/pytype .. [tagged-unions] https://en.wikipedia.org/wiki/Tagged_union

2 2

tkinter: time for round buttons?
by Abdur-Rahmaan Janhangeer 17 Jan '19

17 Jan '19

without starting a should we ban tkinter discussion, i'd like to propose that we add rounded corners buttons. that might make the aesthetic level go up a bit more poor me, if only py had some really nice native gui Abdur-Rahmaan Janhangeer http://www.pythonmembers.club Mauritius

8 11

Fwd: NAN handling in the statistics module
by David Mertz 10 Jan '19

10 Jan '19

> > One possible argument for making PASS the default, even if that means >> implementation-dependent behaviour with NANs, is that in the absense of a >> clear preference for FAIL or RETURN, at least PASS is backwards compatible. >> >> You might shoot yourself in the foot, but at least you know its the same >> foot you shot yourself in using the previous version *wink* >> > I've lost attribution chain. I think this is Steven, but it doesn't really matter. This statement is untrue, or at least only accidentally true at most. The behavior of sorted() against partially ordered collections is unspecified. The author of Timsort says exactly this. If stastics.median() keeps the same implementation—or keeps it with a PASS argument—it may or may not produce the same result in a later Python versions. Timsort is great, but even that has been tweaked sightly over time. I guess the statement is true if "same foot" means "meaningless answer" not some specific value. But that hardly feels like a defense of the behavior. >

2 1

NAN handling in the statistics module
by Steven D'Aprano 10 Jan '19

10 Jan '19

Bug #33084 reports that the statistics library calculates median and other stats wrongly if the data contains NANs. Worse, the result depends on the initial placement of the NAN: py> from statistics import median py> NAN = float('nan') py> median([NAN, 1, 2, 3, 4]) 2 py> median([1, 2, 3, 4, NAN]) 3 See the bug report for more detail: https://bugs.python.org/issue33084 The caller can always filter NANs out of their own data, but following the lead of some other stats packages, I propose a standard way for the statistics module to do so. I hope this will be uncontroversial (he says, optimistically...) but just in case, here is some prior art: (1) Nearly all R stats functions take a "na.rm" argument which defaults to False; if True, NA and NAN values will be stripped. (2) The scipy.stats.ttest_ind function takes a "nan_policy" argument which specifies what to do if a NAN is seen in the data. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.… (3) At least some Matlab functions, such as mean(), take an optional flag that determines whether to ignore NANs or include them. https://au.mathworks.com/help/matlab/ref/mean.html#bt5b82t-1-nanflag I propose adding a "nan_policy" keyword-only parameter to the relevant statistics functions (mean, median, variance etc), and defining the following policies: IGNORE: quietly ignore all NANs FAIL: raise an exception if any NAN is seen in the data PASS: pass NANs through unchanged (the default) RETURN: return a NAN if any NAN is seen in the data WARN: ignore all NANs but raise a warning if one is seen PASS is equivalent to saying that you, the caller, have taken full responsibility for filtering out NANs and there's no need for the function to slow down processing by doing so again. Either that, or you want the current implementation-dependent behaviour. FAIL is equivalent to treating all NANs as "signalling NANs". The presence of a NAN is an error. RETURN is equivalent to "NAN poisoning" -- the presence of a NAN in a calculation causes it to return a NAN, allowing NANs to propogate through multiple calculations. IGNORE and WARN are the same, except IGNORE is silent and WARN raises a warning. Questions: - does anyone have an serious objections to this? - what do you think of the names for the policies? - are there any additional policies that you would like to see? (if so, please give use-cases) - are you happy with the default? Bike-shed away! -- Steve

8 12

Re: [Python-ideas] NAN handling in the statistics module
by Steven D'Aprano 09 Jan '19

09 Jan '19

On Sun, Jan 06, 2019 at 10:52:47PM -0500, David Mertz wrote: > Playing with Tim's examples, this suggests that statistics.median() is > simply outright WRONG. I can think of absolutely no way to characterize > these as reasonable results: > > Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) > In [4]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4, 5]) > Out[4]: 1 > In [5]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4]) > Out[5]: nan The second is possibly correct if one thinks that the median of a list containing NAN should return NAN -- but its only correct by accident, not design. As I wrote on the bug tracker: "I agree that the current implementation-dependent behaviour when there are NANs in the data is troublesome." The only reason why I don't call it a bug is that median() makes no promises about NANs at all, any more than it makes promises about the median of a list of sets or any other values which don't define a total order. help(median) says: Return the median (middle value) of numeric data. By definition, data containing Not A Number values isn't numeric :-) I'm not opposed to documenting this better. Patches welcome :-) There are at least three correct behaviours in the face of data containing NANs: propogate a NAN result, fail fast with an exception, or treat NANs as missing data that can be ignored. Only the caller can decide which is the right policy for their data set. Aside: the IEEE-754 standard provides both signalling and quiet NANs. It is hard and unreliable to generate signalling float NANs in Python, but we can do it with Decimal: py> from statistics import median py> from decimal import Decimal py> median([1, 3, 4, Decimal("sNAN"), 2]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.5/statistics.py", line 349, in median data = sorted(data) decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>] In principle, one ought to be able to construct float signalling NANs too, but unfortunately that's platform dependent: https://mail.python.org/pipermail/python-dev/2018-November/155713.html Back to the topic on hand: I agree that median() does "the wrong thing" when NANs are involved, but there is no one "right thing" that we can do in its place. People disagree as to whether NANs should propogate, or raise, or be treated as missing data, and I see good arguments for all three. -- Steve

6 16

Re: [Python-ideas] NAN handling in the statistics module
by Steven D'Aprano 08 Jan '19

08 Jan '19

On Tue, Jan 08, 2019 at 04:25:17PM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > By definition, data containing Not A Number values isn't numeric :-) > > Unfortunately, that's just a joke, because in fact numeric functions > produce NaNs. I'm not sure if you're agreeing with me or disagreeing, so I'll assume you're agreeing and move on :-) > I agree that this can easily be resolved by documenting that it is the > caller's responsibility to remove NaNs from numeric data, but I prefer > your proposed flags. > > > The only reason why I don't call it a bug is that median() makes no > > promises about NANs at all, any more than it makes promises about the > > median of a list of sets or any other values which don't define a total > > order. > > Pedantically, I would prefer that the promise that ordinal data > (vs. specifically numerical) has a median be made explicit, as there > are many cases where statistical data is ordinal. I think that is reasonable. Provided the data defines a total order, the median is well-defined when there are an odd number of data points, or you can use median_low and median_high regardless of the number of data points. > This may be a moot > point, as in most cases ordinal data is represented numerically in > computation (Likert scales, for example, are rarely coded as "hate, > "dislike", "indifferent", "like", "love", but instead as 1, 2, 3, 4, > 5), and from the point of view of UI presentation, IntEnums do the > right thing here (print as identifiers, sort as integers). > > Perhaps a better way to document this would be to suggest that ordinal > data be represented using IntEnums? (Again to be pedantic, one might > want OrderedEnums that can be compared but don't allow other > arithmetic operations.) That's a nice solution. -- Steve (the other one)

1 0

Possible PEP regarding the use of the continue keyword in try/except blocks
by Simon 08 Jan '19

08 Jan '19

I was writing some python code earlier, and I noticed that in a code that looks somwhat like this one : try: i = int("string") print("continued on") j = int(9.0) except ValueError as e: print(e) >>> "invalid literal for int() with base 10: 'string'" this code will handle the exception, but the code in the try block will not continue. I propose to be able to use the continue keyword to continue the execution of the try block even when an error is handled. The above could then be changed to : try: i = int("string") print("continued on") j = int(9.0) except ValueError as e: print(e) continue >>> "invalid literal for int() with base 10: 'string'" >>> "continued on"

13 16

Re: [Python-ideas] NAN handling in the statistics module
by Tim Peters 07 Jan '19

07 Jan '19

[David Mertz <david.mertz(a)gmail.com>] > Thanks Tim for clarifying. Is it even the case that sorts are STABLE in > the face of non-total orderings under __lt__? A couple quick examples > don't refute that, but what I tried was not very thorough, nor did I > think much about TimSort itself. I'm not clear on what "stable" could mean in the absence of a total ordering. Not only does sort not assume __lt__ is a total ordering, it doesn't assume it's transitive, or even deterministic. We really can't assume anything about potentially user-defined functions. What sort does guarantee is that the result list is some permutation of the input list, regardless of how insanely __lt__ may behave. If __lt__ sanely defines a deterministic total order, then "stable" and "sorted" are guaranteed too, with their obvious meanings.

3 4

Re: [Python-ideas] NAN handling in the statistics module
by David Mertz 07 Jan '19

07 Jan '19

[... apologies if this is dup, got a bounce ...] > [David Mertz <mertz(a)gnosis.cx>] >> I have to say though that the existing behavior of `statistics.median[_low|_high|]` >> is SURPRISING if not outright wrong. It is the behavior in existing Python, >> but it is very strange. >> >> The implementation simply does whatever `sorted()` does, which is an >> implementation detail. In particular, NaN's being neither less than nor >> greater than any floating point number, just stay where they are during >> sorting. > > I expect you inferred that from staring at a handful of examples, but > it's illusion. Python's sort uses only __lt__ comparisons, and if > those don't implement a total ordering then _nothing_ is defined about > sort's result (beyond that it's some permutation of the original > list). Thanks Tim for clarifying. Is it even the case that sorts are STABLE in the face of non-total orderings under __lt__? A couple quick examples don't refute that, but what I tried was not very thorough, nor did I think much about TimSort itself. > So, certainly, if you want median to be predictable in the presence of > NaNs, sort's behavior in the presence of NaNs can't be relied on in > any respect. Playing with Tim's examples, this suggests that statistics.median() is simply outright WRONG. I can think of absolutely no way to characterize these as reasonable results: Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) In [4]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4, 5]) Out[4]: 1 In [5]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4]) Out[5]: nan

1 0

Fixed point format for numbers with locale based separators
by Stefan Krah 06 Jan '19

06 Jan '19

Eric V. Smith wrote: > If the locale character is "*", use locale-aware formatting for the > given "type", with LC_NUMERIC. So, "*g" would be equivalent to the > existing "n", and "*f" would give you the current "f" formatting, except > using LC_NUMERIC for the decimal point. If the locale character is "$" > use locale-aware LC_MONETARY. So then we could use "$g", "$f", etc. > These locale characters would also work with int, so "*d" would make "n" > obsolete (but I'm not proposing to remove it). +1. I also think it's best to have a modifier and cover all cases. > But it's probably worth keeping '#' orthogonal to the locale character. > Maybe someday we'll want to use them together. Yes, somehow it feels right to keep them separate, even if we never use them together. > I think it might also require a PEP, but it would be as simple as > PEP 378 for adding comma grouping formatting. Somewhere to memorialize > the decision and how we got there, including rejected alternate > proposals, would be a good thing. It would be nice (if anyone wants to do the work), but your proposal is already perfect for me. I like the "*" and "$" choices. Stefan Krah

1 0

回复：Python-ideas Digest, Vol 146, Issue 13
by Moon丶sun 06 Jan '19

06 Jan '19

Thanks for your reply.But the answer is not I except, I will show you some examples to explain what result I except: @contextmanagerdef cm(): print('open file') yield print('close file')with cm(): 1/0 If I use a contextmanager ,I except it can help me to close the file anytime,even raise an error,but if I define a function with @contextmanager like the example which I have showed for you, it will never print('close file') I can only modify it like this:@contextmanagerdef cm(): try: print('open file') yield except Exception as e: print('Error',e) finally: print('close file') It is not friendly for us to use it, so I modify the contextlib to fix it,you can catch it from the e-mail attachment.It's in the line 79 and line 97---------------------------------------------------------------------- 发件人：python-ideas-request(a)python.org 收件人：python-ideas(a)python.org 主题：Python-ideas Digest, Vol 146, Issue 13 日期：2019年01月06日 01点05分 Send Python-ideas mailing list submissions to python-ideas(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-ideas or, via email, send a message with subject or body 'help' to python-ideas-request(a)python.org You can reach the person managing the list at python-ideas-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-ideas digest..." Today's Topics: 1. Re: Make the @contextmanager of contextlib to be a real contextmanager (Serhiy Storchaka) 2. Re: Fixed point format for numbers with locale based separators (?ukasz Stelmach) ---------------------------------------------------------------------- Message: 1 Date: Sat, 5 Jan 2019 16:45:34 +0200 From: Serhiy Storchaka <storchaka(a)gmail.com> To: python-ideas(a)python.org Subject: Re: [Python-ideas] Make the @contextmanager of contextlib to be a real contextmanager Message-ID: <q0qfqd$q1q$1(a)blaine.gmane.org> Content-Type: text/plain; charset=UTF-8; format=flowed 05.01.19 14:52, Moon?sun ????: > As we know,when we import the module--'contextlib',we can use the > decorator '@contextmanager' and keyword ?yield? to make a 'instance' of > Class '_GeneratorContextManager' in 'contextlib' module,then we can use > it like: > with 'instance' as 'xx': > ? ? 'code block' > ? ? pass > But there is a little bug,when the code block raise a error,the instance > cannot run the code which after the keyword 'yield'. This is not a bug. Consider the following example: @contextmanager def cm(): try: yield except BaseException as err: print('Fail:', err) raise else: print('Success') with cm(): 1/0 What result would you expect? Test it with the stdlib implementation and with your implementation. ------------------------------ Message: 2 Date: Sat, 05 Jan 2019 16:41:20 +0100 From: ?ukasz Stelmach <steelman(a)post.pl> To: Steven D'Aprano <steve(a)pearwood.info> Cc: python-ideas(a)python.org Subject: Re: [Python-ideas] Fixed point format for numbers with locale based separators Message-ID: <87y37z9knz.fsf%steelman(a)post.pl> Content-Type: text/plain; charset="utf-8" Steven D'Aprano <steve(a)pearwood.info> writes: > On Fri, Jan 04, 2019 at 03:57:53PM +0100, ?ukasz Stelmach wrote: >> Hi, >> >> I would like to present two pull requests[1][2] implementing fixed point >> presentation of numbers and ask for comments. The first is mine. I >> learnt about the second after publishing mine. > > Before I look at the implementation, can you explain the functional > requirements please? As I stated in the original message below the table: >> In the application I want to create I am going to present users numbers >> ranging up to 3 orders of magnitude and I (my users) want them to be >> presented consistently with regards to number of decimal digits AND I >> want to conform to rules of languages of my users. And I would like to >> avoid the exponent notation by all means. The pint[1] library I use, implements formatting of physical quantities using the format()/__format__ code. As far as I can tell my patch for Python is shorter and more straightforward than a patch for pint to use locale.format(). Because the "g" based "n" formatter has been present since the advanced string formatting was described in PEP-3101, I think it is necessary to add the "m" formatter based on "f". The advanced string formatting facility in Python is very convenient and programmers shouldn't forced to use locale.format() like this "The total length of {} sticks is {} meters.".format(n_sticks, locale.format(".2f", l_sticks)) instead of "The total length of {} sticks is {:.2f} meters.".format(n_sticks, l_sticks) > In other words, what is the new feature you hope to have excepted? > Explain the intention and the API (the interface). The implementation is > the least important part :-) I wish to add a new formatter "m" for float/complex/decimal numbers, which behaves like the existing "f", but uses the decimal separator from the locale database. There is "n" formmatter which behaves like "g" but it does not fit my needs. > [...] >> Formatting 1.23456789 * n (LC_ALL=3Dpl_PL.UTF-8) >> | n | ".2f" | ".3n" | >> |---+----------+----------| >> | 1 | 1.23 | 1,23 | >> | 2 | 12.35 | 12,3 | >> | 3 | 123.46 | 123 | >> | 4 | 1234.57 | 1,23e+03 | > > I'm afraid I cannot work out what that table means. You say "Formatting > 1.23... * n" (multiplying by n) but the results shown aren't multiplied > by n=2, n=3, n=4 as the table suggests. > > Can you show what Python code you expect will produce the expected > output? for n in range(1,5): print("| {} | {:8.2f} | {:8.3n} |".format(n,1.23456789 * 10**n, 1.23456789 * 10**n)) [1] http://pint.readthedocs.io/ -- By?o mi bardzo mi?o. --- Rurku. --- ... >?ukasz< --- To dobrze, ?e mnie s?uchasz.

2 1

Make the @contextmanager of contextlib to be a real contextmanager
by Moon丶sun 05 Jan '19

05 Jan '19

As we know,when we import the module--'contextlib',we can use the decorator '@contextmanager' and keyword ‘yield’ to make a 'instance' of Class '_GeneratorContextManager' in 'contextlib' module,then we can use it like:with 'instance' as 'xx': 'code block' passBut there is a little bug,when the code block raise a error,the instance cannot run the code which after the keyword 'yield'.So I try to repair this bug, I add a method '_next()' in the Class '_GeneratorContextManager',and then insert it to the the method '__exit__',then we can make the instance like a realcontextmanager.You can cat the concrete content in the attachment. Conmments:the method '_next()' in in line 79,the method '__exit__' in line 97.

2 1

Add regex pattern literal p""
by Ma Lin 02 Jan '19

02 Jan '19

We can use this literal to represent a compiled pattern, for example: >>> p"(?i)[a-z]".findall("a1B2c3") ['a', 'B', 'c'] >>> compiled = p"(?<=abc)def" >>> m = compiled.search('abcdef') >>> m.group(0) 'def' >>> rp'\W+'.split('Words, words, words.') ['Words', 'words', 'words', ''] This allows peephole optimizer to store compiled pattern in .pyc file, we can get performance optimization like replacing constant set by frozenset in .pyc file. Then such issue [1] can be solved perfectly. [1] Optimize base64.b16decode to use compiled regex [1] https://bugs.python.org/issue35559 Two shortcomings: 1, Elevating a class in a module (re.Pattern) to language level, this sounds not very natural. This makes Python looks like Perl. 2, We can't use regex module as a drop-in replacement: import regex as re IMHO, I would like to see regex module be adopted into stdlib after cutting off its "full case-folding" and "fuzzy matching" features. Related links: [2] Chris Angelico conceived of "compiled regexes be stored in .pyc file" in March 2013. [2] https://mail.python.org/pipermail/python-ideas/2013-March/020043.html [3] Ken Hilton conceived of "Give regex operations more sugar" in June 2018. [3] https://mail.python.org/pipermail/python-ideas/2018-June/051395.html

14 27