Mailman 3 August 2018 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 12, 2021

March 12, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/… [View More]clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman [View Less]

12 35

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Add the imath module
by Serhiy Storchaka March 22, 2020

March 22, 2020

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math … [View More]

14 38

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar March 7, 2020

March 7, 2020

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to … [View More]

2 2

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Please consider adding of functions file system operations to pathlib
by George Fischhof Nov. 16, 2019

Nov. 16, 2019

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger Oct. 23, 2019

Oct. 23, 2019

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed … [View More]for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel [View Less]

17 49

Re: [Python-ideas] Trigonometry in degrees
by Hugh Fisher May 8, 2019

May 8, 2019

> Date: Thu, 7 Jun 2018 12:33:29 +0000 > From: Robert Vanden Eynde <robertvandeneynde(a)hotmail.com> > To: python-ideas <python-ideas(a)python.org> > Subject: [Python-ideas] Trigonometry in degrees > Message-ID: > > > I suggest adding degrees version of the trigonometric functions in the math module. > > - Useful in Teaching and replacing calculators by python, importing something is seen by the young students much more easy than to define a … [View More]function. I agree that degrees are useful for teaching. They are also very useful for graphics programming, especially with my favourite OpenGL API. But I think that the use of radians in programming language APIs is more prevalent, so the initial advantage of easy learning will be outweighed by the long term inconvenience of adjusting to what everyone else is doing. Writing degrees(x) and radians(x) is a little inconvenient, but it does make it clear what units are being used. And even if your proposal is adopted, there is still going to be a lot of code around that uses the older math routines. With the current API it is a least safe to assume that angles are radians unless stated otherwise. > - Special values could be treated, aka when the angle is a multiple of 90, young students are often surprise to see that cos(pi/2) != 0 > > Testing for a special value Isn't very costly (x % 90 == 0) but it could be pointed out that there is a small overhead using the "degrees" equivalent of trig function because of the radians to degrees conversion And the special values testing. Not just young students :-) I agree with this, but I would prefer the check to be in the implementation of the existing functions as well. Any sin/cos very close to 0 becomes 0, any close to 1 becomes 1. > - Standard names will be chosen so that everyone will use the same name convention. I suggest adding a "d" like sind, cosd, tand, acosd, asind, atand, atan2d. Not "d". In the OpenGL 3D API, and many 3D languages/APIs since, appending "d" means "double precision". It's even sort of implied by the C math library which has sinf and friends for single precision. > > Creating a new package like 'from math.degrees import cos' however I would not recommend that because "cos" in the source code would mean to lookup the import to know if it's in degrees or radians (and that leads to very filthy bugs). Also "degrees" is already so the name would have to change the name of the package. Agree, not a good idea. -- cheers, Hugh Fisher [View Less]

16 29

Support localization of unicode descriptions
by Pander Feb. 25, 2019

Feb. 25, 2019

Dear all, Who might also be interested in setting up a project that supports localization for Unicode block description and character description. Translations are available from https://github.com/unicode-table/unicode-table-data/tree/master/loc <https://github.com/unicode-table/unicode-table-data/tree/master/loc> If possible, use a gettext approach similar to https://pypi.org/project/pycountry/ <https://pypi.org/project/pycountry/> Implementing this feature will allow users to … [View More]

5 8

Dart like multi line strings identation
by Marius Räsener Feb. 8, 2019

Feb. 8, 2019

Hey List, this is my very first approach to suggest a Python improvement I'd think worth discussing. At some point, maybe with Dart 2.0 or a little earlier, Dart is now supporting multiline strings with "proper" identation (tried, but I can't find the according docs at the moment. probably due to the rather large changes related to dart 2.0 and outdated docs.) What I have in mind is probably best described with an Example: print(""" I am a multiline String. """) the closing … [View More]

23 47

Pypi private repo's
by Nick Humrich Oct. 29, 2018

Oct. 29, 2018

I am sure this has been discussed before, and this might not even be the best place for this discussion, but I just wanted to make sure this has been thought about. What if pypi.org supported private repos at a cost, similar to npm? This would be able to help support the cost of pypi, and hopefully make it better/more reliable, thus in turn improving the python community. If this discussion should happen somewhere else, let me know. Nick

5 5

Re: [Python-ideas] [Distutils] Pypi private repo's
by Nick Coghlan Oct. 28, 2018

Oct. 28, 2018

On 5 April 2018 at 07:58, Jannis Gebauer <ja.geb(a)me.com> wrote: > What if there was some kind of “blessed” entity that runs these services and puts the majority of the revenue into a fund that funds development on PyPi (maybe trough the PSF)? Having a wholly owned for-profit subsidiary that provides commercial services as a revenue raising mechanism is certainly one way to approach something like this without alienating sponsors or tax authorities (although it may still alienate the … [View More]vendors of now competing services). It would require a big time commitment on the PSF side to get everything set up though, as well as interest from key folks in joining what would essentially be a single-language-focused start up in an already crowded cross-language developer tools marketplace. When the PSF as a whole is still operating with only a handful of full or part time employees, it's far from clear that setting something like that up would be the most effective possible use of their time and energy. At a more basic level, that kind of arrangement technically doesn't require anyone's blessing, it could be as straightforward as downstream tooling vendors signing up as PSF sponsors and saying "please allocate our sponsorship contribution to the Packaging WG's budget so that PyPI keeps operating well and the PyPA tooling keeps improving, increasing the level of demand for our commercial Python repository management services". Historically that wouldn't have helped much, since the PSF itself has struggled with effective project management (for a variety of reasons), but one of the things I think the success of the MOSS grant has shown is the significant strides that the PSF has made in budget management in recent years, such that if funding is made available, it can and will be spent effectively. Cheers, Nick. P.S. PyPA contributors are also free agents in their own right, so folks offering Python-centric developer workflow management tools or features may decide that it's worth their while to invest more directly in smoothing out some of the rough edges that currently still exist. It's a mercenary way of looking at things, but in many cases, it is *absolutely* possible to pay for the time and attention of existing contributors, and if you can persuade them that your proposals are reasonable, they'll often have an easier time than most convincing other community contributors that it's a good way to go :) -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

3 5

Asynchronous exception handling around with/try statement borders
by Erik Bray Sept. 24, 2018

Sept. 24, 2018

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution … [View More]as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts? [View Less]

7 15

Pre-conditions and post-conditions
by Marko Ristin-Kaufmann Sept. 22, 2018

Sept. 22, 2018

Hi, I would be very interested to bring design-by-contract into python 3. I find design-by-contract particularly interesting and indispensable for larger projects and automatic generation of unit tests. I looked at some of the packages found on pypi and also we rolled our own solution (https://github.com/Parquery/icontract/). I also looked into https://www.python.org/dev/peps/pep-0316/. However, all the current solutions seem quite clunky to me. The decorators involve an unnecessary … [View More]

19 53

PEP 420: implicit namespace sub-package
by Gallian Colombeau Sept. 12, 2018

Sept. 12, 2018

Hello, the PEP 420 (Implicit Namespace Packages) is quite descriptive about the problem and the solution implemented back in Python 3.3 but I feel there may be a part missing (but maybe this is categorized as an implementation detail). As I understand, for a package to allow being extended in this way, it must be a namespace package and not contain a marker file. As a matter of fact, no sub-package until the top level package can have a marker file: Lib/test/namespace_pkgs project1 … [View More]

3 2

Positional-only parameters
by Victor Stinner Sept. 11, 2018

Sept. 11, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the … [View More]

19 27

Re: [Python-ideas] Syntactic sugar to declare partial functions
by Michel Desmoulin Sept. 8, 2018

Sept. 8, 2018

I'd rather have functools.partial() to be added as a new method on function objects. > > fromfunctools importpartial > > > def add(x:int,y:int)->int: > returnx +y > > > add_2 = partial(add,2) > Would become: add_2 = add.partial(2) Nothing to change on the parser, no obscure syntax for future readers, and we can get the opportunity of rewriting partial() in C as right now it is amazingly way, way slower than a lambda.

30 142

Re: [Python-ideas] Python-ideas Digest, Vol 141, Issue 145
by James Lu Sept. 6, 2018

Sept. 6, 2018

> As Matthew points out, you could use numpy.array. Or code your own > class, by providing __add__ and __iadd__ methods. > > >>> import numpy > >>> a = numpy.array([1, 2]) > >>> b = numpy.array([3, 4]) > >>> a + b > array([4, 6]) > >>> a += b > >>> a > array([4, 6]) I could, but I don't think that justifies not having this functionality in python standard. From the language experience perspective, numpy is … [View More]often a pain to install on most systems. If I'm designing card games and I just want to run a quick monte carlo simulation, the experience should be as smooth as possible. This is something I think most students will expect while learning python, especially if they're implementing algorithms. On Mon, Aug 27, 2018 at 4:24 AM <python-ideas-request(a)python.org> wrote: > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Unpacking iterables for augmented assignment (Matthew Einhorn) > 2. Re: Unpacking iterables for augmented assignment (Jonathan Fine) > 3. Re: Pre-conditions and post-conditions (Jacco van Dorp) > 4. Re: Pre-conditions and post-conditions (Ivan Levkivskyi) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 27 Aug 2018 01:29:14 -0400 > From: Matthew Einhorn <moiein2000(a)gmail.com> > To: python-ideas(a)python.org > Subject: Re: [Python-ideas] Unpacking iterables for augmented > assignment > Message-ID: > <CALCauYeAjPWek5L=BU-= > yMM-FNYw3Bza2hJqsGmdgtrvBUa7XA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Sun, Aug 26, 2018, 9:24 PM James Lu <jamtlu(a)gmail.com> wrote: > > > Hi Johnathan > > > > I echo your points. Indeed, the PEP referenced to refers to a "tuple > > expression" in the grammatical and not the programmatic sense. > > > > Finally, here's something that surprised me a little bit > > > > >>> x = [1, 2]; id(x) > > 140161160364616 > > >>> x += [3, 4]; id(x) > > 140161160364616 > > > > >>> x = (1, 2); id(x) > > 140161159928520 > > >>> x += (3, 4); id(x) > > 140161225906440 > > > > Notice that '+=' creates uses the same object when the object is > > a > > list, but creates a new object. This raises the question: Why and > > how > > does Python behave in this way? > > > > It's because lists are mutable are tuples are immutable. > > There's a dunder iadd method and a dunder add method. > > iadd magic methods, operating on the left hand side, return None and > > modify the object in-place. add magic methods return the result and > > don't modify the object it's called on. > > iadd is mutable add, whereas add is "return a copy with the result > > added" > > > > >>> tuple.__iadd__ > > Traceback (most recent call last): > > File "<stdin>", line 1, in <module> > > AttributeError: type object 'tuple' has no attribute '__iadd__' > > type object 'tuple' has no attribute '__iadd__' > > >>> tuple.__add__ > > <slot wrapper '__add__' of 'tuple' objects> > > >>> list.__iadd__ > > <slot wrapper '__iadd__' of 'list' objects> > > >>> list.__add__ > > <slot wrapper '__add__' of 'list' objects> > > > > > > tuple1 = tuple1.__add__(tuple2) > > > > list1.__iadd__(list2) > > > > > Does it IN PRACTICE bring sufficient benefits to users? > > > > I found myself needing this when I was writing a monte-carlo > > simulation in python that required incrementing a tallying counter > > from a subroutine. > > > > > Wouldn't a numpy array be very suited for this kind of task? > > > > [View Less]

15 23

Re: [Python-ideas] Pre-conditions and post-conditions
by Hugh Fisher Sept. 4, 2018

Sept. 4, 2018

> Date: Thu, 30 Aug 2018 00:07:04 +0200 > From: Marko Ristin-Kaufmann <marko.ristin(a)gmail.com> ... > I think we got entangled in a discussion about whether design-by-contract > is useful or not. IMO, the personal experience ("I never used/needed this > feature") is quite an inappropriate rule whether something needs to be > introduced into the language or not. > > There seems to be evidence that design-by-contract is useful. Let me cite > Bertrand Meyer from … [View More]his article "Why not program right?" that I already > mentioned before: I don't think that being useful by itself should be enough. I think new features should also be "Pythonic" and I don't see design by contract notation as a good fit. For example, C has the useful & operator which lets you pass &foo as a pointer/array argument despite foo being a scalar, so assignment to bar[0] in the called function actually sets the value of foo. It might be possible to create some kind of aliasing operator for Python so that two or more variables were bound to the same location, but would we want it? No, because Python is not intended for that style of programming. For another example, GPU shading languages have the special keywords uniform and varying for distinguishing definitions that won't change across parallel invocations and definitions that will. Demonstrably very useful in computer games and supercomputer number crunching, so why doesn't Python have those keywords? Because it's not designed to be used for such. For design by contract, as others have noted Python assert statements work fine for simple preconditions and postconditions. I don't see any significant difference in readability between existing def foo(x, y): assert(x > 0) # Do stuff assert(x == y) and new style def foo(x, y): require: x > 0 # Do stuff ensure: x == y Yes there's more to design by contract than simple assertions, but it's not just adding syntax. Meyer often uses the special "old" construct in his post condition examples, a trivial example being ensure count = old.count + 1 How do we do that in Python? And another part of design by contract (at least according to Meyer) is that it's not enough to just raise an exception, but there must be a guarantee that it is handled and the post conditions and/or invariants restored. So there's more syntax for "rescue" and "retry" If you want to do simple pre and post conditions, Python already has assert. If you want to go full design by contract, there's no law saying that Python is the only programming language allowed. Instead of trying to graft new and IMHO alien concepts onto Python, what's wrong with Eiffel? -- cheers, Hugh Fisher [View Less]

9 12

Why shouldn't Python be better at implementing Domain Specific Languages?
by James Lu Sept. 3, 2018

Sept. 3, 2018

Why shouldn't Python be better at implementing Domain Specific Languages? >From Johnathan Fine: > I really do wish we could have language that had all of Ruby's > strengths, and also all of Python's. That would be really nice. Quite > something indeed. > Languages do influence each other. Ruby is good at internal Domain > Specific Languages (DSLs). And there's Perrotta's influential book on > Ruby Metaprogramming. That's something I think Python could learn > from. &… [View More]

12 20

Re: [Python-ideas] Fix some special cases in Fractions?
by Jeroen Demeyer Sept. 3, 2018

Sept. 3, 2018

On 2018-08-30 06:39, Neil Girdhar wrote: > I'd like these to be Fraction(1), Fraction(1), and Fraction(0). Why? I cannot think of any natural use case why you would want Fractions for a few special cases on an operation which returns non-Fractions generically. I consider it a feature to know in advance the type of the output of an operation, given the types of the input. Having an unexpected type suddenly show up because you happen to hit a special case is a recipe for bugs. Jeroen.

11 31

Extend the warnings module with Deprecation utility methods and classes
by Ilya Kulakov Aug. 31, 2018

Aug. 31, 2018

Sooner or later authors and maintainers of libraries change public interfaces of their creations. Usually one of the two approaches is taken: 1. Outright breaking change 2. Soft deprecation later followed by [1] While [1] is perfectly suitable for libraries with limited audience, [2] is what popular libraries have to do (to remain popular). Python's stdlib contributed to the process via PEP 230 and, recently, PEP 565. The `warn` function and subclasses of Warning are omnipresent. However,… [View More] when it comes to practical application it's still up to a developer to write utility methods and classes. I propose that Python should extend the warnings module with a set of utilities to cover basic needs as seen across the industry. The extension should include: 1. A decorator for functions 2. A decorator for classes 3. A context manager Function Decorator A function can be deprecated: - Via a rename when an author decides there is a better name - Via a change of its interface - By becoming obsolete Therefore the job of the decorator is to modify the behavior such as upon a call: - A warning is issued - Optionally an aliased (new) function is called - Optionally both positional and keyword arguments are mapped to satisfy new interface; might be followed by an extra warning for each remap Class Decorator A class can be deprecated: - Via a rename when an author decides there is a better name - By becoming obsolete Therefore a job of the decorator is to ensure that: - A warning is issued when class is either instantiated or used as a base class - If aliased, subclasses of a deprecated class (from the end user perspective) pass `issubclass` and `isinstance` checks against an aliased new class - If aliased, subclasses of a new class pass `issubclass` and `isinstance` checks against a deprecated class Context Manager Context manager is useful when behavior of a particular function changes without visible changes to its name or interface. E.g. when author realizes (too late) that arguments are mutually exclusive. Instead of introducing a breaking change outright, better solution is to wrap that particular case with a warning and consistent behavior. Each of these utilities should allow to specify a condition via a callable. E.g. when deprecation only makes sense for certain platforms or python interpreter versions. I think most if not all can be implemented using existing runtime and stdlib. But before diving into code, what do you think of this idea overall? [View Less]

2 2

Fix some special cases in Fractions?
by Neil Girdhar Aug. 31, 2018

Aug. 31, 2018

Would there be any problem with changing: In [4]: Fraction(1, 1) ** Fraction(2, 3) Out[4]: 1.0 In [5]: Fraction(-1, 1) ** Fraction(2, 3) Out[5]: (-0.4999999999999998+0.8660254037844387j) In [6]: Fraction(0, 1) ** Fraction(2, 3) Out[6]: 0.0 I'd like these to be Fraction(1), Fraction(1), and Fraction(0).

5 5

Re: [Python-ideas] Fix some special cases in Fractions?
by Jeroen Demeyer Aug. 30, 2018

Aug. 30, 2018

On 2018-08-30 11:11, Jonathan Fine wrote: > If anyone has time and ready access, it would help to know what > https://www.sympy.org/en/index.html does with this. It handles such powers symbolically, not actually returning a numerical result: >>> from sympy import Rational >>> Rational(1,2) ** Rational(2,3) 2**(1/3)/2 >>> Rational(1,1) ** Rational(2,3) 1 >>> Rational(-1,1) ** Rational(2,3) (-1)**(2/3) >>> Rational(0,1) ** Rational(2,3) 0

2 1

Re: [Python-ideas] Fix some special cases in Fractions?
by Jeroen Demeyer Aug. 30, 2018

Aug. 30, 2018

With gmpy2 (note that mpq=fractions, mpfr=floating-point reals): >>> from gmpy2 import mpq >>> mpq("1/1") ** mpq("2/3") mpfr('1.0') >>> mpq("-1/1") ** mpq("2/3") mpfr('nan') >>> mpq("0/1") ** mpq("2/3") mpfr('0.0')

1 0

Re: [Python-ideas] Fix some special cases in Fractions?
by Jonathan Fine Aug. 30, 2018

Aug. 30, 2018

Hi Neil We wrote >> This gives the same results as Python's Fraction, except for your >> example [6]. There, it gives the Fraction(0) you ask for. >> >> If the smart mathematicians and computer scientists that wrote gp/pari >> get the same answers, it suggests to me that improvement would be >> hard. > That's because these are just floating point numbers. These are not being > processed symbolically. Wolfram Alpha gives all three roots, including … [View More]

2 1

Re: [Python-ideas] Fix some special cases in Fractions?
by Jeroen Demeyer Aug. 30, 2018

Aug. 30, 2018

On 2018-08-30 11:05, Jonathan Fine wrote: > I'm used to using a number theory computer algebra system > https://pari.math.u-bordeaux.fr/. I don't think that a comparison with PARI is very relevant because PARI doesn't really have a type system the way that Python does. For example the fraction 3/1 doesn't really exist in PARI, only the integer 3 does.

1 0

zipfile encryption function
by 大野隆弘 Aug. 30, 2018

Aug. 30, 2018

Dear all, I would like to use zipfile encryption as python standard library. https://github.com/python/cpython/blob/master/Lib/zipfile.py Below document says "currently" cannot. https://github.com/python/cpython/blob/master/Doc/library/zipfile.rst "but it currently cannot create an encrypted file." Current pythonians like me have to use 3rd party like below, but I believe it is worth to include. https://pypi.org/project/pyminizip/ https://github.com/wllm-rbnt/py-zipcrypt please forgive me … [View More]

2 2

Re: [Python-ideas] Pre-conditions and post-conditions
by Ivan Levkivskyi Aug. 29, 2018

Aug. 29, 2018

replying to the list now... On Thu, 30 Aug 2018 at 00:11, Ivan Levkivskyi <levkivskyi(a)gmail.com> wrote: > On Wed, 29 Aug 2018 at 13:18, Steven D'Aprano <steve(a)pearwood.info> wrote: > >> I didn't want to embarass Ivan any further by seemingly picking on his >> opinion about contracts being always statically checked, but when I >> asked off-list I got told to reword and post it here. So here it is. >> >> Sorry Ivan if this makes you feel I'm … [View More]

1 0

Unpacking iterables for augmented assignment
by James Lu Aug. 28, 2018

Aug. 28, 2018

I propose we apply PEP 3132 to PEP 203. That is, for every statement where "<lhs> = <rhs>" is valid I propose "lhs += rhs" should also be valid. Simple example: a = 0 b = 0 a, b += 1, 2 # a is now 1 # b is now 2

7 14

A GUI for beginners and experts alike
by Mike Barnett Aug. 28, 2018

Aug. 28, 2018

Python has dropped the GUI ball, at least for beginners (in my opinion) While the Python language is awesomely compact, the GUI code is far from compact. Tkinter will create a nice looking GUI, but you've got to be skilled to use it. A student in their first week of Python programming is not going to be working with tkinter. It would be nice if beginners could double click a .py file and have it launch straight into a GUI, like most of the programs people are used to using. I think I've … [View More]

21 39

Re: [Python-ideas] Unpacking iterables for augmented assignment
by James Lu Aug. 27, 2018

Aug. 27, 2018

[Kirill Balunov] It may be worth taking a look at + and +=. However, the semantic difference is due to the dunder add and dunder iadd methods- necessary for supporting both mutable and immutable sequences. See my earlier mail for discussion on this topic. By the way, > the absence of literals for `set` and `frozenset` types, > The literal for the set is {1, 2, 3, 4}.

1 0

Re: [Python-ideas] A GUI for beginners and experts alike (Mike Barnett)
by Hugh Fisher Aug. 24, 2018

Aug. 24, 2018

> Date: Thu, 23 Aug 2018 18:49:48 +0000 > From: Mike Barnett <mike_barnett(a)hotmail.com> > > Python has dropped the GUI ball, at least for beginners (in my opinion) > > While the Python language is awesomely compact, the GUI code is far from compact. Tkinter will create a nice looking GUI, but you've got to be skilled to use it. A student in their first week of Python programming is not going to be working with tkinter. > > It would be nice if beginners could … [View More]

4 3

Fwd: Replacing Infinite while Loops with an Iterator: async edition
by jab＠math.brown.edu Aug. 24, 2018

Aug. 24, 2018

On Jun 23, 2018, at 21:11, Nathaniel Smith <n...(a)pobox.com> wrote: > He's asking for an async version of the 'iter' builtin, presumably > something like: > async def aiter(async_callable, sentinel): > while True: > value = await async_callable() > if value == sentinel: > break > yield value > -n Yes, exactly (thanks, Nathaniel). Wouldn't that be a useful built-in? (Greg, I too would be surprised if this were the first … [View More]

3 4

A simple proposal concerning lambda
by Jonathan Fine Aug. 23, 2018

Aug. 23, 2018

Here's what I call a simple proposal. Some people might find it outrageous. Today is not 1st April. BACKGROUND ============ Many Python users don't like the name lambda. But many Python users don't want any change here. This is true because there are millions of Python users, and even 100 is many. And it's hard to introduce a new keyword, which might break existing code. (And perhaps even worse, break the community.) https://wiki.python.org/moin/LocalUserGroups > There about 1,637 Python … [View More]

13 24

Off topic: 'strike a balance' - second language English
by Jonathan Fine Aug. 23, 2018

Aug. 23, 2018

Summary: I look at the phrase 'strike a balance' in different languages, and rewrite some wikipedia text on accessibility. I found in https://en.wikipedia.org/wiki/Jargon#Accessibility_issues === There is a balance to be struck, as excessive removal of technical terminology from a document leads to an equally undesirable outcome—dumbing down. === Aside: Found while writing https://mail.python.org/pipermail/python-ideas/2018-August/052819.html. I wondered how the phrase 'strike a balance' … [View More]would translate into other languages (which is an accessibility issue). Using google translate I did round-tripping and other loops. (Simply curiosity driven, no agenda.) === en: strike a balance fr: trouver un équilibre en: find a balance de: finde ein Gleichgewicht arabic: العثور على التوازن en: Find balance fi: Etsi tasapaino en: Find the balance de: Finde das Gleichgewicht === en: strike a balance al: të krijojë një ekuilibër en: create a balance basque: oreka sortu en: create balance === >From this I found that 'balance' was the key to the phrase. And that the verb could variously be 'strike', 'find' or 'create'. There may be other verbs. The work 'strike' by itself often means 'stoppage' or 'industrial action'. Is this a hazard? So what's good, when English is the reader's second (or third) language? Surely, here, it's best not to use the word 'strike'. (In English 'strike out' means 'remove', not 'find'.) To try this out, let's rewrite: === There is a balance to be struck, as excessive removal of technical terminology from a document leads to an equally undesirable outcome—dumbing down. == How about === There is a balance to be found [or made] ... === Or we could use 'balance' as a verb (rather than as a noun). === It can be hard to balance removal of technical terminology against retaining essential meaning. === Or even not use the word 'balance' === Harmony between removal of technical terminology and retaining essential meaning can be hard. === Which is the best way to write the sentence, for a second-language English speaker? English is my first (and by far best) language. So I lack the experience, to make a good judgement. However, as an English speaker, I prefer the last > Harmony between removal of technical terminology and retaining essential meaning can be hard. By the way: the rewriting has changed the meaning. For this, how about > Removal of technical terminology may also remove essential meaning. -- Jonathan [View Less]

15 24

Jump to function as an an alternative to call function
by Jacob Solinsky Aug. 21, 2018

Aug. 21, 2018

-Jumping to a function as opposed to calling a function When a function is jumped to, it inherits the variables in the caller’s local namespace and is free to modify them or add new local variables, unlike a normal function call, wherein the caller’s namespace is inaccesible. At present, the only way I know of to accomplish this is to bundle all variables in the caller method’s namespace as properties of the method’s instance and have the callee method modify those properties. Though it is … [View More]easier to read code written this way, it resulted in a great deal of redundancy in the code I was writing. The task I was attempting to accomplish was approximately this: class Verb: def __init__(self, parameters): self.parameters = parameters def conjugate(self): #Using the parameters, chose an appropriate list of morphemes to prefix and suffix to the verb self.morphemelist = morphemelist for morpheme in self.morphemelist: morpheme.mutate() returnstring = ‘' for morpheme in self.morphemelist: returnstring = returnstring + morpheme.form returnstring = returnstring class Morpheme: def __init__(self, verb, form, precedingmorpheme, succeedingmorpheme): self.verb = verb self.form = form self.precedingmorpheme = precedingmorpheme self.succeedingmorpheme = succeedingmorpheme def mutate(self): #Using the verb’s parameters and the type and form of the preceding and succeeding morpheme, mutate this morpheme’s form so that #a correct verb form is produced self.form = newform class Ban(Morpheme): def __init__(self, verb, form): super().__init__(verb, ‘ban’) def mutate(self): #This morpheme has mutation logic unique to itself but with many similarities to the default morpheme’s mutation logic self.form = newform Each subclass of Morpheme has its own slightly different mutate method. Some subclasses of Morpheme needed to access and manipulate a great deal of information about the verb and their surroundings, while other subclasses’ mutate methods differed little from the default. Most of the variables manipulated by the mutate method are not used by any other methods of class Morpheme and thus should not be made properties of the attached instance. What I wish I were able to do is write many little methods that modify the same set of local variables and compose them together into a complete mutate method. This would be effectively equivalent to having the “jump to function” calls be replaced with the text in the function’s body, like a C language macrosubstitution, but without using anything like preprocessor directives. Let foo(a, b) be the syntax to call foo(a, b), and foo(a, b)% be the syntax to jump to foo(a, b) def foo(a): return a + c def bar(a, c): return foo(a) def bazz(a, c): return foo(a)% c = 5 call = bar(1, 3) jump = bazz(1, 3) After execution, call in the above code would be 6 and jump in the above code would be 4. [View Less]

14 40

Asynchronous friendly iterables
by Simon De Greve Aug. 21, 2018

Aug. 21, 2018

Hello everyone, I'm quite new working with asyncio and thus maybe missing some things about it, but wouldn't it be quite easier to have some iterables to support async for loops "natively", since asyncio is now part of the Stdlib? I've tried to work with asyncio while using discord.py, and has some struggle with an "async for" loop on a dictionary, so I had to implement a new dict subclass that would just reimplement items(), keys() and values() functions. I think that it would be a cool … [View More]

5 7

Documentation of locals()
by Jonathan Fine Aug. 20, 2018

Aug. 20, 2018

Summary: There's prior art in bug.python.org relating to off-thread topic discussion of locals(). Suggest work on closing open documentation issues relating to locals(). In the thread === Jump to function as an an alternative to call function https://mail.python.org/pipermail/python-ideas/2018-August/052761.html === there was post === https://mail.python.org/pipermail/python-ideas/2018-August/052807.html I wonder why locals doesn't return a Mapping Proxy, or other read-only mapping object? ===… [View More]

1 0

Redefining method
by Jamesie Pic Aug. 20, 2018

Aug. 20, 2018

Hi all, Not sure if this subject has been brought up already, but why can't we redefine methods as such for example: c = MyClass o = c() def c.foo(cls): ... def o.bar(self): ... Thanks in advance for sharing some of your insight -- ∞

8 9

Re: [Python-ideas] Jump to function as an an alternative to call function
by Jacob Solinsky Aug. 17, 2018

Aug. 17, 2018

So when getx is executed inside a let form, if it tries to read/write the value of X it interacts with the X entry in the let form's symbol table before moving to find X in the global environment, right? That is similar to what I was trying to accomplish in python, but with the local symbol table of the calling function rather than a let form. I think the way a "jump to" function rather than a "call function" would be implemented would be by removing the prologue and epilogue of the function's … [View More]

4 5

Re: [Python-ideas] Jump to function as an an alternative to call function
by Jacob Solinsky Aug. 16, 2018

Aug. 16, 2018

On Thu, 16 Aug 2018, 12:44 Jacob Solinsky, <jacobsolinsky(a)gmail.com> wrote: > I wanted the called, jumped to function to change state in the caller. > From what I tried to do, passing locals() cannot accomplish this. I have > made it happen in other languages though. > In the R language, one can do this > > foo = function(){ > localenv = environment() > eval(bar, environment = localenv) > > The above code captures the environment of the … [View More]calling function and > executes the called function as if the calling function's local environment > was the global environment. bar doesn't have to be a function, it can be > any valid R expression captured with the expr() function, and everything in > R is an expression thus allowing for the full usage code blocks. Outside of > lisp-like languages this feat seems to usually be impossible though. > > > On Thu, 16 Aug 2018, 12:34 Jonathan Fine, <jfine2358(a)gmail.com> wrote: > >> Hi Chris >> >> Steve and you wrote: >> >> >> there >> >> are times where I have really wanted to access the caller's >> environment, >> >> not the environment where my function was defined. >> >> > what am I missing? can't you get that by passing locals() in to a >> function? >> >> I think this will fail when values are changed. According to >> https://docs.python.org/3/library/functions.html#locals >> > The contents of this dictionary should not be modified; changes may not >> affect the values of local and free variables used by the interpreter. >> >> I suspect that Jacob, the original poster, wants to change state, and >> also to read changed state. Jacob, could you confirm or correct, that >> you want the called function to be able to change state in caller (or >> perhaps even further up the stack). >> >> To finish, here's an interactive example of changing the value of >> locals(). >> >> >>> def f(a=1): loc = locals(); yield loc; yield a >> >>> it = f() >> >>> ctx = next(it) >> >>> ctx >> {'a': 1} ## This surprised me. >> >>> >> >>> >> >>> def f(a=1): loc = locals(); yield locals(); yield a >> >>> it = f() >> >>> ctx = next(it) >> >>> ctx >> {'a': 1, 'loc': {...}} >> >>> ctx['a'] = 3 >> >>> ctx['loc']['a'] = 5 >> >>> next(it) ## Is it 1 or 3 or 5? >> 1 ## The value of 'a' hasn't changed. >> >> -- >> Jonathan >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > [View Less]

2 1

Python docs page: In what ways is None special
by Jonathan Fine Aug. 16, 2018

Aug. 16, 2018

This arises out of PEP 505 - None-aware operators. I thought, a page on how None is special would be nice. I've not found such a page on the web. We do have === https://docs.python.org/3/library/constants.html None The sole value of the type NoneType. None is frequently used to represent the absence of a value, as when default arguments are not passed to a function. Assignments to None are illegal and raise a SyntaxError. === So decided to start writing such a page, perhaps to be added to the … [View More]

14 31

Hi!
by Marko Ristin-Kaufmann Aug. 16, 2018

Aug. 16, 2018

Hi, Please let me briefly introduce myself. I'm Marko and live in Zurich, Switzerland. I learned Python and Django during an intership in 2007 and followed its development ever since. I did research in computer vision (at ETH Zurich) and now I work at Parquery AG (Zurich, Switzerland) where we implement most of our infra-structure in Python 3. Cheers, Marko

1 0

Re: [Python-ideas] Python docs: page: In what ways in None special
by Rhodri James Aug. 15, 2018

Aug. 15, 2018

(Sorry to break threading on this. In a fit of idiocy I deleted the original email before realising I wanted to reply.) First off, thanks for doing this Jonathan. Documentation is usually a thankless task, so we ought to start by thanking you! I have a few comments on both content and style, good and bad. Starting with style, I notice you tend to write in short sentences. This is generally a good thing, but sometimes you make them too short. This gets really visible when you start … [View More]sentences with "And" or "But"; my old English teacher would have menaced you with a ruler for such bad grammar. You can start a sentence with a conjunction, but it adds extra emphasis that you don't usually want. The start of the "Why None?" section is a good example. It reads like: "Sometimes a value is required. But (pay careful attention to this, it's important and there will be a quiz later) we're not able to provide one." I exaggerate for effect, of course, but it would read more easily as: "Sometimes a value is required but we're not able to provide one." On 'None is a constant': Erm. I think you've got carried away with simplifying this and gone down a blind alley. None is a literal, and like any other literal can't be rebound. Either this entire section is irrelevant or you meant to explain that there is only one "NoneType" object. Constant is a bit of a loaded term in Python, and I think you've fallen foul of it here. On 'None is the default return value': I really dislike the term "falls off the bottom". I can't think of anything similarly short and expressive, but I grimaced when I saw it. The 'list gotcha' is a good example of how None is used as the default return value and why programmers should pay attention, but it doesn't deserve it's own subsection. It's a direct consequence of None being the return value. On 'None can signal failure': Here's where I think None stops being special in the document. None can signal failure. So False, 0, an empty string, a negative number or pretty much anything else. If you want to have a section on how None is used, that great, but having this in the section "How None is special" is just wrong. On 'None as a placeholder default': Ditto. It's common to use None as a placeholder for a mutable type; explaining that common gotcha here would be good. Epigraphs: If you're going to quote Sherman, you need to expand on the uniqueness of None. Not doing that just makes it look irrelevant. It's not irrelevant, it's a tigger [1] [1] Misquoting Michael Flanders from the introduction to "The Hippopotamus Song" on "At The Drop Of A Hat". -- Rhodri James *-* Kynesim Ltd [View Less]

4 5

Conduct on python-ideas
by Brett Cannon Aug. 14, 2018

Aug. 14, 2018

I'm starting a new thread on this to make sure it receives appropriate visibility for everyone on this list. *There are ramifications for those involved in this whole situation* as outlined at the end of this email. To review, this mailing list operates under the PSF Community Code of Conduct: https://www.python.org/psf/codeofconduct/ . That means everyone is expected to be open, considerate, and respectful. Titus and I have operated this list such that people who don't follow the CoC get a … [View More]warning, then a temporary ban of a month or two, then a year-long ban, then a permanent ban (and we reserve the right to skip any of these steps as we see fit based on how bad the infraction was). Since the traffic here can get a bit heavy and thus we don't read every email, we rely on people reporting things as appropriate to python-ideas-owner@ if things have gotten out of hand. And boy did things get out of hand on the "Syntactic sugar to declare partial functions" thread. Here is what I saw play out (bolding is mine): ---------- [Abe] Words like 'partial', 'curry', 'lambda', and 'closure' are fine for text books, published papers, and technical discussion, *but I think they would (or do in the case of 'lambda') harm Python*. [Steven] *That's an extreme overreaction*. Do you mean to imply that there are people who looked at Python, loved the language, but decided to use something else because they didn't like the choice of the keyword "lambda"? If not, in what way is Python harmed? Would it be faster if the keyword was "function", or use less memory, or more expressive? Remember that to millions of programmers in the world, "function" is just as much an obscure foreign piece of jargon they have to memorise as "lambda" is to English-speakers. [Abe] Extreme? I thought it was a rather benign opinion.* I'm not exactly frothing at the mouth here. It's not like I'm declaring holy war on Python for using the word 'lambda'.* I just think it was a mistake (and thatdeath should come to all non-believers). [Steve] You've said that the choice of keyword, "lambda", has caused harm. Given the chance to clarify what you meant, *you stood by your comment that thechoice of keyword "lambda" has done real, significant, non-trivial harm* to Python (the language, or the community). Presumably you fear the same thing will happen again if we choose "partial" (otherwise, why raise the issue?). [Abe] Notice:* I never said "real, significant, non-trivial harm" anywhere* in this entire discussion. I never said anything close to that. *Stop jamming bullshit in my mouth* to suit your narrative that I'm "extremely overreacting". It's not cute. ---------- The way I read this going down is Abe thought something "would harm Python". "Harm" is a strong word to use, so Steven called it an "extreme overreaction". Now stating it like a fact is unnecessary, abrasive, and a bit of hyperbole. Abe thought we he thought and that's fine, but it isn't universally considered an overreaction; it was an opinion and Abe correctly stated it as such. Steven could have easily left out the hyperbole and framing like a statement of fact and still made the same point. Abe then reacted with more hyperbole as he disagreed with Steven's statement that he was overreacting. So now we are reacting to hyperbole with hyperbole. Steven then reacted by overstating the strength of Abe's initial intentions. That's a misunderstanding at best, entirely misleading at worst. At that point Abe crossed a line and basically yelled at Steven to "stop jamming bullshit in [his] mouth". So how should have this been handled? First, dropping all hyperbole, not stating opinions as facts, and not being so abrasive would have probably stopped all of this from happening. Had Steven just said "I think that's an overreaction; can you please clarify what harm you think happened?" then he would have gotten the answer he wanted and Abe would not have taken offense. Abe replying with his own hyperbole didn't help. Steven also didn't need to misrepresent what Abe said. Choosing to re-interpret what level of harm Abe meant was not appropriate (unless Steven got a bit sloppy and replied from memory instead of going back and reading the emails he was replying to). And finally, Abe's response was totally uncalled for. I don't care how out of line someone on this list is, you don't react like that. You come to Titus and me and we will handle it. If you need to, step back for an hour or day before replying if that's what it takes to not react in a rude manner. I shouldn't be having to explain to adults on how to communicate among strangers of different cultures, but here we are. I did an entire PyCon US keynote on why we need to treat open source as a series of kindnesses and react as such: https://youtu.be/tzFWz5fiVKU?t=49m29s . If we don't treat everything as a kindness then open source simply doesn't work and people end up walking way from open source and the Python community. So,* ramifications from all of this* ... Steven, stop stating your opinion as fact and being needlessly abrasive. Your abrasiveness in responses has pushed multiple threads to breaking points like this before and I know you have the skills to not do that if you chose to because I've seen it here and on other mailing lists. Constructive conversations never need anything abrasive in them in order to get your intentions across. Consider this a warning to scale back the tone in response to be more respectful. Abe, you're receiving a warning about how you eventually reacted. In the future please just step away from the keyboard until you can react appropriately. The way you responded to Steven is not acceptable as it isn't respectful to others on this list and there shouldn't be "vigilante responses"; if you feel someone has violated the CoC then tell us admins and we will handle it. -Brett & Titus [View Less]

3 2

Python Community Code of Conduct
by Jonathan Fine Aug. 13, 2018

Aug. 13, 2018

https://www.python.org/psf/codeofconduct/ Python Community Code of Conduct The Python community is made up of members from around the globe with a diverse set of skills, personalities, and experiences. It is through these differences that our community experiences great successes and continued growth. When you're working with members of the community, we encourage you to follow these guidelines which help steer our interactions and strive to keep Python a positive, successful, and growing … [View More]

1 0

Consider adding an iterable option to dataclass
by Neil Girdhar Aug. 13, 2018

Aug. 13, 2018

It would be nice if dataclasses (https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass) had an option to make them a sequence. This would make dataclass(frozen=True, order=True, sequence=True) an optionally-typed version of namedtuple. It would almost totally supplant it except that namedtuples have a smaller memory footprint. sequence would simply inherit from collections.abc.Sequence and implement the two methods __len__ and __getitme__. Best, Neil

4 4

Is this PEP-able? "with" statement inside genexps / list comprehensions
by Rudy Matela Aug. 11, 2018

Aug. 11, 2018

Hello, Do you think it would be nice to allow with statements inside genexps or list comprehensions? The functions __enter__ and __exit__ would be automatically called as iterables are traversed. I am thinking of drafting a PEP about this. Examples: This g = (f.read() for fn in filenames with open(fn) as f) would be equivalent to the following use of a generator function: def __gen(): for fn in filenames: with open(fn) as f: yield f.read() g = __gen() This list = [f.… [View More]

7 6

Python certification
by Patrick Morris Aug. 10, 2018

Aug. 10, 2018

Hi all I know this might be a contentious question but ... Does anyone know of any PSF approved training or certifications for Python developers? There is this 10 year old page https://wiki.python.org/psf/Certification%20Proposal but I dont see any evidence that this was ever progressed past the discussion of it. Assuming there is no such thing as PSF approval, are there any certifications / training course that people would recommend, or would recognise for someone seeking employment … [View More]

6 8

Revisiting dedicated overloadable boolean operators
by Todd Aug. 9, 2018

Aug. 9, 2018

Coming back to the previous discussion about a new set of overloadable boolean operators [1], I have an idea for overloadable boolean operators that I think might work. The idea would be to define four new operators that take two inputs and return a boolean result based on them. This behavior can be overridden in appropriate dunder methods. These operators would have similar precedence to existing logical operators. The operators would be: bNOT - boolean "not" bAND - boolean "and" bOR - … [View More]boolean "or" bXOR - boolean "xor" With corresponding dunder methods: __bNOT__ and _rbNOT__ (or __r_bNOT__) __bAND__ and _rbAND__ (or __r_bAND__) __bOR__ and _rbOR__ (or __r_bOR__) __bXOR__ and _rbXOR__ (or __r_bXOR__) The basic idea is that the "b" is short for "boolean", and we change the rest of the operator to upercase to avoid confusions with the existing operators. I think these operators would be preferably to the proposals so far (see [1] again) for a few reasons: 1. They are not easy to mistake with existing operators. They are clearly not similar to the existing bitwise operators like & or |, and although they are clearly related to the "not", "and", and "or" I think they are distinct enough that it should not be easy to confuse the two or accidentally use one in place of the other. 2. They are related to the operations they carry out, which is also an advantage over the existing bitwise operators. 3. The corresponding dunder methods (such as __bAND__ and _rbAND__) are obvious and not easily confused with anything else. 4. The unusual capitalization means they are not likely to be used much in existing Python code. It doesn't fall under any standard capitalization scheme I am aware of. 5. At least for english the capitalization means they are not easy to confuse with existing words. For example Band is a word, but it is not likely to be capitalized as bAND. As to why this is useful, the overall problem is that the current logical operators, like and, or, and not, cannot be overloaded, which means projects like numpy and SQLAlchemy instead have to (ab)use bitwise operators to define their own boolean operations (for example elementwise "and" in numpy arrays). This has a variety of problems, such not having appropriate precedence leading to precedence errors being common, and the simple fact that this precludes them from using the bitwise operators for bitwise operations. There was a proposal to allow overloading boolean operators in Pep-335 [2], but that PEP was rejected for a variety of very good reasons. I think none of those reasons (besides the conversation fizzling out) apply to my proposal. So the alternative proposal that has been floating around is to instead define new operators specifically for this. Although there seemed to be some support for this in principle, the actually operators so far have not met with much enthusiasm. So far the main operators proposed so far seem to be: 1. Double bitwise operators, such as && and ||. These have the disadvantage of looking like they should be a type of bitwise operator. 2. the existing operators, with some non-letter character at the front and back, like ".and.". These have the advantage that they are currently not valid syntax in most cases, but I think are too similar to existing logical operators, to easy to confuse, and it is not immediately obvious in what way they should differ from existing operators. They also mean different things in other languages. So I think my proposal addresses the main issues raised with existing proposals, but has the downside that it requires new keywords. Thoughts? [1] https://mail.python.org/pipermail/python-ideas/2015-November/037207.html [2] https://www.python.org/dev/peps/pep-0335/ [View Less]

14 32

File format for automatic and manual tests
by Victor Porton Aug. 9, 2018

Aug. 9, 2018

This is an idea of a new PEP. I propose to create a portable file format which will list command line options to run Python scripts with dual purpose: 1. for automatic tests executing scripts from this file (and optionally checking their stdout against specified values). 2. for running the scripts in a Python debugger. The scripts may be either real command line applications written in Python or scripts created specifically for testing. Currently I use PyCharm to debug my command line … [View More]

9 10

Make "yield" inside a with statement a SyntaxError
by Ken Hilton Aug. 8, 2018

Aug. 8, 2018

This mostly springs off of a comment I saw in some thread. The point of a with statement is that it ensures that some resource will be disposed of, yes? For example, this: with open(filename) as f: contents = f.read() is better than this: contents = open(filename).read() because the former definitely closes the file while the latter relies on garbage collection? The point of a yield expression is to suspend execution. This is nice for efficient looping because instead of … [View More]having to hold all results in memory, each result can be consumed immediately, yes? Therefore this: def five_to_one(): for i in range(4): yield 5 - i is better than this: def five_to_one(): result = [] for i in range(4): result.append(5 - i) return result because the former suspends execution of "five_to_one" while the latter holds all five results in memory? Now, let's take a look at the following scenario: def read_multiple(*filenames): for filename in filenames: with open(filename) as f: yield f.read() Can you spot the problem? The "with open(filename)" statement is supposed to ensure that the file object is disposed of properly. However, the "yield f.read()" statement suspends execution within the with block, so if this happened: for contents in read_multiple('chunk1', 'chunk2', 'chunk3'): if contents == 'hello': break and the contents of "chunk2" were "hello" then the loop would exit, and "chunk2" would never be closed! Yielding inside a with block, therefore, doesn't make sense and can only lead to obscure bugs. The proper way to define the "read_multiple" function would be like so: def read_multiple(*filenames): for filename in filenames: with open(filename) as f: contents = f.read() yield contents Save the contents in a variable somewhere, then yield the variable, instead of suspending execution within a context manager. I believe all possible cases where one would yield inside a context manager can be covered by saving anything required from the context manager and then yielding the results outside. Therefore, I propose making a "yield" inside a with block become a SyntaxError. This means the first "read_multiple" definition I presented will become illegal and fail *at compile-time*. However, it is still legal to define a generator inside a with block: def pass_file_chars(oldfunc): with open('secretfile') as f: contents = f.read() @functools.wraps def newfunc(*args, **kwargs): for char in contents: yield oldfunc(char, *args, **kwargs) return newfunc This is probably a bad example, but I hope it still explains why it should be legal to define generators in context managers - as long as the with block serves its purpose correctly, everything else should still work normally. For those concerned about backwards compatibility: I believe that those who attempt to yield inside a context manager will already discover that results are undefined when doing so; this will simply make it more obvious that suspending execution in a with block is not meant to happen, and convert undefined behavior into a straight-up SyntaxError. What are your thoughts? Sharing, Ken Hilton; [View Less]

15 20

Syntactic sugar to declare partial functions
by Fabrizio Messina Aug. 6, 2018

Aug. 6, 2018

Hello, I would like to propose a new method to create a partial function. At the moment we have to load the *partial* function from the *functool* library, and apply it to an existing function, e.g. from functools import partial def add(x: int, y: int) -> int: return x + y add_2 = partial(add, 2) While partial expose the mechanism excellently its instantiation method is, at times, not very friendly, I would like to propose a syntactic sugar to create partial functions, in … [View More]

6 7

With expressions
by Ken Hilton Aug. 4, 2018

Aug. 4, 2018

Hi, I don't know if someone has already suggested this before, but here goes: With expressions allow using the enter/exit semantics of the with statement inside an expression context. Examples: contents = f.read() with open('file') as f #the most obvious one multiplecontents = [f.read() with open(name) as f for name in names] #reading multiple files I don't know if it's worth making the "as NAME" part of the with mandatory in an expression - is this a valid use case? data = … [View More]

13 26

Re: [Python-ideas] With expressions
by Jeroen Demeyer Aug. 4, 2018

Aug. 4, 2018

On 2018-08-04 09:43, Steven D'Aprano wrote: > If you want to > propose a general exception suppressing mechanism (aside from the > existing try...except statement) then propose it as an independent PEP. This was already tried in PEP 463 (which IMHO is a pity that it wasn't accepted because it seems really useful).

1 0

Re: [Python-ideas] PEP 505: None-aware operators: operators ?= and ?? and OR
by Jonathan Fine Aug. 4, 2018

Aug. 4, 2018

Hi > There is a formatted version of this PEP at > https://www.python.org/dev/peps/pep-0505/ I've taken a look at this, and have some comments on the first two examples drawn from standard library code. (And a very grateful +10 for writing a script to find such examples.) I've started a subthread, just to discuss the ?= and ?? operators. And something newish, that I call OR. FIRST EXAMPLE The first example is --- >From bisect.py: def insort_right(a, x, lo=0, hi=None): # ... … [View More]

17 28

Change repr of collections.OrderedDict to be more dict-like
by Miro Hrončok Aug. 2, 2018

Aug. 2, 2018

Hi, now when dicts are sorted, can we change the repr of collections.OrderedDict to look like dict? I mean from: OrderedDict([('a', '1'), ('b', '2')]) To: OrderedDict({'a': '1', 'b': '2'}) I assume the current list-of-tuples repr is there so copy pasting the repr to the interpreter results in the same OrderedDict. When dicts preserve insertion order, this now works with dict. I consider the dict-based repr much easier to read. Thoughts? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

13 23

PEP 505: None-aware operators
by Steve Dower Aug. 1, 2018

Aug. 1, 2018

Possibly this is exactly the wrong time to propose the next big syntax change, since we currently have nobody to declare on it, but since we're likely to argue for a while anyway it probably can't hurt (and maybe this will become the test PEP for whoever takes the reins?). FWIW, Guido had previously indicated that he was generally favourable towards most of this proposal, provided we could figure out coherent semantics. Last time we tried, that didn't happen, so this time I've made the … [View More]semantics much more precise, have implemented and verified them, and made much stronger statements about why we are proposing these. Additional thanks to Mark Haase for writing most of the PEP. All the fair and balanced parts are his - all the overly strong opinions are mine. Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we went through this - if you're unhappy with "None" being treated as a special kind of value, I recommend reading those before you start repeating them. There is a formatted version of this PEP at https://www.python.org/dev/peps/pep-0505/ My current implementation is at https://github.com/zooba/cpython/tree/pep-505 (though I'm considering removing some of the new opcodes I added and just generating more complex code - in any case, let's get hung up on the proposal rather than the implementation :) ) Let the discussions begin! --- PEP: 505 Title: None-aware operators Version: $Revision$ Last-Modified: $Date$ Author: Mark E. Haase <mehaase(a)gmail.com>, Steve Dower <steve.dower(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Sep-2015 Python-Version: 3.8 Abstract ======== Several modern programming languages have so-called "``null``-coalescing" or "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, and PHP (starting in version 7). These operators provide syntactic sugar for common patterns involving null references. * The "``null``-coalescing" operator is a binary operator that returns its left operand if it is not ``null``. Otherwise it returns its right operand. * The "``null``-aware member access" operator accesses an instance member only if that instance is non-``null``. Otherwise it returns ``null``. (This is also called a "safe navigation" operator.) * The "``null``-aware index access" operator accesses an element of a collection only if that collection is non-``null``. Otherwise it returns ``null``. (This is another type of "safe navigation" operator.) This PEP proposes three ``None``-aware operators for Python, based on the definitions and other language's implementations of those above. Specifically: * The "``None`` coalescing`` binary operator ``??`` returns the left hand side if it evaluates to a value that is not ``None``, or else it evaluates and returns the right hand side. A coalescing ``??=`` augmented assignment operator is included. * The "``None``-aware attribute access" operator ``?.`` evaluates the complete expression if the left hand side evaluates to a value that is not ``None`` * The "``None``-aware indexing" operator ``?[]`` evaluates the complete expression if the left hand site evaluates to a value that is not ``None`` Syntax and Semantics ==================== Specialness of ``None`` ----------------------- The ``None`` object denotes the lack of a value. For the purposes of these operators, the lack of a value indicates that the remainder of the expression also lacks a value and should not be evaluated. A rejected proposal was to treat any value that evaluates to false in a Boolean context as not having a value. However, the purpose of these operators is to propagate the "lack of value" state, rather that the "false" state. Some argue that this makes ``None`` special. We contend that ``None`` is already special, and that using it as both the test and the result of these operators does not change the existing semantics in any way. See the `Rejected Ideas`_ section for discussion on the rejected approaches. Grammar changes --------------- The following rules of the Python grammar are updated to read:: augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=' | '**=' | '//=' | '??=') power: coalesce ['**' factor] coalesce: atom_expr ['??' factor] atom_expr: ['await'] atom trailer* trailer: ('(' [arglist] ')' | '[' subscriptlist ']' | '?[' subscriptlist ']' | '.' NAME | '?.' NAME) Inserting the ``coalesce`` rule in this location ensures that expressions resulting in ``None`` are natuarlly coalesced before they are used in operations that would typically raise ``TypeError``. Like ``and`` and ``or`` the right-hand expression is not evaluated until the left-hand side is determined to be ``None``. For example:: a, b = None, None def c(): return None def ex(): raise Exception() (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3)) (a * b ?? c // d) == a * (b ?? c) // d (a ?? True and b ?? False) == (a ?? True) and (b ?? False) (c() ?? c() ?? True) == True (True ?? ex()) == True (c ?? ex)() == c() Augmented coalescing assignment only rebinds the name if its current value is ``None``. If the target name already has a value, the right-hand side is not evaluated. For example:: a = None b = '' c = 0 a ??= 'value' b ??= undefined_name c ??= shutil.rmtree('/') # don't try this at home, kids assert a == 'value' assert b == '' assert c == '0' and any(os.scandir('/')) Adding new trailers for the other ``None``-aware operators ensures that they may be used in all valid locations for the existing equivalent operators, including as part of an assignment target (more details below). As the existing evaluation rules are not directly embedded in the grammar, we specify the required changes here. Assume that the ``atom`` is always successfully evaluated. Each ``trailer`` is then evaluated from left to right, applying its own parameter (either its arguments, subscripts or attribute name) to produce the value for the next ``trailer``. Finally, if present, ``await`` is applied. For example, ``await a.b(c).d[e]`` is currently parsed as ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated:: _v = a _v = _v.b _v = _v(c) _v = _v.d _v = _v[e] await _v When a ``None``-aware operator is present, the left-to-right evaluation may be short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated:: _v = a if _v is not None: _v = _v.b _v = _v(c) _v = _v.d if _v is not None: _v = _v[e] await _v .. note:: ``await`` will almost certainly fail in this context, as it would in the case where code attempts ``await None``. We are not proposing to add a ``None``-aware ``await`` keyword here, and merely include it in this example for completeness of the specification, since the ``atom_expr`` grammar rule includes the keyword. If it were in its own rule, we would have never mentioned it. Parenthesised expressions are handled by the ``atom`` rule (not shown above), which will implicitly terminate the short-circuiting behaviour of the above transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as:: # a?.b _v = a if _v is not None: _v = _v.b # ... ?? c if _v is None: _v = c # (...).d?.e _v = _v.d if _v is not None: _v = _v.e When used as an assignment target, the ``None``-aware operations may only be used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will raise ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is permitted, though unlikely to be useful unless combined with a coalescing operation:: (a?.b ?? d).c = 1 Examples ======== This section presents some examples of common ``None`` patterns and shows what conversion to use ``None``-aware operators may look like. Standard Library ---------------- Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 standard library discovered up to 678 code snippets that could be replaced with use of one of the ``None``-aware operators:: $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py <snip> Total None-coalescing `if` blocks: 449 Total [possible] None-coalescing `or`: 120 Total None-coalescing ternaries: 27 Total Safe navigation `and`: 13 Total Safe navigation `if` blocks: 61 Total Safe navigation ternaries: 8 Some of these are shown below as examples before and after converting to use the new operators. From ``bisect.py``:: def insort_right(a, x, lo=0, hi=None): # ... if hi is None: hi = len(a) # ... After updating to use the ``??=`` augmented assignment statement:: def insort_right(a, x, lo=0, hi=None): # ... hi ??= len(a) # ... From ``calendar.py``:: encoding = options.encoding if encoding is None: encoding = sys.getdefaultencoding() optdict = dict(encoding=encoding, css=options.css) After updating to use the ``??`` operator:: optdict = dict(encoding=encoding ?? sys.getdefaultencoding(), css=options.css) From ``dis.py``:: def _get_const_info(const_index, const_list): argval = const_index if const_list is not None: argval = const_list[const_index] return argval, repr(argval) After updating to use the ``?[]`` and ``??`` operators:: def _get_const_info(const_index, const_list): argval = const_list?[const_index] ?? const_index return argval, repr(argval) From ``inspect.py``:: for base in object.__bases__: for name in getattr(base, "__abstractmethods__", ()): value = getattr(object, name, None) if getattr(value, "__isabstractmethod__", False): return True After updating to use the ``?.`` operator (and deliberately not converting to use ``any()``):: for base in object.__bases__: for name in base?.__abstractmethods__ ?? (): if object?.name?.__isabstractmethod__: return True From ``os.py``:: if entry.is_dir(): dirs.append(name) if entries is not None: entries.append(entry) else: nondirs.append(name) After updating to use the ``?.`` operator:: if entry.is_dir(): dirs.append(name) entries?.append(entry) else: nondirs.append(name) jsonify ------- This example is from a Python web crawler that uses the Flask framework as its front-end. This function retrieves information about a web site from a SQL database and formats it as JSON to send to an HTTP client:: class SiteView(FlaskView): @route('/site/<id_>', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) return jsonify( first_seen=site.first_seen.isoformat() if site.first_seen is not None else None, id=site.id, is_active=site.is_active, last_seen=site.last_seen.isoformat() if site.last_seen is not None else None, url=site.url.rstrip('/') ) Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the database, and they are also allowed to be ``null`` in the JSON response. JSON does not have a native way to represent a ``datetime``, so the server's contract states that any non-``null`` date is represented as an ISO-8601 string. Without knowing the exact semantics of the ``first_seen`` and ``last_seen`` attributes, it is impossible to know whether the attribute can be safely or performantly accessed multiple times. One way to fix this code is to replace each conditional expression with an explicit value assignment and a full ``if``/``else`` block:: class SiteView(FlaskView): @route('/site/<id_>', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) first_seen_dt = site.first_seen if first_seen_dt is None: first_seen = None else: first_seen = first_seen_dt.isoformat() last_seen_dt = site.last_seen if last_seen_dt is None: last_seen = None else: last_seen = last_seen_dt.isoformat() return jsonify( first_seen=first_seen, id=site.id, is_active=site.is_active, last_seen=last_seen, url=site.url.rstrip('/') ) This adds ten lines of code and four new code paths to the function, dramatically increasing the apparent complexity. Rewriting using the ``None``-aware attribute operator results in shorter code with more clear intent:: class SiteView(FlaskView): @route('/site/<id_>', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) return jsonify( first_seen=site.first_seen?.isoformat(), id=site.id, is_active=site.is_active, last_seen=site.last_seen?.isoformat(), url=site.url.rstrip('/') ) Grab ---- The next example is from a Python scraping library called `Grab <https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf72… ab/upload.py>`_:: class BaseUploadObject(object): def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) if ctype is None: return 'application/octet-stream' else: return ctype class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content if filename is None: self.filename = self.get_random_filename() else: self.filename = filename if content_type is None: self.content_type = self.find_content_type(self.filename) else: self.content_type = content_type class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path if filename is None: self.filename = os.path.split(path)[1] else: self.filename = filename if content_type is None: self.content_type = self.find_content_type(self.filename) else: self.content_type = content_type This example contains several good examples of needing to provide default values. Rewriting to use conditional expressions reduces the overall lines of code, but does not necessarily improve readability:: class BaseUploadObject(object): def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) return 'application/octet-stream' if ctype is None else ctype class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content self.filename = (self.get_random_filename() if filename is None else filename) self.content_type = (self.find_content_type(self.filename) if content_type is None else content_type) class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path self.filename = (os.path.split(path)[1] if filename is None else filename) self.content_type = (self.find_content_type(self.filename) if content_type is None else content_type) The first ternary expression is tidy, but it reverses the intuitive order of the operands: it should return ``ctype`` if it has a value and use the string literal as fallback. The other ternary expressions are unintuitive and so long that they must be wrapped. The overall readability is worsened, not improved. Rewriting using the ``None`` coalescing operator:: class BaseUploadObject(object): def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) return ctype ?? 'application/octet-stream' class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content self.filename = filename ?? self.get_random_filename() self.content_type = content_type ?? self.find_content_type(self.filename) class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path self.filename = filename ?? os.path.split(path)[1] self.content_type = content_type ?? self.find_content_type(self.filename) This syntax has an intuitive ordering of the operands. In ``find_content_type``, for example, the preferred value ``ctype`` appears before the fallback value. The terseness of the syntax also makes for fewer lines of code and less code to visually parse, and reading from left-to-right and top-to-bottom more accurately follows the execution flow. Rejected Ideas ============== The first three ideas in this section are oft-proposed alternatives to treating ``None`` as special. For further background on why these are rejected, see their treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and `PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated discussions. No-Value Protocol ----------------- The operators could be generalised to user-defined types by defining a protocol to indicate when a value represents "no value". Such a protocol may be a dunder method ``__has_value__(self)` that returns ``True`` if the value should be treated as having a value, and ``False`` if the value should be treated as no value. With this generalization, ``object`` would implement a dunder method equivalent to this:: def __has_value__(self): return True ``NoneType`` would implement a dunder method equivalent to this:: def __has_value__(self): return False In the specification section, all uses of ``x is None`` would be replaced with ``not x.__has_value__()``. This generalization would allow for domain-specific "no-value" objects to be coalesced just like ``None``. For example the ``pyasn1`` package has a type called ``Null`` that represents an ASN.1 ``null``:: >>> from pyasn1.type import univ >>> univ.Null() ?? univ.Integer(123) Integer(123) Similarly, values such as ``math.nan`` and ``NotImplemented`` could be treated as representing no value. However, the "no-value" nature of these values is domain-specific, which means they *should* be treated as a value by the language. For example, ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting ``math.nan?.imag`` to return ``math.nan`` would be incorrect. As ``None`` is already defined by the language as being the value that represents "no value", and the current specification would not preclude switching to a protocol in the future (though changes to built-in objects would not be compatible), this idea is rejected for now. Boolean-aware operators ----------------------- This suggestion is fundamentally the same as adding a no-value protocol, and so the discussion above also applies. Similar behavior to the ``??`` operator can be achieved with an ``or`` expression, however ``or`` checks whether its left operand is false-y and not specifically ``None``. This approach is attractive, as it requires fewer changes to the language, but ultimately does not solve the underlying problem correctly. Assuming the check is for truthiness rather than ``None``, there is no longer a need for the ``??`` operator. However, applying this check to the ``?.`` and ``?[]`` operators prevents perfectly valid operations applying Consider the following example, where ``get_log_list()`` may return either a list containing current log messages (potentially empty), or ``None`` if logging is not enabled:: lst = get_log_list() lst?.append('A log message') If ``?.`` is checking for true values rather than specifically ``None`` and the log has not been initialized with any items, no item will ever be appended. This violates the obvious intent of the code, which is to append an item. The ``append`` method is available on an empty list, as are all other list methods, and there is no reason to assume that these members should not be used because the list is presently empty. Further, there is no sensible result to use in place of the expression. A normal ``lst.append`` returns ``None``, but under this idea ``lst?.append`` may result in either ``[]`` or ``None``, depending on the value of ``lst``. As with the examples in the previous section, this makes no sense. As checking for truthiness rather than ``None`` results in apparently valid expressions no longer executing as intended, this idea is rejected. Exception-aware operators ------------------------- Arguably, the reason to short-circuit an expression when ``None`` is encountered is to avoid the ``AttributeError`` or ``TypeError`` that would be raised under normal circumstances. As an alternative to testing for ``None``, the ``?.`` and ``?[]`` operators could instead handle ``AttributeError`` and ``TypeError`` raised by the operation and skip the remainder of the expression. This produces a transformation for ``a?.b.c?.d.e`` similar to this:: _v = a try: _v = _v.b except AttributeError: pass else: _v = _v.c try: _v = _v.d except AttributeError: pass else: _v = _v.e One open question is which value should be returned as the expression when an exception is handled. The above example simply leaves the partial result, but this is not helpful for replacing with a default value. An alternative would be to force the result to ``None``, which then raises the question as to why ``None`` is special enough to be the result but not special enough to be the test. Secondly, this approach masks errors within code executed implicitly as part of the expression. For ``?.``, any ``AttributeError`` within a property or ``__getattr__`` implementation would be hidden, and similarly for ``?[]`` and ``__getitem__`` implementations. Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed. Existing conventions for handling these kinds of errors in the form of the ``getattr`` builtin and the ``.get(key, default)`` method pattern established by ``dict`` show that it is already possible to explicitly use this behaviour. As this approach would hide errors in code, it is rejected. ``None``-aware Function Call ---------------------------- The ``None``-aware syntax applies to attribute and index access, so it seems natural to ask if it should also apply to function invocation syntax. It might be written as ``foo?()``, where ``foo`` is only called if it is not None. This has been deferred on the basis of the proposed operators being intended to aid traversal of partially populated hierarchical data structures, *not* for traversal of arbitrary class hierarchies. This is reflected in the fact that none of the other mainstream languages that already offer this syntax have found it worthwhile to support a similar syntax for optional function invocations. A workaround similar to that used by C# would be to write ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the expression will not be evaluated. (The C# equivalent uses ``?.Invoke()`` on its callable type.) ``?`` Unary Postfix Operator ---------------------------- To generalize the ``None``-aware behavior and limit the number of new operators introduced, a unary, postfix operator spelled ``?`` was suggested. The idea is that ``?`` might return a special object that could would override dunder methods that return ``self``. For example, ``foo?`` would evaluate to ``foo`` if it is not ``None``, otherwise it would evaluate to an instance of ``NoneQuestion``:: class NoneQuestion(): def __call__(self, *args, **kwargs): return self def __getattr__(self, name): return self def __getitem__(self, key): return self With this new operator and new type, an expression like ``foo?.bar[baz]`` evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty generalization, but it's difficult to use in practice since most existing code won't know what ``NoneQuestion`` is. Going back to one of the motivating examples above, consider the following:: >>> import json >>> created = None >>> json.dumps({'created': created?.isoformat()})`` The JSON serializer does not know how to serialize ``NoneQuestion``, nor will any other API. This proposal actually requires *lots of specialized logic* throughout the standard library and any third party library. At the same time, the ``?`` operator may also be **too general**, in the sense that it can be combined with any other operator. What should the following expressions mean?:: >>> x? + 1 >>> x? -= 1 >>> x? == 1 >>> ~x? This degree of generalization is not useful. The operators actually proposed herein are intentionally limited to a few operators that are expected to make it easier to write common code patterns. Built-in ``maybe`` ------------------ Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_ that encapsulates the idea of an optional value without relying on any special keyword (e.g. ``null``) or any special instance (e.g. ``None``). In Haskell, the purpose of ``Maybe`` is to avoid separate handling of "something" and nothing". A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_ provides a rough approximation. The documentation shows the following example:: >>> maybe('VALUE').lower() 'value' >>> maybe(None).invalid().method().or_else('unknown') 'unknown' The function ``maybe()`` returns either a ``Something`` instance or a ``Nothing`` instance. Similar to the unary postfix operator described in the previous section, ``Nothing`` overrides dunder methods in order to allow chaining on a missing value. Note that ``or_else()`` is eventually required to retrieve the underlying value from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short circuit any evaluation. Although ``pymaybe`` has some strengths and may be useful in its own right, it also demonstrates why a pure Python implementation of coalescing is not nearly as powerful as support built into the language. The idea of adding a builtin ``maybe`` type to enable this scenario is rejected. Just use a conditional expression --------------------------------- Another common way to initialize default values is to use the ternary operator. Here is an excerpt from the popular `Requests package <https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e2323… a8149f5/requests/models.py#L212>`_:: data = [] if data is None else data files = [] if files is None else files headers = {} if headers is None else headers params = {} if params is None else params hooks = {} if hooks is None else hooks This particular formulation has the undesirable effect of putting the operands in an unintuitive order: the brain thinks, "use ``data`` if possible and use ``[]`` as a fallback," but the code puts the fallback *before* the preferred value. The author of this package could have written it like this instead:: data = data if data is not None else [] files = files if files is not None else [] headers = headers if headers is not None else {} params = params if params is not None else {} hooks = hooks if hooks is not None else {} This ordering of the operands is more intuitive, but it requires 4 extra characters (for "not "). It also highlights the repetition of identifiers: ``data if data``, ``files if files``, etc. When written using the ``None`` coalescing operator, the sample reads:: data = data ?? [] files = files ?? [] headers = headers ?? {} params = params ?? {} hooks = hooks ?? {} References ========== .. [1] C# Reference: Operators (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx) .. [2] A Tour of the Dart Language: Operators (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators) .. [3] Associated scripts (https://github.com/python/peps/tree/master/pep-0505/) Copyright ========= This document has been placed in the public domain. [View Less]

44 273