Python-ideas July 2020

python-ideas@python.org

93 participants
54 discussions

by Guido van Rossum

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

14 hours, 35 minutes

Fwd: Re: Experimenting with dict performance, and an immutable dict

by Marco Sulla

On Wed, 22 Jul 2020 at 15:35, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > The deltas also look small for a micro-benchmark. I certainly don't > think this is a sufficient reason to add a new datatype to Python. > I think some of the optimizations can be experimented with dict itself. Deltas seem to be small, because I divide them by the number of timeit runs. I think that a 30-45% gain in constructor speed is not something small. The benchmarks in question have the Name that starts with "constructor". PS: sorry for double mail, I sent this message to only you by mistake. On Wed, 22 Jul 2020 at 18:26, Guido van Rossum <guido(a)python.org> wrote: > Did you study PEP 416 (frozendict) and PEP 603 (frozenmap)? > Yes. About frozenmap, at the very start I added to the benchmarks also immutables.Map, but I removed it when I realized that it was slower than frozendict in every bench. Maybe this is because immutables.Map is a C extension and not a builtin type. About PEP 416, yes, I tried to follow it. Indeed hash is calculated using the strategy described in the PEP. I also take a look to frozenset and tuple code. Frankly I must admit that the rejection of PEP 416 was a good idea. Indeed I started to implement an immutable dict only for a symmetry reason, and because I am somewhat fascinated by functional programming and immutables, without a rational reason. I suppose I would have given up and admitted that a frozendict is quite useless in Python, if I did not see that the constructor speed can be faster. I'm not sure 100% of the result. This is why I'm suggesting to try to implement the speed optimizations to the constructor of dict first: 1. the optimizations could be not safe. Indeed recently I got a segfault that I have to investigate 2. maybe dict can be really optimized further. After that, the difference between dict and frozendict performance could be minimal That said, maybe there are four good use cases for a frozendict: 1. they could be used for __annotations__ of function objects, and similar cases 2. they could be used to implement "immutable" modules or classes 3. frozendict can be easily cached, like tuples. 4. as I said, as an alternative to MappingProxyType, since it's very slow and it's used also in the C code of CPython and in the Python stdlib. This maybe is not useful because: a. the speed of MappingProxyType can be improved b. MappingProxyType can't be replaced because you *really* want and need a proxy

1 week, 1 day

Access (ordered) dict by index; insert slice

by Hans Ginzel

Date: Fri, 26 Jun 2020 18:47:44 +0200 From: Hans Ginzel <hans(a)matfyz.cz> To: Hans Ginzel <hans(a)artax.karlin.mff.cuni.cz> Subject: Access (ordered) dict by index; insert slice Hello, thank you for making dict ordered. Is it planned to access key,value pair(s) by index? See https://stackoverflow.com/a/44687752/2556118 for example. Both for reading and (re)writing? Is it planned to insert pair(s) on exact index? Or generally to slice? See splice() in Perl, https://perldoc.perl.org/functions/splice.html. Use case: Represent database table metadata (columns). It is useful as to access columns both by name and by index as to insert column on specific position, https://dev.mysql.com/doc/refman/8.0/en/alter-table.html, “ALTER TABLE ADD COLUMN [FIRST |AFTER col]” (consider default order or table storage size optimisation by aligning). Thank you in advance, Hans PS1: Named tuples cannot be used, are immutable. PS2: See https://metacpan.org/pod/perlref#Pseudo-hashes:-Using-an-array-as-a-hash

1 month, 1 week

Compound with .. else statement

by Jimmy Thrasibule

Hi, In Python, there are multiple [compound statements](https://docs.python.org/3/reference/compound_stmts.html) with the `else` keyword. For example: ``` for x in iterable: if x == sentinel: break else: print("Sentinel not found.") ``` or: ``` try: do_something_sensitive() except MyError: print("Oops!") else: print("We're all safe.") ``` In my situation, I would like to mix the `with` statement with `else`. In this case I would like that if no exception is raised within the `with` to run the `else` part. For example: ``` with my_context(): do_something_sensitive() else: print("We're all safe.") ``` Now imagine that in my `try .. except` block I have some heavy setup to do before `do_something_sensitive()` and some heavy cleanup when the exception occurs. I'd like my context manager to do the preparation work, execute the body, and cleanup. Or execute my else block only if there is no exception. Is there already a way to accomplish this in Python or can this be a nice to have? Regards, Jimmy

1 month, 2 weeks

What about having a .get(index, default) method for arrays like we have for dicts?

by Daniel.

When I need to traverse nested dicts, is a common pattern to do somedict.get('foo', {}).get('bar', {}) But there is no such equivalent for arrays, wouldn't be nice if we can follow somedict.get('foo', {}).get('bar', []).get(10) ... ? What I do in this case is surround with try/except IndexError and set some variable to None, or wrap the try/except block in a function. But this is noise in my opinion, I just want to follow the same reasoning that I have with dicts: I'm searching for some path deep nested, and want a default if not found. What do you think? Regards -- “If you're going to try, go all the way. Otherwise, don't even start. ..." Charles Bukowski

1 month, 3 weeks

Aside RE: Re: macaddress or networkaddress module

by Steve Barnes

As an aside I have a perfect example to back up what Paul is saying below. I work for a large corporation where developers are permitted to install python modules on their development machines, (subject to some licence restrictions), but the proxy settings required to get to PyPi vary from user to user, site to site, day to day (sometimes hour to hour) and connection type to connection type, e.g. If you are connected via WiFi you need to use a different proxy to that needed if you are on a wired connection and on a VPN (we have more than one in use) different again. To help with this I have put together a script that will try pip, if it fails will check the current location for pac.pac, get and parse it for proxies, correct a common incompatibility and then try each discovered proxy and if sill none work fall back to a list of common proxies. Once it finds a working setting it advises the user what to use in the current situation. Obviously, since the script is intended to work out how to get to PyPi, I needed to stick 100% to the standard libraries. -----Original Message----- From: Paul Moore <p.f.moore(a)gmail.com> Sent: 06 April 2020 09:11 To: Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp> Cc: Pete Wicken <petewicken(a)gmail.com>; Python-Ideas <python-ideas(a)python.org> Subject: [Python-ideas] Re: macaddress or networkaddress module On Mon, 6 Apr 2020 at 04:14, Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp> wrote: > That's OK for developers on their "own" machines, but I think there > are still a lot of folks who need approval from corporate (though I > haven't heard from Paul Moore on that theme in quite a while, maybe > it's gotten better?) Howdy :-) It's actually got a bit worse, to the point where I don't bother trying to fight the system as much any more, so things have gone quieter because of that :-) But yes, I still believe that there are important use cases for code that only uses the stdlib, and "not being able to get to the internet" is a valid use case that we shouldn't just dismiss. On the other hand, with regard to the comment you were replying to: > Up to around a decade ago, installing third-party libraries > was a huge mess, but nowadays, PyPI works, Python comes with pip > pre-installed, and telling developers they need internet access > when they first start a new project isn’t considered onerous. I think all of that is true. What I think is more important, though, is around things that aren't so much "projects" as "scripts", and that *distributing* such scripts that depend on 3rd party libraries is still problematic. Here, I'm not talking about making full-fledged packages, or py2exe style fully bundled apps, but "here's this monitoring script I wrote, let's ship it across 10 production systems and why don't you try it out on your dev boxes?" There's a *significant* step change in complexity between doing that if the script depends on just the stdlib, versus if it depends on a 3rd party module. And we don't have a really good answer to that yet (zipapps, and tools like shiv that wrap zipapps, are pretty good in practice, but they are not well known or commonly used). Paul _______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5BN5D… Code of Conduct: http://python.org/psf/codeofconduct/

2 months, 1 week

default parameter in fuctions to clean up flow

by Peter Moore

I have had a long standing unanswered question on on stackoverflow: is it possible to pass a function to a default parameter so that you could do in essence things like this. def time_diff(target_time, curr_time= lambda : datetime.now() ): return curr_time - target_time this would be an syntactical improvement over this style where you have if statement to initialize a missing parameter. def time_diff(target_time, curr_time=None): if curr_time == None: curr_time = datetime.datetime.now() return curr_time - target_time

2 months, 1 week

PEP 472 -- Support for indexing with keyword arguments

by Andras Tantos

Hello! I'm not sure I'm addressing the right audience here, so please direct me to the appropriate channel if that's the case... My name is Andras Tantos and I'm working on a Python library desscribing HW designs. I came across this problem of __getitem__ and co. not supporting kwargs. Apparently this extension was proposed and rejected as PEP 472. Apart from my use-case, which is arguably a corner-case and not worth modifying the language for, I believe there are two important use-cases that are worth considering with the latest improvements in the language: 1. With the recent type-hint support, the feature could be made way more descriptive if this PEP got implemented. For example, instead of doing the following: def func(in: Dict[str, int]) one could write: def func(in: Dict[key=str, value=int]) 2. It would also make 'generic classes' much cleaner to implement, similar to the way type-hints look. Consider the following code: class_Generic(object): Specializations = [] @classmethod def__getitem__(cls, *args): name = f"Generic_{len(cls.Specializations)}" Specialized = type(name, (cls,), {"specials": tuple(args)}) cls.Specializations.append(Specialized) returnSpecialized def__init__(self, value= None): self.value = value def__str__(self): ifhasattr(self, "specials"): return(f"[{type(self)}- "+ ",".join(str(special) forspecial inself.specials) + f"] - {self.value}") else: return(f"[{type(self)}- GENERIC"+ f"] - {self.value}") Generic = _Generic() #g = Generic() - fails because of no specialization is given s1 = Generic[12]() s2 = Generic[42]("Hi!") print(s1) print(s2) Running this simple example results in: python3 -i python_test.py [<class '__main__.Generic_0'> - 12] - None [<class '__main__.Generic_1'> - 42] - Hi! You can see how the specialized parameters got passed as well as the ones to '__init__'. Obviously, in real code the idea would be to filter generic parameters and set up 'Specialized' with the right set of methods and arguments. Now, without kwargs support for __getitem__, it's impossible to pass named arguments to the specialization list, which greatly limits the usability of this notation. I don't know how convincing these arguments and use-cases are for you, but could you advise me about how to start the 'ball rolling' to drum-up support for re-activating this PEP? Thanks again, Andras Tantos

2 months, 1 week

Function suggestion: itertools.one()

by Noam Yorav-Raphael

Hello, There's a simple function that I use many times, and I think may be a good fit to be added to itertools. A function that gets an iterator, and if it has exactly one element returns it, and otherwise raises an exception. This is very useful for cases where I do some sort of query that I expect to get exactly one result, and I want an exception to be raised if I'm wrong. For example: jack = one(p for p in people if p.id == '1234') sqlalchemy already has such a function for queries: https://docs.sqlalchemy.org/en/13/orm/query.html#sqlalchemy.orm.query.Query… more-itertools has this exact function: https://more-itertools.readthedocs.io/en/latest/api.html#more_itertools.one Here is a simple implementation: def one(iterable): it = iter(iterable) try: first = next(it) except StopIteration: raise ValueError("Iterator is empty") try: second = next(it) except StopIteration: return first else: raise ValueError("Iterator has more than one item") I brought this up on python-dev, but it should be discussed here. Here is the discussion: https://mail.python.org/archives/list/python-dev@python.org/thread/D52MPKLI… Brett Cannon said that this idea has been brought up at least twice before. I found: https://mail.python.org/archives/list/python-ideas@python.org/thread/FTJ6JR… https://mail.python.org/archives/list/python-ideas@python.org/thread/REYDJF… The first thread hasn't reached any operative conclusion. The second thread was very long, and seemed to focus mostly on another function, first(), that doesn't check if there is more than one item. Joao S. O Bueno said that it passed "general approval". I think that perhaps a new discussion, focused just on one (no pun intended) function in the itertools module may reach a conclusion. It was suggested that instead of an additional function, one can use iterator unpacking: jack, = (p for p in people if p.id == '1234') or [jack] = (p for p in people if p.id == '1234') I still think that having a one() function would be useful, since: 1. I think it spells the intention more clearly. It is not symbols that you need to understand their meaning in order to understand that I expect the iterable to have exactly one item, it's spelled in code. 2. The exception would be easier to understand, since errors in tuple unpacking usually mean something else. 3. The one() function allows you to use the result inside an expression without assigning it to a variable. Therefore, I think it encourages writing better code. It's very easy to write: print([p for p in people if p.id == '1234][0]) (which has the problem of not verifying the assumption that there's no more than one result), and I find it easier to replace _[0] with one(_) than to be required to name a new variable, and instead of having an operation on the iterable, change the way I'm assigning to it. WDYT? Cheers, Noam

2 months, 2 weeks

Idea: Extend "for ... else ..." to allow "for ... if break ..." else

by Jonathan Fine

There's been a discussion of the "for ... else ..." construction in Python. Here's a suggestion as to how we might improve matters, and perhaps usefully extend the language. I hope it might benefit some, without harming anyone else. Aside: I've not read the whole of the previous discussion, and it may be that some or all of what I say has already been expressed there. So apologies for any missing credits to others. The errors, I'm responsible for. Here's what happens at present. for i in items: LOOP_BODY else: print('exit via StopIteration') I find the following clearer (where 'if break' is a new language 'compound keyword'). for i in items: LOOP_BODY if break: pass else: print('exit via StopIteration') This would now allow us to write for i in items: LOOP_BODY if break: print('exit via break') else: print('exit via StopIteration') Sometimes a loop has multiple break commands, which have different purposes. So a further extension might be for i in items: LOOP_BODY if break left: print('exit via break left') elif break right: print('exit via break right') elif break: print('exit via some other break') else: print('exit via StopIteration') To allow this to work, we extend the language to allow commands such as break left break right as well as the usual break Here the identifiers 'left' and 'right' look like variable names, but in fact they are labels. The compiler can, and I think should, produce a hard-coded jump from command break right to the code block under elif break right: This is the basic idea, keeping the syntax close to Python as it now is (but of course adding to the semantics). Some programmers, and I think I'm one, might want a clearer distinction between for i in items: LOOP_BODY if break: do_something and for i in items: LOOP_BODY if broke: do_something For this reason I tentatively suggest instead for i in items: LOOP_BODY case: break left: print('exit via break left') break right: print('exit via break right') break: print('exit via some other break') else: print('exit via StopIteration') Some people have asked a built-in means to detect, at the end of the iteration, whether theitems was empty to begin with. The syntax could easily provide this, for example via case: zero: print('no items to process') However, if we adopt this perhaps we'd also like to retain the simpler for ... if break ... else ... construct. I hope this helps. -- Jonathan

2 months, 2 weeks

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas July 2020