Mailman 3 October 2021 - Python-ideas

More efficient list copying
by Steven D'Aprano 05 Oct '21

05 Oct '21

This half-baked idea is inspired by this thread here: https://mail.python.org/archives/list/python-ideas@python.org/message/5LGWV… which in turn was inspired by this Stackoverflow post: https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… Problem: today, more and more people are using Python to work with Big Data, or at least Biggish Data. For some folks, it is not unusual to have lists with a gigabyte or more or data. So imagine you have a list with a hundred million items, and you need a copy of the entire list. Easy: alist = [...] # 100 million items blist = alist[:] And that ought to be pretty efficient too, making the slice is basically just a copy of a bunch of pointers, plus a bit of overhead. A good implementation should have lightning fast list slicing, close to the speed of memcopy in C. Give or take. But now suppose you want a copy of all the items except the item in position (let's say) 1000. Here's a bad way: blist = alist[:] del blist[1000] That's inefficient because the list has to shift almost a hundred million items over, and I think they have to be moved one at a time. Which is slowish, even in C. Here's another way: blist = alist[:1000] + alist[1001:] That has to make a slice (copy 1000 items), a second slice (copy another many million items), then make a third copy to concatenate them, then garbage collect the two temporary slices. And while you might have enough memory for double the original list (alist and blist) you might not have enough for triple (alist, the two slices, blist). There are lots of other variants we could come up with, but ultimately we're doing lots of extra work that has to be thrown away, and that costs time or memory or both. How about if lists had an in-place method for doing a fast copy of pointers from one list to another? We could do this: blist = [None]*(len(alist)-1) # Preallocate. blist.copyfrom(alist, index=0, start=0, end=1000) blist.copyfrom(alist, index=1000, start=1001) No temporary slices are needed. Here's a pure-Python (and therefore slow) implementation: def copyfrom(dest, source, index=0, start=0, end=None): if end is None: end = len(source) if len(dest) - index < end - start: raise ValueError('destination too small') for i in range(start, end): dest[index+i-start] = source[i] In pure Python, copying each item over one by one will save memory but cost a lot of time. Doing it as a method in C should save both memory and time. The implementation could optimize the cases where the source is a list, and fall back on iteration for other sequence types. Now *personally* I don't work with a lot of Big Data, so this doesn't really scratch any of my personal itches, but I thought I'd throw it out there for others to tell me why it's a dumb idea :-) -- Steve

6 6

Syntax Sugar for __name__ == "__main__" boilerplate?
by Jonathan Crall 05 Oct '21

05 Oct '21

I was curious if / what sort of proposals have been considered for simplifying the pattern: ``` def main(): ... if __name__ == "__main__": main() ``` I imagine this topic must have come up before, so I'd be interested in any relevant history. But unless I'm missing something, it seems like adding some easier alternative to this cumbersome entrypoint syntax would be worth considering. My motivation for writing this suggestion is in an attempt to stop a common anti-pattern, where instead of defining a `main` function (or a function by any other name) an simply calling that by adding the above two lines, a lot of Python users I work with will just start dumping their logic into the global scope of the module. Needless to say, this can have consequences. If there was some default builtin, let's call it `__main__` for now (open to suggestions), that took a function as an argument and conditionally executed it if `__name__ == "__main__"` in the caller's scope, that would allow us to simplify the above boilerplate to a single line with no extra indentation: ``` def main(): ... __main__(main) ``` In addition to being simpler, it would allow users to avoid the trap of adding logic that impacts the global scope. It would also save me some keystrokes, which I'm always grateful for. Furthermore, it could be used as a decorator (and the use-case wouldn't be unreasonable!), and we all know how much new Python users love decorators when they find out about them. ``` @__main__ def main(): ... ``` Maybe having such a builtin would discourage globals and help new users get the use-decorators-everywhere bug out of their system. -- -Dr. Jon Crall (him)

19 38

dict_items.__getitem__?
by Erik Demaine 05 Oct '21

05 Oct '21

Have folks thought about allowing indexing dictionary views as in the following code, where d is a dict object? d.keys()[0] d.keys()[-1] d.values()[0] d.values()[-1] d.items()[0] d.items()[-1] # item that would be returned by d.popitem() I could see value to the last form in particular: you might want to inspect the last item of a dictionary before possibly popping it. I've also often wanted to get an arbitrary item/key from a dictionary, and d.items()[0] seems natural for this. Of course, the universal way to get the first item from an iterable x is item = next(iter(x)) I can't say this is particularly readable, but it is functional and fast. Or sometimes I use this pattern: for item in x: break If you wanted the last item of a dictionary d (the one to be returned from d.popitem()), you could write this beautiful code: last = next(iter(reversed(d.items()))) Given the dictionary order guarantee from Python 3.7, adding indexing (__getitem__) to dict views seems natural. The potential catch is that (I think) it would require linear time to access an item in the middle, because you need to count the dummy elements. But accessing [i] and [-i] should be doable in O(|i|) time. (I've wondered about the possibility of doing binary or interpolation search, but without some stored index signposts, I don't think it's possible.) Python is also full of operations that take linear time to do: list.insert(0, x), list.pop(0), list.index(), etc. But it may be that __getitem__ takes constant time on all built-in data structures, and the apparent symmetry but very different performance between dict()[i] and list()[i] might be confusing. That said, I really just want d[0] and d[-1], which is when these are fast. I found some related discussion in https://mail.python.org/archives/list/python-ideas@python.org/thread/QVTGZD… but not this exact idea. Erik -- Erik Demaine | edemaine(a)mit.edu | http://erikdemaine.org/

3 2

Power Assertions: Is it PEP-able?
by noam＠10ne.org 04 Oct '21

04 Oct '21

Hi all, I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language. Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' ``` Will result in the output: ```python Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up. In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure. Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime. [1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html… [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expr… [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/pow…

8 17

Re: Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet 04 Oct '21

04 Oct '21

Le dim. 3 oct. 2021 à 16:21, <python-ideas-request(a)python.org> a écrit : > Date: Mon, 4 Oct 2021 01:03:34 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-ideas] Re: Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: <20211003140333.GR16229(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Laurent, Hello Steve, > It is not clear to me what you mean by "filter by indices". > > On Sat, Oct 02, 2021 at 10:25:05PM +0200, Laurent Lyaudet wrote: > > > The idea is to filter a list by indices : > [...] > > Since filter() returns an iterator instead of a list, it could do what > > is needed... if the callback had access to the index like the > > Javascript array filter function. > > Do you mean this? > > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… Yes exactly, > You shouldn't assume we are all experts on Javascript :-) > No problem, I'll give more links next time :) > If that is what you want, it is easy to get access to the index. We can > just do: > > filter(function, enumerate(items)) > That's exactly what I proposed. Except for the fact that I incorrectly permuted the two arguments of filter. I quote my second email : Le sam. 2 oct. 2021 à 22:35, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 22:25:05 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> >... > Currently, the following solution is available : > filter(enumerate(my_list), lambda x: x[0] != i) > But it is slightly ugly and unefficient to have two function calls for > such a simple task I think. I feel uncomfortable because it is often the case that I write explicitly something, and I get an answer as if the person answered without reading all my email or reading another version. > If you want something else, I'm afraid you will have to explain in more > detail what you want, sorry. Again for my second email, I would like one of these 3 options. > What would be your prefered way of doing this ? > enumerate(my_list, filter_callback=(lambda x: x != i)) > filter_by_index(my_list, lambda x: x != i) > # à la JS > filter(my_list, lambda _, x: x != i) filter_by_index(lambda x: x != i, my_list) could be defined as follow (I permuted args to have similar order to python's filter()): def filter_by_index(function, my_list): for i, item in enumerate(my_list) if function(i): yield item like filter is just (without the case function=None) filter(function, my_list): for i, item in enumerate(my_list) if function(item): yield item The problem is not that it is hard to code in Python. The problem is that it is a basic building block for an iterators tools library like itertools. enumerate(my_list, filter_callback=(lambda x: x != i)) could be defined in a similar way but there are two variants. It just needs to return one of the two indices of the item: - either the index of the item in the original sequence, - or the index of the item in the filtered sequence (new counter). For my use case, I do not need the index so I have no argument in favor of either one of the two indices. > # à la JS > filter(my_list, lambda _, x: x != i) You provided the link from MDN and an helper function for that. The lambda takes 2 or 3 parameters: item, index, array/list/sequence and returns true for item that must be output. I see nothing more to say. Best regards, Laurent Lyaudet

1 0

Re: Python-ideas Digest, Vol 179, Issue 25
by Laurent Lyaudet 04 Oct '21

04 Oct '21

Le dim. 3 oct. 2021 à 16:21, <python-ideas-request(a)python.org> a écrit : > Date: Mon, 4 Oct 2021 01:03:34 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-ideas] Re: Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: <20211003140333.GR16229(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Laurent, Hello Steve, > It is not clear to me what you mean by "filter by indices". > > On Sat, Oct 02, 2021 at 10:25:05PM +0200, Laurent Lyaudet wrote: > > > The idea is to filter a list by indices : > [...] > > Since filter() returns an iterator instead of a list, it could do what > > is needed... if the callback had access to the index like the > > Javascript array filter function. > > Do you mean this? > > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… Yes exactly, > You shouldn't assume we are all experts on Javascript :-) > No problem, I'll give more links next time :) > If that is what you want, it is easy to get access to the index. We can > just do: > > filter(function, enumerate(items)) > That's exactly what I proposed. Except for the fact that I incorrectly permuted the two arguments of filter. I quote my second email : Le sam. 2 oct. 2021 à 22:35, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 22:25:05 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> >... > Currently, the following solution is available : > filter(enumerate(my_list), lambda x: x[0] != i) > But it is slightly ugly and unefficient to have two function calls for > such a simple task I think. I feel uncomfortable because it is often the case that I write explicitly something, and I get an answer as if the person answered without reading all my email or reading another version. > If you want something else, I'm afraid you will have to explain in more > detail what you want, sorry. Again for my second email, I would like one of these 3 options. > What would be your prefered way of doing this ? > enumerate(my_list, filter_callback=(lambda x: x != i)) > filter_by_index(my_list, lambda x: x != i) > # à la JS > filter(my_list, lambda _, x: x != i) filter_by_index(lambda x: x != i, my_list) could be defined as follow (I permuted args to have similar order to python's filter()): def filter_by_index(function, my_list): for i, item in enumerate(my_list) if function(i): yield item like filter is just (without the case function=None) filter(function, my_list): for i, item in enumerate(my_list) if function(item): yield item The problem is not that it is hard to code in Python. The problem is that it is a basic building block for an iterators tools library like itertools. enumerate(my_list, filter_callback=(lambda x: x != i)) could be defined in a similar way but there are two variants. It just needs to return one of the two indices of the item: - either the index of the item in the original sequence, - or the index of the item in the filtered sequence (new counter). For my use case, I do not need the index so I have no argument in favor of either one of the two indices. > # à la JS > filter(my_list, lambda _, x: x != i) You provided the link from MDN and an helper function for that. The lambda takes 2 or 3 parameters: item, index, array/list/sequence and returns true for item that must be output. I see nothing more to say. Best regards, Laurent Lyaudet

1 0

Shorthand syntax for lambda functions that have a single parameter
by Dominik Vilsmeier 04 Oct '21

04 Oct '21

Lambda functions that have a single parameter are a common thing, e.g. for "key" functions: `sorted(items, key=lambda x: x['key'])`. For these cases however, the rather long word "lambda" together with the repetition of the parameter name, results in more overhead than actual content (the `x['key']`) and thus makes it more difficult to read. This is also a difficulty for `map` and `filter` for example where there's lots of visual overhead when using a lambda function and hence it's difficult to read: `filter(lambda x: x > 0, items)` or `map(lambda x: f'{x:.3f}', items)`. Hence the proposal is to add a new syntax via the new token `?`. For the examples above: * `sorted(items, key=?['key'])` * `filter(? > 0, items)` * `map(f'{?:.3f}', items)` The rules are simple: whenever the token `?` is encountered as part of an expression (at a position where a name/identifier would be legal), the expression is replaced by a lambda function with a single parameter which has that expression as a return value, where any instances of `?` are replaced by the name of that single parameter. For example: * `?['key']` translates to `lambda x: x['key']` * `? > 0` translates to `lambda x: x > 0` * `f'{?:.3f}'` translates to `lambda x: f'{x:.3f}'` * `?*?` translates to `lambda x: x*x` With the difference that the replacement function would use an unnamed parameter, i.e. not shadowing any names from outer scopes. So `? * x` would use `x` from enclosing scopes, not as the lambda parameter. Regarding operator precedence, the rules would be similar as for lambda functions, i.e. it will include as much in the expression as it would for a lambda function. To find more example use cases, I used `grep -rnE 'lambda [_a-zA-Z][_a-zA-Z0-9]*:' Lib/` on the CPython standard library to find single parameter lambda functions and there are various results coming up (although not all are applicable, e.g. because they return a constant value). I'll include a few examples below, but the actual list is much longer: ``` modes[char] = max(items, key=lambda x: x[1]) modes[char] = max(items, key=?[1]) # proposed new syntax obj = unwrap(obj, stop=(lambda f: hasattr(f, "__signature__"))) obj = unwrap(obj, stop=hasattr(?, "__signature__")) s = re.sub(r"-[a-z]\b", lambda m: m.group().upper(), s) s = re.sub(r"-[a-z]\b", ?.group().upper(), s) members.sort(key=lambda t: (t[1], t[0])) members.sort(key=(?[1], ?[0])) ``` Of course, there is no requirement to use `?` as the token, it could be any other character that's currently not used by Python, i.e. `$` or `!` would be possible too.

13 22

Re: Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet 03 Oct '21

03 Oct '21

Le sam. 2 oct. 2021 à 15:54, <python-ideas-request(a)python.org> a écrit : > Date: Sat, 2 Oct 2021 12:45:33 +0200 > From: Laurent Lyaudet <laurent.lyaudet(a)gmail.com> > Subject: [Python-ideas] Feature request enumerate_with_rest or > enumerate with skip or filter callback > To: python-ideas(a)python.org > Message-ID: > <CAB1LBmsXQJbeLuyK6tG8aP=kb36DAR==VcajO-zHuHJGnfvAJA(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hello, > > This is a very simple feature request that does not break anything but > I don't know if you may find it interesting. > It would be nice to have a function or method of list objects that does this : > - First idea : > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, my_list[:i] + my_list[i + 1:] > > It could be called easily with: > > for i, item, rest in enumerate_with_rest(my_list): > # do something > > or > for i, item, rest in my_list.enumerate_with_rest(): > # do something > > I am not the only one who had the same need : > https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… > > It would be nice to have an optimized C function for this. > However, it may be less interesting than this : > - Second idea > enumerate_with_rest above has quadratic complexity. > It is probably true that most processes that use it will also have > quadratic complexity. > However, it would be better to return an iterator instead of a list > for the rest, > it would use less space. > For this a param skip to enumerate would do the job > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, enumerate(my_list, skip=i) > There could be variants of this idea like : > - enumerate(my_list, skip=i) > - enumerate(my_list, skip=[i]) > - enumerate(my_list, filter_callback=(lambda x: x != i)) > > Please let me know what you think of it :) > > Thanks for your time, best regards, > Laurent Lyaudet Hello, Regarding the last suggestion. - enumerate(my_list, filter_callback=(lambda x: x != i)) The idea is to filter a list by indices : a quick search for that just yields : https://stackoverflow.com/questions/11847491/python-filtering-lists-by-indi… which is not really helpful Since filter() returns an iterator instead of a list, it could do what is needed... if the callback had access to the index like the Javascript array filter function. > def enumerate_with_rest(my_list): > for i, item in enumerate(my_list): > yield i, item, my_list[:i] + my_list[i + 1:] could also be : def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, filter_by_index(my_list, lambda x: x != i) I think my feature request for enumerate_with_rest is maybe too specific. However, I think there should definitely be in itertools something to filter indices like JS permits. I have not found a function to do this there: https://docs.python.org/3/library/functions.html https://docs.python.org/3/library/itertools.html What would be your prefered way of doing this ? enumerate(my_list, filter_callback=(lambda x: x != i)) filter_by_index(my_list, lambda x: x != i) # à la JS filter(my_list, lambda _, x: x != i) Currently, the following solution is available : filter(enumerate(my_list), lambda x: x[0] != i) But it is slightly ugly and unefficient to have two function calls for such a simple task I think. I would appreciate any feedback; even if none of my ideas are accepted, I may learn something :) Best regards, Laurent Lyaudet

2 1

Feature request enumerate_with_rest or enumerate with skip or filter callback
by Laurent Lyaudet 03 Oct '21

03 Oct '21

Hello, This is a very simple feature request that does not break anything but I don't know if you may find it interesting. It would be nice to have a function or method of list objects that does this : - First idea : def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, my_list[:i] + my_list[i + 1:] It could be called easily with: for i, item, rest in enumerate_with_rest(my_list): # do something or for i, item, rest in my_list.enumerate_with_rest(): # do something I am not the only one who had the same need : https://stackoverflow.com/questions/56966429/getting-pairs-of-one-item-and-… It would be nice to have an optimized C function for this. However, it may be less interesting than this : - Second idea enumerate_with_rest above has quadratic complexity. It is probably true that most processes that use it will also have quadratic complexity. However, it would be better to return an iterator instead of a list for the rest, it would use less space. For this a param skip to enumerate would do the job def enumerate_with_rest(my_list): for i, item in enumerate(my_list): yield i, item, enumerate(my_list, skip=i) There could be variants of this idea like : - enumerate(my_list, skip=i) - enumerate(my_list, skip=[i]) - enumerate(my_list, filter_callback=(lambda x: x != i)) Please let me know what you think of it :) Thanks for your time, best regards, Laurent Lyaudet

3 3

Better exception handling hygiene
by Soni L. 03 Oct '21

03 Oct '21

So uh, this is a hardly at all fleshed out idea, but one thing we really don't like about python is having to do stuff like this so as to not swallow exceptions: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() except ExceptionWeCareAbout as exc: raise RuntimeError from exc code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout It'd be nice if there was a way to... make this easier to deal with. Perhaps something where, something like this: def a_potentially_recursive_function(some, args) with ExceptionWeCareAbout: """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ some.user_code() code_we_assume_is_safe() if args.something and some_condition: raise ExceptionWeCareAbout becomes: def a_potentially_recursive_function(some, args): """ Does stuff and things. Raises ExceptionWeCareAbout under so and so conditions. """ try: some.user_code() code_we_assume_is_safe() if args.something and some_condition: let_exception_through = True raise ExceptionWeCareAbout except ExceptionWeCareAbout as exc: if let_exception_through: raise else: raise RuntimeError from exc (or equivalent) and something like: def foo() with Bar: try: baz() except Bar: raise becomes: def foo(): try: try: baz() except Bar: allow_exception = True raise except Bar as exc: if allow_exception: raise else: raise RuntimeError from exc (thus allowing explicit exception propagation) Additionally, note that it would only apply to the raise itself - something like `raise may_raise_the_same_error()` would desugar as if: exception = may_raise_the_same_error() allow_exception = True raise exception Obviously this doesn't solve exception handling, doesn't require the caller to catch the exceptions, etc etc. It does, however, encourage better exception hygiene. Chances are something like this would significantly reduce the amount of swallowed exceptions, if it gets widely adopted. We know something like this would've saved us a lot of trouble, so we're sharing the idea in the hopes it can, in the future, help others.

5 28