Mailman 3 February 2018 - Python-ideas

(no subject)
by guido 19 Feb '18

19 Feb '18

1 0

Re: [Python-ideas] Temporary variables in comprehensions
by Robert Vanden Eynde 18 Feb '18

18 Feb '18

Hello, talking about this syntax : [y+2 for x in range(5) let y = x+1] *Previous talks* I've had almost exactly the same idea in June 2017, see subject "variable assignment in functional context here: https://mail.python.org/ pipermail/python-ideas/2017-June/subject.html. Currently this can easily be done by iterating over a list of size 1: [y+2 for x in range(5) for y in [x+1]] (Other ways exist, see section "current possibilities" below). This comparison would answer a lot of questions like "can we write [x+2 for x in range(5) for x in [x+1]], so [x+2 for x in range(5) let x = x+1]" the new variable would indeed shadow the old one. In June 2017 I introduced the "let" syntax using the existing "for" keyword, using the "=" instead of a "in" like this: [y+2 for x in range(5) for y = x+1] The only difference I introduced was that it would be logical to accept the syntax in any expression: x = 5 print(y+2 for y = x + 1) or with the "let" keyword: x = 5 print(y+2 let y = x + 1) *Previous talk: Pep* In the June conversation one conclusion was that someone wrote a pep ( https://www.python.org/dev/peps/pep-3150/) in 2010 that's still pending (not exactly the same syntax but the same idea, he used a "given:" syntax) that looked like that : print(y+2 given: y=x+1) *Previous talk: GitHub implementation on Cython* Another conclusion was that someone on GitHub implemented a "where" statement in Cython (url: https://github.com/thektulu/cpython/commit/ 9e669d63d292a639eb6ba2ecea3ed2c0c23f2636) where one could write : print(y+2 where y = x+1) So in a list comprehension: [y+2 where y = x + 1 for x in range(5)] As the author thektulu said "just compile and have fun". *New syntax in list comprehension* However, we probably would like to be able to use this "where" after the for: [y+2 for x in range(5) where y = x+1] This would allow the new variable to be used in further "for" and "if" statement : [y+z for x in range(5) where y = x+1 for z in range(y+1)] *Choice of syntax* Initially I thought re-using the "for" keyword would be a good idea for backward comptability (a variable named "where" in old code wouldn't be a problem), however some people pointed out that the python Grammar wouldn't be LL1, when reading the "for" token it wouldn't be able to choose directly if the rest would be a "for in" or "for =" so actually introducing a dedicated keyword is probably better, like this : [y+2 for x in range(5) where y = x+1] print(y+2 where y = x+1) The "where" keyword is very readable in my opinion, very close to English sentences we use. Sometimes we introduce a new variable before using it, generally using "let" or after using it using "where". For example "let y = x+1; print(y+2)". Or "print(y+2 where y = x+1)". The first syntax is chosen in the "let in" syntax in haskell : print(let y = x+2 in y+2) Or chained: print(let x = 2 in let y = x+1 in y+2) But Haskell user would probably line break for clarity : print(let x = 2 in let y = x+1 in y+2) A postfix notation using "where" would probably be less verbose in my opinion. Another example of "postfix notation" python has is with the "a if condition else b" so adding a new one wouldn't be surprising. Furthermore, the postfix notation is preferred in the context of "presenting the result first, then the implementation" (context discussed already in the 2010 pep), the "presenting the result first" is also a goal of the list comprehension, indeed one does write [x+3 for x in range(5)] and not [for x in range(5): x+3], the latter would be more "imperative programming" style, and would be translated to a normal loop. The problem of chaining without parenthesis, how to remove the parenthesis in the following statement ? print((y+2 where y = x+1) where x = 2) We have two options : print(y+2 where x = 2 where y = x+1) print(y+2 where y = x+1 where x = 2) The first option would be probably closer to the way multiple "for" are linked in a list comprehension: [y+2 for x in range(5) for y in [x+1]] But the second option would be more "present result first" and more close to the parenthesized version, the user would create new variables as they go, "I want to compute y+2 but hey, what is y ? It's x+1 ! But what is x ? It's 5 !)). However, keeping the same order as in "multiple for in list comprehension" is better, so I'd choose first option. In the implementation on GitHub of thektulu the parenthesis are mandatory. Another syntax issue would probably surprise some users, the following statement, parenthesized : [(y+2 where y = x+1) for x in range(5)] Would have two totally legal ways to be done: [y+2 where y = x+1 for x in range(5)] [y+2 for x in range(5) where y = x+1] The first one is a consequence of the "where" keyword usable in an expression and the second one is a consequence of using it in a list comprehension. However I think it doesn't break the "there is only one obvious way to do it", because depending on the case, the "y" variable would be a consequence of the iteration or a consequence of the computation. *Goals of the new syntax* Personally, I find it very useful to do an assignment in such context, I use list comprehension for example to generate a big json with multiple for and if. I don't have here a list of big real world example thar would be simplified using this syntax but people interested could search arguments here. People could argue only the "list comprehension" case would be useful and not the "any expression" case: [y+2 for x in range(5) where y = x+1] would be accepted but not : print(y+2 where y=x+1) Because the latter could be written: y = x + 1 print(y+2) However, the idea of having an isolated scope can be a good idea. *Current possibilities* Currently we have multiple options : [y+2 for x in range(5) for y in [x+1]] [y+2 for (x+1 for x in range(5))] [(lambda y:y+2)(y=x+1) for x in range(5)] The first one works well but it's not obvious we just want to assign a new variable, especially when the expression is long, or multiple, or both: [y+z for x in range(5) for y,z in [((x + 1) * 2, x ** 2 - 5)]] The second one makes it impossible to reuse the "x" variable and the y = x+1 relation is not obvious. The third example is what a functional programmer would think but is really too much complex for a beginner and very verbose. *Proposed syntax : Conclusion* As I said, I like the "where" syntax with the "where" keyword. [y+2 for x in range(5) where y = x+1] Also usable in any expression : print(y+2 where y = x+1) *Conclusion* Here is all the talk/work/argument I've already found about this syntax. Apparently it's been a while (2010) since such an idea was thought but I think having a new pep listing all the pros and cons would be a good idea. So that we can measure how much the community would want this concept to be introduced, and if it's refused, the community would have a document where the "cons" are clearly written. Robert Vanden Eynde

2 1

Give ipaddresses an __index__ method
by Steven D'Aprano 15 Feb '18

15 Feb '18

This idea is inspired by Eric Osborne's post "Extending __format__ method in ipaddress", but I wanted to avoid derailing that thread. I notice what seems to be an inconsistency in the ipaddress objects: py> v4 = ipaddress.IPv4Address('1.2.3.4') py> bin(v4) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'IPv4Address' object cannot be interpreted as an integer But that's surely not right: we just need to explicitly do so: py> bin(int(v4)) '0b1000000100000001100000100' IP addresses are, in a strong sense, integers: either 32 or 128 bits. And they can be explicitly converted losslessly to and from integers: py> v4 == ipaddress.IPv4Address(int(v4)) True Is there a good reason not to give them an __index__ method so that bin(), oct() and hex() will work directly? py> class X(ipaddress.IPv4Address): ... def __index__(self): ... return int(self) ... py> a = X('1.2.3.4') py> bin(a) '0b1000000100000001100000100' I acknowledge one potentially undesirable side-effect: this would allow using IP addresses as indexes into sequences: py> 'abcdef'[X('0.0.0.2')] 'c' but while it's weird to do this, I don't think it's logically wrong. -- Steve

4 6

Extending __format__ method in ipaddress
by Eric Osborne 15 Feb '18

15 Feb '18

Folks- The ipaddress library returns an IP address object which can represent itself in a number of ways: In [1]: import ipaddress In [2]: v4 = ipaddress.IPv4Address('1.2.3.4') In [3]: print(v4) 1.2.3.4 In [4]: v4 Out[4]: IPv4Address('1.2.3.4') In [6]: v4.packed Out[6]: b'\x01\x02\x03\x04' In [9]: str(v4) Out[9]: '1.2.3.4' In [10]: int(v4) Out[10]: 16909060 In [13]: bin(int(v4)) Out[13]: '0b1000000100000001100000100' In [14]: hex(int(v4)) Out[14]: '0x1020304' In [15]: oct(int(v4)) Out[15]: '0o100401404' There are IPv6 objects as well: In [6]: v6 = ipaddress.IPv6Address('2001:0db8:85a3:0000:0000:8a2e:0370:7334') In [7]: int(v6) Out[7]: 42540766452641154071740215577757643572 and what I'm proposing will work for both address families. In either case, bin/hex/oct don't work on them directly, but on the integer representation. This is a little annoying but not such a big deal. What is a big deal (at least to me) is that the binary representation isn't zero-padded. This makes it harder to compare two IP addresses by eye to see what the differences are, i.e.: In [16]: a = ipaddress.IPv4Address('0.2.3.4') In [30]: bin(int(a)) Out[30]: '0b100000001100000100' In [31]: bin(int(v4)) Out[31]: '0b1000000100000001100000100' It would be nice if there was a way to have an IP address always present itself in fully zero-padded binary (32 bits for IPv4, 128 bits for IPv6). I find this particularly convenient when putting together training material, as it's easier to show subnetting and aggregation if you point at the binary than if you give people dotted-quad addresses and ask them to do the binary conversion in their head. Hex is also handy when you're comparing a dotted-quad IP address to a hex sniffer trace. It's possible to do this in a one-liner (thanks to Eric Smith): f'{int(v4):#0{34}b}'. But this is a little cryptic. I opened bpo-32820 (https://github.com/python/cpython/pull/5627) to contribute a way to do this. I started with an __index__ method but Issue 15559 ( https://github.com/python/cpython/commit/e0c3f5edc0f20cc28363258df501758c1b…) rules this out. I instead added a bits() class method so that v4.bits would return the fully padded string. This was not terribly pretty, but it mirrored packed(), at least. Nick Coghlan suggested I instead extend __format__, which is what the diffs in the current pull request do. This allows a great deal more flexibility: the current code takes 'b', 'n', or 'x' types, as well as the '#' option and support for the '_' separator. I realize now I didn't add 'o' but I certainly can for completeness. I debated adding rfc1924 encoding for IPv6 addresses but decided it was entirely too silly. This is just a convenience function, but IMO fills a need. Is this worth pursuing? eric

3 2

Complicate str methods
by Franklin? Lee 13 Feb '18

13 Feb '18

Let s be a str. I propose to allow these existing str methods to take params in new forms. s.replace(old, new): Allow passing in a collection of olds. Allow passing in a single argument, a mapping of olds to news. Allow the olds in the mapping to be tuples of strings. s.split(sep), s.rsplit, s.partition: Allow sep to be a collection of separators. s.startswith, s.endswith: Allow argument to be a collection of strings. s.find, s.index, s.count, x in s: Similar. These methods are also in `list`, which can't distinguish between items, subsequences, and subsets. However, `str` is already inconsistent with `list` here: list.M looks for an item, while str.M looks for a subsequence. s.[r|l]strip: Sadly, these functions already interpret their str arguments as collections of characters. These new forms can be optimized internally, as a search for multiple candidate substrings can be more efficient than searching for one at a time. See https://stackoverflow.com/questions/3260962/algorithm-to-find-multiple-stri… The most significant change is on .replace. The others are simple enough to simulate with a loop or something. It is harder to make multiple simultaneous replacements using one .replace at a time, because previous replacements can form new things that look like replaceables. The easiest Python solution is to use regex or install some package, which uses (if you're lucky) regex or (if unlucky) doesn't simulate simultaneous replacements. (If possible, just use str.translate.) I suppose .split on multiple separators is also annoying to simulate. The two-argument form of .split may be even more of a burden, though I don't know when a limited multiple-separator split is useful. The current best solution is, like before, to use regex, or install a package and hope for the best.

11 14

List the methods of a metaclass in the help of its instances
by Yahya Abou 'Imran 12 Feb '18

12 Feb '18

For example, in the help of ABCs, the register method witch is defined in ABCMeta is not listed. It's a little bit weird, since it's accessible from the class just like any classmethod or staticmethod, and it's a real feature of the class. I think it would be great if there was sections in the help that look like: | ---------------------------------------------------------------------- | Metaclass methods defined in <TheMetaclass>: | ---------------------------------------------------------------------- | Metaclass methods inherited from <TheMetaclassParent>: The same rules of visibility would apply of course.

1 0

Easing set-up of of console python applications
by Mario Corchero 11 Feb '18

11 Feb '18

Hello All! I got asked how to configure the logging stack to be able to output directly to console using both stdout and stderr and I could not really find a great answer as adding both as StreamHandlers will result in error and above messages going to stdout. The usecase is having a cli or app that wants to log to console as other tools. Errors and bove to stderr and normal information to stdout. I know the recommended way in Python is to just use print on simple scripts, but it can happen that you import a library you want to see the logs of, and therefore you need to set the logging stack. I have drafted two implementations but I am open to suggestions: 1) A "Console Handler" that uses multiple streams and chooses based on the level. 2) An inverted filter that can be used to filter everything above info for the stdout Stream handler. What do you people think? If people like it I'll send an issue + PR. Regards, Mario Corchero

2 4

Consider generalizing Decimal to support arbitrary radix
by Neil Girdhar 09 Feb '18

09 Feb '18

Arbitrary radix comes up every now and then and Decimal already has a radix() method. It would be nice when initializing a Decimal object to be able to specify an arbitrary radix>=2.

7 14

Re: [Python-ideas] Complicate str methods
by Franklin? Lee 09 Feb '18

09 Feb '18

On Feb 8, 2018 13:06, "Serhiy Storchaka" <storchaka(a)gmail.com> wrote: 08.02.18 12:45, Franklin? Lee пише: > Could it be that re uses an optimization that can also be used in str? > CPython uses a modified Boyer-Moore for str.find: > https://github.com/python/cpython/blob/master/Objects/string > lib/fastsearch.h > http://effbot.org/zone/stringlib.htm > Maybe there's a minimum length after which it's better to precompute a > table. > Yes, there is a special optimization in re here. It isn't free, you need to spend some time for preparing it. You need a special object that keeps an optimized representation for faster search. This makes it very unlikely be used in str, because you need either spend the time for compilation on every search, or use some kind of caching, which is not free too, adds complexity and increases memory consumption. Note also in case of re the compiler is implemented in Python. This reduces the complexity. The performance of the one-needle case isn't really relevant, though, is it? This idea is for the multi-needle case, and my tests showed that re performs even worse than a loop of `.find`s. How do re and .find scale with both number and lengths of needles on your machine?

1 0

Re: [Python-ideas] Complicate str methods
by Franklin? Lee 08 Feb '18

08 Feb '18

On Thu, Feb 8, 2018 at 11:24 AM, Steve Dower <steve.dower(a)python.org> wrote: > Easily fixed by installing one of the alternate regex libraries. MRAB's regex library, the most prominent alternative, does not use the linear-time search algorithm. The only libraries I know that do are the ones with re2, though I haven't looked deeply. Let it be known that I tried to install pyre2 (re2 on PyPI) on Ubuntu For Windows for the tests, and after hours of no success, I decided that the alternative libraries were not the point. I eventually got it working (https://github.com/axiak/pyre2/issues/51), and here are the results: # Same setup as before, with findsub_re2 using `find=re2.search, esc=re2.escape`. pattern2 = re2.compile('|'.join(map(re.escape, needles))) %timeit findsub(haystack, needles) #=> 1.26 ms ± 15.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) %timeit findsub_re(haystack, needles) #=> 745 ms ± 19.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit findsub_re_cached(haystack, pattern) #=> 733 ms ± 16.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit findsub_regex(haystack, needles) #=> 639 ms ± 12.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit findsub_re2(haystack, needles) #=> 34.1 ms ± 1.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) %timeit findsub_re_cached(haystack, pattern2) #=> 9.56 µs ± 268 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) In any case, installing re2 is even more of an advanced solution for a basic problem than using re. > re performance and its edge cases have been discussed endlessly. Please look > it up before restarting that discussion. I'm not trying to restart the discussion. I'm trying to say that the assumptions being made about its superior performance are unfounded. Two members have suggested that re is the performant option, and that's just not true.

1 0