Mailman 3 April 2018 - Python-ideas

Asynchronous exception handling around with/try statement borders
by Erik Bray 24 Sep '18

24 Sep '18

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts?

7 15

Positional-only parameters
by Victor Stinner 11 Sep '18

11 Sep '18

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor

19 27

Proposal: A Reduce-Map Comprehension and a "last" builtin
by Peter O'Connor 31 May '18

31 May '18

Dear all, In Python, I often find myself building lists where each element depends on the last. This generally means making a for-loop, create an initial list, and appending to it in the loop, or creating a generator-function. Both of these feel more verbose than necessary. I was thinking it would be nice to be able to encapsulate this common type of operation into a more compact comprehension. I propose a new "Reduce-Map" comprehension that allows us to write: signal = [math.sin(i*0.01) + random.normalvariate(0, 0.1) for i in range(1000)] smooth_signal = [average = (1-decay)*average + decay*x for x in signal from average=0.] Instead of: def exponential_moving_average(signal: Iterable[float], decay: float, initial_value: float=0.): average = initial_value for xt in signal: average = (1-decay)*average + decay*xt yield average signal = [math.sin(i*0.01) + random.normalvariate(0, 0.1) for i in range(1000)] smooth_signal = list(exponential_moving_average(signal, decay=0.05)) I've created a complete proposal at: https://github.com/petered/peps/blob/master/pep-9999.rst , (and a pull-request <https://github.com/python/peps/pull/609>) and I'd be interested to hear what people think of this idea. Combined with the new "last" builtin discussed in the proposal, this would allow u to replace "reduce" with a more Pythonic comprehension-style syntax. - Peter

22 65

Allow multiple imports from a package while preserving its namespace
by Julian DeMille 22 May '18

22 May '18

I personally would like a feature where instead of doing `from ... import ...` (which imports the specified items into the current namespace), one could use something along the lines of `import <lib>.{ <mod1>, <mod2>, ... }` such that the imported modules/attributes could be accessed as `<lib>.<mod1>`, etc. -- Thanks, Julian DeMille CEO, demilleTech, LLC This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

10 19

A cute Python implementation of itertools.tee
by Tim Peters 14 May '18

14 May '18

Just for fun - no complaint, no suggestion, just sharing a bit of code that tickled me. The docs for `itertools.tee()` contain a Python work-alike, which is easy to follow. It gives each derived generator its own deque, and when a new value is obtained from the original iterator it pushes that value onto each of those deques. Of course it's possible for them to share a single deque, but the code gets more complicated. Is it possible to make it simpler instead? What it "really" needs is a shared singly-linked list of values, pointing from oldest value to newest. Then each derived generator can just follow the links, and yield its next result in time independent of the number of derived generators. But how do you know when a new value needs to be obtained from the original iterator, and how do you know when space for an older value can be recycled (because all of the derived generators have yielded it)? I ended up with almost a full page of code to do that, storing with each value and link a count of the number of derived generators that had yet to yield the value, effectively coding my own reference-count scheme by hand, along with "head" and "tail" pointers to the ends of the linked list that proved remarkably tricky to keep correct in all cases. Then I thought "this is stupid! Python already does reference counting." Voila! Vast swaths of tedious code vanished, giving this remarkably simple implementation: def mytee(xs, n): last = [None, None] def gen(it, mylast): nonlocal last while True: mylast = mylast[1] if not mylast: mylast = last[1] = last = [next(it), None] yield mylast[0] it = iter(xs) return tuple(gen(it, last) for _ in range(n)) There's no need to keep a pointer to the start of the shared list at all - we only need a pointer to the end of the list ("last"), and each derived generator only needs a pointer to its own current position in the list ("mylast"). What I find kind of hilarious is that it's no help at all as a prototype for a C implementation: Python recycles stale `[next(it), None]` pairs all by itself, when their internal refcounts fall to 0. That's the hardest part. BTW, I certainly don't suggest adding this to the itertools docs either. While it's short and elegant, it's too subtle to grasp easily - if you think "it's obvious", you haven't yet thought hard enough about the problem ;-)

7 11

string method count()
by Julia Kim 08 May '18

08 May '18

Hi, There’s an error with the string method count(). x = ‘AAA’ y = ‘AA’ print(x.count(y)) The output is 1, instead of 2. I write programs on SoloLearn mobile app. Warm regards, Julia Kim

7 8

A "local" pseudo-function
by Tim Peters 03 May '18

03 May '18

A brain dump, inspired by various use cases that came up during the binding expression discussions. Idea: introduce a "local" pseudo-function to capture the idea of initialized names with limited scope. As an expression, it's "local" "(" arguments ")" - Because it "looks like" a function call, nobody will expect the targets of named arguments to be fancier than plain names. - `a=12` in the "argument" list will (& helpfully so) mean pretty much the same as "a=12" in a "def" statement. - In a "local call" on its own, the scope of a named argument begins at the start of the next (if any) argument, and ends at the closing ")". For the duration, any variable of the same name in an enclosing scope is shadowed. - The parentheses allow for extending over multiple lines without needing to teach editors (etc) any new tricks (they already know how to format function calls with arglists spilling over multiple lines). - The _value_ of a local "call" is the value of its last "argument". In part, this is a way to sneak in C's comma operator without adding cryptic new line noise syntax. Time for an example. First a useless one: a = 1 b = 2 c = local(a=3) * local(b=4) Then `c` is 12, but `a` is still 1 and `b` is still 2. Same thing in the end: c = local(a=3, b=4, a*b) And just to be obscure, also the same: c = local(a=3, b=local(a=2, a*a), a*b) There the inner `a=2` temporarily shadows the outer `a=3` just long enough to compute `a*a` (4). This is one that little else really handled nicely: r1, r2 = local(D = b**2 - 4*a*c, sqrtD = math.sqrt(D), twoa = 2*a, ((-b + sqrtD)/twoa, (-b - sqrtD)/twoa)) Everyone's favorite: if local(m = re.match(regexp, line)): print(m.group(0)) Here's where it's truly essential that the compiler know everything about "local", because in _that_ context it's required that the new scope extend through the end of the entire block construct (exactly what that means TBD - certainly through the end of the `if` block, but possibly also through the end of its associated (if any) `elif` and `else` blocks - and similarly for while/else constructs). Of course that example could also be written as: if local(m = re.match(regexp, line), m): print(m.group(0)) or more specifically: if local(m = re.match(regexp, line), m is not None): print(m.group(0)) or even: if local(m = re.match(regexp, line)) is not None: print(m.group(0)) A listcomp example, building the squares of integers from an iterable but only when the square is a multiple of 18: squares18 = [i2 for i in iterable if local(i2=i*i) % 18 == 0] That's a bit mind-bending, but becomes clear if you picture the kinda-equivalent nest: for i in iterable: if local(i2=i*i) % 18 == 0: append i2 to the output list That should also make clear that if `iterable` or `i` had been named `i2` instead, no problem. The `i2` created by `local()` is in a wholly enclosed scope. Drawbacks: since this is just a brain dump, absolutely none ;-) Q: Some of those would be clearer if it were the more Haskell-like local(...) "in" expression A: Yup, but for some of the others needing to add "in m" would be annoyingly redundant noise. Making an "in" clause optional doesn't really fly either, because then local(a='z') in 'xyz' would be ambiguous. Is it meant to return `'xyz'`, or evaluate `'z' in 'xyz'`? And any connector other than "in" would make the loose resemblance to Haskell purely imaginary ;-) Q: Didn't you drone on about how assignment expressions with complex targets seemed essentially useless without also introducing a "comma operator" - and now you're sneaking the latter in but _still_ avoiding complex targets?! A. Yes, and yes :-) The syntactic complexity of the fully general assignment statement is just too crushing to _sanely_ shoehorn into any "expression-like" context. Q: What's the value of this? local(a=7, local(a=a+1, a*2)) A: 16. Obviously. Q: Wow - that _is_ obvious! OK, what about this, where there is no `a` in any enclosing scope: local(a) A: I think it should raise NameError, just like a function call would. There is no _intent_ here to allow merely declaring a local variable without supplying an initial value. Q: What about local(2, 4, 5)? A: It should return 5, and introduce no names. I don't see a point to trying to outlaw stupidity ;-) Then again, it would be consistent with the _intent_ to require that all but the last "argument" be of the `name=expression` form. Q: Isn't changing the meaning of scope depending on context waaaay magical? A: Yup! But in a language with such a strong distinction between statements and expressions, without a bit of deep magic there's no single syntax I can dream up that could work well for both that didn't require _some_ deep magic. The gimmick here is something I expect will be surprising the first time it's seen, less so the second, and then you're never confused about it again. Q: Are you trying to kill PEP 572? A: Nope! But since this largely subsumes the functionality of binding expressions, I did want to put this out there before 572's fate is history. Binding expressions are certainly easier to implement, and I like them just fine :-) Note: the thing I'm most interested in isn't debates, but in whether this would be of real use in real code.

21 85

Objectively Quantifying Readability
by Matt Arcidy 02 May '18

02 May '18

The number and type of arguments about readability as a justification, or an opinion, or an opinion about an opinion seems counter-productive to reaching conclusions efficiently. I think they are very important either way, but the justifications used are not rich enough in information to be very useful. A study has been done regarding readability in code which may serve as insight into this issue. Please see page 8, fig 9 for a nice chart of the results, note the negative/positive coloring of the correlations, grey/black respectively. https://web.eecs.umich.edu/~weimerw/p/weimer-tse2010-readability-preprint.p… The criteria in the paper can be applied to assess an increase or decrease in readability between current and proposed changes. Perhaps even an automated tool could be implemented based on agreed upon criteria. Opinions about readability can be shifted from: - "Is it more or less readable?" to - "This change exceeds a tolerance for levels of readability given the scope of the change." Still need to argue "exceeds ...given" and "tolerance", but at least the readability score exists, and perhaps over time there will be consensus. Note this is an attempt to impact rhetoric in PEP (or other) discussions, not about supporting a particular PEP. Please consider this food for thought to increase the efficacy and efficiency of PEP discussions, not as commenting on any specific current discussion, which of course is the motivating factor of sending this email today. I think using python implicitly accepts readability being partially measurable, even if the resolution of current measure is too low to capture the changes currently being discussed. Perhaps using this criteria can increase that resolution. Thank you, - Matt

13 24

Should __builtins__ have some kind of pass-through print function, for debugging?
by Nathaniel Smith 02 May '18

02 May '18

Hi all, This came up in passing in one of the PEP 572 threads, and I'm curious if folks think it's a good idea or not. When debugging, sometimes you have a somewhat complicated expression that's not working: # Hmm, is func2() returning the right thing? while (func1() + 2 * func2()) < func3(): ... It'd be nice to print out what func2() returns, but to do that we have to refactor this code, which might be rather tricky in a case like this. I think if you want to use print() directly here, the simplest way to do that is: while True: tmp = func2() print(tmp) if not (func1() + 2 * func2()) < func3()): break ... Obviously this is annoying and error prone – especially for beginners, who are the ones most likely to need to print out lots of stuff to figure out why their code isn't working. (Chris Angelico mentioned that he finds this to be a common problem when teaching beginners.) There is a better way: if you define a trivial helper like: # "debug print": prints and then returns its argument def dp(obj): print(repr(obj)) return obj then the rewritten code becomes: while (func1() + 2 * dp(func2())) < func3(): ... Of course, this is trivial -- for me or you. But the leap to first realize that this is a useful thing, and then implement it correctly, is really asking a lot of beginners, who by assumption are struggling to do *anything* with Python syntax. And similarly, putting a package on PyPI is useful (cf. the venerable 'q' package), but still adds a significant barrier to entry: you need to be able to install packages, and you need to add an import. In fact, I can imagine that you might want to teach this trick even before you teach what imports are. So, would it make sense to include a utility like this in __builtins__? PEP 553, the breakpoint() builtin, provides some relevant precedent. Looking at it, I see it also emphasized the value of letting IDEs override the debugger, and I can see some similar value here: e.g. fancy REPLs like Spyder or Jupyter could potentially capture the objects passed to dp() and make them available for interactive viewing (imagine if they're like a large dataframe or something). Points to argue over if people like the general idea: - The name: p(), dp(), debug(), debugprint(), ...? - __str__ or __repr__? Presumably __repr__ since it's a debugging tool. - Exact semantics: there should probably be some way to add a bit of metadata that gets printed out, for cases like: while (dp(func1(), "func1") + 2 * dp(func2(), "func2")) < dp(func3(), "func3"): ... Maybe other tweaks would be useful as well. -n -- Nathaniel J. Smith -- https://vorpus.org

9 16

Sublocal scoping at its simplest
by Chris Angelico 30 Apr '18

30 Apr '18

There's been a lot of talk about sublocal scopes, within and without the context of PEP 572. I'd like to propose what I believe is the simplest form of sublocal scopes, and use it to simplify one specific special case in Python. There are no syntactic changes, and only a very slight semantic change. def f(): e = 2.71828 try: 1/0 except Exception as e: print(e) print(e) f() The current behaviour of the 'except... as' statement is as follows: 1) Bind the caught exception to the name 'e', replacing 2.71828 2) Execute the suite (printing "Division by zero") 3) Set e to None 4) Unbind e Consequently, the final print call raises UnboundLocalError. I propose to change the semantics as follows: 1) Bind the caught exception to a sublocal 'e' 2) Execute the suite, with the reference to 'e' seeing the sublocal 3) Set the sublocal e to None 4) Unbind the sublocal e At the unindent, the sublocal name will vanish, and the original 'e' will reappear. Thus the final print will display 2.71828, just as it would if no exception had been raised. The above definitions would become language-level specifications. For CPython specifically, my proposed implementation would be for the name 'e' to be renamed inside the block, creating a separate slot with the same name. With no debates about whether "expr as name" or "name := expr" or "local(name=expr)" is better, hopefully we can figure out whether sublocal scopes are themselves a useful feature :) ChrisA

5 7