Mailman 3 April 2016 - Python-ideas

Exposing flat bytecode representation to optimizers
by Andrew Barnert 15 May '16

15 May '16

The biggest pain with dealing with the peephole optimizer is that it happens after all the complicated flattening and fixup[^1] the compiler does, which means you have to hack up the jump targets as you go along. The future bytecode optimizers that PEP 511 enables will have the same headache. But this isn't actually necessary. The optimizer could work on a flat array of instructions[^2] instead of an array of bytes, with relocatable jump targets instead of fixed byte offsets, and then the compiler could do the fixup _after_ the optimization.[^3] It would break the optimizer APIs, but `PyCode_Optimize` isn't public, and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. I don't think we need to expose the intermediate representation any farther along than the `PyCode_Optimize` step.[^4] Just moving the optimize one step earlier in the chain solves more than enough to be worth it. [^1]: If you think numbering the offsets and emitting the jump targets is easy: Every time you fix up a jump, that may require adding an `EXTENDED_ARG`, which means you have to redo any fixups that cross the the current instruction. The compiler does this by just looping until it hasn't added any more `EXTENDED_ARG`s. [^2]: In case anyone's thinking that wordcode would solve this problem, it doesn't. The `EXTENDED_ARG` jump targets are a much bigger hassle than the 1-or-3-byte-ops, and wordcode not only can't eliminate those, but makes `EXTENDED_ARG` more common. [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. That's actually even better to work with, but too complicated to expose to optimizers. Flattening it would be trivial. Or, if that's too expensive, we could do something almost as simple and much cheaper: convert it in-line to a deque-like linked list of arrays, with jump targets being indices or pointers into that. Or we could just expose the list of blocks as-is, as an opaque thing with a mutable-deque-of-instructions API around it. [^4]: Later stages--import hooks, optimizing decorators, etc.--have the same pain as the peephole optimizer, but they also have code objects that contain other code objects, and they can be written in Python, and so on, so the same solution isn't feasible there. Of course we could add a function to convert bytecode back to a list of instructions, in some format that can be exposed to Python (in fact, `dis` already does 90% of that), and then another one to convert that back to bytecode and redo the fixup (which is basically the third-party `byteplay` module). But that's almost certainly overkill. (If we wanted that, I'd rather add `byteplay` to the stdlib, port it and `dis` to C, and expose a C API for them. And then we could use that for everything, including the peephole optimizer and PEP 511 optimizers. Which would all be really cool, but I think it's more work than we want to do, and I don't know if we'd actually want something like `byteplay` builtin even if it were easy...)

6 16

unittest: assertEqual(a, b, msg): Show diff AND msg
by Thomas Güttler 27 Apr '16

27 Apr '16

Up to now assertEqual(a, b, msg) outputs only the msg, not the diff. I know that setting longMessage to True shows the diff and the msg[1] I think the sane default is to show the diff and the message for assertEqual(). What do you think? Regards, Thomas Güttler [1] longMessage: https://docs.python.org/3/library/unittest.html#unittest.TestCase.longMessa… -- Thomas Guettler http://www.thomas-guettler.de/

2 3

Allow with (x as y, z as w):
by Ram Rachum 26 Apr '16

26 Apr '16

Hi, I have a problem: I've got code that has a `with` statement with multiple long names, so I have to break the line awkwardly using the \ character: with some_darn_object.my_long_named_method_on_it(argument='yep') as foo, \ a_different_darn_object.and_yet_another_method() as bar: I tried wrapping the context managers in parentheses, but it looks like it's not legal syntax: File "<stdin>", line 1 with (some_darn_object.my_long_named_method_on_it(argument='yep') as foo, ^ SyntaxError: invalid syntax Another demonstration from the REPL: Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> with (x as y, z as w): pass File "<stdin>", line 1 with (x as y, z as w): pass ^ SyntaxError: invalid syntax I wish this syntax to be legal. Thanks, Ram.

4 3

Re: [Python-ideas] Shrink recursion error tracebacks (was: Have REPL print less by default)
by Émanuel Barry 24 Apr '16

24 Apr '16

I have been thinking, and especially after Nick’s comments, that it might be better to keep it as simple as possible to reduce the risk of errors happening while we’re printing the tracebacks. Recursive functions (that go deep enough to trigger an exception) are usually within one function, in the REPL, or by using __getattr__ / __getattribute__. Nice to have? Sure. Necessary? Don’t think as much. I’m also a very solid -1 on the idea of prompting the user to do something like full_ex() to get the full stack trace. My rationale for such a change is that we need to be extremely precise in our message, to leave absolutely no room for a different interpretation than what we mean. Your message, for instance, is ambiguous. The fact it says that calls are hidden would let me think I just lost information about the stack trace (though that’s but a wording issue). As a user, just seeing "Call _full_ex() to print full trace." would be an immediate red flag: Crap, I just lost precious debugging information to save on lines! Might be just me, but that’s my opinion anyway :) -Emanuel Example: File "<stdin>", line 1, in f File "<stdin>", line 1, in g [Mutually recursive calls hidden: f (300), g (360)] File "<stdin>", line 1, in h File "<stdin>", line 1, in f File "<stdin>", line 1, in g [Mutual-recursive calls hidden: f (103), g (200)] RuntimeError: maximum recursion depth exceeded [963 calls hidden. Call _full_ex() to print full trace.] This rule is easily modified to, "have been seen at least three times before." For functions that recurse at multiple lines, it can print out one message per line, but maybe it should count them all together in the "hidden" summary.

4 5

Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.
by Neil Girdhar 22 Apr '16

22 Apr '16

Sometimes users inherit from builtin types only to find that their overridden methods are not called. Instead of this being a trap for unsuspecting users, I suggest overriding the __new__ method of these types so that it will raise with an informative exception explaining that, e.g., instead of inheriting from dict, you should inherit from UserDict. I suggest this modification to any Python implementation that has special versions of classes that cannot easily be extended, such as CPython. If another Python implementation allows dict (e.g.) to be extended easily, then it doesn't have to raise. Best, Neil

12 35

Shrink recursion error tracebacks (was: Have REPL print less by default)
by Émanuel Barry 22 Apr '16

22 Apr '16

This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, the function definition that triggered the exception. Of course, the same is true for actual programs, where tracebacks drown off everything else useful. For all we know, a typo caused a NameError in one of the functions which somehow triggered it to call itself over again until Python decided it was enough, but you can't know that because your entire scrollback is full of File "<stdin>", line 1, in func And, for programs that are user-facing, I take all tracebacks and redirect them to a file that the user can submit me so I can debug the issue, and having tens of hundreds of the same line hurts readability. The issue that the output from running a certain script in the REPL and as a script would differ has been raised, and I am of the opinion that in no way, shape or form should the two have different behaviours (the exception being sys.ps1, sys.ps2 and builtins._ in interactive mode, which are 100% fine). I'm suggesting a change to how Python handles specifically recursion limits, to cut after a sane number of times (someone suggested to shrink the output after 3 times), and simply stating how many more identical messages there are. I would also like to extend this to any arbitrary loop of callers (for example foo -> bar -> baz -> foo and so on would be counted as one "call" for the purposes of this proposal). Under this proposal, something similar to this would happen: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in func File "<stdin>", line 1, in func File "<stdin>", line 1, in func [Previous 1 message(s) repeated 996 more times] RecursionError: maximum recursion depth exceeded With multiple chained calls (I don't know how hard it would be to implement this, probably not trivial): Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz [Previous 3 message(s) repeated 330 more times] RecursionError: maximum recursion depth exceeded We can probably afford to cut out immediately as we notice the recursion, but even just 3 times with multiple chained calls isn't going to be anywhere near as long as it currently is. Thoughts? Suggestions? Something I forgot? -Emanuel

9 13

Why was Path.path added?
by Antony Lee 21 Apr '16

21 Apr '16

Well, it's already there and it's probably too late to remove it, but introducing the idiom path_str = getattr(arg, "path", arg) was not necessary: you could also write path_str = str(Path(arg)) which works just as well. My 2c., Antony

5 4

Have REPL print less by default
by Franklin? Lee 21 Apr '16

21 Apr '16

Two different (probably radical) ideas, with the same justification: reduce often-useless output in the REPL, which can flood out terminal history and overwhelm the user. 1. Limit the output per entered command: If you type into the REPL (AKA interactive shell), list(range(n)) and you forgot that you set n to 10**10, the interpreter should not print more than a page of output. Instead, it will print a few lines ("... and approximately X more lines"), and tell you how to print more. (E.g. "Call '_more()' for more. Call '_full()' for full output.") Alternatively, have "less"-like behavior. 2. Only print a few parts of the stack trace. In particular, for a recursive or mutually recursive function, if the error was due to maximum recursion (is this reasonably distinguishable? the error is `RuntimeError('maximum recursion depth exceeded')`), try to print each function on the stack once each. Again, there should be a message telling you how to get the full stacktrace printed. EXACTLY how, preferably in a way that is easy to type, so that a typo won't cause the trace to be lost. It should not use `sys.something()`, because the user's first few encounters with this message will result in, "NameError: name 'sys' is not defined". A few possible rules to reduce stacktrace size: - Always show the last (top) frame(?). - Hide any other stdlib funcs directly below (above) it. - If a function appears more than once in a row, show it once, with the note, "(and X recursive calls)". - If functions otherwise appears more than once (usually by mutual recursion?), and there is a run of them, list them as, "(Mutual recursion: 'x' (5 times), 'y' (144 times), 'z' (13 times).)". These two behaviors and their corresponding functions could go into a special module which is (by default) loaded by the interactive shell. The behaviors can be turned off with some command-line verbosity flag, or tuned with command-line parameters (e.g. how many lines/pages to print). Justification: - Excessive output floods the terminal window. Some terminals have a limit on output history (Windows *defaults* to 300 lines), or don't let you scroll up at all (at least, my noob self couldn't figure it out when I did get flooded). - Students learning Python, and also everyone else using Python, don't usually need 99% of a 10000-line output or stack trace. - People who want the full output are probably advanced users with, like, high-limit or unlimited window size, and advanced users are more likely to look for a verbosity flag, or use a third-party REPL. Default should be newbie friendly, because advanced users can work around it. Thoughts? Even if the specific proposals are unworkable, is limiting REPL output (by default) a reasonable goal?

16 39

Re: [Python-ideas] Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.
by Neil Girdhar 21 Apr '16

21 Apr '16

Good point, I withdraw my suggestion. I think it's unfortunate for Python to have traps like this, but I don't see a nice way to protect users while still letting people do whatever they want. On Thu, Apr 21, 2016 at 3:03 AM Neil Girdhar <mistersheik(a)gmail.com> wrote: > I guess that's fair. > > On Thu, Apr 21, 2016 at 2:36 AM Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> On 21 April 2016 at 12:51, Neil Girdhar <mistersheik(a)gmail.com> wrote: >> >>> Sometimes users inherit from builtin types only to find that their >>> overridden methods are not called. Instead of this being a trap for >>> unsuspecting users, I suggest overriding the __new__ method of these types >>> so that it will raise with an informative exception explaining that, e.g., >>> instead of inheriting from dict, you should inherit from UserDict. >>> >>> I suggest this modification to any Python implementation that has >>> special versions of classes that cannot easily be extended, such as >>> CPython. If another Python implementation allows dict (e.g.) to be >>> extended easily, then it doesn't have to raise. >>> >> >> Builtins can be extended, you just have to override all the methods where >> you want to change the return type: >> >> >>> from collections import defaultdict, Counter, OrderedDict >> >>> issubclass(defaultdict, dict) >> True >> >>> issubclass(Counter, dict) >> True >> >>> issubclass(OrderedDict, dict) >> True >> >> This isn't hard as such, it's just tedious, so it's often simpler to use >> the more subclass friendly variants that dynamically look up the type to >> return and hence let you get away with overriding a smaller subset of the >> methods. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia >> >

1 0

Anonymous namedtuples
by Joseph Martinot-Lagarde 20 Apr '16

20 Apr '16

Hi, list ! namedtuples are really great. I would like to use them even more, for example for functions that return multiple arguments. The problem is that namedtuples have to be "declared" beforehand, so it would be quite tedious to declare a nameedtuple by function, that's why I very rarely do it. Another hting I don't like about namedtuples is the duplication of the name. TYpical declarations look like `Point = namedtuple('Point', ['x', 'y'])`, where `Point` is repeated two times. I'll go one step further and say that the name is useless most of the time, so let's just get rid of it. Proposal ======== So I thought about a new (ok, maybe it has been proposed before but I couldn't find it) syntax for anonymous namedtuples (I put the prints as comments, otherwise gmane is complainig about top-posting): my_point = (x=12, y=16) # (x=12, y=16) my_point[0] # 12 my_point.y # 16 type(my_point) # <class 'anonymousnamedtuple'> It's just a tuple, but with names. Parenthesis would be mandatory because `my_point = x = 12, y = 16` wouldn't work. Single elements anonymouns namedtuples would require a trailing comma, similarely to tuples. I'd be happy to make a factory function for my personal use, but the order of kwargs is not respected. To have an elegant syntax, it has to be a construct of the language. The created objects would all be of the same class (with a better name, of course). As it adds no keyword to the language, it would not break compatibility. The created objects could support some namedtuples methods: _asdict, _replace and _fields. The two other methods _make and _source would not apply. Performance-wise, I guess that they would be slower than tuples and namedtuples, But to me the additionnal usability trumps the performance hit. If you have to care about performance, you can still use a namedtuple ot bare tuples. Even further... =============== This initial ideas is useful in itself, but it could be extended even further. I've thought about a possible evolution. The idea is that not all values has to be named: similarely to how args and kwargs work for functions, there could be non-named values, with the same limitations as the arguments (unnamed first, named afterwards). All values could be retrived by indexing, and named values could also be retrieved by their attribute name. This way they could be used in __getitem__ as proposed in PEP 472[0], so that __getitem__ supports additionnal keywords arguments. It would correspond to Strategy "named tuple", with some of the cons removed: "The namedtuple fields, and thus the type, will have to change according to the passed arguments. This can be a performance bottleneck, and makes it impossible to guarantee that two subsequent index accesses get the same Index class;" That would still be true for the performance bottleneck, but since the class would always be the same the second problem disappears. To minimize the performace hit, a standard tuple would be passed to __getitiem__ if there is no keyword argument, and an anonymous namedtuple would be passed if there is a keyword. "the _n "magic" fields are a bit unusual, but ipython already uses them for result history." Those wouldn't be needed if both named and unnamed values are allowed. "Differently from a function, the two notations gridValues[x=3, y=5, z=8] and gridValues[3,5,8] would not gracefully match if the order is modified at call time (e.g. we ask for gridValues[y=5, z=8, x=3]) . In a function, we can pre-define argument names so that keyword arguments are properly matched. Not so in __getitem__ , leaving the task for interpreting and matching to __getitem__ itself." Indexing already doesn't behave like a function call. Keeping the argument order (like this proposal would imply) is a special case of not keeping it, while the reverse is not true, so the more general solution would be implemented. Finally, I admit that I have no idea of how to implement this, I love python but never looked at it's internals. Maybe my proposal is too naive, I really hope not. Thanks for your attention, Joseph [0] https://www.python.org/dev/peps/pep-0472/

12 40