Mailman 3 October 2015 - Python-ideas

Allowing def to assign to anything
by Alexander Walters 27 Oct '15

27 Oct '15

In my code, I write a lot of dispatch dictionaries (for lack of a switch statement, but I will not hold my breath for that). In trying to make writing these dictionaries less annoying, I tend to use many lambdas. I can let you guess at what problems that has resulted in. Of course, the preferred way to write such dictionaries is by using a regular function, and adding that function to a dictionary. This isn't exactly a problem - it works, and works well, but it is annoying to write, and leaves artifacts of those functions in module scope. I propose a little bit of sugar to make this a little less annoying. If `def` is allowed to assign to anything (anything that is legal at the left hand side of an = in that scope), annoying artifacts go away. The syntax I propose should be backwards compatible. ``` dispatch = {} def dispatch['foo'](bar): return bar * bar ``` Does this make anything possible that is impossible now? No. But it does make the intent of the module author clear - the function is only ever intended to live inside that dict, or list, or other structure. This, to me, is less annoying to write, and is more readable. This obviously could be used outside of creating dispatch dictionaries, but that is the use case I would benefit from.

9 22

Re: [Python-ideas] PEP 8: raw strings & regular expressions
by Yury Selivanov 26 Oct '15

26 Oct '15

On 2015-10-22 9:42 PM, Steven D'Aprano wrote: > If you want my advice, it would be to either just pick whichever style > of formatting you like, and people will either like your code > highlighter and use it, or they won't, or you should consider a plug-in > architecture where your users can choose the string-highlighter style > they want, and even extend it to other DSLs. Yeah, I finally figured out a way to make the highlighter easily configurable, so that users can turn on/off highlighting of regexes, formatting literals, python2 support etc. This discussion was actually quite useful to me. Thanks to everybody for feedback! Yury

1 0

"else if" as equivalent for "elif"
by Ian Kelly 26 Oct '15

26 Oct '15

Parse the sequence "else if" equivalently to "elif". This would be 100% backward compatible. It wouldn't require adding any new keywords to the language. It shouldn't be burdensome to maintain, since "elif" only appears in one place in the grammar. And it solves a minor readability complaint. Thoughts?

14 25

Re: [Python-ideas] Allowing def to assign to anything
by Alexander Heger 26 Oct '15

26 Oct '15

how about class Dispatch(object): @staticmethod def foo(bar): return bar * bar def __getitem__(self, key): return self.__getattribute__(key) dispatch = Dispatch() >>> test.dispatch['foo'](2) 4 this leaves the module namespace less polluted. -Alexander On 26 October 2015 at 18:27, Ian Kelly <ian.g.kelly(a)gmail.com> wrote: >> On Sun, Oct 25, 2015 at 11:02 PM, Alexander Walters >> <tritium-list(a)sdamon.com> wrote: >>> >>> In my code, I write a lot of dispatch dictionaries (for lack of a switch >>> statement, but I will not hold my breath for that). In trying to make >>> writing these dictionaries less annoying, I tend to use many lambdas. I can >>> let you guess at what problems that has resulted in. Of course, the >>> preferred way to write such dictionaries is by using a regular function, and >>> adding that function to a dictionary. This isn't exactly a problem - it >>> works, and works well, but it is annoying to write, and leaves artifacts of >>> those functions in module scope. I propose a little bit of sugar to make >>> this a little less annoying. >>> >>> If `def` is allowed to assign to anything (anything that is legal at the >>> left hand side of an = in that scope), annoying artifacts go away. The >>> syntax I propose should be backwards compatible. >>> >>> ``` >>> dispatch = {} >>> >>> def dispatch['foo'](bar): >>> return bar * bar >>> ``` > > What about: > > def foo(bar)[baz](x): > return x > > This seems like it would complicate parsing as the parser can't be > sure whether (bar) is a parameter list or an argument list until it > reaches the following [baz]. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Re: [Python-ideas] PEP 8: raw strings & regular expressions
by Barry Warsaw 26 Oct '15

26 Oct '15

On Oct 22, 2015, at 06:56 PM, Yury Selivanov wrote: >In principle, there is no reason why *both* of these groups >of users can't use one tool and be happy. I propose to >establish a convention in PEP 8, explaining that, while both >literals are semantically equivalent, > >- r'..' strings *should* be used for regexps, > >- R'..' strings *should* be used for unstyled raw strings, > >and tools *should* treat them as such. > >All of this is merely about codifying the current status quo. What about all the regexps that are written without raw strings? Won't PEP 8 codification shame everyone into turn them into raw strings when they really don't need to? Cheers, -Barry

2 1

Re: [Python-ideas] PEP 8: raw strings & regular expressions
by Eric V. Smith 26 Oct '15

26 Oct '15

On 10/22/2015 06:56 PM, Yury Selivanov wrote: > On the other hand, *a lot* of users are annoyed, when their > raw strings are highlighted as regexps (why \n is highlighted > as escaped?). Often that results not in just wrongly > highlighted raw strings, but in broken highlighting of > the whole source file [1]. ... > [1] https://gist.github.com/1st1/d3da5d69a9b6c088c26f I don't see how this matters to the subject at hand. No matter what you think an r or R string contains, not detecting the end of the string is a failure of the syntax highlighter. It's failing to find the end of the triple-quoted r"""[""" token in this case. Eric.

1 0

PEP 8: raw strings & regular expressions
by Yury Selivanov 23 Oct '15

23 Oct '15

Hi, I was a little bit frustrated that Sublime Text and Atom didn't support all Python 3 features (mainly annotations & async/await syntax), so I decided to write a new highlighter: https://github.com/MagicStack/MagicPython In the process, we had to make a decision on how to highlight raw string literals -- r''. Many existing highlighters assume that all raw strings are regexps, and highlight them as such, i.e. '\s' and '\n' will be highlighted. I think that it might be a good idea to state the following in PEP 8: - use r'...' strings for raw strings that describe regular expressions; these strings might be highlighted specially in some editors. - use R'...' strings for raw strings; editors *should not* highlight any escaped characters in them. What do you think? Yury

12 18

Re: [Python-ideas] PEP 8: raw strings & regular expressions
by Sven R. Kunze 23 Oct '15

23 Oct '15

I think you shouldn't push it, Yury. Most people here seem not to like it that much for many reasons. Best, Sven On 23.10.2015 00:56, Yury Selivanov wrote: > Hi Barry, > > On 2015-10-22 5:17 PM, Barry Warsaw wrote: >>> I think that it might be a good idea to state the following in PEP 8: >>> > >>> >- use r'...' strings for raw strings that describe regular >>> expressions; >>> >these strings might be highlighted specially in some editors. >>> > >>> >- use R'...' strings for raw strings; editors*should not* highlight >>> >any escaped characters in them. >> Aside from other comments in this thread, I think it's generally a >> bad idea to >> codify semantics where there are none defined by the language. So r'' >> and R'' >> are equivalent. It would be like assigning different semantics or >> syntax >> highlighting for 'foo' and "foo". > > The thing is that *a lot* of Python users have already made > the choice -- many tools (including GitHub) do highlight > r'' strings as regexps, and people love this feature, and > complain when it's not implemented in their tool of choice. > This is an extremely useful feature, since any regexp longer > than 20 characters is unreadable at a glance. > > On the other hand, *a lot* of users are annoyed, when their > raw strings are highlighted as regexps (why \n is highlighted > as escaped?). Often that results not in just wrongly > highlighted raw strings, but in broken highlighting of > the whole source file [1]. > > In principle, there is no reason why *both* of these groups > of users can't use one tool and be happy. I propose to > establish a convention in PEP 8, explaining that, while both > literals are semantically equivalent, > > - r'..' strings *should* be used for regexps, > > - R'..' strings *should* be used for unstyled raw strings, > > and tools *should* treat them as such. > > All of this is merely about codifying the current status quo. > > Thanks, > Yury > > > [1] https://gist.github.com/1st1/d3da5d69a9b6c088c26f > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Add specialized bytecode with guards to functions
by Victor Stinner 22 Oct '15

22 Oct '15

Hi, I would like to share with you an idea to try to optimize CPython. My previous attempts to optimize CPython failed because they changed the language semantic. For example, it's not possible to replace len('abc') with 3 because it is technically possible to override the builtin len() function. I propose to add a new "FAT" mode to Python to allow to add specialized bytecode to function with guards. Example: >>> import builtins >>> def f(): return len("abc") ... >>> f() 3 >>> def g(): return 3 ... >>> i=f.add(g) # i is the index of the new specialized bytecode >>> f.add_dict_key_guard(i, builtins.__dict__, 'len') >>> f() 3 >>> builtins.len = lambda obj: "len" >>> f() 'len' In this example, the f() function gets a fast version (g()) returning directly 3 instead of calling len("abc"). The optimization is disabled when the builtin len() function is modified (when builtins.__dict__['len'] is modified). (The example is not complete, we need a second guard on the current global namespace, but I wanted to write a short example.) With such machinery, it becomes possible to implement interesting optimizations. Some examples: * inline a function instead of calling it: calling a Python function is "expensive". For example, if you inline the _get_sep() call in posixpath.isabs(), the function becomes 20% faster. * move invariant out of the loop: classic micro-optimization done manually. For example, set "append = obj.append" before the loop and then call "append(item)" instead of "obj.append(item)" in the loop body. On a microbenchmark, it's between 5% (1 item) and 30% faster (10 items or more) * call pure functions at the compilation. A pure function has no side effect. Example: len(str). Technical Challenges ==================== The problem is the cost of "guards". A guard is a check done before calling a specialized function. Example of guards: * Type of a function parameter * Watch a dictionary key * Watch a function (especially it's __code__ attribute) Watching a dictionary key is a very important guard to disable an optimization when a builtin function is modified, when a function is replaced in a namespace, etc. The problem is that a dictionary lookup is not cheap: we have to get the hash value of the key, browse the hash table to find the bucket which may require multiple iterations, then we have to compare the key value, etc. To have faster guard, I propose to create a subtype of dict which provides a global "version" of the dictionary, incremented at each modification (create a new key, modify a key, delete a key) and a "version" per key=>value mapping, incremented each time that the mapping is modified. The lookup can be avoided when the dictionary is not modified. If a different key is modified, we need a lookup, but only once (we store the new global version to avoid a lookup at the next guard check). Limitations =========== Specialized bytecode is build at compilation, *not* at runtime: it's not a just-in-time (JIT) compiler. A JIT can implement even more efficient optimizations and can use better guards. Please continue to use PyPy for best performances! ;-) The name "FAT" comes from the fact that multiple bytecode versions of a function are stored in the memory, so a larger memory footprint and larger .pyc files on disk can be expected. Checking guards can also be more expensive than the optimization of the specialized bytecode. The optimizer should use an heuristic to decide if it's worth to use a specialized bytecode or not depending on the theoric speeup and the cost of guards. My motivation to write FAT Python is that CPython remains the reference implementation where new features are implemented, and other Python implementations still have issues with C extensions. JIT also has some issues, like longer startup time, slow warmup and memory footprint (this one may also be an issue of FAT Python!). Implementation ============== I wrote a proof-of-concept of my idea: https://hg.python.org/sandbox/fatpython/ It's a fork of CPython 3.6. Try it with: hg clone https://hg.python.org/sandbox/fatpython/ cd fatpython ./configure && make ./python -F See bench.py, Lib/posixpath.py (isabs) and Lib/test/test_fat.py for examples of optimizations. The command line -F flag enables the FAT mode. By default, *all* optimizations are disabled, there is a negligible overhead when the FAT mode is not used. In the FAT mode, the dictionary for modules, classes and instances becomes "versionned". Functions gets new methods to support adding specialized bytecode with guards. You can add manually a specialized bytecode for microbenchmarks, but functions are *not* optimized automatically. (See the Roadmap below.) How to write an optimizer? ========================== Currently, my proof-of-oncept is only the machinery to support adding specialized bytecodes with guards. The next step is to generate automatically these specialized versions. IMHO an optimizer should be implemented in Python and not in C, it's easier to write Python code. We can begin with simple optimizations on AST. See for example my astoptimizer which implements basic optimizations: https://bitbucket.org/haypo/astoptimizer/ At the beginning, we may use type hints (PEP 484) and other manual hints to help the optimizer. Later, a profiler learning the most common types of function parameters can be written for automatic optimizations. Example of hints: a list of constants which should not be modified in the application. A global "DEBUG" flag is common in applications, relying on DEBUG=False helps to remove dead code. Roadmap ======= * Finish the proof-of-concept: implement new guards, optimize method calls. For example, guards on module imports are required. Checking if a module is the expected mode can be tricky. * Write an optimizer based on manual hints * Write a profiler to generate hints Right now, my main question is if it will be possible and efficient to optimize classes, not only simple functions, especially classes defined in two different modules. I wrote a proof-of-concept of optimized method (with inlining), but I'm not sure yet that it's safe (don't change Python semantic). For more information on FAT Python, read: https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst So, what do you think? Do you expect real speedup on applications, not only on microbechmarks? Do you know similar optimizations in other scripting languages? Victor Stinner

11 23

PEP-505: Draft (Coalescing Operators)
by Mark E. Haase 22 Oct '15

22 Oct '15

I've finally finished a reasonably complete draft of PEP-505. It's taken me longer than I expected due to work and also a new addition to our family :) I tried to keep the PEP humorous; but please don't mistake silliness for a lack of seriousness. Let's try to keep separate two different debates: (1) the utility of the proposed operators (2) how to spell them. Arguing about #2 is a waste of time if #1 is a non-starter. If we can achieve some consensus on #1, then with Guido's consent, I think the next logical step would be to conduct a survey of opinions about proposed spellings, just as the authors of PEP-308 did. The survey is a data-gathering exercise, of course, not a democratic vote. The PEP has more details on that. I envision the PEP as being severable: it might be accepted in whole, in part, or not at all. For this reason, there are quite a few alternatives proposed, and some of them are mutually conflicting. The conflicts aren't a problem unless and until any parts of the proposal get the green light. (Again, similar to PEP-308.) https://www.python.org/dev/peps/pep-0505/ -- Mark E. Haase

2 2