Mailman 3 December 2017 - Python-ideas

Repr of lambda
by Serhiy Storchaka 24 Dec '17

24 Dec '17

Currently repr of doesn't contain much of information besides that it is a lambda. >>> lambda x: x**2 <function <lambda> at 0x7f3479b74488> All lambdas have the same repr, different only by unreadable hexadecimal address. What if include the signature and the expression of the lambda in its repr? >>> lambda x: x**2 <lambda x: x**2> This would return an old feature of Python 0.9.1 (https://twitter.com/dabeaz/status/934835068043956224). Repr of function could contain just the signature. <function foo(bar, baz)> But there is a problem with default values. Their reprs can be very long, especially in the following case with mutable default value: def foo(x, _cache={}): try: return _cache[x] except KeyError: pass _cache[x] = y = expensive_calculation(x) return y Maybe the placeholder should be always used instead of default values.

13 16

Support floating-point values in collections.Counter
by Joel Croteau 20 Dec '17

20 Dec '17

It would be useful in many scenarios for values in collections.Counter to be allowed to be floating point. I know that Counter nominally emulates a multiset, which would suggest only integer values, but in a more general sense, it could be an accumulator of either floating point or integer data. As near as I can tell, Collection already does support float values in both Python 2.7 and 3.6, and the way the code is implemented, this change should be a no-op. All that is required is to update the documentation to say floating-point values are allowed, as it currently says only integers are allowed.

6 7

Raise StopIteration with same value on subsequent `next`s
by Franklin? Lee 12 Dec '17

12 Dec '17

Consider the following code, which creates a generator that immediately returns 1, and then catches the StopIteration twice. def generatorfunction(): if False: yield return 1 def get_return(gen): try: next(gen) except StopIteration as e: return e.value else: raise ValueError("Generator was not ready to stop.") gen = generatorfunction() get_return(gen) #=> 1 get_return(gen) #=> None The first time StopIteration is raised, it contains the returned value. If StopIteration is forced again, the value is missing. What about keeping the return value for subsequent raises? Perhaps as an attribute on the generator object? The main disadvantage is that it'd add another reference, keeping the return value alive as long as the generator is alive. However, I don't think you'll want to keep a dead generator around anyway. Note: Javascript, which added generators, agrees with the Python status quo: the return value is only available on the first stop. C# does not have the concept of returning from an iterator. ---- Background: I made a trampoline for a toy problem, using generators as coroutines to recurse. When it came time to memoize it, I had to couple the memoization with the trampoline, because I could not cache the answer before it was computed, and I could not cache the generator object because it would not remember its return value later. I would have attached the returned value to the generator object, but for some reason, generator and coroutine objects can't take attributes. Maybe I should ask for that feature instead. Either feature would allow the concept of a coroutine that is also a thunk.

3 4

Decorator for creating enumeration?
by Kirill Balunov 10 Dec '17

10 Dec '17

Since PEP 557 "Data Classes"[1] and PEP 526 "Syntax for Variable Annotations"[2] are accepted and become part of the language. Is it worth writing a proposal about decorator-version for creating an enumeration? Something like: from enum import enum @enum(unique=True, int_=False, flag=False, ...): class Color: RED : auto GREEN: auto BLUE : auto Despite the fact that Functional API to create enums already exists, it seems to me that decorator-version will allow to unify these two relatively young residents of the standard library. In addition, the idea of "Not having to specify values for enums"[3], which at the time seemed to involve much magic in the implementation, becomes part of the language. Of course, PEP 526 unequivocally says that it does not allow one to annotate the types of variables when tuple unpacking is used. But in any case, I find the variant with the decorator to be an interersting idea. With kind regards, -gdg [1] PEP 557 "Data Classes" <https://www.python.org/dev/peps/pep-0557/> [2] PEP 526 "Syntax for Variable Annotations" <https://www.python.org/dev/peps/pep-0526/> [3] Not having to specify values for enums. <https://www.python.org/dev/peps/pep-0435/#not-having-to-specify-values-for-…>

3 2

a sorting protocol dunder method?
by Chris Barker 09 Dec '17

09 Dec '17

I can't believe this hasn't been brought up before, but searching the web, and python-ideas, and all the PEPs has found nothing (could be my lame google-fu), so here goes: Recent python has moved toward a "key" function for customized sorting: list.sort(key=key_fun) key is also used (according to https://docs.python.org/3.6/library/functools.html#functools.cmp_to_key) in: min(), max(), heapq.nlargest(), heapq.nsmallest(), itertools.groupby() with this fairly broad use, it seems it's becoming a fairly universal protocol for ordering. However, if you are writing a custom class, and want to make it "sortable", you need to define (some of) the total comparison operators, which presumably are then called O(n logn) number of times for comparisons when sorting. Or provide a sort key function when you actually do the sorting, which requires some inside knowledge of the objects you are sorting. But what if there was a sort key magic method: __key__ or __sort_key__ (or whatever) that would be called by the sorting functions if: no key function was specified and it exists It seems this would provide a easy way to make custom classes sortable that would be nicer for end users (not writing key functions), and possibly more performant in the "usual" case. In fact, it's striking me that there may well be classes that are defining the comparison magic methods not because they want the objects to "work" with the comparison operators, but because that want them to work with sort and min, and max, and... hmm, perhaps a __key__ method could even be used by the comparison operators, though that could result in pretty weird results when comparing two different types. So: has this already been brought up and rejected? Am I imagining the performance benefits? Is sorting-related functionally too special-case to deserve a protocol? Thoughts? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

20 37

f-string literals by default?
by Neil Schemenauer 05 Dec '17

05 Dec '17

I think most people who have tried f-strings have found them handy. Could we transition to making default string literal into an f-string? I think there is a smooth migration path. f-strings without embedded expressions already compile to the same bytecode as normal string literals. I.e. no overhead. The issue will be literal strings that contain the f-string format characters. We could add a future import, e.g. from __future__ import fstring_literals that would make all literal strings in the module into f-strings. In some future release, we could warn about literal strings in modules without the future import that contain f-string format characters. Eventually, we can change the default. To make migration easier, we can provide a source-to-source translation tool. It is quite simple to do that using the tokenizer module.

5 6

Data Classes in Kotlin
by Juancarlo Añez 05 Dec '17

05 Dec '17

I thought this might be interesting input for the discussions about "data classes" in Python: https://kotlinlang.org/docs/reference/data-classes.html I think that the use of "data class" as syntax is kind of cool, but what really matters is the semantics they chose for Kotlin. -- Juancarlo *Añez*

1 0

PEP 447: Adding type.__getdescriptor__
by Ronald Oussoren 03 Dec '17

03 Dec '17

Hi, I’m trying to get up to speed again w.r.t. PEP 447 (and to be honest, with CPython’s development workflow as its been too long since I’ve actually done work on the CPython codebase…). Let me start by recapping the PEP (<https://www.python.org/dev/peps/pep-0447/ <https://www.python.org/dev/peps/pep-0447/>>) is about: The problem I’m trying to solve is that super.__getattribute__ basicy assumes looking at the __dict__ of classes on the MRO is all that’s needed to check if those classes provide an attribute. The core of both object.__getattribute__ and super.__getattribute__ for finding a descriptor on the class is basically (from the PEP): def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple) for base in mro: assert isinstance(base, type) try: return base.__dict__[name] except KeyError: pass return None Note that the real implementation in in C, and accesses base.__dict__ directly instead of through Python’s attribute lookup (which would lead to infinite recursion). This is problematic when the class is dynamically populated, as the descriptor may not yet be present in the class __dict__. This is not a problem for normal attribute lookup, as you can replace the __getattribute__ method of the class to do the additional work (at the cost of having to reimplement all of __getattribute__). This is a problem for super() calls though, as super will unconditionally use the lookup code above. The PEP proposed to replace “base.__dict__[name]” by “base.__class__.__getdescriptor__(name)” (once again, in C with direct access to the two slots instead of accessing them through normal attribute lookup). I have two open questions about this PEP: 1) Last time around Mark Shannon worried that this introduces infinite recursion in the language itself (in my crummy summary, please read this message to get the real concern <https://mail.python.org/pipermail/python-dev/2015-July/140938.html <https://mail.python.org/pipermail/python-dev/2015-July/140938.html>>). Is this truly a problem? I don’t think there is a problem, but I’m worried that I don’t fully understand Mark’s concerns. 2) PEP 487 introduced __init_subclass__ as a class method to avoid having to write a metaclass for a number of use cases. My PEP currently does require a metaclass, but it might be nicer to switch to a regular class method instead (like __init_subclass__). My primary usecase for this PEP is PyObjC, which does populate the class dictionary on demand for performance and correctness reasons and PyObC therefore currently includes a replacement for super. For PyObjC’s implementation making __getdescriptor__ a classmethod would be a win as this would avoid creating yet another level of metaclasses in C code (PyObjC already has a Python class and metaclass for every Objective-C class, PEP 447 would require the introduction of a metaclass for those metaclasses). BTW. Two other open issues w.r.t. this PEP are forward porting the patch (in issue 18181) and performing benchmarks. The patch doesn’t apply cleanly to the current tip of the tree, I’ll try to update it tomorrow. And with some luck will manage to update PyObjC’s implementation as well to validate the design. Ronald

2 9

Provide a way to import module without exec body
by Neil Schemenauer 03 Dec '17

03 Dec '17

I have been working on reducing Python statup time. It would be nice if there was some way to load a module into memory without exec of its body code. I'm sure other people have wished for this. Perhaps there could be a new special function, similar to __import__ for this purpose. E.g. __load_module__(). To actually execute the module, I had the idea to make module objects callable, i.e. tp_call for PyModule_Type. That's a little too cute though and will cause confusion. Maybe instead, add a function attribute to modules, e.g. mod.__exec__(). I have a little experimental code, just a small step: https://github.com/nascheme/cpython/tree/import_defer_exec We need importlib to give us the module object and the bytecode without doing the exec(). My hackish solution is to set properties on __spec__ and then have PyImport_ImportModuleLevelObject() do the exec().

7 10

Re: [Python-ideas] Provide a way to import module without exec body
by Brett Cannon 03 Dec '17

03 Dec '17

On Fri, 1 Dec 2017 at 10:11 Neil Schemenauer <neil(a)python.ca> wrote: > On 2017-12-01, Chris Angelico wrote: > > Can you elaborate on where this is useful, please? > > Introspection tools, for example, might want to look at the module > without executing it. Also, it is a building block to make lazy loading > of modules work. As Nick points out, importlib can do this already. > > Currently, the IMPORT_NAME both loads the code for a module and also > executes it. The exec happens fairly deep in the guts of importlib. > This makes import.c and ceval.c mutually recursive. The locking gets > complicated. There are hacks like _call_with_frames_removed() to hide > the recursion going on. > > Instead, we could have two separate opcodes, one that gets the module > but does not exec it (i.e. a function like __import__() that returns a > future) and another opcode that actually does the execution. Figuring > out all the details is complicated. > > Possible benefits: > > - importlib is simpler > > - reduce the amount of stack space used (removing recursion by > "continuation passing style"). > > - makes profiling Python easier. Tools like valgrind get confused > by call cycle between ceval.c and import.c. > > - easier to implement lazy loading of modules (not necessarily a > standard Python feature but will make 3rd party implementations > cleaner) > > I'm CCing Brett as I'm sure he has thoughts on this, given his intimate > knowledge of importlib. To me, it seems like __import__() has a > terribly complicated API because it does so many different things. > I have always assumed the call signature for __import__() was because the import-related opcodes pushed so much logic into the function instead of doing it in opcodes (I actually blogged about this at https://snarky.ca/if-i-were-designing-imort-from-scratch/). Heck, the thing takes in locals() and yet never uses them (and its use of globals() is restricted to specific values so it really doesn't need to be quite so broad). Basically I wished __import__() looked like importlib.import_module(). > > Maybe two opcodes is not even enough. Maybe we should have one to > resolve relative imports (i.e. import.c:resolve_name), one to load but > not exec a module given its absolute name (i.e. _find_and_load() > without the exec), one to exec a loaded module, one or more to handle > the horror of "fromlist" (i.e. _handle_fromlist()). > I have always wanted to at least break up getting the module and fromlist as separate opcodes, so +1 for that. Name resolution could potentially be done as an opcode as it relies on execution state pulled from the globals of the module, but the logic also isn't difficult so +0 for that (i.e. making an opcode that calls something more like importlib.import_module() is more critical to me than eliminating the 'package' argument to that call, but I don't view it as a bad thing to have another opcode for that either). As for the completely separating the loading and execution, I don't have a need for what's being proposed so I don't have an opinion. I basically made sure Eric Snow structured specs so that lazy loading as currently supported works so I got what I wanted for basic lazy importing (short of the PyPI package I keep talking about writing to add a nicer API around lazy importing :) . -Brett > > Regards, > > Neil >

3 3