Mailman 3 December 2015 - Python-ideas

Have max and min functions ignore None
by Franklin? Lee 30 Dec '15

30 Dec '15

What do people think about having `max` and `min` ignore `None`? Examples: max(1, None) == 1 min(1, None) == 1 max([1, None]) == 1 max(None, None) == max() or max(None, None) == max([]) (The last one currently throws two different errors.) This change would allow one to use `None` as a default value. For example, def my_max(lst): best = None for x in lst: best = max(best, x) return best Currently, you would initialize `best` to the first element, or to float('-inf'). There are more complicated examples, which aren't just replications of `max`'s functionality. The example I have in mind wants to update several running maximums during iteration. I know that there are other ways to do it (having given one above). What if this becomes _the_ obvious way to do it? I'm concerned about this silencing some bugs which would have been caught before. I'm also worried about whether it would make sense to people learning Python. I'm less concerned about custom types which allow comparisons to `None`, because I don't understand why you would want that, but you can change my mind.

8 11

Re: [Python-ideas] Using functools.lru_cache only on some arguments of a function
by Franklin? Lee 30 Dec '15

30 Dec '15

(Replying to an email I didn't get in my inbox. I hope this goes through correctly.) On Friday, December 4, 2015 at 4:44:47 PM UTC-5, Bill Winslow wrote: > This is a question I posed to reddit, with no real resolution: https://www.reddit.com/r/learnpython/comments/3v75g4/using_functoolslru_cac… > The summary for people here is the following: > Here's a pattern I'm using for my code: > def deterministic_recursive_ calculation(input, partial_state=None): > condition = do_some_calculations(input) > if condition: > return deterministic_recursive_calculation(reduced_input, some_state) > Basically, in calculating the results of the subproblem, the subproblem can be calculated quicker by including/sharing some partial results from the superproblem. (Calling the subproblem without the partial state still gives the same result, but takes substantially longer.) Solutions: 1. Rewrite your recursive function so that the partial state is a nonlocal variable (in the closure), and memoize the recursive part. 2. Attach the state to the function, rather than passing it as a parameter. (Also suggested in this comment: https://www.reddit.com/r/learnpython/comments/3v75g4/using_functoolslru_cac…) 3. Wrap the state in a class for which __eq__ is defined to say that all instances of it are the same, and __hash__ always returns a constant (say, the id of the class). I think #1 is the best: The state is a property of the current recursive execution. You make a function which solves the problem, and the recursive function is an inner function of the solver. The state will be lost when you finish computing the answer. If you want to keep the partial state even after the recursion finished, then #2 is the right answer: the state is a property of the solver. I feel that #3 is icky and heavy-handed. The function won't be semantically responsible for the state. Hacking/extending lru_cache is saying the wrong thing about what the partial state is. Cached information (which is what your state is) isn't really an argument to the function as a mathematical input-output box. Also, if you're calling this function multiple times, you as the caller shouldn't be expected to keep track of the cache, so you shouldn't be expected to store it (in global scope). But if it's reusable, it should be stored, so the function should be responsible for storing it. If you want to keep the lru_cache, too, then you can store the recursive function as an attribute. It will even keep itself and your partial state in its closure. Sample code so you can confirm that the state is stored between calls. Just call solve(50) twice in a row, and compare the times. # WARNING: I don't recommend x > 100. # I've killed two interpreters while messing with this. def solve(x): try: rec = solve.rec # To trigger the exception the first time. except: @lru_cache(None) def rec(n, m): return sum( rec(i, j) for i in reversed(range(n)) for j in reversed(range(m))) solve.rec = rec return rec(x, x)

2 4

Have max and min functions ignore None
by Gerald Britton 29 Dec '15

29 Dec '15

On *Tue Dec 29 06:22:58 EST 2015, Stephen D'Aprano wrote:* > So that's three perfectly reasonable behaviours: max(x, None) is an error > and should raise; > max(x, None) ignores None and returns x; > max(x, None) is unknown or missing and returns None > (or some other sentinel representing NA/Missing/Unknown). For comparison's sake, SQL ignores NULL when doing MAX: e.g. select max(val) from (values (1),(null)) v(val) returns 1 In Python, None is sorta-kinda a bit like NULL in SQL, so one could make the argument that None should be handled similarly in min and max. OTOH I wouldn't want to see Python implement 3-valued logic. -- Gerald Britton, MCSE-DP, MVP LinkedIn Profile: http://ca.linkedin.com/in/geraldbritton

3 2

Fwd: Re: Unambiguous repr for recursive objects
by Guido van Rossum 28 Dec '15

28 Dec '15

--Guido (mobile) ---------- Forwarded message ---------- From: <python-ideas-owner(a)python.org> Date: Dec 26, 2015 9:33 AM Subject: Re: [Python-ideas] Unambiguous repr for recursive objects To: <gvanrossum(a)gmail.com> Cc: Your message has been rejected, probably because you are not subscribed to the mailing list and the list's policy is to prohibit non-members from posting to it. If you think that your messages are being rejected in error, contact the mailing list owner at python-ideas-owner(a)python.org. ---------- Forwarded message ---------- From: Guido van Rossum <gvanrossum(a)gmail.com> To: Serhiy Storchaka <storchaka(a)gmail.com> Cc: Python-Ideas <python-ideas(a)python.org> Date: Sat, 26 Dec 2015 09:33:37 -0700 Subject: Re: [Python-ideas] Unambiguous repr for recursive objects I disagree. We should not take this guideline too literally. The dots are easily understood and nobody has been fooled by a list containing an ellipsis yet. (Also, isn't the repr of an ellipsis the string 'Ellipsis'?) --Guido (mobile)

9 16

Missing Core Feature: + - * / | & do not call __getattr__
by Stephan Sahm 28 Dec '15

28 Dec '15

Dear all, I just stumbled upon a very weird behaviour of python 2 and python 3. At least I was not able to find a solution. *The point is to dynamically define __add__, __or__ and so on via __getattr__* (for example by deriving them from __iadd__ or similar in a generic way). However this very intuitive idea is currently NOT POSSIBLE because * - * / & | and so on just bypass this standard procedure. I found two stackoverflow contributions stating this: http://stackoverflow.com/questions/11629287/python-how-to-forward-an-instan… http://stackoverflow.com/questions/33393474/lazy-evaluation-forward-operati… Neither the mentioned posts, nor I myself can see any reason why this is the way it is, nor how the operators are actually implemented to maybe bypass this unintuitive behaviour. Any help? Comments? Ides? best, Stephan

9 19

Adding collections.abc.Ordered
by Amir Rachum 27 Dec '15

27 Dec '15

I am suggesting the addition of a collections abstract base class called "Ordered". Its meaning is that a collection's iteration order is part of its API. The bulk of this mail describes a use case for this. The reason I believe that such abstract base class is required is that there is no way to test this behavior in a given class. An ordered collection has the exact same interface as an unordered collection (e.g, dict and OrderedDict), other than a _promise_ of the API that the order in which this collection will be iterated has some sort of meaning (In OrderedDict, it is the order in which keys were added to it.) As examples, set, frozenset, dict and defaultdict should *not* be considered as ordered. list, OrderedDict, deque and tuple should be considered ordered. Before I dive into the use case I am presenting, I would like to point out that a similar discussion was already done on the subject (suggesting at first that OrderedDict would abstract-subclass from Sequence) at the following thread* - http://code.activestate.com/lists/python-ideas/29532/. That thread was abandoned largely from a lack of a clear use case, which I hope will be more clear in this suggestion. I am working on package called basicstruct ( https://pypi.python.org/pypi/basicstruct). The way it works is that you define an object that inherits from basicstruct.BasicStruct, define __slots__ , and automatically get behaviors such as a nice __init__, __str__, comparison functions, attribute access, dict-like access, pickleability, etc. It is similar to namedtuple, except that it is mutable. In the Python documentation, the following is said regarding __slots__: Any non-string iterable may be assigned to __slots__. Mappings may also be used; however, in the future, special meaning may be assigned to the values corresponding to each key. Here's how the current__init__ method of BasicStruct looks like: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): setattr(self, key, None) Notice the following line: arg_pairs = zip(self.__slots__, args) It assumes that __slots__ defines attributes in a certain order. So as a use I would expect that the following code class MyStruct(BasicStruct): __slots__ = ('x', 'y') MyStruct(0, 1) ... will create a struct in which x is 0 and y is 1. However, if I define __slots__ as a set, or dict, the result is undefined. class MyStruct(BasicStruct): __slots__ = {'x', 'y'} # No order is defined here MyStruct(0, 1) # Which is which? So, In BasicStruct's __init__ method it is required to test whether __slots__ was defined with a collection that has a meaningful order. If it was _not_, using non-keyword arguments in __init__ should be forbidden, since the order of __slots__ is arbitrary. Here is how I wish the code would look like: class BasicStruct(object): """Class for holding struct-like objects.""" __slots__ = () # should be extended by deriving classes def __init__(self, *args, **kwargs): ordered = isinstance(self.__slots__, Ordered) if args and not ordered: raise ValueError("Can't pass non-keyword arguments to {}, since " "__slots__ was declared with an unordered " "iterable.".format(self.__class__.__name__)) arg_pairs = zip(self.__slots__, args) for key, value in chain(arg_pairs, six.iteritems(kwargs)): setattr(self, key, value) for key in self.__slots__: if not hasattr(self, key): setattr(self, key, None) Thanks, Amir Rachum * The author of the original thread is Ram Rachum. To avoid some confusion - yes, we are related - Ram is my big brother. It's just so happens that I was looking around for this issue and found that he did so in the past as well.

14 48

Re: [Python-ideas] Adding collections.abc.Ordered
by Wes Turner 26 Dec '15

26 Dec '15

On Dec 26, 2015 8:11 PM, "Wes Turner" <wes.turner(a)gmail.com> wrote: > > * collections.abc.Ordered > * collections.abc.Reversible > * collections.abc.Infinite [...] > > * collections.abc.Sorted ? > * collections.abc.Recursive ? > > Rationale: > > These are all attributes of collections that would allow us to reason about [complexity, types, runtime]? > > [While someone is at it, annotating functions and methods with complexity class URI fragments accessible at runtime could also be useful for [dynamic programming].] > > * Ordered is justified by this thread > * Reversible is distinct from Ordered (because Infinite sequences) > * Infinite is the distinction between Ordered and Reversible > > * collections.abc.Sorted would be a useful property to keep track of (because then you don't have to do element-wise comparisons for each collection member) > > ... > > * collections.abc.Recursive could also be a useful property to mixin [again for dynamic programming] > > https://en.wikipedia.org/wiki/Dynamic_programming https://en.wikipedia.org/wiki/Goal_programming There are more properties of sequences listed here; IDK if this is out of scope for OT: https://en.wikipedia.org/wiki/Sequence#Formal_definition_and_basic_properti… Vague use case: algorithmic selection / unhalting-avoidance with combinatorial data/logic sequences. [e.g. find the fastest halting solution] Practically, Ordered is a property of various types [e.g. is this a poset or not]. There is currently no way to check for .ordered with hasattr. These properties are things we currently keep in mind (some of our 7±2 things) and haven't yet figured out how to annotate with and access at runtime. > > On Dec 26, 2015 7:34 PM, "Wes Turner" <wes.turner(a)gmail.com> wrote: >> >> * collections.abc.Ordered >> * collections.abc.Reversible >> * collections.abc.Infinite [...] >> >> * collections.abc.Sorted ? >> >> On Dec 26, 2015 7:24 PM, "Andrew Barnert via Python-ideas" < python-ideas(a)python.org> wrote: >>> >>> On Dec 26, 2015, at 16:09, Serhiy Storchaka <storchaka(a)gmail.com> wrote: >>> > >>> >> On 27.12.15 01:05, Guido van Rossum wrote: >>> >> There is a precedent for declaring that a method isn't implemented: >>> >> __hash__. The convention is to set it to None in the subclass that >>> >> explicitly doesn't want to implement it. The __subclasshook__ in >>> >> collections.Hashable checks for this. The pattern is also used for >>> >> __await__. >>> > >>> > Yes, this was the first thing that I tried, but it doesn't work, as shown in my example in issue25864. This is yet one thing that should be fixed in Reversible. >>> >>> As Guido pointed out in your typehinting #170, that isn't a bug with typing.Reversible. You can't use typing types in runtime type tests with issubclass. That isn't supposed to work, and the fact that it often kind of does work is actually a bug that he's fixing. So the fact that it doesn't work in this case is correct. >>> >>> That also means your attempted solution to this thread is wrong; typing.Reversible cannot be used as a substitute for collections.abc.Ordered. >>> >>> > May be we have to use this idiom more widely, and specially handle assigning special methods to None. The error message "'sometype' can't be reverted" looks better than "'NoneType' is not callable". >>> >>> I agree with this. A new collections.abc.Reversible (interposed between Iterable and Sequence) would be a potential substitute for Ordered, and would have this problem, which would be solvable by treating __reversed__ = None specially, just like __hash__ = None. And I'm pretty sure it would come up in practice (see issue 25864). And once we've got two or three special methods doing this instead of one, making it more general does sound like a good idea. So, if we need Reversible as a substitute for Ordered, then I think we want the general "is None" test. >>> >>> But I'm still not sure Reversible is a good substitute for Ordered (again, consider an infinitely long collection, or just a lazy proxy that doesn't compute values until needed and doesn't know its length in advance--they're clearly ordered, and just as clearly not reversible), and I'm not sure we actually need either Reversible or Ordered in the first place. >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas(a)python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Unambiguous repr for recursive objects
by Serhiy Storchaka 26 Dec '15

26 Dec '15

Currently repr() for recursive object replaces its recurred representation wish a placeholder containing "...". For list this is "[...]": >>> a = [1, 2] >>> a.append(a) >>> a [1, 2, [...]] For dict this is "{...}": >>> d = {1: 2} >>> d[3] = d >>> d {1: 2, 3: {...}} For OrderedDict (Python implementation or 3.4-) this is just "...": >>> from collections import OrderedDict >>> od = OrderedDict({1: 2}) >>> od[3] = od >>> od OrderedDict([(1, 2), (3, ...)]) The problem is that "[...]", and "{...}", and just "..." are valid Python expressions and above representations can be evaluated to different objects. I propose to use uniform and unambiguous non-evaluable representation for recursive objects. I have two ideas: 1. "<...>". Plus: this is as short as "[...]" and "{...}". Minus: we loss even a little tip about the type of recurred object. 2. Use the default implementation, object.__repr__(). E.g. "<list object at 0xb7111498>". Plus: we get even more information than before. Not just exact name of the type, but the identifier of the object. This can be useful in the case of complex structure containing a number of potentially recursive objects. Minus: it is longer.

3 3

Buffering iterators?
by Franklin? Lee 22 Dec '15

22 Dec '15

(This would be a lot of work that I wouldn't know how to do, but is it worth thinking about? Maybe it's already been done at the level necessary. Also, this is a proposal for the sake of theoretical optimization, in case you don't like that, and it will require a lot of work in a lot of code everywhere. As I see it, even if it's possible and desirable to do this, it would take years of work and testing to make it beneficial.) The move from Python 2 (disclaimer: which I barely touched, so I have little sentimental attachment) to Python 3 resulted in many functions returning iterators instead of lists. This saves a lot of unnecessary memory when iterating, say, over the indices of a large list, especially if we break in the middle. I'm wondering, though, about a step backwards: generating values before they're needed. The idea is based on file buffered reading and memory prefetching (https://en.wikipedia.org/wiki/Synchronous_dynamic_random-access_memory#DDR_…). In fact, I'm hoping to take advantage of such things. For example, in `sum(lst[i] * i for i in range(10000))`, `sum` will exhaust the iterator, so it can ask the generator to return buffers, and it will internally read the elements off the lists. It would be the responsibility of the iterator to decide whether to respect the request, and to determine the size of the buffer. It would be the responsibility of the consumer to request it, and consumers should only request it if they think they'll almost definitely consume a lot at a time. The idea is, especially for complex nested iterators, instead of running A B C A B C A B C..., where each is the code for generating a next thing from the previous, that the interpreter runs A A A A A..., B B B B B..., C C C..., which could mean a lot more memory locality in both instructions and objects. There's the possibility that a function has side-effects, so buffering would have different semantics than normal. There's also the possibility that getting the next element is complex enough that it wouldn't help to buffer. If the iterator can't tell, then it should just not buffer. Here's an obnoxious example of where you can't tell: def f(i): return i s = 0 for i in (f(x) for x in range(100)): s += f(i) def f(x): return x + s In fact, all Python function calls are probably unsafe, including with operators (which can be legally replaced during the iteration). Well, `map` and `filter` are possible exceptions in special cases, because the lookup for their function is bound at the call to `map`. It's usually safe if you're just using reversed, enumerate, builtin operators, dict views, etc. on an existing data structure, as long as your iteration doesn't modify entries, unlike so: for i, x in enumerate(reversed(lst)): lst[i+1] = x But I'm looking toward the future, where it might be possible for the interpreter to analyze loops and functions before making such decisions. Then again, if the interpreter is that smart, it can figure out where and when to buffer without adding to the API of iterators. Anyway, here's an idea, how it might be helpful, and how it might not be helpful.

3 2

Re: [Python-ideas] Optimizing global names via dict references
by Franklin? Lee 20 Dec '15

20 Dec '15

(Sent this from the wrong email and it got rejected. Sorry, Victor, for the double post.) On Dec 20, 2015 2:03 AM, "Franklin?" <redacted> wrote: > > Oops, that was supposed to private. No point now. > > Anyway, after I read Tim's remark, I realized that I overestimated the cost of dict lookups. It's not the complexity, but the constant factor in terms of branches and dereferences. It means that if I want to make an improvement, it would have to be very optimized C. They were discussing tricks like having a cell consider itself as its own parent, just to avoid a branch, which is not the level at which I was thinking. > > I think that normal dict operations won't have to slow down, except resize/destruction. I don't use registry, so there is no O(n) notification. This is an important difference from Neil Toronto's original method. > > Reading PEP 266, and rethinking your own work, I'm now considering a separate idea for temporary registries. But I think it'd be reaching into JIT territory, and I'm not confident at all in my JIT knowledge. I can't say that this idea would be an improvement. I'm just putting this out there in the hope that it can make some sense and inspire someone to have a real idea. > > The idea is, when code is running, if a function is called (within a loop?), load its global names into an array and register them with the globals/builtins (which can be done with user dicts by temporarily swapping out their get/set/del). Then replace the function with one that does LOAD_CONSTANT (like in FAT Python) and replace its function calls with "compile the functions the same way before calling" bytecodes. > > This array is for the whole stack. As I imagine it, it will be used during an execution. So for the REPL, it can be a new array per eval, or just globally. > > The trick is, each name will be added to the array once, so the registry will slow down the normal dict operations but each dict change only needs (at most) one notification per stack. > > This can be made to (only) slow down the dicts which are used as scopes, by dynamically swapping out its function pointers upon (temporary?) conversion. > > (You sent this as private?) > > On Dec 19, 2015 7:09 PM, "Victor Stinner" <victor.stinner(a)gmail.com> wrote: >> >> >> Le 19 déc. 2015 21:07, "Franklin? Lee" <leewangzhong+python(a)gmail.com> a écrit : >> > Tim Peters points out how it might be more expensive than a lookup. >> >> Your whole idea rely on the assumption than a dict lookup (two lookups for builtins) is slow. Remember that a dict lookup has a complexity of O(1)! That's why I suggested you to start to benchmark, especially know the time in nanoseconds of a dict lookup. >> >> > He ended up using versioning, because the notification was expensive. >> >> Yeah I also began with notification with a registry of functions before moving to versionning: >> http://faster-cpython.readthedocs.org/readonly.html >> >> Versionning is simple to implement and doesn't make dict operations slower. >> >> Victor

2 1