Mailman 3 September 2013 - Python-ideas

An exhaust() function for iterators
by Clay Sweetser 12 Oct '13

12 Oct '13

Currently, several strategies exist for exhausting an iterable when one does not care about what the iterable returns (such as when one merely wants a side effect of the iteration process). One can either use an empty for loop: for x in side_effect_iterable: pass A throwaway list comprehension: [x for x in side_effect_iterable] A try/except and a while: next = side_effect_iterable.next try: while True: next() except StopIteration: pass Or a number of other methods. The question is, which one is the fastest? Which one is the most memory efficient? Though these are all obvious methods, none of them are both the fastest and the most memory efficient (though the for/pass method comes close). As it turns out, the fastest and most efficient method available in the standard library is collections.deque's __init__ and extend methods. from collections import deque exhaust_iterable = deque(maxlen=0).extend exhaust_iterable(side_effect_iterable) When a deque object is initialized with a max length of zero or less, a special function, consume_iterator, is used instead of the regular element insertion calls. This function, found at http://hg.python.org/cpython/file/tip/Modules/_collectionsmodule.c#l278, merely iterates through the iterator, without doing any work allocating the object to the deque's internal structure. I would like to propose that this function, or one very similar to it, be added to the standard library, either in the itertools module, or the standard namespace. If nothing else, doing so would at least give a single *obvious* way to exhaust an iterator, instead of the several miscellaneous methods available. -- "Evil begins when you begin to treat people as things." - Terry Pratchett

6 6

pprint in displayhook
by Serhiy Storchaka 04 Oct '13

04 Oct '13

What are you think about using pprint.pprint() to output the result of evaluating an expression entered in an interactive Python session (and in IDLE)?

11 25

Have os.unlink remove junction points on Windows
by Kim Gräsman 29 Sep '13

29 Sep '13

Hi all, I posted this bug a while back, but haven't had any feedback on it, so I figured I'd open discussion here: http://bugs.python.org/issue18314 I want to have os.unlink recognize NTFS junction points (a prequel to proper symbolic links) and remove them transparently. Currently, os.unlink on a path to a junction point will fail with a WindowsError and the error code 5, access denied. Does this sound controversial, or would anybody be interested in reviewing a patch to this effect? Thanks, - Kim

6 20

String formatting
by Ram Rachum 28 Sep '13

28 Sep '13

Any reason why string formatting using % doesn't work when the list of arguments is in a list rather than a tuple?

3 2

AST Hash
by anatoly techtonik 28 Sep '13

28 Sep '13

Hi, We need a checksum for code pieces. The goal of the checksum is to reliably detect pieces of code with absolutely identical behaviour. Borders of such checksum can be functions, classes, modules,. Practical application for such checksums are: - detecting usage of recipes and examples across PyPI packages - detecting usage of standard stdlib calls - creating execution safe serialization formats for data - choosing class to deserialize data fields of the object based on its hash - enable consistent validation and testing of results across various AST tools There can be two approaches to build such checksum: 1. Code Section Hash 2. AST Hash Code Section Hash is built from a substring of a source code, cut on function or class boundaries. This hash is flaky - whitespace and comment differences ruin it, even when behaviour (and bytecode) stays the same. It is possible to reduce the effect of whitespace and comment changes by normalizing the substring - dedenting, reindenting with 4 spaces, stripping empty lines, comments and trailing whitespace. And it still will be unreliable and affected by whitespace changes in the middle of the string. Therefore a 2nd way of hashing is more preferable. AST Hash is build on AST. This excludes any comments, whitespace etc. and makes the hash strict and reliable. This is a canonical Default AST Hash. There are cases when Default AST Hash may not be enough for comparison. For example, if local variables are renamed, or docstrings changed, the behaviour of a function may not change, but its AST hash will. In these cases additional normalization rules apply. Such as changing all local variable names to var1, var2, ... in order of appearance, stripping docstrings etc. Every set of such normalization rules should have a name. This will also be the name of resulting custom AST Hash. Explicit naming of AST Hashes and hardlinking of names to rules that are used to build them will settle common ground (base) for AST tools interoperability and research papers. As such, it most likely require a separate PEP. -- anatoly t.

6 8

Re: [Python-ideas] Should we improve `dir`?
by David Halter 26 Sep '13

26 Sep '13

Sorry for answering so late, but I've stayed in a very rural area of Afghanistan and enjoyed my life :-) I also realized that this discussion has been removed from python-ideas, sorry! 2013/9/15 Nick Coghlan <ncoghlan(a)gmail.com> > On 15 September 2013 16:06, David Halter <davidhalter88(a)gmail.com> wrote: > > > > 2013/9/15 Nick Coghlan <ncoghlan(a)gmail.com> > >> > >> If introspection tools want to show all the operations available *on the > >> class*, then they need to include "dir(type(cls))" as well. So there > may be > >> a legitimate feature request for a new section in the pydoc output > showing > >> "class only" methods and attributes. > > > > > > How about adding a keyword argument to `dir`: ``dir(object, > > with_class_methods=False)``? > > > > I get that there are compatibility issues with changing the default `dir` > > functionality. But at the same time adding such an option could make it > > easier for beginners, why type attributes are not being listed (because > one > > could read that in the `dir` docstring). > > It's actually the metaclass methods/attributes that are missing. The > trick with dir is it *stops at the class*, and thus always leaves out > the metaclass. While in a important sense "classes are just objects", > attribute access is a critical area where they're *different* from > most other objects, because they play different roles in the > descriptor protocol. > > That means the question is whether it is worth adding an appropriate > flag to dir(), over updating introspection tools (like IDLE's tab > completion as Chris points out) to consider "dir(type(cls))" when > appropriate. > > "dir" currently works roughly as follows for instances: > > - check the instance > - check the class MRO > > And for classes: > > - check the class MRO > > If an "include_metaclass" flag is added, then setting it to True has > an obvious meaning for classes: > > - check the class MRO > - check the metaclass MRO > > But what does "include_metaclass=True" mean for instances? You can't > access metaclass attributes and methods from an instance - the > attribute lookup only traverses one step. So, it could be reasonable > to have "include_metaclass=True" do nothing for instances, and only > change dir() behaviour for classes. > Good point. I haven't thought about this, but an "include_metaclass" option for dir can also be quite confusing (in the case of instances). > On the other hand, if the flag was called "include_class", then it > would need to be tri-valued: > > None: use appropriate default based on the kind of object > True: default for instances, forces inclusion of the metaclass MRO for > classes > False: default for classes, forces omission of the class MRO (and > thus all descriptors) for instances > > Alternatively, if we don't change dir() at all and just document that > getting a complete list of attributes means doing "sorted(set(dir(obj) > + dir(type(obj))", we'd have something that works for all versions of > Python, rather than something that was only available in 3.4+: > Yes, IMHO that's the least we should do. But I would strongly suggest to adjust the `dir` method docstrings (not only the online docs). I think that the current documentation really needs improvement (it is quite confusing now). >>> def full_dir(obj): > ... return sorted(set(dir(obj) + dir(tyape(obj)))) > ... > >>> len(set(full_dir(1)) - set(dir(1))) > 0 > >>> len(set(full_dir(int)) - set(dir(int))) > 19 > >>> len(set(dir(type)) - set(dir(object))) > 19 > > That's why my preference is for the latter approach - this isn't new > behaviour, and it's introspection tools that don't handle metaclasses > properly that need updating, rather than changing the dir() builtin. Well, I would still opt for changing dir, but I can understand that that would cause serious backwards-compatibility issues. If you would do that, it would be something for a Python 4 (and we're not even close to that). So for now that really leaves us with documenting it better. I don't really like the "include_metaclass" option. Maybe your "include_class" might make a little bit more sense. But even that one would complicate things. Cheers! Dave

1 0

`OrderedDict.sort`
by Ram Rachum 25 Sep '13

25 Sep '13

What do you think about providing an `OrderedDict.sort` method? I've been using my own `OrderedDict` subclass that defines `sort` for years, and I always wondered why the stdlib one doesn't provide `sort`. I can write the patch if needed. Thanks, Ram.

13 31

Subject: Re: `OrderedDict.sort`
by Vernon D. Cole 25 Sep '13

25 Sep '13

> > > > And yes, NaN is a problem, but it's exactly the same problem it is > > everywhere else in Python. > > I was serious about wanting to know how dictionaries handle NaN as a > key. Is it a special case? The obvious way of implementing it would > conclude it is a hash collision but not a match. I notice that > Decimal('NaN') and float nan don't match each other (as do any other > float/Decimal with the same value) but they do both work as dictionary > keys. > NaN is, by definition, never equal to another NaN, which is why the following happens: >>> nan = float('NAN') >>> n2 = nan >>> n2 == nan False >>> n2 is nan True It turns out that many other things which I never thought about before can be dictionary keys... >>> d = {'a':1, nan: 2} >>> d[n2] 2 >>> d[NotImplemented] = 3 >>> d[...] = 4 >>> d[None] = 5 >>> d[True] = 6 >>> d[False] = 7 >>> d {'a': 1, nan: 2, False: 7, True: 6, NotImplemented: 3, Ellipsis: 4, None: 5}

2 1

+1 button/counter for bugs.python.org
by anatoly techtonik 25 Sep '13

25 Sep '13

Does anybody think it is a good idea to personally approve good issues and messages on bugs.python.org? If yes, should it be a Google's +1 (easier to add), or a pythonic solution for Roundup? -- anatoly t.

10 12

Indicate if an iterable is ordered or not
by Eric Snow 25 Sep '13

25 Sep '13

Iterables are not necessarily ordered (e.g. dict vs. OrderedDict). Sequences are but Sets aren't. I'm not aware of any good way currently to know if an arbitrary iterable is ordered. Without an explicit indicator of ordered-ness, you must know in advance for each specific type. One possible solution is an __isordered__ attribute (on the class), set to a boolean. The absence of the attribute would imply False. Such an attribute would be added to existing types: * collections.abc.Iterable (default: False) * list (True) * tuple (True) * set (False) * dict (False) * collections.OrderedDict (True) * ... Thoughts? -eric

6 10