Mailman 3 October 2013 - Python-ideas

Another "little" combination, permutation, iterator, yield from, example.
by Ron Adam 18 Oct '13

18 Oct '13

On 10/13/2013 02:02 PM, Tim Peters wrote:> [MRAB, posts a beautiful solution] > > I don't really have a use for this, but it was a lovely programming > puzzle, so I'll include an elaborate elaboration of MRAB's algorithm > below. And that's the end of my interest in this;-) A what the heck... :-) This is what I came up with which works like MRAB's by removing the first element and using recursion on the rest. (It's how you do it in lisp or scheme.) It's just two generators working together. You can get items with specified length by using a list comprehension or filter. (Disclaimer... I didn't do any more testing (performance or otherwise) than you see below.) Cheers, Ron Adam def combine_with_all(x, seq): """Combine item x to every item in seq.""" for s in seq: if (not (isinstance(s, list))): s = [s] yield s + [x] def unique_sets(seq): """Return all unique combinations of elements in seq.""" if len(seq) == 0: return [] first, rest = seq[0], seq[1:] yield [first] yield from unique_sets(rest) yield from combine_with_all(first, unique_sets(rest)) ### EXAMPLES ### for args in [('x', []), ('x', [1, 2]), ('x', [[1], [2]]), ('x', [[1, 2], 3]), ('x', [[1, 2], [3]]), ('x', ['abc']), ('x', ['ab', 'c'])]: print(list(combine_with_all(*args))) # [] # [[1, 'x'], [2, 'x']] # [[1, 'x'], [2, 'x']] # [[1, 2, 'x'], [3, 'x']] # [[1, 2, 'x'], [3, 'x']] # [['a', 'x'], ['b', 'x'], ['c', 'x']] # [['abc', 'x']] # [['ab', 'x'], ['c', 'x']] print(list(unique_sets('abc'))) # [['a'], ['b'], ['c'], ['c', 'b'], ['b', 'a'], ['c', 'a'], # ['c', 'b', 'a']] print(list(unique_sets(['abc']))) # [['abc']] print(list(unique_sets(['a', 'b', 'c']) )) # [['a'], ['b'], ['c'], ['c', 'b'], ['b', 'a'], ['c', 'a'], # ['c', 'b', 'a']] print(list (unique_sets( [(1,2), (3,4)] ))) # [[(1, 2)], [(3, 4)], [(3, 4), (1, 2)]] print(list (unique_sets( [[1,2], [3,4]] ))) # [[[1, 2]], [[3, 4]], [[3, 4], [1, 2]]] print([x for x in unique_sets(['a', 'b', 'c']) if len(x) == 2]) # [['c', 'a'], ['b', 'a'], ['c', 'b']] print([x for x in unique_sets(["a1", "b", "a2"]) if len(x) == 2]) # [['a2', 'b'], ['b', 'a1'], ['a2', 'a1']]

4 7

Extremely weird itertools.permutations
by Neil Girdhar 17 Oct '13

17 Oct '13

"It is universally agreed that a list of n distinct symbols has n! permutations. However, when the symbols are not distinct, the most common convention, in mathematics and elsewhere, seems to be to count only distinct permutations." — http://stackoverflow.com/questions/6534430/why-does-pythons-itertools-permu…. Should we consider fixing itertools.permutations and to output only unique permutations (if possible, although I realize that would break code). It is completely non-obvious to have permutations returning duplicates. For a non-breaking compromise what about adding a flag? Best, Neil

18 56

pytaint: taint tracking in python
by Felix Gröbert 17 Oct '13

17 Oct '13

Hi, I'd like to start a discussion on adding a security feature: taint tracking. As part of his internship, Marcin (cc) has been working on a patch to cpython-2.7.5 which is available online. We also published a design document and slides. https://github.com/felixgr/pytaint The idea behind taint tracking (or taint checking) is that we mark ('taint') untrusted data and prevent the programmer from using it in sensitive places (called sinks). A standard use case would be in a web application, where data extracted from HTTP requests is tainted and a database connection is sensitive sink. In other words: objects returned by http request have a property indicating taint, and when one of them is passed to database connection, a TaintException is raised. The idea itself is not new (Ruby and Perl have it; there are also some python libraries floating around) and pretty much noone uses it - however with a few improvements, it can be made viable. Firstly, we introduce different kinds of taint (motivation: a string may be attack vector for many classes of attacks - e.g. XSS, SQLi - and we need different escaping for that). Secondly, we allow to easily apply it to existing software - a programmer can simply write a config file specifying taint sources, sensitive sinks and taint cleaners, and enable tracking by adding one line to his app. We think it's a very useful feature for developing most of webapps and other security-sensitive application in Python, any thoughts on this? Thanks, Felix

9 15

Fwd: Extremely weird itertools.permutations
by David Mertz 15 Oct '13

15 Oct '13

What would you like this hypothetical function to output here: >>> from itertools import permutations >>> from decimal import Decimal as D >>> from fractions import Fraction as F >>> items = (3, 3.0, D(3), F(3,1), "aa", "AA".lower(), "a"+"a") >>> list(permutations(items)) It's neither QUITE equality nor identity you are looking for, I think, in nonredundant_permutation(): >> "aa" == "AA".lower(), "aa" is "AA".lower() (True, False) >>> "aa" == "a"+"a", "aa" is "a"+"a" (True, True) >>> D(3) == 3.0, D(3) is 3.0 (True, False) On Fri, Oct 11, 2013 at 11:38 AM, Neil Girdhar <mistersheik(a)gmail.com>wrote: > "It is universally agreed that a list of n distinct symbols has n! > permutations. However, when the symbols are not distinct, the most common > convention, in mathematics and elsewhere, seems to be to count only > distinct permutations." — > http://stackoverflow.com/questions/6534430/why-does-pythons-itertools-permu… > . > > > Should we consider fixing itertools.permutations and to output only unique > permutations (if possible, although I realize that would break code). It is > completely non-obvious to have permutations returning duplicates. For a > non-breaking compromise what about adding a flag? > > Best, > Neil > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

9 36

An exhaust() function for iterators
by Clay Sweetser 12 Oct '13

12 Oct '13

Currently, several strategies exist for exhausting an iterable when one does not care about what the iterable returns (such as when one merely wants a side effect of the iteration process). One can either use an empty for loop: for x in side_effect_iterable: pass A throwaway list comprehension: [x for x in side_effect_iterable] A try/except and a while: next = side_effect_iterable.next try: while True: next() except StopIteration: pass Or a number of other methods. The question is, which one is the fastest? Which one is the most memory efficient? Though these are all obvious methods, none of them are both the fastest and the most memory efficient (though the for/pass method comes close). As it turns out, the fastest and most efficient method available in the standard library is collections.deque's __init__ and extend methods. from collections import deque exhaust_iterable = deque(maxlen=0).extend exhaust_iterable(side_effect_iterable) When a deque object is initialized with a max length of zero or less, a special function, consume_iterator, is used instead of the regular element insertion calls. This function, found at http://hg.python.org/cpython/file/tip/Modules/_collectionsmodule.c#l278, merely iterates through the iterator, without doing any work allocating the object to the deque's internal structure. I would like to propose that this function, or one very similar to it, be added to the standard library, either in the itertools module, or the standard namespace. If nothing else, doing so would at least give a single *obvious* way to exhaust an iterator, instead of the several miscellaneous methods available. -- "Evil begins when you begin to treat people as things." - Terry Pratchett

6 6

Add "has_surrogates" flags to string object
by Serhiy Storchaka 11 Oct '13

11 Oct '13

Here is an idea about adding a mark to PyUnicode object which allows fast answer to the question if a string has surrogate code. This mark has one of three possible states: * String doesn't contain surrogates. * String contains surrogates. * It is still unknown. We can combine this with "is_ascii" flag in 2-bit value: * String is ASCII-only (and doesn't contain surrogates). * String is not ASCII-only and doesn't contain surrogates. * String is not ASCII-only and contains surrogates. * String is not ASCII-only and it is still unknown if it contains surrogate. By default a string is created in "unknown" state (if it is UCS2 or UCS4). After first request it can be switched to "has surrogates" or "hasn't surrogates". State of the result of concatenating or slicing can be determined from states of input strings. This will allow faster UTF-16 and UTF-32 encoding (and perhaps even a little faster UTF-8 encoding) and converting to wchar_t* if string hasn't surrogates (this is true in most cases).

12 25

pprint in displayhook
by Serhiy Storchaka 04 Oct '13

04 Oct '13

What are you think about using pprint.pprint() to output the result of evaluating an expression entered in an interactive Python session (and in IDLE)?

11 25

Allow from foo import bar*
by Giampaolo Rodola' 03 Oct '13

03 Oct '13

I suppose this has already been proposed in past but couldn't find any online reference so here goes. When it comes to module constant imports I usually like being explicit it's OK with me as long as I have to do: >>> from resource import (RLIMIT_CORE, RLIMIT_CPU, RLIMIT_FSIZE) Nevertheless in case the existence of certain constants depends on the platform in use I end up doing: >>> if hasattr(resource, "RLIMIT_MSGQUEUE"): # linux only .... import resource.RLIMIT_MSGQUEUE .... >>> if hasattr(resource, "RLIMIT_NICE"): # linux only .... import resource.RLIMIT_NICE .... ...or worse, if for simplicity I'm willing to simply import all RLIMIT_* constants I'll have to do this: >>> import resource >>> import sys >>> for name in dir(resource): .... if name.startswith('RLIMIT_'): .... setattr(sys.modules[__name__], name, getattr(resource, name)) ...or just give up and use: from resource import * ...which of course will pollute the namespace with unnecessary stuff. So why not just allow "from resource import RLIMIT_*" syntax? Another interesting variation might be: >>> from socket import AF_*, SOCK_* >>> AF_INET, AF_INET6, SOCK_STREAM, SOCK_DGRAM (2, 10, 1, 2) On the other hand mixing "*" and "common" imports would be forbidden: >>> from socket import AF_*, socket, File "<stdin>", line 1 from socket import AF_*, socket ^ SyntaxError: invalid syntax; Thoughts? --- Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/

7 9