Mailman 3 July 2013 - Python-ideas

Enum and serialization
by Giampaolo Rodola' July 20, 2013

July 20, 2013

I just received this bug report in psutil: https://code.google.com/p/psutil/issues/detail?id=408 Long story short, in psutil I have defined a constant type: http://code.activestate.com/recipes/577984-named-constant-type/ https://code.google.com/p/psutil/source/browse/psutil/_common.py#31 ...behaving along these lines: >>> FOO = constant(0, 'FOO') >>> FOO 0 >>> str(FOO) 'FOO' It's a nasty problem and I'm still not sure how to solve it in psutil but this got me thinking about PEP 435 (Adding an Enum type to the Python standard library) since the two things are kind of related. I haven't properly gone through the PEP yet (I will) but I notice it doesn't talk about serialization. Has it been considered? Regards, - Giampaolo https://code.google.com/p/pyftpdlib/ https://code.google.com/p/psutil/ https://code.google.com/p/pysendfile/

5 11

Fast sum() for non-numbers
by Sergey July 19, 2013

July 19, 2013

Hello, python-ideas. Trying to cover everything in one shot, so the message is long, sorry. sum() is a great function. It is the "obvious way" to add things. Unfortunately sometimes it's slower than it could be. The problem is that code: sum([[1,2,3]]*1000000, []) takes forever to complete. Let's fix that! This problem was there since the sum() introduction, but it is almost unknown among python-beginners. When people look at sum(seq,start=0) signature they most probably expect it to be like this: def sum(seq, start = 0): for item in seq: start += item return start But it is not, because in cases like: empty = [] result = sum(list_of_lists, empty) such implementation would modify content of "empty". This use-case is known, it is checked in test_builtin.py, and explicitly mentioned in comments inside sum() code. So actually sum() looks like this: def sum(seq, start = 0): for item in seq: start = start + item return start it creates a copy of the partial result on every "start + item". What I suggest is instead of making a copy for every item make just one copy and reuse it as many times as needed. For example: def sum(seq, start = 0): start = start + seq[0] for item in seq[1:]: start += item return start Patch implementing this idea attached to issue18305. Patch is simple, it just rearranges existing code. It should work for python 2.7, 3.3 and hg-tip. Except sum() becoming faster there should be no behavior change. Can this implementation break anything? Advantages: * Performance boost is like 200000% [1], nothing else becomes slower Disadvantages (inspired by Terry J. Reedy): * Other pythons (pypy, jython, older cpython versions) do not have such optimisation, people will move code that depends on the internal optimization to pythons that do not have it. And they will complain about their program 'freezing'. * It discourages people from carefully thinking about whether they actually need a concrete list or merely the iterator for a virtual list. Alternatives to sum() for this use-case: * list comprehension is 270% slower than patched sum [2] * itertools.chain is 50%-100% slower than patched sum [3] Main questions: * Whether to accept the patch * If yes, whether it should go to the new version or to a bugfix release Alternatives to this patch: * Reject the idea as a whole. Intentionally keep the sum() slow. * Accept the idea, but reject this particular implementation, instead use something different, e.g. implement special case optimization or use different approach (I thought about start=copy.copy(start)) In my opinion (looking though python changelog) performance patches were accepted for bugfix releases many times before. So as long as this patch does not break anything, it may go even to python 2.7. I think bad performance is a bug that should be fixed. Rejecting performance fixes just because older versions don't have it would mean no performance improvements to anything, ever. Sum is a great basic function. It keeps the simple code simple. This patch puts it on par with more advanced features like itertools, and allows people to use sum for simple things and itertools for complex memory-efficient algorithms. Questions that may arise if the patch is accepted: * sum() was rejecting strings because of this bug. If the bug gets fixed should another patch allow sum() to accept strings? * maybe in some distant future drop the second parameter (or make it None by default) and allow calling sum for everything, making sum() "the one obvious way" to sum up things? It would be nice if sum "just worked" for everything (e.g. sum() of empty sequence would return None, i.e. if there's nothing to sum then nothing is returned). But I think it needs more work for that, because even with this patch sum() is still ~20 times slower than "".join() [4] That's all. Any suggestions welcome. -- [1] Python 2.7.5 Before patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 10 loops, best of 3: 885 msec per loop After patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "sum(x,[])" 1000 loops, best of 3: 524 usec per loop [2] Python 2.7.5 with patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" "[i for l in x for i in l]" 1000 loops, best of 3: 1.94 msec per loop [3] Python 2.7.5 with patch: $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain.from_iterable(x))" 1000 loops, best of 3: 821 usec per loop $ ./python -mtimeit --setup="x=[[1,2,3]]*10000" --setup="from itertools import chain" "list(chain(*x))" 1000 loops, best of 3: 1.03 msec per loop [4] Python 2.7.5 with patch and string check removed: $ ./python -mtimeit --setup="x=['a']*10000" "sum(x,'')" 100 loops, best of 3: 3.98 msec per loop $ ./python -mtimeit --setup="x=['a']*10000" "''.join(x)" 10000 loops, best of 3: 170 usec per loop

26 179

Fwd: Allow Enum members to refer to each other during execution of body
by Antony Lee July 19, 2013

July 19, 2013

Sorry for the duplicate, seems like sending just to the googlegroups address doesn't cc to the python.org address. My previous email below. Antony 2013/7/12 Antony Lee <antony.lee(a)berkeley.edu> > Is there any specific reason why you do not wish to change the behavior of > Enum to this one (which does seem more logical to me)? The patch is fairly > simple in its logic (compared to the rest of the implementation, at > least...), and I could even change it to remove the requirement of defining > __new__ before the members as long as there are no references to other > members (because as long as there are no references to other members, I > obviously don't need to actually create the members), thus making it fully > compatible with the current version. > Antony > > > 2013/7/11 Ethan Furman <ethan(a)stoneleaf.us> > >> On 07/11/2013 02:07 PM, Antony Lee wrote: >> >>> On Wednesday, July 10, 2013 6:52:24 PM UTC-7, stoneleaf wrote: >>> >>>> On 07/10/2013 03:47 PM, Antony Lee wrote: >>>> >>>>> >>>>> Forward references are now implemented (https://github.com/anntzer/** >>>>> enum <https://github.com/anntzer/enum> <https://github.com/anntzer/** >>>>> enum <https://github.com/anntzer/enum>>). >>>>> >>>> >>>> Do they work with a custom __new__ ? __init__ ? >>>> >>> >>> In the current version, they work with a custom __init__ (though of >>> course, as long as the actual arguments that need to >>> be passed to __init__ are provided, the pre-declared members are just >>> "empty"). They do not work with a custom __new__ >>> (not sure how I could make this work, given that at declaration time an >>> "empty" member needs to be created but we don't >>> know what arguments we need to pass to __new__...). >>> As a side effect, however, the whole patch adds a new requirement: >>> custom __new__s must be defined before the members >>> themselves; otherwise they won't be called, for the same reason as >>> above: if I don't know what __new__ is, I can't call >>> it... >>> >> >> Hmm. Well, at this point I can offer kudos for getting it this far, but >> that's about it. The use-case this addresses seems fairly rare, and is >> definitely not a typical enumeration, and can be solved fairly easily with >> some extra post-processing code on a per-enumeration basis. >> >> >> -- >> ~Ethan~ >> ______________________________**_________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas> >> >> -- >> >> --- You received this message because you are subscribed to a topic in >> the Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/** >> topic/python-ideas/PC_**Ej19qj5w/unsubscribe<https://groups.google.com/d/topic/python-ideas/PC_Ej19qj5w/unsubscribe> >> . >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe(a)**googlegroups.com<python-ideas%2Bunsubscribe(a)googlegroups.com> >> . >> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >> . >> >> >> >

2 1

Intermediate Summary: Fast sum() for non-numbers
by Sergey July 19, 2013

July 19, 2013

Hello, python-ideas. I don't want to call it "summary" yet because I hope there're other ideas out there that would be suggested later. So let's call it "Intermediate Summary". I also shifted emphasis in this summary a little, to show that a faster sum() may be just a side-effect of some ideas. Introduction ============ sum() is a great function because it makes the code simple. It was never restricted to numbers, since it's also useful for adding types like timedelta or different numpy types. Unfortunately sum() performance is INCONSISTENT. I.e. it's O(N) for numbers, but it's O(N*N) for many containers. As a result: sum( [123]*1000000, 0 ) is instant, but: sum( [[1,2,3]]*1000000, [] ) takes forever to complete. It's worth to note, that sum() is one of the most commonly suggested options to add lists [1], despite usually someone also comes and says that it may be slow. This means, that people at least often try to use sum() for lists. That case was also explicitly mentioned in comments to sum() sources. So the problem is not hypothetical. This thread was not the first time when people discussed how this problem can be solved [2]. But this is probably the most deep and detailed discussion of it. Alternatives ============ During the thread there were many alternatives suggested. Among them: * Sum is not obvious (for everyone) way to add lists, so people should not use it, as there're alternatives, i.e. instead of - sum(list_of_lists, []) one can use: - reduce(operator.iadd, list_of_lists, []) - list(itertools.chain.from_iterable(list_of_lists)) - result = [] for x in list_of_lists: result.extend(x) * Alternatives should be more visible, so that people would spotted them earlier. For example there was a suggestion to move chain (or chain.from_iterable) from itertools into builtins, optionally changing its name. * Another alternative was to introduce "reduce" in builtins under a new name "fold", optionally with adjusted parameters to have them easier to understand. * And one more alternative suggested was do nothing. I can't understand it, so I'll just quote: > You're saying what IS NOT your preferred way. But I'm asking what > IS your preferred way. Do you prefer to have a slow sum in python > and people asking why it's slow forever? Do you see that as a best > possible case for python? YES. Godamnit YES. 100% true-to-form YES. But I believe that all of them are workarounds (better workarounds, maybe, but still workaround), that do not solve original problem. It's like if you have a similar bug [3] in httplib you could answer that python is not the best tool to download 110Mb files, or you could suggest some customhttpclient instead, but that would not fix the bug in httplib. I'm trying to address this particular issue of sum() inconsistency being O(N) for numbers, but O(N*N) for many containers. And trying to find options that could give sum a chance to be O(N) for most use cases. Ideas ===== 0. Original patch. Original patch [4] was suggesting to use "+" for first item and "+=" for everything else. That could make sum faster not only for lists but also for all the types having __iadd__ faster than __add__. But a clever example from Oscar [5] shown, that "+" can be different from "+=" e.g. due to coercion: >>> from numpy import array >>> a = array([1, 2, 3], dtype=int) >>> b = array([1.4, 2.5, 3.6], dtype=float) >>> a + b array([ 2.4, 4.5, 6.6]) >>> a += b >>> a array([2, 4, 6]) I suggest (patch [6]) to mention this example explicitly in comments for sum(), so that future developers would not be tempted to follow this approach. 1. Fast custom types. Is it fine to have types that are O(N) summable? This question looks weird, since sum() is already O(N) for some types and O(N*N) for others, but I was several times told things like: > Today, sum is not the best way to concatenate sequences. Making > it work better for some sequences but not others would mean it's > still not the best way to concatenate sequences, but it would > _appear_ to be. That's the very definition of an attractive > nuisance. So I want to clear this out: is it normal that sum() is fast for some types (e.g. numbers, timedeltas or FastTuples from [7]) and slow for others? Or sum just MUST BE slow and the very existence of faster containers is "attractive nuisance", and those should be expelled or slowed down too? 2. Fast built-in types. If the answer is "Yes", i.e. there's nothing bad in some types being faster than others, and there's nothing bad in sum() being faster for some types. Then what do you think about optimizing built-in types, like lists, tuples and strings? The optimization itself could have different benefits: * lower memory usage * instant creation of one instance from another * faster add operation (i.e. "c = a + b") * having it common to lists and tuples could give instant conversion of lists to tuples and back But as a side-effect of such optimization sum() will be O(N) for those types. So the question is: if it's fine for some types to be faster than others, then is it fine if those "some types" are lists or tuples? I.e. what if lists and tuples would be O(N) to sum, while some other types are not? To clear the question whether this is possible I wrote [7] a very limited version of tuple, using that approach, so that: from fasttuple import ft x = sum( [ft([1,2,3])] * 1000000, ft([]) ) takes a few seconds under a usual unpatched python. 3. Fast built-in types just for sum [8]. If it's not bad to have lists/tuples faster than other types, then how about just implementing a small special case for them? Advantage: patch is much shorter [8], than optimization for lists and tuples would be, and it would probably cover most (if not all) usages of sum() for containers. Disadvantage: this patch is limited to sum(), i.e. it gives the same performance for sum() as #2, but it does not optimize general "c = a + b" code as #2 would. Side-note: sum() was designed with special cases in mind from the beginning [9]. Initial sum() patch had a special case for strings (''.join() was used for them) but due to a complexity of code for mixed lists it was replaced by another special case rejecting strings. Currently sum has 5 special cases: three to reject strings/bytes and two optimizations for ints and floats. 4. If that would be considered a bad idea, i.e. if we decide that other custom types should have a fair chance to implement some optimization, then how about declaring a unified way to initialize a container type from iterable? Something like an optional method __init_concatenable_sequence_from_iterable__. Most containers already have such method as a default constructor, i.e. it should be trivial to implement it for any existing type. As a side-effect of this unification sum() could have a general optimization for such types, something like: if hasattr(type(start), "__init_concatenable_container_from_iterable__"): optimized_for = type(start) l_result = list() l_result.extend(start) try: while True: item = next(it) if type(item) is not optimized_for: start = optimized_for.__init_concatenable_container_from_iterable__(l_result) start = start + item break l_result.extend(item) except StopIteration: return optimized_for.__init_concatenable_container_from_iterable__(l_result) (I don't like that long __ name, it was just first I thought about. Better names are welcome. __iccfi__?) 5. Alternative unification idea was suggested by MRAB: . Get the first item, or the start value . Ask the item for an 'accumulator' (item.__accum__()). . Add the first item to the accumulator. . Add the remaining items to the accumulator. . Ask the accumulator for the result. If there's no accumulator available (the "__accum__" method isn't implemented), then either fall back to the current behaviour or raise a TypeError like it currently does for strings. I *guess* sum() was never discussed before so deeply because many people either hate sum() or believe that nothing can be done, so they don't even try to find any options. Or maybe they just prefer to use some easily available workaround and don't spend time trying to find other solutions. PS: If you can, please, explain what exactly you [don't] like in the ideas above, so I could have a chance to modify the ideas according to your opinions. PPS: I'm sorry if I missed some of the suggestions, please, add them too. PPPS: I'm also sorry not to answer to some emails during the thread. I read all of them, but there were already too many of messages in the thread, so I was trying to limit number of emails I write. If I missed something important, please, forward your email to me privately and I'll answer you either directly or on list, if you wish. -- [1] Some stackoverflow questions where sum() is suggested for lists: http://stackoverflow.com/questions/406121/flattening-a-shallow-list-in-pyth… http://stackoverflow.com/questions/716477/join-list-of-lists-in-python http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of… http://stackoverflow.com/questions/3021641/concatenation-of-many-lists-in-p… http://stackoverflow.com/questions/7895449/merging-a-list-of-lists http://stackoverflow.com/questions/10424219/combining-lists-into-one http://stackoverflow.com/questions/11331908/how-to-use-reduce-with-list-of-… http://stackoverflow.com/questions/17142101/concatenating-sublists-python [2] Some earlier attempts to discuss solutions to sum() slowness: 2006-01-12 http://article.gmane.org/gmane.comp.python.general/441831 > A fast implementation would probably allocate the output list just > once and then stream the values into place with a simple index. That's what I hoped "sum" would do, but instead it barfs with a type error. 2010-03-27 http://article.gmane.org/gmane.comp.python.general/658537 The mildly surprising part of sum() is that is does add vs. add-in- place, which leads to O(N) vs. O(1) for the inner loop calls, for certain data structures, notably lists, even though none of the intermediate results get used by the caller. For lists, you could make a more efficient variant of sum() that clones the start value and does add-in-place. [3] http://bugs.python.org/issue6838 *** httplib's _read_chunked extremely slow for lots of chunks *** As the comment in this method suggests, accumulating the value by repeated string concatenation is slow. Appending to a list speeds this up dramatically. [4] http://bugs.python.org/file30705/fastsum.patch Original patch using "+" for first item and "+=" for the rest. [5] http://bugs.python.org/issue18305#msg192873 Oscar Benjamin: This "optimisation" is a semantic change. It breaks backward compatibility in cases where a = a + b and a += b do not result in the name a having the same value. In particular this breaks backward compatibility for numpy users: [...] [6] http://bugs.python.org/issue18305#msg192956 http://bugs.python.org/file30904/fastsum-iadd_warning.patch Extended warning in sum() comments with more important numpy cases. [7] http://bugs.python.org/issue18305#msg193048 http://bugs.python.org/file30917/fasttuple.py fasttuple.py is a Proof-of-Concept implementation of tuple, that reuses same data storage when possible. Its possible usage looks similar to built-in tuples [...]. An interesting side-effect of this implementation is a faster __add__ operator: Adding 100000 of fasttuples took 0.23242688179 seconds Adding 100000 of built-in tuples took 25.2749021053 seconds [8] http://bugs.python.org/issue18305#msg192919 http://bugs.python.org/file30897/fastsum-special-tuplesandlists.patch Patch implementing a special case for tuples and lists. Should work for both Python 2.7 and Python 3.3, and should introduce no behavior change. [9] http://mail.python.org/pipermail/python-dev/2003-April/034767.html Alex Martelli (author of sum()) > for the simple reason that I special-case this -- when the first > item is a PyBaseString_Type, I delegate to ''.join

10 34

A limited exec
by Hua Lu July 19, 2013

July 19, 2013

Hi, I've attempted to make exec/eval a bit safer. May I please have some feedback? https://github.com/cag/execgate Thanks, Alan

2 3

Re: [Python-ideas] regex module - proper implementation of alternation?
by Eli Bendersky July 16, 2013

July 16, 2013

On Tue, Jul 16, 2013 at 8:53 AM, MRAB <python(a)mrabarnett.plus.com> wrote: > On 16/07/2013 16:41, Eli Bendersky wrote: > >> >> >> >> On Tue, Jul 16, 2013 at 8:00 AM, MRAB <python(a)mrabarnett.plus.com >> <mailto:python@mrabarnett.**plus.com <python(a)mrabarnett.plus.com>>> >> wrote: >> >> On 16/07/2013 04:10, Eli Bendersky wrote: >> >> Since the 'regex' module is a candidate for inclusion into the >> stdlib, I >> figured this would be a good place to ask. >> >> While discussing something related in pss >> (https://github.com/eliben/**pss__<https://github.com/eliben/pss__>), >> Ben Hoyt brought to my >> >> attention that >> the implementation of alternation (foo|bar) in Python's default >> regex >> module (the SRE implementation) is very inefficient. And indeed, >> looking >> there it seems that | gets translated to an opcode that simply >> means >> going over all the alternatives in a loop trying to match each. >> This is >> not how a proper regex engine should implement alternation! >> >> A common advice given to Python programmers is to combine their >> regexes >> into a single one with |. This makes code faster, but it turns >> out that >> it's far from its full potential because the combination doesn't >> go full >> way to the DFA as it's supposed to. >> >> A question about 'regex' - is it implemented properly there? >> >> There are 2 ways of implementing regex: DFA and NFA. >> >> DFA is faster, but those using NFA do so because the implementation >> offers additional features that make DFA tricky or impossible, such as >> backreferences. >> >> Of course, you could use DFA when it's possible, NFA when it isn't, at >> the cost of yet more code. >> >> The regex module uses NFA, just like re. >> >> If you want to improve regex, making it use DFA when possible, well, >> the source code is open, and your contributions are welcome. Good >> luck! >> :-) >> >> >> You can use NFA without backtracking, though, by keeping track of the >> set of possible states. I believe (but am not 100% sure) this is the way >> re2 works, for example. >> >> In the particular case of alternations, such approach is vastly superior >> because the "possible states" set never grows large (assuming the >> alternatives are not mostly the same). Whereas with backtracking you >> always have to iterate over all of them. >> >> That said, I think you have answered my question - regex also uses a >> backtracking implementation of NFA and iterates in case of alternations. >> OK :-) >> >> Have you tried timing them (re, re2, regex, and possibly others) to see > whether > it's a problem in practice? > I have not; this page - http://swtch.com/~rsc/regexp/regexp1.html - explains the problems with backtracking, but focuses on a different aspect (where backtracking leads to exponential behavior). The problem did, however, come up in practice in pss (see https://github.com/eliben/pss/issues/4); there was an attempt to make heavier use of regex alternations and the result ended up being much slower than one would expect. I had this experience in a different place as well (writing regex-based lexers). There is a performance improvement gained from moving from explicit Python-level looping to alternations, but it's a small gain - something that makes sense for just moving a loop from Python into C, but not for doing something actually smart with the alternation (like looking at each incoming character only once). [P.S. please use reply-all in this thread] Eli

2 1

Re: [Python-ideas] regex module - proper implementation of alternation?
by Eli Bendersky July 16, 2013

July 16, 2013

On Tue, Jul 16, 2013 at 8:00 AM, MRAB <python(a)mrabarnett.plus.com> wrote: > On 16/07/2013 04:10, Eli Bendersky wrote: > >> Since the 'regex' module is a candidate for inclusion into the stdlib, I >> figured this would be a good place to ask. >> >> While discussing something related in pss >> (https://github.com/eliben/pss**), Ben Hoyt brought to my attention that >> the implementation of alternation (foo|bar) in Python's default regex >> module (the SRE implementation) is very inefficient. And indeed, looking >> there it seems that | gets translated to an opcode that simply means >> going over all the alternatives in a loop trying to match each. This is >> not how a proper regex engine should implement alternation! >> >> A common advice given to Python programmers is to combine their regexes >> into a single one with |. This makes code faster, but it turns out that >> it's far from its full potential because the combination doesn't go full >> way to the DFA as it's supposed to. >> >> A question about 'regex' - is it implemented properly there? >> >> There are 2 ways of implementing regex: DFA and NFA. > > DFA is faster, but those using NFA do so because the implementation > offers additional features that make DFA tricky or impossible, such as > backreferences. > > Of course, you could use DFA when it's possible, NFA when it isn't, at > the cost of yet more code. > > The regex module uses NFA, just like re. > > If you want to improve regex, making it use DFA when possible, well, > the source code is open, and your contributions are welcome. Good luck! > :-) > > You can use NFA without backtracking, though, by keeping track of the set of possible states. I believe (but am not 100% sure) this is the way re2 works, for example. In the particular case of alternations, such approach is vastly superior because the "possible states" set never grows large (assuming the alternatives are not mostly the same). Whereas with backtracking you always have to iterate over all of them. That said, I think you have answered my question - regex also uses a backtracking implementation of NFA and iterates in case of alternations. OK :-) Eli

1 0

regex module - proper implementation of alternation?
by Eli Bendersky July 16, 2013

July 16, 2013

Since the 'regex' module is a candidate for inclusion into the stdlib, I figured this would be a good place to ask. While discussing something related in pss (https://github.com/eliben/pss), Ben Hoyt brought to my attention that the implementation of alternation (foo|bar) in Python's default regex module (the SRE implementation) is very inefficient. And indeed, looking there it seems that | gets translated to an opcode that simply means going over all the alternatives in a loop trying to match each. This is not how a proper regex engine should implement alternation! A common advice given to Python programmers is to combine their regexes into a single one with |. This makes code faster, but it turns out that it's far from its full potential because the combination doesn't go full way to the DFA as it's supposed to. A question about 'regex' - is it implemented properly there? Eli

1 0

PEP for issue2292, "Missing *-unpacking generalizations"
by Joshua Landau July 15, 2013

July 15, 2013

The PEP is attached. I'm not sure if I've covered the basics, but it's a try. If anyone knows how to get the patch (from the bug report) working, or where to find http://code.python.org/python/users/twouters/starunpack after code.python.org was deleted in favour of hg.python.org (which seems not to have it), it'd be nice to know.

7 32

Iterable function calls and unpacking [was: PEP for issue2292, "Missing *-unpacking generalizations"]
by Joshua Landau July 15, 2013

July 15, 2013

I've moved this to a different thread, as I agree with Guido that it's a different PEP. On 15 July 2013 21:06, Guido van Rossum <guido(a)python.org> wrote: > On Mon, Jul 15, 2013 at 1:01 PM, Oscar Benjamin > <oscar.j.benjamin(a)gmail.com> wrote: >> On 15 July 2013 12:08, Joshua Landau <joshua.landau.ws(a)gmail.com> wrote: >>> On 15 July 2013 11:40, Oscar Benjamin <oscar.j.benjamin(a)gmail.com> wrote: >>> >>> In fact, I'd much like it if there was an iterable "unpacking" method >>> for functions, too, so "chain.from_iterable()" could use the same >>> interface as "chain" (and str.format with str.format_map, etc.). I >>> feel we already have a good deal of redundancy due to this. >> >> I've also considered this before. I don't know what a good spelling >> would be but lets say that it uses *args* so that you have a function >> signature like: >> >> def chain(*iterables*): >> for iterable in iterables: >> yield from iterable Personally some form of decorator would be simpler: @lazy_unpack() def chain(*iterables): ... (and it could theoretically work for mappings too, by just "bundling" them; useful for circumstances where the mapping is left untouched and just passed to the next function in line.) >> And then if the function is called with >> >> for line in chain(first_line, *inputfile): >> # do stuff >> >> then iterables would be bound to a lazy generator that chains >> [first_line] and inputfile. Then you could create the unpacking >> iterator I wanted by just using chain e.g.: >> >> chain(prepend, *iterable, append) > > But how could you do this without generating different code depending > on how the function you are calling is declared? Python's compiler > doesn't have access to that information. You could simply make the code such that if has an unpack inside the call it does a run-time check. Whilst this will be slower for the false-positives, the number of times *args is pass-through (and thus you save a redundant copy of the argument tuple) and *args is a simple loop-once construct makes it plausible that those losses would be outweighed. It doesn't even reduce efficiency that much, too, as the worst case scenario is immediately falling back after checking a single C-level attribute of the function, and the function doesn't need to be fetched again or anything suchlike. Then again, I'm guessing. You'd also need to add a call to exhaust the iterator at the end of every function utilising this (transparently, probably) to make this have no obvious externally-visible effects. There would still be a call-order change, but that's much more minor.

2 2