<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div style="-webkit-text-size-adjust: auto; ">On Oct 4, 2014, at 15:27, Steven D'Aprano <<a href="mailto:steve@pearwood.info">steve@pearwood.info</a>> wrote:</div><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; "><span></span></div><blockquote type="cite" style="-webkit-text-size-adjust: auto; "><div><span>On Fri, Oct 03, 2014 at 05:09:20PM +0200, Thomas Chaumeny wrote:</span><br><blockquote type="cite"><span>Hi!</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I have just come across some code counting a generator comprehension</span><br></blockquote><blockquote type="cite"><span>expression by doing len([foo for foo in bar if some_condition]) and I</span><br></blockquote><blockquote type="cite"><span>realized it might be better if we could just use len(foo for foo in bar if</span><br></blockquote><blockquote type="cite"><span>some_condition) as it would avoid a list allocation in memory.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Another possibility is to write sum(1 for foo in bar if some_condition),</span><br></blockquote><blockquote type="cite"><span>but that is not optimal either as it generates a lot of intermediate</span><br></blockquote><blockquote type="cite"><span>additions which should not be needed.</span><br></blockquote><span></span><br><span>I don't understand this reasoning. Why do you think that they are </span><br><span>unnecessary?</span><br><span></span><br><span>I believe that, in the general case of an arbitrary generator </span><br><span>expression, there are only two ways to tell what the length will be. The </span><br><span>first is to produce a list (or other sequence), then return the </span><br><span>length of the list: len([obj for obj in generator if condition]). The </span><br><span>second is to count each item as it is produced, but without storing them </span><br><span>all: sum(1 for obj in generator if condition). The first is optimized </span><br><span>for readability, the second for memory.</span><br></div></blockquote><div style="-webkit-text-size-adjust: auto; "><br></div><div style="-webkit-text-size-adjust: auto; ">Also more that the itertools recipes include a function for doing exactly this:</div><div style="-webkit-text-size-adjust: auto; "><br></div><div><pre style="overflow-x: auto; overflow-y: hidden; padding: 5px; border: 1px solid rgb(170, 204, 153); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; "><font face="Helvetica"><span style="white-space: normal; -webkit-text-size-adjust: auto; background-color: rgba(255, 255, 255, 0);"><span class="k" style="font-weight: bold; ">def</span> <span class="nf">quantify</span><span class="p">(</span><span class="n">iterable</span><span class="p">,</span> <span class="n">pred</span><span class="o">=</span><span class="nb">bool</span><span class="p">):</span></span></font></pre><pre style="overflow-x: auto; overflow-y: hidden; padding: 5px; border: 1px solid rgb(170, 204, 153); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; "><span class="s" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; "> "Count how many times the predicate is true"</span></pre><pre style="overflow-x: auto; overflow-y: hidden; padding: 5px; border: 1px solid rgb(170, 204, 153); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; "><span class="k" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; "> </span><span class="k" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; font-weight: bold; ">return</span><span style="background-color: rgba(255, 255, 255, 0); white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; "> </span><span class="nb" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">sum</span><span class="p" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">(</span><span class="nb" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">map</span><span class="p" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">(</span><span class="n" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">pred</span><span class="p" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">,</span><span style="background-color: rgba(255, 255, 255, 0); white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; "> </span><span class="n" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">iterable</span><span class="p" style="white-space: normal; -webkit-text-size-adjust: auto; font-family: Helvetica; ">))</span></pre></div><div>The more-itertools library on PyPI has this, plus an even simpler `ilen` function that just does, IIRC, sum(1 for _ in iterable). Of course they're both trivial one-liners, but maybe calling more_itertools.ilen is a nice way to clarify that you're intentionally consuming an iterator just to get its length.</div><div><br></div><blockquote type="cite" style="-webkit-text-size-adjust: auto; "><div><span>I don't believe that there is any other general way to work out the </span><br><span>length of an arbitrary generator. (Apart from trivial, or obfuscated, </span><br><span>variations on the above two, of course.) </span></div></blockquote><div><br></div><div>Would teeing the iterator and consuming the extra copy count as an obfuscated variation? In practice, it's kind of the worst of both worlds--you're constructing a sequence (a deque instead of a list) in memory, but can only access it as a one-shot iterator. Conceptually, it looks somewhat nice--you're copying the iterator to count it without destroying it--but I think that's only an illusion for those who don't understand lists as iterables.</div><br><blockquote type="cite" style="-webkit-text-size-adjust: auto; "><div><span>How would you tell what the </span><br><span>length of this generator should be, apart from actually running it to </span><br><span>exhaustion?</span><br></div></blockquote><div><br></div>Obviously we just need to rewrite Python around lazy dataflow variables, so whatever you assign len(generator()) to doesn't consume the generator until necessary, meaning you could still use the iterator's vales elsewhere in the mean time. :)<div><div><blockquote type="cite" style="-webkit-text-size-adjust: auto; "><div><span></span></div></blockquote></div></div></body></html>