Hello, Sometimes it's useful to get the number of elements yield by an iterator. For example (if ilen is the name of the function): def pi(n): return ilen(for e in xrange(n) if isprime(e)) def count_pred(pred, iterator): return ilen(itertools.ifilter(pred, iterator)) A few solutions were discussed here http://stackoverflow.com/questions/3393431/how-to-counting-not-0-elements-in... with the best two below: 1) sum(1 for e in iterator) 2) len(list(iterator)) First solution is slow, the second solution uses O(N) extra memory. I propose the addition of a new function ilen() which is functionally equivalent to: def ilen(iterator): return sum(1 for e in iterator) This function should be different from len() because it's time complexity is O(N) (most people assume that len() takes O(1)) and it consumes the iterator. Regards, -- Alexandru Moșoi http://www.alexandru.mosoi.ro/
On 9 August 2010 18:16, Alexandru Moșoi <brtzsnr@gmail.com> wrote:
You say that this solution "is slow" and then you propose it? I'm confused. Besides which, as you define it, it exhausts the iterator, which makes it useless. It may be useful for an *iterable*, but most of them support len in any case.
Precisely. So how is it useful? If you could show some real code that uses your ilen function, that would help clarify. But it still won't explain why the function should be built in rather than just defined by your code where it's needed - you'll have to have some very common and compelling use cases to argue that. Paul.
2010/8/9 Paul Moore <p.f.moore@gmail.com>:
My requirements was to count the non-zero elements from a list like this: sum(1 for e in iterator if not e) What I'm really looking for is the number of elements in a list comprehension: len(list(for e in iterator if not e)) but this is not generally useful nor optimal in terms of memory requirements. My idea was to implement the above with the aid of itertools.ifilter: ilen(itertools.ifilter(pred, iterable)) if pred is None, this would translate in my usecase. Since I first post this I learned that ilen (or something similar) was rejected before due to similar concerns: it consumes the iterator, it's not a real optimization. How about: count(pred, iterable) which returns the same value as len(filter(pred, iterable))? -- Alexandru Moșoi http://www.alexandru.mosoi.ro/
On 8/9/2010 4:17 PM, Alexandru Moșoi wrote:
You are responding to his request for a specific example with a generic class of examples, which is what prompted his request for a specific example in the first place. Please give a *specific* example and be prepared to be told that you are going about it the wrong way, since, at this point, nobody has replied as having recognized this as a problem they've encountered before. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
2010/8/10 Scott Dial <scott+python-ideas@scottdial.com>:
My exact need is to count the not-None elements in a list. My current solution is, as described before: sum(1 for e in iterator if not e) -- Alexandru Moșoi http://www.alexandru.mosoi.ro/
It seems to me that, to count the not-None elements, your logic is inverted. Surely you want sum(1 for e in iterator if e) or more accurately sum(1 for e in iterator if e is not None) Rob Cliffe ----- Original Message ----- From: "Alexandru Moșoi" <brtzsnr@gmail.com> To: "Scott Dial" <scott+python-ideas@scottdial.com> Cc: <python-ideas@python.org> Sent: Tuesday, August 10, 2010 5:45 PM Subject: Re: [Python-ideas] iterator length
Can you all take this off-line? It is turning into programming help instead of a feature discussion. -- --Guido van Rossum (python.org/~guido)
Hello,
Sometimes it's useful to get the number of elements yield by an iterator. For example (if ilen is the name of the function):
A generic iterator is something that inherently, by definition, has unknown and unpredictable length (think, for example, about berries you gather in a forest). There is no guarantee two tries give the same results, there isn't even a guarantee that it's finite. You just can't, absolutely, by any means, know its length otherwise than by exhausting it. If you can, it's not a generic iterator but something else. -- Regards, Ivan mailto:vano@mail.mipt.ru
On Tue, Aug 10, 2010 at 7:42 AM, Ivan Pozdeev <vano@mail.mipt.ru> wrote:
Indeed - iterating over a completely arbitrary iterator with no exit criteria is a recipe for infinite loops when someone passes in something like itertools.count or itertools.repeat.
Yep, hence the existence of the __length_hint__ API as an internal optimisation when dealing with iterators that do have some idea of their length (see http://mail.python.org/pipermail/python-dev/2009-April/088108.html for discussion as to why it is undocumented). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 9 August 2010 18:16, Alexandru Moșoi <brtzsnr@gmail.com> wrote:
You say that this solution "is slow" and then you propose it? I'm confused. Besides which, as you define it, it exhausts the iterator, which makes it useless. It may be useful for an *iterable*, but most of them support len in any case.
Precisely. So how is it useful? If you could show some real code that uses your ilen function, that would help clarify. But it still won't explain why the function should be built in rather than just defined by your code where it's needed - you'll have to have some very common and compelling use cases to argue that. Paul.
2010/8/9 Paul Moore <p.f.moore@gmail.com>:
My requirements was to count the non-zero elements from a list like this: sum(1 for e in iterator if not e) What I'm really looking for is the number of elements in a list comprehension: len(list(for e in iterator if not e)) but this is not generally useful nor optimal in terms of memory requirements. My idea was to implement the above with the aid of itertools.ifilter: ilen(itertools.ifilter(pred, iterable)) if pred is None, this would translate in my usecase. Since I first post this I learned that ilen (or something similar) was rejected before due to similar concerns: it consumes the iterator, it's not a real optimization. How about: count(pred, iterable) which returns the same value as len(filter(pred, iterable))? -- Alexandru Moșoi http://www.alexandru.mosoi.ro/
On 8/9/2010 4:17 PM, Alexandru Moșoi wrote:
You are responding to his request for a specific example with a generic class of examples, which is what prompted his request for a specific example in the first place. Please give a *specific* example and be prepared to be told that you are going about it the wrong way, since, at this point, nobody has replied as having recognized this as a problem they've encountered before. -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
2010/8/10 Scott Dial <scott+python-ideas@scottdial.com>:
My exact need is to count the not-None elements in a list. My current solution is, as described before: sum(1 for e in iterator if not e) -- Alexandru Moșoi http://www.alexandru.mosoi.ro/
It seems to me that, to count the not-None elements, your logic is inverted. Surely you want sum(1 for e in iterator if e) or more accurately sum(1 for e in iterator if e is not None) Rob Cliffe ----- Original Message ----- From: "Alexandru Moșoi" <brtzsnr@gmail.com> To: "Scott Dial" <scott+python-ideas@scottdial.com> Cc: <python-ideas@python.org> Sent: Tuesday, August 10, 2010 5:45 PM Subject: Re: [Python-ideas] iterator length
Can you all take this off-line? It is turning into programming help instead of a feature discussion. -- --Guido van Rossum (python.org/~guido)
Hello,
Sometimes it's useful to get the number of elements yield by an iterator. For example (if ilen is the name of the function):
A generic iterator is something that inherently, by definition, has unknown and unpredictable length (think, for example, about berries you gather in a forest). There is no guarantee two tries give the same results, there isn't even a guarantee that it's finite. You just can't, absolutely, by any means, know its length otherwise than by exhausting it. If you can, it's not a generic iterator but something else. -- Regards, Ivan mailto:vano@mail.mipt.ru
On Tue, Aug 10, 2010 at 7:42 AM, Ivan Pozdeev <vano@mail.mipt.ru> wrote:
Indeed - iterating over a completely arbitrary iterator with no exit criteria is a recipe for infinite loops when someone passes in something like itertools.count or itertools.repeat.
Yep, hence the existence of the __length_hint__ API as an internal optimisation when dealing with iterators that do have some idea of their length (see http://mail.python.org/pipermail/python-dev/2009-April/088108.html for discussion as to why it is undocumented). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Alexandru Moșoi -
Guido van Rossum -
Ivan Pozdeev -
MRAB -
Nick Coghlan -
Paul Moore -
Rob Cliffe -
Scott Dial