[Tutor] when is a generator "smart?"

[Steven D'Aprano]

On 02/06/13 13:58, Jim Mooney wrote:
> It's a little unclear to me where generators are more efficient.

When there are a lot of items, and you access the items one at a time, not all at once. If there are only a few items, a list or tuple has less overhead and is easier to use.


> Below I'm creating a list of non-even squares.

By creating a list, you completely lose any benefit of a generator, since you access all the items in one big list.


> I figured Python would be
> smart enough to see I only wanted a small slice of a large generated
> list,

Nope. The standard CPython implementation is deliberately dumb. Advanced optimizing Python implementations like PyPy *may* be smarter.

There are good reasons why Python *cannot* be very smart about this:

- it cannot tell what list(...) will do until runtime, since "list" is just a name and may be replaced by a different function;

- consequently it cannot tell what the slice [2:10] does until it has built the entire list.

A smarter compiler, like PyPy, *may* be able to tell at runtime that list is the built-in list, and therefore do something special, but that requires a much more complicated compiler. As the reference implementation, CPython is deliberately simple, and will always remain that way.



-- 
Steven