Geneator/Iterator Nesting Problem - Any Ideas? 2.4

[ChaosKCW]

Hi

Using Python 2.4 I am trying to procduce a generator which will return
the results of dbi SQL statement using fetchmany for performance.

So instead of fetching one record for each call, I will fetch batches
of X (eg 100) and yeild each record in turn.

For reasons of pure asthetics and my own learning I wanted it to look
like this:

</B>
def resultsetbatchgen(cursor, size=100):
    for results in (recordbatch for recordbatch in
cursor.fetchmany(size)):
        for rec in results:
            yield rec
</B>

Problem is this this gives spurious results. To understand the problem
if I excute the following in the console I get the correct results:

</B>
>>> cur.execute(<QUERY WITH MOER THAN 1000 records>)
>>> recs = (recordbatch for recordbatch in cur.fetchmany(1000))
>>> sum(map(lambda x: 1, (rec for rec in recs)))
1000
</B>

This is PERFECT! 1000 is what I expected. but now lets nest this inside
another for construct like so

</B>
>>> cur.execute(<QUERY WITH MOER THAN 1000 records>)
>>> for results in (recordbatch for recordbatch in
cur.fetchmany(1000)):
... 	print sum(map(lambda x: 1, (rec for rec in results)))
76
76
76
76
76
76
76
76
...........
</B>

Now it thinks each batch size is 76 ... ? This completly wrong, and
baffling.

The commands are exactly the same as far as I can tell, the only
difference is that it is now nested wihtin another for loop?

Any help would be greatly aprpeciated.

PS a working but far less elegant version of this below but I would
like to understand why teh above doesnt work and doesnt work
consitantly:

def resultsetbatch(cursor, size=100):
    results = cursor.fetchmany(size)
    while results <> []:
        for rec in results:
            yield rec
        results = cursor.fetchmany(size)