fromiter shape argument -- was Re: For loop tips
![](https://secure.gravatar.com/avatar/3f692386259303b90d1967ad662a9eb0.jpg?s=120&d=mm&r=g)
return uL,asmatrix(fromiter((idx[x] for x in L),dtype=int))
Is it possible for fromiter to take an optional shape (or count) argument in addition to the dtype argument? If both is given it could preallocate memory and we only have to iterate over L once. //Torgil On 8/29/06, Keith Goodman <kwgoodman@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/5b2449484c19f8e037c5d9c71e429508.jpg?s=120&d=mm&r=g)
Torgil Svensson wrote:
If both is given it could preallocate memory and we only have to iterate over L once.
Regardless, L is only iterated over once. In general you can't rewind iterators, so that's a requirement. This is accomplished by doing successive overallocation similar to the way appending to a list is handled. By specifying the count up front you save a bunch of reallocs, but no iteration. -tim
![](https://secure.gravatar.com/avatar/3f692386259303b90d1967ad662a9eb0.jpg?s=120&d=mm&r=g)
Yes. fromiter(iterable, dtype, count) works.
Oh. Thanks. I probably had too old documentation to see this (15 June). If it's not updated since I'll give Travis a rest about this, at least until 1.0 is released :)
Regardless, L is only iterated over once.
How can this be true? If no size is given, mustn't numpy either loop over L twice or build an internal representation on which it'll iterate or copy in chunks? I just found out that this works
but what's wrong with this?
//Torgil On 8/30/06, Tim Hochberg <tim.hochberg@ieee.org> wrote:
![](https://secure.gravatar.com/avatar/5b2449484c19f8e037c5d9c71e429508.jpg?s=120&d=mm&r=g)
Torgil Svensson wrote:
Actually I just knew 'cause I wrote it. I don't see a docstring for fromiter, although I though I wrote one. Maybe I just forgot?
Well, it can't in general loop over L twice since the only method that L is guaranteed to have is next(); that's the extent of the iterator protocol. What it does is allocate an initial chunk of memory (the size of which I forget -- I did some tuning) and start filling it up. Once it's full, it does a realloc, which expands the existing chunk or memory, if possible, or returns a new, larger, chunk of memory with the data copied into it. Then we iterate on L some more until we fill up the new larger chunk, in which case we go get another one, etc. This is exactly how list.append works, although in that case the chunk of data is acutally a chunk of pointers to objects. -tim
![](https://secure.gravatar.com/avatar/5b2449484c19f8e037c5d9c71e429508.jpg?s=120&d=mm&r=g)
Torgil Svensson wrote:
If both is given it could preallocate memory and we only have to iterate over L once.
Regardless, L is only iterated over once. In general you can't rewind iterators, so that's a requirement. This is accomplished by doing successive overallocation similar to the way appending to a list is handled. By specifying the count up front you save a bunch of reallocs, but no iteration. -tim
![](https://secure.gravatar.com/avatar/3f692386259303b90d1967ad662a9eb0.jpg?s=120&d=mm&r=g)
Yes. fromiter(iterable, dtype, count) works.
Oh. Thanks. I probably had too old documentation to see this (15 June). If it's not updated since I'll give Travis a rest about this, at least until 1.0 is released :)
Regardless, L is only iterated over once.
How can this be true? If no size is given, mustn't numpy either loop over L twice or build an internal representation on which it'll iterate or copy in chunks? I just found out that this works
but what's wrong with this?
//Torgil On 8/30/06, Tim Hochberg <tim.hochberg@ieee.org> wrote:
![](https://secure.gravatar.com/avatar/5b2449484c19f8e037c5d9c71e429508.jpg?s=120&d=mm&r=g)
Torgil Svensson wrote:
Actually I just knew 'cause I wrote it. I don't see a docstring for fromiter, although I though I wrote one. Maybe I just forgot?
Well, it can't in general loop over L twice since the only method that L is guaranteed to have is next(); that's the extent of the iterator protocol. What it does is allocate an initial chunk of memory (the size of which I forget -- I did some tuning) and start filling it up. Once it's full, it does a realloc, which expands the existing chunk or memory, if possible, or returns a new, larger, chunk of memory with the data copied into it. Then we iterate on L some more until we fill up the new larger chunk, in which case we go get another one, etc. This is exactly how list.append works, although in that case the chunk of data is acutally a chunk of pointers to objects. -tim
participants (2)
-
Tim Hochberg
-
Torgil Svensson