Creating Long Lists

[Jorgen Grahn]

On Tue, 2011-02-22, Ben Finney wrote:
> Kelson Zawack <zawackkfb at gis.a-star.edu.sg> writes:
>
>> I have a large (10gb) data file for which I want to parse each line
>> into an object and then append this object to a list for sorting and
>> further processing.
>
> What is the nature of the further processing?
>
> Does that further processing access the items sequentially? If so, they
> don't all need to be in memory at once, and you can produce them with a
> generator <URL:http://docs.python.org/glossary.html#term-generator>.

He mentioned sorting them -- you need all of them for that.

If that's the *only* such use, I'd experiment with writing them as
sortable text to file, and run GNU sort (the Unix utility) on the file.
It seems to have a clever file-backed sort algorithm.

/Jorgen

-- 
  // Jorgen Grahn <grahn@  Oo  o.   .  .
\X/     snipabacken.se>   O  o   .