Creating Long Lists

Jorgen Grahn grahn+nntp at
Wed Feb 23 14:57:12 CET 2011

On Tue, 2011-02-22, Ben Finney wrote:
> Kelson Zawack <zawackkfb at> writes:
>> I have a large (10gb) data file for which I want to parse each line
>> into an object and then append this object to a list for sorting and
>> further processing.
> What is the nature of the further processing?
> Does that further processing access the items sequentially? If so, they
> don't all need to be in memory at once, and you can produce them with a
> generator <URL:>.

He mentioned sorting them -- you need all of them for that.

If that's the *only* such use, I'd experiment with writing them as
sortable text to file, and run GNU sort (the Unix utility) on the file.
It seems to have a clever file-backed sort algorithm.


  // Jorgen Grahn <grahn@  Oo  o.   .  .
\X/>   O  o   .

More information about the Python-list mailing list