Creating Long Lists
grahn+nntp at snipabacken.se
Wed Feb 23 14:57:12 CET 2011
On Tue, 2011-02-22, Ben Finney wrote:
> Kelson Zawack <zawackkfb at gis.a-star.edu.sg> writes:
>> I have a large (10gb) data file for which I want to parse each line
>> into an object and then append this object to a list for sorting and
>> further processing.
> What is the nature of the further processing?
> Does that further processing access the items sequentially? If so, they
> don't all need to be in memory at once, and you can produce them with a
> generator <URL:http://docs.python.org/glossary.html#term-generator>.
He mentioned sorting them -- you need all of them for that.
If that's the *only* such use, I'd experiment with writing them as
sortable text to file, and run GNU sort (the Unix utility) on the file.
It seems to have a clever file-backed sort algorithm.
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
More information about the Python-list