Unsung Python modules
tim at vegeta.ath.cx
Sat Dec 15 16:22:53 EST 2001
Tim Hammerquist <tim at vegeta.ath.cx> graced us by uttering:
> Skip Montanaro <skip at pobox.com> graced us by uttering:
>> I agree on the readline module (don't have a lot of need to text
>> file processing that sys.stdin.readlines() won't handle).
> ... Sure, sys.stdin.readlines() will do until I have to parse a
> daemon log or flat text db; then there's no good reason or even
> advantage to loading the whole file in memory at once.
I found an excellent example of this in the course of almost crashing
Using kpackage I tried to get the list of files included in the
mozilla-0.9.5 rpm that came with my Mandrake 7.2 distro. There were
a lot of files. Not to put too find a point on it, but if I hadn't
doubled the recommended size of my swap partition, kpackage would've
crashed the machine!
This is no big deal for the command-line rpm because it iterates over
the filenames as it reaches them, and doesn't have to store them a
single, huge memory glob.
...which is why:
for x in fileobject.xreadlines():
...is more scalable than:
for x in fileobject.readlines():
...and isn't scalability the buzzword that everyone drops when
promoting Python? =)
The biggest problem with communication is the illusion that it has occurred.
More information about the Python-list