memory usage

[Batista, Facundo]

It could be very useful if you include in the mail your hardware/software
configuration.

.	Facundo


#- -----Mensaje original-----
#- De: Nagy Gabor [mailto:linux42 at freemail.c3.hu]
#- Enviado el: Martes 6 de Mayo de 2003 10:21 AM
#- Para: Python
#- Asunto: memory usage
#- 
#- 
#- I wrote a simple datafile parser, and it is quite memory 
#- hungry, and I
#- don't know if this is what I should expect, or is there a 
#- bug in my code.
#- 
#- I can profile the CPU time spent here and there, but I have 
#- no information
#- about the memory used meanwhile, nor about the time spent 
#- handling this
#- memory.
#- 
#- My input file is cleartext, with fixed width fields.
#- 
#- My first input to test with was about 18MiB long, about 115k lines.
#- I ran out of physical memory parsing this file, then I 
#- cancelled, as the
#- swapping took ages.
#- Then I trimmed the file to 15k lines, about 2,3MiB in size.
#- It took about 250MiB of virtual memory.
#- Is this normal? I hope I don't keep extra copies around to 
#- multiple the
#- memory needed.
#- 
#- OK, what do I want to keep in memory? Here is some of the code.
#- 
#- I have two classes, T, and TD.
#- 
#- def Parse():
#-   recordset = TD()
#-   recordset.Tag = T(Name = 'recordset', Flag=1)
#-   recordset.Data = []
#-   while 1:
#-     record = ParseRecord()
#-     recordset.append(record)
#- 
#- def ParseRecord():
#-   record = TD()
#-   record.Tag = T( Value='42', Name = 'record', Class='C', Flag=1)
#-   record.Data = ParseFields()
#-   return record
#- 
#- def ParseFields():
#-   fields = []
#-   for ...:
#-     Data = StringIO.read( length)
#-     tmp = TD()
#-     tmp.Tag = T(name = 'name')
#-     tmp.Data = Data
#-     fields.append(tmp)
#-   return fields
#- 
#- class T:
#-     def __init__(self, Value = '', Flag = 0, Name = '', Class = ''):
#-         self.Flag = Flag
#- 	self.Value = Value
#- 	self.Class = Class
#- 	self.Name = Name
#- 
#- class TD:
#-     def __init__(self):
#-         self.Tag = T()
#- 	self.Data = None
#- 
#- That's all. Storing the whole text file in the Data 
#- attributes of TDs is
#- OK. But what is the remaining 247MiB?
#- T.Value is two characters,
#- T.Name is about 5-25 characters
#- T.Class is always 'C'
#- 
#- Parse was called once, took 190s cumtime
#- ParseRecord was called 15000 times, 186s cumtime
#- ParseFields was called 14999 times, 176s cumtime
#- T.__init__ was called 599901 times, 16.5s cumtime
#- TD.__init__ was called 599901 times, 76.8s cumtime
#- 
#- I don't understand why TD.__init__ took 76.8s, (doing almost 
#- nothing), and
#- I don't understand what my memory was used for.
#- 
#- Can someone please explain what goes on, when I pass around 
#- objects, etc.
#- What is the (memory) overhead of having a list, an instance 
#- of a class, a
#- string, etc.
#- 
#- Regards, Gee
#- 
#- -- 
#- http://mail.python.org/mailman/listinfo/python-list
#-