Memory Problem

[Marc 'BlackJack' Rintsch]

On Tue, 18 Sep 2007 14:06:22 +0200, Christoph Scheit wrote:

> Then the data is added to a table, which I use for the actual Post-Processing.
> The table is actually a Class with several "Columns", each column internally
> being represented by array.

Array or list?

> # create reader
> breader = BDBReader("<var>", "<type>", "#")
>  
> # read data
> bData = breader.readDB(dbFileList[0])
> 
> # create table
> dTab = DBTable(breader.headings, breader.converters, [1,2])
> addRows(bData, dTab)
> 
> Before I add a new entry to the table, I check if there is already an entry 
> like this. To do so, I store keys for all the entries with row-number in a
> dictionary. What about the memory consumption of the dictionary?

The more items you put into the dictionary the more memory it uses.  ;-)

> Here the code for adding a new row to the table:
> 
> # check if data already exists
> if (self.keyDict.has_key(key)):
>         rowIdx = self.keyDict[key]
>         for i in self.mutableCols:
>             self.cols[i][rowIdx] += rowData[i]
>         return
>             
>  # key is still available - insert row to table
>  self.keyDict[key] = self.nRows
>         
>  # insert data to the columns
>  for i in range(0, self.nCols):
>      self.cols[i].add(rowData[i])
>         
>  # add row i and increment number of rows
>  self.rows.append(DBRow(self, self.nRows))
>  self.nRows += 1
> 
> Maybe somebody can help me. If you need, I can give more implementation
> details.

IMHO That's not enough code and/or description of the data structure(s). 
And you also left out some information like the number of rows/columns and
the size of the data.

Have you already thought about using a database?

Ciao,
	Marc 'BlackJack' Rintsch