[Tutor] Am I storeing up problems ?

[Dave S]

Danny Yoo wrote:

>On Sun, 2 Jan 2005, Dave S wrote:
>
>  
>
>>My matrix 'self.data' consists of a list of 110 items, each item is a
>>dictionary of 250 keys, each key holds two lists, one of four items, one
>>of 12 items.
>>    
>>
>
>Hi Dave,
>
>Hmmm... what kind of data is being copied here?  Python's data structures
>are desigined for flexibility, but sometimes that flexibility comes at the
>cost of some space.  If we knew more about the data that's being stored,
>maybe we can think of a more compact representation.
>
>  
>
OK my app is tracking the ftse 250, the 110 items are time slots through 
a trading day, the 250 keys are ftse company names added dynamicly , the 
two lists are (4) a series of linked cookies to transmit info forward 
per company, and (12) financial stats for the particular company at a 
particular time.

Companies in the 250 change randomly and there are not always exactly 
250 (realy!), a linked cookie system is needed for a variety of info 
going forward not least as a count of how long any particular company 
has been on the ftse 250.

The copy is needed so two days ftse figures can be viewed at any one time.

Thurther apps can pull out ftse figures for a particular company going 
backwards .. you get the idea !

It all works & its been a great learning experience to code.

>>I needed to copy this matrix to 'self.old_data', so I have been using
>>.deepcopy(), which works OK but is SLOW .... (12+ secs)
>>    
>>
>
>Perhaps we can avoid making a separate copy of the data?  Copying data
>structures can often be avoided.  Why does your program try to copy the
>data structure?
>
>(Aside: one nonobvious example where copying can be avoided is in Conway's
>Game of Life:  when we calculate what cells live and die in the next
>generation, we can actually use the 'Command' design pattern to avoid
>making a temporary copy of the world.  We can talk about this in more
>detail if anyone is interested.)
>
>
>
>  
>
>>Once the matrix is copied, 'self.data' is re-constructed as the
>>programme gathers data.
>>
>>To speed things up I changed my code to
>>
>># This is the speeded up deepcopy()
>>self.old_data = self.data
>>self.data = []
>>for i in range(110):
>>    self.data.append({})
>>
>> Query: Is all this OK, it works and quick as well, but I am concerned I
>>may be leaving garbage in the Python PVM since I am shrugging off old
>>'self.old_data's which may be mounting up ?
>>    
>>
>
>
>Whenever we reassign to self.old_data:
>
>    self.old_data = self.data
>
>then whatever self.old_data was pointing at, before the assignment, should
>get garbage collected, if there are no other references to the old data.
>
>The code there isn't really doing any copying at all, but is rather just
>constructing a whole new empty data structure into 'self.data'.  If so,
>then it does seem that you can avoid copying.
>
>I'd love to know a little bit more about the data that the program is
>collecting; if you have time, please feel free to tell us more details.
>Good luck to you!
>
>
>
>  
>