[Tutor] pickling?

[Jeff Shannon]

Jeff Peery wrote:

> hello, I am not sure what pickling is, could someone give me a quick 
> synopsis for a newbie?
>  
> the reason I ask is that I think it might be useful for an application I 
> am working on. Is pickling the process of storing data for later use? I 
> have a system that measures liquid flow rate and we do regular 
> measurements with flow meters, and I want to setup a statistical quality 
> control system. the fiirst thing I am doing is sorting the measurement 
> into categories and then I will do the stats. anyhow, I will have to 
> store lots of information in a file and access it regularly to read and 
> append new measurements.  Is pickling a good way to store matricies and 
> then pick then up later for use in python?

Pickling is indeed storing data for later use.  However, from what 
you've said about your project, you may be better off using a simple 
database rather than plain pickles.

A pickle is just a way of storing a Python object on disk (or sending 
it across a network).  In order to use it, you would first unpickle 
any old data that you have, append any new data to it, and then pickle 
the whole bundle again.  If you've really got lots of data, that 
unpickling and re-pickling could get pretty expensive, even if you 
only do it at startup and shutdown.  (And if you repickle only at 
shutdown, you risk losing data when your program crashes...)

A fairly lightweight database, however, will be much more efficient 
for storing large amounts of data.  You won't have to do nearly as 
much unpickle-repickle style dancing; the database handles all that 
stuff for you transparently.

I've never needed to use such a database myself, so I can't offer any 
specific advice.  But I've heard people (here and on comp.lang.python) 
speak very well of mysql and sqlite, and their respective Python 
language bindings.

Jeff Shannon
Technician/Programmer
Credit International