SQLite or files?
python.list at tim.thechases.com
Thu Sep 17 12:26:08 CEST 2009
> info was stored in the simple RTF files.
> However now I'd like to rewrite this program in Python (using PyQt) as
> I want to make it cross-platform and add/remove some features. Now I'm
> thinking about where to store my information. Would it be better to
> use files as I used to do or use the database, SQLite in particular?
> What will be faster and more flexible in the long run? This
> application will be in the memory most of the time so I'm also
> concerned about memory usage.
Not knowing what you do with the files, nor what sort of data
they contain makes it a bit difficult to make suggestions in
light of the context. However:
- sqlite will let you perform arbitrary queries against your
data, so if you want to aggregate or perform complex conditional
tests, it's just SQL.
- I don't know if you're currently keeping the RTF in memory the
whole time, or if you repeatedly reload (whether in one go, or
streaming it) and reparse the file. This sounds memory and/or
processor intensive. Using sqlite, the processing is done at the
C-module level, the data is kept on disk and only brought into
memory as-requested, being released when you're done with it.
- concurrently sharing a sqlite database should have minimal
issues. Sharing RTF files concurrently means locking/contention
issues. May not be an issue for you.
- sqlite comes built-in with Python2.5+ while RTF processing is
not batteries-included from what I can tell
So in the general case, I see sqlite being a notable win over
RTF. Depending on your data, if it's just key/value pairs of
strings, you might look into the anydbm module which has an
even simpler interface.
More information about the Python-list