(in memory) database

[Bruno Desthuilliers]

mark a écrit :
> Hi there,
> 
> I need to extract data from text files (~4 GB) on this data some
> operations are performed like avg, max, min, group etc. The result is
> formated and written in some other text files (some KB).
> 
> I currently think about database tools might be suitable for this. I
> would just write the import from the text files and ... the tool does
> the rest. The only problem I can imagine is that this would not be
> fast enough.

Is this an a priori, or did you actually benchmark and found out it 
would not fit your requirements ?

> But I would give it a shoot.
> Unfortunately I have only some knowledge of SQLite which is not an
> option here.
 >
> Some additional requirements I can think of are:
> - Python (I want to hone my programming skills too)
> - Python-only (no C-lib) for simplicity (installation, portability).
> Therefore SQLite is not an option
> - must be fast

These two requirements can conflict for some values of "fast".

> - I like SQL (select a, b from ...) this would be nice (row[..] + ...
> is a little hard getting used to)
> 
> So far I found PyDBLite, PyTables, Buzhug but they are difficult to
> compare for a beginner.

Never used any of them - I have sqlite, mysql and pgsql installed on all 
my machines -, so I can't help here.