handeling very large dictionaries

Aahz aahz at pythoncraft.com
Sun Jun 28 22:39:00 EDT 2009


In article <9efff087-bd6e-49fb-ad30-a955a64b85f9 at j32g2000yqh.googlegroups.com>,
mclovin  <hanooter at gmail.com> wrote:
>
>I need to have a dictionary of about 8 gigs (well the data it is
>processing is around 4gb). so naturally i am running into memory
>errors.
>
>So i looked around and found bsddb which acts like a dictionary object
>only offloads the data from the RAM to the HDD, however that only
>supports strings.

Look at the pickle module.  Personally, I'd use SQLite instead of bsddb.
You might also look into a full-blown ORM such as SQLAlchemy.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha



More information about the Python-list mailing list