Guido van Rossum wrote:
Someone has asked me for a dbm clone that can store 16M keys of 350 bytes each, and runs on Linux, HPUX, and NT. That's 5.6 Gigabyte in keys alone! I presume most classic approaches won't cut it since total file size is typicall limited by the seek system call, internal data structures and/or file index format to 2Gb (signed longs) or 4Gb (unsigned longs).
Does anyone have an idea where to start looking? Would a Python extension already exist?
I'd suggest using a dbm style wrapper around the DB-API and then trying out the many cross-platform databases. IBM DB2 comes to mind... it can certainly handle these sizes given the right hardware.