[sapug] Large dictionaries
chris at inetd.com.au
Thu May 11 15:15:28 CEST 2006
On Thu, 11 May 2006, Darryl Ross wrote:
>> p.s. Disk-based DBs are out of the question because most key lookups
>> will result in a miss, and lookup time is critical for this application.
> In memory sqlite database (use ':memory:' as the filename) with
> appropriate indexes?
> Make sure you put any inserts into a transaction. Using a disk based
> database, inserting 10,000 rows went from being a multi-minute operation
> to a split-second operation, just by putting them inside a BEGIN/COMMIT.
I just ran a test - the overhead of parsing SQL is too high:
dictionary metakit sqlite
---------- ------- ---------
1M numbers 8.8s 22.2s 89.6s
2M numbers 24.0s 43.7s 190.0s
5M numbers 115.3s 105.4s N/A
 pysqlite V1 & sqlite V3.
No go :-(
Chris Foote <chris at inetd.com.au>
Inetd Pty Ltd T/A HostExpress
Phone: (08) 8410 4566
More information about the sapug