[sapug] Large dictionaries

Chris Foote chris at inetd.com.au
Thu May 11 15:15:28 CEST 2006


On Thu, 11 May 2006, Darryl Ross wrote:

>> p.s. Disk-based DBs are out of the question because most key lookups
>> will result in a miss, and lookup time is critical for this application.
>
> In memory sqlite database (use ':memory:' as the filename) with
> appropriate indexes?
>
> Make sure you put any inserts into a transaction. Using a disk based
> database, inserting 10,000 rows went from being a multi-minute operation
> to a split-second operation, just by putting them inside a BEGIN/COMMIT.

I just ran a test - the overhead of parsing SQL is too high:

               dictionary  metakit sqlite[1]
               ----------  ------- ---------
1M numbers       8.8s       22.2s     89.6s
2M numbers      24.0s       43.7s    190.0s
5M numbers      115.3s     105.4s       N/A

[1] pysqlite V1 & sqlite V3.



No go :-(

-- 
Chris Foote <chris at inetd.com.au>
Inetd Pty Ltd T/A HostExpress
Web:   http://www.hostexpress.com.au
Blog:  http://www.hostexpress.com.au/drupal/chris
Phone: (08) 8410 4566


More information about the sapug mailing list