dicts,instances,containers, slotted instances, et cetera.

[Aaron Brady]

On Jan 28, 2:38 pm, ocsch... at gmail.com wrote:
> Hi, all.
>
> I have an application that that creates, manipulates, and finally
> archives on disk 10^6 instances of an object that in CS/DB terms is
> best described as a relation.
>
> It has 8 members, all of them common Python datatypes. 6 of these are
> set once and then not modified. 2 are modified around 4 times before
> the instance's archving. Large collections (of small lists) of these
> objects are created, iterated through, and sorted using any and all of
> the 8 members as sorting keys.
>
> It neither has nor needs custom methods.
>
> I used a simple dictionary to create the application prototype. Now I
> need to speed things up.
> I first tried changing to a new style class, with __slots__, __init__,
> __getstate__& __setstate__ (for pickling) and was shocked to see
> things SLOW down over dictionaries.
>
> So of these options, where should I go first to satisfy my need for
> speed?
>
> 0. Back to dict
> 1. old style class
> 2. new style class
> 3. new style class, with __slots__, with or without some nuance I'm
> missing.
> 4. tuple, with constants to mark the indices
> 5. namedTuple
> 6. other...

Hello, quoting myself from another thread today:

There is the 'shelve' module.  You could create a shelf that tells you
the filename of the 5 other ones.  A million keys should be no
problem, I guess.  (It's standard library.)  All your keys have to be
strings, though, and all your values have to be pickleable.  If that's
a problem, yes you will need ZODB or Django (I understand), or another
relational DB.

There is currently no way to store live objects.