persistent deque (continued)

M.-A. Lemburg mal at
Mon Jul 21 23:54:21 CEST 2008

On 2008-07-21 21:08, castironpi wrote:
> Some time ago, I was asking about the feasibility of a persistent
> deque, a double-ended queue.

You might want to have a look at mxBeeBase:

Using the integer index you could probably write an on-disk

The B*Tree indexes available in mxBeeBase keep the indexes sorted,
so you'd only have to keep incrementing the index as you add new
values and keep a record of the highest and lowest index currently
in use.

Details are left as exercise for the interested reader ;-)

> It runs into the typical space allocation problems.  If you're storing
> a pickle, you have to allocate and fragment the file you've opened,
> since pickles can be variable-length strings; i.e. if the new data is
> too long, blank out its entry, and grow the file.  If you're storing a
> data-type, you lose Python's dynamic-type advantages, as well as its
> long integers, as they can be any length.  If you change the object in
> the deque, such as when using any mutable type, you have to update the
> container too.
> Does anyone have any experience storing pickles (I am aware of the
> similarities to shelf) to a database?
> Can the file system facilitate the variable-length string problem?
> How feasible is a length-of-length - length - data solution to the
> unbounded integer problem?
> Is there any alternative to completely re-pickling a large (say 1k
> pickled) object you only change slightly?
> What other issues are there?
> Is a hybrid-storage type possible, that stores the contents of its
> Python-allocated memory block to disk, including reference count, even
> if it's a lot slower?  The object could not contain any references to
> objects not allocated on disk.
> A first approach is for the file to look like this:
> 00 data 01 data 02
> 01 data 03 data 04
> 02 data 05 data 06
> Append would add:
> 03 data 07 data 08
> AppendLeft would add:
> -01 data 09 data 0a
> Pop would remove 03, PopLeft would remove -01.  You would need a
> length-and-head index to make 'rotate' available.  Remove would run a
> worst-case risk of renumbering half of the indices stored, plus a
> rotate.
> --

Marc-Andre Lemburg

Professional Python Services directly from the Source  (#1, Jul 21 2008)
 >>> Python/Zope Consulting and Support ...
 >>> mxODBC.Zope.Database.Adapter ...   
 >>> mxODBC, mxDateTime, mxTextTools ...

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the Python-list mailing list