Running queries on large data structure

[Amit Khemka]

On 8/3/06, Christoph Haas <email at christoph-haas.de> wrote:
> On Wednesday 02 August 2006 22:24, Christoph Haas wrote:
> > I have written an application in Perl some time ago (I was young and
> > needed the money) that parses multiple large text files containing
> > nested data structures and allows the user to run quick queries on the
> > data. [...]
>
> I suppose my former posting was too long and concrete. So allow me to try
> it in a different way. :)
>
> The situation is that I have input data that take ~1 minute to parse while
> the users need to run queries on that within seconds. I can think of two
> ways:
>
> (1) Database
>     (very quick, but the input data is deeply nested and it would be
>      ugly to convert it into some relational shape for the database)
> (2) cPickle
>     (Read the data every now and then, parse it, write the nested Python
>      data structure into a pickled file. The let the other application
>      that does the queries unpickle the variable and use it time and
>      again.)

Though some sugggested maintaining data in some XML structures, I was
wondering that
if you considered using some native XML database like BDB XML.

1. It allows you to retain hierarchical structure of data.
2. It also has support for precompiled queries.
3. You can open a persistent connection.

cheers,
amit.

-- 
----
Amit Khemka -- onyomo.com
Home Page: www.cse.iitd.ernet.in/~csd00377
Endless the world's turn, endless the sun's Spinning, Endless the quest;
I turn again, back to my own beginning, And here, find rest.