[XML-SIG] Re: XML for scientific data storage and search

[Walter Underwood]

Instead of inventing something from scratch, I'd recommend an
existing, standard, flexible, free format, like NetCDF. It even
has Python bindings.

  http://my.unidata.ucar.edu/content/software/netcdf/

If you must invent it yourself, use Python marshal format over
HTTP. Works fine.

XML is wonderful stuff, but it is really unsuitable for large
amounts of data or fast transfer.

SAX should not require reading in the whole document, though.
It is almost always the right choice if you are extracting the
data from a document instead of manipulating it.

wunder

--On Friday, January 14, 2005 10:34:02 AM +0800 Ping Yeh <ping at pingyeh.net> wrote:

> That's true.  I'm willing to trade some speed for language neutrality.
> But after some study it seems the speed is really toooooooo slow...
> One big reason is that with DOM and SAX the whole data tree has to
> be built before I can do anything with the data.  Pull DOM is my
> hope of using smaller memory footprint to reduce memory allocation
> overheads in python.
>
> I may be forced to use binary format if I can't find better performance.
> I'm reluctant to do that because the type and amount of data my experiment
> will produce is not yet known exactly and adding flexibility to binary
> format is painful.
>
> Ping
>
> Walter Underwood wrote:
>> There are some XML speed issues that won't go away with a better parser.
>> Sending floating point numbers as formatted ASCII is never going to be
>> really fast.
>>
>> wunder
>>
>> --On January 13, 2005 3:33:49 PM +0800 Ping Yeh <ping at pingyeh.net> wrote:
>>
>>
>>> Thanks a lot for the reference pointers!  I'm now studying
>>> pull DOM, and will go through other modules later.  I'll make
>>> performance comparisons available just in case they might be useful.
>>>
>>> cheers,
>>> Ping
>>>
>>> Fredrik Lundh wrote:
>>>
>>>>>> But I haven't found any.  I'm not sure this is possible with current
>>>>>> architecture of parsers.  Any advise is highly appreciated.
>>>>>
>>>>> http://online.effbot.org/2004_12_01_archive.htm#element-generator
>>>>> http://online.effbot.org/2004_12_01_archive.htm#element-generator-2
>>>>
>>>>
>>>> also:
>>>>
>>>> http://www-106.ibm.com/developerworks/xml/library/x-tipulldom.html
>>>> http://cvs.sourceforge.net/viewcvs.py/splice/kid/pulltree.py?view=markup
>>>>
>>>> </F>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> XML-SIG maillist  -  XML-SIG at python.org
>>>> http://mail.python.org/mailman/listinfo/xml-sig
>>>
>>> _______________________________________________
>>> XML-SIG maillist  -  XML-SIG at python.org
>>> http://mail.python.org/mailman/listinfo/xml-sig
>>
>>
>>
>>
>> --
>> Walter Underwood
>> Principal Architect, Verity
>



--
Walter Underwood
Principal Architect
Verity Ultraseek