<HTML dir=ltr><HEAD></HEAD>
<BODY>
<DIV><FONT face=Arial color=#000000 size=2>Hi,</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>I'm a newbie at python but we are using it daily in our research, so I'm getting the hang of it.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>My RA and I are developing about 30 large unicode xml databases, averaging about 100 Mb each. They have to retain their XML format as they are used in a specialised program designed to read them in as they are.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>We use Python to do our programming logic, which so far is mainly data manipulation. For example, each day we read through the data, which we save *for the time being* in python dictionaries (and thus they are about 10Mb each!) many times, processing various values to modify existing and create new values.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Our question is this: when we finish porting our 300Mb "python" data into 3Gb of XML data, how can we continue to read it from disk in its xml format and manipulate it?</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>We are looking at Berkeley XML with the Python API, but are concerned this is not the best solution. we have also dabbled with Amara and ElementTree, but the size our our XML is giving us problems.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>We want to focus on the programming logic, which is all we do in python, and working with python structures is great, but this is not a viable option (as far as we can tell) once we move to XML. </FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Or is there a way to read large XML files into python structures and then write them out again? </FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Our ideal solution (we think!) would be to have an xml database whose elements we could directly modify using python scripts and where the disk and memory handling is done for us somehow.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Any advice would be appreciated. </FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Regards,</FONT></DIV>
<DIV><FONT face=Arial size=2>Matthew</FONT></DIV>
<DIV><FONT face=Arial size=2>Postdoctoral Fellow, Charles Sturt University</FONT></DIV></BODY></HTML>