reading large file
bokr at oz.net
Wed Sep 3 21:07:45 CEST 2003
On 3 Sep 2003 05:00:39 -0700, g_alleon at yahoo.fr (guillaume) wrote:
>I have to read and process a large ASCII file containing a mesh : a
>list of points and triangles.
>The file is 100 MBytes.
>I first tried to do it in memory but I think I am running out of
>memory therefore I decide to use the shelve
>module to store my points and elements on disks.
>Despite the fact it is slow ... Any hint ? I think I have the same
>memory problem but I don't understand why
>since my aPoint should be removed by the gc.
>Have you any idea ?
Since your data is very homogeneous, why don't you store it in a couple of
homogeneous arrays? You could easily create a class to give you convenient
access via indices or iterators etc. Also you could write load and store
methods that could write both arrays in binary to a file. You could
consider doing this as a separate conversion from your source file, and
then run your app using the binary files and wrapper class.
Arrays are described in the array module docs ;-)
I imagine you'd want to use the 'd' type for ponts and 'l' for faces.
More information about the Python-list