reading large file
alleon at club-internet.fr
Fri Sep 5 08:26:12 CEST 2003
Thanks to your comments, it is now possible to read my large file in a
couple of minutes
on my machine.
"Bengt Richter" <bokr at oz.net> a écrit dans le message de news:
bj5e61$pjr$0 at 18.104.22.168...
> On 3 Sep 2003 05:00:39 -0700, g_alleon at yahoo.fr (guillaume) wrote:
> >I have to read and process a large ASCII file containing a mesh : a
> >list of points and triangles.
> >The file is 100 MBytes.
> >I first tried to do it in memory but I think I am running out of
> >memory therefore I decide to use the shelve
> >module to store my points and elements on disks.
> >Despite the fact it is slow ... Any hint ? I think I have the same
> >memory problem but I don't understand why
> >since my aPoint should be removed by the gc.
> >Have you any idea ?
> Since your data is very homogeneous, why don't you store it in a couple of
> homogeneous arrays? You could easily create a class to give you convenient
> access via indices or iterators etc. Also you could write load and store
> methods that could write both arrays in binary to a file. You could
> consider doing this as a separate conversion from your source file, and
> then run your app using the binary files and wrapper class.
> Arrays are described in the array module docs ;-)
> I imagine you'd want to use the 'd' type for ponts and 'l' for faces.
> Bengt Richter
More information about the Python-list