The whole story

[andrew cooke]

Paul Hemans wrote:
> Hi Andrew,
> The reason I am using mapped objects is that I need to abstract from the
> database implementation allowing the replication to target a number of
> different platforms. This will definitely slow things down.

have you looked at sqlalchemy's generic sql support?  you can construct
sql "statements" using python classes/functions called "select", "update"
etc and sqlalchemy automatically makes things work with different database
implementations.  it's not perfect, but i have the same code working with
mysql and oracle, which covers quite a range :o)

>> process a whole pile in memory and then (perhaps every 10,000 - when
>> your
>> memory is about to run out and start paging) flush the session.
> Under windows how can I tell when memory is about to run out? I guess
> there
> is no cross-platform solution to this.

i don't know, sorry.

> Writing external files has all come about from a post titled "Memory
> problems (garbage collection) by Carbon Man" which I never got a
> resolution
> to.
> I was trying to execute gc.collect() when a process was complete because I
> was having huge problems with memory (though the problem still remains).
> If
> I stop at "import schema" There are 2524104 objects processed by
> gc.collect()

you shouldn't need to call gc.collect().

when you write everything out to the database, if you have no other
references to the objects, python will clean them up automatically. 
calling gc.collect() won't make any difference - python's gc already works
just fine.

if your memory use isn't going down then either (1) you are not writing
out/flushing correctly (if you want i can check my code later today and
tell you exactly what i do) or you are keeping references to your objects
elsewhere (eg in a dictionary in the code you use to construct them).

andrew