Python for large projects?

[Alexander Williams]

On Sat, 26 Jun 1999 16:52:44 -0400, Darrell <news at dorb.com> wrote:
>This isn't a memory cycle but a problem still.
>Don't:
>    buf=open(file).read()
>    doTheBigJob(buf)
>        # buf won't be freed until this function exits. Which might be the
>entire run of the program.
>        # The idea is don't leave a reference at the top level
>        # Even worse is if at a lower level you want to change buf.  Try
>this and monitor memory usage.
>>>> s="        "*999999
>>>> b=s
>>>> b=b+" "
>
>Do something like this:
>   parm={}
>   parm['buf']=open(file).read()
>   doTheBigJob(parm)
>        # Now doTheBigJob can "del parm['buf']" to free the memory.
>        # Or pass the buffer with in an object of some kind.

Actually, this isn't fully necessary.  Borrowing from your first
example above, all you need is:

  buf = open(file).read()
  doTheBigJob(buf)
  del buf

No need to introduce new passing schema or wrap the buffer object in
another object at all; this will clean it up for you.

Mind you, ideally (for a number of reasons) you want to avoid opening
a file in the top level of a module /anyway/, better to embed the
opening and data-extraction in its own function which you then exit,
returning the space used by the variables therein to the pool
naturally.

If the only thing you're doing with the content of file is passing it
to doTheBigJob, then its even /better/ to simply call it as:

  doTheBigJob(open(file).read())

.... since that will only contain the content for the temporal extent
of the local its bound to within the function.

>Look out for:
>    dict1={}
>    dict1['dict']=dict1
>    # This and variations on this theme become immortal. Unless you kill it
>off carefully.

Cycles in any structure are dangerous and usually indicate a failure
of the author to understand the actual structure of the data he's
trying to model.  On the other hand, some data representations really
/do/ require recursive representations and, as such, probably bear
watching closely whether you're in a strictly reference-counting
environ or in a language implimentation with GC.

>6. No const or private.
>    You have to trust everyone.

This always struck me as a really /odd/ concern.  If you don't trust
the people on your programming team, why are they on your team?  If
you're afraid the hooks into your inner-code are visible, then rethink
why you're concerned.  Mathods/slots on Python objects can be prefixed
with _ and __ to make them harder to casually examine, but I've always
equated 'reference hiding' with 'security through obscurity,' it
doesn't protect you and it lends a false sense of hope.

-- 
Alexander Williams (thantos at gw.total-web.net)
"In the end ... Oblivion Always Wins."