[Pythonmac-SIG] PackageManager / Python 2.3.x - 2.4 ideas

[Donovan Preston]

On Friday, August 1, 2003, at 6:26 AM, Schollnick, Benjamin wrote:
>
> This is a little too general...
>
> Concurrency does require a good examination of what is happening, and 
> the
> project flow...

This may be possible for smaller programs, but since threading 
introduces
nondeterministic context switching (a context switch may happen at any 
time),
I would argue that this is impossible for a program of sufficient 
complexity.

> But what I found worked in multiple cases, was reducing the issue, 
> until I
> had
> the "actions" being handled by a object.  Thus I would create multiple
> objects, and
> pass them into the threads for servicing.  Once the thread finished, I 
> would
> retrieve
> the data from the object, destroy the object, and continue on...

How do you know the thread is finished? Do you have a callback 
mechanism?
Or Polling?

> Yes, this model does not work every where...  But often, when working 
> in a
> OOP model,
> I keep coming back to a similiar model...

So how is this any different than making your API properly 
asynchronous? If
you are already dividing your functions up into "actions", you might as 
well be
handling control flow explicitly through the use of callbacks.

The problem with threading is nondeterminacy. Since a context switch can
happen at any time, and is beyond the control of the programmer, your
program may work properly 99.999999999% of the time. However, it may
fail 0.000000001% of the time due to lock contention issues (deadlocks) 
or
data-corruption issues due to state sharing between threads, and when it
does, it is utterly impossible to debug, because chances are the 
deadlock
condition will be impossible to reproduce in the debugger due to timing
issues.

It is mathematically impossible to prove the correctness of a threaded
program. The number of possible control flows is just too large. Would
you rather be up all night debugging deadlocks (because all programs
have bugs) or doing real work?

But, you say, a single-threaded application does not take advantage of 
my
dual 2ghz G5 machine with it's two processors. Well, for one, neither 
does
Python. So the only way to take advantage of both of those processors is
to fork another process and explicitly share state using IPC mechanisms,
which is what you should be doing anyway. Operating systems implement
protected memory mechanisms and process context switching for a reason.

> Threading in a non-OOP environment is harder...  When I rewrote parts 
> of
> Stathost,
> and made it into a multithreaded application, to speed it up...  I had 
> to
> basically
> graft a OOP manager on top of the system for the threading...  But it 
> was
> not difficult..

There's no excuse for not using an asynchronous model in a language like
Python which includes many features which make it very easy and 
convenient.
Bound method references, closures, and lambdas make it very easy to 
capture
state for use in your callbacks, while automatically and correctly 
managing the
reference counting for garbage collection purposes.

dp