Scripting *of* Python

[Phil Mayers]

On Thu, 30 Jan 2003 12:31:23 +0000, Aahz wrote:

> In article <b1bj7a$i9o$1 at jura.cc.ic.ac.uk>,
> Phil Mayers <p.mayers at imperial.ac.uk> wrote:
>>
>>I've got an application (actually a framework) written in Python based on
>>a medusa-like framework for asynchronously talking to thousands of remote
>>services in parallel. I have been unable to equal the performance of this
>>using a 1:1 or 1:N (pooled) thread model, so I need to stick with the
>>async version.
> 
> Based on what you've said, I'd suggest looking into switching to
> Twisted, http://www.twistedmatrix.com/products/twisted

I have seen Twisted. It seems to be a very popular recommendation in c.l.p
of late ("I have incurable narcolepsy", "I recommend using Twisted" - :o)
However, the main problem is not with the async engine, which works fine,
it's with the code that runs *inside* the async engine *looking like*
async code.

My brain can cope with that - the people I'm writing it for probably
cannot, and I'm being paid to come up with a solution that middle- to
low-tier programmers can drive.

http://www.twistedmatrix.com/documents/howto/async

...this document seems to imply that the only extra stuff Twisted gives
you is the Deferred class/type for return values, which is basically
what I've written in my engine. In fact, the Twisted doco more succinctly
summarises the problem:

"""Since non-volatile state cannot be kept in local variables, because
each method must return quickly, it is usually kept in instance variables.
In cases where recursion would have been tempting, it is usually necessary
to keep stacks manually, using Python's list and the .append and .pop
method. Because those state machines frequently get non-trivial, it is
better to layer them such that each one state machine does one thing --
converting events from one level of abstraction to the next higher level
of abstraction. This allows the code to be clearer, as well as easier to
debug."""

Yes, quite - but I don't think this state machine can be abstracted out to
be "clear enough" - hence the scripting language and virtual
multithreading suggestion.

I will almost certainly switch to using Twisted if I can:

a) Shoehorn the virtual multithreading idea in, or find a better one
b) Find the time before I need the code working

...since it's clearly going to be better than an async engine I can write.

Thanks for the pointer though!

Cheers,
Phil