Scripting *of* Python

[Phil Mayers]

I thought I posted this already, but I can't see it anywhere - apologies
if this is a repeat.

I've got an application (actually a framework) written in Python based on
a medusa-like framework for asynchronously talking to thousands of remote
services in parallel. I have been unable to equal the performance of this
using a 1:1 or 1:N (pooled) thread model, so I need to stick with the
async version.

However, I have a problem - the code you write in such an application is
fairly... complex. The app acts as the client, the endpoints as the
server, so the logic looks like this:

def func(endpoint,a,b):
	# do some more get/set/process
	pass

val1 = get("var1")
if val1==1:
	tab = {}
	for index in set("var2", "val2"):
		tab[index] = get("var3:"+index),get("var4:"+index)
	for index,(a,b) in tab.items():
		func(index,a,b)

...but this ends up reading:

def me(*args):
	if not args:
		# First time
		queue("get","var1","val1")
		queue("call",me,"var1","val1")
		return
	elif args[1]=='var1':
		# args = ('var1','val1')
		args[2]==1:
			queue("set","var2","val2")
			queue("call",me,"index",LASTRETVAL)
			return
	elif args[1]=='index':
		tab = {}
		for index in args[2]:
			queue("get","var3:"+index,"val3:"+index)
			queue("get","var4:"+index,"val4:"+index)
			queue("call",me,"item",index,
				"val3:"+index,"val4:"+index,tab)
		queue("call",me,"done",tab)
		return
	elif args[1]=='item':
		index,val3,val4,table = args[2:]
		table[index] = (val3,val4)
		queue("call",func,index,val3,val4)

It's a contrived example, but I'm sure you see what I mean - the function
cannot block for very long, so it has to queue itself. This code quickly
gets unreadable (the code it *would* be in a thread is over 1k lines!) and
non-technical (well, less technical) people have to be able to write it.

I considered Stackless, but there appears to be no equivalent of the
select() timeout on channels and that makes things tough - then I came up
with the bright idea of *scripting* the Python engine.

Basically, you have a compiled bytecode and "stack" for each "thread", and
when you come across an opcode which would block, you store the state and
return from the "run_stack()" function, the scheduler runs, and all is
well.

The question is - does anyone know of such a thing i.e. a bytecode script
engine *for* a Python appliction - something like a JavaScript or lua
engine where I can hook enough of the system to implement
microthreads/fibres/whatever you want to call them.

Cheers,
Phil