[IronPython] Debugging hosted python scripts

[Dino Viehland]

Jeff wrote:
> On Wed, Mar 30, 2011 at 8:52 PM, Keith Rome <rome at wintellect.com>
> wrote:
> > I have been going down the path of using ScriptEngine.SetTrace() and
> > inspecting frames in the callback. This works fine if I am not doing
> > anything interactive. For example, dumping some information to
> > Debug.WriteLine(). However what I really need (I think?) is to be able
> > to suspend the script execution during a trace callback. I don't see a
> > clear way to do this though. The script runtime simply continues
> > execution when my callback returns. I have done some work around
> > running the debugged script on a background thread, and then blocking
> > it during "breakpoint" callbacks - but these scripts are normally run
> > within the UI thread because they interact with data structures that
> > are often databound to UI controls, and running them from a background
> > thread is becoming a minefield of cross-thread violations. I cannot
> > simply run the script in the UI thread, because blocking in the trace
> > callback would make the application unresponsive.
> 
> I think this is going to be your biggest issue with debugging - AFAIK the
> Python engine is not designed to be suspendable; it relies on .NET's normal
> thread suspension mechanism to handle that case. If the scripts are on the UI
> thread, and the debugger is on the UI thread, that 's going to be an issue.
> 
> Now, if the engine were running in interpreted mode (i.e. it doesn't try to
> convert Python to IL), it would probably be possible to suspend it without
> suspending the thread, but I have no idea how much work that would be.
> (Heck, it might already support it - Dino?) My hunch is that it wouldn't be a
> huge amount of work, but I'm not familiar with the interpreter loop at all.

There's certainly nothing that does this today - this is effectively "how to do
stackless on IronPython."  There've been similar questions on the mailing list
on how to do this (for greenlets) in the past.  There's basically two changes
which need to happen to get this working:

	1. We need to re-write the trees we produce so all local variables are
hoisted into a heap allocated data structure.  We already do this for generators
and for sys.settrace support so that one's not so hard.
	2. We need to re-write the trees so that rather than performing a dynamic
operation they yield control back to a loop which then performs the dynamic 
operation.
	3. We might need to do #2 but with certain built in calls (for example 
import if you don't want imports to block, maybe some other built-in operations
as well).

Doing #2 isn't that difficult either - it's really just a tree rewrite that changes each
Dynamic node (or one of the DLR outer layers dynamic like nodes) into a node which
returns a call site plus arguments.  The outer loop can then dispatch into the call
site or it can yield appropriately.  

Finally this needs to be combined with the debugging mechanism which it's self
is just an AST re-write.  sys.settrace uses the same basic rewrite we need for #1 to 
hoist the variables.  In addition to that it introduces the line, call, exception, etc...
callbacks as well.  I think the final tweak here would be to make those yield control
back to the dispatch loop which then can then make the sys.settrace call.

Finally you'll probably need to update functions so that if they're called "externally"
(not directly from the dispatch loop) that they setup a new dispatch loop.  So the
delegates that are held in the FunctionCode object will need to be distinct from the
normal delegates (this is similar to how we have a normal delegate and a light throwing
delegate today).  

I'd suggest by trying to add the stackless re-write and dispatch loop to the DebugInfoRewriter
class (or a fork of it) and then getting simple stackless function calls going.    Then you can
worry about fitting it in with the rest of the system.

I could even see us adding this back to the core if it worked out to be sufficiently on the side
which if implemented just as these AST rewrites it could be.