[Twisted-Python] integrating a python REPL into my existing twisted application
Dear twisted-python, I’m trying to integrate a python REPL into my existing twisted 8.2.0 client-side program (which talks to another twisted server running a custom protocol). This client side program uses twisted.internet.stdio.StandardIO already to interact with the user, to read commands from the user and send them to the server. What’s the best way to add functionality to drop into a python REPL? I adapted the doc/core/examples/stdin.py code below to drop into the ‘code’ stdlib module to try to model for the mailing list what my larger client program is trying to do. It works fine on linux, until you enter a command which outputs a large amount of data, at which point you get a “Resource temporarily unavailable” as can be seen below. I’m assuming the issue is related to StandardIO taking control of reads/writes to the stdio file descriptors, and not integrating well with python code that writes directly to sys.stderr, but I’m not sure what the right solution is. I don’t mind that the call to ‘code.interact()’ blocks the reactor for my use case, but I don’t mind if better solutions are available which don’t have that problem, either. Modified doc/core/examples/stdin.py: --------------------------------------- """ An example of reading a line at a time from standard input without blocking the reactor. """ from twisted.internet import stdio from twisted.protocols import basic class Echo(basic.LineReceiver): from os import linesep as delimiter def connectionMade(self): self.transport.write('> ') def lineReceived(self, line): self.sendLine('Echo: ' + line) self.transport.write('> ') import code code.interact() def main(): stdio.StandardIO(Echo()) from twisted.internet import reactor reactor.run() if __name__ == '__main__': main() --------------------------------------- A sample run: $ ./twisted-stdio.py line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/sw/external/twisted-8.2.0/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/sw/external/twisted-8.2.0/lib/python/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/sw/external/twisted-8.2.0/lib/python/twisted/internet/process.py", line 242, in doRead return fdesc.readFromFD(self.fd, self.dataReceived) File "/sw/external/twisted-8.2.0/lib/python/twisted/internet/fdesc.py", line 70, in readFromFD callback(output) File "/sw/external/twisted-8.2.0/lib/python/twisted/internet/process.py", line 245, in dataReceived self.proc.childDataReceived(self.name, data) File "/sw/external/twisted-8.2.0/lib/python/twisted/internet/_posixstdio.py", line 73, in childDataReceived self.protocol.dataReceived(data) File "/sw/external/twisted-8.2.0/lib/python/twisted/protocols/basic.py", line 231, in dataReceived why = self.lineReceived(line) File "./twisted-stdio.py", line 27, in lineReceived code.interact() File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 306, in interact console.interact(banner) File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 243, in interact more = self.push(line) File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 265, in push more = self.runsource(source, self.filename) File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 87, in runsource self.runcode(code) File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 107, in runcode self.showtraceback() File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 162, in showtraceback map(self.write, list) File "/sw/external/python-2.6.1/lib/python2.6/code.py", line 171, in write sys.stderr.write(data) exceptions.IOError: [Errno 11] Resource temporarily unavailable My colleague discovered the below workaround which temporarily changes the nonblocking status of sys.stderr (and restores it afterwards), but feels like a hack which violates across abstraction layers and which might not work correctly in future versions of twisted: + import fcntl + oldval = fcntl.fcntl(sys.stderr.fileno(), fcntl.F_GETFL) + fcntl.fcntl(sys.stderr.fileno(), fcntl.F_SETFL, oldval & ~os.O_NONBLOCK) code.interact() + fcntl.fcntl(sys.stderr.fileno(), fcntl.F_SETFL, oldval) So, any tips on integrating a REPL into our existing program without resorting to this hack would be appreciated! Thanks. -- Benjamin
Benjamin Rutt wrote:
Use twisted.conch.manhole.ManholeInterpreter. Here's its docstring: class ManholeInterpreter(code.InteractiveInterpreter): """Interactive Interpreter with special output and Deferred support. Aside from the features provided by L{code.InteractiveInterpreter}, this class captures sys.stdout output and redirects it to the appropriate location (the Manhole protocol instance). It also treats Deferreds which reach the top-level specially: each is formatted to the user with a unique identifier and a new callback and errback added to it, each of which will format the unique identifier and the result with which the Deferred fires and then pass it on to the next participant in the callback chain. """ -Andrew.
On 02:27 am, rutt.4@osu.edu wrote:
In addition to what Andrew said, I'll also point out that rather than making code that expects blocking writes to stdout work by putting stdout back into blocking mode, you can make it work by instead providing a stdout which will buffer anything that cannot immediately be written non-blockingly. StandardIO gives you exactly this; its `write` knows how to deal with non-blocking descriptors and buffer as necessary. This is why manhole doesn't have the same problem as your code calling directly into `code.interact`. For details, <http://twistedmatrix.com/trac/browser/trunk/twisted/conch/manhole.py#L45>. Jean-Paul
On Thu, May 26, 2011 at 9:38 AM, <exarkun@twistedmatrix.com> wrote:
Thanks. I'm having trouble integrating this or Andrew's idea into my example. The simplest case doesn't quite work right for me (I see '>>>' prompts, and can interact with the python interpreter but the banner appears at the end of the REPL after I press Ctrl+D, and then I never see the '>' prompts from my original program anymore)...was this what you had in mind? See the diff below. @@ -6,6 +6,7 @@ without blocking the reactor. """ +import sys from twisted.internet import stdio from twisted.protocols import basic @@ -22,7 +23,8 @@ code.interact() def main(): - stdio.StandardIO(Echo()) + stdio_obj = stdio.StandardIO(Echo()) + sys.stderr = stdio_obj from twisted.internet import reactor reactor.run() -- Benjamin
Benjamin Rutt wrote: […]
What I had in mind was that you not use the 'code' module directly at all: the manhole module already implements the intergration you are looking for (and more). e.g. what happens if you replace “stdio.StandardIO(Echo())” in your original example with this: from twisted.conch import manhole stdio.StandardIO(manhole.Manhole()) I certainly didn't mean to suggest assigning a StandardIO instance to sys.stderr; it doesn't implement Python's file-like object protocol, so it will fail to do anything useful in that context. -Andrew.
participants (3)
-
Andrew Bennetts -
Benjamin Rutt -
exarkun＠twistedmatrix.com