[Twisted-Python] stdio.StandardIO, ServerProtocol and Services.
Hi all. I'm trying to write a "terminal" component, wrapped in a service.Service, to drop into an application framework so that I can use it to build interactive command-line tools. I've almost, but not quite, got it working. I know about manhole, but I want to write a simple command-based syntax, rather than exposing a python namespace and interpreter. I could also just go with a telnet or ssh interface, but that would require an additional step (the actual telnet or ssh). There's also stdiodemo.py, but that doesn't offer readline-like functionality, which would be a Very Good Thing. (Besides, I'm really curious as to why this approach is not working!) The code below is a minimal stripped-down example of what I've got so far through frantic grepping and glimpseing through the Twisted codebase. (although there are some fragmentary holdovers from various experiments; ignore these). At the moment, the command handler (the lineReceived() method) just echoes the argument with some '+' signs prepended. The termios stuff comes from twisted/conch/stdio.py, via a mailing list post saying something to the effect of, "this is the minimal amount you have to do in order to hook up your terminal to your process's stdio"; I don't have the ML link handy right now, sorry. Note that whereas stdio.py passes a Protocol argument to ServerProtocol, I've created an intermediate class, CLIServerProtocol, which has a protocolFactory attribute. An examination of twisted/conch/insults/insults.py shows that this should be equivalent. For the curious bystanders, HistoricRecvLine is derived as follows: HistoricRecvLine -> RecvLine -> TerminalProtocol ... and TerminalProtocol implements things like connectionMade, etc. Here's the behaviour I see: With the code as is, I can type in lines and get a reply from lineReceived. However, none of the cursor keys or other things for which there is special code in twisted/conch/recvline.py work; I just get control codes all over the terminal. If I uncomment the "reactor.run()", however, the behaviour changes completely, and becomes much closer to what I expect: keypresses like Home, cursor keys, etc, are honoured, as well as the history. However, Ctrl-C merely loses the connection, and the terminal hangs and I have to kill the twistd process externally. The fact that adding reactor.run() (which shouldn't be needed in a Service, right?) implies to me that the reactor isn't starting up properly without it, and I can't see why. Any advice would be greatly appreciated. Ricky -- import os, tty, sys, termios from twisted.application import service from twisted.internet import reactor, stdio, protocol, defer from twisted.python import failure, reflect, log from twisted.protocols import basic from twisted.application import internet from twisted.conch.insults import insults from twisted.conch.manhole import ColoredManhole # from twisted.conch.stdio import ConsoleManhole from twisted.application import service from twisted.conch import recvline class CLIProtocol ( recvline.HistoricRecvLine ): service = None def connectionMade ( self ): recvline.HistoricRecvLine.connectionMade ( self ) self.keyHandlers [ '\x01' ] = self.handle_HOME self.keyHandlers [ '\x03' ] = self.handle_QUIT self.keyHandlers [ '\x1a' ] = self.handle_QUIT def connectionLost ( self, reason ): log.msg ( "Connection Lost" ) def handle_QUIT ( self ): self.terminal.loseConnection() def lineReceived ( self, line ): self.terminal.write ( '+++' + line ) class CLIServerProtocol ( insults.ServerProtocol ): protocolFactory = CLIProtocol class CLIService ( service.Service ): def startService ( self ): fd = sys.__stdin__.fileno() oldSettings = termios.tcgetattr ( fd ) tty.setraw ( fd ) try: p = CLIServerProtocol() stdio.StandardIO ( p ) # reactor.run() finally: termios.tcsetattr ( fd, termios.TCSANOW, oldSettings ) os.write ( fd, "\r\x1bc\r" ) return service.Service.startService ( self ) ###################################################################### # Create the application service hierarchy. ###################################################################### application = service.Application ( 'cliapp' ) cs = CLIService() cs.setServiceParent ( application )
The fact that adding reactor.run() (which shouldn't be needed in a Service, right?) implies to me that the reactor isn't starting up properly without it, and I can't see why.
How are you starting this code? Are you just executing it? If so, don't do that. Use: twistd -noy thefile.py The Twisted application/service code just creates the relevant objects; the application object still needs to be run. "twistd" will do this for you.
I'm trying to Jelly decimal.Decimals, using Twisted 2.4. I've tried a few approaches, such as deriving my own Decimal class from decimal.Decimal, pb.RemoteCopy, and pb.Copyable, but no luck. Is there an easy way to do this, without patching twisted itself? Worse case I could always fall back on passing strings across the wire I guess, but that's not very enticing. :-/ -Jasper
On Sat, 06 Oct 2007 14:45:42 -0700, Jasper <jasper@peak.org> wrote:
I'm trying to Jelly decimal.Decimals, using Twisted 2.4. I've tried a few approaches, such as deriving my own Decimal class from decimal.Decimal, pb.RemoteCopy, and pb.Copyable, but no luck.
Is there an easy way to do this, without patching twisted itself? Worse case I could always fall back on passing strings across the wire I guess, but that's not very enticing. :-/
Jelly isn't currently extensible in this way. There isn't really any good reason that it *shouldn't* be. However, someone will need to do the work. Jean-Paul
Jean-Paul Calderone wrote:
On Sat, 06 Oct 2007 14:45:42 -0700, Jasper <jasper@peak.org> wrote:
I'm trying to Jelly decimal.Decimals, using Twisted 2.4. I've tried a few approaches, such as deriving my own Decimal class from decimal.Decimal, pb.RemoteCopy, and pb.Copyable, but no luck.
Is there an easy way to do this, without patching twisted itself? Worse case I could always fall back on passing strings across the wire I guess, but that's not very enticing. :-/
Jelly isn't currently extensible in this way. There isn't really any good reason that it *shouldn't* be. However, someone will need to do the work.
Jean-Paul
That was my impression from looking through the code as well; jelly requires old style python classes, and that's that. I've hacked in a special case work around for Decimals, but it's just a hack and not really suitable for general inclusion. I'm reluctant to put further work into it, as I recall that there is a new version PB in the works, which I suspect will completely redo how things are "jellied". -Jasper
On Tue, 09 Oct 2007 15:18:26 -0700, Jasper <jasper@peak.org> wrote:
Jean-Paul Calderone wrote:
That was my impression from looking through the code as well; jelly requires old style python classes, and that's that.
I've hacked in a special case work around for Decimals, but it's just a hack and not really suitable for general inclusion. I'm reluctant to put further work into it, as I recall that there is a new version PB in the works, which I suspect will completely redo how things are "jellied".
Just to clarify, there isn't a new version of PB in the works. There's Foolscap, which was called "newpb" for a while, but is developed outside of Twisted now. There's also AMP, which doesn't really resemble PB at all, but is developed as part of Twisted. PB itself is still maintained. Jean-Paul
On Tue, 09 Oct 2007 15:18:26 -0700, Jasper <jasper@peak.org> wrote:
Jean-Paul Calderone wrote:
That was my impression from looking through the code as well; jelly requires old style python classes, and that's that.
I've hacked in a special case work around for Decimals, but it's just a hack and not really suitable for general inclusion. I'm reluctant to put further work into it, as I recall that there is a new version PB in the works, which I suspect will completely redo how things are "jellied".
Just to clarify, there isn't a new version of PB in the works. There's Foolscap, which was called "newpb" for a while, but is developed outside of Twisted now. There's also AMP, which doesn't really resemble PB at all, but is developed as part of Twisted. PB itself is still maintained.
Jean-Paul I understand that PB is still maintained, but it's also a dead end without further development, and rather lacking in a few areas. From
Jean-Paul Calderone wrote: the bits I've seen Foolscap looks like it'll be enough of an improvement that I'll happily switch when it's done. I'd actually switch now, but I need something stable for the time being. I hadn't been aware of AMP; after a quick look at the docs it seems a bit low level for my purposes though, as I need to do more than just send messages. -Jasper
On 9 Oct, 11:55 pm, jasper@peak.org wrote:
I understand that PB is still maintained, but it's also a dead end without further development, and rather lacking in a few areas. From the bits I've seen Foolscap looks like it'll be enough of an improvement that I'll happily switch when it's done. I'd actually switch now, but I need something stable for the time being.
There is nobody currently working on a major overhaul of PB, but that doesn't meant that further development is prohibited. If you have specific features you would like added or bugs fixed in PB, you can feel free to contribute patches and we will integrate them (subject to the normal review process, of course). That said, if Foolscap better suits your purposes, by all means use it!
I hadn't been aware of AMP; after a quick look at the docs it seems a bit low level for my purposes though, as I need to do more than just send messages.
Since the FAQ of "isn't PB dead" has been raised yet again, I figure it's high time to give my own opinions of what's going on here. AMP is intentionally low level. AMP, PB, and Foolscap are, to me, at least, on a continuum where, on AMP's side, there is an extremely simple protocol implementing only the bare minimum required of an asynchronous messaging protocol. On the other end, Foolscap is extremely complex but featureful. This makes AMP ideal for applications where there is a narrow interface where two parties have to exchange a few simple and well-defined messages, with each party potentially being implemented many times by different languages and environments. There are already ad-hoc AMP implementations in a half a dozen languages already (I myself have prototyped elisp and Java implementations) and they are so easy to do that I'm sure more are on the way. You can do one yourself if you want. PB falls somewhere in the middle; using PB involves understanding fewer concepts than using Foolscap (in particular, it has no concept of URLs or object-spaces like tubs) but provides commensurately fewer features. It has conveniences like deep object serialization. It's well defined enough that you *can* write implementations for other languages if you really want to (there's one that mostly works for Java) but it's not the trivial affair that knocking off an AMP implementation is. PB was originally designed to support shared simulation spaces (online games) where lots of clients talk to a server about a potentially complex graph of objects, and that's where it remains ideal. Foolscap is geared towards very complex distributed applications where you need to serialize deep objects and coordinate arbitrary numbers of hosts which are connected in a mesh configuration. There's only the one implementation, but if you have an application that is well-suited to foolscap, the application itself is probably hard enough to implement that you won't be worrying about integrating alternative implementations for quite a while. Personally I've been more focused on AMP in the last year because I think there are currently a lot of applications in that first category which are currently being poorly served by bloated, complex, text-based protocols like XMLRPC, SOAP, and XMPP. However, this mostly represents a shift in my personal area of interest, and is not a comment on the validity of either of the other two application domains. That said, there may one day be some convergence or refactoring where a future version of PB becomes a layer on top of AMP, but given the obvious lack of any *application* drivers for such functionality, that seems like it will be very far off indeed.
On Saturday 06 October 2007 18:04:46 Phil Mayers wrote:
The fact that adding reactor.run() (which shouldn't be needed in a Service, right?) implies to me that the reactor isn't starting up properly without it, and I can't see why.
How are you starting this code? Are you just executing it? If so, don't do that. Use:
twistd -noy thefile.py
The Twisted application/service code just creates the relevant objects; the application object still needs to be run. "twistd" will do this for you.
Hi Phil; thanks for your reply. Sadly, I am already using twistd to start this code. Virtually all my Twisted code is service.Service-based, and I am accustomed to just boshing the various services into an application hierarchy and letting twistd Do The Right Thing, which is why I'm confused as to why this snippet seems not to work. Or rather, let me rephrase that: I'm not surprised that my code *overall* doesn't work quite how I would like, because I simply don't understand the inner workings of insults.ServerProtocol and recvline.RecvLine well enough, and I'm probably just being bone-headed. However, I *am* surprised that adding an "illegal" reactor.run() to a service's startService() method seems to "kick" the snippet into behaving differently. Cheers, Ricky
On Mon, 8 Oct 2007 11:15:33 +0300, kgi <iacovou@gmail.com> wrote:
On Saturday 06 October 2007 18:04:46 Phil Mayers wrote:
The fact that adding reactor.run() (which shouldn't be needed in a Service, right?) implies to me that the reactor isn't starting up properly without it, and I can't see why.
How are you starting this code? Are you just executing it? If so, don't do that. Use:
twistd -noy thefile.py
The Twisted application/service code just creates the relevant objects; the application object still needs to be run. "twistd" will do this for you.
Hi Phil; thanks for your reply.
Sadly, I am already using twistd to start this code. Virtually all my Twisted code is service.Service-based, and I am accustomed to just boshing the various services into an application hierarchy and letting twistd Do The Right Thing, which is why I'm confused as to why this snippet seems not to work.
You want to do things with the terminal. twistd also wants to do things with the terminal (particularly if you use -n). These two things come into conflict. The added call to reactor.run() seems to "fix" the issue because it prevents twistd from finishing the usual startup tasks, so your code gets to run and has the terminal to itself. It's probably not feasible at this point to provide a terminal interface to an application run with twistd on the controlling pty for that application. This is a feature which would need to be added to twistd. Jean-Paul
On Tuesday 09 October 2007 05:02:27 Jean-Paul Calderone wrote:
You want to do things with the terminal. twistd also wants to do things with the terminal (particularly if you use -n). These two things come into conflict. The added call to reactor.run() seems to "fix" the issue because it prevents twistd from finishing the usual startup tasks, so your code gets to run and has the terminal to itself.
It's probably not feasible at this point to provide a terminal interface to an application run with twistd on the controlling pty for that application. This is a feature which would need to be added to twistd.
Hi Jean-Paul, thanks for your reply (in fact, you seem to have gone on a "reply to all unanswered questions" rampage!). Thanks for clearing that up. It's good to know, even though it's not what I wanted to hear, because now I can stop bashing my head against that particular brick wall and take another approach. Regards, Ricky
participants (5)
-
glyph＠divmod.com -
Jasper -
Jean-Paul Calderone -
kgi -
Phil Mayers