[Twisted-Python] flash's rtmp protocol
![](https://secure.gravatar.com/avatar/627b464c215e4b7507bb74cefa725698.jpg?s=120&d=mm&r=g)
RTMP is a secret protocol used by Macromedia flash servers to do fancy interactive/streaming stuff. I found a reference to it on one of Bob Ippolito's TODO lists: http://pythonmac.org/wiki/BobIppolito "TODO: [....] Reverse engineer proprietary RTMP protocol (Flash Communication Server) and implement Python version (Twisted)" I'm also interested in doing this, but am unlikely to start soon. So this is just to let you (Bob) know, lest we do too much in parallel. douglas
![](https://secure.gravatar.com/avatar/b932b1e5a3e8299878e579f51f49b84a.jpg?s=120&d=mm&r=g)
On Dec 30, 2003, at 1:52 AM, Douglas Bagnall wrote:
I've given the TCP sessions a cursory glance, and my best guess is that it uses something quite similar to its stateless "Flash Remoting MX" feature. There are implementations of the remoting stuff in PHP and Java, but not Python (last I checked). I have not written any code or documentation yet either way. -bob
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
Hi, Has anyone written an IRC client in Twisted, using protocol.irc? Do I make a "reactor", or is a reactor only used for making servers? All I need is some really simple code illustrating how to connect to an IRC Server, doing an "identd" to satisfy some IRC Servers, and then having a simple imput line, and an output line... It doesn't have to be fancy, just some really basic example code on how to connect, specify a nick, joining a channel, and typing text into a channel, and of course how to quit. Unfortunately, no such example exists, all the parts seem to be there, but none of the methods in the protocol.irc.IRCClient appear to be implemented, which I suspect I would have to make my own sub-class... but I have no clue how to put all this together. There doesn't seem to be any docs that explain this. John
![](https://secure.gravatar.com/avatar/3c6f9c62d1015960866823348f070ed7.jpg?s=120&d=mm&r=g)
There's an IRC log bot with the examples.... Look here: http://twistedmatrix.com/documents/examples/ JD wrote:
![](https://secure.gravatar.com/avatar/fb44df761a72cca8686ea7faf897344b.jpg?s=120&d=mm&r=g)
On Tuesday 30 December 2003 08:04 pm, JD wrote:
ident is a seperate protocol from IRC. You will need to run a seperate daemon accessible from the Internet to provide this. Twisted doesn't have an ident server. I use the one here http://freshmeat.net/projects/fauxident/
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Dec 31, 2003, at 7:30 PM, darryl wrote:
Why can't anyone post just a teeny little bit of information on how to tie all this together? One of my main problems is where to put my code. Do I put it in the factory class, or do I put it in the Client class? One always points to the other. I suppose it don't really matter, but I'm sure the author(s) of twisted have their OWN ideas of where to put their specific code, but they sure don't seem to have been very sucessful in conveying this info to others. I know it's mostly due to my lack of knowledge, but the only part that's really well documented is the Twisted Web section. Anyway, if anyone has any suggestions or tips on what goes where, please post it here. Inquiring minds want to know, and I'm not the only one who wants to know. John
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Jan 1, 2004, at 2:22 AM, JD wrote:
It depends. The general pattern for Twisted code is to separate protocol logic from application logic. Your protocol subclass should translate from protocol-level events (IRCClient.signedOn, in this case) to application-level events (like a hypothetical 'jd.ChatterRobot.nowOnIRCAs(nick)' method). As for division between "protocol" and "factory" - try to make it so that your factory class can support multiple connected clients at once. It should be small. Any logic associated with a particular connection should go in the protocol class, and any logic associated with all connections simultaneously must at least be referenced from the factory class. Generally it's a bad idea to put that logic actually into the protocol class because that establishes a hard dependency between network code and application logic which makes your application logic harder to write test cases for. This pattern breaks down by necessity when dealing with very network-heavy code, or abstractions over things that have to logically understand IP addresses and the like. However, try to follow the general rule as much as possible even in this kind of code. The reason that nobody has suggested a particular answer to your problem is that the best approach involves keeping your code in an organization that best fits your problem domain, and is not dictated by Twisted. As long as the network events make it from the reactor to your python objects, there are few constraints as to how to organize those objects, and most of those have to do with starting up and shutting down cleanly in the context of bin/twistd.
I know our docs aren't the best, but I think that these cover the relevant organization issues: http://www.twistedmatrix.com/documents/howto/servers http://www.twistedmatrix.com/documents/howto/clients and they have nothing to do with Twisted Web. If you disagree, and there is really a particular documentation problem, open a doc bug in the tracker (http://www.twistedmatrix.com/bugs) and assign it to "hypatia".
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
Hi, I'm open for suggestions on where I can put my customized code. I seem to have a choice between putting it in the "factory" or in the "protocol", and I cannot seem to decide where the best place would be. My application is simple..... here is all it should do.... 1) establish a connection to an IRC server. 2) send an IRC command, like a "who" command, and extract the info 3) then exit the client I could put my code in the IRCClient's "connectionMade" method. Or I could put it in the factory's "startedConnecting" method. The code would be simple, issue a "who" command, and take it's output and return it. So, in my connectionMade method I would issue the command... then call reactor.stop() which would exit from my reactor.run(). I think the factory's "startedConnecting" might get called first, then somewhere down in this mess, the Client would call me back with the "connectionMade", and I could do it there. I just need to know the appropriate place to put my code. None of these examples in the 'examples' directories make this clear. My next task is to know how to extract the 'who' data back from my initial "who" The ONLY callback I see is the "action()" method, where it has the "msg" as the argument. Is that the one I sub-class to read back the result of the "who"? Which I assume is called back from a deferred. There are a number of other callbacks for the private messages and such. Which one do I use? When testing the code, I'm not getting any callbacks to the "action()" method. Is this because something else is fucked up. I don't suppose anyone on this last has an Mac OS-X system, do they? If so, I would love to establish a VIOP meeting if anyone has the time to help me out. My "iChat" AIM handle is 'jdcrunchman' John
![](https://secure.gravatar.com/avatar/fb44df761a72cca8686ea7faf897344b.jpg?s=120&d=mm&r=g)
On Wednesday 31 December 2003 04:58 pm, JD wrote:
Since this is IRC you shouldn't go issuing commands until your signed on, thus the signedOn method of your protocol would be the right place.
Yeah, it's not obvious how to do that. I'll send another mail when I've a chance to test something, but New Years is coming up right now... :)
No. An ACTION in IRC terms is a user doing an /me.
Yeah, your code.
-Eric
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Dec 31, 2003, at 7:43 PM, Eric Mangold wrote:
Ok, good point.... but how do I issue commands? let's say I have a IRCClient class called "testIRC". do I use: testIRC.sendLine('signedOn') to send the command to the IRC server to signon? or do I: testIRC.register('mynick', 'myhost', 'irc.debian.com') then, when I want to do the "who" command, would I do: testIRC.sendLine('who') then, i would expect to call me back, so I would override the "action" method, like so: def action(self, user, channel, msg): < extract the info I want from msg and store it somewhere, probably in an instance variable in testIRC object > reactor.stop() --- to exit the client and disconnect, or would I have to put the reactor.stop() in the factory's "clientConnectionFailed" method, which I would override of course. It's things like this that throw me totally for a loop. Where to put things.... what gets the call back...
Yeah, it's not obvious how to do that. I'll send another mail when I've a chance to test something, but New Years is coming up right now... :)
Yea - and while everyone is partying, i take this wonderful opportunity to get some serious coding done, because the place is so quiet, and I'm too broke to do anything anyway.
No. An ACTION in IRC terms is a user doing an /me.
Ok, if "action" isn't the callback which I override to get the output of the who command, then what is? myInfo? luserClient? luserChannels - Called with the number of channels existant on the server (ALMOST what I want) luserOp - Called with the number of ops logged on to the server (GETTING CLOSER - NOT QUITE) luserMe - Called with information about the server connected to. (CLOSEST YET - but no IP address info) Ok, so how do these get called? As far as I can tell, "handleCommand" calls method = getattr(self, "irc_%s" % command, None) which it then somehow then calls back to: def irc_RPL_LUSERME(self, prefix, params): self.luserMe(params[1]) I really want: RPL_WHOISSERVER, but (sigh) it's not there. Nobody bothered to finish this module. Could I add this to my IRCClient's subclass? IE: def irc_RPL_WHOISSERVER(self, prefix, params): self.whois(params[1]) Then.... def whois(self, prefix, info): < then in this code, get the info? - then after that do the reactor.stop() so I can fall out of the reactor.run() call. > Is this all I have to do.... but then would 'getattr' as you see above, know how to get the method name? Or would I have to add this somehow to another dictionary? If so, how do I do it? and if i have to do it, how do I find the reference to the dictionary? Anyway, other then being totally confused, is there any hope? I think I'm just missing a few critical steps. Or do I have them all?
No shit sherlock.... now i got to figure out why it's not calling my "whois" method. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Jan 01, 2004 at 11:25:37AM -0800, JD wrote:
No. IRCClient is a LineReceiver subclass, because that's how the low-level protocol works. However, you're not using the low-level protocol, you're using an abstraction on top of it -- IRCClient. Don't break the abstraction (by calling LineReceiver's sendLine method) unless you have a good reason to and you know the protocol spec well enough to know what you're doing (which would be RFC 2812 in this case, I think). As far as I'm aware, sending a line saying 'signedOn' to the server doesn't make any sense in the IRC protocol, so this definitely wouldn't achieve anything useful, except causing an error from the server that the IRCClient instance probably won't expect (because you broke the abstraction and snuck behind its back to do it).
or do I: testIRC.register('mynick', 'myhost', 'irc.debian.com')
That looks much saner. And noticeably more full of information than testIRC.sendLine('signedOn')...
then, when I want to do the "who" command, would I do: testIRC.sendLine('who')
Well, I believe the WHO command requires an argument, so it would be more like testIRC.sendLine('WHO ' + name)... but see my next answer.
That's not what the action method is for. As the docstring for action says: "Called when I see a user perform an ACTION on a channel." An ACTION is a specific thing in IRC, not a catch-all for every type of message from server to client. See the RFC if you're not familar with the terms. The right way to do this is implemented in sandbox/exarkun/irc2.py -- see the AdvancedClient.who method. I'd follow JP's earlier suggestion and use it.
It's things like this that throw me totally for a loop. Where to put things.... what gets the call back...
It depends on the protocol implementation. There's no single generic answer for the whole of Twisted.
The answer is of course none of those. You should've already noticed that IRCClient doesn't implement a who() method, so it's hardly surprising it hasn't implemented a command to deal with its response. irc2.py does, though (and uses Deferreds for dealing with the response, which is cleaner anyway).
I thought you wanted to use the WHO command not WHOIS? Please try to be precise. Anyway, irc2.py also implements WHOIS.
If you look at irc2.py, you'll see it does override this method. Note that 'whois' is a bad name for the response handler -- the existing convention in IRCClient is that 'join', 'nick', etc, are the methods that *issue* the command, not that handle the response.
Yes, it would. Please see the docs for getattr in the Python standard library reference.
No idea. You tell me :) I think the biggest critical step you're missing is that you're busily trying to reinvent a wheel that's already been made in sandbox/exarkun/irc2.py, though. -Andrew.
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 1, 2004, at 6:52 PM, Andrew Bennetts wrote:
I think I understand. what is needed in the docs, is a better description of just what the roles of these objects are supposed to do, and where we, as users of Twisted, would have a much better chance to use Twisted in the way it is intended. So the usage of 'sendLine' in a 'lineReceiver' would be inappropriate? but if that's true, then how does one send a command to the IRC server? By the way, I've chosen this as a simple test application, not unlike what I want to really do, so I have chosen a relatively simple application, where I just extract all the users of a specified IRC server, and return the data. I have not specified the form of the data, that's not important, but what's important to me, is the understand how to use Twisted in the way it was intended. I have this: class IRC_test(irc.IRCClient): def whois_on(self): """ This method issues a 'WHO' command to see who is on the server. """ self.sendLine("WHO") def signedOn(self): """Called after sucessfully signing on to the server. So we immediately issue the 'whois_on' message to let the server go and do it's stuff. """ self.whois_on(self) def whois_result(self, prefix, info): """ This is the code the IRC is somehow going to call. Then, after it is called, it stores the info in the factory and returns by calling 'reactor.stop()' to exit the reactor.run() """ self.result = extract_whois_data(info) reactor.stop() def irc_RPL_WHOISSERVER(self, prefix, params): """ This was not included in the parent class IRCClient, So I add it here. 'lineReceived' gets the command = numeric_to_symbolic[command], which is now the string 'RPL_WHOISSERVER'. To get THIS method to call, the 'handleCommand' method first generates a key to the class's 'attribute' dictionary, by pre-pending 'irc_' in front of it to get this method, which is then called. 'params' would then have the result data. We need to hack around to see what form it's in. """ self.whois_result(params[1]) def test_irc(host, port): """ This function scans a particular IRC server, and returns a dictionary of all the nicks and IP addresses. Nick is the key, the IP or reverse dns is the value. """ my_nick = "jd6969" data_dict = {} # place where I want my IRC data myTestFactory = IRC_test(data_dict, my_nick) reactor.connectTCP(host, port, myTestFactory) reactor.run() # I just wait for the server to respond return data_dict # when it does, I just want to return with the data I know this is a pretty stupid and useless application, but this is very close to what I want it to do, without revealing anything. I haven't tested this yet, because I just want to be sure I'm doing it right, rather then to spend weeks down a particular path to nowhere.
I believe I do, because I still have to tell the IRC server what I want it to do. And the only way I can see, is with the sendLine command.
Ok, so it is clear I'm doing this wrong. If that's the case, then what is the RIGHT way of doing it?
hmmm - in checking the RFC it mentions I can either "WHO <channel>" or "WHO" without specifying a channel, in which it gets the names of ALL the users of the IRC server not hidden.
Yes - I was mislead... I interpreted this as an action that might come from the server, such as the result of a WHO command (in using my example) I think if you take a look at the shell of my application, I'm no longer using this.
I was NOT aware that anything in "sandbox" had anything to do with IRC. Expecially when words like this are chosen. One can interpret 'sandbox' in many zillion ways, so how would I even know this. I took a look at this, and WOW - this might be just what I need. I'm going to have to spend a lot of time studying this. I had no clue this even existed, primarily because the name 'sandbox' really had nothing to do with IRC, and I just couldn't make the connection.
yes - as it also doesn't implement a LOT of things, and yet they are defined as overrideable functions.
Kewl - and now just being aware of that, I definately want to check it out. And also I could use some connents on my above approach, as I have avoided the pitfalls you outlined above.
I thought they were almost the same.... at least that's what I read in the RFC's. What is the difference?
Anyway, irc2.py also implements WHOIS.
Kewl
I haven't seen any mention of naming conventions, but if there is one, I certainly would want to know where it is.
that is obviously true.... I explained above why I just didn't even know of it's existance until it was pointed out to me. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Jan 02, 2004 at 06:13:39PM -0800, JD wrote:
No, using sendLine in a LineReceiver is fine -- that's why LineReceiver defines sendLine. What isn't so good is using sendLine in an IRCClient subclass. While IRCClient subclasses LineReceiver, in general you should be using the abstractions IRCClient provides to send commands to and receive responses from the server (if you don't want to use them, then why not just use a plain LineReceiver?). The only reason you should have for calling sendLine in an IRCClient subclass is to extend its functionality to support more of the protocol than it already does. IRCClient already has support for signing on (the 'register' method and the 'signedOn' handler), so you shouldn't use sendLine for that. IRCClient currently lacks some IRC features, such as WHO, so to do those you do need sendLine. Also, good style suggests that you probably shouldn't mix your application code with your IRCClient extensions, i.e. subclass IRCClient for WHO support (and other other unimplemented commands you need), and then subclass that for your application.
(This identation is really painful.)
(This identation won't even work!) But creating a method called 'irc_RPL_WHOISSERVER' will do what you want here. Handler methods with a common prefix that are dispatched with a getattr call are a common, and very nice, idiom in Twisted.
(Ugh, and now you're using tabs...) IRC_test is a protocol, but you're passing it as a factory to connectTCP. See http://twistedmatrix.com/documents/howto/clients.
This code is short. Why didn't you run it yourself to get some idea of how close it is to working? Working code is usually a good sign that you're close to the right path, if not already on it...
Only because you need to extend it to cover parts of the protocol that aren't implemented. You started off talking about 'signedOn', which is already covered by IRCClient, and only later mentioned trying to do a WHO command, which isn't. Confusing issues like this makes it hard to figure out what you're trying to do.
Calling register, of course. Where in the RFC does it say 'signedOn' is a recognised command?
Fair enough. Believe it or not, but I actually know very very little about IRC, I mainly figured it out for that email to you by skimming irc.py and one or two brief google searches to confirm e.g. that 'signedOn' doesn't appear in the RFC.
Because of a mail from earlier in the thread from JP: http://twistedmatrix.com/pipermail/twisted-python/2004-January/006893.html Which part of "WHO is one of the commands I did manage to improve, but I never moved any of the work out of my sandbox. You can find it in Twisted/sandbox/exarkun/irc2.py." was unclear?
Hence the email.
I don't understand what you're saying here. You mean things like irc_RPL_WHOISSERVER? That's just the way the prefixed getattr idiom works; all you need to do support a new command define a handler method for it, and that's it. No explicit registration or anything like that needed. Python's a dynamic language, and Twisted uses that to its advantage.
No idea. Again, I don't know IRC very well :) I notice that irc2.py implements both WHO and WHOIS seperately, though, so I figure there are some differences, which is why you saying WHO and then later WHOIS confused me.
It's not explicitly written anywhere that I know of. It was just obvious to me from skimming the code of irc.py and irc2.py that that's what the convention was, because that's what the existing code *does*. I generally try to follow the conventions of code I'm using to avoid getting horribly confused.
Yes, but it was pointed out to you before this... -Andrew.
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 3, 2004, at 8:32 AM, Andrew Bennetts wrote:
Yes, exactly.
I think I did that, but not sure I did it right. I sent a code fragment up to the list earlier, did you see how I did it? Was I on the right track? John
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 2, 2004, at 7:20 PM, Andrew Bennetts wrote:
That's what I thought. Did you take a look at that code sample I sent earlier? I hacked that up myself, with few examples, so I can see where i might have put things in places they weren't intended to be. I want to really learn Twisted, and to be able to put pieces together without having to be constantly digging for example code, and twiddling with that.
I don't quite understand. Because in my case, my application just intantiates a client long enough to extract some information from the IRC server, issue a "who" command and extract the data. I always thought it was appropriate to sub-class the part i want to do, or parts that are not implemented by the parent, as in my case of wanting to do the "who" command, so I treated the command like any other IRC command that issues commands and gets back data. So If I can't mix my application code with a sub_class of a IRCClient, then how is the right way to do it. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Mon, Jan 05, 2004 at 06:59:26PM -0800, JD wrote:
I did. The basic idea seemed fine, except for the trivial flaws I pointed out. I don't have time to test and debug your code for you, though, especially when it's obvious you haven't tried to do so either. If your code isn't working, then it would help us if you could specify *how* it isn't working, i.e. what error or misbehaviour you are seeing. Your previous mail didn't indicate that you'd tested the example code you sent (and the trivial syntax errors suggest this is also the case).
What I was trying to say is rather than subclass like this: IRCClient (from twisted.protocols.irc) | \--YourIRCApp (adds WHO command, has your application logic) It might be a better idea to do it like this: IRCClient (from twisted.protocols.irc) | \--ExtendedIRCClient (adds WHO command) | \--YourIRCApp (has your application logic) But that's really just a cosmetic issue. In an ideal world, Twisted would already have the WHO command implemented in its standard IRCClient. Until we reach that point, though, I still like the idea of keeping the protocol implementation and the application using that protocol seperate, even if the "protocol implementation" part of that picture is currently split between what Twisted already provides (IRCClient) and what you need to add (what I called ExtendedIRCClient above). Or you could not implement WHO yourself at all, and use the more complete IRC implementation in irc2.py... -Andrew.
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 5, 2004, at 7:28 PM, Andrew Bennetts wrote:
Don't worry about the code.... it was put up there to only show to general construct, and how I plan to put things together. it was a simple lame example totally different then what my intended APP was going to be, I just used that just to illustrate to others, my understanding, or lack thereof..
I've already decided that.... because when I first started to use Twisted, I had no clue of even the existance of ExtendedIRCClient because it was so cleverly mis-labeled and hidden so well in the examples, I never would have found it, had I not posted this in the first place. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Jan 08, 2004 at 01:04:18PM -0800, JD wrote:
You misunderstand me. I wasn't referring to the AdvancedClient class in sandbox/exarkun/irc2.py, as you seem to be thinking. I was offering a suggestion on how to structure *your* code that extends twisted.protocols.irc.IRCClient, if you choose to extend it. (The fact that AdvancedClient is an extension to IRCClient that subclasses it as I describe above suggests that my advice is sound, but the existence of AdvancedClient is incidental to the point I was trying to make.) -Andrew.
![](https://secure.gravatar.com/avatar/56e4cc78ea7fcf3bb37888ebf23bc1f0.jpg?s=120&d=mm&r=g)
On Wed, Dec 31, 2003 at 03:58:50PM -0800, JD wrote:
startedConnecting isn't the right place. connectionMade is a little closer, but since you need to log in (tell the IRC server your nickname, etc) before you can issue commands like WHO, it isn't totally appropriate either. signedOn is the best place, since it is called by IRCClient once the server acknowledges the login information.
Indeed. This is not very easy to do with IRCClient, since the response the server sends doesn't have a whole lot to associate it with the WHO command you originally issued. You need to keep track of outstanding commands and try to match them up with server replies. I recognized this deficiency some time ago and did some initial work towards improving the API. WHO is one of the commands I did manage to improve, but I never moved any of the work out of my sandbox. You can find it in Twisted/sandbox/exarkun/irc2.py. It is under the same license as the rest of Twisted, even though it currently lacks any indication fo that. If it would be helpful to you, please feel free to use it. The improvements take one general form: commands which will produce server responses are exposed as functions which return a Deferred. When the server replies, the parsed information will be passed to that Deferred's callback. Here is a simple example of how you might use it (untested): class WhoingIRCClient(irc2.AdvancedClient): def signedOn(self): # Find everyone IRC'ing from Twisted Matrix's server. self.who('*twistedmatrix.com' ).addCallbacks(self._cbWho, self._ebWho ).addBoth(self._shutdownEverything ) def _cbWho(self, results): print 'channel user host server nick flags hops realname' for r in results: print ', '.join(r) def _ebWho(self, failure) print 'Error in who!' log.err(failure) def _shutdownEverything(self, ignored): reactor.stop() Jp
![](https://secure.gravatar.com/avatar/b932b1e5a3e8299878e579f51f49b84a.jpg?s=120&d=mm&r=g)
On Dec 30, 2003, at 1:52 AM, Douglas Bagnall wrote:
I've given the TCP sessions a cursory glance, and my best guess is that it uses something quite similar to its stateless "Flash Remoting MX" feature. There are implementations of the remoting stuff in PHP and Java, but not Python (last I checked). I have not written any code or documentation yet either way. -bob
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
Hi, Has anyone written an IRC client in Twisted, using protocol.irc? Do I make a "reactor", or is a reactor only used for making servers? All I need is some really simple code illustrating how to connect to an IRC Server, doing an "identd" to satisfy some IRC Servers, and then having a simple imput line, and an output line... It doesn't have to be fancy, just some really basic example code on how to connect, specify a nick, joining a channel, and typing text into a channel, and of course how to quit. Unfortunately, no such example exists, all the parts seem to be there, but none of the methods in the protocol.irc.IRCClient appear to be implemented, which I suspect I would have to make my own sub-class... but I have no clue how to put all this together. There doesn't seem to be any docs that explain this. John
![](https://secure.gravatar.com/avatar/3c6f9c62d1015960866823348f070ed7.jpg?s=120&d=mm&r=g)
There's an IRC log bot with the examples.... Look here: http://twistedmatrix.com/documents/examples/ JD wrote:
![](https://secure.gravatar.com/avatar/fb44df761a72cca8686ea7faf897344b.jpg?s=120&d=mm&r=g)
On Tuesday 30 December 2003 08:04 pm, JD wrote:
ident is a seperate protocol from IRC. You will need to run a seperate daemon accessible from the Internet to provide this. Twisted doesn't have an ident server. I use the one here http://freshmeat.net/projects/fauxident/
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Dec 31, 2003, at 7:30 PM, darryl wrote:
Why can't anyone post just a teeny little bit of information on how to tie all this together? One of my main problems is where to put my code. Do I put it in the factory class, or do I put it in the Client class? One always points to the other. I suppose it don't really matter, but I'm sure the author(s) of twisted have their OWN ideas of where to put their specific code, but they sure don't seem to have been very sucessful in conveying this info to others. I know it's mostly due to my lack of knowledge, but the only part that's really well documented is the Twisted Web section. Anyway, if anyone has any suggestions or tips on what goes where, please post it here. Inquiring minds want to know, and I'm not the only one who wants to know. John
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Jan 1, 2004, at 2:22 AM, JD wrote:
It depends. The general pattern for Twisted code is to separate protocol logic from application logic. Your protocol subclass should translate from protocol-level events (IRCClient.signedOn, in this case) to application-level events (like a hypothetical 'jd.ChatterRobot.nowOnIRCAs(nick)' method). As for division between "protocol" and "factory" - try to make it so that your factory class can support multiple connected clients at once. It should be small. Any logic associated with a particular connection should go in the protocol class, and any logic associated with all connections simultaneously must at least be referenced from the factory class. Generally it's a bad idea to put that logic actually into the protocol class because that establishes a hard dependency between network code and application logic which makes your application logic harder to write test cases for. This pattern breaks down by necessity when dealing with very network-heavy code, or abstractions over things that have to logically understand IP addresses and the like. However, try to follow the general rule as much as possible even in this kind of code. The reason that nobody has suggested a particular answer to your problem is that the best approach involves keeping your code in an organization that best fits your problem domain, and is not dictated by Twisted. As long as the network events make it from the reactor to your python objects, there are few constraints as to how to organize those objects, and most of those have to do with starting up and shutting down cleanly in the context of bin/twistd.
I know our docs aren't the best, but I think that these cover the relevant organization issues: http://www.twistedmatrix.com/documents/howto/servers http://www.twistedmatrix.com/documents/howto/clients and they have nothing to do with Twisted Web. If you disagree, and there is really a particular documentation problem, open a doc bug in the tracker (http://www.twistedmatrix.com/bugs) and assign it to "hypatia".
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
Hi, I'm open for suggestions on where I can put my customized code. I seem to have a choice between putting it in the "factory" or in the "protocol", and I cannot seem to decide where the best place would be. My application is simple..... here is all it should do.... 1) establish a connection to an IRC server. 2) send an IRC command, like a "who" command, and extract the info 3) then exit the client I could put my code in the IRCClient's "connectionMade" method. Or I could put it in the factory's "startedConnecting" method. The code would be simple, issue a "who" command, and take it's output and return it. So, in my connectionMade method I would issue the command... then call reactor.stop() which would exit from my reactor.run(). I think the factory's "startedConnecting" might get called first, then somewhere down in this mess, the Client would call me back with the "connectionMade", and I could do it there. I just need to know the appropriate place to put my code. None of these examples in the 'examples' directories make this clear. My next task is to know how to extract the 'who' data back from my initial "who" The ONLY callback I see is the "action()" method, where it has the "msg" as the argument. Is that the one I sub-class to read back the result of the "who"? Which I assume is called back from a deferred. There are a number of other callbacks for the private messages and such. Which one do I use? When testing the code, I'm not getting any callbacks to the "action()" method. Is this because something else is fucked up. I don't suppose anyone on this last has an Mac OS-X system, do they? If so, I would love to establish a VIOP meeting if anyone has the time to help me out. My "iChat" AIM handle is 'jdcrunchman' John
![](https://secure.gravatar.com/avatar/fb44df761a72cca8686ea7faf897344b.jpg?s=120&d=mm&r=g)
On Wednesday 31 December 2003 04:58 pm, JD wrote:
Since this is IRC you shouldn't go issuing commands until your signed on, thus the signedOn method of your protocol would be the right place.
Yeah, it's not obvious how to do that. I'll send another mail when I've a chance to test something, but New Years is coming up right now... :)
No. An ACTION in IRC terms is a user doing an /me.
Yeah, your code.
-Eric
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Dec 31, 2003, at 7:43 PM, Eric Mangold wrote:
Ok, good point.... but how do I issue commands? let's say I have a IRCClient class called "testIRC". do I use: testIRC.sendLine('signedOn') to send the command to the IRC server to signon? or do I: testIRC.register('mynick', 'myhost', 'irc.debian.com') then, when I want to do the "who" command, would I do: testIRC.sendLine('who') then, i would expect to call me back, so I would override the "action" method, like so: def action(self, user, channel, msg): < extract the info I want from msg and store it somewhere, probably in an instance variable in testIRC object > reactor.stop() --- to exit the client and disconnect, or would I have to put the reactor.stop() in the factory's "clientConnectionFailed" method, which I would override of course. It's things like this that throw me totally for a loop. Where to put things.... what gets the call back...
Yeah, it's not obvious how to do that. I'll send another mail when I've a chance to test something, but New Years is coming up right now... :)
Yea - and while everyone is partying, i take this wonderful opportunity to get some serious coding done, because the place is so quiet, and I'm too broke to do anything anyway.
No. An ACTION in IRC terms is a user doing an /me.
Ok, if "action" isn't the callback which I override to get the output of the who command, then what is? myInfo? luserClient? luserChannels - Called with the number of channels existant on the server (ALMOST what I want) luserOp - Called with the number of ops logged on to the server (GETTING CLOSER - NOT QUITE) luserMe - Called with information about the server connected to. (CLOSEST YET - but no IP address info) Ok, so how do these get called? As far as I can tell, "handleCommand" calls method = getattr(self, "irc_%s" % command, None) which it then somehow then calls back to: def irc_RPL_LUSERME(self, prefix, params): self.luserMe(params[1]) I really want: RPL_WHOISSERVER, but (sigh) it's not there. Nobody bothered to finish this module. Could I add this to my IRCClient's subclass? IE: def irc_RPL_WHOISSERVER(self, prefix, params): self.whois(params[1]) Then.... def whois(self, prefix, info): < then in this code, get the info? - then after that do the reactor.stop() so I can fall out of the reactor.run() call. > Is this all I have to do.... but then would 'getattr' as you see above, know how to get the method name? Or would I have to add this somehow to another dictionary? If so, how do I do it? and if i have to do it, how do I find the reference to the dictionary? Anyway, other then being totally confused, is there any hope? I think I'm just missing a few critical steps. Or do I have them all?
No shit sherlock.... now i got to figure out why it's not calling my "whois" method. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Jan 01, 2004 at 11:25:37AM -0800, JD wrote:
No. IRCClient is a LineReceiver subclass, because that's how the low-level protocol works. However, you're not using the low-level protocol, you're using an abstraction on top of it -- IRCClient. Don't break the abstraction (by calling LineReceiver's sendLine method) unless you have a good reason to and you know the protocol spec well enough to know what you're doing (which would be RFC 2812 in this case, I think). As far as I'm aware, sending a line saying 'signedOn' to the server doesn't make any sense in the IRC protocol, so this definitely wouldn't achieve anything useful, except causing an error from the server that the IRCClient instance probably won't expect (because you broke the abstraction and snuck behind its back to do it).
or do I: testIRC.register('mynick', 'myhost', 'irc.debian.com')
That looks much saner. And noticeably more full of information than testIRC.sendLine('signedOn')...
then, when I want to do the "who" command, would I do: testIRC.sendLine('who')
Well, I believe the WHO command requires an argument, so it would be more like testIRC.sendLine('WHO ' + name)... but see my next answer.
That's not what the action method is for. As the docstring for action says: "Called when I see a user perform an ACTION on a channel." An ACTION is a specific thing in IRC, not a catch-all for every type of message from server to client. See the RFC if you're not familar with the terms. The right way to do this is implemented in sandbox/exarkun/irc2.py -- see the AdvancedClient.who method. I'd follow JP's earlier suggestion and use it.
It's things like this that throw me totally for a loop. Where to put things.... what gets the call back...
It depends on the protocol implementation. There's no single generic answer for the whole of Twisted.
The answer is of course none of those. You should've already noticed that IRCClient doesn't implement a who() method, so it's hardly surprising it hasn't implemented a command to deal with its response. irc2.py does, though (and uses Deferreds for dealing with the response, which is cleaner anyway).
I thought you wanted to use the WHO command not WHOIS? Please try to be precise. Anyway, irc2.py also implements WHOIS.
If you look at irc2.py, you'll see it does override this method. Note that 'whois' is a bad name for the response handler -- the existing convention in IRCClient is that 'join', 'nick', etc, are the methods that *issue* the command, not that handle the response.
Yes, it would. Please see the docs for getattr in the Python standard library reference.
No idea. You tell me :) I think the biggest critical step you're missing is that you're busily trying to reinvent a wheel that's already been made in sandbox/exarkun/irc2.py, though. -Andrew.
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 1, 2004, at 6:52 PM, Andrew Bennetts wrote:
I think I understand. what is needed in the docs, is a better description of just what the roles of these objects are supposed to do, and where we, as users of Twisted, would have a much better chance to use Twisted in the way it is intended. So the usage of 'sendLine' in a 'lineReceiver' would be inappropriate? but if that's true, then how does one send a command to the IRC server? By the way, I've chosen this as a simple test application, not unlike what I want to really do, so I have chosen a relatively simple application, where I just extract all the users of a specified IRC server, and return the data. I have not specified the form of the data, that's not important, but what's important to me, is the understand how to use Twisted in the way it was intended. I have this: class IRC_test(irc.IRCClient): def whois_on(self): """ This method issues a 'WHO' command to see who is on the server. """ self.sendLine("WHO") def signedOn(self): """Called after sucessfully signing on to the server. So we immediately issue the 'whois_on' message to let the server go and do it's stuff. """ self.whois_on(self) def whois_result(self, prefix, info): """ This is the code the IRC is somehow going to call. Then, after it is called, it stores the info in the factory and returns by calling 'reactor.stop()' to exit the reactor.run() """ self.result = extract_whois_data(info) reactor.stop() def irc_RPL_WHOISSERVER(self, prefix, params): """ This was not included in the parent class IRCClient, So I add it here. 'lineReceived' gets the command = numeric_to_symbolic[command], which is now the string 'RPL_WHOISSERVER'. To get THIS method to call, the 'handleCommand' method first generates a key to the class's 'attribute' dictionary, by pre-pending 'irc_' in front of it to get this method, which is then called. 'params' would then have the result data. We need to hack around to see what form it's in. """ self.whois_result(params[1]) def test_irc(host, port): """ This function scans a particular IRC server, and returns a dictionary of all the nicks and IP addresses. Nick is the key, the IP or reverse dns is the value. """ my_nick = "jd6969" data_dict = {} # place where I want my IRC data myTestFactory = IRC_test(data_dict, my_nick) reactor.connectTCP(host, port, myTestFactory) reactor.run() # I just wait for the server to respond return data_dict # when it does, I just want to return with the data I know this is a pretty stupid and useless application, but this is very close to what I want it to do, without revealing anything. I haven't tested this yet, because I just want to be sure I'm doing it right, rather then to spend weeks down a particular path to nowhere.
I believe I do, because I still have to tell the IRC server what I want it to do. And the only way I can see, is with the sendLine command.
Ok, so it is clear I'm doing this wrong. If that's the case, then what is the RIGHT way of doing it?
hmmm - in checking the RFC it mentions I can either "WHO <channel>" or "WHO" without specifying a channel, in which it gets the names of ALL the users of the IRC server not hidden.
Yes - I was mislead... I interpreted this as an action that might come from the server, such as the result of a WHO command (in using my example) I think if you take a look at the shell of my application, I'm no longer using this.
I was NOT aware that anything in "sandbox" had anything to do with IRC. Expecially when words like this are chosen. One can interpret 'sandbox' in many zillion ways, so how would I even know this. I took a look at this, and WOW - this might be just what I need. I'm going to have to spend a lot of time studying this. I had no clue this even existed, primarily because the name 'sandbox' really had nothing to do with IRC, and I just couldn't make the connection.
yes - as it also doesn't implement a LOT of things, and yet they are defined as overrideable functions.
Kewl - and now just being aware of that, I definately want to check it out. And also I could use some connents on my above approach, as I have avoided the pitfalls you outlined above.
I thought they were almost the same.... at least that's what I read in the RFC's. What is the difference?
Anyway, irc2.py also implements WHOIS.
Kewl
I haven't seen any mention of naming conventions, but if there is one, I certainly would want to know where it is.
that is obviously true.... I explained above why I just didn't even know of it's existance until it was pointed out to me. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Jan 02, 2004 at 06:13:39PM -0800, JD wrote:
No, using sendLine in a LineReceiver is fine -- that's why LineReceiver defines sendLine. What isn't so good is using sendLine in an IRCClient subclass. While IRCClient subclasses LineReceiver, in general you should be using the abstractions IRCClient provides to send commands to and receive responses from the server (if you don't want to use them, then why not just use a plain LineReceiver?). The only reason you should have for calling sendLine in an IRCClient subclass is to extend its functionality to support more of the protocol than it already does. IRCClient already has support for signing on (the 'register' method and the 'signedOn' handler), so you shouldn't use sendLine for that. IRCClient currently lacks some IRC features, such as WHO, so to do those you do need sendLine. Also, good style suggests that you probably shouldn't mix your application code with your IRCClient extensions, i.e. subclass IRCClient for WHO support (and other other unimplemented commands you need), and then subclass that for your application.
(This identation is really painful.)
(This identation won't even work!) But creating a method called 'irc_RPL_WHOISSERVER' will do what you want here. Handler methods with a common prefix that are dispatched with a getattr call are a common, and very nice, idiom in Twisted.
(Ugh, and now you're using tabs...) IRC_test is a protocol, but you're passing it as a factory to connectTCP. See http://twistedmatrix.com/documents/howto/clients.
This code is short. Why didn't you run it yourself to get some idea of how close it is to working? Working code is usually a good sign that you're close to the right path, if not already on it...
Only because you need to extend it to cover parts of the protocol that aren't implemented. You started off talking about 'signedOn', which is already covered by IRCClient, and only later mentioned trying to do a WHO command, which isn't. Confusing issues like this makes it hard to figure out what you're trying to do.
Calling register, of course. Where in the RFC does it say 'signedOn' is a recognised command?
Fair enough. Believe it or not, but I actually know very very little about IRC, I mainly figured it out for that email to you by skimming irc.py and one or two brief google searches to confirm e.g. that 'signedOn' doesn't appear in the RFC.
Because of a mail from earlier in the thread from JP: http://twistedmatrix.com/pipermail/twisted-python/2004-January/006893.html Which part of "WHO is one of the commands I did manage to improve, but I never moved any of the work out of my sandbox. You can find it in Twisted/sandbox/exarkun/irc2.py." was unclear?
Hence the email.
I don't understand what you're saying here. You mean things like irc_RPL_WHOISSERVER? That's just the way the prefixed getattr idiom works; all you need to do support a new command define a handler method for it, and that's it. No explicit registration or anything like that needed. Python's a dynamic language, and Twisted uses that to its advantage.
No idea. Again, I don't know IRC very well :) I notice that irc2.py implements both WHO and WHOIS seperately, though, so I figure there are some differences, which is why you saying WHO and then later WHOIS confused me.
It's not explicitly written anywhere that I know of. It was just obvious to me from skimming the code of irc.py and irc2.py that that's what the convention was, because that's what the existing code *does*. I generally try to follow the conventions of code I'm using to avoid getting horribly confused.
Yes, but it was pointed out to you before this... -Andrew.
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 3, 2004, at 8:32 AM, Andrew Bennetts wrote:
Yes, exactly.
I think I did that, but not sure I did it right. I sent a code fragment up to the list earlier, did you see how I did it? Was I on the right track? John
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 2, 2004, at 7:20 PM, Andrew Bennetts wrote:
That's what I thought. Did you take a look at that code sample I sent earlier? I hacked that up myself, with few examples, so I can see where i might have put things in places they weren't intended to be. I want to really learn Twisted, and to be able to put pieces together without having to be constantly digging for example code, and twiddling with that.
I don't quite understand. Because in my case, my application just intantiates a client long enough to extract some information from the IRC server, issue a "who" command and extract the data. I always thought it was appropriate to sub-class the part i want to do, or parts that are not implemented by the parent, as in my case of wanting to do the "who" command, so I treated the command like any other IRC command that issues commands and gets back data. So If I can't mix my application code with a sub_class of a IRCClient, then how is the right way to do it. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Mon, Jan 05, 2004 at 06:59:26PM -0800, JD wrote:
I did. The basic idea seemed fine, except for the trivial flaws I pointed out. I don't have time to test and debug your code for you, though, especially when it's obvious you haven't tried to do so either. If your code isn't working, then it would help us if you could specify *how* it isn't working, i.e. what error or misbehaviour you are seeing. Your previous mail didn't indicate that you'd tested the example code you sent (and the trivial syntax errors suggest this is also the case).
What I was trying to say is rather than subclass like this: IRCClient (from twisted.protocols.irc) | \--YourIRCApp (adds WHO command, has your application logic) It might be a better idea to do it like this: IRCClient (from twisted.protocols.irc) | \--ExtendedIRCClient (adds WHO command) | \--YourIRCApp (has your application logic) But that's really just a cosmetic issue. In an ideal world, Twisted would already have the WHO command implemented in its standard IRCClient. Until we reach that point, though, I still like the idea of keeping the protocol implementation and the application using that protocol seperate, even if the "protocol implementation" part of that picture is currently split between what Twisted already provides (IRCClient) and what you need to add (what I called ExtendedIRCClient above). Or you could not implement WHO yourself at all, and use the more complete IRC implementation in irc2.py... -Andrew.
![](https://secure.gravatar.com/avatar/cdedd0f07d3d5976128eea43ea89c50b.jpg?s=120&d=mm&r=g)
On Jan 5, 2004, at 7:28 PM, Andrew Bennetts wrote:
Don't worry about the code.... it was put up there to only show to general construct, and how I plan to put things together. it was a simple lame example totally different then what my intended APP was going to be, I just used that just to illustrate to others, my understanding, or lack thereof..
I've already decided that.... because when I first started to use Twisted, I had no clue of even the existance of ExtendedIRCClient because it was so cleverly mis-labeled and hidden so well in the examples, I never would have found it, had I not posted this in the first place. John
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Jan 08, 2004 at 01:04:18PM -0800, JD wrote:
You misunderstand me. I wasn't referring to the AdvancedClient class in sandbox/exarkun/irc2.py, as you seem to be thinking. I was offering a suggestion on how to structure *your* code that extends twisted.protocols.irc.IRCClient, if you choose to extend it. (The fact that AdvancedClient is an extension to IRCClient that subclasses it as I describe above suggests that my advice is sound, but the existence of AdvancedClient is incidental to the point I was trying to make.) -Andrew.
![](https://secure.gravatar.com/avatar/56e4cc78ea7fcf3bb37888ebf23bc1f0.jpg?s=120&d=mm&r=g)
On Wed, Dec 31, 2003 at 03:58:50PM -0800, JD wrote:
startedConnecting isn't the right place. connectionMade is a little closer, but since you need to log in (tell the IRC server your nickname, etc) before you can issue commands like WHO, it isn't totally appropriate either. signedOn is the best place, since it is called by IRCClient once the server acknowledges the login information.
Indeed. This is not very easy to do with IRCClient, since the response the server sends doesn't have a whole lot to associate it with the WHO command you originally issued. You need to keep track of outstanding commands and try to match them up with server replies. I recognized this deficiency some time ago and did some initial work towards improving the API. WHO is one of the commands I did manage to improve, but I never moved any of the work out of my sandbox. You can find it in Twisted/sandbox/exarkun/irc2.py. It is under the same license as the rest of Twisted, even though it currently lacks any indication fo that. If it would be helpful to you, please feel free to use it. The improvements take one general form: commands which will produce server responses are exposed as functions which return a Deferred. When the server replies, the parsed information will be passed to that Deferred's callback. Here is a simple example of how you might use it (untested): class WhoingIRCClient(irc2.AdvancedClient): def signedOn(self): # Find everyone IRC'ing from Twisted Matrix's server. self.who('*twistedmatrix.com' ).addCallbacks(self._cbWho, self._ebWho ).addBoth(self._shutdownEverything ) def _cbWho(self, results): print 'channel user host server nick flags hops realname' for r in results: print ', '.join(r) def _ebWho(self, failure) print 'Error in who!' log.err(failure) def _shutdownEverything(self, ignored): reactor.stop() Jp
participants (8)
-
Andrew Bennetts
-
Bob Ippolito
-
darryl
-
Douglas Bagnall
-
Eric Mangold
-
Glyph Lefkowitz
-
JD
-
Jp Calderone