[Twisted-Python] implementing NAMES in IRC
First, I apologize if this is not the appropriate place for this. I have a basic bot written as a mechanism for learning classes and methods. I think I've got methods down, but classes are kicking my butt. As a personal exercise, I'm trying to implement a check in an IRC bot to see if the person issuing a command has oper (@) status. in IRC this is done with the 'NAMES <#channel>' command. Google gives me this: http://stackoverflow.com/questions/6671620/list-users-in-irc-channel-using-t... My own research points me to this: http://twistedmatrix.com/trac/browser/trunk/twisted/words/protocols/irc.py#L... Now, is stackoverflow overriding that method or was it perhaps written prior to the implementation of the irc.py portion I linked to? My code is here. I am attempting to parse the names list at line 110 http://bpaste.net/show/Fll7at9Z3b8nD6GDNN14/ I don't know if I should be using twisted/words/protocols/irc.py or twisted/words/im/ircsupport.py. Can someone point me in the right direction as to what I need to do to grab the list of names at that particular spot using Twisted's in methods? If it cannot be done with the Twisted's methods, am I to understand that's what the StackOverflow post was doing? Yes, I know the code is ugly. I can clean it up after I understand what I'm doing more fully. -- Molon Labe "Come and take them" "The Marines I have seen around the world have the cleanest bodies, the filthiest minds, the highest morale, and the lowest morals of any group of animals I have ever seen. Thank God for the United States Marine Corps!" -Eleanor Roosevelt
On Sep 10, 2012, at 6:24 PM, Art Scheel <ascheel@gmail.com> wrote:
Now, is stackoverflow overriding that method or was it perhaps written prior to the implementation of the irc.py portion I linked to?
The StackOverflow answer is implementing a new method, on IRCClient. The implementation in irc.py that you're linking to is from the IRC server protocol implementation. They're not really related, except insofar as they are talking about the same protocol command - but they are opposite sides of the connection for that command. For what it's worth, the StackOverflow answer really ought to be linking to a Twisted ticket for a method like .names() on IRCClient within Twisted itself. This is important functionality, and we're missing it; the right thing to do is always to file a bug, not to implement it without tests in a forum post. Such a bug may already exist - if you wouldn't mind, would you search for it, and if you can't find it, file a new one?
twisted.words.im.* implements a multi-protocol chat abstraction. I don't think that your bot necessarily needs to be IRC specific, I think it would be a good idea for you to try to use twisted.words.im; you will probably discover a sad lack of documentation, but you can feel free to ask questions and file doc bugs. Or perhaps even contribute doc patches, once you've figured out what's going on! At the very least, twisted.words.im.ircsupport already implements irc_RPL_NAMREPLY to convert this message into a series of memberJoined/memberLeft API calls, so you don't have to do something like that yourself. See here: <http://twistedmatrix.com/trac/browser/trunk/twisted/words/im/ircsupport.py#L...>.
Can someone point me in the right direction as to what I need to do to grab the list of names at that particular spot using Twisted's in methods? If it cannot be done with the Twisted's methods, am I to understand that's what the StackOverflow post was doing?
Yes, that's exactly what the SO post was doing. It is using Twisted's methods though, so clearly it can be done with them, it's just a small matter of work :).
Yes, I know the code is ugly. I can clean it up after I understand what I'm doing more fully.
Ugliness is forgivable, but in the future, you should try to post much more minimal programs. It's a lot easier to help if one can read the whole thing. See <http://sscce.org> for details. -glyph
On Tue, Sep 11, 2012 at 6:05 AM, Glyph <glyph@twistedmatrix.com> wrote:
Such a bug may already exist - if you wouldn't mind, would you search for it, and if you can't find it, file a new one?
I don't know if there's an existing bug or not, but implementing a names() method is problematic. The natural API would be something like names('#foo') -> Deferred firing with a list of names in the channel. The difficulty in implementing this comes from the fact that there are two possible results from issuing a NAMES command on IRC: a serious of zero or more RPL_NAMREPLY messages followed by an RPL_ENDOFNAMES message, *or* any error message the IRC server feels like. There are a number of partial workarounds for this, involving various combinations of queues, timeouts, and unreliable detection of responses, but none of them seem particularly satisfactory to me. (On the other hand, a method that just sends NAMES #foo without any response handling would be pretty straightforward to implement, maybe that's what you meant?) -- mithrandi, i Ainil en-Balandor, a faer Ambar
On 10:49 am, mithrandi@mithrandi.net wrote:
This is the excuse that is always given for not implementing a new feature on `IRCClient`. However, here's another equivalent way of stating the objection: IRC is a terrible protocol and it is very difficult to implement a method like `names` reliably, due to the various vague and obscure corner cases presented. Therefore, instead of Twisted tackling this problem and providing a single (perhaps imperfect) implementation, every single IRC application developer should instead rediscover this sad reality for themselves and then implement their own uniquely buggy version of this functionality. Hopefully, cast in that light, it's a bit more clear why this might not be a reasonable argument. I can think of other arguments against adding a `names` (or similar) method to `IRCClient`, but since it's not likely anyone is actually going to step up and do any work on `irc.py` anyway, I don't feel the need to make them (and I'd much rather someone actually maintain `irc.py`). Jean-Paul
On Tue, Sep 11, 2012 at 3:43 PM, <exarkun@twistedmatrix.com> wrote:
Actually, my recommendation would be to avoid trying to implement functionality of this nature in any application at all. The only way to reliably use the NAMES command is the way twisted.words.im.ircsupport and real IRC clients do: handle RPL_NAMREPLY / RPL_ENDOFNAMES without regard to any NAMES command you may or may not have issued, and then issue a NAMES command at certain points without any regard for a response that may or may not be sent to you by the IRC server. Having said that, I guess IRCClient could handle the parsing of RPL_NAMREPLY / RPL_ENDOFNAMES messages into a single list of names, which could then be delivered as a single event. My main concern with most of the more complex implementation strategies is that they impose a non-zero cost on users of IRCClient, as well as encouraging the implementation of unreliable features; at the very least, I think "anti-IRC" code like this should be an extra layer on top of the basic IRC protocol implementation. -- mithrandi, i Ainil en-Balandor, a faer Ambar
On Sep 11, 2012, at 9:12 AM, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
So... you're recommending exactly what twisted.words.im already does? :) (This is another area of the code that could use a maintainer, but its design is less horrible than its lack of testing and lack of coding standard compliance might imply.) Keep in mind that we can assume to have control over outgoing traffic on our own IRC connection. If you've already issued one .names(), then another outgoing one doesn't have to send an additional command and try to do weird Deferred-stacking stuff. It just has to delay sending the actual NAMES request until the previous one has come back, or perhaps just fire both Deferreds with a single response. -glyph
On Tue, Sep 11, 2012 at 6:46 PM, Glyph <glyph@twistedmatrix.com> wrote:
So... you're recommending exactly what twisted.words.im already does? :)
Well, yeah: "The only way to reliably use the NAMES command is the way twisted.words.im.ircsupport [already does]" :)
Keep in mind that we can assume to have control over outgoing traffic on our own IRC connection. If you've already issued one .names(), then another outgoing one doesn't have to send an additional command and try to do weird Deferred-stacking stuff. It just has to delay sending the actual NAMES request until the previous one has come back, or perhaps just fire both Deferreds with a single response.
On the other hand, if you're writing code that doesn't need to match requests and responses, you should be able to issue another NAMES command before the previous one has completed (and bear in mind, if you want to queue NAMES requests independently of other requests, the only ways for it to complete will be 1. RPL_ENDOFNAMES and 2. timeout). That's why I would prefer this not to be baked into the lowest level of the IRC client API. My earlier responses were probably worded too harshly; people *are* going to attempt to implement things like this, no matter how much of a bad idea it might be, so going LALALALA WE WON'T LET YOU won't improve things. On the other hand, the compromises needed to do things this way probably shouldn't penalize people who are doing it the "right" way (insofar as anything about IRC can be said to be "right"). -- mithrandi, i Ainil en-Balandor, a faer Ambar
On Tue, Sep 11, 2012 at 9:43 AM, <exarkun@twistedmatrix.com> wrote:
I know you know what this means, exarkun, but for those who don't know exactly what this means, it means that you can send two requests out, and not be guaranteed to get replies in any particular order (or at all), and that means that you can get ambiguous replies to requests. Additionally, IRC daemons may play tricks and send out "replies" that doesn't associate with any request you've sent, as a means to have a client perform a specific action; for instance, bouncers usually send join replies without any associated requests method of forcing a client to open a buffer for a particular channel, and certain networks also use this same technique to force users on specific channels all the time (and they will also make a PART request on this particular channel fail). These two things mean that you cannot have a Deferred-based API for IRC, where each request correlates to one response, and instead need to have a reactionary system where you cast a NAMES out to the wind, and just implement some sane behavior on every RPL_NAMREPLY you get, independent of what triggered it.
-- Jasper
On Sep 11, 2012, at 3:01 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
It's not exactly that they're "forcing" users (or clients) to do anything. They're just notifying the user that they're in a particular channel. In other words, the "forcing" is all done server-side, not in the protocol itself.
This is actually a better way to structure applications in general, even for much more sanely structured protocols than IRC. One subtle point about Twisted is that while Deferreds make it a lot easier to manage sequential work-flow in an event driven system, it is sequential work-flow itself that is the problem in many cases. You should always try to write your protocols so they can react to any valid input at any time, not rigidly structure everything so that your state is always shoehorned into a request/response structure so all your state is encapsulated in stack frames in closures that are added as callbacks on Deferreds. If I were designing a new chat protocol today, I'd definitely make sure it had a "you joined a conversation" message that did not have to be a response to any particular "I'd like to join a conversation" message :-). -glyph
On 12/09/2012, at 8:01 AM, "Jasper St. Pierre" <jstpierre@mecheye.net> wrote:
However it is entirely possible to sanely parse and store replies in a reactionary system because the first component of an RPL_NAMREPLY message is the name of the channel it applies to. You can log or quietly drop replies that aren't for channels you're currently following, and use the others to build out some kind of structure that maintains an active list of users in that channel and their status (maintained through subsequent joins and parts). Is this not so? It doesn't sound that complicated to me.
On Wed, Sep 12, 2012 at 2:12 AM, Adrian Overbury <skaarjj@gmail.com> wrote:
However it is entirely possible to sanely parse and store replies in a reactionary system because the first component of an RPL_NAMREPLY message is the name of the channel it applies to.
You can log or quietly drop replies that aren't for channels you're currently following, and use the others to build out some kind of structure that maintains an active list of users in that channel and their status (maintained through subsequent joins and parts). Is this not so? It doesn't sound that complicated to me.
Handling RPL_NAMREPLY/RPL_ENDOFNAMES responses in order to update a list of users is perfectly feasible (most user-interactive IRC clients do something of this nature); the problem is directly associating a response with a request. If you send NAMES #code, there are three possibilities: 1) the ircd sends you a sequence of RPL_NAMREPLY/RPL_ENDOFNAMES with the list of users, 2) you get an error response, 3) nothing happens. While 3) is unlikely, there's no particular reason an ircd couldn't behave that way (I don't specifically know of one, but in general, the IRC protocol and ircds are written and designed with primarily the interactive use case in mind, not automated agents). However, the real fly in the ointment is 2); there is no way to know what errors might be returned in response to a NAMES message, and many of those responses will not even mention the channel name, thus giving you no way to recognize that the request will never "complete". If you're keeping a deferred around to fire with the response, then unless something else intervenes (a cancellation timer?), the deferred will just hang around, which also means keeping its callbacks and the objects they reference alive. Thus, the problem is not with tracking the users in a channel in general (although there are issues with tracking the channel modes of those users...), but rather with specifically implementing an API like names(channel) -> Deferred firing with a list of names. If you're willing to accept an arbitrary timeout and use that as the only (or at least the primary) mechanism of failure detection, as well as a few other compromises (in the area of "overlapping" requests for the same channel), it would be possible, I just don't feel that the base IRCClient class is the right place to implement things like this, and I'd also discourage designing IRC clients (whether they are bots or user-interactive clients) in a way that requires such an API. -- mithrandi, i Ainil en-Balandor, a faer Ambar
On Sep 10, 2012, at 6:24 PM, Art Scheel <ascheel@gmail.com> wrote:
Now, is stackoverflow overriding that method or was it perhaps written prior to the implementation of the irc.py portion I linked to?
The StackOverflow answer is implementing a new method, on IRCClient. The implementation in irc.py that you're linking to is from the IRC server protocol implementation. They're not really related, except insofar as they are talking about the same protocol command - but they are opposite sides of the connection for that command. For what it's worth, the StackOverflow answer really ought to be linking to a Twisted ticket for a method like .names() on IRCClient within Twisted itself. This is important functionality, and we're missing it; the right thing to do is always to file a bug, not to implement it without tests in a forum post. Such a bug may already exist - if you wouldn't mind, would you search for it, and if you can't find it, file a new one?
twisted.words.im.* implements a multi-protocol chat abstraction. I don't think that your bot necessarily needs to be IRC specific, I think it would be a good idea for you to try to use twisted.words.im; you will probably discover a sad lack of documentation, but you can feel free to ask questions and file doc bugs. Or perhaps even contribute doc patches, once you've figured out what's going on! At the very least, twisted.words.im.ircsupport already implements irc_RPL_NAMREPLY to convert this message into a series of memberJoined/memberLeft API calls, so you don't have to do something like that yourself. See here: <http://twistedmatrix.com/trac/browser/trunk/twisted/words/im/ircsupport.py#L...>.
Can someone point me in the right direction as to what I need to do to grab the list of names at that particular spot using Twisted's in methods? If it cannot be done with the Twisted's methods, am I to understand that's what the StackOverflow post was doing?
Yes, that's exactly what the SO post was doing. It is using Twisted's methods though, so clearly it can be done with them, it's just a small matter of work :).
Yes, I know the code is ugly. I can clean it up after I understand what I'm doing more fully.
Ugliness is forgivable, but in the future, you should try to post much more minimal programs. It's a lot easier to help if one can read the whole thing. See <http://sscce.org> for details. -glyph
On Tue, Sep 11, 2012 at 6:05 AM, Glyph <glyph@twistedmatrix.com> wrote:
Such a bug may already exist - if you wouldn't mind, would you search for it, and if you can't find it, file a new one?
I don't know if there's an existing bug or not, but implementing a names() method is problematic. The natural API would be something like names('#foo') -> Deferred firing with a list of names in the channel. The difficulty in implementing this comes from the fact that there are two possible results from issuing a NAMES command on IRC: a serious of zero or more RPL_NAMREPLY messages followed by an RPL_ENDOFNAMES message, *or* any error message the IRC server feels like. There are a number of partial workarounds for this, involving various combinations of queues, timeouts, and unreliable detection of responses, but none of them seem particularly satisfactory to me. (On the other hand, a method that just sends NAMES #foo without any response handling would be pretty straightforward to implement, maybe that's what you meant?) -- mithrandi, i Ainil en-Balandor, a faer Ambar
On 10:49 am, mithrandi@mithrandi.net wrote:
This is the excuse that is always given for not implementing a new feature on `IRCClient`. However, here's another equivalent way of stating the objection: IRC is a terrible protocol and it is very difficult to implement a method like `names` reliably, due to the various vague and obscure corner cases presented. Therefore, instead of Twisted tackling this problem and providing a single (perhaps imperfect) implementation, every single IRC application developer should instead rediscover this sad reality for themselves and then implement their own uniquely buggy version of this functionality. Hopefully, cast in that light, it's a bit more clear why this might not be a reasonable argument. I can think of other arguments against adding a `names` (or similar) method to `IRCClient`, but since it's not likely anyone is actually going to step up and do any work on `irc.py` anyway, I don't feel the need to make them (and I'd much rather someone actually maintain `irc.py`). Jean-Paul
On Tue, Sep 11, 2012 at 3:43 PM, <exarkun@twistedmatrix.com> wrote:
Actually, my recommendation would be to avoid trying to implement functionality of this nature in any application at all. The only way to reliably use the NAMES command is the way twisted.words.im.ircsupport and real IRC clients do: handle RPL_NAMREPLY / RPL_ENDOFNAMES without regard to any NAMES command you may or may not have issued, and then issue a NAMES command at certain points without any regard for a response that may or may not be sent to you by the IRC server. Having said that, I guess IRCClient could handle the parsing of RPL_NAMREPLY / RPL_ENDOFNAMES messages into a single list of names, which could then be delivered as a single event. My main concern with most of the more complex implementation strategies is that they impose a non-zero cost on users of IRCClient, as well as encouraging the implementation of unreliable features; at the very least, I think "anti-IRC" code like this should be an extra layer on top of the basic IRC protocol implementation. -- mithrandi, i Ainil en-Balandor, a faer Ambar
On Sep 11, 2012, at 9:12 AM, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
So... you're recommending exactly what twisted.words.im already does? :) (This is another area of the code that could use a maintainer, but its design is less horrible than its lack of testing and lack of coding standard compliance might imply.) Keep in mind that we can assume to have control over outgoing traffic on our own IRC connection. If you've already issued one .names(), then another outgoing one doesn't have to send an additional command and try to do weird Deferred-stacking stuff. It just has to delay sending the actual NAMES request until the previous one has come back, or perhaps just fire both Deferreds with a single response. -glyph
On Tue, Sep 11, 2012 at 6:46 PM, Glyph <glyph@twistedmatrix.com> wrote:
So... you're recommending exactly what twisted.words.im already does? :)
Well, yeah: "The only way to reliably use the NAMES command is the way twisted.words.im.ircsupport [already does]" :)
Keep in mind that we can assume to have control over outgoing traffic on our own IRC connection. If you've already issued one .names(), then another outgoing one doesn't have to send an additional command and try to do weird Deferred-stacking stuff. It just has to delay sending the actual NAMES request until the previous one has come back, or perhaps just fire both Deferreds with a single response.
On the other hand, if you're writing code that doesn't need to match requests and responses, you should be able to issue another NAMES command before the previous one has completed (and bear in mind, if you want to queue NAMES requests independently of other requests, the only ways for it to complete will be 1. RPL_ENDOFNAMES and 2. timeout). That's why I would prefer this not to be baked into the lowest level of the IRC client API. My earlier responses were probably worded too harshly; people *are* going to attempt to implement things like this, no matter how much of a bad idea it might be, so going LALALALA WE WON'T LET YOU won't improve things. On the other hand, the compromises needed to do things this way probably shouldn't penalize people who are doing it the "right" way (insofar as anything about IRC can be said to be "right"). -- mithrandi, i Ainil en-Balandor, a faer Ambar
On Tue, Sep 11, 2012 at 9:43 AM, <exarkun@twistedmatrix.com> wrote:
I know you know what this means, exarkun, but for those who don't know exactly what this means, it means that you can send two requests out, and not be guaranteed to get replies in any particular order (or at all), and that means that you can get ambiguous replies to requests. Additionally, IRC daemons may play tricks and send out "replies" that doesn't associate with any request you've sent, as a means to have a client perform a specific action; for instance, bouncers usually send join replies without any associated requests method of forcing a client to open a buffer for a particular channel, and certain networks also use this same technique to force users on specific channels all the time (and they will also make a PART request on this particular channel fail). These two things mean that you cannot have a Deferred-based API for IRC, where each request correlates to one response, and instead need to have a reactionary system where you cast a NAMES out to the wind, and just implement some sane behavior on every RPL_NAMREPLY you get, independent of what triggered it.
-- Jasper
On Sep 11, 2012, at 3:01 PM, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
It's not exactly that they're "forcing" users (or clients) to do anything. They're just notifying the user that they're in a particular channel. In other words, the "forcing" is all done server-side, not in the protocol itself.
This is actually a better way to structure applications in general, even for much more sanely structured protocols than IRC. One subtle point about Twisted is that while Deferreds make it a lot easier to manage sequential work-flow in an event driven system, it is sequential work-flow itself that is the problem in many cases. You should always try to write your protocols so they can react to any valid input at any time, not rigidly structure everything so that your state is always shoehorned into a request/response structure so all your state is encapsulated in stack frames in closures that are added as callbacks on Deferreds. If I were designing a new chat protocol today, I'd definitely make sure it had a "you joined a conversation" message that did not have to be a response to any particular "I'd like to join a conversation" message :-). -glyph
On 12/09/2012, at 8:01 AM, "Jasper St. Pierre" <jstpierre@mecheye.net> wrote:
However it is entirely possible to sanely parse and store replies in a reactionary system because the first component of an RPL_NAMREPLY message is the name of the channel it applies to. You can log or quietly drop replies that aren't for channels you're currently following, and use the others to build out some kind of structure that maintains an active list of users in that channel and their status (maintained through subsequent joins and parts). Is this not so? It doesn't sound that complicated to me.
On Wed, Sep 12, 2012 at 2:12 AM, Adrian Overbury <skaarjj@gmail.com> wrote:
However it is entirely possible to sanely parse and store replies in a reactionary system because the first component of an RPL_NAMREPLY message is the name of the channel it applies to.
You can log or quietly drop replies that aren't for channels you're currently following, and use the others to build out some kind of structure that maintains an active list of users in that channel and their status (maintained through subsequent joins and parts). Is this not so? It doesn't sound that complicated to me.
Handling RPL_NAMREPLY/RPL_ENDOFNAMES responses in order to update a list of users is perfectly feasible (most user-interactive IRC clients do something of this nature); the problem is directly associating a response with a request. If you send NAMES #code, there are three possibilities: 1) the ircd sends you a sequence of RPL_NAMREPLY/RPL_ENDOFNAMES with the list of users, 2) you get an error response, 3) nothing happens. While 3) is unlikely, there's no particular reason an ircd couldn't behave that way (I don't specifically know of one, but in general, the IRC protocol and ircds are written and designed with primarily the interactive use case in mind, not automated agents). However, the real fly in the ointment is 2); there is no way to know what errors might be returned in response to a NAMES message, and many of those responses will not even mention the channel name, thus giving you no way to recognize that the request will never "complete". If you're keeping a deferred around to fire with the response, then unless something else intervenes (a cancellation timer?), the deferred will just hang around, which also means keeping its callbacks and the objects they reference alive. Thus, the problem is not with tracking the users in a channel in general (although there are issues with tracking the channel modes of those users...), but rather with specifically implementing an API like names(channel) -> Deferred firing with a list of names. If you're willing to accept an arbitrary timeout and use that as the only (or at least the primary) mechanism of failure detection, as well as a few other compromises (in the area of "overlapping" requests for the same channel), it would be possible, I just don't feel that the base IRCClient class is the right place to implement things like this, and I'd also discourage designing IRC clients (whether they are bots or user-interactive clients) in a way that requires such an API. -- mithrandi, i Ainil en-Balandor, a faer Ambar
participants (6)
-
Adrian Overbury -
Art Scheel -
exarkun＠twistedmatrix.com -
Glyph -
Jasper St. Pierre -
Tristan Seligmann