Mailman 3 January 2011 - Twisted

[Twisted-Python] Design advice
by benjamin.bertrand＠lfv.se 17 Jan '11

17 Jan '11

Hi, I'm trying to do a proof of concept and need some advice about the design. We today have an external system sending messages to several of our tests systems (one specific connection per system). We'd like to get our own server for more flexibility (easily setup connection to a new system, filtering, re-sending...). The general idea so far is: - A server receives all the messages from the external system, parses them and writes them to a database. The server will be running as a daemon (should be on all the time). - Clients can retrieve messages from the database and send them to test systems (one client per system). Clients should as well be able to send "live" data to the system (after some filtering on message content). Clients are independent. They can be started/stopped depending on the need. The server and clients use a specific protocol to communicate with the other system. I already have implemented a server that receives messages from the external system. Parsing and writing them to a database shouldn't be too difficult. I see how I can tell a client to retrieve specific messages from the database to send them. What I'm not sure about is the "live" part (sending messages coming from the external system). Note that when I say "live", a small delay (up to 2-3 seconds) is not critical. 1) Each client could poll the database for change... They would need to poll it quite frequently, so don't think it's an option. 2) I read about being notified by a database on table changes (http://www.divillo.com/). That's much nicer than polling, but I don't think that's the best for my application. 3) I think a better solution would be to directly ask the server receiving the messages. Not sure about the best way to do that: 3.1) The server could forward the raw data to all the clients and let the client do the filtering. Lot of unneeded data would be sent. 3.2) The server could do the filtering and send only the needed messages. Is using twisted's perspective broker a good solution for that? 4) Another idea? In term of dimensioning, we can get around 200 messages per minute (max message length is 10000 bytes). Usually more around 20-30 messages per minute. The number of clients is small (around 10). Any advice? Thanks Benjamin

2 2

[Twisted-Python] Twisted and ipython
by Alessandro Dentella 17 Jan '11

17 Jan '11

Hi, I'm starting a project where I plan to use twisted for the first time. I'd like to practice using ipython and after looking around I found a recepe [1] by Metthew Scott and a comment by Bob Ippolito on that recepe that suggests a different approch [2]. In the ipython manual there's a reference to twisted in the -*thread section, but it's unclear to me what should be the option (-pylab as for matplotlib?): -gthread, -qthread, -q4thread, -wthread, -pylab Only ONE of these can be given, and it can only be given as the first option passed to IPython (it will have no effect in any other position). They provide threading support for the GTK, QT3, QT4 and WXWidgets toolkits, for the mat- plotlib library and Twisted reactor. So what's the suggested way to play with twisted? thanks in advance sandro *:-) [1] http://code.activestate.com/recipes/410670-integrating-twisted-reactor-with… [2] http://bob.pythonmac.org/archives/2005/04/17/twisted-and-foreign-event-loop… -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy

4 6

Re: [Twisted-Python] Boston Area Mini-Sprint
by Andrew Francis 16 Jan '11

16 Jan '11

Hi Glyph et al: Message: 4 Date: Sat, 15 Jan 2011 15:44:55 -0500 From: Glyph Lefkowitz <glyph(a)twistedmatrix.com> Subject: Re: [Twisted-Python] Boston Area Mini-Sprint To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <D5E13202-730B-4784-A29F-56180FB0A71D(a)twistedmatrix.com> Content-Type: text/plain; charset="us-ascii" >It's being held in somebody's home office in Somerville MA, and it's from >2-9PM EST. I'm excited about it myself, but it is *probably* overkill to >drive over 300 miles to get there :) - it's not close to a hotel or >anything. For that kind of long-distance thing, we have the sprints at >PyCon, which are substantially longer. Fair enough although I have never been to Boston. I will stay home and watch the New England Patriots/NY Giants game. About Pycon. For starters, congratulations on your talk acceptance. I will be a Pycon 2011 and if there is a Twisted Sprint (and no other competing sprint like Stackless), I would participate. However I have to make these plans in advance. Cheers, Andrew

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 15 Jan '11

15 Jan '11

1 0

[Twisted-Python] "small" jobs
by Glyph Lefkowitz 15 Jan '11

15 Jan '11

Hello Twisted Community, Tomorrow, we're going to have a sprint in Massachusetts, where core developers and several contributors will get together to improve Twisted's code and documentation. If you can't make it there, we'd be happy for you to play along at home, reviewing tickets and submitting patches. We'll be on the twisted IRC channel as usual. However, there are probably many of you who don't quite feel up to a sprint; who might not know enough about Python, or Twisted's internals, or the development process. Or maybe you're just not that confident in your coding skills. Twisted needs your help too. In the past year, we've had two people take over non-coding responsibilities. Jason J.W. Williams has become our official Success Stories ambassador (feel free to get in touch with him at <success(a)twistedmatrix.com> if you've got one!) and David Sturgis has become (somewhat less formally) our impromptu Boston-area sprint director. Partially as a result, the core team has been able to stop thinking about sprint organization and success stories, and finish several new features in time for 10.2 as well as getting that release out before the end of 2010. There are lots of other small jobs that need some attention. By committing to one of the following responsibilities, you'd take a load off the core developers' minds so that we can focus on doing that core development: Maintain a project page - one of the ones linked from <http://twistedmatrix.com/trac/wiki/TwistedProjects>. These should be the preferred "landing" pages for people with a particular interest in Twisted, like "I want to run a chat server", but right now they don't contain enough useful information. Maintain contact with our sponsors. This would involve remembering when particular people donated, getting in touch with them when their sponsor membership is about to expire, and putting up and taking down logos from the front page of twistedmatrix.com. Give credit where credit is due. <http://twistedmatrix.com/trac/wiki/TwistedMatrixLaboratories> and the CREDITS file in the Twisted release are kind of a mess; they refer to people who are long gone and they don't recognize people who worked hard on the current release. Please reply if you'd like to accept one of these responsibilities, or if you can think of another smallish, non-coding responsibility that someone could take over.

2 1

Re: [Twisted-Python] Boston Area Mini-Sprint
by Andrew Francis 15 Jan '11

15 Jan '11

Hi Folks: > Message: 1 > Date: Thu, 13 Jan 2011 12:32:08 -0500 > From: Tenth <tenth(a)twistedmatrix.com> > Subject: [Twisted-Python] Boston Area Mini-Sprint > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: > > <AANLkTim0WuyPW219MH+cxXJuKi1WVvBSiCZOW+sJogqc(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Apologies in advance for the short notice, but I'm planning > to host a Boston > area Twisted mini-sprint this weekend on Sunday, January > 16th. > > Space is fairly limited (hence the mini- prefix), but if > you're on > twisted-python and would be interested in the details of > this and/or future > Boston area sprints, let me know, and I'll add you to the > invitation. Yes this is short notice. Still, I am tempted to drive in for the sprint from Montreal, Canada. However where is the sprint being held, what time does it start, and how long will it be? Cheers, Andrew

2 1

[Twisted-Python] Boston Area Mini-Sprint
by Tenth 14 Jan '11

14 Jan '11

Apologies in advance for the short notice, but I'm planning to host a Boston area Twisted mini-sprint this weekend on Sunday, January 16th. Space is fairly limited (hence the mini- prefix), but if you're on twisted-python and would be interested in the details of this and/or future Boston area sprints, let me know, and I'll add you to the invitation. Thanks, - Dave

4 6

[Twisted-Python] Pantheon sponsorship logo
by David Strauss 13 Jan '11

13 Jan '11

Hi Twisters, I recently sent in a $1000/silver sponsorship to the Twisted project from Pantheon Systems, my new company. For the sponsor listing, I've attached a logo with the specified dimensions. Please link to our home page [1]. Thanks, and happy 2011. [1] http://getpantheon.com/ -- David Strauss | david(a)getpantheon.com | +1 512 577 5827 [mobile] Pantheon Systems | http://getpantheon.com/

3 2

Re: [Twisted-Python] Twisted-Python Digest, Vol 82, Issue 12
by williem75＠gmail.com 13 Jan '11

13 Jan '11

Sent from my LG phone twisted-python-request(a)twistedmatrix.com wrote: >Send Twisted-Python mailing list submissions to > twisted-python(a)twistedmatrix.com > >To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >or, via email, send a message with subject or body 'help' to > twisted-python-request(a)twistedmatrix.com > >You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of Twisted-Python digest..." > > >Today's Topics: > > 1. Re: client connecting to 2 servers (nonsimultaneously) > (benjamin.bertrand(a)lfv.se) > 2. Re: client connecting to 2 servers (nonsimultaneously) > (Jason Rennie) > 3. Re: client connecting to 2 servers (nonsimultaneously) > (exarkun(a)twistedmatrix.com) > 4. Re: client connecting to 2 servers (nonsimultaneously) > (Jason Rennie) > 5. Re: client connecting to 2 servers (nonsimultaneously) > (Glyph Lefkowitz) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Wed, 12 Jan 2011 09:49:34 +0000 >From: <benjamin.bertrand(a)lfv.se> >Subject: Re: [Twisted-Python] client connecting to 2 servers > (nonsimultaneously) >To: <twisted-python(a)twistedmatrix.com> >Message-ID: <2C9A58914594E34AA28179DB776FABDF92E4(a)xw-exch04.lfv.se> >Content-Type: text/plain; charset="iso-8859-1" > >One small question about the following code: >Why did you use reactor.callWhenRunning in the __init__ method? >Why not calling directly reactor.connectTCP? > >Cheers, > >Benjamin > >Fr?n: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] F?r Jason Rennie >Skickat: den 11 januari 2011 14:11 >Till: Twisted general discussion >?mne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) > >You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each?reactor.connectTCP. ?Also, no need for multiple reactor imports---one at the top of the code is fine. ?Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. ?IIUC, "fail" only handles the case that no connection is made (protocol is never created). > >import collections >from twisted.internet import reactor > >class MyClientFactory(object): >?? ?protocol = MyClientProtocol >?? ?def __init__(self, hosts): >?? ? ? ?self.hosts = collections.deque(hosts) >?? ? ? ?reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) >?? ? ? ?self.hosts.rotate(1) >?? ?def clientConnectionFailed(self, connector, reason): >?? ? ? ?reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) >?? ? ? ?self.hosts.rotate(1) > >factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) >reactor.run() > >Cheers, > >Jason > >On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand(a)lfv.se> wrote: >Hi, > >I'm new to twisted and I have started to write a new protocol with a TCP >client and server. >In my protocol, a client should be able to connect to 2 servers >(master/slave node - only the master accepts connection). >The client should try to connect to server1. If it fails, try to connect >to server2 (after a specific timeout). If that fails, try server1... >I came up with a solution (see below). >As I'm new to twisted and I haven't seen anything like that in the >examples, I'd like to check if that's a proper way to do it. >Any comments is welcome. > >Thanks > >Benjamin > >*********************************************** >class MyClientFactory(ClientFactory): > >? ?protocol = MyClientProtocol > >? ?def __init__(self, host2=None): >? ? ? ?self.host1 = None >? ? ? ?self.host2 = host2 > >? ?def clientConnectionFailed(self, connector, reason): >? ? ? ?from twisted.internet import reactor >? ? ? ?if self.host2 is None: >? ? ? ? ? ?# host2 is not defined, reconnect to host1 >? ? ? ? ? ?reactor.callLater(2.0, connector.connect) >? ? ? ?else: >? ? ? ? ? ?destination = connector.getDestination() >? ? ? ? ? ?if self.host1 is None: >? ? ? ? ? ? ? ?# First connection failed, initialize host1, and try >host2 >? ? ? ? ? ? ? ?self.host1 = destination.host >? ? ? ? ? ? ? ?host = self.host2 >? ? ? ? ? ?elif destination.host == self.host1: >? ? ? ? ? ? ? ?# Connection to host1 failed, try host2 >? ? ? ? ? ? ? ?host = self.host2 >? ? ? ? ? ?else: >? ? ? ? ? ? ? ?# Connection to host2 failed, try host1 >? ? ? ? ? ? ? ?host = self.host1 >? ? ? ? ? ?reactor.callLater(2.0, reactor.connectTCP, host, >destination.port, self) > > >factory = MyClientFactory(server2) >from twisted.internet import reactor >reactor.connectTCP(server1, 8010, factory) >reactor.run() >*********************************************** > > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python(a)twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > >-- >Jason Rennie >Research Scientist, ITA Software >617-714-2645 >http://www.itasoftware.com/ > > > >------------------------------ > >Message: 2 >Date: Wed, 12 Jan 2011 10:15:59 -0500 >From: Jason Rennie <jrennie(a)gmail.com> >Subject: Re: [Twisted-Python] client connecting to 2 servers > (nonsimultaneously) >To: Twisted general discussion <twisted-python(a)twistedmatrix.com> >Message-ID: > <AANLkTi=ge7JGPY3_woNu6LCapUEXjH88pRZ=TB05g5dY(a)mail.gmail.com> >Content-Type: text/plain; charset="iso-8859-1" > >Habit, mostly. I think it's a good habit, though. The reason is that I >wanted to make sure the __init__ code completed before the connectTCP was >called. In this case, it doesn't matter, but if I had a number of things I >wanted to do in __init__, it might matter. Consider: > >class MyClientProtocol(Protocol): > def connectionMade(self): > self.factory.numConnections += 1 > >class MyClientFactory(Factory): > def __init__(self): > reactor.connectTCP(host, port, self) > self.numConnections = 0 > >'course, I'm sure you'd put reactor.connectTCP *after* the numConnections >initializer, but using callWhenRunning ensures that the order doesn't >matter, so it's one less thing you have to worry about :-) > >Jason > >On Wed, Jan 12, 2011 at 4:49 AM, <benjamin.bertrand(a)lfv.se> wrote: > >> One small question about the following code: >> Why did you use reactor.callWhenRunning in the __init__ method? >> Why not calling directly reactor.connectTCP? >> >> Cheers, >> >> Benjamin >> >> >-- >Jason Rennie >Research Scientist, ITA Software >617-714-2645 >http://www.itasoftware.com/ >

1 0

[Twisted-Python] client connecting to 2 servers (non simultaneously)
by benjamin.bertrand＠lfv.se 13 Jan '11

13 Jan '11

Hi, I'm new to twisted and I have started to write a new protocol with a TCP client and server. In my protocol, a client should be able to connect to 2 servers (master/slave node - only the master accepts connection). The client should try to connect to server1. If it fails, try to connect to server2 (after a specific timeout). If that fails, try server1... I came up with a solution (see below). As I'm new to twisted and I haven't seen anything like that in the examples, I'd like to check if that's a proper way to do it. Any comments is welcome. Thanks Benjamin *********************************************** class MyClientFactory(ClientFactory): protocol = MyClientProtocol def __init__(self, host2=None): self.host1 = None self.host2 = host2 def clientConnectionFailed(self, connector, reason): from twisted.internet import reactor if self.host2 is None: # host2 is not defined, reconnect to host1 reactor.callLater(2.0, connector.connect) else: destination = connector.getDestination() if self.host1 is None: # First connection failed, initialize host1, and try host2 self.host1 = destination.host host = self.host2 elif destination.host == self.host1: # Connection to host1 failed, try host2 host = self.host2 else: # Connection to host2 failed, try host1 host = self.host1 reactor.callLater(2.0, reactor.connectTCP, host, destination.port, self) factory = MyClientFactory(server2) from twisted.internet import reactor reactor.connectTCP(server1, 8010, factory) reactor.run() ***********************************************

6 14