Mailman 3 April 2009 - Twisted

[Twisted-Python] Two-way telnet client problem
by Greg Taylor April 11, 2009

April 11, 2009

Hello all, I'm still pretty new with Twisted, so the documentation is a little bit cryptic and strange to wade through. I'm writing a MUD server that I'd like to add InterMUD Protocol (IMC) to. This basically allows any of the games running the MUD server to connect to an IMC server that lets players from other games talk to one another. I'm thinking I'll create another thread and run the IMC client from there so as to not block the main server thread. I've started tinkering with how to get the IMC client connected and able to send and receive, but I'm hitting a problem. If you look at my pasted example at http://python.pastebin.com/m155730ed you will see at the end that once the reactor is running, I _think_ this blocks things up for this thread. This is just fine, I assumed it'd be the case. However, let's go back to my main MUD server's thread for a moment. How would I tell the IMC client to send a line of text now? I assume it'd be something to do with the factory or reactor objects, but I'm just not sure. I've seen mention of something like: p = reactor.buildProtocol() p.sendLine('blah\n') But can't quite seem to piece it together. I'd really love to figure this out, as it'd do a lot for my game. I may be doing this horribly wrong, any direction would be great. Greg

2 3

Re: [Twisted-Python] Fork/Spawn children to accept connections on the same port.
by Eric York April 11, 2009

April 11, 2009

> > From: Jean-Paul Calderone <exarkun(a)divmod.com> > Date: April 10, 2009 8:57:19 AM PDT > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Subject: Re: [Twisted-Python] Fork/Spawn children to accept > connections on the same port. > Reply-To: Twisted general discussion <twisted- > python(a)twistedmatrix.com> > > > On Mon, 6 Apr 2009 15:25:39 -0700, Eric York <eyork(a)apple.com> wrote: >> I am trying to get the highest level of performance using all of >> the processors cores on a server. >> >> In the past, a unix app would bind/listen to a socket and then fork >> or spawn children to accept connections on that socket. I can’t see >> how to do that in Twisted. Can someone point me in the right >> direction? >> >> The solution of using a single process to accept connections and >> then farm out work to child processes, while a workable solution, >> isn’t at the same level of performance as children processes that >> are doing their own accepts. > It's also possible to bind a port in one process and then send it > over a UNIX socket to another process; this comes closer to the bind/ > fork > approach, but accomplishes the resource sharing explicitly via fd > passing > rather than through fork. This is the path that I would like to follow. I can see how to spawn a child process and pass the fd in Twisted. What I can't see how to do in Twisted is to have a parent process just bind and listen to a socket and have a child process accept on that socket. In twisted/ internet/tcp.py in the class Port, there is a function startListening which does bind/listen/startReading all in this one function. It seems that a small refactoring would allow the parent to bind and a child to do startReading, if the call to startReading was moved out of startListening. The reactor calls would also need a small refactoring to allow this type of setup. How does that sound? Eric

2 1

[Twisted-Python] PBClientFactory and getRootObject
by Fabrizio Mancini April 10, 2009

April 10, 2009

Hi everyone, I'm writing a server and a client using Perspective Broker with no Avatar and no auth. The server accumulates some data using a DeferredQueue. The clients using a remote call retrieve one item per time from the queue and elaborates it making some changes on a database using adbapi. What i want to know is: 1) how can i recursively call the remote method to retrieve the items from the server? 2) can i defer the database operations to a thread so every client can elaborates multiple requests per time? The code i'm using is this: from twisted.internet import reactor, defer, pb from twisted.enterprise import adbapi from twisted.python import log import os globalConfigurationFile=os.path.abspath('conf' + os.sep + 'configuration.ini') class DataBasePreparerClient(object): ''' DataBasePreparerClient() ''' def __init__(self, globalConfigurationFile): self._gcf = globalConfigurationFile self.data = {} self._parseConfig() self._createDbPool() self.d = defer.Deferred() self.clientfactory = pb.PBClientFactory() def _createDbPool(self): self.dbpool = adbapi.ConnectionPool('cx_Oracle', self.data['db_username'], self.data['db_password'], self.data['db_tns']) def connect(self): reactor.connectTCP(self.data['server_ip'], self.data['server_port'], self.clientfactory) self.d = self.clientfactory.getRootObject() self.dbpool.connect() self.d.addCallbacks(self.get_item, self._eb) def _eb(self, reason): print "Failure: ", reason.getErrorMessage() def get_item(self, result): d = result.callRemote("get_item") d.addCallback(self.got_item) def got_item(self, item): query = "update table where ..." res = self.dbpool.runOperation(query) res.addErrback(self._eb) self.d = self.clientfactory.getRootObject() self.d.addCallbacks(self.get_item, self._eb) if __name__ == "__main__": DataBasePreparerClient(globalConfigurationFile).connect() reactor.run() To call recursively the remote object in function connect i call clientfactory.getRootObject() and addCallback() for the first time, and the i recall always self.clientfactory.getRootObject() and self.d.addCallback when the db query has completed. Is this correct? Do I have to always call self.clientfactory.getRootObject() every time i have to call a remote method? Can i deferToThread the function got_item or self.dbpool.runOperation(query)? Thanks in advance Fabrizio

2 2

[Twisted-Python] adbapi and long-running transactions
by Wolfgang Powisch (privat) April 10, 2009

April 10, 2009

Hi, I'm using twisted.enterprise.adbapi for non-blocking database interaction with a postgres database. Currently I'm only using runQuery or runOperation for running single queries or some own little helper methods "runQueries" or "runOperations" for running multiple queries within one Transaction (using runInteraction). What I need now is to start a Transaction and access this Transaction from multiple callbacks within my application-code in the main-thread. In the last callback (and/or errback) I will then either commit or rollback the transaction. Any ideas how to acomplish that, didn't find a working solution yet. The two options I tested are: 1.) running my code within the adbapi thread using runInteraction: -> this is really not what I want and will screw up my code 2.) create a transaction and pass it as an argument to a function called with connectionpool._deferToThread -> this doesn't work, because I seem to end up in the wrong thread and another (or no) transaction. thx in advance for your help regards, Wolfgang

1 1

[Twisted-Python] RE: Question about Descriptors and SelectReactor
by Andrew Francis April 10, 2009

April 10, 2009

Hi Alec and Drew: Thanks for answering my question. From reading an excerpt of Steven's "UNIX Network Programming," it implies that the FD_SETSIZE is defined in the kernel. If you want a bigger FD_SETSIZE, you need to recompile the kernel. At least that is the case in BSD 4.4 UNIX. However I am using Linux. The Twisted documentation is recommending that it may be better to use pollreactor. Cheers, Andrew --- On Tue, 4/7/09, twisted-python-request(a)twistedmatrix.com <twisted-python-request(a)twistedmatrix.com> wrote: > From: twisted-python-request(a)twistedmatrix.com <twisted-python-request(a)twistedmatrix.com> > Subject: Twisted-Python Digest, Vol 61, Issue 10 > To: twisted-python(a)twistedmatrix.com > Date: Tuesday, April 7, 2009, 9:00 AM > Send Twisted-Python mailing list > submissions to > twisted-python(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > or, via email, send a message with subject or body 'help' > to > twisted-python-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more > specific > than "Re: Contents of Twisted-Python digest..." > > > Today's Topics: > > 1. RE: Question about Descriptors and > SelectReactor (Alec Matusis) > 2. trac server error (Johann Borck) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 6 Apr 2009 18:25:45 -0700 > From: "Alec Matusis" <matusis(a)yahoo.com> > Subject: RE: [Twisted-Python] Question about Descriptors > and > SelectReactor > To: "'Twisted general discussion'" <twisted-python(a)twistedmatrix.com> > Message-ID: <009701c9b71f$ca82aac0$5f880040$@com> > Content-Type: text/plain; > charset="utf-8" > > > I would say a function of the operating system - or > specifically > > ulimit on a *nix system. > > I do not think this is correct, on linux: ulimit -n is the > number of fds that can be written/read by a process, but the > 1024 limit on select is actually hard-compiled into the > kernel and can be changed only in the headers with kernel > recompilation. > > > -----Original Message----- > > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python- > > bounces(a)twistedmatrix.com] > On Behalf Of Drew Smathers > > Sent: Monday, April 06, 2009 8:40 AM > > To: Twisted general discussion > > Subject: Re: [Twisted-Python] Question about > Descriptors and > > SelectReactor > > > > On Sat, Apr 4, 2009 at 6:04 PM, Andrew Francis <andrewfr_ice(a)yahoo.com> > > wrote: > > > > > > Hi: > > > > > > Perhaps this question has been asked before but > is there a limit to > > the number of descriptors the SelectReactor can > handle? Is this a > > function of the operating system? > > > > > > > I would say a function of the operating system - or > specifically > > ulimit on a *nix system. > > > > -Drew > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python(a)twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

4 5

[Twisted-Python] trac server error
by Johann Borck April 10, 2009

April 10, 2009

there seems to be some serious problem with http://twistedmatrix.com/trac/ this is what the browser shows me on about one third of the requests: Traceback (most recent call last): File "/srv/trac/.local/lib/python2.4/site-packages/trac/web/api.py", line 382, in send_error 'text/html') File "/srv/trac/.local/lib/python2.4/site-packages/trac/web/chrome.py", line 475, in render_template return stream.render(method, doctype=doctype) File "/srv/trac/.local/lib/python2.4/site-packages/genshi/core.py", line 146, in render output = u''.join(list(generator)) File "/srv/trac/.local/lib/python2.4/site-packages/genshi/output.py", line 200, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/output.py", line 486, in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File "/srv/trac/.local/lib/python2.4/site-packages/genshi/output.py", line 436, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.4/site-packages/trac/web/chrome.py", line 478, in _strip_accesskeys for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/template.py", line 1145, in _match content = list(content) File "/srv/trac/.local/lib/python2.4/site-packages/genshi/filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/template.py", line 1120, in _match for event in stream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/template.py", line 1109, in _strip event = stream.next() File "/srv/trac/.local/lib/python2.4/site-packages/genshi/template.py", line 941, in _eval for event in substream: File "/srv/trac/.local/lib/python2.4/site-packages/genshi/template.py", line 920, in _eval ctxt): File "/srv/trac/.local/lib/python2.4/site-packages/genshi/template.py", line 930, in _eval result = data.evaluate(ctxt) File "/srv/trac/.local/lib/python2.4/site-packages/genshi/eval.py", line 101, in evaluate {'data': data}) File "/home/trac/.local/share/trac/templates/error.html", line 36, in <Expression u"get_reporter_id(req)"> <input type="hidden" name="reporter" value="${get_reporter_id(req)}" /> File "/srv/trac/.local/lib/python2.4/site-packages/trac/util/__init__.py", line 50, in get_reporter_id name = req.session.get('name', None) AttributeError: 'NoneType' object has no attribute 'get'

2 1

[Twisted-Python] Fork/Spawn children to accept connections on the same port.
by Eric York April 10, 2009

April 10, 2009

I am trying to get the highest level of performance using all of the processors cores on a server. In the past, a unix app would bind/listen to a socket and then fork or spawn children to accept connections on that socket. I can’t see how to do that in Twisted. Can someone point me in the right direction? The solution of using a single process to accept connections and then farm out work to child processes, while a workable solution, isn’t at the same level of performance as children processes that are doing their own accepts. Eric

2 1

[Twisted-Python] HTTP Proxy client?
by Alec Flett April 9, 2009

April 9, 2009

Is there any way to make twisted.web.client talk through an HTTP proxy? I don't know enough about http to know if its as simple as opening the connection to the proxy and just including the full URL in the GET line? Alec

2 1

[Twisted-Python] ZeroC Ice + Twisted?
by Einar S. Idsø April 9, 2009

April 9, 2009

Dear listmembers, I have a project that requires me to use ZeroC Ice for communicating with a server application. I would very much like to use Twisted for this project, and am therefore wondering if anyone has any experience with using Twisted with ZeroC Ice? Previous posts on this list have mentioned that ZeroC Ice has asynchronous operations and therefore should be easily twistable, but its Python bindings and type system are unpythonic (Stephen Waterbury, May 30th 2006) . But I haven't found any posts by people that have actually used Ice with Twisted, and whether it is possible to achieve proper twistified asynchronous behaviour. Any tales of experience, code snippets or anything else that could help me get started using Ice with Twisted and avoid *not* using Twisted would be greatly appreciated :) Cheers, Einar S. Idsø

1 0

[Twisted-Python] How to hook at StandardIO factory to a twistd app
by akrherz＠iastate.edu April 8, 2009

April 8, 2009

Hi Twisted! Could a kind soul provide me an example of how to connect my StandardIO factory to a twistd app? Here is an overview of my code class MyProtocol(basic.LineReceiver): # Blah Blah class MyFactory( stdio.StandardIO ): def __init__(self, protocol): self.protocol = protocol # then by .tac has application = service.Application("Ingestor") serviceCollection = service.IServiceCollection(application) proto = MyProtocol() factory = MyFactory( proto ) It is not clear to me how to connect my factory to the application, I do this many other times for network related factories by connecting via a TCPServer or TCPClient . Sorry for what I suspect is an embarrassingly simple question. :( thank you! daryl

3 2