Mailman 3 September 2004 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 05 Aug '21

05 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool 28 Apr '13

28 Apr '13

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def xmlrpc_whatIsMyAddress(self): return self.transport.getPeer().host > > if __name__ == '__main__': > from twisted.internet import reactor > r = Example() > reactor.listenTCP(7080, server.Site(r)) > reactor.run() > > I want to be able to get the address of the client that calls the xmlrpc > method can anyone help me with this? This solution didn't work because 'transport' isn't a property of the Example class. I'm currently in the process of changing from a customized SimpleXMLRPCServer to a twisted XMLRPC server solution and I need to insert the client IP into the attributes passed to the called xmlrpc method. Anyone who knows the answer and is willing to share the info? Regards, Remy

3 3

[Twisted-Python] twisted and threading
by twisted-10152003＠atlas.lcs.mit.edu 26 Nov '04

26 Nov '04

I wrapped a c library using swig and was hoping to make use of it in a twisted server I am writing. The issue is that the c library fundamentally blocks waiting for input that may never come. Is there a way from twisted to deal with this cleanly within the framework? I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet. (I saw all the previous posts on this topic, but the recommendations don't seem applicable in my case since the blocking is occurring outside my control in a c library and the blocking call may never return.) Thanks, Steve Bauer

8 10

[Twisted-Python] twisted.internet and IPv6
by Stefan Behnel 18 Oct '04

18 Oct '04

Hi! I'm curious: how much work would it be to enable IPv6 support in twisted.internet? When I do --------------------------------------- cd twisted/internet fgrep INET *.py --------------------------------------- I get a number of references to IPv4 that make me believe that t.i.* is not quite IPv6 clean by itself: --------------------------------------- default.py: server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) default.py: client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) javareactor.py: return ('INET', InetAddress.getLocalHost().getHostAddress(), self.jport.port) javareactor.py: return ('INET', addr.getHostAddress(), self.skt.getPort()) javareactor.py: return ("INET", self.host, self.port) tcp.py: addressFamily = socket.AF_INET tcp.py: return address.IPv4Address('TCP', *(self.socket.getsockname() + ('INET',))) tcp.py: return address.IPv4Address('TCP', *(self.addr + ('INET',))) tcp.py: return address.IPv4Address('TCP', *(self.socket.getsockname() + ('INET',))) tcp.py: return address.IPv4Address('TCP', *(self.client + ('INET',))) tcp.py: addressFamily = socket.AF_INET tcp.py: return address.IPv4Address('TCP', *(self.socket.getsockname() + ('INET',))) tcp.py: return address.IPv4Address('TCP', self.host, self.port, 'INET') udp.py: addressFamily = socket.AF_INET udp.py: return address.IPv4Address('UDP', *(self.socket.getsockname() + ('INET_UDP',))) udp.py: Returns a tuple of ('INET_UDP', hostname, port), indicating udp.py: return address.IPv4Address('UDP', self.remotehost, self.remoteport, 'INET_UDP') --------------------------------------- What would be needed? Would a new reactor be a good solution? "reactor6" ? Or rather extending the current one? I'd say a reactor6 would be cleaner, but on the other hand, a simple flag could do the trick from the API perspective. Any comments on this? Stefan

3 3

[Twisted-Python] new to twisted
by m. smadi 18 Oct '04

18 Oct '04

hi; i was writing third party sip call controller when a friend suggested i use twisted. I think it will save me a lot of time, but i had trouble with the following when installing: cvs the pserial and pcrypto required package. the cvs server complians about a wrong password (any hints) i continued without these package and installed using setup.py. Where is a good place to start (the tutorial provided is not really very great) - any other suggestions? moe smadi

4 3

[Twisted-Python] Twisted scalability with hundreds of outbound (client) connections
by Phil Mayers 04 Oct '04

04 Oct '04

I have an SNMP poller written in Python that I'd like to turn into a long-running server process. The obvious thing (it is obvious - I really need the support infrastructure) is to convert it to twisted, since it provides easy support to turn the process into a server as well as a client (e.g. XMLRPC query mapping onto an SNMP fetch; using DBAPI to update node status, all hooked in via deferred, etc.) However, I've done a couple of proof-of-concept ports and am having scalability problems related to the way reactors work. I'm not entirely certain I've got the problem nailed, since the reactor sources are... complex... but I think what happens is this: The framework is heavily oriented towards getting IO out as soon as possible, and since there's no "scheduling" of receive events, protocol instances talking to "fast" clients eat all the CPU, and the socket buffer overflows since each datagramReceived triggers another write(). Possibly an example would help: class agent: def start(self): d = self.get_table('ifTable') d.addCallbacks(self.step2, self.die) def step2(self, ifTable): # Do some processing, then more data fetch d = self.get_table('ipAddrTable') d.addCallbacks(self.step3, self.die) def step3(self, ipAddrTable): # etc. etc. def do_timeouts(reactor): # get the list of sent PDUs now = time.time() for pdu in pdus: if pdu.due < now: pdu.deferred.errback(Timeout()) reactor.callLater(1, do_timeouts, reactor) import sys from twisted.internet import reactor for host in sys.argv[1:]: a = agent(host) reactor.callLater(0, a.start) reactor.callLater(1, do_timeouts, reactor) reactor.run() Now, I know there are scalability problems with callLater, but that's only being used for the startup because I can't think of a cleaner way of doing it - future activity is all triggered by deferreds. I'm handling the timeouts by running a once-per-second expiry which is sufficient for my needs. The *problem* is that the SNMP agents I'm talking to have an extremely wide range of performances, response times and quantity of data to retrieve, and the faster endpoints are consuming a disproportionate quantity of the CPU time; so much so in fact, that the UDP socket receive queue limit is being exceeded (I'm using a single UDP socket for all the clients, and breaking the responses back out based on SNMP message ID - this is to overcome FD limits). We're talking 750-1000 clients here BTW. What I really need is some way of making the reactor do a fair-queuing style of transmit/receive; I have some not-very-clear notion in my head of transport.write not actually doing the write, but just putting the output data onto a per-agent queue, and the reactor emptying those queues in a strictly round-robin fashion, *but* stopping immediately to service input data if it arrives. Hmm. I suppose I could implement that by subclassing the UDP transport: class queuedtransport: def write(self, data, host=None): if not host in self.host2queue: queue = max(self.host2queue.values()) + 1 self.host2queue[host] = queue else: queue = self.host2queue[host] self.queues.setdefault(queue, []).append(data) def queue(self): if not self.queues: return reactor.callLater(1, self.queue) while True: self.pos = (self.pos + 1) % len(self.queues) if not self.queues[self.pos]: continue self._write(self.queues[self.pos].pop(0)) break Or something - I'm fairly sure that code wouldn't quite work, but something similar. Does anyone have any suggestions how to handle this? My existing (home-grown, nasty) async framework can handle a lot of queries a second; upwards of several hundred, and it does a very good job of sharing out the load evenly so that fast systems complete quickly but the slower ones still get enough time to a) not suffer PDU timeouts and b) not exhaust inbound socket buffers. The way my current code works is: def receive(): r,w,x = select.select(fds,[],[]) for rfd in r: while True: try: pdu = r.read() if not pdu: break mark_answered(pdu) except: pass return len(r) else: return None while self.jobs: while True: if not receive(): break job = job.nextptr if job.waiting: if job.waiting.answered: job.receive() elif job.waiting.due < now: try: job.waiting.retry() except: # Out of retries job.cancel() You can probably see the way this works - on every loop, receive IO takes absolute precedence but goes into an input queue, and the input queue only has one "receive" event per IO-empty - *and* there's a pointer into the queue of jobs to ensure that they're serviced in a round-robin fashion if ready. Sadly, the pseudo-code above makes it look much cleaner than the real code, and I want the other protocol / service support along with other things like the deferred pattern and such. I would welcome any suggestions.

4 11

[Twisted-Python] problems with trial and adbapi
by donfu 04 Oct '04

04 Oct '04

Dear Twisters I'm diving into the twisted matrix. Mostly really enjoying it, but sometimes it's a tough ground :) Now i'm trying to set up unittests with trial for a small twisted datastore. the store itself works outside of trial, but strange things happen when running the tests. The first thing I had to learn, is to explicitly start the dbpool with dbpool.start(). Now the tests work sometimes, but sometimes not. I guess the problem has something to do with adbapi's usage of threads and my using of 'deferredResult()' in the tests. can someone help me here? thanks, donfu.

6 8

RE: [Twisted-Python] Problems using Twisted adbapi
by Simon 30 Sep '04

30 Sep '04

In response to Thomas Weholts question regarding writing a general database-abstraction layer for Twisted I thought I'd share this with you all. I have made a rewrite of the adbapi to use ADOdb for Python (http://adodb.sourceforge.net/#pydownload). It seems to be working fine but I haven't really tested it since I just finished it yesterday. Let me know of this is something that sounds useful to any of you. I'm also thinking of doing a rewrite of the sqlreflector and rowobject to support a more dynamic type checking and validation. Since I'm quite new in this community I don't really know what has been done before and not. So any input is appreciated. Cheers, Simon

2 1

[Twisted-Python] Seeking Twisted/Python Contractors
by Clark C. Evans 30 Sep '04

30 Sep '04

Hello. We have a core toolset written with Twisted (that will most likely be made open source) and have several applications of this toolset to the medical research field. I'm looking for someone to continue to build out the product by implementing customer requested features. In particular, we need several user interface improvements, as well as a whole slew of custom, one-off programming tasks. Depending on the particular task, some of the code could quite possibly be donated to the Twisted project. I'm looking for someone very familiar with Python, who has good track record building test-driven code, and who is responsive over the Internet. Experience with Python, HTTP and HTML is a must, Javascript a plus, and XSLT is a very nice bonus. This is a contract position, and need not be local; although if you are within 'visiting' distance to New Haven, CT (say NYC or Boston) that is icing. If you are interested, please send me information you consider relevant, your hourly rate expectations, and your availability; if you are not a 'visible' person in the Twisted community a few references would be quite helpful. Cheers! Clark -- Clark C. Evans Prometheus Research, LLC. http://www.prometheusresearch.com/ o office: +1.203.777.2550 ~/ , mobile: +1.203.444.0557 // (( Prometheus Research: Transforming Data Into Knowledge \\ , \/ - Research Exchange Database /\ - Survey & Assessment Technologies ` \ - Software Tools for Researchers ~ *

2 1

[Twisted-Python] Problems using Twisted adbapi
by Thomas Weholt 30 Sep '04

30 Sep '04

Hi, My code below is an attempt to write a general database-abstraction layer for Twisted. I got one allready supporting SQLite and MySQL ( MSSQL and PostgreSQL in the works ) but it blocks and doesn't use deffereds etc so I thought I'd try to write one that fits better with the Twisted framework. It will mainly be used in web-applications. The basic example is ( and don't laugh, this is my first attempt at using deferreds etc. ) : from twisted.enterprise import adbapi class TestConn: def __init__(self, dbname): self.dbname = dbname self.dbpool = adbapi.ConnectionPool("sqlite", db=dbname) def returnOk(self, o): return True def returnFailure(self, o): return False def returnResult(self, result): return result def _returnResult(self, deferred, count = None): if count: return self.dbpool.fetchmany(count) else: return self.dbpool.fetchall() def execSql(self, sql, params = {}): def run(sql, params): return self.dbpool.runQuery(sql, params) d = run(sql, params) d.addCallback(self._returnResult) d.addErrback(self.returnFailure) d.addCallback(self.returnResult) def fetch(self, sql, params = {}): def run(sql, params): return self.dbpool.runQuery(sql, params) d = run(sql, params) d.addCallback(self.returnResult) d.addErrback(self.returnFailure) s = TestConn(r'c:\temp\db1.db') print s.execSql('create table table1 (a int, b int)') print s.execSql('insert into table1 (a,b) values (1,1)') print s.execSql('insert into table1 (a,b) values (1,2)') print s.execSql('insert into table1 (a,b) values (2,2)') print s.fetch('select * from table1') print s.fetch('select a from table1') And the result is : None None None None None None Why ??? What I want is a general purpose "connection"-class which can execute queries and fetch data from a database, without having to create special methods returning deferreds, adding callbacks etc. to get the result of the query. -- Mvh/Best regards, Thomas Weholt http://www.weholt.org

2 1