Mailman 3 September 2004 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz Aug. 5, 2021

Aug. 5, 2021

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |… [View More]

4 5

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool April 28, 2013

April 28, 2013

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def … [View More]

3 3

[Twisted-Python] twisted and threading
by twisted-10152003＠atlas.lcs.mit.edu Nov. 26, 2004

Nov. 26, 2004

I wrapped a c library using swig and was hoping to make use of it in a twisted server I am writing. The issue is that the c library fundamentally blocks waiting for input that may never come. Is there a way from twisted to deal with this cleanly within the framework? I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet. (I saw all the previous posts on this topic, but the … [View More]

8 10

[Twisted-Python] twisted.internet and IPv6
by Stefan Behnel Oct. 18, 2004

Oct. 18, 2004

Hi! I'm curious: how much work would it be to enable IPv6 support in twisted.internet? When I do --------------------------------------- cd twisted/internet fgrep INET *.py --------------------------------------- I get a number of references to IPv4 that make me believe that t.i.* is not quite IPv6 clean by itself: --------------------------------------- default.py: server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) default.py: client = socket.socket(socket.AF_INET, … [View More]

3 3

[Twisted-Python] new to twisted
by m. smadi Oct. 18, 2004

Oct. 18, 2004

hi; i was writing third party sip call controller when a friend suggested i use twisted. I think it will save me a lot of time, but i had trouble with the following when installing: cvs the pserial and pcrypto required package. the cvs server complians about a wrong password (any hints) i continued without these package and installed using setup.py. Where is a good place to start (the tutorial provided is not really very great) - any other suggestions? moe smadi

4 3

[Twisted-Python] Twisted scalability with hundreds of outbound (client) connections
by Phil Mayers Oct. 4, 2004

Oct. 4, 2004

I have an SNMP poller written in Python that I'd like to turn into a long-running server process. The obvious thing (it is obvious - I really need the support infrastructure) is to convert it to twisted, since it provides easy support to turn the process into a server as well as a client (e.g. XMLRPC query mapping onto an SNMP fetch; using DBAPI to update node status, all hooked in via deferred, etc.) However, I've done a couple of proof-of-concept ports and am having scalability … [View More]problems related to the way reactors work. I'm not entirely certain I've got the problem nailed, since the reactor sources are... complex... but I think what happens is this: The framework is heavily oriented towards getting IO out as soon as possible, and since there's no "scheduling" of receive events, protocol instances talking to "fast" clients eat all the CPU, and the socket buffer overflows since each datagramReceived triggers another write(). Possibly an example would help: class agent: def start(self): d = self.get_table('ifTable') d.addCallbacks(self.step2, self.die) def step2(self, ifTable): # Do some processing, then more data fetch d = self.get_table('ipAddrTable') d.addCallbacks(self.step3, self.die) def step3(self, ipAddrTable): # etc. etc. def do_timeouts(reactor): # get the list of sent PDUs now = time.time() for pdu in pdus: if pdu.due < now: pdu.deferred.errback(Timeout()) reactor.callLater(1, do_timeouts, reactor) import sys from twisted.internet import reactor for host in sys.argv[1:]: a = agent(host) reactor.callLater(0, a.start) reactor.callLater(1, do_timeouts, reactor) reactor.run() Now, I know there are scalability problems with callLater, but that's only being used for the startup because I can't think of a cleaner way of doing it - future activity is all triggered by deferreds. I'm handling the timeouts by running a once-per-second expiry which is sufficient for my needs. The *problem* is that the SNMP agents I'm talking to have an extremely wide range of performances, response times and quantity of data to retrieve, and the faster endpoints are consuming a disproportionate quantity of the CPU time; so much so in fact, that the UDP socket receive queue limit is being exceeded (I'm using a single UDP socket for all the clients, and breaking the responses back out based on SNMP message ID - this is to overcome FD limits). We're talking 750-1000 clients here BTW. What I really need is some way of making the reactor do a fair-queuing style of transmit/receive; I have some not-very-clear notion in my head of transport.write not actually doing the write, but just putting the output data onto a per-agent queue, and the reactor emptying those queues in a strictly round-robin fashion, *but* stopping immediately to service input data if it arrives. Hmm. I suppose I could implement that by subclassing the UDP transport: class queuedtransport: def write(self, data, host=None): if not host in self.host2queue: queue = max(self.host2queue.values()) + 1 self.host2queue[host] = queue else: queue = self.host2queue[host] self.queues.setdefault(queue, []).append(data) def queue(self): if not self.queues: return reactor.callLater(1, self.queue) while True: self.pos = (self.pos + 1) % len(self.queues) if not self.queues[self.pos]: continue self._write(self.queues[self.pos].pop(0)) break Or something - I'm fairly sure that code wouldn't quite work, but something similar. Does anyone have any suggestions how to handle this? My existing (home-grown, nasty) async framework can handle a lot of queries a second; upwards of several hundred, and it does a very good job of sharing out the load evenly so that fast systems complete quickly but the slower ones still get enough time to a) not suffer PDU timeouts and b) not exhaust inbound socket buffers. The way my current code works is: def receive(): r,w,x = select.select(fds,[],[]) for rfd in r: while True: try: pdu = r.read() if not pdu: break mark_answered(pdu) except: pass return len(r) else: return None while self.jobs: while True: if not receive(): break job = job.nextptr if job.waiting: if job.waiting.answered: job.receive() elif job.waiting.due < now: try: job.waiting.retry() except: # Out of retries job.cancel() You can probably see the way this works - on every loop, receive IO takes absolute precedence but goes into an input queue, and the input queue only has one "receive" event per IO-empty - *and* there's a pointer into the queue of jobs to ensure that they're serviced in a round-robin fashion if ready. Sadly, the pseudo-code above makes it look much cleaner than the real code, and I want the other protocol / service support along with other things like the deferred pattern and such. I would welcome any suggestions. [View Less]

4 11

[Twisted-Python] problems with trial and adbapi
by donfu Oct. 4, 2004

Oct. 4, 2004

Dear Twisters I'm diving into the twisted matrix. Mostly really enjoying it, but sometimes it's a tough ground :) Now i'm trying to set up unittests with trial for a small twisted datastore. the store itself works outside of trial, but strange things happen when running the tests. The first thing I had to learn, is to explicitly start the dbpool with dbpool.start(). Now the tests work sometimes, but sometimes not. I guess the problem has something to do with adbapi's usage of threads … [View More]

6 8

RE: [Twisted-Python] Problems using Twisted adbapi
by Simon Sept. 30, 2004

Sept. 30, 2004

In response to Thomas Weholts question regarding writing a general database-abstraction layer for Twisted I thought I'd share this with you all. I have made a rewrite of the adbapi to use ADOdb for Python (http://adodb.sourceforge.net/#pydownload). It seems to be working fine but I haven't really tested it since I just finished it yesterday. Let me know of this is something that sounds useful to any of you. I'm also thinking of doing a rewrite of the sqlreflector and rowobject to support a … [View More]

2 1

[Twisted-Python] Seeking Twisted/Python Contractors
by Clark C. Evans Sept. 30, 2004

Sept. 30, 2004

Hello. We have a core toolset written with Twisted (that will most likely be made open source) and have several applications of this toolset to the medical research field. I'm looking for someone to continue to build out the product by implementing customer requested features. In particular, we need several user interface improvements, as well as a whole slew of custom, one-off programming tasks. Depending on the particular task, some of the code could quite possibly be donated to the … [View More]

2 1

[Twisted-Python] Problems using Twisted adbapi
by Thomas Weholt Sept. 29, 2004

Sept. 29, 2004

Hi, My code below is an attempt to write a general database-abstraction layer for Twisted. I got one allready supporting SQLite and MySQL ( MSSQL and PostgreSQL in the works ) but it blocks and doesn't use deffereds etc so I thought I'd try to write one that fits better with the Twisted framework. It will mainly be used in web-applications. The basic example is ( and don't laugh, this is my first attempt at using deferreds etc. ) : from twisted.enterprise import adbapi class TestConn: … [View More]

2 1