Mailman 3 May 2009 - Twisted

Re: [Twisted-Python] RTFM about custom reactors
by andrewfr_ice＠yahoo.com 05 May '09

05 May '09

Message: 3 Date: Mon, 4 May 2009 02:15:25 -0400 From: Mohamed Lrhazi <lrhazi(a)gmail.com> Subject: [Twisted-Python] RTFM about custom reactors To: twisted-python(a)twistedmatrix.com Message-ID: <c4ab18df0905032315m4f0b3f85vf900adddd7557000(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Hi Mohamed: >I was wondering if someone could point me in the right direction.... >I am attempting to connect a proprietary network OS plugin system to >Python, to allow for writing traffic handling plugins in Python, and >was wondering what it would take to connect such a system into >Twisted, thus allowing the writing of normal twisted apps on top.... >Is it matter of learning and writing a custom reactor? My current >understanding is that the reactor is what detects and dispatches >events... but does it also dispatch the "data", as in the traffic >payload? what can I read to learn more? I strongly suspect what you want to implement is a custom protocol rather than a custom reactor. Look at that first. Based on past e-mail and examples, I believe the rule of thumb is one writes a custom reactor if you are porting to a new operating system/or taking advantage of some specific OS feature. There are exceptions like wxReactor that is to make Twisted interoperate with a windowing system but there functions that make the regular reactor interoperate with those systems. Cheers, Andrew

1 0

[Twisted-Python] addCallback0 convenience method
by Terry Jones 05 May '09

05 May '09

There are two minor niggles I frequently run into when writing code with deferreds and callbacks: 1. I very often write callbacks that ignore the incoming result: I tend to write this as def cb(_): ... 2. I sometimes write functions that I want to call both from within a callback chain and when not in a callback chain. Usually this is another case of writing def x(_, arg1, arg2) etc. and documenting that the extra unused arg is for when the function is called as part of a callback chain. This looks odder when the function is some kind of utility living in different Python source file, and is thus distant from its use in a callback chain. Seeing as n-1 of the add* methods of t.i.defer.Deferred are convenience methods for calling addCallbacks, I propose another: def addCallback0(self, callback, *args, **kw): self.addCallback(lambda result: callback(*args, **kw)) This allows you to write callback functions that will not be passed the result of the callback chain. That means I never have to deal with 1 and 2 above. They're minor niggles, of course, but they make code look less attractive and a little harder to understand. Yes, I could write my own standalone function def addCallback0(d, callback, *args, **kw): d.addCallback(lambda result: callback(*args, **kw)) and pass it my deferreds. And yes, I can also write my own wrapper to call other methods which don't take a callback result as their first argument. But all this would be nicer if addCallback0 was part of the Deferred class. I don't think we need a corresponding addBoth0 or addErrback0 etc., though. Terry

3 2

[Twisted-Python] RTFM about custom reactors
by Mohamed Lrhazi 04 May '09

04 May '09

Hello all, I was wondering if someone could point me in the right direction.... I am attempting to connect a proprietary network OS plugin system to Python, to allow for writing traffic handling plugins in Python, and was wondering what it would take to connect such a system into Twisted, thus allowing the writing of normal twisted apps on top.... Is it matter of learning and writing a custom reactor? My current understanding is that the reactor is what detects and dispatches events... but does it also dispatch the "data", as in the traffic payload? what can I read to learn more? Thanks a lot, Mohamed.

2 2

[Twisted-Python] sending out RAW IP packets using sockets
by Lars Reimann 04 May '09

04 May '09

hello, this question may be a bit off-topic, but nevertheless I am stuck atm. I'm trying to send out RAW IP packets (incl. full header) which i obtained previously using packet capture. It has to be RAW IP packets because I want the kernel to do ARP lookup stuff, etc for me . So an obtained packet should go through the normal lookup process. I've tried to use RAW sockets in python for this, but it constantly tells me, that the "Protocol (is) not supported" using this code: ------- self.rawserver = socket.socket(socket.AF_INET, socket.SOCK_RAW, self.proto) self.rawserver.bind((self.interface, self.proto)) ------- I would then simply 'send' the packets over that raw socket. I've tried to specify IPPROTO_RAW, and others for 'self.proto' but my python 2.5 won't accept this because they are unknown. My question is why that error occurs and how I can enable my program to send out RAW IP packets. Maybe the twisted framework could be helpful. As I am new to network programming I ask for your help. links to examples would also be greatly appreciated. greetings, l.r.

1 0

[Twisted-Python] design question
by Pet 04 May '09

04 May '09

Hi, I've written my first Twisted Server, which accept request from clients, does DB queries, sends response back and logs some events in DB. I wonder, if my design was good. Basically, I have a Factory where is db connectionPool created and Protocol(basic.LineReceiver), which processes requests and sends data back. In Protocol, I instantiate my utility object which I try to keep independent from twisted framework, so it can be easily used somewhere else or can be replaced by other class. def m(self, *args, **kwargs ): try: d = self.getData(params=kwargs['params']) if d: d.addCallback(self.sendResult) d.addErrback(log.err) d.addCallback(self.logFailedQuery) d.addErrback(log.err) return None except Exception, e: log.msg('Unknown error in m:', e) def getData(self, *args, **kwargs): obj = MyObj(self.db) return self.db.runInteraction(obj.getDataFromDB,kwargs['parameter']) What I don't like, I must pass db variable to obj.getDataFromDb and may be other function called there. I think it would be better obj = MyObj(self.db), but it doesn't work and runInteraction passes automatically db variable to interaction method. I'm new to Python and Twisted and would appreciate your advises. Pet

1 0

Re: [Twisted-Python] cx_Oracle, Twisted - rollback failed
by Steve Lewis 04 May '09

04 May '09

> Is there some way it can determine that the connection has been lost aside > from trying the rollback (and is compatible with other DB-API 2.0 modules)? > > Jean-Paul Oops, I actually did see my message going through now. Sorry for the duplicate. To answer your question, I was under the impression that the cp_reconnect would check to see if the connection existed already. Is there a way to have it check that query before rolling back? I'm guessing other databases/apis don't throw exceptions on a rollback if the connection is gone? Steve

2 1

[Twisted-Python] May 2nd Sprint
by Jean-Paul Calderone 04 May '09

04 May '09

Hello all, The sprint yesterday went quite well. He's a quick peek at what everyone was up to: * Drake Smith and Itamar Shtull-Trauring spent time fixing problems with our documentation and addressing a potential problem with the __str__ of DelayedCall * Dave Sturgis worked on adding frame-drop notification to LoopingCall * Thomas Hervé contributed remotely with code for the Conch "session" branch and tests for a regression fix in Conch's ssh agent support * Michael Hudson also participated remotely, continuing to work towards support for multiple byte ranges in HTTP range requests in the Twisted Web server * Glyph Lefkowitz finished the removal of a lot of decrepit gtk1-based application code, reviewed much of the other work done at the sprint, and triaged a bunch of tickets. * Chris Armstrong added an API to the new HTTP client currently under development to allow applications to interrupt it and disconnect at arbitrary times. He also paired with me to fix a memory leak triggered by trial's --until-failure option. * I also worked on some Conch fixes, reviewed some code, and triaged some tickets. Thanks to Divmod and the TSF sponsors who picked up lunch for all the sprinters. Jean-Paul

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 03 May '09

03 May '09

1 0

Re: [Twisted-Python] TCP Proxy scalability issue
by Mohapatra Saurav 02 May '09

02 May '09

Glyph, Thanks for the response. We're trying to proxy multiple VNC processes for a screen sharing application. So though CPU load is low, the data being pushed out is pretty high. We have around 16Kb of data being pushed out per second per client for a normal case. The test server is running on EC2, but we see the degradation in the production deployment (on physical machines running on a cogent fiber connection). I was thinking it might have something to do with amount of data being written and number of simultenous writes issued. Something to do with transport.write() I'll take a look at txloadbalancer. I can make a benchmarking application available to you, please mail me off the list at saurav.mohapatra(a)dimdim.com Regards Saurav -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of twisted-python-request(a)twistedmatrix.com Sent: Friday, May 01, 2009 12:00 PM To: twisted-python(a)twistedmatrix.com Subject: Twisted-Python Digest, Vol 62, Issue 2 Send Twisted-Python mailing list submissions to twisted-python(a)twistedmatrix.com To subscribe or unsubscribe via the World Wide Web, visit http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python or, via email, send a message with subject or body 'help' to twisted-python-request(a)twistedmatrix.com You can reach the person managing the list at twisted-python-owner(a)twistedmatrix.com When replying, please edit your Subject line so it is more specific than "Re: Contents of Twisted-Python digest..." Today's Topics: 1. Re: TCP Proxy scalability issue (glyph(a)divmod.com) ---------------------------------------------------------------------- Message: 1 Date: Fri, 01 May 2009 15:54:56 -0000 From: glyph(a)divmod.com Subject: Re: [Twisted-Python] TCP Proxy scalability issue To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <20090501155456.12555.1002021953.divmod.xquotient.10153(a)weber.divmod.com > Content-Type: text/plain; charset="utf-8"; format="flowed" On 02:02 pm, saurav.mohapatra(a)dimdim.com wrote: >Worker processes W1...n run listening on P1..n on the loopback and one >router process (twisted based) runs on public port P0 exposed to the >real world. I haven't used it myself, but that sounds a bit like txloadbalancer: https://launchpad.net/txloadbalancer Are you using that? >The clients connect to P0 and the first few bytes they send indicates >which worker process they wish to connect to. The Twisted protocol >implementation then creates a relay TCP connection to loopback worker >process port and after that forwards received data on P0 to the worker >port on loop back and sends back data received from worker port to the >external connection. >We're noticing significant degradation / starvation of the clients >under >load (around 25 concurrent connections are enough to simulate this). >We're running on Linux (CentOS 5.2) using python 2.5 and Twisted latest >source tarball using the epoll reactor and all settings are default. >Each connection sends back around 8-24kb data per second. While every application is a unique beast in terms of performance tuning, these numbers all sound surprisingly low to me for something as simple as a TCP proxy. Are you saturating your CPU? What is the load like on the box in question, both from the Twisted proxy and from the other processes? Can you provide a benchmark that we can run somewhere else, to demonstrate the issue you're having? >My question is are there any "production" settings I need to do >(threadpool etc.) to get the maximum out of twisted. Nothing that you described will make any use of threads - is there some reason you mention threadpool size? If you *are* using threads then perhaps you shouldn't be, and they're causing performance problems :) However, aside from reactor selection, Twisted is designed to have very few knobs to turn; it has one button for performance tuning and we push it before it leaves the factory. So if there's a performance problem, the issue is that we need to optimize something in Twisted, or you need to optimize something in your application. You may be able to tweak various linux kernel parameters to be able to improve things a bit, but if you're running into problems at 25 connections, it doesn't sound like you're running into kernel issues to me. ------------------------------ _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python End of Twisted-Python Digest, Vol 62, Issue 2 *********************************************

2 1

[Twisted-Python] TCP Proxy scalability issue
by Mohapatra Saurav 01 May '09

01 May '09

Hi, We're currently running a TCP proxy / port multiplexer written in Twisted to route to multiple worker server processes running on loopback ports. The architecture is something like this Worker processes W1...n run listening on P1..n on the loopback and one router process (twisted based) runs on public port P0 exposed to the real world. The clients connect to P0 and the first few bytes they send indicates which worker process they wish to connect to. The Twisted protocol implementation then creates a relay TCP connection to loopback worker process port and after that forwards received data on P0 to the worker port on loop back and sends back data received from worker port to the external connection. We're noticing significant degradation / starvation of the clients under load (around 25 concurrent connections are enough to simulate this). We're running on Linux (CentOS 5.2) using python 2.5 and Twisted latest source tarball using the epoll reactor and all settings are default. Each connection sends back around 8-24kb data per second. My question is are there any "production" settings I need to do (threadpool etc.) to get the maximum out of twisted. Regards Saurav Saurav Mohapatra Director of Technology ( 508.875.9501 | Mobile: 774.242.7422 * saurav(a)dimdim.com Skype: mohaps | AIM: mohaps Meet freely.

2 1