Mailman 3 December 2008 - Twisted

[Twisted-Python] twisted eat the memory
by Muaaz Hussain 15 Dec '08

15 Dec '08

I wrote web application on twisted and my application read from ulogd daemon and push the output to the client browser using AJAX .also I have another application displaying ethernets traffic using AJAX. but after some minutes the twisted process in the server eat the memory and the OOM-killer of the linux will kill the twisted process. are there any solution to make the twisted release the memory . note: my ram is 256mb -- Muaaz Osman Hussain IT Researcher

4 3

[Twisted-Python] Re: Twisted FTP
by Adriano Marques 15 Dec '08

15 Dec '08

Hello Gergely, > I just started to learn twisted. I tried to look after a working FTP example > on the net for twisted, but unfortunately I could not really found any. This > one seemed to be the most completed, but it says "Unknown error code" when I > try to connect to it. I'm sorry, but I don't have the chance of looking it further right now, but I took a look at your code, and I say that instead of making FTPServerFactory inherit from FtpFactory, you opted to instantiate it inside a object inherited class. I never tried this approach, and that may be the root of your problem. I'm redirecting you e-mail to twisted mailing list, so people there can help you better. I advice you to subscribe to that mailing list to get help (if you aren't subscribed yet). You may also try the IRC channel for some help. Twisted folks are very responsive and willing to help. > In my opinion the code is fine. May I ask you to take a look on it please, > and give me any suggestions, if you can see, why it is not working? > > I would really appreciate your help. Kind Regards, -- Adriano Monteiro Marques http://adriano-marques.blogspot.com http://www.umitproject.org http://www.pythonbenelux.org "Don't stay in bed, unless you can make money in bed." - George Burns

1 0

[Twisted-Python] Twisted and MVC
by Christian Scharkus 15 Dec '08

15 Dec '08

Hi folks. I want to build a MVC with the Model based on twisted with a custom protocol. At the moment all my attemps terrible failed. So far the client build a valid ssl-connection to the server and send the lines he is supposed to send but don't recognise any incomming lines from the server. So I decided to build the model from scratch again. The controller inits the model-class (implementation of the custom chat-protocol) to be able to call methods of the instance and send messages to the server. So how do I connect the model to he server? In former implementations I were using the Clientfactory but this didn't work very well. Thanks for your help Christian Scharkus

2 2

[Twisted-Python] Non-blocking http client?
by Erik Wickstrom 14 Dec '08

14 Dec '08

Hi all, I have an application that is doing some web spidering. Right now I'm using urllib to retrieve the URLs, but it is painfully slow. I was wondering if it's feasible to swap out urllib with a twisted client that uses deferds so I can process urls in a more "parallel" fashion? I've done a bunch of Googleing, but I haven't come across anything that I can use as a drop in replacement. If you can point me in the right direction I'd really appreciate it! Thanks for your help! Erik

4 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 14 Dec '08

14 Dec '08

1 0

[Twisted-Python] twisted.mail.relaymanager.MXCalculator.getMX in Win32 got problem
by biziap biziap 13 Dec '08

13 Dec '08

The problem is that: the UDP listening ports increase rapidly and eventually come to the end: too many open file in selector. As many as the hosts you lookup, those UDP ports are still alive. It is very different from running this script in Linux (gentoo 2.6). In Linux, the listening UDP port keep only 1 instance no matter how many lookups . Try the script below: ------------------------------ from twisted.mail import relaymanager from twisted.internet import reactor,defer global MXC MXC = relaymanager.MXCalculator() def getMailExchange(host): global MXC def cbMX(mxRecord): print 'Got mxRecord: %s' % mxRecord.name return MXC.getMX(host).addCallback(cbMX) def run(): d = getMailExchange('gmail.com') def next(mxRecord): d = getMailExchange('yahoo.com') #d.addCallback(next) ## uncomment the line to increase your UDP listening ports as many as you want reactor.callWhenRunning(run) reactor.run() -------------------------------------------------- Any hints?

2 1

[Twisted-Python] PDF I thought may be of interest
by Michael 13 Dec '08

13 Dec '08

Hi, I came across this livejournal entry today: * http://four.livejournal.com/934085.html Which has this link: http://s3.amazonaws.com/four.livejournal/20081209/threads-hotos-2003.pdf .. which is a paper by Rob von Behren, Jeremy Condit and Eric Brewer. I personally raised an eyebrow when I saw that because Eric Brewer was the CTO of Inktomi, who back the had a very highly scaleable network system called Traffic Server amongst other things (I worked at Inktomi between 2000 & 2002, which is why I recognised the name). Traffic Server's architecture was in many respects reactor based, and in many respects it's architecture and twisted's architecture weren't *massively* different. OK, TS was written in C++, and twisted is python, but a reactor approach is still a reactor approach. Specifically I found this part particularly surprising: Abstract Event-based programming has been highly touted in recent years as the best way to write highly concurrent applications. Having worked on several of these systems, we now believe this approach to be a mistake. Specifically, we believe that threads can achieve all of the strengths of events, including support for high concurrency, low overhead, and a simple concurrency model. The paper appears to be 5 years old already, so the impact of this paper appears minimal, but I thought it may be of interest to others. (The reason it raised such a surprise for me is because it would be as surprising to me as it would be to read of any of the core twisted devs to suddenly come out in favour of a threading approach rather than non-threading!) Looking slightly below the surface of things, it appears to be advocating something more like stackless's tasklets, rather than OS level threads (since it makes reference near the end to Erlang's threading model). Either why, hopefully interesting :) Regards, Michael -- http://yeoldeclue.com/blog

4 3

[Twisted-Python] Re: PDF I thought may be of interest
by Andrew Francis 11 Dec '08

11 Dec '08

Hi Michael and Colleagues: > .. which is a paper by Rob von Behren, Jeremy Condit and > Eric Brewer. As a part of writing "Adventures in Stackless Python/Twisted Integration," I have read "The Why Events are" paper as well as the Ousterhout "Why Threads Are A Bad Idea (for most purposes). The "Why Events" paper led me to the Adya et al, "Cooperative task management without manual stack management" and the original Lauer and Needham "On the duality of operating system structures." I also visited the Schmitt papers on the Reactor pattern. >From what I recalled, I preferred the latter three papers because they provide a much better analytic framework for describing the issues. In particular, the Adya provides terminology. I have two basic sets of take-aways from the paper survey 1) "Events are great papers" state that threads are bad because of A,B, and C. "Threads are great papers" state that events are bad because of X,Y, and Z. As a programmer using these systems, perhaps I should pay special attention to issues A,B,C,X,Y and Z whenever applicable (and by the way papers thanks for telling me C and Y were issues since I didn't know). Perhaps out of all this, we can come up with appropriate guidelines for lockless programming? 2) The "Duality" paper argues that messaging passing and procedural systems can be described in terms of each other (bi-simulation?). I found this to be a very powerful way to look at Stackless Python and Twisted's threading models. A simple example. At the time I presented "Adventures," to avoid deadlock, I ran a reactor protocol instance in its own Stackless thread. This bugged me. It turned out that by replacing channels with deferreds (and doing a little buffering), I could replace the extraneous tasklet. My way of looking at it was, after all, protocol instances/deferreds are just another way of doing continuations... Use them, don't replace them.... > Looking slightly below the surface of things, it appears to > be advocating something more like stackless's tasklets, rather than > OS level threads (since it makes reference near the end to Erlang's >threading model). A quote from the paper: "To evaluate the ability of threads to support high concurrency, we designed and implemented a simple (5000 line) user-level cooperative threading package for Linux. Our thread package uses the coro coroutine library [15] for minimalist context switching, and it translates blocking I/O requests to asynchronous requests internally." > The paper appears to be 5 years old already, so the impact > of this paper appears minimal, but I thought it may be of interest to > others. (The reason it raised such a surprise for me is because it >would be as surprising to me as it would be to read of any of the core >twisted devs to suddenly come out in favour of a threading approach >rather than non-threading!) "The Why Events" is essentially the approach I took in regards to Stackless and Twisted. Right now, I am analyzing Stackless's scheduling model to see if improvements can be made. I feel I have much to learn... That said, I feel inline callbacks are a step in this direction. As Stackless/PyPy concepts get into mainstream Python, I can see Twisted following in this direction. Already one can see that with stuff like Christopher Armstrong's Corotwine. Again, I think the key to incorporate threads and keep to the spirit of Twisted, simplicity and safety, is to come up with some guidelines for lockless programming. Cheers, Andrew

1 0

[Twisted-Python] Running commands / collecting datas via SSH on many machines: Twisted vs plain old scripts ?
by Alessio Pace 11 Dec '08

11 Dec '08

Hi, I have to run commands and (later in time) collect logs from many machines, connecting through SSH. I was _genuinely_ wondering what an SSH client written in Twisted could give me more than executing plain old shell scripts in a for loop. >From what I saw, given the fact that my run/collect commands remain still shell commands (cp, tar, etc...), the only thing that may change is that using Twisted I could have a better utilization of resources rather than a simple shell for loop in which I iterate in order over all the machines on which I have to issue commands. Am I missing something? I would like to stress the fact that I'm asking this as a very specific question for Twisted usage in the context of running/retrieving results on remote machines (as indeed I already happily use Twisted for running custom network protocols) and I would right to choose a simple but effective solution for this case. Thanks in advance, Alessio Pace.

2 2

[Twisted-Python] twisted reverse proxy for web filtering
by dave fowler 11 Dec '08

11 Dec '08

I'm making a simple app that will run a local proxy and redirect me away from distracting websites. For example, if I go to facebook.com I'd like it to instead send me to google.com I'd like to use python and in my research I think a twisted reverse proxy is what I want. I've done the examples for setting up a proxy, including one that counts the words on the pages you visit, but i cannot find anything on fetching the destination and re-directing. I've been sifting through the many classes of the proxy examples and am pretty lost. Does anyone have some example code for something like this? Thanks Dave

2 1