Mailman 3 December 2003 - Twisted

Re: [Twisted-Python] Using Twisted in a network project
by Brian Warner Dec. 4, 2003

Dec. 4, 2003

"Hegedus, Matthew S" <matthew.s.hegedus(a)lmco.com> writes: > I would like to use Twisted if I could to write the code running on these > Custom Routers to achieve what I've described. Sounds perfectly reasonable. That's the same approach used by standard IPv4 multicast routers. Most of the multicast routing protocols are concerned with optimizing a tree of possible sources and members: your design simplifies that to a single source and a strict two-level hierarchy (level 1 are … [View More]the distant gateway router, level 2 is the set of remote endpoints), which should remove much of the complexity from those protocols. You'll need to implement a protocol from the end point to their local "router" (a host that can multicast to the end point and which is running your router program) which says "I want to join group X". That router then needs a protocol to the data source to say "somebody behind me wants to join group X, please sign me up". It will probably be easier to have the original data source do both the local multicast and the series of remote-router UDP unicasts. (I'm assuming you're writing the code on that end too). You can have a local-router too, but then you have to coordinate an additional host. Think of things in terms of trees of subscriptions. You probably have reasonable multicast support within your local subnet (most OSes ship with multicast-capable IP stacks), so use it (i.e. don't broadcast your local data packets, because then you'll waste the time and interrupts of all the local non-participating nodes). The data source should pick a multicast group address, everyone local should join that group (the IP stack and the ethernet hardware will take care of the details), the local router can join it too. You'll need some kind of service advertisement so the remote routers and remote nodes can know which groups they can join. > Question #3: If this idea is reasonable in theory, should I be worried about > Twisted/Python network performance? Depends upon how much data you're sending and what kind of connection you have to the outside world. Unless you've got a really nice pipe, it is likely that Twisted running on reasonable hardware will be able to saturate it. The local traffic is all multicast (one packet on the wire each), the remote traffic to N remote routers is unicast (N packets on the wire each), so the number of packets that your userspace code has to deal with is N+1 each, and your upstream bandwidth has to handle N of those. Once you've got more than two or three remote routers, N/(N+1) goes to 1 and Twisted doesn't have to run any faster than it takes to saturate your upsteam pipe. If you live on an OC-48 or something then you can set up multiple local routers and have them decide amongst themselves which one should be responsible for sending to each remote router. sounds like fun, -Brian [View Less]

1 0

[Twisted-Python] Twisted inclusion in other software
by RITA Y/O RODRIGO DIAZ Y/O BENENSON Dec. 4, 2003

Dec. 4, 2003

It looks like, but I need to be sure. Twisted is 99% pure python code. For each optimised C module there is a Python remplacement. So, it is absolutelly viable to include some Twisted modules directly into another app, eliminating the requirement of "please install www.twistedmatrix.com" ? For example: twisted.internet and twisted...woven ? Obviously product references will be explicited in the Readme. Can this be done ? Have it be done ? Except for Banana application, it is a solution … [View More]

3 2

[Twisted-Python] Using Twisted in a network project
by Hegedus, Matthew S Dec. 4, 2003

Dec. 4, 2003

Howdy all, Here's my problem: I'm in a situation where applications on a LAN need to be able to transmit UDP packets to other apps on the same LAN and on other LANS acroos the WAN. On the WAN there may be many routers in between that are not configured for multicast and cannot be reconfigured by users interested in multicasting. So multicast is out, and we are left with broadcast and point-to-point transmissions. For performance concerns, it's not viable to have each application do a point-to-… [View More]

2 1

[Twisted-Python] Generators as MicroThread and Twisted
by RITA Y/O RODRIGO DIAZ Y/O BENENSON Dec. 4, 2003

Dec. 4, 2003

Hi, Is there any relation between the generators microthread execution idea and the Twisted deffered/reactors schema ? http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html I'm observing how circle.org.au is reinventing the wheel. It does it very fine, but can be twisted used ? How does one idea relates to the other ? Pro/Cons ? Thanks. RodrigoB.

3 2

Re: [Twisted-Python] way of dealing with returning a deferred from a method/function
by Brian Warner Dec. 4, 2003

Dec. 4, 2003

> On Dec 3, 2003, at 8:48 PM, Jonathan Simms wrote: > >> This may seem obvious to everyone, but I had an insight into the use of >> deferreds today. It would always confuse me, I've got a method that's >> going to take quite a while to run. It occurred to me that the wording here ("quite a while to run") is significant, and perhaps we should take a closer look at the Deferred documentation because it might be fueling some of the confusion surrounding when and where … [View More]Deferreds will help you. bad language: The function takes a long time to run. good language: The function will not be complete for a long time. The operation will not finish right away. Functions which take a long time to run, like "for i in range(10**10):", are going to take many seconds of continuous CPU time to finish. To do something else while it is running means time-slicing (run it in a thread), or task-switching (break it into pieces and chain them with callLater so that other events can be processed in between the chunks), or a different form of task-switching (run the operation in a separate OS process and watch for the child to exit). These are all fundamentally the same technique, the only difference is who gets to decide how the CPU cycles are allocated and when the job-switches take place. A function which may not be complete for a while, like "waitForUserInput()", or "waitForWebResponse()", use a little CPU to set up the operation and then sits around doing nothing until somebody else does something (at least they do when implemented in the old-style synchronous single-tasking blocking-IO way). They aren't burning CPU in the meantime: they are waiting for someone else to burn CPU (the web server, the human thinking about which button they want to push), or for packets to reach their destination and back, or whatever. Because the operation is not consuming CPU time, and because you aren't really sure exactly when the operation is going to be complete, it feels silly to sit around and do nothing for all that time. This is where Deferreds are useful. They provide a standard idiom for "tell me when something happens" callbacks. The important distinction to make is that, in the Deferred case, the operation is completed by some *external* event (external to the function that creates the Deferred, that is). It could be network activity (in which case the event is when the select loop sees the readable socket and calls dataReceived()), it could be a callLater timer expiring, it could be some third piece of code responding to an externality and running your callback, allowing your Deferred to complete. If the completing activity were performed by the same code that generated (and then returned) the Deferred, you might as well just do the entire operation synchronously. The term "blocking" is frequently used in this context. A function that has to sort a million numbers will take a long time to run, but it is not usually described as blocking, because there is no way it could return sooner than that and still get its job done. A function that sends a packet and busy-waits until another packet comes back *is* said to be blocking. It's all a question of terminology :) -Brian [View Less]

1 0

[Twisted-Python] way of dealing with returning a deferred from a method/function
by Jonathan Simms Dec. 4, 2003

Dec. 4, 2003

This may seem obvious to everyone, but I had an insight into the use of deferreds today. It would always confuse me, I've got a method that's going to take quite a while to run. When something calls it, i want to immediately return a deferred. But how to accomplish this? returns (most frequently) come at the *end* of a function! I was thinking about it like this (hence my confusion): def aLongWait(stuff): # obviously incorrect d = defer.Deferred() return d result = … [View More]

5 5

[Twisted-Python] Please try - experimental wxPython reactor
by Itamar Shtull-Trauring Dec. 3, 2003

Dec. 3, 2003

Attached is an attempt at making a wxPython reactor, allowing easy integration of Twisted applications with wxPython. I'm hoping it solves the issues that the current wx support has. See docstring for usage. Please, if you have a wxPython/Twisted app, try this out and tell me if it works. -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python & Twisted consulting

3 2

[Twisted-Python] deferred
by vicky lupien Dec. 3, 2003

Dec. 3, 2003

I have a problem with deferred. I have a wmfactory that return the results of a sql query but sometimes it occurs that the result is an sql-exception error because it's an insert query and sometimes the data are already in the database. When this kind of error occurs, the page is not render and I got an error. Here's what I talking about: class MessagePage(pages.BasePage): templateDirectory = str(os.sep).join([pages.BasePage().templateDirectory]) templateFile = "message.html" … [View More]

4 3

[Twisted-Python] t.web.proxy ignores response body if no Content-Length
by Nicola Larosa Dec. 3, 2003

Dec. 3, 2003

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The connectionLost method of t.web.proxy.ProxyClient neglects to call its namesake in the t.protocols.http.HTTPClient superclass. The later thoughtfully calls self.handleResponseEnd() when the connection is lost; such behaviour is needed as a fallback, when a broken web server (the so called "Apache Coyote 1.0" inside Jakarta Tomcat 4.0) omits to send a "Content-Length" header (nor a "Transfer-Encoding" one) with a non-null (and non-chunked) … [View More]

1 0

[Twisted-Python] daemon thread support
by Martin Waite Dec. 3, 2003

Dec. 3, 2003

Hi, I have written a simple network listener that collects requests, stores these in a database, pulls responses from a database, and forwards these back over the network. This uses three threads: * main twisted reactor thread * database writer thread * database reader thread The database threads are created using reactor.callInThread(). Everything works fine, but I cannot get the application to close down cleanly - it always blocks waiting for the database threads to stop. Is there … [View More]

5 14