Mailman 3 August 2011 - Twisted

[Twisted-Python] Status of current proxy support (of various kinds)
by Damiano ALBANI Aug. 14, 2011

Aug. 14, 2011

Hello, I'm trying to determine the current status of proxy support in Twisted. I've googled quite a bit to find an answer but I couldn't get a crystal clear picture. What I mean by "proxy support" is the capability to: - act as a proxy server - use an external proxy server (i.e. act as a client) And by proxy, I'm talking in pratice of SOCKS v{4(a),5} and HTTP "CONNECT" standards. >From what I've read so far, it looks like Twisted is currently only capable of acting as a proxy server (correct me if I'm wrong). A quick search for "proxy" on Trac returns quite a lot of results so it's not easy to have a good grasp of the situation. Personnaly, what I'm interested in is the ability for Twisted to act as a proxy client and thus to connect to an external proxy server. I've seen bug reports on that topic, of which some are still open: Add a SOCKS client API http://twistedmatrix.com/trac/ticket/3508 Socks V5 functionality http://twistedmatrix.com/trac/ticket/1330 Add CONNECT support to HTTP client http://twistedmatrix.com/trac/ticket/4969 Regarding this last bug, I was wondering if there could a generic HTTP proxy mechanism, like with SOCKS, and not tied to the Web client. Because we might want to use an HTTP proxy for something else that Web traffic. What would be nice in Twisted is a generic, upper-protocol-agnostic proxy mechanism -- and one that would support indistinctly SOCKS and HTTP kinds of proxies. By the way, I'm totally new to Twisted, so forgive me if what I'm missing something obvious :-) Thanks, -- Damiano ALBANI

7 8

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 14, 2011

Aug. 14, 2011

1 0

[Twisted-Python] Flush socket
by Tobias Oberstein Aug. 12, 2011

Aug. 12, 2011

For testing implementations of a TCP-based protocol, I am testing if an implementation is agnostic wrt to in what chops in receives octets from the wire ("stream clean"). To do that, I force out octets from the protocol test driver (written in Twisted) in differently sized chops, down to single octets. Currently, I am using reactor.doSelect(0) after a transport.write() to force out outstanding octets to wire. I know this should not be done like this, but is there a clean and supported way to do that? def syncSocket(self): ## FIXME: find suitable replacement for this code, which appears to break ## sometimes .. ## ## From the web: "You should never call reactor.doSelect. This isn't portable across ## reactors, and it could easily break the reactor by re-entering it where it isn't ## expecting to be re-entered." ## try: reactor.doSelect(0) return True except: return False # socket has already gone away ..

5 14

[Twisted-Python] Flow control
by Tobias Oberstein Aug. 11, 2011

Aug. 11, 2011

For a test driver for a TCP-based protocol (written in Twisted) I want to send data to a receiver as fast as the receiver can consume data, but not faster. Apparently, a naive loop which just does transport.write() does not work. When I do reactor.select(0) between the writes, it somehow works, but memory consumption of sender is growing fast, since I guess data is buffered within Twisted, right? I then got over implementing my own application level flow control, where the receiver gives feedback of how much it has consumed, and the sender throttles itself to that. However, isn't there a simpler way (without application level flow control)? In the end, the sender TCP stack must have knowledge of it's window size .. is there a way to get at that info? What is the recommended way?

3 3

[Twisted-Python] A problem with very long blocking thread in Twisted
by Heungsub Lee Aug. 10, 2011

Aug. 10, 2011

Hi, fellow programmers. I have a problem with very long blocking thread(may not be finished) in Twisted. from socket import * > from twisted.internet import reactor > def consuming(): > while reactor.running: > amqp_consumer.wait() # waiting for very long time or forever > reactor.callInThread(consuming) The process follows the below timeline: >--: Started ---: Process --X: Terminated ^ : Keyboard Input Now main loop |>--------------------X thread | >--------------------------------------------... long time later ...---X ^ [Ctrl]+C Ideal main loop |>--------------------X thread | >-------------------X ^ [Ctrl]+C To solve this problem, I'd like to set threads as daemon. So that all related threads are terminated when main loop is interrupted. Any idea on this problem will be appreciated. ––––––––––––––––––––––––––– * Heungsub Lee sublee Mobile: +82-10-3215-2380 Site: http://subl.ee/ GitHub: http://github.com/sublee *

4 3

[Twisted-Python] ANN: MV3D 0.60 Released!
by Michael Handverger Aug. 8, 2011

Aug. 8, 2011

I am pleased to announce the release of MV3D version 0.60! This release focused on scalability of worlds and includes support for splitting a single area across multiple server processes with automatic load balancing and redundancy. Areas can now be connected together using gateways to build worlds limited in size only by the amount of available hardware. The Overseer Cluster Management tool was upgraded in order to better handle many processes across multiple physical servers. Camera controls across all content tools have been unified. Better support for Panda3D clients was added as well. MV3D is an open source virtual world and multiplayer game framework written in Python which uses Twisted for networking. It was designed with scalability in mind and is able to distribute a world across as many servers as needed while dynamically balancing the load. The simulation framework is not specifically slanted towards any one genre of online game, and can just as easily be used for a space game as a fantasy setting. Objects on an MV3D server are simulated using the ODE physics engine. A single server can host thousands of simulated objects. MV3D uses either Python-Ogre or Panda3D on the client. For more information on MV3D and this or future releases, please visit the website at http://www.mv3d.com . The full release notes for version 0.6 are available online at this URL: http://www.mv3d.com/trac/browser/tags/mv3d-0.60/release-notes.txt . For further inquiries, feel free to stop by our IRC channel on irc.freenode.net #MV3D. Here's a video of the new demo world: http://www.youtube.com/watch?v=iZiKr0t1TqU Enjoy, Mike

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 7, 2011

Aug. 7, 2011

1 0

[Twisted-Python] scaling with lots of clients
by Grant Mckenzie Aug. 4, 2011

Aug. 4, 2011

Hi, I have a twisted Perspective Broker based server. Recently with an increased number of clients we are seeing a decrease in throughput. Profiling the server it seems that it is spending the majority of time sending messages to the clients (unsuprisingly). Messages are sent to the client from the server using a callRemote, invoked from the event queue using reactor.callFromThread - nice and simple. What designs have people used to improve throughput in a heavily loaded server? My experience is telling me that I need to implement some sort of thread/process pool to divide the messaging to clients. Various documents on the twisted website strongly suggets that this is not a good idea. Thoughts? Cheers, G.

3 3

[Twisted-Python] Reconnection to Serialport after connectionLost
by Rene Klootwijk Aug. 1, 2011

Aug. 1, 2011

I have written several daemons that listen to the serialport. Everything works well, however the serial port connections are actually USB connections over IP (using an USB network hub). This means the connection is lost once in a while which is detected and triggers the connectionLost method of the protocol. I tried numerous things to reconnect to the serial port but without success. When I reconnect to the serial port I doubt what to do with the reactor parameter, when I specify the current reactor, I am able to reconnect to the serial port, the connectionMade method of the protocol is triggered, but it will never trigger dataReceived although I know data is arriving. Does any body have a glue how you can reconnect to the serial port within a running reactor loop? -- Rene

6 10