Mailman 3 October 2004 - Twisted

[Twisted-Python] Weekly Bug Summary
by James Y Knight 12 Oct '04

12 Oct '04

A little late this week, sorry. ;) Bug summary ______________________ Since 2004-10-03 20:16:57.250000: Bugs opened: 11 Bugs closed: 2 Total open bugs: 311 (+9) New / Reopened Bugs ______________________ [CORE] writeSequence totally broken (1/4 hour ago) http://www.twistedmatrix.com/bugs/issue745 opened by jknight SMTP Client upgrade and bug fixes (7 days ago) http://www.twistedmatrix.com/bugs/issue735 opened by bkirsch imap.py LOGINAuthenticator missing functionality (7 days ago) CLOSED http://www.twistedmatrix.com/bugs/issue736 opened by bkirsch [DOCUMENTATION, CORE] Add "why zope interfaces?" FAQ (3 days ago) http://www.twistedmatrix.com/bugs/issue744 opened by hypatia [CORE] t.p.usage uses passed arguments rather than real defaults when displaying help (3 days ago) http://www.twistedmatrix.com/bugs/issue743 opened by itamarst twistd --version failed on windows? I'm using 1.3.0 (3 days ago) http://www.twistedmatrix.com/bugs/issue742 opened by maoy Twisted logging howto (4 days ago) http://www.twistedmatrix.com/bugs/issue741 opened by exarkun twisted.cred.portal.IRealm docstring for parameters (5 days ago) http://www.twistedmatrix.com/bugs/issue740 opened by thomasvs roundup should send .patch files as text/plain (6 days ago) http://www.twistedmatrix.com/bugs/issue738 opened by jknight Minor code cleanups (6 days ago) http://www.twistedmatrix.com/bugs/issue737 opened by HelmutG Type enforcement for twisted.python.usage (5 days ago) http://www.twistedmatrix.com/bugs/issue739 opened by itamarst Bugs Closed ______________________ imap.py LOGINAuthenticator missing functionality (7 days ago) http://www.twistedmatrix.com/bugs/issue736 opened by bkirsch twisted/xish/domish.py bug breaks jabber protocol implementation (1 week ago) http://www.twistedmatrix.com/bugs/issue734 opened by doko

1 0

[Twisted-Python] Message Queue
by Stefan Majer 11 Oct '04

11 Oct '04

Hi, We are in the design phase for a new project. We want to write a new kind of a network monitoring applikation. This application is designed to spread over several monitoring agents across the network. Therefore we want to do the following: 1. a central scheduler puts new measurement actions into a queue 2. all executors are subscribed to this queue an execute the specified actions and report all back to a central repository. We are all relatively new to python (most of us have java background) and we are searching for a good framework to help us to get the job done. Is it possible to use twisted to achieve the above. Or has twisted another idea how to distribute jobs to other machines ? with kind regards -- Stefan Majer x-cellent technologies GmbH Rosenkavalierplatz 5 81925 Muenchen fon +49 (0)89 929 274 - 245 fax +49 (0)89 929 274 - 250 e-mail stefan(a)x-cellent.com http://www.x-cellent.com

3 2

[Twisted-Python] UDP and multiple access
by Sebastien Kirche 11 Oct '04

11 Oct '04

Hi, I am working on a little project to list the servers of the online game «Enemy Territory» and also to list/find a known player. I try to use twisted to simplify the network access. I am new both to twisted and python, but as i am also a programmer, learning is quite easy. The task consists in 2 times: 1) one UDP request to the master server at idsoftware that returns the list of the active (slaves) servers (currently around 2020 hosts) 2) one UDP request to *each* slave server to obtain characteristics (name, map that is played, ...) and the list of the players For the time i managed to perform the task 1) and i can obtain the list of the hosts:ports to ask for. If someone would like to peek an eye on my code, i uploaded it [1]. My problem is now to send around 2000+ non blocking requests and to take the answers as they come back. Data treatment should not be a problem, but i don't really see how to perform the mass request. I don't understand clearly the twisted terminology (protocols, factory, ...). I have seen in another script [2] a case where there is just one datagramReceived that checks on the host:port of the datagram to distinguish the answers, but that is kind of a library, and it only shows 2 requests for a test. I have no idea for the mass request. Would someone be kind to guide me a little about it ? For example by just giving me a snippet for a multiple UDP request with the hosts are in a list, and if the datagramReceived will suit the response treatment. I hope to be clear enough. TIA. Footnotes: [1] http://sebastien.kirche.free.fr/python_stuff/master_query.py (comments welcome) [2] http://sebastien.kirche.free.fr/python_stuff/twist_p2p.py (initially published in the twisted mailing-list) Sébastien Kirche

2 2

Re: [Twisted-Python] Message Queue
by Jeff Pitman 11 Oct '04

11 Oct '04

On Monday 11 October 2004 15:46, Stefan Majer wrote: > Is it possible to use twisted to achieve the above. Or has twisted > another idea how to distribute jobs to other machines ? Brian Warner's buildbot: http://buildbot.sourceforge.net/, can help stoke the fire of ideas. Maybe it won't map 1:1 to what you need, but it's got some good concepts. take care, -- -jeff

1 0

[Twisted-Python] More on docs
by Mary Gardiner 10 Oct '04

10 Oct '04

Sorry, I appreciate that I spend more time ranting about getting help writing docs than I do writing, but... again. Unfortunately the fastest way for me to write docs is to take rough drafts by someone else and edit them into a helpful style with examples and whatnot. Writing docs from scratch is fairly slow because I have to spend a lot of time with the code, far more than the developers would have to spend. And usually anything I say about design designs or high level rationale will subsequently need to be corrected. So, it would be fastest *for me* if developers wrote scratch drafts and I edited them and improved the examples etc. Which is great, but also unlikely to happen. I figure something that is more likely to happen is that developers will be explaining something to someone and accidently generate drafts as they go. For example: mailing list posts, IRC logs and blog entries. So, if you see or participate in a good "using something" discussion, particularly on IRC, please send me a copy and I'll do my best to edit/re-write it into a usable piece of documentation. As my part of the bargain, it really is time I went back through the archives and mined them for good posts. Ta, Mary

2 1

[Twisted-Python] Re: [Twisted-commits] r11937 - untested beginnings of distutils hacks for zpkgtools. also move packages to dist
by Itamar Shtull-Trauring 10 Oct '04

10 Oct '04

<3

1 0

Re: [Twisted-Python] Sending other things than strings in UDP packets
by Paul Campbell 09 Oct '04

09 Oct '04

> Message: 5 > Date: Thu, 7 Oct 2004 08:53:08 +1100 > From: Christopher Armstrong <radeex(a)gmail.com> > Subject: Re: [Twisted-Python] Sending other things than strings in UDP > packets > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: <60ed19d404100614531935658a(a)mail.gmail.com> > Content-Type: text/plain; charset=US-ASCII > Christopher Armstrong <radeex(a)gmail.com> wrote: > > Ergh. Please don't do anything _close_ to suggesting this. This is not > the "python way", it is the "stupid, insecure, > let-people-rm-rf-your-home-directory way". You apparently read part of the message and then you failed to read the next paragraph, right? At the risk of being redundant, let me reiterate: "Read the documentation on the pickle module for more information. And be forewarned: pickle will dump/load ANYTHING. For safety reasons, there's also a 'safe_pickle' variant floating around." I guess writing instructive and marginal (and probably nonworking) code for helping out newbies is not what this mailing list is all about. I thought I said "nonworking", "no error checking", and "lots of issues" enough times to get the point across. Sorry, I'll "mail.compose.elitist_mode=true" next time and give some flippant answer like "UDP sucks dude. Just use PB under TCP." At the risk of leaving you hanging with regards to whether such variants actually exist, below are pointers to two that can be used off the shelf, and possibly a third already contained within the twisted code base. Here's one variant that includes the a "safe pickle" call: http://zgp.org/pipermail/p2p-hackers/2004-August/002046.html Incidentally, the protocol contains a lot of the extras that I mentioned my stripped-down code was lacking (as well as a few small bugs). It also handles long messages and retries as well within UDP. It has an interesting "microprotocol" sort of structure (where each layer of the protocol builds on the previous one). Read it in addition to the first couple functions that handle pickling/unpickling. Another "safe pickle" module is buried in the code for "thecircle" (www.thecircle.org.au). Just download it and rip out the "safe_pickle.py" module from circlelib. It is stand-alone, and designed for UDP transportation (although not currently using Twisted). I haven't dug really deep, but banana (part of PB) appears to be essentially yet another incarnation of exactly the same idea. The code pattern looked identical to the two pieces of code I just mentioned. In fact, I haven't looked at it but I suspect that even pickle itself has the same pattern, other than being more generalized (it will handle executables and instances, while the safe variants will reject that). The code for all of these modules has an identical structure. It takes a structure and walks down it. It reads each piece and codes it in a "Type+data" format. It rejects anything that it can't inherently decode without aid (such as class instances). In those cases, at least the banana variant does allow the possibility to kick it up to a higher level (via Jelly) to handle user-level structures. The "unpickle" code does exactly the same thing except in reverse, converting the coded data back into a structure. There are standards (S-expressions or XML) for the format itself but I haven't seen any truly compelling reasons to follow those. They seem to add lots of overhead without any additional benefit.

2 1

[Twisted-Python] xml-rpc, single-instance... how does that work
by Guyon Morée 08 Oct '04

08 Oct '04

Hi all, I am trying to get my head around some things here. I have been looking at the twisted xml-rpc examples and I know you can get concurrency through deferreds. I like that :) The concept(which i dont get) is having a *single* instance of some class, which is published through xml-rpc. What happens when multiple 'users' try to do stuff like updating and retrieving. How does one do transactional processing? The way I see a user having a transaction is having it's own instance of some (business)object. For example: # semi pseudo code :) x = xmlserverobject() transaction = x.SomeTransActionalBusinessObject() # do stuff transaction.init(blablabla) # save the transaction if everything went correct to for example a database transactiob.save() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope it's clear what I am looking for.I dont see how you can do that through xml-rpc, but I might be just stupid :P I hope I missed something because the webservices/xmlrpc looks really appealing. regards, Guyon Morée

1 0

[Twisted-Python] Sending other things than strings in UDP packets
by Paul Campbell 08 Oct '04

08 Oct '04

Sending anything via any protocol is fairly easy with python in general. There are two ways to do it depending on your particular goals. The "python way" is as follows: message = pickle.dumps(my structures) my structures = pickle.loads(message) Read the documentation on the pickle module for more information. And be forewarned: pickle will dump/load ANYTHING. For safety reasons, there's also a "safe_pickle" variant floating around. The underlying banana protocol is really just another pickle module for use inside PB. I haven't tried using it separately from PB though. The other "unpythonese" way of doing it is using the struct module: message = struct.pack("format", param1, param2, param3) param1, param2, param3 = struct.unpack("format", message) The advantage of the struct library is that you have complete and total control over every byte so you can deal with non-python data. In reality, I personally use both. pickle is great for providing a totally unstructured interface to higher level protocols that have no desire in looking directly at low level packet formats. struct does just the opposite. For instance, here's a very simple UDP RPC module with no error checking whatsoever, no safety, no handling of very long packets (or checking for that case), no timeouts, or anything but a very basic rudimentary idea of how to assemble an RPC protocol. class UDPRPC(DatagramProtocol): REQUEST = 0 RESPONSE = 1 def start(self): """Create a dictionary of pending RPC's""" self.pending = dict() def call(self, address, remote_class, remote_method, *remote_args, **remote_kargs): """The actual RPC call local (stub) interface. Note that we can even get creative by overloading __calls__ and similar interfaces to make it look 100% like a real class.""" d = defer.Deferred() # Create a deferred return. # The following is an index to look up the RPC response later nonce = random.randomint(0, 2^32-1) index = struct.pack("!I!I!H", nonce, address[0], address[1]) self.pending[index] = d # Store the deferred for the response to follow self.sendMessage(self.REQUEST, address, (remote_class, remote_method, remote_args, remote_kargs)) return d def sendMessage(self, type, address, nonce, data): "Do the conversion to a packet and send it." packet = chr(type)+struct.pack("!I", nonce)+ pickle.dumps(data) transport.write(packet) return d def datagramReceived(self, packet, address): """Handle incoming packets.""" type = ord(packet[0]) nonce = struct.unpack("!I", packet[1:4]) message = pickle.loads(packet[5:]) if type == REQUEST: # Handling a remote call remote_class, remote_method, remote_args, remote_kargs = message result = remote_class.remote_method(*remote_args, **remote_kargs) self.sendMessage(self.RESPONSE, address, nonce, result) else: # Assume that this is the response, so return the deferred index = struct.pack("!I!I!H", nonce, address[0], address[1]) caller = self.pending[index] del self.pending[index] caller.Callback(message)

4 3

[Twisted-Python] epoll and other questions
by Andrea Arcangeli 08 Oct '04

08 Oct '04

Is there any plan to use epoll instead of poll to make twisted scalabile with hundred thousand simultanous sockets connected? I'm evaluating if to use twisted or C/C++ for my cpushare.com server side. I'd expect network to be the main bottleneck in the short term, and so I feel safe at spending extra cycles in userspace (at least in the short term), but even in the short it should at least handle hundred thousand tcp connections, and so I'm really scared about poll. I think poll would be the biggest showstopper, and that's why I'm asking to make twisted use epoll. I assume my application would require no change, so I can start developing with current twisted, I can test it with poll, and then later fix the internals when the slowdown becomes noticeable. Right? I understand there's no limitation on the number of sockets simultanously open, I just need to use ulimit to boost the limit of fds. A slightly separated issue: I assume it's best for me not to do any blocking I/O in the main network server handling the 100k connections and to create a secondary internal server communicating again through tcp/ip (loopback device) with the primary server to do the real blocking I/O. Is this correct? Best would be to use asynchronous I/O for the IO, but I think using a second process will be a lot simpler in practice since I don't need bulk I/O performance (I only need to avoid blocking). I only want to keep the network pipeline full even when some disk-read is happening. Best would be to use threading (or shared memory with MAP_SHARED in tmpfs), but it seems twisted is not mature enough for threading and shared memory communication using futex, right? If I would write it in C I could probably get various performance bits faster but I doubt the time spent on those bits would payoff significantly, opinions? Another thing I plan doing is to ship the public key (matching the private key stored only on the server) on the client source tarball, this way as far as people downloaded the right tarball, they will be able to securely connect to the server since they will be able to check the signature. Is there any example of this idea (public key stored in a file in the client package) available somewhere? thank you! PS. if somebody has some spare time and wants to have a look at the python _client_ code downloadable at www.cpushare.com/download.php and tell me if it's decent code and if I interfaced correctly between twisted and pyqt that's welcome. I'm at the point that I've to open up the connections from the QT event handlers etc.. I still have some unsolved issue on the QT side (like error windows instead of using sys.stderr and disabling one side of the tab when the other is running), but those are low priority at the moment, the GUI design was strightforward and the real big issues will start now in the network protocol implementation with twisted.

3 4