Mailman 3 October 2004 - Twisted

[Twisted-Python] Weekly Bug Summary
by James Y Knight Oct. 11, 2004

Oct. 11, 2004

A little late this week, sorry. ;) Bug summary ______________________ Since 2004-10-03 20:16:57.250000: Bugs opened: 11 Bugs closed: 2 Total open bugs: 311 (+9) New / Reopened Bugs ______________________ [CORE] writeSequence totally broken (1/4 hour ago) http://www.twistedmatrix.com/bugs/issue745 opened by jknight SMTP Client upgrade and bug fixes (7 days ago) http://www.twistedmatrix.com/bugs/issue735 opened by bkirsch imap.py LOGINAuthenticator missing … [View More]

1 0

[Twisted-Python] Message Queue
by Stefan Majer Oct. 11, 2004

Oct. 11, 2004

Hi, We are in the design phase for a new project. We want to write a new kind of a network monitoring applikation. This application is designed to spread over several monitoring agents across the network. Therefore we want to do the following: 1. a central scheduler puts new measurement actions into a queue 2. all executors are subscribed to this queue an execute the specified actions and report all back to a central repository. We are all relatively new to python (most of us have java … [View More]

3 2

[Twisted-Python] UDP and multiple access
by Sebastien Kirche Oct. 11, 2004

Oct. 11, 2004

Hi, I am working on a little project to list the servers of the online game «Enemy Territory» and also to list/find a known player. I try to use twisted to simplify the network access. I am new both to twisted and python, but as i am also a programmer, learning is quite easy. The task consists in 2 times: 1) one UDP request to the master server at idsoftware that returns the list of the active (slaves) servers (currently around 2020 hosts) 2) one UDP request to *each* slave server … [View More]

2 2

Re: [Twisted-Python] Message Queue
by Jeff Pitman Oct. 11, 2004

Oct. 11, 2004

On Monday 11 October 2004 15:46, Stefan Majer wrote: > Is it possible to use twisted to achieve the above. Or has twisted > another idea how to distribute jobs to other machines ? Brian Warner's buildbot: http://buildbot.sourceforge.net/, can help stoke the fire of ideas. Maybe it won't map 1:1 to what you need, but it's got some good concepts. take care, -- -jeff

1 0

[Twisted-Python] More on docs
by Mary Gardiner Oct. 9, 2004

Oct. 9, 2004

Sorry, I appreciate that I spend more time ranting about getting help writing docs than I do writing, but... again. Unfortunately the fastest way for me to write docs is to take rough drafts by someone else and edit them into a helpful style with examples and whatnot. Writing docs from scratch is fairly slow because I have to spend a lot of time with the code, far more than the developers would have to spend. And usually anything I say about design designs or high level rationale will … [View More]

2 1

[Twisted-Python] Re: [Twisted-commits] r11937 - untested beginnings of distutils hacks for zpkgtools. also move packages to dist
by Itamar Shtull-Trauring Oct. 9, 2004

Oct. 9, 2004

<3

1 0

Re: [Twisted-Python] Sending other things than strings in UDP packets
by Paul Campbell Oct. 8, 2004

Oct. 8, 2004

> Message: 5 > Date: Thu, 7 Oct 2004 08:53:08 +1100 > From: Christopher Armstrong <radeex(a)gmail.com> > Subject: Re: [Twisted-Python] Sending other things than strings in UDP > packets > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: <60ed19d404100614531935658a(a)mail.gmail.com> > Content-Type: text/plain; charset=US-ASCII > Christopher Armstrong <radeex(a)gmail.com> wrote: > > Ergh. Please don't do … [View More]anything _close_ to suggesting this. This is not > the "python way", it is the "stupid, insecure, > let-people-rm-rf-your-home-directory way". You apparently read part of the message and then you failed to read the next paragraph, right? At the risk of being redundant, let me reiterate: "Read the documentation on the pickle module for more information. And be forewarned: pickle will dump/load ANYTHING. For safety reasons, there's also a 'safe_pickle' variant floating around." I guess writing instructive and marginal (and probably nonworking) code for helping out newbies is not what this mailing list is all about. I thought I said "nonworking", "no error checking", and "lots of issues" enough times to get the point across. Sorry, I'll "mail.compose.elitist_mode=true" next time and give some flippant answer like "UDP sucks dude. Just use PB under TCP." At the risk of leaving you hanging with regards to whether such variants actually exist, below are pointers to two that can be used off the shelf, and possibly a third already contained within the twisted code base. Here's one variant that includes the a "safe pickle" call: http://zgp.org/pipermail/p2p-hackers/2004-August/002046.html Incidentally, the protocol contains a lot of the extras that I mentioned my stripped-down code was lacking (as well as a few small bugs). It also handles long messages and retries as well within UDP. It has an interesting "microprotocol" sort of structure (where each layer of the protocol builds on the previous one). Read it in addition to the first couple functions that handle pickling/unpickling. Another "safe pickle" module is buried in the code for "thecircle" (www.thecircle.org.au). Just download it and rip out the "safe_pickle.py" module from circlelib. It is stand-alone, and designed for UDP transportation (although not currently using Twisted). I haven't dug really deep, but banana (part of PB) appears to be essentially yet another incarnation of exactly the same idea. The code pattern looked identical to the two pieces of code I just mentioned. In fact, I haven't looked at it but I suspect that even pickle itself has the same pattern, other than being more generalized (it will handle executables and instances, while the safe variants will reject that). The code for all of these modules has an identical structure. It takes a structure and walks down it. It reads each piece and codes it in a "Type+data" format. It rejects anything that it can't inherently decode without aid (such as class instances). In those cases, at least the banana variant does allow the possibility to kick it up to a higher level (via Jelly) to handle user-level structures. The "unpickle" code does exactly the same thing except in reverse, converting the coded data back into a structure. There are standards (S-expressions or XML) for the format itself but I haven't seen any truly compelling reasons to follow those. They seem to add lots of overhead without any additional benefit. [View Less]

2 1

[Twisted-Python] xml-rpc, single-instance... how does that work
by Guyon Morée Oct. 8, 2004

Oct. 8, 2004

Hi all, I am trying to get my head around some things here. I have been looking at the twisted xml-rpc examples and I know you can get concurrency through deferreds. I like that :) The concept(which i dont get) is having a *single* instance of some class, which is published through xml-rpc. What happens when multiple 'users' try to do stuff like updating and retrieving. How does one do transactional processing? The way I see a user having a transaction is having it's own instance of some (… [View More]

1 0

[Twisted-Python] Sending other things than strings in UDP packets
by Paul Campbell Oct. 8, 2004

Oct. 8, 2004

Sending anything via any protocol is fairly easy with python in general. There are two ways to do it depending on your particular goals. The "python way" is as follows: message = pickle.dumps(my structures) my structures = pickle.loads(message) Read the documentation on the pickle module for more information. And be forewarned: pickle will dump/load ANYTHING. For safety reasons, there's also a "safe_pickle" variant floating around. The underlying banana protocol is really just another … [View More]pickle module for use inside PB. I haven't tried using it separately from PB though. The other "unpythonese" way of doing it is using the struct module: message = struct.pack("format", param1, param2, param3) param1, param2, param3 = struct.unpack("format", message) The advantage of the struct library is that you have complete and total control over every byte so you can deal with non-python data. In reality, I personally use both. pickle is great for providing a totally unstructured interface to higher level protocols that have no desire in looking directly at low level packet formats. struct does just the opposite. For instance, here's a very simple UDP RPC module with no error checking whatsoever, no safety, no handling of very long packets (or checking for that case), no timeouts, or anything but a very basic rudimentary idea of how to assemble an RPC protocol. class UDPRPC(DatagramProtocol): REQUEST = 0 RESPONSE = 1 def start(self): """Create a dictionary of pending RPC's""" self.pending = dict() def call(self, address, remote_class, remote_method, *remote_args, **remote_kargs): """The actual RPC call local (stub) interface. Note that we can even get creative by overloading __calls__ and similar interfaces to make it look 100% like a real class.""" d = defer.Deferred() # Create a deferred return. # The following is an index to look up the RPC response later nonce = random.randomint(0, 2^32-1) index = struct.pack("!I!I!H", nonce, address[0], address[1]) self.pending[index] = d # Store the deferred for the response to follow self.sendMessage(self.REQUEST, address, (remote_class, remote_method, remote_args, remote_kargs)) return d def sendMessage(self, type, address, nonce, data): "Do the conversion to a packet and send it." packet = chr(type)+struct.pack("!I", nonce)+ pickle.dumps(data) transport.write(packet) return d def datagramReceived(self, packet, address): """Handle incoming packets.""" type = ord(packet[0]) nonce = struct.unpack("!I", packet[1:4]) message = pickle.loads(packet[5:]) if type == REQUEST: # Handling a remote call remote_class, remote_method, remote_args, remote_kargs = message result = remote_class.remote_method(*remote_args, **remote_kargs) self.sendMessage(self.RESPONSE, address, nonce, result) else: # Assume that this is the response, so return the deferred index = struct.pack("!I!I!H", nonce, address[0], address[1]) caller = self.pending[index] del self.pending[index] caller.Callback(message) [View Less]

4 3

[Twisted-Python] epoll and other questions
by Andrea Arcangeli Oct. 7, 2004

Oct. 7, 2004

Is there any plan to use epoll instead of poll to make twisted scalabile with hundred thousand simultanous sockets connected? I'm evaluating if to use twisted or C/C++ for my cpushare.com server side. I'd expect network to be the main bottleneck in the short term, and so I feel safe at spending extra cycles in userspace (at least in the short term), but even in the short it should at least handle hundred thousand tcp connections, and so I'm really scared about poll. I think poll would be the … [View More]biggest showstopper, and that's why I'm asking to make twisted use epoll. I assume my application would require no change, so I can start developing with current twisted, I can test it with poll, and then later fix the internals when the slowdown becomes noticeable. Right? I understand there's no limitation on the number of sockets simultanously open, I just need to use ulimit to boost the limit of fds. A slightly separated issue: I assume it's best for me not to do any blocking I/O in the main network server handling the 100k connections and to create a secondary internal server communicating again through tcp/ip (loopback device) with the primary server to do the real blocking I/O. Is this correct? Best would be to use asynchronous I/O for the IO, but I think using a second process will be a lot simpler in practice since I don't need bulk I/O performance (I only need to avoid blocking). I only want to keep the network pipeline full even when some disk-read is happening. Best would be to use threading (or shared memory with MAP_SHARED in tmpfs), but it seems twisted is not mature enough for threading and shared memory communication using futex, right? If I would write it in C I could probably get various performance bits faster but I doubt the time spent on those bits would payoff significantly, opinions? Another thing I plan doing is to ship the public key (matching the private key stored only on the server) on the client source tarball, this way as far as people downloaded the right tarball, they will be able to securely connect to the server since they will be able to check the signature. Is there any example of this idea (public key stored in a file in the client package) available somewhere? thank you! PS. if somebody has some spare time and wants to have a look at the python _client_ code downloadable at www.cpushare.com/download.php and tell me if it's decent code and if I interfaced correctly between twisted and pyqt that's welcome. I'm at the point that I've to open up the connections from the QT event handlers etc.. I still have some unsolved issue on the QT side (like error windows instead of using sys.stderr and disabling one side of the tab when the other is running), but those are low priority at the moment, the GUI design was strightforward and the real big issues will start now in the network protocol implementation with twisted. [View Less]

3 4

[Twisted-Python] Using wxreactor on Windows
by Joshua Ritter Oct. 6, 2004

Oct. 6, 2004

On Windows, I was having a problem with my wxreactor application intermittently freezing on initialization. It was also taking a long time to exit. wxdemo.py doesn't exhibit the problem at startup, most likely due to the simplicity of the GUI. The demo does however exhibit the exit problem. What I did to alleviate both problems was to move the timer creation and starting code: class MoMWorldServer(wx.App): def OnInit(self): #SNIP wx.EVT_IDLE(self,self.OnFirstIdle) … [View More]

1 0

[Twisted-Python] Sending other things than strings in UDP packets
by Ibrahim Mubarak Oct. 6, 2004

Oct. 6, 2004

Hi, I am working on a UDP server. I can send and recieve packets with their data in strings with no problems. However, I was wondering if there is a way to populate the data portion of the packet with data types other than strings. And to be able to stick a number of values, might be of different types, together in the same packet. I looked at PB. It is not what I need. I need a low level control on what I add to the packets I send and how I can read packets I recieve. Thanks, ib … [View More]

2 1

[Twisted-Python] assignKeyAttr in row object
by Tsai Li Ming Oct. 5, 2004

Oct. 5, 2004

Dear all, What is the assignKeyAttr method used for in a row object? Is it similar to a primary key that can be used to identity the instance? Liming

3 4

[Twisted-Python] About adodb
by Simon Oct. 5, 2004

Oct. 5, 2004

Dave, thanks for your reply. >I'm not too familiar with ADOdb, what are the benefits? Quoting the author: "You might ask why Python needs a database abstraction library when Python provides the official DB API. Unfortunately the DB API does not encapsulate differences in the database implementations. For example, to select a limited number of rows, say 10 rows, you would have to use very different SQL for different databases. [...] These differences are handled by ADOdb (using SelectLimit), … [View More]

2 3

[Twisted-Python] Twisted scalability with hundreds of outbound (client) connections
by Phil Mayers Oct. 4, 2004

Oct. 4, 2004

I have an SNMP poller written in Python that I'd like to turn into a long-running server process. The obvious thing (it is obvious - I really need the support infrastructure) is to convert it to twisted, since it provides easy support to turn the process into a server as well as a client (e.g. XMLRPC query mapping onto an SNMP fetch; using DBAPI to update node status, all hooked in via deferred, etc.) However, I've done a couple of proof-of-concept ports and am having scalability … [View More]problems related to the way reactors work. I'm not entirely certain I've got the problem nailed, since the reactor sources are... complex... but I think what happens is this: The framework is heavily oriented towards getting IO out as soon as possible, and since there's no "scheduling" of receive events, protocol instances talking to "fast" clients eat all the CPU, and the socket buffer overflows since each datagramReceived triggers another write(). Possibly an example would help: class agent: def start(self): d = self.get_table('ifTable') d.addCallbacks(self.step2, self.die) def step2(self, ifTable): # Do some processing, then more data fetch d = self.get_table('ipAddrTable') d.addCallbacks(self.step3, self.die) def step3(self, ipAddrTable): # etc. etc. def do_timeouts(reactor): # get the list of sent PDUs now = time.time() for pdu in pdus: if pdu.due < now: pdu.deferred.errback(Timeout()) reactor.callLater(1, do_timeouts, reactor) import sys from twisted.internet import reactor for host in sys.argv[1:]: a = agent(host) reactor.callLater(0, a.start) reactor.callLater(1, do_timeouts, reactor) reactor.run() Now, I know there are scalability problems with callLater, but that's only being used for the startup because I can't think of a cleaner way of doing it - future activity is all triggered by deferreds. I'm handling the timeouts by running a once-per-second expiry which is sufficient for my needs. The *problem* is that the SNMP agents I'm talking to have an extremely wide range of performances, response times and quantity of data to retrieve, and the faster endpoints are consuming a disproportionate quantity of the CPU time; so much so in fact, that the UDP socket receive queue limit is being exceeded (I'm using a single UDP socket for all the clients, and breaking the responses back out based on SNMP message ID - this is to overcome FD limits). We're talking 750-1000 clients here BTW. What I really need is some way of making the reactor do a fair-queuing style of transmit/receive; I have some not-very-clear notion in my head of transport.write not actually doing the write, but just putting the output data onto a per-agent queue, and the reactor emptying those queues in a strictly round-robin fashion, *but* stopping immediately to service input data if it arrives. Hmm. I suppose I could implement that by subclassing the UDP transport: class queuedtransport: def write(self, data, host=None): if not host in self.host2queue: queue = max(self.host2queue.values()) + 1 self.host2queue[host] = queue else: queue = self.host2queue[host] self.queues.setdefault(queue, []).append(data) def queue(self): if not self.queues: return reactor.callLater(1, self.queue) while True: self.pos = (self.pos + 1) % len(self.queues) if not self.queues[self.pos]: continue self._write(self.queues[self.pos].pop(0)) break Or something - I'm fairly sure that code wouldn't quite work, but something similar. Does anyone have any suggestions how to handle this? My existing (home-grown, nasty) async framework can handle a lot of queries a second; upwards of several hundred, and it does a very good job of sharing out the load evenly so that fast systems complete quickly but the slower ones still get enough time to a) not suffer PDU timeouts and b) not exhaust inbound socket buffers. The way my current code works is: def receive(): r,w,x = select.select(fds,[],[]) for rfd in r: while True: try: pdu = r.read() if not pdu: break mark_answered(pdu) except: pass return len(r) else: return None while self.jobs: while True: if not receive(): break job = job.nextptr if job.waiting: if job.waiting.answered: job.receive() elif job.waiting.due < now: try: job.waiting.retry() except: # Out of retries job.cancel() You can probably see the way this works - on every loop, receive IO takes absolute precedence but goes into an input queue, and the input queue only has one "receive" event per IO-empty - *and* there's a pointer into the queue of jobs to ensure that they're serviced in a round-robin fashion if ready. Sadly, the pseudo-code above makes it look much cleaner than the real code, and I want the other protocol / service support along with other things like the deferred pattern and such. I would welcome any suggestions. [View Less]

4 11

[Twisted-Python] problems with trial and adbapi
by donfu Oct. 4, 2004

Oct. 4, 2004

Dear Twisters I'm diving into the twisted matrix. Mostly really enjoying it, but sometimes it's a tough ground :) Now i'm trying to set up unittests with trial for a small twisted datastore. the store itself works outside of trial, but strange things happen when running the tests. The first thing I had to learn, is to explicitly start the dbpool with dbpool.start(). Now the tests work sometimes, but sometimes not. I guess the problem has something to do with adbapi's usage of threads … [View More]

6 8

[Twisted-Python] Proposed change to buffering
by Itamar Shtull-Trauring Oct. 4, 2004

Oct. 4, 2004

Now that we don't do write-at-once, lists are easier to use. Benefits: Allows protocols with many small writes to not have to do their own buffering (e.g. http, imap4 as currently coded, PB, IRC, telnet, etc. etc.) Problems: Bit more overhead for large writes. There's also a bit of code change in tcp.py for the SSL code. Index: abstract.py =================================================================== --- abstract.py (revision 11908) +++ abstract.py (working copy) @@ -43,7 +43,9 @@ … [View More]

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight Oct. 3, 2004

Oct. 3, 2004

Bug summary ______________________ Since 2004-09-25 17:50:20.840000: Bugs opened: 3 Bugs closed: 12 Total open bugs: 302 (-9) New / Reopened Bugs ______________________ twisted/xish/domish.py bug breaks jabber protocol implementation (12 hours ago) http://www.twistedmatrix.com/bugs/issue734 opened by doko twisted's SIGCHLD handler breaks popen. (3 days ago) http://www.twistedmatrix.com/bugs/issue733 opened by jknight test_journal.Service defines addtime twice in the … [View More]

1 0

[Twisted-Python] PATCH: twisted.web.client Basic/Digest Authentication
by Mathieu Fenniak Oct. 1, 2004

Oct. 1, 2004

The attached patch adds support for HTTP authentication to twisted.web.client. It supports both basic and digest authentication. This patch adds a new module, twisted.web.auth, which I hope to expand in the future to support digest server-side authentication. A new class is also added to twisted.web.error, which passes on the WWW-Authenticate headers such that authentication could be handled externally. I have never contributed before to Twisted, so I'd appreciate feedback on this … [View More]

2 1

[Twisted-Python] ANN: Fusion 0.1, C++ / Twisted integration
by Itamar Shtull-Trauring Oct. 1, 2004

Oct. 1, 2004

This allows you to write protocols for UDP, multicast and TCP in C++. In theory this can make the protocol much faster since it's easier to prevent data copying etc.. The TCP writing support is not efficient yet, though, as it still uses Twisted's current buffering system. Tested with 1.3.0 and 2.0 from trunk. No API stability guaranteed. http://itamarst.org/software/fusion-0.1.tar.gz or sandbox/itamar/cppreactor in Twisted SVN. Disclaimer: if you want to make your Twisted code go faster … [View More]

1 1