Mailman 3 August 2001 - Twisted

[Twisted-Python] TWN #6, Text version!
by Chris Armstrong Aug. 15, 2001

Aug. 15, 2001

(Someone pointed out that lynx -dump produces nice text output with references. So here's the text version of TWN #6!) The Twisted Weekly News 6th Issue. Wednesday, August 15, 2001 QOTW: <Inhibitor> this is commercial software - there are no security holes <Inhibitor> not like your crappy open source - written by students - stuff <glyph> right, I had forgotten Welcome to the 6th issue of the Twisted Weekly News! Last week I mentioned how much Twisted had grown in developers and users -- well, obviously I'm not a psychic. This week has seen a much larger growth-spurt. Rock on, Twisted! Progress Here's what's been happening this week in the code base. * Twisted 0.9.5: We made another [1]release. This one was mainly bug fixes, code cleanup, and documentation. The cool thing about this release is that we put [2]publicly available Twisted Words and Twisted Reality servers up on twistedmatrix.com. Get ready for a new release, though, Twisted 0.9.6 is planned to be out soon! * Twisted Manhole: Chris Armstrong got sick and tired of the telnet interface to Twisted, so he wrote a nice PB service as a replacement for it, with cooler features. The GTK+ interface supports coloured output, command history, and editing of full blocks of code before executing them. Glyph also helped out a lot when he cleaned up Chris' mess of code, and made the GUI much prettier. ;-) * Docs: Well, people have been responding to repeated requests for documentation. Acapnotic has been on a rampage through docstrings, and moshez wrote a [3]page on writing low-level protocol implementations for Twisted. * Mail: Moshez is still working hard on twisted.mail - the latest checkin comments show that he has implemented relaying successfully! (So when is twistedmatrix.com going to switch to twisted.mail? ;-)) * Twisted.Words: There's a lot of interest in this little package lately: the IRC interface is becoming more robust, and the InstanceMessenger client has been getting a lot better (there is now a Tk version, too). * utilities: Moshez hacked a little on the utility scripts included with Twisted -- he made a neato utility for generating policy-conformant Debian packages, added a --python parameter to twistd for reading from plain python files instead of .taps, and added --append to mktap, which adds another server to an existing twisted application. from tap files. * Twisted TOC: z3p's been hacking more on Twisted TOC. He made a client for it, but unfortunately had to pull it due to a nasty copyright on some widget library he was using. Instead of creating a separate client, z3p now plans on adding TOC support to Instance Messenger (right, z3p? ;-)) * Twisted SSL: Itamar implemented a really cool SSL layer for Twisted. All that is required to use it is to basically change one line of code in the typical Twisted server. This will be really cool for manhole, so passwords and code will be able to be encrypted. Thanks a lot Itamar, we've had lots of requests for SSL support. :-) Other Stuff Here are summaries of some of the more significant mailing list threads. Moshe [4]suggested refactoring the daemonize() function into a separate module, only to start somewhat of a holy war with glyph which later moved to IRC. I can't really explain why glyph was against it myself, but I know it had something to do with philosophy. ;-) Kevin Turner made a very confusing [5]attempt at proposing a new "Multiplexing" feature for Perspective Broker. The gist of it was that there can be an object shared between many hosts, and whenever the object is updated by anyone, the changes propagate to everyone else. The discussion must've been half on IRC, though, because I couldn't really make out what the conclusion was. It seems that what Acapnotic wants is in the Cached class.. Itamar had [6]Some suggestions for Twisted after he read a lot of the source while he was on a plane trip. He thought that 0) Twisted was cool, 1) Pickling on server shutdown should be optional (he has servers which aren't pickleable), 2) number of clients which socket.listen() queues shouldn't be hard-coded to 5, 3) threadable servers should be faster, and 4) more docstrings should be put in. Moshe Zadka replied that Most of these things had been solved or are being worked on. The thread quickly changed topic to one about [7]threading, and Itamar [8]implementing some more threading capabilities for Twisted. Cool Thing Of the Week: Twisted Enterprise Twisted Enterprise is the new database interface for Twisted, written by Sean Riley. This was an especially interesting problem, because most (all?) database servers are blocking beasts, so servers talking to it must be multi-threaded. The Twisted solution was to have a separate multi-threaded Twisted process which talks to the database in the background and listens and answers for requests from other (thread or not) Twisted processes. This way you do not have to deal with threads in your own application if you don't want to, even if you want it to talk to a database. Enterprise is going to be a big thing in Twisted -- Glyph is working on a great authentication system that uses this new package. Get ready for enterprise b2b internet solution!! Plea for help from the pathetic editor I've been very busy lately, and I haven't even started school yet. So, I'll probably be even more busy very soon -- If anyone is willing to help me out with TWN on a weekly basis (maybe only half an hour every week) It would be *incredibly* helpful. The two areas where I need it the most are the mailing list summaries and the conversion to text format for the mailing list (if you know of some software that will do this and nicely generate references for links at the bottom, that'd help *tons*). But even if you can just summarise a new feature in Twisted, it'd still help. Thanks! Ok, that's it for this week. Thanks for reading! [9]Chris Armstrong References 1. http://www.twistedmatrix.com/downloads/ 2. http://www.twistedmatrix.com/users/pb.twistd 3. http://www.twistedmatrix.com/page.epy/docs/writing-serves.html 4. http://www.twistedmatrix.com/pipermail/twisted-python/2001-August/000199.ht… 5. http://www.twistedmatrix.com/pipermail/twisted-python/2001-August/000203.ht… 6. http://www.twistedmatrix.com/pipermail/twisted-python/2001-August/000228.ht… 7. http://www.twistedmatrix.com/pipermail/twisted-python/2001-August/000240.ht… 8. http://www.twistedmatrix.com/pipermail/twisted-python/2001-August/000231.ht… 9. mailto:carmstro@twistedmatrix.com

1 0

[Twisted-Python] TWN #6
by Chris Armstrong Aug. 15, 2001

Aug. 15, 2001

The new TWN is at http://twistedmatrix.com/page.epy/twn-6.html I'm terribly sorry for not making a text version this week, but I've been busy. I'm starting school soon, so I doubt I'm going to get less busy. Please see my "plea for help" in this weeks TWN. I'm going to go try to write some PB docs now.

1 0

[Twisted-Python] Patch to internet.main
by Itamar Aug. 13, 2001

Aug. 13, 2001

This makes the threaded scheduler work, if we are running in threaded mode: Index: twisted/internet/main.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/main.py,v retrieving revision 1.13 diff -c -r1.13 main.py *** twisted/internet/main.py 2001/08/10 06:38:06 1.13 --- twisted/internet/main.py 2001/08/13 18:13:00 *************** *** 22,28 **** # Sibling Imports ! import task, tcp class Application(log.Logger): running = 0 --- 22,28 ---- # Sibling Imports ! import task, threadtask, tcp class Application(log.Logger): running = 0 *************** *** 135,140 **** --- 135,142 ---- writes = {} running = None delayeds = [task.theScheduler] + if threadable.threaded: + delayeds.append(threadtask.theScheduler) shutdowns = [] def shutDown(a=None, b=None):

1 0

[Twisted-Python] Some suggestions
by Itamar Aug. 13, 2001

Aug. 13, 2001

Hi all, I spent some time on my 11 hour flight to the US porting my servers to twisted, and here's what I've come up with so far: 0) Twisted is very cool. 1) Pickle on save should be optional. One of my objects is unpicklable, so every time I shutdown I get an exception. You could say I should rewrite, but I feel developing a server should not have to support all kinds of extra features that twisted provides. Obviously it'd be nice to have them, but it's one more thing you have to learn up front that isn't really necessary. 2) Why is the number of clients that socket.listen() queues hard-coded to 5? this should be an option. 3) If writing to a connection and closing a connection are thread safe then writing threaded servers is trivial. I'll try to write some patches that don't slow down the default non-threaded code tommorow. Again, rewriting my code from scratch as event driven is a waste of a time - I might do it someday, but I want working code now. 4) I'll submit a working twisted.internet.ssl after I do (3). 5) More docstrings! And if this an HTML email it's Mozilla's fault, not mine.

5 10

[Twisted-Python] Setting Up Us The Bomb^H^H^H^HMail server
by Moshe Zadka Aug. 13, 2001

Aug. 13, 2001

Hi! I am happy to say the relaying stuff works, but there is still plenty to fix. However, I actually tested "smart host" relaying and it worked. That means, twisted.mail is a real mail server, if a bit young. I'll be writing a tutorial about setting up a twisted.mail server soon enough. -- The Official Moshe Zadka FAQ: http://moshez.geek The Official Moshe Zadka FAQ For Dummies: http://moshez.org Read the FAQ

1 0

[Twisted-Python] Mail Relaying
by Moshe Zadka Aug. 12, 2001

Aug. 12, 2001

Don't get your panties in a knot, this isn't ISP-class mail relaying yet, just simple localhost-to-smarthost mail relaying. These classes are supposed to work in tandem with a DomainPickler which leaves messages for them to relay. Without further ado, here it is: from twisted.python import delayed from twisted.mail import relay class SMTPManagedRelay(relay.SMTPRelay): def __init__(self, messages, manager): SMTPRelay.__init__(self, messages) self.manager = manager def sentMail(self, addresses): SMTPRelay.sentMail(self, addresses) if addresses: self.manager.notifySuccess(self, self.messages[0][0]) if addresses: self.manager.notifyFailure(self, self.messages[0][0]) def connectionLost(self): self.manager.notifyDone(self) class SmartHostSMTPRelayingManager: def __init__(self, directory, smartHostIP, maxConnections=1, maxMessagesPerConnection=10): self.directory = directory self.maxConnections = maxConnections self.maxMessagesPerConnection = maxMessagesPerConnection self.delayed = smartHostIP self.managed = {} self.relayingMessages = {} self.readDirectory() def notifySuccess(self, relay, message): message = os.path.basename(message) self.managed[relay].remove(message) self.messages[message] = 1 del self.relayingMessages[message] notifyFailure = no.tifySuccess def notifyDone(self, relay): for message in self.managed[relay]: self.notifyFailure(relay, message) del self.managed[relay] def __getstate__(self): dct = self.__dict__.copy() del dct['managed'], dct['relayingMessages'], dct['messages'] return dct def __setstate__(self, state): self.__dict__.update(state) self.relayingMessages = {} self.managed = {} self.readDirectory() def readDirectory(self): self.messages = {} for message in os.listdir(self.directory): if not self.relayingMessages.has_key(message): self.messages[message] = 1 def checkState(self): self.readDirectory() if self.messages: return if len(self.managed) >= self.maxConnections: return nextMessages = self.messages.keys()[:self.maxMessagesPerConnection] toRelay = [] for message in nextMessages: self.relayingMessages[message] = 1 del self.messages[message] toRelay.append(os.path.join(self.directory, message)) protocol = SMTPManagedRelay(toRelay, self) self.managed[protocol] = nextMessages transport = tcp.Client(smartHostIP, 25, protocol) # It's difficult to pickle methods # So just have a function call the method def checkState(manager): manager.checkState() def attachManagerToDelay(manager, delay, time): loop = delayed.Loop(delay, time, checkState, manager) loop.loop() -- Moshe Zadka - http://moshez.geek (if you're not cool, http://moshez.org is still working)

1 0

[Twisted-Python] Checking in SSL support
by Itamar Aug. 11, 2001

Aug. 11, 2001

Ok, I have basic working SSL support. I also made an echoserv-ssl.py so I could test the code. I can telnet-ssl in and get data echoed back. May I checkin the following? 1) twisted.internet.ssl 2) docs/examples/echoserv-ssl.py 3) docs/examples/server.pem (a SSL ceritificate so echoserv-ssl can run). SSL contexts are created per socket and not stored in the ssl.Port/ssl.Client instances, since they are not picklable.

2 2

[Twisted-Python] Alternative method for thread-safe transport code
by Itamar Aug. 11, 2001

Aug. 11, 2001

The scheduler is integrated into twisted's event loop I presume. So, instead of making FileDescriptor thread-safe, how abou making the scheduler htread safe, and then threads that want to write or close a socket can add this operation to the scheduler?

1 0

[Twisted-Python] Wierd Error
by Itamar Aug. 10, 2001

Aug. 10, 2001

Might this is be the result of a connection being closed by a thread, implying that this is not thread safe? If so (even if the error resulted from something else, actually) I think the FileDescriptor.loseConnection() method should also be synchronized. Unexpected error in Selector.run. Traceback (most recent call last): File "twisted/internet/main.py", line 277, in run doSelect(timeout) File "twisted/internet/main.py", line 191, in doSelect [], timeout) File "twisted/internet/abstract.py", line 232, in fileno raise NotImplementedError(str(self.__class__)+' has no fileno method') NotImplementedError: twisted.internet.tcp.Server has no fileno method

2 1

[Twisted-Python] First try at thread-safe FileDescriptor
by Itamar Aug. 10, 2001

Aug. 10, 2001

First, my threaded server works great when using threaded mode. However, having a lock on every write call seems excessive, especially since some of my servers aren't threaded, so I tried rewriting abstract.FileDescriptor so that the write() and doWrite() methods are thread-safe without using locks. The issues: 1) It doesn't actually work - if I turn of the threadble.synchronize() call at the end of the module then the server gets deadlocks. 2) How can I make the FileDescriptor backwards compatible? Add a custom __getstate__ that adds the unsentChunks list? If writeable FileDescriptors are never pickled, which makes sense to me, then this is not issue. 3) Will this slow things down? I may just give up and write a ThreadSafeFileDescriptor class, allowigng threaded servers to use it while non-threaded servers can use the regular FileDescriptor. Index: abstract.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/abstract.py,v retrieving revision 1.2 diff -r1.2 abstract.py 1d0 < 3c2 < import types --- > import types, operator, string 22c21,24 < --- > > def __init__(self): > self.unsentChunks = [] > 53a56,63 > numChunks = len(self.unsentChunks) > if numChunks: > chunks = [self.unsent,] > for i in range(numChunks): > chunks.append(self.unsentChunks.pop(0)) > > self.unsent = string.join(chunks, '') > 80a91,92 > > This method is thread-safe. 86c98 < self.unsent = self.unsent + data --- > self.unsentChunks.append(data) 88c100 < if len(self.unsent) > self.bufferSize: --- > if len(self.unsent) + reduce(operator.add, map(len, self.unsentChunks)) > self.bufferSize: Index: tcp.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/tcp.py,v retrieving revision 1.10 diff -r1.10 tcp.py 41a42 > abstract.FileDescriptor.__init__(self) 228,229c229,230 < self.factory = factory < self.backlog = backlog --- > self.factory = factory > self.backlog = backlog

4 4