Mailman 3 July 2003 - Twisted

[Twisted-Python] problems with callback in an iteration
by Justin Johnson July 30, 2003

July 30, 2003

In the code below, I create a Trigger object (something I defined) and call the "make" method on it. Then I add a callback so that I can "lock" it after it has been made. I'm getting some wierd results though. The output from the first print statement... print "trigger name", self.name ...prints the names of the triggers correctly, as they appear in the list (trigger1 followed by trigger2). However, the callback to _lock is only called with the last element in the list (trigger2), but it is called twice. So instead of calling _lock on trigger1 and then on trigger2, it is called twice on trigger2. Does anyone see what I'm doing wrong? [snip] def applyTriggers(self): """Apply the standard triggers to the VOB. """ deferreds = [] trigger1 = trigger.PreRmverTrigger("pre_rmver","Prevent removing of versions",config.thisServer,self.tag) trigger2 = trigger.PreRmelemTrigger("pre_rmelem","Prevent removing of elements",config.thisServer,self.tag) for t in [trigger1, trigger2]: print "trigger name:", t.name def _lock(results): print "*** locking trigger:", t.name d = t.lock() return d d = t.make() d.addCallbacks(_lock, log.err) deferreds.append(d) return defer.DeferredList(deferreds) [/snip]

1 0

[Twisted-Python] Moshe wins the 1000th test award!
by Andrew Bennetts July 30, 2003

July 30, 2003

Ok, so there's no prize. But one of Moshe's recent checkins brought us over the 1000 tests mark, as judged by BuildBot: http://twistematrix.com/buildbot/ Woohoo! :) -Andrew.

1 0

[Twisted-Python] callback wierdness
by Justin Johnson July 30, 2003

July 30, 2003

In the attached files I defined process.openProcess to spawn a process using the reactor, and return a deferred from within my ProcessProtocol. Then in trigger.py I define methods "make" and "remove" that utilize process.openProcess to run a command. make() adds callbacks to the deferred to display the results of the execution, and at the bottom of trigger.py I add a callback to remove the trigger after creating it. The problem is, none of my callbacks ever get called. The initial spawnProcess works and I can see the trigger is created, but nothing is printed to the screen (callback 1 added in "make") and the trigger is not removed (callback 2 added at end of file). Can someone tell me what I'm doing wrong here? Also, is spawnProcess a special case that actually runs before reactor.run is called? That is what I am seeing, and I'm not sure if that makes sense. Just trying to get my head around this. Thanks much. -Justin

2 2

[Twisted-Python] twisted.web.xmlrpc doesn't generate content length header
by Steve Freitas July 30, 2003

July 30, 2003

Hi all, I'm experimenting with a C++ XML-RPC client library, and it refused to talk to my Twisted sample server. Why? It reported that no content length was given, so it gave up and sulked. So I dumped what Twisted coughed up, and sure enough, it's lacking a content length field (I'm calling the sample xmlrpc_add(self, a, b) function): HTTP/1.0 200 OK Date: Wed, 30 Jul 2003 04:41:56 GMT Content-type: text/xml Server: TwistedWeb/1.0.6 <?xml version='1.0'?> <methodResponse> <params> <param> <value><int>3</int></value> </param> </params> </methodResponse> Connection closed by foreign host. Could it be that I'm doing something wrong? The XML-RPC spec states that the server must respond with a correct content length, so I would've expected Twisted to behave accordingly. So am I missing something, or does Twisted not comply with the spec (as Dave Winer defines it)? Thanks, Steve P.S. For the DW-standard inclined, see http://www.xml-rpc.com/spec, under "Response format."

2 2

[Twisted-Python] questions about Twisted mail
by Matthew R. Scott July 29, 2003

July 29, 2003

Hello Twisted hackers extraordinaire! I am evaluating the use of Twisted mail for small- to medium-sized installations. My goal is to get up and running with it fairly soon, with only minor changes in the areas of email account database / authentication. As I've been evaluating it, I have come up with a few questions regarding best practices for its use. First, I'll give some background on how I am evaluating it. Disclaimer: I know my way around "talking SMTP and POP3" on a basic level (telnetting to a host and going through a session manually) but I am not too familiar with the ins and outs of "talking ESMTP AUTH" yet. :) Software ======== Mozilla 1.5a Kmail 1.5.2 Twisted CVS 2003-07-28 Goals ===== Ultimately, I want Twisted mail to support these outcomes from common e-mail scenarios: POP3 access to account: PASS SMTP Mail from outside domain to outside domain: FAIL SMTP Mail from outside domain to user: PASS SMTP Mail from user to user: PASS SMTP Mail from user to outside domain: FAIL ESMTP Mail from user to user: PASS ESMTP Mail from user to outside domain: PASS mktap variations ================ I've run through several 'mktap mail' variations and these are the results I've gathered. In all cases, there is a 'queue' and 'test.example.com' directory in the directory where mail.tap is created and started from. 'twistd -nof mail.tap' is the command line used to start the mail server. Also, when doing regular SMTP sessions, I am sending the following commands in a session with the SMTP server to try to send mail: HELO localhost MAIL FROM:<from(a)email.address> RCPT TO:<to(a)email.address> DATA testing 1 2 3 . (!) indicates passing when it should be failing, or failing when it should be passing mktap mail --maildirdbmdomain=test.example.com=test.example.com --esmtp --user=test=1234 --default POP3 access to account: PASS SMTP Mail from outside domain to outside domain: PASS (!) SMTP Mail from outside domain to user: PASS SMTP Mail from user to user: PASS SMTP Mail from user to outside domain: PASS (!) ESMTP Mail from user to user: FAIL (!) ESMTP Mail from user to outside domain: FAIL (!) mktap mail --maildirdbmdomain=test.example.com=test.example.com --esmtp --user=test=1234 --default --disable-anonymous POP3 access to account: PASS SMTP Mail from outside domain to outside domain: FAIL SMTP Mail from outside domain to user: FAIL (!) SMTP Mail from user to user: FAIL (!) SMTP Mail from user to outside domain: FAIL ESMTP Mail from user to user: FAIL (!) ESMTP Mail from user to outside domain: FAIL (!) Failure details for ESMTP ========================= When I configured Mozilla's SMTP settings, it did not give me an option to specify what kind of authentication to perform; it only allowed me to specify whether or not a username and password were required. This is what was in the twistd log when I tried sending using Mozilla in the second mktap variation: 2003/07/28 16:11 CDT [twisted.mail.protocols.ESMTPFactory] Connection from ('127.0.0.1', 33291) 2003/07/28 16:11 CDT [ESMTP,0,127.0.0.1] Failure: twisted.cred.error.UnhandledCredentials: No checker for twisted.cred.credentials.IAnonymous ,twisted.cred.credentials.ICredentials 2003/07/28 16:11 CDT [ESMTP,0,127.0.0.1] /home/gldnspud/p/PYTHONPATH/twisted/protocols/smtp.py:553: exceptions.DeprecationWarning: Returning None from validateFrom is deprecated. Raise smtp.SMTPBadSender instead Interestingly, in the first mktap variation, Mozilla was able to send mail just fine, which makes me wonder if it is even using ESMTP AUTH. When using Kmail to send mail via both variations, it complains with an "authentication failed" message. This was after setting the authentication type to CRAM-MD5 and verifying that the correct username "test" was in place. The log file from twistd shows the following: 2003/07/28 16:13 CDT [twisted.mail.protocols.ESMTPFactory] Connection from ('127.0.0.1', 33293) 2003/07/28 16:13 CDT [ESMTP,1,127.0.0.1] Failure: twisted.cred.error.UnhandledCredentials: No checker for twisted.cred.credentials.IUsernameHashedPassword ,twisted.cred.credentials.ICredentials Questions ========= Some more questions, to which I'll post my own answers if I find them out after tinkering some more: Are any of the problems I'm running into (such as the "No checker for ...IUsernameHashedPassword" failure) due to the transitional nature of Twisted CVS? (I freely admit I have not yet tried any of the above on the last "stable" Twisted release) Would it be easy (or at least feasible) to implement ESMTP "AUTH PLAIN" in Twisted's SMTP server? (I'm assuming so, since "AUTH CRAM-MD5" is already implemented) How close is IMAP server capability to being integrated with Twisted mail? How would one keep usernames on different domains separate in terms of logging in with POP3 and authenticating with ESMTP? E.g. "jdoe(a)domain1.com" and "jdoe(a)domain2.com" What are some common configurations of Twisted mail that are currently deployed in production environments? Thanks, -- Matthew R. Scott OMAjA / http://www.omaja.com/

4 4

[Twisted-Python] One big smile ...
by Thomas Weholt July 28, 2003

July 28, 2003

I implemented a small newsfetching webserver today. At first I used some blocking code at startup and then reactor.callLater(some_intervall, sameBlockingMethod) to call it later and refresh the content later. It processed about 8-10 RSS/RDF feeds and then handed my server back to me so I could see the output. I was quite happy about the result. Just for fun I tried to do the blocking code in a thread instead, - with mixed results. Read some more docs. And did a deferToThread for each call to the blocking method. Added a callback-method that parsed the results. A bit more hacking and ...then .. suddenly ... it was just .... so fast. No hogging the server at startup, all content fetched,, *much* faster than before. Everything seemed great. I had finally got the grip on threads and deferred. But something was wrong with my data. Some data were in the wrong lists etc. so I guess I need some mutex-locking or something. This is were you guys come in. Do I need to aquire a lock to avoid updating wrong parts of my datastructure? In the docs it says: "This will make certain parts of Twisted thread-safe so you can use them safely. However, note that most parts of Twisted are not thread-safe." What parts are still not thread-safe? My datastructures? They don't seem thread-safe. If I just solve the drunken attribute update process going on in my server this will be a real break-thru for me Twisted-wise. Anyway, I still have a HUGE smile on my face. Thomas

9 19

[Twisted-Python] Re: [Zope3-dev] twisted zope
by Phillip J. Eby July 28, 2003

July 28, 2003

At 02:42 PM 7/28/03 -0400, Shane Hathaway wrote: >Phillip J. Eby wrote: >>And there's yet another possible approach... Have Zope depend on >>specific I*Reactor interfaces in place of asyncore, but don't necessarily >>depend on Twisted itself. I'm not sure if this meets the intended goals >>for Zope/Twisted integration, but it's a useful first step. If this is >>useful, PEAK has an "UntwistedReactor" class that implements a useful >>subset of the Twisted reactor interfaces, enough to be able to have a >>select loop and scheduled callbacks. It's ZPL/PSF licensed, so you can >>steal it fair and square. So, as long as the existing Zope 3 server code >>just expects to be woken up by asyncore for reads and writes, it should >>be pretty adaptable to the reactor interface, at which point it could be >>hooked into Twisted or into a Zope 3 mini-reactor. > >That sounds great. Do you have a ViewCVS link? Interface is at: http://cvs.eby-sarna.com/PEAK/src/peak/running/interfaces.py?rev=HEAD&conte… See 'IBasicReactor'. For the implementation, see: http://cvs.eby-sarna.com/PEAK/src/peak/running/scheduler.py?rev=HEAD&conten… It's class 'UntwistedReactor'. The implementation depends on peak.binding, but not in any deep way. You should be able to replace most of the 'binding.bindSomehow' attributes with an __init__ method. The class also expects to have a 'logger' attribute that's a PEP 282-compatible logger. Last, but not least, you can interpret the 'self._delBinding()' calls as meaning that the specified attribute should be reinitialized. You can pretty much ignore the rest of the module for your purposes, except the _Appt class, which is a private class representing a scheduled callback. >The one feature that we'd need beyond the simple "wake me up when >something is ready" interface is the trigger interface. It lets a thread >tell the main thread to wake up immediately. Ah. I didn't implement that, as I haven't needed it for anything. I believe you can emulate it over the primitives I've supplied, though, as I believe Twisted normally does it by setting up some kind of pipe or socket that the thread writes to, with a reader hooked up on the reactor. The reader just throws away the meaningless data, but of course the main thread has now been reawakened from its select() call.

1 0

[Twisted-Python] New PB connection API
by Itamar Shtull-Trauring July 28, 2003

July 28, 2003

First sketch is in CVS. Please comment if you have any issues with it, I'd like to refactor old APIs to use new API next. -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting

3 3

[Twisted-Python] Re: [Zope3-dev] twisted zope
by Phillip J. Eby July 28, 2003

July 28, 2003

At 11:26 AM 7/28/03 -0400, Shane Hathaway wrote: >Gary Poster wrote: >>As to the request parsing, I like Twisted's design a lot. My impression >>is that if you used it, rather than Zope's request parsing, you would be >>in for a lot more work, but that work would probably end up being a big >>eventual win in clarity and simplicity. If I were doing it, I'd punt and >>stick with Zope's, with a regretful glance backward. If you have the gas >>to go up the mountain, though, I betcha the view's worth it. :-) > >May I suggest a different approach? > >Zope 3's HTTP request parsing is superior to Twisted's because it can more >easily handle complex features of HTTP like pipelining. However, >Twisted's core is a better-executed implementation of the same idea behind >asyncore. Therefore, this is how I'd like to see the integration work: > >- Enhance Twisted's HTTP server with notions taken from Zope 3's HTTP >server. The most important notion is that there is a request object that >does nothing but receive the request data. Once the entire request is >received, the request is used to construct an HTTP transaction; at that >time the request is considered immutable. Also, the generic memory/disk >buffer concept could be a big win for Twisted. > >- Rename Zope 3's HTTPTask to HTTPTransaction. Create >IHTTPTransaction. (It might become part of Twisted instead of Zope.) > >- Add only specific modules of Twisted to the Zope 3 source, such as >twisted.internet (I think that's its name). Straighten out the licensing >first. And there's yet another possible approach... Have Zope depend on specific I*Reactor interfaces in place of asyncore, but don't necessarily depend on Twisted itself. I'm not sure if this meets the intended goals for Zope/Twisted integration, but it's a useful first step. If this is useful, PEAK has an "UntwistedReactor" class that implements a useful subset of the Twisted reactor interfaces, enough to be able to have a select loop and scheduled callbacks. It's ZPL/PSF licensed, so you can steal it fair and square. So, as long as the existing Zope 3 server code just expects to be woken up by asyncore for reads and writes, it should be pretty adaptable to the reactor interface, at which point it could be hooked into Twisted or into a Zope 3 mini-reactor.

1 0

[Twisted-Python] Re: [Zope3-dev] twisted zope
by Andrew Bennetts July 28, 2003

July 28, 2003

On Sat, Jul 26, 2003 at 04:48:18PM -0700, Butch Landingin wrote: > Hi all, > > I've been mucking about with Zope3 and twisted, trying to get Zope3 to > run under twisted's reactor framework. I'm still at code-spiking stage [...] > > Some of the issues so far: > [...] > > 2. Configuration: zope's configuration machinery (i.e. ZConfig) or twisted's > configuration machinery (e.g. coil)? Which is better? Or maybe can I use > both? Or maybe a way for the two to work together? Twisted's COIL is currently old and not very functional. Work is underway to rewrite it properly (thanks Donovan!), but seeing as that isn't done yet, the pragmatic answer would have to be ZConfig. > PS. I cross-posted to both the zope3-dev and twisted-python lists > so if you're listed on both, sorry. Hey, I'm interested in both (and especially in them working together), so I can't complain! :) -Andrew.

1 0