hi there, folks:
I'd really like to release 0.7.0 but I would like it to be at least a
little bit tested before I do so. Could those of you with CVS trees check
everything out and see if it performs as advertised? Deeper bugs than
that will have to wait for the next release, but I'd at least like to know
if it works for someone other than me.
______ __ __ _____ _ _
| ____ | \_/ |_____] |_____|
|_____| |_____ | | | |
@ t w i s t e d m a t r i x . c o m
The following things are undocumented -- any volunteers would be welcome:
- conch HOWTO
- Twisted chat HOWTO (im, words server, writing bots)
- Add tkmktap/tktwistd to basics.html
- Twisted news HOWTO
- Twisted names HOWTO
I've been working on a project using twisted for the last few weeks. In
the process I've written quite a few unittests most of which need to run
a twisted reactor either directly as reactor.run or via an Application
instance's run() method.
It's amazing how difficult that is, given that twisted itself has an
extensive set of unittests. Running only one test with a main loop is
simple but running several tests that need to listen and unlisten the
same ports or that have timeouts, etc. seems much more difficult than it
How difficult would it be to implement at least one of the following
1. a way to reset a reactor so that e.g. a new unit test could start
with a clean state. It's OK for me if the unittest itself has to
take care to do some things automatically such as disconnecting
connections or unlistenting.
2. A base class (derived from unittest.TestCase or as a mix-in class)
for tests that need to run the main loop. There should perhaps be
two of them, one for plain reactor usage and for test with
Obviously this would require having 1. to a certain extent. However,
it's OK if the class knew how to achieve a reset in as far as that
is necessary for test cases. For instance, the base class could
require that derived classes don't call the reactor directly to
listen or to add delayed calls and go through some method instead to
e.g. keep track of the ids returned by reactor.callLater.
AFAICT neither is currently implemented in twisted. There even seems to
be one unit test (test_sister) that's not implemented because of this.
I have effectively implemented no. 2 as part of my unit-tests but only
as much as I needed it so far. It didn't take much code but there are
some tricky things:
1. After and unlisten when the loop has already been left, call
reactor.iterate() to make sure the port is properly freed again.
This was relatively easy to figure out because some twisted test
cases do this.
2. Application.run() will only run the reactor mainloop if main.running
is not true. Unfortunately, main.running is not unset when
main.shutdown or main.crash is called, so when trying to enter
application main loops more than once, even for different
application instances, once has to explicitly unset main.running.
My code will eventually be available under GPL as part of the
'GREAT-ER' software we're developing, but it's not yet in CVS.
If there's interest in including my test case base class in twisted I
could make it available sooner and under LGPL.
Intevation GmbH http://intevation.de/
Over the weekend I was working on adding POP3 support to the messaging
library I'm working on. I noticed that while downloading large messages
from a POP3 server on my local network my test app would use 100% CPU,
and would stop responding to events until it finished downloading the
Looking at the twisted.protocols.pop3.POP3Client code I saw that it was
handling downloaded messages by concatenating a string, adding one line
at a time. This seemed like an inefficient way to do it, so I modified
the code to use a file-like object instead. The result is much faster,
and gives you the option of writing a downloading message to a file as
it comes in rather than keeping it all in memory.
The modified POP3Client (attached) still has some work to be done, but I
was hoping to get some comments from the list first.
Currently it works like this: When you send a command that returns a
multi-line response (LIST or RETR), you pass an optional file-like
object that the response should be written to. If no file argument is
supplied a StringIO is used. When the server is finished sending the
response, handle_COMMANDNAME is called, passing back the file object
containing the downloaded response.
At this point I'm still not correcting byte-stuffed lines in downloaded
messages, so I need to do that. Also, if it would be OK with everyone,
I think it might be nice to make POP3Client a little higher-level, so
that LIST, for example, would return an actual list instead of a file
(or, in the current implementation, a string) that the implementer has
to parse. And it might be good to offer support for more POP3 commands
If somebody could take a look at my code and tell me what I'd need to do
to make it meet the Twisted standards I'd appreciate it.
(There are also some improvements that could probably be made to the
POP3 server code, but I thought those could wait 'till later).
Hi. I wanted to have SRV record (RFC 2782) support connecting
to services. So here we go.
The following patch adds a new Connector class that takes
service (smtp, http, ldap, ..) and domain (example.com), looks
for a DNS SRV record and falls back sanely if not found.
itamar pointed out that this might be more suitable in
twisted.names.client -- feel free to point out what is the
Please give feedback.
The patch also fixes the output of SRV-using example; it used
to not output the interesting bits.
The patch is attached, and the latest version is at
Please don't kick me in head. For some reason, Cornell's webmail thingie decided to send the previous e-mail with "twisted-python(a)twistedmatrix.com" in the From: field or something. Or maybe the mailing list is broken. The previous message was from me :)
I noticed your message with an example of using CreateProcess():
I actually only need a version of the os.system() command that does
return the exit value. The problem is that under Windows 95 and 98 the
return value if always zero, thus I don't know if the command actually
worked. For my work on A-A-P (http://www.a-a-p.org) I do need to get
the exit value.
The key to solving the problem apparently is to avoid using command.com.
It seems your example can be changed to redirect the stdout and stderr
of the command, invoke CreateProcess() and obtain the exit value. But
it's not that simple.
Do you perhaps know a better solution?
hundred-and-one symptoms of being an internet addict:
244. You use more than 20 passwords.
/// Bram Moolenaar -- Bram(a)moolenaar.net -- http://www.moolenaar.net \\\
/// Creator of Vim - Vi IMproved -- http://www.vim.org \\\
\\\ Project leader for A-A-P -- http://www.a-a-p.org ///
\\\ Lord Of The Rings helps Uganda - http://iccf-holland.org/lotr.html ///
As it is now, twisted.python.usage will not
allow parameters to have empty strings as
values. That is, these are invalid currently:
--foo='' (the same thing, more explicitly)
I think t.p.usage should not be dictating
such policy for its users.
It seems this only concerns optParameters users, "def
opt_foo(self, x)" seems to be immune to this problem if
I'm reading the code correctly.
All the above cases are easily detectable, Here's a suggested
patch; tell me if you want me to commit.
RCS file: /cvs/Twisted/twisted/python/usage.py,v
retrieving revision 1.36
diff -u -r1.36 usage.py
--- twisted/python/usage.py 10 Nov 2002 10:45:36 -0000 1.36
+++ twisted/python/usage.py 29 Dec 2002 17:06:03 -0000
@@ -242,7 +242,7 @@
self.opts[flagName] = 1
def _generic_parameter(self, parameterName, value):
- if value in ('', None):
+ if value is None:
raise UsageError, ("Parameter '%s' requires an argument."
I spent some time yesterday looking into the intermittent test failures in
test_tcp and test_threadpool. These failures only occurred when the full
test suite was run (./admin/runtests -v -t), not when test_tcp was run by
itself (./admin/runtests -v -t twisted.test.test_tcp). I finally figured out
what was happening and wanted to share my findings to help others track down
Early in the test sequence, test_woven did some kind of session/login
thing. One particular test case was verifying an Unauthorized response,
which involved creating a Deferred that was then .errback()'ed. However,
there was no errback handler added to the Deferred.
Inside that test, web.guard started a 60 second session timeout, with a
callLater() function that referenced that Deferred (or an object holding
test_woven finishes. The timer is still running. The Deferred (with pending
error) is still alive, since the callLater timer prevents it from being
Other test cases run.
[60 seconds pass]
Some other innocent test case is running, depending upon the speed of the
machine. On my CPU, it was usually test_tcp.
During some call to reactor.iterate, the web.guard session timer expires,
the callback does whatever callbacky things it likes, and an object gets
freed. That leads to the Deferred being freed. The Deferred has a pending
error but no .errback handler. Deferred.__del__ does log.msg() with the
"Unhandled error in Deferred" message, and log.err() with the
"Unauthorized" Failure object that wanted an .errback but had none.
log.err() puts the message in the error queue. log.msg() goes into the
test.log file, but not into the error queue.
test_tcp finishes without problem, but unittest.py looks in the error queue
and sees the Failure. It adds the traceback of the Failure (which is the
call trace leading up to the creation of the Unauthorized exception, which
is in web.guard, and doesn't mention test_woven at all) to the error
results of the test.
test_tcp is flunked, and as explanation unittest.py emits the error results
(the Unauthorized exception and the web.guard stack trace)
The basic issue is timers that "leak" over from one unit test to another.
Unit tests are supposed to be isolated from each other, but if one test
changes the state of the reactor (by adding timers or establishing listenTCP
ports) without cleaning it up when it finishes, that state can affect other
This is not always obvious, as several modules create timers internally.
This is very convenient, and works fine in normal applications. It is only
in the unit tests where it causes a problem.
I've added IReactorTime.getDelayedCalls() as a reactor-portable way to
retrieve the list of pending callLater() timers, and changed unittest.py to
cancel all pending timers between test cases. This succeeded in moving the
failure back to the test that caused it. I've also added a warning message
(which goes into test.log) that indicates when a test leaves timers running
after it completes. There are a handful that do; most are benign but should
probably be cleaned up. This warning message may turn into a test failure
Some tools I used to track this down, in case they prove useful to others:
added code to Deferred and DelayedCall to record the stack track when they
are created, then print them in __repr__ or added to the "unhandled error
in deferred" Failure. The traceback of creation is often more useful than
the traceback of .callback, .errback, or timer execution.
added .extra attribute to Failures, insert its contents in the traceback.
I used this to display the Deferred-creation traceback.
There are still a few issues: I need to implement getDelayedCalls for the
cReactor, and my unit test for getDelayedCalls is stymied by a recurring
timer created when the 'popsicle' module is imported, but in general it
seems to make the unit test suite behave sensibly (no more intermittent