Mailman 3 September 2004 - Twisted

[Twisted-Python] TCP wrapper support for twisted servers
by Eugene Coetzee 21 Sep '04

21 Sep '04

Hi everyone, I was wondering if anybody knows about a Python wrapper for libwrap. I'm aware of libwrap function libraries for PHP and libwrap-ruby for Ruby - but could not google anything similar for Python. I want to control access to my twisted servers using hosts.allow/deny. I have read someting about "inetd setup" options planned for tap2deb (couldn't get any docs regarding inetd/tcp -wrapper support though). Alternatively, is there anything included in twisted that can be used get tcp wrapper support without having to run from inetd. regards, Eugene Web -> www.reedflute.com ===============================================

3 5

Re: [Twisted-Python] log rotation
by J C Lawrence 21 Sep '04

21 Sep '04

On Tue, 21 Sep 2004 13:21:09 GMT exarkun <exarkun(a)divmod.com> wrote: > Why you'd _want_ to rotate them this way, instead of just letting them > automatically be rotated by twistd (which they will be), I dunno. Is > there some feature logrotate has that twistd lacks that you're > interested in? In my case its the simple fact that logrotate is responsible for all log rotation for every system on the network. Making twisted-based apps a special case which I then have to tell everybody about, remind them repeatedly etc, is not particularly welcome. -- J C Lawrence ---------(*) Satan, oscillate my metallic sonatas. claw(a)kanga.nu He lived as a devil, eh? http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.

2 1

[Twisted-Python] Fwd: [Twisted-commits] r11721 - revert r11685
by exarkun＠divmod.com 20 Sep '04

20 Sep '04

Begin forwarded message: > From: Jp Calderone <exarkun(a)wolfwood.twistedmatrix.com> > Date: Sat, 18 Sep 2004 21:25:57 -0600 > To: twisted-commits(a)twistedmatrix.com > Subject: [Twisted-commits] r11721 - revert r11685 > Reply-to: twisted-python(a)twistedmatrix.com > > Author: exarkun > Date: Sat Sep 18 21:25:57 2004 > New Revision: 11721 > > Removed: > trunk/doc/web/howto/newweb.xhtml > trunk/twisted/web2/ > Log: > revert r11685 > > twisted/web2 has an unacceptable dependency on nevow that will need to be > discussed or removed before this subpackage can be added. > > > _______________________________________________ > Twisted-commits mailing list > Twisted-commits(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-commits >

10 22

[Twisted-Python] On the lighter side...
by James Y Knight 20 Sep '04

20 Sep '04

Some publicity: From LWN.net <http://lwn.net/Articles/102742/>: > ZDNet Australia covers an MPAA screwup > <http://news.zdnet.co.uk/business/0,39020645,39167160,00.htm>. "Linux > Australia president Pia Smith told Builder AU the MPAA had issued > Linux Australia with a notice of claimed infringement demanding the > group cease providing access to two copyrighted movies -- one called > 'Grind' and the other 'Twisted' -- and ordering it to 'take > appropriate action against the account holder'. However, the files in > question had nothing to do with those movies. The file entitled > Twisted is a download of the popular framework written in Python and > Grind refers to a download of Valgrind, a tool for developers to > locate memory management." Linux Australia looks set to have some fun > with this one. The ZDNet article even links to the twistedmatrix website. Woo. :) James

1 0

[Twisted-Python] ftp: LIST blocking issues
by Andy Gayton 19 Sep '04

19 Sep '04

Hi there, I'm finding that the ftp server hangs the client when performing a LIST. I thought it was something to do with what I was doing, but find it happens with the standard tap anonymous ftp server as well. It looks like processCommand adds requests to a blocked queue if dtpInstance is None - which it is in my case: ---- 2004/09/13 14:03 EST [FTP,0,127.0.0.1] 'PORT 127,0,0,1,152,65' 2004/09/13 14:03 EST [FTP,0,127.0.0.1] processing command ['PORT', '127,0,0,1,152,65'] 2004/09/13 14:03 EST [FTP,0,127.0.0.1] FTP.processCommand: cmd = PORT, args = ('127,0,0,1,152,65',) 2004/09/13 14:03 EST [FTP,0,127.0.0.1] FTP.processCommand: cmd PORT in blockingCommands 2004/09/13 14:03 EST [FTP,0,127.0.0.1] FTP.processCommand: self.dtpInstance = None 2004/09/13 14:03 EST [FTP,0,127.0.0.1] during dtp setup, blocked = [('PORT', ('127,0,0,1,152,65',))] ---- I created the server with 'mktap ftp --root=/tmp' against Twisted trunk r11660 Is there something I'm missing? thanks, Andy.

2 2

[Twisted-Python] retrial, the new-and-improved unittesting framework for twisted
by Jonathan Simms 19 Sep '04

19 Sep '04

Hello twisted people, Many of you know that I've been working on rewriting the twisted-framework's unittesting program "trial". This message is for those of you who don't ;) I've nearly completely redesigned trial in terms of both it's user-facing features, and it's internal workings. In initial testing, it's shown a 4x speed increase over regular trial, running twisted's tests in roughly 220s on a p3-1Ghz machine (as compared to 800+ seconds on the same box). here are some of the features you can look forward to (from the twisted.trial.unittest docstring, so please excuse the epydoc markup): 1. B{Trial now understands deferreds!} - There is no reason to use L{twisted.trial.util.wait} or L{twisted.trial.util.deferredResult}. Write your deferred handling code as you normally would, make your assertions in your callbacks and errbacks, then I{return the deferred from your test method}. Trial will spin the reactor (correctly) and will wait for the results before running the next test. This will allow developers to write more natural-looking tests for their asynchronous code. - there is a new attribute that has been introduced, C{.timeout}. Trial will wait a default 4 seconds for a result from a deferred that is returned from a test method. If you wish to make this value smaller or larger: >>> from twisted.trial.unittest import TestCase >>> from twisted.internet import defer >>> class MyTestCase(TestCase): ... def testThatReturnsADeferred(self): ... return defer.success('Hooray!') ... testThatReturnsADeferred.timeout = 2.8 ... >>> This would cause trial to wait up to 2.8 seconds (quite needlessly in this case) for the deferred to either callback or errback 2. B{Trial is now 100% compatible with new-style classes and zope interfaces} - Some people (the maintainer included), have been bitten in the past by trial's mediocre support for new-style classes (classes which inherit from object). In v2.0, nearly all of the classes that comprise the framework inherit from object, so support is built-in. Whereas before the TestCase finding machinery used a test for inheritance from L{twisted.trial.unittest.TestCase}, the new mechanism tests that L{twisted.trial.interfaces.ITestCaseFactory} is supplied by your class B{type}. You can write a custom TestCase, and trial will detect it and use it as a class to test, if you do: >>> import zope.interface as zi >>> from twisted.trial.interfaces import ITestCaseFactory >>> from twisted.trial.interfaces import ITestCase >>> class MyTestCase(object): ... zi.classProvides(ITestCaseFactory) ... zi.implements(ITestCase) >>> Naturally, the class should actually provide an implementation of L{twisted.trial.interfaces.ITestCase}. - To avoid any possible conflicts (and to provide component de-registration), trial uses it's own private adapter registry, see L{twisted.trial.__init__} for details. - Trial makes use of zope.interface.Interfaces to allow flexibility and adaptation. All objects implement interfaces, and those interfaces are centralized and documented in L{twisted.trial.interfaces}. 3. B{All assert* and fail* methods are now top-level functions of the unittest module} - Previously they were only available as instance-methods on the TestCase. You can now import all of the assert* and fail* variants and use them as functions. This will allow you to use this functionality in helper classes and functions that aren't part of your TestCase (plus less typing ;]) - Note: these methods are no longer part of the ITestCase API, but are provided as a backwards-compatability to classes written to use the original TestCase class. 4. B{The trial script now accepts a --reporter option} - This is to allow for custom reporter classes. If you want to run a trial process remotely, and gain access to the output, or if you would just like to have your reporting formatted differently, you can supply the fully-qualified class name (of a class that implements L{twisted.trial.interfaces.IReporter}) to --reporter, and trial will report results to your class. - The Reporter is now (almost) totally stateless. All stats on the test run are held in the TestSuite and are reported as necessary using the ITestStats interface. This allows for greatly simplified design in the Reporter implementation. - The Reporter API has been greatly simplified by changing the method signatures so that methods are called with a single object that can easily be adapted to provide all information necessary about a given test phase. 5. B{Compatibility for PyUnit tests} - Trial now supports stdlib unittest.TestCase classes transparently. This functionality is unstable, and has not been heavily tested. - Note: Trial accomplishes this by monkey-patching unittest.TestCase in L{twisted.trial.__init__}. - Please report any bugs you find with this feature to the twisted-python mailing list 6. B{Experimental support for doctests} - The trial script now supports a --doctest=[module] option. The argument is a fully-qualified module name, and trial will use a modified version of DocTestSuite to run the doctests it finds. - My support for doctests is broken when using Python 2.4-alpha3, hopefully, i'll get this fixed by the time the first major-release comes out. - Note: you cannot use .skip or .todo attributes with doctests, all tests will be reported as pass/fail - Please report any bugs you find with this feature to the twisted-python mailing list 7. B{expectedAssertions} is no longer supported - it was just too difficult to make radix's clever deferred-doublecheck feature work with this code revision. With his permisison, this feature has been removed. Trial's 'special' attributes: 1. .todo attributes can either be set on the TestCase or on an individual test* method, and indicate that the test is expected to fail. New tests (for which the underlying functionality has not yet been added) should set this flag while the code is being written. Once the feature is added and the test starts to pass, the flag should be removed. 2. Tests of highly-unstable in-development code should consider using .skip to turn off the tests until the code has reached a point where the success rate is expected to be monotonically increasing. 3. Tests that return deferreds may alter the default timeout period of 4.0 seconds by adding a method attribute C{.timeout} which is the number of seconds as a float that trial should wait for a result. I am currently developing this in a branch in the twisted repository. I would like to encourage anyone who is curious to check it out and play with it and its new features. I'd like to make it as bug free as possible before I merge it back into trunk, and the more eyes I can get on it, the better. you can check it out by doing "svn co svn://svn.twistedmatrix.com/svn/Twisted/branches/slyphon/retrial-2" and thank you for your support.... -Jonathan "slyphon" Simms

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight 17 Sep '04

17 Sep '04

Bug summary ______________________ Since 2004-09-10 22:45:53.770000: Bugs opened: 9 Bugs closed: 9 Total open bugs: 305 (+0) New / Reopened Bugs ______________________ [WORDS] IRC examples bug fix/improvment (9 hours ago) http://www.twistedmatrix.com/bugs/issue721 opened by yacc cfreactor breaks LoopingCall (yesterday) CLOSED http://www.twistedmatrix.com/bugs/issue717 opened by anthony [MAIL] MaildirMailbox.appendMessage implementation. (4 days ago) http://www.twistedmatrix.com/bugs/issue714 opened by yacc Pluggable Application loaders (yesterday) http://www.twistedmatrix.com/bugs/issue720 opened by exarkun tempfile.mkstemp in twisted.python.compat (4 days ago) http://www.twistedmatrix.com/bugs/issue713 opened by yacc [CORE] better pluggable reactor support for twistd (yesterday) http://www.twistedmatrix.com/bugs/issue719 opened by itamarst [CORE] Allow specifying exit code in generic way (supporting use and non-use of twistd) (yesterday) http://www.twistedmatrix.com/bugs/issue718 opened by itamarst [CONCH] weirdness in conch's connection forwarding (yesterday) http://www.twistedmatrix.com/bugs/issue716 opened by z3p [CORE] No way to override the default signal handler (2 days ago) CLOSED http://www.twistedmatrix.com/bugs/issue715 opened by loic Bugs Closed ______________________ cfreactor breaks LoopingCall (yesterday) http://www.twistedmatrix.com/bugs/issue717 opened by anthony [MAIL] twisted.mail.pop3.POP3 unconditionally chatty (2 weeks ago) http://www.twistedmatrix.com/bugs/issue700 opened by yacc [DOCUMENTATION, RELEASE MANAGEMENT] Twisted 1.3 identifying itself as 1.3.0rc1 (2 months ago) http://www.twistedmatrix.com/bugs/issue641 opened by br1 [DOCUMENTATION, TWISTD] twistd or twistw? (2 months ago) http://www.twistedmatrix.com/bugs/issue640 opened by br1 spec file in Twisted-1.3.0 tarball has wrong version and source information (1 month ago) http://www.twistedmatrix.com/bugs/issue679 opened by jawildman [WEB] SCGI server protocol implementation (1 week ago) http://www.twistedmatrix.com/bugs/issue703 opened by fog Implement spawnProcess for default reactor on Windows (5 months ago) http://www.twistedmatrix.com/bugs/issue591 opened by justinj [CORE] Port t.p.c to Zope3 components (5 months ago) http://www.twistedmatrix.com/bugs/issue572 opened by itamarst [CORE] No way to override the default signal handler (2 days ago) http://www.twistedmatrix.com/bugs/issue715 opened by loic

1 0

[Twisted-Python] PyHeartbeat cookbook recipe
by Nicola Larosa 17 Sep '04

17 Sep '04

My old PyHeartbeat recipe on the Cookbook site, included in the O'Reilly book too, has got a fresh new look, and a new Twisted implementation besides the old threaded one. Its shortness and double implementation may make it a nice first example for those trying to disentangle themselves from those pesky threads. ;-) PyHeartbeat - detecting inactive computers http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52302 -- Nicola Larosa - nico(a)tekNico.net "Laugh. But you too are going to sound like and old fart one day. And the respect you show or don't show for those that came before you is going to be what you instill in those that come after you." -- EvilTwinSkippy, Slashdot, June 2004

3 4

[Twisted-Python] domish.Element and deepcopy problem
by Arthur Lutz 17 Sep '04

17 Sep '04

Hi, I'm having a problem with domish.Element, here it is (isolated) : from twisted.xish import domish from copy import deepcopy invitation = domish.Element(('jabber:client', 'message')) invitation2 = deepcopy(invitation) gives me : Traceback (most recent call last): File "modules/isolated_bug.py", line 5, in ? invitation2 = deepcopy(invitation) File "/usr/lib/python2.3/copy.py", line 198, in deepcopy rv = reductor(2) TypeError: 'NoneType' object is not callable Python version : Python 2.3.4 (#2, Jul 5 2004, 09:15:05) [GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Hope this is enough for a diagnostic. Thanks a lot for the good work, -- Arthur LUTZ LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org Advanced Computing - Knowledge Management - Artificial Intelligence

2 1

[Twisted-Python] Memory leak//problem in twisted write procedures
by Joshua Moore-Oliva 17 Sep '04

17 Sep '04

I believe I have identified a memory problem in twisted. A quick background.. I've been developing a program that has to use mssql for reasons beyond my control. Due to the chaotic nature of mssql tds implementations, I created a program to pipe queries through to a windows box. In the beginning, I was using asyn* for my communications, however asyn* was very inefficient when writing large amounts of data as after each write the string would be spliced to remove the data written. I then moved to twisted to use it's far more efficient buffer implementation, and the performance problems went away. However, I have since been hit with an apparent memory leak. After reading through the twisted source (specifically the write routines in twisted/internet/abstract.py) I believe I have come across the problem. *** the write implementation continually increases it's write databuffer, only freeing up memory when the data buffer is all sent However, successive write operations append data to the data buffer.. So if a program is able to keep the data buffer from ever completely emptying (which mine is) then the data buffer will forever grow, resulting in a memory problem. I have added some output to print out the length of the data buffer and the offset, and after only 10 minutes of running the numbers are already offset == 76906496 dataBuffer len == 120768384 *** Now, reading through the source to fix this problem, the fastest solution (requiring the least change to the existing code) would be to splice//reduce the size of the dataBuffer after offset exceeds a certain number. However, I'm not too intimate with the source.. I will try that above solution to see if it does any good.. otherwise, i think that this is a fairly serious bug that should be addressed by someone more intimate with the source. Thanks, Joshua Moore-Oliva

2 2

[Twisted-Python] Problem with process ?
by Philippe Bouige 14 Sep '04

14 Sep '04

Hello, I have problem with getProcessOutput, I attempt this code (*) and i change the line : return utils.getProcessOutput("ls",["-l", "-a"]) by : return utils.getProcessOutput("ls",["-l, "."]) it's Ok for this... But when i change by : return utils.getProcessOutput("ls",["-l *"]) or : return utils.getProcessOutput("ls",["-l -a"]) or : return utils.getProcessOutput("ls",["-l ."]) and i have allways this error, but I don't undestand ?? *--- Failure #5 --- --- <exception caught here> --- *--- End of Failure #5 --- Traceback (most recent call last): File "/home/foo/twisted/internet/default.py", line 526, in doSelect ....etc.. "/home/foo/twisted/internet/utils.py", line 44, in errReceived self.deferred.errback(failure.Failure(IOError("got stderr"))) exceptions.AttributeError: 'NoneType' object has no attribute 'errback' *--- Failure #9 --- --- <exception caught here> --- *--- End of Failure #9 --- Traceback (most recent call last): File "/home/foo/twisted/internet/default.py", line 526, in doSelect _logrun(selectable, _drdw, selectable, method, dict) ......etc.. File "/home/foo/twisted/internet/utils.py", line 44, in errReceived self.deferred.errback(failure.Failure(IOError("got stderr"))) exceptions.AttributeError: 'NoneType' object has no attribute 'errback' *--- Failure #13 --- --- <exception caught here> --- *--- End of Failure #13 --- Thank you very much for your help. (*)-------------------------------------------------------- from twisted.internet import protocol, reactor, defer, utils from twisted.protocols import basic class FingerProtocol(basic.LineReceiver): def lineReceived(self, user): self.factory.getUser(user ).addCallback(lambda m: (self.transport.write(m+"\r\n\n")) ).addErrback(lambda x : x.printDetailedTraceback()) class FingerFactory(protocol.ServerFactory): protocol = FingerProtocol def getUser(self, user): return utils.getProcessOutput("ls",["-l -a"]) reactor.listenTCP(1079, FingerFactory()) reactor.run() -- Cordialement Philippe

3 3

[Twisted-Python] Weekly Bug Summary
by James Y Knight 13 Sep '04

13 Sep '04

Okay, I made a script that generates this from Roundup -- I'm going to put it in a crontab and have it email out every week from now on. Alternate formatting suggestions welcome. Bug summary ______________________ Since 2004-09-03 22:45:53.767335: Bugs opened: 10 Bugs closed: 5 Total open bugs: 305 (+5) New / Reopened Bugs ______________________ returning None from buildProtocol in ReconnectingClientFactory causes exception (2 hours ago) http://www.twistedmatrix.com/bugs/issue710 opened by djk121 Oscar html function broken (yesterday) CLOSED http://www.twistedmatrix.com/bugs/issue709 opened by jadestorm [CORE] ReactorTime has serious performance issues (yesterday) http://www.twistedmatrix.com/bugs/issue707 opened by stefan Add ClientCreator example to TCP Client howto (yesterday) http://www.twistedmatrix.com/bugs/issue708 opened by mcfletch [DOCUMENTATION] failure.check() docstring (2 days ago) http://www.twistedmatrix.com/bugs/issue706 opened by thomasvs retrial util.deferredError blows up on failure (3 days ago) http://www.twistedmatrix.com/bugs/issue705 opened by warner [CHAT] MSN Protocol bugs (with patches & unit tests) (5 days ago) http://www.twistedmatrix.com/bugs/issue702 opened by delx [XISH] Jabber xmlstream parse bug (with patch and unit test) (5 days ago) http://www.twistedmatrix.com/bugs/issue701 opened by delx SCGI server protocol implementation (3 days ago) http://www.twistedmatrix.com/bugs/issue704 opened by fog SCGI server protocol implementation (3 days ago) http://www.twistedmatrix.com/bugs/issue703 opened by fog Bugs Closed ______________________ Oscar html function broken (yesterday) http://www.twistedmatrix.com/bugs/issue709 opened by jadestorm [PB] callRemote exception handler relies on possibly unbound local `req' (8 months ago) http://www.twistedmatrix.com/bugs/issue459 opened by exarkun [PB] Copyable and Referenceable undefined in pb.py (sandbox) (8 months ago) http://www.twistedmatrix.com/bugs/issue461 opened by exarkun reconnecting ConnectionPools (1 month ago) http://www.twistedmatrix.com/bugs/issue657 opened by davep Error reporting for unslicable objects isn't very informative (8 months ago) http://www.twistedmatrix.com/bugs/issue462 opened by exarkun

3 2

[Twisted-Python] SSH server silently failing to open port
by Jason Stitt 13 Sep '04

13 Sep '04

Hi, I'm still trying to get my SFTP server up. After looking over SFTPGateway in some detail, I think I have an idea of where all the pieces fit, but I'm still having the problem I had before with the sshsimpleserver example. Even though my factory is being instantiated and the log output says it's starting on the port I specified, the port is not, in fact, open to any connections. Because there is no error reported, I'm having a hard time debugging this. I know that my factory's __init__ method completes successfully, and startFactory is also called. After that... nothing. Can anyone help me figure out where to start looking, at least? Thanks, Jason Here's my Factory class, minus a few comments and debug print statements: class Factory(factory.SSHFactory): def __init__(self): self.portal = portal.Portal(Realm()) def makeConnection(*args, **kw): return Connection(*args, **kw) self.services = { 'ssh-userauth': userauth.SSHUserAuthServer, 'ssh-connection': makeConnection, } # TODO: replace this with keys loaded from a file self.__private_keys = { 'ssh-rsa': keys.getPublicKeyString(data=publicKey) } self.__public_keys = { 'ssh-rsa': keys.getPrivateKeyObject(data=privateKey) } def buildProtocol(self, address): # Note: this function does not use its address parameter. I don't know # why it's there. return factory.SSHFactory.buildProtocol(self, address) def getPrivateKeys(self): return self.__private_keys.copy() def getPublicKeys(self): return self.__public_keys.copy()

2 5

[Twisted-Python] Including code under PSF license for backwards compatibility?
by James Y Knight 13 Sep '04

13 Sep '04

Regarding licensing: Is it okay to include Python library code in twisted for purposes of backwards compatibility. Two examples recently: - Heapq is new in python 2.3. Therefore, regardless of whether we modify it or not, if we use it, we either need to depend on python >= 2.3, or include a copy for compatibility purposes. It has a pure python implementation. - yacc on IRC wanted to use tempfile.mkstemp. This only exists on >= 2.3. Again, pure python implementation. Therefore, Glyph: a pronouncement is necessary on whether, and under what conditions, it is acceptable to include PSF-licensed code in twisted. Select one: a) It is always okay. Copy freely from python into twisted. b) It is okay, as long as the modules are included for backwards compatibility only and can be removed as soon as the minimum required version moves up to 2.3 (or whatever the latest and greatest version we need modules from). Also the code under PSF must be segregated in a separate file. c) It is okay <fill in the blank> d) It is never okay. As you can probably tell, my initial inclination is (b). James

2 1

[Twisted-Python] Async Message Queue?
by Tom Samplonius 13 Sep '04

13 Sep '04

Is there an implementation of an async message queue? I would like to be send messages to another server, with the guarrentee that accepted messages can not be lost, if even if the destination server is down. Tom

1 0

[Twisted-Python] Handling (intercepting) exceptions in twisted
by Joshua Moore-Oliva 12 Sep '04

12 Sep '04

Whenever an exception is raised, I would like to handle it instead of twisted so I can syslog and exit the application. Is there an excepthook for twisted that I can override to provide this functionality? Thanks, Joshua Moore-Oliva

2 1

[Twisted-Python] [repost] twistd quietness
by Tsai Li Ming 12 Sep '04

12 Sep '04

Hi Is there an option in twistd from showing stderr messages from appearing, specifically "Removing stale pidfile /var/run/myProg.pid"? # twistd -y myProg.tac -q --pid /var/run/myProg.pid I'm already using the quiet option. Do I simply redirect stderr to /dev/null? -Liming

1 0

[Twisted-Python] MSN & Jabber patches
by James Bunton 11 Sep '04

11 Sep '04

Hiya all. Just a friendly reminding prod to whoever that issue 701 and 702 are both sitting in the Twisted tracker with patches and unit tests. =) --- James

2 1

[Twisted-Python] Things that Need to be Done for 2.0
by Christopher Armstrong 10 Sep '04

10 Sep '04

glyph: Come up with a precise license and copyright policy. Do that, and someone (probably me) can change trunk HEAD to use it. Everyone: Get rid of all deprecation warnings when running current APIs. Me: Finish converting our packaging to use zpkgtools, obsoleting half of release-twisted. Then improve release-twisted (fix bugs in SVN workflow, etc). Me: Write documentation on release process and teach people how to do it. Moshez: At least do the initial work for Debian packages for each of our subprojects. Web Sites, in order of importance: Someone: Move existing Twisted-core-specific content to a new Core project web site. foom, fzZzy, dreid: Write a project web site for Web. z3p: Write a project web site for Conch. exarkun: Write project web sites for names, news, and mail. spiv: Write a project web site for Lore and Runner. glyph, z3p, exarkun, ???: Write a project web site for Words. cce: Write a project web site for Flow. Tv: Write a project web site for Pair. Project web sites are to be placed at http://twistedmatrix.com/projects/*/, where * is 'core', 'conch', 'web', etc. Put your content in the WebSite svn repository in "twistedmatrix.com/projects/". If anybody needs access to this repository shoot me a message. Obviously, not all of these are terribly urgent. The urgent ones are core, names, conch, lore, and web. If nobody gets around to creating real web sites for the others, I'll put in place a basic stock web page. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com

3 2

[Twisted-Python] Subclassing processProtocol.
by Joshua Moore-Oliva 10 Sep '04

10 Sep '04

I have an application that needs to communicate via netstrings over pipes that I am porting to twisted. Is it possible to use the netstring protocol that already exists with a processProtocol? SInce my terminology is probably all incorrect, in simpler terms.. I would like to be able to spawn a process, and communicate with that process via netstrings. The netstring module would be perfect for this, except that I only know how to get it to work with sockets not processes. I could write my own netstring protocol on top of the processProtocol, but I would prefer to use the current Netstring protocol if possible as it would save time and would probably be better than code I would write :) Is it possible to launch a process and communicate with it via the twisted.protocols.basic.NetstringReceiver class? Thanks, Joshua Moore-Oliva

1 0

[Twisted-Python] What does this error mean? (Port 12345 Closed)
by Joshua Moore-Oliva 10 Sep '04

10 Sep '04

To start off with, last night I had a major bottleneck in python's generic iasynchat modules for large amounts of data.. so I switched to twisted last night. It ran great, 50x faster!!. It ran for about 2.5 hours, and then I received this error.. 2004/09/10 02:54 CDT [-] (Port 12345 Closed) 2004/09/10 02:54 CDT [-] Stopping factory <twisted.internet.protocol.ServerFactory instance at 0x4076628c> 2004/09/10 02:54 CDT [-] Saving chatserver application to chatserver-shutdown.tap... 2004/09/10 02:54 CDT [-] Traceback (most recent call last): File "/usr/lib/python2.3/site-packages/twisted/internet/defer.py", line 227, in callback self._startRunCallbacks(result) File "/usr/lib/python2.3/site-packages/twisted/internet/defer.py", line 288, in _startRunCallbacks self._runCallbacks() File "/usr/lib/python2.3/site-packages/twisted/internet/defer.py", line 313, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 317, in _cbContinueSystemEvent self._continueSystemEvent(eventType) --- <exception caught here> --- File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 325, in _continueSystemEvent callable(*args, **kw) File "/usr/lib/python2.3/site-packages/twisted/persisted/sob.py", line 135, in save self._saveTemp(filename, passphrase, dumpFunc) File "/usr/lib/python2.3/site-packages/twisted/persisted/sob.py", line 103, in _saveTemp dumpFunc(self.original, f) File "/usr/lib/python2.3/site-packages/twisted/persisted/sob.py", line 119, in dumpFunc pickle.dump(obj, file, 1) cPickle.PicklingError: Can't pickle __builtin__.SqlSocket: attribute lookup __builtin__.SqlSocket failed I am rather confused as to how a port can be closed? I am very new to twisted. Could anyone shed some light on this? I need a stable long running daemon, and I am attempting to figure out why this shutdown occurred.. Thanks, Joshua Moore-Oliva

2 2

[Twisted-Python] Recursive filescanning in Twisted??
by Thomas Weholt 09 Sep '04

09 Sep '04

Hi, I want to run a web-server which displays information about files on my filesystems. The info will be stored in either a mysql or sqlite database. The number of files to be scanned can be many and huge in size. I want to click on a link which goes to a specified page/resource on the server which starts the filescanning, generates checksums for the files processed ( sha/md5 ) and stores this info in the database. The process cannot block the server, at least not the filescanning itself. The database interactions is not that critical. It would be nice to have some kind of indication available for users to see that a filescan was in progress, perhaps even some kind of progress indication. Can anybody show me how to do a non-blocking filescan like I've described in a Twisted-based web-application which generates sha/md5-checkums in the process? -- Mvh, Thomas Weholt http://www.weholt.org - thomas krøll weholt dått org ----------------------------------------------------------------------------------------------------------------- Alltid morsomt å høre røykere snakke om inneklima og luftkvalitet.

5 8

[Twisted-Python] Performance issue in reactor.callLater
by Stefan Behnel 09 Sep '04

09 Sep '04

Hi everyone! I'm running a network simulator on top of Twisted and that made me realize a heavy performance bottleneck. When I'm simulating a larger number of network nodes each of which uses two or three delayed calls, that adds up and then adding and cancelling calls becomes /really/ expensive. We are talking about ~ one millisecond per add and even a few milliseconds per cancel. The problem is the implementation that uses a linear list. A heap would be /much/ more appropriate. But especially the cancel could be spead up by factors by not actually removing the object but just setting a flag that it was canceled. The canceled DelayedCall objects would then be filtered out when looking for the next delay. I already implemented that in my simulator when I was trying to eliminate the calls to dc.cancel(). Works perfectly well, but I'd rather see Twisted scale better itself. Are there any objections to replacing the current implementation? Otherwise I'd just send a patch to the list as soon as I have integrated my version into the respective classes. Stefan

7 22

[Twisted-Python] Why don't Connectors use deferreds?
by Mike C. Fletcher 09 Sep '04

09 Sep '04

I've been wasting yet more time today looking for the "right" way to do a very simple thing: * Given a protocol * Given a destination (e.g. a TCP port) * Connect to the TCP port using the given protocol and return the protocol object to be used by client code Now, I know the "normal" way to do this is to create a one-off factory such as is seen in twisted.web.client and then use the factory to do the after-connection operations. What I'm wondering is why the code doesn't use deferred's to make the operations simpler. That is, a connector succeeds if it completes connecting, and fails otherwise. The return value would be a protocol (and thereby an attached transport). So something like: df = reactor.connectTCP( address, port, factory ) df.addCallbacks( doSomethingWithPortObject,doSomethingIfFailsOrCancelled ) Sure, that doesn't give you the ability to cancel the connection, but I'd guess it's a fairly common pattern for client-side code. With less modification of current code, simply adding a deferred to the connector that is fired on either failure-to-connect or connection would seem reasonable, so: connector = reactor.connectTCP( address, port, factory ) connector.defer.addCallbacks( doSomethingWithPortObject,doSomethingIfFailsOrCancelled ) After all, we're dealing with a classic case of a deferred operation (completion of the connection), so why not make it use the standard deferred mechanisms. I've hacked in a solution for my current code where the factory calls defer.errback in clientConnectionFailed and the protocol calls defer.callback in connectionMade, but that seems extremely inelegant (and to be honest, I'm not sure it really guarantees that the deferred will eventually be called, there may be cases where neither of those methods are called). Maybe I'm just missing something about the "proper" way to do this in Twisted, if so, feel free to point out the one-true-way, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com

2 2

[Twisted-Python] PEP 334 - Simple Coroutines via SuspendIteration
by William la Forge Jr 09 Sep '04

09 Sep '04

Clark, I'll note that you are still coupling data flow and control flow. If you think of coroutines as light-weight threads that operate very nicely in an asynchronous environment, then it becomes apparent that coroutines could easily move data via an asynchronously implemented pipe or queue. Which is exactly what I'm doing in CompStrm. ;-) Bill la Forge http://compstrm.sourceforge.net/

1 0