Mailman 3 March 2007 - Twisted

[Twisted-Python] Can I still run (most of) Twisted on Python 2.2?
by Erik Forsberg 27 Mar '07

27 Mar '07

Hi! In TwistedCore/setup.py, I find that python 2.3 is required for Twisted. I'd like to use Twisted for a project that currently requires only Python 2.2, which raises the question - which parts of Twisted requires 2.3, and how much of it can I run on Python 2.2? On the same matter - are there any plans on when to stop supporting Python 2.3, Python 2.4, etc? Thanks, \EF -- Erik Forsberg OpenSource-based Thin Client Technology Systems Analyst/Developer Phone: +46-13-21 46 00 Cendio AB Web: http://www.cendio.com

6 7

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 25 Mar '07

25 Mar '07

Bug summary ______________________ Summary for 2007-03-18 through 2007-03-25 Bugs opened: 11 Bugs closed: 9 Total open bugs: 881 (+2) |== Type Changes |== Priority Changes |== Component Changes |Defect: -3 |Highest: -1 |Conch: +0 |Enhancement: +3 |High: -3 |Core: +2 |Task: +2 |Normal: +4 |Mail: -1 |Low: +2 |Pydoctor: +1 |Release Management: +0 |Web: -1 |Web2: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#2528] t.conch.ssh.userauth.ssh_USERAUTH_REQUEST creates a Deferred and doesn't return it (opened by jml) (CLOSED, fixed) defect conch http://twistedmatrix.com/trac/ticket/2528 [#2530] Remove some of the warnings of process-docs (opened by therve) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/2530 ===== Normal ===== [#2529] AMP streaming protocol extension (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/2529 [#2531] twisted.conch.stdio.runWithProtocol fails in Eclipse/pyDev console (opened by neurophyre) (CLOSED, invalid) defect conch http://twistedmatrix.com/trac/ticket/2531 [#2532] How is a source listing generated (opened by duncanwebb) (CLOSED, fixed) task pydoctor http://twistedmatrix.com/trac/ticket/2532 [#2533] Pydoctor SVN and TortoiseSVN don't work (opened by duncanwebb) enhancement pydoctor http://twistedmatrix.com/trac/ticket/2533 [#2535] Allow Twisted to work without SIGCHLD on posix (opened by spiv) enhancement core http://twistedmatrix.com/trac/ticket/2535 [#2536] Deprecated TimeoutQueue so we can remove it (opened by exarkun) task core http://twistedmatrix.com/trac/ticket/2536 [#2538] twisted.web2.client.http has no unittests for pipelining support (opened by dreid) enhancement web2 http://twistedmatrix.com/trac/ticket/2538 ===== Low ===== [#2534] investigate event dispatch optimization (opened by exarkun) task core http://twistedmatrix.com/trac/ticket/2534 [#2537] doc/current/api source links point to SVN HEAD (opened by mwh) defect release management http://twistedmatrix.com/trac/ticket/2537 Closed Bugs ______________________ ===== Highest ===== [#2528] t.conch.ssh.userauth.ssh_USERAUTH_REQUEST creates a Deferred and doesn't return it (opened by jml, closed by jml, fixed) defect conch http://twistedmatrix.com/trac/ticket/2528 [#2530] Remove some of the warnings of process-docs (opened by therve, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2530 [#2521] SMTP authentication tracebacks when receiving an empty line (opened by washort, closed by washort, fixed) defect mail http://twistedmatrix.com/trac/ticket/2521 ===== High ===== [#1272] selectreactor's addReader unexpectedly assumes a logPrefix attr (opened by marienz, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/1272 [#1366] TwistedQuotes documentation references external source files (opened by exarkun, closed by exarkun, duplicate) defect core http://twistedmatrix.com/trac/ticket/1366 [#927 ] Add description of Twisted Web to its project site (opened by hypatia, closed by oubiwann, fixed) defect web http://twistedmatrix.com/trac/ticket/927 ===== Normal ===== [#1126] Package Twisted using Eggs/EasyInstall (opened by glyph, closed by therve, duplicate) enhancement release management http://twistedmatrix.com/trac/ticket/1126 [#2531] twisted.conch.stdio.runWithProtocol fails in Eclipse/pyDev console (opened by neurophyre, closed by glyph, invalid) defect conch http://twistedmatrix.com/trac/ticket/2531 [#2532] How is a source listing generated (opened by duncanwebb, closed by duncanwebb, fixed) task pydoctor http://twistedmatrix.com/trac/ticket/2532

1 0

[Twisted-Python] thread pool question
by Rutt, Benjamin 23 Mar '07

23 Mar '07

From http://twistedmatrix.com/projects/core/documentation/howto/threading.htm l: I copied the code below and ran it as: $ cat twisted-threadpool.py #!/usr/bin/env python from twisted.internet import reactor def aSillyBlockingMethod(x): import time time.sleep(2) print x # run method in thread reactor.callInThread(aSillyBlockingMethod, "2 seconds have passed") $ ./twisted-threadpool.py 2 seconds have passed But I note that the program never stops. I infer that least one thread is getting set as a non-daemon thread, according to python's thread rules on program termination: `thread.setDaemon(daemonic)' Set the thread's daemon flag to the Boolean value DAEMONIC. This must be called before `start()' is called. The initial value is inherited from the creating thread. The entire Python program exits when no active non-daemon threads are left. Is there a way to set this program up such that the program will exit when all of the threads registered via callInThread, and the main thread, have exited? Thanks, -- Benjamin Rutt

4 7

[Twisted-Python] Twisted friendly hosting services?
by Vincent Cate 23 Mar '07

23 Mar '07

I am working on an application that runs using twisted.web. I want to host it on a machine with plenty of bandwidth as cheaply as possible. Is there a hosting service that uses Twisted.web so that I can just add my resource to the namespace? Or will I have to get a "virtual dedicated server" or a real "dedicated server"? If so, does anyone know a virtual one that uses Ubuntu? Thanks, -- Vince

4 3

[Twisted-Python] Using twisted.conch
by pico 23 Mar '07

23 Mar '07

Hi all. I'm struggling a bit with twisted.conch. I've read the howto at http://twistedmatrix.com/projects/conch/documentation/howto/conch_client.ht…. I can understand this example fine. For a simple application, it works fine to subclass SSHChannel and implement all the code in that classes methods. However I would like to be able to implement something as follows: class SSHWrapper: def connect(self,hostname,username,password): #Create SshTransport object def runCommand(self,command) #Use SshChannel object to run command, return a deferred def disconnect(self): #Cleanup The problem is that, due to the ClientFactory creating instances of the relevant objects for you (SSHClientTransport,SSHUserAuthClient,etc), you are left without a reference to the SSHChannel object,required to run the command. Is there any way to get the SSHChannel object from the SSHClientTransport instance. Any advice with this matter would be appreciated. Thanks Pico

2 1

[Twisted-Python] Twistd memory usage
by Jarek Zgoda 22 Mar '07

22 Mar '07

I have an application, ran as twistd plugin. If I run it as daemon, twistd consumes all available memory (of 4GB available), then after few seconds of swapping twistd is killed by kernel's OOM killer. I cann't reproduce this when I run this application in no-daemon mode (twistd -n), the memory footprint is constant and does not exceed 150MB. Is this normal behaviour? -- Jarek Zgoda "We read Knuth so you don't have to."

2 2

[Twisted-Python] hardcoded max in threadpool or connection pool ?
by Jonathan Vanasco 22 Mar '07

22 Mar '07

forgive my ignorance this week - i'm rusty on twisted. I seem to be hitting a limit on threads / db connection ( not sure which one , as db could block connection ) at 11 The only code i have influencing the sizes are as follows: ======= class MY_dbConnectionPool( adbapi.ConnectionPool ): def __init__( self ,): adbapi.ConnectionPool.__init__( self, 'psycopg2' , host="localhost", database="xxxxx", user="xxxxx", password="xxxxx", cp_min= 16, cp_max= 16 ) ======= reactor.suggestThreadPoolSize( 17 ) reactor.run() ======= i never see more than 11 connections - in fact, i always see 11 connections -- 10 for the db the thread pool, and 1 for the db in the main i'm fairly sure if there's any error here it's going to be my fault -- can someone point me in the right direction ?

1 0

[Twisted-Python] suggestion for tracking down cause of 2 bugs sought: possibly threads / defer.DeferredSemaphore ?
by Jonathan Vanasco 22 Mar '07

22 Mar '07

I've got 2 bugs in a twisted app that I'm trying to narrow in on. Overview of the app: A twisted service connects to a db and requests about 500 web pages to download and analyze. The service than uses defer.DeferredSemaphore service to handle dispatching each request to a thread via deferToThread . defer.DeferredSemaphore( tokens= MAX_DOWNLOADERS ) reactor.suggestThreadPoolSize( MAX_DOWNLOADERS + 1 ) Each thread runs a python script that is not written in Twisted and has a lot of blocking. And to pre-empt many comments that will inevitably pop up: It would be great to convert all the code to twisted , but 90% of the code + underlying modules are used by command line maintenance routines that were already in place. I just don't have the time or resources to rewrite it all in twisted. One day I'd love to have the funding to hire someone to manage them, but I just don't have that time myself. Overview of the bugs: 1) The query for figuring out which pages to download can take quite some time , so I run the getNextDownloads routine in a new thread , so the pool always remains full. It works perfect, except when the number of items queued up to defer.DeferredSemaphore reaches ~ 1000. After ~1000, I seem to only be running with 1-2 semaphores in place not 15 Question: Can anyone suggest a way to test this to see where the error is? Off the top of my head, the following seem plausible: a- hitting 1000 just breaks something in defer.DeferredSemaphore . the module is listed as unstable, so I'm guessing this could be the cause. b- the defer.DeferredSemaphore connection is just coincidental -- its really an issue with threads locking up c- something entirely different Unfortunately, I'm at a loss on trying to figure out whats going on. 2) I'm experiencing an obscene amount of process growth. I'm fairly certain that its from constant errors in urllib2 , which has been nothing of a nightmare with countless known bugs and patches ( downloading takes place via Mechanize , which is wrapped via urllib2 ), and makes me consider just wrapping wget or curl. but this could very well be threading related. Question: Is there any way to profile memory usage by twisted in a granular fashion ? I'd like to figure out where this memory usage is ( it grows steadily to ~1.7gb after 8 hours ). Thanks, // Jonathan Vanasco | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | SyndiClick.com | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | RoadSound.com - Tools For Bands, Stuff For Fans | Collaborative Online Management And Syndication Tools | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

2 2

[Twisted-Python] Twisted.Names assistance with auth and additional sections
by Sean Leach 21 Mar '07

21 Mar '07

Hey all, I have a client I have written using twisted.names that will send a DNS request. Here is the gist of the relevant code: (tfp, tfile) = tempfile.mkstemp() os.write(tfp, 'nameserver %s' % ip) os.close(tfp) resolver = client.createResolver(resolvconf=tfile) deferred = resolver.lookupAddress(somevalue) deferred.addCallback(resultHandler) hacky way to set the IP, but I am having issues with passing in the "servers" list I don't want to troubleshoot until I figure out this problem. Anyways, as you can see, this code calls the lookupAddress method for the name I want. The problem I am having is it is only returning the answer section of the response. I need the additional and authority sections as well (and I know for sure those sections exist in the response). I have looked at the docs and the code, and I don't know what I am missing. I am sure it's something silly, but I am missing it right now. So long question short, what do I need to change in my code so I can get the additional sections and authority sections in the response so I can get to those values? Thanks, Sean

4 9

[Twisted-Python] question about thread and threadpool
by wang wei 21 Mar '07

21 Mar '07

the document said: # run method in thread reactor.callInThread(aSillyBlockingMethod, "2 seconds have passed") but after above code, Is that Thread dead? and how can I re-use this thread? and if I create a thread using python thread module, can I use reactor to re-use it? there is a threadpool in twisted, is there any example of how to use it? thanks for any advices.

2 2