Mailman 3 February 2008 - Twisted

[Twisted-Python] Throttling a web2 server
by Cameron Dale 28 Feb '08

28 Feb '08

Hi, I'm relatively new to Twisted, and I was wondering if anyone has had success with the ThrottlingFactory, or any other methods for throttling push producers. I'm attempting to throttle the bandwidth of the web2 server using the twisted.protocols.policies.ThrottlingFactory. I know this isn't possible with web1 (see Ticket #601), but it should be possible with web2 since web2 uses a streaming (push) producer. The trouble I'm having is that the ThrottlingFactory doesn't throttle the server at all. I've traced the output, and it seems to be due to the twisted.internet.tcp.Server also calling the pauseProducing() and resumeProducing() methods of the producer. The way the ThrottlingFactory works, is that it schedules periodic checks of the number of bytes written using reactor.callLater. If too many bytes have been written, then the producer is paused. However, these periodic checks always occur when the producer is already paused by the twisted.internet.tcp.Server, and so have no effect. Later the producer is resumed by the twisted.internet.tcp.Server, and so the throttling is non-existent. Thanks, Cameron

3 5

[Twisted-Python] connection identification string
by coder_gus 28 Feb '08

28 Feb '08

Is there a way to identify a connection by a number/string for later use? It could be easily implemented or given the async character of twisted probably is not worth it ... but just as a hint? Is there? Thanks.

4 3

[Twisted-Python] thread created from tcp connection
by big_tyo＠inf.its-sby.edu 28 Feb '08

28 Feb '08

hi,my name is raditya from indonesia,i'm newbie at twisted... i tried to develop a server that receive data from many clients and send the same data to all connected clients,and somehow i found some problem here. The Problem is i still haven't figure out how to send some parameters to a thread that created from established tcp connection. please,can anyone help me with this little problem of mine? best regard, raditya ps : sorry for the bad english ----------------------------------- Fakultas Teknologi Informasi Institut Teknologi Sepuluh Nopember http://www.ftif.its.ac.id/ -----------------------------------

2 1

[Twisted-Python] How to get an unused port
by Gabriel Rossetti 27 Feb '08

27 Feb '08

Hello, I need to be able to get an unused port for a server, it will then communicate it to a third party. I don't have any restrictions other than it shouldn't be a known used port (one used by other apps (above 1023)), but for that I'll probably have to create a list and verify manually. Does anyone know how to do this (without iterating all ports and trying), it must be possible since a client gets the next unused port from the OS, I'd like to do this with Twisted if possible. Thanks, Gabriel

5 8

[Twisted-Python] DeferredList?
by Don Smith 26 Feb '08

26 Feb '08

Greetings, I have a problem I hope someone here can assist with. I'm using TwistedSNMP to query a bunch of SNMP devices asynchronously. The behavior I'm trying to get is to get is to query all the devices via SNMP, each query returns a deferred, and when all their callbacks have been fired then I want to stop the reactor and thereby exit the program. I thought perhaps I could put each of these SNMP deferreds in a DeferredList and add a callback to the DeferredList that would stop the reactor but it does not do that. Enclosed is a code sample. Am I doing something incorrectly, or should I do something different? Thanks! -Don """Trivial example to retrieve an OID from a remote Agent""" from twisted.internet import reactor from twistedsnmp import snmpprotocol, agentproxy from twisted.enterprise import adbapi from twisted.internet import defer import os APPNAME = 'ClearSNMP' d_results = {} #dictionary to store results device_name = 'Unknown' db_conn = {'user':'sa', 'password':'password', 'host':'localhost', 'database':'ClearSNMP' } community_string = 'public' deferred_list = [] #list to help group the snmp requests outfile = open('outfile.csv','wb',0) #add headers to the outfile outfile.write ("device_name,link_oid,link_name,link_capacity_oid,link_capacity,traffic_in_oid,traffic_in,traffic_out_oid,traffic_out") #create the database connection pool dbpool = adbapi.ConnectionPool("pymssql", user=db_conn['user'], password=db_conn['password'], host=db_conn['host'], database=db_conn['database']) def main( class_handler, proxy, oids ): """Do a getTable on proxy for OIDs and store in oidStore""" df = proxy.getTable( oids, timeout=.25, retryCount=5 ) if class_handler == 'tasman': df.addCallback( tasmanResults ) else: df.addCallback( results ) #df.addCallback( exiter ) df.addErrback( errorReporter, proxy ) #df.addErrback( exiter ) return df def tasmanResults( result ): """Results 'appear' to be a nested dictionary, but it is really an object of OIDs. I figured out how to get to the OIDs by casting them as a dictionary using the built_in dict() function. Now I can iterate over all the OIDs.""" #print 'Results:' d_table_key = {} for table_key in result.keys(): #get the device name. for some reason i have to do this in a loop as just saying dict(result[table_key])['.1.3.6.1.2.1.1.5.0'] doesn't work for oid in dict(result[table_key]).keys(): if oid=='.1.3.6.1.2.1.1.5.0': device_name=str(dict(result[table_key])[oid]) d_oid = {} for oid in dict(result[table_key]).keys(): d_oid[str(oid)]=dict(result[table_key])[oid] d_table_key[str(table_key)] = d_oid d_results[device_name]=d_table_key #specify the table oids so we can match them appropriately later link_name_table = ".1.3.6.1.2.1.2.2.1.2" link_capacity_table = ".1.3.6.1.2.1.2.2.1.5" traffic_in_table = ".1.3.6.1.2.1.2.2.1.10" traffic_out_table = ".1.3.6.1.2.1.2.2.1.16" # For each link name in the table I need to get the values from the link_capacity, traffic_in and traffic_out tables and put them in the same line for i in d_results.keys(): d_row = {} #holds the column values for a row #set device_name in Row d_row['device_name'] = device_name for k in d_results[i][link_name_table]: #set link_oid and link_name in Row d_row['link_oid'] = k d_row['link_name'] = d_results[i][link_name_table][d_row['link_oid']] #lookup the capacity metric for this link_oid #create the oid to lookup d_row['link_capacity_oid'] = d_row['link_oid'].replace(link_name_table,link_capacity_table) d_row['link_capacity'] = d_results[i][link_capacity_table][d_row['link_capacity_oid']] #lookup the traffic_in metric for this link_oid #create the oid to lookup d_row['traffic_in_oid'] = d_row['link_oid'].replace(link_name_table,traffic_in_table) d_row['traffic_in'] = d_results[i][traffic_in_table][d_row['traffic_in_oid']] #lookup the traffic_out metric for this link_oid #create the oid to lookup d_row['traffic_out_oid'] = d_row['link_oid'].replace(link_name_table,traffic_out_table) d_row['traffic_out'] = d_results[i][traffic_out_table][d_row['traffic_out_oid']] #Calculate Utilization - if we can #print d_row out = d_row['device_name']+","+d_row['link_oid']+","+d_row['link_name']+","+d_row['link_capacity_oid']+","+str(d_row['link_capacity'])+","+d_row['traffic_in_oid']+","+str(d_row['traffic_in'])+","+d_row['traffic_out_oid']+","+str(d_row['traffic_out'])+'\r\n' #print out outfile.write(out) return result def errorReporter( err, proxy ): #print 'ERROR', err.getTraceback() #log the failed snmp query attempt print 'Failed to retrieve SNMP counters from agent:',proxy return err def exiter( value ): reactor.stop() outfile.close() return value def getNetworkElements(): return dbpool.runQuery("select top 10 ip, mkt, dns_name, dns_fqdn from dns where dns_type='TASMAN'") def printResult(l): for item in l: print "Fetching counters for "+item[2] #deferred_list.append(snmpSetup(item[0], 161, 'ctipublic','tasman')) ipAddress = item[0] portno = 161 community = community_string class_handler = 'tasman' print ipAddress,portno # choose random port in range 25000 to 30000 port = snmpprotocol.port() targetPort = int(portno) proxy = agentproxy.AgentProxy(ipAddress, targetPort, community = community, snmpVersion = 'v1', protocol = port.protocol, ) d_oids = {'.1.3.6.1.2.1.1':"System Tables", '.1.3.6.1.2.1.2.2.1.2':"Circuit Name", '.1.3.6.1.2.1.2.2.1.5':"Capacity", '.1.3.6.1.2.1.2.2.1.10':"Traffic In", '.1.3.6.1.2.1.2.2.1.16':"Traffic Out" } """Do a getTable on proxy for OIDs and store in oidStore""" df = proxy.getTable( d_oids, timeout=.25, retryCount=5 ) if class_handler == 'tasman': df.addCallback( tasmanResults ) else: df.addCallback( results ) df.addErrback( errorReporter, proxy ) deferred_list.append(df) return if __name__ == "__main__": import sys #start the log service from twisted.python import log from twisted.python import logfile # rotate every 100000000 bytes f = logfile.LogFile(APPNAME+str(os.getpid())+".log", "Logs", rotateLength=100000000) # setup logging to use our new logfile #log.startLogging(f) g = getNetworkElements().addCallback(printResult) dl = defer.DeferredList(deferred_list, 0, 0, 1 ) print dir(dl) dl.addCallback(exiter) reactor.run()

2 2

[Twisted-Python] Reactor is left dirty after calling twisted.mail.smtp.sendmail via trial
by Michał Pasternak 26 Feb '08

26 Feb '08

Hi, please consider attached minimal non-working code example. If I send an e-mail from a Trial testcase - using t.m.smtp.sendmail, the reactor is left dirty. Is it a bug, feature, or... how should we treat this situation? In my case, I already substituted t.m.smtp.sendmail in my tests, unless hitting a situation, where testing if my parameters to that function caused it to raise an exception. That's why I'm trying to test it this way. And, it leaves reactor dirty. Regards, -- m

4 6

[Twisted-Python] Installing Twisted on Win32 Cygwin
by Stephen Leake 26 Feb '08

26 Feb '08

I'm attempting to install Twisted using the Cygwin installation of Python. This is my first time using Python; I'm setting up a buildbot for monotone (http://www.venge.net/mtn-wiki/BuildBot). I could probably make things work with a native Win32 installation of Python, but I suspect it would be simpler in the long run to get the Cygwin version working. The first problem is the location and name of the Python library. Cygwin puts it in: /lib/python2.5/config/python2.5.dll.a But the default Twisted install scripts use this link option: -lpython2.5 Changing that to: -L/lib/python2.5/config -lpython2.5.dll lets some of the links succeed. I'm doing this by manually re-executing the link step, after editing the command line options. How do I tell the Twisted install system where the Python library is, so it gets the link command correct? I read setup.py, and poked around in some other scripts, but I don't see where this option is set. I'm used to debuging Gnu configure scripts, but Python setup is new to me. Another link problem is with TwistedRunner: gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.5.25-i686-2.5/twisted/runner/portmap.o -L/lib/python2.5/config -lpython2.5.dll -o build/lib.cygwin-1.5.25-i686-2.5/twisted/runner/portmap.dll /Apps/Twisted-2.5.0/TwistedRunner-0.2.0/twisted/runner/portmap.c:24: undefined reference to `_pmap_unset' /Apps/Twisted-2.5.0/TwistedRunner-0.2.0/twisted/runner/portmap.c:25: undefined reference to `_pmap_set' build/temp.cygwin-1.5.25-i686-2.5/twisted/runner/portmap.o: In function `portmap_unset': /Apps/Twisted-2.5.0/TwistedRunner-0.2.0/twisted/runner/portmap.c:39: undefined reference to `_pmap_unset' pmap_set and pmap_unset are defined in <rpc/pmap_clnt.h>. What library are these functions in? -- -- Stephe

3 3

[Twisted-Python] Open Spaces Revisited
by Andrew Francis 25 Feb '08

25 Feb '08

Hi Folks: Earlier someone asked whether there was going to be a PyCon 2008 Open Space for Twisted. I will be attending the PyCon 2008 conference. Since this is my first PyCon, I don't know how Open Spaces work. I am wondering if any agreement was reached? It would be neat to meet other folks interested in Twisted to exchange information and experiences. Cheers, Andrew ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping

2 1

[Twisted-Python] http proxy
by Vasil Vangelovski 25 Feb '08

25 Feb '08

I'm trying to set up a http proxy which will write redirects to the client if it requests certain uri but work as a normal proxy for others. Can this be done with the higher level proxy classes, if so, how?

3 2

[Twisted-Python] Twisted Consumes 100% CPU
by Matthew Glubb 24 Feb '08

24 Feb '08

Hi All, Hope you are all enjoying your weekends. I have a simple twisted proxy server, obviously passed on twisted.web.proxy. Periodically, it makes a call to a MySQL database, using adbapi.ConnectionPool. When I initially boot the server, I see traffic going through the proxy and everything works fine, twistd consumes less than 5% CPU. I return 24 hours later, the server is still responsive (ie. functions fine) but it consumes 100% of the CPU. I have only noticed this behaviour since I added the db connection via adbapi. I assume, because the proxy still functions well that the threads in the adbapi pool are working overtime. When I manhole in to the server, gc.collect() shows objects are being cleaned up, and gc.garbage reports nothing. I am running Twisted 2.5, Python 2.5.1, Linux 2.6, epoll, MySQL 5.0.51a. Does anyone have any experience of this problem? Is there an easy workaround? Thanks for your time :) Matt Matthew Glubb Technical Partner email: matthew.glubb(a)madebykite.com phone: 44 (0) 7715 754017 skype: mglubb Kite http://madebykite.com -- GPG: 96FF DE0E 0B7B 37F0 7F8D C54C E285 3D8F 5625 9244

2 4