Mailman 3 September 2010 - Twisted

[Twisted-Python] Strange error: SQL-Server tries to rollback
by Pantelis Theodosiou 11 Sep '10

11 Sep '10

I have a simple INSERT INTO statement sent to SQL-Server using pymssql module. I use the runOperation method to send the query and I get this strange error. The data goes through OK and is written in the database but SQL-Server tries to rollback. Perhaps this is a pymssql problem. In the http://code.google.com/p/pymssql/wiki/PymssqlExamples page, they state: import pymssql conn = pymssql.connect(host='SQL01', user='user', password='password', database='mydatabase') cur = conn.cursor() cur.execute('CREATE TABLE persons(id INT, name VARCHAR(100))') cur.executemany("INSERT INTO persons VALUES(%d, %s)", \ [ (1, 'John Doe'), (2, 'Jane Doe') ]) conn.commit() # you must call commit() to persist your data if you don't set autocommit to True But how can I use the con.commit method with runOperation and runQuery that I use in my application? thank you, Pandelis Theodosiou Here is the error: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.6/dist-packages/twisted/python/threadpool.py", line 210, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib/python2.6/dist-packages/twisted/enterprise/adbapi.py", line 436, in _runInteraction conn.rollback() File "/usr/lib/python2.6/dist-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() File "/usr/lib/pymodules/python2.6/pymssql.py", line 496, in rollback raise OperationalError, "cannot roll back transaction: " + e[0] pymssql.OperationalError: cannot roll back transaction: SQL Server message 3903, severity 16, state 1, line 1: The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION. DB-Lib error message 3903, severity 16: General SQL Server error: Check messages from the SQL Server

4 7

Re: [Twisted-Python] Twisted-Python Digest, Vol 78, Issue 15
by lance.barisdale＠gmail.com 11 Sep '10

11 Sep '10

Sent from my HTC on the Now Network from Sprint! ----- Reply message ----- From: twisted-python-request(a)twistedmatrix.com Date: Fri, Sep 10, 2010 12:09 pm Subject: Twisted-Python Digest, Vol 78, Issue 15 To: <twisted-python(a)twistedmatrix.com> Send Twisted-Python mailing list submissions to twisted-python(a)twistedmatrix.com To subscribe or unsubscribe via the World Wide Web, visit http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python or, via email, send a message with subject or body 'help' to twisted-python-request(a)twistedmatrix.com You can reach the person managing the list at twisted-python-owner(a)twistedmatrix.com When replying, please edit your Subject line so it is more specific than "Re: Contents of Twisted-Python digest..." Today's Topics: 1. Re: Strange error: SQL-Server tries to rollback (Paul Goins) 2. reactor for both udp and tcp/ timer issue (belk Dj) 3. Re: twisted cred: why does avatarId need to be a str? (Laurens Van Houtven) 4. Re: reactor for both udp and tcp/ timer issue (Pantelis Theodosiou) ---------------------------------------------------------------------- Message: 1 Date: Sat, 11 Sep 2010 01:39:13 +0900 From: Paul Goins <general(a)vultaire.net> Subject: Re: [Twisted-Python] Strange error: SQL-Server tries to rollback To: twisted-python(a)twistedmatrix.com Message-ID: <4C8A5F31.8020305(a)vultaire.net> Content-Type: text/plain; charset=UTF-8; format=flowed Thanks for the code example. I can't offer to take this work over, but maybe I can make a comment. There's one part that stands out to me. > try: > deferred = self.dbpool.runOperation(sql) > #print("DATA sent") > except Exception as e: > print("error in insertDATA") > print(e) > return > return deferred It seems like maybe you have a misunderstanding about how Deferreds generally work in Twisted. (Been there myself.) Basically, the above try/except block won't work to catch errors from most Deferreds... well, at least not without some extra magic. You really should read the Deferred section of the Twisted documentation to understand how errors are handled. Docs are here: http://twistedmatrix.com/documents/current/core/howto/defer.html ---- Basically, to "fix" the above code's error catching, you have two choices: 1. You can add an errback to the deferred. This is the "standard" Twisted way, and would replace the try/except block entirely. 2. You can use the @inlineCallbacks decorator (from twisted.internet.defer), and yield on the Deferred. This is easier, and it allows try/except blocks, but there's some gotchas. inlineCallbacks is what I used when I was learning Twisted, and you may want to try that for now. But please understand that it hides details about how Deferreds and callbacks really work. When you find time, read the Deferred docs. ---- Best of luck, - Paul Goins ------------------------------ Message: 2 Date: Fri, 10 Sep 2010 16:55:01 +0000 (GMT) From: belk Dj <d_belkhiter(a)yahoo.fr> Subject: [Twisted-Python] reactor for both udp and tcp/ timer issue To: twisted-python(a)twistedmatrix.com Message-ID: <228404.72208.qm(a)web24406.mail.ird.yahoo.com> Content-Type: text/plain; charset="utf-8" I need to communicate with servers over tcp and also communicate with other servers over udp. It seems not possible to launch two reactor objects one for tcp and one for udp. The code after a reactor.run is not called. How can this can be solved ? One more thing that deals with timer. When i send a message to a server if after some time i don't get an acknowledge i re send the message. And that x times. To do that i use Timer. When i send the message i start the timer if the acknowledge comes before the timeout i cancel the timer. If i don't receive this acknwledge in time i cancel the timer i re send the previous message and i re start the timer. In this two cases the program fails with the message the timer is still started. What is the issue ? Regards

1 0

[Twisted-Python] reactor for both udp and tcp/ timer issue
by belk Dj 10 Sep '10

10 Sep '10

I need to communicate with servers over tcp and also communicate with other servers over udp. It seems not possible to launch two reactor objects one for tcp and one for udp. The code after a reactor.run is not called. How can this can be solved ? One more thing that deals with timer. When i send a message to a server if after some time i don't get an acknowledge i re send the message. And that x times. To do that i use Timer. When i send the message i start the timer if the acknowledge comes before the timeout i cancel the timer. If i don't receive this acknwledge in time i cancel the timer i re send the previous message and i re start the timer. In this two cases the program fails with the message the timer is still started. What is the issue ? Regards

2 1

[Twisted-Python] conchoctopus: APIs for working with multiple SSH servers
by Kuba Kończyk 10 Sep '10

10 Sep '10

Hi Everyone, Conchoctopus was started as a library that glue together various parts of twisted.conch so that it can be easily used in everyday work with SSH servers.Currently there are two APIs you can use. First one takes care of most low level details of t.c such as authentication and error handling and provides ready to use classes that you can use in your twisted code.I would like to make it as usable for other twisted users as possible. There's also high level API where all deferred management machinery is hidden. User have to create a python module with a Task class that specifies 'what to do' using inlineCallbacks and Config class with a list of servers to work on.This modules can then be run with an simple console utility. Both APIs are documented and armed with test suite.Please play around and let me know what can be improved or added. Code is available from pypi: http://pypi.python.org/pypi/conchoctopus/ Thanks, Kuba

1 0

[Twisted-Python] How to hook onto or append functionality to a class (from another class)?
by Einar S. Idsø 10 Sep '10

10 Sep '10

Hi, I am trying to make one class/service hook onto another class/service to add to its functionality without disturbing the original class/service. My situation is this: I currently have a class which at specific intervals queries one server per instance (via t.u.getProcessOutput()) for some basic information. Each instance has a method which analyses its returned data and stores the result it to ts internal data structure. My dilemma is that I wish to leave this class alone, but I also wish to be able to add functionality from external classes based on the result of said method. For instance: If the method analysing the result from getProcessOutput() discovers a certain user on the server being queried, I wish to store the data to a mysql database. Under other circumstances I wish to send a notification to myself over MSN or via mail. I can easily envision myself wanting to respond to the results with other actions as well in the future. How can this be achieved? I can of course modify the original class so that I pass an adbapi.ConnectionPool as optional argument to __init__() and modify the method to check for the existence of such an object before querying the database. I would then also need to add another optional argument which is whatever object needs to be passed to allow the class to send a message to MSN, and still more optional arguments for any such new service I'd like to add. However, as mentioned I'd rather not modify the original class, and doing this seems like too much of a hack. Or is it? Perhaps inheritance is the key, although that would require me to modify existing methods which I know to be working just to tack on some additional functionality at the end. What are my options here? I took a look at t.p.hook, and it seems like it is perfect for the job: After having instantiated my "global" adbapi.ConnectionPool(), I can just add a pre- or post-hook to any class's method from which I'd like to store data. That way I can make a db-instance which hooks in to any part of my application and stores any result imaginable from any method. However, it seems that t.p.hook is very rarely used, so perhaps there is a better way of achieving what I want? I hope this was somewhat comprehensible, and that there is an elegant solution out there. :) Cheers, Einar

2 3

[Twisted-Python] txdev: utility for working on Twisted
by Jonathan Lange 10 Sep '10

10 Sep '10

Hello everyone, I wrote a tool to help do Twisted development, especially with Bazaar. You can get the code with: $ bzr branch lp:~jml/+junk/twisteddev Basically, it's a few things to help with the interactions between branches and tickets. It might well work with Subversion, since it's mostly just putting strings together. There's some support for Bazaar configuration. It might even work with other Twisted-like projects, (e.g. Divmod projects). There are some examples below. I hope they are self-explanatory. I'll try to fix bugs if you find them. Features, well, you're on your own. Let me know if it helps you. cheers, jml $ txdev --help Usage: txdev [options] Options: -h, --help Display this help and exit. --trac-url= The URL of the Trac instance --repo-url= The Bazaar repository with the branches. --version Commands: branch Get the branch for a ticket open-ticket Open ticket in web browser reviews Display tickets needing review $ txdev branch 4616 http://svn.twistedmatrix.com/bzr/Twisted/branches/remove-lineno-patch-4616 $ bzr branch `txdev branch 4616` $ cd remove-lineno-patch-4616 $ txdev open-ticket Created new window in existing browser session. $ txdev reviews #4416 - IRCClient: Detect and handle lines that are too long #4536 - Credentials materials are compared unsafely throughout Twisted #4567 - Improve lore2sphinx buildbot results for `projects/core/development` #4568 - Improve lore2sphinx buildbot results for `projects/core/howto` (except ‘tutorial/’) #4616 - trial documentation says --coverage requires python 2.3.3 #4617 - Make t.i.a._SocketCloser a new style class #4623 - Missing interface IQueue #4632 - ability to cascade canceling inlineCallbacks's deferred #4641 - twisted.words.im.ircsupport.IRCProto sends USER command with the wrong data $ txdev reviews --branches-only http://svn.twistedmatrix.com/bzr/Twisted/branches/handle-long-IRC-lines-4416 http://svn.twistedmatrix.com/bzr/Twisted/branches/password-comparison-4536-2 http://svn.twistedmatrix.com/bzr/Twisted/branches/improve-lore2sphinx-outpu… http://svn.twistedmatrix.com/bzr/Twisted/branches/lore2sphinx-projects-core… http://svn.twistedmatrix.com/bzr/Twisted/branches/remove-lineno-patch-4616 http://svn.twistedmatrix.com/bzr/Twisted/branches/newstyle-socketcloser-4617 http://svn.twistedmatrix.com/bzr/Twisted/branches/ircsupport-user-4641

2 1

[Twisted-Python] How to get original error message in failure object
by Pet 09 Sep '10

09 Sep '10

Hello, if an error occurs in HTTP11ClientProtocol request method (twisted.web._newclient) then error is stored in RequestGenerationFailed Exception and wrapped in Failure object: def ebRequestWriting(err): if self._state == 'TRANSMITTING': self._state = 'GENERATION_FAILED' self.transport.loseConnection() self._finishedRequest.errback( Failure(RequestGenerationFailed([err]))) else: log.err(err, "foo") but if I print detailed traceback of failure in my code it doesn't show all information it contains Original Failure (err in function above): [Failure instance: Traceback: <type 'exceptions.TypeError'>: Data must not be unicode This is detailed traceback, where I don't see the message "Data must not be unicode" *--- Failure #7 (pickled) --- Failure: twisted.web._newclient.RequestGenerationFailed: [<twisted.python.failure.Failure <type 'exceptions.TypeError'>>] *--- End of Failure #7 --- None Is there a way to get original error message and where it was raised in Twisted code? Thanks a lot!

2 2

[Twisted-Python] connection to many servers
by dbcourriel-4all＠yahoo.fr 07 Sep '10

07 Sep '10

Hi, i need to connect to many servers so i've found an example that shows how to connect to many servers see the source below. But i need to get the way to retrieve each server in order to see the satus of the connection, to send to it a message. Is there an connection object with sendData action, getCxStatus for example Thanks class ClntProtocol(protocol.Protocol): def sendData(self, data): if data: self.transport.write(data) else: self.transport.loseConnection() def connectionMade(self, data=''): connectedClientlist.append(self) if data: self.sendData(data) def dataReceived(self, data): if data: self.sendData(data) def closeConnection(self): pass class cxFactory(protocol.ClientFactory): protocol = ClntProtocol def __init__(self): self.deferred = defer.Deferred( ) def clientConnectionFailed(self, connector, reason): self.deferred.errback(reason) def svcConnect(address): clntFactory = cxFactory() reactor.connectTCP(address[0], int(address[1]), clntFactory) return clntFactory.deferred def connectAll(addresses): logger.info("connect all") servers = [svcConnect(address) for address in addresses] defer.DeferredList(servers, consumeErrors=True).addCallback(handleAllResults, addresses) reactor.run( )

2 2

[Twisted-Python] connection to many servers
by dbcourriel-4all＠yahoo.fr 07 Sep '10

07 Sep '10

Hi, i need to connect (tcp) to many servers. so i wrote def connectAll(addresses): logger.info("connect all") servers = [svcConnect(address) for address in addresses] defer.DeferredList(servers, consumeErrors=True).addCallback(handleAllResults, addresses) reactor.run( )

1 0

[Twisted-Python] Memory Usage Problem
by Ozgur Pekcagliyan 06 Sep '10

06 Sep '10

Hi All, I have a problem which is that memory usage of twisted is increasing but not decreasing over time. After few days, it reaches 520MB. You may find the codes below; How may I reduce the usage of memory? Twisted version: 8.2.0 Python version: 2.4.3 [root@labris init.d]# cat /usr/bin/labris-flyng.tac #!/usr/bin/python import os from twisted.application import internet, service from twisted.internet import reactor from twisted.enterprise import adbapi from twisted.plugin import getPlugins from twisted.python import log import labris.flyng.iflyng as iflyng import labris.flyng.config as config import labris.flyng.plugins as pplugins import labris.flyng.protocols as flyng_protocols from labris.flyng.config import flyng_config # Database type to package dictionary DB_PACKAGE_DICT = { "postgresql" : "psycopg2" } def initDatabase(config): """ Create database connection pool """ dbtype = config.getDBType() dbpackage = DB_PACKAGE_DICT.get(dbtype, None) if not dbpackage: print "Database type % is not supported yet!" %(dbtype) sys.exit(1) if dbtype == "mysql": dbpool = adbapi.ConnectionPool( dbpackage, config.getDBHost(), config.getDBUser(), config.getDBPassword(), config.getDBName() ) else: dbpool = adbapi.ConnectionPool( dbpackage, database=config.getDBName(), user=config.getDBUser(), password=config.getDBPassword(), host=config.getDBHost() ) dbpool.connect() return dbpool def setupPlugins(config, dbpool): """ Initialize and setup plugins """ for plugin in getPlugins(iflyng.IFlyngPlugin, pplugins): print "* Plugin %s" %(plugin.name) if not config.isEnabled(plugin.name): print " - Not enabled" continue plugin.init(config, dbpool) for type, port in config.getListeningPorts(plugin.name): port = port.replace('"', "") if type == "tcp": print " -> TCP port: %s" %(port) factory = flyng_protocols.FlyngServerFactory(plugin) internet.TCPServer( int(port), factory ).setServiceParent(application) elif type == "udp": print " -> UDP port: %s" %(port) internet.UDPServer( int(port), flyng_protocols.FlyngUDPProtocol(plugin) ).setServiceParent(application) elif type == "unix": print " -> UNIX stream: %s" %(port) if os.path.exists(port): os.unlink(port) factory = flyng_protocols.FlyngServerFactory(plugin) internet.UNIXServer( port, factory ).setServiceParent(application) else: raise "connection type %s is not supported" %(port) #log.startLogging(open("/var/log/labris-flyng.log", "a")) config = config.Configuration() uid, gid = config.getRunAs() application = service.Application('labris-flyng', uid, gid) dbpool = initDatabase(config) setupPlugins(config, dbpool) ------------------------------------------------------------------- [root@labris init.d]# cat /usr/bin/labris-pyng.tac #!/usr/bin/python """ A syslog-ng to database mapping service """ import os from twisted.internet import protocol, reactor, defer from twisted.application import internet, service from twisted.python import log from labris.logutils import factories from labris.logutils.config import pyng_config class LogFactory(protocol.ServerFactory): pass class UnixLogFactory(protocol.ServerFactory): pass def openUNIXStreams(configuration, application): for logtype, stream in configuration.getUnixStreams(): try: if not pyng_config.getboolean(logtype.strip(), "enabled"): print "Warning: UNIX stream: ", logtype, " is DISABLED" continue print "\nPlugin: %s [UNIX-Stream(%s)]" %(logtype, stream) factory = UnixLogFactory() factory.protocol = factories.ProtocolFactory.getLogProtocol(logtype.strip(), "stream") # delete old stream if os.path.exists(stream): os.unlink(stream) internet.UNIXServer(stream.strip(), factory).setServiceParent(application) except factories.NoSuchProtocolException, e: print "Error: no such protocol:", e.getProtoname() for logtype, sock in configuration.getUnixDgrams(): try: if not pyng_config.getboolean(logtype.strip(), "enabled"): print "Warning: UNIX dgram: ", logtype, " is DISABLED" continue print "\nPlugin: %s [UNIX-Datagram(%s)]" %(logtype, stream) protocol = factories.ProtocolFactory.getLogProtocol(logtype.strip(), "datagram") # delete old stream if os.path.exists(sock): os.unlink(sock) internet.UNIXDatagramServer(sock.strip(), protocol()).setServiceParent(application) except factories.NoSuchProtocolException, e: print "Error: no such protocol:", e.getProtoname() def openSockets(configuration, application): for logtype, hostname, port in configuration.getTCP(): try: if not pyng_config.getboolean(logtype.strip(), "enabled"): print "Warning: TCP socket: ", logtype, " is DISABLED" continue print "\nPlugin: %s [TCP => port:%d]" %(logtype, int(port)) factory = LogFactory() factory.protocol = factories.ProtocolFactory.getLogProtocol(logtype.strip(), "stream") internet.TCPServer(int(port), factory).setServiceParent(application) except factories.NoSuchProtocolException, e: print "Error: no such protocol:", e.getProtoname() for logtype, hostname, port in configuration.getUDP(): try: if not pyng_config.getboolean(logtype.strip(), "enabled"): print "Warning: UDP socket: ", logtype, " is DISABLED" continue print "\nPlugin: %s [UDP => port:%d]" %(logtype, int(port)) protocol = factories.ProtocolFactory.getLogProtocol(logtype.strip(), "datagram") internet.UDPServer(int(port), protocol()).setServiceParent(application) except factories.NoSuchProtocolException, e: print "Error: no such protocol:", e.getProtoname() #log.startLogging(open("/var/log/labris-pyng.log", "a")) from labris.logutils.dao.pyngdb import PyNgDB PyNgDB.initDatabase(pyng_config.bind()) application = service.Application('labris-pyng', pyng_config.uid(), pyng_config.gid()) openUNIXStreams(pyng_config, application) openSockets(pyng_config, application)

3 3