Mailman 3 April 2011 - Twisted

[Twisted-Python] Anyone ever used successfully sqlanydb together with Twisted and adbapi?
by Hynek Schlawack 27 Apr '11

27 Apr '11

Hi, I've essentially written a whois daemon that serves via TCP as well as via web, fetching its data from a sqlanywhere 12 server, using the official sqlanydb driver. Everything works fine except for occasional SIGABRTs (or inside of gdb: SIGSEGVs inside of the sqlanywhere binary driver) in the first query. If that one works, it keeps running. However it's impossible to run tests as at some point it always crashes while doing queries because I recreate the pool for every test. Essentially I get a domain record from the database and fetch some related data afterward. I tried to implement it using DeferredList and using runInteraction and it always aborts before doing the second query. I've already posted some code and gdb backtraces to stackoverflow: http://stackoverflow.com/questions/5790435/python-twisted-sqlanydb-abort maybe there's some helpful data I missed here. To stress it: The lookup of the domain record usually works. It's the following defers that crash my application. My essential question is: Has someone already used Twisted and SQLAnywhere together successfully? Is it me or is it sqlanywhere or even Twisted? What are my options? TIA, -h

1 0

[Twisted-Python] Instrumenting Reactors
by Paul Thomas 26 Apr '11

26 Apr '11

I have a reactor which is getting busier over time and I'd like to find out where the cycles are going. Using the profiler isn't really practical on a server running for days, so I'd like to instrument the reactor to show me which I/O events are taking the longest to deal with. Should I just hack into the reactor somewhere? Or is there something sitting in a library I haven't seen that will help with this? Paul.

8 7

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 24 Apr '11

24 Apr '11

1 0

[Twisted-Python] Twisted Plugins - Implementation Discussion
by Stephen Thorne 22 Apr '11

22 Apr '11

G'day, So Glyph and I had a discussion about the architecture and implementation of plugins on IRC this week, I raised some issues that I've seen with implementing plugins in that discussion, and he said that I should take the discussion to the list because IRC wasn't the right place for it. First of all, a quick discussion of the current plugin architecture, so that we're on the same page. 'twistd' automatically imports python modules from under twisted/plugins relative to sys.path[1], or it loads a cache of those plugins from dropin.cache, or it loads a cache of those plugins from dropin.cache. The reason it loads all of those plugins is so that 'tapname' and 'description' can be grabbed out of all of the serviceMaker attributes of all those modules. Then running 'twistd' shows a helpful list of commands, and 'twistd $tapname' uses the correct serviceMaker to start whatever service is specified. Part of the discussion was about how to rewrite this in such a way that no python code needs to be run in order to discover all the tapname+description combinations that are available to twistd, this is because of a perceived performance and sanity deficit in using 'twistd'. In the course of the discussion I raised several things that I consider annoyances in the twisted plugin system. I will repeat them here. First, the reason i use twisted plugins is because they're the a way to easily do sensible things with logging, daemonisation and interaction from init.d files. * The number of imports required to compose a plugin is annoying. 2 interfaces from two different packages, plus needing zope.interface.implements. * I've never liked the twisted arg parser, I use it only grudingly, it would be nice to be able to throw argv at my make_service call. * The entire task of having this python plugin is to link up the metadata with a more or less standard * It's very easy to accidentally make your plugin load your package for every other twistd daemon running out of the same plugin cache. * The practice of putting a module under twisted/plugins/$mymodulehere.py upsets my equilibrium, the only reason I tolerate this kind of thing is that I install my python code via rpms and I automatically get installation, ownership and uninstallation done in a stable manner. For your consideration, and (constructive) critcism, here is a twisted plugin that is nearly identical to 6 that I have running in production: from zope.interface import implements from twisted.python import usage from twisted.plugin import IPlugin from twisted.application.service import IServiceMaker class Options(usage.Options): optFlags = [['debug', 'd', 'Emit debug messages']] class ExampleServiceMaker(object): implements(IServiceMaker, IPlugin) tapname = 'example' description = 'Example Twistd Plugin' options = Options def makeService(self, options): from examplepackage.examplemodule import make_service return make_service(debug=options['debug']) serviceMaker = ExampleServiceMaker() >From these 19 lines of code there are 4 things that are relevent: tapname = example description = 'Example Twistd Plugin' options = {'debug':True} if args == ['--debug'] else {'debug':False} service = examplepackage.examplemodule.make_service(**options) Only 2 of which are relevent for running a twistd daemon other than 'example' So the goal of my post to this mailing list is: * I would like glyph's goal of having less arbitary code executed at twistd launch time to become a realisation, * I would like the process of creating a twisted plugin to be less of a cut+paste+fill-in-blanks hassle. -- Regards, Stephen Thorne Development Engineer Netbox Blue

15 34

[Twisted-Python] Understanding deferred and error handling
by David 22 Apr '11

22 Apr '11

Hi, I have a hard time figuring out error handling with deferred in twisted. More exactly, I don't understand how to always get meaningful tracebacks to understand where the error actually happened. For a simple example: import sys import twisted.web.client from twisted.internet import defer from twisted.internet import reactor from twisted.python import log def remote_call(): # No process bound to 8083 -> connection refused d = twisted.web.client.getPage("http://localhost:8083") return d def main(): d = remote_call() def _stop(arg): reactor.stop() d.addBoth(_stop) log.startLogging(sys.stdout) reactor.callWhenRunning(main) reactor.run() This will simply print no error in the log: 2011-04-20 12:37:40+0900 [-] Log opened. 2011-04-20 12:37:40+0900 [-] Starting factory <HTTPClientFactory: http://localhost:8083> 2011-04-20 12:37:40+0900 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://localhost:8083> 2011-04-20 12:37:40+0900 [-] Main loop terminated. This already bothers me at a fundamental level, because it means it is very easy to "swallow" errors without being aware of it. Is there a "systematic" solution to this issue, or am I condemned to handle errors systematically everywhere in my code ? Now, if I add an errback for logging purpose: def main(): d = remote_call() def _stop(arg): reactor.stop() d.addErrback(log.err) d.addBoth(_stop) I get something like: 2011-04-20 12:38:35+0900 [-] Log opened. 2011-04-20 12:38:35+0900 [-] Starting factory <HTTPClientFactory: http://localhost:8083> 2011-04-20 12:38:35+0900 [HTTPPageGetter,client] Unhandled Error Traceback (most recent call last): Failure: twisted.web.error.Error: 404 Not Found 2011-04-20 12:38:35+0900 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://localhost:8083> 2011-04-20 12:38:35+0900 [-] Main loop terminated. I do get an error, but I don't get a traceback. Interestingly enough, if I use printTraceback: def main(): d = remote_call() def _stop(arg): reactor.stop() def log_error(failure): log.err(failure.printTraceback()) return failure d.addErrback(log_error) d.addBoth(_stop) I don't get any traceback either. I found quite a few documents on how to handle errors in twisted, but I did not see much information on how to at least make sure unexpected errors are being shown (and fixed later). cheers, David

7 8

[Twisted-Python] Running twistd with a compiled file
by Albert Brandl 21 Apr '11

21 Apr '11

Hi! It is sometimes necessary to deploy an application without the sources. But when I try to start the twistd daemon with a compiled Python file, it raises the following traceback: $ twistd -ny application.pyc Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/application/app.py", line 631, in run runApp(config) File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/scripts/twistd.py", line 23, in runApp _SomeApplicationRunner(config).run() File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/application/app.py", line 374, in run self.application = self.createOrGetApplication() File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/application/app.py", line 439, in createOrGetApplication application = getApplication(self.config, passphrase) --- <exception caught here> --- File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/application/app.py", line 450, in getApplication application = service.loadApplication(filename, style, passphrase) File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/application/service.py", line 400, in loadApplication application = sob.loadValueFromFile(filename, 'application', passphrase) File "/usr/local/lib/python2.6/dist-packages/Twisted-10.0.0-py2.6-linux-i686.egg/twisted/persisted/sob.py", line 210, in loadValueFromFile exec fileObj in d, d exceptions.SyntaxError: Non-ASCII character '\xd1' in file application.pyc on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details (application.pyc, line 1) Failed to load application: Non-ASCII character '\xd1' in file application.pyc on line 1, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details (application.pyc, line 1) This behaviour was observed with twistd 10.2.0. Of course, it's possible to move most of the logic to other (compiled) files, but I'd prefer if the application itself could also be deployed in compiled form. Is this a known problem? Can you suggest a workaround? Best regards, Albert -- Albert Brandl Weiermayer Solutions GmbH | Abteistraße 12, A-4813 Altmünster phone: +43 (0) 720 70 30 14 | fax: +43 (0) 7612 20 3 56 web: http://www.weiermayer.com

5 5

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 18 Apr '11

18 Apr '11

2 1

[Twisted-Python] Python 2.7 on 64 bit Fedora 11
by exarkun＠twistedmatrix.com 17 Apr '11

17 Apr '11

Greetings, We've had a Fedora 11 slave for a little while now (hosted by Flumotion, thanks!). It's been green just about since it came up, so I think we should call it a supported platform now. It will show up on the buildbot supported platforms page and force- builds.py will include it (once you update your checkout). Jean-Paul

1 0

[Twisted-Python] Announcing pyOpenSSL 0.12
by exarkun＠twistedmatrix.com 12 Apr '11

12 Apr '11

Exciting news everyone, I have just released pyOpenSSL 0.12. pyOpenSSL provides Python bindings to a number of OpenSSL APIs, including certificates, public and private keys, and of course running TLS (SSL) over sockets or arbitrary in- memory buffiers. This release fixes an incompatibility with Python 2.7 involving memoryviews. It also exposes the "info callback" constants used to report progress of the TLS handshake and later steps of SSL connections. Perhaps most interestingly, it also adds support for inspecting arbitrary X509 extensions. http://python.org/pypi/pyOpenSSL - check it out. Jean-Paul

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 10 Apr '11

10 Apr '11

1 0