Mailman 3 April 2011 - Twisted

[Twisted-Python] WaitForMultipleObjects socket limitation
by anatoly techtonik 09 Apr '11

09 Apr '11

Hi, I've made a proof of concept for asynchronous console input on Windows [1] and now I am trying to understand the limits of WaitForMultipleObjects API I've used. Documentation on win32eventreactor mentions limit for 64 objects: http://twistedmatrix.com/documents/11.0.0/api/twisted.internet.win32eventre… However, it is completely opaque what these objects are? For console handles and process handles it is quite obvious, but not for sockets. Is 64 the limit for total amount sockets opened on different ports? Is 64 the limit for connections made to a socket on specified port? 1. http://techtonik.rainforce.org/2011/03/asynchronous-input-from-windows-cons… -- anatoly t.

5 7

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion
by Stefano Debenedetti 08 Apr '11

08 Apr '11

On Apr 7, 2011, at 1:08 AM, Tim Allen wrote: > If you need a non-Turing-complete config language and rule out .ini and > XML, I'm not sure what's left. JSON, perhaps. I bet a lot of people have a deja-vu feeling about a config file syntax debate so I'll propose an alternative approach: RDF. Perhaps most people in this community will not like it, yet some might find it more fun that revamping their 2002 arguments about merits and pitfalls of various syntaxes. One of the reasons why I like RDF so much is that I can focus on what I need to express and let people pick the serialization syntax that better suits their mood, habits, tools and use-cases. I know that the use-case that's being discussed is slightly different (config files for the plugin system as opposed to config files for a specific plugin) but as an example: my twistd-plugin-driven webserver will gladly accept any of the attached configuration files, they are equivalent and there are commonly available tools to switch back and forth, including pure python ones. It could as well accept any other standard RDF serialization syntax, for example there are several other XML formats, a line-based grep-friendly syntax (NTriples) and a JSON format. The code that parses this and turns it into running twisted Services and web applications is about the same size of your average TAC file. If anybody wants to see it please email me privately, I'm not proud enough of other parts of my open source project containing it to advertise it on this list. Other than mentioning that RDF also comes with a standard query and update language and protocol (SPARQL), I won't enumerate other advantages here so let's see what some of drawbacks (and their counter-arguments) are: 1) it's not widely known yet (but so was XML in 2000 and JSON in 2002 and INI in 2011) 2) it would require to add a dependency for an RDF parser (people often argued the same way about XML, remember when libxml2 became a Gnome dependency? RDF is now becoming a requirement of Gnome and KDE...) 3) it's not python (yet the several python object-RDF-mapper libraries available seem to me much easier to use and way more simple than SQLAlchemy so I already switched from pickle to RDF whenever I want to serialize some object graph, BTW it's also safer and hand-editable) Sorry if I went too off-topic, ciao ste

4 4

[Twisted-Python] Asynchronous context in Twisted
by Fantix King 07 Apr '11

07 Apr '11

Hi, I tried to make python.context work in asynchronous code between main loops. Anyone has similar experience to share please? Not sure if I am rebuilding a wheel :P http://code.google.com/p/little-site/source/browse/littlesite/custom_reacto… BR, Fantix.

4 4

Re: [Twisted-Python] Twisted Plugins - Implementation Discussion
by Tristan Seligmann 07 Apr '11

07 Apr '11

On 7 Apr 2011 06:56, "Glyph Lefkowitz" <glyph(a)twistedmatrix.com> wrote: >> - Each section blah blah terrible user interface stuff about 'sections' and other misfeatures of ini files. > > > I don't want a solution that is hard-coded to deal with the metadata that 'twistd' specifically needs, as Twisted plugins are already used for more than just twistd plugins, and I'd like them to be used for even more. An interface that limits the metadata to ConfigParser sections would make it awkward to fit into a management GUI or web page. I'd like to jump in with a +1000 here. I have literally written over 1000 plugins[1] using twisted.plugin, and not one of them was a twistd plugin. I would very much like to take advantage of any improvements in this area in my own projects. [1] http://bazaar.launchpad.net/~dosage-dev/dosage/trunk/files/head:/dosage/plu…

1 0

[Twisted-Python] Persisted protocol?
by anatoly techtonik 06 Apr '11

06 Apr '11

Hi again, >From http://twistedmatrix.com/documents/current/core/howto/udp.html """As you can see, the protocol is registered with the reactor. This means it may be persisted if it's added to an application...""" What does 'persisted` here mean? -- anatoly t.

6 10

Re: [Twisted-Python] AMP Client disconnecting from server on high memory usage
by Dan-Cristian Bogos 06 Apr '11

06 Apr '11

Hey Jean-Paul, For record's sake, I thought I announce here that my issue got fixed. The problem was that I was loading the data in startService method and it looks like there are some timers which prevent the method to block for a longer period of time. Loading the data in __init__ of service.Service object did the trick and I have no problem anymore with large data. Have a good one! DanB

1 0

[Twisted-Python] Twisted 11.0 and Foolscap incompatibility
by Bill Noon 06 Apr '11

06 Apr '11

It looks like the new ssl/tls code doesn't work with Foolscap. The ConnectionMixin is treating the self.protocol object as a transport and calls write() and loseConnection() on it. Here is a traceback: Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/slicers/root.py", line 107, in send d.callback(None) File "/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 361, in callback self._startRunCallbacks(result) File "/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 455, in _startRunCallbacks self._runCallbacks() File "/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) --- <exception caught here> --- File "/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py", line 216, in produce self.pushSlicer(slicer, obj) File "/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py", line 352, in pushSlicer openID = self.sendOpen() File "/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py", line 481, in sendOpen int2b128(openID, self.transport.write) File "/usr/lib64/python2.6/site-packages/foolscap-0.6.1_-py2.6.egg/foolscap/banana.py", line 24, in int2b128 stream(chr(0)) File "/usr/lib64/python2.6/site-packages/Twisted-11.0.0_r31548-py2.6-linux-x86_64.egg/twisted/internet/_newtls.py", line 176, in write self.protocol.write(bytes) exceptions.AttributeError: 'Broker' object has no attribute 'write' Any workarounds or suggestions? Reverting to Twisted 10.2 works for now. --Bill Noon Northeast Regional Climate Center Cornell University

2 5

Re: [Twisted-Python] [Twisted-web] Creating a hybrid server in Twisted
by Glyph Lefkowitz 06 Apr '11

06 Apr '11

On Apr 4, 2011, at 7:51 PM, David Bolen wrote: > Jashank Jeremy <jashank.jeremy(a)optusnet.com.au> writes: > >> So I'm pretty much stuck in a rut. I don't want to totally reinvent the >> wheel just to be able to protocol-switch; I'd prefer to make use of >> existing code from Twisted. How do I hijack Twisted.Web to add protocol >> switching? > > To insert your own protocol code, you want to set the "protocol" > attribute of your Site (or HTTPFactory if using that directly) > instance to your own "hybrid" protocol class. It would be slightly better to override buildProtocol, because that way you can call the superclass's implementation more idiomatically. But, the idea is the same. Note that we'd really like to have protocol switch functionality built into Twisted proper so that you don't have to jump through all these hoops. Please feel free to contribute patches to fix this ticket: <http://twistedmatrix.com/trac/ticket/3204>.

1 0

[Twisted-Python] Gtk2Reactor: what are the reasons for avoiding GLib functions?
by Jason Heeris 05 Apr '11

05 Apr '11

I'd like to try and get to the bottom of the various problems with Twisted + GTK + Windows (mainly issue #4932[1], but see also issues #4376[2], #4862[3], #1759[4]). On issue #4376 there's a comment from Exarkun: "glib handles timeouts much less efficiently than any of the existing Twisted reactors. Letting glib take care of these would probably make the glib-based reactors unusable for applications with many timeouts." Is there a benchmark or piece of code somewhere that verifies this (or at least gives some sort of demonstration of "unuseable")? If not, is there some alternative implementation lying around in version control that tried to use GLib's mainloop functions and was thrown out for performance reasons? I ask because my first approach to fixing this would be to ignore what's already there and try to write something that's quite closely coupled to the GLib/GTK mainloop. But it sounds like this was already tried and abandoned, and I want to be able to make sure I'm not (a) writing something that was already written, or (b) performs too poorly to be of any use. It would be good to have a metric available. Cheers, Jason [1] http://twistedmatrix.com/trac/ticket/4932 [2] http://twistedmatrix.com/trac/ticket/4376 [3] http://twistedmatrix.com/trac/ticket/4862 [4] http://twistedmatrix.com/trac/ticket/1759

4 4

[Twisted-Python] @inlinecallbacks and AlreadyCalledError in test cases
by Brad Milne 05 Apr '11

05 Apr '11

Hi all I have recently started switching to trial.unittest from python's own. The trouble I'm experiencing is when a timeout occurs in my test, it errbacks(). Then the @inlineCallbacks decorator sees the error and errbacks(). But then a second @inlineCallback in the chain subsequently sees *that* errback and tries to errback itself. This results in AlreadyCalledError. In the test setup, various services are started. These are tracked and then shutdown again in the teardown. Also, there is some polling that happens as part of the tests (waiting on db activities, for example). These use deferLater calls, which are also tracked and torn down in the teardown. I've tried _suppressAlreadyCalled in various places to no avail. Below is a simple example that shows the problem. Many thanks Brad <code> from twisted.trial.unittest import TestCase from twisted.internet import task, reactor from twisted.internet.defer import inlineCallbacks #import twisted #twisted.internet.base.DelayedCall.debug = True #twisted.internet.defer.setDebugging(True) class Test1(TestCase): def setUp(self): # timeout test in 1 second self.timeout = 1 self.jobs = [] self.addCleanup(self._tearDown) def _tearDown(self): for d in self.jobs: if d and not d.called: d.cancel() @inlineCallbacks def _waitForChange(self): # do stuff d = task.deferLater(reactor, 0.5, lambda : None) self.jobs.append(d) yield d self.jobs.remove(d) # do more stuff @inlineCallbacks def testHere(self): # do stuff yield self._waitForChange() # do more stuff # This one passes OK #def testHere(self): # return self._waitForChange() if __name__ == "__main__": import sys from twisted.scripts import trial sys.argv.extend([sys.argv[0]]) trial.run() </code>

2 2