Mailman 3 January 2009 - Twisted

Re: [Twisted-Python] high server loads on FreeBSD - does kqueue reactor help?
by Maarten ter Huurne 20 Jan '09

20 Jan '09

On Tuesday 20 January 2009, Werner Thie wrote: > With a four core, four twisted http server processes (nevow/athena on > top), dual 100 MBps interfaces, all four cores showing CPU loads of 50% > plus, I observe the following degradation in the network traffic with > tcpdump: > > - 'TCP checksum incorrect' raising to almost 25% of all packets observed > from one client IP > - TCP Ack'ed lost segment raising to 0.5% of all packets observed Incorrect checksums sounds like a hardware problem. Or possibly a driver problem if the checksum calculation is done by the CPU instead of the network card. In any case it is not possible for user space code like Twisted to generate incorrect TCP checksums. Bye, Maarten

1 0

[Twisted-Python] Newbee searching test script for web-servers
by Sven-Erik Tiberg 20 Jan '09

20 Jan '09

Hi Been lurcking this list for some time w/o contributing with only some questions. Mostly working with Disrtibuted Interactive 3D simulations and have used twisted as one solution transfere states between 3D browser nodes. As I'm using both apache and www.plone.org to store files that builds up 3D applications, I'm curious on how fast a file are delivered from a web-server. What I would like to use are a test scripts that accesses files by url and log access-times. If I can define multipel files it would be even better. Will use the same mashine to run the test-script as are hosting both apache and plone servers. Sven-Erik Tiberg / ITS Lulea Univ of Technology sweden

1 1

[Twisted-Python] pb: callRemotes execute serially on server-side
by David Karnowski 20 Jan '09

20 Jan '09

Hi all, I'm having a client-side pb app call the same server-side function 10 times with this client-side code: def process_test(rootObject): for i in range(10): _logger.debug("Calling remote with arg: %d" % (i)) deferred = rootObject.callRemote("test_it", i) deferred.addCallbacks(success, problem, callbackKeywords={'i': i}, errbackKeywords={'i': i}) The server-side "remote_test_it" function takes 5 seconds to complete (per call). I find the following. The first point is what I'd expect, but points 2 & 3 are not: 1) The above for-loop completes asynchronously, with no blocking. It's complete in a few-thousand milliseconds. 2) The server-side executes serially. The second client-side call to "remote_test_it" doesn't start executing "test_it" on the server-side until after the first call has finished. I'd have thought these 10 calls would occur in parallel (in separate server-side threads). No? 3) None of the callbacks you see in the above client-side for-loop get executed until ALL 10 callRemotes have finished on the server-side. e.g. the callRemote for the first loop iteration finishes on the server-side in 5 seconds, but the associated client-side callback isn't executed until after 50 seconds (after the 10th loop iteration's server-side call finishes). I'm just starting with Twisted. Are points 2 & 3 expected, and is there a way for me to have them run in parallel rather than serially? thanks, David

3 2

[Twisted-Python] Mailing list archive search?
by David Karnowski 19 Jan '09

19 Jan '09

Is there a search function available for the mailing list archives? I don't see a search field anywhere<http://twistedmatrix.com/pipermail/twisted-python/>. Am I missing something obvious? The search field on your home page appears to just search the Tickets, Changesets, & the Wiki, and not the mailing list archives. I ended just doing it manually through a custom Google search<http://www.google.com/advanced_search?q=+site:http://twistedmatrix.com/pipe…>(adding a "site: http://twistedmatrix.com/pipermail/twisted-python" parameter to the Google query). If there isn't search functionality for the mailing list archives it'd be nice to have it added. regards, David

3 2

[Twisted-Python] spawnProcess() works after reactor.stop()
by Paul Thomas 19 Jan '09

19 Jan '09

I'm using SelectReactor on Linux for a system that spawns a lot of sub- processes. I'm finding that there are some cases where a signal to the twistd process doesn't reap the children and I think it's because the SIGINT handler defers the shutdown but doesn't prevent spawnProcess() from working in the meantime. What's happening is the signal gets handled by calling stop(). This sets a flag and schedules the shutdown event for later. - Then Linux signals all the sub-processes. - Then my code gets scheduled and spawns a process or 2. - The shutdown event runs The result is that any process spawned after the signal is orphaned. So, my question is: is this a bug in the reactor (should it check self._stopped)? If it's a bug, I expect it's a bit more general than just processes - would it be better to have the select loop check _stopped. If it's not a bug, would it be possible to add a public accessor for the stopped state so application code can decide whether to run or not? ta, pt.

1 0

[Twisted-Python] strangely getting 8002 can't serialise output when returning some list objects
by Krishnakant 19 Jan '09

19 Jan '09

hello hackers, I am a new user so perhaps I might be bothering with some question already answered. I am getting a strange response from a published object with twisted. I find no problem returning lists from a function. but when that same function returns records from a postgresql database using psycopg2 (again converted to list ) I get the dredded 8002 can't serialise output error. I have confirmed that i indeed casted the results into a list and the terminal where I run the twisted published object, I can print the list properly. but on the python prompt i use xmlrpclib and when I execute the function from the client, I get that error. how is this possible that a list created normally is getting returned and when i create an exact same list this time with a database record with psycopg2, i get error? can any one solve the suspence? happy hacking. Krishnakant.

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones 18 Jan '09

18 Jan '09

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> Just to drive that point home, Terry, I found an interesting error glyph> in your initial example. Your example does this: glyph> info = sys.exc_info() glyph> f = failure.Failure(*info) glyph> but sys.exc_info() is a tuple of (type, value, traceback), whereas glyph> Failure takes a tuple of (value, type, traceback). OMG! I guess I should learn to read. glyph> So, we still have some diagnosis to do on why you don't seem to be glyph> getting useful tracebacks from inlineCallbacks :). Now, not only glyph> can I not reproduce the bug, your reasoning doesn't make sense any glyph> more either. Glyph, sometimes I think you must be the most patient human ever. What a friendly and gentle correcting paragraph that is. I'm sure you'd make a great parent. (And I'm still a child, besides.) Still, I'd love to have been a fly on the wall when you realized I had the args reversed :-) I'm a big fan of expletives. Thanks a lot. I'm sure that's going to save me many hours of frustration and confusion. Terry

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 17 Jan '09

17 Jan '09

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones 17 Jan '09

17 Jan '09

Hi all. >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> 1. More of this discussion should be on a ticket. See http://twistedmatrix.com/trac/ticket/3622 Terry

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones 17 Jan '09

17 Jan '09

Hi Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> It sounds like the real issue here is that we don't have a channel glyph> to communicate a "cleaned traceback" (i.e., failure with stringified glyph> frames, but no traceback) from one bit of code to another? Yes, now that I better understand what's going on with cleaning failures, I think that's completely accurate. glyph> 1. More of this discussion should be on a ticket. The mailing list glyph> is good for getting discussions started and figuring out what needs glyph> to be done, but now we're talking about the technical specifics of glyph> resolving a specific problem. Someone (terry?) should take the glyph> specifics in the email that started this thread and file an glyph> appropriate ticket. OK, I will do that, because I do still think there is a problem. Code below. glyph> 2. When filing that ticket, I'd really like a full working example glyph> of inlineCallbacks not showing a traceback to play with, not just glyph> examples of portions of the problem. The obvious example I tried, glyph> from twisted.internet.defer import inlineCallbacks glyph> def buggy2(): glyph> raise RuntimeError("Whoops!") glyph> @inlineCallbacks glyph> def buggy(): glyph> yield buggy2() glyph> buggy() glyph> actually prints a nice descriptive traceback. That's because buggy2 is raising an exception, there's no failure coming back via a deferred into buggy (where its traceback would ultimately be replaced with None in cleanFailure). Here's a example: from twisted.internet import reactor, defer def ok(r): print 'OK', r def nok(r): print 'FAILED', r def two(): try: 1 / 0 except Exception: return defer.fail() @defer.inlineCallbacks def one(): yield two() def zero(): one().addCallbacks(ok, nok).addBoth(lambda _: reactor.stop()) reactor.callLater(0, zero) reactor.run() Because two returns a deferred whose errback has been called, one gets a result that's a failure and that failure gets cleaned. The output of running this example is: FAILED [Failure instance: Traceback: <type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero /usr/lib/python2.5/site-packages/twisted/internet/posixbase.py:228:mainLoop /usr/lib/python2.5/site-packages/twisted/internet/base.py:561:runUntilCurrent five.py:17:zero /usr/lib/python2.5/site-packages/twisted/internet/defer.py:813:unwindGenerator --- <exception caught here> --- /usr/lib/python2.5/site-packages/twisted/internet/defer.py:724:_inlineCallbacks five.py:14:one ] Showing the exception caught on line 14 (in one) not line 8. If you look at the deferred that comes back from calling two, its failure result has None as its tb attribute (via the process described in my original mail). However if we replace one with a version that doesn't use inlineCallbacks: def one(): return two() Then the failure has the original traceback: FAILED [Failure instance: Traceback: <type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero /usr/lib/python2.5/site-packages/twisted/internet/posixbase.py:228:mainLoop /usr/lib/python2.5/site-packages/twisted/internet/base.py:561:runUntilCurrent five.py:17:zero five.py:14:one --- <exception caught here> --- five.py:8:two ] I'll file a ticket and we can take discussion about what to do into the tracker. Thanks! Terry

1 0