Mailman 3 August 2007 - Twisted

[Twisted-Python] why is Twsited efficiently in one thread? what about in multi-processor system?
by Yan Zhu 15 Aug '07

15 Aug '07

hi all, I read a text about twisted vs thread, who can tell mewhy Twsited is efficiently in one thread? especially, if clients require long-running processing above and beyond socket calls, or in the multi-processor system? look at this: http://kaishaku.org/twisted-vs-threads/ -- eSX

3 2

[Twisted-Python] Cookies in Client?
by Andy Fundinger 15 Aug '07

15 Aug '07

I'm trying to use twisted to drive a website with forms-based authentication. The target site is a php-based, so I need to grab and return a PHPSESSIONID cookie to preserve the session. Is there some twisted support for that which I can hook into? -Andy Fundinger -- Blog: http://channel3b.wordpress.com Second Life Name: Ciemaar Flintoff #Python 3001: from future import time.travel

1 0

[Twisted-Python] threads monitoring
by Ladislav Andel 15 Aug '07

15 Aug '07

Hello, How can I monitor the number of created threads when I use just function deferToThreds in my code? Could you give me an example, please? If I do following: ThreadPool.dumpStats() I get TypeError: unbound method dumpStats() must be called with ThreadPool instance as first argument (got nothing instead) Here I understand I have to make an instance of ThreadPool, but if I just use deferToThreads from threads module I'm not sure how to accomplish it. Thanks, Ladislav

2 1

AW: [Twisted-Python] threads monitoring
by Andreas Kostyrka 15 Aug '07

15 Aug '07

Well, I'm not at the keyboard, but you'll probably need to take a look at the source code which threadpool instance is used by deferToThread. Andreas -- Ursprüngl. Mitteil. -- Betreff: [Twisted-Python] threads monitoring Von: Ladislav Andel <ladaan(a)iptel.org> Datum: 15.08.2007 16:53 Hello, How can I monitor the number of created threads when I use just function deferToThreds in my code? Could you give me an example, please? If I do following: ThreadPool.dumpStats() I get TypeError: unbound method dumpStats() must be called with ThreadPool instance as first argument (got nothing instead) Here I understand I have to make an instance of ThreadPool, but if I just use deferToThreads from threads module I'm not sure how to accomplish it. Thanks, Ladislav _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

[Twisted-Python] Handling PBConnectionLost errors
by Daniel Miller 15 Aug '07

15 Aug '07

Hello, Twisted PB sometimes loses its connection to the server. In this case, a PBConnectionLost exception is raised on the client. It would be nice to implement a fail-safe(er) way of calling remote methods that would retry when necessary until the remote method has been called successfully and the result has been returned. Note that this is only necessary when the remote method call should be invoked exactly once on the server (i.e. for POST-like calls that change server state). In the case of GET-like requests, a simpler retry mechanism will do. The motivation for this is based on my experience of using Twisted in an application I am developing. The network communications are all happening on a LAN. The good end of the network is connected directly to a 100Mbps switch at the server. Failures occur more frequently at the other end (my end) of the network, which is connected through a 10/100 hub that is connected to the main switch. I rigged up a quick test with a 1000-request sample size; failures ranged from 28/1000 on the good end of the network to 83/1000 on the bad end of the network. One request consists of a single remote method call through PB. A success was when I got the expected result, a failure was when I got a PBConnectionLost error. The following is pseudo code that I came up with to mitigate the problem. Simple request (GET - repeatedly call method until success or RETRY_LIMIT is reached) Client flow: for x in range(RETRY_LIMIT) invoke remote method without unique call identifier if result is not PBConnectionLost break if result is PBConnectionLost raise server not responding error Server flow: (nothing special, just plain PB) Complex request (POST - server-side method is invoked exactly once) Client flow: use simple retry method to get a unique call identifier from server a timeout value is also sent along to tell the server how long to hold the results of this request for x in range(RETRY_LIMIT) invoke remote method with identifier if return value is not PBConnectionLost break if result is PBConnectionLost raise server not responding error using simple retry method tell server to discard unique call identifier Server flow: receive request for unique call identifier create and store identifier with UNCALLED token schedule identifier to be discarded with timeout value supplied by client return identifier to client receive remote method invocation with unique call identifier branch on value stored with unique call identifier if UNCALLED update identifier with CALLED token invoke method while result is deferred get defer result store COMPLETED token and unique with unique call identifier if there is another invocation WAITING this means the connection was lost signal the WAITING request with the result else return result to client if CALLED store WAITING token with unique identifier (must not overwrite other call tokens) defer until COMPLETED if COMPLETED return result to client if unique call identifier does not exist raise error receive request to discard unique call identifier if identifier exists discard identifier, tokens, and result return True I realize that implementing this would not eliminate network errors. It would simply reduce the likelyhood of failed method calls due to dropped connections. If I have my math correct (I always struggle a bit with statistics), even a RETRY_LIMIT of 2 would reduce the probability of a lost connection to 0.6% at the worst (<0.1% on the good end of the network). I have two questions: 1. Does something like this already exist? 2. Is this a totally stupid idea? (would it be better to improve our physical network than to try to band-aid the problem with something like this?) ~ Daniel

4 8

[Twisted-Python] Where to start: log reader/analysis
by Andrew E 13 Aug '07

13 Aug '07

Dear Twisted Experts (... meant in a nice way :) ) I'm not sure where to start. I need to write a small server that: - reads lines in a log file as they are appended - reads input from a socket as it becomes available - does an analysis of both (like, what time was input received in the log, and the output received via the socket) - outputs a summary report Socket I/O is easy - but I'm not sure how to include file reading ... its bound to be easy. Any tips? Thanks :) Andrew

4 3

[Twisted-Python] Specifications
by Jonathan Lange 12 Aug '07

12 Aug '07

Hello all, I've seen a couple of specifications on the Twisted wiki referred to in Twisted tickets. This is great. I'm all for specs, particularly when they are there simply to help clarify ideas and provoke discussion. However, if we are going to have more than one spec, it'd be nice to have a list of them somewhere. It would also be very helpful if specs had some sort of status. Launchpad provides something like this at https://blueprints.launchpad.net/twisted. jml

3 4

[Twisted-Python] returning value after addCallback
by Ladislav Andel 12 Aug '07

12 Aug '07

Hi, you could tell me that I should stop programming since I don't know properly OOP but I quite need to sort out this problem. Here is another piece of code I have problem with. Please be patient with me. Where is mistake in my code. It is not returning any reasonable result. Thank you for help. Lada -------------------------------------- code ------------------------------------------------- from twisted.internet import reactor, defer, threads from siptest import siptest_f test_opts = {'username':'test','transport':'udp','localport':'5085','password':'test'} domain = ['sip.1und1.de', 'sip.babble.net'] class PingTracker: def __init__(self): self.dns = None self.sip = None self.ping = None def gotSipServerResults(self, domain, test_opts): if domain: return siptest_f(domain, test_opts) raise ValueError("Error occured") def checkSIP(self, domain, test_opts): return threads.deferToThread(self.gotSipServerResults, domain, test_opts) def check(self, host, test_opts): checks = [] #checks.append(checkDNS().addCallback(self.setDNSResult)) checks.append(self.checkSIP(host, test_opts).addCallback(self.setSIPResult)) #checks.append(checkPing().addCallback(self.setPingResult)) return defer.gatherResults(checks) def setDNSResult(self, result): self.dns = result def setSIPResult(self, result): self.sip = result def setPingResult(self, result): self.ping = result def SipCallback(self): print self.sip test = PingTracker() d = test.check(domain[0], test_opts) print d # this returns Deferred... test.SipCallback() # this returns None reactor.run()

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 11 Aug '07

11 Aug '07

Bug summary ______________________ Summary for 2007-08-05 through 2007-08-12 Bugs opened: 10 Bugs closed: 9 Total open bugs: 1010 (+1) |== Type Changes |== Priority Changes |== Component Changes |Defect: +2 |Highest: -4 |Conch: +1 |Enhancement: -1 |High: -1 |Core: +2 |Task: +1 |Normal: +7 |Lore: +1 |Mail: -1 |Names: +0 |Trial: +0 |Vfs: +2 |Words: -3 New / Reopened Bugs ______________________ ===== Highest ===== [#2775] Crunchy Output (opened by tehminkeh) enhancement lore http://twistedmatrix.com/trac/ticket/2775 [#2783] twisted.internet.task.Clock does not implement IReactorTime (opened by dreid) defect core http://twistedmatrix.com/trac/ticket/2783 [#2772] Move XML stream header processing to respective authenticators (opened by ralphm) (CLOSED, fixed) enhancement words http://twistedmatrix.com/trac/ticket/2772 ===== Normal ===== [#2776] Define a uniform way to make objects immutable. (opened by ralphm) task core http://twistedmatrix.com/trac/ticket/2776 [#2777] Fix SRVConnector to deal with bad nameservers. (opened by ralphm) enhancement names http://twistedmatrix.com/trac/ticket/2777 [#2778] TestCase instances re-used for test method runs when trial is invoked with --until-failure (opened by exarkun) defect trial http://twistedmatrix.com/trac/ticket/2778 [#2779] os.setsid throws exception in PTYProcess on AIX (opened by chrispritchard) defect core http://twistedmatrix.com/trac/ticket/2779 [#2780] Remove vfs.pathutils (opened by cablehead) defect vfs http://twistedmatrix.com/trac/ticket/2780 [#2781] Just use producers and consumers for the vfs ftp adapter (opened by cablehead) defect vfs http://twistedmatrix.com/trac/ticket/2781 [#2782] Channel is not notified, if Connection fails between openChannel and ssh_CHANNEL_OPEN_CONFIRM (opened by cebewee) defect conch http://twistedmatrix.com/trac/ticket/2782 Closed Bugs ______________________ ===== Highest ===== [#2771] exceptionFromStreamError fails to parse errors correctly (opened by ralphm, closed by ralphm, fixed) defect words http://twistedmatrix.com/trac/ticket/2771 [#2290] nameservers resolved in wrong order (opened by ilia, closed by exarkun, fixed) defect names http://twistedmatrix.com/trac/ticket/2290 [#1917] Typo in twisted.mail.mail.DomainWithDefaultDict (opened by therve, closed by exarkun, fixed) defect mail http://twistedmatrix.com/trac/ticket/1917 [#2773] t.w.p.j.xmlstream.toResponse fails on stanzas without id attribute (opened by ralphm, closed by ralphm, fixed) enhancement words http://twistedmatrix.com/trac/ticket/2773 [#2770] Allow JIDs to be used as key in dicts (opened by ralphm, closed by ralphm, fixed) enhancement words http://twistedmatrix.com/trac/ticket/2770 [#2772] Move XML stream header processing to respective authenticators (opened by ralphm, closed by ralphm, fixed) enhancement words http://twistedmatrix.com/trac/ticket/2772 [#2772] Move XML stream header processing to respective authenticators (opened by ralphm, closed by ralphm, fixed) enhancement words http://twistedmatrix.com/trac/ticket/2772 [#2757] give TrafficLoggingFactory a public API to reset its counter (opened by exarkun, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/2757 ===== High ===== [#1630] trial test hangs if using the twisted enterprise connection pool (opened by harshaw, closed by exarkun, fixed) defect trial http://twistedmatrix.com/trac/ticket/1630 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2783] twisted.internet.task.Clock does not implement IReactorTime (since 2007-08-11 18:14:57). Mean open ticket age: 500 days, 21:14:39.812723. Median: 391 days, 8:26:03.380050. Standard deviation: 395 days, 15:09:23.171000. Interquartile range: 483 days, 5:01:00. Mean time between ticket creation and ticket resolution: 119 days, 14:46:10.149769. Median: 14 days, 3:27:47. Standard deviation is 222 days, 21:59:34.405886. The interquartile range is 125 days, 12:43:31. Mean time spent in review: 60 days, 17:22:38.145920. Median: 4 days, 0:03:10. Standard deviation: 132 days, 8:20:06.481949. Interquartile range: 28 days, 13:23:16. Mean number of times a ticket is reviewed: 1.85922330097. Median: 1 Standard deviation: 1.4260364579. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 21 unique ticket reporters 10 unique ticket reviewers 5 unique ticket resolvers In the last 24 weeks, 86 unique ticket reporters 14 unique ticket reviewers 17 unique ticket resolvers In the last 48 weeks, 158 unique ticket reporters 21 unique ticket reviewers 26 unique ticket resolvers

1 0

[Twisted-Python] Job offer for Python Lead, LA or the Bay Area
by Andy Gayton 10 Aug '07

10 Aug '07

DigiSynd is an experienced team of Los Angeles-based digital pioneers with a unique vision: to create, produce, syndicate, and market online entertainment. DigiSynd uses time-honored storytelling techniques and valuable lessons from traditional media, but applies them in a medium of online communities, virtual experiences, interactivity and user-generated content. They're based in LA, Burbank - across the street from the Warner Bros Lot. Digisynd are seeking a seasoned python engineer. Ideally you: * live and breath python * have exceptional SQL skills * have experience with capturing, and reporting off large datasets in real time * have extensive experience with web services - both creating and utilizing You should be based in LA, or in the bay area. This is a great opportunity to put together and lead a team of developers, while working in a very interesting space. If interested, send your resume through to jobs14(a)digisynd.com I'm trekking out to the SuperHappyDevHouse at Los Gatos tomorrow. If you know anyone who is going that might be interested, just tell them to grab me. cheers! Andy.

1 0