Mailman 3 December 2006 - Twisted

Re: [Twisted-Python] pb.cacheable: adjustable cache depth?
by Anand Patil Dec. 12, 2006

Dec. 12, 2006

>They don't have that capability, no. I guess you could modify them to do >that, but I can't think of why you would want to. > >If you're not sure that's what you want to do, then it might help to >explain what you're trying to accomplish and someone on this list may be >able to help you. > > This is a statistical algorithm I'm trying to parallelize to run on a cluster, and the serial version strongly suggests that the behavior I've asked for would be a big help. I'll try to describe my serial algorithm succinctly, any advice on parallelizing it would be extremely welcome. I have a bunch of objects which are nodes in a directed acyclic graph. I've pseudocoded them in the attached file. What I'd like to do is push some of the nodes off to other processes running on the cluster and have them retain references to their parents. I'll then make repeated calls to all nodes' get_prob() methods and periodically update their values. It would be nice to make Node.value something like a Cacheable. I'd like it to store n remote references, with a timestamp (this just needs to be an integer counter, as in the pseudocode) attached to each. When queried, it could return its timestamp and value as a tuple. Thanks again, Anand class Node(object): value #A sizeable object, usually a numpy array. timestamp #An integer, incremented every time value is changed. parents #The parent Nodes of self parent_timestamp_caches # Remembered values of parents' timestamps the last n times # compute_prob() was called. prob_caches # Remembered outputs of compute_prob() the last n times it # was called. # It turns out to be optimal to have n = 2. def compute_prob(): """ A heavyweight function, should be called as little as possible. Returns a float value. """ return some_function([parent.value for parent in parents]) def get_prob(): """ Check my parents' timestamps. If this combination is present in parent_timestamp_caches, return the appropriate value from prob_caches. Otherwise, - Call compute_prob() - Store this combination of parents' timestamps in parent_timestamp_caches - Store the output of compute_prob() in prob_caches - Return the output of compute_prob(). """

2 1

[Twisted-Python] pb.cacheable: adjustable cache depth?
by Anand Patil Dec. 12, 2006

Dec. 12, 2006

Hi all, Quick question from a pre-beginner with Twisted: Can Cacheable store not just the most recent value of a remote object, but the n most recent values? Thanks, Anand Patil

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Dec. 10, 2006

Dec. 10, 2006

Bug summary ______________________ Summary for 2006-12-03 through 2006-12-10 Bugs opened: 36 Bugs closed: 29 Total open bugs: 791 (+7) |== Type Changes |== Priority Changes |== Component Changes |Defect: +3 |Highest: -2 |Conch: +0 |Enhancement: +7 |High: -4 |Core: +6 |Task: -1 |Normal: +14 |Mail: +2 |Low: +1 |Names: +0 |Trial: +2 |Web: -3 |Web2: +1 |Web2.Dav: +1 |Website: +0 New / Reopened Bugs ______________________ ===== Highest ===== [#2286] abstract.FileDescriptor calls pauseProducing even if producer is not streaming (opened by zseil) defect core http://twistedmatrix.com/trac/ticket/2286 [#2091] Make reactor unclean warnings reported errors (opened by jml) enhancement trial http://twistedmatrix.com/trac/ticket/2091 ===== High ===== [#2267] twisted.conch.test.test_conch tests leave the reactor unclean (opened by jml) defect conch http://twistedmatrix.com/trac/ticket/2267 [#2268] twisted.test.test_socks.Bind.test_simple leaves the reactor unclean (opened by jml) defect core http://twistedmatrix.com/trac/ticket/2268 [#2269] twisted.test.test_pb.NewStyleCachedTestCase leaves the reactor unclean (opened by jml) defect core http://twistedmatrix.com/trac/ticket/2269 ===== Normal ===== [#2256] test_tcp.HalfClose3TestCase.testReadNotificationRaisesTest fails sometimes on OSX (opened by jerub) (CLOSED, duplicate) defect core http://twistedmatrix.com/trac/ticket/2256 [#2257] Weak ETag in first second is bogus (opened by wsanchez) enhancement web2 http://twistedmatrix.com/trac/ticket/2257 [#2259] stdio fails when output redirected to a file (opened by jknight) defect core http://twistedmatrix.com/trac/ticket/2259 [#2260] Allow errors and failures to be strings (opened by jml) (CLOSED, invalid) defect trial http://twistedmatrix.com/trac/ticket/2260 [#2261] Front Page Sidebar CSS needs some work (opened by oubiwann) enhancement website http://twistedmatrix.com/trac/ticket/2261 [#2262] Fix Header Search CSS (opened by oubiwann) enhancement website http://twistedmatrix.com/trac/ticket/2262 [#2263] Fix "logged in as..." CSS (opened by oubiwann) enhancement website http://twistedmatrix.com/trac/ticket/2263 [#2264] Fix CSS on "Reply" buttons in issue tracker (opened by oubiwann) enhancement website http://twistedmatrix.com/trac/ticket/2264 [#2265] signal delivery vs reactor.stop (opened by titty) defect core http://twistedmatrix.com/trac/ticket/2265 [#2266] twisted.mail.test.test_mail.ProcessAliasTestCase leaves the reactor unclean (opened by jml) defect mail http://twistedmatrix.com/trac/ticket/2266 [#2270] Steal the bzr progress bar / realtime reporter for Trial (opened by jml) enhancement trial http://twistedmatrix.com/trac/ticket/2270 [#2272] epytext errors in svn trunk, set up pydoctor buildbot (opened by mwh) enhancement core http://twistedmatrix.com/trac/ticket/2272 [#2273] AMP documentation on command errors is reversed or code is (opened by htj) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/2273 [#2274] Twisted test suite is too slow (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/2274 [#2275] Twisted test suite requires too much memory (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/2275 [#2276] Changes to TwistedNames to make it support NAPTR records (opened by agatonsax) enhancement names http://twistedmatrix.com/trac/ticket/2276 [#2277] getcontenttype property should be explicitly set on PUT (opened by Cyrus Daboo) defect web2.dav http://twistedmatrix.com/trac/ticket/2277 [#2278] IMAP4 server fails for commands like "11 search uid 2:*" (opened by exarkun) defect mail http://twistedmatrix.com/trac/ticket/2278 [#2279] Add a link to a Windows build of PyOpenSSL to the WindowsBuild page (opened by exarkun) enhancement website http://twistedmatrix.com/trac/ticket/2279 [#2280] DigestCredential needs a new interface that supports verifying credentials when given a prehashed user:realm:password (opened by dreid) enhancement web2 http://twistedmatrix.com/trac/ticket/2280 [#2283] text color in search box hard to read (opened by itamarst) (CLOSED, duplicate) defect website http://twistedmatrix.com/trac/ticket/2283 [#2284] web2 submitRequest result does not have content-length (opened by ghazel) defect web2 http://twistedmatrix.com/trac/ticket/2284 [#2287] deferredCoroutine using new generator features in Python 2.5 (opened by mneeley) (CLOSED, duplicate) enhancement core http://twistedmatrix.com/trac/ticket/2287 [#2288] deferredCoroutine using new generator features in Python 2.5 (opened by mneeley) (CLOSED, duplicate) enhancement core http://twistedmatrix.com/trac/ticket/2288 [#2289] Page Titles and Metadata (opened by oubiwann) enhancement website http://twistedmatrix.com/trac/ticket/2289 ===== Low ===== [#2258] Use producer/consumer API in POP3 server for list responses (opened by exarkun) enhancement mail http://twistedmatrix.com/trac/ticket/2258 [#2271] Hello! This is a temporary test" appears on localhost:8080 if you "$ twistd web" (opened by jerub) enhancement web http://twistedmatrix.com/trac/ticket/2271 [#2281] Annotations for Twisted Finger Tutorial (opened by sluggo) enhancement website http://twistedmatrix.com/trac/ticket/2281 [#2282] Annotations for Twisted Finger Tutorial (opened by sluggo) (CLOSED, duplicate) enhancement website http://twistedmatrix.com/trac/ticket/2282 [#2285] AMP should have an additional client-specific tag->error mapping (opened by radix) enhancement core http://twistedmatrix.com/trac/ticket/2285 [#2285] AMP should have an additional client-specific tag->error mapping (opened by radix) enhancement core http://twistedmatrix.com/trac/ticket/2285 Closed Bugs ______________________ ===== Highest ===== [#2240] Add twisted.mail SMTP client example which demonstrates TLS and authentication (opened by exarkun, closed by exarkun, fixed) enhancement mail http://twistedmatrix.com/trac/ticket/2240 [#2091] Make reactor unclean warnings reported errors (opened by jml, closed by jml, ) enhancement trial http://twistedmatrix.com/trac/ticket/2091 [#2200] Resource lookup is slow (opened by Cyrus Daboo, closed by wsanchez, fixed) enhancement web2 http://twistedmatrix.com/trac/ticket/2200 [#2248] Do error reporting in twisted.names.client.Resolver (opened by exarkun, closed by exarkun, fixed) task names http://twistedmatrix.com/trac/ticket/2248 [#1199] twisted.test.test_protocols calls reactor.iterate() (opened by exarkun, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1199 ===== High ===== [#345 ] twcgi not unregistering producer correctly (possibly related to distrib). (opened by mesozoic, closed by exarkun, worksforme) defect web http://twistedmatrix.com/trac/ticket/345 [#676 ] twisted.persisted.sob._EverythingEphemeral is totally broken (opened by itamarst, closed by exarkun, duplicate) defect core http://twistedmatrix.com/trac/ticket/676 [#681 ] [PATCH] Make microdom preserve boolean of "was there whitespace or not" (opened by Tv, closed by exarkun, duplicate) defect web http://twistedmatrix.com/trac/ticket/681 [#2141] Twisted Trac Content Re-org (opened by oubiwann, closed by oubiwann, fixed) enhancement website http://twistedmatrix.com/trac/ticket/2141 [#1929] twisted.web2.test.test_xmlrpc should not use twisted.web.xmlrpc.Proxy (opened by dreid, closed by dreid, fixed) defect web2 http://twistedmatrix.com/trac/ticket/1929 [#1019] /projects/index.html in website (opened by mikewarren, closed by exarkun, wontfix) defect website http://twistedmatrix.com/trac/ticket/1019 [#981 ] cgi fails to close stdin of the child process (opened by warner, closed by exarkun, fixed) defect web http://twistedmatrix.com/trac/ticket/981 ===== Normal ===== [#2260] Allow errors and failures to be strings (opened by jml, closed by jml, invalid) defect trial http://twistedmatrix.com/trac/ticket/2260 [#1677] can't find manpages on the website (opened by synapsis, closed by oubiwann, fixed) defect website http://twistedmatrix.com/trac/ticket/1677 [#492 ] Document bug reporting methods (who to assign bugs to) (opened by hypatia, closed by jknight, fixed) enhancement website http://twistedmatrix.com/trac/ticket/492 [#2202] The News on the front page is no longer news. (opened by jml, closed by oubiwann, fixed) defect website http://twistedmatrix.com/trac/ticket/2202 [#2156] New Twisted Site Preparing for Release (opened by oubiwann, closed by oubiwann, fixed) enhancement website http://twistedmatrix.com/trac/ticket/2156 [#995 ] contribpackages are not updated for Python 2.4 (opened by manlio, closed by exarkun, fixed) enhancement website http://twistedmatrix.com/trac/ticket/995 [#2256] test_tcp.HalfClose3TestCase.testReadNotificationRaisesTest fails sometimes on OSX (opened by jerub, closed by exarkun, duplicate) defect core http://twistedmatrix.com/trac/ticket/2256 [#2283] text color in search box hard to read (opened by itamarst, closed by exarkun, duplicate) defect website http://twistedmatrix.com/trac/ticket/2283 [#2287] deferredCoroutine using new generator features in Python 2.5 (opened by mneeley, closed by exarkun, duplicate) enhancement core http://twistedmatrix.com/trac/ticket/2287 [#2288] deferredCoroutine using new generator features in Python 2.5 (opened by mneeley, closed by glyph, duplicate) enhancement core http://twistedmatrix.com/trac/ticket/2288 [#2273] AMP documentation on command errors is reversed or code is (opened by htj, closed by radix, fixed) defect core http://twistedmatrix.com/trac/ticket/2273 ===== Low ===== [#307 ] (don't) switch to logging.py / add support for logging.py (opened by glyph, closed by exarkun, fixed) enhancement core http://twistedmatrix.com/trac/ticket/307 [#313 ] Convention for accessing TLS certificates from protocols (opened by exarkun, closed by exarkun, wontfix) enhancement core http://twistedmatrix.com/trac/ticket/313 [#467 ] [PATCH] sandbox/exarkun/tls/record.py (opened by symbiont, closed by exarkun, wontfix) enhancement conch http://twistedmatrix.com/trac/ticket/467 [#578 ] Problems using twisted.web.client.getPage() (opened by dialtone, closed by exarkun, invalid) enhancement web http://twistedmatrix.com/trac/ticket/578 [#2282] Annotations for Twisted Finger Tutorial (opened by sluggo, closed by exarkun, duplicate) enhancement website http://twistedmatrix.com/trac/ticket/2282 [#2285] AMP should have an additional client-specific tag->error mapping (opened by radix, closed by glyph, ) enhancement core http://twistedmatrix.com/trac/ticket/2285

1 0

Re: [Twisted-Python] delay_connect problems in twisted.enterprise.adbapi
by glyph＠divmod.com Dec. 8, 2006

Dec. 8, 2006

On 04:39 am, hoadley(a)gmail.com wrote: >I'm using twisted, sybase 12.5, and the most recent Sybase module. The >only difference is the way I'm connecting is one program is a tap created >from mktap and the other a standalone script. Sadly, I am totally unfamiliar with the sybase module and so I can't help you myself. However, if someone else using the sybase module were posting to this list, it would be good to know whether this issue also occurs with the most recent trunk checkout of Twisted SVN. If so, then at least it might be worthwhile to open a bug (with a minimal example attached) in the tracker. If not, then at least you could use a recent release of trunk to work around the issue.

2 1

[Twisted-Python] What Win32 issues are important to you?
by Eric Mangold Dec. 8, 2006

Dec. 8, 2006

I'm going to set aside some time to work on win32 issues, and was wondering what tickets are particularly important for the folks out there? -- Eric Mangold Twisted/Win32 Maintainer

3 3

[Twisted-Python] Perspective Broker Anonymous access..
by Carlos Eduardo Dec. 8, 2006

Dec. 8, 2006

I´m just learning how to use twisted PB and playing with the chat server... I´m trying to create a way to allow authenticated and anonymous users into the system but can't figure how to make it. Here what i´m trying to do: ----------SERVER------------ def requestAvatar(self, avatarId, mind, *interfaces): assert pb.IPerspective in interfaces return pb.IPerspective, MyPerspective(avatarId), lambda:None p = portal.Portal(MyRealm()) c = checkers.InMemoryUsernamePasswordDatabaseDontUse(user1="pass1", user2="pass2") p.registerChecker(checkers.AllowAnonymousAccess(), credentials.Anonymous()) p.registerChecker(c) reactor.listenTCP(8800, pb.PBServerFactory(p)) reactor.run() ----------------------------- CLIENT factory = pb.PBClientFactory() reactor.connectTCP("localhost", 8800, factory) def1 = factory.login(credentials.Anonymous()) def1.addCallback(connected) reactor.run() ----------------------------- It dont work this way, giving me this error: Traceback (most recent call last): Failure: twisted.cred.error.UnauthorizedLogin: Another way i tried to register only the AllowAnonymousAccess in the server but the following error popped: Traceback (most recent call last): Failure: twisted.cred.error.UnhandledCredentials: No checker for twisted.cred.cr edentials.IUsernameHashedPassword, twisted.spread.pb.IUsernameMD5Password, twist ed.spread.interfaces.IJellyable Looks like PB doesn´t support anonymous access but from documentation i saw that i can do the following: # anonymous users share one Avatar, named users each get their own def requestAvatar(self, avatarID, mind, *interfaces): assert pb.IPerspective in interfaces if avatarID == checkers.ANONYMOUS: return pb.IPerspective, self.anonAvatar, lambda:None else: return pb.IPerspective, self.avatars[avatarID], lambda:None Just dont know how.... I already checked the PB internals (pb, portal, checkers, credentials) and havent figured out how to add anonymous support to PB... if you guys could help me I´m commited to add this support... Thanks for all, Carlos Eduardo

2 3

[Twisted-Python] delay_connect problems in twisted.enterprise.adbapi
by Rob Hoadley Dec. 8, 2006

Dec. 8, 2006

Hello, I'm having a problem using twisted.enterprise.adbapi. The problem has to do with the 'delay_connect' option I send with my db connection kwargs. I want to send sybase connection appname and hostname info. The only way I can do this is using the delayed connect option. I have the following standalone program example that works just fine: vrdb = vr_db('billing_db', 'appname') # vr_db just is a helper class for logical dbname -> db connection info db_kwargs = vrdb.return_kwargs() db_kwargs['cp_min'] = 1 # one thread min db_kwargs['cp_max'] = 1 # one thread max = only one db connection db_kwargs['cp_reconnect'] = 1 # reconnect if die db_kwargs['cp_noisy'] = 1 db_kwargs['delay_connect'] = 1 # do the delayed connect db_kwargs['cp_openfun'] = vrdb._vrdbOptions print db_kwargs dbpool = adbapi.ConnectionPool(vrdb.return_import_name(), **db_kwargs) # _vrdbOptions adds the following two things to the connection # conn.set_property (Sybase.CS_HOSTNAME, self.hostname) # conn.set_property(Sybase.CS_APPNAME, self.appname) a = storage(dbpool, d) a.run() reactor.run() run does a sql query. def run(self): ''' utility method if the script is called directly ''' self.get_campaign_info('id', 588103).addCallback( self.print_result ).addErrback(self.error_happened) This works fine and dandy. With the hostname and appname showing up in my sp_who optput (show's you who's logged in from where with what appname ) Now my problem is when I use the database connection part in makeService(config) and pass a dbpool to a service, my service starts up fine, but when I have a database operation I get this: 2006/12/07 15:46:17 PST [-] [Failure instance: Traceback: twisted.enterprise.adbapi.ConnectionLost: /usr/lib/python2.4/threading.py:422:run /usr/lib/python2.4/site-packages/twisted/python/threadpool.py:148:_worker /usr/lib/python2.4/site-packages/twisted/python/context.py:59:callWithContext /usr/lib/python2.4/site-packages/twisted/python/context.py:37:callWithContext --- <exception caught here> --- /usr/lib/python2.4/site-packages/twisted/internet/threads.py:25:_putResultInDeferred /usr/lib/python2.4/site-packages/twisted/enterprise/adbapi.py:379:_runInteraction /usr/lib/python2.4/site-packages/twisted/enterprise/adbapi.py:64:rollback ] So I've reverted my code to pass db_kwargs['delay_connect'] = 0 and it's working fine as long as I don't try the delay_connect. I'm hoping someone has seen this before and can point me in the right direction. I'd really like to have my appname and hostname show up on the db side. I'm using twisted, sybase 12.5, and the most recent Sybase module. The only difference is the way I'm connecting is one program is a tap created from mktap and the other a standalone script. >>> import Sybase >>> Sybase.__version__ '0.37' >>> import twisted >>> twisted.__version__ '2.4.0' Thanks -rob

1 0

[Twisted-Python] Re: Twisted-Python Digest, Vol 33, Issue 6
by Steve Han Dec. 8, 2006

Dec. 8, 2006

Thank you very much Robert.I have been using SuSE10.0 that contained twisted.I could not find twisted when updating to SuSE10.1Wired.I am back to suse10.0 again.I will learn more about twisted from now. On 12/7/06, twisted-python-request(a)twistedmatrix.com < twisted-python-request(a)twistedmatrix.com> wrote: > > Send Twisted-Python mailing list submissions to > twisted-python(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > or, via email, send a message with subject or body 'help' to > twisted-python-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Twisted-Python digest..." > > > Today's Topics: > > 1. Re: web2: http Content-Length header (Pieter Grimmerink) > 2. twisted dohop ( Atli ?orbj?rnsson ) > 3. Re: twisted dohop (Christopher Armstrong) > 4. Re: web2: http Content-Length header (David Reid) > 5. Problems with building of twisted-2.4.0 (Steve Han) > 6. Re: Problems with building of twisted-2.4.0 (Robert Gravina) > 7. Re: web2: http Content-Length header (Pieter Grimmerink) > 8. Thread (Gabriel) > 9. Re: Thread (Jean-Paul Calderone) > 10. Re: web2: http Content-Length header (James Y Knight) > 11. Re: web2: http Content-Length header (Pieter Grimmerink) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 5 Dec 2006 22:55:37 +0100 > From: Pieter Grimmerink <mailinglists(a)grimmerink.nl> > Subject: Re: [Twisted-Python] web2: http Content-Length header > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: <200612052255.37751.mailinglists(a)grimmerink.nl> > Content-Type: text/plain; charset="us-ascii" > > On Monday 04 December 2006 22:16, Pieter Grimmerink wrote: > > Is it really necessary to remove the content-length header from the > > response? > > Correction, it turns out indeed it has to be removed. If it isn't stripped > from the response, somehow the download stops after a single block of data > has been received. > > Since the content-length is saved in self.length, the following could be a > workaround (and it does work fine here): > > Add the Content-Length header again after calling setConnectionParams, at > the bottom of allHeadersReceived (line 195 twisted/web2/channel/http.py) > > self.inHeaders.setHeader('Content-Length', str(self.length)) > > What exactly goes wrong with setConnectionParams when the Content-Length > header is not (temporarily) removed is not 100% clear to me, so there > probably are better solutions for this problem. > > Regards, Pieter > > > > ------------------------------ > > Message: 2 > Date: Tue, 5 Dec 2006 22:51:06 +0000 > From: " Atli ?orbj?rnsson " <atlithorn(a)gmail.com> > Subject: [Twisted-Python] twisted dohop > To: twisted-python(a)twistedmatrix.com > Message-ID: > <91142a380612051451k441243f8vd438f5daf618c3f8(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > We would like to be added to the ProjectsUsingTwisted page. dohop.com is a > flight planner and twisted runs what we call our "priceserver". The > priceserver is under a substantial amount of load and twisted handles the > job handsomely. > > Try it out at http://www.dohop.com > > The backend of the site runs otherwise almost wholly on Python with bits > and > pieces in C. We do a lot of django under mod_python with a mysql db. > > Hope you see us fit for a mention and thank you for the amazing framework > that is twisted. > > Atli Thorbjornsson > dohop.com >

1 0

[Twisted-Python] Problems with building of twisted-2.4.0
by Steve Han Dec. 7, 2006

Dec. 7, 2006

Hi all, I am new to Twisted.When building ZopeInterface-3.1.0-0c1 got the messages as the following: ...build/temp.linux-i686-2.4/Dependencies/zope.interface- ZopeInterface-3.1.0c1/zope.interface/_zope_interface_coptimizations.o unable to execute gcc: No such file or directory error: command 'gcc' failed with exit status 1 What does mean by that? Which packages I need for building Twisted/ZopeInterface? Thanks alot.

3 2

[Twisted-Python] web2: http Content-Length header
by Pieter Grimmerink Dec. 6, 2006

Dec. 6, 2006

Hi all, I'm using twisted.web2.client.HTTPClientProtocol to implement a HTTP downloader object. In order to show the download progress, I need the Content-Length header, which is found in the server response. But it turns out this header is being removed from the response, because it is listed in connHeaderNames, twisted/web2/channel/http.py line 217. Leaving 'content-length' out of this list gives me the valid content-length header in the response. Is it really necessary to remove the content-length header from the response? Regards, Pieter

3 5