Mailman 3 August 2009 - Twisted

[Twisted-Python] How to debug an AMP connection?
by Peter Westlake Aug. 10, 2009

Aug. 10, 2009

Hello, I'm having trouble with an AMP connection that doesn't fire the Deferred returned by callRemote. The AMP command copies files from the client (called a "worker" in the code below) to a server ("controller"). It sends a chunk of text at a time - 16K originally, but I tried smaller amounts too. The server appends the text to a queue to be written; it shouldn't take long, because it doesn't do the I/O synchronously, just starts a thread. What happens is that the client calls callRemote, but the Deferred that callRemote returns is never fired. I know the command reaches the server, because the client keeps track of how many bytes have been sent successfully, and the file on the server is bigger than that. I know that the Deferred doesn't fire, because I can examine it using a manhole and it has called = False. There are no error messages in the log. That's consistent with the Deferred not firing, because the error callback (self.next_or_retry) would log a message if the remote command failed. So, what can possibly be happening? And how can I find out? All suggestions most gratefully received. Peter. Client-side code: class ConnectionBuffer(object): """Class to store AMP commands and retry them after network outages. ... """ def send(self, adv_arg, *a, **kw): if self.protocol: # call_d is here purely to make the Deferred accessible when debugging. self.call_d = self.protocol.callRemote(*a, **kw).addBoth(self.next_or_retry, adv_arg) def connected(self, protocol): self.protocol = protocol self.send_next() def disconnected(self): self.protocol = None Server: class ControllerProtocol(amp.AMP): """Protocol used by the controller to communicate with a worker. ... """ def logoutput(self, product, branch, site, job, action, number, data, stream): """Receive stdout or stderr from an action running on a worker.""" actid = (product, branch, site, job, action) ac = self.factory.controller.actions.find(actid) ac.instances[number].logfile[stream].write(data) return {'status': True} commands.LogOutput.responder(logoutput)

4 7

[Twisted-Python] sending 'tail -f' command to a remote host using conch?
by Harsha Aug. 10, 2009

Aug. 10, 2009

Hi All, A day back I install Twisted and Conch on RHEL5. I am a novice user of Twisted software. I copied an example from dev-shed website (attached herewith). This example script uses conch to talk to a ssh deamon. This script was able to capture and return the output of simple commands such as - ls, cat etc. I mean, when I run the script to execute 'ls -al' on a remote machine it does and returns the prompt. Where as, when I run the script to execute '*tail -f /path/to/mylog.txt*' I get the ouput of this command but the prompt never returns. When I press ^C (Control-C) I get the following run-time error. *Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/internet/tcp.py", line 520, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/conch/ssh/transport.py", line 166, in connectionLost self.service.serviceStopped() File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/conch/ssh/connection.py", line 59, in serviceStopped map(self.channelClosed, self.channels.values()) File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/conch/ssh/connection.py", line 580, in channelClosed log.callWithLogger(channel, channel.closed) --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "shankha.py", line 56, in closed reactor.stop() File "/usr/lib/python2.4/site-packages/Twisted-8.2.0-py2.4-linux-i686.egg/twisted/internet/base.py", line 526, in stop raise error.ReactorNotRunning( twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.* *I know that 'tail -f' command is expecting "Control-C" or Break to return the prompt on remote host. But my attempt to send the ^C character is failing.. I am doing somethink like this: python sshc.py <hostname> 'tail -f mylog.txt ; \x03' What is the recommended method to adopt such cases? *Cheers!! Harsha Reddy

2 1

[Twisted-Python] twisted.positioning -- Interface, or class with method stubs?
by Laurens Van Houtven Aug. 10, 2009

Aug. 10, 2009

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi :-) As you probably know, the interfaces for t.p live in t.p.ipositioning. I still haven't quite understood why they are going to be awesome, but anyway -- I'm running into a problem where keeping them would (I think) imply a bunch of duplicated code. The thing is, t.p.nmea.NMEAAdapter needs to (at least, I think so) implement all of the methods of t.p.ipositioning.IPositioningProvider, because people will subclass t.p.nmea.NMEAAdapter and override the corresponding (stub) methods. In order to get all of the stub methods implemented (otherwise people using the class need to implement *every* method, which is dumb, since a lot of people won't care about things like GPS climb), I would basically be copying every line of the body of the interface into the t.p.nmea.NMEAAdapter. This sounds like a dumb idea. Subclassing the interface sounds like *entirely* the wrong thing to do, since the class is-not-an Interface. Another alternative is using getattr-with-default-value in order to call the methods, since then missing (non-implemented) methods will be handled gracefully. I kinda like this one, mainly because it's very little work ;-) A third alternative would be to scrap the interface all together and use a base class with method stubs (I'm deliberately not using the word abstract there). This sounds like a bad idea because we'd be losing the interface, but I'm not entirely sure why we want it in the first place... thanks for your time Laurens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.8) iEYEARECAAYFAkp/W9QACgkQT5v5zGkvKT7inwCg0UP9d0zoiFnq1J8LrC/T1yUr gXQAoIDbcp2jWg2Q5qIXTCRPsSGMiYCr =kwgP -----END PGP SIGNATURE-----

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 9, 2009

Aug. 9, 2009

1 0

[Twisted-Python] Changing threadpool size in reactor
by Darren Govoni Aug. 6, 2009

Aug. 6, 2009

Hi, I could see from the docs I read how to increase the allotted threads in the reactor. I use reactor.callInThread(....) and it seems to max out at 10. I read about ThreadPool class, but need to run mine in the reactor. thanks for any tips and sorry if it was more obvious than i could see. Darren

2 1

[Twisted-Python] Adbapi issues
by Gerrat Rickert Aug. 6, 2009

Aug. 6, 2009

I'd like to use twisted's adbapi module (twisted 8.2.0 for python 2.5) with cx_Oracle, but I'm having some issues with it. Specifically: 1. It doesn't seem to reconnect (or possibly I just need enlightenment on how reconnecting works): If I tweak part of the test_adbapi.py script to work for Oracle (using a proper "conn_str", and ignoring the irrelevant parts), I get something like: class OracleTests(unittest.TestCase): """Test adbapi for Oracle""" timeout = 10 def setUp(self): self.dbpool = adbapi.ConnectionPool('cx_Oracle', conn_str, cp_noisy=True, cp_min=1, cp_reconnect=True, cp_max=3, cp_good_sql='select * from dual', threaded=True) def test_reconnect(self): d = defer.succeed(None) d.addCallback(self._testPool_1) d.addCallback(self._testPool_2) d.addCallback(self._testPool_3) return d def _testPool_1(self, res): def _success(rslt): self.failUnless(rslt[0][0] == 'X', "Select from dual not working") d = self.dbpool.runQuery("select * from dual") d.addCallback(_success) return d def _testPool_2(self, res): self.dbpool.connections.values()[0].close() def _testPool_3(self, res): sql = "select * from dual" d = self.dbpool.runQuery(sql) def _check(row): self.failUnless(row[0][0] == 'X', " Select from dual not working ") d.addCallback(_check) return d I get this traceback: [ERROR]: test_oracle.OracleTests.test_reconnect Traceback (most recent call last): File "C:\Python25\Lib\site-packages\twisted\python\threadpool.py", line 210, i n _worker result = context.call(ctx, function, *args, **kwargs) File "C:\Python25\Lib\site-packages\twisted\python\context.py", line 59, in ca llWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "C:\Python25\Lib\site-packages\twisted\python\context.py", line 37, in ca llWithContext return func(*args,**kw) File "c:\python25\lib\site-packages\twisted\enterprise\adbapi.py", line 429, i n _runInteraction result = interaction(trans, *args, **kw) File "c:\python25\lib\site-packages\twisted\enterprise\adbapi.py", line 443, i n _runQuery trans.execute(*args, **kw) cx_Oracle.InterfaceError: not connected ...am I doing something wrong, or is this something specific to Oracle? I apologize for the seemingly rhetorical question, but does this actually work for other databases (I don't have any others installed, so trial just skips most of the tests when I run test_adbapi.py)? 2. ...since I can't get it to reconnect properly, perhaps I could just close the connections in the old connection pool, and create a new one... dbpool = adbapi.ConnectionPool('cx_Oracle', conn_str, cp_noisy=True, cp_min=1, cp_reconnect=True, cp_max=3, cp_good_sql='select * from dual', threaded=True) dbpool.close() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\python25\lib\site-packages\twisted\enterprise\adbapi.py", line 359, i n close if self.shutdownID: AttributeError: ConnectionPool instance has no attribute 'shutdownID' ...shouldn't shutdownID be initialized in the class, or at least in __init__ ? 3. ...as stated in PEP 249, under fetchall(), "... Note that the cursor's arraysize attribute can affect the performance of this operation." In the PEP, cursor.arraysize defaults to 1, which results in absolutely terrible performance when retrieving a large number of rows from the database (empirically tested). Since twisted has already gone through the trouble of wrapping simple calls to the database (like it's runQuery), it would be ideal if this was an optional parameter that could be passed in. It doesn't look possible to do this in the current adbapi module (or perhaps, I might just need enlightening again): def _runQuery(self, trans, *args, **kw): trans.execute(*args, **kw) return trans.fetchall() ...perhaps it could be something more like: def _runQuery(self, trans, *args, **kw): if kw.has_key('arraysize'): trans._cursor.arraysize = kw['arraysize'] trans.execute(*args, **kw) return trans.fetchall() 4. Timeouts. ...well, since Deferred.setTimeout is deprecated, and we can't cancel deferreds, most protocols (or asynchronous "mechanisms" if protocol isn't the right term here) should probably have a timeout mechanism. (...or is everyone looking at me like I'm from outer-space?) Is there a canonical way of timing out a connection? If I were to do a dbpool.runQuery("select some_cols from some_table"); and attach a timeout mechanism to the deferred that it returns, is there a way to drop/recycle that particular connection from the pool if it didn't respond in a timely fashion? ...or (assuming the dbpool.close() issue went away), would "best-practice" be just closing the old pool, and re-creating it? ...is this module widely used in production, or are most people using something like SqlAlchemy (or "rolling their own")? Thanks, Gerrat

3 6

[Twisted-Python] twisted.positioning preliminary review
by Laurens Van Houtven Aug. 6, 2009

Aug. 6, 2009

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey :-) I've been told that I need to produce a patch faster, and most of the eyesores that I was really unhappy with have been removed (no more mutable sentenceData dict \o/). As usual, comments welcome. I could produce an actual patch, but perhaps it would be more useful to do this with the Launchpad-published bzr branch? Obviously that means I can't attach a real patch to a trac ticket. If people want me to, I will. Some of the obvious problems that I've seen myself or that come up on the mailing list are tagged with "REVIEW", so if you only have 10 minutes to look at it, grep nmea.py for it :-) Obvious TODO is howto documentation, but I'm not starting on that until the API has been reviewed thoroughly for obvious reasons. Here it is: http://bit.ly/3wWL2c Or you could just: bzr branch lp:~lvh/twisted/positioning Thanks for your time Laurens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.8) iEYEARECAAYFAkp3hvsACgkQT5v5zGkvKT4vTwCdG41LE0snM8vdK1L9ohF8wZRo +T4An2RQz22QmjT1YFEAmOT8IHJyoT8C =WbHv -----END PGP SIGNATURE-----

2 2

[Twisted-Python] Really Basic clarification on defers
by John Aherne Aug. 6, 2009

Aug. 6, 2009

This is a really basic problem we are trying to decide about, We have programs that run quite happily, so far. Its main task is to receive data from port A and send it out via port B. Then receive data via port B and send it out via port A. It's pretty much like a chat setup. You just build up a list of connected clients and send data to them as required One side A receives some input from a tcp port - about 100-200 characters, and forwards it to another port B. We do not need to wait for any response. If we get a response we pick that up through line receiver. We also run a calllater to check if we got a response on linereceiver within the timeframe specified. If not we drop the connection. Traffic coming in from port B is analysed and some subset is sent back to port A. Ignoring port A for the moment, just concentrating on port B, we have tried three options:-- 1. We set up a defer to handle the sendline to port B so that the reactor would schedule it in its own good time. No threads involved using the standard twisted setup. When we get a response through receiveline we fire the callback defer. If we timeout via callLater we fire the errback to clear the defer. In this case the defer does not seem to be doing very much 2. Now a fresh pair of eyes is looking at the code and saying why are we using a deferred for sending data to port B. We could just issue a straight sendline as part of the main code and carry on. If we get a response via linereceiver,we process it normally, otherwise we set our callLater running and timeout and lose the connection. So no deferreds required at all. It does seem to work.What we are not sure about is what penalty is incurred in terms of reliability or throughput by using sendline without a deferred. We are not too sure what the holdup will be and whether it could end up halting the show. Is it better to schedule these messages via deferreds or am I missing something obvious 3. So we then did an experiment and used defertothread to run the sendline in a separate thread with its own defer to maximise the asynchronous running of the code. So now we are running threads when one of the reasons for looking at twisted was that we could avoid threads as much as possible. The conundrum we are trying to resolve now is which option should we use. Do any of the options have a built-in problem awaiting the unwary. In theory all 3 options work. But if No 1 works well enough for our volume of traffic should we adopt that one. Or is it better to start using the defertothread option. Is there a simple answer The traffic is not large, upto a 100-200 remote devices on port B. They will send GPS data every 20 secs, and about 500 messages of about 200 bytes average throught the day. The remote devices will respond in an irregular manner without dropping the connection, so we force a disconnectf if important messages are not getting through. They are then forced to reconnect. We have looked through the code searching for enlightment and it does seem to be well documented, but the information we are looking for comes well before the doc strings. Hopefully, someone can give us some pointers in the right direction. Thanks for any help. John Aherne

6 14

[Twisted-Python] Deferred documentation rewrite
by Edward Z. Yang Aug. 6, 2009

Aug. 6, 2009

Hello all, I have been prodded by the members of #twisted into rewriting the Deferred documentation. You can check out the plan at this ticket: http://twistedmatrix.com/trac/ticket/3943 Comments would be appreciated. Cheers, Edward P.S. If you reply on the mailing list, please CC me.

9 13

[Twisted-Python] My first wxPython App with Twisted
by Thalassery Koran Aug. 4, 2009

Aug. 4, 2009

Hi, I am creating the first GUI app (wxPython based) using Twisted. I am facing an issue which I cannot seem to solve/understand due to lack of trace or anything. I had a big app written on twisted and now I am creating a login screen to the app. The app used to work without any issues. I added the login screen and the login screen comes up. But then everything freezes up. I have to force quit the python application to get anything back. Can anyone help me solve it? The code is as follows: import wx import os import logging from twisted.internet import wxreactor wxreactor.install() from twisted.internet import reactor #Called only after wxreactor.install() from twisted.internet.task import LoopingCall import clientapp def opj(path): """ Convert paths to the platform-specific separator """ st = apply(os.path.join, tuple(path.split('/'))) if path.startswith('/'): st = '/' + st return st class APPApp(wx.App): def OnInit(self): self.frame = APPFrame(None, -1, " APP Login", size=(200, 150)) self.panel = APPPanel(self.frame,-1) client = clientapp.setup_app() call = LoopingCall(client.run) call.start(.51) return True def main(): app = APPApp(0) reactor.registerWxApp(app) try: reactor.run() except StandardError: raise class APPPanel(wx.Panel): """ APP GUI PANEL The panel in which the user name and password is typed. @var ID_LOGIN: Login buttons id. When this button is clicked, the data is stored for logging. """ ID_LOGIN = wx.NewId() def __init__(self, parent, id): self.parent = parent wx.Panel.__init__(self, parent, id, size=(200,150)) self.quote = wx.StaticText(self, -1, "APP Login",wx.Point(60, 10)) wx.StaticText(self, wx.ID_ANY, "User Name ", wx.Point(20, 33)) wx.StaticText(self, wx.ID_ANY, "Password ", wx.Point(20, 58)) self.usernameBox = wx.TextCtrl(self, wx.ID_ANY, "", wx.Point(80, 31), wx.Size(100, wx.ID_ANY)) self.usernameBox.SetFocus() self.passwordBox = wx.TextCtrl(self, wx.ID_ANY, "", wx.Point(80, 56), wx.Size(100, wx.ID_ANY), style=wx.TE_PASSWORD) self.button =wx.Button(self, self.ID_LOGIN, "Login", wx.Point(60, 88)) wx.EVT_BUTTON(parent, self.ID_LOGIN, self.on_login) def on_login(self, event): username = self.usernameBox.GetString(0, 128) password = self.passwordBox.GetString(0, 128) if username and password: self.parent.on_login_click(username, password) class APPTaskBarIcon(wx.TaskBarIcon): ''' TaskBar Icon So that the application is always alive on the taskbar. 3 operations are possible, DoubleClick on the taskbar would bring up the window, right click will bring up options to restore or exit the tool. ''' TBMENU_RESTORE = wx.NewId() TBMENU_CLOSE = wx.NewId() def __init__(self, frame): wx.TaskBarIcon.__init__(self) self.frame = frame # Set the image icon = self.MakeIcon(self.frame.get_image()) self.SetIcon(icon, "APP") self.imgidx = 1 # bind some events self.Bind(wx.EVT_TASKBAR_LEFT_DCLICK, self.OnTaskBarActivate) self.Bind(wx.EVT_MENU, self.OnTaskBarActivate, id=self.TBMENU_RESTORE) self.Bind(wx.EVT_MENU, self.OnTaskBarClose, id=self.TBMENU_CLOSE) def CreatePopupMenu(self): """ Right Click Popup Menu This method is called by the base class when it needs to popup the menu for the default EVT_RIGHT_DOWN event. """ menu = wx.Menu() menu.Append(self.TBMENU_RESTORE, "Restore Window") menu.Append(self.TBMENU_CLOSE, "Exit APP") return menu def MakeIcon(self, img): """ Return the generated Icon. This function is required because various platforms have different requirements for the icon size. """ if "wxMSW" in wx.PlatformInfo: img = img.Scale(16, 16) elif "wxGTK" in wx.PlatformInfo: img = img.Scale(22, 22) icon = wx.IconFromBitmap(img.ConvertToBitmap() ) return icon def OnTaskBarActivate(self, evt): ''' Activation code - say DoubleClick etc. It raises the window ''' if self.frame.IsIconized(): self.frame.Iconize(False) if not self.frame.IsShown(): self.frame.Show(True) self.frame.Raise() def OnTaskBarClose(self, evt): ''' On right click and close, the frame itself is closed ''' wx.CallAfter(self.frame.Close) class APPFrame(wx.Frame): def __init__(self, parent, id, title, size=None): wx.Frame.__init__(self, parent, id, title, size=size, pos=wx.Point(400, 300)) icon = self.get_icon() self.SetIcon(icon) self.tbicon = APPTaskBarIcon(self) self.Show(True) self.Bind(wx.EVT_CLOSE, self.OnCloseWindow) self.Bind(wx.EVT_ICONIZE, self.OnIconify) def on_login_click(self, username, password): self.Hide() def OnCloseWindow(self, event): if self.tbicon is not None: self.tbicon.Destroy() event.Skip() def OnIconify(self, event): self.Hide() def get_image(self): img= wx.Image(opj('client.ico'), wx.BITMAP_TYPE_ICO) return img def get_icon(self): img = self.get_image() icon = wx.IconFromBitmap(img.ConvertToBitmap() ) return icon if __name__ == '__main__': main() -------------- If somebody can help me out, I would be much obliged.

2 1

Re: [Twisted-Python] Deferred documentation rewrite
by Andrew Francis Aug. 4, 2009

Aug. 4, 2009

Hi Edward: Message: 2 Date: Mon, 03 Aug 2009 18:00:07 -0400 From: "Edward Z. Yang" <ezyang(a)MIT.EDU> Subject: Re: [Twisted-Python] Deferred documentation rewrite To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <1249336297-sup-7263@javelin> Content-Type: text/plain; charset=UTF-8 >Why callbacks? > - Asynchronous interaction to synchronous interaction > - Delocalized execution (the parser example) > - High level functions in Python review I don't know if this is an aid but here goes: When I was preparing my Pycon 2008 talk, amongst the many papers I read, was the Doug Schmitt's concurrency design pattern papers including the original Reactor Pattern paper. I also read the Microsoft "Cooperative Task Management without Manual Stack Management Event-driven Programming is Not the Opposite of Threaded Programming" paper. In the light of those papers, this is the way I see things: 1) Reactors provide a portable form of non-preemptive multitasking. By implication, reactors are schedulers. 2) If you buy Reactor as scheduler, then Deferreds can be viewed as representing a thread (or chain) of execution, each callback analogous to a continuation - the next address to resume execution when a result is ready. 3) One of the main differences between asychronous and synchronous processing becomes who is responsible for setting up shared state between the links in the execution chain. >Quite frankly, I'm stumped on "defining synchronous and asynchronous." Simple definition: In a synchronous call, the caller blocks until a result is ready. Upon return the next statement is executed (barring something like an exception). In an asynchronous call, the caller does not wait for a result and continues. > I just don't know what direction people are coming from. I would suggest most people are trying to solve simple problems and want the least surprise. Unfortunately asynchronous programming has lots of surprises. In case of terminology, try looking up how terms like 'asynchronous' or 'synchronous' as used in a few of the more popular network programming books. Cheers, Andrew

2 1

[Twisted-Python] OS for Twisted Framework to run on
by vitaly＠synapticvision.com Aug. 4, 2009

Aug. 4, 2009

hi, is there any significant reason/advantage to run the Twisted Framework on Linux or on FreeBSD? Appreciate the help.

2 1

[Twisted-Python] Twisted.web.client.getPage with proxy?
by disappearedng Aug. 3, 2009

Aug. 3, 2009

Hello world, 1) I have a proxy server running on my computer on port 8888. It's listening on localhost. I am wondering whether Twisted.web.client.getPage has some sort of proxy kwargs that can do this for me. ( I checked the source and I really doubt it has something like that) What are my alternatives? I have looked at low-level alternatives, and ProxyClient doesn't really seem to fit my situation. 2) I sometimes have 2 proxy servers running. I know urllib has a method which allows you to build opener, then you can access webpages with opener.open('site').read(). Does twisted have something like this which allows me to keep multiple "proxy space" to access different websites at the same time on different proxy servers? Thx

1 0

[Twisted-Python] Questions about adding documentation
by Jessica McKellar Aug. 3, 2009

Aug. 3, 2009

Hi everyone, I would like to help Twisted by adding documentation or reviewing existing documentation. However, I don't see many (any?) unassigned tickets regarding documentation of specific items, and because I am quite new to Twisted this makes it hard for me to determine where you wish I would focus my attention. Where should I focus my attention? Want to open some tickets for me to claim? Is adding to API docs more important than updating the examples and howtos? It looks like to add to the API docs you just update the doc strings for functions and let someone let pydoctor do its magic later. Is that true? Is there a special Twisted pydoctor incant to see how they'd look on the web before doing any hasty patch-submitting? I'd like to update http://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs to include some of this information as well as help I got earlier for updating xhtml documentation. Thanks, -Jessica

15 40

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 2, 2009

Aug. 2, 2009

1 0

[Twisted-Python] ANNOUNCING Tahoe, the Lofty-Atmospheric Filesystem, v1.5
by Zooko Wilcox-O'Hearn Aug. 2, 2009

Aug. 2, 2009

Dear people of the twisted mailing list: This release announcement is relevant to this list because Tahoe-LAFS is built on Twisted's networking and concurrency framework. Also, we strive to emulate the UQDS. Regards, Zooko The Tahoe-LAFS team is pleased to announce the immediate availability of version 1.5 of Tahoe, the Lofty Atmospheric File System. Tahoe-LAFS is the first cloud storage technology which offers security and privacy in the sense that the cloud storage service provider itself can't read or alter your data. Here is the one-page explanation of its unique security and fault-tolerance properties: http://allmydata.org/source/tahoe/trunk/docs/about.html This release is the successor to v1.4.1, which was released April 13, 2009 [1]. This is a major new release, improving the user interface and performance and fixing a few bugs, and adding ports to OpenBSD, NetBSD, ArchLinux, NixOS, and embedded systems built on ARM CPUs. See the NEWS file [2] for more information. In addition to the functionality of Tahoe-LAFS itself, a crop of related projects have sprung up to extend it and to integrate it into operating systems and applications. These include frontends for Windows, Macintosh, JavaScript, and iPhone, and plugins for duplicity, bzr, Hadoop, and TiddlyWiki, and more. See the Related Projects page on the wiki [3]. COMPATIBILITY Version 1.5 is fully compatible with the version 1 series of Tahoe-LAFS. Files written by v1.5 clients can be read by clients of all versions back to v1.0. v1.5 clients can read files produced by clients of all versions since v1.0. v1.5 servers can serve clients of all versions back to v1.0 and v1.5 clients can use servers of all versions back to v1.0. This is the sixth release in the version 1 series. The version 1 series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with Tahoe-LAFS v1. The version 1 series of Tahoe-LAFS is the basis of the consumer backup product from Allmydata, Inc. -- http://allmydata.com . WHAT IS IT GOOD FOR? With Tahoe-LAFS, you can distribute your filesystem across a set of servers, such that if some of them fail or even turn out to be malicious, the entire filesystem continues to be available. You can share your files with other users, using a simple and flexible access control scheme. We believe that the combination of erasure coding, strong encryption, Free/Open Source Software and careful engineering make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or other Cloud storage systems. This software comes with extensive tests, and there are no known security flaws which would compromise confidentiality or data integrity in typical use. (For all currently known issues please see the known_issues.txt file [4].) LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file "COPYING.GPL" [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to wait for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file "COPYING.TGPPL.html" [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with "docs/install.html" [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed thanks to the sponsorship of Allmydata, Inc. [12], a provider of commercial backup services. Allmydata, Inc. created the Tahoe-LAFS project and contributed hardware, software, ideas, bug reports, suggestions, demands, and money (employing several Tahoe-LAFS hackers and instructing them to spend part of their work time on this Free Software project). Also they awarded customized t-shirts to hackers who found security flaws in Tahoe-LAFS (see http://hacktahoe.org ). After discontinuing funding of Tahoe-LAFS R&D in early 2009, Allmydata, Inc. has continued to provide servers, co-lo space and bandwidth to the open source project. Thank you to Allmydata, Inc. for their generous and public-spirited support. This is the second release of Tahoe-LAFS which was created solely as a labor of love by volunteers; developer time is no longer funded by allmydata.com (see [13] for details). Zooko Wilcox-O'Hearn on behalf of the Tahoe-LAFS team Special acknowledgment goes to Brian Warner, whose superb engineering skills and dedication are primarily responsible for the Tahoe implementation, and significantly responsible for the Tahoe design as well, not to mention most of the docs and tests. Tahoe-LAFS wouldn't exist without him. August 1, 2009 Boulder, Colorado, USA P.S. Just kidding about that acronym. "LAFS" actually stands for "Lightweight Authorization File System". Or possibly for "Least-Authority File System". There is no truth to the rumour that it actually stands for "Long-lived Axe-tolerant File System". [1] http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=3853 [2] http://allmydata.org/trac/tahoe/browser/NEWS?rev=4033 [3] http://allmydata.org/trac/tahoe/wiki/RelatedProjects [4] http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt [5] http://allmydata.org/trac/tahoe/browser/COPYING.GPL [6] http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html [7] http://allmydata.org/source/tahoe/trunk/docs/install.html [8] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev [9] http://allmydata.org/trac/tahoe/roadmap [10] http://allmydata.org/trac/tahoe/browser/CREDITS?rev=4035 [11] http://allmydata.org/trac/tahoe/wiki/Dev [12] http://allmydata.com [13] http://allmydata.org/pipermail/tahoe-dev/2009-March/001461.html

1 0

[Twisted-Python] what's a workaround for Ticket #3472 (sendmail has no timeout)?
by Dave Britton Aug. 2, 2009

Aug. 2, 2009

Trying to implement a custom mail service I have run into the problem that this ticket describes: ==== Ticket #3472 (new defect ) Opened 10 months ago Last modified 10 months ago twisted.mail.smtp sendmail() should [have] parameters to be passed for the retry and timeout logic supported by SMTPClientFactory: The method: def sendmail() spawns SMTPSenderFactories without a default timeout value. These cause factories to wait forever if there is no response from the remote server. On prolonged periods of time, stale SMTPSenderFactories accumulates and will cause file descriptors to ran out (Couldn't bind: 24: Too many open files.) ==== What would be the best way for me to work around this? Specifically, when I use twisted.mail.smtp sendmail() for a lengthy list, I fairly quickly get the "too many open files" message. I tried using a coiterator to send small enough batches of emails, waiting for each batch to be done before sending the next, because the "too many files" error seemed to be caused by creating too many deferreds when a too large batch of emails was sent at once. This worked in a test simulation, but with real emails the system never completes the batch, it just hangs waiting for sendmail to return. Is there a best timeout mechanism I should use to force sendmail's return? What I can think of so far are these: option 1: Call setTimeout() on the deferred returned by sendmail despite this warning in the source for Deferred.setTimeout(): "warnings.warn("Deferred.setTimeout is deprecated. Look for timeout support specific to the API you are using instead.", DeprecationWarning, stacklevel=2). or option2: Patch a timeout argument into sendmail to have it passed to SMTPSenderFactory(from_addr, to_addrs, msg, d)? Would this work? I can't figure out how or if SMTPSenderFactory would handle a timeout argument. Sorry for the long message. Thanks for your advice! -Dave

2 1

[Twisted-Python] AssertionError: '' for <module '__init__' from 'nfscraper\__init__.pyc'> not in import cache
by Konrads Smelkovs Aug. 1, 2009

Aug. 1, 2009

Hello, I am running latest Twisted Python release 8.2.0 on Windows / Python 2.6. When running trial on my unittests, I get an AssertionError not in import cache: C:\Documents and Settings\Konrads\My Documents\nfscraper\src>tree /f Folder PATH listing Volume serial number is A899-B4C8 C:. └───nfscraper │ httpclient.py │ httpclient.pyc │ interfaces.py │ interfaces.pyc │ scraper.py │ scraper.pyc │ settings.py │ settings.pyc │ util.py │ util.pyc │ __init__.py │ __init__.pyc │ └───test test_scrapers.py test_scrapers.pyc __init__.py __init__.pyc C:\Documents and Settings\Konrads\My Documents\nfscraper\src>set PYTHONPATH="C:\ Documents and Settings\Konrads\My Documents\nfscraper\src" C:\Documents and Settings\Konrads\My Documents\nfscraper\src>trial nfscraper Traceback (most recent call last): File "C:\Python26\Scripts\trial.py", line 24, in <module> run() File "C:\Python26\lib\site-packages\twisted\scripts\trial.py", line 357, in ru n suite = _getSuite(config) File "C:\Python26\lib\site-packages\twisted\scripts\trial.py", line 315, in _g etSuite return loader.loadByNames(config['tests'], recurse) File "C:\Python26\lib\site-packages\twisted\trial\runner.py", line 647, in loa dByNames for thing in set(things)] File "C:\Python26\lib\site-packages\twisted\trial\runner.py", line 604, in loa dAnything return self.loadPackage(thing, recurse) File "C:\Python26\lib\site-packages\twisted\trial\runner.py", line 543, in loa dPackage pkgobj = modules.getModule(package.__name__) File "C:\Python26\lib\site-packages\twisted\python\modules.py", line 741, in g etModule return theSystemPath[moduleName] File "C:\Python26\lib\site-packages\twisted\python\modules.py", line 677, in _ _getitem__ self._findEntryPathString(moduleObject)), File "C:\Python26\lib\site-packages\twisted\python\modules.py", line 630, in _ findEntryPathString rval, modobj, pformat(self.importerCache)) AssertionError: '' for <module '__init__' from 'nfscraper\__init__.pyc'> not in import cache {'"C:\\Documents and Settings\\Konrads\\My Documents\\nfscraper\\sr c"': <imp.NullImporter object at 0x00A8F028>, 'C:\\Documents and Settings\\Konrads\\My Documents\\nfscraper\\src\\"C:\\Docume nts and Settings\\Konrads\\My Documents\\nfscraper\\src"': <imp.NullImporter obj ect at 0x00A8F488>, 'C:\\Documents and Settings\\Konrads\\My Documents\\nfscraper\\src\\"C:\\Docume nts and Settings\\Konrads\\My Documents\\nfscraper\\src"\\twisted\\plugins': <im p.NullImporter object at 0x00A8F528>, 'C:\\Python26': None, 'C:\\Python26\\DLLs': None, 'C:\\Python26\\DLLs\\twisted\\plugins': <imp.NullImporter object at 0x00A8F538> , 'C:\\Python26\\Scripts': None, 'C:\\Python26\\Scripts\\trial.py': <imp.NullImporter object at 0x00A8F498>, 'C:\\Python26\\Scripts\\twisted\\plugins': <imp.NullImporter object at 0x00A8F5 00>, 'C:\\Python26\\lib': None, 'C:\\Python26\\lib\\ctypes': None, 'C:\\Python26\\lib\\distutils': None, 'C:\\Python26\\lib\\encodings': None, 'C:\\Python26\\lib\\lib-tk': None, 'C:\\Python26\\lib\\lib-tk\\twisted\\plugins': <imp.NullImporter object at 0x00 A8F550>, 'C:\\Python26\\lib\\logging': None, 'C:\\Python26\\lib\\plat-win': <imp.NullImporter object at 0x00A8F490>, 'C:\\Python26\\lib\\plat-win\\twisted\\plugins': <imp.NullImporter object at 0x 00A8F548>, 'C:\\Python26\\lib\\site-packages': None, 'C:\\Python26\\lib\\site-packages\\PIL': None, 'C:\\Python26\\lib\\site-packages\\PIL\\twisted\\plugins': <imp.NullImporter ob ject at 0x00A8F560>, 'C:\\Python26\\lib\\site-packages\\Pythonwin': None, 'C:\\Python26\\lib\\site-packages\\Pythonwin\\twisted\\plugins': <imp.NullImpor ter object at 0x00A8F578>, 'C:\\Python26\\lib\\site-packages\\beautifulsoup-3.1.0.1-py2.6.egg': <zipimport er object "C:\Python26\lib\site-packages\beautifulsoup-3.1.0.1-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\beautifulsoup-3.1.0.1-py2.6.egg\\twisted\\pl ugins': <zipimporter object "C:\Python26\lib\site-packages\beautifulsoup-3.1.0.1 -py2.6.egg\twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\django-1.1_beta_1-py2.6.egg': None, 'C:\\Python26\\lib\\site-packages\\django-1.1_beta_1-py2.6.egg\\twisted\\plugin s': <imp.NullImporter object at 0x00A8F508>, 'C:\\Python26\\lib\\site-packages\\django_authopenid-1.0.1-py2.6.egg': None, 'C:\\Python26\\lib\\site-packages\\django_authopenid-1.0.1-py2.6.egg\\twisted\\ plugins': <imp.NullImporter object at 0x00A8F510>, 'C:\\Python26\\lib\\site-packages\\django_registration-0.7-py2.6.egg': <zipimpo rter object "C:\Python26\lib\site-packages\django_registration-0.7-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\django_registration-0.7-py2.6.egg\\twisted\\ plugins': <zipimporter object "C:\Python26\lib\site-packages\django_registration -0.7-py2.6.egg\twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\docutils-0.5-py2.6.egg': None, 'C:\\Python26\\lib\\site-packages\\docutils-0.5-py2.6.egg\\twisted\\plugins': < imp.NullImporter object at 0x00A8F520>, 'C:\\Python26\\lib\\site-packages\\epydoc-3.0.1-py2.6-win32.egg': None, 'C:\\Python26\\lib\\site-packages\\epydoc-3.0.1-py2.6-win32.egg\\twisted\\plugi ns': <imp.NullImporter object at 0x00A8F518>, 'C:\\Python26\\lib\\site-packages\\python_openid-2.2.4-py2.6.egg': <zipimporter object "C:\Python26\lib\site-packages\python_openid-2.2.4-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\python_openid-2.2.4-py2.6.egg\\twisted\\plug ins': <zipimporter object "C:\Python26\lib\site-packages\python_openid-2.2.4-py2 .6.egg\twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c9-py2.6.egg': <zipimporter ob ject "C:\Python26\lib\site-packages\setuptools-0.6c9-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c9-py2.6.egg\\twisted\\plugins ': <zipimporter object "C:\Python26\lib\site-packages\setuptools-0.6c9-py2.6.egg \twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\twisted': None, 'C:\\Python26\\lib\\site-packages\\twisted\\application': None, 'C:\\Python26\\lib\\site-packages\\twisted\\cred': None, 'C:\\Python26\\lib\\site-packages\\twisted\\internet': None, 'C:\\Python26\\lib\\site-packages\\twisted\\lore': None, 'C:\\Python26\\lib\\site-packages\\twisted\\lore\\scripts': None, 'C:\\Python26\\lib\\site-packages\\twisted\\persisted': None, 'C:\\Python26\\lib\\site-packages\\twisted\\plugins': None, 'C:\\Python26\\lib\\site-packages\\twisted\\python': None, 'C:\\Python26\\lib\\site-packages\\twisted\\scripts': None, 'C:\\Python26\\lib\\site-packages\\twisted\\trial': None, 'C:\\Python26\\lib\\site-packages\\twisted\\web': None, 'C:\\Python26\\lib\\site-packages\\twisted\\web2': None, 'C:\\Python26\\lib\\site-packages\\twisted\\words': None, 'C:\\Python26\\lib\\site-packages\\win32': None, 'C:\\Python26\\lib\\site-packages\\win32\\lib': None, 'C:\\Python26\\lib\\site-packages\\win32\\lib\\twisted\\plugins': <imp.NullImpo rter object at 0x00A8F570>, 'C:\\Python26\\lib\\site-packages\\win32\\twisted\\plugins': <imp.NullImporter object at 0x00A8F568>, 'C:\\Python26\\lib\\site-packages\\zope': None, 'C:\\Python26\\lib\\site-packages\\zope\\interface': None, 'C:\\Python26\\lib\\twisted\\plugins': <imp.NullImporter object at 0x00A8F540>, 'C:\\Python26\\twisted\\plugins': <imp.NullImporter object at 0x00A8F558>, 'C:\\WINDOWS\\system32\\python26.zip': <imp.NullImporter object at 0x00A8F030>, 'C:\\WINDOWS\\system32\\python26.zip\\twisted\\plugins': <imp.NullImporter obje ct at 0x00A8F530>} C:\Documents and Settings\Konrads\My Documents\nfscraper\src> -- Konrads Smelkovs Applied IT sorcery.

1 1

[Twisted-Python] getPage using ssl proxy
by Konrads Smelkovs Aug. 1, 2009

Aug. 1, 2009

Hello, I am writing some scraper scripts and need to pass them through an intercepting proxy. getPage does not support a proxy argument and this code I found on internet won't work with SSL proxy (stalls indefinitely): def getPage(url, contextFactory=None, *args, **kwargs): scheme, host, port, path = _parse(url) factory = HTTPClientFactory(url, *args, **kwargs) if 0: # use a proxy host, port = 'localhost', 8080 factory.path = url if scheme == 'https': from twisted.internet import ssl if contextFactory is None: contextFactory = ssl.ClientContextFactory() reactor.connectSSL(host, port, factory, contextFactory) else: reactor.connectTCP(host, port, factory) return factory.deferred Plain http proxying works. My guess is that there is an issue with self-signed or otherwise invalid certificate the http proxy supplies. Any clues? -- Konrads Smelkovs Applied IT sorcery.

1 1

Re: [Twisted-Python] [Fwd: Re: Deferred documentation rewrite]
by Edward Z. Yang Aug. 1, 2009

Aug. 1, 2009

Excerpts from Itamar Shtull-Trauring's message of Fri Jul 31 22:26:29 -0400 2009: > The problem with this is that it perpetuates the misunderstanding the > Deferreds *make* things asynchronous, even with the intro that says > otherwise. I think it's better to assume already asynchronous code, > handling the transition from synchronous to async in an intro event loop > howto. Either way, the function that the first segment of the new docs do belong somewhere. The documentation that traditionally served this purpose has been removed. As for perpetuating the misunderstanding of Deferreds making things asynchronous, I completely agree! However, I think this is something that can be fixed by spelling out the distinction between "writing asynchronous code" and "interacting with asynchronous code", and not just omitting the important paradigm shift that comes with sync->async. > A better comparative exposition might be with normal callbacks, e.g.: > > "def foo(x, gotResultCallback): pass" vs. "def foo(x): # return > Deferred". > > At the very least having that async but callbacky version in the middle > helps understanding. I briefly gloss on this, but I agree that this is an important point that could be further expanded. We could have implemented asynchronous mechanisms using normal callbacks, but we decided to use Deferreds instead. It's not clear to me if the common case of confusion of Deferreds occurs in people who "know callbacks" but "don't know Deferreds." As an incoming developer who was familiar with asynchronous programming, my primary problem was the ill-defined behavior of callback chains (which I resolved by hunkering down and reading the source code) rather than any fundamental misunderstanding of what Deferreds were supposed to do. > It also omits half the story: how you *create* Deferreds. There should > be a section on that as well. I agree. In fact, it might be worth making the document a little longer to address this point, because I realize now that even if you're not writing asynchronous code, you'll often need to baton Deferreds to make the execution flow work the way you want them to. > An example involving a parser, where you just wave your hands about who > pushes data in to the parser exactly (so no need to go into event loop > details), may work well. In particular, the object that wants the result > of the parsing wants to get parse errors, *not* whoever pushes data in. > Often it's same object, but not always. Deferreds help with that. That's a good distinction. I'll see how I can work that in. Cheers, Edward

2 1