Mailman 3 February 2010 - Twisted

[Twisted-Python] Conch channel blocked
by james bardin 04 Feb '10

04 Feb '10

Hello, I'm trying to get a feel for twisted.conch, and not getting very far. I know the documentation is lacking, but I'm not here to complain, I'm here to figure this out. I only have a cursory knowledge of twisted so far, so that may be some of my problem. I am however fairly familiar with the internals of ssh, so I thought I could start there... I started with the sshsimpleclient example, took some pieces, and began experimenting. The example worked as expected. Next, I wanted to pipe arbitrary data over the channel, as if I called "ssh hostname 'cat > file'" This however fills the remote buffer, and hangs Here's a small example ############### class ClientConnection(connection.SSHConnection): def serviceStarted(self): self.openChannel(CatChannel(conn = self)) class CatChannel(channel.SSHChannel): name = 'session' def channelOpen(self, data): d = self.conn.sendRequest(self, 'exec', common.NS('cat > /tmp/testfile'), wantReply = 1) d.addCallback(self._cbSendRequest) def _cbSendRequest(self, ignored): f = open('testfile') chunk = f.read(4096) while chunk: self.write(chunk) chunk = f.read(4096) f.close() self.conn.sendEOF(self) self.loseConnection() ################ Also, setting the channel name to 'session', sending the 'exec' request with the command as a netstring, etc. seem like something that should be abstracted away. SSHSession with request_exec seemed like the obvious choice, but I had no success in using that class for anything. Does anyone know if any projects making good use of twisted.conch, where I can see the source in action, and in depth? Thanks, -jim

3 3

[Twisted-Python] Using Twisted and GTK2
by Ramiro Alba Queipo 04 Feb '10

04 Feb '10

Hi everybody, I recently new about Twisted and saw it as the best way to deal with a client/server network application (I use python long time ago). Really wonderful. I now intend to write a GTK client that can send commands and transfers files to the server and from it. I started to learn how to write applications not using 'Perspective Broker', but when I reached to the example pbgtk2.py to authenticate and send echo commands, I could no see a way the same with a PB. Is that true?. Is, anyway the best alternative (I mean PB) to write the application I intend to? Thanks in advance Regards -- Ramiro Alba Centre Tecnològic de Tranferència de Calor http://www.cttc.upc.edu Escola Tècnica Superior d'Enginyeries Industrial i Aeronàutica de Terrassa Colom 11, E-08222, Terrassa, Barcelona, Spain Tel: (+34) 93 739 86 46 -- Aquest missatge ha estat analitzat per MailScanner a la cerca de virus i d'altres continguts perillosos, i es considera que està net.

5 7

[Twisted-Python] twisted + virtual synchrony
by K. Richard Pixley 04 Feb '10

04 Feb '10

Has anyone done any work towards integrating any virtual synchrony system, (like spread - http://spread.org), into twisted? --rich

1 0

[Twisted-Python] How do I get data from a Queue (or "outside") into a Telnet Server?
by Mark Bailey 03 Feb '10

03 Feb '10

Good day, everyone: I'm trying to learn Python and Twisted at the same time and having fun (mostly). I'm writing an application that is collecting data from multiple sources, filtering the data, and providing it to users through a Telnet server. I can set up a polling loop for the server but I can't figure out how to get send the data to users connected to my server. I thought I'd put the server in a thread and use a Queue to send data to it. But, I could do the work within the server application... The server I'm playing with, including the polling loop, is: from twisted.conch.telnet import StatefulTelnetProtocol from twisted.internet import reactor, protocol class TelnetEcho(StatefulTelnetProtocol): def lineReceived(self, data): print "Type of self: " + str(type(self)) data = data.rstrip('\n\r') self.sendLine("Unrecognized command: %r\r" % (data,)) def checkforspots(): print "running checkforspots()" reactor.callLater(1.0,checkforspots) def createTelnetServer(): factory = protocol.ServerFactory() instance = TelnetEcho factory.protocol = instance port = reactor.listenTCP(8023,factory) print "Listening on port 8023" print "Type of port: " + str(type(port)) checkforspots() return port if __name__ == "__main__": reactor.callWhenRunning(createTelnetServer) reactor.run() Any suggestions greatly appreciated. I've been unable to find anything using my "google-fu" on this issue - I'm stuck. Thanks. Mark Bailey

2 3

[Twisted-Python] [ANN] Ampoule 0.2.0 released
by Valentino Volonghi 02 Feb '10

02 Feb '10

After a LOT of time I am finally, with Canonical Ltd. (Aaron Bentley) help, happy to release a new version of Ampoule. The changelog is not terribly long, hopefully this means that the library kind of works well for most use cases :). Have fun! For those that don't know Ampoule (a lot): ampoule is a process pool implementation written on top of Twisted Matrix. Its name comes from the use of AMP as the default communication protocol between the pool and all its children. It's different from other alternative solutions because it provides an API that is very close to the ThreadPool built into Twisted itself. It also provides the helper function deferToAMPProcess that is also conceptually similar to deferToThread. Ampoule 0.2.0 (2010-02-02) ========================== Changes -------- - Fixed bug #317287: Twisted was required by setup.py due to import of application code to get the version number. The setup.py now tries to do the same thing but if it fails it uses a separate hardcoded version. - Fixed bug #317077: Ampoule didn't work on windows xp due to childFD customization, now on windows ampoule uses the standard 0,1,2 fds. - Added pyOpenSSL as an explicit dependency since Twisted doesn't require it but AMP does. - Greatly simplify setup.py and support plain distutils too. - Bootstrap code now supports context managers - Support for execution deadline on child calls - Parametrize the timeout signal to send to the child - Pass arguments directly from the ProcessPool object for the child process creation. -- Valentino Volonghi aka Dialtone Now Running MacOSX 10.6 http://www.adroll.com/

3 2

Re: [Twisted-Python] understanding deferreds
by Terry Jones 02 Feb '10

02 Feb '10

>>>>> "Michael" == Michael P Soulier <msoulier(a)digitaltorque.ca> writes: Michael> I'm a bit confused as to the "visual explanation". It seems like Michael> if you have a single callback and errback, that if the errback Michael> wants to swallow the error, the callback will not be called at Michael> all, since deferred processing will look for the second callback Michael> at that point by the diagram. Michael> So, right now I'm calling my own callback from the errback if I Michael> want to swallow the error. Is there a better way? Yes. Add the errback to the deferred and then add the callback. d = somethingReturningADeferred() d.addErrback(cleanup) d.addCallback(dealWithResult) If cleanup doesn't return the failure object it's passed or raise some an exception, processing will pass to dealWithResult. Note that dealWithResult will get as an argument whatever cleanup returns (that means None if cleanup doesn't explicitly return something). You're thinking about this in the right way. Understanding the difference between the above and d = somethingReturningADeferred() d.addCallbacks(dealWithResult, cleanup) is important. Terry

4 4

[Twisted-Python] ANNOUNCING Tahoe, the Least-Authority File System, v1.6
by Zooko O'Whielacronx 02 Feb '10

02 Feb '10

This highly reliable distributed data store is built on top of Twisted. Thanks for the excellent tool! Regards, Zooko Wilcox-O'Hearn ANNOUNCING Tahoe, the Least-Authority File System, v1.6 We are pleased to announce the immediate availability of version 1.6 of Tahoe-LAFS, an extremely reliable distributed data store. Tahoe-LAFS is the first cloud storage system which offers "provider-independent security" -- meaning that not even your cloud service provider can read or alter your data without your consent. Here is the one-page explanation of its unique security and fault-tolerance properties: http://allmydata.org/source/tahoe/trunk/docs/about.html Tahoe-LAFS v1.6.0 is the successor to v1.5.0, which was released August 1, 2009 [1]. This release includes major performance improvements, usability improvements, and one major new feature: deep-immutable directories (cryptographically unalterable permanent snapshots). See the NEWS file [2] for details. WHAT IS IT GOOD FOR? With Tahoe-LAFS, you spread your filesystem across multiple servers, and even if some of the servers fail or are taken over by an attacker, the entire filesystem continues to work correctly, and continues to preserve your privacy and security. You can easily and securely share chosen files and directories with others. In addition to the core storage system itself, volunteers have developed related projects to integrate it with other tools. These include frontends for Windows, Macintosh, JavaScript, and iPhone, and plugins for Hadoop, bzr, duplicity, TiddlyWiki, and more. As of this release, contributors have added an Android frontend and a working read-only FUSE frontend. See the Related Projects page on the wiki [3]. We believe that the combination of erasure coding, strong encryption, Free/Open Source Software and careful engineering make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or other Cloud storage systems. This software is developed under thorough unit tests, and there are no known bugs or security flaws which would compromise confidentiality or data integrity under normal use. (For all currently known issues please see the known_issues.txt file [4].) COMPATIBILITY This release is fully compatible with the version 1 series of Tahoe-LAFS. Clients from this release can write files and directories in the format used by clients of all versions back to v1.0 (which was released March 25, 2008). Clients from this release can read files and directories produced by clients of all versions since v1.0. Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0. This is the seventh release in the version 1 series. The version 1 series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with Tahoe-LAFS v1. In addition, version 1.6 improves forward-compatibility with planned future directory formats, allowing updates to a directory containing both current and future links, without loss of information. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file "COPYING.GPL" [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to wait for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file "COPYING.TGPPL.html" [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with "docs/install.html" [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed thanks to the sponsorship of Allmydata, Inc. [12], a provider of commercial backup services. Allmydata founded the Tahoe-LAFS project and contributed hardware, software, ideas, bug reports, suggestions, demands, and they employed several Tahoe-LAFS hackers and instructed them to spend part of their work time on this Free Software project. Also they awarded customized t-shirts to hackers who found security flaws in Tahoe-LAFS (see the Hack Tahoe-LAFS Hall Of Fame [13]). After discontinuing funding of Tahoe-LAFS R&D in early 2009, Allmydata, Inc. has continued to provide servers, co-lo space, bandwidth, and small personal gifts as tokens of appreciation. (Also they continue to provide bug reports.) Thank you to Allmydata, Inc. for their generous and public-spirited support. This is the third release of Tahoe-LAFS to be created solely as a labor of love by volunteers. Thank you very much to the dedicated team of "hackers in the public interest" who make Tahoe-LAFS possible. Zooko Wilcox-O'Hearn on behalf of the Tahoe-LAFS team February 1, 2010 Boulder, Colorado, USA [1] http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=4042 [2] http://allmydata.org/trac/tahoe/browser/NEWS?rev=4189 [3] http://allmydata.org/trac/tahoe/wiki/RelatedProjects [4] http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt [5] http://allmydata.org/trac/tahoe/browser/COPYING.GPL [6] http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html [7] http://allmydata.org/source/tahoe/trunk/docs/install.html [8] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev [9] http://allmydata.org/trac/tahoe/roadmap [10] http://allmydata.org/trac/tahoe/browser/CREDITS?rev=4186 [11] http://allmydata.org/trac/tahoe/wiki/Dev [12] http://allmydata.com [13] http://hacktahoe.org

1 0

Re: [Twisted-Python] understanding deferreds
by Terry Jones 02 Feb '10

02 Feb '10

>>>>> "Alexey" == twisted-web <twisted-web(a)udmvt.ru> writes: Alexey> Is it true, that adding many callback functions as filters is Alexey> elegant, but impractical solution? Many callbacks as a filter is elegant & practical. But there's a difference with errbacks, as people are touching on. That's, with a bit of hand-waving, because when things are going right there's only one thing happening - you're in the middle of a computation figuring out the single result. With errors though, there are multiple types of errors, many ways of dealing with them, multiple different possible outcomes when an error occurs. So the callback chain is great when everything is working, but the errback chain can sometimes feel like it's not quite the right tool for the job. But there are some techniques to mitigate that, and I think they work quite well. Alexey> Since there is only a [linear] list of pairs (callback, errback), Alexey> the last errbacks have to be complex to distinguish the failures, Alexey> that came from the previous callback from failures, that traverse Alexey> the errback chain from the beginning. Alexey> Should I only use errbacks as a means of catching errors only from Alexey> who have produced the Deferred? And never use them to catch Alexey> errors, that came from my callbacks? No. One thing to think about is the flow down the call/errback chain. You can do something like this. d = IReturnADeferred() d.addErrback(e1) d.addCallback(c1) d.addErrback(e2, 'first') d.addCallback(c2) d.addCallback(c3) d.addErrback(e2, 'second') Here e1 will only get errors created by IReturnADeferred. If IReturnADeferred has no problem, you're into c1 and e1 is "behind" you on the call/errback chain. Note that I add e2 to the errback chain twice. That's a way of having the same function be called for (perhaps) the same reason, but on the chain in 2 places. By the time the flow of control gets to c2 or c3, e2 can only be called with 'second' as its argument. Alexey> Nevertheless the Deferred mechanism always catch the exceptions in Alexey> my callbacks, I have to always catch them myself and never let them Alexey> out uncontrolled (since I cannot distinguish, say TypeError that Alexey> came from the first callback from TypeError, that came from the Alexey> next callback, and they are not the same for me, since I try to do Alexey> a cleanup)? But then I have to translate all errors to my invented Alexey> exceptions, so they will differ. See the above example. Alexey> Or just tell me, that I have missed something, I'm still getting Alexey> into the theme! The other thing that's very useful is to write errbacks like this: def e1(fail): fail.trap(TypeError, KeyError) # Do stuff to handle just TypeError, KeyError def e2(fail): fail.trap(MyException) # Do stuff to handle MyException then you just do d = IReturnADeferred() d.addErrback(e1) d.addErrback(e2) The fail.trap will cause the function to immediately return the current failure if it's not one of the mentioned types. See also fail.check (which is what trap uses). I.e., go read the source of twisted.python.failure That approach is nice because you can write your err handling methods completely separately, and can probably not worry about the order you list them in, etc. The failure just falls down the errback chain until it hits something that can deal with it ("trap" it). Another thing you could do (a bit of a hack) is to add an attribute to a failure object to mark that you've processed it (e.g., that it has been logged). And you might want to read the recent post to this list by Brian Warner (which I've also been meaning to reply to - hi Brian!). See http://twistedmatrix.com/pipermail/twisted-python/2010-January/021330.html Probably others have suggestions on nice ways to handle particular cases. Those are just some that I've used. Terry

1 0

Re: [Twisted-Python] PyCon 2010 Twisted Sprint
by Terry Jones 01 Feb '10

01 Feb '10

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> I will be there for one day of the sprints. I'm interested in web2, Glyph> deferred cancellation, endpoints, and getting a new release out. I could probably stay for most/all of the 22nd. Doing some work on deferred cancellation would be good. Terry

1 0