Mailman 3 February 2011 - Twisted

[Twisted-Python] ANNOUNCING Tahoe, the Least-Authority File System, v1.8.2
by Zooko O'Whielacronx 02 Feb '11

02 Feb '11

ANNOUNCING Tahoe, the Least-Authority File System, v1.8.2 The Tahoe-LAFS team is pleased to announce the immediate availability of version 1.8.2 of Tahoe-LAFS, an extremely reliable distributed storage system. Get it here: http://tahoe-lafs.org/source/tahoe/trunk/docs/quickstart.html Tahoe-LAFS is the first distributed storage system to offer "provider-independent security" — meaning that not even the operators of your storage servers can read or alter your data without your consent. Here is the one-page explanation of its unique security and fault-tolerance properties: http://tahoe-lafs.org/source/tahoe/trunk/docs/about.html The previous stable release of Tahoe-LAFS was v1.8.1, which was released October 28, 2010 [1]. v1.8.2 is a stable bugfix release, adding compatibility with the recently-released Twisted-10.2, and correcting a number of minor issues. See the NEWS file [2] for details. WHAT IS IT GOOD FOR? With Tahoe-LAFS, you distribute your filesystem across multiple servers, and even if some of the servers fail or are taken over by an attacker, the entire filesystem continues to work correctly, and continues to preserve your privacy and security. You can easily share specific files and directories with other people. In addition to the core storage system itself, volunteers have built other projects on top of Tahoe-LAFS and have integrated Tahoe-LAFS with existing systems, including Windows, JavaScript, iPhone, Android, Hadoop, Flume, Django, Puppet, bzr, mercurial, perforce, duplicity, TiddlyWiki, and more. See the Related Projects page on the wiki [3]. We believe that strong cryptography, Free and Open Source Software, erasure coding, and principled engineering practices make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or cloud storage. This software is developed under test-driven development, and there are no known bugs or security flaws which would compromise confidentiality or data integrity under recommended use. (For all important issues that we are currently aware of please see the known_issues.rst file [4].) COMPATIBILITY This release is compatible with the version 1 series of Tahoe-LAFS. Clients from this release can write files and directories in the format used by clients of all versions back to v1.0 (which was released March 25, 2008). Clients from this release can read files and directories produced by clients of all versions since v1.0. Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0. This is the thirteenth release in the version 1 series. This series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with this series. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file "COPYING.GPL" [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to delay for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file "COPYING.TGPPL.html" [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with "docs/quickstart.html" [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed by Allmydata, Inc., a provider of commercial backup services. After discontinuing funding of Tahoe-LAFS R&D in early 2009, they continued to provide servers, bandwidth, small personal gifts as tokens of appreciation, and bug reports. Google, Inc. sponsored Tahoe-LAFS development as part of the Google Summer of Code 2010. They awarded four sponsorships to students from around the world to hack on Tahoe-LAFS that summer. Thank you to Allmydata and Google for their generous and public-spirited support. HACK TAHOE-LAFS! If you can find a security flaw in Tahoe-LAFS which is serious enough that we feel compelled to warn our users and issue a fix, then we will award you with a customized t-shirts with your exploit printed on it and add you to the "Hack Tahoe-LAFS Hall Of Fame" [12]. ACKNOWLEDGEMENTS This is the seventh release of Tahoe-LAFS to be created solely as a labor of love by volunteers. Thank you very much to the team of "hackers in the public interest" who make Tahoe-LAFS possible. Brian Warner on behalf of the Tahoe-LAFS team January 30, 2011 San Francisco, California, USA [1] http://tahoe-lafs.org/trac/tahoe/browser/relnotes.txt?rev=4865 [2] http://tahoe-lafs.org/trac/tahoe/browser/NEWS?rev=5000 [3] http://tahoe-lafs.org/trac/tahoe/wiki/RelatedProjects [4] http://tahoe-lafs.org/trac/tahoe/browser/docs/known_issues.rst [5] http://tahoe-lafs.org/trac/tahoe/browser/COPYING.GPL [6] http://tahoe-lafs.org/source/tahoe/trunk/COPYING.TGPPL.html [7] http://tahoe-lafs.org/source/tahoe/trunk/docs/quickstart.html [8] http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev [9] http://tahoe-lafs.org/trac/tahoe/roadmap [10] http://tahoe-lafs.org/trac/tahoe/browser/CREDITS?rev=5000 [11] http://tahoe-lafs.org/trac/tahoe/wiki/Dev [12] http://tahoe-lafs.org/hacktahoelafs/

1 0

[Twisted-Python] Guidance needed on serial device interaction
by Jason Heeris 02 Feb '11

02 Feb '11

This is a bit long, sorry... I have a PyGTK program that uses threads and pyserial's blocking methods to interact with an RS232 connected device. I'd like to throw out the threading awfulness and redo it in Twisted, if possible, but I'm a little lost. The real protocol is a bit convoluted, but basically: - You can issue single character commands to the device that give a fixed length response, such as sending 'C' and getting an eight-digit hex string back (the program flash CRC) - You can put the device into "programming mode" (command 'P'), where it takes an arbitrary length sequence of records, verifying each record and stopping when it sees the special "end record" - The device will send back '!' to indicate an error - The device will send back '>' to indicate that it's ready for more commands I know I need to use the SerialPort transport, and since that takes a protocol I've tried to sketch one out but can't seem to get very far. I want to have methods that can be called from the UI, returning deferreds to which UI responses can be added as callbacks. I also want to be able to monitor progress of the programming, so I need a callback for that separate from the deferred itself. I figure that calling, say, device.program(program, progress_cb) should return a Deferred and queue the command, and that somehow the dataReceived() method should be the start of a chain of events that either: - updates state and waits for the next piece of data, or - calls back on the appropriate Deferred I'd also like to structure things so that successive calls to the DeviceProtocol object queue up, something like: ---- class DeviceProtocol(Protocol): def dataReceived(self, data): # ...? def checksum(self): res = defer.Deferred() # When ready, send 'C' over the serial line (ie. # self.transport.write('C')) # Somehow callback via a deferred when the checksum comes back return res def program(self, program_records, progress_cb): res = defer.Deferred() # When ready, send 'P' over the serial line # Write all the data in "program_records", checking each # response, calling back via a deferred when done. # ... return res def connectionLost(self): # maybe have something here, like calling the errback of all # pending deferreds def go(reactor): # callbacks/errbacks not shown device = DeviceProtocol() transport = SerialPort(device, portname, reactor) program_result_1 = device.program(records, good_programming_progress) program_result_1.addCallbacks(programming_done, programming_err) checksum_result = device.checksum() checksum_result.addCallbacks(checksum, checksum_err) # This will fail at some point: program_result_2 = device.program(bad_records) cp_deferred.addCallbacks(programming_done, programming_err) if __name__ == "__main__": reactor.callWhenRunning(go, reactor) reactor.run() ---- But I'm really lost as to how to start structuring things within the protocol object itself. Should I even be doing this in a Protocol subclass, or should I be putting some of this functionality into a Factory of some sort (eg. a ClientFactory)? And if so, how do I actually connect the factory to the serial port transport? Or am I on the wrong track altogether? More to the point, has a problem like this already been solved somewhere? Is the solution obvious to Twisted gurus? Any help would be appreciated. Cheers, Jason

4 6

[Twisted-Python] Fingering Finger
by Tom Davis 01 Feb '11

01 Feb '11

In this thread, I hope to find a resolution to the issue of the Finger tutorial and efforts to sufficiently improve it or remove it. In the course of reviewing documentation-related tickets, I stumbled upon #1148 (http://twistedmatrix.com/trac/ticket/1148). Therein, Glyph first(?) put down a lot of things we've been discussing and agreeing upon in the Refactoring Documentation thread. One of the issues still up for debate is whether or not the Finger tutorial is sufficiently strong to survive the documentation overhaul. There are various points against it right now: - It isn't tested or even test*able* - It doesn't cover "best practices" as they relate to writing testable, maintainable code, etc. - It attempts to implement basically every main Twisted concept, often in contrived or poorly-executed ways - It has been said it has, "...at best, the potential for mediocrity." There are also enough tickets related to refactoring / rewriting it that a resolution would make a significant dent in the list of stale documentation tickets. Among these two year-old tickets are: - http://twistedmatrix.com/trac/ticket/532 - Big jump from finger18.py to finger19.py in tutorial - http://twistedmatrix.com/trac/ticket/626 - Split tutorial finger code into libraries - http://twistedmatrix.com/trac/ticket/2205 - Documentation codelistings need updating and tests This shouldn't be a blocker on anything Kevin and I are doing, but it'd be nice to concurrently have discussions on issues we'll need to address later. I'm also pretty anal about ticket lists and if these aren't going anywhere I'd love to close them ;) Cheers, Tom

6 5

[Twisted-Python] High-level view of Twisted
by Jason Heeris 01 Feb '11

01 Feb '11

Hi, I'd like to come to grips with Twisted, and so far I've been following and tweaking example after example. But I'm finding it incredibly hard to get a high level view of the Twisted "landscape" — and because I don't know what I don't know, it's hard to know what questions to even ask. I want to get to the point where I can just sit down and know where to start, and hopefully someone here can give me a bit of direction. I understand many of the microscopic components. Deferred results, file descriptors and sockets, the reactor loop concept — I get those. I've read the tutorial[1], another Twisted Introduction[2], and about 50% of the API documentation (possibly not the right 50%, though). But I just can't seem to wrap my head around the interfaces, factories and wiring it all together. Maybe some examples of the speedbumps I keep encountering will help: 1. The "finger" intro "Drop Connections" example[3] just straight out uses the self.transport member of a protocol. But how do I, a Twisted newbie, know that this even exists? After hours of digging around, I find it in the docs for a BaseProtocol method[4]. Later on, this same example uses the "self.factory" member. I still haven't found that one. Where do I look up these assumed-to-exist members? How do I know what other members exist? 2. What is a Factory, anyway? What are they *for*? 3. The finger example goes over writing a server — what about an asynchronous client that must follow a particular protocol? Where do I start with that? Is there an example? 4. What if I'm not interested in networking? I primarily deal with serial lines, files and subprocesses... are there examples for those? Points #3 and #4 are my motivation for using Twisted — I am currently mired in a pygtk program that uses threads and all sorts of locks and synchronisation mechanisms to send and receive data over a serial line, updating UI with progress, errors and success. I'd like to see if using Twisted simplifies things a bit, and I'll ask a more specific question about how to structure that when I can condense it to a reasonable level :) I hope I'm not being too vague, but Twisted is rather large, conceptually, to absorb. I keep being pointed to the documentation for specific parts, but I have no idea how they fit together and that's what I'm really after. Thanks, Jason [1] http://twistedmatrix.com/documents/current/core/howto/tutorial/index.html [2] http://krondo.com/?page_id=1327 [3] http://twistedmatrix.com/documents/current/core/howto/tutorial/intro.html#a… [4] http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Ba…

3 5

[Twisted-Python] A Python enum library with a Twist
by Jonathan Jacobs 31 Jan '11

31 Jan '11

Hi, First off there isn't actually a twist, it's just for Twisted addressing the points in #4671 [1]. Sorry to disappoint. Secondly I realise there are already at least a thousand Python enum libraries, most of which I haven't explored, feel free to mention these with supporting arguments. My point of view is that a great many number of these enum libraries go to long lengths to make stuff too "cute". While having less not-networking-related code in Twisted may be a more desirable route, Twisted makes enough use of enum-esque things in the current code base that it may be easier (maybe even warranted) to have our own implementation that can cater directly to the needs of Twisted without being subjected to the usual irritations of third-party software; also Twisted's high-qualilty coding and testing standards are only a good thing. The current code is hosted on Launchpad [2] and the source code is viewable on the web [3]. I am aware that there are currently no unittests or any kind of useful documentation, hopefully the examples are good enough for now. I'm hoping for some feedback to decide whether to press ahead or turn back. I've tried to look at as many Twisted enum use-cases as possible, without trying to cater to every one individually, however I do still have a few questions: 1. #4671 uses an "asInt" method in the examples, this is obviously going to require changing all code that uses these enums (of which there is plenty.) Is this a good idea? It seems easy to forget that "F_FOO | F_BAR" is not giving you a number. 2. I use "__int__", I don't know if this is horrible or not and whether I should just use the "asInt" method proposed in the ticket. 3. It seems to me that there are two general cases: 1. The enum just holds values that have an interesting meaning on their own (e.g. AMP special keys (ASK, ANSWER, etc.), IRC status codes (RPL_WHOIS, etc.)) and combining them makes no sense. 2. The enum holds values that are intended to be combined. i.e. flags. Does it make sense to try and combine this behaviour into the same type? Currently it is and you get an exception when doing certain things (none of which make any sense, in this context): >>> (Words.AYE | Words.BEE) <EnumValueGroup '|': [<EnumValue Words.AYE: 'a'>, <EnumValue Words.BEE: 'b'>]> >>> (Words.AYE | Words.BEE).value Traceback (most recent call last): File "<stdin>", line 1, in <module> File "enum.py", line 77, in value return reduce(self.op, values) TypeError: unsupported operand type(s) for |: 'str' and 'str' >>> Words.fromValue('a') <EnumValue Words.AYE: 'a'> >>> Words.fromValue('c') # This value is not in the enumeration and ORing values to get it is impossible. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "enum.py", line 133, in fromValue if target - v < 0: TypeError: unsupported operand type(s) for -: 'str' and 'str' Comments appreciated. [1] <http://twistedmatrix.com/trac/ticket/4671> [2] <http://code.launchpad.net/~jjacobs/+junk/twistedenum> [3] <http://bazaar.launchpad.net/~jjacobs/+junk/twistedenum/view/head:/enum.py> -- Jonathan

4 5

Twisted February 2011