Mailman 3 February 2009 - Twisted

[Twisted-Python] Communication with Async Java NIO Socketchannel
by Johann Höchtl 15 Feb '09

15 Feb '09

Is it possible to communicate with a nonblocking Java SocketChannel from Java.nio? What has to be attended?

2 1

[Twisted-Python] ANNOUNCING allmydata.org "Tahoe", the Least-Authority Filesystem, v1.3
by zooko 15 Feb '09

15 Feb '09

Folks: This project is all based on Twisted, and many of the Related Projects that are springing up (see below) are, too. Thanks to everyone for making Twisted into such a high-quality tool! Regards, Zooko ANNOUNCING allmydata.org "Tahoe", the Least-Authority Filesystem, v1.3 We are pleased to announce the release of version 1.3.0 of "Tahoe", the Least Authority Filesystem. Tahoe-LAFS is a secure, decentralized, fault-tolerant filesystem. All of the source code is available under a choice of two Free Software, Open Source licences. This filesystem is encrypted and distributed over multiple peers in such a way it continues to function even when some of the peers are unavailable, malfunctioning, or malicious. Here is the one-page explanation of the security and fault-tolerance properties that it offers: http://allmydata.org/source/tahoe/trunk/docs/about.html This is the successor to v1.2, which was released July 21, 2008 [1]. This is a major new release, adding a repairer, an efficient backup command, support for large files, an (S)FTP server, and much more. See the NEWS file [2] and the known_issues.txt file [3] for more information. In addition to the many new features of Tahoe itself, a crop of related projects have sprung up, including Tahoe frontends for Windows and Macintosh, two front-ends written in JavaScript, a Tahoe plugin for duplicity, a Tahoe plugin for TiddlyWiki, a project to create a new backup tool, CIFS/SMB integration, an iPhone app, and three incomplete Tahoe frontends for FUSE. See Related Projects on the wiki: [4]. COMPATIBILITY The version 1 branch of Tahoe is the basis of the consumer backup product from Allmydata, Inc. -- http://allmydata.com . Tahoe v1.3 is fully compatible with the version 1 branch of Tahoe. Files written by v1.3 clients can be read by clients of all versions back to v1.0 unless the file is too large -- files greater than about 12 GiB (depending on the configuration) can't be read by older clients. v1.3 clients can read files produced by clients of all versions since v1.0. v1.3 servers can serve clients of all versions back to v1.0 and v1.3 clients can use servers of all versions back to v1.0 (but can't upload large files to them). This is the fourth release in the version 1 series. We believe that this version of Tahoe is stable enough to rely on as a permanent store of valuable data. The version 1 branch of Tahoe will be actively supported and maintained for the forseeable future, and future versions of Tahoe will retain the ability to read files and directories produced by Tahoe v1 for the forseeable future. WHAT IS IT GOOD FOR? With Tahoe, you can distribute your filesystem across a set of computers, such that if some of the computers fail or turn out to be malicious, the entire filesystem continues to be available, thanks to the remaining computers. You can also share your files with other users, using a simple and flexible access control scheme. Because this software is new, we do not categorically recommend it as the sole repository of data which is extremely confidential or precious. However, we believe that erasure coding, strong encryption, Free/Open Source Software and careful engineering make Tahoe safer than common alternatives, such as RAID, removable drive, tape, or "on-line storage" or "Cloud storage" systems. This software comes with extensive unit tests [5], and there are no known security flaws which would compromise confidentiality or data integrity. (For all currently known issues please see the known_issues.txt file [2].) This release of Tahoe is suitable for the "friendnet" use case [6] -- it is easy to create a filesystem spread over the computers of you and your friends so that you can share disk space and files. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file "COPYING.GPL" [7] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1.0. The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to wait for up to twelve months after you redistribute a derived work before releasing the source code of your derived work. See the file "COPYING.TGPPL.html" [8] for the terms of the Transitive Grace Period Public Licence, version 1.0. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe works on Linux, Mac OS X, Windows, Cygwin, and Solaris, and probably most other systems. Start with "docs/install.html" [9]. HACKING AND COMMUNITY Please join us on the mailing list [10]. Patches that extend and improve Tahoe are gratefully accepted -- the RoadMap page [11] shows the next improvements that we plan to make and CREDITS [12] lists the names of people who've contributed to the project. The wiki Dev page [13] contains resources for hackers. SPONSORSHIP Tahoe is sponsored by Allmydata, Inc. [14], a provider of commercial backup services. Allmydata, Inc. created the Tahoe project, and contributes hardware, software, ideas, bug reports, suggestions, demands, and money (employing several Tahoe hackers and instructing them to spend part of their work time on this Free Software project). Also they award customized t-shirts to hackers who find security flaws in Tahoe (see http://hacktahoe.org ). Thank you to Allmydata, Inc. for their generous and public-spirited support. Zooko Wilcox-O'Hearn on behalf of the allmydata.org team Special acknowledgment goes to Brian Warner, whose superb engineering skills and dedication are primarily responsible for the Tahoe implementation, and largely responsible for the Tahoe design as well, not to mention most of the docs and many other things besides. February 13, 2009 Boulder, Colorado, USA [1] http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=2789 [2] http://allmydata.org/trac/tahoe/browser/NEWS [3] http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt [4] http://allmydata.org/trac/tahoe/wiki/RelatedProjects [5] http://allmydata.org/trac/tahoe/wiki/Dev [6] http://allmydata.org/trac/tahoe/wiki/UseCases [7] http://allmydata.org/trac/tahoe/browser/COPYING.GPL [8] http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html [9] http://allmydata.org/source/tahoe/trunk/docs/install.html [10] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev [11] http://allmydata.org/trac/tahoe/roadmap [12] http://allmydata.org/trac/tahoe/browser/CREDITS?rev=2677 [13] http://allmydata.org/trac/tahoe/wiki/Dev [14] http://allmydata.com

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 14 Feb '09

14 Feb '09

1 0

[Twisted-Python] half-closing SSL connection, unexpected connectionLost() error
by Denis Bilenko 13 Feb '09

13 Feb '09

Hi, What's the correct way to call loseWriteConnection on SSL connection? I keep getting "Connection was closed in a non-clean fashion" errors when I do that. Here's a minimal server and a client that trigger the error. Note that the server does not implement IHalfCloseableProtocol and I'd like to keep it that way (primarily because I'm building a client, not a server, so I'd like my client to work with as many servers as possible) --------------------------------------------------------------------------- server.py from twisted.internet.protocol import Factory, Protocol from twisted.internet import reactor class MyProtocol(Protocol): def connectionMade(self): print self.transport.getHost(), self.transport.getPeer() def connectionLost(self, reason): print 'connectionLost %r' % reason.value class MyFactory(Factory): protocol = MyProtocol from twisted.internet import ssl contextFactory = ssl.DefaultOpenSSLContextFactory('valid.key', 'valid.crt', sslmethod=ssl.SSL.SSLv3_METHOD) reactor.listenSSL(10000, MyFactory(), contextFactory) reactor.run() --------------------------------------------------------------------------- client.py import sys from zope.interface import implements from twisted.internet.protocol import ClientFactory, Protocol from twisted.internet.interfaces import IHalfCloseableProtocol from twisted.internet import reactor class MyProtocol(Protocol): implements(IHalfCloseableProtocol) def connectionMade(self): print 'connectionMade %s -> %s' % (self.transport.getHost(), self.transport.getPeer()) self.transport.loseWriteConnection() def readConnectionLost(self): print 'readConnectionLost' def writeConnectionLost(self): print 'writeConnectionLost' def connectionLost(self, reason): print 'connectionLost %r' % reason.value class MyFactory(ClientFactory): protocol = MyProtocol def clientConnectionFailed(self, connector, err): print err.value reactor.stop() from twisted.internet import ssl contextFactory = ssl.ClientContextFactory() reactor.connectSSL(sys.argv[1], int(sys.argv[2]), MyFactory(), contextFactory) reactor.run() --------------------------------------------------------------------------- console1 $ python server.py IPv4Address(TCP, '127.0.0.1', 10000) IPv4Address(TCP, '127.0.0.1', 40291) connectionLost ConnectionDone() --------------------------------------------------------------------------- console2 $ python client.py localhost 10000 connectionMade IPv4Address(TCP, '127.0.0.1', 40291) -> IPv4Address(TCP, 'localhost', 10000) writeConnectionLost connectionLost ConnectionLost() The question is why it's not ConnectionDone but ConnectionLost that client receives in connectionLost callback? If I convert the examples to use TCP, I get ConnectionDone, as expected. Thank you for your time.

2 1

[Twisted-Python] Slow Performance of Twisted Script
by Thomas Greenwood 13 Feb '09

13 Feb '09

Hi everyone, I have been developing my first python script and jumped straight into the world of twisted. I like the concept but I seem to be doing something very wrong. The script takes an NZB file and parses it (using HellaNZB's parser) and then makes a specified number of connections to the news server and checks that all articles are there using the STAT command. I also know nothing about newsservers or nntp so this script is a complete shot in the dark. First time with python, first time with twisted and first time with nntp seems to equal disaster! The problem is that the script is very slow, even with 20 simultaneous connections to the internet I only produce 111k or so of traffic and only achieve 2 article checks per connection per second. On a 24meg connection I can download the whole thing quicker than checking it with my script! Any ideas why it is so slow? I have attached the script. Thanks for any help. Tom Greenwood

1 0

[Twisted-Python] thread + signal + reactor.stop problem
by Gabriel Rossetti 13 Feb '09

13 Feb '09

Hello everyone, I was able to write a small example that reproduces my problem, when I redefine signal and I have threads and I try to stop the reactor, it hangs. I wasn't sure if it was loopingCall or callLater or threads at first, but from my tests it's the threads that cause the problem. I attached the example, it's a modified version of the Twisted echo server/client example. Run the server, run the client, press Ctrl + C in the client's terminal (or Ctrl + break on windows) and it doesn't quit. I still have to test if it's caused by the signal redefinition at all or not. Thank you, Gabriel Rossetti

2 2

[Twisted-Python] SQLAlchemy and Twisted
by alex clemesha 13 Feb '09

13 Feb '09

Hi, I didn't want to hijack the 'Twisted Presentation Materials..' thread, so I started this one ... First off, in the other thread, Glenn, you mentioned Storm. I have heard of Storm, but never got any deeper that noticing that it is still a work in progress (maybe things have settled down ?) If anyone has anything to say on how it might be a good choice with Twisted, please feel free to enlighten us! But, I hoping to solve some immediate problems with Twisted and SQLAlchemy, so ... >> >> * A main "App Server", that controls high level access with "twisted.cred" >> * Web frontend: "twisted.web2" >> * Data: some "twisted.enterprise.adbapi", moving to SQLAlchemy. > > Interesting; how are you handling asynchrony in SA? This is the issue that we're stilling unclear on, and I really would like to know of the best way of integrating Twisted with SQLAlchemy. Twisted devs / experts: Please, could you comment on the right way of using Twisted and SQLAlchemy together. Anyone: If anyone know of existing code that integrates Twisted and SQLAlchemy nicely, please share! RIght away I just went about the problem be wrapping each call to SQLAlchemy in a "deferToThread" inside my "DatabaseManager" class. At startup of my app, I make an instance of this class, and pass it to each "avatar" (in "requestAvatar", in my portal) Here is my Database Manager: (you will actually see that I 'turned off' the deferToThread wrapping for now): http://trac.knoboo.com/browser/branches/knoboo-sqlalchemy-take2/knoboo/data… but I really haven't convinced myself that this is the best way of doing things. Hell, it could be totally wrong, so if anyone has any advice, I would be very appreciative! > I'm aware of sAsync, > but haven't looked much into it. (I'm a step or two behind you, actively > using adbapi, and thinking about moving to SA when I get some breathing > room.) I've looked into SAsync as well, but it is not totally clear to me what the extra benefit of it is (again, anyone who has a good description, please tell) The main developer of sAsync use to post to this mailing list, but I haven't seen him post in a fairly long while. So, if others are thinking about using SQLAlchemy with Twisted, let's discuss it more. Thanks very much, Alex

5 5

[Twisted-Python] GSoC
by Esteve Fernandez 12 Feb '09

12 Feb '09

Hi all, the GSoC FAQ was recently published [1] and I wondered if Twisted could be a mentoring organization for the next installment of the GSoC program. I think it's a good opportunity to bring new developers to Twisted by having a mentor. I was a GSoC student last year and it was pretty fun. I'd offer my help to mentor a student, but I think I couldn't become an official mentor (though I'd like to be one) as I'm not a Twisted developer. In any case, I could bring my experience as an ex-student :-) Anyway, some areas a student could give a helping hand are: - porting some Twisted.web2 feature(s) to Twisted.web - improving support for Twisted in Jython or IronPython - porting Amp to Java, C++, etc. - adding or improving support for Twisted in an external library (e.g. Thrift, Amazon services, Storm, Protocol Buffers, etc.) What do you think? Cheers. 1 - http://code.google.com/opensource/gsoc/2009/faqs.html

2 1

[Twisted-Python] need help for twisted FTPClient
by 柳锴 12 Feb '09

12 Feb '09

hello, everyone: I'm new to twisted and ftp protocol. for some purpose, i need a python ftp client to do list, get, put, remove operations upon a FTP server which is implemented with twisted. here is my codes, testing for 'get' passed, but 'put' failed. i checked the api of storeFile, abd got these: ''' *def storeFile(self, path, offset=0): **(source) *<http://twistedmatrix.com/trac/browser/tags/releases/twisted-8.2.0/twisted/p…> ** *Store a file at the given path.* *This method issues the 'STOR' FTP command. * *Returns* *A tuple of two ** Deferred*<http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Defer…> *s: * - *Deferred*<http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Defer…> * **IFinishableConsumer*<http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.…> *. You must call the finish method on the IFinishableConsumer when the file is completely transferred. * - *Deferred*<http://twistedmatrix.com/documents/current/api/twisted.internet.defer.Defer…> * list of control-connection responses. * ''' but i don't know how to handle it, :) sorry for the rubbish codes, but i really need help from you guys, it's hard to find some useful info from twisted official doc, Any suggestion on how to write a nice ftp client with twisted is welcome. Thanks in advance. IOSCAS ______________________________________________________________________________ from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.python import usage from twisted.internet import reactor, defer class SaveFile(Protocol): ''' save the ftp file to local filesystem ''' def __init__(self, output): self.fout = open(output, 'w') def dataReceived(self, data): self.fout.write(data) self.fout.close() class FTP: ''' a simple ftp client ''' def __init__(self, host, port): ''' init ''' self.__host = host self.__port = port self.__username = 'aaa' self.__passwd = 'bbb' self.__passive = 1 def __get(self, ftp_client, src, des): ''' ''' save_file = SaveFile(des) d = ftp_client.retrieveFile(src, save_file) d = ftp_client.quit() d.addCallback(lambda result: reactor.stop()) return d def get(self, src, des): ''' get src file from ftp server, store it in des ''' creator = ClientCreator(reactor, FTPClient, self.__username, self.__passwd, self.__passive) defer = creator.connectTCP(self.__host, self.__port).addCallback(self.__get, src, des) reactor.run() return defer.result def __put(self, ftp_client, src, des): ''' ''' source_file = os.path.basename(src) target_dir = os.path.dirname(des) ftp_client.changeDirectory(target_dir) d = ftp_client.storeFile(src) d = ftp_client.quit() d.addCallback(lambda result: reactor.stop()) return d def put(self, src, des): ''' put src file to ftp server, store it in des ''' creator = ClientCreator(reactor, FTPClient, self.__username, self.__passwd, self.__passive) defer = creator.connectTCP(self.__host, self.__port).addCallback(self.__put, src, des) reactor.run() return defer.result

3 3

[Twisted-Python] Bad signature using conch ssh
by Jonathan Marshall 11 Feb '09

11 Feb '09

I'm receiving 'Bad signature' when trying to connect to an SSH server, however I'm able to connect OK using OpenSSH. I'm hoping that somebody here can help me get to the bottom of this! Using twisted: $ python -m twisted.conch.scripts.conch -v xxx [-] Log opened. [-] Starting factory <twisted.conch.client.unix.SSHUnixClientFactory instance at 0x707a30> [-] Stopping factory <twisted.conch.client.unix.SSHUnixClientFactory instance at 0x707a30> [-] Starting factory <twisted.conch.client.direct.SSHClientFactory instance at 0x707a58> [SSHClientTransport,client] kex alg, key alg: diffie-hellman-group- exchange-sha1 ssh-rsa [SSHClientTransport,client] outgoing: aes256-cbc hmac-sha1 none [SSHClientTransport,client] incoming: aes256-cbc hmac-sha1 none [SSHClientTransport,client] Disconnecting with error, code 3 reason: bad signature [SSHClientTransport,client] connection lost [SSHClientTransport,client] Stopping factory <twisted.conch.client.direct.SSHClientFactory instance at 0x707a58> [-] Main loop terminated. Connection to xxx closed. conch: exiting with error [Failure instance: Traceback (failure with no frames): <class 'twisted.conch.error.ConchError'>: ('bad signature', 3) Using ssh (some lines removed for brevity): $ ssh -vvv xxx OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006 debug1: Remote protocol version 2.0, remote software version 0.0 debug1: no match: 0.0 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.1 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie- hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman- group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128- cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc(a)lysator.liu.se ,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64(a)openssh.com,hmac- ripemd160,hmac-ripemd160(a)openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib(a)openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group1-sha1,diffie-hellman- group-exchange-sha1 debug2: kex_parse_kexinit: ssh-rsa debug2: kex_parse_kexinit: aes256-cbc,aes192-cbc,aes128-cbc,twofish- cbc,blowfish-cbc,3des-cbc,arcfour debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96 debug2: kex_parse_kexinit: zlib,none debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-md5 debug1: kex: server->client aes128-cbc hmac-md5 none debug2: mac_setup: found hmac-md5 debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug2: dh_gen_key: priv key bits set: 132/256 debug2: bits set: 515/1025 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: check_host_in_hostfile: filename /Users/jon/.ssh/known_hosts debug3: check_host_in_hostfile: match line 13 debug1: Host 'xxx' is known and matches the RSA host key. debug1: Found key in /Users/jon/.ssh/known_hosts:13 debug2: bits set: 521/1025 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /Users/jon/.ssh/id_rsa (0x107ec0) debug2: key: /Users/jon/.ssh/identity (0x0) debug2: key: /Users/jon/.ssh/id_dsa (0x0) debug3: input_userauth_banner

1 0

Twisted February 2009