Mailman 3 December 2005 - Twisted

[Twisted-Python] Re: [Twisted-commits] r15355 - Add twisted.python.pymodules.datetime and use it if "import datetime" fails.
by Itamar Shtull-Trauring 21 Dec '05

21 Dec '05

On Tue, 2005-12-20 at 18:26 -0700, Wilfredo Sanchez wrote: > +try: > + import datetime > +except ImportError: > + import twisted.python.pymodules.datetime as datetime twisted.python.compat is an even better place, so other modules can benefit (check out how other compat stuff is handled).

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight 20 Dec '05

20 Dec '05

Bug summary ______________________ Since 2005-12-14 07:00:01.820000: Bugs opened: 10 Bugs closed: 9 Total open bugs: 501 (+1) Urgent: 1 (+0) Bug: 240 (+2) Feature: 177 (-1) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [CORE, TRIAL] Raising SkipTest in setUp is treated as an error (12 hours ago) http://twistedmatrix.com/bugs/issue1372 opened by exarkun [CORE, TESTS] Possible race condition in test_tcp (yesterday) CLOSED http://twistedmatrix.com/bugs/issue1370 opened by jml [CORE, TESTS] Possible race condition in test_tcp (yesterday) http://twistedmatrix.com/bugs/issue1371 opened by jml iocpreactor (udp) socket.error: (10049, "Can't assign requested address") (yesterday) http://twistedmatrix.com/bugs/issue1369 opened by ghazel [CORE, FTP] FTP server tests are a mess (2 days ago) http://twistedmatrix.com/bugs/issue1368 opened by exarkun [CORE, TRIAL] If setUp returns an unfired Deferred, test methods cannot use wait() (2 days ago) http://twistedmatrix.com/bugs/issue1367 opened by exarkun [CORE, DOCUMENTATION] TwistedQuotes documentation references external source files (2 days ago) http://twistedmatrix.com/bugs/issue1366 opened by exarkun select.error: (10022, 'Windows Error 0x2726') (3 days ago) http://twistedmatrix.com/bugs/issue1365 opened by ghazel ==== Feature ==== [CORE] [PATCH] re-installation of the same reactor (6 days ago) http://twistedmatrix.com/bugs/issue1364 opened by antony [CORE] [PATCH] broadcasting over udp causes warning (6 days ago) http://twistedmatrix.com/bugs/issue1363 opened by antony Bugs Closed ______________________ ==== Bug ==== Separate reporter result handling from output (2 weeks ago) http://twistedmatrix.com/bugs/issue1349 opened by jml, closed by jml [CORE, TESTS] Possible race condition in test_tcp (yesterday) http://twistedmatrix.com/bugs/issue1370 opened by jml, closed by jml [CORE, TRIAL] Test case run() methods to use Deferreds internally (3 weeks ago) http://twistedmatrix.com/bugs/issue1341 opened by jml, closed by jml [TRIAL] Trial doesn't cut tracebacks properly (13 months ago) http://twistedmatrix.com/bugs/issue773 opened by jknight, closed by jml [TRIAL] [PATCH] Traceback running a test with broken setUp method (1 week ago) http://twistedmatrix.com/bugs/issue1359 opened by headly, closed by jml [BACKPORT, WEB2] twisted.web2.stream and wsgi (7 days ago) http://twistedmatrix.com/bugs/issue1362 opened by mkerrin, closed by jknight ==== Feature ==== Trial spews color codes by default on win32 (4 months ago) http://twistedmatrix.com/bugs/issue1141 opened by TimothyFitz, closed by jml [DOCUMENTATION, TRIAL] Draft release notes for the recent and upcoming Trial changes (3 months ago) http://twistedmatrix.com/bugs/issue1172 opened by jml, closed by jml [TRIAL] TASK: Deprecate setUpClass/tearDownClass. Provide replacement (2 months ago) http://twistedmatrix.com/bugs/issue1206 opened by jml, closed by jml

1 0

[Twisted-Python] Problem w/ TNPE Example on Windows
by Viktor Ransmayr 20 Dec '05

20 Dec '05

Hello, I'm new to Twisted and just started reading "Twisted Network Programming Essentials". The example "dataforward.py" in chapter 2 creates a traceback and an import error, if executed on my laptop running WinXP/Pro-SP2, using Python-2.4.2 and Twisted-2.1.0 The import error complains about a missing module "fcntl" required by ...\twisted\internet\fdesc.py Reading the python documentation states that the "fcntl" module is available on Unix only. - Does that mean, that this example is working on Unix only or what am I missing? Kind regards, Viktor Ransmayr

6 8

[Twisted-Python] I'm missing something on SSL-encrypted PB connections...
by Mike C. Fletcher 20 Dec '05

20 Dec '05

Hi all, I'm working on refining a load-sharing mechanism to run across a number of pb connections (sharing processor time and/or access to a network). One of the requirements I have is that I must be able to run the pb connections over encrypted channels. However, when I try to set up an SSL-encrypted server (using the same mechanisms I've used for setting up SSL-encrypted Nevow sites and the same ones in the echo examples) I always get a connection refused error on the client when I try to connect. I'm using self-generated keys from: openssl genrsa > privkey.pem openssl req -new -x509 -key privkey.pem -out cacert.pem -days 1000 I'm guessing this has to be some incredibly stupid error, but I'm not seeing it in a few hours of poking now, so this is basically a request for more eyes. I've reduced the failing code down to a modification of the pbecho* example code (attached). With keys generated as above in the current directory the server starts fine, but the client always receives: Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused. when it tries to connect. The only other code I've found trying to use SSL + PB is connecting it up to a multi-service of some sort, which I don't *think* should be necessary. Thoughts appreciated, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com

2 2

[Twisted-Python] Twisted SCP
by Phil Mayers 20 Dec '05

20 Dec '05

All, This is a very very horrible bit of code (stupid undocumented RCP junk), but a basic single-file remote -> local SCP can be done using the attached code. It's clearly as dumb as a box of rocks, but seems to work against OpenSSH and Cisco IOS, so is fine for what we need. If anyone knows why the sftp stuff in it doesn't work that would be useful. Anyway, hope it's helpful to someone. If I get time over or after xmas I'll clean it up - it seems pretty clear to me how to make the code pretty modular without it descending into the mess that OpenSSH and Putty's SCP/SFTP code has become (with the greatest of respect to the developers involved of course) #!/usr/bin/env python from twisted.conch.ssh import transport, userauth, connection, channel from twisted.conch.ssh.common import NS from twisted.internet import defer, protocol, reactor from twisted.python import log import sys, os, getpass USER, PASS, HOST, CMD, SRC, DST = None, None, None, None, None, None # The "Transport" is the crypto layer on top of port 22 class Transport(transport.SSHClientTransport): def verifyHostKey(self, hostKey, fingerprint): # For info only log.msg('host key fingerprint: %s' % fingerprint) # FIXME: this is insecure return defer.succeed(1) def connectionSecure(self): # Once we're secure (server key valid) we ask for userauth service # on a new "connection" object self.requestService(UserAuth(USER, Connection())) class UserAuth(userauth.SSHUserAuthClient): def getPassword(self): # Ack! Globals! return defer.succeed(PASS) def getPublicKey(self): # returning None means always use username/password auth return class Connection(connection.SSHConnection): def serviceStarted(self): # Once userauth has succeeded we ask for a channel on this # connection self.openChannel(ScpChannel(2**16, 2**15, self)) class XferChannelBase(channel.SSHChannel): name = 'session' state = None todo = 0 buf = '' def openFailed(self, reason): log.err(reason) def channelOpen(self, data): # Might display/process welcome screen self.welcome = data # We might be an SCP or SFTP requests if 'scp' in CMD or CMD.startswith('/'): kind = 'exec' else: kind = 'subsystem' # Call our handler d = self.conn.sendRequest(self, kind, NS(CMD), wantReply=1) d.addCallbacks(self.channelOpened, log.err) def closed(self): self.loseConnection() reactor.stop() class SftpChannel(XferChannelBase): def channelOpened(self, data): log.msg("channelOpened: %r" % (data,)) self.client = filetransfer.FileTransferClient() self.client.makeConnection(self) self.dataReceived = self.client.dataReceived d = self.client.openFile(SRC, filetransfer.FXF_READ, {}) d.addCallbacks(self.fileOpened, log.err) def fileOpened(self, rfile): rfile.getAttrs().addCallbacks(self.fileStatted, log.err, (rfile,)) def fileStatted(self, attrs, rfile): rfile.readChunk(0, 4096).addCallbacks(self.did_read, log.err, (rfile, 0, attrs['size'])).addCallback(self.done) def did_read(self, data, f, pos, todo): if len(data)>todo: log.msg("got %i bytes more than expected, trimming" % (len(data)-todo,)) data = data[:todo] DST.write(data) todo -= len(data) pos += len(data) if todo<=0: return pos return f.readChunk(pos, 4096).addCallbacks(self.did_read, log.err, (f, pos, todo)) def done(self, l): log.msg("done %i bytes" % (l,)) self.loseConnection() class ScpChannel(XferChannelBase): def channelOpened(self, data): # once the scp is exec'ed, start the SCP transfer self.write('\0') # we're a state machine self.state = 'waiting' def dataReceived(self, data): #log.msg('dataReceived: %s %r' % (self.state, data)) # What we do with the data depends on where we are if self.state=='waiting': # we've started the transfer, and are expecting a C # Coctalperms size filename\n # might not get it all at once, buffer self.buf += data if not self.buf.endswith('\n'): return b = self.buf self.buf = '' # Must be a C if not b.startswith('C'): log.msg("expecting C command: %r" % (self.buf,)) self.loseConnection() return # Get the file info p, l, n = b[1:-1].split(' ') perms = int(p, 8) self.todo = int(l) log.msg("getting file %s mode %s len %i" % (n, oct(perms), self.todo)) # Tell the far end to start sending the content self.state = 'receiving' self.write('\0') elif self.state=='receiving': # we've started the file body #log.msg('got %i bytes' % (len(data),)) if len(data)>self.todo: extra = data[self.todo:] data = data[:self.todo] if extra!='\0': log.msg("got %i more bytes than we expected, ignoring: %r" % (len(extra), extra)) DST.write(data) self.todo -= len(data) if self.todo<=0: log.msg('done') self.loseConnection() else: log.err("dataReceived in unknown state: %r" % (self.state,)) def usage(ex): print >>sys.stderr, """%s: [user[:pass]@]hostname:sourcefile destfile""" % (sys.argv[0],) if ex: sys.exit(ex) if __name__=='__main__': args = sys.argv[1:] if len(args)<2: usage(1) SRC = args[0] DST = args[1] if '@' in SRC: USER, SRC = SRC.split('@', 1) else: USER = os.environ['USERNAME'] if not ':' in SRC: usage(1) HOST, SRC = SRC.split(':', 1) if ':' in USER: USER, PASS = USER.split(':', 1) else: PASS = getpass.getpass('password for %s@%s: ' % (USER, HOST)) DST = open(DST, 'wb') if 'sftp' in sys.argv[0]: CMD = 'sftp' else: CMD = 'scp -f %s' % (SRC,) protocol.ClientCreator(reactor, Transport).connectTCP(HOST, 22) log.startLogging(sys.stderr) reactor.run()

1 0

[Twisted-Python] xpost: multiple job openings
by Paul G 19 Dec '05

19 Dec '05

folks, i'm crossposting this to twisted-python-l and twisted-web-l for the benefit of those who don't read both. apologies to those who will consequently have to read this twice. --- We've got multiple job openings for great engineers familiar with python and twisted, senior and junior positions available. We're building a small team of folks who like working hard, solving interesting problems, writing clean code and having fun doing it. If you got into coding because that's where the money was in the '90s, please skip this one and see my next job posting where I'm recruiting bodyguards for work in the Persian Gulf. The dev team is run by an engineer ('run' in the sense that I shield you from the crazy business folk), so it is a strict meritocracy. You will be working on rebuilding/refactoring an existing PHP (ugh) web application into python services built on top of twisted as well as implementing new features. This is a high volume system, with millions of transactions a day - if you dig concurrency, you won't be bored. Must be able to handle the excitement ;) Requirements: * We're looking for great engineers - while your ability to discuss Wittgenstein over our 5th beer is welcome, a piece of paper^W^W^Wdiploma is not required. * You should be old enough not to get us in trouble with Amnesty International for child exploitation. * Experience working on hobby, open source and commercial projects counts with me. * Python/Twisted and SQL a must. * Reading/grokking bad PHP code a big plus. Writing PHP code useful, since we'll be hacking in the twisted/python bits over time. * Javascript/DHTML/AJAX(ugh, buzzowrd) familiarity needed in at least one hire, so a plus. * Experience with C and ASM good in that it is a good predictor of your understanding of memory allocation and performance issues. * Don't mention Java, unless it has caused a medical (eg mental health) problem we should know about. <grin> * Understanding of performance and scalability principles and issues. Work environment: * Onsite in NYC/NY Metro area strongly preferred. * Don't let the above deter you - we can make telecommuting work under the right conditions. * Competitive compensation - I know what good people are worth and have managed to beat that into the management with a clue-by-four. * Flex time - relaxed cluture - getting your job done is all that matters. --- Shoot me a resume, CV, writeup of what you've done, absolutely anything. Put [twisted-resume] in the subject. Include your phone number, time zone and a good time to call. I'll respond to everyone. cheers, -p

1 0

[Twisted-Python] Is it necessary to create a new reactor to daemonize a File Alteration Monitoring system
by Kaladan 19 Dec '05

19 Dec '05

Hello, I manage to write a file system monitoring daemon based on gamin ( http://gnome.org/~veillard/gamin/) thanks to twisted API. Therefore I have created a new reactor, which inherit from the ReactorBase. In order to use the twistd application runner I have a GaminService which define a simple StartService. Am I in the good way ? The code is in attachment -- Olivier Guilloux

2 1

[Twisted-Python] Some beginner questions about "twisted.names.client" and ".tac" environment
by Jesus Cea 18 Dec '05

18 Dec '05

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Twisted 2.1, twisted.named 0.2, here. I'm taking my first steps with Twisted (documentation -inexistence- nightmare :-), and my first project will be a bulk mailer as the backend of my mailing list system. The application would take the message and the subscriber list and a) resolve the MX for the domains and b) connect to the MX's and send the message, trying to minimice traffic sending a single envelope for several recipients sharing the domain or the MX's. I'm doing currently the DNS stuff. The result are promising, resolving about 200 domains per second in a 1.4GHz P4, so my biggest mailing list (about 31500 unique domains, múltiple subscribers per domain) is "resolved" in less than three minutes. Nice so far. The demo code (2Kbytes) is the following (if I'm violating the rules posting this code, please tell me): ===== # File "dns.tac" from twisted.application import service application = service.Application("DNS test") import time t=time.time() class resolucion(object) : def __init__(self,dominio) : from twisted.names import client d = client.lookupMailExchange(dominio,timeout=(60,)) d.addCallbacks(self._cbMailExchange, self._ebMailExchange) self.dominio=dominio def _cbMailExchange(self,results): # Callback for MX query global aun_pendientes aun_pendientes-=1 if not aun_pendientes : print "OK",time.time()-t return from twisted.internet import reactor reactor.stop() return if not len(pendientes) : return resolucion(pendientes.pop()) from twisted.names.dns import QUERY_TYPES for i in results[0] : n=i.payload.name tipo=QUERY_TYPES[i.payload.TYPE] if tipo=="MX" : return p=i.payload.preference print n,p, for j in results[2] : if n==j.name : print j.payload.dottedQuad(),"(%d)" %j.ttl break else : print "???" elif tipo=="CNAME" : redirigidos.append((self.dominio,i.payload.name)) def _ebMailExchange(self,failure): # Error callback for MX query global aun_pendientes aun_pendientes-=1 if not aun_pendientes : print "ERROR",time.time()-t return from twisted.internet import reactor reactor.stop() return if not len(pendientes) : return resolucion(pendientes.pop()) print "XXX",self.dominio print 'Lookup failed:' failure.printTraceback() pendientes=[] redirigidos=[] f=open("domain_list") for i in f : pendientes.append(i) aun_pendientes=len(pendientes) concurrencia=1000 for i in pendientes[:concurrencia] : resolucion(i) from twisted.names import client client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)] # client.theResolver.resolvers=[client.theResolver.resolvers[-1]] pendientes=pendientes[concurrencia:] ===== I launch the code as "twistd -ny dns.tac". The demo does 1000 resolutions in parallel. If you experiment with the code, reduce the value. Questions: 1. I get a warning: "[Uninitialized] /usr/local/lib/python2.4/site-packages/twisted/names/dns.py:1227: exceptions.DeprecationWarning: Deferred.setTimeout is deprecated. Look for timeout support specific to the API you are using instead." I'm using, the native "twisted.names" timeout API, as far as I know... 2. By default "twisted.names.client" uses the "/etc/resolv.conf" file to know which nameservers to use. I, nevertheless, want to use a particular nameserver, so: 2.1. I couldn't to find an appropiate API. I had to do a "hack", reading the "twisted.names" core to know implementation details: "client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)]" 2.2. The previous "hack" is only effective for future "twisted.names.client" instances. The previous ones use the "/etc/resolv.conf" entries. Putting the "hack" code before any instance creation doesn't work. 2.3. While reading the framework code, I saw that "client" uses a resolver chain: host, cache, network. But the cache is initially clear (of course) and NEVER ever gets populated, so we are not using it but checking missing entries eats CPU: 155 seconds for the unchanged code, 125 seconds if I drop the host and cache resolvers. A caching client would be very nice, if the client is long running (my original idea). 2.4. The resolution failure code is only called if the resolution timeouts. But if the domain doesn't exists, the code called is the "success" one, with a "nil" answer. So we can't diferenciate between inexistant domains and inexistant RRs. 3. How can I stop this ".tac"?. If I do "reactor.stop()", I get an infinite error, repeated forever: ===== [twisted.names.dns.DNSDatagramProtocol (UDP)] Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/udp.py", line 113, in doRead data, addr = self.socket.recvfrom(self.maxPacketSize) exceptions.AttributeError: 'Port' object has no attribute 'socket' ===== I must kill -9 the "twistd" process. Thank you for your time and attention. Help greatly appreciated :-) - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBQ6SNsplgi5GaxT1NAQLMQwP/czYFLQ6+olTCvM0jdmMlaBgwxHsHdvxT /2mhWqtyhIf1Kdh6FioFQq13xqCfZxFIkwuUwTlG+ZmkSYK1iWZEmaS0CGa5YmuA d7miIFfL9Tfa3OLyV1nvqdCR3YtzH/ws9UuJ2DGnACRI++Of6gBVwGlhFDa7S57o wZcsYWAS6Sk= =/Pks -----END PGP SIGNATURE-----

2 5

[Twisted-Python] WSGI Thread-management strategy
by Jim Fulton 16 Dec '05

16 Dec '05

The strategy used by twisted WSGI, as I understand it, doesn't meet our needs. Currently, a thread is created for each request. The total number of threads is throttled, I gather using a general Twisted thread limit. WSGI applications are called as soon as input headers have been received completely. An application may be called before all body input is received. We need application calls to be delayed until all request input has been received, In an application like Zope, application threads are expensive. There are application resources, such as database connections and caches needed by the application threads. To limit resource consumption, we have to limit the number of application threads. Because the number of threads is limited, application threads are scarce and thus valuable resources that need to be used very efficiently. In particular, we don't want to tie up an application thread waiting for request body input. I propose that the default thread-management strategy should be to delay calling an application until all request input has been received. If this isn't the default, then there should at least be an option to get this behavior. (Of course, the buffering strategy needs to be clever enough to switch to a file when the input gets over some size.) Thoughts? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 3

[Twisted-Python] how to run a twisted daemon without tcpserver
by Kaladan 15 Dec '05

15 Dec '05

Can I run a twisted daemon (twistd) without TCPServer, basically I want to run a reactor within an application, something like : application = service.Application('myApp', uid=505, gid=506) reactor.run(application) Thank's -- Olivier Guilloux

1 0