Mailman 3 March 2002 - Twisted

[Twisted-Python] Re: Twisted protocol
by Andrew Bennetts March 12, 2002

March 12, 2002

On Mon, Mar 11, 2002 at 06:51:57PM +0100, David wrote: > Hello spiv, this is supposed to be reaching you =) > I was told by radix on #python@opn that you had written "a little proxy > protocol which dispatches to multiple other protocol objects" in > twisted. This is exactly what i need for a project of mine. > I'd very much appriciate if you would share this little piece of code > with me or alternativly explain how it works. Heh. I'm CC'ing the list with this, just for … [View More]the sake of it :) Here is the class. I created it for a case where I had a protocol that was sometimes inheriting from one Protocol, and sometimes another, and I didn't want to diddle with __bases__ or munge the two bases into one class (although perhaps that is the "better" solution?). If you like, imagine PacketBasedProtocolA/B to be basic.LineReceiver and Int32StringReceiver, where lineReceived and stringReceived are both aliased to packetReceived. class ProxiedPacketProtocol(Protocol): """General packet-based protocol. This class provides a simple way to use either PacketBasedProtocolA or PacketBasedProtocolB (or potentially others) as protocol for a data feed, without having to change which one you inherit from at runtime. It basically acts as a proxy protocol between the low-level protocol and your subclass. Simply override parentProtocol with the "parent" protocol (the default is PacketBasedProtocolA) and then when packetReceived is triggered in the parent, packetReceived is called in me. Inherit from this class, and override packetReceived.""" parentProtocol = PacketBasedProtocolA packets = {} # This is copied to the parent def __init__(self): # Create a private subclass of the parentProtocol so that we can # override its class-level data attribute(s). class ParentProtocol(self.parentProtocol): packets = self.packets # Create the parent protocol self.parent = ParentProtocol() # Raw data should get delivered to the parent... self.dataReceived = self.parent.dataReceived # ...and processed packets should come back to me self.parent.packetReceived = self.packetReceived self.parent.unknownPacket = self.unknownPacket # Data from the pinger needs to come via me self.parent.transport = self.transport def packetReceived(self, packet): """Override this. Called whenever the parentProtocol receives a packet.""" pass def unknownPacket(self, messageType, data): print "Received unknown packet of type '%s': %s" \ % (messageType, repr(data)) David, I hope this is what you were after :) It should be fairly easy to use and adapt -- but my question for the list is, is it worth the extra effort it would take to make this more general? i.e. take arbitrary lists of attributes to copy from the parent to the proxy (and vice versa?), instead of hard-coded names in __init__? My other question is, is this a useful enough concept to include it in Twisted? :) -Andrew. [View Less]

2 1

[Twisted-Python] Win32 updates
by Itamar Shtull-Trauring March 11, 2002

March 11, 2002

1) win32 process support is checked in and (lightly) tested. Just twisted.internet.win32.install() and it's there. For CGI support we're gonna need to write a small parser thing that can grok unix style "#!c:\python2.1\python.exe" interpreter lines. 2) Because the process tests install the win32 event loop, I discovered the win32 event loop breaks the the DNS client/server testing code, so currently these tests do *not* pass. Anyone know about issues with WSAEventSelect and UDP?

2 1

[Twisted-Python] docs
by Christopher Armstrong March 11, 2002

March 11, 2002

I just shwacked all the HTML files in the 'doc/' directory and put them in the TwistedDocs repository. This repository will be populated with HTML documentation which have '.htc' suffixes -- these files *are* just plain old HTML. I copied/hacked 'generate.py' from the WebSite repository to be able to convert regular HTML to HTML-without-headers-and-footers, and also be able to extract the title from the <title> tags of the .htc document. the generate.py script generates "prettier" html … [View More]

1 1

[Twisted-Python] Yet another win32 process
by Itamar Shtull-Trauring March 10, 2002

March 10, 2002

This one uses the win32 event loop for reading from stdout and stderr, and starts one thread per Process for writing to stdin. So we're down to one thread per Process instead of 3, thanks to Andrew's suggestion. """ Win32 process support. """ # Twisted imports from twisted.internet import abstract, main, win32 # System imports import win32api import win32pipe import win32file import win32process import win32security import win32con import win32event import pywintypes import msvcrt import … [View More]os import sys import threading import Queue import string class Process(abstract.FileDescriptor): """A process that integrates with the Twisted event loop.""" buffer = '' def __init__(self, command, args, environment, path): # security attributes for pipes sAttrs = win32security.SECURITY_ATTRIBUTES() sAttrs.bInheritHandle = 1 # create pipes hStdin_r, self.hStdin_w = win32pipe.CreatePipe(sAttrs, 0) self.hStdout_r, hStdout_w = win32pipe.CreatePipe(sAttrs, 0) self.hStderr_r, hStderr_w = win32pipe.CreatePipe(sAttrs, 0) # set the info structure for the new process. StartupInfo = win32process.STARTUPINFO() StartupInfo.hStdInput = hStdin_r StartupInfo.hStdOutput = hStdout_w StartupInfo.hStdError = hStderr_w StartupInfo.dwFlags = win32process.STARTF_USESTDHANDLES # Mark doesn't support wShowWindow yet. # StartupInfo.dwFlags = StartupInfo.dwFlags | win32process.STARTF_USESHOWWINDOW # StartupInfo.wShowWindow = win32con.SW_HIDE # Create new output read handles and the input write handle. Set # the inheritance properties to FALSE. Otherwise, the child inherits # the these handles; resulting in non-closeable handles to the pipes # being created. pid = win32api.GetCurrentProcess() tmp = win32api.DuplicateHandle( pid, self.hStdin_w, pid, 0, 0, # non-inheritable!! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStdin_w) self.hStdin_w = tmp tmp = win32api.DuplicateHandle( pid, self.hStdout_r, pid, 0, 0, # non-inheritable! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStdout_r) self.hStdout_r = tmp tmp = win32api.DuplicateHandle( pid, self.hStderr_r, pid, 0, 0, # non-inheritable! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStderr_r) self.hStderr_r = tmp # start the process. print "creating process" cmdline = "%s %s" % (command, string.join(args, ' ')) hProcess, hThread, dwPid, dwTid = win32process.CreateProcess( None, # program cmdline,# command line None, # process security attributes None, # thread attributes 1, # inherit handles, or USESTDHANDLES won't work. # creation flags. Don't access the console. 0, # Don't need anything here. # If you're in a GUI app, you should use # CREATE_NEW_CONSOLE here, or any subprocesses # might fall victim to the problem described in: # KB article: Q156755, cmd.exe requires # an NT console in order to perform redirection.. environment, # new environment path, # new directory StartupInfo) # normally, we would save the pid etc. here... print "process created" # Child is launched. Close the parents copy of those pipe handles # that only the child should have open. # You need to make sure that no handles to the write end of the # output pipe are maintained in this process or else the pipe will # not close when the child process exits and the ReadFile will hang. win32file.CloseHandle(hStderr_w) win32file.CloseHandle(hStdout_w) win32file.CloseHandle(hStdin_r) self.outQueue = Queue.Queue() self.closed = 0 self.stdoutClosed = 0 self.stderrClosed = 0 threading.Thread(target=self.doWrite).start() win32.addEvent(self.hStdout_r, self, self.doReadOut) win32.addEvent(self.hStderr_r, self, self.doReadErr) def write(self, data): """Write data to the process' stdin.""" self.outQueue.put(data) def closeStdin(self): """Close the process' stdin.""" self.outQueue.put(None) def connectionLost(self): """Will be called twice, by the stdout and stderr threads.""" if not self.closed: win32.removeEvent(self.hStdout_r) win32.removeEvent(self.hStderr_r) abstract.FileDescriptor.connectionLost(self) print "connection lost" self.closed = 1 self.closeStdin() win32file.CloseHandle(self.hStdout_r) win32file.CloseHandle(self.hStderr_r) def doWrite(self): """Runs in thread.""" while 1: data = self.outQueue.get() if data == None: break try: win32file.WriteFile(self.hStdin_w, data, None) except win32api.error: break win32file.CloseHandle(self.hStdin_w) def doReadOut(self): """Runs in thread.""" try: hr, data = win32file.ReadFile(self.hStdout_r, 8192, None) except win32api.error: self.stdoutClosed = 1 if self.stderrClosed: return main.CONNECTION_LOST else: return self.handleChunk(data) def doReadErr(self): """Runs in thread.""" try: hr, data = win32file.ReadFile(self.hStderr_r, 8192, None) except win32api.error: self.stderrClosed = 1 if self.stdoutClosed: return main.CONNECTION_LOST else: return self.handleError(data) if __name__ == '__main__': win32.install() def printer(x): print "Got", repr(x) exe = win32api.GetModuleFileName(0) print exe p = Process(exe, ['-u', 'processtest.py'], None, None) print "ok, made process object" p.handleChunk = printer p.handleError = printer p.write("hello, world") p.closeStdin() main.run() [View Less]

1 0

[Twisted-Python] twisted.internet.process for Win32
by Itamar Shtull-Trauring March 9, 2002

March 9, 2002

I've attached my current code, which seems to be working. This will hopefully be checked in today, but some testing by others would be useful. """ Win32 process support. """ # Twisted imports from twisted.internet import task, abstract # System imports import win32api import win32pipe import win32file import win32process import win32security import win32con import win32event import pywintypes import msvcrt import os import sys import threading import Queue import string class Process(… [View More]abstract.FileDescriptor): """A process that integrates with the Twisted event loop.""" buffer = '' def __init__(self, command, args, environment, path): # security attributes for pipes sAttrs = win32security.SECURITY_ATTRIBUTES() sAttrs.bInheritHandle = 1 # create pipes hStdin_r, self.hStdin_w = win32pipe.CreatePipe(sAttrs, 0) self.hStdout_r, hStdout_w = win32pipe.CreatePipe(sAttrs, 0) self.hStderr_r, hStderr_w = win32pipe.CreatePipe(sAttrs, 0) # set the info structure for the new process. StartupInfo = win32process.STARTUPINFO() StartupInfo.hStdInput = hStdin_r StartupInfo.hStdOutput = hStdout_w StartupInfo.hStdError = hStderr_w StartupInfo.dwFlags = win32process.STARTF_USESTDHANDLES # Mark doesn't support wShowWindow yet. # StartupInfo.dwFlags = StartupInfo.dwFlags | win32process.STARTF_USESHOWWINDOW # StartupInfo.wShowWindow = win32con.SW_HIDE # Create new output read handles and the input write handle. Set # the inheritance properties to FALSE. Otherwise, the child inherits # the these handles; resulting in non-closeable handles to the pipes # being created. pid = win32api.GetCurrentProcess() tmp = win32api.DuplicateHandle( pid, self.hStdin_w, pid, 0, 0, # non-inheritable!! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStdin_w) self.hStdin_w = tmp tmp = win32api.DuplicateHandle( pid, self.hStdout_r, pid, 0, 0, # non-inheritable! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStdout_r) self.hStdout_r = tmp tmp = win32api.DuplicateHandle( pid, self.hStderr_r, pid, 0, 0, # non-inheritable! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStderr_r) self.hStderr_r = tmp # start the process. print "creating process" cmdline = "%s %s" % (command, string.join(args, ' ')) hProcess, hThread, dwPid, dwTid = win32process.CreateProcess( None, # program cmdline,# command line None, # process security attributes None, # thread attributes 1, # inherit handles, or USESTDHANDLES won't work. # creation flags. Don't access the console. 0, # Don't need anything here. # If you're in a GUI app, you should use # CREATE_NEW_CONSOLE here, or any subprocesses # might fall victim to the problem described in: # KB article: Q156755, cmd.exe requires # an NT console in order to perform redirection.. environment, # new environment path, # new directory StartupInfo) # normally, we would save the pid etc. here... print "process created" # Child is launched. Close the parents copy of those pipe handles # that only the child should have open. # You need to make sure that no handles to the write end of the # output pipe are maintained in this process or else the pipe will # not close when the child process exits and the ReadFile will hang. win32file.CloseHandle(hStderr_w) win32file.CloseHandle(hStdout_w) win32file.CloseHandle(hStdin_r) self.outQueue = Queue.Queue() self.closed = 0 threading.Thread(target=self.doWrite).start() threading.Thread(target=self.doReadOut).start() threading.Thread(target=self.doReadErr).start() def write(self, data): """Write data to the process' stdin.""" self.outQueue.put(data) def closeStdin(self): """Close the process' stdin.""" self.outQueue.put(None) def connectionLost(self): """Will be called twice, by the stdout and stderr threads.""" if not self.closed: abstract.FileDescriptor.connectionLost(self) print "connection lost" self.closed = 1 self.closeStdin() win32file.CloseHandle(self.hStdout_r) win32file.CloseHandle(self.hStderr_r) def doWrite(self): """Runs in thread.""" while 1: data = self.outQueue.get() if data == None: break try: win32file.WriteFile(self.hStdin_w, data, None) except win32api.error: break win32file.CloseHandle(self.hStdin_w) def doReadOut(self): """Runs in thread.""" while 1: try: hr, data = win32file.ReadFile(self.hStdout_r, 8192, None) except win32api.error: task.schedule(self.connectionLost) return task.schedule(self.handleChunk, data) def doReadErr(self): """Runs in thread.""" while 1: try: hr, data = win32file.ReadFile(self.hStderr_r, 8192, None) except win32api.error: task.schedule(self.connectionLost) return task.schedule(self.handleError, data) if __name__ == '__main__': from twisted.internet import main def printer(x): print "Got", repr(x) exe = win32api.GetModuleFileName(0) print exe p = Process(exe, ['-u', 'processtest.py'], None, None) print "ok, made process object" p.handleChunk = printer p.handleError = printer p.write("hello, world") p.closeStdin() main.run() [View Less]

2 5

[Twisted-Python] Commits list problems
by Itamar Shtull-Trauring March 6, 2002

March 6, 2002

Commits that change stuff in the root aren't sent to the commits list. So if I made a change to the ChangeLog along with my change, like the good programmer I am, nothing goes out to the commit list. So, things that were lost - refactor of tap making APIs, and the fact that I checked in the win32 event loop. This is BAD. This is ANNOYING. Someone fix it.

1 0

[Twisted-Python] Web Widgets Examples
by cpaul＠telemetrybox.org March 6, 2002

March 6, 2002

I was wondering if anybody on this list has an example of using Web Widgets to render an HTML page? Aloha, Charles

2 2

[Twisted-Python] Mime Types ins static.File
by cpaul＠telemetrybox.org March 5, 2002

March 5, 2002

Aloha, Today I shut down Apache on my workstation and started using twisted.web on port 80 - everything is sailing smooth, I even have a startup script in rc.d. The only issue I am running into so far has to do with mime types. In static, mime types are defined by the dictionary File.contentTypes - Is there an elegant way to dynamically set the mime types at mktap time? Or would it be better to hack static.py to use Python's mimetypes module along with a common mime.types file? I am … [View More]

1 0

[Twisted-Python] Win32 events main loop
by Andrew Bennetts March 5, 2002

March 5, 2002

Hi all, Here is the first *rough* cut of the Twisted main loop using Win32 events, based upon internet.poll. It seems to be okay for simpleserv.py, but I haven't tested it for anything else. I'll leave that to a day when I'm not feeling so ill... I pretty much guarantee it is mostly broken -- think of it as a proof of concept. I think if I experiment with MsgWaitForMultipleObjects instead of WaitForMultipleObjects, I can make it respond to Win32 window messages properly; e.g. Ctrl-C and … [View More]stopping when running as a service might work nicely -- COM stuff would probably benefit too. Enjoy! -Andrew. --- win32.py --- # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """A win32event based implementation of the twisted main loop. To install the event loop (and you should do this before any connections, listeners or connectors are added): from twisted.internet import win32 win32.install() """ # Win32 imports from win32file import WSAEventSelect, FD_READ, FD_WRITE, FD_CLOSE, \ FD_ACCEPT, FD_CONNECT from win32event import CreateEvent, WaitForMultipleObjects, \ WAIT_OBJECT_0, WAIT_TIMEOUT, INFINITE # Twisted imports from twisted.python import log, threadable import main # globals reads = {} writes = {} events = {} def _addEvent(fd, why, events=events): print '_addEvent:', fd, why event = CreateEvent(None, 0, 0, None) WSAEventSelect(fd, event, why) events[event] = (fd, why) return event def addReader(reader, reads=reads): """Add a FileDescriptor for notification of data available to read. """ print 'addReader:', reader if not reads.has_key(reader): reads[reader] = _addEvent(reader, FD_READ|FD_ACCEPT|FD_CONNECT) def addWriter(writer, writes=writes): """Add a FileDescriptor for notification of data available to write. """ print 'addWriter:', writer if not writes.has_key(writer): writes[writer] =_addEvent(writer, FD_WRITE) def removeReader(reader): """Remove a Selectable for notification of data available to read. """ if reads.has_key(reader): del events[reads[reader]] del reads[reader] def removeWriter(writer, writes=writes): """Remove a Selectable for notification of data available to write. """ if writes.has_key(writer): del events[writes[writer]] del writes[writer] def removeAll(): """Remove all selectables, and return a list of them.""" result = reads.keys() + writes.keys() reads.clear() writes.clear() events.clear() return result def doWaitForMultipleEvents(timeout, reads=reads, writes=writes): if timeout is None: #timeout = INFINITE timeout = 5000 else: timeout = timeout * 1000 print 'timeout is:', timeout handles = events.keys() val = WaitForMultipleObjects(handles, 0, timeout) print 'WaitForMultipleObjects returned:', val if val == WAIT_TIMEOUT: return elif val >= WAIT_OBJECT_0 and val < WAIT_OBJECT_0 + len(handles): print 'Here' fd, why = events[handles[val - WAIT_OBJECT_0]] print fd, why log.logOwner.own(fd) try: if why & FD_READ|FD_ACCEPT|FD_CONNECT: fd.doRead() elif why == FD_WRITE: fd.doWrite() except: log.deferr() why = FD_CLOSE if why == FD_CLOSE: removeReader(fd) removeWriter(fd) try: fd.connectionLost() except: log.deferr() log.logOwner.disown(fd) def install(): """Install the poll()-based event loop.""" main.addReader = addReader main.addWriter = addWriter main.removeReader = removeReader main.removeWriter = removeWriter #main.doSelect = doPoll main.doSelect = doWaitForMultipleEvents main.removeAll = removeAll def initThreads(): """Do initialization for threads.""" if main.wakerInstalled: # make sure waker is registered with us removeReader(main.waker) addReader(main.waker) threadable.whenThreaded(initThreads) __all__ = ["install"] [View Less]

2 2

[Twisted-Python] twistedmatrix client
by Kelly Roestel March 3, 2002

March 3, 2002

I am new to python and twistedmatrix, so any help would be great. I have been able to write a server and connect to it via python sockets. But I would like to figure out how to create a twisted client. But, I have found no docs on creating clients. Is there some place that I can look for docs, or can some one help me with client setup. What I am passing to the client from the server is a simple list with four items. Thanks for the help -- Kelly Roestel Network Admin DolphinSearch, Inc

2 1

[Twisted-Python] Aloha, New Member
by cpaul＠telemetrybox.org March 3, 2002

March 3, 2002

Aloha, My name is Charles Paul, and I am interested in learning more about how to use Twisted. Aloha, Charles

3 3

[Twisted-Python] A new testing strategy for protocols
by Itamar Shtull-Trauring March 3, 2002

March 3, 2002

A long time issue we've had is how to test protocols under "real" circumstances. For DNS testing, I decided to do just that - see the code in twisted/test/test_names.py. Basically, I start a server listening using UDP, and have a client connect to it, all within the testing code and automatically. It works pretty well, after a bit of debugging. Data is pushed by calling main.iterate(). Now, in a lot of cases twisted.protocols.loopback may be good enough, but this approach is really … [View More]

1 0

[Twisted-Python] Psyco is our friend
by Itamar Shtull-Trauring March 2, 2002

March 2, 2002

Psyco == http://sf.net/projects/psyco/ So, I ran a Banana benchmark, attached. The results: Psyco on 1.8585100174 seconds (5380.65434483 messages a second) Now I edit the file and turn off psyco: Psyco off 4.81016397476 seconds (2078.93120743 messages a second) Nice, eh? import time, psyco from twisted.spread import banana PSYCO = 1 if PSYCO: psyco.bind(banana.encode) psyco.bind(banana.int2b128) psyco.bind(banana.b1282int) psyco.bind(banana.decode) … [View More]

1 0

[Twisted-Python] Unused variables in doSelect
by Andrew Bennetts March 1, 2002

March 1, 2002

This is a minor thing, but... doSelect has the following comment (in 0.15.2): # Since this loop should really be as fast as possible, # I'm caching these global attributes so the interpreter # will hit them in the local namespace. reads=reads, writes=writes, rhk=reads.has_key, whk=writes.has_key): But rhk and whk are unused. Further on in the function, it does: for selectables, method, dict in ((r, … [View More]

3 2

[Twisted-Python] Twisted DNS issues
by Itamar Shtull-Trauring March 1, 2002

March 1, 2002

<itamar> next, if you fail to connect to a DNS server, you get a traceback <itamar> something about passing extra args to connectionLost <moshez> itamar: yeah, I know. <moshez> itamar: ugh. <itamar> now, moving onwards <itamar> consider this code: <moshez> itamar: I didn't know about that <itamar> r = dns.ResolveConfResolver() <itamar> d = defer.Deferred().addCallback(printer).addErrback(printer) <itamar> d.arm() <itamar> r.… [View More]

1 0