Mailman 3 September 2004 - Twisted

[Twisted-Python] Implementing a custom FTP server
by Jason Stitt 08 Sep '04

08 Sep '04

Hi, all, I'm trying to implement a custom FTP server (eventually SFTP, but I want to get the basics down.) It won't operate on the file system; the back end will be a database, and the server will generate virtual directories and folders on the fly. I have a basic Twisted-based FTP server running. I can log in, and I have successfully overridden ftp_LIST to display a custom "file" list. I have gotten stuck on ftp_STOR and ftp_RETR. There is no implementation of ftp_STOR in the base FTP class, so I have no reference to work against, and I am also having a hard time figuring out how ftp_RETR works. How or where is the actual file data passed back and forth? Any pointers would be greatly appreciated. Thanks, Jason P.S. A lack of documentation aside, though, Twisted has been quite easy to work with so far. Even fun :) P.P.S. is SFTP different enough from FTP that I should be thinking about implementing it from the start, rather than later, or can I just graft an SSL layer onto my FTP server?

8 23

[Twisted-Python] FWD: PEP 334 - Simple Coroutines via SuspendIteration
by Clark C. Evans 08 Sep '04

08 Sep '04

Perhaps with a bit of low-level help from the Python Interpreter, "flow" can be rendered obsolete. - Clark ----- Forwarded message from "Clark C. Evans" <cce(a)clarkevans.com> ----- Date: Tue, 7 Sep 2004 21:48:45 -0400 From: "Clark C. Evans" <cce(a)clarkevans.com> To: python-dev(a)python.org Subject: [Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration I've packaged up the idea of a coroutine facility using iterators and an exception, SuspendIteration. This would require some rather deep changes to how generators are implemented, however, it seems backwards compatible, implementable /w JVM or CLR, and would make most of my database/web development work far more pleasant. http://www.python.org/peps/pep-0334.html Cheers! Clark ... PEP: 334 Title: Simple Coroutines via SuspendIteration Version: $Revision: 1.1 $ Last-Modified: $Date: 2004/09/08 00:11:18 $ Author: Clark C. Evans <info(a)clarkevans.com> Status: Draft Type: Standards Track Python-Version: 3.0 Content-Type: text/x-rst Created: 26-Aug-2004 Post-History: Abstract ======== Asynchronous application frameworks such as Twisted [1]_ and Peak [2]_, are based on a cooperative multitasking via event queues or deferred execution. While this approach to application development does not involve threads and thus avoids a whole class of problems [3]_, it creates a different sort of programming challenge. When an I/O operation would block, a user request must suspend so that other requests can proceed. The concept of a coroutine [4]_ promises to help the application developer grapple with this state management difficulty. This PEP proposes a limited approach to coroutines based on an extension to the iterator protocol [5]_. Currently, an iterator may raise a StopIteration exception to indicate that it is done producing values. This proposal adds another exception to this protocol, SuspendIteration, which indicates that the given iterator may have more values to produce, but is unable to do so at this time. Rationale ========= There are two current approaches to bringing co-routines to Python. Christian Tismer's Stackless [6]_ involves a ground-up restructuring of Python's execution model by hacking the 'C' stack. While this approach works, its operation is hard to describe and keep portable. A related approach is to compile Python code to Parrot [7]_, a register-based virtual machine, which has coroutines. Unfortunately, neither of these solutions is portable with IronPython (CLR) or Jython (JavaVM). It is thought that a more limited approach, based on iterators, could provide a coroutine facility to application programmers and still be portable across runtimes. * Iterators keep their state in local variables that are not on the "C" stack. Iterators can be viewed as classes, with state stored in member variables that are persistent across calls to its next() method. * While an uncaught exception may terminate a function's execution, an uncaught exception need not invalidate an iterator. The proposed exception, SuspendIteration, uses this feature. In other words, just because one call to next() results in an exception does not necessarily need to imply that the iterator itself is no longer capable of producing values. There are four places where this new exception impacts: * The simple generator [8]_ mechanism could be extended to safely 'catch' this SuspendIteration exception, stuff away its current state, and pass the exception on to the caller. * Various iterator filters [9]_ in the standard library, such as itertools.izip should be made aware of this exception so that it can transparently propagate SuspendIteration. * Iterators generated from I/O operations, such as a file or socket reader, could be modified to have a non-blocking variety. This option would raise a subclass of SuspendIteration if the requested operation would block. * The asyncore library could be updated to provide a basic 'runner' that pulls from an iterator; if the SuspendIteration exception is caught, then it moves on to the next iterator in its runlist [10]_. External frameworks like Twisted would provide alternative implementations, perhaps based on FreeBSD's kqueue or Linux's epoll. While these may seem dramatic changes, it is a very small amount of work compared with the utility provided by continuations. Semantics ========= This section will explain, at a high level, how the introduction of this new SuspendIteration exception would behave. Simple Iterators ---------------- The current functionality of iterators is best seen with a simple example which produces two values 'one' and 'two'. :: class States: def __iter__(self): self._next = self.state_one return self def next(self): return self._next() def state_one(self): self._next = self.state_two return "one" def state_two(self): self._next = self.state_stop return "two" def state_stop(self): raise StopIteration print list(States()) An equivalent iteration could, of course, be created by the following generator:: def States(): yield 'one' yield 'two' print list(States()) Introducing SuspendIteration ---------------------------- Suppose that between producing 'one' and 'two', the generator above could block on a socket read. In this case, we would want to raise SuspendIteration to signal that the iterator is not done producing, but is unable to provide a value at the current moment. :: from random import randint from time import sleep class SuspendIteration(Exception): pass class NonBlockingResource: """Randomly unable to produce the second value""" def __iter__(self): self._next = self.state_one return self def next(self): return self._next() def state_one(self): self._next = self.state_suspend return "one" def state_suspend(self): rand = randint(1,10) if 2 == rand: self._next = self.state_two return self.state_two() raise SuspendIteration() def state_two(self): self._next = self.state_stop return "two" def state_stop(self): raise StopIteration def sleeplist(iterator, timeout = .1): """ Do other things (e.g. sleep) while resource is unable to provide the next value """ it = iter(iterator) retval = [] while True: try: retval.append(it.next()) except SuspendIteration: sleep(timeout) continue except StopIteration: break return retval print sleeplist(NonBlockingResource()) In a real-world situation, the NonBlockingResource would be a file iterator, socket handle, or other I/O based producer. The sleeplist would instead be an async reactor, such as those found in asyncore or Twisted. The non-blocking resource could, of course, be written as a generator:: def NonBlockingResource(): yield "one" while True: rand = randint(1,10) if 2 == rand: break raise SuspendIteration() yield "two" It is not necessary to add a keyword, 'suspend', since most real content generators will not be in application code, they will be in low-level I/O based operations. Since most programmers need not be exposed to the SuspendIteration() mechanism, a keyword is not needed. Application Iterators --------------------- The previous example is rather contrived, a more 'real-world' example would be a web page generator which yields HTML content, and pulls from a database. Note that this is an example of neither the 'producer' nor the 'consumer', but rather of a filter. :: def ListAlbums(cursor): cursor.execute("SELECT title, artist FROM album") yield '<html><body><table><tr><td>Title</td><td>Artist</td></tr>' for (title, artist) in cursor: yield '<tr><td>%s</td><td>%s</td></tr>' % (title, artist) yield '</table></body></html>' The problem, of course, is that the database may block for some time before any rows are returned, and that during execution, rows may be returned in blocks of 10 or 100 at a time. Ideally, if the database blocks for the next set of rows, another user connection could be serviced. Note the complete absence of SuspendIterator in the above code. If done correctly, application developers would be able to focus on functionality rather than concurrency issues. The iterator created by the above generator should do the magic necessary to maintain state, yet pass the exception through to a lower-level async framework. Here is an example of what the corresponding iterator would look like if coded up as a class:: class ListAlbums: def __init__(self, cursor): self.cursor = cursor def __iter__(self): self.cursor.execute("SELECT title, artist FROM album") self._iter = iter(self._cursor) self._next = self.state_head return self def next(self): return self._next() def state_head(self): self._next = self.state_cursor return "<html><body><table><tr><td>\ Title</td><td>Artist</td></tr>" def state_tail(self): self._next = self.state_stop return "</table></body></html>" def state_cursor(self): try: (title,artist) = self._iter.next() return '<tr><td>%s</td><td>%s</td></tr>' % (title, artist) except StopIteration: self._next = self.state_tail return self.next() except SuspendIteration: # just pass-through raise def state_stop(self): raise StopIteration Complicating Factors -------------------- While the above example is straight-forward, things are a bit more complicated if the intermediate generator 'condenses' values, that is, it pulls in two or more values for each value it produces. For example, :: def pair(iterLeft,iterRight): rhs = iter(iterRight) lhs = iter(iterLeft) while True: yield (rhs.next(), lhs.next()) In this case, the corresponding iterator behavior has to be a bit more subtle to handle the case of either the right or left iterator raising SuspendIteration. It seems to be a matter of decomposing the generator to recognize intermediate states where a SuspendIterator exception from the producing context could happen. :: class pair: def __init__(self, iterLeft, iterRight): self.iterLeft = iterLeft self.iterRight = iterRight def __iter__(self): self.rhs = iter(iterRight) self.lhs = iter(iterLeft) self._temp_rhs = None self._temp_lhs = None self._next = self.state_rhs return self def next(self): return self._next() def state_rhs(self): self._temp_rhs = self.rhs.next() self._next = self.state_lhs return self.next() def state_lhs(self): self._temp_lhs = self.lhs.next() self._next = self.state_pair return self.next() def state_pair(self): self._next = self.state_rhs return (self._temp_rhs, self._temp_lhs) This proposal assumes that a corresponding iterator written using this class-based method is possible for existing generators. The challenge seems to be the identification of distinct states within the generator where suspension could occur. Resource Cleanup ---------------- The current generator mechanism has a strange interaction with exceptions where a 'yield' statement is not allowed within a try/finally block. The SuspendIterator exception provides another similar issue. The impacts of this issue are not clear. However it may be that re-writing the generator into a state machine, as the previous section did, could resolve this issue allowing for the situation to be no-worse than, and perhaps even removing the yield/finally situation. More investigation is needed in this area. API and Limitations ------------------- This proposal only covers 'suspending' a chain of iterators, and does not cover (of course) suspending general functions, methods, or "C" extension function. While there could be no direct support for creating generators in "C" code, native "C" iterators which comply with the SuspendIterator semantics are certainly possible. Low-Level Implementation ======================== The author of the PEP is not yet familiar with the Python execution model to comment in this area. References ========== .. [1] Twisted (http://twistedmatrix.com) .. [2] Peak (http://peak.telecommunity.com) .. [3] C10K (http://www.kegel.com/c10k.html) .. [4] Coroutines (http://c2.com/cgi/wiki?CallWithCurrentContinuation) .. [5] PEP 234, Iterators (http://www.python.org/peps/pep-0234.html) .. [6] Stackless Python (http://stackless.com) .. [7] Parrot /w coroutines (http://www.sidhe.org/~dan/blog/archives/000178.html) .. [8] PEP 255, Simple Generators (http://www.python.org/peps/pep-0255.html) .. [9] itertools - Functions creating iterators (http://docs.python.org/lib/module-itertools.html) .. [10] Microthreads in Python, David Mertz (http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- Clark C. Evans Prometheus Research, LLC. http://www.prometheusresearch.com/ o office: +1.203.777.2550 ~/ , mobile: +1.203.444.0557 // (( Prometheus Research: Transforming Data Into Knowledge \\ , \/ - Research Exchange Database /\ - Survey & Assessment Technologies ` \ - Software Tools for Researchers ~ * _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/cce%40clarkevans.com ----- End forwarded message ----- -- Clark C. Evans Prometheus Research, LLC. http://www.prometheusresearch.com/ o office: +1.203.777.2550 ~/ , mobile: +1.203.444.0557 // (( Prometheus Research: Transforming Data Into Knowledge \\ , \/ - Research Exchange Database /\ - Survey & Assessment Technologies ` \ - Software Tools for Researchers ~ *

1 0

[Twisted-Python] twistd quietness
by Tsai Li Ming 08 Sep '04

08 Sep '04

Hi Is there an option in twistd from showing stderr messages from appearing, specifically "Removing stale pidfile /var/run/myProg.pid"? # twistd -y myProg.tac -q --pid /var/run/myProg.pid I'm already using the quiet option. Do I simply redirect stderr to /dev/null? -Liming

1 0

[Twisted-Python] Is anyone working on CFReactor?
by vng1＠mac.com 07 Sep '04

07 Sep '04

Is anyone looking at what's wrong with the CFReactor in Twisted 1.3 and OSX? http://www.twistedmatrix.com/users/roundup.twistd/twisted/issue648 I vaguely remember seeing a posting about the GIL causing problems, but I can't seem to find that article now. vic

3 2

[Twisted-Python] Using setuid/setgid correctly
by Abe Fettig 07 Sep '04

07 Sep '04

Hi folks, Can anyone give me a quick primer (or point me to some documentation) on using uid/gids with twisted.application and mktap? I'm looking to run a server process under an unprivileged user account, while still being able to bind to reserved ports at startup. Thanks! Abe

1 0

[Twisted-Python] Twisted XML-RPC Database Interface
by Patrick Kelly 07 Sep '04

07 Sep '04

Hi, I'm new to twisted and fairly new to python. I'm toying with a way to use Twisted/XML-RPC to make calls to a mysql database. I've gone through the examples on the web page, but I'm still having trouble putting it all together. Is there some good example code somewhere else that will help write the server and client? Thanks for you help Patrick

1 0

[Twisted-Python] PB / XML
by Sergio Trejo 07 Sep '04

07 Sep '04

I read with interest a thread that was discussed over a year ago on the mailing list which was initiated by Uwe C. Schroeder discussing use of Perspective Broker to pipe around data (in which PB has some size restrictions on how much it will vend over the wire between PB Client and PB Server): http://twistedmatrix.com/pipermail/twisted-python/2003-May/003986.html Similar to Uwe, I want to push around some XML data. I don't need SOAP and I don't need XML-RPC. I just want to push little chunks of XML between PB Client and PB Server, and I want to do so in a manner that adheres to the strictness of PB and its data size limits per the thread mentioned above. What I am curious about is if any twisted developers can recommend a means for encoding and decoding an XML document before / after sending such encoded document over the wire via PB? It would be nice to be able to validate the XML). I have taken notice of twisted.protocols.sux but in reading the API documents, it doesn't appear as if sux is going to be applicable to PB usage since sux can not perform *any* external actions under *any* circumstances" ... plus sux doesn't validate. So, can anyone recommend a known method for taking XML data into a generic Python object which is compatable with PB? Eventually I want to take the XML data and persist it into an atop data base but that's later down the pipeline and better left for the divmod list. Thanks for any feedback / experiences, Serg _________________________________________________________________ MSN 8 with e-mail virus protection service: 2 months FREE* http://join.msn.com/?page=features/virus

1 0

Re: [Twisted-Python] Patch for MSN protocol & Jabber protocol
by James Bunton 05 Sep '04

05 Sep '04

On Sun, 5 Sep 2004 04:00 am, Jp Calderone <exarkun(a)divmod.com> wrote: > Thanks for the patches, James. In addition to Glyph's comments, I > wonder if you could include unit tests which: > > 1) fail when run against the current Twisted code and > 2) pass with your changes > > This is especially important for protocol code. Without extremely > good test coverage, different people begin to depend on different > (unknown) chunks of functionality. Then there is no way to know what > future changes can be made without breaking existing code (in this case, > potentially yours ;). > > Jp Ok. They're both in the tracker as Issue701 (Jabber stuff, assigned to dizzyd) and Issue702 (MSN stuff, assigned to nobody). I wrote a unit test for the Jabber stuff, and for the parts of the MSN fixes where it makes sense (the NOT message, aka MSN alerts fixes). It'd be great for these to get into the next Twisted release. --- James

1 0

[Twisted-Python] Patch for MSN protocol & Jabber protocol
by James Bunton 04 Sep '04

04 Sep '04

Here's a patch that fixes a few bugs in the Twisted MSN support. * An exception will no longer be thrown on an incoming NOT (notification) message. * A fix for occasional exceptions on receiving messages. * Fix receiving of contact status and friendly name changes to be safer, and to actually update the factory =P (friendly name updates didn't occur previously) * Removed a stray print statement I've been using these fixes for a while, as they were necessary to get the Jabber PyMSNt to work. They seem to be stable. It'd be great to get them into the next version. Now for the Jabber support in Twisted. There's a list of patches on this website here that it would be great to have included. Especially the domish.py patch, without that an exception is thrown on some XML that many Jabber clients send with every packet. That makes it almost unusable. http://idavoll.jabberstudio.org/2.html --- James

3 2

[Twisted-Python] HTTPS proxy
by James Bunton 04 Sep '04

04 Sep '04

Hi all. Me again =) Is there any way to specify a proxy to use for an HTTPS connection? I've had a bit of a look around and I can't seem to find anything. Any help would be much appreciated. --- James

1 0

[Twisted-Python] Making non-blocking code blocking?
by Paul Campbell 04 Sep '04

04 Sep '04

Here's the situation... I've been writing all my code to defer correctly. Every time it gets to a point where the call can block, I've inserted the appropriate callback function. However, it occurred to me (for testing purposes) that it would be nice some times to INTENTIONALLY block. Right now, it seems like the only point where you can actually "block" is when you reach the top level where the code is interacting with the reactor interface. So far, it seems like once you get the idea of writing deferred code (which is essentially continuation passing style, right?) that it is hard to get OUT of the mindset of writing deferreds into everything. I'm writing a whole lot of RPC-based code right now, which means there are LOTS of places where blocking actions can happen, so almost everything ends up wrapped with deferreds. The alternative is to use threads, but...ugh! I think those are even uglier.

2 1

[Twisted-Python] How to install to another directory?
by junk＠mnjdesigns.com 03 Sep '04

03 Sep '04

How do I install to another directory other then the default using python setup.py install. Any help appreciated! - M

3 2

[Twisted-Python] Small bug fix for twisted.mail.pop3
by Andreas Kostyrka 03 Sep '04

03 Sep '04

Hi! I'd propose to include this small patch, as it seems to be unconditionally verbose: Index: twisted/mail/pop3.py =================================================================== --- twisted/mail/pop3.py (revision 11428) +++ twisted/mail/pop3.py (working copy) @@ -148,7 +148,8 @@ self.magic = self.generateMagic() self.successResponse(self.magic) self.setTimeout(self.timeOut) - log.msg("New connection from " + str(self.transport.getPeer())) + if self.factory.noisy: + log.msg("New connection from " + str(self.transport.getPeer())) def connectionLost(self, reason): if self._onLogout is not None: As noisy is defined twisted.internet.protocol.Factory, it should benign, running trial on twisted/test doesn't seem to produce anything worse than a heap of Deprecation warnings. TIA, Andreas -- Andreas Kostyrka Josef-Mayer-Strasse 5 83043 Bad Aibling

1 0

[Twisted-Python] Spawning processes and multiple CPU's.
by Matt Feifarek 02 Sep '04

02 Sep '04

I'm using reactor.spawnProcess to perform some inherently-blocking code (imagemagick scripts). I'm curious if multiple CPU's would help me out. I know that a single python (and therefore a single reactor) cannot benefit from multiple CPU's, but since the spawn process is a kernel call, will it be sent to both processors? Thanks.

2 3

[Twisted-Python] Re: Perspective Broker and UDP, is it possible?
by Ed Suominen 02 Sep '04

02 Sep '04

> I'm doing my master thesis in computer science, and the topic > for the project is "Managment of network delays in computer games". > The software im going to use is Python with Twisted and Pygame. > Right now I'm trying to get to know how different parts of Twisted > interact with each other. And my first question is: > > Is it possible to use Perspective Broker with UDP? > I've tried modifying the examples here > (http://www.twistedmatrix.com/documents/current/howto/pb-usage), > with DatagramProtocol and so on, but I haven't got it to work. > I suspect that it is impossible due to that UDPs is connectionless, > but there's a lot of "magic" in Twisted, so I thought I'd better > ask the pros'. It is certainly possible, just not with the current version of PB. In any modified version (go for it!), I think you would need to provide client identifying info with each burst of data, or limit "connections" to just one client at a time per port, to keep track of things. I'm not a twisted expert, but I do use it extensively. I have also learned an embarrassingly large amount of stuff about low-level TCP/IP due to too much experience with a vastly inferior high-level language [1]. [1] http://tinyurl.com/5nrkp Ed Suominen Registered Patent Agent Open-Source Software Author (yes, both...) Web Site: http://www.eepatents.com

2 1

[Twisted-Python] Perspective Broker and UDP, is it possible?
by Joakim Bech 01 Sep '04

01 Sep '04

I'm doing my master thesis in computer science, and the topic for the project is "Managment of network delays in computer games". The software im going to use is Python with Twisted and Pygame. Right now I'm trying to get to know how different parts of Twisted interact with each other. And my first question is: Is it possible to use Perspective Broker with UDP? I've tried modifying the examples here (http://www.twistedmatrix.com/documents/current/howto/pb-usage), with DatagramProtocol and so on, but I haven't got it to work. I suspect that it is impossible due to that UDPs is connectionless, but there's a lot of "magic" in Twisted, so I thought I'd better ask the pros'. Mvh // Joakim Bech Mail: d00jb(a)efd.lth.se

2 1

[Twisted-Python] Resource Tree
by Tsai Li Ming 01 Sep '04

01 Sep '04

Hi For example, as given in the howto to handle url path segment: root = Hello() root.putChild('fred', Hello()) root.putChild('bob', Hello()) Therefore, the following URLs are valid: http://example.com/ http://example.com/fred http://example.com/bob http://example.com/fred/ http://example.com/bob/ How do I handle non-existent url like http://example.com/foo? Liming

3 2