Mailman 3 March 2003 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 06 Aug '21

06 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] documentation issue?
by rprice＠collegenet.com 27 Apr '03

27 Apr '03

Hi, I am working my way through "The Twisted Documentation" Jan 29, 2003 and found a problem with an example (or with me). At http://twistedmatrix.com/documents/howto/plugin it says: % mktap qotd Saving qotd application to qotd.tap... Saved. % twistd -f qotd.tap % nc localhost 8007 An apple a day keeps the doctor away. % kill `cat twistd.pid` The problem is when I run 'mktap qotd' I get the following error: Unknown command: qotd and then it prints the usage and options help. I am using python 2.2.1 and Twisted 1.0.3 Any help would be appreciated. Ray

3 3

[Twisted-Python] t.i.process uid/gid suckiness
by Tommi Virtanen 23 Apr '03

23 Apr '03

The handling of uid/gid changes and SIGCHLD all around twisted are in an appalling state. I've tried to bitch at the right people, but that hasn't helped, so I'm taking the more heavyhanded approach of just fixing stuff. Let the code talk. However, I still don't consider myself an authority on what Twisted should do, don't have that much time longer term (I will not maintain this code permanently), and don't know what silly limitations the windows port of python has. So, I'll try to write down what changes I'm planning on doing, so you have a chance of vetoing. If I get no comments, I'll just basically go ahead and commit what I happen to want to, touching a many areas of mktap/t.i.process/procmon/whatever. If you want to avoid a de facto hijack-fix-abandon process, TALK NOW 0) make the t.i.process setuid/setgid code actually work (see patch; BTW is the initgroups part really needed? I feel my pure-python 6-liner does the same thing.) 1) make all places that take uid/gid just silently take in strings, too, and use pwd/grp to convert them to uids and gids as necessary 2) make .taps store uids and gids as strings 3) defer procmon startup so the processes are forked only after setuid has happened. 4) try to fix the child process races, atleast including deferring signal-triggered processing to happen outside the actual signal handler, rewriting the reap logic silliness, and stopping procmon from trying to kill reaped children. 5) fix whatever problems I noticed while fixing the above 6) whatever else I feel like improving at the time 7) ??? 8) profit! diff -u -u -r1.50 process.py --- twisted/internet/process.py 10 Mar 2003 20:16:57 -0000 1.50 +++ twisted/internet/process.py 11 Mar 2003 13:08:00 -0000 @@ -33,10 +33,25 @@ pty = None try: - from initgroups import initgroups import pwd + try: + from initgroups import initgroups + except: + import grp + def initgroups(username, dummy): + l=[] + for groupname, password, gid, userlist in grp.getgrall(): + if username in userlist: + l.append(gid) + os.setgroups(l) + def switch_uid(uid, gid): + os.setgid(gid) + initgroups(pwd.getpwuid(uid)[0], gid) + os.setuid(uid) except: - def initgroups(*args): pass + def switch_uid(uid, gid): + os.setgid(gid) + os.setuid(uid) from twisted.persisted import styles from twisted.python import log, failure @@ -229,9 +244,7 @@ os.chdir(path) # set the UID before I actually exec the process if settingUID: - os.setgid(gid) - initgroups(pwd.getpwuid(uid)[0], gid) - os.setuid(uid) + switch_uid(uid, gid) os.execvpe(command, args, environment) except: # If there are errors, bail and try to write something @@ -482,9 +495,7 @@ # set the UID before I actually exec the process if settingUID: - os.setgid(gid) - initgroups(pwd.getpwuid(uid)[0], gid) - os.setuid(uid) + switch_uid(uid, gid) os.execvpe(command, args, environment) except: stderr = os.fdopen(1, 'w') -- :(){ :|:&};:

5 7

[Twisted-Python] Lore vs. latex2html: Fight to the Finish
by Moshe Zadka 22 Apr '03

22 Apr '03

Dear Twisted developers, I have started converting Thinking Like a Computer Scientist in Python to Lore, to try and see what Lore is missing. It has been a most education experience. The two prime things Lore is currently missing for lossless conversion (of the specific chapter I tested) are: * EPS images support * Indexing support However, beyond that, I am interested to know which looks better: latex2html: http://www.ibiblio.org/obp/thinkCSpy/chap01.htm Lore: http://twistedmatrix.com/users/moshez/loreprop/ (The Lore is using the Twisted documentation stylesheet.) Any comments would be nice. Comments which come with a patch to the stylesheet will be doubly appreciated. Thanks in advance, Moshe Zadka -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness.

4 3

[Twisted-Python] flow - a state-machine consumer
by Clark C. Evans 09 Apr '03

09 Apr '03

After being convinced on IRC that my use cases arn't what Deferred is meant to solve, Donovan "wzZzy" and I put together a first pass at a more granular 'consumer'. I'm now using it to build some fairly complicated web pages from a database, and may find its way into Woven (although perhaps not in its current form). Anyway, the tool includes: 0. the Flow dispatcher, which is a reactor like mechanism driving events. 1. a mechanism to execute a function in a thread, where the function returns an iterator which is used to generate events in the main thread. 2. an superclass of adbapi.ConnectionPool with a queryChunked function which returns a result set incrementally (row by row) to the main thread for further processing Future plans: 0. Work on exception handling and trapping just to verify that it works as coded. 1. Incorporate some way for the reactor to 'pause' itself (and throw it self into the event queue) so that other events can be processed 2. Update the code so that the execute to thread function can take an initial state for an argument 3. Update the SQL linkage so that it can take more than one query in parallel, sequencing the result sets, and returning them to a shared flow object using different state arguments. It isn't stable, but you can find it in CVS, or at http://clarkevans.com/flow.py Comments are encouraged. Best, Clark

3 3

[Twisted-Python] How do I upload files using FTPClient?
by Daniel Burr 09 Apr '03

09 Apr '03

I am currently designing a code-deploying service, to be used in conjunction with a variety of source control tools. I'm looking to use Twisted in the implementation of the file-transfer mechanism, as Twisted's broad protocol support would make it easy for me to switch from say, FTP to HTTP, or to some custom protocol I roll myself. I've experimented with twisted.protocols.ftp.FTPClient, and the one thing I can't seem to grok is how to perform file uploads. I've read through the code, and it seems that the FTP server class supports RETR and STOR, using DTPFactory, while FTPClient only supports RETR, and implements it independently. Basically, I'm lost as to how to connect FTPClient to DTPFactory, such that I can send files from FTPClient. I just need some pointers on how to accomplish this; it seems like the code is there to do it, but I'm missing the necessary Twisted mindset in this case. Thanks for your help. L. Daniel Burr

2 3

[Twisted-Python] more thoughts on resumable async data flows
by Clark C. Evans 08 Apr '03

08 Apr '03

Howdy. I've made quite a few changes to flow.py and it's quickly approaching maturity. To summarize, flow.py is a way to express sequential, streaming data flows that are interruptable as a collection of small, atomic operations. This is needed since sometimes an operation must block, flow.py takes care of resuming your flow at a later time. It's trivial, but here is an example... def printResult(data): print data def addOne(data): return data+1 def finished(): print "finished" def dataSource(data): return [1, 1+data, 1+data*2] a = Flow() a.addBranch(dataSource, finished) a.addCallable(addOne) a.addCallable(printResult) a.execute(2) a.execute(8) The above code constructs a flow, starting with a 'branch' operation that generates a series of events, in this case, the lists [1,3,5] and then in a second execution, [1,9,17]. For each one of these events, a function addOne is called on them, and then the output of that function is directed to be printed. This mechanism works with generators... def simpleGenerator(data): for x in range(data): yield x b = Flow() b.addBranch(simpleGenerator) b.addCallable(printResult) b.execute(5) While this may not be all that useful, what Flow brings to the table is the ability to PauseFlow within a processing stage, for example class simpleIterator: def __init__(self, data): self.data = data def __iter__(self): return self def next(self): print "." if self.data < 0: raise StopIteration ret = self.data self.data -= 1 # # imagine a blocking operation here... sometime if ret % 2: raise PauseFlow # <= Goes to main event loop return ret c = Flow() c.addBranch(simpleIterator) c.addCallable(printResult) c.execute(5) The above code produces 4, 2, 0 ; while this may not be all that interesting, if there were N stages above this code, they would all be resumed properly. By tossing PauseFlow, the entire Flow event loop is stopped, and a reactor.callLater(0, loop-again) is called; giving other events in the queue to work. Thus, the flow construct provides a way to not only manage a bunch of useful code snippets into a process; but more importantly gives a way that the entire flow can be interruped and then resumed later when data arrives. Anyway, it's in the sandbox if anyone wants to play... Clark P.S. Unfortunately, it looks like PauseFlow doesn't work in the context of a generator... pity. Hopefully I'm doing something wrong.

3 8

[Twisted-Python] Problems with PB and Jelly...
by Jasper Phillips 04 Apr '03

04 Apr '03

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to twisted, and attempting to use it for networkable gaming; unfortunately I've run into some problems. I have a Perspective <-> Referenceable server/client setup, and am trying to pass objects between them. Following the online docs I've set all the relavent classes to inherit from pb.Copyable + pb.RemoteCopy, and passed them all to pb.setUnjellyableForClass. However, I still get "Module not allowed" exceptions raised in twisted.spread.jelly.py line 523. Looking more closely, I see that modules are not allowed unless passed to SecurityOptions.allowModules(), which is only called from SecurityOptions.allowInstancesOf(), which is not exposed through pb... In short I don't see how the example code on: http://twistedmatrix.com/documents/howto/pb-copyable could possibly work! Is it indeed necessary to call allowInstancesOf() directly? Hopefully I am just missing something... :-) I'm also unsure what to make of the last bullet under "Things To Watch Out For" on the above webpage. It seems unsure whether using __init__ to initialize transferable objects is ok... Must I truly go and hack all my objects to not use __init__, and instead use setCopyableState()? This, especially when combined with the need to inherit from pb.Copyable + pb.RemoteCopy, seems onerous when compared to the minimal interface of pickle. I would like to be able to simple pass objects of arbitrary classes, provided I've registered them as UnJellyable. Hopefully I'm missing something here as well! Thanks in advance, - -Jasper -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE+eQVb8EpjZ7/X9bIRAoGmAKDOp/mMXUV0wGG057+CK5Shf75k9ACgu9Cb EuFB62b9vaObg3LZZlJ81jg= =8Dte -----END PGP SIGNATURE-----

5 12

[Twisted-Python] wxPython and Twisted event loops
by Uwe C. Schroeder 03 Apr '03

03 Apr '03

Hi, for those interested, I just posted a recipe on the activestate.com python cookbook about how to get wxPython running with Twisted withoud having blocked menu's and modal dialogs. It's a simple and dirty hack, but it's working on linux and win (basically all systems using the default select reactor) enjoy UC -- Open Source Solutions 4U, LLC 2570 Fleetwood Drive Phone: +1 650 872 2425 San Bruno, CA 94066 Cell: +1 650 302 2405 United States Fax: +1 650 872 2417

3 3

[Twisted-Python] win32guireactor
by Thomas Heller 02 Apr '03

02 Apr '03

Here's my win32guireactor code, which at least for me, works better than win32eventreactor in my application. I have some problems in my app when using win32eventreactor: If the app displays a message box, no callLater calls are done (because a win32 messagebox has it's own event loop, and doIteration() is not called during this time. There are also other things that the user can do which shows the same behaviour, dragging a scrollbar for example. The second problem is that win32eventreactor uses a lot a cpu time - nearly 100% even on my brandnew 2GHz P4. Looks much like polling behaviour instead of event driven. So I wrote my own reactor implementation, which passes all events (those I've covered so far, reactor.callLater and network events) through a win32 message loop. Deferred's are not yet passed through the loop, so there is still some polling to be done. Looking at the comments at the top of the win32eventreactor code, my win32guireactor module shouldn't have the problems mentioned there. I post this module here asking for peer review and critical comments. This module requires win32all and my ctypes module as well, see the source for details. Thanks, Thomas PS: I tried to run unittests with this and other reactors with the command line py22 bin\trial -p twisted.test.test_internet -r <reactor> -v here are the results: default reactor FAILED (failures=5, errors=18) win32eventreactor twisted.test.test_ftp:FTPClientAndServerTests.testLongFileListings hangs forver. win32guireactor twisted.test.test_ftp:FTPClientAndServerTests.testLongFileListings crashes with this traceback, and then also hangs forever. twisted.test.test_ftp FTPClientAndServerTests testBadLogin ... testLongFileListings ... Traceback (most recent call last): File "C:\sf\Twisted\twisted\internet\win32guireactor.py", line 85, in __wm_timerevent tple.func(*tple.args, **tple.kw) File "C:\sf\Twisted\twisted\test\test_ftp.py", line 124, in errback except self.failureException, e: AttributeError: FTPClientAndServerTests instance has no attribute 'failureException'

2 2