Mailman 3 June 2008 - Twisted

[Twisted-Python] chaining deferreds
by Andy Hird 20 Jun '08

20 Jun '08

Hi all, I've been trying to get a really basic example of chaining deferreds going with no luck. I was hoping someone could give me some help and tell me where I'm going wrong. I've attached the basic code to this message. I'd expect output to be: constructed deferred setup everything printing: hello mr doing something else: printed something finishing up with: done something else But instead the callback chain stops after print_something() has been called (do_something inbetween and finish_up are never called). Thanks for any help in advance! Andy Hird

2 1

[Twisted-Python] RE: ipython & twisted: no module named threading
by Mike Winter (miwinter) 20 Jun '08

20 Jun '08

This is a record of analysis and consequent workaround to enable running ipython and Twisted's trial. Posting to twisted, it probably requires no action from twisted team since I suspect IPython is not dealing with the situation correctly. Ipython needs to resolve the problem w/ a try: import thread except ImportException: print "failed to import thread" import dummy_thread as thread For ipython to 'work' I copy dummy_thread.py thread.py, then when I run trial or twistd I see the following exception. If I delete the thread.py this does not show up, but ipython stops working. Exception exceptions.ImportError: 'No module named threading' in <bound method local.__del__ of <_threading_local.local object at 0xb72f6874>> ignored (venv)[miwinter@sawwing test]$ twistd -ny finger.tac shows the same exception (from twisted's finger tutorial). Python -V 2.4.5 and 2.5.1 and 2.5.2 Ipython-0.8.2 * Twisted 8.1.0 virtualenv 1.0 CVSToys 1.0.10 python -m virtualenv venv . venv/bin/activate (venv)# make InstallTwisted (cd Twisted-8.1.0; python setup.py install) (venv)# make InstallBuildbot (easy_install buildbot) (venv)# trial buildbot.test.test_twisted .... Shows the above problem: [skip] PASSED (skips=37, successes=271) Exception exceptions.AttributeError: "'NoneType' object has no attribute 'StringIO'" in <bound method RemoteReference.__del__ of <twisted.spread.pb.RemoteReference instance at 0xb6856aec>> ignored Exception exceptions.ImportError: 'No module named threading' in <bound method local.__del__ of <_threading_local.local object at 0xb72dacfc>> ignored test main-trunk]$ md5sum lib/python2.4/thread*py lib/python2.4/dummy_thread*py 3783492a0969521ff1266ec6c3e570c2 lib/python2.4/threading.py 38db2323ac32ddbcad69888abd30d21f lib/python2.4/dummy_threading.py 52a6d92c98cc5a2eb68a3b1dde16e58b lib/python2.4/thread.py 52a6d92c98cc5a2eb68a3b1dde16e58b lib/python2.4/dummy_thread.py Without thread.py ipython will fail to start, so in my makefile I do: fixThread: bin/python lib/python$(PY_MACRO_VERSION)/thread.py lib/python$(PY_MACRO_VERSION)/thread.py: (cd lib/python$(PY_MACRO_VERSION); cp dummy_thread.py thread.py) This exception (see start of email) is perhaps caused by dummy.py lacking some of the guts that some inappropriately intimate part of threading.py depends upon. w/o target fixThread ipython will not start! Since resolving this by bandaiding Makefile I thought it worthy to report that my patch to my Makefile was to add target unfixThread: to reverse the effects of fixThread which was a hack for ipython to work, better fix ipython somehow.

1 0

[Twisted-Python] web client and proxy
by Olivier Fournier 19 Jun '08

19 Jun '08

Hi, My english is very bad, sorry... I don't understand how to access to a web server behind a proxy from web.client ? somebody could explain me how to do? thanks Olivier Fournier

2 1

[Twisted-Python] Something like a ThreadPool
by kevin beckford 19 Jun '08

19 Jun '08

Is there any sort of job queue or similar , besides AsynQueue, which looked like exactly what i wanted, but is sadly undocumented, native to twisted? What is the twisted way to do such a thing? Is there something already available? -- Kevin Beckford Technical Lead, Lazyweb Construction Company, http://lazyweb.ca kevin(a)lazyweb.ca

4 6

[Twisted-Python] The twisted way for basic file i/o?
by Nathan 19 Jun '08

19 Jun '08

I just want to do some basic text-file manipulations without blocking the rest of twisted. I've found "FilePath", which has an open()...which is undocumented: http://twistedmatrix.com/documents/current/api/twisted.python.filepath.File… Is that what I should be using??? ~ Nathan

2 2

Re: AW: [Twisted-Python] Twisted Trial
by Curt 18 Jun '08

18 Jun '08

Torsten A. wrote: > I have never really used a version control system I have to admit. I thought > of control systems to log changes which can in case of severe damage or what > ever reason be made undone. But if I can execute different versions of the > project, e.g. in my case the initial version and the last modified, just by > saying run this one or that one, I am glad to use it. Can this be done > (excuse my probably dumb question)? Can someone please give me catchword to > search for? As I looked briefly in the bazaar guide > (http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/index.html) I wasn't able > to find anything helpful. > > Thanks again for your help! I really appreciate it! > > I use bazaar myself for version control, and I highly recommend it. If you're popping in and out on different branches/versions all the time though, you may want to look at combinator from Divmod. Sounds like just what you're looking for. Here is a link to it, the link isn't working for me at this particular moment, but I know it is a good link, hopefully will be back up sometime today. http://divmod.org/trac/wiki/DivmodCombinator

1 0

Re: [Twisted-Python] Twisted Trial
by Terry Jones 18 Jun '08

18 Jun '08

The various answers recommending version control are just heading off in a direction that wasn't asked about at all! :-) Sorry if I sound grouchy, but the OP hasn't done anything to warrant being curtly told to just use version control, especially when that's not a solution to the problem he's running into. There should be no need to add that people often have multiple versions (branches, often) checked out and want to swap between using them and have the right version picked up by Python. That's part of the point/use of combinator and the *branch commands, as many people here already know. Torsten, you say your code doesn't work when you have PYTHONPATH set. Can you be more specific? Send us the output showing it not working, etc? Terry

2 1

[Twisted-python] Writing industrial grade twisted servers
by kevin beckford 18 Jun '08

18 Jun '08

Hi all, I've been reading through the twisted documentation, trying to get a handle on how to write a real life twisted app. ( I need to use the AsynQueue for my project, btw..) So I've been reading the documentation, and I've been putting together some things.. but i note that much of the docs says that "this is not the way you usually do this" Where do i read to find out how to simply create a production quality twisted application? -- Kevin Beckford Technical Lead, Lazyweb Construction Company, http://lazyweb.ca kevin(a)lazyweb.ca

7 6

[Twisted-Python] Executing a number of commands over SSH
by James Brady 18 Jun '08

18 Jun '08

Hi all, I'd like to run a few commands on a remote machine over SSH. Looking at the conch packages, I'm not sure what the best approach is for this. The only way I can see to pass the set of commands down into the SSHConnection and SSHChannel objects is through my transport factory, i.e. I can't add or remove commands to be run on an already instantiated SSHConnection. Also, I'm a newcomer to the Twisted way of doing things, so it's not clear to me how I would go about returning the status and output of these commands back up to my code. I'm guessing I should create a Deferred somewhere, but some guidance on how these should bubble up through the twisted layers would be much appreciated. Thanks, James

2 2

Re: [Twisted-Python] Re-working a synchronous iterator to use Twisted
by Terry Jones 18 Jun '08

18 Jun '08

For the record, here's a followup to my own posting, with working code. The earlier untested code was a bit of a mess. The below runs fine. In case it wasn't clear before, you're pulling "results" (e.g., from a search engine) in off the web. Each results pages comes with an indicator to tell you whether there are more results. I wanted to write a function (see processResults below) that, when called, would call the process function below on each result, all done asynchronously. This solution feels cumbersome, but it does work (aka prints the expected output). Comments welcome (just don't tell me to use version control :-)) Terry from twisted.internet import defer def process(result): # Stub: process a single result. print 'processed result', result def getPage(uri, offset=0): # Stub: return some results and an indicator of whether more are available. data = [[0, 1, 2], [4, 7, 8, 10], [12, 14, 18, 30]] results = data[offset] more = offset < len(data) - 1 return defer.succeed((results, more)) def getResultsFromPage(page): # Stub: get the results from page, return them in a list return page[0] def needToCallAgain(page): # Stub: determine if there are more results, given current page. return bool. return page[1] # ASYNCHRONOUS result producer def getResults(uri, offset=0): def parsePage(page, offset): results = getResultsFromPage(page) if needToCallAgain(page): d = getResults(uri, offset + 1) else: d = None return results, d def returnTheseResults(page, offset): resultIterator, done = parsePage(page, offset) return resultIterator, done return getPage(uri, offset).addCallback(returnTheseResults, offset) # ASYNCHRONOUS calling def processResults(uri): def cb((resultIterator, deferred)): for result in resultIterator: process(result) if deferred is not None: deferred.addCallback(cb) return getResults(uri).addCallback(cb) if __name__ == '__main__': def finished(x): print 'finished' processResults('uri').addCallback(finished) The above prints $ python xxx.py processed result 0 processed result 1 processed result 2 processed result 4 processed result 7 processed result 8 processed result 10 processed result 12 processed result 14 processed result 18 processed result 30 finished

1 0