Twisted-web October 2009

twisted-web@python.org

13 participants
13 discussions

Start a nNew thread

Test

by Philippe Lafoucrière

Hi all just a test

5 months, 3 weeks

[Athena] Is ReliableMessageDelivery really necessary?

by Paul Thomas

Hi, I've hit a problem as my app has got bigger (about 30-40 widgets now, all chattering roughly once every 2 seconds) where the reliable message delivery mechanism is spiralling out of control. It seems that the constant back and forth means that large 'baskets' of messages are resent. The more this happens, the busier everything gets until the browser becomes unresponsive. There's a fix for it: [Divmod-dev] athena duplicate messages issue but I'm slightly concerned about the potential for lost messages - and also confused about how this could happen. Given that HTTP is a reliable connection-oriented transport, where is the gap that messages can fall through? I think I can cope with lost messages in most cases, so would it be useful to add a kind of 'sendRemote' that was like 'callRemote' but didn't care about a response? Or maybe this already exists and I've missed it? Paul. P.S. this app is likely to get more noisy - is it likely that I'll have to abandon Athena for Orbited or similar? I mean, are there architectural differences that will prevent Athena scaling?

12 years, 1 month

using css with nevow

by Eric Medina

Hi, I have been using twisted quite a bit these last 6 months for a networked application i'm working on. Recently, i've decided to port the web part of the app from spyce to nevow. However, i have encountered a problem with using css file. My css file has references to image files like below: *... body { background: #EEE url(img/bg.gif); color: #444; font: normal 62.5% "Lucida Sans Unicode",sans-serif; margin: 0; } ...* The following is a test file that loads an external xml file which references a css file: *class TestPage(rend.Page): addSlash = True child_images = static.File('img') children = {'default.css': static.File('default.css')} docFactory = loaders.xmlfile('webtest.html') site = appserver.NevowSite(TestPage()) reactor.listenTCP(8080, site) reactor.run()* The problem is that the images referenced inside the css file is not being loaded when the page is rendered. Can someone tell me how to fix this? This is the first time i've used nevow or even twisted.web. Thank you very much. Regards, Eric

12 years, 3 months

Limit the simultaneous twisted.web.client.downloadPage requests

by Igor Katson

Hello! I am a newbie in twisted, sorry if my question sounds awkward. I have written a pretty simple recursive page downloader, which parses an html, extracts all the needed links from it, and starts dowloading them. The links are the videofiles, so they are pretty large. The problem is, that the downloader works TOO FAST :) I want to set something like the global bandwidth limit or the maximum limit of concurrently downloading files. I am using the twisted.web.client.downloadPage to download the files and using the Deferred, that it returns. I can't understand how to make it still return a Deferred, corresponding to that file, but not start downloading right away, but instead start downloading it on some kind of event (make a manger-like wrapper for that function). So I want the code to still look simple like this: for link in links: d = downloadPage_limited(link, filename) And the wrapper(function downloadPage_limited) will manage the amount of concurrent downloads, and will still return the Deferred, which will be returned by twisted.web.client.downloadPage. Is my idea about a "wrapper" practical and what's the general way to write it? On which event is it better to decrement the counter of the amount currently downloading files? Hope it is clear enough. Thanks in advance, Igor Katson.

12 years, 3 months

Tracking the progress of the server

by naman jain

Hi, I have a client server model in twisted, where the server spawns a thread ( basically a test script in python that runs for about 20 mins) I want to track the progress of the thread, and send the progress to the client back So, I write something like this in my server: parent_conn, child_conn = Pipe() thread = Process(target = start_test.main_func, args=(SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn)) thread.start() response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) . . . thread.join() But the transport.write calls don't send at the same time. instead they wait for the thread to finish (coz of thread.join) and then append all the response and send it back; like "initialization doneconfiguration done...done" thereby defeating the purpose of creating a thread. How do I give the control to the reactor to write the data back, and still keep the thread running? or is there any other way these kinda of progress can be tracked ? I am kinda stuck with this :(

12 years, 3 months

Re: [Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

by Terry Jones

> Thanks for quick and great help, Terry and Jean-Paul! You're welcome. If in any doubt about conflicting advice, do what JP suggests. The reason I pointed you to JP's Cooperator solution was that when you extract the set of links from a page that you also want to fetch, you could make another generator and submit that to the same cooperator's coiterate function. That would just schedule the additional getPage calls to be run. The only wrinkle (I think) in that is that in doing the very last getPage, the cooperator will run out of more work and fire the deferred you got originally. But the final page may contain more links. I'm not sure if I'm thinking clearly about that final bit. That's your job :-) Terry

12 years, 3 months

Re: [Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

by Terry Jones

> That list seems like far from the most active one, but the one filled > with friendly people :) The thing I like when dealing with Twisted and > it's documentation is that it is seen often that people who make it, and > who use it, really love Twisted. Yes. Without the community, Twisted would mainly be an adjective :-) Terry

12 years, 3 months

Re: [Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

by Terry Jones

Hi Igor Check out http://jcalderone.livejournal.com/24285.html from JP Calderone. It does a good bit of what you want. It might be a little difficult to understand, so feel free to come back with questions. Also check the underlying Cooperator API docs at http://twistedmatrix.com/documents/current/api/twisted.internet.task.Cooper… Terry

12 years, 3 months

docFactory at Nevow Elements

by Juanjo Conti

I am using Nevow's LiveElements and I am gettings troubles with the docFactory attribute. In the examples I've seen, docFactory = loaders.stan(...) I'd like to load a string representing my html page[0] Can this be done? I really don't like stan :) Greets! [0] I generate the string using Django's templates systems' render_to_string function. -- Juanjo Conti blog: http://www.juanjoconti.com.ar

12 years, 3 months

Exceptions in twisted.web.client

by Chris Were

Hi, I have an application that fetches many urls, following any redirects. Every so often I get a variety of unhandled exceptions from within twisted.web.client (see snippet at the bottom of email). My code has an addErrback attached to the deferred and a try/except around the getPage() call, neither of which get invoked. Are these exceptions that web.client needs to catch internally and then call self.deferred.errback so that my error handler will receive it? If not, how can I catch them? Cheers, Chris --- Exception 1 --- 2009-10-06 20:21:16-0700 [HTTPPageGetter,client] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 463, in doRead return self.protocol.dataReceived(data) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 231, in dataReceived why = self.lineReceived(line) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 420, in lineReceived self.handleEndHeaders() File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 74, in handleEndHeaders m() File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 93, in handleStatus_301 _parse(url, defaultPort=self.transport.getPeer().port) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 426, in _parse port = int(port) exceptions.ValueError: invalid literal for int() with base 10: '' --- Exception 2 --- 2009-10-06 20:28:42-0700 [HTTPPageGetter,client] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py", line 463, in doRead return self.protocol.dataReceived(data) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py", line 231, in dataReceived why = self.lineReceived(line) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/http.py", line 420, in lineReceived self.handleEndHeaders() File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 74, in handleEndHeaders m() File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 93, in handleStatus_301 _parse(url, defaultPort=self.transport.getPeer().port) File "/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py", line 425, in _parse host, port = host.split(':') exceptions.ValueError: too many values to unpack --- Exception 3 --- 2009-10-06 20:31:11-0700 [HTTPPageGetter,client] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.6/site-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.6/site-packages/twisted/internet/tcp.py", line 463, in doRead return self.protocol.dataReceived(data) File "/usr/local/lib/python2.6/site-packages/twisted/protocols/basic.py", line 231, in dataReceived why = self.lineReceived(line) File "/usr/local/lib/python2.6/site-packages/twisted/web/http.py", line 404, in lineReceived status = l[1] exceptions.IndexError: list index out of range

12 years, 3 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

Twisted-web October 2009