Mailman 3 February 2006 - Twisted-web

web client & "TimeoutError: User timeout caused connection failure"
by Lenny G Arbage 11 Feb '06

11 Feb '06

I'm running into a problem under high client (not server) load. If I create a client process that makes only light requests to the server, everything works perfectly. If, however, I spawn a bunch of client requests simultaneously, using HTTPDownloader and HTTPClientFactory (or getPage and downloadPage, it doesn't seem to matter), I invariably get a bunch of "TimeoutError: User timeout caused connection failure" failures. These failures have no stack associated with them, so it isn't clear where they are originating, but it appears that they are due to timeouts on waiting for the connection setup, not on the response. I know that there is a lot of activity on redoing the client in web2, but I didn't see any mention of this particular problem (or potential ways to avoid it) in the list archives. Is this a known problem? Is there a workaround I can rely on until something better comes along in web2? The breakpoint (on my 2.8Ghz Pentium) seems to be around 40-50 simultaneous client connections. On my slower 2.53 Ghz machine, it is closer to 70-80. thanks, Lenny (I can produce a short example that reproduces the behavior if needed) __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

Web2 Script.py seems to be broken
by Todd Thomas 09 Feb '06

09 Feb '06

from twisted.application import strports, service from twisted.web2 import static, server, http, script root = static.File("./adir") root.indexNames=['index.rpy'] root.processors = {'.rpy': script.ResourceScript} application = service.Application("web") site = server.Site(root) s = strports.service('tcp:8000', http.HTTPFactory(site)) s.setServiceParent(application) Get error, noRsrc = error.ErrorPage(500, "Whoops! Internal Error", rpyNoResource) exceptions.AttributeError: 'module' object has no attribute 'ErrorPage' Failed to load application: 'module' object has no attribute 'ErrorPage' Looking in api docs for web2.error I don't see an ErrorPage, but it does exist in web.error Todd Thomas

4 14

making streams optional in web2
by glyph＠divmod.com 09 Feb '06

09 Feb '06

A few days ago, JP showed me some benchmarks of web2 vs. CherryPy hosting a trivial WSGI application. It turns out it's 4x slower, clocking in only ~250 req/sec on a middle-spec Opteron, as opposed to CherryPy's ~1000. From previous benchmarks, I happen to know that Apache can do ~3000 req/sec on that same hardware, making web2 well over 10x slower. I didn't look at the profile too closely, but preliminarily it seemed clear that the major bottleneck at this point was the creation and invocation of tons of Deferreds in _NotifyingProducerStream, while generating the response. Considering that streams aren't used at all in the actual HTTP protocol implementation (and thank goodness for that, given this discovery) it strikes me that the use of them in the response-generating API ought to be more optional, for cases where efficiency is important. Like, for example, DAV, which also seems to be making embarrassingly heavy use of the streams API - I haven't benchmarked that yet, but I'm sure it will be an adventure if I do :). I'm not sure I totally understand the use of streams here, anyway. I notice that Request purports to implement IConsumer, but I don't see any of IConsumer's methods actually present on Request. I'd prefer it if it actually did implement IConsumer and I could just write to that as an API. Finally, a minor nit; can we rename the various "http.py"s? I'd rather a naming convention than subdirectories anyway, http_channel.py rather than channel/http. Right now we have web2/http.py, web2/channel/http.py, and web2/dav/http.py. I can only see this problem getting worse.

4 5

sprinting at PyCon
by Duncan McGreggor 09 Feb '06

09 Feb '06

From an email exchange with dreid: > From: David Reid <dreid(a)dreid.org> > Date: February 9, 2006 1:04:30 PM MST > To: Duncan McGreggor <duncan.mcgreggor(a)gmail.com> > Subject: Re: t.w.xmlrpc Basic Auth patch > >>> Help is always great, unfortunately the web2 client isn't _done_ yet. >>> It has pretty decent test coverage and works fairly well but doesn't >>> have any high level api. I'm not sure when that stuff will get done. >>> If you're interested in the XML-RPC server side that'd be great. :) >> >> Awesome. Perfect for sprinting at pycon. > > I don't think foom or I are going to be there. Perhaps some benefit > might come of organizing a virtual sprint to coincide with pycon's > sprints. :( I will be at pycon this year. I had planned on sprinting with some of my own projects, but I can do that anytime. I'd much rather do some group work, even if it's virtual sprinting with those that can't be there. Is there any interest in a sprint/vsprint? In particular, I want to help with t.w2 xmlrpc (server, maybe some client work). Any thoughts? What about #twisted.pycon or some such? d

1 0

Re: [Twisted-Python] Re: [Twisted-commits] r15860 - Move checkPreconditions() from web2.dav.static.DAVFile to web2.resource.RenderMixIn.
by James Y Knight 09 Feb '06

09 Feb '06

Sorry, I meant to send this to twisted-web instead. Plz make any reply there, instead. James On Feb 9, 2006, at 1:34 PM, James Y Knight wrote: > On Feb 8, 2006, at 7:49 PM, Wilfredo Sanchez wrote: >> Move checkPreconditions() from web2.dav.static.DAVFile to >> web2.resource.RenderMixIn. >> Add test suite (ported foom's tests for server.checkpreconditions()). >> Fix some bugs found by the tests. (Holy cow, checkpreconditions >> are a PITA.) > > I don't like this part. You've made the precondition check code > much longer, IMO harder to understand (figuring out how the return > values of one method influenced the rest of the execution is not > easy), and pinned it to being part of Resource, rather than being > reusable. I'm not sure of the need, given that the previous > implementation was callable from Resource as it was. Why not just > use it... > > Nextly, I'm somewhat concerned about the change to calling the > precondition check before GET rendering instead of afterwards: I > think your use cases are probably not typical. I believe most > people will write resources which generate dynamic content, and if > they're _really good_ about it, they'll simultaneously generate an > etag and a last-modified time. This is also the only way that such > information can come out of wsgi, cgi, proxies to other servers, etc. > > The way things were set up before made Not Modified responses just > "magically" show up correctly when you set those headers in your > response. No more interface was required than that, which means > that the response could come from anywhere. Now, instead, you force > the author to implement resource.etag() and resource.lastModified() > in order to get that behavior. It's really easy for file-like > resources to do so, but not so much so for most other kinds. > > And the thing is, it's OK to check afterwards for the kinds of > requests a web browser makes: GET/HEAD/POST. GET/HEAD don't change > things so it's acceptable to do their check at any time, and while > POST does change things, as a practical matter nobody actually > sends conditionals on post requests, because what would it be > useful for? (and nothing would be set up to handle such a thing). > > However, I'm currently ignoring conditionals on PostableResource's > http_POST, which while okay as far as real-world interoperability > goes, it is certainly not _right_. And to make things worse, the > post-processing filter doesn't confine itself to checking only GET/ > HEAD requests, so the client will get back a precondition failed > message, even though the method was executed. At the very least a > POST with an If-Unmodified-Since/If-Match should return > precondition failed. If there was code on Resource to handle that > automatically, I could not have forgetten to do it. > > So, um, I dunno what to conclude. :) > > James > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >

1 0

How do I add a doctype using stan?
by Michael M 09 Feb '06

09 Feb '06

Okay, I have searched but have not found an answer and I am sure it is obvious. I am using nevow tags and want to have the doctype so that validation works. I am not using xmlfile or xmlstr. I tried to do the following. docFactory = stan.loaders( tags.xml('DOCTYPE etc etc'), tags.html[ tags.head[tags.title['Help']], tags.body['How do I add a doctype'] ] ) and of course it failed. Any help appreciated. Thanks Michael

2 4

twistd syslog import error
by Gustavo Rahal 09 Feb '06

09 Feb '06

Hi All twistd -y app.py Traceback (most recent call last): File "C:\Python24\Lib\site-packages\twisted\scripts\twistd.py", line 6, in ? from twisted.python import log, syslog File "C:\Python24\lib\site-packages\twisted\python\syslog.py", line 5, in ? syslog = __import__('syslog') ImportError: No module named syslog The module is there! I'm running on windows 2000. I tried to adjust PATH and PYTHONPATH (I don't know if pythonpath applies for windows) and still didn't work. Any ideas? Gustavo

3 2

xmlrpc server daemon
by Gustavo Rahal 08 Feb '06

08 Feb '06

Hi How do I make my xmlrpc server a daemon? The code that starts the server is: r = rmsWebServices() xmlrpc.addIntrospection(r) reactor.listenTCP(7080, server.Site(r)) reactor.run() What should I change? Thanks Gustavo

3 3

debian sarge, twistd: command not found
by Arp 08 Feb '06

08 Feb '06

Hi All, I'm very new to all this - the problem is probably a no-brainer - please be kind ;) In the proces of trying to get a MoinMoin wiki running on a Debian Sarge server I used aptitude to install python2.3, moinmoin (1.3.4) and twisted (as listed below) - I know there are newer versions of all these around, but this what seems the best what the system config offers me through aptitude in terms of dependencies. The problem is after installing all this python and moin seem to run fine, but when I try to invoke mointwisted the script complains that sh can't find twistd I've searched the filesystem with "find / -name twistd" but that comes up empty too so it's probably not just a missing path setting ... So what am I missing to get twisted installed _including_ twistd ===================================== List of "twisted" packages installed by aptitude: ---------------------- i A python-twisted 2.1.0-4 2.1.0-4 i A python-twisted-conch 1:0.6.0-1 1:0.6.0-1 i python2.3-twisted 2.1.0-4 2.1.0-4 i A python2.3-twisted-bin 2.1.0-4 2.1.0-4 i A python2.3-twisted-conch 1:0.6.0-1 1:0.6.0-1 i python-twisted-core 2.1.0-4 2.1.0-4 i python-twisted-lore 0.1.0-2 0.1.0-2 i python-twisted-mail 0.2.0-2 0.2.0-2 i A python-twisted-names 0.2.0-1 0.2.0-1 i A python-twisted-news 0.1.0-1 0.1.0-1 i A python-twisted-runner 0.1.0-2 0.1.0-2 i A python-twisted-web 0.5.0-2 0.5.0-2 i python-twisted-web2 0.1.0-1 0.1.0-1 i A python-twisted-words 0.3.0-1 0.3.0-1 .... i A python2.3-twisted-core 2.1.0-4 2.1.0-4 i A python2.3-twisted-lore 0.1.0-1 0.1.0-2 i A python2.3-twisted-mail 0.2.0-1 0.2.0-2 i A python2.3-twisted-names 0.2.0-1 0.2.0-1 i A python2.3-twisted-news 0.1.0-1 0.1.0-1 i A python2.3-twisted-runner 0.1.0-1 0.1.0-2 i A python2.3-twisted-web 0.5.0-1 0.5.0-2 i A python2.3-twisted-web2 0.1.0-1 0.1.0-1 i A python2.3-twisted-words 0.3.0-1 0.3.0-1 ========================= Thanks in advance for any pointers ! -- Arp

3 3

Problems uploading data in web2
by Henrik Thostrup Jensen 07 Feb '06

07 Feb '06

Hi I'm trying to use web2 to stream uploading of data, but I ran into some problems. First I tried building from the example in http://twistedmatrix.com/projects/web2/documentation/examples/demo.html, however args and files objects in the request did not show anything. However the length of stream was correct, and using tcpdump, showed the correct value. So I tried building my own uploader which look like this: class Uploader(resource.PostableResource): def http_POST(self, ctx): request = iweb.IRequest(ctx) self.parse_stream(request.stream) @defer.deferredGenerator def parse_stream(self, stream): from twisted.web2 import fileupload s = fileupload.parse_urlencoded_stream(stream) while True: datas = s.read() datas = fileupload.wait(datas) yield datas datas = datas.getResult() Using the 0.1.0 release of web2 didn't get past the yield line. Using latest from subversion (revision 15836), got past, but datas was None, even though data had been send. I've used both cURL (using the -d switch) and wget (using the --post-file switch) for uploading data, so I do not think the client as malfunctioning. What am I doing wrong? I'm aware that web2 is under development and all that, but I need streaming upload, so the web module is not an option. -- - Henrik

2 2