
Itamar and I have just created a few new tickets and the "Web2-Gold-Master" milestone for the ascension of web2 to the status of The Web Server For Twisted. Obviously I'll also have to get in my car and cast the original twisted.web back into the volcano where I originally forged it, but I still visit my parents' house pretty regularly so I don't think that should be a problem. Details are here: http://twistedmatrix.com/trac/ticket/2085 Please read this description, and create / mark any tickets with the appropriate milestone if you believe they are a requirement for this objective. We may not agree, but we'd appreciate community involvement. In particular, this is a favorite topic of discussion for pundits. I have heard many times that our web server situation is why Twisted is not as popular as Linux, or Quake, or Ruby on Rails, or MVS, or BitchX, or whatever today's infrastructure flavor of the month is. I've had to shrug and admit fault in the past, but now that there is a concrete list of tasks for you all to help out with, I sincerely hope that some of you will choose to do so.

glyph@divmod.com wrote:
In particular, this is a favorite topic of discussion for pundits. I have heard many times that our web server situation is why Twisted is not as popular as Linux, or Quake, or Ruby on Rails, or MVS, or BitchX, or whatever today's infrastructure flavor of the month is. I've had to shrug and admit fault in the past, but now that there is a concrete list of tasks for you all to help out with, I sincerely hope that some of you will choose to do so.
We surely will. In the meantime, I have an urgent request: *Please fix the wrapping of your emails!* Many messages in the archives of mailing lists around Twisted and Divmod require constant horizontal scrolling, and that's very annoying. It's been this way for *years*, it shouldn't be that difficult to fix Quotient, or whatever is the cause of this. Just to be clear, I read your emails perfectly, wrapped and all, via Gmane and Thunderbird. The point are *the archives*. There's lots of valuable stuff in there, but: you do a Google search, find a message, and then are confronted with the pain of horizontal scrolling. There's a good chance many people will turn away without ever reading them. In many years of reading mailing lists and newsgroups, the Quotient-originated emails are *the only ones* exhibiting this problem: that's why I think it's you that should be fixing it. Sorry for ranting, this has been getting on my nerves for too long. :-) -- Nicola Larosa - http://www.tekNico.net/ Many software developers have become hostage to the development frameworks that they utilise. In turn, many frameworks have made session state a fundamental building block of web development because it permits sloppy design. -- Alan Dean, April 2006

On Wed, 13 Sep 2006 10:09:21 +0200, Nicola Larosa <nico@teknico.net> wrote:
glyph@divmod.com wrote:
In particular, this is a favorite topic of discussion for pundits. I have heard many times that our web server situation is why Twisted is not as popular as Linux, or Quake, or Ruby on Rails, or MVS, or BitchX, or whatever today's infrastructure flavor of the month is. I've had to shrug and admit fault in the past, but now that there is a concrete list of tasks for you all to help out with, I sincerely hope that some of you will choose to do so.
We surely will. In the meantime, I have an urgent request:
*Please fix the wrapping of your emails!*
I think you accidentally sent this message to the twisted-web mailing list, when you intended to send it to the mailman-dev mailing list. Jean-Paul

Nicola Larosa wrote:
*Please fix the wrapping of your emails!*
Jean-Paul Calderone wrote:
I think you accidentally sent this message to the twisted-web mailing list, when you intended to send it to the mailman-dev mailing list.
I definitely did not intend anything of the sort. How did you get that impression? And why should have I done so? Most known email-generating programs wrap the lines of message bodies, in keeping with RFC2822: 2.1.1. Line Length Limits There are two limits that this standard places on the number of characters in a line. Each line of characters MUST be no more than 998 characters, and SHOULD be no more than 78 characters, excluding the CRLF. Disregarding common practice is bad enough: disregarding the accessibility of useful information about your own work is even worse. I have difficulty understanding this stance; can you help me? -- Nicola Larosa - http://www.tekNico.net/ I think the first I ever heard that user session state was "bad" was from REST proponents, and that was fairly recently. As REST becomes better and more widely understood, I believe people are starting to think differently about how to architect a web application. I certainly am. -- Bill Venners, April 2006

On Wed, 13 Sep 2006 19:07:31 +0200, Nicola Larosa <nico@teknico.net> wrote:
Nicola Larosa wrote:
*Please fix the wrapping of your emails!*
Jean-Paul Calderone wrote:
I think you accidentally sent this message to the twisted-web mailing list, when you intended to send it to the mailman-dev mailing list.
I definitely did not intend anything of the sort. How did you get that impression?
And why should have I done so? Most known email-generating programs wrap the lines of message bodies, in keeping with RFC2822:
2.1.1. Line Length Limits
There are two limits that this standard places on the number of characters in a line. Each line of characters MUST be no more than 998 characters, and SHOULD be no more than 78 characters, excluding the CRLF.
Disregarding common practice is bad enough: disregarding the accessibility of useful information about your own work is even worse. I have difficulty understanding this stance; can you help me?
I don't have time to deal with the line-wrapping issues in pipermail. I can confidently say that neither does Glyph. If you'd like to do something about this, feel free - I'm not opposed to a solution, I'm just not going to provide one. Jean-Paul

On Wednesday 13 September 2006 09:11, Jean-Paul Calderone wrote:
I think you accidentally sent this message to the twisted-web mailing list, when you intended to send it to the mailman-dev mailing list.
79 columns; it's not the law, just a really good idea. Mike.

On 9/13/06, Nicola Larosa <nico@teknico.net> wrote:
We surely will. In the meantime, I have an urgent request:
*Please fix the wrapping of your emails!*
Many messages in the archives of mailing lists around Twisted and Divmod require constant horizontal scrolling, and that's very annoying. It's been this way for *years*, it shouldn't be that difficult to fix Quotient, or whatever is the cause of this.
Just to be clear, I read your emails perfectly, wrapped and all, via Gmane and Thunderbird. The point are *the archives*. There's lots of valuable stuff in there, but: you do a Google search, find a message, and then are confronted with the pain of horizontal scrolling. There's a good chance many people will turn away without ever reading them.
Perhaps your browser might be capable of wrapping the text for you? Opera can do it (press ctrl-F11) and there is a bookmarklet to do it in Firefox as well.

On Tue, 12 Sep 2006 21:21:16 -0400, glyph@divmod.com wrote:
Itamar and I have just created a few new tickets and the "Web2-Gold-Master" milestone for the ascension of web2 to the status of The Web Server For Twisted. Obviously I'll also have to get in my car and cast the original twisted.web back into the volcano where I originally forged it, but I still visit my parents' house pretty regularly so I don't think that should be a problem.
Details are here:
http://twistedmatrix.com/trac/ticket/2085
Please read this description, and create / mark any tickets with the appropriate milestone if you believe they are a requirement for this objective. We may not agree, but we'd appreciate community involvement.
In particular, this is a favorite topic of discussion for pundits. I have heard many times that our web server situation is why Twisted is not as popular as Linux, or Quake, or Ruby on Rails, or MVS, or BitchX, or whatever today's infrastructure flavor of the month is. I've had to shrug and admit fault in the past, but now that there is a concrete list of tasks for you all to help out with, I sincerely hope that some of you will choose to do so.
Lest anyone misunderstand, please take note: This does *not* mean that twisted.web is unmaintained, obsolete, or deprecated. You do not need to drop all of your code in favor of a BaseHTTPServer re-write. Bugs in twisted.web will continue to be fixed, and if twisted.web2 is ever finished it will provide a compatible API layer so that it can run your existing twisted.web-based applications. Jean-Paul

glyph@divmod.com wrote:
Itamar and I have just created a few new tickets and the "Web2-Gold-Master" milestone for the ascension of web2 to the status of The Web Server For Twisted. ... In particular, this is a favorite topic of discussion for pundits. I have heard many times that our web server situation is why Twisted is not as popular as Linux, or Quake, or Ruby on Rails, or MVS, or BitchX, or whatever today's infrastructure flavor of the month is.
Linux? Quake? MVS? BitchX? I thought this was about web. ;-) The web server is just a part, the matter is usually about web *frameworks*; if you want to compare something to RoR and others, that should be Nevow, not twisted.web2 . twisted.web(2) is a great web server that works well (albeit in a threaded environment) with most Python web frameworks, being complementary to them. I think the main reason Twisted is not popular as other Python web technologies is its event-based structure. As soon as one tries to interface other libraries with Twisted, one either uses threads, or wraps the library's API with Deferreds. The first solution is suboptimal for Twisted; the second is incompatible with already existing code that uses that library. This makes Twisted basically incompatible with all the code not written for it, if one wants to fully embrace its concurrency model. This kind of mismatch does not seem something that can be "fixed". I would however be very glad to be corrected, on this point. :-) -- Nicola Larosa - http://www.tekNico.net/ I think the first I ever heard that user session state was "bad" was from REST proponents, and that was fairly recently. As REST becomes better and more widely understood, I believe people are starting to think differently about how to architect a web application. I certainly am. -- Bill Venners, April 2006

I hadn't noticed until now that y'all have added a twisted.web2.client to trunk. (Copied from a web2-client4 branch; I guess the previous castle burned down, fell over, and then sank into the swamp. ;) How does the pipelining work? I ask because I've grown quite picky about the closing states of HTTP. I'm almost certainly preaching to the choir, but I'll quote RFC 2616 section 8.1.2.2: Clients which assume persistent connections and pipeline immediately after connection establishment SHOULD be prepared to retry their connection if the first pipelined attempt fails. If a client does such a retry, it MUST NOT pipeline before it knows the connection is persistent. Clients MUST also be prepared to resend their requests if the server closes the connection before sending all of the corresponding responses. and...umm, I can't find the RFCs, but some combination says that: (A) the kernel's TCP stack should immediately dump all its buffers and abort on a RST, so effectively the application gets a ECONNRESET sent back from the future (B) the sequence the HTTP server takes when receiving a request after a FIN causes a RST to be sent So when using pipelining, the client must be prepared for this case: 1. send request A 2. send request B 3. receive an ECONNRESET...and not even know if the webserver received or handled request A, much less what the response might have been For non-idempotent HTTP requests (a lot of SOAP calls), that ambiguity is not acceptable, so I need a way to toggle pipelining. Looks like that's this? p = HTTPClientProtocol() p.readPersistent = PERSIST_(NO_)?PIPELINE Second, if it is using pipelining, it has to do the retry. That part I don't see. Is the caller expected to do it? twisted/web2/client/ http.py has this: def connectionLost(self, reason): self.readPersistent = False self.setTimeout(None) self.manager.clientGone(self) # Tell all requests to abort. for request in self.inRequests: if request is not None: request.connectionLost(reason) Warm regards, Scott -- Scott Lamb <http://www.slamb.org/>

On Sep 13, 2006, at 2:00 PM, Scott Lamb wrote:
I hadn't noticed until now that y'all have added a twisted.web2.client to trunk. (Copied from a web2-client4 branch; I guess the previous castle burned down, fell over, and then sank into the swamp. ;)
<pipelining/retry/etc>
The client that's in there right now is a low-level http client. The higher level web client (the connection manager, queuing, pooling, etc.) is not completed. That is where the retry logic and the determination of which requests can be pipelined and which cannot would go. Please note that even *keepalive* can only be (reliably) used for idempotent actions. For example, client sends request A, gets response A, sends request B, but at the same time, server closes the connection from a timeout condition. The twisted.web2.client.HTTPClientProtocol.submitRequest has a keyword arg "closeAfter", which defaults to True. Thus, without you doing anything to change it, there is no keepalive and no pipelining. Each connection will only be usable for one request. Pipelining and keepalive features should really only be enabled when a request is submitted by a client manager which is able to handle the appropriate failure retry conditions. James

On Sep 14, 2006, at 11:36 AM, James Y Knight wrote:
The client that's in there right now is a low-level http client. The higher level web client (the connection manager, queuing, pooling, etc.) is not completed. That is where the retry logic and the determination of which requests can be pipelined and which cannot would go.
Okay, cool. So is twisted.web.client (and its dependencies) not going to be included in this deprecation, or is there going to be a temporary regression in functionality? I have no strong opinion; an HTTP/1.0 client wasn't that useful to me to begin with.
Please note that even *keepalive* can only be (reliably) used for idempotent actions. For example, client sends request A, gets response A, sends request B, but at the same time, server closes the connection from a timeout condition.
Ugh, you're right. Looking over RFC 2616 section 8.1.4, they say the same thing you do: "Non-idempotent methods or sequences MUST NOT be automatically retried, although user agents MAY offer a human operator the choice of retrying the request(s)." It does seem like this problem could have been avoided by the server (after gracefully closing) simply reading and discarding that request - the client would get the FIN and no RST. There's no circumstance in which a server handles a request and then cleanly closes the connection without a response, so it would know the server did not handle request B. It would have to retry, but it could do so safely even for non-idempotent requests. I wonder why they didn't do that...bandwidth? This could be a practical problem for me. I'm stuck using a protocol created by a wannabe standards body that has mandated (1) a non- idempotent sequence and (2) the client never closing the connection. (And no, this is not the first time they've contradicted an underlying standard...) -- Scott Lamb <http://www.slamb.org/>

On Sep 14, 2006, at 3:25 PM, Scott Lamb wrote:
Okay, cool. So is twisted.web.client (and its dependencies) not going to be included in this deprecation, or is there going to be a temporary regression in functionality? I have no strong opinion; an HTTP/1.0 client wasn't that useful to me to begin with.
twisted.web isn't deprecated, and won't be until web2 is an adequate replacement.
This could be a practical problem for me. I'm stuck using a protocol created by a wannabe standards body that has mandated (1) a non-idempotent sequence and (2) the client never closing the connection. (And no, this is not the first time they've contradicted an underlying standard...)
Can you assume the server doesn't close the connection on you except on errors? If so, that should be okay. I don't know which protocol you're talking about, but others like this I've seen which assign significance to a long-lived HTTP connection have also had the property that the connection doesn't get closed out from under the client after approx 30s of inactivity, like a normal HTTP server would. James

On Sep 14, 2006, at 3:03 PM, James Y Knight wrote:
On Sep 14, 2006, at 3:25 PM, Scott Lamb wrote:
Okay, cool. So is twisted.web.client (and its dependencies) not going to be included in this deprecation, or is there going to be a temporary regression in functionality? I have no strong opinion; an HTTP/1.0 client wasn't that useful to me to begin with.
twisted.web isn't deprecated, and won't be until web2 is an adequate replacement.
I thought the desire to deprecate it is what started this thread? I don't see any mention of web2.client in http://twistedmatrix.com/trac/ ticket/2085 or http://twistedmatrix.com/trac/query? status=new&status=assigned&status=reopened&milestone=Web2-Gold-Master.
This could be a practical problem for me. I'm stuck using a protocol created by a wannabe standards body that has mandated (1) a non-idempotent sequence and (2) the client never closing the connection. (And no, this is not the first time they've contradicted an underlying standard...)
Can you assume the server doesn't close the connection on you except on errors? If so, that should be okay.
No, I believe in the last version it was only the client that can't close it, and not a word about timeouts.
I don't know which protocol you're talking about, but others like this I've seen which assign significance to a long-lived HTTP connection have also had the property that the connection doesn't get closed out from under the client after approx 30s of inactivity, like a normal HTTP server would.
Most server implementations are using J2EE code on off-the-shelf webservers. I don't think they've touched the timeout settings at all... If you like staring at train wrecks, you'll love CWMP. Here's the original standard: http://www.dslforum.org/techwork/tr/TR-069.pdf I think the latest version isn't public yet. -- Scott Lamb <http://www.slamb.org/>

On Thu, 14 Sep 2006 15:36:27 -0700, Scott Lamb <slamb@slamb.org> wrote:
On Sep 14, 2006, at 3:03 PM, James Y Knight wrote:
On Sep 14, 2006, at 3:25 PM, Scott Lamb wrote:
Okay, cool. So is twisted.web.client (and its dependencies) not going to be included in this deprecation, or is there going to be a temporary regression in functionality? I have no strong opinion; an HTTP/1.0 client wasn't that useful to me to begin with.
twisted.web isn't deprecated, and won't be until web2 is an adequate replacement.
I thought the desire to deprecate it is what started this thread?
Note, however, that this isn't a contradiction of James' statement. See my post earlier in this thread. Jean-Paul

On Sep 14, 2006, at 11:36 AM, James Y Knight wrote:
The twisted.web2.client.HTTPClientProtocol.submitRequest has a keyword arg "closeAfter", which defaults to True. Thus, without you doing anything to change it, there is no keepalive and no pipelining. Each connection will only be usable for one request. Pipelining and keepalive features should really only be enabled when a request is submitted by a client manager which is able to handle the appropriate failure retry conditions.
Maybe this could be replaced with "idempotent" with an appropriate docstring? It'd accomplish the same thing, but it would then be obvious to people who haven't read a pile of RFCs under what circumstances they need it set in a particular way. With a "closeAfter" argument, it's tempting to say "of course I want closeAfter=False; I'm sending more stuff and one connection is faster" without realizing the consequences. If idempotency is what decides the behavior, then that's what the client should tell you. -- Scott Lamb <http://www.slamb.org/>

On Sep 14, 2006, at 3:33 PM, Scott Lamb wrote:
Maybe this could be replaced with "idempotent" with an appropriate docstring? It'd accomplish the same thing, but it would then be obvious to people who haven't read a pile of RFCs under what circumstances they need it set in a particular way. With a "closeAfter" argument, it's tempting to say "of course I want closeAfter=False; I'm sending more stuff and one connection is faster" without realizing the consequences. If idempotency is what decides the behavior, then that's what the client should tell you.
This is the low level http API. In the nonexistent high level client API, the user would not concern themselves with this: the connection pool manager would default to reusing connections when it's safe to do so, and not reusing connections when it's not safe. If you're using the low level API directly, you should never set it to False, regardless of whether your request is idempotent, unless you really know what you're doing (ie: you're basically writing a connection manager). James
participants (7)
-
glyph@divmod.com
-
James Y Knight
-
Jean-Paul Calderone
-
Mike Pelletier
-
Nicola Larosa
-
Pavel Pergamenshchik
-
Scott Lamb