[Web-SIG] PEP 444 / WSGI 2 Async

[Alice Bevan–McGregor]

Alex Grönholm and I have been discussing async implementation details 
(and other areas of PEP 444) for some time on IRC.  Below is the 
cleaned up log transcriptions with additional notes where needed.

Note: The logs are in mixed chronological order — discussion of one 
topic is chronological, potentially spread across days, but separate 
topics may jump around a bit in time.  Because of this I have 
eliminated the timestamps as they add nothing to the discussion.  
Dialogue in square brackets indicates text added after-the-fact for 
clarity.  Topics are separated by three hyphens.  Backslashes indicate 
joined lines.

This should give a fairly comprehensive explanation of the rationale 
behind some decisions in the rewrite; a version of these conversations 
(in narrative style vs. discussion) will be added to the rewrite Real 
Soon Now™ under the Rationale section.

	— Alice.


--- General

agronholm: my greatest fear is that a standard is adopted that does not 
solve existing problems

GothAlice: [Are] there any guarantees as to which thread / process a 
callback [from the future instance] will be executed in?



--- 444 vs. 3333

agronholm: what new features does pep 444 propose to add to pep 3333? \ 
async, filters, no buffering?

GothAlice: Async, filters, no server-level buffering, native string 
usage, the definition of "byte string" as "the format returned by 
socket read" (which, on Java, is unicode!), and the allowance for 
returned data to be Latin1 Unicode. \ All of this together will allow a 
'''def hello(environ): return "200 OK", [], ["Hello world!"]''' example 
application to work across Python versions without modification (or use 
of b"" prefix)

agronholm: why the special casing for latin1 btw? is that an http thing?

GothAlice: Latin1 = \u0000 → \u00FF — it's one of the only formats that 
can be decoded while preserving raw bytes, and if another encoding is 
needed, transcode safely. \ Effectively requiring Latin1 for unicode 
output ensures single byte conformance on the data. \ If an application 
needs to return UTF-8, for example, it can return an encoded UTF-8 
bytestream, which will be passed right through,



--- Filters

agronholm: regarding middleware, you did have a point there -- 
exception handling would be pretty difficult with ingress/egress filters

GothAlice: Yup.  It's pretty much a do or die scenario in filter-land.

agronholm: but if we're not ditching middleware, I wonder about the 
overall benefits of filtering \ it surely complicates the scenario so 
it'd better be worth it \ I don't so much agree with your reasoning 
that [middleware] complicates debugging \ I don't see any obvious 
performance improvements either (over middleware)

GothAlice: Simplified debugging of your application w/ reduced stack to 
sort through, reduced nested stack overhead (memory allocation 
improvement), clearer separation of tasks (egress compression is a good 
example).  This follows several of the Zen of Python guidelines: \ 
Simple is better than complex. \ Flat is better than nested. \ There 
should be one-- and preferably only one --obvious way to do it. \ If 
the implementation is hard to explain, it's a bad idea. \ If the 
implementation is easy to explain, it may be a good idea.

agronholm: I would think that whatever memory the stack elements 
consume is peanuts compared to the rest of the application \ 
ingress/egress isn't exactly simpler than middleware

GothAlice: The implementation for ingress/egress filters is two lines 
each: a for loop and a call to the elements iterated over.  Can't get 
much simpler or easier to explain.  ;) \ Middleware is pretty complex… 
\ The majority of ingress filters won't have to examine wsgi.input, and 
supporting async on egress would be relatively easy for the filters 
(pass-through non-bytes data in body_iter). \ If you look at a system 
that offers input filtering, output filtering, and decorators 
(middleware), modifying input should "obviously" be an input filter, 
and vice-versa.

agronholm: how does a server invoke the ingress filters \ in my 
opinion, both ingress and egress filters should essentially be pipes \ 
compression filters are a good example of this \ once a block of 
request data (body) comes through from the client, it should be sent 
through the filter chain

agronholm: consider an application that receives a huge gzip encoded 
upload \ the decompression filter decompresses as much as it can using 
the incoming data \ the application only gets the next block once the 
decompression filter has enough raw data to decompress

GothAlice: Ingress decompression, for example, would accept the environ 
argument, detect gzip content-encoding, then decompress the wsgi.input 
into its own buffer, and finally replace wsgi.input in the environ with 
its decompressed version. \ Alternatively, it could decompress chunks 
and have a more intelligent replacement for wsgi.input (to delay 
decompression until it is needed).

agronholm: are you saying that the filter should decompress all of the 
data at once? how would this work with async?

GothAlice: The first example is the easiest to implement, but you are 
correct in that it would buffer all the data up-front.  The second I 
described (intelligent wsgi.input replacement) would work in an async 
application environment. (But would be harder to code and unit-test.)

agronholm: I don't really see how it would work

GothAlice: environ = parse_headers() ; decompression_filter(environ)

agronholm: wouldn't it be simpler to just have ingress filters return 
the data chunk, altered or not?

GothAlice: decompression_filter(environ): if 
environ.get('HTTP_TRANSFER_ENCODING', None) == 'gzip': 
environ['wsgi.input'] = StreamDecompression(environ['wsgi.input'])

agronholm: I'm not very comfortable with the idea of wsgi.input in 
async apps \ I'm just thinking what would happen when you do 
environ['wsgi.input'].read()

GothAlice: One of two things: in a sync environment, it blocks until it 
can read, in an async environment [combined with yield] it 
pauses/shelves your application until the data is available.

agronholm: I'd rather do away with wsgi.input altogether, but I haven't 
yet figured out how the application would read the entire request body 
then

agronholm: it should be fairly easy to write a helper function for that though

GothAlice: Returning the internal socket representation would improve 
some things, and make things generally worse. :/

agronholm: returning socket from what?

GothAlice: In Tornado's HTTP server, you read and write directly 
from/to the IOStream. \ wsgi.input, though, is more abstracted

agronholm: argh, I can't think of a way to make this work beautifully

GothAlice: Yeah.  :(

agronholm: the requirements of async apps are a big problem

agronholm: returning magic values from the app sounds like a bad idea

agronholm: the best solution I can come up with is to have 
wsgi.async_input or something, which returns an async token for any 
given read operation


agronholm: most filters only deal with the headers \ so what if we made 
it so that the filter chain is only accessed once, and filters that 
need to modify the body as well would return a generator \ and when the 
server receives more data, it would feed it to the first generator in 
the chain, feed the results from that to the next etc.

agronholm: the generators could also return futures, at which point the 
server adjourns processing of the chain until the callback fires \ in 
multithreaded mode, the server would simply call .result() which would 
block, and in single threaded mode, add a callback to the reactor

GothAlice: Hmm.

agronholm: the ingress filters' return values would affect what is sent 
to the application

agronholm: [I'm] trying to solve the inherent difficulties with having 
a file-like object in the environ \ my solution would allow them to 
work transparently with sync and async apps alike

GothAlice: Hmm.  What would the argspec of an ingress filter be, then?  
(The returned value, via yield, being wsgi.input chunks.)

agronholm: probably environ, body_generator or something

agronholm: the beauty in wsgi in general is of course that it requires 
no importing of predefined functions or anything \ so there should be 
some way for the application to read the entire request at once

GothAlice: I think combining wsgi.async with specific attributes on 
wsgi.input which can be yielded as async tokens might be a way to go.


GothAlice: agronholm: yielding None from the application being a polite 
way to re-schedule the application after a reactor cycle to give other 
connections a chance before doing something potentially blocking.

agronholm: I thought None meant "I'm done here" \ otoh, the app has to 
return some response

GothAlice: That's yielding an application response tuple followed by 
StopIteration. \ (Not necessarily immediately returning StopIteration 
after yielding the response; there may be clean-up to do; which is a 
nice addition.)

GothAlice: Three options: yield None (reschedule to be nice/cooperative 
behaviour), yield status, headers, body (deliver a response), and yield 
AsyncToken.

agronholm: so what would the application yield if it wanted to generate 
the body in chunks? (potentially a slow process)

GothAlice: A body_iter that generates the body in chunks, as per a 
standard (non-generator) application callable. \ That wouldn't change. 
\ But often an application would want to async stream the response body 
in before starting body generation.

GothAlice: An application MUST be a callable returning (status_bytes, 
header_list, body_iter) OR a generator.  IF the application is a 
generator, it MUST yield EITHER None (delay execution), a 
(status_bytes, header_list, body_iter) tuple, or an async token.  After 
yielding a response the application generator MAY perform additional 
actions before raising StopIteration, but MUST NOT yield anything but 
None or async tokens from that point onward.

agronholm: one of my concerns is how a request body modifying 
middleware will work with async apps unless it's specifically designed 
with those in mind \ you suggested that such middleware replace 
wsgi.input with their own

GothAlice: It would have to be; or it could simply yield through 
non-bytes chunks, returning the result of the yield back up (which may 
be ignored).

agronholm: what guarantee is there that the replacement has 
.async_read() unless the filter was specifically designed to be async 
aware?

GothAlice: Or, if the developer was in a particularly black mood, the 
middleware could re-set wsgi.async to be false.  ;)

agronholm: I don't quite understand the meaning or point of wsgi.async

GothAlice: wsgi.async is a boolean representing the capability of the 
underlying server to accept async tokens.

agronholm: why would that ever be false? \ in a blocking/threaded 
server, implementing support for that is trivial

GothAlice: Why does no HTTP server in Python conform to the HTTP/1.1 
spec properly?  Lazy developers.  ;)  [And lack of interest 
down-stream.  Calling server authors idiots was not my intention.]

agronholm: they could just as well forgo setting wsgi.async altogether

GothAlice: environ.get('wsgi.async', False) is the only way to armor 
against that, I guess.

agronholm: well I think we're talking about *conforming* servers here \ 
there's not much that can be done about incomplete implementations

GothAlice: However, if wsgi.async is going to be in the WSGI2 spec, 
it'll be required.  if the server hasn't gotten around to implementing 
async yet, it should be False.

agronholm: I think wsgi.async is useless \ "hasn't gotten around to"? 
that's not a lot of work, really \
that flag just paves way for half assed implementations

GothAlice: Still, some method to detect the capability should be 
present.  Something more than attempting to access wsgi.input's 
async_read attribute and catching the AttributeError exception.

agronholm: the capability should be *required* \ given how easy it is 
to implement \ I don't see any justification not to require it

GothAlice: We'll have to see how easy it is to add to m.s.http before 
I'll admit it's "easy" in the general sense.  ;)  If it turns out to be 
simple (and not-too-badly-performance-impacting) I'll make it required.

agronholm: fair enough


agronholm: robert pointed out the difficulty of executing them in the 
right order

GothAlice: Indeed; this problem exists with the current middleware system, too.

agronholm: it'd probably be easier to specify them as a list in the 
deployment descriptor

GothAlice: (My note about appending to ingress_filters, prepending to 
egress_filters to simulate middleware behaviour is functional, though 
non-optimal; the filters, if co-dependant, should be middleware 
instead.)

agronholm: webcore's current middleware system is too much magic imho

GothAlice: I agree. \ A init.d-style ordering system would have to be 
its own PEP.

agronholm: also, I was thinking if we could filters that needed both 
ingress/egress capabilities (such as session middleware) in a way that 
only required specifying it once

GothAlice: … wouldn't that be middleware?  ;) \ Thus far I've defined 
ingress and egress filters as distinct and separate, with 
dual-functionality requirements being fulfilled by middleware.
agronholm: we could probably simplify that

GothAlice: "There should be one, and preferably only one, right way to 
do something."  ;)

agronholm: yes, and that is the point of my idea :)

GothAlice: Replacing middleware isn't a small task; the power of 
potentially redirecting application flow (amongst other gems the 
middleware structure brings to the table) would be very difficult to 
model cleanly when separated into ingress/egress.

agronholm: btw, I very much agreed with PJE's suggestion of making 
filtering its own middleware instead of a static part of the interface

GothAlice: The problem with not mentioning filtering in the PEP is that 
middleware authors wont take it into consideration when coding. (That's 
why it's optional for servers to implement and includes an example 
middleware implementation of the API.)



--- Async

agronholm: +1 for async wsgi using the new concurrent.futures stdlib feature


agronholm: I still don't like the idea of wsgi.executor \ imho that 
should be left up to the application or framework \ not the web server 
\ and I still disapprove of the wsgi.async flag

GothAlice: The server does, however, need to be able to capture async 
read requests across environ['wsgi.input'].async_read*

GothAlice: What would the semantics be for a worker on a 
single-threaded async server to wait for a long-running task?  Was my 
code example (the simplified try/except block) inadequate?

agronholm: if the app needs to do heavy lifting, it delegates the task 
to a thread/process pool, which returns a future, which the app yields 
back \ when the callback is activated, the reactor will resume 
execution of that app \ I think you pretty much got it right in your 
revised example code

GothAlice: Just replace environ['wsgi.executor'] with an 
application-specific one?

agronholm: essentially, yes \ that would greatly simplify the 
implementation of the interface

GothAlice: And it is all done via done_callbacks… hmm.  For the 
purposes of the callbacks, though, exceptions are ignored.  :/

agronholm: what is your concern with this specifically?

GothAlice: That my desired syntax (try/except around a value=yield 
future) won't be able to capture and elevate exceptions back to the 
WSGI application.

agronholm: oh, that is not a problem since the reactor will call 
.result() on it anyway and send any exceptions back to the application

GothAlice: Back to the environment issue for a moment: not providing an 
executor in the environment means middleware will not be able to 
utilize async features without having their own executor in addition to 
the application's.  How about I explicitly require that servers allow 
overriding of the executor used? \ How often would an application want 
to utilize multiple executors at once?

agronholm: the middleware could have a constructor argument for passing 
an executor

GothAlice: That would then require passing an executor to multiple 
layers of middleware, creating a number of additional references and 
local variables, vs. configuring a "default executor" at the server 
level.

agronholm: there are pros and cons with the wsgi.executor approach

GothAlice: There would be no requirement for the application to use 
wsgi.executor; if an application has a default threaded executor 
(wsgi.executor), it can use a multi-process one for specific jobs 
[ignoring the one in the env] without too much worry.

agronholm: essentially wsgi.executor would be a convenience then

GothAlice: Exactly. \ (And mostly a helper to middleware so they don't 
each need explicit configuration or management of their own executors.)



--- Optional Components

GothAlice: I think full HTTP/1.1 conformance should be a requirement 
for WSGI2 servers, too.  (chunked requests, not just chunked responses) 
\ Because there's really no point in writing a -new- HTTP/1.0 server.  
;)

agronholm: indeed

GothAlice: One thing I've been grappling [while] rewriting PEP 444 is 
that pretty much everything marked 'optional' or 'may' in WSGI 1 / PEP 
333 no developer actually gets around to implementing.  Thus making 
HTTP/1.1 support non-optional [in PEP 444].

GothAlice: Something I've noticed with Python HTTP code: none of it is 
complete, and all of the servers that report HTTP/1.1 compliance 
straight up lie.  Zero I found support chunked response bodies, and 
zero support chunked requests (which is required by HTTP/1.1). \ (The 
servers I looked at universally had embedded comments along the lines 
of: "Chunked responses are left up to application developers.")

GothAlice: If it's too demanding [or appears too daunting], a "may 
implement" feature becomes a "never will be implemented" feature.

agronholm: I would prefer requiring HTTP/1.1 support from all WSGI2 servers

GothAlice: I mean, if I can do it in 172 Python opcodes, I'm certain it 
can't be -that- hard to implement.  ;)