From graham.dumpleton at gmail.com  Sun Jan  2 02:02:51 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sun, 2 Jan 2011 12:02:51 +1100
Subject: [Web-SIG] PEP 444 != WSGI 2.0
Message-ID: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>

Can we please clear up a matter.

GothAlice (don't know off hand there real name), keeps going around
and claiming:

"""
After some discussion on the Web-SIG mailing list, PEP 444 is now
"officially" WSGI 2, and PEP 3333 is WSGI 1.1
"""

In this instance on web.py forum on Google Groups.

I have pointed out a couple of times to them that there is no way that
PEP 444 has been blessed as being the official WSGI 2.0 but they are
not listening and are still repeating this claim. They can't also get
right that PEP 3333 clearly says it is still WSGI 1.0 and not WSGI
1.1.

If the people here who's opinion matters are quite happy for GothAlice
to hijack the WSGI 2.0 moniker for PEP 444 I will shut up. But if that
happens, I will voice my objections by simply not having anything to
do with WSGI 2.0 any more.

Graham

From graham.dumpleton at gmail.com  Sun Jan  2 02:28:07 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sun, 2 Jan 2011 12:28:07 +1100
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTinVHQQLuypuAEphsMwNu9Fc+8gPbvu_bAs6Q7Om@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTinVHQQLuypuAEphsMwNu9Fc+8gPbvu_bAs6Q7Om@mail.gmail.com>
Message-ID: <AANLkTikj7LE6_1DENFBNmyKKrOs+o_5_wJiWxMTozwsq@mail.gmail.com>

On 2 January 2011 12:09, Jonas Galvez <jonasgalvez at gmail.com> wrote:
> Graham Dumpleton wrote:
>> If the people here who's opinion matters are quite happy for GothAlice
>> to hijack the WSGI 2.0 moniker for PEP 444 I will shut up. But if that
>> happens, I will voice my objections by simply not having anything to
>> do with WSGI 2.0 any more.
>
> Hi Graham, I'm interested in learning what is your motivation for
> objecting that PEP 444 be WSGI 2.0. I'm assuming you must have already
> voiced your criticism on a technical level somewhere by this point.
> Can you point me to it?

Because for all we know right now what will be WSGI 2.0 may look a lot
different to what PEP 444 is now. Already they have taken the original
PEP 444 that was put out by Chris, and which had never actually been
updated based on feedback on the Python WEB-SIG list to address
perceived shortcomings, and started injecting his own ideas on top of
it without any real consultation with those on the WEB-SIG who have
had a lot of experience with all this stuff.

Thus what he is working on is a very fluid specification that keeps
changing. Ie., it is thus a work in progress, yet the way they talk
about it is if it already is the official WSGI 2.0 specification when
it is still no more than a bunch of ideas of what could be done. I am
thus manly objecting at this point on a matter of process and how they
are portraying what PEP 444 is. The PEP 444 by rights should have been
completely withdrawn and marked as rejected. If they want to carry on
and take PEP 444 and turn it into something else, then give it some
other working name, but where it still isn't labelled as WSGI 2.0.
When they have fleshed it out sufficiently and it has passed review on
the Python WEB-SIG that is fine and then gets put up as a PEP with
some blessing, only then should it notionally be anointed as WSGI 2.0
if the community wants that. Don't do this and all you do is cause
ongoing confusion in the community as to what WSGI 2.0 is given that
the definition of what it may be keeps changing.

I also have a various technical issues with the original WSGI
specification and they aren't being addressed in PEP 444 from what I
have seen so far, as well as having issues with new things in PEP 444.
I have blogged and posted on the WEB-SIG list about a number of them
and am now starting to get back into documenting what some of those
other issues are. Overall though, I believe a big step needs to be
taken back and fresh look at this stuff needs to be made. It needs to
be cast into the greater context of how we deploy this stuff as well,
otherwise deployment is going to continue to be a PITA with all the
systems using different ways when there could be a better level of
compatibility across them all to make deployment easier.

Graham

From ianb at colorstudy.com  Sun Jan  2 05:34:40 2011
From: ianb at colorstudy.com (Ian Bicking)
Date: Sat, 1 Jan 2011 23:34:40 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
Message-ID: <AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>

Until the PEP is approved, it's just a suggestion.  So for it to "really" be
WSGI 2 it will have to go through at least some approval process; which is
kind of ad hoc, but not so ad hoc as just to implicitly happen.  For WSGI 2
to happen, someone has to write something up and propose it.  Alice has
agreed to do that, working from PEP 444 which several other people have
participated in.  Calling it "WSGI 2" instead of "Web 3" was brought up on
this list, and the general consensus seemed to be that it made sense -- some
people felt a little funny about it, but ultimately it seemed to be
something everyone was okay with (with some people like myself feeling
strongly it should be "WSGI 2").

I'm not sure why you are so stressed out about this?  If you think it's
really an issue, perhaps 2 could be replaced with "2alpha" until such time
as it is approved?


On Sat, Jan 1, 2011 at 8:02 PM, Graham Dumpleton <graham.dumpleton at gmail.com
> wrote:

> Can we please clear up a matter.
>
> GothAlice (don't know off hand there real name), keeps going around
> and claiming:
>
> """
> After some discussion on the Web-SIG mailing list, PEP 444 is now
> "officially" WSGI 2, and PEP 3333 is WSGI 1.1
> """
>
> In this instance on web.py forum on Google Groups.
>
> I have pointed out a couple of times to them that there is no way that
> PEP 444 has been blessed as being the official WSGI 2.0 but they are
> not listening and are still repeating this claim. They can't also get
> right that PEP 3333 clearly says it is still WSGI 1.0 and not WSGI
> 1.1.
>
> If the people here who's opinion matters are quite happy for GothAlice
> to hijack the WSGI 2.0 moniker for PEP 444 I will shut up. But if that
> happens, I will voice my objections by simply not having anything to
> do with WSGI 2.0 any more.
>
> Graham
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/ianb%40colorstudy.com
>



-- 
Ian Bicking  |  http://blog.ianbicking.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110101/2e9306d6/attachment.html>

From guido at python.org  Sun Jan  2 06:28:28 2011
From: guido at python.org (Guido van Rossum)
Date: Sat, 1 Jan 2011 21:28:28 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
Message-ID: <AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>

On Sat, Jan 1, 2011 at 5:02 PM, Graham Dumpleton
<graham.dumpleton at gmail.com> wrote:
> Can we please clear up a matter.
>
> GothAlice (don't know off hand there real name), keeps going around
> and claiming:
>
> """
> After some discussion on the Web-SIG mailing list, PEP 444 is now
> "officially" WSGI 2, and PEP 3333 is WSGI 1.1
> """
[...]

>From past posts here, that's Alice Bevan?McGregor
<alice at gothcandy.com>, added to the thread.

On Sat, Jan 1, 2011 at 8:34 PM, Ian Bicking <ianb at colorstudy.com> wrote:
> Until the PEP is approved, it's just a suggestion.? So for it to "really" be
> WSGI 2 it will have to go through at least some approval process; which is
> kind of ad hoc, but not so ad hoc as just to implicitly happen.? For WSGI 2
> to happen, someone has to write something up and propose it.? Alice has
> agreed to do that, working from PEP 444 which several other people have
> participated in.? Calling it "WSGI 2" instead of "Web 3" was brought up on
> this list, and the general consensus seemed to be that it made sense -- some
> people felt a little funny about it, but ultimately it seemed to be
> something everyone was okay with (with some people like myself feeling
> strongly it should be "WSGI 2").
>
> I'm not sure why you are so stressed out about this?? If you think it's
> really an issue, perhaps 2 could be replaced with "2alpha" until such time
> as it is approved?

I'm guessing that Graham is concerned that Alice's assertion implies
that the PEP is approved. IOW that the *future* WSGI 2.0 is equal to
the *current* PEP 444. While we don't know for sure, this is likely
wrong (at least in some details).

OTOH I agree with Ian that it seems correct to say that PEP 444 (which
is still under development) is striving to arrive at a consensus for
what will be named WSGI 2.0. This is not unusual in the world of
standards -- future standards usually are given names and document IDs
long before there is agreement on the contents of the standard. In
this sense Alice's use of "officially" is not incorrect, although out
of context it could be misunderstood to imply PEP approval. I would
recommend adding caution about this whenever the equivalency between
PEP 444 and WSGI 2.0 is mentioned -- perhaps it is enough to state
that "PEP 444 is the draft for WSGI 2.0".

Often people or companies draw premature conclusions from draft
standards and prepare implementations that comply with the draft
standard in the hope that the draft won't change before it is set in
stone, and sometimes such implementations are incorrectly billed as
compliant with the standard (rather than with a specific version of
the draft). I don't know if that is what Alice is doing -- an equally
likely theory is that she's just excited.

I haven't followed the development of PEP 4444 much, so I can't
comment on how much agreement there is on the draft; Ian's use of
"alpha" suggests that there's some way to go still.

One clearly factual error in Alice's (quoted) post: PEP 3333 is WSGI
1.0.1, not WSGI 1.1. AFAIK there's no such thing as WSGI 1.1 now.

Alice, since you have in the past posted here suggesting you are
interested in carrying PEP 444 / WSGI 2.0 forward, please acknowledge
that you understand the concerns raised in this thread.

Graham, I suggest that you don't worry about this issue, and instead
focus on helping the draft turn into a standard by providing feedback
on PEP 444. Unless there's part of the story you're not telling here.

-- 
--Guido van Rossum (python.org/~guido)

From graham.dumpleton at gmail.com  Sun Jan  2 06:33:09 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sun, 2 Jan 2011 16:33:09 +1100
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
Message-ID: <AANLkTi=i9KgWAgrxcqS7tnW4cryiJUhBv1YiiaQb=99d@mail.gmail.com>

On 2 January 2011 15:34, Ian Bicking <ianb at colorstudy.com> wrote:
> Until the PEP is approved, it's just a suggestion.? So for it to "really" be
> WSGI 2 it will have to go through at least some approval process; which is
> kind of ad hoc, but not so ad hoc as just to implicitly happen.? For WSGI 2
> to happen, someone has to write something up and propose it.? Alice has
> agreed to do that, working from PEP 444 which several other people have
> participated in.? Calling it "WSGI 2" instead of "Web 3" was brought up on
> this list, and the general consensus seemed to be that it made sense

Only about 2 or 3 people commented directly on it from what I
recollect. I would hardly say that is consensus. For myself it came at
a really bad time, coming off the back of a one month trip and
finishing up in a job after 8 1/2 years. If I had known that this
would lead to Alice going around and making comments like 'official'
on what is far from being  I would have objected quite strongly at the
time rather than just treating it like yet another one of the
distractions that has kept coming up over the years when dealing with
the WSGI on Python 3 issue.

> -- some
> people felt a little funny about it, but ultimately it seemed to be
> something everyone was okay with (with some people like myself feeling
> strongly it should be "WSGI 2").
>
> I'm not sure why you are so stressed out about this?

You say I am stressed and Alice in private email likes to think I am
bitter. What I am is passionate. The whole WSGI stuff over the last
few years has been handled so badly by the Python web community it is
embarrassing. We can continue this shambles or try and bring some
sanity with this process. As is, what Alice is now working on will
effectively be the 3rd or 4th variation of what WSGI 2.0 should be
depending on how you count it. First off there was PJEs basic
suggestion of dropping start_response and leaving everything else.
Others then pitched in with there wish list for that. Armin worked on
a proposal for a while and then there was PEP 444 which has mutated
again with what Alice is doing. Some of these have been seen by
outsiders as being what WSGI 2.0 will be and there have at times been
hosting mechanisms or frameworks claiming to support what these
proposals described and calling themselves WSGI 2.0. Luckily those
third party packages claiming to support some form of WSGI 2.0 have
never taken off, but either way the risk of confusion is still there.

> If you think it's
> really an issue, perhaps 2 could be replaced with "2alpha" until such time
> as it is approved?

If it is going to be done under the guise of a continually changing
PEP 444, then refer to it as PEP 444, including the environ tag
prefixes being pep444.

By allowing it to claim in some way that it is going to be WSGI 2.0
you are effectively fixing yourself down a course that that can be the
only successor for WSGI 1.0. So, if someone comes up with a much
better solution, they will be forced to call it something completely
different because you are hardly going to be able to go back and say,
sorry, we made a mistake and all you people who genuinely thought you
were coding to what was going to be a WSGI 2.0 are screwed.

Graham

> On Sat, Jan 1, 2011 at 8:02 PM, Graham Dumpleton
> <graham.dumpleton at gmail.com> wrote:
>>
>> Can we please clear up a matter.
>>
>> GothAlice (don't know off hand there real name), keeps going around
>> and claiming:
>>
>> """
>> After some discussion on the Web-SIG mailing list, PEP 444 is now
>> "officially" WSGI 2, and PEP 3333 is WSGI 1.1
>> """
>>
>> In this instance on web.py forum on Google Groups.
>>
>> I have pointed out a couple of times to them that there is no way that
>> PEP 444 has been blessed as being the official WSGI 2.0 but they are
>> not listening and are still repeating this claim. They can't also get
>> right that PEP 3333 clearly says it is still WSGI 1.0 and not WSGI
>> 1.1.
>>
>> If the people here who's opinion matters are quite happy for GothAlice
>> to hijack the WSGI 2.0 moniker for PEP 444 I will shut up. But if that
>> happens, I will voice my objections by simply not having anything to
>> do with WSGI 2.0 any more.
>>
>> Graham
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/ianb%40colorstudy.com
>
>
>
> --
> Ian Bicking? |? http://blog.ianbicking.org
>

From graham.dumpleton at gmail.com  Sun Jan  2 07:04:35 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sun, 2 Jan 2011 17:04:35 +1100
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
Message-ID: <AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>

On 2 January 2011 16:28, Guido van Rossum <guido at python.org> wrote:
> On Sat, Jan 1, 2011 at 5:02 PM, Graham Dumpleton
> <graham.dumpleton at gmail.com> wrote:
>> Can we please clear up a matter.
>>
>> GothAlice (don't know off hand there real name), keeps going around
>> and claiming:
>>
>> """
>> After some discussion on the Web-SIG mailing list, PEP 444 is now
>> "officially" WSGI 2, and PEP 3333 is WSGI 1.1
>> """
> [...]
>
> From past posts here, that's Alice Bevan?McGregor
> <alice at gothcandy.com>, added to the thread.
>
> On Sat, Jan 1, 2011 at 8:34 PM, Ian Bicking <ianb at colorstudy.com> wrote:
>> Until the PEP is approved, it's just a suggestion.? So for it to "really" be
>> WSGI 2 it will have to go through at least some approval process; which is
>> kind of ad hoc, but not so ad hoc as just to implicitly happen.? For WSGI 2
>> to happen, someone has to write something up and propose it.? Alice has
>> agreed to do that, working from PEP 444 which several other people have
>> participated in.? Calling it "WSGI 2" instead of "Web 3" was brought up on
>> this list, and the general consensus seemed to be that it made sense -- some
>> people felt a little funny about it, but ultimately it seemed to be
>> something everyone was okay with (with some people like myself feeling
>> strongly it should be "WSGI 2").
>>
>> I'm not sure why you are so stressed out about this?? If you think it's
>> really an issue, perhaps 2 could be replaced with "2alpha" until such time
>> as it is approved?
>
> I'm guessing that Graham is concerned that Alice's assertion implies
> that the PEP is approved.

I certainly don't take it as being approved because I know that PEP
444 is quite incomplete at this time. It is the perceptions others get
when they are being told that PEP 444 is WSGI 2.0 that I am worried
about.

> IOW that the *future* WSGI 2.0 is equal to
> the *current* PEP 444. While we don't know for sure, this is likely
> wrong (at least in some details).
>
> OTOH I agree with Ian that it seems correct to say that PEP 444 (which
> is still under development) is striving to arrive at a consensus for
> what will be named WSGI 2.0. This is not unusual in the world of
> standards -- future standards usually are given names and document IDs
> long before there is agreement on the contents of the standard. In
> this sense Alice's use of "officially" is not incorrect, although out
> of context it could be misunderstood to imply PEP approval. I would
> recommend adding caution about this whenever the equivalency between
> PEP 444 and WSGI 2.0 is mentioned -- perhaps it is enough to state
> that "PEP 444 is the draft for WSGI 2.0".

I'd suggest that that is even too strong. You are giving the
impression that no one else can separately come up with another
proposal, especially one that is significantly different.

> Often people or companies draw premature conclusions from draft
> standards and prepare implementations that comply with the draft
> standard in the hope that the draft won't change before it is set in
> stone, and sometimes such implementations are incorrectly billed as
> compliant with the standard (rather than with a specific version of
> the draft). I don't know if that is what Alice is doing -- an equally
> likely theory is that she's just excited.
>
> I haven't followed the development of PEP 4444 much, so I can't
> comment on how much agreement there is on the draft; Ian's use of
> "alpha" suggests that there's some way to go still.
>
> One clearly factual error in Alice's (quoted) post: PEP 3333 is WSGI
> 1.0.1, not WSGI 1.1. AFAIK there's no such thing as WSGI 1.1 now.
>
> Alice, since you have in the past posted here suggesting you are
> interested in carrying PEP 444 / WSGI 2.0 forward, please acknowledge
> that you understand the concerns raised in this thread.
>
> Graham, I suggest that you don't worry about this issue, and instead
> focus on helping the draft turn into a standard by providing feedback
> on PEP 444.

That may be too late. Alice and I haven't exactly hit it off well.
Using the #webcore IRC channel as the forum to work on it as Alice
wants is also totally inadequate. IRC is just not a suitable forum for
discussing this. It is just not possible to fit into a few lines what
takes pages to explain to a level that people can understand. As much
as this mailing list has caused seemingly never ending discussions in
the past that go no where, it is still the appropriate forum for any
discussion.

> Unless there's part of the story you're not telling here.

It should be no secret that I have been progressively poking holes in
the existing WSGI specification through my blog posts and suggesting
remedies. Overall I still see what is being done as more of a band aid
solution. The whole thing needs a rethink because even when you patch
some things, the design in some areas is arguably broken, eg.
wsgi.file_wrapper. Just dropping features or ignoring the fact they
are broken isn't the answer though.

There may also be merit in seeing whether the minimum Python version
be increased such that a solution could may use of new language
features added since Python 2.3. In Python 2.3 you couldn't yield in a
try block. You also didn't have the with statement. No one is looking
at how WSGI might be made better by utilising such new features or at
least understanding properly how the existing specification interacts
with them.

Take for example PEP 325 which the close() method for resource
reclamation was based on. That PEP was rejected in the end and was
replaced with PEP 342 which worked quite differently, yet I cant see
that the WSGI specification was revisited in light of how it ended up
being implemented and the implications of that.

As I have said before, I also still believe that any revised
specification needs to take into consideration deployment. Right now
deployment is still a bit of a mess. If standardisation around a
strategy for how you deploy is reached then likely would be a separate
PEP, but you don't want a situation where it still a bit of a fudge
because the base WSGI specification doesn't make it easy and an
opportunity was missed to change it.

So, what am I saying that isn't being heard, only that I don't want to
see a band aid solution. I had hoped to devote a lot more time to all
this in the coming year now that I have changed jobs and have more
flexibility, but am concerned that it is being taken down a specific
path with no ability to change it.

Graham

From alice at gothcandy.com  Sun Jan  2 10:02:34 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 2 Jan 2011 01:02:34 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
Message-ID: <ifpeva$5c6$1@dough.gmane.org>

Graham,

> GothAlice (don't know off hand there real name), keeps going around and 
> claiming:

Alice Zo? Bevan-McGregor.  If the unicode in my newsreader doesn't 
work, that's an e with an umlaut.

> I have pointed out a couple of times to them that there is no way that 
> PEP 444 has been blessed as being the official WSGI 2.0 but they are 
> not listening and are still repeating this claim. They can't also get 
> right that PEP 3333 clearly says it is still WSGI 1.0 and not WSGI 1.1.

Interestingly, we're both wrong.  PEP 3333 clearly states it's WSGI 
1.0.1 -- it's in the title.  My excuse is 60 hours straight of sleep 
depravation over the holidays.  ;)

> Because for all we know right now what will be WSGI 2.0 may look a lot 
> different to what PEP 444 is now.

That's not a technical reason against, that's a reason to step in and 
help shape the future.

> Already they have taken the original PEP 444 that was put out by Chris, 
> and which had never actually been updated based on feedback on the 
> Python WEB-SIG list to address perceived shortcomings, and started 
> injecting his own ideas on top of it without any real consultation with 
> those on the WEB-SIG who have had a lot of experience with all this 
> stuff.

I, too, have a fair amount of experience with WSGI, having utilized a 
nubmer of alternate frameworks, worked around several common issues, 
ripped apart the top 10 HTTP servers to examine their inner workings, 
torn apart a framework or two, written a performant HTTP/1.1 server 
against both the published PEP 444 draft and my rewrite, and have 
written my own framework used on a number of production deployments in 
at least four countries.

If you'll pardon the expression, it's not like I'm a random ass-hat off 
the street with two weeks of Python experience.

> Thus what he is working on is a very fluid specification that keeps 
> changing. Ie., it is thus a work in progress, yet the way they talk 
> about it is if it already is the official WSGI 2.0 specification when 
> it is still no more than a bunch of ideas of what could be done.

"He" and "they."  Better than "it", I suppose, though last time I 
checked I'm not "people". ;P

It's a draft PEP.  Looking at it, it prominently states at the top of 
both the published and rewritten versions: draft.  Pretty much any time 
I mention my rewrite of it (for clearer RFC-style language and 
disambiguation) I either explicitly call it a "draft" or use language 
indicative of a work in progress.

> [snip] Don't do this and all you do is cause ongoing confusion in the 
> community as to what WSGI 2.0 is given that the definition of what it 
> may be keeps changing.

>From a comment I made on your blog:

> PEP 3333 is a simple-to-implement organic evolution of PEP 333, and 
> thus requires (or should require) little work on the part of the WSGI 
> application developer or WSGI server author. PEP 444 is a clean break.

And further elaborated in further comments:

> PEP 3333 can do a lot of good while maintaining gross compatibility 
> between Py2K and 3K with the same WSGI1 API across both. Also as stated 
> before, PEP 3333 should be ratified As Soon As Possible?. The support 
> is there from the developers implementing it, the implementation is a 
> refinement of PEP 333 w/ clarifications, and all can be happy.
> 
> PEP 444 / WSGI 2 is completely different in goals and approach.

I think it's fairly clear that 3333 is closer to ratification and 
simpler to migrate to, and is a clear way forward for the short- to 
mid-term.

> I also have a various technical issues with the original WSGI 
> specification and they aren't being addressed in PEP 444 from what I 
> have seen so far, as well as having issues with new things in PEP 444.

Have you read my draft rewrite? [1] It incorporates solutions to a 
number of the problems you have described in various articles on your 
blog.  In fact, it's modeled fairly heavily after your "definition 4" 
and its use of native strings.

> I have blogged and posted on the WEB-SIG list about a number of them 
> and am now starting to get back into documenting what some of those 
> other issues are.

I look forward to further input.

> Overall though, I believe a big step needs to be taken back and fresh 
> look at this stuff needs to be made.

That was the goal of the rewrite.  PEP 444 / web3 as published on the 
Python website does fail to solve a number of problems.

> It needs to be cast into the greater context of how we deploy this 
> stuff as well, otherwise deployment is going to continue to be a PITA 
> with all the systems using different ways when there could be a better 
> level of compatibility across them all to make deployment easier.

Deployment isn't really the domain of the web interface API; we have 
distribute, pypi, eggs, etc., etc. for literal package distribution 
(including deployment).  I think a separate PEP should be used to 
describe, say, entry point-based high-level deployment similar to 
PasteDeploy, as an example.

	- Alice

[1] http://bit.ly/e7rtI6



From alice at gothcandy.com  Sun Jan  2 10:47:28 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 2 Jan 2011 01:47:28 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
References: <AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
Message-ID: <ifphjf$g0e$1@dough.gmane.org>

>>> "After some discussion on the Web-SIG mailing list, PEP 444 is now 
>>> "officially" WSGI 2, and PEP 3333 is WSGI 1.1"

This out-of-context quote came after prior discussion of PEP 444 (and 
clear statements of draft-ness) throughout a rather excellent thread on 
HTTP server performance.

The majority of the quotes I find on the CherryPy and WebPy mailing 
lists are better worded: "...updated version of PEP 444[1] is availble 
in draft form", "...[m.s.http] supports PEP 444 (with modifications) 
and, soon, my draft rewrite" and "...PEP 444, I'm trying as hard as I 
can to get my rewrite polished" show up in a quick search.

Apologies for the confusion.

> I'm guessing that Graham is concerned that Alice's assertion implies 
> that the PEP is approved.

He has also raised concerns over my fundamentally not having a mandate, 
having not been granted control over anything, etc.  For completeness 
sake, here's a link to the beginning of the thread in his blog comments:

	http://bit.ly/htHLAz

>> OTOH I agree with Ian that it seems correct to say that PEP 444 (which 
>> is still under development) is striving to arrive at a consensus for 
>> what will be named WSGI 2.0. This is not unusual in the world of 
>> standards -- future standards usually are given names and document IDs 
>> long before there is agreement on the contents of the standard. In this 
>> sense Alice's use of "officially" is not incorrect, although out of 
>> context it could be misunderstood to imply PEP approval. I would 
>> recommend adding caution about this whenever the equivalency between 
>> PEP 444 and WSGI 2.0 is mentioned -- perhaps it is enough to state that 
>> "PEP 444 is the draft for WSGI 2.0".
> 
> I'd suggest that that is even too strong. You are giving the impression 
> that no one else can separately come up with another proposal, 
> especially one that is significantly different.

People have been referring to "anything beyond WSGI 1" as "WSGI 2" for 
a long time.  As in, "I'd like to see X in WSGI 2", or "I hope they fix 
Y in WSGI 2", etc.

As it stands, and since you have refused to give evidence towards 
(links to) others working on things they refer to as "WSGI 2", my 
references to PEP 444 as a draft of WSGI 2 are not unreasonable.  
Quotes taken out of context to support your arguments aren't needed.

>> Often people or companies draw premature conclusions from draft 
>> standards and prepare implementations that comply with the draft 
>> standard in the hope that the draft won't change before it is set in 
>> stone, and sometimes such implementations are incorrectly billed as 
>> compliant with the standard (rather than with a specific version of the 
>> draft). I don't know if that is what Alice is doing -- an equally 
>> likely theory is that she's just excited.

Darn tootin' I'm excited.  :D  The only things that depress me in the 
slightest are the lack of current discussion on the Web-SIG list (I did 
post a thread announcing my rewrite and asking for input, but there 
were no takers) and being told (indirectly) that my solution is crap, 
it'll never be adopted, that I'm wasting my time, and wasting everybody 
else's time.

That's not constructive critisizm or technical faults, that's 
attempting to beat someone down with the weight of your opinion.

>> I haven't followed the development of PEP 444 much, so I can't comment 
>> on how much agreement there is on the draft; Ian's use of "alpha" 
>> suggests that there's some way to go still.

There's a long way to go; I don't expect to see even hints of 
ratification in less than 6 months, really.  I have made that clear in 
prior discussions.

>> One clearly factual error in Alice's (quoted) post: PEP 3333 is WSGI 
>> 1.0.1, not WSGI 1.1. AFAIK there's no such thing as WSGI 1.1 now.

Indeed; apologies for that.  On the mailing lists where Graham has 
stepped in and corrected me I've responsed by thanking him and 
confirming that 3333 is, in fact, 1.0.1.

>> Alice, since you have in the past posted here suggesting you are 
>> interested in carrying PEP 444 / WSGI 2.0 forward, please acknowledge 
>> that you understand the concerns raised in this thread.

Guido, I definitely do; and I have attempted to be as careful as I can 
when attempting to stir discussion over the future of WSGI on any forum 
where I mention it.  I may ocasionally slip up and not give the 
appropriately sized warnings, but I do not do so intentionally!

(Up here in the Canadark the government is now mandating 3/4 of the 
pack graphic warnings on cigarette packages; surely discussion over a 
document which clearly states DRAFT in the header don't need to be 
couched in that level of politial correctness!)

I see it as unfortunate, however, that a thread discussing the 
validitiy of PEP 444 going forward with the moniker "WSGI 2" using 
biased out-of-context quotes has spawned more discussion than the 
threads asking for technical input on the spec.

Politics is more juicy than dry techincal documents, it seems.

:(

> That may be too late. Alice and I haven't exactly hit it off well. 
> Using the #webcore IRC channel as the forum to work on it as Alice 
> wants is also totally inadequate.

I asked you to join me on IRC to better voice your technical complaints 
against PEP 444, not to attempt to get into a verbal sparring match.  
IRC is higher fidelity than threads in blog comments and isn't an 
unreasonable request after a comment thread begins to visibly derail.

> It is just not possible to fit into a few lines what takes pages to 
> explain to a level that people can understand. As much as this mailing 
> list has caused seemingly never ending discussions in the past that go 
> no where, it is still the appropriate forum for any discussion.

Then please resurrect my "PEP 444 Draft Rewrite" thread from the 24th 
of Dec (merry x-mas!) and describe the technical faults you see within 
it.  Using the same argument, your blog (and comments system) is a less 
desireable place for discussion than this mailing list, too.

> ... Overall I still see what is being done as more of a band aid 
> solution. The whole thing needs a rethink because even when you patch 
> some things, the design in some areas is arguably broken, eg. 
> wsgi.file_wrapper. Just dropping features or ignoring the fact they are 
> broken isn't the answer though.
> [snip] ... There may also be merit in seeing whether the minimum Python 
> version be increased such that a solution could may use of new language 
> features added since Python 2.3. [snip]

Now I'm sure: you haven't read my rewrite.

> So, what am I saying that isn't being heard, only that I don't want to 
> see a band aid solution. I had hoped to devote a lot more time to all 
> this in the coming year now that I have changed jobs and have more 
> flexibility, but am concerned that it is being taken down a specific 
> path with no ability to change it.

You are free to write your own PEP and attempt to garner wide-spread 
review and adoption.  You are also in a position in which to encourage 
adoption by adding support to mod_wsgi, which is a signifigantly more 
impressive position to be in than my simply having written a 
proof-of-concept HTTP server to test out PEP 444 ideas, no matter how 
efficient or light-weight it may be.

	- Alice.



From pje at telecommunity.com  Sun Jan  2 13:55:31 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 02 Jan 2011 07:55:31 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.c
 om>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
Message-ID: <20110102125539.2E2063A40A8@sparrow.telecommunity.com>

At 05:04 PM 1/2/2011 +1100, Graham Dumpleton wrote:
>That PEP was rejected in the end and was
>replaced with PEP 342 which worked quite differently, yet I cant see
>that the WSGI specification was revisited in light of how it ended up
>being implemented and the implications of that.

Part of my contribution to PEP 342 was ensuring that it was 
sufficiently PEP 325-compatible to ensure that PEP 333 wouldn't 
*need* revisiting.  At least, not with respect to generator close() 
methods anyway.  ;-)


From guido at python.org  Sun Jan  2 18:21:29 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Jan 2011 09:21:29 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifphjf$g0e$1@dough.gmane.org>
References: <AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
Message-ID: <AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>

Alice and Graham,

I worry a lot that there's a fight brewing here that will lead to
disappointment all around. I see accusations, demands, and passion. I also
see a lot of apathy in the web-sig. This is not a recipe for a successful
standard.

Since it appears we have or are about to face a breakdown of communication,
I'd like to start with some remarks on the mechanics of communication.

   - Alice hasn't posted a link to her rewrite of PEP 444 in a while. AFAICT
   it's this: https://github.com/GothAlice/wsgi2/blob/master/pep444.textile.
I find it a bit disturbing that the "official" copy of PEP 444 (
   http://www.python.org/dev/peps/pep-0444/ ) hasn't been updated. This is
   confusing for occasional observers (like myself), since the
python.orgcopy looks quite dead. It also is not in line with the PEP
workflow as
   written down in PEP 1 (
   http://www.python.org/dev/peps/pep-0001/#pep-work-flow ).
   - It is not reasonable to demand a discussion on IRC. In fact I think it
   is one of the worst media for arriving agreement over a standard. IRC
   doesn't have public logs for those who didn't participate in real time
   (apparently intentionally so); it is pretty hostile to people who don't use
   it regularly (I am one of those); it doesn't work well for people in
   different timezones. Blog comments are not much better (they are archived,
   but as a medium they get too much spam and are too scattered to be worth
   tracking down for other participants); the web-sig mailing list is the
   preferred forum.
   - If you are going to quote stuff from earlier in the thread and respond
   to it using "you", please don't strip the attributions (or add them if your
   mailer doesn't). Also it's best to keep the person you address in the To:
   line of the message (or add them back if your mailer doesn't automatically
   do this).

With that out of the way, let me try to analyze the crux of the matter. The
WSGI 1.0 standard (PEP 333) has been widely successful, but, like any
standard, it has some flaws. People thought that Python 3 would be a good
opportunity to fix the flaws. A proposal was drafted, and posted as PEP 444,
but no agreement was reached. In order to fix some obvious flaws due to
Python 3's different treatment of bytes and text, a much less ambitious
update was produced as PEP 3333, and labeled WSGI 1.0.1. Although this is
still marked as draft, I personally think of it as accepted; it is really
just a very small set of clarifications and disambiguations of PEP 333,
specifically for the benefit of interoperability between WSGI 1.0 apps and
servers across the Python 2 / Python 3 boundary.

But there still was no WSGI 2.0, and the original PEP 444 authors stopped
pushing for their more ambitious draft (I recall this from the web-sig list;
the PEP itself was not updated to reflect this). Then Alice came along, with
much enthusiasm (though perhaps not enough prior understanding of Python's
PEP process) and offered to rewrite PEP 444 to make it better, and to aim to
make the updated version the agreed-upon WSGI 2.0 standard.

I can't quite tell what happened from there; IIRC Alice's proposal did not
receive much objection but neither did she get much support from the folks
on web-sig -- perhaps people were tired of the discussion (which had already
gone around once and not reached any agreement), perhaps people were too
busy to read the list. It seems Graham, at least, falls in the latter
category and is now regretting that he didn't speak up earlier. Ian, OTOH,
seems to implicitly endorse Alice's actions, and seems to be hoping that a
widely accepted WSGI 2.0 standard will come out of her work.

In the mean time, Alice (understandably) has looked for other forums where
she got more feedback -- I may not like IRC, but I can see how the general
apathy on the web-sig is not exactly encouraging. (This is a general problem
with Python -- we always complain that there aren't enough people to do the
work, but when someone shows up and offers to do some work, they don't get
much support. On python-dev we've acknowledged this and are trying to get
better about it.)

In order to get WSGI 2.0 back on the standards track, I think a number of
things have to happen.

First, it would be great if Alice could prepare a version of her draft in
the format required for PEPs, and submit it to the PEP editors (
peps at python.org). Note that the PEP editors do *not* judge a PEP by its
technical merits or have a say in its approval (though they may reject
outright nonsense) -- they merely facilitate the discussion by promptly
checking in the submitted draft, so that PEP authors don't need SVN (or,
soon, I hope Hg) access. The PEP editors also ensure that the PEP is
formatted right, so that it can be automatically processed into HTML for
publication on python.org, and they may make simple fixes for
typos/spelling/grammar. In my experience (being one of them), the PEP
editors usually respond within 24 hours, and don't have any problem with
frequent submissions, as long as new submissions of the same PEP incorporate
any changes that the PEP editors made (e.g. ReST formatting fixes).

(Another way to get new versions submitted to SVN would be to ask the other
author of PEP 444, Armin Ronacher, to check it in; he has submit
privileges.)

The longer Alice's draft lingers on github, the less likely I think it is
that it will ever be elevated to the official WSGI 2.0 standard.

At the same time, I hope that web-sig and Graham can start discussing the
technical merits of Alice's proposal, so that Alice can incorporate the
feedback into her draft. It is my understanding that Alice primarily offered
to improve the writing and precision of the PEP (although that doesn't mean
she can't have an opinion on specific features), and that she is open to
changes proposed by others. (If I were wrong about this, and Alice had an ax
to grind, that would change things, and it might even make sense to have
multiple competing proposals, each hopeful to once earn the WSGI 2.0
moniker. But I hope not.)

Alice, I hope you can live with these recommendations. While it may place a
burden on you to convert your draft to ReST and to have to maintain it that
way, I think there is a much better chance of an open community discussion
leading to a widely accepted standard if you start following the PEP rules
set out in PEP 1 (and a few other low-numbered PEPs).

Graham, I hope that you can stop being grumpy about the process that is
being followed and start using your passion to write up a critique of the
technical merits of Alice's draft. You don't have to attack the whole draft
at once -- you can start by picking one or two important issues and try to
guide a discussion here on web-sig to tease out the best solutions.  Please
understand that given the many different ways people use and implement WSGI
there may be no perfect solution within reach -- writing a successful
standard is the art of the compromise. (If you still think the process going
forward should be different, please write me off-list with your concerns.)

Everyone else on this list, please make a new year's resolution to help the
WSGI 2.0 standard become a reality in 2011.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110102/7bff54a0/attachment.html>

From chrism at plope.com  Sun Jan  2 20:14:00 2011
From: chrism at plope.com (Chris McDonough)
Date: Sun, 02 Jan 2011 14:14:00 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
References: <AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
Message-ID: <1293995640.2726.51.camel@thinko>

On Sun, 2011-01-02 at 09:21 -0800, Guido van Rossum wrote:
> Graham, I hope that you can stop being grumpy about the process that
> is being followed and start using your passion to write up a critique
> of the technical merits of Alice's draft. You don't have to attack the
> whole draft at once -- you can start by picking one or two important
> issues and try to guide a discussion here on web-sig to tease out the
> best solutions.  Please  understand that given the many different ways
> people use and implement WSGI there may be no perfect solution within
> reach -- writing a successful standard is the art of the compromise.
> (If you still think the process going forward should be different,
> please write me off-list with your concerns.)
> 
> Everyone else on this list, please make a new year's resolution to
> help the WSGI 2.0 standard become a reality in 2011.

I think Graham mostly has an issue with this thing being called "WSGI
2".

FTR, avoiding naming arguments is why I titled the original PEP "Web3".
I knew that if I didn't (even though personally I couldn't care less if
it was called Buick or McNugget), people would expend effort arguing
about the name rather than concentrate on the process of creating a new
standard.  They did anyway of course; many people argued publically
wishing to rename Web3 to WSGI2.  On balance, though, I think giving the
standard a "neutral" name before it's widely accepted as a WSGI
successor was (and still is) a good idea, if only as a conflict
avoidance strategy. ;-)

That said, I have no opinion on the technical merits of the new PEP 444
draft; I've resigned myself to using derivatives of PEP 3333 "forever".
It's good enough.  Most of the really interesting stuff seems to happen
at higher levels anyway, and the benefit of a new standard doesn't
outweigh the angst caused by trying to reach another compromise.  I'd
suggest we just embrace it, adding minor tweaks as necessary, until we
reach some sort of technical impasse it doesn't address.

- C



From pje at telecommunity.com  Sun Jan  2 20:57:19 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 02 Jan 2011 14:57:19 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
Message-ID: <20110102195722.953513A4076@sparrow.telecommunity.com>

At 01:47 AM 1/2/2011 -0800, Alice Bevan?McGregor wrote:
>The only things that depress me in the slightest are the lack of 
>current discussion on the Web-SIG list (I did post a thread 
>announcing my rewrite and asking for input, but there were no takers)

FWIW, my lack of interest has been due to two factors.  First, the 
original version of PEP 444 before you worked on it was questionable 
in my book, due to the addition of new optional features (e.g. 
async), and second, when I saw your "filters" proposal, it struck me 
as more of the same, and put me off from investigating further.  (The 
whole idea of having a WSGI 2 (IMO at least), was to get rid of cruft 
and fix bugs, not to add new features.)

Reading the draft now (I had not done so previously), I would suggest 
making your filters proposal available as a standalone module (or an 
addition to a future version of wsgiref.util), for anybody who wants 
that feature, e.g. via:

   def filter_app(app, ingress=(), egress=()):
       def wrapped(environ):
           for f in ingress: f(environ)
           s, h, b = app(environ)
           for f in egress:
               s, h, b = f(environ, s, h, b)
           return s, h, b
       return wrapped

(Writing this implementation highlights one problem with the notion 
of filters, though, and that's that egress filters can't trap an 
error in the application.)

As far as the rest of the draft is concerned, in order to give 
*thorough* technical feedback, I'd need to first make sure that all 
of the important rules from 3333 are still present, which is hard to 
do without basically printing out both PEPs and doing a line-by-line analysis.

I notice that file_wrapper is missing, for example, but am not clear 
on the rationale for its removal.  Is it your intention that servers 
wishing to support file iteration should check for a fileno() directly?

There are a number of minor errors in the draft, such as saying that 
__iter__ must return a bytes instance (it should return an iterator 
yielding bytes instances) and __iter__ itself has broken markup.

On other matters:

* I remain a strong -1 on the .script_name and .path_info variables 
(one of which is named incorrectly in the draft), for reasons 
outlined here previously.  (Specifically, that environ redundancy is 
a train wreck for middleware, which must be written to support both 
ways of providing the same variable, or to delete the extended 
version when modifying the environment.)

* -1 on the key-specific encoding schemes for the various CGI 
variables' values -- for continuity of coding (not to mention 
simplicity) PEP 3333's approach to environ encodings should be 
used.  (That is, the environ consists of bytes-in-unicode-form, 
rather than true unicode strings.)

* +1 to requiring Python 2.6 -- standardizing on b'' notation makes 
good sense and improves forward-portability to Python 3.

* Where is the PARAMETERS variable defined in the CGI spec?  What 
servers actually support it?

* The language about empty vs. missing environment variables appears 
to have disappeared; without it, the spec is ambiguous.

Those are the issues I have identified on a first reading, without 
doing a full analysis.  However, in lieu of such an analysis, I would 
take Graham's word on whether or not your spec addresses the HTTP 
compliance/implementation issues found in previous WSGI specs.

If I may offer a recommendation from previous experience steering 
this PEP, I would say that just because other people know (say) HTTP 
better than you, doesn't mean that you can't still make a better spec 
than they can.  You don't have to make *your* proposal into 
*Graham's* spec, or even the spec that Graham would have wanted you 
to make.  But you *do* need to take Graham's *goals* into consideration.

During the original drafting of PEP 333, Ian proposed quite a lot of 
features that I shot down...  but I modified what I was doing so that 
Ian could still achieve his *goals* within the spec, without 
compromising my core vision for the spec (which was that it should be 
as close to trivial for server implementers as possible, and not 
expose any application API that might be perceived by framework 
developers as competition).

So, I urge you to pay attention when Graham says something about what 
the spec is lacking or how it's broken.  You don't have to fix it 
*his* way, but you do need to fix such that he can still get 
there.  WSGI really does need some fixes for chunking and streaming, 
and Graham absolutely has the expertise in that area, and I would 
100% defer to his judgment on whether some proposal of mine got it 
right in that area.

That doesn't mean I would just adopt whatever he proposed, though, if 
it didn't meet *my* goals.

That's the hart part of making a spec, but it's also the part that 
makes one great.


From guido at python.org  Sun Jan  2 21:00:39 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Jan 2011 12:00:39 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <1293995640.2726.51.camel@thinko>
References: <AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
Message-ID: <AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>

On Sun, Jan 2, 2011 at 11:14 AM, Chris McDonough <chrism at plope.com> wrote:

> On Sun, 2011-01-02 at 09:21 -0800, Guido van Rossum wrote:
> > Graham, I hope that you can stop being grumpy about the process that
> > is being followed and start using your passion to write up a critique
> > of the technical merits of Alice's draft. You don't have to attack the
> > whole draft at once -- you can start by picking one or two important
> > issues and try to guide a discussion here on web-sig to tease out the
> > best solutions.  Please  understand that given the many different ways
> > people use and implement WSGI there may be no perfect solution within
> > reach -- writing a successful standard is the art of the compromise.
> > (If you still think the process going forward should be different,
> > please write me off-list with your concerns.)
> >
> > Everyone else on this list, please make a new year's resolution to
> > help the WSGI 2.0 standard become a reality in 2011.
>
> I think Graham mostly has an issue with this thing being called "WSGI
> 2".
>
> FTR, avoiding naming arguments is why I titled the original PEP "Web3".
> I knew that if I didn't (even though personally I couldn't care less if
> it was called Buick or McNugget), people would expend effort arguing
> about the name rather than concentrate on the process of creating a new
> standard.  They did anyway of course; many people argued publically
> wishing to rename Web3 to WSGI2.  On balance, though, I think giving the
> standard a "neutral" name before it's widely accepted as a WSGI
> successor was (and still is) a good idea, if only as a conflict
> avoidance strategy. ;-)
>

Well, it seems too late for that now. :-)

Note that a new standard, even with a familiar name, doesn't automatically
get wide adoption. This wouldn't be the first time that a new version of a
popular standard is put forward by some well-meaning folks, which is
subsequently ignored by most users. (IIRC this happened to several versions
of IMAP, there's been an "improvement" of HTTP that nobody uses, and HTML 5
seems to be saying that XHTML was a mistake.

So, let's discuss the merits.


> That said, I have no opinion on the technical merits of the new PEP 444
> draft; I've resigned myself to using derivatives of PEP 3333 "forever".
> It's good enough.  Most of the really interesting stuff seems to happen
> at higher levels anyway, and the benefit of a new standard doesn't
> outweigh the angst caused by trying to reach another compromise.  I'd
> suggest we just embrace it, adding minor tweaks as necessary, until we
> reach some sort of technical impasse it doesn't address.
>

Actually that does sound like an opinion on the technical merits. I can't
tell though, because I'm not familiar enough with PEP 444 to know what the
critical differences are compared to PEP 3333. Could someone summarize?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110102/bc34307a/attachment.html>

From alice at gothcandy.com  Sun Jan  2 21:14:46 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 2 Jan 2011 12:14:46 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
Message-ID: <ifqmbm$dmg$1@dough.gmane.org>

On 2011-01-02 09:21:29 -0800, Guido van Rossum said:
> Alice hasn't posted a link to her rewrite of PEP 444 in a while. AFAICT 
> it's this: 
> https://github.com/GothAlice/wsgi2/blob/master/pep444.textile . I find 
> it a bit disturbing that the "official" copy of PEP 444 
> (http://www.python.org/dev/peps/pep-0444/ ) hasn't been updated. This 
> is confusing for occasional observers (like myself), since the 
> python.org copy looks quite dead. It also is not in line with the PEP 
> workflow as written down in PEP 1 
> (http://www.python.org/dev/peps/pep-0001/#pep-work-flow ).

I am unsure of the policy behind updating a PEP on the website from a 
partial (with glaring holes) source.  In my rewrite there are several 
sections that need to be fleshed out before I would consider pushing it 
up-stream.

I'm tentative that way.  ;)

> It is not reasonable to demand a discussion on IRC. In fact I think it 
> is one of the worst media for arriving agreement over a standard. IRC 
> doesn't have public logs for those who didn't participate in real time 
> (apparently intentionally so); it is pretty hostile to people who don't 
> use it regularly (I am one of those); it doesn't work well for people 
> in different timezones. Blog comments are not much better (they are 
> archived, but as a medium they get too much spam and are too scattered 
> to be worth tracking down for other participants); the web-sig mailing 
> list is the preferred forum.

Understood.

> If you are going to quote stuff from earlier in the thread and respond 
> to it using "you", please don't strip the attributions (or add them if 
> your mailer doesn't). Also it's best to keep the person you address in 
> the To: line of the message (or add them back if your mailer doesn't 
> automatically do this).

I do forget that not all newsreaders are the same, and that while mine 
may make quoted text relate to the threading of the messages, not all 
do.  Apologies.  Unfortunately, I subscribe and post using a Usenet 
news reader, so e-mail functionality is not included.  :/

> In order to fix some obvious flaws due to Python 3's different 
> treatment of bytes and text, a much less ambitious update was produced 
> as PEP 3333, and labeled WSGI 1.0.1. Although this is still marked as 
> draft, I personally think of it as accepted; it is really just a very 
> small set of clarifications and disambiguations of PEP 333, 
> specifically for the benefit of interoperability between WSGI 1.0 apps 
> and servers across the Python 2 / Python 3 boundary.

PEP 3333 is an excellent solution that should be quick to adopt.  My 
PEP 444 rewrite takes a fundamentally different approach in an attempt 
to simplify and solve broader problems than pure compatibility.

> In the mean time, Alice (understandably) has looked for other forums 
> where she got more feedback -- I may not like IRC, but I can see how 
> the general apathy on the web-sig is not exactly encouraging. (This is 
> a general problem with Python -- we always complain that there aren't 
> enough people to do the work, but when someone shows up and offers to 
> do some work, they don't get much support. On python-dev we've 
> acknowledged this and are trying to get better about it.)

I have recieved valuable input from a co-conspirator on IRC (who is on 
the other side of the world from me) and mentioning PEP 444 on other 
mailing lists (webpy, cherrypy, pylons) has garnered some interesting 
discussion.

> First, it would be great if Alice could prepare a version of her draft 
> in the format required for PEPs, and submit it to the PEP editors.

I will make this a priority.

> (If I were wrong about this, and Alice had an ax to grind, that would 
> change things, and it might even make sense to have multiple competing 
> proposals, each hopeful to once earn the WSGI 2.0 moniker. But I hope 
> not.)

Despite being a framework author, I am distinctly attempting to tackle 
PEP 444 from an independant viewpoint.  I want a workable solution for 
the majority-not nessicarily /everybody/-not something that caters only 
for solution A or solution B.  The HTTP/1.1 server I've written is, 
ATM, merely a platform from which to brainstorm ideas for PEP 444, and 
to check, using code, that 444 is viable.

> Alice, I hope you can live with these recommendations. While it may 
> place a burden on you to convert your draft to ReST and to have to 
> maintain it that way, I think there is a much better chance of an open 
> community discussion leading to a widely accepted standard if you start 
> following the PEP rules set out in PEP 1 (and a few other low-numbered 
> PEPs).

I'll give the low-numbered PEPs a thurough read through and reformat 
the rewrite using ReST; I knew I was going to have to do this anyway at 
some point.

	- Alice



From alice at gothcandy.com  Sun Jan  2 21:38:53 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 2 Jan 2011 12:38:53 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
Message-ID: <ifqnos$jd3$1@dough.gmane.org>

On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>> I'd suggest we just embrace it, adding minor tweaks as necessary, until 
>> we reach some sort of technical impasse it doesn't address.

Async is one area that 3333 does not cover, and that by not having a 
standard which incorporates async means competing, incompatible 
solutions have been created.

On 2011-01-02 12:00:39 -0800, Guido van Rossum said:
> Actually that does sound like an opinion on the technical merits. I 
> can't tell though, because I'm not familiar enough with PEP 444 to know 
> what the critical differences are compared to PEP 3333. Could someone 
> summarize?

Async, distinction between byte strings (type returned by socket.read), 
native strings, and unicode strings, thorough unicode decoding (moving 
some of the work from middleware to the server), simplified call syntax 
(no more start_response), and clear, consice language with easy 
references via numbered lists.

The async part is an idea in my head that I really do need to write 
down, clarified with help from agronholm on IRC.  The futures PEP is 
available as a pypi installable module, is core in 3.2, and seems to 
provide a simple enough abstract (duck-typed) interface that it should 
be usable as a basis for async in PEP 444.

There are parts I need to add back, of course.  file_wrapper, examples, 
rationale, and references being a few of the items not yet rewritten.

	- Alice.



From guido at python.org  Sun Jan  2 21:55:36 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Jan 2011 12:55:36 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifqnos$jd3$1@dough.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
Message-ID: <AANLkTin7j2MqY5TcH7SRXa5PqpXUd5PgBd60-ctvadz_@mail.gmail.com>

It sounds like async could be a separate PEP, if that will make acceptance
of PEP 444 easier.

On Sun, Jan 2, 2011 at 12:38 PM, Alice Bevan?McGregor
<alice at gothcandy.com>wrote:

> On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>
>> I'd suggest we just embrace it, adding minor tweaks as necessary, until we
>>> reach some sort of technical impasse it doesn't address.
>>>
>>
> Async is one area that 3333 does not cover, and that by not having a
> standard which incorporates async means competing, incompatible solutions
> have been created.
>
>
> On 2011-01-02 12:00:39 -0800, Guido van Rossum said:
>
>> Actually that does sound like an opinion on the technical merits. I can't
>> tell though, because I'm not familiar enough with PEP 444 to know what the
>> critical differences are compared to PEP 3333. Could someone summarize?
>>
>
> Async, distinction between byte strings (type returned by socket.read),
> native strings, and unicode strings, thorough unicode decoding (moving some
> of the work from middleware to the server), simplified call syntax (no more
> start_response), and clear, consice language with easy references via
> numbered lists.
>
> The async part is an idea in my head that I really do need to write down,
> clarified with help from agronholm on IRC.  The futures PEP is available as
> a pypi installable module, is core in 3.2, and seems to provide a simple
> enough abstract (duck-typed) interface that it should be usable as a basis
> for async in PEP 444.
>
> There are parts I need to add back, of course.  file_wrapper, examples,
> rationale, and references being a few of the items not yet rewritten.
>
>
>        - Alice.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110102/2745264a/attachment.html>

From masklinn at masklinn.net  Sun Jan  2 21:55:30 2011
From: masklinn at masklinn.net (Masklinn)
Date: Sun, 2 Jan 2011 21:55:30 +0100
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifqnos$jd3$1@dough.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
Message-ID: <142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>

On 2011-01-02, at 21:38 , Alice Bevan?McGregor wrote:
> On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>>> I'd suggest we just embrace it, adding minor tweaks as necessary, until we reach some sort of technical impasse it doesn't address.
> Async is one area that 3333 does not cover, and that by not having a standard which incorporates async means competing, incompatible solutions have been created.
> 
If I remember the previous Web3 discussion correctly, the result was basically that async has no business being shoehorned in WSGI, that WSGI's model is fundamentally unfit for async and you can't correctly support sync and async with the same spec, and therefore an asynchronous equivalent to WSGI should be developed separately, in order to correctly match the needs of asynchronous servers and interfaces, without the cruft and inadequacies of being forked from a synchronous gateway model.

From guido at python.org  Sun Jan  2 21:54:06 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Jan 2011 12:54:06 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifqmbm$dmg$1@dough.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<ifqmbm$dmg$1@dough.gmane.org>
Message-ID: <AANLkTik_QYfem1SYBTL8iZK3yVeiqQ3A0GkUoXhbA8VX@mail.gmail.com>

On Sun, Jan 2, 2011 at 12:14 PM, Alice Bevan?McGregor
<alice at gothcandy.com>wrote:

> On 2011-01-02 09:21:29 -0800, Guido van Rossum said:
>
>> Alice hasn't posted a link to her rewrite of PEP 444 in a while. AFAICT
>> it's this: https://github.com/GothAlice/wsgi2/blob/master/pep444.textile. I find it a bit disturbing that the "official" copy of PEP 444 (
>> http://www.python.org/dev/peps/pep-0444/ ) hasn't been updated. This is
>> confusing for occasional observers (like myself), since the python.orgcopy looks quite dead. It also is not in line with the PEP workflow as
>> written down in PEP 1 (
>> http://www.python.org/dev/peps/pep-0001/#pep-work-flow ).
>>
>
> I am unsure of the policy behind updating a PEP on the website from a
> partial (with glaring holes) source.  In my rewrite there are several
> sections that need to be fleshed out before I would consider pushing it
> up-stream.
>
> I'm tentative that way.  ;)
>

Please drop the perfectionism. It is unpythonic. :) We believe in "release
early, release often" here.

Even if *you* believe there are no glaring holes, others will find them
anyway. The point of having incomplete drafts up on the website (and in SVN)
is that everybody has the same chance of seeing the draft. (Also note that
the website version is the first hit if you type "PEP 444" into a search
engine.) If it's incomplete in places, just put XXX markers in it. People
are used to this. Publication to the website does not imply any kind of
approval -- it just is a mechanism for people to comment on a draft in a
uniform way. (Also, a detailed SVN history might help people see the
relevant changes -- a big bulk update doesn't shed much light on what
changed since it looks like *everything* changed.)


> PEP 3333 is an excellent solution that should be quick to adopt.  My PEP
> 444 rewrite takes a fundamentally different approach in an attempt to
> simplify and solve broader problems than pure compatibility.


It might be good if you summarized the differences, either here (on the
list) or in the PEP itself. Unlike the documents issued by some standards
bodies, Python PEPs are meant to be readable documents, and can contain
sections about motivation, history, examples, whatever else the author deems
likely to sway its acceptance. The formal specification itself can be in a
section labeled as such.  (PS: I saw you just posted a quick summary, for
which I am grateful. Maybe you can add a more thorough one to the PEP
itself.)

 First, it would be great if Alice could prepare a version of her draft in
>> the format required for PEPs, and submit it to the PEP editors.
>>
>
> I will make this a priority.


Great!

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110102/8eb6ed89/attachment-0001.html>

From guido at python.org  Sun Jan  2 22:31:45 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Jan 2011 13:31:45 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
	<142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>
Message-ID: <AANLkTimpK-bDKKp2OPFU3DGTmi7m5gX8-b79d4nq8bDo@mail.gmail.com>

On Sun, Jan 2, 2011 at 12:55 PM, Masklinn <masklinn at masklinn.net> wrote:

> On 2011-01-02, at 21:38 , Alice Bevan?McGregor wrote:
> > On 2011-01-02 11:14:00 -0800, Chris McDonough said:
> >>> I'd suggest we just embrace it, adding minor tweaks as necessary, until
> we reach some sort of technical impasse it doesn't address.
> > Async is one area that 3333 does not cover, and that by not having a
> standard which incorporates async means competing, incompatible solutions
> have been created.
> >
> If I remember the previous Web3 discussion correctly, the result was
> basically that async has no business being shoehorned in WSGI, that WSGI's
> model is fundamentally unfit for async and you can't correctly support sync
> and async with the same spec, and therefore an asynchronous equivalent to
> WSGI should be developed separately, in order to correctly match the needs
> of asynchronous servers and interfaces, without the cruft and inadequacies
> of being forked from a synchronous gateway model.
>

Masklinn, those are pretty strong words (bordering on offensive). I'm sure
Alice has a different opinion. Alice, hopefully you can write down your
ideas for all to see? Perhaps you have an implementation too? Maybe seeing a
concrete proposal will help us all see how big or small of a shoehorn will
be needed.

(Just trying to keep this thread from degenerating into a shouting match.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110102/c2db1dff/attachment.html>

From alice at gothcandy.com  Sun Jan  2 23:16:48 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 2 Jan 2011 14:16:48 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
	<142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>
	<AANLkTimpK-bDKKp2OPFU3DGTmi7m5gX8-b79d4nq8bDo@mail.gmail.com>
Message-ID: <ifqtgh$bb6$1@dough.gmane.org>

On 2011-01-02 13:31:45 -0800, Guido van Rossum said:
> Masklinn, those are pretty strong words (bordering on offensive). I'm 
> sure Alice has a different opinion. Alice, hopefully you can write down 
> your ideas for all to see? Perhaps you have an implementation too? 
> Maybe seeing a concrete proposal will help us all see how big or small 
> of a shoehorn will be needed.

I will be experimenting with a futures-based async implementation in 
marrow.server.http while writing, as I have been with the draft rewrite 
so far.

> (Just trying to keep this thread from degenerating into a shouting match.)

I missed how his statements could be construed as offensive.  :/  I 
interpreted the multiple "you can't" references to be careless 
shorthand, not explicitly me, so no harm done.

On 2011-01-02 12:55:30 -0800, Masklinn said:
>> If I remember the previous Web3 discussion correctly, the result was 
>> basically that async has no business being shoehorned in WSGI, that 
>> WSGI's model is fundamentally unfit for async and you can't correctly 
>> support sync and async with the same spec, and therefore an 
>> asynchronous equivalent to WSGI should be developed separately, in 
>> order to correctly match the needs of asynchronous servers and 
>> interfaces, without the cruft and inadequacies of being forked from a 
>> synchronous gateway model.

That may have been the result of the previous discussions, however I 
belive I can both write a specification that may be acceptable to 
enough developers, and write a reference implementation illustrating 
both asynchronous and synchronous requests while remaining performant.

	- Alice.



From alice at gothcandy.com  Sun Jan  2 23:21:01 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 2 Jan 2011 14:21:01 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<20110102195722.953513A4076@sparrow.telecommunity.com>
Message-ID: <ifqtod$c8k$1@dough.gmane.org>

On 2011-01-02 11:57:19 -0800, P.J. Eby said:
> FWIW, my lack of interest has been due to two factors.  First, 
> theoriginal version of PEP 444 before you worked on it was questionable 
> in my book, due to the addition of new optional features (e.g. async), 
> and second, when I saw your "filters" proposal, it struck meas more of 
> the same, and put me off from investigating further.  (Thewhole idea of 
> having a WSGI 2 (IMO at least), was to get rid of cruft and fix bugs, 
> not to add new features.)

Interestingly, I see PEP 444 / WSGI 2 as the oppostie; 3333 fixes the 
major outstanding bugs that block Python 3 compatibility without 
improving anything else, 444 is an opportunity to shape the future by 
making previously optional things required (like HTTP/1.1 support) and 
adding features that individual framework authors have all implemented 
in incompatible ways (async).

> Reading the draft now (I had not done so previously), I would 
> suggestmaking your filters proposal available as a standalone module 
> (or anaddition to a future version of wsgiref.util), for anybody who 
> wantsthat feature, e.g. via:

The filtering API is marked as optional, with an example of using it as 
middleware.

> (Writing this implementation highlights one problem with the notionof 
> filters, though, and that's that egress filters can't trap anerror in 
> the application.)

Egress filters only run on successful (non-exceptional) resopnses; if 
you need to capture exceptions to return non-exceptional responses 
(e.g. debugging), you use middleware.  This is not a bug, it's a design 
feature.  :)

> As far as the rest of the draft is concerned, in order to give 
> *thorough* technical feedback, I'd need to first make sure that all of 
> the important rules from 3333 are still present, which is hard todo 
> without basically printing out both PEPs and doing a line-by-line 
> analysis.

I'll do so.

> I notice that file_wrapper is missing, for example, but am not clear on 
> the rationale for its removal.  Is it your intention that 
> serverswishing to support file iteration should check for a fileno() 
> directly?

No, file_wrapper was missing because it's simply one thing I haven't 
gotten around to declaring in a point-by-point RFC manner yet.  :/

> There are a number of minor errors in the draft, such as saying that 
> __iter__ must return a bytes instance (it should return an iterator 
> yielding bytes instances) and __iter__ itself has broken markup.

I've corrected my local copy and will push after reading some 
additional messages.

> * I remain a strong -1 on the .script_name and .path_info variables 
> (one of which is named incorrectly in the draft), for reasons outlined 
> here previously.  (Specifically, that environ redundancy is a train 
> wreck for middleware, which must be written to support bothways of 
> providing the same variable, or to delete the extended version when 
> modifying the environment.)

Agreed.

> * -1 on the key-specific encoding schemes for the various CGI 
> variables' values -- for continuity of coding (not to mention 
> simplicity) PEP 3333's approach to environ encodings should beused.  
> (That is, the environ consists of bytes-in-unicode-form, rather than 
> true unicode strings.)

Does ISO-8859-1 not accomodate this for all but a small number of the 
environment variables in PEP 444?  (Specifically, variables refering to 
parts of the URL/URI.)  And for those variables decoded differently, 
there is an environment variable containing the encoding used.

> * +1 to requiring Python 2.6 -- standardizing on b'' notation makesgood 
> sense and improves forward-portability to Python 3.

In /theory/ you could write a PEP 444-compliant server specifically for 
versions of Python < 2.6, this is not likely to be done, however.

> * Where is the PARAMETERS variable defined in the CGI spec?  
> Whatservers actually support it?

It's defined in the HTTP spec by way of 
http://www.ietf.org/rfc/rfc2396.txt URI Syntax.  These values are 
there, they should be available.  (Specifically semi-colon separated 
parameters and hash-separated fragment.)

> * The language about empty vs. missing environment variables appears to 
> have disappeared; without it, the spec is ambiguous.

I will re-examine the currently published PEP 444.

> Those are the issues I have identified on a first reading, without 
> doing a full analysis.  However, in lieu of such an analysis, I would 
> take Graham's word on whether or not your spec addresses the HTTP 
> compliance/implementation issues found in previous WSGI specs.

In point of fact, I have been, but I need more in comment than "that 
won't work".

> If I may offer a recommendation from previous experience steering this 
> PEP, I would say that just because other people know (say) HTTP better 
> than you, doesn't mean that you can't still make a better specthan they 
> can.  You don't have to make *your* proposal into *Graham's* spec, or 
> even the spec that Graham would have wanted youto make.  But you *do* 
> need to take Graham's *goals* into consideration.

Indeed.  I do try to understand the issues covered in a broader scope 
before writing; for example, I do consider the ability for new 
developers to get up and running without worrying about the example 
applications they are trying to use work in their version of Python; 
thus /allowing/ native strings to be used as response values on Python 
3.

Byte strings are still perferred, and may be more performant, but 
getting started becomes much easier.  This is just one of the points I 
happened to agree on.  ;)

> During the original drafting of PEP 333, Ian proposed quite a lot of 
> features that I shot down...  but I modified what I was doing so that 
> Ian could still achieve his *goals* within the spec, without 
> compromising my core vision for the spec (which was that it should be 
> as close to trivial for server implementers as possible, and not expose 
> any application API that might be perceived by framework developers as 
> competition).

There's a difference between /competition/ and /eliminating pointless, 
incompatible reimplementation/.  A -simple for the application 
developer- async specification will ensure portable, more easily 
tested, consistent solutions.

> So, I urge you to pay attention when Graham says something about 
> whatthe spec is lacking or how it's broken.

I certainly will; I just need to see concrete points against the 
technical merits of the rewritten PEP, not pointless arguing about the 
name of a draft document.  ;)

Thank you for your input!

	- Alice.



From masklinn at masklinn.net  Sun Jan  2 23:27:04 2011
From: masklinn at masklinn.net (Masklinn)
Date: Sun, 2 Jan 2011 23:27:04 +0100
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifqtgh$bb6$1@dough.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
	<142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>
	<AANLkTimpK-bDKKp2OPFU3DGTmi7m5gX8-b79d4nq8bDo@mail.gmail.com>
	<ifqtgh$bb6$1@dough.gmane.org>
Message-ID: <7601B91A-7812-4413-8E0F-41E49B1094BD@masklinn.net>

On 2011-01-02, at 23:16 , Alice Bevan?McGregor wrote:
>> (Just trying to keep this thread from degenerating into a shouting match.)
> I missed how his statements could be construed as offensive.  :/
I missed it as well (though my report might have been brusque), and definitely didn't intent it to be offensive.

>  I interpreted the multiple "you can't" references to be careless shorthand
Yes, it was intended as a short report from the previous discussion's gist/result (or more precisely my potentially incorrect recollection of it), and aimed at suggesting that you check out that previous thread, not as a statement of fact (let alone personal criticism or attacks).


From pje at telecommunity.com  Sun Jan  2 23:41:08 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 02 Jan 2011 17:41:08 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
Message-ID: <20110102224110.ED3043A4076@sparrow.telecommunity.com>

At 12:38 PM 1/2/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>>>I'd suggest we just embrace it, adding minor tweaks as necessary, 
>>>until we reach some sort of technical impasse it doesn't address.
>
>Async is one area that 3333 does not cover, and that by not having a 
>standard which incorporates async means competing, incompatible 
>solutions have been created.

Actually, it's the other way 'round.  I wrote off async for PEP 333 
because the competing, incompatible solutions that already existed 
lacked sufficient ground to build a spec on.

In effect, any direction I took would effectively have either blessed 
one async paradigm as the correct one, or else been a mess that 
nobody would've used anyway.

And this condition largely still exists today: about the only common 
ground between at least *some* async Python frameworks today is the 
use of greenlets...  but if you have greenlets, then you don't need a 
fancy async API in the first place, because you can just "block" 
during I/O from the POV of the app.

The existence of a futures API in the stdlib doesn't alter this much, 
either, because the async frameworks by and large already had their 
own API paradigms for doing such things (e.g. Twisted deferreds and 
thread/process pools, or generator/greenlet-based APIs in other frameworks).

The real bottleneck isn't even that, so much as that if you're going 
to write an async WSGI application, WSGI itself can't define enough 
of an async API to let you do anything useful.  For example, you may 
still need database access that's compatible with the async 
environment you're using...  so you'd only be able to write portable 
async applications if they didn't do ANY I/O outside of HTTP itself!

That being the case, I don't see how a meaningfully cross-platform 
async WSGI can ever really exist, and be attractive both to 
application developers (who want to run on many platforms) and 
framework developers (who want many developers to choose their platform).


>On 2011-01-02 12:00:39 -0800, Guido van Rossum said:
>>Actually that does sound like an opinion on the technical merits. I 
>>can't tell though, because I'm not familiar enough with PEP 444 to 
>>know what the critical differences are compared to PEP 3333. Could 
>>someone summarize?
>
>Async, distinction between byte strings (type returned by 
>socket.read), native strings, and unicode strings,

What distinction do you mean?  I see a difference in *how* you're 
distinguishing byte, native, and unicode strings, but not *that* 
they're distinguished from one another.  (i.e., PEP 3333 
distinguishes them too.)


>thorough unicode decoding (moving some of the work from middleware 
>to the server),

What do you mean by "thorough decoding" and "moving from middleware 
to server"?  Are these references to the redundant environ variables, 
to the use of decoded headers (rather than bytes-in-unicode ones), or 
something else?


>The async part is an idea in my head that I really do need to write 
>down, clarified with help from agronholm on IRC.  The futures PEP is 
>available as a pypi installable module, is core in 3.2, and seems to 
>provide a simple enough abstract (duck-typed) interface that it 
>should be usable as a basis for async in PEP 444.

I suggest reviewing the Web-SIG history of previous async 
discussions; there's a lot more to having a meaningful API spec than 
having a plausible approach.  It's not that there haven't been past 
proposals, they just couldn't get as far as making it possible to 
write a non-trivial async application that would actually be portable 
among Python-supporting asynchronous web servers.

(Now, if you're proposing that web servers run otherwise-synchronous 
applications using futures, that's a different story, and I'd be 
curious to see what you've come up with.  But that's not the same 
thing as an actually-asynchronous WSGI.)  


From bchesneau at gmail.com  Sun Jan  2 23:51:06 2011
From: bchesneau at gmail.com (Benoit Chesneau)
Date: Sun, 2 Jan 2011 23:51:06 +0100
Subject: [Web-SIG] Fwd:  PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTikEDJ_xref8kB7mzYOfq2SuS3LMVRYq1BPFKbyz@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
	<AANLkTikEDJ_xref8kB7mzYOfq2SuS3LMVRYq1BPFKbyz@mail.gmail.com>
Message-ID: <AANLkTi=tQxD9uA+eMn1Up+G4o7w=Bk7=9HqDZvza4LvG@mail.gmail.com>

---------- Forwarded message ----------
From: Benoit Chesneau <bchesneau at gmail.com>
Date: Sun, Jan 2, 2011 at 10:51 PM
Subject: Re: [Web-SIG] PEP 444 != WSGI 2.0
To: Alice Bevan?McGregor <alice at gothcandy.com>


On Sun, Jan 2, 2011 at 9:38 PM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
> On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>>>
>>> I'd suggest we just embrace it, adding minor tweaks as necessary, until
>>> we reach some sort of technical impasse it doesn't address.
>
> Async is one area that 3333 does not cover, and that by not having a
> standard which incorporates async means competing, incompatible solutions
> have been created.
>

I don't see why async need a specific handling in the spec. What
differences do you need ? Today users are perfectly abble to manage
async using gevent, eventlet, ... behind WSGI 1.0 .

- beno?t

From pje at telecommunity.com  Mon Jan  3 00:48:07 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 02 Jan 2011 18:48:07 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifqtod$c8k$1@dough.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<20110102195722.953513A4076@sparrow.telecommunity.com>
	<ifqtod$c8k$1@dough.gmane.org>
Message-ID: <20110102234813.A4FA83A4076@sparrow.telecommunity.com>

At 02:21 PM 1/2/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-02 11:57:19 -0800, P.J. Eby said:
>>* -1 on the key-specific encoding schemes for the various CGI 
>>variables' values -- for continuity of coding (not to mention 
>>simplicity) PEP 3333's approach to environ encodings should beused.
>>(That is, the environ consists of bytes-in-unicode-form, rather 
>>than true unicode strings.)
>
>Does ISO-8859-1 not accomodate this for all but a small number of 
>the environment variables in PEP 444?

PEP 3333 requires that environment variables contain the bytes of the 
HTTP headers, decoded using ISO-8859-1.  The unicode strings, in 
other words are restricted to code points in the 0-255 range, and are 
really just a representation of bytes, rather than being a unicode 
decoding of the contents of the bytes.

What I saw in your draft of PEP 444 (which I admittedly may be 
confused about) is language that simply loosely refers to unicode 
environment variables, which could easily be misconstrued as meaning 
that the values could actually contain other code points.

To be precise, in PEP 333, the "true" unicode value of an environment 
variable is:

     environ[key].encode('iso-8859-1').decode(appropriate_encoding_for_key)

Whereas, my reading of your current draft implies that this has to 
already be done by the server.

As I understand it, the problem with this is that the server 
developer can't always provide such a decoding correctly, and would 
require that the server guess, in the absence of any information that 
it could use to do the guessing.  An application developer is in a 
better position to deal with this ambiguity than the server 
developer, and adding configuration to the server just makes 
deployment more complicated, and breaks application composability if 
two sub-applications within a larger application need different decodings.

That's the rationale for the PEP 3333 approach -- it essentially 
acknowledges that HTTP is bytes, and we're only using strings for the 
API conveniences they afford.



>>* Where is the PARAMETERS variable defined in the CGI spec?
>>Whatservers actually support it?
>
>It's defined in the HTTP spec by way of 
>http://www.ietf.org/rfc/rfc2396.txt URI Syntax.  These values are 
>there, they should be available.  (Specifically semi-colon separated 
>parameters and hash-separated fragment.)

I mean, what web servers currently provide PARAMETERS as a CGI 
variable?  If it's not a CGI variable, it doesn't go in all caps.

What's more, the spec you reference points out that parameters can be 
placed in *each* path-segment, which means that they would:

1) already be in PATH_INFO, and
2) have multiple values

So, -1 on the notion of PARAMETERS, since AFAICT it is redundant, not 
CGI, and would only hold one parameter segment.


>>* The language about empty vs. missing environment variables 
>>appears to have disappeared; without it, the spec is ambiguous.
>
>I will re-examine the currently published PEP 444.

I don't know if it's in there or not; I've read your spec more 
thoroughly than that one.  I'm referring to the language from PEP 333 
and its successor, with which I'm much more intimately familiar.


>Indeed.  I do try to understand the issues covered in a broader 
>scope before writing; for example, I do consider the ability for new 
>developers to get up and running without worrying about the example 
>applications they are trying to use work in their version of Python; 
>thus /allowing/ native strings to be used as response values on Python 3.

I don't understand.  If all the examples in your PEP use b'' strings 
(per the 2.6+ requirement), where is the problem?

They can't use WSGI 1(.0.1) code examples at all (as your draft isn't 
backward-compatible), so I don't see any connection there, either.


>Byte strings are still perferred, and may be more performant,

Performance was not the primary considerations; they were:

* One Obvious Way
* Refuse The Temptation To Guess
* Errors Should Not Pass Silently

The first two would've been fine with unicode; the third was the 
effective tie-breaker.  (Since if you use Unicode, at some point you 
will send garbled data and end up with an error message far away from 
the point where the error occurred.)


>I certainly will; I just need to see concrete points against the 
>technical merits of the rewritten PEP

Well, I've certainly given you some, but it's hard to comment other 
than abstractly on an async spec you haven't proposed yet.  ;-)

Nonetheless, it's really important to understand that the PEP process 
(especially for Informational-track standards) is not so much about 
technical merits in an absolute sense, as it is about *community consensus*.

And that means it's actually a political and marketing process at 
least as much as it is a technical one.  If you miss that, you may 
well end up with a technically-perfect spec (in the sense that nobody 
gives you any additional "concrete points against the technical 
merits"), that nobody cares to actually *implement*.

And from a marketing perspective, the people who must "buy" a WSGI 2 
spec are the *server implementers*.

Sure, if you have a well-defined mapping from WSGI 1.0.x to your 
spec, then you could write a wrapper that provides your spec in any 
WSGI 1.0.x server -- and you can then promote that API for people to 
use.  However, you would then be in the web development API business, 
competing against dozens of existing web frameworks for 
mindshare.  Why use it over any other such API?

So, in order to make the spec a meaningful point of coordination, 
both sides of the spec need some reasonable expectation that the 
people on the other side are really going to use/implement it.

But the *costs* of implementing are asymmetric: writing code against 
(any given spec for) WSGI 2 is a lower commitment than actually 
implementing the corresponding WSGI 2 server.  Which means it's 
harder to convince a server implementer to just do something 
different because it's nicer for the end-user.  (Nice for the 
end-user, after all, is the application framework's job, from the 
server developer's POV!)

This isn't to say you shouldn't make things nicer for users; but your 
spec will likely be successful to the degree it *also* makes things 
nicer for server developers...  like Graham and Robert, among others.

So, the good news is, your real target audience is already here, 
listening, and occasionally commenting on these matters.  The bad 
news is, making them happy has less to do with technical merit per 
se, and much more to do with how much work they see your spec making 
for them.  Welcome to the PEP process.  ;-)

Hm.  That didn't sound quite right: it's not that server developers 
don't care about technical merit.  It's that the technical merits 
they're most interested in are going to fall on the side of things 
like needing to handle ambiguities at the interface between HTTP and 
the WSGI spec.  So a new spec that doesn't address *known* issues 
with HTTP-WSGI relations may be perceived by some as pointless noise 
that isn't helping anything...  and from *their* perspective, they're right.

While it'd be nice if the burden of proof were on them to tell you 
where your spec is wrong, this simply isn't practical.  Reviewing 
every spec that comes along is time-consuming and *hard*.  (To be 
quite honest, I was previously hoping that somebody else besides me 
would do the initial reviewing of your spec -- and I don't plan to 
review every draft or do a super-detailed assessment unless it 
clearly becomes necessary!)

And, whether it's true or fair, a quick glance at your draft can 
easily give the impression to a long-time Web-SIGger that you haven't 
studied the pitfalls that Graham has so extensively documented in the 
past, or that if you did, you didn't give much thought to resolving them.

Again, I don't know if you've done so, but if you have, it's not 
immediately apparent to me from your current draft. And it really is 
up to you to promote and defend your PEP, not up to others to shoot it down.

All that being said, I've pitched in today to give some feedback and 
support because I *want you to succeed*.  The disadvantage to being 
an old hand at this is that it's easy to be discouraged by one's own 
extensive knowledge of the problems, and it's great to have somebody 
with some fresh enthusiasm stepping up to the plate.  (Lord knows I 
don't want to have to write the damn thing myself!)

But that won't change the part where to get *other* people on board, 
you're going to have to convince them that you have a credible 
solution to the problems *they* care about, not just the ones you 
care about.  Writing a rationale that will convince Graham and the 
others that you've got a solution to the problems they've already 
posted about so many times is really Job 1 here if you want to get 
their buy-in.

If you get stuck on some element of what they're asking 
for/complaining about, feel free to post a question here, and I will 
certainly chime in and try to help out.  However, if you keep 
thinking about the part where the overall process is inherently 
unfair to you -- which I totally agree it *is* -- it's not going to 
do anything but stress you out.

And I don't want *that*, because I don't want you to get burned out 
before you do the job of writing the next spec for me.  ;-)


From tseaver at palladion.com  Mon Jan  3 04:08:27 2011
From: tseaver at palladion.com (Tres Seaver)
Date: Sun, 02 Jan 2011 22:08:27 -0500
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <AANLkTimpK-bDKKp2OPFU3DGTmi7m5gX8-b79d4nq8bDo@mail.gmail.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>	<ifphjf$g0e$1@dough.gmane.org>	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>	<1293995640.2726.51.camel@thinko>	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>	<ifqnos$jd3$1@dough.gmane.org>	<142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>
	<AANLkTimpK-bDKKp2OPFU3DGTmi7m5gX8-b79d4nq8bDo@mail.gmail.com>
Message-ID: <ifrej8$3ka$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/02/2011 04:31 PM, Guido van Rossum wrote:
> On Sun, Jan 2, 2011 at 12:55 PM, Masklinn <masklinn at masklinn.net> wrote:
> 
>> On 2011-01-02, at 21:38 , Alice Bevan?McGregor wrote:
>>> On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>>>>> I'd suggest we just embrace it, adding minor tweaks as necessary, until
>> we reach some sort of technical impasse it doesn't address.
>>> Async is one area that 3333 does not cover, and that by not having a
>> standard which incorporates async means competing, incompatible solutions
>> have been created.
>>>
>> If I remember the previous Web3 discussion correctly, the result was
>> basically that async has no business being shoehorned in WSGI, that WSGI's
>> model is fundamentally unfit for async and you can't correctly support sync
>> and async with the same spec, and therefore an asynchronous equivalent to
>> WSGI should be developed separately, in order to correctly match the needs
>> of asynchronous servers and interfaces, without the cruft and inadequacies
>> of being forked from a synchronous gateway model.
>>
> 
> Masklinn, those are pretty strong words (bordering on offensive). I'm sure
> Alice has a different opinion. Alice, hopefully you can write down your
> ideas for all to see? Perhaps you have an implementation too? Maybe seeing a
> concrete proposal will help us all see how big or small of a shoehorn will
> be needed.

I didn't read that as offensive, but as an accurate description of the
de-facto consensus of the SIG:  there is a *very* small minority who
want next-WSGI to address async issues (AFAIK, Alice and Manlio are the
only active proponents).


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0hPasACgkQ+gerLs4ltQ6x5wCfdoX2n+Dpt5wQkY6i8Q8r7hlD
J60An1ZszrZmyfB3S+IGmloHXM4JYTBg
=WCCE
-----END PGP SIGNATURE-----


From guido at python.org  Mon Jan  3 04:21:53 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 2 Jan 2011 19:21:53 -0800
Subject: [Web-SIG] PEP 444 != WSGI 2.0
In-Reply-To: <ifrej8$3ka$1@dough.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<AANLkTi=YmXNMyykh8OestW8_ECQcgiDeMO+pu0D9_Sga@mail.gmail.com>
	<AANLkTinVbFW5_mh5tyzGtvBPEM7zaG2+PKdfXpxBmz0=@mail.gmail.com>
	<AANLkTi=AV1O_=Md1aX-aaxvX1F+n=n2tjPG7_ocK14kB@mail.gmail.com>
	<ifphjf$g0e$1@dough.gmane.org>
	<AANLkTimRSGbq-MzsB17bzQdZH6MXL5W-yR-yHLJa3TS6@mail.gmail.com>
	<1293995640.2726.51.camel@thinko>
	<AANLkTikG64jR6-ULE_ZCHHC8gyqYq7NpafTHSAYV03oM@mail.gmail.com>
	<ifqnos$jd3$1@dough.gmane.org>
	<142088AD-3F24-47BF-A19D-63DA31FBBA0D@masklinn.net>
	<AANLkTimpK-bDKKp2OPFU3DGTmi7m5gX8-b79d4nq8bDo@mail.gmail.com>
	<ifrej8$3ka$1@dough.gmane.org>
Message-ID: <AANLkTi=oYtyPo2ByO=61-n+2ri3EucL2fZCUP8qRT=D2@mail.gmail.com>

On Sun, Jan 2, 2011 at 7:08 PM, Tres Seaver <tseaver at palladion.com> wrote:
> On 01/02/2011 04:31 PM, Guido van Rossum wrote:
>> On Sun, Jan 2, 2011 at 12:55 PM, Masklinn <masklinn at masklinn.net> wrote:
>>
>>> On 2011-01-02, at 21:38 , Alice Bevan?McGregor wrote:
>>>> On 2011-01-02 11:14:00 -0800, Chris McDonough said:
>>>>>> I'd suggest we just embrace it, adding minor tweaks as necessary, until
>>> we reach some sort of technical impasse it doesn't address.
>>>> Async is one area that 3333 does not cover, and that by not having a
>>> standard which incorporates async means competing, incompatible solutions
>>> have been created.
>>>>
>>> If I remember the previous Web3 discussion correctly, the result was
>>> basically that async has no business being shoehorned in WSGI, that WSGI's
>>> model is fundamentally unfit for async and you can't correctly support sync
>>> and async with the same spec, and therefore an asynchronous equivalent to
>>> WSGI should be developed separately, in order to correctly match the needs
>>> of asynchronous servers and interfaces, without the cruft and inadequacies
>>> of being forked from a synchronous gateway model.

>> Masklinn, those are pretty strong words (bordering on offensive). I'm sure
>> Alice has a different opinion. Alice, hopefully you can write down your
>> ideas for all to see? Perhaps you have an implementation too? Maybe seeing a
>> concrete proposal will help us all see how big or small of a shoehorn will
>> be needed.
>
> I didn't read that as offensive, but as an accurate description of the
> de-facto consensus of the SIG: ?there is a *very* small minority who
> want next-WSGI to address async issues (AFAIK, Alice and Manlio are the
> only active proponents).

That's not what Masklinn appeared to be saying though. He implied that
it was a bad idea, not that nobody was interested. Also he used an
excessive number of negative terms ("has no business", "shoehorn",
"fundamentally unfit", "cruft"). If he had said it the way you said it
I would have remained silent. But I accept that I overreacted, and no
offense was taken by Alice, so this is all academic. The best way
forward for async is likely a separate PEP.

-- 
--Guido van Rossum (python.org/~guido)

From jacob at jacobian.org  Tue Jan  4 00:13:27 2011
From: jacob at jacobian.org (Jacob Kaplan-Moss)
Date: Mon, 3 Jan 2011 15:13:27 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
Message-ID: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>

On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum <guido at python.org> wrote:
> Although [PEP 3333] is still marked as draft, I personally think of it
> as accepted; [...]

What does it take to get PEP 3333 formally marked as accepted? Is
there anything I can do to push that process forward?

The lack of a WSGI answer on Py3 is the main thing that's keeping me,
personally, from feeling excited about the platform. Once that's done
I can feel comfortable coding to it -- and browbeating those who don't
support it.

I understand that PEP 444/Web3/WSGI 2/whatever might be a better
answer, but it's clearly got some way to go. In the meantime, what's
next to get PEP 3333 officially endorsed and accepted?

Jacob

From bchesneau at gmail.com  Tue Jan  4 00:19:15 2011
From: bchesneau at gmail.com (Benoit Chesneau)
Date: Tue, 4 Jan 2011 00:19:15 +0100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
Message-ID: <AANLkTi=j_pQvCVhxrTyn52r74=gx3dFCvZpF1S8_E6P_@mail.gmail.com>

On Tue, Jan 4, 2011 at 12:13 AM, Jacob Kaplan-Moss <jacob at jacobian.org> wrote:
> On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum <guido at python.org> wrote:
>> Although [PEP 3333] is still marked as draft, I personally think of it
>> as accepted; [...]
>
> What does it take to get PEP 3333 formally marked as accepted? Is
> there anything I can do to push that process forward?
>
> The lack of a WSGI answer on Py3 is the main thing that's keeping me,
> personally, from feeling excited about the platform. Once that's done
> I can feel comfortable coding to it -- and browbeating those who don't
> support it.
>
> I understand that PEP 444/Web3/WSGI 2/whatever might be a better
> answer, but it's clearly got some way to go. In the meantime, what's
> next to get PEP 3333 officially endorsed and accepted?
>
> Jacob

I we pep3333 is considered accepted, I will start the port of gunicorn
based on it too. It shouldn't be hard since we are using our own http
parser.

- benoit

From guido at python.org  Tue Jan  4 01:43:45 2011
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jan 2011 16:43:45 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
Message-ID: <AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>

On Mon, Jan 3, 2011 at 3:13 PM, Jacob Kaplan-Moss <jacob at jacobian.org> wrote:
> On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum <guido at python.org> wrote:
>> Although [PEP 3333] is still marked as draft, I personally think of it
>> as accepted; [...]
>
> What does it take to get PEP 3333 formally marked as accepted? Is
> there anything I can do to push that process forward?
>
> The lack of a WSGI answer on Py3 is the main thing that's keeping me,
> personally, from feeling excited about the platform. Once that's done
> I can feel comfortable coding to it -- and browbeating those who don't
> support it.
>
> I understand that PEP 444/Web3/WSGI 2/whatever might be a better
> answer, but it's clearly got some way to go. In the meantime, what's
> next to get PEP 3333 officially endorsed and accepted?

I haven't heard anyone speak up against it, ever, since it was
submitted. If no-one speaks up in the next 24 hours consider it
accepted (and after that delay, anyone with SVN privileges can mark it
thus).

-- 
--Guido van Rossum (python.org/~guido)

From rsyring at inteli-com.com  Tue Jan  4 02:04:02 2011
From: rsyring at inteli-com.com (Randy Syring)
Date: Mon, 03 Jan 2011 20:04:02 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
Message-ID: <4D227202.9090902@inteli-com.com>

In the server/gateway example, there is a comment in the code that says:

# TODO: this needs to be binary on Py3

The "TODO" part confuses me.  In other areas of the PEP, there are 
comments like:

# call must be byte-safe on Py3

which make sense.  But is the TODO meant to be a TODO for the PEP or is 
it meant to be a note to the person running the example on Py3.  If the 
latter, maybe "TODO" isn't the best prefix.

FWIW, don't consider this an objection, it is just a question I had as I 
read through the PEP.

--------------------------------------
Randy Syring
Intelicom
Direct: 502-276-0459
Office: 502-212-9913

For the wages of sin is death, but the
free gift of God is eternal life in
Christ Jesus our Lord (Rom 6:23)


On 01/03/2011 07:43 PM, Guido van Rossum wrote:
> On Mon, Jan 3, 2011 at 3:13 PM, Jacob Kaplan-Moss<jacob at jacobian.org>  wrote:
>> On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum<guido at python.org>  wrote:
>>> Although [PEP 3333] is still marked as draft, I personally think of it
>>> as accepted; [...]
>> What does it take to get PEP 3333 formally marked as accepted? Is
>> there anything I can do to push that process forward?
>>
>> The lack of a WSGI answer on Py3 is the main thing that's keeping me,
>> personally, from feeling excited about the platform. Once that's done
>> I can feel comfortable coding to it -- and browbeating those who don't
>> support it.
>>
>> I understand that PEP 444/Web3/WSGI 2/whatever might be a better
>> answer, but it's clearly got some way to go. In the meantime, what's
>> next to get PEP 3333 officially endorsed and accepted?
> I haven't heard anyone speak up against it, ever, since it was
> submitted. If no-one speaks up in the next 24 hours consider it
> accepted (and after that delay, anyone with SVN privileges can mark it
> thus).
>

From pje at telecommunity.com  Tue Jan  4 02:27:09 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 03 Jan 2011 20:27:09 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
Message-ID: <20110104012707.AFE8C3A4119@sparrow.telecommunity.com>

At 04:43 PM 1/3/2011 -0800, Guido van Rossum wrote:
>On Mon, Jan 3, 2011 at 3:13 PM, Jacob Kaplan-Moss <jacob at jacobian.org> wrote:
> > On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum <guido at python.org> wrote:
> >> Although [PEP 3333] is still marked as draft, I personally think of it
> >> as accepted; [...]
> >
> > What does it take to get PEP 3333 formally marked as accepted? Is
> > there anything I can do to push that process forward?
> >
> > The lack of a WSGI answer on Py3 is the main thing that's keeping me,
> > personally, from feeling excited about the platform. Once that's done
> > I can feel comfortable coding to it -- and browbeating those who don't
> > support it.
> >
> > I understand that PEP 444/Web3/WSGI 2/whatever might be a better
> > answer, but it's clearly got some way to go. In the meantime, what's
> > next to get PEP 3333 officially endorsed and accepted?
>
>I haven't heard anyone speak up against it, ever, since it was
>submitted. If no-one speaks up in the next 24 hours consider it
>accepted (and after that delay, anyone with SVN privileges can mark it
>thus).

There are a few minor changes to the code samples needed to make them 
proper Python 3; I just checked in the hairiest of them, though.


From pje at telecommunity.com  Tue Jan  4 02:29:29 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 03 Jan 2011 20:29:29 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <4D227202.9090902@inteli-com.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
Message-ID: <20110104012922.D45143A4119@sparrow.telecommunity.com>

At 08:04 PM 1/3/2011 -0500, Randy Syring wrote:
>In the server/gateway example, there is a comment in the code that says:
>
># TODO: this needs to be binary on Py3
>
>The "TODO" part confuses me.  In other areas of the PEP, there are 
>comments like:
>
># call must be byte-safe on Py3
>
>which make sense.  But is the TODO meant to be a TODO for the PEP or 
>is it meant to be a note to the person running the example on 
>Py3.  If the latter, maybe "TODO" isn't the best prefix.
>
>FWIW, don't consider this an objection, it is just a question I had 
>as I read through the PEP.

Those are my TODO's for the PEP itself, and I've fixed a couple of 
them in SVN (probably around the time you were writing the 
above).  If somebody can point me to the proper Py3 incantation for 
writing bytes to stdout, I'll fix the one remaining TODO marker as well.


From guido at python.org  Tue Jan  4 03:30:44 2011
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jan 2011 18:30:44 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <20110104012922.D45143A4119@sparrow.telecommunity.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
Message-ID: <AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>

On Mon, Jan 3, 2011 at 5:29 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 08:04 PM 1/3/2011 -0500, Randy Syring wrote:
>>
>> In the server/gateway example, there is a comment in the code that says:
>>
>> # TODO: this needs to be binary on Py3
>>
>> The "TODO" part confuses me. ?In other areas of the PEP, there are
>> comments like:
>>
>> # call must be byte-safe on Py3
>>
>> which make sense. ?But is the TODO meant to be a TODO for the PEP or is it
>> meant to be a note to the person running the example on Py3. ?If the latter,
>> maybe "TODO" isn't the best prefix.
>>
>> FWIW, don't consider this an objection, it is just a question I had as I
>> read through the PEP.
>
> Those are my TODO's for the PEP itself, and I've fixed a couple of them in
> SVN (probably around the time you were writing the above). ?If somebody can
> point me to the proper Py3 incantation for writing bytes to stdout, I'll fix
> the one remaining TODO marker as well.

Would

  sys.stdout.buffer.write(b'abc')

do?

(If you mix this with writing strings to sys.stdout directly, you may
have to call sys.stdout.flush() first.)

-- 
--Guido van Rossum (python.org/~guido)

From graham.dumpleton at gmail.com  Tue Jan  4 04:39:17 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 4 Jan 2011 14:39:17 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
Message-ID: <AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>

On 4 January 2011 11:43, Guido van Rossum <guido at python.org> wrote:
> On Mon, Jan 3, 2011 at 3:13 PM, Jacob Kaplan-Moss <jacob at jacobian.org> wrote:
>> On Sun, Jan 2, 2011 at 9:21 AM, Guido van Rossum <guido at python.org> wrote:
>>> Although [PEP 3333] is still marked as draft, I personally think of it
>>> as accepted; [...]
>>
>> What does it take to get PEP 3333 formally marked as accepted? Is
>> there anything I can do to push that process forward?
>>
>> The lack of a WSGI answer on Py3 is the main thing that's keeping me,
>> personally, from feeling excited about the platform. Once that's done
>> I can feel comfortable coding to it -- and browbeating those who don't
>> support it.
>>
>> I understand that PEP 444/Web3/WSGI 2/whatever might be a better
>> answer, but it's clearly got some way to go. In the meantime, what's
>> next to get PEP 3333 officially endorsed and accepted?
>
> I haven't heard anyone speak up against it, ever, since it was
> submitted. If no-one speaks up in the next 24 hours consider it
> accepted (and after that delay, anyone with SVN privileges can mark it
> thus).

I note one issue which I have expressed concern over previously. In
section 'Handling the Content-Length Header; it says:

"""
Under some circumstances, however, the server or gateway may be able
to either generate a Content-Length header, or at least avoid the need
to close the client connection. If the application does not call the
write() callable, and returns an iterable whose len() is 1, then the
server can automatically determine Content-Length by taking the length
of the first bytestring yielded by the iterable.
"""

As documented in:

  http://blog.dscpl.com.au/2009/10/wsgi-issues-with-http-head-requests.html

the automatic addition of a Content-Length response header where
len(iterable) is 1, can cause wrong output for cases where WSGI
application believes that it can itself decide not to return any
actual content for a HEAD response, ignoring the fact that there could
be output filters which rely on headers or content being exactly the
same as for GET.

Do we therefore still want to promote the idea that the optimisation
is a good idea or even allowed?

Next thing is the reference to Jython in section 'Supporting Older
(<2.2) Versions of Python' which are quite out of date with respect to
version of Python it supports. Should that be updated? Should that
whole section be removed now?

Finally, I'd still like to see the CGI gateway example be updated to
properly protect stdin/stdout so that people using print() without
redirecting it to stderr don't stuff themselves up. For example:

    # Keep a reference to the original stdin. We then replace
    # stdin with an empty stream. This is to protect against
    # code from accessing sys.stdin directly and consuming the
    # request content.

    stdin = sys.stdin

    sys.stdin = cStringIO.StringIO('')

    # Keep a reference to the original stdout. We then replace
    # stdout with stderr. This is to protect against code that
    # wants to use 'print' to output debugging. If stdout wasn't
    # protected, then anything output using 'print' would end up
    # being sent as part of the response itself and interfere
    # with the operation of the CGI protocol.

    stdout = sys.stdout

    sys.stdout = sys.stderr

The adapter would then use stdin/stdout local variables and not
sys.stdin/sys.stdout.

The CGI adapter in wsgiref should also really be updated to do
something similar. This would solve the portability problem where code
is written for non CGI hosting environment and they leave 'print'
statements in which breaks on CGI. Even if people who have copied the
CGI gateway from PEP previously don't update their implementations,
have at least set what is best practice for the future.

BTW, I am not across how stdin/stdout works on Windows, but is there
an issue with that CGI example working on Windows because of text
stream vs byte stream issues and CRLF translation?

Graham

From graham.dumpleton at gmail.com  Tue Jan  4 06:01:05 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 4 Jan 2011 16:01:05 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <ACB65F44-1DB3-49DA-94C5-EFFE97B0AE76@fuhm.net>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
	<ACB65F44-1DB3-49DA-94C5-EFFE97B0AE76@fuhm.net>
Message-ID: <AANLkTimpC4VUauRFHCf8+eqNN0OfPFes+RzpA=L771n4@mail.gmail.com>

On 4 January 2011 15:43, James Y Knight <foom at fuhm.net> wrote:
>
> On Jan 3, 2011, at 10:39 PM, Graham Dumpleton wrote:
>
>> As documented in:
>>
>> ?http://blog.dscpl.com.au/2009/10/wsgi-issues-with-http-head-requests.html
>>
>> the automatic addition of a Content-Length response header where
>> len(iterable) is 1, can cause wrong output for cases where WSGI
>> application believes that it can itself decide not to return any
>> actual content for a HEAD response, ignoring the fact that there could
>> be output filters which rely on headers or content being exactly the
>> same as for GET.
>>
>> Do we therefore still want to promote the idea that the optimisation
>> is a good idea or even allowed?
>
> I think it would be nice if it was allowed -- it makes simple apps easier. Just because some WSGI applications may be broken w.r.t. HEAD, that doesn't make this optimization undesirable.
>
> However, the current description does leave things a bit ambiguous. Why, for example, does it suggest only adding Content-Length if the length of the iterable is 1? Surely "if type(iterable) in (list, tuple)", the server could also set the Content-Length header to "sum(len(s) for s in iterable)". Is that forbidden, or just not explicitly spelled out as allowed?

It just doesn't mention it. From memory it also doesn't mention what
to do about case when len(iterable) is 0 either, which presumably if
such an optimisation was allowed could allow you to set Content-Length
to 0.

> If your app *wants* to special-case HEAD handling so as to avoid generating the body when it doesn't need to, how can it do that correctly/reliably? If you normally return content with hard-to-determine length, and you want the HEAD processing to thus also omit Content-Length (and not, say, have the server decide it should return Content-Length: 0), what do you have to return to ensure this happens?

Which is further support for the WSGI server not making decisions
itself to set Content-Length. But then, if an application isn't going
to generate Content-Length for HEAD, then it can't by rights do it for
GET either for same request else the response headers are different.

Graham

From foom at fuhm.net  Tue Jan  4 05:43:34 2011
From: foom at fuhm.net (James Y Knight)
Date: Mon, 3 Jan 2011 23:43:34 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
Message-ID: <ACB65F44-1DB3-49DA-94C5-EFFE97B0AE76@fuhm.net>


On Jan 3, 2011, at 10:39 PM, Graham Dumpleton wrote:

> As documented in:
> 
>  http://blog.dscpl.com.au/2009/10/wsgi-issues-with-http-head-requests.html
> 
> the automatic addition of a Content-Length response header where
> len(iterable) is 1, can cause wrong output for cases where WSGI
> application believes that it can itself decide not to return any
> actual content for a HEAD response, ignoring the fact that there could
> be output filters which rely on headers or content being exactly the
> same as for GET.
> 
> Do we therefore still want to promote the idea that the optimisation
> is a good idea or even allowed?

I think it would be nice if it was allowed -- it makes simple apps easier. Just because some WSGI applications may be broken w.r.t. HEAD, that doesn't make this optimization undesirable. 

However, the current description does leave things a bit ambiguous. Why, for example, does it suggest only adding Content-Length if the length of the iterable is 1? Surely "if type(iterable) in (list, tuple)", the server could also set the Content-Length header to "sum(len(s) for s in iterable)". Is that forbidden, or just not explicitly spelled out as allowed?

If your app *wants* to special-case HEAD handling so as to avoid generating the body when it doesn't need to, how can it do that correctly/reliably? If you normally return content with hard-to-determine length, and you want the HEAD processing to thus also omit Content-Length (and not, say, have the server decide it should return Content-Length: 0), what do you have to return to ensure this happens?

James

From guido at python.org  Tue Jan  4 06:39:25 2011
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Jan 2011 21:39:25 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
Message-ID: <AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>

On Mon, Jan 3, 2011 at 7:39 PM, Graham Dumpleton
<graham.dumpleton at gmail.com> wrote:
> I note one issue which I have expressed concern over previously. In
> section 'Handling the Content-Length Header; it says:
>
> """
> Under some circumstances, however, the server or gateway may be able
> to either generate a Content-Length header, or at least avoid the need
> to close the client connection. If the application does not call the
> write() callable, and returns an iterable whose len() is 1, then the
> server can automatically determine Content-Length by taking the length
> of the first bytestring yielded by the iterable.
> """

That is copied exactly from PEP 333, i.e. WSGI 1.0. I didn't mean to
solicit objections to parts of PEP 3333 that are the same as PEP 333;
PEP 3333 is intended only to specify how WSGI 1.0 compliance is
supposed to work in Python 3. Some clarifications to the original WSGI
1.0 wordings were actually added to PEP 333 around the same time that
PEP 3333 was spun off; AFAIK the changes to PEP 333 were
noncontroversial and merely clarifications of how WSGI already works.
I don't think you can change the above bit of specification (no matter
how bad it is) and still call the resulting spec WSGI 1.0(.x) -- we
don't want to rule out WSGI 1.0 compliance of apps or frameworks that
would be considered compliant under the original 1.0 spec.

-- 
--Guido van Rossum (python.org/~guido)

From graham.dumpleton at gmail.com  Tue Jan  4 06:49:37 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 4 Jan 2011 16:49:37 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
	<AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>
Message-ID: <AANLkTimTkHi8_WgJVLX2UR1Cp9iye4fCKJ9VD+twCH29@mail.gmail.com>

On 4 January 2011 16:39, Guido van Rossum <guido at python.org> wrote:
> On Mon, Jan 3, 2011 at 7:39 PM, Graham Dumpleton
> <graham.dumpleton at gmail.com> wrote:
>> I note one issue which I have expressed concern over previously. In
>> section 'Handling the Content-Length Header; it says:
>>
>> """
>> Under some circumstances, however, the server or gateway may be able
>> to either generate a Content-Length header, or at least avoid the need
>> to close the client connection. If the application does not call the
>> write() callable, and returns an iterable whose len() is 1, then the
>> server can automatically determine Content-Length by taking the length
>> of the first bytestring yielded by the iterable.
>> """
>
> That is copied exactly from PEP 333, i.e. WSGI 1.0. I didn't mean to
> solicit objections to parts of PEP 3333 that are the same as PEP 333;
> PEP 3333 is intended only to specify how WSGI 1.0 compliance is
> supposed to work in Python 3. Some clarifications to the original WSGI
> 1.0 wordings were actually added to PEP 333 around the same time that
> PEP 3333 was spun off; AFAIK the changes to PEP 333 were
> noncontroversial and merely clarifications of how WSGI already works.
> I don't think you can change the above bit of specification (no matter
> how bad it is) and still call the resulting spec WSGI 1.0(.x) -- we
> don't want to rule out WSGI 1.0 compliance of apps or frameworks that
> would be considered compliant under the original 1.0 spec.

I don't believe this really causes a compliance issue as it is a
requirement on the WSGI server, not the WSGI application and doesn't
cause any existing WSGI applications to break.

It also says 'can' and not 'must' so technically WSGI servers are not
currently obligated to do it as I read it and certainly mod_wsgi
doesn't do it any more because it was causing problems for people.

But then, since it does say 'can' and not 'must' any WSGI server
implementers who know better can just ignore it anyway if it if left
in, and it can be dealt with in any new major revision.

Graham

From graham.dumpleton at gmail.com  Tue Jan  4 10:53:04 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 4 Jan 2011 20:53:04 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTimTkHi8_WgJVLX2UR1Cp9iye4fCKJ9VD+twCH29@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
	<AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>
	<AANLkTimTkHi8_WgJVLX2UR1Cp9iye4fCKJ9VD+twCH29@mail.gmail.com>
Message-ID: <AANLkTi=9wcXBGzeTg6OE7ke5S9rwtwPiiR7YExs4Y5YL@mail.gmail.com>

BTW, to what extent are the examples in the PEP meant to be able to
work on both Python 2.X and Python 3.X as is.

Does it need to be clarified where examples will only work on Python
3.X, in particular the CGI gateway.

Graham

On 4 January 2011 16:49, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> On 4 January 2011 16:39, Guido van Rossum <guido at python.org> wrote:
>> On Mon, Jan 3, 2011 at 7:39 PM, Graham Dumpleton
>> <graham.dumpleton at gmail.com> wrote:
>>> I note one issue which I have expressed concern over previously. In
>>> section 'Handling the Content-Length Header; it says:
>>>
>>> """
>>> Under some circumstances, however, the server or gateway may be able
>>> to either generate a Content-Length header, or at least avoid the need
>>> to close the client connection. If the application does not call the
>>> write() callable, and returns an iterable whose len() is 1, then the
>>> server can automatically determine Content-Length by taking the length
>>> of the first bytestring yielded by the iterable.
>>> """
>>
>> That is copied exactly from PEP 333, i.e. WSGI 1.0. I didn't mean to
>> solicit objections to parts of PEP 3333 that are the same as PEP 333;
>> PEP 3333 is intended only to specify how WSGI 1.0 compliance is
>> supposed to work in Python 3. Some clarifications to the original WSGI
>> 1.0 wordings were actually added to PEP 333 around the same time that
>> PEP 3333 was spun off; AFAIK the changes to PEP 333 were
>> noncontroversial and merely clarifications of how WSGI already works.
>> I don't think you can change the above bit of specification (no matter
>> how bad it is) and still call the resulting spec WSGI 1.0(.x) -- we
>> don't want to rule out WSGI 1.0 compliance of apps or frameworks that
>> would be considered compliant under the original 1.0 spec.
>
> I don't believe this really causes a compliance issue as it is a
> requirement on the WSGI server, not the WSGI application and doesn't
> cause any existing WSGI applications to break.
>
> It also says 'can' and not 'must' so technically WSGI servers are not
> currently obligated to do it as I read it and certainly mod_wsgi
> doesn't do it any more because it was causing problems for people.
>
> But then, since it does say 'can' and not 'must' any WSGI server
> implementers who know better can just ignore it anyway if it if left
> in, and it can be dealt with in any new major revision.
>
> Graham
>

From graham.dumpleton at gmail.com  Tue Jan  4 11:51:04 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Tue, 4 Jan 2011 21:51:04 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTi=9wcXBGzeTg6OE7ke5S9rwtwPiiR7YExs4Y5YL@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
	<AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>
	<AANLkTimTkHi8_WgJVLX2UR1Cp9iye4fCKJ9VD+twCH29@mail.gmail.com>
	<AANLkTi=9wcXBGzeTg6OE7ke5S9rwtwPiiR7YExs4Y5YL@mail.gmail.com>
Message-ID: <AANLkTinTt6ihOx8RhBLymx-OMNPGnJ4B8KvkryK+tEsm@mail.gmail.com>

Add another point. FWIW, these are coming up because of questions
being asked on python-dev IRC channel about PEP 3333.

The issue as it came down to was that the PEP may not be clear enough
in explaining that where str() is unicode and as such something like
PATH_INFO, although unicode, is actually bytes decoded as ISO-8859-1,
needed to be re encoded/decoded to get it back to Unicode in the
charset required before use.

They were thinking that because it was unicode already they could use
it as is and not need to do anything. Ie., didn't realise that need to
do:

  path_info = environ.get('PATH_INFO', '')
  path_info = path_info.encode('ISO-8859-1').decode('UTF-8')

for example to get it interpreted as UTF-8 first. They were simply
looking at concatenating new URL bits to the ISO-8859-1 variant from
other unicode strings that weren't bytes represented as ISO-8859-1.

In Python 2.X it was obvious that since it wasn't unicode that you had
to decode it, but confusion may arise for Python 3.X if this
requirement is not explicitly spelled out with a code example like
above.

We all may see it as obvious and yes perhaps it could be covered in
separate articles or commentaries be people, but given this person was
new to it, maybe it is deserving of more explanation in the PEP itself
if they were confused.

It could also be that the PEP covers it adequately already. I am too
tired to read through it again right now.

Graham

On 4 January 2011 20:53, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> BTW, to what extent are the examples in the PEP meant to be able to
> work on both Python 2.X and Python 3.X as is.
>
> Does it need to be clarified where examples will only work on Python
> 3.X, in particular the CGI gateway.
>
> Graham
>
> On 4 January 2011 16:49, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
>> On 4 January 2011 16:39, Guido van Rossum <guido at python.org> wrote:
>>> On Mon, Jan 3, 2011 at 7:39 PM, Graham Dumpleton
>>> <graham.dumpleton at gmail.com> wrote:
>>>> I note one issue which I have expressed concern over previously. In
>>>> section 'Handling the Content-Length Header; it says:
>>>>
>>>> """
>>>> Under some circumstances, however, the server or gateway may be able
>>>> to either generate a Content-Length header, or at least avoid the need
>>>> to close the client connection. If the application does not call the
>>>> write() callable, and returns an iterable whose len() is 1, then the
>>>> server can automatically determine Content-Length by taking the length
>>>> of the first bytestring yielded by the iterable.
>>>> """
>>>
>>> That is copied exactly from PEP 333, i.e. WSGI 1.0. I didn't mean to
>>> solicit objections to parts of PEP 3333 that are the same as PEP 333;
>>> PEP 3333 is intended only to specify how WSGI 1.0 compliance is
>>> supposed to work in Python 3. Some clarifications to the original WSGI
>>> 1.0 wordings were actually added to PEP 333 around the same time that
>>> PEP 3333 was spun off; AFAIK the changes to PEP 333 were
>>> noncontroversial and merely clarifications of how WSGI already works.
>>> I don't think you can change the above bit of specification (no matter
>>> how bad it is) and still call the resulting spec WSGI 1.0(.x) -- we
>>> don't want to rule out WSGI 1.0 compliance of apps or frameworks that
>>> would be considered compliant under the original 1.0 spec.
>>
>> I don't believe this really causes a compliance issue as it is a
>> requirement on the WSGI server, not the WSGI application and doesn't
>> cause any existing WSGI applications to break.
>>
>> It also says 'can' and not 'must' so technically WSGI servers are not
>> currently obligated to do it as I read it and certainly mod_wsgi
>> doesn't do it any more because it was causing problems for people.
>>
>> But then, since it does say 'can' and not 'must' any WSGI server
>> implementers who know better can just ignore it anyway if it if left
>> in, and it can be dealt with in any new major revision.
>>
>> Graham
>>
>

From solipsis at pitrou.net  Tue Jan  4 13:43:15 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 4 Jan 2011 12:43:15 +0000 (UTC)
Subject: [Web-SIG] CGI in PEP 444
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
Message-ID: <loom.20110104T134106-719@post.gmane.org>

Alice Bevan?McGregor <alice at ...> writes:
> 
> [1] http://bit.ly/e7rtI6

So, while we are at it, could we get rid of the "CGI server example" in this new
SWGI spec?
This is 2011, and we should promote modern idioms, not encourage people to do
1995 Web programming. 10 years ago, CGI was already frown upon.
(and even the idea that WSGI should provide some kind of CGI compatibility
sounds a bit ridiculous to me)

Regards

Antoine.



From hidura at gmail.com  Tue Jan  4 15:51:21 2011
From: hidura at gmail.com (Hidura)
Date: Tue, 4 Jan 2011 10:51:21 -0400
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <loom.20110104T134106-719@post.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
Message-ID: <AANLkTi=F4yYWNGAme6jEVmX=spE-jPP8r1pNCtNqyZeg@mail.gmail.com>

Agree, i develop an excelent wsgi server in Py3k, without use the cgi
library to extract the data, so is better if in the manual kills the
part that indicate use the cgi lib to extract the data.

2011/1/4, Antoine Pitrou <solipsis at pitrou.net>:
> Alice Bevan?McGregor <alice at ...> writes:
>>
>> [1] http://bit.ly/e7rtI6
>
> So, while we are at it, could we get rid of the "CGI server example" in this
> new
> SWGI spec?
> This is 2011, and we should promote modern idioms, not encourage people to
> do
> 1995 Web programming. 10 years ago, CGI was already frown upon.
> (and even the idea that WSGI should provide some kind of CGI compatibility
> sounds a bit ridiculous to me)
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/hidura%40gmail.com
>

-- 
Enviado desde mi dispositivo m?vil

Diego I. Hidalgo D.

From pje at telecommunity.com  Tue Jan  4 16:48:25 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 04 Jan 2011 10:48:25 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
Message-ID: <20110104154827.B2DB93A4890@sparrow.telecommunity.com>

At 06:30 PM 1/3/2011 -0800, Guido van Rossum wrote:
>Would
>
>   sys.stdout.buffer.write(b'abc')
>
>do?
>
>(If you mix this with writing strings to sys.stdout directly, you may
>have to call sys.stdout.flush() first.)

The current code is:

             sys.stdout.write(data)  # TODO: this needs to be binary on Py3
             sys.stdout.flush()

Should I be using sys.stdout.buffer for both, or just the write?

For the CGI example in the PEP, I don't want to bother trying to make 
it fully production-usable; that's what we have wsgiref in the stdlib 
for.  So I won't worry about mixing strings and regular output in the 
example, even if perhaps wsgiref should add the StringIO's proposed by Graham.


From pje at telecommunity.com  Tue Jan  4 16:53:17 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 04 Jan 2011 10:53:17 -0500
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <loom.20110104T134106-719@post.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
Message-ID: <20110104155312.8C24B3A4890@sparrow.telecommunity.com>

At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
>Alice Bevan?McGregor <alice at ...> writes: > > [1] 
>http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of 
>the "CGI server example" in this new SWGI spec? This is 2011, and we 
>should promote modern idioms, not encourage people to do 1995 Web 
>programming. 10 years ago, CGI was already frown upon. (and even the 
>idea that WSGI should provide some kind of CGI compatibility sounds 
>a bit ridiculous to me) Regards Antoine.

I still use CGI for the odd one-off, testing, prototyping, etc., and 
it's by far the easiest thing to deploy on a lot of web hosts.  Hell, 
even Google App Engine *emulates* CGI in its default deployment 
configuration, IIRC.  So it's not exactly obsolete.

Also, the main purpose of the example is to show what a web server 
developer needs to do to hook up their own piping to provide WSGI 
services...  and most web server developers have something like CGI 
code already lying around, or at least know what CGI looks like.




>  _______________________________________________ Web-SIG mailing 
> list Web-SIG at python.org Web SIG: http://www.python.org/sigs/web-sig 
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com


From pje at telecommunity.com  Tue Jan  4 16:59:12 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 04 Jan 2011 10:59:12 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTinTt6ihOx8RhBLymx-OMNPGnJ4B8KvkryK+tEsm@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
	<AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>
	<AANLkTimTkHi8_WgJVLX2UR1Cp9iye4fCKJ9VD+twCH29@mail.gmail.com>
	<AANLkTi=9wcXBGzeTg6OE7ke5S9rwtwPiiR7YExs4Y5YL@mail.gmail.com>
	<AANLkTinTt6ihOx8RhBLymx-OMNPGnJ4B8KvkryK+tEsm@mail.gmail.com>
Message-ID: <20110104155909.34CE93A40A8@sparrow.telecommunity.com>

At 09:51 PM 1/4/2011 +1100, Graham Dumpleton wrote:
>Add another point. FWIW, these are coming up because of questions
>being asked on python-dev IRC channel about PEP 3333.
>
>The issue as it came down to was that the PEP may not be clear enough
>in explaining that where str() is unicode and as such something like
>PATH_INFO, although unicode, is actually bytes decoded as ISO-8859-1,
>needed to be re encoded/decoded to get it back to Unicode in the
>charset required before use.
>
>They were thinking that because it was unicode already they could use
>it as is and not need to do anything. Ie., didn't realise that need to
>do:
>
>   path_info = environ.get('PATH_INFO', '')
>   path_info = path_info.encode('ISO-8859-1').decode('UTF-8')
>
>for example to get it interpreted as UTF-8 first. They were simply
>looking at concatenating new URL bits to the ISO-8859-1 variant from
>other unicode strings that weren't bytes represented as ISO-8859-1.
>
>In Python 2.X it was obvious that since it wasn't unicode that you had
>to decode it, but confusion may arise for Python 3.X if this
>requirement is not explicitly spelled out with a code example like
>above.
>
>We all may see it as obvious and yes perhaps it could be covered in
>separate articles or commentaries be people, but given this person was
>new to it, maybe it is deserving of more explanation in the PEP itself
>if they were confused.

It would be really awesome if somebody would write separate 
Application Authors' Guide and Middleware Authors' Guides to 
WSGI.  They don't need to know absolutely everything in the PEP, 
unlike server authors.


>It could also be that the PEP covers it adequately already. I am too
>tired to read through it again right now.

It's pretty prominently stated early on that NO strings in the spec 
are really unicode, they're just bytes packed into unicode objects.

Obviously, no matter how prominently this is stated, some people will 
still make this mistake, but if desired, we could always put some 
additional info near the environ part of the spec for clarification.

(It occurs to me in retrospect that I should probably have updated 
wsgiref in the stdlib to check the bytesy-ness of strings used to 
create Header objects.  Too late for 3.2, though.)


From pje at telecommunity.com  Tue Jan  4 16:55:05 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 04 Jan 2011 10:55:05 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTi=9wcXBGzeTg6OE7ke5S9rwtwPiiR7YExs4Y5YL@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<AANLkTikU-BHiiHHtKX1qMVLianH4FrvMnwG8B2Heas9d@mail.gmail.com>
	<AANLkTinKi8nJLOrc9ZR-5kNYDbyn2gQ_5J=-meaHpeR6@mail.gmail.com>
	<AANLkTimTkHi8_WgJVLX2UR1Cp9iye4fCKJ9VD+twCH29@mail.gmail.com>
	<AANLkTi=9wcXBGzeTg6OE7ke5S9rwtwPiiR7YExs4Y5YL@mail.gmail.com>
Message-ID: <20110104155914.9FAAC3A40A8@sparrow.telecommunity.com>

At 08:53 PM 1/4/2011 +1100, Graham Dumpleton wrote:
>BTW, to what extent are the examples in the PEP meant to be able to
>work on both Python 2.X and Python 3.X as is.
>
>Does it need to be clarified where examples will only work on Python
>3.X, in particular the CGI gateway.

The intention is that PEP 3333 will have examples specific to Python 
3 in future.


From solipsis at pitrou.net  Tue Jan  4 18:19:48 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 4 Jan 2011 17:19:48 +0000 (UTC)
Subject: [Web-SIG] CGI in PEP 444
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
Message-ID: <loom.20110104T181726-729@post.gmane.org>

P.J. Eby <pje at ...> writes:
> 
> At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
> >Alice Bevan?McGregor <alice at ...> writes: > > [1] 
> >http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of 
> >the "CGI server example" in this new SWGI spec? This is 2011, and we 
> >should promote modern idioms, not encourage people to do 1995 Web 
> >programming. 10 years ago, CGI was already frown upon. (and even the 
> >idea that WSGI should provide some kind of CGI compatibility sounds 
> >a bit ridiculous to me) Regards Antoine.
> 
> I still use CGI for the odd one-off, testing, prototyping, etc., and 
> it's by far the easiest thing to deploy on a lot of web hosts.

Really? Isn't that the kind of thing for which wsgiref should be the preferred 
choice?
As for deployment, why would anyone recommend using CGI in production?

Regards

Antoine.



From rdsteph at mac.com  Tue Jan  4 17:23:17 2011
From: rdsteph at mac.com (Ron Stephens)
Date: Tue, 04 Jan 2011 08:23:17 -0800
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <20110104155312.8C24B3A4890@sparrow.telecommunity.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
Message-ID: <9033B105-8447-484F-8998-6013C045FF48@mac.com>

CGI is by far the quickest and easiest way to write and deploy very simple web site scripts. As you move to improve Python for important industrial strength web programming, why not also continue to support quick and dirty web interactivity scripts? 

Ron Stephens

Sent from my iPhone

On Jan 4, 2011, at 7:53 AM, "P.J. Eby" <pje at telecommunity.com> wrote:

> At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
>> Alice Bevan McGregor <alice at ...> writes: > > [1] http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of the "CGI server example" in this new SWGI spec? This is 2011, and we should promote modern idioms, not encourage people to do 1995 Web programming. 10 years ago, CGI was already frown upon. (and even the idea that WSGI should provide some kind of CGI compatibility sounds a bit ridiculous to me) Regards Antoine.
> 
> I still use CGI for the odd one-off, testing, prototyping, etc., and it's by far the easiest thing to deploy on a lot of web hosts.  Hell, even Google App Engine *emulates* CGI in its default deployment configuration, IIRC.  So it's not exactly obsolete.
> 
> Also, the main purpose of the example is to show what a web server developer needs to do to hook up their own piping to provide WSGI services...  and most web server developers have something like CGI code already lying around, or at least know what CGI looks like.
> 
> 
> 
> 
>> _______________________________________________ Web-SIG mailing list Web-SIG at python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com
> 
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/rdsteph%40mac.com

From hidura at gmail.com  Tue Jan  4 18:57:17 2011
From: hidura at gmail.com (hidura at gmail.com)
Date: Tue, 04 Jan 2011 17:57:17 +0000
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <9033B105-8447-484F-8998-6013C045FF48@mac.com>
Message-ID: <20cf30433cfed41ea6049908ff58@google.com>

> CGI is by far the quickest and easiest way to write and deploy very  
> simple web site scripts. As you move to improve Python for important  
> industrial strength web programming, why not also continue to support  
> quick and dirty web interactivity scripts?
Because CGI is an old fashion way to make the things and is very different  
from the strength web programming. I don't say that we have to abandon the  
easy way but wsgi provides an easy way to work and with the PEP-3333 will  
be able in Py3k so CGI will be no useless.

El , Ron Stephens <rdsteph at mac.com> escribi?:
> CGI is by far the quickest and easiest way to write and deploy very  
> simple web site scripts. As you move to improve Python for important  
> industrial strength web programming, why not also continue to support  
> quick and dirty web interactivity scripts?



> Ron Stephens



> Sent from my iPhone



> On Jan 4, 2011, at 7:53 AM, "PJ Eby" pje at telecommunity.com> wrote:



> > At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:

> >> Alice Bevan McGregor writes: > > [1] http:://bit.ly/e7rtI6 So, while  
> we are at it, could we get rid of the "CGI server example" in this new  
> SWGI spec? This is 2011, and we should promote modern idioms, not  
> encourage people to do 1995 Web programming. 10 years ago, CGI was  
> already frown upon. (and even the idea that WSGI should provide some kind  
> of CGI compatibility sounds a bit ridiculous to me) Regards Antoine.

> >

> > I still use CGI for the odd one-off, testing, prototyping, etc., and  
> it's by far the easiest thing to deploy on a lot of web hosts. Hell, even  
> Google App Engine *emulates* CGI in its default deployment configuration,  
> IIRC. So it's not exactly obsolete.

> >

> > Also, the main purpose of the example is to show what a web server  
> developer needs to do to hook up their own piping to provide WSGI  
> services... and most web server developers have something like CGI code  
> already lying around, or at least know what CGI looks like.

> >

> >

> >

> >

> >> _______________________________________________ Web-SIG mailing list  
> Web-SIG at python.org Web SIG: http://www.python.org/sigs/web-sig  
> Unsubscribe:  
> http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com

> >

> > _______________________________________________

> > Web-SIG mailing list

> > Web-SIG at python.org

> > Web SIG: http://www.python.org/sigs/web-sig

> > Unsubscribe:  
> http://mail.python.org/mailman/options/web-sig/rdsteph%40mac.com

> _______________________________________________

> Web-SIG mailing list

> Web-SIG at python.org

> Web SIG: http://www.python.org/sigs/web-sig

> Unsubscribe:  
> http://mail.python.org/mailman/options/web-sig/hidura%40gmail.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110104/b681227e/attachment.html>

From eric at ionrock.org  Tue Jan  4 19:04:58 2011
From: eric at ionrock.org (Eric Larson)
Date: Tue, 04 Jan 2011 12:04:58 -0600
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <loom.20110104T181726-729@post.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
	<loom.20110104T181726-729@post.gmane.org>
Message-ID: <877hekv92t.wl%eric@ionrock.org>

At Tue, 4 Jan 2011 17:19:48 +0000 (UTC),
Antoine Pitrou wrote:
> 
> P.J. Eby <pje at ...> writes:
> > 
> > At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
> > >Alice Bevan?McGregor <alice at ...> writes: > > [1] 
> > >http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of 
> > >the "CGI server example" in this new SWGI spec? This is 2011, and we 
> > >should promote modern idioms, not encourage people to do 1995 Web 
> > >programming. 10 years ago, CGI was already frown upon. (and even the 
> > >idea that WSGI should provide some kind of CGI compatibility sounds 
> > >a bit ridiculous to me) Regards Antoine.
> > 
> > I still use CGI for the odd one-off, testing, prototyping, etc., and 
> > it's by far the easiest thing to deploy on a lot of web hosts.
> 
> Really? Isn't that the kind of thing for which wsgiref should be the preferred 
> choice?
> As for deployment, why would anyone recommend using CGI in production?
> 
> Regards
> 
> Antoine.
> 

It is important to recognize that "production" doesn't necessarily
have to be some ultra powerful server somewhere that is central to
some organization. A simple server running Apache with CGI is just as
valid a production environment as an EC2 cluster. This is especially
true when you are the only person using the application and
requirements are minimal. The point being that in terms of the
specification, it should be plausible a person could use a WSGI
application without heavy server requirements. Shared hosting is the
obvious example here but minimal virtual machines may also fit into
this category.


Eric Larson

> 
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org

From solipsis at pitrou.net  Tue Jan  4 19:24:32 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 4 Jan 2011 18:24:32 +0000 (UTC)
Subject: [Web-SIG] CGI in PEP 444
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
	<loom.20110104T181726-729@post.gmane.org>
	<877hekv92t.wl%eric@ionrock.org>
Message-ID: <loom.20110104T191847-208@post.gmane.org>

Eric Larson <eric at ...> writes:
> It is important to recognize that "production" doesn't necessarily
> have to be some ultra powerful server somewhere that is central to
> some organization. A simple server running Apache with CGI is just as
> valid a production environment as an EC2 cluster. This is especially
> true when you are the only person using the application and
> requirements are minimal. The point being that in terms of the
> specification, it should be plausible a person could use a WSGI
> application without heavy server requirements. Shared hosting is the
> obvious example here but minimal virtual machines may also fit into
> this category.

That sounds backwards. If you use a shared hosting or some minimal virtual
machine, you are much more concerned about resource consumption than if you have
ample processing power (*). And indeed, big companies often have wasteful setups
(including overblown and/or badly-written applications/frameworks/etc.) while
individuals and non-profits have to optimize their setups much more carefully.

(*) (and not only *you* are concerned, but so is your host ;-))

Also, the latency imposed by CGI to each request (due to startup overhead) can
be a problem not only for resource consumption but also for responsiveness and
therefore usability of the Web site/app. Why not use wsgiref directly, at least?

(for the record, the leading Web scripting language, PHP, has moved away from
CGI and standardized on mod_php eons ago)

Regards

Antoine.



From klaus.bremer at bmcct.de  Tue Jan  4 19:18:41 2011
From: klaus.bremer at bmcct.de (Klaus Bremer)
Date: Tue, 4 Jan 2011 19:18:41 +0100
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <20cf30433cfed41ea6049908ff58@google.com>
References: <20cf30433cfed41ea6049908ff58@google.com>
Message-ID: <D89E4525-A857-410E-A617-2C76ECA1553B@bmcct.de>


CGI may be old fashioned, but in my opinion Ron got the point. And for simple applications CGI is fast enough.
I don't want to miss this option.

Klaus.

--

Am 04.01.2011 um 18:57 schrieb hidura at gmail.com:

> Because CGI is an old fashion way to make the things and is very different from the strength web programming. I don't say that we have to abandon the easy way but wsgi provides an easy way to work and with the PEP-3333 will be able in Py3k so CGI will be no useless.
> 
> El , Ron Stephens <rdsteph at mac.com> escribi?:
> > CGI is by far the quickest and easiest way to write and deploy very simple web site scripts. As you move to improve Python for important industrial strength web programming, why not also continue to support quick and dirty web interactivity scripts?
> > 
> > 
> > 
> > Ron Stephens
> > 
> > 
> > 
> > Sent from my iPhone
> > 
> > 
> > 
> > On Jan 4, 2011, at 7:53 AM, "P.J. Eby" pje at telecommunity.com> wrote:
> > 
> > 
> > 
> > > At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
> > 
> > >> Alice Bevan McGregor writes: > > [1] http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of the "CGI server example" in this new SWGI spec? This is 2011, and we should promote modern idioms, not encourage people to do 1995 Web programming. 10 years ago, CGI was already frown upon. (and even the idea that WSGI should provide some kind of CGI compatibility sounds a bit ridiculous to me) Regards Antoine.
> > 
> > >
> > 
> > > I still use CGI for the odd one-off, testing, prototyping, etc., and it's by far the easiest thing to deploy on a lot of web hosts.  Hell, even Google App Engine *emulates* CGI in its default deployment configuration, IIRC.  So it's not exactly obsolete.
> > 
> > >
> > 
> > > Also, the main purpose of the example is to show what a web server developer needs to do to hook up their own piping to provide WSGI services...  and most web server developers have something like CGI code already lying around, or at least know what CGI looks like.
> > 
> > >
> > 
> > >
> > 
> > >
> > 
> > >
> > 
> > >> _______________________________________________ Web-SIG mailing list Web-SIG at python.org Web SIG: http://www.python.org/sigs/web-sig Unsubscribe: http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com
> > 
> > >
> > 
> > > _______________________________________________
> > 
> > > Web-SIG mailing list
> > 
> > > Web-SIG at python.org
> > 
> > > Web SIG: http://www.python.org/sigs/web-sig
> > 
> > > Unsubscribe: http://mail.python.org/mailman/options/web-sig/rdsteph%40mac.com
> > 
> > _______________________________________________
> > 
> > Web-SIG mailing list
> > 
> > Web-SIG at python.org
> > 
> > Web SIG: http://www.python.org/sigs/web-sig
> > 
> > Unsubscribe: http://mail.python.org/mailman/options/web-sig/hidura%40gmail.com
> > 
> >_______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/klaus.bremer%40bmcct.de


From guido at python.org  Tue Jan  4 19:33:12 2011
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jan 2011 10:33:12 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <20110104154827.B2DB93A4890@sparrow.telecommunity.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
Message-ID: <AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>

On Tue, Jan 4, 2011 at 7:48 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 06:30 PM 1/3/2011 -0800, Guido van Rossum wrote:
>>
>> Would
>>
>> ?sys.stdout.buffer.write(b'abc')
>>
>> do?
>>
>> (If you mix this with writing strings to sys.stdout directly, you may
>> have to call sys.stdout.flush() first.)
>
> The current code is:
>
> ? ? ? ? ? ?sys.stdout.write(data) ?# TODO: this needs to be binary on Py3
> ? ? ? ? ? ?sys.stdout.flush()
>
> Should I be using sys.stdout.buffer for both, or just the write?

For both.

But the flush() I was referring to is actually *before* either of
these, suggesting

sys.stdout.flush()
sys.stdout.buffer.write(data)
sys.stdout.buffer.flush()

However the first flush() is only necessary is there's a possibility
that previously something had been written to sys.stdout (not to
sys.stdout.buffer).

> For the CGI example in the PEP, I don't want to bother trying to make it
> fully production-usable; that's what we have wsgiref in the stdlib for. ?So
> I won't worry about mixing strings and regular output in the example, even
> if perhaps wsgiref should add the StringIO's proposed by Graham.

Not sure what you mean. Surely copying and pasting the examples should
work? What are the details you'd like to leave out?

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Tue Jan  4 19:38:16 2011
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jan 2011 10:38:16 -0800
Subject: [Web-SIG] [Python-Dev] PEP 3333: wsgi_string() function
In-Reply-To: <ifvhfl$o2t$1@dough.gmane.org>
References: <1294109093.14661.4.camel@marge> <ifvhfl$o2t$1@dough.gmane.org>
Message-ID: <AANLkTi=FofrCLL6a4r5n=YEcAy6+iCh1OEhyX9aoPY0w@mail.gmail.com>

On Tue, Jan 4, 2011 at 8:22 AM, Tres Seaver <tseaver at palladion.com> wrote:
> Note that Guido just recently wrote on that list that he considers that
> PEP to be de facto accepted.

That was conditional on there not being any objections in the next 24
hours. There have been plenty, so I'm retracting that.

-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Tue Jan  4 19:45:48 2011
From: guido at python.org (Guido van Rossum)
Date: Tue, 4 Jan 2011 10:45:48 -0800
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <20110104155312.8C24B3A4890@sparrow.telecommunity.com>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
Message-ID: <AANLkTi=R5aMhOU4TjURk+4E_BaLi-Gu_iTd=xSermtKW@mail.gmail.com>

On Tue, Jan 4, 2011 at 7:53 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
>> Alice Bevan?McGregor <alice at ...> writes: > > [1] http:://bit.ly/e7rtI6 So,
>> while we are at it, could we get rid of the "CGI server example" in this new
>> SWGI spec? This is 2011, and we should promote modern idioms, not encourage
>> people to do 1995 Web programming. 10 years ago, CGI was already frown upon.
>> (and even the idea that WSGI should provide some kind of CGI compatibility
>> sounds a bit ridiculous to me) Regards Antoine.
>
> I still use CGI for the odd one-off, testing, prototyping, etc., and it's by
> far the easiest thing to deploy on a lot of web hosts. ?Hell, even Google
> App Engine *emulates* CGI in its default deployment configuration, IIRC. ?So
> it's not exactly obsolete.

Right. Note that App Engine does not copy the full CGI mechanism -- it
doesn't start a new process for each request. But it does use
os.environ to set the request parameters for each request. However, in
practice, all but the simplest test apps use a custom WSGI bridge, and
we are considering dropping CGI in favor of WSGI in a future version
of the App Engine runtime.

-- 
--Guido van Rossum (python.org/~guido)

From hidura at gmail.com  Tue Jan  4 20:37:34 2011
From: hidura at gmail.com (Hidura)
Date: Tue, 4 Jan 2011 15:37:34 -0400
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <877hekv92t.wl%eric@ionrock.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
	<loom.20110104T181726-729@post.gmane.org>
	<877hekv92t.wl%eric@ionrock.org>
Message-ID: <AANLkTi=KpOycKj9eu96y6h3GiWcxSvi9u_CJPsiz_GcZ@mail.gmail.com>

I used wsgi and have my app on a shared server and runs excelent,
better than if was using cgi, that cause me a lot of headache with the
incompatiblities with Py3k.

2011/1/4, Eric Larson <eric at ionrock.org>:
> At Tue, 4 Jan 2011 17:19:48 +0000 (UTC),
> Antoine Pitrou wrote:
>>
>> P.J. Eby <pje at ...> writes:
>> >
>> > At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:
>> > >Alice Bevan?McGregor <alice at ...> writes: > > [1]
>> > >http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of
>> > >the "CGI server example" in this new SWGI spec? This is 2011, and we
>> > >should promote modern idioms, not encourage people to do 1995 Web
>> > >programming. 10 years ago, CGI was already frown upon. (and even the
>> > >idea that WSGI should provide some kind of CGI compatibility sounds
>> > >a bit ridiculous to me) Regards Antoine.
>> >
>> > I still use CGI for the odd one-off, testing, prototyping, etc., and
>> > it's by far the easiest thing to deploy on a lot of web hosts.
>>
>> Really? Isn't that the kind of thing for which wsgiref should be the
>> preferred
>> choice?
>> As for deployment, why would anyone recommend using CGI in production?
>>
>> Regards
>>
>> Antoine.
>>
>
> It is important to recognize that "production" doesn't necessarily
> have to be some ultra powerful server somewhere that is central to
> some organization. A simple server running Apache with CGI is just as
> valid a production environment as an EC2 cluster. This is especially
> true when you are the only person using the application and
> requirements are minimal. The point being that in terms of the
> specification, it should be plausible a person could use a WSGI
> application without heavy server requirements. Shared hosting is the
> obvious example here but minimal virtual machines may also fit into
> this category.
>
>
> Eric Larson
>
>>
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/hidura%40gmail.com
>

-- 
Enviado desde mi dispositivo m?vil

Diego I. Hidalgo D.

From hidura at gmail.com  Tue Jan  4 20:44:07 2011
From: hidura at gmail.com (hidura at gmail.com)
Date: Tue, 04 Jan 2011 19:44:07 +0000
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <AANLkTi=KpOycKj9eu96y6h3GiWcxSvi9u_CJPsiz_GcZ@mail.gmail.com>
Message-ID: <90e6ba4fc126e2499704990a7d1f@google.com>

> Right. Note that App Engine does not copy the full CGI mechanism -- it
> doesn't start a new process for each request. But it does use
> os.environ to set the request parameters for each request. However, in
> practice, all but the simplest test apps use a custom WSGI bridge, and
> we are considering dropping CGI in favor of WSGI in a future version
> of the App Engine runtime.

I can give the code of my lib that i use as bridge with my server on
Py3k with wsgi that is small but replace very well the work of cgi
in some functions and the community can start with that, and is
simple to make it run.

El , Hidura <hidura at gmail.com> escribi?:
> I used wsgi and have my app on a shared server and runs excelent,

> better than if was using cgi, that cause me a lot of headache with the

> incompatiblities with Py3k.



> 2011/1/4, Eric Larson eric at ionrock.org>:

> > At Tue, 4 Jan 2011 17:19:48 +0000 (UTC),

> > Antoine Pitrou wrote:

> >>

> >> PJ Eby writes:

> >> >

> >> > At 12:43 PM 1/4/2011 +0000, Antoine Pitrou wrote:

> >> > >Alice BevanMcGregor writes: > > [1]

> >> > >http:://bit.ly/e7rtI6 So, while we are at it, could we get rid of

> >> > >the "CGI server example" in this new SWGI spec? This is 2011, and we

> >> > >should promote modern idioms, not encourage people to do 1995 Web

> >> > >programming. 10 years ago, CGI was already frown upon. (and even the

> >> > >idea that WSGI should provide some kind of CGI compatibility sounds

> >> > >a bit ridiculous to me) Regards Antoine.

> >> >

> >> > I still use CGI for the odd one-off, testing, prototyping, etc., and

> >> > it's by far the easiest thing to deploy on a lot of web hosts.

> >>

> >> Really? Isn't that the kind of thing for which wsgiref should be the

> >> preferred

> >> choice?

> >> As for deployment, why would anyone recommend using CGI in production?

> >>

> >> Regards

> >>

> >> Antoine.

> >>

> >

> > It is important to recognize that "production" doesn't necessarily

> > have to be some ultra powerful server somewhere that is central to

> > some organization. A simple server running Apache with CGI is just as

> > valid a production environment as an EC2 cluster. This is especially

> > true when you are the only person using the application and

> > requirements are minimal. The point being that in terms of the

> > specification, it should be plausible a person could use a WSGI

> > application without heavy server requirements. Shared hosting is the

> > obvious example here but minimal virtual machines may also fit into

> > this category.

> >

> >

> > Eric Larson

> >

> >>

> >> _______________________________________________

> >> Web-SIG mailing list

> >> Web-SIG at python.org

> >> Web SIG: http://www.python.org/sigs/web-sig

> >> Unsubscribe:

> >> http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org

> > _______________________________________________

> > Web-SIG mailing list

> > Web-SIG at python.org

> > Web SIG: http://www.python.org/sigs/web-sig

> > Unsubscribe:

> > http://mail.python.org/mailman/options/web-sig/hidura%40gmail.com

> >



> --

> Enviado desde mi dispositivo m?vil



> Diego I. Hidalgo D.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110104/d4599394/attachment-0001.html>

From foom at fuhm.net  Tue Jan  4 21:04:23 2011
From: foom at fuhm.net (James Y Knight)
Date: Tue, 4 Jan 2011 15:04:23 -0500
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <loom.20110104T191847-208@post.gmane.org>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
	<loom.20110104T181726-729@post.gmane.org>
	<877hekv92t.wl%eric@ionrock.org>
	<loom.20110104T191847-208@post.gmane.org>
Message-ID: <2B22029C-CF02-4631-AE49-7E9E6EB14D44@fuhm.net>

On Jan 4, 2011, at 1:24 PM, Antoine Pitrou wrote:
> (for the record, the leading Web scripting language, PHP, has moved away from
> CGI and standardized on mod_php eons ago)

Yet also still offers a cgi if you want to use that instead.

CGI is a wonderful lowest-common-denominator. It's a great option for little scripts which are infrequently run.

I don't get the hatred: it's old (this is a GOOD thing!), it works, and everything supports it.

Back to the subject of this thread: A simple CGI server is useful because it's simple enough that you can include it in the spec, to demonstrate how to handle various bits of WSGI. And anyone writing a webserver understands CGI, and can understand that. A complete HTTP implementation would not be simple enough to write into the spec.

Obviously if you write your app to the WSGI spec, you will want to deploy it on an existing HTTP server or cgi wrapper in the stdlib (or whatever else you want, that's the whole point!). Users aren't going to be copy and pasting the text from the spec to run their app. So, really, what's the problem?

James

From graham.dumpleton at gmail.com  Tue Jan  4 22:38:37 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Wed, 5 Jan 2011 08:38:37 +1100
Subject: [Web-SIG] CGI in PEP 444
In-Reply-To: <2B22029C-CF02-4631-AE49-7E9E6EB14D44@fuhm.net>
References: <AANLkTi=oc-gzUdnyiz4Y0h+sG-wUBHJZhd56ZbiXQVti@mail.gmail.com>
	<ifpeva$5c6$1@dough.gmane.org>
	<loom.20110104T134106-719@post.gmane.org>
	<20110104155312.8C24B3A4890@sparrow.telecommunity.com>
	<loom.20110104T181726-729@post.gmane.org>
	<877hekv92t.wl%eric@ionrock.org>
	<loom.20110104T191847-208@post.gmane.org>
	<2B22029C-CF02-4631-AE49-7E9E6EB14D44@fuhm.net>
Message-ID: <AANLkTim0N04QdHShF1xXi6LWqm8bgBZo4rZfLtZ2oOO8@mail.gmail.com>

On 5 January 2011 07:04, James Y Knight <foom at fuhm.net> wrote:
> Back to the subject of this thread: A simple CGI server is useful because it's simple enough that you can include it in the spec, to demonstrate how to handle various bits of WSGI. And anyone writing a webserver understands CGI, and can understand that. A complete HTTP implementation would not be simple enough to write into the spec.

+1

And this is the crux of the issue. It doesn't matter whether people
use CGI or not, CGI provides a good basis for showing the mechanics of
how a WSGI server/adapter should process stuff. If not that, what are
you going to do, try and use pseudo code, include a much larger socket
based web server solution?

Graham

From alice at gothcandy.com  Thu Jan  6 05:01:47 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Wed, 5 Jan 2011 20:01:47 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
Message-ID: <ig3erb$vll$1@dough.gmane.org>

[Apologies if this is a double- or triple-post; I seem to be having a 
stupid number of connectivity problems today.]

Howdy!

Apologies for the delay in responding, it?s been a hectic start to the 
new year. ?:)

On 2011-01-03, at 6:22 AM, Timothy Farrell wrote:

> You don't know me but I'm the author of the Rocket Web Server 
> (http://pypi.python.org/pypi/rocket) and have, in the past, been 
> involved in the web2py community. ?Like you, I'm interested in seeing 
> web development come to Python3. ?I'm glad you're taking up WSGI2. ?I 
> have a feature-request for it that perhaps we could work in.

Of course; in fact, I hope you don?t mind that I?ve re-posted this 
response to the web-sig mailing list. ?Async needs significantly 
broader discussion. ?I would appreciate it if you could reply to the 
mailing list thread.

> I would like to see futures added as a server option. ?This way, 
> controllers could dispatch emails (or run some other blocking or 
> long-running task) that would not block the web-response. ?WSGI2 
> Servers could provide a futures executor as environ['wsgi.executor'] 
> that the app could use to offload processes that need not complete 
> before the web-request is served to the client.

E-mail dispatch is one of the things I solved a long time ago with 
TurboMail; it uses a dedicated thread pool and can deliver > 100 unique 
messages per second (more if you use BCC) in the default configuration, 
so I don?t really see that one use case as one that can benefit from 
the futures module. ?Updating TurboMail to use futures would be an 
interesting exercise. ?;)

I was thinking of exposing the executor as 
environ[?wsgi.async.executor?], with ?wsgi.async? being a boolean value 
indicating support.

> What should the server do with the future instances?

The executor returns future instances when running executor.submit/map; 
the application never generates its own Future instances. ?The 
application may, however, use whatever executor it sees fit; it can, 
for example, have one thread pool executor and one process pool, used 
for different tasks.

The server itself can utilize any combination of single-threaded 
IO-based async (see further on in this message), and multi-threaded or 
multi-process management of WSGI requests. ?Resuming suspended 
applications (ones pending future results) is an implementation detail 
of the server.

> Should future.add_done_callback() be allowed? ?I'm not sure how 
> practical/reliable this would be. (By the time the callback is called, 
> the calling environment could be gone. ?Is this undefined behavior?)

If you wrap your callback in a partial(my_callback, environ) the 
environ will survive the end of the request/response cycle (due to the 
incremented reference count), and should be allowed to enable 
intelligent behaviour in the callbacks. ?(Obviously the callbacks will 
not be able to deliver a response to the client at the time they are 
called; the body iterator can, however, wait for the future instance to 
complete and/or timeout.)

A little bit later in this message I describe a better solution than 
the application registering its own callbacks.

> Do we need to also specify what type of executor is provided (threaded 
> vs. separate process)?

I think that?s an application-specific configuration issue, not really 
the concern of the PEP.

> Do you have any thoughts about this?

I believe that intelligent servers need some way to ?pause? a WSGI 
worker rather than relying on the worker executing in a thread and 
blocking while waiting for the return value of a future. ?Using 
generator syntax (yield) with the following rules is my initial idea:

* The application may yield None. ?This is a polite way to have the 
async reactor (in the WSGI server/gateway) reschedule the worker for 
the next reactor cycle. ?Useful as a hint that ?I?m about do do 
something that may take a moment?, allowing other workers to get a 
chance to perform work. (Cooperative multi-tasking on single-threaded 
async servers.)

* The application must yield one 3-tuple WSGI response, and must not 
yield additional data afterwords. ?This is usually the last thing the 
WSGI application would do, with possible cleanup code afterwords 
(before falling off the bottom / raising StopIteration / returning 
None).

* The application may yield Future instances returned by 
environ[?wsgi.executor?].submit/map; the worker will then be paused 
pending execution of the future; the return value of the future will be 
returned from the yield statement. ?Exceptions raised by the future 
will be re-raised from the yield statement and can thus be captured in 
a natural way. ?E.g.:

	try:
	?? ?complex_value = yield environ[?wsgi.executor?].submit(long_running)
	except:
	?? ?pass # handle exceptions generated from within long_running

Similar rules apply to the response body iterator: it yields 
bytestrings, may yield unicode strings where native strings are unicode 
strings, and may yield Future instances which will pause the body 
iterator as per the application callable.

Servers must:

* Allow configuration of the future implementation for options like 
threading / processes.

* Allow developers to override the executor completely.

* Provide additional attributes on wsgi.input: async_ prefixed versions 
of the read methods, which are factories returning server-specific 
Future instances. ?(Allowing a single-threaded async server to handle 
socket IO intelligently with select/epoll/etc.)

To the libraries you use, futures make async pretty much transparent. 
?E.g. libraries (such as a DB layer) must not create their own Future 
objects, but must instead utilize an executor passed to them explicitly 
by the application.

My ideas thus far,

	??Alice.

P.s. a number of these ideas (wsgi.executor, wsgi.async, some of the 
yield syntax described above) have been soundly argued against by a 
co-conspirator over IRC. ?I?ll re-read my IRC logs and reply with those 
considerations in mind (and transcribed logs) shortly.

P.p.s. my kernel panicked while I was translating my rewrite into ReST; 
I'll re-do the conversion tonight or tomorrow morning and submit it 
downstream ASAP.



From alice at gothcandy.com  Thu Jan  6 06:51:44 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Wed, 5 Jan 2011 21:51:44 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org>
Message-ID: <ig3l9g$lqc$1@dough.gmane.org>

Alex Gr?nholm and I have been discussing async implementation details 
(and other areas of PEP 444) for some time on IRC.  Below is the 
cleaned up log transcriptions with additional notes where needed.

Note: The logs are in mixed chronological order ? discussion of one 
topic is chronological, potentially spread across days, but separate 
topics may jump around a bit in time.  Because of this I have 
eliminated the timestamps as they add nothing to the discussion.  
Dialogue in square brackets indicates text added after-the-fact for 
clarity.  Topics are separated by three hyphens.  Backslashes indicate 
joined lines.

This should give a fairly comprehensive explanation of the rationale 
behind some decisions in the rewrite; a version of these conversations 
(in narrative style vs. discussion) will be added to the rewrite Real 
Soon Now? under the Rationale section.

	? Alice.


--- General

agronholm: my greatest fear is that a standard is adopted that does not 
solve existing problems

GothAlice: [Are] there any guarantees as to which thread / process a 
callback [from the future instance] will be executed in?



--- 444 vs. 3333

agronholm: what new features does pep 444 propose to add to pep 3333? \ 
async, filters, no buffering?

GothAlice: Async, filters, no server-level buffering, native string 
usage, the definition of "byte string" as "the format returned by 
socket read" (which, on Java, is unicode!), and the allowance for 
returned data to be Latin1 Unicode. \ All of this together will allow a 
'''def hello(environ): return "200 OK", [], ["Hello world!"]''' example 
application to work across Python versions without modification (or use 
of b"" prefix)

agronholm: why the special casing for latin1 btw? is that an http thing?

GothAlice: Latin1 = \u0000 ? \u00FF ? it's one of the only formats that 
can be decoded while preserving raw bytes, and if another encoding is 
needed, transcode safely. \ Effectively requiring Latin1 for unicode 
output ensures single byte conformance on the data. \ If an application 
needs to return UTF-8, for example, it can return an encoded UTF-8 
bytestream, which will be passed right through,



--- Filters

agronholm: regarding middleware, you did have a point there -- 
exception handling would be pretty difficult with ingress/egress filters

GothAlice: Yup.  It's pretty much a do or die scenario in filter-land.

agronholm: but if we're not ditching middleware, I wonder about the 
overall benefits of filtering \ it surely complicates the scenario so 
it'd better be worth it \ I don't so much agree with your reasoning 
that [middleware] complicates debugging \ I don't see any obvious 
performance improvements either (over middleware)

GothAlice: Simplified debugging of your application w/ reduced stack to 
sort through, reduced nested stack overhead (memory allocation 
improvement), clearer separation of tasks (egress compression is a good 
example).  This follows several of the Zen of Python guidelines: \ 
Simple is better than complex. \ Flat is better than nested. \ There 
should be one-- and preferably only one --obvious way to do it. \ If 
the implementation is hard to explain, it's a bad idea. \ If the 
implementation is easy to explain, it may be a good idea.

agronholm: I would think that whatever memory the stack elements 
consume is peanuts compared to the rest of the application \ 
ingress/egress isn't exactly simpler than middleware

GothAlice: The implementation for ingress/egress filters is two lines 
each: a for loop and a call to the elements iterated over.  Can't get 
much simpler or easier to explain.  ;) \ Middleware is pretty complex? 
\ The majority of ingress filters won't have to examine wsgi.input, and 
supporting async on egress would be relatively easy for the filters 
(pass-through non-bytes data in body_iter). \ If you look at a system 
that offers input filtering, output filtering, and decorators 
(middleware), modifying input should "obviously" be an input filter, 
and vice-versa.

agronholm: how does a server invoke the ingress filters \ in my 
opinion, both ingress and egress filters should essentially be pipes \ 
compression filters are a good example of this \ once a block of 
request data (body) comes through from the client, it should be sent 
through the filter chain

agronholm: consider an application that receives a huge gzip encoded 
upload \ the decompression filter decompresses as much as it can using 
the incoming data \ the application only gets the next block once the 
decompression filter has enough raw data to decompress

GothAlice: Ingress decompression, for example, would accept the environ 
argument, detect gzip content-encoding, then decompress the wsgi.input 
into its own buffer, and finally replace wsgi.input in the environ with 
its decompressed version. \ Alternatively, it could decompress chunks 
and have a more intelligent replacement for wsgi.input (to delay 
decompression until it is needed).

agronholm: are you saying that the filter should decompress all of the 
data at once? how would this work with async?

GothAlice: The first example is the easiest to implement, but you are 
correct in that it would buffer all the data up-front.  The second I 
described (intelligent wsgi.input replacement) would work in an async 
application environment. (But would be harder to code and unit-test.)

agronholm: I don't really see how it would work

GothAlice: environ = parse_headers() ; decompression_filter(environ)

agronholm: wouldn't it be simpler to just have ingress filters return 
the data chunk, altered or not?

GothAlice: decompression_filter(environ): if 
environ.get('HTTP_TRANSFER_ENCODING', None) == 'gzip': 
environ['wsgi.input'] = StreamDecompression(environ['wsgi.input'])

agronholm: I'm not very comfortable with the idea of wsgi.input in 
async apps \ I'm just thinking what would happen when you do 
environ['wsgi.input'].read()

GothAlice: One of two things: in a sync environment, it blocks until it 
can read, in an async environment [combined with yield] it 
pauses/shelves your application until the data is available.

agronholm: I'd rather do away with wsgi.input altogether, but I haven't 
yet figured out how the application would read the entire request body 
then

agronholm: it should be fairly easy to write a helper function for that though

GothAlice: Returning the internal socket representation would improve 
some things, and make things generally worse. :/

agronholm: returning socket from what?

GothAlice: In Tornado's HTTP server, you read and write directly 
from/to the IOStream. \ wsgi.input, though, is more abstracted

agronholm: argh, I can't think of a way to make this work beautifully

GothAlice: Yeah.  :(

agronholm: the requirements of async apps are a big problem

agronholm: returning magic values from the app sounds like a bad idea

agronholm: the best solution I can come up with is to have 
wsgi.async_input or something, which returns an async token for any 
given read operation


agronholm: most filters only deal with the headers \ so what if we made 
it so that the filter chain is only accessed once, and filters that 
need to modify the body as well would return a generator \ and when the 
server receives more data, it would feed it to the first generator in 
the chain, feed the results from that to the next etc.

agronholm: the generators could also return futures, at which point the 
server adjourns processing of the chain until the callback fires \ in 
multithreaded mode, the server would simply call .result() which would 
block, and in single threaded mode, add a callback to the reactor

GothAlice: Hmm.

agronholm: the ingress filters' return values would affect what is sent 
to the application

agronholm: [I'm] trying to solve the inherent difficulties with having 
a file-like object in the environ \ my solution would allow them to 
work transparently with sync and async apps alike

GothAlice: Hmm.  What would the argspec of an ingress filter be, then?  
(The returned value, via yield, being wsgi.input chunks.)

agronholm: probably environ, body_generator or something

agronholm: the beauty in wsgi in general is of course that it requires 
no importing of predefined functions or anything \ so there should be 
some way for the application to read the entire request at once

GothAlice: I think combining wsgi.async with specific attributes on 
wsgi.input which can be yielded as async tokens might be a way to go.


GothAlice: agronholm: yielding None from the application being a polite 
way to re-schedule the application after a reactor cycle to give other 
connections a chance before doing something potentially blocking.

agronholm: I thought None meant "I'm done here" \ otoh, the app has to 
return some response

GothAlice: That's yielding an application response tuple followed by 
StopIteration. \ (Not necessarily immediately returning StopIteration 
after yielding the response; there may be clean-up to do; which is a 
nice addition.)

GothAlice: Three options: yield None (reschedule to be nice/cooperative 
behaviour), yield status, headers, body (deliver a response), and yield 
AsyncToken.

agronholm: so what would the application yield if it wanted to generate 
the body in chunks? (potentially a slow process)

GothAlice: A body_iter that generates the body in chunks, as per a 
standard (non-generator) application callable. \ That wouldn't change. 
\ But often an application would want to async stream the response body 
in before starting body generation.

GothAlice: An application MUST be a callable returning (status_bytes, 
header_list, body_iter) OR a generator.  IF the application is a 
generator, it MUST yield EITHER None (delay execution), a 
(status_bytes, header_list, body_iter) tuple, or an async token.  After 
yielding a response the application generator MAY perform additional 
actions before raising StopIteration, but MUST NOT yield anything but 
None or async tokens from that point onward.

agronholm: one of my concerns is how a request body modifying 
middleware will work with async apps unless it's specifically designed 
with those in mind \ you suggested that such middleware replace 
wsgi.input with their own

GothAlice: It would have to be; or it could simply yield through 
non-bytes chunks, returning the result of the yield back up (which may 
be ignored).

agronholm: what guarantee is there that the replacement has 
.async_read() unless the filter was specifically designed to be async 
aware?

GothAlice: Or, if the developer was in a particularly black mood, the 
middleware could re-set wsgi.async to be false.  ;)

agronholm: I don't quite understand the meaning or point of wsgi.async

GothAlice: wsgi.async is a boolean representing the capability of the 
underlying server to accept async tokens.

agronholm: why would that ever be false? \ in a blocking/threaded 
server, implementing support for that is trivial

GothAlice: Why does no HTTP server in Python conform to the HTTP/1.1 
spec properly?  Lazy developers.  ;)  [And lack of interest 
down-stream.  Calling server authors idiots was not my intention.]

agronholm: they could just as well forgo setting wsgi.async altogether

GothAlice: environ.get('wsgi.async', False) is the only way to armor 
against that, I guess.

agronholm: well I think we're talking about *conforming* servers here \ 
there's not much that can be done about incomplete implementations

GothAlice: However, if wsgi.async is going to be in the WSGI2 spec, 
it'll be required.  if the server hasn't gotten around to implementing 
async yet, it should be False.

agronholm: I think wsgi.async is useless \ "hasn't gotten around to"? 
that's not a lot of work, really \
that flag just paves way for half assed implementations

GothAlice: Still, some method to detect the capability should be 
present.  Something more than attempting to access wsgi.input's 
async_read attribute and catching the AttributeError exception.

agronholm: the capability should be *required* \ given how easy it is 
to implement \ I don't see any justification not to require it

GothAlice: We'll have to see how easy it is to add to m.s.http before 
I'll admit it's "easy" in the general sense.  ;)  If it turns out to be 
simple (and not-too-badly-performance-impacting) I'll make it required.

agronholm: fair enough


agronholm: robert pointed out the difficulty of executing them in the 
right order

GothAlice: Indeed; this problem exists with the current middleware system, too.

agronholm: it'd probably be easier to specify them as a list in the 
deployment descriptor

GothAlice: (My note about appending to ingress_filters, prepending to 
egress_filters to simulate middleware behaviour is functional, though 
non-optimal; the filters, if co-dependant, should be middleware 
instead.)

agronholm: webcore's current middleware system is too much magic imho

GothAlice: I agree. \ A init.d-style ordering system would have to be 
its own PEP.

agronholm: also, I was thinking if we could filters that needed both 
ingress/egress capabilities (such as session middleware) in a way that 
only required specifying it once

GothAlice: ? wouldn't that be middleware?  ;) \ Thus far I've defined 
ingress and egress filters as distinct and separate, with 
dual-functionality requirements being fulfilled by middleware.
agronholm: we could probably simplify that

GothAlice: "There should be one, and preferably only one, right way to 
do something."  ;)

agronholm: yes, and that is the point of my idea :)

GothAlice: Replacing middleware isn't a small task; the power of 
potentially redirecting application flow (amongst other gems the 
middleware structure brings to the table) would be very difficult to 
model cleanly when separated into ingress/egress.

agronholm: btw, I very much agreed with PJE's suggestion of making 
filtering its own middleware instead of a static part of the interface

GothAlice: The problem with not mentioning filtering in the PEP is that 
middleware authors wont take it into consideration when coding. (That's 
why it's optional for servers to implement and includes an example 
middleware implementation of the API.)



--- Async

agronholm: +1 for async wsgi using the new concurrent.futures stdlib feature


agronholm: I still don't like the idea of wsgi.executor \ imho that 
should be left up to the application or framework \ not the web server 
\ and I still disapprove of the wsgi.async flag

GothAlice: The server does, however, need to be able to capture async 
read requests across environ['wsgi.input'].async_read*

GothAlice: What would the semantics be for a worker on a 
single-threaded async server to wait for a long-running task?  Was my 
code example (the simplified try/except block) inadequate?

agronholm: if the app needs to do heavy lifting, it delegates the task 
to a thread/process pool, which returns a future, which the app yields 
back \ when the callback is activated, the reactor will resume 
execution of that app \ I think you pretty much got it right in your 
revised example code

GothAlice: Just replace environ['wsgi.executor'] with an 
application-specific one?

agronholm: essentially, yes \ that would greatly simplify the 
implementation of the interface

GothAlice: And it is all done via done_callbacks? hmm.  For the 
purposes of the callbacks, though, exceptions are ignored.  :/

agronholm: what is your concern with this specifically?

GothAlice: That my desired syntax (try/except around a value=yield 
future) won't be able to capture and elevate exceptions back to the 
WSGI application.

agronholm: oh, that is not a problem since the reactor will call 
.result() on it anyway and send any exceptions back to the application

GothAlice: Back to the environment issue for a moment: not providing an 
executor in the environment means middleware will not be able to 
utilize async features without having their own executor in addition to 
the application's.  How about I explicitly require that servers allow 
overriding of the executor used? \ How often would an application want 
to utilize multiple executors at once?

agronholm: the middleware could have a constructor argument for passing 
an executor

GothAlice: That would then require passing an executor to multiple 
layers of middleware, creating a number of additional references and 
local variables, vs. configuring a "default executor" at the server 
level.

agronholm: there are pros and cons with the wsgi.executor approach

GothAlice: There would be no requirement for the application to use 
wsgi.executor; if an application has a default threaded executor 
(wsgi.executor), it can use a multi-process one for specific jobs 
[ignoring the one in the env] without too much worry.

agronholm: essentially wsgi.executor would be a convenience then

GothAlice: Exactly. \ (And mostly a helper to middleware so they don't 
each need explicit configuration or management of their own executors.)



--- Optional Components

GothAlice: I think full HTTP/1.1 conformance should be a requirement 
for WSGI2 servers, too.  (chunked requests, not just chunked responses) 
\ Because there's really no point in writing a -new- HTTP/1.0 server.  
;)

agronholm: indeed

GothAlice: One thing I've been grappling [while] rewriting PEP 444 is 
that pretty much everything marked 'optional' or 'may' in WSGI 1 / PEP 
333 no developer actually gets around to implementing.  Thus making 
HTTP/1.1 support non-optional [in PEP 444].

GothAlice: Something I've noticed with Python HTTP code: none of it is 
complete, and all of the servers that report HTTP/1.1 compliance 
straight up lie.  Zero I found support chunked response bodies, and 
zero support chunked requests (which is required by HTTP/1.1). \ (The 
servers I looked at universally had embedded comments along the lines 
of: "Chunked responses are left up to application developers.")

GothAlice: If it's too demanding [or appears too daunting], a "may 
implement" feature becomes a "never will be implemented" feature.

agronholm: I would prefer requiring HTTP/1.1 support from all WSGI2 servers

GothAlice: I mean, if I can do it in 172 Python opcodes, I'm certain it 
can't be -that- hard to implement.  ;)



From solipsis at pitrou.net  Thu Jan  6 12:53:14 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 6 Jan 2011 11:53:14 +0000 (UTC)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
Message-ID: <loom.20110106T122356-956@post.gmane.org>

Alice Bevan?McGregor <alice at ...> writes:
> 
> agronholm: what new features does pep 444 propose to add to pep 3333? \ 
> async, filters, no buffering?
> 
> GothAlice: Async, filters, no server-level buffering, native string 
> usage, the definition of "byte string" as "the format returned by 
> socket read" (which, on Java, is unicode!), and the allowance for 
> returned data to be Latin1 Unicode.

Regardless of the rest, I think the latter would be a large step backwards.
Clear distinction between bytes and unicode is a *feature* of Python 3.
Unicode-ignorant programmers should use frameworks which do the encoding work
for them.

(by the way, why you are targeting both Python 2 and 3?)

> agronholm: I'm not very comfortable with the idea of wsgi.input in 
> async apps \ I'm just thinking what would happen when you do 
> environ['wsgi.input'].read()
> 
> GothAlice: One of two things: in a sync environment, it blocks until it 
> can read, in an async environment [combined with yield] it 
> pauses/shelves your application until the data is available.

Er, for the record, in Python 3 non-blocking file objects return None when
read() would block. For example:

>>> r, w = os.pipe()
>>> flags = fcntl.fcntl(r, fcntl.F_GETFL, 0); fcntl.fcntl(r, fcntl.F_SETFL,
flags | os.O_NONBLOCK)
0
>>> os.read(r, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 11] Resource temporarily unavailable
>>> f = open(r, "rb")
>>> f.read(1) is None
True

> agronholm: the requirements of async apps are a big problem
> 
> agronholm: returning magic values from the app sounds like a bad idea
> 
> agronholm: the best solution I can come up with is to have 
> wsgi.async_input or something, which returns an async token for any 
> given read operation

The idiomatic abstraction for non-blockingness under POSIX is file descriptors.
So, at the low level (the WSGI level), exchanging fds between server and app
could be enough to allow both to wake up each other (perhaps two fds: one the
server can wait on, one the app can wait on). Similarly to what signalfd() does.
Then higher-level tools can wrap inside Futures or whatever else.

However, this also means Windows compatibility becomes more complicated, unless
the fds are sockets.

Regards

Antoine.



From chris.dent at gmail.com  Thu Jan  6 14:03:15 2011
From: chris.dent at gmail.com (chris.dent at gmail.com)
Date: Thu, 6 Jan 2011 13:03:15 +0000 (GMT)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig3l9g$lqc$1@dough.gmane.org>
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
Message-ID: <alpine.OSX.2.00.1101061302550.7296@crank.local>


On Wed, 5 Jan 2011, Alice Bevan?McGregor wrote:

> This should give a fairly comprehensive explanation of the rationale behind 
> some decisions in the rewrite; a version of these conversations (in narrative 
> style vs. discussion) will be added to the rewrite Real Soon Now? under the 
> Rationale section.

Thanks for this. I've been trying to follow along with this
conversation as an interested WSGI app developer and admit that much
of the thrust of things is getting lost in the details and people's
tendency to overquote.

One thing that would be useful is if, when you post, Alice, you could
give the URL of whatever and wherever your current draft is.

That out of the way some comments:

For me WSGI is a programmers' aid used to encourage ecapsulation and
separation of concerns in web applications I develop. After that there's
a bit about reuability and portability, but the structure of the
apps/middleware themselves are the most important concerns for me. I
don't use frameworks, or webob or any of that stuff. I just cook up
callables that take environ and start_response. I don't want my
awareness of the basics of HTTP abstracted away, because I want to make
sure that my apps behave well.

Plain WSGI is a good thing, for me, because it means that my
applications are a) very webby (in the stateless HTTP sense) and b)
very testable.

This is all works because WSGI is very simple, so my tendency is to be
resistant to ideas which appear to add complexity.

> --- 444 vs. 3333
> GothAlice: Async, filters, no server-level buffering, native string usage, 
> the definition of "byte string" as "the format returned by socket read" 
> (which, on Java, is unicode!), and the allowance for returned data to be 
> Latin1 Unicode. \ All of this together will allow a '''def hello(environ): 
> return "200 OK", [], ["Hello world!"]''' example application to work across 
> Python versions without modification (or use of b"" prefix)

On async:

I agree with some others who have suggested that maybe async should be
its own thing, rather than integrated into a WSGI2. A server could
choose to be WSGI2 compliant or AWSGI compliant, or both.

Having spent some time messing about with node.js recently, I can say
that the coding style for happy little async apps is great fun, but
actually not what I really want to be doing in my run-of-the-mill as-
RESTful-as-possible web apps.

This might make me a bit of a dinosaur. Or a grape.

That said I can understand why an app author might like to be able to
read or write in an async way, and being able to shelf an app to wait
around for the next cycle would be a good thing. I just don't want
efforts to make that possible to make writing a boring wsgi thing more
annoying.

On filters:

I can't get my head around filters yet. They sound like a different
way to do middleware, with a justification of something along the
lines of "I don't like middleware for filtering". I'd like to be
(directly) pointed at a more robust justification. I suspect you have
already pointed at such a thing, but it is lost in the sands of
time...

Filters seem like something that could be added via a standardized piece
of middleware, rather than being part of the spec. I like minimal specs.

> GothAlice: Latin1 = \u0000 ? \u00FF ? it's one of the only formats that can 
> be decoded while preserving raw bytes, and if another encoding is needed, 
> transcode safely. \ Effectively requiring Latin1 for unicode output ensures 
> single byte conformance on the data. \ If an application needs to return 
> UTF-8, for example, it can return an encoded UTF-8 bytestream, which will be 
> passed right through,

There's a rule of thumb about constraints. If you must constrain, do
none, one or all, never some. Ah, here it is:
http://en.wikipedia.org/wiki/Zero_One_Infinity

Does that apply here? It seems you either allow unicode strings or you
don't, not a certain subsection.

My own personal method is: textual apps _always_ return unicode
producing iterators and a piece of (required, thus not offical by some
people's biases) middleware turns it into UTF-8 on the way out. I've
naively never understood why you want do anything else? My general
rule is unicode inside, UTF-8 at the boundaries.

That's all I got so far. I applaud you for taking on this challenge.
It's work that needs to be done. I hope to be able to comment more and
make a close reading of the various documents, but time is tough
sometimes. I'll do what I can as I can.

Thanks.
-- 
Chris Dent                                   http://burningchrome.com/
                                  [...]

From pje at telecommunity.com  Thu Jan  6 15:07:52 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 06 Jan 2011 09:07:52 -0500
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <alpine.OSX.2.00.1101061302550.7296@crank.local>
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
Message-ID: <20110106140751.6B8083A40A8@sparrow.telecommunity.com>

At 01:03 PM 1/6/2011 +0000, chris.dent at gmail.com wrote:
>Does that apply here? It seems you either allow unicode strings or you
>don't, not a certain subsection.

That's why PEP 3333 requires bytes instead - only the application 
knows what it's sending, and the server and middleware shouldn't have to guess.


>My general rule is unicode inside, UTF-8 at the boundaries.

Which would be easy to enforce if you can only yield bytes, as is the 
case with PEP 3333.

I worry a bit that right now, there may be Python 3.2 servers (other 
than the ones built on wsgiref.handlers) that may not be enforcing 
this rule yet. 


From alice at gothcandy.com  Thu Jan  6 16:27:50 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 07:27:50 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
Message-ID: <ig4n1m$g6j$1@dough.gmane.org>

On 2011-01-06 03:53:14 -0800, Antoine Pitrou said:
> Alice Bevan-??McGregor <alice at ...> writes:
>> GothAlice: ... native string usage, the definition of "byte string" as 
>> "the format returned by socket read" (which, on Java, is unicode!) ...

Just so no-one feels the need to correct me; agronholm made sure I 
didn't drink the kool-aid of one article I was reading and basing some 
ideas on.  Java socket ojects us byte-based buffers, not unicode.  My 
bad!

> Regardless of the rest, I think the latter would be a large step backwards.
> Clear distinction between bytes and unicode is a *feature* of Python 3.
> Unicode-ignorant programmers should use frameworks which do the encoding work
> for them.

+0.5

I'm beginning to agree; with the advent of b'' syntax in 2.6, the only 
compelling reason to include this "feature" (examples that work without 
modification across major versions of Python) goes up in smoke.  The 
examples should use the b'' syntax and have done with it.

> (by the way, why you are targeting both Python 2 and 3?)

For the same reason that Python 3 features are introduced to 2.x; 
migration.  Users are more likely to adopt something that doesn't 
require them to change production environments, and 3.x is far away 
from being deployed in production anywhere but on Gentoo, it seems.  ;)

Broad development and deployment options are a Good Thing?, and with 
b'', there is no reason -not- to target 2.6+.  (There is no requirement 
that a PEP 444 / WSGI 2 server even try to be a cross-compatible 
polygot; there is room for 2.x-specific and 3.x-specific solutions, 
and, in theory, it should be possible to support Python < 2.6, I just 
don't feel it's worthwhile to lock your application into Very Old? 
interpreters.)

>> agronholm: I'm not very comfortable with the idea of wsgi.input in
>> async apps \ I'm just thinking what would happen when you do
>> environ['wsgi.input'].read()
>> 
>> GothAlice: One of two things: in a sync environment, it blocks until it
>> can read, in an async environment [combined with yield] it
>> pauses/shelves your application until the data is available.
> 
> Er, for the record, in Python 3 non-blocking file objects return None when
> read() would block.

-1

I'm aware, however that's not practically useful.  How would you detect 
from within the WSGI 2 application that the file object has become 
readable?  Implement your own async reactor / select / epoll loop?  
That's crazy talk!  ;)

>> agronholm: the requirements of async apps are a big problem
>> 
>> agronholm: returning magic values from the app sounds like a bad idea
>> 
>> agronholm: the best solution I can come up with is to have
>> wsgi.async_input or something, which returns an async token for any
>> given read operation
> 
> The idiomatic abstraction for non-blockingness under POSIX is file descriptors.
> So, at the low level (the WSGI level), exchanging fds between server and app
> could be enough to allow both to wake up each other (perhaps two fds: one the
> server can wait on, one the app can wait on). Similarly to what 
> signalfd() does.
> Then higher-level tools can wrap inside Futures or whatever else.

-0

Hmm; I'll have to mull that over.  Initial thoughts: having a magic 
yield value that combines a fd and operation (read/write) is too 
magical.

> However, this also means Windows compatibility becomes more complicated, unless
> the fds are sockets.

+1 for pure futures which (in theory) eliminate the need for dedicated 
async versions of absolutely everything at the possible cost of 
slightly higher overhead.

	- Alice.



From alice at gothcandy.com  Thu Jan  6 16:44:19 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 07:44:19 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
Message-ID: <ig4o0j$lgv$1@dough.gmane.org>

Chris,

On 2011-01-06 05:03:15 -0800, Chris Dent said:
> On Wed, 5 Jan 2011, Alice Bevan?McGregor wrote:
>> This should give a fairly comprehensive explanation of the rationale 
>> behind > some decisions in the rewrite; a version of these 
>> conversations (in narrative > style vs. discussion) will be added to 
>> the rewrite Real Soon Now? under the > Rationale section.
> 
> Thanks for this. I've been trying to follow along with this 
> conversation as an interested WSGI app developer and admit that much of 
> the thrust of things is getting lost in the details and people's 
> tendency to overquote.

Yeah; I knew the IRC log dump was only so useful.  It's a lot of 
material to go through, and much of it was discussed at strange hours 
with little sleep.  ;)

> One thing that would be useful is if, when you post, Alice, you could 
> give the URL of whatever and wherever your current draft is.

Tomorrow (ack, today!) I'll finish converting over the PEP from Textile 
to ReStructuredText and get it re-submitted to the Python website.

	https://github.com/GothAlice/wsgi2/blob/master/pep444.textile
	http://www.python.org/dev/peps/pep-0444/

> I don't use frameworks, or webob or any of that stuff. I just cook up 
> callables that take environ and start_response. I don't want my 
> awareness of the basics of HTTP abstracted away, because I want to make
> sure that my apps behave well.

Kudos!  That approach is heavily frowned upon in the #python IRC 
channel, but I fully agree that working solutions can be reasonably 
made using that methedology.  There are some details that are made 
easier by frameworks, though.  Testing benefits from MVC: you can test 
the dict return value of the controller, the templates, and the model 
all separately.

> Plain WSGI is a good thing, for me, because it means that my 
> applications are a) very webby (in the stateless HTTP sense) and b) 
> very testable.

c) And very portable.  You need not depend on some pre-arranged stack 
(including web server).

> I agree with some others who have suggested that maybe async should be 
> its own thing, rather than integrated into a WSGI2. A server could 
> choose to be WSGI2 compliant or AWSGI compliant, or both.

-1

That is already the case with filters, and will be when I ratify the 
async idea (after further discussion here).  My current thought process 
is that async will be optional for server implementors and will be 
easily detectable by applications and middleware and have zero impact 
on middleware/applications if disabled (by configuration) or missing.

> That said I can understand why an app author might like to be able to 
> read or write in an async way, and being able to shelf an app to wait 
> around for the next cycle would be a good thing.

Using futures, async covers any callable at all; you can queue up a 
dozen DB calls at the top of your application, then (within a body 
generator) yield those futures to be paused pending the data.  That 
would, as an example, allow complex pages to be generated and streamed 
to the end-user in a efficient way -- the user would see a page begin 
to appear, and the browser downloading static resources, while 
intensive tasks complete.

> I just don't want efforts to make that possible to make writing a 
> boring wsgi thing more
> annoying.

+9001

See above.

> I can't get my head around filters yet. They sound like a different way 
> to do middleware, with a justification of something along the lines of 
> "I don't like middleware for filtering". I'd like to be (directly) 
> pointed at a more robust justification. I suspect you have already 
> pointed at such a thing, but it is lost in the sands of time...

Filters offer several benefits, some of which are mild:

:: Simplified application / middleware debugging via smaller stack.
:: Clearly defined tasks; ingress = altering the environ / input, 
egress = altering the output.
:: Egress filters are not executed if an unhandled exception is raised.

The latter point is important; you do not want badly written middleware 
to absorb exceptions that should bubble, etc.  (I'll need to elaborate 
on this and add a few more points when I get some sleep.)

> Filters seem like something that could be added via a standardized 
> piece of middleware, rather than being part of the spec. I like minimal 
> specs.

Filters are optional, and an example is/will be provided for utilizing 
ingress/egress filter stacks as middleware.

The problem with /not/ including the filtering API (which, by itself is 
stupidly simple and would barely warrant its own PEP, IMHO) is that a 
separate standard would not be seen and taken into consideration when 
developers are writing what they will think /must/ be middleware.  
Seing as a middleware version of a filter is trivial to create (just 
execute the filter in a thin middleware wrapper), it should be a 
consideration up front.

>> Latin1 = \u0000 ? \u00FF [snip]
> 
> There's a rule of thumb about constraints. If you must constrain, do 
> none, one or all, never some. Ah, here it is: 
> http://en.wikipedia.org/wiki/Zero_One_Infinity
> 
> Does that apply here? It seems you either allow unicode strings or you 
> don't, not a certain subsection.

+1

See my post from a few minutes ago which covers this.

> That's all I got so far. I applaud you for taking on this challenge. 
> It's work that needs to be done. I hope to be able to comment more and 
> make a close reading of the various documents, but time is tough 
> sometimes. I'll do what I can as I can.

Thank you, and I look forward to additional input!

	- Alice.



From chris.dent at gmail.com  Thu Jan  6 18:06:10 2011
From: chris.dent at gmail.com (chris.dent at gmail.com)
Date: Thu, 6 Jan 2011 17:06:10 +0000 (GMT)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig4o0j$lgv$1@dough.gmane.org>
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<ig4o0j$lgv$1@dough.gmane.org>
Message-ID: <alpine.OSX.2.00.1101061705390.8190@crank.local>


On Thu, 6 Jan 2011, Alice Bevan?McGregor wrote:

> Yeah; I knew the IRC log dump was only so useful.  It's a lot of material to 
> go through, and much of it was discussed at strange hours with little sleep. 
> ;)

I wasn't actually talking about the log dump. That was useful. What I
was talking about were earlier messages in the thread where people
were making responses, quoting vast swaths of text for no clear
reason.

> 	https://github.com/GothAlice/wsgi2/blob/master/pep444.textile

Thanks, watching that now.

>> I don't use frameworks, or webob or any of that stuff. I just cook up 
>> callables that take environ and start_response. I don't want my awareness 
>> of the basics of HTTP abstracted away, because I want to make
>> sure that my apps behave well.
>
> Kudos!  That approach is heavily frowned upon in the #python IRC channel, but 
> I fully agree that working solutions can be reasonably made using that 
> methedology.  There are some details that are made easier by frameworks, 
> though.  Testing benefits from MVC: you can test the dict return value of the 
> controller, the templates, and the model all separately.

I should have been more explicit here as I now feel I must defend
myself from frowns. I'm not talking about single methods that do the
entire app. I nest a series of middleware that bottom out at Selector
which then does url based dispatch to applications, which themselves
are defined as handlers (simple wsgi functions) and access
StorageInterfaces and Serializations. The middleware, handlers, stores
and serializers are all independently testable (and usable).

I find the MVC language ineffective when thinking about the HTTP
verbs and the resources the app(s) present(s). In fact I think it
encourages _bad_ thinking.

Anyway, the point of the methodolgy is, that from the perspective of
the web data, the entry and exit points are unadulterated WSGI.

> That is already the case with filters, and will be when I ratify the async 
> idea (after further discussion here).  My current thought process is that 
> async will be optional for server implementors and will be easily detectable 
> by applications and middleware and have zero impact on 
> middleware/applications if disabled (by configuration) or missing.

This notion of being detectable seems weird to me. Are we actually
expecting an application to query the server, find out it is not async
capable, and choose a different code path as a result? Seems much more
likely that the installer will choose a server or app that meets their
needs. That is: you don't need to detect, you need to know (presumably
at install/config time).

Or maybe I am imagining the use cases incorrectly here. I think of app
being async as an explicit choice made by the builder to achieve some
goal.

>> I can't get my head around filters yet.[snip]
>
> Filters offer several benefits, some of which are mild:
>
> :: Simplified application / middleware debugging via smaller stack.
> :: Clearly defined tasks; ingress = altering the environ / input, egress = 
> altering the output.
> :: Egress filters are not executed if an unhandled exception is raised.

Taken individually none of these seem super critical to me.

Or to put it another way: Yeah, so?

(This is the aforementioned resistance showing through. The above
sounds perfectly nice, reasonable and desireable, but not
_necessary_.)

> Filters are optional, and an example is/will be provided for utilizing 
> ingress/egress filter stacks as middleware.

In a conversation with some people about the Atom Publishing Protocol
I tried to convince them that the terms SHOULD and MAY had no place in
a spec. WSGI* is not really the same kind of spec, but optionality
still grates in the same way.

> The problem with /not/ including the filtering API (which, by itself is 
> stupidly simple and would barely warrant its own PEP, IMHO) is that a 
> separate standard would not be seen and taken into consideration when 
> developers are writing what they will think /must/ be middleware.

Yeah, so? :)

> See my post from a few minutes ago which covers this.

Yay!

-- 
Chris Dent                                   http://burningchrome.com/
                                  [...]

From rsyring at inteli-com.com  Thu Jan  6 18:20:48 2011
From: rsyring at inteli-com.com (Randy Syring)
Date: Thu, 06 Jan 2011 12:20:48 -0500
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig4o0j$lgv$1@dough.gmane.org>
References: <ig3erb$vll$1@dough.gmane.org>
	<ig3l9g$lqc$1@dough.gmane.org>	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<ig4o0j$lgv$1@dough.gmane.org>
Message-ID: <4D25F9F0.3070306@inteli-com.com>

Alice,

Being a web application developer and relying on frameworks like 
Werkzeug and WebOb, I may not have much of a dog in this fight.  
However, I have been following web-sig for a couple years and I have 
seen the difficulties involved in reaching consensus on 
modifying/updating the WSGI spec.  Its clear to me that most people on 
this list who can contribute in meaningful ways to the creation of WSGI 
2 have very little time to do so.  Jobs, family, and life in general 
limit the amount of time that can be spent on something like this.  
Motivation seems generally low anyway, because what we have currently 
works.  It may have warts, but it works, and that very fact seems to 
limit the number of people interested in donating time to improving the 
spec.

So, getting a WSGI 2 spec discussed and approved is going to be hard 
enough.  Every time something controversial is added to the spec, its 
going to make it that much harder to move forward.  I'm not saying that 
means all controversial items should be dropped, some things are worth 
fighting for, just pointing out that IMO we are already working uphill, 
and adding weights to our rucksacks should only be done when absolutely 
necessary.

On 01/06/2011 10:44 AM, Alice Bevan?McGregor wrote:
>
> On 2011-01-06 05:03:15 -0800, Chris Dent said:
>> I agree with some others who have suggested that maybe async should 
>> be its own thing, rather than integrated into a WSGI2. A server could 
>> choose to be WSGI2 compliant or AWSGI compliant, or both.
Adding async to the spec is a death blow IMO.  You gain nothing by 
putting it in and lose a lot of interest and time spent discussing it.  
Make it a separate PEP that references the first.  That way, those who 
don't really care about it can still work on WSGI 2 without the 
distraction of the async parts.  If you make the new async PEP dependent 
on the WSGI 2 spec, then those ideas can be tossed around all day long 
without distracting from or taking energy away from the core WSGI 2 ideas.

So, I agree with Chris and others who have said async should be a 
separate PEP.

-1 on having async in PEP 444 / WSGI 2

With respect to filters:

On 12/14/2010 04:25 PM, Ian Bicking wrote:
> <...>
> GzipFilter is wonky at best (it interacts oddly with range requests 
> and etags).  Prefix handling is useful (e.g., 
> paste.deploy.config.PrefixMiddleware), and usually global and 
> unconfigured.  Debugging and logging stuff often needs per-path 
> configuration, which can mean multiple instances applied after 
> dispatch.  Encoding and Decoding don't apply to WSGI.  Tidy is 
> intrusive and I think questionable on a global level.  I don't think 
> the use cases are there.  Tightly bound pre-filters and post-filters 
> are particularly problematic.  This all seems like a lot of work to 
> avoid a few stack frames in a traceback.
>

I agree with Ian's analysis of filters.  I don't see the benefit and its 
just another item to detract from other core issues that could be 
addressed.  -1 on filters.

Alice, I do appreciate the time you are giving this issue.  But my 
feeling so far is that the things you have focused on are not the things 
that concern most of the people pushing towards a WSGI 2.  On January 
2nd, Phillip Eby sent three emails to web-sig.  IMO, they had great 
wisdom concerning different parts of WSGI 2, async, and the political 
aspects of the PEP process.  Your approach so far doesn't seem to have 
benefited from that wisdom, especially regarding the latter two items, 
and IMO this ship is dead in the water until that changes.

What I mean is, your approach just doesn't seem to take the history of 
the web-sig and the main contributor's opinions into account enough.  I 
have been on the web-sig long enough to have respect for the opinions of 
guys like Ian, Graham, Phillip, Armin, etc.  They have shown their 
competence and their care through their contributions via code and via 
posts to this list.  I have personally benefited greatly by being able 
to use the code they have built and I trust them and their opinions with 
regards to issues discussed on web-sig.  So when I see Ian write the 
above about filters or I see Phillip write the following about async:

> I suggest reviewing the Web-SIG history of previous async discussions; 
> there's a lot more to having a meaningful API spec than having a 
> plausible approach.  It's not that there haven't been past proposals, 
> they just couldn't get as far as making it possible to write a 
> non-trivial async application that would actually be portable among 
> Python-supporting asynchronous web servers. 
and I don't see you respond to their concerns/suggestions, it makes it 
difficult for me to trust the direction you are heading.

Overall, you have shown energy and a willingness to contribute, which I 
greatly appreciate.  But, I have the opinion that you are coming into 
this process ignorant of or mostly disregarding the many discussions 
that have taken place before on this list, are pushing an agenda that 
seems mostly defined by your likes and dislikes, and are mostly 
disregarding the suggestions/concerns of the list "heavy-weights".  That 
opinion may not be true, in fact, *I am not* even saying that is what is 
going on.  What I am saying is that this is what it *appears* like to a 
no-name follower of the web-sig.  We only see what you write here, the 
burden of proof is on you to communicate your attentions and agenda.  
That may not be fair, but if life were to suddenly get fair, I doubt it 
would happen on web-sig... :)

Then again, my opinion and impression could be completely off, and if 
that is the case, feel free to ignore me.  :)

--------------------------------------
Randy Syring
Intelicom
Direct: 502-276-0459
Office: 502-212-9913

For the wages of sin is death, but the
free gift of God is eternal life in
Christ Jesus our Lord (Rom 6:23)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110106/02a09e73/attachment.html>

From eric at ionrock.org  Thu Jan  6 19:02:33 2011
From: eric at ionrock.org (Eric Larson)
Date: Thu, 06 Jan 2011 12:02:33 -0600
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <alpine.OSX.2.00.1101061302550.7296@crank.local>
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
Message-ID: <87y66xlxl2.wl%eric@ionrock.org>

At Thu, 6 Jan 2011 13:03:15 +0000 (GMT),
chris dent wrote:
> <snip>
> On async:
> 
> I agree with some others who have suggested that maybe async should be
> its own thing, rather than integrated into a WSGI2. A server could
> choose to be WSGI2 compliant or AWSGI compliant, or both.
> </snip>

+1

After seeing some of the ideas regarding how to add async into a new
version of WSGI, it isn't the specific problem the async feature
addresses in terms of WSGI servers. Is the goal is to support long
running connections? Are we trying to support WebSockets and other
long running connection interfaces? If that is the case, async is a
*technique* for handling this paradigm, but it doesn't address the
real problem. There are techniques that have sounded reasonable like
making available the socket such that a server can give it to the
application to do something use with it (here is an example doing
something similar with CherryPy - http://trac.defuze.org/browser/oss/ws4cp/ws4cp.py).

Just to summarize, I'm for making async something else while finding a
way to support long running connections in WSGI outside of adopting a
particular technique a potentially viable goal.

Just my $.02 on the issue.

Eric Larson

From solipsis at pitrou.net  Thu Jan  6 19:15:19 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 6 Jan 2011 18:15:19 +0000 (UTC)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
Message-ID: <loom.20110106T185706-961@post.gmane.org>

Alice Bevan?McGregor <alice at ...> writes:
> > Er, for the record, in Python 3 non-blocking file objects return None when
> > read() would block.
> 
> -1
> 
> I'm aware, however that's not practically useful.  How would you detect 
> from within the WSGI 2 application that the file object has become 
> readable?  Implement your own async reactor / select / epoll loop?  
> That's crazy talk!  ;)

I was just pointing out that if you need to choose a convention for signaling
blocking reads on a non-blocking object, it's already there.

By the way, an event loop is the canonical implementation of asynchronous
programming, so I'm not sure what you're complaining about. Or perhaps you're
using "async" in a different meaning? (which one?)

> >> agronholm: the requirements of async apps are a big problem
> >> 
> >> agronholm: returning magic values from the app sounds like a bad idea
> >> 
> >> agronholm: the best solution I can come up with is to have
> >> wsgi.async_input or something, which returns an async token for any
> >> given read operation
> > 
> > The idiomatic abstraction for non-blockingness under POSIX is file
descriptors.
> > So, at the low level (the WSGI level), exchanging fds between server and
app
> > could be enough to allow both to wake up each other (perhaps two fds: one
the
> > server can wait on, one the app can wait on). Similarly to what 
> > signalfd() does.
> > Then higher-level tools can wrap inside Futures or whatever else.
> 
> -0
> 
> Hmm; I'll have to mull that over.  Initial thoughts: having a magic 
> yield value that combines a fd and operation (read/write) is too 
> magical.

I don't understand why you want a "yield" at this level. IMHO, WSGI needn't
involve generators. A higher-level wrapper (framework, middleware, whatever) can
wrap fd-waiting in fancy generator stuff if so desired. Or, in some other
environments, delegate it to a reactor with callbacks and deferreds. Or whatever
else, such as futures.

By the way, the concurrent.futures module is new. Though it will be there in
3.2, it's not guaranteed that its API and semantics will be 100% stable while
people start to really flesh it out.

> +1 for pure futures which (in theory) eliminate the need for dedicated 
> async versions of absolutely everything at the possible cost of 
> slightly higher overhead.

I don't understand why futures would solve the need for a low-level async
facility. You still need to define a way for the server and the app to wake each
other (and for the server to wake multiple apps). This isn't done "naturally" in
Python (except perhaps with stackless or greenlets). Using fds give you
well-known flexible possibilities.

If you want to put the futures API in WSGI, think of the poor authors of a WSGI
server written in C who will have to write their own executor and future
implementation. I'm sure they have better things to do.

Regards

Antoine.



From tfarrell at owassobible.org  Thu Jan  6 20:36:17 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Thu, 6 Jan 2011 13:36:17 -0600 (CST)
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <1928823.0.1294335538137.JavaMail.root@boaz>
Message-ID: <11174894.6.1294342577349.JavaMail.root@boaz>

Hello web-sig.  My name is Timothy Farrell.  I'm the developer of the Rocket web server.  I understand that most of you are more experienced and passionate than myself.  But I'm come here because I want to see certain things standardized.  I'm pretty new to this forum but I've read through all the recent discussions on PEP 444.  That being said, I'll try to take a humble approach.

It seems to me that the spec that Alice is working on could be something great but the problems are not well defined (in the PEP).  This causes confusion about what the goals are.  There's some disagreement about whether or not certain features should be in PEP 444.  I think those people have a different idea for what PEP 444 ought to be.  The first thing that should be done is clearly defining the shortcomings with PEP 3333 that PEP 444 seeks to address and limit our PEP 444 discussions to solving those problems.

Since Alice is rewriting the PEP perhaps we should all sit back for a second until we have a PEP to work off of.  That will help the discussion be a little more focused.

Sorry if I've stepped on anyone's toes.

-tim

From alice at gothcandy.com  Thu Jan  6 20:59:11 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 11:59:11 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<ig4o0j$lgv$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061705390.8190@crank.local>
Message-ID: <ig56uf$9tr$1@dough.gmane.org>

On 2011-01-06 09:06:10 -0800, 
chris.dent at gmail.com said:
> I wasn't actually talking about the log dump. That was useful. What I 
> was talking about were earlier messages in the thread where people were 
> making responses, quoting vast swaths of text for no clear reason.

Ah.  :)  I do make an effort to trim quoted text to only the relevant parts.

> On Thu, 6 Jan 2011, Alice Bevan?McGregor wrote:
>> 	https://github.com/GothAlice/wsgi2/blob/master/pep444.textile
> 
> Thanks, watching that now.

The textile document will no longer be updated; the pep-444.rst 
document is where it'll be at.

> I should have been more explicit here as I now feel I must defend 
> myself from frowns. I'm not talking about single methods that do the 
> entire app. I nest a series of middleware that bottom out at Selector 
> which then does url based dispatch to applications, which themselves 
> are defined as handlers (simple wsgi functions) and access 
> StorageInterfaces and Serializations. The middleware, handlers, stores 
> and serializers are all independently testable (and usable).

*nods* My framework (WebCore) is basically a packaged up version of a 
custom middleware stack so I can easily re-use it from project to 
project.  I assumed (in my head) you were "rolling your own" 
framework/stack.

>> That is already the case with filters, and will be when I ratify the 
>> async idea (after further discussion here).  My current thought process 
>> is that async will be optional for server implementors and will be 
>> easily detectable by applications and middleware and have zero impact 
>> on middleware/applications if disabled (by configuration) or missing.
> 
> This notion of being detectable seems weird to me. Are we actually 
> expecting an application to query the server, find out it is not async 
> capable, and choose a different code path as a result? Seems much more
> likely that the installer will choose a server or app that meets their 
> needs. That is: you don't need to detect, you need to know (presumably 
> at install/config time).
> 
> Or maybe I am imagining the use cases incorrectly here. I think of app 
> being async as an explicit choice made by the builder to achieve some 
> goal.

More to the point it needs to be detectable by middleware without 
explicitly configuring every layer of middleware, potentially with 
differing configuration mechanics and semantics.  (I.e. arguments like 
enable_async, async_enable, iLoveAsync, ...)

>>> I can't get my head around filters yet.[snip]
>> 
>> Filters offer several benefits, some of which are mild:
>> 
>> :: Simplified application / middleware debugging via smaller stack.
>> :: Clearly defined tasks; ingress = altering the environ / input, 
>> egress = > altering the output.
>> :: Egress filters are not executed if an unhandled exception is raised.
> 
> Taken individually none of these seem super critical to me.
> 
> Or to put it another way: Yeah, so?
> 
> (This is the aforementioned resistance showing through. The above 
> sounds perfectly nice, reasonable and desireable, but not _necessary_.)

It isn't necessary; it is, however, an often re-implemented feature of 
a framework on top of WSGI.  CherryPy, Paste, Django, etc. all 
implement some form of non-WSGI (or, hell, Paste uses WSGI middleware) 
thing they call a 'filter'.

>> Filters are optional, and an example is/will be provided for utilizing 
>> > ingress/egress filter stacks as middleware.
> 
> In a conversation with some people about the Atom Publishing Protocol I 
> tried to convince them that the terms SHOULD and MAY had no place in a 
> spec. WSGI* is not really the same kind of spec, but optionality
> still grates in the same way.

I fully agree; that's why a lot of the PEP 333 "optionally" or "may" 
features have become "must".  "Optionally" and "may" simply never get 
implemented.

Filters are optional because a number of people have raised valid 
arguments that it might not be entirely needed.  Thus, it's not 
required.  But I strongly feel that some defined API should be present 
in (or /at least/ referred to by) the PEP, otherwise the future will 
hold the same server-specific incompatible implementations.

	- Alice.



From alice at gothcandy.com  Thu Jan  6 21:52:29 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 12:52:29 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
Message-ID: <ig5a2d$r5s$1@dough.gmane.org>

> It seems to me that the spec that Alice is working on could be 
> something great but the problems are not well defined (in the PEP).  
> This causes confusion about what the goals are.

For completeness sake, here's a slightly simplified Abstract:

:: A proposed second-generation standard interface between web servers 
and Python 2.6+ and 3.1+ applications.

The rationale for even having such an interface is outlined in PEP 333.

Ignoring async for the moment, the goals of the PEP 444 rewrite are:

:: Clear separation of "narrative" from "rules to be followed".  This 
allows developers of both servers and applications to easily run 
through a confomance "check list".

:: Isolation of examples and rationale to improve readability of the 
core rulesets.

:: Clarification of often mis-interpreted rules from PEP 333 (and those 
carried over in 3333).

:: Elimination of unintentional non-conformance, esp. re: cgi.FieldStorage.

:: Massive simplification of call flow.  Replacing start_response with 
a returned 3-tuple immensely simplifies the task of middleware that 
needs to capture HTTP status or manipulate (or even examine) response 
headers. [1]

:: Reduction of re-implementation / NIH syndrome by incorporating the 
most common (1%) of features most often relegated to middleware or 
functional helpers.  Unicode decoding of a small handful of values (CGI 
values that pull from the request URI) is the biggest example. [2, 3]

:: Cross-compatibility considerations.  The definition and use of 
native strings vs. byte strings is the biggest example of this in the 
rewrite.

:: Making optional (and thus rarely-implemented) features non-optional. 
 E.g. server support for HTTP/1.1 with clarifications for interfacing 
applications to 1.1 servers.  Thus pipelining, chunked encoding, et. 
al. as per the HTTP 1.1 RFC.

There are likely others I can't think of at the moment.  ;)  If I 
remember anything else as I wake up more fully (caffeine zombie, here) 
I'll post an additional reply.

Footnotes:

[1] This also happens to be a very Pythonic solution.

[2] This does not mean WSGI 2 will attempt to "compete" with 
frameworks; merely reduce the multiplication of effort for the common 
denominator.

[3] Filters are covered under re-implementation.

> Since Alice is rewriting the PEP perhaps we should all sit back for a 
> second until we have a PEP to work off of.  That will help the 
> discussion be a little more focused.

I'll have a direct translation of my current rewritten draft into ReST 
for incorporation on the Python.org website within a few hours.  
Unfortunately, in the short term, it still doesn't include a high-level 
goal overview, though will incorporate the consensus (thus far) on 
removing the ability to return unicode response data.

> Sorry if I've stepped on anyone's toes.

No worries; you do raise a very valid point.

	- Alice.



From alex.gronholm at nextday.fi  Thu Jan  6 21:48:21 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Thu, 06 Jan 2011 22:48:21 +0200
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <87y66xlxl2.wl%eric@ionrock.org>
References: <ig3erb$vll$1@dough.gmane.org>
	<ig3l9g$lqc$1@dough.gmane.org>	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<87y66xlxl2.wl%eric@ionrock.org>
Message-ID: <4D262A95.5070703@nextday.fi>

06.01.2011 20:02, Eric Larson kirjoitti:
> At Thu, 6 Jan 2011 13:03:15 +0000 (GMT),
> chris dent wrote:
>> <snip>
>> On async:
>>
>> I agree with some others who have suggested that maybe async should be
>> its own thing, rather than integrated into a WSGI2. A server could
>> choose to be WSGI2 compliant or AWSGI compliant, or both.
>> </snip>
> +1
>
> After seeing some of the ideas regarding how to add async into a new
> version of WSGI, it isn't the specific problem the async feature
> addresses in terms of WSGI servers. Is the goal is to support long
> running connections? Are we trying to support WebSockets and other
> long running connection interfaces? If that is the case, async is a
> *technique* for handling this paradigm, but it doesn't address the
> real problem. There are techniques that have sounded reasonable like
> making available the socket such that a server can give it to the
> application to do something use with it (here is an example doing
> something similar with CherryPy - http://trac.defuze.org/browser/oss/ws4cp/ws4cp.py).
The primary idea behind asynchronous servers/applications is the ability 
to efficiently serve a huge number of concurrent connections with a 
small number of threads. Asynchronous applications tend to be faster 
because there is less thread context switching happening in the CPU. Any 
application that runs on top of a web server that allocates less threads 
to the application than the number of connections has to be quick to 
respond so as not to starve the thread pool or block the event loop. 
This is true regardless of whether nonblocking I/O or some other 
technique is used. I'm a bit unclear as to how else you would do this. 
Care to elaborate on that? I looked at the Cherrypy code, but I couldn't 
yet figure that out.
> Just to summarize, I'm for making async something else while finding a
> way to support long running connections in WSGI outside of adopting a
> particular technique a potentially viable goal.
>
> Just my $.02 on the issue.
>
> Eric Larson
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From foom at fuhm.net  Thu Jan  6 22:06:36 2011
From: foom at fuhm.net (James Y Knight)
Date: Thu, 6 Jan 2011 16:06:36 -0500
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig5a2d$r5s$1@dough.gmane.org>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
Message-ID: <736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>

On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
> :: Making optional (and thus rarely-implemented) features non-optional. E.g. server support for HTTP/1.1 with clarifications for interfacing applications to 1.1 servers.  Thus pipelining, chunked encoding, et. al. as per the HTTP 1.1 RFC.

Requirements on the HTTP compliance of the server don't really have any place in the WSGI spec. You should be able to be WSGI compliant even if you don't use the HTTP transport at all (e.g. maybe you just send around requests via SCGI). 

The original spec got this right: chunking etc are something which is not relevant to the wsgi application code -- it is up to the server to implement the HTTP transport according to the HTTP spec, if it's purporting to be an HTTP server. 

James

From fumanchu at aminus.org  Thu Jan  6 22:08:04 2011
From: fumanchu at aminus.org (Robert Brewer)
Date: Thu, 6 Jan 2011 13:08:04 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig56uf$9tr$1@dough.gmane.org>
References: <ig3erb$vll$1@dough.gmane.org>
	<ig3l9g$lqc$1@dough.gmane.org><alpine.OSX.2.00.1101061302550.7296@crank.local><ig4o0j$lgv$1@dough.gmane.org><alpine.OSX.2.00.1101061705390.8190@crank.local>
	<ig56uf$9tr$1@dough.gmane.org>
Message-ID: <F1962646D3B64642B7C9A06068EE1E64101E55CB@ex10.hostedexchange.local>

Alice Bevan?McGregor wrote:
> chris.dent at gmail.com said:
> > I can't get my head around filters yet...
> 
> It isn't necessary; it is, however, an often re-implemented feature of
> a framework on top of WSGI.  CherryPy, Paste, Django, etc. all
> implement some form of non-WSGI (or, hell, Paste uses WSGI middleware)
> thing they call a 'filter'.

Or, if you had actually read what I wrote weeks ago, you'd say "CherryPy used to have a thing they call a 'filter', but then replaced it with a much better mechanism ("hooks and tools") once the na?ve categories of ingress/egress were shown in practice to be inadequate." Not to mention that, even when CherryPy had something called a 'filter', that it not only predated WSGI but ran at the innermost WSGI layer, not the outermost. It's apples and oranges at best, or reinventing the square wheel at worst.

We don't need Yet Another Way of hooking in processing components; if anything, we need a standard mechanism to compose existing middleware graphs so that invariant orderings are explicit and guaranteed. For example, "encode, then gzip, then cache". By introducing egress filters as described in PEP 444 (which mentions gzip as a candidate for an egress filter), you're then stuck in a tug-of-war as to whether to build a new caching component as middleware, as an egress filter, or (most likely, in order to compete) both.


Robert Brewer
fumanchu at aminus.org

From sh at defuze.org  Thu Jan  6 22:11:33 2011
From: sh at defuze.org (Sylvain Hellegouarch)
Date: Thu, 6 Jan 2011 22:11:33 +0100
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <4D262A95.5070703@nextday.fi>
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<87y66xlxl2.wl%eric@ionrock.org> <4D262A95.5070703@nextday.fi>
Message-ID: <AANLkTimWJJJG+4B1j+_QTE1BXynwujyzU-m6kV5=oRUW@mail.gmail.com>

2011/1/6 Alex Gr?nholm <alex.gronholm at nextday.fi>

> 06.01.2011 20:02, Eric Larson kirjoitti:
>
>> At Thu, 6 Jan 2011 13:03:15 +0000 (GMT),
>> chris dent wrote:
>>
>>> <snip>
>>> On async:
>>>
>>> I agree with some others who have suggested that maybe async should be
>>> its own thing, rather than integrated into a WSGI2. A server could
>>> choose to be WSGI2 compliant or AWSGI compliant, or both.
>>> </snip>
>>>
>> +1
>>
>> After seeing some of the ideas regarding how to add async into a new
>> version of WSGI, it isn't the specific problem the async feature
>> addresses in terms of WSGI servers. Is the goal is to support long
>> running connections? Are we trying to support WebSockets and other
>> long running connection interfaces? If that is the case, async is a
>> *technique* for handling this paradigm, but it doesn't address the
>> real problem. There are techniques that have sounded reasonable like
>> making available the socket such that a server can give it to the
>> application to do something use with it (here is an example doing
>> something similar with CherryPy -
>> http://trac.defuze.org/browser/oss/ws4cp/ws4cp.py).
>>
> The primary idea behind asynchronous servers/applications is the ability to
> efficiently serve a huge number of concurrent connections with a small
> number of threads. Asynchronous applications tend to be faster because there
> is less thread context switching happening in the CPU. Any application that
> runs on top of a web server that allocates less threads to the application
> than the number of connections has to be quick to respond so as not to
> starve the thread pool or block the event loop. This is true regardless of
> whether nonblocking I/O or some other technique is used. I'm a bit unclear
> as to how else you would do this. Care to elaborate on that? I looked at the
> Cherrypy code, but I couldn't yet figure that out.


Since I wrote that piece of code, I guess I ought to chime in. First of all,
the code isn't part of CherryPy, simply it's one idea to provide WebSocket
to CherryPy. Considering WebSocket bootstraps on HTTP but once that's done,
it's just a raw socket with bits and pieces on top, I wanted to find a way
not to block CherryPy from serving other requests once a WebSocket handshake
had been performed. The idea was simply to detach the socket from the worker
thread once the handshake had been performed. Then the application had a
socket at hand and this particular instance, I simply decided to use
asyncore to loop through those sockets aside from the CherryPy HTTP
server. In effect, you end up with asyncore for WS sockets and CherryPy for
any HTTP serving but from within one single process, using CherryPy's main
loop.

By large this is not a generic solution for implementing async in WSGI but a
specific example on how one can have both threads and an async loop playing
together. It's merely a proof of concept :)

Hope that clarifies that piece of code.
-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110106/f0921572/attachment.html>

From alice at gothcandy.com  Thu Jan  6 22:56:09 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 13:56:09 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
Message-ID: <ig5dpo$eui$1@dough.gmane.org>

On 2011-01-06 13:06:36 -0800, James Y Knight said:

> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>> :: Making optional (and thus rarely-implemented) features non-optional. 
>> E.g. server support for HTTP/1.1 with clarifications for interfacing 
>> applications to 1.1 servers.  Thus pipelining, chunked encoding, et. 
>> al. as per the HTTP 1.1 RFC.
> 
> Requirements on the HTTP compliance of the server don't really have any 
> place in the WSGI spec. You should be able to be WSGI compliant even if 
> you don't use the HTTP transport at all (e.g. maybe you just send 
> around requests via SCGI).
> The original spec got this right: chunking etc are something which is 
> not relevant to the wsgi application code -- it is up to the server to 
> implement the HTTP transport according to the HTTP spec, if it's 
> purporting to be an HTTP server.

Chunking is actually quite relevant to the specification, as WSGI and 
PEP 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) 
allow for chunked bodies regardless of higher-level support for 
chunking.  The body iterator.  Previously you /had/ to define a length, 
with chunked encoding at the server level, you don't.

I agree, however, that not all gateways will be able to implement the 
relevant HTTP/1.1 features.  FastCGI does, SCGI after a quick Google 
search, seems to support it as well. I should re-word it as:

"For those servers capable of HTTP/1.1 features the implementation of 
such features is required."

+1

	- Alice.



From foom at fuhm.net  Thu Jan  6 23:01:09 2011
From: foom at fuhm.net (James Y Knight)
Date: Thu, 6 Jan 2011 17:01:09 -0500
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig5dpo$eui$1@dough.gmane.org>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
Message-ID: <6D1FBE4B-46CB-41D7-B5DB-6374B007B661@fuhm.net>


On Jan 6, 2011, at 4:56 PM, Alice Bevan?McGregor wrote:

> On 2011-01-06 13:06:36 -0800, James Y Knight said:
> 
>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>> :: Making optional (and thus rarely-implemented) features non-optional. E.g. server support for HTTP/1.1 with clarifications for interfacing applications to 1.1 servers.  Thus pipelining, chunked encoding, et. al. as per the HTTP 1.1 RFC.
>> Requirements on the HTTP compliance of the server don't really have any place in the WSGI spec. You should be able to be WSGI compliant even if you don't use the HTTP transport at all (e.g. maybe you just send around requests via SCGI).
>> The original spec got this right: chunking etc are something which is not relevant to the wsgi application code -- it is up to the server to implement the HTTP transport according to the HTTP spec, if it's purporting to be an HTTP server.
> 
> Chunking is actually quite relevant to the specification, as WSGI and PEP 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for chunked bodies regardless of higher-level support for chunking.  The body iterator.  Previously you /had/ to define a length, with chunked encoding at the server level, you don't.

No you don't -- HTTP 1.0 allows indeterminate-length output. The server simply must close the connection to indicate the end of the response if either the client version HTTP/1.0, or the server doesn't implement HTTP/1.1. 

James

From alice at gothcandy.com  Thu Jan  6 22:59:54 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 13:59:54 -0800
Subject: [Web-SIG] WSGI Middleware Dependancy Graphing (was: PEP 444 / WSGI
	2 Async)
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<ig4o0j$lgv$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061705390.8190@crank.local>
	<ig56uf$9tr$1@dough.gmane.org>
	<F1962646D3B64642B7C9A06068EE1E64101E55CB@ex10.hostedexchange.local>
Message-ID: <ig5e0q$fs0$1@dough.gmane.org>

On 2011-01-06 13:08:04 -0800, Robert Brewer said:

> Or, if you had actually read what I wrote weeks ago...

I did.  Apologies for forgetting the detail of the implementation being 
deprecated.

> We don't need Yet Another Way of hooking in processing components; if 
> anything, we need a standard mechanism to compose existing middleware 
> graphs so that invariant orderings are explicit and guaranteed. For 
> example, "encode, then gzip, then cache". By introducing egress filters 
> as described in PEP 444 (which mentions gzip as a candidate for an 
> egress filter), you're then stuck in a tug-of-war as to whether to 
> build a new caching component as middleware, as an egress filter, or 
> (most likely, in order to compete) both.

I do, in fact, have a proposal for declaring dependancies, however such 
declaration is utterly useless unless differing middleware-based 
implementations (e.g. sessions) can agree on a common API for their 
feature sets.  I feel strongly that this idea does not belong in PEP 
444; it's one of the few things I think should be its own PEP.

My mechanism (for which I do have a working implementation against WSGI 
1; my web framework uses it) involves middleware layers declaring 
several attributes on themselves:

provides - abstract API names
uses - ordering hint, no dependancy
needs - die if dependancy is not met
before - explicit ordering, including "*"
after - explicit ordering, including "*"

For this to really work, however, it'd also need either an 
entrypoint-based way of looking up components (making the graph truly 
dynamic), or it needs to be combined with explicit packages a la 
setuptools.require.  In that instance, you've already done the ordering 
yourself, so dependancy graphing is moot.

	- Alice.



From alice at gothcandy.com  Thu Jan  6 23:14:32 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 14:14:32 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<6D1FBE4B-46CB-41D7-B5DB-6374B007B661@fuhm.net>
Message-ID: <ig5es8$kj0$1@dough.gmane.org>

On 2011-01-06 14:01:09 -0800, James Y Knight said:

> No you don't -- HTTP 1.0 allows indeterminate-length output. The server 
> simply must close the connection to indicate the end of the response if 
> either the client version HTTP/1.0, or the server doesn't implement 
> HTTP/1.1.

Ah, you are correct.  There was something, somewhere I was reading 
related to WSGI about requiring content-length... but no matter.

Interestingly enough, HTTP/1.0 also supports pipelining (though 
obviously not if content-length is missing) via the `Connection: 
keep-alive` header.  HTTP/1.1 mandates keep-alive by default (which is 
a good thing, IMHO) and offers a work-around for missing content-length 
to preserve the connection: chunked encoding.  Add to that 100-Continue 
(allowing delayed /transfer/ of the request body until the first 
wsgi.input.read() operation) and allows proper, full URLs to be 
requested, amongst other goodies.

Arguing against mandated HTTP/1.1 support (where possible) seems... 
silly to me.  HTTP/1.1 has been around for a long time (adopted by the 
major browsers in 1996), is well understood, is /trivial/ to implement 
(I managed it as part of my 172 Python opcode HTTP server 
implementation), and Just Makes Sense.

If there can be a good technical reason why the adapted language ("if 
possible, it's required") can not be used, I'll definitely re-consider 
this point.  Considering that detection is easy (SERVER_PROTOCOL == 
"1.0"), adaption by the application to either case is easy (detect and 
if not present consume the body_iter and determine length) and it's a 
15 year old standard: welcome to the 20'th century.  ;)

	- Alice.



From graham.dumpleton at gmail.com  Thu Jan  6 23:29:36 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 09:29:36 +1100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig5dpo$eui$1@dough.gmane.org>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
Message-ID: <AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>

On 7 January 2011 08:56, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>
>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>>
>>> :: Making optional (and thus rarely-implemented) features non-optional.
>>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>>> applications to 1.1 servers. ?Thus pipelining, chunked encoding, et. al. as
>>> per the HTTP 1.1 RFC.
>>
>> Requirements on the HTTP compliance of the server don't really have any
>> place in the WSGI spec. You should be able to be WSGI compliant even if you
>> don't use the HTTP transport at all (e.g. maybe you just send around
>> requests via SCGI).
>> The original spec got this right: chunking etc are something which is not
>> relevant to the wsgi application code -- it is up to the server to implement
>> the HTTP transport according to the HTTP spec, if it's purporting to be an
>> HTTP server.
>
> Chunking is actually quite relevant to the specification, as WSGI and PEP
> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for
> chunked bodies regardless of higher-level support for chunking. ?The body
> iterator. ?Previously you /had/ to define a length, with chunked encoding at
> the server level, you don't.
>
> I agree, however, that not all gateways will be able to implement the
> relevant HTTP/1.1 features. ?FastCGI does, SCGI after a quick Google search,
> seems to support it as well. I should re-word it as:
>
> "For those servers capable of HTTP/1.1 features the implementation of such
> features is required."

I would question whether FASTCGI, SCGI or AJP support the concept of
chunking of responses to the extent that the application can prepare
the final content including chunks as required by the HTTP
specification. Further, in Apache at least, the output from a web
application served via those protocols is still pushed through the
Apache output filter chain so as to allow the filters to modify the
response, eg., apply compression using mod_deflate. As a consequence,
the standard HTTP 'CHUNK' output filter is still a part of the output
filter stack. This means that were a web application to try and do
chunking itself, then Apache would rechunk such that the original
chunking became part of the content, rather than the transfer
encoding.

So, in order to be able to achieve what I think you want, with a web
application being able to do chunking itself, you would need to modify
the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
also like mod_cgi and mod_cgid of Apache.

The only WSGI implementation I know of for Apache where you might even
be able to do what you want is uWSGI. This is because I believe from
memory it uses a mode in Apache by default called assbackwords. What
this allows is for the output from the web application to bypass the
Apache output filter stack and directly control the raw HTTP output.
This gives uWSGI a little bit less overhead in Apache, but at the loss
of the ability to actually use Apache output filters and for Apache to
fix up response headers in any way. There is a flag in uWSGI which can
optionally be set to make it use the more traditional mode and not use
assbackwords.

Thus, I believe you would be fighting against server implementations
such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
chunking to be supported at the level of the web application.

About all you can do is ensure that the WSGI specification doesn't
include anything in it which would prevent a web application
harnessing indirectly such a feature as chunking where the web server
supports it.

As it is, it isn't chunked responses which is even the problem,
because if a underlying web server supports chunking for responses,
all you need to do is not set the content length.

The problem area with chunking is the request content as the way that
the WSGI specification is written prevents being able to have chunked
request content. I have described the issue previously and made
suggestions about alternate way that wsgi.input could be used.

Graham

> +1
>
> ? ? ? ?- Alice.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>

From graham.dumpleton at gmail.com  Thu Jan  6 23:51:07 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 09:51:07 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
Message-ID: <AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>

Can we not let the PEP 444 discussion side track getting PEP 3333
sorted out? This is exactly what has happened numerous times before
when we have been trying to sort out core issues of WSGI on Python 3.
And people wander why I get grumpy now every time this happens. :-(

So, where are we at? It seems the only real issue that needs to be
resolved is the correctness or otherwise of the CGI/WSGI bridge
example. Everything else could be left as is, even if dubious.

For the CGI/WSGI bridge I take it the issue is that for bytes need to
use the internal buffer directly. I don't see a need to flush the top
level stdout because nothing should be writing to stdout besides the
WSGI bridge and so it just needs to ensure it is consistent.

Is that the last thing or do I need to go spend some time and write my
own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
lying around and just do some final validation checks with a parallel
implementation as a sanity check to make sure we got everything? This
might be a good idea anyway.

Graham

On 5 January 2011 05:33, Guido van Rossum <guido at python.org> wrote:
> On Tue, Jan 4, 2011 at 7:48 AM, P.J. Eby <pje at telecommunity.com> wrote:
>> At 06:30 PM 1/3/2011 -0800, Guido van Rossum wrote:
>>>
>>> Would
>>>
>>> ?sys.stdout.buffer.write(b'abc')
>>>
>>> do?
>>>
>>> (If you mix this with writing strings to sys.stdout directly, you may
>>> have to call sys.stdout.flush() first.)
>>
>> The current code is:
>>
>> ? ? ? ? ? ?sys.stdout.write(data) ?# TODO: this needs to be binary on Py3
>> ? ? ? ? ? ?sys.stdout.flush()
>>
>> Should I be using sys.stdout.buffer for both, or just the write?
>
> For both.
>
> But the flush() I was referring to is actually *before* either of
> these, suggesting
>
> sys.stdout.flush()
> sys.stdout.buffer.write(data)
> sys.stdout.buffer.flush()
>
> However the first flush() is only necessary is there's a possibility
> that previously something had been written to sys.stdout (not to
> sys.stdout.buffer).
>
>> For the CGI example in the PEP, I don't want to bother trying to make it
>> fully production-usable; that's what we have wsgiref in the stdlib for. ?So
>> I won't worry about mixing strings and regular output in the example, even
>> if perhaps wsgiref should add the StringIO's proposed by Graham.
>
> Not sure what you mean. Surely copying and pasting the examples should
> work? What are the details you'd like to leave out?
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>

From graham.dumpleton at gmail.com  Fri Jan  7 00:14:16 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 10:14:16 +1100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
Message-ID: <AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>

One other comment about HTTP/1.1 features.

You will always be battling to have some HTTP/1.1 features work in a
controllable way. This is because WSGI gateways/adapters aren't often
directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
AJP, CGI etc. In this sort of situation you are at the mercy of what
the modules implementing those protocols do, or even are hamstrung by
how those protocols work.

The classic example is 100-continue processing. This simply cannot
work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
mechanisms where proxying is performed as the protocol being used
doesn't implement a notion of end to end signalling in respect of
100-continue.

The current WSGI specification acknowledges that by saying:

"""
Servers and gateways that implement HTTP 1.1 must provide transparent
support for HTTP 1.1's "expect/continue" mechanism. This may be done
in any of several ways:

* Respond to requests containing an Expect: 100-continue request with
an immediate "100 Continue" response, and proceed normally.
* Proceed with the request normally, but provide the application with
a wsgi.input stream that will send the "100 Continue" response if/when
the application first attempts to read from the input stream. The read
request must then remain blocked until the client responds.
* Wait until the client decides that the server does not support
expect/continue, and sends the request body on its own. (This is
suboptimal, and is not recommended.)
"""

If you are going to try and push for full visibility of HTTP/1.1 and
an ability to control it at the application level then you will fail
with 100-continue to start with.

So, although option 2 above would be the most ideal and is giving the
application control, specifically the ability to send an error
response based on request headers alone, and with reading the response
and triggering the 100-continue, it isn't practical to require it, as
the majority of hosting mechanisms for WSGI wouldn't even be able to
implement it that way.

The same goes for any other feature, there is no point mandating a
feature that can only be realistically implementing on a minority of
implementations. This would be even worse where dependence on such a
feature would mean that the WSGI application would no longer be
portable to another WSGI server and destroys the notion that WSGI
provides a portable interface.

This isn't just restricted to HTTP/1.1 features either, but also
applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
that are directly hooked into the URL parsing of the base HTTP server
can provide that information, which basically means that only pure
Python HTTP/WSGI servers are likely able to provide it without
guessing, and in that case such servers usually are always used where
WSGI application mounted at root anyway.

Graham

On 7 January 2011 09:29, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> On 7 January 2011 08:56, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
>> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>>
>>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>>>
>>>> :: Making optional (and thus rarely-implemented) features non-optional.
>>>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>>>> applications to 1.1 servers. ?Thus pipelining, chunked encoding, et. al. as
>>>> per the HTTP 1.1 RFC.
>>>
>>> Requirements on the HTTP compliance of the server don't really have any
>>> place in the WSGI spec. You should be able to be WSGI compliant even if you
>>> don't use the HTTP transport at all (e.g. maybe you just send around
>>> requests via SCGI).
>>> The original spec got this right: chunking etc are something which is not
>>> relevant to the wsgi application code -- it is up to the server to implement
>>> the HTTP transport according to the HTTP spec, if it's purporting to be an
>>> HTTP server.
>>
>> Chunking is actually quite relevant to the specification, as WSGI and PEP
>> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for
>> chunked bodies regardless of higher-level support for chunking. ?The body
>> iterator. ?Previously you /had/ to define a length, with chunked encoding at
>> the server level, you don't.
>>
>> I agree, however, that not all gateways will be able to implement the
>> relevant HTTP/1.1 features. ?FastCGI does, SCGI after a quick Google search,
>> seems to support it as well. I should re-word it as:
>>
>> "For those servers capable of HTTP/1.1 features the implementation of such
>> features is required."
>
> I would question whether FASTCGI, SCGI or AJP support the concept of
> chunking of responses to the extent that the application can prepare
> the final content including chunks as required by the HTTP
> specification. Further, in Apache at least, the output from a web
> application served via those protocols is still pushed through the
> Apache output filter chain so as to allow the filters to modify the
> response, eg., apply compression using mod_deflate. As a consequence,
> the standard HTTP 'CHUNK' output filter is still a part of the output
> filter stack. This means that were a web application to try and do
> chunking itself, then Apache would rechunk such that the original
> chunking became part of the content, rather than the transfer
> encoding.
>
> So, in order to be able to achieve what I think you want, with a web
> application being able to do chunking itself, you would need to modify
> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
> also like mod_cgi and mod_cgid of Apache.
>
> The only WSGI implementation I know of for Apache where you might even
> be able to do what you want is uWSGI. This is because I believe from
> memory it uses a mode in Apache by default called assbackwords. What
> this allows is for the output from the web application to bypass the
> Apache output filter stack and directly control the raw HTTP output.
> This gives uWSGI a little bit less overhead in Apache, but at the loss
> of the ability to actually use Apache output filters and for Apache to
> fix up response headers in any way. There is a flag in uWSGI which can
> optionally be set to make it use the more traditional mode and not use
> assbackwords.
>
> Thus, I believe you would be fighting against server implementations
> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
> chunking to be supported at the level of the web application.
>
> About all you can do is ensure that the WSGI specification doesn't
> include anything in it which would prevent a web application
> harnessing indirectly such a feature as chunking where the web server
> supports it.
>
> As it is, it isn't chunked responses which is even the problem,
> because if a underlying web server supports chunking for responses,
> all you need to do is not set the content length.
>
> The problem area with chunking is the request content as the way that
> the WSGI specification is written prevents being able to have chunked
> request content. I have described the issue previously and made
> suggestions about alternate way that wsgi.input could be used.
>
> Graham
>
>> +1
>>
>> ? ? ? ?- Alice.
>>
>>
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>>
>

From alex.gronholm at nextday.fi  Fri Jan  7 01:00:25 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Fri, 07 Jan 2011 02:00:25 +0200
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <AANLkTimWJJJG+4B1j+_QTE1BXynwujyzU-m6kV5=oRUW@mail.gmail.com>
References: <ig3erb$vll$1@dough.gmane.org>	<ig3l9g$lqc$1@dough.gmane.org>	<alpine.OSX.2.00.1101061302550.7296@crank.local>	<87y66xlxl2.wl%eric@ionrock.org>	<4D262A95.5070703@nextday.fi>
	<AANLkTimWJJJG+4B1j+_QTE1BXynwujyzU-m6kV5=oRUW@mail.gmail.com>
Message-ID: <4D265799.5060106@nextday.fi>

06.01.2011 23:11, Sylvain Hellegouarch kirjoitti:
>
>
> 2011/1/6 Alex Gr?nholm <alex.gronholm at nextday.fi 
> <mailto:alex.gronholm at nextday.fi>>
>
>     06.01.2011 20:02, Eric Larson kirjoitti:
>
>         At Thu, 6 Jan 2011 13:03:15 +0000 (GMT),
>         chris dent wrote:
>
>             <snip>
>             On async:
>
>             I agree with some others who have suggested that maybe
>             async should be
>             its own thing, rather than integrated into a WSGI2. A
>             server could
>             choose to be WSGI2 compliant or AWSGI compliant, or both.
>             </snip>
>
>         +1
>
>         After seeing some of the ideas regarding how to add async into
>         a new
>         version of WSGI, it isn't the specific problem the async feature
>         addresses in terms of WSGI servers. Is the goal is to support long
>         running connections? Are we trying to support WebSockets and other
>         long running connection interfaces? If that is the case, async
>         is a
>         *technique* for handling this paradigm, but it doesn't address the
>         real problem. There are techniques that have sounded
>         reasonable like
>         making available the socket such that a server can give it to the
>         application to do something use with it (here is an example doing
>         something similar with CherryPy -
>         http://trac.defuze.org/browser/oss/ws4cp/ws4cp.py).
>
>     The primary idea behind asynchronous servers/applications is the
>     ability to efficiently serve a huge number of concurrent
>     connections with a small number of threads. Asynchronous
>     applications tend to be faster because there is less thread
>     context switching happening in the CPU. Any application that runs
>     on top of a web server that allocates less threads to the
>     application than the number of connections has to be quick to
>     respond so as not to starve the thread pool or block the event
>     loop. This is true regardless of whether nonblocking I/O or some
>     other technique is used. I'm a bit unclear as to how else you
>     would do this. Care to elaborate on that? I looked at the Cherrypy
>     code, but I couldn't yet figure that out.
>
>
> Since I wrote that piece of code, I guess I ought to chime in. First 
> of all, the code isn't part of CherryPy, simply it's one idea to 
> provide WebSocket to CherryPy. Considering WebSocket bootstraps on 
> HTTP but once that's done, it's just a raw socket with bits and pieces 
> on top, I wanted to find a way not to block CherryPy from serving 
> other requests once a WebSocket handshake had been performed. The idea 
> was simply to detach the socket from the worker thread once the 
> handshake had been performed. Then the application had a socket at 
> hand and this particular instance, I simply decided to use asyncore to 
> loop through those sockets aside from the CherryPy HTTP server. In 
> effect, you end up with asyncore for WS sockets and CherryPy for any 
> HTTP serving but from within one single process, using CherryPy's main 
> loop.
>
> By large this is not a generic solution for implementing async in WSGI 
> but a specific example on how one can have both threads and an async 
> loop playing together. It's merely a proof of concept :)
Yes, this is how I figured it too. In the end, what really matters is 
that code that doesn't get a dedicated thread has to be designed a 
little differently. The purpose of this discussion is to come up with a 
standard interface for such applications. I'd also like to explore the 
possibility of incorporating such a mechanism in PEP 444, provided that 
it does not complicate the implementation too much. Otherwise, a 
separate specification may be necessary.
>
> Hope that clarifies that piece of code.
> -- 
> - Sylvain
> http://www.defuze.org
> http://twitter.com/lawouach

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110107/b12c2c57/attachment.html>

From alex.gronholm at nextday.fi  Fri Jan  7 01:46:48 2011
From: alex.gronholm at nextday.fi (=?windows-1252?Q?Alex_Gr=F6nholm?=)
Date: Fri, 07 Jan 2011 02:46:48 +0200
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
References: <11174894.6.1294342577349.JavaMail.root@boaz>	<ig5a2d$r5s$1@dough.gmane.org>	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>	<ig5dpo$eui$1@dough.gmane.org>	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
Message-ID: <4D266278.8010702@nextday.fi>

07.01.2011 01:14, Graham Dumpleton kirjoitti:
> One other comment about HTTP/1.1 features.
>
> You will always be battling to have some HTTP/1.1 features work in a
> controllable way. This is because WSGI gateways/adapters aren't often
> directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
> AJP, CGI etc. In this sort of situation you are at the mercy of what
> the modules implementing those protocols do, or even are hamstrung by
> how those protocols work.
>
> The classic example is 100-continue processing. This simply cannot
> work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
> mechanisms where proxying is performed as the protocol being used
> doesn't implement a notion of end to end signalling in respect of
> 100-continue.
I think we need some concrete examples to figure out what is and isn't 
possible with WSGI 1.0.1.
My motivation for participating in this discussion can be summed up in 
that I want the following two applications to work properly:

- PlasmaDS (Flex Messaging implementation)
- WebDAV

The PlasmaDS project is the planned Python counterpart to Adobe's 
BlazeDS. Interoperability with the existing implementation requires that 
both the request and response use chunked transfer encoding, to achieve 
bidirectional streaming. I don't really care how this happens, I just 
want to make sure that there is nothing preventing it.

The WebDAV spec, on the other hand, says 
(http://www.webdav.org/specs/rfc2518.html#STATUS_102):

The 102 (Processing) status code is an interim response used to inform 
the client that the server has accepted the complete request, but has 
not yet completed it. This status code /SHOULD/ only be sent when the 
server has a reasonable expectation that the request will take 
significant time to complete. As guidance, if a method is taking longer 
than 20 seconds (a reasonable, but arbitrary value) to process the 
server /SHOULD/ return a 102 (Processing) response. The server /MUST/ 
send a final response after the request has been completed.

Again, I don't care how this is done as long as it's possible.
> The current WSGI specification acknowledges that by saying:
>
> """
> Servers and gateways that implement HTTP 1.1 must provide transparent
> support for HTTP 1.1's "expect/continue" mechanism. This may be done
> in any of several ways:
>
> * Respond to requests containing an Expect: 100-continue request with
> an immediate "100 Continue" response, and proceed normally.
> * Proceed with the request normally, but provide the application with
> a wsgi.input stream that will send the "100 Continue" response if/when
> the application first attempts to read from the input stream. The read
> request must then remain blocked until the client responds.
> * Wait until the client decides that the server does not support
> expect/continue, and sends the request body on its own. (This is
> suboptimal, and is not recommended.)
> """
>
> If you are going to try and push for full visibility of HTTP/1.1 and
> an ability to control it at the application level then you will fail
> with 100-continue to start with.
>
> So, although option 2 above would be the most ideal and is giving the
> application control, specifically the ability to send an error
> response based on request headers alone, and with reading the response
> and triggering the 100-continue, it isn't practical to require it, as
> the majority of hosting mechanisms for WSGI wouldn't even be able to
> implement it that way.
>
> The same goes for any other feature, there is no point mandating a
> feature that can only be realistically implementing on a minority of
> implementations. This would be even worse where dependence on such a
> feature would mean that the WSGI application would no longer be
> portable to another WSGI server and destroys the notion that WSGI
> provides a portable interface.
>
> This isn't just restricted to HTTP/1.1 features either, but also
> applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
> that are directly hooked into the URL parsing of the base HTTP server
> can provide that information, which basically means that only pure
> Python HTTP/WSGI servers are likely able to provide it without
> guessing, and in that case such servers usually are always used where
> WSGI application mounted at root anyway.
>
> Graham
>
> On 7 January 2011 09:29, Graham Dumpleton<graham.dumpleton at gmail.com>  wrote:
>> On 7 January 2011 08:56, Alice Bevan?McGregor<alice at gothcandy.com>  wrote:
>>> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>>>
>>>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>>>> :: Making optional (and thus rarely-implemented) features non-optional.
>>>>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>>>>> applications to 1.1 servers.  Thus pipelining, chunked encoding, et. al. as
>>>>> per the HTTP 1.1 RFC.
>>>> Requirements on the HTTP compliance of the server don't really have any
>>>> place in the WSGI spec. You should be able to be WSGI compliant even if you
>>>> don't use the HTTP transport at all (e.g. maybe you just send around
>>>> requests via SCGI).
>>>> The original spec got this right: chunking etc are something which is not
>>>> relevant to the wsgi application code -- it is up to the server to implement
>>>> the HTTP transport according to the HTTP spec, if it's purporting to be an
>>>> HTTP server.
>>> Chunking is actually quite relevant to the specification, as WSGI and PEP
>>> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for
>>> chunked bodies regardless of higher-level support for chunking.  The body
>>> iterator.  Previously you /had/ to define a length, with chunked encoding at
>>> the server level, you don't.
>>>
>>> I agree, however, that not all gateways will be able to implement the
>>> relevant HTTP/1.1 features.  FastCGI does, SCGI after a quick Google search,
>>> seems to support it as well. I should re-word it as:
>>>
>>> "For those servers capable of HTTP/1.1 features the implementation of such
>>> features is required."
>> I would question whether FASTCGI, SCGI or AJP support the concept of
>> chunking of responses to the extent that the application can prepare
>> the final content including chunks as required by the HTTP
>> specification. Further, in Apache at least, the output from a web
>> application served via those protocols is still pushed through the
>> Apache output filter chain so as to allow the filters to modify the
>> response, eg., apply compression using mod_deflate. As a consequence,
>> the standard HTTP 'CHUNK' output filter is still a part of the output
>> filter stack. This means that were a web application to try and do
>> chunking itself, then Apache would rechunk such that the original
>> chunking became part of the content, rather than the transfer
>> encoding.
>>
>> So, in order to be able to achieve what I think you want, with a web
>> application being able to do chunking itself, you would need to modify
>> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
>> also like mod_cgi and mod_cgid of Apache.
>>
>> The only WSGI implementation I know of for Apache where you might even
>> be able to do what you want is uWSGI. This is because I believe from
>> memory it uses a mode in Apache by default called assbackwords. What
>> this allows is for the output from the web application to bypass the
>> Apache output filter stack and directly control the raw HTTP output.
>> This gives uWSGI a little bit less overhead in Apache, but at the loss
>> of the ability to actually use Apache output filters and for Apache to
>> fix up response headers in any way. There is a flag in uWSGI which can
>> optionally be set to make it use the more traditional mode and not use
>> assbackwords.
>>
>> Thus, I believe you would be fighting against server implementations
>> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
>> chunking to be supported at the level of the web application.
>>
>> About all you can do is ensure that the WSGI specification doesn't
>> include anything in it which would prevent a web application
>> harnessing indirectly such a feature as chunking where the web server
>> supports it.
>>
>> As it is, it isn't chunked responses which is even the problem,
>> because if a underlying web server supports chunking for responses,
>> all you need to do is not set the content length.
>>
>> The problem area with chunking is the request content as the way that
>> the WSGI specification is written prevents being able to have chunked
>> request content. I have described the issue previously and made
>> suggestions about alternate way that wsgi.input could be used.
>>
>> Graham
>>
>>> +1
>>>
>>>         - Alice.
>>>
>>>
>>> _______________________________________________
>>> Web-SIG mailing list
>>> Web-SIG at python.org
>>> Web SIG: http://www.python.org/sigs/web-sig
>>> Unsubscribe:
>>> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>>>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110107/9827b23d/attachment-0001.html>

From graham.dumpleton at gmail.com  Fri Jan  7 02:15:21 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 12:15:21 +1100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <4D266278.8010702@nextday.fi>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
	<4D266278.8010702@nextday.fi>
Message-ID: <AANLkTi=sw8pczQk86sBEx2767oHjmLaaBX+JT-5u=4OP@mail.gmail.com>

2011/1/7 Alex Gr?nholm <alex.gronholm at nextday.fi>:
> 07.01.2011 01:14, Graham Dumpleton kirjoitti:
>
> One other comment about HTTP/1.1 features.
>
> You will always be battling to have some HTTP/1.1 features work in a
> controllable way. This is because WSGI gateways/adapters aren't often
> directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
> AJP, CGI etc. In this sort of situation you are at the mercy of what
> the modules implementing those protocols do, or even are hamstrung by
> how those protocols work.
>
> The classic example is 100-continue processing. This simply cannot
> work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
> mechanisms where proxying is performed as the protocol being used
> doesn't implement a notion of end to end signalling in respect of
> 100-continue.
>
> I think we need some concrete examples to figure out what is and isn't
> possible with WSGI 1.0.1.
> My motivation for participating in this discussion can be summed up in that
> I want the following two applications to work properly:
>
> - PlasmaDS (Flex Messaging implementation)
> - WebDAV
>
> The PlasmaDS project is the planned Python counterpart to Adobe's BlazeDS.
> Interoperability with the existing implementation requires that both the
> request and response use chunked transfer encoding, to achieve bidirectional
> streaming. I don't really care how this happens, I just want to make sure
> that there is nothing preventing it.

That can only be done by changing the rules around wsgi.input is used.
I'll try and find a reference to where I have posted information about
this before, otherwise I'll write something up again about it.

> The WebDAV spec, on the other hand, says
> (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
>
> The 102 (Processing) status code is an interim response used to inform the
> client that the server has accepted the complete request, but has not yet
> completed it. This status code SHOULD only be sent when the server has a
> reasonable expectation that the request will take significant time to
> complete. As guidance, if a method is taking longer than 20 seconds (a
> reasonable, but arbitrary value) to process the server SHOULD return a 102
> (Processing) response. The server MUST send a final response after the
> request has been completed.

That I don't offhand see a way of being able to do as protocols like
SCGI and CGI definitely don't allow interim status. I am suspecting
that FASTCGI and AJP don't allow it either.

I'll have to even do some digging as to how you would even handle that
in Apache with a normal Apache handler.

Graham

> Again, I don't care how this is done as long as it's possible.
>
> The current WSGI specification acknowledges that by saying:
>
> """
> Servers and gateways that implement HTTP 1.1 must provide transparent
> support for HTTP 1.1's "expect/continue" mechanism. This may be done
> in any of several ways:
>
> * Respond to requests containing an Expect: 100-continue request with
> an immediate "100 Continue" response, and proceed normally.
> * Proceed with the request normally, but provide the application with
> a wsgi.input stream that will send the "100 Continue" response if/when
> the application first attempts to read from the input stream. The read
> request must then remain blocked until the client responds.
> * Wait until the client decides that the server does not support
> expect/continue, and sends the request body on its own. (This is
> suboptimal, and is not recommended.)
> """
>
> If you are going to try and push for full visibility of HTTP/1.1 and
> an ability to control it at the application level then you will fail
> with 100-continue to start with.
>
> So, although option 2 above would be the most ideal and is giving the
> application control, specifically the ability to send an error
> response based on request headers alone, and with reading the response
> and triggering the 100-continue, it isn't practical to require it, as
> the majority of hosting mechanisms for WSGI wouldn't even be able to
> implement it that way.
>
> The same goes for any other feature, there is no point mandating a
> feature that can only be realistically implementing on a minority of
> implementations. This would be even worse where dependence on such a
> feature would mean that the WSGI application would no longer be
> portable to another WSGI server and destroys the notion that WSGI
> provides a portable interface.
>
> This isn't just restricted to HTTP/1.1 features either, but also
> applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
> that are directly hooked into the URL parsing of the base HTTP server
> can provide that information, which basically means that only pure
> Python HTTP/WSGI servers are likely able to provide it without
> guessing, and in that case such servers usually are always used where
> WSGI application mounted at root anyway.
>
> Graham
>
> On 7 January 2011 09:29, Graham Dumpleton <graham.dumpleton at gmail.com>
> wrote:
>
> On 7 January 2011 08:56, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
>
> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>
> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>
> :: Making optional (and thus rarely-implemented) features non-optional.
> E.g. server support for HTTP/1.1 with clarifications for interfacing
> applications to 1.1 servers. ?Thus pipelining, chunked encoding, et. al. as
> per the HTTP 1.1 RFC.
>
> Requirements on the HTTP compliance of the server don't really have any
> place in the WSGI spec. You should be able to be WSGI compliant even if you
> don't use the HTTP transport at all (e.g. maybe you just send around
> requests via SCGI).
> The original spec got this right: chunking etc are something which is not
> relevant to the wsgi application code -- it is up to the server to implement
> the HTTP transport according to the HTTP spec, if it's purporting to be an
> HTTP server.
>
> Chunking is actually quite relevant to the specification, as WSGI and PEP
> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for
> chunked bodies regardless of higher-level support for chunking. ?The body
> iterator. ?Previously you /had/ to define a length, with chunked encoding at
> the server level, you don't.
>
> I agree, however, that not all gateways will be able to implement the
> relevant HTTP/1.1 features. ?FastCGI does, SCGI after a quick Google search,
> seems to support it as well. I should re-word it as:
>
> "For those servers capable of HTTP/1.1 features the implementation of such
> features is required."
>
> I would question whether FASTCGI, SCGI or AJP support the concept of
> chunking of responses to the extent that the application can prepare
> the final content including chunks as required by the HTTP
> specification. Further, in Apache at least, the output from a web
> application served via those protocols is still pushed through the
> Apache output filter chain so as to allow the filters to modify the
> response, eg., apply compression using mod_deflate. As a consequence,
> the standard HTTP 'CHUNK' output filter is still a part of the output
> filter stack. This means that were a web application to try and do
> chunking itself, then Apache would rechunk such that the original
> chunking became part of the content, rather than the transfer
> encoding.
>
> So, in order to be able to achieve what I think you want, with a web
> application being able to do chunking itself, you would need to modify
> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
> also like mod_cgi and mod_cgid of Apache.
>
> The only WSGI implementation I know of for Apache where you might even
> be able to do what you want is uWSGI. This is because I believe from
> memory it uses a mode in Apache by default called assbackwords. What
> this allows is for the output from the web application to bypass the
> Apache output filter stack and directly control the raw HTTP output.
> This gives uWSGI a little bit less overhead in Apache, but at the loss
> of the ability to actually use Apache output filters and for Apache to
> fix up response headers in any way. There is a flag in uWSGI which can
> optionally be set to make it use the more traditional mode and not use
> assbackwords.
>
> Thus, I believe you would be fighting against server implementations
> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
> chunking to be supported at the level of the web application.
>
> About all you can do is ensure that the WSGI specification doesn't
> include anything in it which would prevent a web application
> harnessing indirectly such a feature as chunking where the web server
> supports it.
>
> As it is, it isn't chunked responses which is even the problem,
> because if a underlying web server supports chunking for responses,
> all you need to do is not set the content length.
>
> The problem area with chunking is the request content as the way that
> the WSGI specification is written prevents being able to have chunked
> request content. I have described the issue previously and made
> suggestions about alternate way that wsgi.input could be used.
>
> Graham
>
> +1
>
> ? ? ? ?- Alice.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>
>

From alice at gothcandy.com  Fri Jan  7 02:47:55 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 17:47:55 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <4D25F9F0.3070306@inteli-com.com>
Message-ID: <ig5rcb$csc$1@dough.gmane.org>

On 2011-01-06 09:20:48 -0800, Randy Syring said:
> Being a web application developer and relying on frameworks like 
> Werkzeug and WebOb, I may not have much of a dog in this fight.

All input is welcome; I do want to hear from both framework developers 
and users of frameworks.  I suspect this discussion ocurring on the 
Web-SIG list would be somewhat of an impediment for users to 
contribute, so thank you for posting!

> However, I have been following web-sig for a couple years and I have 
> seen the difficulties involved in reaching consensus on 
> modifying/updating the WSGI spec.

I've read through the archives and seen the issues as well.  I do 
believe that, on this one topic, it will be simply impossible to please 
everyone.  Up here in Canada we have

> Its clear to me that most people on this list who can contribute in 
> meaningful ways to the creation of WSGI 2 have very little time to do 
> so.

One benefit of mailing lists over other communications channels (IRC, 
etc.) is that mailing list traffic sticks around for a while and 
doesn't require realtime effort.

> Motivation seems generally low anyway, because what we have currently works.

The burst of traffic after Guido offered to push PEP 3333 ratification 
proves that what we have /doesn't/ currently work, at least, for 
everyone.  Python 3 continues to be a problem.

> It may have warts, but it works, and that very fact seems to limit the 
> number of people interested in donating time to improving the spec.

Limiting the scope to Python 2; PEP 333 has a number of issues 
including, I feel the worst sins for a "standard": ambiguity and 
complexity.  While people may feel comfortable with the standard they 
have learned thus far, I don't think they should be complacent when it 
comes to examining possible improvements.

> Every time something controversial is added to the spec, its going to 
> make it that much harder to move forward.

Thus my pushing for the controversial parts to be optional.  While, 
demonstrably, not everyone will use these parts, having them be present 
is important to capture mind-space and get people thinking about the 
broader implications of what they code.

> On 2011-01-06 05:03:15 -0800, Chris Dent said:
>> I agree with some others who have suggested that maybe async should be 
>> its own thing, rather than integrated into a WSGI2. A server could 
>> choose to be WSGI2 compliant or AWSGI compliant, or both.
> 
> Adding async to the spec is a death blow IMO.? You gain nothing by 
> putting it in and lose a lot of interest and time spent discussing it.? 
> Make it a separate PEP that references the first.? That way, those who 
> don't really care about it can still work on WSGI 2 without the 
> distraction of the async parts.? If you make the new async PEP 
> dependent on the WSGI 2 spec, then those ideas can be tossed around all 
> day long without distracting from or taking energy away from the core 
> WSGI 2 ideas.

Tossing the idea around all day long will then, of course, be happening 
regardless.  Unfortunately for that particular discussion, PEP 3148 / 
Futures seems to have won out in the broader scope.  Having a ratified 
and incorporated language PEP (core in 3.2 w/ compatibility package for 
2.5 or 2.6+ support) reduces the scope of async discussion down to: 
"how do we integrate futures into WSGI 2" instead of "how do we define 
an async API at all".

> I suggest reviewing the Web-SIG history of previous async discussions; 
> there's a lot more to having a meaningful API spec than having a 
> plausible approach.? It's not that there haven't been past proposals, 
> they just couldn't get as far as making it possible to write a 
> non-trivial async application that would actually be portable among 
> Python-supporting asynchronous web servers.

See my previous paragraph about futures.

> [snip]
> 
> We only see what you write here, the burden of proof is on you to 
> communicate your attentions and agenda.

Unfortunately the reality is that no solution will be agreeable to 
everyone, and, for lack of a better phrase, no amount of hand-holding 
can make it otherwise.

I am attempting to improve my PR somewhat through the many posts today 
and with as much thorough information as possible.  ;)

> Then again, my opinion and impression could be completely off, and if 
> that is the case, feel free to ignore me.? :)

That, despite popular opinion otherwise, is something I rarely do, 
though my brain is so full these days that some things might get 
accidentally expunged from my LRU heap.  ;)

	- Alice.



From graham.dumpleton at gmail.com  Fri Jan  7 03:09:58 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 13:09:58 +1100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <AANLkTi=sw8pczQk86sBEx2767oHjmLaaBX+JT-5u=4OP@mail.gmail.com>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
	<4D266278.8010702@nextday.fi>
	<AANLkTi=sw8pczQk86sBEx2767oHjmLaaBX+JT-5u=4OP@mail.gmail.com>
Message-ID: <AANLkTim==2PU7nVKd235Xsu5u8P+iE7+Q_oy3X0xbyjJ@mail.gmail.com>

2011/1/7 Graham Dumpleton <graham.dumpleton at gmail.com>:
> 2011/1/7 Alex Gr?nholm <alex.gronholm at nextday.fi>:
>> 07.01.2011 01:14, Graham Dumpleton kirjoitti:
>>
>> One other comment about HTTP/1.1 features.
>>
>> You will always be battling to have some HTTP/1.1 features work in a
>> controllable way. This is because WSGI gateways/adapters aren't often
>> directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
>> AJP, CGI etc. In this sort of situation you are at the mercy of what
>> the modules implementing those protocols do, or even are hamstrung by
>> how those protocols work.
>>
>> The classic example is 100-continue processing. This simply cannot
>> work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
>> mechanisms where proxying is performed as the protocol being used
>> doesn't implement a notion of end to end signalling in respect of
>> 100-continue.
>>
>> I think we need some concrete examples to figure out what is and isn't
>> possible with WSGI 1.0.1.
>> My motivation for participating in this discussion can be summed up in that
>> I want the following two applications to work properly:
>>
>> - PlasmaDS (Flex Messaging implementation)
>> - WebDAV
>>
>> The PlasmaDS project is the planned Python counterpart to Adobe's BlazeDS.
>> Interoperability with the existing implementation requires that both the
>> request and response use chunked transfer encoding, to achieve bidirectional
>> streaming. I don't really care how this happens, I just want to make sure
>> that there is nothing preventing it.
>
> That can only be done by changing the rules around wsgi.input is used.
> I'll try and find a reference to where I have posted information about
> this before, otherwise I'll write something up again about it.

BTW, even if WSGI specification were changed to allow handling of
chunked requests, it would not work for FASTCGI, SCGI, AJP, CGI or
mod_wsgi daemon mode. Also not likely to work on uWSGI either.

This is because all of these work on the expectation that the complete
request body can be written across to the separate application process
before actually reading the response from the application.

In other words, both way streaming is not possible.

The only solution which would allow this with Apache is mod_wsgi
embedded mode, which in mod_wsgi 3.X already has an optional feature
which can be enabled so as to allow you to step out of current bounds
of the WSGI specification and use wsgi.input as I will explain, to do
this both way streaming.

Pure Python HTTP/WSGI servers which are a front facing server could
also be modified to handle this is WSGI specification were changed,
but whether those same will work if put behind a web proxy will depend
on how the front end web proxy works.

Graham

>> The WebDAV spec, on the other hand, says
>> (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
>>
>> The 102 (Processing) status code is an interim response used to inform the
>> client that the server has accepted the complete request, but has not yet
>> completed it. This status code SHOULD only be sent when the server has a
>> reasonable expectation that the request will take significant time to
>> complete. As guidance, if a method is taking longer than 20 seconds (a
>> reasonable, but arbitrary value) to process the server SHOULD return a 102
>> (Processing) response. The server MUST send a final response after the
>> request has been completed.
>
> That I don't offhand see a way of being able to do as protocols like
> SCGI and CGI definitely don't allow interim status. I am suspecting
> that FASTCGI and AJP don't allow it either.
>
> I'll have to even do some digging as to how you would even handle that
> in Apache with a normal Apache handler.
>
> Graham
>
>> Again, I don't care how this is done as long as it's possible.
>>
>> The current WSGI specification acknowledges that by saying:
>>
>> """
>> Servers and gateways that implement HTTP 1.1 must provide transparent
>> support for HTTP 1.1's "expect/continue" mechanism. This may be done
>> in any of several ways:
>>
>> * Respond to requests containing an Expect: 100-continue request with
>> an immediate "100 Continue" response, and proceed normally.
>> * Proceed with the request normally, but provide the application with
>> a wsgi.input stream that will send the "100 Continue" response if/when
>> the application first attempts to read from the input stream. The read
>> request must then remain blocked until the client responds.
>> * Wait until the client decides that the server does not support
>> expect/continue, and sends the request body on its own. (This is
>> suboptimal, and is not recommended.)
>> """
>>
>> If you are going to try and push for full visibility of HTTP/1.1 and
>> an ability to control it at the application level then you will fail
>> with 100-continue to start with.
>>
>> So, although option 2 above would be the most ideal and is giving the
>> application control, specifically the ability to send an error
>> response based on request headers alone, and with reading the response
>> and triggering the 100-continue, it isn't practical to require it, as
>> the majority of hosting mechanisms for WSGI wouldn't even be able to
>> implement it that way.
>>
>> The same goes for any other feature, there is no point mandating a
>> feature that can only be realistically implementing on a minority of
>> implementations. This would be even worse where dependence on such a
>> feature would mean that the WSGI application would no longer be
>> portable to another WSGI server and destroys the notion that WSGI
>> provides a portable interface.
>>
>> This isn't just restricted to HTTP/1.1 features either, but also
>> applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
>> that are directly hooked into the URL parsing of the base HTTP server
>> can provide that information, which basically means that only pure
>> Python HTTP/WSGI servers are likely able to provide it without
>> guessing, and in that case such servers usually are always used where
>> WSGI application mounted at root anyway.
>>
>> Graham
>>
>> On 7 January 2011 09:29, Graham Dumpleton <graham.dumpleton at gmail.com>
>> wrote:
>>
>> On 7 January 2011 08:56, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
>>
>> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>>
>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>
>> :: Making optional (and thus rarely-implemented) features non-optional.
>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>> applications to 1.1 servers. ?Thus pipelining, chunked encoding, et. al. as
>> per the HTTP 1.1 RFC.
>>
>> Requirements on the HTTP compliance of the server don't really have any
>> place in the WSGI spec. You should be able to be WSGI compliant even if you
>> don't use the HTTP transport at all (e.g. maybe you just send around
>> requests via SCGI).
>> The original spec got this right: chunking etc are something which is not
>> relevant to the wsgi application code -- it is up to the server to implement
>> the HTTP transport according to the HTTP spec, if it's purporting to be an
>> HTTP server.
>>
>> Chunking is actually quite relevant to the specification, as WSGI and PEP
>> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for
>> chunked bodies regardless of higher-level support for chunking. ?The body
>> iterator. ?Previously you /had/ to define a length, with chunked encoding at
>> the server level, you don't.
>>
>> I agree, however, that not all gateways will be able to implement the
>> relevant HTTP/1.1 features. ?FastCGI does, SCGI after a quick Google search,
>> seems to support it as well. I should re-word it as:
>>
>> "For those servers capable of HTTP/1.1 features the implementation of such
>> features is required."
>>
>> I would question whether FASTCGI, SCGI or AJP support the concept of
>> chunking of responses to the extent that the application can prepare
>> the final content including chunks as required by the HTTP
>> specification. Further, in Apache at least, the output from a web
>> application served via those protocols is still pushed through the
>> Apache output filter chain so as to allow the filters to modify the
>> response, eg., apply compression using mod_deflate. As a consequence,
>> the standard HTTP 'CHUNK' output filter is still a part of the output
>> filter stack. This means that were a web application to try and do
>> chunking itself, then Apache would rechunk such that the original
>> chunking became part of the content, rather than the transfer
>> encoding.
>>
>> So, in order to be able to achieve what I think you want, with a web
>> application being able to do chunking itself, you would need to modify
>> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
>> also like mod_cgi and mod_cgid of Apache.
>>
>> The only WSGI implementation I know of for Apache where you might even
>> be able to do what you want is uWSGI. This is because I believe from
>> memory it uses a mode in Apache by default called assbackwords. What
>> this allows is for the output from the web application to bypass the
>> Apache output filter stack and directly control the raw HTTP output.
>> This gives uWSGI a little bit less overhead in Apache, but at the loss
>> of the ability to actually use Apache output filters and for Apache to
>> fix up response headers in any way. There is a flag in uWSGI which can
>> optionally be set to make it use the more traditional mode and not use
>> assbackwords.
>>
>> Thus, I believe you would be fighting against server implementations
>> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
>> chunking to be supported at the level of the web application.
>>
>> About all you can do is ensure that the WSGI specification doesn't
>> include anything in it which would prevent a web application
>> harnessing indirectly such a feature as chunking where the web server
>> supports it.
>>
>> As it is, it isn't chunked responses which is even the problem,
>> because if a underlying web server supports chunking for responses,
>> all you need to do is not set the content length.
>>
>> The problem area with chunking is the request content as the way that
>> the WSGI specification is written prevents being able to have chunked
>> request content. I have described the issue previously and made
>> suggestions about alternate way that wsgi.input could be used.
>>
>> Graham
>>
>> +1
>>
>> ? ? ? ?- Alice.
>>
>>
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>>
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi
>>
>>
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>>
>>
>

From alex.gronholm at nextday.fi  Fri Jan  7 03:36:32 2011
From: alex.gronholm at nextday.fi (=?windows-1252?Q?Alex_Gr=F6nholm?=)
Date: Fri, 07 Jan 2011 04:36:32 +0200
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <AANLkTim==2PU7nVKd235Xsu5u8P+iE7+Q_oy3X0xbyjJ@mail.gmail.com>
References: <11174894.6.1294342577349.JavaMail.root@boaz>	<ig5a2d$r5s$1@dough.gmane.org>	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>	<ig5dpo$eui$1@dough.gmane.org>	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>	<4D266278.8010702@nextday.fi>	<AANLkTi=sw8pczQk86sBEx2767oHjmLaaBX+JT-5u=4OP@mail.gmail.com>
	<AANLkTim==2PU7nVKd235Xsu5u8P+iE7+Q_oy3X0xbyjJ@mail.gmail.com>
Message-ID: <4D267C30.9080105@nextday.fi>

07.01.2011 04:09, Graham Dumpleton kirjoitti:
> 2011/1/7 Graham Dumpleton<graham.dumpleton at gmail.com>:
>> 2011/1/7 Alex Gr?nholm<alex.gronholm at nextday.fi>:
>>> 07.01.2011 01:14, Graham Dumpleton kirjoitti:
>>>
>>> One other comment about HTTP/1.1 features.
>>>
>>> You will always be battling to have some HTTP/1.1 features work in a
>>> controllable way. This is because WSGI gateways/adapters aren't often
>>> directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
>>> AJP, CGI etc. In this sort of situation you are at the mercy of what
>>> the modules implementing those protocols do, or even are hamstrung by
>>> how those protocols work.
>>>
>>> The classic example is 100-continue processing. This simply cannot
>>> work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
>>> mechanisms where proxying is performed as the protocol being used
>>> doesn't implement a notion of end to end signalling in respect of
>>> 100-continue.
>>>
>>> I think we need some concrete examples to figure out what is and isn't
>>> possible with WSGI 1.0.1.
>>> My motivation for participating in this discussion can be summed up in that
>>> I want the following two applications to work properly:
>>>
>>> - PlasmaDS (Flex Messaging implementation)
>>> - WebDAV
>>>
>>> The PlasmaDS project is the planned Python counterpart to Adobe's BlazeDS.
>>> Interoperability with the existing implementation requires that both the
>>> request and response use chunked transfer encoding, to achieve bidirectional
>>> streaming. I don't really care how this happens, I just want to make sure
>>> that there is nothing preventing it.
>> That can only be done by changing the rules around wsgi.input is used.
>> I'll try and find a reference to where I have posted information about
>> this before, otherwise I'll write something up again about it.
> BTW, even if WSGI specification were changed to allow handling of
> chunked requests, it would not work for FASTCGI, SCGI, AJP, CGI or
> mod_wsgi daemon mode. Also not likely to work on uWSGI either.
>
> This is because all of these work on the expectation that the complete
> request body can be written across to the separate application process
> before actually reading the response from the application.
>
> In other words, both way streaming is not possible.
>
> The only solution which would allow this with Apache is mod_wsgi
> embedded mode, which in mod_wsgi 3.X already has an optional feature
> which can be enabled so as to allow you to step out of current bounds
> of the WSGI specification and use wsgi.input as I will explain, to do
> this both way streaming.
>
> Pure Python HTTP/WSGI servers which are a front facing server could
> also be modified to handle this is WSGI specification were changed,
> but whether those same will work if put behind a web proxy will depend
> on how the front end web proxy works.
Then I suppose this needs to be standardized in PEP 444, wouldn't you agree?
> Graham
>
>>> The WebDAV spec, on the other hand, says
>>> (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
>>>
>>> The 102 (Processing) status code is an interim response used to inform the
>>> client that the server has accepted the complete request, but has not yet
>>> completed it. This status code SHOULD only be sent when the server has a
>>> reasonable expectation that the request will take significant time to
>>> complete. As guidance, if a method is taking longer than 20 seconds (a
>>> reasonable, but arbitrary value) to process the server SHOULD return a 102
>>> (Processing) response. The server MUST send a final response after the
>>> request has been completed.
>> That I don't offhand see a way of being able to do as protocols like
>> SCGI and CGI definitely don't allow interim status. I am suspecting
>> that FASTCGI and AJP don't allow it either.
>>
>> I'll have to even do some digging as to how you would even handle that
>> in Apache with a normal Apache handler.
>>
>> Graham
>>
>>> Again, I don't care how this is done as long as it's possible.
>>>
>>> The current WSGI specification acknowledges that by saying:
>>>
>>> """
>>> Servers and gateways that implement HTTP 1.1 must provide transparent
>>> support for HTTP 1.1's "expect/continue" mechanism. This may be done
>>> in any of several ways:
>>>
>>> * Respond to requests containing an Expect: 100-continue request with
>>> an immediate "100 Continue" response, and proceed normally.
>>> * Proceed with the request normally, but provide the application with
>>> a wsgi.input stream that will send the "100 Continue" response if/when
>>> the application first attempts to read from the input stream. The read
>>> request must then remain blocked until the client responds.
>>> * Wait until the client decides that the server does not support
>>> expect/continue, and sends the request body on its own. (This is
>>> suboptimal, and is not recommended.)
>>> """
>>>
>>> If you are going to try and push for full visibility of HTTP/1.1 and
>>> an ability to control it at the application level then you will fail
>>> with 100-continue to start with.
>>>
>>> So, although option 2 above would be the most ideal and is giving the
>>> application control, specifically the ability to send an error
>>> response based on request headers alone, and with reading the response
>>> and triggering the 100-continue, it isn't practical to require it, as
>>> the majority of hosting mechanisms for WSGI wouldn't even be able to
>>> implement it that way.
>>>
>>> The same goes for any other feature, there is no point mandating a
>>> feature that can only be realistically implementing on a minority of
>>> implementations. This would be even worse where dependence on such a
>>> feature would mean that the WSGI application would no longer be
>>> portable to another WSGI server and destroys the notion that WSGI
>>> provides a portable interface.
>>>
>>> This isn't just restricted to HTTP/1.1 features either, but also
>>> applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
>>> that are directly hooked into the URL parsing of the base HTTP server
>>> can provide that information, which basically means that only pure
>>> Python HTTP/WSGI servers are likely able to provide it without
>>> guessing, and in that case such servers usually are always used where
>>> WSGI application mounted at root anyway.
>>>
>>> Graham
>>>
>>> On 7 January 2011 09:29, Graham Dumpleton<graham.dumpleton at gmail.com>
>>> wrote:
>>>
>>> On 7 January 2011 08:56, Alice Bevan?McGregor<alice at gothcandy.com>  wrote:
>>>
>>> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>>>
>>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>>
>>> :: Making optional (and thus rarely-implemented) features non-optional.
>>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>>> applications to 1.1 servers.  Thus pipelining, chunked encoding, et. al. as
>>> per the HTTP 1.1 RFC.
>>>
>>> Requirements on the HTTP compliance of the server don't really have any
>>> place in the WSGI spec. You should be able to be WSGI compliant even if you
>>> don't use the HTTP transport at all (e.g. maybe you just send around
>>> requests via SCGI).
>>> The original spec got this right: chunking etc are something which is not
>>> relevant to the wsgi application code -- it is up to the server to implement
>>> the HTTP transport according to the HTTP spec, if it's purporting to be an
>>> HTTP server.
>>>
>>> Chunking is actually quite relevant to the specification, as WSGI and PEP
>>> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow for
>>> chunked bodies regardless of higher-level support for chunking.  The body
>>> iterator.  Previously you /had/ to define a length, with chunked encoding at
>>> the server level, you don't.
>>>
>>> I agree, however, that not all gateways will be able to implement the
>>> relevant HTTP/1.1 features.  FastCGI does, SCGI after a quick Google search,
>>> seems to support it as well. I should re-word it as:
>>>
>>> "For those servers capable of HTTP/1.1 features the implementation of such
>>> features is required."
>>>
>>> I would question whether FASTCGI, SCGI or AJP support the concept of
>>> chunking of responses to the extent that the application can prepare
>>> the final content including chunks as required by the HTTP
>>> specification. Further, in Apache at least, the output from a web
>>> application served via those protocols is still pushed through the
>>> Apache output filter chain so as to allow the filters to modify the
>>> response, eg., apply compression using mod_deflate. As a consequence,
>>> the standard HTTP 'CHUNK' output filter is still a part of the output
>>> filter stack. This means that were a web application to try and do
>>> chunking itself, then Apache would rechunk such that the original
>>> chunking became part of the content, rather than the transfer
>>> encoding.
>>>
>>> So, in order to be able to achieve what I think you want, with a web
>>> application being able to do chunking itself, you would need to modify
>>> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
>>> also like mod_cgi and mod_cgid of Apache.
>>>
>>> The only WSGI implementation I know of for Apache where you might even
>>> be able to do what you want is uWSGI. This is because I believe from
>>> memory it uses a mode in Apache by default called assbackwords. What
>>> this allows is for the output from the web application to bypass the
>>> Apache output filter stack and directly control the raw HTTP output.
>>> This gives uWSGI a little bit less overhead in Apache, but at the loss
>>> of the ability to actually use Apache output filters and for Apache to
>>> fix up response headers in any way. There is a flag in uWSGI which can
>>> optionally be set to make it use the more traditional mode and not use
>>> assbackwords.
>>>
>>> Thus, I believe you would be fighting against server implementations
>>> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
>>> chunking to be supported at the level of the web application.
>>>
>>> About all you can do is ensure that the WSGI specification doesn't
>>> include anything in it which would prevent a web application
>>> harnessing indirectly such a feature as chunking where the web server
>>> supports it.
>>>
>>> As it is, it isn't chunked responses which is even the problem,
>>> because if a underlying web server supports chunking for responses,
>>> all you need to do is not set the content length.
>>>
>>> The problem area with chunking is the request content as the way that
>>> the WSGI specification is written prevents being able to have chunked
>>> request content. I have described the issue previously and made
>>> suggestions about alternate way that wsgi.input could be used.
>>>
>>> Graham
>>>
>>> +1
>>>
>>>         - Alice.
>>


From graham.dumpleton at gmail.com  Fri Jan  7 03:55:15 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 13:55:15 +1100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <4D267C30.9080105@nextday.fi>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
	<4D266278.8010702@nextday.fi>
	<AANLkTi=sw8pczQk86sBEx2767oHjmLaaBX+JT-5u=4OP@mail.gmail.com>
	<AANLkTim==2PU7nVKd235Xsu5u8P+iE7+Q_oy3X0xbyjJ@mail.gmail.com>
	<4D267C30.9080105@nextday.fi>
Message-ID: <AANLkTin2uEcn_KPkjpg5oCZggxx5q3RtFip-NtGthJO6@mail.gmail.com>

2011/1/7 Alex Gr?nholm <alex.gronholm at nextday.fi>:
> 07.01.2011 04:09, Graham Dumpleton kirjoitti:
>>
>> 2011/1/7 Graham Dumpleton<graham.dumpleton at gmail.com>:
>>>
>>> 2011/1/7 Alex Gr?nholm<alex.gronholm at nextday.fi>:
>>>>
>>>> 07.01.2011 01:14, Graham Dumpleton kirjoitti:
>>>>
>>>> One other comment about HTTP/1.1 features.
>>>>
>>>> You will always be battling to have some HTTP/1.1 features work in a
>>>> controllable way. This is because WSGI gateways/adapters aren't often
>>>> directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
>>>> AJP, CGI etc. In this sort of situation you are at the mercy of what
>>>> the modules implementing those protocols do, or even are hamstrung by
>>>> how those protocols work.
>>>>
>>>> The classic example is 100-continue processing. This simply cannot
>>>> work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
>>>> mechanisms where proxying is performed as the protocol being used
>>>> doesn't implement a notion of end to end signalling in respect of
>>>> 100-continue.
>>>>
>>>> I think we need some concrete examples to figure out what is and isn't
>>>> possible with WSGI 1.0.1.
>>>> My motivation for participating in this discussion can be summed up in
>>>> that
>>>> I want the following two applications to work properly:
>>>>
>>>> - PlasmaDS (Flex Messaging implementation)
>>>> - WebDAV
>>>>
>>>> The PlasmaDS project is the planned Python counterpart to Adobe's
>>>> BlazeDS.
>>>> Interoperability with the existing implementation requires that both the
>>>> request and response use chunked transfer encoding, to achieve
>>>> bidirectional
>>>> streaming. I don't really care how this happens, I just want to make
>>>> sure
>>>> that there is nothing preventing it.
>>>
>>> That can only be done by changing the rules around wsgi.input is used.
>>> I'll try and find a reference to where I have posted information about
>>> this before, otherwise I'll write something up again about it.
>>
>> BTW, even if WSGI specification were changed to allow handling of
>> chunked requests, it would not work for FASTCGI, SCGI, AJP, CGI or
>> mod_wsgi daemon mode. Also not likely to work on uWSGI either.
>>
>> This is because all of these work on the expectation that the complete
>> request body can be written across to the separate application process
>> before actually reading the response from the application.
>>
>> In other words, both way streaming is not possible.
>>
>> The only solution which would allow this with Apache is mod_wsgi
>> embedded mode, which in mod_wsgi 3.X already has an optional feature
>> which can be enabled so as to allow you to step out of current bounds
>> of the WSGI specification and use wsgi.input as I will explain, to do
>> this both way streaming.
>>
>> Pure Python HTTP/WSGI servers which are a front facing server could
>> also be modified to handle this is WSGI specification were changed,
>> but whether those same will work if put behind a web proxy will depend
>> on how the front end web proxy works.
>
> Then I suppose this needs to be standardized in PEP 444, wouldn't you agree?

Huh! Not sure you understand what I am saying. Even if you changed the
WSGI specification to allow for it, the bulk of implementations
wouldn't be able to support it. The WSGI specification has no
influence over distinct protocols such as FASTCGI, SCGI, AJP or CGI or
proxy implementations and so cant be used to force them to be changed.

So, as much as I would like to see WSGI specification changed to allow
it, others may not on the basis that there is no point if few
implementations could support it.

Graham

>> Graham
>>
>>>> The WebDAV spec, on the other hand, says
>>>> (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
>>>>
>>>> The 102 (Processing) status code is an interim response used to inform
>>>> the
>>>> client that the server has accepted the complete request, but has not
>>>> yet
>>>> completed it. This status code SHOULD only be sent when the server has a
>>>> reasonable expectation that the request will take significant time to
>>>> complete. As guidance, if a method is taking longer than 20 seconds (a
>>>> reasonable, but arbitrary value) to process the server SHOULD return a
>>>> 102
>>>> (Processing) response. The server MUST send a final response after the
>>>> request has been completed.
>>>
>>> That I don't offhand see a way of being able to do as protocols like
>>> SCGI and CGI definitely don't allow interim status. I am suspecting
>>> that FASTCGI and AJP don't allow it either.
>>>
>>> I'll have to even do some digging as to how you would even handle that
>>> in Apache with a normal Apache handler.
>>>
>>> Graham
>>>
>>>> Again, I don't care how this is done as long as it's possible.
>>>>
>>>> The current WSGI specification acknowledges that by saying:
>>>>
>>>> """
>>>> Servers and gateways that implement HTTP 1.1 must provide transparent
>>>> support for HTTP 1.1's "expect/continue" mechanism. This may be done
>>>> in any of several ways:
>>>>
>>>> * Respond to requests containing an Expect: 100-continue request with
>>>> an immediate "100 Continue" response, and proceed normally.
>>>> * Proceed with the request normally, but provide the application with
>>>> a wsgi.input stream that will send the "100 Continue" response if/when
>>>> the application first attempts to read from the input stream. The read
>>>> request must then remain blocked until the client responds.
>>>> * Wait until the client decides that the server does not support
>>>> expect/continue, and sends the request body on its own. (This is
>>>> suboptimal, and is not recommended.)
>>>> """
>>>>
>>>> If you are going to try and push for full visibility of HTTP/1.1 and
>>>> an ability to control it at the application level then you will fail
>>>> with 100-continue to start with.
>>>>
>>>> So, although option 2 above would be the most ideal and is giving the
>>>> application control, specifically the ability to send an error
>>>> response based on request headers alone, and with reading the response
>>>> and triggering the 100-continue, it isn't practical to require it, as
>>>> the majority of hosting mechanisms for WSGI wouldn't even be able to
>>>> implement it that way.
>>>>
>>>> The same goes for any other feature, there is no point mandating a
>>>> feature that can only be realistically implementing on a minority of
>>>> implementations. This would be even worse where dependence on such a
>>>> feature would mean that the WSGI application would no longer be
>>>> portable to another WSGI server and destroys the notion that WSGI
>>>> provides a portable interface.
>>>>
>>>> This isn't just restricted to HTTP/1.1 features either, but also
>>>> applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
>>>> that are directly hooked into the URL parsing of the base HTTP server
>>>> can provide that information, which basically means that only pure
>>>> Python HTTP/WSGI servers are likely able to provide it without
>>>> guessing, and in that case such servers usually are always used where
>>>> WSGI application mounted at root anyway.
>>>>
>>>> Graham
>>>>
>>>> On 7 January 2011 09:29, Graham Dumpleton<graham.dumpleton at gmail.com>
>>>> wrote:
>>>>
>>>> On 7 January 2011 08:56, Alice Bevan?McGregor<alice at gothcandy.com>
>>>> ?wrote:
>>>>
>>>> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>>>>
>>>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>>>
>>>> :: Making optional (and thus rarely-implemented) features non-optional.
>>>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>>>> applications to 1.1 servers. ?Thus pipelining, chunked encoding, et. al.
>>>> as
>>>> per the HTTP 1.1 RFC.
>>>>
>>>> Requirements on the HTTP compliance of the server don't really have any
>>>> place in the WSGI spec. You should be able to be WSGI compliant even if
>>>> you
>>>> don't use the HTTP transport at all (e.g. maybe you just send around
>>>> requests via SCGI).
>>>> The original spec got this right: chunking etc are something which is
>>>> not
>>>> relevant to the wsgi application code -- it is up to the server to
>>>> implement
>>>> the HTTP transport according to the HTTP spec, if it's purporting to be
>>>> an
>>>> HTTP server.
>>>>
>>>> Chunking is actually quite relevant to the specification, as WSGI and
>>>> PEP
>>>> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow
>>>> for
>>>> chunked bodies regardless of higher-level support for chunking. ?The
>>>> body
>>>> iterator. ?Previously you /had/ to define a length, with chunked
>>>> encoding at
>>>> the server level, you don't.
>>>>
>>>> I agree, however, that not all gateways will be able to implement the
>>>> relevant HTTP/1.1 features. ?FastCGI does, SCGI after a quick Google
>>>> search,
>>>> seems to support it as well. I should re-word it as:
>>>>
>>>> "For those servers capable of HTTP/1.1 features the implementation of
>>>> such
>>>> features is required."
>>>>
>>>> I would question whether FASTCGI, SCGI or AJP support the concept of
>>>> chunking of responses to the extent that the application can prepare
>>>> the final content including chunks as required by the HTTP
>>>> specification. Further, in Apache at least, the output from a web
>>>> application served via those protocols is still pushed through the
>>>> Apache output filter chain so as to allow the filters to modify the
>>>> response, eg., apply compression using mod_deflate. As a consequence,
>>>> the standard HTTP 'CHUNK' output filter is still a part of the output
>>>> filter stack. This means that were a web application to try and do
>>>> chunking itself, then Apache would rechunk such that the original
>>>> chunking became part of the content, rather than the transfer
>>>> encoding.
>>>>
>>>> So, in order to be able to achieve what I think you want, with a web
>>>> application being able to do chunking itself, you would need to modify
>>>> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
>>>> also like mod_cgi and mod_cgid of Apache.
>>>>
>>>> The only WSGI implementation I know of for Apache where you might even
>>>> be able to do what you want is uWSGI. This is because I believe from
>>>> memory it uses a mode in Apache by default called assbackwords. What
>>>> this allows is for the output from the web application to bypass the
>>>> Apache output filter stack and directly control the raw HTTP output.
>>>> This gives uWSGI a little bit less overhead in Apache, but at the loss
>>>> of the ability to actually use Apache output filters and for Apache to
>>>> fix up response headers in any way. There is a flag in uWSGI which can
>>>> optionally be set to make it use the more traditional mode and not use
>>>> assbackwords.
>>>>
>>>> Thus, I believe you would be fighting against server implementations
>>>> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
>>>> chunking to be supported at the level of the web application.
>>>>
>>>> About all you can do is ensure that the WSGI specification doesn't
>>>> include anything in it which would prevent a web application
>>>> harnessing indirectly such a feature as chunking where the web server
>>>> supports it.
>>>>
>>>> As it is, it isn't chunked responses which is even the problem,
>>>> because if a underlying web server supports chunking for responses,
>>>> all you need to do is not set the content length.
>>>>
>>>> The problem area with chunking is the request content as the way that
>>>> the WSGI specification is written prevents being able to have chunked
>>>> request content. I have described the issue previously and made
>>>> suggestions about alternate way that wsgi.input could be used.
>>>>
>>>> Graham
>>>>
>>>> +1
>>>>
>>>> ? ? ? ?- Alice.
>>>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>

From foom at fuhm.net  Fri Jan  7 03:55:20 2011
From: foom at fuhm.net (James Y Knight)
Date: Thu, 6 Jan 2011 21:55:20 -0500
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <4D266278.8010702@nextday.fi>
References: <11174894.6.1294342577349.JavaMail.root@boaz>	<ig5a2d$r5s$1@dough.gmane.org>	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>	<ig5dpo$eui$1@dough.gmane.org>	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
	<4D266278.8010702@nextday.fi>
Message-ID: <87F7F151-0C29-4FBE-93CA-8613C8DE499F@fuhm.net>


On Jan 6, 2011, at 7:46 PM, Alex Gr?nholm wrote:

> The WebDAV spec, on the other hand, says (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
> 
> The 102 (Processing) status code is an interim response used to inform the client that the server has accepted the complete request, but has not yet completed it. This status code SHOULD only be sent when the server has a reasonable expectation that the request will take significant time to complete. As guidance, if a method is taking longer than 20 seconds (a reasonable, but arbitrary value) to process the serverSHOULD return a 102 (Processing) response. The server MUST send a final response after the request has been completed.
> 
> Again, I don't care how this is done as long as it's possible.

This pretty much has to be generated by the server implementation. One thing that could be done in WSGI is a callback function inserted into the environ to suggest to the server that it generate a certain 1xx response. That is, something like:
  if 'wsgi.intermediate_response' in environ:
    environ['wsgi.intermediate_response'](102, {'Random-Header': 'Whatever'})

If a server implements this, it should probably ignore any requests from the app to send a 100 or 101 response. The server should be free to ignore the request, or not implement it. Given that the only actual use case (WebDAV) is rather rare and marks it as a SHOULD, I don't see any real practical issues with it being optional.

The other thing that could be done is simply have a server-side configuration to allow sending 102 after *any* request takes > 20 seconds to process. That wouldn't require any changes to WSGI.

I'd note that HTTP/1.1 clients are *required* to be able to handle any number of 1xx responses followed by a final response, so it's supposed to be perfectly safe for a server to always send a 102 as a response to any request, no matter what the app is, or what client user-agent is (so long as it advertised HTTP/1.1), or even whether the resource has anything to do with WebDAV. Of course, I'm willing to bet that's patently false back here in the Real World -- no doubt plenty of "HTTP/1.1" clients incorrectly barf on 1xx responses.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20110106/7e854ddb/attachment.html>

From graham.dumpleton at gmail.com  Fri Jan  7 04:08:20 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 14:08:20 +1100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <87F7F151-0C29-4FBE-93CA-8613C8DE499F@fuhm.net>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>
	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>
	<4D266278.8010702@nextday.fi>
	<87F7F151-0C29-4FBE-93CA-8613C8DE499F@fuhm.net>
Message-ID: <AANLkTin=25bNGNx6vZoWafPV2k3WmL8vxhnOAGoYZCqM@mail.gmail.com>

2011/1/7 James Y Knight <foom at fuhm.net>:
>
> On Jan 6, 2011, at 7:46 PM, Alex Gr?nholm wrote:
>
> The WebDAV spec, on the other hand, says
> (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
>
> The 102 (Processing) status code is an interim response used to inform the
> client that the server has accepted the complete request, but has not yet
> completed it. This status code?SHOULD?only be sent when the server has a
> reasonable expectation that the request will take significant time to
> complete. As guidance, if a method is taking longer than 20 seconds (a
> reasonable, but arbitrary value) to process the serverSHOULD?return a 102
> (Processing) response. The server?MUST?send a final response after the
> request has been completed.
>
> Again, I don't care how this is done as long as it's possible.
>
> This pretty much has to be generated by the server implementation. One thing
> that could be done in WSGI is a callback function inserted into the environ
> to suggest to the server that it generate a certain 1xx response. That is,
> something like:
> ??if?'wsgi.intermediate_response' in?environ:
> ?? ?environ['wsgi.intermediate_response'](102, {'Random-Header':
> 'Whatever'})
> If a server implements this, it should probably ignore any requests from the
> app to send a 100 or 101 response.?The server should be free to ignore the
> request, or not implement it.?Given that the only actual use case (WebDAV)
> is rather rare and marks it as a SHOULD, I don't see any real practical
> issues with it being optional.
> The other thing that could be done is simply have a server-side
> configuration to allow sending 102 after *any* request takes > 20 seconds to
> process. That wouldn't require any changes to WSGI.
> I'd note that HTTP/1.1 clients are *required* to be able to handle any
> number of 1xx responses followed by a final response, so it's supposed to be
> perfectly safe for a server to always send a 102 as a response to any
> request, no matter what the app is, or what client user-agent is (so long as
> it advertised HTTP/1.1), or even whether the resource has anything to do
> with WebDAV. Of course, I'm willing to bet that's patently false back here
> in the Real World -- no doubt plenty of "HTTP/1.1" clients incorrectly barf
> on 1xx responses.

FWIW, Apache provides ap_send_interim_response() to allow interim status.

This is used by mod_proxy, but no where else in Apache core code. So,
you would be fine if proxying to a pure Python HTTP/WSGI server which
could generate interim responses, but would be out of luck with
FASTCGI, SCGI, AJP, CGI and any modules which do custom proxying using
own protocol such as uWSGI or mod_wsgi daemon mode.

In all the latter, the wire protocols for proxy connection would
themselves need to be modified as well as module implementation, which
isn't going to happen for any of those which are generic protocols.

Graham

From alex.gronholm at nextday.fi  Fri Jan  7 05:12:20 2011
From: alex.gronholm at nextday.fi (=?windows-1252?Q?Alex_Gr=F6nholm?=)
Date: Fri, 07 Jan 2011 06:12:20 +0200
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <AANLkTin2uEcn_KPkjpg5oCZggxx5q3RtFip-NtGthJO6@mail.gmail.com>
References: <11174894.6.1294342577349.JavaMail.root@boaz>	<ig5a2d$r5s$1@dough.gmane.org>	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>	<ig5dpo$eui$1@dough.gmane.org>	<AANLkTimsw9G0u735Pu1YvcgEFir3d+FzaZZ8brSgcqdg@mail.gmail.com>	<AANLkTi==hbsLpkNbqWoJ4-ia3goQRyNN23kxoBPpJ2Px@mail.gmail.com>	<4D266278.8010702@nextday.fi>	<AANLkTi=sw8pczQk86sBEx2767oHjmLaaBX+JT-5u=4OP@mail.gmail.com>	<AANLkTim==2PU7nVKd235Xsu5u8P+iE7+Q_oy3X0xbyjJ@mail.gmail.com>	<4D267C30.9080105@nextday.fi>
	<AANLkTin2uEcn_KPkjpg5oCZggxx5q3RtFip-NtGthJO6@mail.gmail.com>
Message-ID: <4D2692A4.8040000@nextday.fi>

07.01.2011 04:55, Graham Dumpleton kirjoitti:
> 2011/1/7 Alex Gr?nholm<alex.gronholm at nextday.fi>:
>> 07.01.2011 04:09, Graham Dumpleton kirjoitti:
>>> 2011/1/7 Graham Dumpleton<graham.dumpleton at gmail.com>:
>>>> 2011/1/7 Alex Gr?nholm<alex.gronholm at nextday.fi>:
>>>>> 07.01.2011 01:14, Graham Dumpleton kirjoitti:
>>>>>
>>>>> One other comment about HTTP/1.1 features.
>>>>>
>>>>> You will always be battling to have some HTTP/1.1 features work in a
>>>>> controllable way. This is because WSGI gateways/adapters aren't often
>>>>> directly interfacing with the raw HTTP layer, but with FASTCGI, SCGI,
>>>>> AJP, CGI etc. In this sort of situation you are at the mercy of what
>>>>> the modules implementing those protocols do, or even are hamstrung by
>>>>> how those protocols work.
>>>>>
>>>>> The classic example is 100-continue processing. This simply cannot
>>>>> work end to end across FASTCGI, SCGI, AJP, CGI and other WSGI hosting
>>>>> mechanisms where proxying is performed as the protocol being used
>>>>> doesn't implement a notion of end to end signalling in respect of
>>>>> 100-continue.
>>>>>
>>>>> I think we need some concrete examples to figure out what is and isn't
>>>>> possible with WSGI 1.0.1.
>>>>> My motivation for participating in this discussion can be summed up in
>>>>> that
>>>>> I want the following two applications to work properly:
>>>>>
>>>>> - PlasmaDS (Flex Messaging implementation)
>>>>> - WebDAV
>>>>>
>>>>> The PlasmaDS project is the planned Python counterpart to Adobe's
>>>>> BlazeDS.
>>>>> Interoperability with the existing implementation requires that both the
>>>>> request and response use chunked transfer encoding, to achieve
>>>>> bidirectional
>>>>> streaming. I don't really care how this happens, I just want to make
>>>>> sure
>>>>> that there is nothing preventing it.
>>>> That can only be done by changing the rules around wsgi.input is used.
>>>> I'll try and find a reference to where I have posted information about
>>>> this before, otherwise I'll write something up again about it.
>>> BTW, even if WSGI specification were changed to allow handling of
>>> chunked requests, it would not work for FASTCGI, SCGI, AJP, CGI or
>>> mod_wsgi daemon mode. Also not likely to work on uWSGI either.
>>>
>>> This is because all of these work on the expectation that the complete
>>> request body can be written across to the separate application process
>>> before actually reading the response from the application.
>>>
>>> In other words, both way streaming is not possible.
>>>
>>> The only solution which would allow this with Apache is mod_wsgi
>>> embedded mode, which in mod_wsgi 3.X already has an optional feature
>>> which can be enabled so as to allow you to step out of current bounds
>>> of the WSGI specification and use wsgi.input as I will explain, to do
>>> this both way streaming.
>>>
>>> Pure Python HTTP/WSGI servers which are a front facing server could
>>> also be modified to handle this is WSGI specification were changed,
>>> but whether those same will work if put behind a web proxy will depend
>>> on how the front end web proxy works.
>> Then I suppose this needs to be standardized in PEP 444, wouldn't you agree?
> Huh! Not sure you understand what I am saying. Even if you changed the
> WSGI specification to allow for it, the bulk of implementations
> wouldn't be able to support it. The WSGI specification has no
> influence over distinct protocols such as FASTCGI, SCGI, AJP or CGI or
> proxy implementations and so cant be used to force them to be changed.
I believe I understand what you are saying, but I don't want to restrict 
the freedom of the developer just because of some implementations that 
can't support some particular feature. If you need to do streaming, use 
a server that supports it, obviously! If Java can do it, why can't we? I 
would hate having to rely on a non-standard implementation if we have 
the possibility to standardize this in a specification.
> So, as much as I would like to see WSGI specification changed to allow
> it, others may not on the basis that there is no point if few
> implementations could support it.
>
> Graham
>
>>> Graham
>>>
>>>>> The WebDAV spec, on the other hand, says
>>>>> (http://www.webdav.org/specs/rfc2518.html#STATUS_102):
>>>>>
>>>>> The 102 (Processing) status code is an interim response used to inform
>>>>> the
>>>>> client that the server has accepted the complete request, but has not
>>>>> yet
>>>>> completed it. This status code SHOULD only be sent when the server has a
>>>>> reasonable expectation that the request will take significant time to
>>>>> complete. As guidance, if a method is taking longer than 20 seconds (a
>>>>> reasonable, but arbitrary value) to process the server SHOULD return a
>>>>> 102
>>>>> (Processing) response. The server MUST send a final response after the
>>>>> request has been completed.
>>>> That I don't offhand see a way of being able to do as protocols like
>>>> SCGI and CGI definitely don't allow interim status. I am suspecting
>>>> that FASTCGI and AJP don't allow it either.
>>>>
>>>> I'll have to even do some digging as to how you would even handle that
>>>> in Apache with a normal Apache handler.
>>>>
>>>> Graham
>>>>
>>>>> Again, I don't care how this is done as long as it's possible.
>>>>>
>>>>> The current WSGI specification acknowledges that by saying:
>>>>>
>>>>> """
>>>>> Servers and gateways that implement HTTP 1.1 must provide transparent
>>>>> support for HTTP 1.1's "expect/continue" mechanism. This may be done
>>>>> in any of several ways:
>>>>>
>>>>> * Respond to requests containing an Expect: 100-continue request with
>>>>> an immediate "100 Continue" response, and proceed normally.
>>>>> * Proceed with the request normally, but provide the application with
>>>>> a wsgi.input stream that will send the "100 Continue" response if/when
>>>>> the application first attempts to read from the input stream. The read
>>>>> request must then remain blocked until the client responds.
>>>>> * Wait until the client decides that the server does not support
>>>>> expect/continue, and sends the request body on its own. (This is
>>>>> suboptimal, and is not recommended.)
>>>>> """
>>>>>
>>>>> If you are going to try and push for full visibility of HTTP/1.1 and
>>>>> an ability to control it at the application level then you will fail
>>>>> with 100-continue to start with.
>>>>>
>>>>> So, although option 2 above would be the most ideal and is giving the
>>>>> application control, specifically the ability to send an error
>>>>> response based on request headers alone, and with reading the response
>>>>> and triggering the 100-continue, it isn't practical to require it, as
>>>>> the majority of hosting mechanisms for WSGI wouldn't even be able to
>>>>> implement it that way.
>>>>>
>>>>> The same goes for any other feature, there is no point mandating a
>>>>> feature that can only be realistically implementing on a minority of
>>>>> implementations. This would be even worse where dependence on such a
>>>>> feature would mean that the WSGI application would no longer be
>>>>> portable to another WSGI server and destroys the notion that WSGI
>>>>> provides a portable interface.
>>>>>
>>>>> This isn't just restricted to HTTP/1.1 features either, but also
>>>>> applies to raw SCRIPT_NAME and PATH_INFO as well. Only WSGI servers
>>>>> that are directly hooked into the URL parsing of the base HTTP server
>>>>> can provide that information, which basically means that only pure
>>>>> Python HTTP/WSGI servers are likely able to provide it without
>>>>> guessing, and in that case such servers usually are always used where
>>>>> WSGI application mounted at root anyway.
>>>>>
>>>>> Graham
>>>>>
>>>>> On 7 January 2011 09:29, Graham Dumpleton<graham.dumpleton at gmail.com>
>>>>> wrote:
>>>>>
>>>>> On 7 January 2011 08:56, Alice Bevan?McGregor<alice at gothcandy.com>
>>>>>   wrote:
>>>>>
>>>>> On 2011-01-06 13:06:36 -0800, James Y Knight said:
>>>>>
>>>>> On Jan 6, 2011, at 3:52 PM, Alice Bevan?McGregor wrote:
>>>>>
>>>>> :: Making optional (and thus rarely-implemented) features non-optional.
>>>>> E.g. server support for HTTP/1.1 with clarifications for interfacing
>>>>> applications to 1.1 servers.  Thus pipelining, chunked encoding, et. al.
>>>>> as
>>>>> per the HTTP 1.1 RFC.
>>>>>
>>>>> Requirements on the HTTP compliance of the server don't really have any
>>>>> place in the WSGI spec. You should be able to be WSGI compliant even if
>>>>> you
>>>>> don't use the HTTP transport at all (e.g. maybe you just send around
>>>>> requests via SCGI).
>>>>> The original spec got this right: chunking etc are something which is
>>>>> not
>>>>> relevant to the wsgi application code -- it is up to the server to
>>>>> implement
>>>>> the HTTP transport according to the HTTP spec, if it's purporting to be
>>>>> an
>>>>> HTTP server.
>>>>>
>>>>> Chunking is actually quite relevant to the specification, as WSGI and
>>>>> PEP
>>>>> 444 / WSGI 2 (damn, that's getting tedious to keep dual-typing ;) allow
>>>>> for
>>>>> chunked bodies regardless of higher-level support for chunking.  The
>>>>> body
>>>>> iterator.  Previously you /had/ to define a length, with chunked
>>>>> encoding at
>>>>> the server level, you don't.
>>>>>
>>>>> I agree, however, that not all gateways will be able to implement the
>>>>> relevant HTTP/1.1 features.  FastCGI does, SCGI after a quick Google
>>>>> search,
>>>>> seems to support it as well. I should re-word it as:
>>>>>
>>>>> "For those servers capable of HTTP/1.1 features the implementation of
>>>>> such
>>>>> features is required."
>>>>>
>>>>> I would question whether FASTCGI, SCGI or AJP support the concept of
>>>>> chunking of responses to the extent that the application can prepare
>>>>> the final content including chunks as required by the HTTP
>>>>> specification. Further, in Apache at least, the output from a web
>>>>> application served via those protocols is still pushed through the
>>>>> Apache output filter chain so as to allow the filters to modify the
>>>>> response, eg., apply compression using mod_deflate. As a consequence,
>>>>> the standard HTTP 'CHUNK' output filter is still a part of the output
>>>>> filter stack. This means that were a web application to try and do
>>>>> chunking itself, then Apache would rechunk such that the original
>>>>> chunking became part of the content, rather than the transfer
>>>>> encoding.
>>>>>
>>>>> So, in order to be able to achieve what I think you want, with a web
>>>>> application being able to do chunking itself, you would need to modify
>>>>> the implementations of mod_fcgid, mod_fastcgi, mod_scgi, mod_ajp and
>>>>> also like mod_cgi and mod_cgid of Apache.
>>>>>
>>>>> The only WSGI implementation I know of for Apache where you might even
>>>>> be able to do what you want is uWSGI. This is because I believe from
>>>>> memory it uses a mode in Apache by default called assbackwords. What
>>>>> this allows is for the output from the web application to bypass the
>>>>> Apache output filter stack and directly control the raw HTTP output.
>>>>> This gives uWSGI a little bit less overhead in Apache, but at the loss
>>>>> of the ability to actually use Apache output filters and for Apache to
>>>>> fix up response headers in any way. There is a flag in uWSGI which can
>>>>> optionally be set to make it use the more traditional mode and not use
>>>>> assbackwords.
>>>>>
>>>>> Thus, I believe you would be fighting against server implementations
>>>>> such as Apache and likely also nginx, Cherokee, lighttpd etc, to allow
>>>>> chunking to be supported at the level of the web application.
>>>>>
>>>>> About all you can do is ensure that the WSGI specification doesn't
>>>>> include anything in it which would prevent a web application
>>>>> harnessing indirectly such a feature as chunking where the web server
>>>>> supports it.
>>>>>
>>>>> As it is, it isn't chunked responses which is even the problem,
>>>>> because if a underlying web server supports chunking for responses,
>>>>> all you need to do is not set the content length.
>>>>>
>>>>> The problem area with chunking is the request content as the way that
>>>>> the WSGI specification is written prevents being able to have chunked
>>>>> request content. I have described the issue previously and made
>>>>> suggestions about alternate way that wsgi.input could be used.
>>>>>
>>>>> Graham
>>>>>
>>>>> +1
>>>>>
>>>>>         - Alice.
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe:
>> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>>


From pje at telecommunity.com  Fri Jan  7 05:18:12 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 06 Jan 2011 23:18:12 -0500
Subject: [Web-SIG] PEP 444 Goals
Message-ID: <20110107041816.8224E3A40A8@sparrow.telecommunity.com>

At 12:52 PM 1/6/2011 -0800, Alice Bevan?McGregor wrote:
>Ignoring async for the moment, the goals of the PEP 444 rewrite are:
>
>:: Clear separation of "narrative" from "rules to be 
>followed".  This allows developers of both servers and applications 
>to easily run through a confomance "check list".
>
>:: Isolation of examples and rationale to improve readability of the 
>core rulesets.
>
>:: Clarification of often mis-interpreted rules from PEP 333 (and 
>those carried over in 3333).
>
>:: Elimination of unintentional non-conformance, esp. re: cgi.FieldStorage.
>
>:: Massive simplification of call flow.  Replacing start_response 
>with a returned 3-tuple immensely simplifies the task of middleware 
>that needs to capture HTTP status or manipulate (or even examine) 
>response headers. [1]

A big +1 to all the above as goals.


>:: Reduction of re-implementation / NIH syndrome by incorporating 
>the most common (1%) of features most often relegated to middleware 
>or functional helpers.

Note that nearly every application-friendly feature you add will 
increase the burden on both server developers and middleware 
developers, which ironically means that application developers 
actually end up with fewer options.


>   Unicode decoding of a small handful of values (CGI values that 
> pull from the request URI) is the biggest example. [2, 3]

Does that mean you plan to make the other values bytes, then?  Or 
will they be unicode-y-bytes as well?  What happens for additional 
server-provided variables?

The PEP 3333 choice was for uniformity.  At one point, I advocated 
simply using surrogateescape coding, but this couldn't be made 
uniform across Python versions and maintain compatibility.

Unfortunately, even with the move to 2.6+, this problem remains, 
unless server providers are required to register a surrogateescape 
error handler -- which I'm not even sure can be done in Python 2.x.


>:: Cross-compatibility considerations.  The definition and use of 
>native strings vs. byte strings is the biggest example of this in the rewrite.

I'm not sure what you mean here.  Do you mean "portability of WSGI 2 
code samples across Python versions (esp. 2.x vs. 3.x)?"


From pje at telecommunity.com  Fri Jan  7 05:28:03 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 06 Jan 2011 23:28:03 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
Message-ID: <20110107042802.B06473A40A8@sparrow.telecommunity.com>

At 10:33 AM 1/4/2011 -0800, Guido van Rossum wrote:
>But the flush() I was referring to is actually *before* either of
>these, suggesting
>
>sys.stdout.flush()
>sys.stdout.buffer.write(data)
>sys.stdout.buffer.flush()
>
>However the first flush() is only necessary is there's a possibility
>that previously something had been written to sys.stdout (not to
>sys.stdout.buffer).

Yeah, that sort of error checking seems out of scope for a PEP 
example.  If something was written to sys.stdout at that point, your 
CGI was already broken.  ;-)


> > For the CGI example in the PEP, I don't want to bother trying to make it
> > fully production-usable; that's what we have wsgiref in the stdlib for.  So
> > I won't worry about mixing strings and regular output in the example, even
> > if perhaps wsgiref should add the StringIO's proposed by Graham.
>
>Not sure what you mean. Surely copying and pasting the examples should
>work? What are the details you'd like to leave out?

I meant that a production CGI gateway should handle various 
boundary/error conditions.  Graham was saying that a CGI gateway 
should replace sys.stdout to avoid stray print()s causing problems, 
and I consider that similar to saying, "what if somebody wrote text 
to sys.stdout?" -- i.e., an error handling case that would be a good 
idea in a production gateway, but which would obscure the common case 
that the example is meant to illustrate.


From pje at telecommunity.com  Fri Jan  7 05:30:11 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 06 Jan 2011 23:30:11 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
Message-ID: <20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>

At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>Is that the last thing or do I need to go spend some time and write my
>own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>lying around and just do some final validation checks with a parallel
>implementation as a sanity check to make sure we got everything? This
>might be a good idea anyway.

It would.  In the meantime, though, I've checked in the two-line 
change to add .buffer in.  ;-)


From pje at telecommunity.com  Fri Jan  7 05:49:57 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 06 Jan 2011 23:49:57 -0500
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig5rcb$csc$1@dough.gmane.org>
References: <4D25F9F0.3070306@inteli-com.com>
 <ig5rcb$csc$1@dough.gmane.org>
Message-ID: <20110107044956.D208F3A40A8@sparrow.telecommunity.com>

At 05:47 PM 1/6/2011 -0800, Alice Bevan?McGregor wrote:
>Tossing the idea around all day long will then, of course, be 
>happening regardless.  Unfortunately for that particular discussion, 
>PEP 3148 / Futures seems to have won out in the broader scope.

Do any established async frameworks or server (e.g. Twisted, 
Eventlet, Gevent,  Tornado, etc.) make use of futures?


>   Having a ratified and incorporated language PEP (core in 3.2 w/ 
> compatibility package for 2.5 or 2.6+ support) reduces the scope of 
> async discussion down to: "how do we integrate futures into WSGI 2" 
> instead of "how do we define an async API at all".

It would be helpful if you addressed the issue of scope, i.e., what 
features are you proposing to offer to the application developer.

While the idea of using futures presents some intriguing 
possibilities, it seems to me at first glance that all it will do is 
move the point where the work gets done.  That is, instead of simply 
running the app in a worker, the app will be farming out work to 
futures.  But if this is so, then why doesn't the server just farm 
the apps themselves out to workers?

I guess what I'm saying is, I haven't heard use cases for this from 
the application developer POV -- why should an app developer care 
about having their app run asynchronously?

So far, I believe you're the second major proponent (i.e. ones with 
concrete proposals and/or implementations to discuss) of an async 
protocol...  and what you have in common with the other proponent is 
that you happen to have written an async server that would benefit 
from having apps operating asynchronously.  ;-)

I find it hard to imagine an app developer wanting to do something 
asynchronously for which they would not want to use one of the 
big-dog asynchronous frameworks.  (Especially if their app involves 
database access, or other communications protocols.)

This doesn't mean I think having a futures API is a bad thing, but 
ISTM that a futures extension to WSGI 1 could be defined right now 
using an x-wsgi-org extension in that case...  and you could then 
find out how many people are actually interested in using it.

Mainly, though, what I see is people using the futures thing to 
shuffle off compute-intensive tasks...  but if they do that, then 
they're basically trying to make the server's life easier...  but 
under the existing spec, any truly async server implementing WSGI is 
going to run the *app* in a "future" of some sort already...

Which means that the net result is that putting in async is like 
saying to the app developer: "hey, you know this thing that you just 
could do in WSGI 1 and the server would take care of it for 
you?  Well, now you can manage that complexity by yourself!  Isn't 
that wonderful?"   ;-)

I could be wrong of course, but I'd like to see what concrete use 
cases people have for async.  We dropped the first discussion of 
async six years ago because someone (I think it might've been James) 
pointed out that, well, it isn't actually that useful.  And every 
subsequent call for use cases since has been answered with, "well, 
the use case is that you want it to be async."

Only, that's a *server* developer's use case, not an app developer's 
use case...  and only for a minority of server developers, at that.


From alice at gothcandy.com  Fri Jan  7 06:13:06 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 21:13:06 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <ig5es8$kj0$1@dough.gmane.org>
Message-ID: <ig67d2$rhh$1@dough.gmane.org>

On 2011-01-06 14:14:32 -0800, Alice Bevan?McGregor said:
> There was something, somewhere I was reading related to WSGI about 
> requiring content-length... but no matter.

Right, I remember now: the HTTP 1.0 specification.  (Honestly not 
trying to sound sarcastic!)  See:

	http://www.w3.org/Protocols/HTTP/1.0/draft-ietf-http-spec.html#Entity-Body

However, after testing every browser on my system (from Links and 
ELinks, through Firefox, Chrome, Safari, Konqueror, and Dillo) across 
the following test code, I find that they all handle a missing 
content-length in the same way: reading the socket until it closes.

	http://pastie.textmate.org/1435415

	- Alice.



From alex.gronholm at nextday.fi  Fri Jan  7 06:24:12 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Fri, 07 Jan 2011 07:24:12 +0200
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <20110107044956.D208F3A40A8@sparrow.telecommunity.com>
References: <4D25F9F0.3070306@inteli-com.com> <ig5rcb$csc$1@dough.gmane.org>
	<20110107044956.D208F3A40A8@sparrow.telecommunity.com>
Message-ID: <4D26A37C.2020902@nextday.fi>

07.01.2011 06:49, P.J. Eby kirjoitti:
> At 05:47 PM 1/6/2011 -0800, Alice Bevan?McGregor wrote:
>> Tossing the idea around all day long will then, of course, be 
>> happening regardless.  Unfortunately for that particular discussion, 
>> PEP 3148 / Futures seems to have won out in the broader scope.
>
> Do any established async frameworks or server (e.g. Twisted, Eventlet, 
> Gevent,  Tornado, etc.) make use of futures?
I understand that Twisted has incorporated futures support to their 
deferreds. Others, I believe, don't support them yet. You have to 
consider that Python 3.2 (the first Python with futures support in 
stdlib) hasn't even been released yet, and it's only been two weeks 
since I released the drop-in backport 
(http://pypi.python.org/pypi/futures/2.1).
>
>
>>   Having a ratified and incorporated language PEP (core in 3.2 w/ 
>> compatibility package for 2.5 or 2.6+ support) reduces the scope of 
>> async discussion down to: "how do we integrate futures into WSGI 2" 
>> instead of "how do we define an async API at all".
>
> It would be helpful if you addressed the issue of scope, i.e., what 
> features are you proposing to offer to the application developer.
>
> While the idea of using futures presents some intriguing 
> possibilities, it seems to me at first glance that all it will do is 
> move the point where the work gets done.  That is, instead of simply 
> running the app in a worker, the app will be farming out work to 
> futures.  But if this is so, then why doesn't the server just farm the 
> apps themselves out to workers?
>
> I guess what I'm saying is, I haven't heard use cases for this from 
> the application developer POV -- why should an app developer care 
> about having their app run asynchronously?
Applications need to be asynchronous to work on a single threaded 
server. There is no other benefit than speed and concurrency, and having 
to program a web app to operate asynchronously can be a pain. AFAIK 
there is no other way if you want to avoid the context switching 
overhead and support a huge number of concurrent connections.

Thread/process pools are only necessary in an asynchronous application 
where the app needs to use blocking network APIs or do heavy 
computation, and such uses can unfortunately present a bottleneck. It 
follows that it's pretty pointless to have an asynchronous application 
that uses a thread/process pool on every request.

The goal here is to define a common API for these mutually incompatible 
asynchronous servers to implement so that you could one day run an 
asynchronous app on Twisted, Tornado, or whatever without modifications.
>
> So far, I believe you're the second major proponent (i.e. ones with 
> concrete proposals and/or implementations to discuss) of an async 
> protocol...  and what you have in common with the other proponent is 
> that you happen to have written an async server that would benefit 
> from having apps operating asynchronously.  ;-)
>
> I find it hard to imagine an app developer wanting to do something 
> asynchronously for which they would not want to use one of the big-dog 
> asynchronous frameworks.  (Especially if their app involves database 
> access, or other communications protocols.)
>
> This doesn't mean I think having a futures API is a bad thing, but 
> ISTM that a futures extension to WSGI 1 could be defined right now 
> using an x-wsgi-org extension in that case...  and you could then find 
> out how many people are actually interested in using it.
>
> Mainly, though, what I see is people using the futures thing to 
> shuffle off compute-intensive tasks...  but if they do that, then 
> they're basically trying to make the server's life easier...  but 
> under the existing spec, any truly async server implementing WSGI is 
> going to run the *app* in a "future" of some sort already...
>
> Which means that the net result is that putting in async is like 
> saying to the app developer: "hey, you know this thing that you just 
> could do in WSGI 1 and the server would take care of it for you?  
> Well, now you can manage that complexity by yourself!  Isn't that 
> wonderful?"   ;-)
>
> I could be wrong of course, but I'd like to see what concrete use 
> cases people have for async.  We dropped the first discussion of async 
> six years ago because someone (I think it might've been James) pointed 
> out that, well, it isn't actually that useful.  And every subsequent 
> call for use cases since has been answered with, "well, the use case 
> is that you want it to be async."
>
> Only, that's a *server* developer's use case, not an app developer's 
> use case...  and only for a minority of server developers, at that.
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From foom at fuhm.net  Fri Jan  7 06:26:32 2011
From: foom at fuhm.net (James Y Knight)
Date: Fri, 7 Jan 2011 00:26:32 -0500
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig67d2$rhh$1@dough.gmane.org>
References: <ig5es8$kj0$1@dough.gmane.org> <ig67d2$rhh$1@dough.gmane.org>
Message-ID: <425C6AF3-FFDF-4277-873F-D8DD2F621E57@fuhm.net>


On Jan 7, 2011, at 12:13 AM, Alice Bevan?McGregor wrote:

> On 2011-01-06 14:14:32 -0800, Alice Bevan?McGregor said:
>> There was something, somewhere I was reading related to WSGI about requiring content-length... but no matter.
> 
> Right, I remember now: the HTTP 1.0 specification.  (Honestly not trying to sound sarcastic!)  See:
> 
> 	http://www.w3.org/Protocols/HTTP/1.0/draft-ietf-http-spec.html#Entity-Body

You've misread that section. In HTTP/1.0, *requests* were required to have a Content-Length if they had a body (HTTP 1.1 fixed that with chunked request support). Responses have never had that restriction: they have always (even since before HTTP 1.0) been allowed to omit Content-Length and terminate by closing the socket. 

HTTP 1.1 didn't really add any new functionality to *responses* by adding chunking, simply bit of efficiency and error detection ability.

James

From guido at python.org  Fri Jan  7 06:30:11 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 6 Jan 2011 21:30:11 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <20110107044956.D208F3A40A8@sparrow.telecommunity.com>
References: <4D25F9F0.3070306@inteli-com.com> <ig5rcb$csc$1@dough.gmane.org>
	<20110107044956.D208F3A40A8@sparrow.telecommunity.com>
Message-ID: <AANLkTimYUChbg1T0YMTa0kTMhZNSv73yo84Raa-fbQVh@mail.gmail.com>

On Thu, Jan 6, 2011 at 8:49 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 05:47 PM 1/6/2011 -0800, Alice Bevan?McGregor wrote:
>>
>> Tossing the idea around all day long will then, of course, be happening
>> regardless. ?Unfortunately for that particular discussion, PEP 3148 /
>> Futures seems to have won out in the broader scope.
>
> Do any established async frameworks or server (e.g. Twisted, Eventlet,
> Gevent, ?Tornado, etc.) make use of futures?

PEP 3148 Futures are meant for a rather different purpose than those
async frameworks. Those frameworks all are trying to minimize the
number of threads using some kind of callback-based non-blocking I/O
system. PEP 3148 OTOH doesn't care about that -- it uses threads or
processes proudly. This is useful for a different type of application,
where there are fewer, larger tasks, and the overhead of threads
doesn't matter.

The Monocle framework, which builds on top of Tornado or Twisted, uses
something not entirely unlike Futures, though they call it Callback.

I don't think the acceptance of PEP 3148 should be taken as forcing
the direction that async frameworks should take.

>> ?Having a ratified and incorporated language PEP (core in 3.2 w/
>> compatibility package for 2.5 or 2.6+ support) reduces the scope of async
>> discussion down to: "how do we integrate futures into WSGI 2" instead of
>> "how do we define an async API at all".
>
> It would be helpful if you addressed the issue of scope, i.e., what features
> are you proposing to offer to the application developer.
>
> While the idea of using futures presents some intriguing possibilities, it
> seems to me at first glance that all it will do is move the point where the
> work gets done. ?That is, instead of simply running the app in a worker, the
> app will be farming out work to futures. ?But if this is so, then why
> doesn't the server just farm the apps themselves out to workers?
>
> I guess what I'm saying is, I haven't heard use cases for this from the
> application developer POV -- why should an app developer care about having
> their app run asynchronously?
>
> So far, I believe you're the second major proponent (i.e. ones with concrete
> proposals and/or implementations to discuss) of an async protocol... ?and
> what you have in common with the other proponent is that you happen to have
> written an async server that would benefit from having apps operating
> asynchronously. ?;-)
>
> I find it hard to imagine an app developer wanting to do something
> asynchronously for which they would not want to use one of the big-dog
> asynchronous frameworks. ?(Especially if their app involves database access,
> or other communications protocols.)
>
> This doesn't mean I think having a futures API is a bad thing, but ISTM that
> a futures extension to WSGI 1 could be defined right now using an x-wsgi-org
> extension in that case... ?and you could then find out how many people are
> actually interested in using it.
>
> Mainly, though, what I see is people using the futures thing to shuffle off
> compute-intensive tasks... ?but if they do that, then they're basically
> trying to make the server's life easier... ?but under the existing spec, any
> truly async server implementing WSGI is going to run the *app* in a "future"
> of some sort already...
>
> Which means that the net result is that putting in async is like saying to
> the app developer: "hey, you know this thing that you just could do in WSGI
> 1 and the server would take care of it for you? ?Well, now you can manage
> that complexity by yourself! ?Isn't that wonderful?" ? ;-)
>
> I could be wrong of course, but I'd like to see what concrete use cases
> people have for async. ?We dropped the first discussion of async six years
> ago because someone (I think it might've been James) pointed out that, well,
> it isn't actually that useful. ?And every subsequent call for use cases
> since has been answered with, "well, the use case is that you want it to be
> async."
>
> Only, that's a *server* developer's use case, not an app developer's use
> case... ?and only for a minority of server developers, at that.
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From alice at gothcandy.com  Fri Jan  7 06:31:52 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 21:31:52 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<736F5855-A80F-40D9-924B-850D76F03F87@fuhm.net>
	<ig5dpo$eui$1@dough.gmane.org>
	<6D1FBE4B-46CB-41D7-B5DB-6374B007B661@fuhm.net>
	<ig5es8$kj0$1@dough.gmane.org> <ig67d2$rhh$1@dough.gmane.org>
	<425C6AF3-FFDF-4277-873F-D8DD2F621E57@fuhm.net>
Message-ID: <ig68g8$v59$1@dough.gmane.org>

On 2011-01-06 21:26:32 -0800, James Y Knight said:
> You've misread that section. In HTTP/1.0, *requests* were required to 
> have a Content-Length if they had a body (HTTP 1.1 fixed that with 
> chunked request support). Responses have never had that restriction: 
> they have always (even since before HTTP 1.0) been allowed to omit 
> Content-Length and terminate by closing the socket.

Ah ha, that explains my confusion, then! Thank you.

	- Alice.



From jacob at jacobian.org  Fri Jan  7 06:35:24 2011
From: jacob at jacobian.org (Jacob Kaplan-Moss)
Date: Thu, 6 Jan 2011 23:35:24 -0600
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig5rcb$csc$1@dough.gmane.org>
References: <4D25F9F0.3070306@inteli-com.com> <ig5rcb$csc$1@dough.gmane.org>
Message-ID: <AANLkTinYn4k-HD40td4XyDhX8XkxnvxKX7U3BbC7h+RQ@mail.gmail.com>

On Thu, Jan 6, 2011 at 7:47 PM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
> All input is welcome; I do want to hear from both framework developers and
> users of frameworks.

OK, here's my input. I'm not comfortable speaking on behalf of the
entire Django core team, but I am consciously wearing my Django BDFL
hat, and I do know that many (most?) of Django's core team feels as I
do.

And I'm feeling incredibly disheartened.

Python 3 was released in December 2008. I assumed we'd have an updated
WSGI spec wthin maybe 6 months. It's been two years, and we still
don't have a WSGI spec. This fundamentally means that it's not worth
my time to port Django to Python 3 -- the bits where Django meets WSGI
are critical, and I simply can't get excited about targeting a moving
spec with potential incompatible implementations. The lack of a
WSGI-for-Py3 is a fundamental enthusiasm killer. Django is, at the end
of the day, a framework designed to be deployed. Until I can do so,
there's no use even starting the porting process.

A few months ago, PJE posted PEP 3333. It looked good... and then
nothing happened. I tried  to prod things forward, and some more
discussion ensued... and now it looks like it's stalling again. Each
time, discussion of PEP 444 seems to derail discussion of PEP 3333.

I have no skin in this game. Frankly, I have a huge amount of trouble
following the discussions, and I can't speak to the technical merits
of one over the other. But even if PEP 444 is a million times better
than PEP 3333, 444 is clearly a *lot farther off. But PEP 444 seems to
be where all the energy keeps ending up.

At this rate, I really wonder if it'll be another two years before we
have a working WSGI for Python 3. I hope I'm being pessimistic. Prove
me wrong. Please.

Can we please, please, PLEASE, pause discussion of PEP 444 until PEP
3333 is finalized?

Jacob

From foom at fuhm.net  Fri Jan  7 06:47:29 2011
From: foom at fuhm.net (James Y Knight)
Date: Fri, 7 Jan 2011 00:47:29 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
Message-ID: <D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>

On Jan 6, 2011, at 11:30 PM, P.J. Eby wrote:
> At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>> Is that the last thing or do I need to go spend some time and write my
>> own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>> lying around and just do some final validation checks with a parallel
>> implementation as a sanity check to make sure we got everything? This
>> might be a good idea anyway.
> 
> It would.  In the meantime, though, I've checked in the two-line change to add .buffer in.  ;-)

So does that mean PEP 3333 can be accepted now?

James

From guido at python.org  Fri Jan  7 06:58:09 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 6 Jan 2011 21:58:09 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
Message-ID: <AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>

On Thu, Jan 6, 2011 at 9:47 PM, James Y Knight <foom at fuhm.net> wrote:
> On Jan 6, 2011, at 11:30 PM, P.J. Eby wrote:
>> At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>>> Is that the last thing or do I need to go spend some time and write my
>>> own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>>> lying around and just do some final validation checks with a parallel
>>> implementation as a sanity check to make sure we got everything? This
>>> might be a good idea anyway.
>>
>> It would. ?In the meantime, though, I've checked in the two-line change to add .buffer in. ?;-)
>
> So does that mean PEP 3333 can be accepted now?

TBH I've totally lost track. Hopefully PJE and Graham can tell you...

-- 
--Guido van Rossum (python.org/~guido)

From graham.dumpleton at gmail.com  Fri Jan  7 07:00:17 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 17:00:17 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
Message-ID: <AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>

Stupid question first. When running 2to3 on the example CGI code, why
would it throw back the following. Is this indicative of anything else
that needs to be changed to satisfy some Python 3 thing. The list()
bit seems redundant, but I don't know what the other stuff is about.

--- xx.py (original)
+++ xx.py (refactored)
@@ -9,7 +9,7 @@
     return u.encode(enc, esc).decode('iso-8859-1')

 def run_with_cgi(application):
-    environ = {k: wsgi_string(v) for k,v in os.environ.items()}
+    environ = {k: wsgi_string(v) for k,v in list(os.environ.items())}
     environ['wsgi.input']        = sys.stdin
     environ['wsgi.errors']       = sys.stderr
     environ['wsgi.version']      = (1, 0)
@@ -45,7 +45,7 @@
             try:
                 if headers_sent:
                     # Re-raise original exception if headers sent
-                    raise exc_info[0], exc_info[1], exc_info[2]
+                    raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])
             finally:
                 exc_info = None     # avoid dangling circular ref
         elif headers_set:




On 7 January 2011 16:58, Guido van Rossum <guido at python.org> wrote:
> On Thu, Jan 6, 2011 at 9:47 PM, James Y Knight <foom at fuhm.net> wrote:
>> On Jan 6, 2011, at 11:30 PM, P.J. Eby wrote:
>>> At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>>>> Is that the last thing or do I need to go spend some time and write my
>>>> own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>>>> lying around and just do some final validation checks with a parallel
>>>> implementation as a sanity check to make sure we got everything? This
>>>> might be a good idea anyway.
>>>
>>> It would. ?In the meantime, though, I've checked in the two-line change to add .buffer in. ?;-)
>>
>> So does that mean PEP 3333 can be accepted now?
>
> TBH I've totally lost track. Hopefully PJE and Graham can tell you...
>
> --
> --Guido van Rossum (python.org/~guido)
>

From graham.dumpleton at gmail.com  Fri Jan  7 07:13:58 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 17:13:58 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
Message-ID: <AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>

The version at:

http://svn.python.org/projects/peps/trunk/pep-3333.txt

still shows:

        elif not headers_sent:
             # Before the first output, send the stored headers
             status, response_headers = headers_sent[:] = headers_set
             sys.stdout.write('Status: %s\r\n' % status)
             for header in response_headers:
                 sys.stdout.write('%s: %s\r\n' % header)
             sys.stdout.write('\r\n')

so not using buffer there and also not converting strings written for
headers to bytes.

Graham

On 7 January 2011 17:00, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> Stupid question first. When running 2to3 on the example CGI code, why
> would it throw back the following. Is this indicative of anything else
> that needs to be changed to satisfy some Python 3 thing. The list()
> bit seems redundant, but I don't know what the other stuff is about.
>
> --- xx.py (original)
> +++ xx.py (refactored)
> @@ -9,7 +9,7 @@
> ? ? return u.encode(enc, esc).decode('iso-8859-1')
>
> ?def run_with_cgi(application):
> - ? ?environ = {k: wsgi_string(v) for k,v in os.environ.items()}
> + ? ?environ = {k: wsgi_string(v) for k,v in list(os.environ.items())}
> ? ? environ['wsgi.input'] ? ? ? ?= sys.stdin
> ? ? environ['wsgi.errors'] ? ? ? = sys.stderr
> ? ? environ['wsgi.version'] ? ? ?= (1, 0)
> @@ -45,7 +45,7 @@
> ? ? ? ? ? ? try:
> ? ? ? ? ? ? ? ? if headers_sent:
> ? ? ? ? ? ? ? ? ? ? # Re-raise original exception if headers sent
> - ? ? ? ? ? ? ? ? ? ?raise exc_info[0], exc_info[1], exc_info[2]
> + ? ? ? ? ? ? ? ? ? ?raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])
> ? ? ? ? ? ? finally:
> ? ? ? ? ? ? ? ? exc_info = None ? ? # avoid dangling circular ref
> ? ? ? ? elif headers_set:
>
>
>
>
> On 7 January 2011 16:58, Guido van Rossum <guido at python.org> wrote:
>> On Thu, Jan 6, 2011 at 9:47 PM, James Y Knight <foom at fuhm.net> wrote:
>>> On Jan 6, 2011, at 11:30 PM, P.J. Eby wrote:
>>>> At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>>>>> Is that the last thing or do I need to go spend some time and write my
>>>>> own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>>>>> lying around and just do some final validation checks with a parallel
>>>>> implementation as a sanity check to make sure we got everything? This
>>>>> might be a good idea anyway.
>>>>
>>>> It would. ?In the meantime, though, I've checked in the two-line change to add .buffer in. ?;-)
>>>
>>> So does that mean PEP 3333 can be accepted now?
>>
>> TBH I've totally lost track. Hopefully PJE and Graham can tell you...
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>

From alice at gothcandy.com  Fri Jan  7 07:19:13 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 22:19:13 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
Message-ID: <ig6b91$8l8$1@dough.gmane.org>

On 2011-01-06 22:00:17 -0800, Graham Dumpleton said:
> -    environ = {k: wsgi_string(v) for k,v in os.environ.items()}
> +    environ = {k: wsgi_string(v) for k,v in list(os.environ.items())}

2to3 takes the conservative route of assuming your application treats 
dict.items() as a list in all cases; this is not nessicarily true (of 
course), but it is safe, and interestingly, backwards compatible.

> -                    raise exc_info[0], exc_info[1], exc_info[2]
> +                    raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])

The exception raising syntax has changed; you can not re-raise an 
exception using tuple notation any more.  The new syntax is far 
clearer, but I'm unsure of back-compatibility or even if it is possible 
to emulate it completely as a polygot (2.x and 3.x w/ same code).

	- Alice.



From graham.dumpleton at gmail.com  Fri Jan  7 07:22:18 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 17:22:18 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
Message-ID: <AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>

On 7 January 2011 17:13, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> The version at:
>
> http://svn.python.org/projects/peps/trunk/pep-3333.txt
>
> still shows:
>
> ? ? ? ?elif not headers_sent:
> ? ? ? ? ? ? # Before the first output, send the stored headers
> ? ? ? ? ? ? status, response_headers = headers_sent[:] = headers_set
> ? ? ? ? ? ? sys.stdout.write('Status: %s\r\n' % status)
> ? ? ? ? ? ? for header in response_headers:
> ? ? ? ? ? ? ? ? sys.stdout.write('%s: %s\r\n' % header)
> ? ? ? ? ? ? sys.stdout.write('\r\n')
>
> so not using buffer there and also not converting strings written for
> headers to bytes.

So:

        elif not headers_sent:
             # Before the first output, send the stored headers
             status, response_headers = headers_sent[:] = headers_set
             sys.stdout.buffer.write(wsgi_header('Status: %s\r\n' % status))
             for header in response_headers:
                 sys.stdout.buffer.write(wsgi_header('%s: %s\r\n' % header))
             sys.stdout.buffer.write(wsgi_header('\r\n'))

where define up start of file:

def wsgi_header(u):
    return u.encode('iso-8859-1')

I am still seeing some issue with CRLF but is in my body and with
conversion of some StringIO in my test.

> Graham
>
> On 7 January 2011 17:00, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
>> Stupid question first. When running 2to3 on the example CGI code, why
>> would it throw back the following. Is this indicative of anything else
>> that needs to be changed to satisfy some Python 3 thing. The list()
>> bit seems redundant, but I don't know what the other stuff is about.
>>
>> --- xx.py (original)
>> +++ xx.py (refactored)
>> @@ -9,7 +9,7 @@
>> ? ? return u.encode(enc, esc).decode('iso-8859-1')
>>
>> ?def run_with_cgi(application):
>> - ? ?environ = {k: wsgi_string(v) for k,v in os.environ.items()}
>> + ? ?environ = {k: wsgi_string(v) for k,v in list(os.environ.items())}
>> ? ? environ['wsgi.input'] ? ? ? ?= sys.stdin
>> ? ? environ['wsgi.errors'] ? ? ? = sys.stderr
>> ? ? environ['wsgi.version'] ? ? ?= (1, 0)
>> @@ -45,7 +45,7 @@
>> ? ? ? ? ? ? try:
>> ? ? ? ? ? ? ? ? if headers_sent:
>> ? ? ? ? ? ? ? ? ? ? # Re-raise original exception if headers sent
>> - ? ? ? ? ? ? ? ? ? ?raise exc_info[0], exc_info[1], exc_info[2]
>> + ? ? ? ? ? ? ? ? ? ?raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])
>> ? ? ? ? ? ? finally:
>> ? ? ? ? ? ? ? ? exc_info = None ? ? # avoid dangling circular ref
>> ? ? ? ? elif headers_set:
>>
>>
>>
>>
>> On 7 January 2011 16:58, Guido van Rossum <guido at python.org> wrote:
>>> On Thu, Jan 6, 2011 at 9:47 PM, James Y Knight <foom at fuhm.net> wrote:
>>>> On Jan 6, 2011, at 11:30 PM, P.J. Eby wrote:
>>>>> At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>>>>>> Is that the last thing or do I need to go spend some time and write my
>>>>>> own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>>>>>> lying around and just do some final validation checks with a parallel
>>>>>> implementation as a sanity check to make sure we got everything? This
>>>>>> might be a good idea anyway.
>>>>>
>>>>> It would. ?In the meantime, though, I've checked in the two-line change to add .buffer in. ?;-)
>>>>
>>>> So does that mean PEP 3333 can be accepted now?
>>>
>>> TBH I've totally lost track. Hopefully PJE and Graham can tell you...
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>

From graham.dumpleton at gmail.com  Fri Jan  7 07:27:25 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 17:27:25 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
Message-ID: <AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>

On 7 January 2011 17:22, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> On 7 January 2011 17:13, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
>> The version at:
>>
>> http://svn.python.org/projects/peps/trunk/pep-3333.txt
>>
>> still shows:
>>
>> ? ? ? ?elif not headers_sent:
>> ? ? ? ? ? ? # Before the first output, send the stored headers
>> ? ? ? ? ? ? status, response_headers = headers_sent[:] = headers_set
>> ? ? ? ? ? ? sys.stdout.write('Status: %s\r\n' % status)
>> ? ? ? ? ? ? for header in response_headers:
>> ? ? ? ? ? ? ? ? sys.stdout.write('%s: %s\r\n' % header)
>> ? ? ? ? ? ? sys.stdout.write('\r\n')
>>
>> so not using buffer there and also not converting strings written for
>> headers to bytes.
>
> So:
>
> ? ? ? ?elif not headers_sent:
> ? ? ? ? ? ? # Before the first output, send the stored headers
> ? ? ? ? ? ? status, response_headers = headers_sent[:] = headers_set
> ? ? ? ? ? ? sys.stdout.buffer.write(wsgi_header('Status: %s\r\n' % status))
> ? ? ? ? ? ? for header in response_headers:
> ? ? ? ? ? ? ? ? sys.stdout.buffer.write(wsgi_header('%s: %s\r\n' % header))
> ? ? ? ? ? ? sys.stdout.buffer.write(wsgi_header('\r\n'))
>
> where define up start of file:
>
> def wsgi_header(u):
> ? ?return u.encode('iso-8859-1')
>
> I am still seeing some issue with CRLF but is in my body and with
> conversion of some StringIO in my test.

Solved my CRLF issue. Was caused by what 2to3 did to my code.

Another thing though. For output changed to sys.stdout.buffer. For
input should we be using sys.stdin.buffer as well if want bytes?

Good thing I tried running this. Did we all assume that someone else
was actually running it to check it? :-)

Graham

>> Graham
>>
>> On 7 January 2011 17:00, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
>>> Stupid question first. When running 2to3 on the example CGI code, why
>>> would it throw back the following. Is this indicative of anything else
>>> that needs to be changed to satisfy some Python 3 thing. The list()
>>> bit seems redundant, but I don't know what the other stuff is about.
>>>
>>> --- xx.py (original)
>>> +++ xx.py (refactored)
>>> @@ -9,7 +9,7 @@
>>> ? ? return u.encode(enc, esc).decode('iso-8859-1')
>>>
>>> ?def run_with_cgi(application):
>>> - ? ?environ = {k: wsgi_string(v) for k,v in os.environ.items()}
>>> + ? ?environ = {k: wsgi_string(v) for k,v in list(os.environ.items())}
>>> ? ? environ['wsgi.input'] ? ? ? ?= sys.stdin
>>> ? ? environ['wsgi.errors'] ? ? ? = sys.stderr
>>> ? ? environ['wsgi.version'] ? ? ?= (1, 0)
>>> @@ -45,7 +45,7 @@
>>> ? ? ? ? ? ? try:
>>> ? ? ? ? ? ? ? ? if headers_sent:
>>> ? ? ? ? ? ? ? ? ? ? # Re-raise original exception if headers sent
>>> - ? ? ? ? ? ? ? ? ? ?raise exc_info[0], exc_info[1], exc_info[2]
>>> + ? ? ? ? ? ? ? ? ? ?raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])
>>> ? ? ? ? ? ? finally:
>>> ? ? ? ? ? ? ? ? exc_info = None ? ? # avoid dangling circular ref
>>> ? ? ? ? elif headers_set:
>>>
>>>
>>>
>>>
>>> On 7 January 2011 16:58, Guido van Rossum <guido at python.org> wrote:
>>>> On Thu, Jan 6, 2011 at 9:47 PM, James Y Knight <foom at fuhm.net> wrote:
>>>>> On Jan 6, 2011, at 11:30 PM, P.J. Eby wrote:
>>>>>> At 09:51 AM 1/7/2011 +1100, Graham Dumpleton wrote:
>>>>>>> Is that the last thing or do I need to go spend some time and write my
>>>>>>> own CGI/WSGI bridge for Python 3 based on my own Python 2 one I have
>>>>>>> lying around and just do some final validation checks with a parallel
>>>>>>> implementation as a sanity check to make sure we got everything? This
>>>>>>> might be a good idea anyway.
>>>>>>
>>>>>> It would. ?In the meantime, though, I've checked in the two-line change to add .buffer in. ?;-)
>>>>>
>>>>> So does that mean PEP 3333 can be accepted now?
>>>>
>>>> TBH I've totally lost track. Hopefully PJE and Graham can tell you...
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>>
>>>
>>
>

From graham.dumpleton at gmail.com  Fri Jan  7 07:32:13 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 17:32:13 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <ig6b91$8l8$1@dough.gmane.org>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<ig6b91$8l8$1@dough.gmane.org>
Message-ID: <AANLkTimg8LUZnvUEt+74Z5EWPd8=G-JeyK79bkdcYrYz@mail.gmail.com>

On 7 January 2011 17:19, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
>> - ? ? ? ? ? ? ? ? ? ?raise exc_info[0], exc_info[1], exc_info[2]
>> + ? ? ? ? ? ? ? ? ? ?raise
>> exc_info[0](exc_info[1]).with_traceback(exc_info[2])
>
> The exception raising syntax has changed; you can not re-raise an exception
> using tuple notation any more. ?The new syntax is far clearer, but I'm
> unsure of back-compatibility or even if it is possible to emulate it
> completely as a polygot (2.x and 3.x w/ same code).

PJE already said that intent is that PEP will only have Python 3
compatible code in it. Not attempting to have examples that work for
both Python 2 and Python 3.

That sounds to me then that we should be using what 2to3 changed it to. Ie.,

                  if headers_sent:
                    # Re-raise original exception if headers sent
                    raise exc_info[0](exc_info[1]).with_traceback(exc_info[2])

Graham


Graham

From alice at gothcandy.com  Fri Jan  7 08:23:52 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 23:23:52 -0800
Subject: [Web-SIG] Python 3 / PEP 3333 (was: PEP 444 / WSGI 2 Async)
References: <AANLkTinYn4k-HD40td4XyDhX8XkxnvxKX7U3BbC7h+RQ@mail.gmail.com>
Message-ID: <ig6f28$lnq$1@dough.gmane.org>

On 2011-01-06 21:35:24 -0800, Jacob Kaplan-Moss said:
> And I'm feeling incredibly disheartened.

As the author of my own small WSGI framework (with world-wide, though 
still limited use) I have the luxury of being able to embrace 
experimental technologies.  The lack of WSGI capability in Python 3 
thoroughly depressed me for the same reasons you describe.

Then I got fed up, tracked down something to tackle, and picked up PEP 
444 knowing full well that PEP 3333 existed and was nearer to 
completion.  PEP 3333 -should- be ratified ASAP in order for developers 
to begin to move forward.  PEP 444, despite the seeming high blood 
pressure on the Web-SIG list, is a long, long way off, and I recognize 
that.  Despite my boundless enthusiasm for debate, I certainly hope 
everyone else realizes this, too.  ;)

I wrote an experimental proof-of-concept HTTP/1.1 server against PEP 
444 (and continue to update it as my rewrite progresses) over the 
course of a week.  It just so happened to be stupidly performant under 
ideal conditions (see the webpy mailing list for a more real-world 
comparison against a CPython extension-based server), extremely simple 
code to maintain and experiment on, and will continue to be my/the 
"reference implementation" for PEP 444.

Other than mod_wsgi, are there any PEP 3333-compliant (or 
near-compliant) components in the wild?  Enough to bring a framework to 
life in Python 3?  What I see is the chicken-and-egg problem endemic 
with Python 3: developers wait on upstream to port before they do, and 
upstream developers are either waiting themselves or don't see the 
demand to port.

Any standard needs early adopters / implementors in order to truly test 
the specification; without such, much of the discussion is pure 
thought-experiment and practical problems may arise after the standard 
is ratified, which is never good.  ;)

With the Marrow suite I'm attempting to brute-force the Python 3 
problem domain within the context of testing PEP 444 and providing 
(after ratification) a solid meta-framework foundation a la Paste.  
Yes, that means I'm re-inventing enough wheels for a 6-axel rig, but it 
also means (in theory) I should have a solid understanding of the 
strengths and weaknesses of the PEP.  (The WebOb equivalent is only 
partially complete as of this writing.)

> A few months ago, PJE posted PEP 3333. It looked good... and then 
> nothing happened. I tried  to prod things forward, and some more 
> discussion ensued... and now it looks like it's stalling again. Each 
> time, discussion of PEP 444 seems to derail discussion of PEP 3333.

I see the opposite in regards to recent traffic on the Web-SIG; PEP 444 
discussion has encouraged PEP 3333 discussion.  See the "Declaring PEP 
3333 accepted" thread (encouraged by Guido himself).

> At this rate, I really wonder if it'll be another two years before we 
> have a working WSGI for Python 3. I hope I'm being pessimistic. Prove 
> me wrong. Please.

I'll do what I can.  :)

> Can we please, please, PLEASE, pause discussion of PEP 444 until PEP 
> 3333 is finalized?

This is something I've seen fairly often around PEP 444 threads; 
instead of reviving (or starting a new) PEP 3333 thread, a complaint is 
levied against PEP 444 discussion itself.  That doesn't help.  ;)

Truthfully, this month already surpasses the amount of activity 
(post-wise) of the last three months combined, and includes quite a 
number of posts about PEP 3333.  (PEP 3333 has had no significant 
discussion - again, by post count - since October.)

	- Alice.



From graham.dumpleton at gmail.com  Fri Jan  7 08:40:53 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Fri, 7 Jan 2011 18:40:53 +1100
Subject: [Web-SIG] Python 3 / PEP 3333 (was: PEP 444 / WSGI 2 Async)
In-Reply-To: <ig6f28$lnq$1@dough.gmane.org>
References: <AANLkTinYn4k-HD40td4XyDhX8XkxnvxKX7U3BbC7h+RQ@mail.gmail.com>
	<ig6f28$lnq$1@dough.gmane.org>
Message-ID: <AANLkTikgCjpAmxgh=f45Zn1jQQS4QfiQPSp2mfQceR7J@mail.gmail.com>

On 7 January 2011 18:23, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
> On 2011-01-06 21:35:24 -0800, Jacob Kaplan-Moss said:
> Other than mod_wsgi, are there any PEP 3333-compliant (or near-compliant)
> components in the wild? ?Enough to bring a framework to life in Python 3?
> ?What I see is the chicken-and-egg problem endemic with Python 3: developers
> wait on upstream to port before they do, and upstream developers are either
> waiting themselves or don't see the demand to port.

There is also uWSGI and CherryPy WSGI server. I recollect that Benoit
may have started looking it over for gunicorn.

Graham

From alice at gothcandy.com  Fri Jan  7 08:44:26 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Thu, 6 Jan 2011 23:44:26 -0800
Subject: [Web-SIG] Python 3 / PEP 3333 (was: PEP 444 / WSGI 2 Async)
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<ig4o0j$lgv$1@dough.gmane.org> <4D25F9F0.3070306@inteli-com.com>
	<ig5rcb$csc$1@dough.gmane.org>
	<AANLkTinYn4k-HD40td4XyDhX8XkxnvxKX7U3BbC7h+RQ@mail.gmail.com>
	<ig6f28$lnq$1@dough.gmane.org>
	<AANLkTikgCjpAmxgh=f45Zn1jQQS4QfiQPSp2mfQceR7J@mail.gmail.com>
Message-ID: <ig6g8q$q61$1@dough.gmane.org>

On 2011-01-06 23:40:53 -0800, Graham Dumpleton said:

> There is also uWSGI and CherryPy WSGI server. I recollect that Benoit 
> may have started looking it over for gunicorn.

Ah, right, I recall seeing CherryPy mentioned in archived discussions.  
So there's hope, then, for relatively quick adoption once ratified.  :)

	- Alice.



From alice at gothcandy.com  Fri Jan  7 09:39:28 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 00:39:28 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <20110107044956.D208F3A40A8@sparrow.telecommunity.com>
Message-ID: <ig6jg0$7ni$1@dough.gmane.org>

On 2011-01-06 20:49:57 -0800, P.J. Eby said:
> It would be helpful if you addressed the issue of scope, i.e., 
> whatfeatures are you proposing to offer to the application developer.

Conformity, predictability, and portability.  That's a lot of y's.  
(Pardon the pun!)

Alex Gr?nholm's post describes the goal quite clearly.

> So far, I believe you're the second major proponent (i.e. ones with 
> concrete proposals and/or implementations to discuss) of an async 
> protocol... and what you have in common with the other proponent is 
> that you happen to have written an async server that would benefit from 
> having apps operating asynchronously.  ;-)

Well, the Marrow HTTPd does operate in multi-process mode, and, one 
day, multi-threaded or a combination.  Integration of a futures 
executor to the WSGI environment would alleviate the major need for a 
multi-threaded implementation in the server core; intensive tasks can 
be deferred to a thread pool vs. everything being deferred to a thread 
pool.  (E.g. template generation, PDF/other text extraction for 
indexing of file uploads, image scaling, etc. all of which are real use 
cases I have which would benefit from futures.)

> I find it hard to imagine an app developer wanting to do something 
> asynchronously for which they would not want to use one of the big-dog 
> asynchronous frameworks.  (Especially if their app involves database 
> access, or other communications protocols.)

Admittedly, a truly async server needs some way to allow file 
descriptors to be registered with the reactor core, with the WSGI 
application being resumed upon some event (e.g. socket is readable or 
writeable for DB access, or even pipe operations for use cases I can't 
think of at the moment).

Futures integration is a Good Idea, IMHO, and being optional and easily 
added to the environ by middleware for servers that don't implement it 
natively is even better.

As for how to provide a generic interface to an async core, I have two 
ideas, but one is magical and the other is more so; I'll describe these 
in a descrete post.

> This doesn't mean I think having a futures API is a bad thing, butISTM 
> that a futures extension to WSGI 1 could be defined right nowusing an 
> x-wsgi-org extension in that case...  and you could thenfind out how 
> many people are actually interested in using it.

I'll add writing up a WSGI middleware layer that configures and adds a 
future.executor to the environ to my already overweight to-do list.  It 
actually is something I have a use for right now on at least one 
commercial project.  :)

> Mainly, though, what I see is people using the futures thing to shuffle 
> off compute-intensive tasks...

That's what it's for.  ;)

> ...but if they do that, then they're basically trying to make the 
> server's life easier...  but under the existing spec, any truly async 
> server implementing WSGI is going to run the *app* in a "future" of 
> some sort already...

Running the application in a future is actually not a half-bad way for 
me to add threading to marrow.server... thanks!

> Which means that the net result is that putting in async is like saying 
> to the app developer: "hey, you know this thing that you just could do 
> in WSGI 1 and the server would take care of it foryou?  Well, now you 
> can manage that complexity by yourself!  Isn't that wonderful?"   ;-)

That's a bit extreme; PEP 444 servers may still implement threading, 
multi-processing, etc. at the reactor level (a la CherryPy or Paste).  
Giving WSGI applications access to a futures executor (possibly the one 
powering the main processing threads) simply gives applications the 
ability to utilize it, not the requirement to do so.

> I could be wrong of course, but I'd like to see what concrete usecases 
> people have for async.

Earlier in this post I illustrated a few that directly apply to a 
commercial application I am currently writing.  I'll elaborate:

:: Image scaling would benefit from multi-processing (spreading the 
load across cores). Also, only one sacle is immediately required before 
returning the post-upload page: the thumbnail.  The other scales can be 
executed without halting the WSGI application's return.

:: Asset content extraction and indexing would benefit from threading, 
and would also not require pausing the WSGI application.

:: Since most templating engines aren't streaming (see my unanswered 
thread in the general mailing list re: this), pausing the application 
pending a particularly difficult render is a boon to single-threaded 
async servers, though true streaming templating (with flush semantics) 
would be the holy grail.  ;)

:: Long-duration calls to non-async-aware libraries such as DB access.  
The WSGI application could queue up a number of long DB queries, pass 
the futures instances to the template, and the template could then 
.result() (block) across them or yield them to be suspended and resumed 
when the result is available.

:: True async is useful for WebSockets, which seem a far superior 
solution to JSON/AJAX polling in addition to allowing real web-based 
socket access, of course.

> We dropped the first discussion of async six years ago because someone 
> (I think it might've been James)pointed out that, well, it isn't 
> actually that useful.  And every subsequent call for use cases since 
> has been answered with, "well, the use case is that you want it to be 
> async."

See the above.  ;)

	- Alice.



From alice at gothcandy.com  Fri Jan  7 10:02:49 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 01:02:49 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <loom.20110106T185706-961@post.gmane.org>
Message-ID: <ig6krp$dpi$1@dough.gmane.org>

On 2011-01-06 10:15:19 -0800, Antoine Pitrou said:

> Alice Bevan?McGregor <alice at ...> writes:
>>> Er, for the record, in Python 3 non-blocking file objects return None when
>>> read() would block.
>> 
>> -1
>> 
>> I'm aware, however that's not practically useful.  How would you detect
>> from within the WSGI 2 application that the file object has become
>> readable?  Implement your own async reactor / select / epoll loop?
>> That's crazy talk!  ;)
> 
> I was just pointing out that if you need to choose a convention for 
> signaling blocking reads on a non-blocking object, it's already there.

I don't.  I need a way to suspend execution of a WSGI application 
pending some operation, often waiting for socket or file read or write 
availability.  (Just as often something entirely unrelated to file 
descriptors, see my previous post from a few moments ago.)

> By the way, an event loop is the canonical implementation of 
> asynchronous programming, so I'm not sure what you're complaining 
> about. Or perhaps you're using "async" in a different meaning? (which 
> one?)

If you use non-blocking sockets, and the WSGI server provides a way to 
directly access the client socket (ack!), utilizing the none response 
on reads would require you to utilize a tight loop within your 
application to wait for actual data.  That's really, really bad, and in 
a single-threaded server, deadly.

> I don't understand why you want a "yield" at this level. IMHO, WSGI 
> needn't involve generators. A higher-level wrapper (framework, 
> middleware, whatever) can wrap fd-waiting in fancy generator stuff if 
> so desired. Or, in some other environments, delegate it to a reactor 
> with callbacks and deferreds. Or whatever else, such as futures.

WSGI already involves generators: the response body.  In fact, the 
templating engine I wrote (and extended to support flush semantics) 
utilizes a generator to return the response body.  Works like a hot 
damn, too.

Yield is the Python language's native way to suspend execution of a 
callable in a re-entrant way.  A trivial example of this is an async 
"ping-pong" reactor.  I wrote one ("you aren't a real Python programmer 
unless...") as an experiment and utilize it for server monitoring with 
tasks being generally scheduled against time, vs. edge-triggered or 
level-triggered fd operation availability.

Everyone has their own idea of what a "deferred" is, and there is only 
one definition of a "future", which (in a broad sense) is the same as 
the general idea of a "deferred".  Deferreds just happen to be 
implementation-specific and often require rewriting large portions of 
external libraries to make them compatible with that specific deferred 
implementation.  That's not a good thing.

Hell; an extension to the futures spec to handle file descriptor events 
might not be a half-bad idea.  :/

> By the way, the concurrent.futures module is new. Though it will be 
> there in 3.2, it's not guaranteed that its API and semantics will be 
> 100% stable while people start to really flesh it out.

Ratification of PEP 444 is a long way off itself.  Also, Alex Gr?nholm 
maintains a pypi backport of the futures module compatible with 2.x+ 
(not sure of the specific minimum version) and < 3.2.  I'm fairly 
certain deprecation warnings wouldn't kill the usefulness of that 
implementation.  Worrying about instability, at this point, may be 
premature.

>> +1 for pure futures which (in theory) eliminate the need for dedicated 
>> async versions of absolutely everything at the possible cost of 
>> slightly higher overhead.
> 
> I don't understand why futures would solve the need for a low-level 
> async facility.

You mis-interpreted; I didn't mean to infer that futures would replace 
an async core reactor, just that long-running external library calls 
could be trivially deferred using futures.

> You still need to define a way for the server and the app to wake each 
> other (and for the server to wake multiple apps).

Futures is a pretty convienent way to have a server wake an app; using 
a future completion callback wrapped (using partial) with the paused 
application generator would do it.  (The reactor Marrow uses, a 
modified Tornado IOLoop, would require calling 
reactor.add_callback(partial(worker, app_gen)) followed by 
reactor._wake() in the future callback.)

"Waking up the server" would be accomplished by yielding a futures 
instance (or fd magical value, etc).

> This isn't done "naturally" in Python (except perhaps with stackless or 
> greenlets). Using fds give you well-known flexible possibilities.

Yield is the natural way for one side of that, re-entering the 
generator on future completion covers the other side.  Stackless and 
greenlets are alternate ideas, but yield is built-in (and soon, so will 
futures).

> If you want to put the futures API in WSGI, think of the poor authors 
> of a WSGI server written in C who will have to write their own executor 
> and future implementation. I'm sure they have better things to do.

If they embed a Python interpreter via C, they can utilize native 
implementations of future executors, though these will obviously be 
slightly less performant than a native C implementation.  (That is, 
unless the stdlib version in 3.2 will have C backing.)

	- Alice.



From chris.dent at gmail.com  Fri Jan  7 10:08:42 2011
From: chris.dent at gmail.com (chris.dent at gmail.com)
Date: Fri, 7 Jan 2011 09:08:42 +0000 (GMT)
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig5a2d$r5s$1@dough.gmane.org>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
Message-ID: <alpine.OSX.1.10.1101070902520.33899@rust.local>

On Thu, 6 Jan 2011, Alice Bevan?McGregor wrote:

> :: Clear separation of "narrative" from "rules to be followed".  This allows 
> developers of both servers and applications to easily run through a 
> confomance "check list".

+1

> :: Isolation of examples and rationale to improve readability of the core 
> rulesets.

+1

> :: Clarification of often mis-interpreted rules from PEP 333 (and those 
> carried over in 3333).

+1

> :: Elimination of unintentional non-conformance, esp. re: cgi.FieldStorage.

+1

> :: Massive simplification of call flow.  Replacing start_response with a 
> returned 3-tuple immensely simplifies the task of middleware that needs to 
> capture HTTP status or manipulate (or even examine) response headers. [1]

+1

I was initially resistant to this one in a we fear change kind of
way, but I've since recognized that a) I was thinking about it
mostly in terms of existing code I have that would need to be
changed b) it _is_ more pythonic.

> :: Reduction of re-implementation / NIH syndrome by incorporating the most 
> common (1%) of features most often relegated to middleware or functional 
> helpers.  Unicode decoding of a small handful of values (CGI values that pull 
> from the request URI) is the biggest example. [2, 3]

0 (as in unsure, need to be convinced, etc)

The zero here is in large part because this particular goal could
cover a large number of things from standardized query string
processing (maybe a good idea) to filters (which I've already
expressed reservations about).

So this goal seems like it ought to be several separate goals.

> :: Cross-compatibility considerations.  The definition and use of native 
> strings vs. byte strings is the biggest example of this in the rewrite.

+1

> :: Making optional (and thus rarely-implemented) features non-optional. E.g. 
> server support for HTTP/1.1 with clarifications for interfacing applications 
> to 1.1 servers.  Thus pipelining, chunked encoding, et. al. as per the HTTP 
> 1.1 RFC.

0

The other option (than non-optional) for optional things is to
remove them.

I think working from a list of goals is an excellent way to make
some headway.

-- 
Chris Dent                                   http://burningchrome.com/
                                 [...]

From alice at gothcandy.com  Fri Jan  7 10:17:05 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 01:17:05 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <20110107041816.8224E3A40A8@sparrow.telecommunity.com>
Message-ID: <ig6lmh$hrj$1@dough.gmane.org>

On 2011-01-06 20:18:12 -0800, P.J. Eby said:
>> :: Reduction of re-implementation / NIH syndrome by incorporating>the 
>> most common (1%) of features most often relegated to middleware>or 
>> functional helpers.
> 
> Note that nearly every application-friendly feature you add will 
> increase the burden on both server developers and middleware 
> developers, which ironically means that application developers actually 
> end up with fewer options.

Some things shouldn't have multiple options in the first place.  ;)  I 
definitely consider implementation overhead on server, middleware, and 
application authors to be important.

As an example, if yield syntax is allowable for application objects (as 
it is for response bodies) middleware will need to iterate over the 
application, yielding up-stream anything that isn't a 3-tuple.  When it 
encounters a 3-tuple, the middleware can do its thing.  If the app 
yield semantics are required (which may be a good idea for consistency 
and simplicity sake if we head down this path) then async-aware 
middleware can be implemented as a generator regardless of the 
downstream (wrapped) application's implementation. That's not too much 
overhead, IMHO.

>> Unicode decoding of a small handful of values (CGI values that> pull 
>> from the request URI) is the biggest example. [2, 3]
> 
> Does that mean you plan to make the other values bytes, then?  Or will 
> they be unicode-y-bytes as well?

Specific CGI values are bytes (one, I believe), specific ones are true 
unicode (URI-related values) and decoded using a configurable encoding 
with a fallback to "bytes in unicode" (iso-8859-1/latin1), are kept 
internally consistent (if any one fails, treat as if they all failed), 
have the encoding used recorded in the environ, and all others are 
native strings ("bytes in unicode" where native strings are unicode).

> What happens for additional server-provided variables?

That is the domain of the server to document, though native strings 
would be nice.  (The PEP only covers CGI variables.)

> The PEP 3333 choice was for uniformity.  At one point, I advocated 
> simply using surrogateescape coding, but this couldn't be made uniform 
> across Python versions and maintain compatibility.

As an open question to anyone: is surrogateescape availabe in Python 
2.6?  Mandating that as a minimum version for PEP 444 has yielded 
benefits in terms of back-ported features and syntax, like b''.

>> :: Cross-compatibility considerations.  The definition and use 
>> of>native strings vs. byte strings is the biggest example of this in 
>> the rewrite.
> 
> I'm not sure what you mean here.  Do you mean "portability of WSGI 
> 2code samples across Python versions (esp. 2.x vs. 3.x)?"

It should be possible (and currently is, as demonstrated by 
marrow.server.http) to create a polygot server, polygot 
middleware/filters (demonstrated by marrow.wsgi.egress.compression), 
and polygot applications, though obviously polygot code demands the 
"lowest common denominator" in terms of feature use.  Application / 
framework authors would likely create Python 3 specific WSGI 
applications to make use of the full Python 3 feature set, with 
cross-compatibility relegated to server and middleware authors.

	- Alice.



From alice at gothcandy.com  Fri Jan  7 10:31:32 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 01:31:32 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<ig5a2d$r5s$1@dough.gmane.org>
	<alpine.OSX.1.10.1101070902520.33899@rust.local>
Message-ID: <ig6mhk$li7$1@dough.gmane.org>

On 2011-01-07 01:08:42 -0800, chris.dent said:
> ... this particular goal [reduction of reimplementation / NIH] could 
> cover a large number of things from standardized query string 
> processing (maybe a good idea) to filters (which I've already expressed 
> reservations about).
> 
> So this goal seems like it ought to be several separate goals.

+1

This definitely needs to be broken out to be explicit over the things 
that can be abstracted away from middleware and applications.  Input 
from framework authors would be valuable here to see what they disliked 
re-implementing the most.  ;)

Query string processing is a difficult task at the best of times, and 
is one area that is reimplemented absolutely everywhere.  (At some 
point I should add up the amount of code + unit testing code that 
covers this topic alone from the top 10 frameworks.)

> The other option (than non-optional) for optional things is to remove them.

True; though optional things already exist as if they were not there.  
Implementors rarely, it seems, expend the effort to implement optional 
components, thus every HTTP server I came across having comments in the 
code saying "up to the application to implement chunked responses" 
indicating -some- thought, but despite chunked /request/ support being 
mandated by HTTP/1.1.  (And other ignored requirements.)

	- Alice.



From pje at telecommunity.com  Fri Jan  7 16:35:36 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 10:35:36 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
Message-ID: <20110107153539.82E8D3A40A8@sparrow.telecommunity.com>

At 05:00 PM 1/7/2011 +1100, Graham Dumpleton wrote:
>Stupid question first. When running 2to3 on the example CGI code,

Don't do that.  It's supposed to already be Python 3 code.  ;-)

It did, however, reveal a bug where I have not in fact done the 
correct Python 3 thing:

>                  if headers_sent:
>                      # Re-raise original exception if headers sent
>-                    raise exc_info[0], exc_info[1], exc_info[2]
>+                    raise 
>exc_info[0](exc_info[1]).with_traceback(exc_info[2])
>              finally:
>                  exc_info = None     # avoid dangling circular ref

Can somebody weigh in on what the correct translation here is?  The 
only real Python 3 coding I've done to date has been experiments to 
test changes to other aspects of WSGI.  ;-)



From pje at telecommunity.com  Fri Jan  7 16:47:33 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 10:47:33 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
Message-ID: <20110107154735.7915E3A40A8@sparrow.telecommunity.com>

At 05:13 PM 1/7/2011 +1100, Graham Dumpleton wrote:
>The version at:
>
>http://svn.python.org/projects/peps/trunk/pep-3333.txt
>
>still shows:
>
>         elif not headers_sent:
>              # Before the first output, send the stored headers
>              status, response_headers = headers_sent[:] = headers_set
>              sys.stdout.write('Status: %s\r\n' % status)
>              for header in response_headers:
>                  sys.stdout.write('%s: %s\r\n' % header)
>              sys.stdout.write('\r\n')
>
>so not using buffer there and also not converting strings written for
>headers to bytes.

Fixed in SVN now.  The main issue now is that we need to fix the 
re-raises and error handling for Python 3, in the text and examples.

I also found some references for with_traceback() and I think I've 
got that sorted now, but if someone can check my work that'd be a good idea.


From pje at telecommunity.com  Fri Jan  7 16:55:45 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 10:55:45 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.c
 om>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
Message-ID: <20110107155545.8AD763A40A8@sparrow.telecommunity.com>

At 05:27 PM 1/7/2011 +1100, Graham Dumpleton wrote:
>Another thing though. For output changed to sys.stdout.buffer. For
>input should we be using sys.stdin.buffer as well if want bytes?

%&$*()&%!!!  Sorry, still getting used to this whole Python 3 
thing.  (Honestly, I don't even use Python 2.6 for anything real yet.)


>Good thing I tried running this. Did we all assume that someone else
>was actually running it to check it? :-)

Well, I only recently started changing the examples to actual Python 
3, vs being the old Python 2 examples.  Though, I'm not sure anybody 
ever ran the Python 2 ones.  ;-)


From guido at python.org  Fri Jan  7 17:01:25 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jan 2011 08:01:25 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <20110107153539.82E8D3A40A8@sparrow.telecommunity.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<20110107153539.82E8D3A40A8@sparrow.telecommunity.com>
Message-ID: <AANLkTinP5ZZx8MK3Y0YPUu2NbXEBGN0PGf2Lgaijuo79@mail.gmail.com>

On Fri, Jan 7, 2011 at 7:35 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 05:00 PM 1/7/2011 +1100, Graham Dumpleton wrote:
>>
>> Stupid question first. When running 2to3 on the example CGI code,
>
> Don't do that. ?It's supposed to already be Python 3 code. ?;-)
>
> It did, however, reveal a bug where I have not in fact done the correct
> Python 3 thing:
>
>> ? ? ? ? ? ? ? ? if headers_sent:
>> ? ? ? ? ? ? ? ? ? ? # Re-raise original exception if headers sent
>> - ? ? ? ? ? ? ? ? ? ?raise exc_info[0], exc_info[1], exc_info[2]
>> + ? ? ? ? ? ? ? ? ? ?raise
>> exc_info[0](exc_info[1]).with_traceback(exc_info[2])
>> ? ? ? ? ? ? finally:
>> ? ? ? ? ? ? ? ? exc_info = None ? ? # avoid dangling circular ref
>
> Can somebody weigh in on what the correct translation here is? ?The only
> real Python 3 coding I've done to date has been experiments to test changes
> to other aspects of WSGI. ?;-)

That translation works.

-- 
--Guido van Rossum (python.org/~guido)

From pje at telecommunity.com  Fri Jan  7 17:10:43 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 11:10:43 -0500
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig6jg0$7ni$1@dough.gmane.org>
References: <20110107044956.D208F3A40A8@sparrow.telecommunity.com>
	<ig6jg0$7ni$1@dough.gmane.org>
Message-ID: <20110107161045.20B0B3A40A8@sparrow.telecommunity.com>

At 12:39 AM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>Earlier in this post I illustrated a few that directly apply to a 
>commercial application I am currently writing.  I'll elaborate:
>
>:: Image scaling would benefit from multi-processing (spreading the 
>load across cores). Also, only one sacle is immediately required 
>before returning the post-upload page: the thumbnail.  The other 
>scales can be executed without halting the WSGI application's return.
>
>:: Asset content extraction and indexing would benefit from 
>threading, and would also not require pausing the WSGI application.
>
>:: Since most templating engines aren't streaming (see my unanswered 
>thread in the general mailing list re: this), pausing the 
>application pending a particularly difficult render is a boon to 
>single-threaded async servers, though true streaming templating 
>(with flush semantics) would be the holy grail.  ;)

In all these cases, ISTM the benefit is the same if you future the 
WSGI apps themselves (which is essentially what most current async 
WSGI servers do, AFAIK).


>:: Long-duration calls to non-async-aware libraries such as DB access.
>The WSGI application could queue up a number of long DB queries, 
>pass the futures instances to the template, and the template could 
>then .result() (block) across them or yield them to be suspended and 
>resumed when the result is available.
>
>:: True async is useful for WebSockets, which seem a far superior 
>solution to JSON/AJAX polling in addition to allowing real web-based 
>socket access, of course.

The point as it relates to WSGI, though, is that there are plenty of 
mature async APIs that offer these benefits, and some of them (e.g. 
Eventlet and Gevent) do so while allowing blocking-style code to be 
written.  That is, you just make what looks like a blocking call, but 
the underlying framework silently suspends your code, without tying 
up the thread.

Or, if you can't use a greenlet-based framework, you can use a 
yield-based framework.  Or, if for some reason you really wanted to 
write continuation-passing style code, you could just use the raw Twisted API.

But in all of these cases you would be better off than if you used a 
half-implementation of the same thing using futures under WSGI, 
because all of those frameworks already have mature and sophisticated 
APIs for doing async communications and DB access.  If you try to do 
it with WSGI under the guise of "portability", all this means is that 
you are stuck rolling your own replacements for those existing APIs.

Even if you've already written a bunch of code using raw sockets and 
want to make it asynchronous, Eventlet and Gevent actually let you 
load a compatibility module that makes it all work, by replacing the 
socket API with an exact duplicate that secretly suspends your code 
whenever a socket operation would block.

IOW, if you are writing a truly async application, you'd almost have 
to be crazy to want to try to do it *portably*, vs. picking a 
full-featured async API and server suite to code against.  And if 
you're migrating an existing, previously-synchronous WSGI app to 
being asynchronous, the obvious thing to do would just be to grab a 
copy of Eventlet or Gevent and import the appropriate compatibility 
modules, not rewrite the whole thing to use futures.


From pje at telecommunity.com  Fri Jan  7 17:28:15 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 11:28:15 -0500
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig6lmh$hrj$1@dough.gmane.org>
References: <20110107041816.8224E3A40A8@sparrow.telecommunity.com>
	<ig6lmh$hrj$1@dough.gmane.org>
Message-ID: <20110107162815.607003A40A8@sparrow.telecommunity.com>

At 01:17 AM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-06 20:18:12 -0800, P.J. Eby said:
>>>:: Reduction of re-implementation / NIH syndrome by 
>>>incorporating>the most common (1%) of features most often 
>>>relegated to middleware>or functional helpers.
>>Note that nearly every application-friendly feature you add will 
>>increase the burden on both server developers and middleware 
>>developers, which ironically means that application developers 
>>actually end up with fewer options.
>
>Some things shouldn't have multiple options in the first place.  ;)

I meant that if a server doesn't implement the spec because of a 
required feature, then the app developer doesn't have the option of 
using that feature anyway -- meaning that adding the feature to the 
spec didn't really help.


>   I definitely consider implementation overhead on server, 
> middleware, and application authors to be important.
>
>As an example, if yield syntax is allowable for application objects 
>(as it is for response bodies) middleware will need to iterate over 
>the application, yielding up-stream anything that isn't a 
>3-tuple.  When it encounters a 3-tuple, the middleware can do its 
>thing.  If the app yield semantics are required (which may be a good 
>idea for consistency and simplicity sake if we head down this path) 
>then async-aware middleware can be implemented as a generator 
>regardless of the downstream (wrapped) application's implementation. 
>That's not too much overhead, IMHO.

The reason I proposed the 3-tuple return in the first place (see 
http://dirtsimple.org/2007/02/wsgi-middleware-considered-harmful.html 
) was that I wanted to make middleware *easy* to write.

Easy enough to write quick, say, 10-line utility functions that are 
correct middleware -- so that you could actually build your 
application out of WSGI functions calling other WSGI-based functions.

The yielding thing wouldn't work for that at all.


>>>Unicode decoding of a small handful of values (CGI values that> 
>>>pull from the request URI) is the biggest example. [2, 3]
>>Does that mean you plan to make the other values bytes, then?  Or 
>>will they be unicode-y-bytes as well?
>
>Specific CGI values are bytes (one, I believe), specific ones are 
>true unicode (URI-related values) and decoded using a configurable 
>encoding with a fallback to "bytes in unicode" (iso-8859-1/latin1), 
>are kept internally consistent (if any one fails, treat as if they 
>all failed), have the encoding used recorded in the environ, and all 
>others are native strings ("bytes in unicode" where native strings 
>are unicode).

So, in order to know what type each CGI variable is, you'll need a reference?


>>What happens for additional server-provided variables?
>
>That is the domain of the server to document, though native strings 
>would be nice.  (The PEP only covers CGI variables.)

I mean the ones required by the spec, not server-specific extensions.



>>The PEP 3333 choice was for uniformity.  At one point, I advocated 
>>simply using surrogateescape coding, but this couldn't be made 
>>uniform across Python versions and maintain compatibility.
>
>As an open question to anyone: is surrogateescape availabe in Python 
>2.6?  Mandating that as a minimum version for PEP 444 has yielded 
>benefits in terms of back-ported features and syntax, like b''.

No, otherwise I'd totally go for the surrogateescape approach.  Heck, 
I'd still go for it if it were possible to write a surrogateescape 
handler for 2.6, and require that a PEP 444 server register one with 
Python's codec system.  I don't know if it's *possible*, though, 
hopefully someone with more knowledge can weigh in on that.


>>>:: Cross-compatibility considerations.  The definition and use 
>>>of>native strings vs. byte strings is the biggest example of this 
>>>in the rewrite.
>>I'm not sure what you mean here.  Do you mean "portability of WSGI 
>>2code samples across Python versions (esp. 2.x vs. 3.x)?"
>
>It should be possible (and currently is, as demonstrated by 
>marrow.server.http) to create a polygot server, polygot 
>middleware/filters (demonstrated by marrow.wsgi.egress.compression), 
>and polygot applications, though obviously polygot code demands the 
>"lowest common denominator" in terms of feature use.  Application / 
>framework authors would likely create Python 3 specific WSGI 
>applications to make use of the full Python 3 feature set, with 
>cross-compatibility relegated to server and middleware authors.

I'm just asking whether, in your statement of goals and rationale, 
you would expand "cross compatibility" as meaning cross-python 
version portability, or whether you meant something else.


From merwok at netwok.org  Fri Jan  7 17:36:56 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Fri, 07 Jan 2011 17:36:56 +0100
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <20110107162815.607003A40A8@sparrow.telecommunity.com>
References: <20110107041816.8224E3A40A8@sparrow.telecommunity.com>	<ig6lmh$hrj$1@dough.gmane.org>
	<20110107162815.607003A40A8@sparrow.telecommunity.com>
Message-ID: <4D274128.1000406@netwok.org>

> No, otherwise I'd totally go for the surrogateescape approach.  Heck, 
> I'd still go for it if it were possible to write a surrogateescape 
> handler for 2.6, and require that a PEP 444 server register one with 
> Python's codec system.  I don't know if it's *possible*, though, 
> hopefully someone with more knowledge can weigh in on that.

This error handler is written in C; I don?t know whether it would be
possible to reimplement it in Python.  See PEP 383 for a description,
Python/codecs.c for the source.

Regards


From solipsis at pitrou.net  Fri Jan  7 17:38:09 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 7 Jan 2011 16:38:09 +0000 (UTC)
Subject: [Web-SIG] Declaring PEP 3333 accepted (CGI example)
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.c om>
	<20110107153539.82E8D3A40A8@sparrow.telecommunity.com>
Message-ID: <loom.20110107T173522-744@post.gmane.org>

Hello,

P.J. Eby <pje at ...> writes:
> 
> >                  if headers_sent:
> >                      # Re-raise original exception if headers sent
> >-                    raise exc_info[0], exc_info[1], exc_info[2]
> >+                    raise 
> >exc_info[0](exc_info[1]).with_traceback(exc_info[2])
> >              finally:
> >                  exc_info = None     # avoid dangling circular ref
> 
> Can somebody weigh in on what the correct translation here is?  The 
> only real Python 3 coding I've done to date has been experiments to 
> test changes to other aspects of WSGI.  

You don't need the "with_traceback". Just "raise exc_info[1]". The original
traceback is already attached to the exception instance (major difference from
Python 2).

Oh and by the way:

    headers_set = []
    headers_sent = []

This is really a recipe for disaster. Please give these two variables clearly
distinct names. Your example is very confusing to read because of this.

Regards

Antoine.



From solipsis at pitrou.net  Fri Jan  7 18:04:07 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 7 Jan 2011 17:04:07 +0000 (UTC)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
Message-ID: <loom.20110107T173907-709@post.gmane.org>

Alice Bevan?McGregor <alice at ...> writes:
> 
> > I don't understand why you want a "yield" at this level. IMHO, WSGI 
> > needn't involve generators. A higher-level wrapper (framework, 
> > middleware, whatever) can wrap fd-waiting in fancy generator stuff if 
> > so desired. Or, in some other environments, delegate it to a reactor 
> > with callbacks and deferreds. Or whatever else, such as futures.
> 
> WSGI already involves generators: the response body.

Wrong. The response body is an arbitrary iterable, which means it can be a
sequence, a generator, or something else. WSGI doesn't mandate any specific
feature of generators, such as coroutine-like semantics, and the server doesn't
have to know about them.

> Everyone has their own idea of what a "deferred" is, and there is only 
> one definition of a "future", which (in a broad sense) is the same as 
> the general idea of a "deferred".

A Twisted deferred is as well defined as a Python stdlib future; actually,
deferreds have been in use by the Python community for much, much longer than
futures. But that's besides the point, since I'm proposing that your spec
doesn't rely on a high-level abstraction at all.

> Ratification of PEP 444 is a long way off itself.

Right, that's why I was suggesting you drop your concern for Python 2
compatibility.


Antoine.



From tfarrell at owassobible.org  Fri Jan  7 18:37:35 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Fri, 7 Jan 2011 11:37:35 -0600 (CST)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <AANLkTimYUChbg1T0YMTa0kTMhZNSv73yo84Raa-fbQVh@mail.gmail.com>
Message-ID: <25078474.24.1294421855233.JavaMail.root@boaz>

When I originally requested a futures executor option (the email that started this thread), this is more like what I had in mind.  I'm not against async...rather indifferent.  But I wanted the ability for the server to run something after the response had been fully served to the client and thus not blocking the response.  The example I gave was sending an email, but there are plenty of other use cases.  Futures seemed like the right way to do this.  I'm also not sure futures is the right way to build an async specification and for that matter, there will be a lot to work out with regard to PEP 444.

Rather than responding to this, I'll start a new thread since this takes the environ["wsgi.executor"] discusssion in a different direction.  Please send your comments there.

-t

----- Original Message -----
From: "Guido van Rossum" <guido at python.org>
To: "P.J. Eby" <pje at telecommunity.com>
Cc: alice at gothcandy.com, web-sig at python.org
Sent: Thursday, January 6, 2011 11:30:11 PM
Subject: Re: [Web-SIG] PEP 444 / WSGI 2 Async

On Thu, Jan 6, 2011 at 8:49 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 05:47 PM 1/6/2011 -0800, Alice Bevan?McGregor wrote:
>>
>> Tossing the idea around all day long will then, of course, be happening
>> regardless. ?Unfortunately for that particular discussion, PEP 3148 /
>> Futures seems to have won out in the broader scope.
>
> Do any established async frameworks or server (e.g. Twisted, Eventlet,
> Gevent, ?Tornado, etc.) make use of futures?

PEP 3148 Futures are meant for a rather different purpose than those
async frameworks. Those frameworks all are trying to minimize the
number of threads using some kind of callback-based non-blocking I/O
system. PEP 3148 OTOH doesn't care about that -- it uses threads or
processes proudly. This is useful for a different type of application,
where there are fewer, larger tasks, and the overhead of threads
doesn't matter.

The Monocle framework, which builds on top of Tornado or Twisted, uses
something not entirely unlike Futures, though they call it Callback.

I don't think the acceptance of PEP 3148 should be taken as forcing
the direction that async frameworks should take.

>> ?Having a ratified and incorporated language PEP (core in 3.2 w/
>> compatibility package for 2.5 or 2.6+ support) reduces the scope of async
>> discussion down to: "how do we integrate futures into WSGI 2" instead of
>> "how do we define an async API at all".
>
> It would be helpful if you addressed the issue of scope, i.e., what features
> are you proposing to offer to the application developer.
>
> While the idea of using futures presents some intriguing possibilities, it
> seems to me at first glance that all it will do is move the point where the
> work gets done. ?That is, instead of simply running the app in a worker, the
> app will be farming out work to futures. ?But if this is so, then why
> doesn't the server just farm the apps themselves out to workers?
>
> I guess what I'm saying is, I haven't heard use cases for this from the
> application developer POV -- why should an app developer care about having
> their app run asynchronously?
>
> So far, I believe you're the second major proponent (i.e. ones with concrete
> proposals and/or implementations to discuss) of an async protocol... ?and
> what you have in common with the other proponent is that you happen to have
> written an async server that would benefit from having apps operating
> asynchronously. ?;-)
>
> I find it hard to imagine an app developer wanting to do something
> asynchronously for which they would not want to use one of the big-dog
> asynchronous frameworks. ?(Especially if their app involves database access,
> or other communications protocols.)
>
> This doesn't mean I think having a futures API is a bad thing, but ISTM that
> a futures extension to WSGI 1 could be defined right now using an x-wsgi-org
> extension in that case... ?and you could then find out how many people are
> actually interested in using it.
>
> Mainly, though, what I see is people using the futures thing to shuffle off
> compute-intensive tasks... ?but if they do that, then they're basically
> trying to make the server's life easier... ?but under the existing spec, any
> truly async server implementing WSGI is going to run the *app* in a "future"
> of some sort already...
>
> Which means that the net result is that putting in async is like saying to
> the app developer: "hey, you know this thing that you just could do in WSGI
> 1 and the server would take care of it for you? ?Well, now you can manage
> that complexity by yourself! ?Isn't that wonderful?" ? ;-)
>
> I could be wrong of course, but I'd like to see what concrete use cases
> people have for async. ?We dropped the first discussion of async six years
> ago because someone (I think it might've been James) pointed out that, well,
> it isn't actually that useful. ?And every subsequent call for use cases
> since has been answered with, "well, the use case is that you want it to be
> async."
>
> Only, that's a *server* developer's use case, not an app developer's use
> case... ?and only for a minority of server developers, at that.
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
_______________________________________________
Web-SIG mailing list
Web-SIG at python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/tfarrell%40owassobible.org

From tfarrell at owassobible.org  Fri Jan  7 18:47:12 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Fri, 7 Jan 2011 11:47:12 -0600 (CST)
Subject: [Web-SIG] PEP 444 feature request - Futures executor
Message-ID: <10086684.26.1294422432945.JavaMail.root@boaz>

There has been much discussion about how to handle async in PEP 444 and that discussion centers around the use of futures.  However, I'm requesting that servers _optionally_ provide environ['wsgi.executor'] as a futures executor that applications can use for the purpose of doing something after the response is fully sent to the client.  This is feature request is designed to be concurrency methodology agnostic.

Some example use cases are:

- send an email that might block on a slow email server (Alice, I read what you said about Turbomail, but one product is not the solution to all situations)
- initiate a database vacuum
- clean a cache
- build a cache
- compile statistics

When serving pages of an application, these are all things that could be done after the response has been sent.  Ideally these things don't need to be done in a request thread and aren't incredibly time-sensitive.  It seems to me that futures would be an ideal way of handling this.

Thoughts?

From guido at python.org  Fri Jan  7 18:59:10 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Jan 2011 09:59:10 -0800
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <10086684.26.1294422432945.JavaMail.root@boaz>
References: <10086684.26.1294422432945.JavaMail.root@boaz>
Message-ID: <AANLkTim3S=AbwmCPQveCS1tWBKK6xwkMMYOv9MMwom8q@mail.gmail.com>

If it's optional, what's the benefit for the app of getting it through
WSGI instead of through importing some other standard module? The API
of the executor will require a lot of thought. I worry that this
weighs down the WSGI standard with the responsibility of coming up
with the perfect executor API, and if it's not quite perfect after
all, servers are additionally required to support the standard but
suboptimal API effectively forever. Or they can choose not to provide
it, in which case it was a waste of time putting it in WSGI.

On Fri, Jan 7, 2011 at 9:47 AM, Timothy Farrell
<tfarrell at owassobible.org> wrote:
> There has been much discussion about how to handle async in PEP 444 and that discussion centers around the use of futures. ?However, I'm requesting that servers _optionally_ provide environ['wsgi.executor'] as a futures executor that applications can use for the purpose of doing something after the response is fully sent to the client. ?This is feature request is designed to be concurrency methodology agnostic.
>
> Some example use cases are:
>
> - send an email that might block on a slow email server (Alice, I read what you said about Turbomail, but one product is not the solution to all situations)
> - initiate a database vacuum
> - clean a cache
> - build a cache
> - compile statistics
>
> When serving pages of an application, these are all things that could be done after the response has been sent. ?Ideally these things don't need to be done in a request thread and aren't incredibly time-sensitive. ?It seems to me that futures would be an ideal way of handling this.
>
> Thoughts?
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From jacob at jacobian.org  Fri Jan  7 19:40:42 2011
From: jacob at jacobian.org (Jacob Kaplan-Moss)
Date: Fri, 7 Jan 2011 12:40:42 -0600
Subject: [Web-SIG] Python 3 / PEP 3333 (was: PEP 444 / WSGI 2 Async)
In-Reply-To: <ig6f28$lnq$1@dough.gmane.org>
References: <AANLkTinYn4k-HD40td4XyDhX8XkxnvxKX7U3BbC7h+RQ@mail.gmail.com>
	<ig6f28$lnq$1@dough.gmane.org>
Message-ID: <AANLkTi=WTeiUF5N3ujrDpLd1etNPz7HF9CrZ9VvkmupB@mail.gmail.com>

On Fri, Jan 7, 2011 at 1:23 AM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
> Other than mod_wsgi, are there any PEP 3333-compliant (or near-compliant)
> components in the wild? ?Enough to bring a framework to life in Python 3?
> ?What I see is the chicken-and-egg problem endemic with Python 3: developers
> wait on upstream to port before they do, and upstream developers are either
> waiting themselves or don't see the demand to port.

I don't see that problem any more. I have at least three WSGI servers
I could test against: modwsgi, CherryPy, and Django's half-assed
built-in server. I guess I could add wsgiref as a 4th, but only sorta.
And looks like Benoit's geting Gunicorn up to snuff.

What happens now goes something like this:

1. Get excited to port Django to Python 3.
2. Hack for a while.
3. Get something working under runserver - woo!
4. Hm, it fails under modwsgi.
5. OK, problem fixed.
6. Wait, no, now it doesn't work under runserver.
7. Or CherryPy. Dammit.
8. Lose interest for another 6 months.

At this point, there's a bug somewhere. It's *probably* in my code --
like I said earlier, I only barely grok WSGI -- but without a spec to
refer to I'm pretty much hosed. See, Django on Py 2 jumps through a
whole bunch of hoops to gloss over the string/unicode distinction and
over the question of encoding, and that's stuff's pretty fiddley. The
"right" way to handle that on Python 3 depends entirely on the issues
hammered out in PEP 3333 -- particularly the byte/str decisions. I'm
starting to assume that PEP 3333 is going to get accepted in a form
fundamentally the same as it appears right now, but if I'm wrong I get
to do this stuff all over again. Nothing's more of an
enthusiasm-killer than knowing I might have to start all over again
later.

I really want a definitive answer. If the spec says it's my fault, I
want people to yell at me loudly until Django is compliant. If the
spec says it's not my fault, I want to be able to be an asshole [1]
until all the app containers are compliant.

[1] That's a technical term; see
http://diveintomark.org/archives/2004/08/16/specs.

>> Can we please, please, PLEASE, pause discussion of PEP 444 until PEP 3333
>> is finalized?
>
> This is something I've seen fairly often around PEP 444 threads; instead of
> reviving (or starting a new) PEP 3333 thread, a complaint is levied against
> PEP 444 discussion itself. ?That doesn't help. ?;)

Unfortunately, I really don't know any other way of helping than being
a pain in the ass. I don't understand the issues well enough to
contribute technically, so I've decided I'm going to continue to
complain loudly. Hopefully you'll get sick of me and give me a spec so
I'll shut the hell up!

I do feel crappy asking you to put 444 on hold. I understand that the
SIG should be perfectly capable of working on more than one thing at
once. However, to my eyes it seems like it keeps getting derailed. I
understand this, too: clearly WSGI isn't perfect, and when you run up
against some of those issues it's a *lot* more fun to ignore 'em just
for a bit longer and work on something more exciting.

I really do appreciate the enthusiasm for PEP 444. I share it: it
seems a lot easier to implement, and it'll certainly make some of the
things Django's doing a lot easier. I just would really like to see
that enthusiasm and energy turned full blast on PEP 3333 until it's
done.

Jacob

(Luckily for you, I'm going on vacation tomorrow, so you won't have to
deal with me me complaining for at least a week!)

From tfarrell at owassobible.org  Fri Jan  7 20:07:36 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Fri, 7 Jan 2011 13:07:36 -0600 (CST)
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <AANLkTim3S=AbwmCPQveCS1tWBKK6xwkMMYOv9MMwom8q@mail.gmail.com>
Message-ID: <25494500.30.1294427256563.JavaMail.root@boaz>

> If it's optional, what's the benefit for the app of getting it through WSGI instead of through importing some other standard module?

Performance primarily.  If you instantiate an executor at every page request, wouldn't that slow things down unnecessarily?  Aside from that, servers currently specify if they are multi-threaded and/or multi-process.  Having the server provide the executor allows it to provide an executor that most matches its own concurrency model...again for performance reasons.

Optional and not manditory because not every application wants or need such functionality.  Maybe this should be a server option instead of a spec option.  But since we already have the module available, it shouldn't be too much of a burden on server/gateway authors to add support for it.

> I worry that this weighs down the WSGI standard with the responsibility of coming up with the perfect executor API, and if it's not quite perfect after all, servers are additionally required to support the standard but suboptimal API effectively forever.

I'm not following you here.  What's wrong with executor.submit() that might need changing?  Granted, it would not be ideal if an application called executor.shutdown().  This doesn't seem difficult to my tiny brain.


----- Original Message -----
From: "Guido van Rossum" <guido at python.org>
To: "Timothy Farrell" <tfarrell at owassobible.org>
Cc: web-sig at python.org
Sent: Friday, January 7, 2011 11:59:10 AM
Subject: Re: [Web-SIG] PEP 444 feature request - Futures executor

If it's optional, what's the benefit for the app of getting it through
WSGI instead of through importing some other standard module? The API
of the executor will require a lot of thought. I worry that this
weighs down the WSGI standard with the responsibility of coming up
with the perfect executor API, and if it's not quite perfect after
all, servers are additionally required to support the standard but
suboptimal API effectively forever. Or they can choose not to provide
it, in which case it was a waste of time putting it in WSGI.

On Fri, Jan 7, 2011 at 9:47 AM, Timothy Farrell
<tfarrell at owassobible.org> wrote:
> There has been much discussion about how to handle async in PEP 444 and that discussion centers around the use of futures. ?However, I'm requesting that servers _optionally_ provide environ['wsgi.executor'] as a futures executor that applications can use for the purpose of doing something after the response is fully sent to the client. ?This is feature request is designed to be concurrency methodology agnostic.
>
> Some example use cases are:
>
> - send an email that might block on a slow email server (Alice, I read what you said about Turbomail, but one product is not the solution to all situations)
> - initiate a database vacuum
> - clean a cache
> - build a cache
> - compile statistics
>
> When serving pages of an application, these are all things that could be done after the response has been sent. ?Ideally these things don't need to be done in a request thread and aren't incredibly time-sensitive. ?It seems to me that futures would be an ideal way of handling this.
>
> Thoughts?
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From pje at telecommunity.com  Fri Jan  7 21:14:20 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 15:14:20 -0500
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <10086684.26.1294422432945.JavaMail.root@boaz>
References: <10086684.26.1294422432945.JavaMail.root@boaz>
Message-ID: <20110107201420.346A43A40A8@sparrow.telecommunity.com>

At 11:47 AM 1/7/2011 -0600, Timothy Farrell wrote:
>There has been much discussion about how to handle async in PEP 444 
>and that discussion centers around the use of futures.  However, I'm 
>requesting that servers _optionally_ provide 
>environ['wsgi.executor'] as a futures executor that applications can 
>use for the purpose of doing something after the response is fully 
>sent to the client.  This is feature request is designed to be 
>concurrency methodology agnostic.
>
>Some example use cases are:
>
>- send an email that might block on a slow email server (Alice, I 
>read what you said about Turbomail, but one product is not the 
>solution to all situations)
>- initiate a database vacuum
>- clean a cache
>- build a cache
>- compile statistics
>
>When serving pages of an application, these are all things that 
>could be done after the response has been sent.  Ideally these 
>things don't need to be done in a request thread and aren't 
>incredibly time-sensitive.  It seems to me that futures would be an 
>ideal way of handling this.
>
>Thoughts?

This seems like a potentially good way to do it; I suggest making it 
a wsgi.org extension; see (and update) 
http://www.wsgi.org/wsgi/Specifications with your proposal.

I would suggest including a simple sample executor wrapper that 
servers could use to block all but the methods allowed by your 
proposal.  (i.e., presumably not shutdown(), for example.)

There are some other issues that might need to be addressed, like 
maybe adding an attribute or two for the level of reliability 
guaranteed by the executor, or allowing the app to request a given 
reliability level.  Specifically, it might be important to distinguish between:

* this will be run exactly once as long as the server doesn't crash
* this will eventually be run once, even if the server suffers a 
fatal error between now and then

IOW, to indicate whether the thing being done is "transactional", so to speak.

I mean, I can imagine building a transactional service on top of the 
basic service, by queuing task information externally, then just 
using executor calls to pump the queue.  But IMO it seems pretty 
intrinsic to want that kind of persistence guarantee for at least the 
email case, or, say, sending off a charge to a credit card or 
something like that.

One other relevant use case: sometimes you want a long-running 
process step that the user checks back in on periodically, so having 
a way to get a "handle" for a future that can be kept in a session or 
something might be important.  Like, say, you're preparing a report 
that will be viewed in the browser, and using meta-refresh or some 
such to poll.  The app needs to check on a previously queued future 
and get its results.

I don't know how easy any of the above are to implement with the 
futures API or your proposal, but they seem like worthwhile things to 
have available, and actually would provide for some rich application 
use cases.  But if they're implementable over the futures API at all, 
it should be possible to implement them as WSGI 1.x middleware or as 
a server extension.

A spec like that definitely needs some thrashing out, but I don't 
think it need derail any PEPs in progress: the API of such an 
extension doesn't affect the basic WSGI protocol at all. 


From alice at gothcandy.com  Fri Jan  7 21:21:35 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 12:21:35 -0800
Subject: [Web-SIG] PEP 444 feature request - Futures executor
References: <10086684.26.1294422432945.JavaMail.root@boaz>
	<AANLkTim3S=AbwmCPQveCS1tWBKK6xwkMMYOv9MMwom8q@mail.gmail.com>
	<25494500.30.1294427256563.JavaMail.root@boaz>
Message-ID: <ig7skf$je$1@dough.gmane.org>

On Fri, Jan 7, 2011 at 9:47 AM, Timothy Farrell wrote:
> There has been much discussion about how to handle async in PEP 444 and 
> that discussion centers around the use of futures. ?However, I'm 
> requesting that servers _optionally_ provide environ['wsgi.executor'] 
> as a futures executor that applications can use for the purpose of 
> doing something after the response is fully sent to the client. ?This 
> is feature request is designed to be concurrency methodology agnostic.

+1

On 2011-01-07 11:07:36 -0800, Timothy Farrell said:
> On 2011-01-07 09:59:10 -0800, Guido van Rossum said:
>> If it's optional, what's the benefit for the app of getting it through 
>> WSGI instead of through importing some other standard module?
> 
> Aside from that, servers currently specify if they are multi-threaded 
> and/or multi-process.  Having the server provide the executor allows it 
> to provide an executor that most matches its own concurrency model...

I think that's the bigger point; WSGI servers do implement their own 
concurrency model for request processing and utilizing a 
server-provided executor which interfaces with whatever the internal 
representation of concurrency is would be highly beneficial.  (Vs. an 
application utilizing a more generic executor implementation that adds 
a second thread pool...)

Taking futures to be separate and distinct from the rest of async 
discussion, I still think it's an extremely useful feature.  I outlined 
my own personal use cases in my slew of e-mails last night, and many of 
them are also not time sensitive.  (E.g. image scaling, full text 
indexing, etc.)

> Maybe this should be a server option instead of a spec option.

It would definitely fall under the Server API spec, not the application 
one.  Being optional, and with simple (wsgi.executor) access via the 
environ would also allow middleware developers to create executor 
implementations (or just reference the concurrent.futures 
implementation).

>> I worry that this weighs down the WSGI standard with the responsibility 
>> of coming up with the perfect executor API, and if it's not quite 
>> perfect after all, servers are additionally required to support the 
>> standard but suboptimal API effectively forever.
> 
> I'm not following you here.  What's wrong with executor.submit() that 
> might need changing?  Granted, it would not be ideal if an application 
> called executor.shutdown().  This doesn't seem difficult to my tiny 
> brain.

The "perfect" executor API is already well defined in PEP 3148 AFIK.  
Specific methods with specific semantics implemented in a duck-typed 
way.  The underlying implementation is up to the server, or the server 
can utilize an external (or built-in in 3.2) futures implementation.

If WSGI 2 were to incorporate futures as a feature there would have to 
be some mandate as to which methods applications and middleware are 
allowed to call; similar to how we do not allow .close() across 
wsgi.input or wsgi.errors.

	- Alice.



From alice at gothcandy.com  Fri Jan  7 21:29:56 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 12:29:56 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
Message-ID: <ig7t44$2tu$1@dough.gmane.org>

On 2011-01-07 09:04:07 -0800, Antoine Pitrou said:
> Alice Bevan?McGregor <alice at ...> writes:
>>> I don't understand why you want a "yield" at this level. IMHO, WSGI 
>>> needn't involve generators. A higher-level wrapper (framework, 
>>> middleware, whatever) can wrap fd-waiting in fancy generator stuff if 
>>> so desired. Or, in some other environments, delegate it to a reactor 
>>> with callbacks and deferreds. Or whatever else, such as futures.
>> 
>> WSGI already involves generators: the response body.
> 
> Wrong.

I'm aware that it can be any form of iterable, from a list-wrapped 
string all the way up to generators or other nifty things.  I 
mistakenly omitted these assuming that the other iterables were 
universally understood and implied.

However, using a generator is a known, vlaid use case that I do see in 
the wild.  (And also rely upon in some of my own applications.)

> Right, that's why I was suggesting you drop your concern for Python 2 
> compatibility.

-1

There is practically no reason for doing so; esp. considering that I've 
managed to write a 2k/3k polygot server that is more performant out of 
the box than any other WSGI HTTP server I've come across and is far 
simpler in implementation than most of the ones I've come across with 
roughly equivelant feature sets.

Cross compatibility really isn't that hard, and arguing that 2.x 
support should be dropped for the sole reason that "it might be dead by 
the time this is ratified" is a bit off.

Python 2.x will be around for a long time.

	- Alice.



From alice at gothcandy.com  Fri Jan  7 21:37:38 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 12:37:38 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<alpine.OSX.2.00.1101061302550.7296@crank.local>
	<ig4o0j$lgv$1@dough.gmane.org> <4D25F9F0.3070306@inteli-com.com>
	<ig5rcb$csc$1@dough.gmane.org>
	<20110107044956.D208F3A40A8@sparrow.telecommunity.com>
	<ig6jg0$7ni$1@dough.gmane.org>
	<20110107161045.20B0B3A40A8@sparrow.telecommunity.com>
Message-ID: <ig7tii$5cp$1@dough.gmane.org>

On 2011-01-07 08:10:43 -0800, P.J. Eby said:
> At 12:39 AM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>> :: Image scaling would benefit from multi-processing (spreading 
>> the>load across cores). Also, only one sacle is immediately 
>> required>before returning the post-upload page: the thumbnail.  The 
>> other>scales can be executed without halting the WSGI application's 
>> return.
>> 
>> :: Asset content extraction and indexing would benefit from>threading, 
>> and would also not require pausing the WSGI application.
> 
> In all these cases, ISTM the benefit is the same if you future theWSGI 
> apps themselves (which is essentially what most current asyncWSGI 
> servers do, AFAIK).

Image scaling and asset content extraction should not block the 
response to a HTTP request; these need to be 'forked' from the main 
request.  Only template generation (where the app needs to effectively 
block pending completion) is solved easily by threading the whole 
application call.

>> :: Long-duration calls to non-async-aware libraries such as DB access.
>> The WSGI application could queue up a number of long DB queries,>pass 
>> the futures instances to the template, and the template could>then 
>> .result() (block) across them or yield them to be suspended and>resumed 
>> when the result is available.
>> 
>> :: True async is useful for WebSockets, which seem a far 
>> superior>solution to JSON/AJAX polling in addition to allowing real 
>> web-based>socket access, of course.
> 
> The point as it relates to WSGI, though, is that there are plenty 
> ofmature async APIs that offer these benefits, and some of them 
> (e.g.Eventlet and Gevent) do so while allowing blocking-style code to 
> bewritten.  That is, you just make what looks like a blocking call, 
> butthe underlying framework silently suspends your code, without 
> tyingup the thread.
> 
> Or, if you can't use a greenlet-based framework, you can use a 
> yield-based framework.  Or, if for some reason you really wanted to 
> write continuation-passing style code, you could just use the raw 
> Twisted API.

But is there really any problem with providing a unified method for 
indication a suspend point?  What the server does when it gets the 
yielded value is entirely up to the implementation of the server; if it 
(the server) wants to use greenlets, it can.  If it has other 
methedologies, it can go nuts.

> Even if you've already written a bunch of code using raw sockets and 
> want to make it asynchronous, Eventlet and Gevent actually let youload 
> a compatibility module that makes it all work, by replacing the socket 
> API with an exact duplicate that secretly suspends your code whenever a 
> socket operation would block.

I generally frown upon magic, and each of these implementations is 
completely specific.  :/

	- Alice.



From paul.joseph.davis at gmail.com  Fri Jan  7 21:42:24 2011
From: paul.joseph.davis at gmail.com (Paul Davis)
Date: Fri, 7 Jan 2011 15:42:24 -0500
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <ig7t44$2tu$1@dough.gmane.org>
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
	<ig7t44$2tu$1@dough.gmane.org>
Message-ID: <AANLkTik_JMrvNAf7KbCpNaEBvriq-=49YZgYD7PzMdWT@mail.gmail.com>

> There is practically no reason for doing so; esp. considering that I've
> managed to write a 2k/3k polygot server that is more performant out of the
> box than any other WSGI HTTP server I've come across and is far simpler in
> implementation than most of the ones I've come across with roughly
> equivelant feature sets.

Is the code for this server online? I'd be interested in reading through it.

From tfarrell at owassobible.org  Fri Jan  7 21:52:46 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Fri, 7 Jan 2011 14:52:46 -0600 (CST)
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <20110107201420.346A43A40A8@sparrow.telecommunity.com>
Message-ID: <31198151.34.1294433566856.JavaMail.root@boaz>



----- Original Message -----
From: "P.J. Eby" <pje at telecommunity.com>
To: "Timothy Farrell" <tfarrell at owassobible.org>, web-sig at python.org
Sent: Friday, January 7, 2011 2:14:20 PM
Subject: Re: [Web-SIG] PEP 444 feature request - Futures executor

> This seems like a potentially good way to do it; I suggest making it 
> a wsgi.org extension; see (and update) 
> http://www.wsgi.org/wsgi/Specifications with your proposal.
> 
> I would suggest including a simple sample executor wrapper that 
> servers could use to block all but the methods allowed by your 
> proposal.  (i.e., presumably not shutdown(), for example.)

OK, will do.




From solipsis at pitrou.net  Fri Jan  7 22:21:36 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 7 Jan 2011 21:21:36 +0000 (UTC)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
	<ig7t44$2tu$1@dough.gmane.org>
Message-ID: <loom.20110107T221211-708@post.gmane.org>

Alice Bevan?McGregor <alice at ...> writes:
> 
> On 2011-01-07 09:04:07 -0800, Antoine Pitrou said:
> > Alice Bevan?McGregor <alice at ...> writes:
> >>> I don't understand why you want a "yield" at this level. IMHO, WSGI 
> >>> needn't involve generators. A higher-level wrapper (framework, 
> >>> middleware, whatever) can wrap fd-waiting in fancy generator stuff if 
> >>> so desired. Or, in some other environments, delegate it to a reactor 
> >>> with callbacks and deferreds. Or whatever else, such as futures.
> >> 
> >> WSGI already involves generators: the response body.
> > 
> > Wrong.
> 
> I'm aware that it can be any form of iterable, [snip]

Ok, so, WSGI doesn't "already involve generators". QED.

> > Right, that's why I was suggesting you drop your concern for Python 2 
> > compatibility.
> 
> -1
> 
> There is practically no reason for doing so;

Of course, there is one: a less complex PEP without any superfluous
compatibility language sprinkled all over. And a second one: a simpler PEP is
probably easier to get contructive comments about, and (perhaps some day)
consensus on.

> esp. considering that I've 
> managed to write a 2k/3k polygot server that is more performant out of 
> the box than any other WSGI HTTP server I've come across and is far 
> simpler in implementation than most of the ones I've come across with 
> roughly equivelant feature sets.

Just because you "managed to write" some piece of code for a *particular* use
case doesn't mean that cross-compatibility is a solved problem. If you think
it's easy, then I'm sure the authors of various 3rd-party libs would welcome
your help achieving it.

> Python 2.x will be around for a long time.

And so will PEP 3333 and even PEP 333. People who value legacy compatibility
will favour these old PEPs over your new one anyway. People who don't will
progressively jump to 3.x.

Antoine.



From alice at gothcandy.com  Fri Jan  7 22:22:23 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 13:22:23 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<20110107041816.8224E3A40A8@sparrow.telecommunity.com>
	<ig6lmh$hrj$1@dough.gmane.org>
	<20110107162815.607003A40A8@sparrow.telecommunity.com>
Message-ID: <ig806f$ilk$1@dough.gmane.org>

On 2011-01-07 08:28:15 -0800, P.J. Eby said:
> At 01:17 AM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>> On 2011-01-06 20:18:12 -0800, P.J. Eby said:
>>>> :: Reduction of re-implementation / NIH syndrome by>>>incorporating>the 
>>>> most common (1%) of features most often>>>relegated to middleware>or 
>>>> functional helpers.
>>> Note that nearly every application-friendly feature you add 
>>> will>>increase the burden on both server developers and 
>>> middleware>>developers, which ironically means that application 
>>> developers>>actually end up with fewer options.
>> 
>> Some things shouldn't have multiple options in the first place.  ;)
> 
> I meant that if a server doesn't implement the spec because of 
> arequired feature, then the app developer doesn't have the option of 
> using that feature anyway -- meaning that adding the feature to the 
> spec didn't really help.

I truly can not worry about non-conformant applications, middleware, or 
servers and still keep my hair.

>> I definitely consider implementation overhead on server,> middleware, 
>> and application authors to be important.
>> 
>> As an example, if yield syntax is allowable for application objects>(as 
>> it is for response bodies) middleware will need to iterate over>the 
>> application, yielding up-stream anything that isn't a>3-tuple.  When it 
>> encounters a 3-tuple, the middleware can do its>thing.  If the app 
>> yield semantics are required (which may be a good>idea for consistency 
>> and simplicity sake if we head down this path)>then async-aware 
>> middleware can be implemented as a generator>regardless of the 
>> downstream (wrapped) application's implementation.>That's not too much 
>> overhead, IMHO.
> 
> The reason I proposed the 3-tuple return in the first place (see 
> http://dirtsimple.org/2007/02/wsgi-middleware-considered-harmful.html) 
> was that I wanted to make middleware *easy* to write.

This was noted several times, and I do agree with that blog article 
which states that a lot of middleware shouldn't be middleware.

> Easy enough to write quick, say, 10-line utility functions that 
> arecorrect middleware -- so that you could actually build 
> yourapplication out of WSGI functions calling other WSGI-based 
> functions.
> 
> The yielding thing wouldn't work for that at all.

Handling a possible generator isn't that difficult.

>>>> Unicode decoding of a small handful of values (CGI values that>>>>pull 
>>>> from the request URI) is the biggest example. [2, 3]
>>> Does that mean you plan to make the other values bytes, then?  Or>>will 
>>> they be unicode-y-bytes as well?
>> 
>> Specific CGI values are bytes (one, I believe), specific ones are>true 
>> unicode (URI-related values) and decoded using a configurable>encoding 
>> with a fallback to "bytes in unicode" (iso-8859-1/latin1),>are kept 
>> internally consistent (if any one fails, treat as if they>all failed), 
>> have the encoding used recorded in the environ, and all>others are 
>> native strings ("bytes in unicode" where native strings>are unicode).
> 
> So, in order to know what type each CGI variable is, you'll need a reference?

Reference?  Re-read what I wrote.  Only URI-specific values utilize an 
encoding reference variable in the environment; that's four values out 
of the entire environ.  There is one, clearly defined bytes value.  The 
rest are native strings, decoded using 
latin1/iso-8859-1/"str-in-unicode" where native strings are unicode.

>>> What happens for additional server-provided variables?
>> 
>> That is the domain of the server to document, though native 
>> strings>would be nice.  (The PEP only covers CGI variables.)
> 
> I mean the ones required by the spec, not server-specific extensions.

The spec clearly defines the expected value types (see above).  If it 
doesn't, I will fix that.  ;)

> I'm just asking whether, in your statement of goals and rationale,you 
> would expand "cross compatibility" as meaning cross-pythonversion 
> portability, or whether you meant something else.

Cross-Python version portability is what it was intended to mean.

	- Alice.



From alice at gothcandy.com  Fri Jan  7 22:23:41 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 13:23:41 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
	<ig7t44$2tu$1@dough.gmane.org>
	<AANLkTik_JMrvNAf7KbCpNaEBvriq-=49YZgYD7PzMdWT@mail.gmail.com>
Message-ID: <ig808t$j0c$1@dough.gmane.org>

On 2011-01-07 12:42:24 -0800, Paul Davis said:

> Is the code for this server online? I'd be interested in reading through it.

https://github.com/pulp/marrow.server.http

There are two branches: master will always refer to the version 
published on Python.org, and draft refers to my rewrite.  (When 
published, draft will be merged.)

	- Alice.



From alice at gothcandy.com  Fri Jan  7 23:33:23 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 14:33:23 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
	<ig7t44$2tu$1@dough.gmane.org>
	<loom.20110107T221211-708@post.gmane.org>
Message-ID: <ig84bj$7ff$1@dough.gmane.org>

On 2011-01-07 13:21:36 -0800, Antoine Pitrou said:
> Ok, so, WSGI doesn't "already involve generators". QED.

This can go around in circles; by allowing all forms of iterable, it 
involves generators.  Geneators are a type of iterable.  QED right 
back.  ;)

>>> Right, that's why I was suggesting you drop your concern for Python 2 
>>> compatibility.
>> 
>> -1
>> 
>> There is practically no reason for doing so;
> 
> Of course, there is one: a less complex PEP without any superfluous 
> compatibility language sprinkled all over.

There isn't any "compatibility language" sprinkled within the PEP.  In 
fact, the only mention of it is in the introduction (stating that < 2.6 
support may be possible but is undefined) and the title of a section 
"Python Cross-Version Compatibility".

Using native strings where possible encourages compatibility, though 
for the environ variables previously mentioned (URI, etc.) explicit 
exceptional behaviour is clearly defined.  (Byte strings and true 
unicode.)

> Just because you "managed to write" some piece of code for a 
> *particular* use case doesn't mean that cross-compatibility is a solved 
> problem.

The particular use case happens to be PEP 444 as implemented using an 
async and multi-process (some day multi-threaded) HTTP server, so I'm 
not quite sure what you're getting at, here.  I think that use case is 
sufficiently broad to be able to make claims about the ease of 
implementing PEP 444 in a compatible way.

> If you think it's easy, then I'm sure the authors of various 3rd-party 
> libs would welcome your help achieving it.

I helped proof a book about Python 3 compatibility and am giving a 
presentation in March that contains information on Python 3 
compatibility from the viewpoint of implementing the Marrow suite.

>> Python 2.x will be around for a long time.
> 
> And so will PEP 3333 and even PEP 333. People who value legacy 
> compatibility will favour these old PEPs over your new one anyway. 
> People who don't will progressively jump to 3.x.

Yup.  Not sure how this is really an issue.  PEP 444 is the /future/, 
333[3] is /now/ [-ish].

	- Alice.



From alice at gothcandy.com  Fri Jan  7 23:38:40 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 14:38:40 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
Message-ID: <ig84lg$8rj$1@dough.gmane.org>

On 2011-01-07 09:04:07 -0800, Antoine Pitrou said:
> WSGI doesn't mandate any specific feature of generators, such as 
> coroutine-like semantics, and the server doesn't have to know about 
> them.

The joy of writing a new specification is that we are not (potentially) 
shackled by old ways of doing things.  Case in point: dropping 
start_response and changing the return value.  PEP 444 isn't WSGI 1, 
and can change things, including additional changes to the allowable 
return value.

	- Alice.



From alex.gronholm at nextday.fi  Sat Jan  8 00:24:30 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sat, 08 Jan 2011 01:24:30 +0200
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <4D26A37C.2020902@nextday.fi>
References: <4D25F9F0.3070306@inteli-com.com>
	<ig5rcb$csc$1@dough.gmane.org>	<20110107044956.D208F3A40A8@sparrow.telecommunity.com>
	<4D26A37C.2020902@nextday.fi>
Message-ID: <4D27A0AE.8060808@nextday.fi>

07.01.2011 07:24, Alex Gr?nholm kirjoitti:
> 07.01.2011 06:49, P.J. Eby kirjoitti:
>> At 05:47 PM 1/6/2011 -0800, Alice Bevan?McGregor wrote:
>>> Tossing the idea around all day long will then, of course, be 
>>> happening regardless.  Unfortunately for that particular discussion, 
>>> PEP 3148 / Futures seems to have won out in the broader scope.
>>
>> Do any established async frameworks or server (e.g. Twisted, 
>> Eventlet, Gevent,  Tornado, etc.) make use of futures?
> I understand that Twisted has incorporated futures support to their 
> deferreds. Others, I believe, don't support them yet. You have to 
> consider that Python 3.2 (the first Python with futures support in 
> stdlib) hasn't even been released yet, and it's only been two weeks 
> since I released the drop-in backport 
> (http://pypi.python.org/pypi/futures/2.1).
Exarkun corrected me on this -- there is currently no futures support in 
Twisted. Sorry about the false information.
>
>>
>>
>>>   Having a ratified and incorporated language PEP (core in 3.2 w/ 
>>> compatibility package for 2.5 or 2.6+ support) reduces the scope of 
>>> async discussion down to: "how do we integrate futures into WSGI 2" 
>>> instead of "how do we define an async API at all".
>>
>> It would be helpful if you addressed the issue of scope, i.e., what 
>> features are you proposing to offer to the application developer.
>>
>> While the idea of using futures presents some intriguing 
>> possibilities, it seems to me at first glance that all it will do is 
>> move the point where the work gets done.  That is, instead of simply 
>> running the app in a worker, the app will be farming out work to 
>> futures.  But if this is so, then why doesn't the server just farm 
>> the apps themselves out to workers?
>>
>> I guess what I'm saying is, I haven't heard use cases for this from 
>> the application developer POV -- why should an app developer care 
>> about having their app run asynchronously?
> Applications need to be asynchronous to work on a single threaded 
> server. There is no other benefit than speed and concurrency, and 
> having to program a web app to operate asynchronously can be a pain. 
> AFAIK there is no other way if you want to avoid the context switching 
> overhead and support a huge number of concurrent connections.
>
> Thread/process pools are only necessary in an asynchronous application 
> where the app needs to use blocking network APIs or do heavy 
> computation, and such uses can unfortunately present a bottleneck. It 
> follows that it's pretty pointless to have an asynchronous application 
> that uses a thread/process pool on every request.
>
> The goal here is to define a common API for these mutually 
> incompatible asynchronous servers to implement so that you could one 
> day run an asynchronous app on Twisted, Tornado, or whatever without 
> modifications.
>>
>> So far, I believe you're the second major proponent (i.e. ones with 
>> concrete proposals and/or implementations to discuss) of an async 
>> protocol...  and what you have in common with the other proponent is 
>> that you happen to have written an async server that would benefit 
>> from having apps operating asynchronously.  ;-)
>>
>> I find it hard to imagine an app developer wanting to do something 
>> asynchronously for which they would not want to use one of the 
>> big-dog asynchronous frameworks.  (Especially if their app involves 
>> database access, or other communications protocols.)
>>
>> This doesn't mean I think having a futures API is a bad thing, but 
>> ISTM that a futures extension to WSGI 1 could be defined right now 
>> using an x-wsgi-org extension in that case...  and you could then 
>> find out how many people are actually interested in using it.
>>
>> Mainly, though, what I see is people using the futures thing to 
>> shuffle off compute-intensive tasks...  but if they do that, then 
>> they're basically trying to make the server's life easier...  but 
>> under the existing spec, any truly async server implementing WSGI is 
>> going to run the *app* in a "future" of some sort already...
>>
>> Which means that the net result is that putting in async is like 
>> saying to the app developer: "hey, you know this thing that you just 
>> could do in WSGI 1 and the server would take care of it for you?  
>> Well, now you can manage that complexity by yourself!  Isn't that 
>> wonderful?"   ;-)
>>
>> I could be wrong of course, but I'd like to see what concrete use 
>> cases people have for async.  We dropped the first discussion of 
>> async six years ago because someone (I think it might've been James) 
>> pointed out that, well, it isn't actually that useful.  And every 
>> subsequent call for use cases since has been answered with, "well, 
>> the use case is that you want it to be async."
>>
>> Only, that's a *server* developer's use case, not an app developer's 
>> use case...  and only for a minority of server developers, at that.
>>
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe: 
>> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From alex.gronholm at nextday.fi  Sat Jan  8 01:39:46 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sat, 08 Jan 2011 02:39:46 +0200
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <AANLkTim3S=AbwmCPQveCS1tWBKK6xwkMMYOv9MMwom8q@mail.gmail.com>
References: <10086684.26.1294422432945.JavaMail.root@boaz>
	<AANLkTim3S=AbwmCPQveCS1tWBKK6xwkMMYOv9MMwom8q@mail.gmail.com>
Message-ID: <4D27B252.9070505@nextday.fi>

07.01.2011 19:59, Guido van Rossum kirjoitti:
> If it's optional, what's the benefit for the app of getting it through
> WSGI instead of through importing some other standard module? The API
> of the executor will require a lot of thought. I worry that this
> weighs down the WSGI standard with the responsibility of coming up
> with the perfect executor API, and if it's not quite perfect after
> all, servers are additionally required to support the standard but
> suboptimal API effectively forever. Or they can choose not to provide
> it, in which case it was a waste of time putting it in WSGI.
The only plausible reason for having a wsgi.executor object is to make 
writing (asynchronous) middleware easier. Otherwise the app could just 
create its own executor (as it is done now).
If there is no wsgi.executor, how will the middleware get ahold of a 
thread/process pool? Having individual middleware maintain their own 
pools is pretty pointless, as I'm sure everyone will agree. On the other 
hand, I believe allowing wsgi.executor to be either a process pool or a 
thread pool is a recipe for disaster. I'm really not sure where to go 
from here.
> On Fri, Jan 7, 2011 at 9:47 AM, Timothy Farrell
> <tfarrell at owassobible.org>  wrote:
>> There has been much discussion about how to handle async in PEP 444 and that discussion centers around the use of futures.  However, I'm requesting that servers _optionally_ provide environ['wsgi.executor'] as a futures executor that applications can use for the purpose of doing something after the response is fully sent to the client.  This is feature request is designed to be concurrency methodology agnostic.
>>
>> Some example use cases are:
>>
>> - send an email that might block on a slow email server (Alice, I read what you said about Turbomail, but one product is not the solution to all situations)
>> - initiate a database vacuum
>> - clean a cache
>> - build a cache
>> - compile statistics
>>
>> When serving pages of an application, these are all things that could be done after the response has been sent.  Ideally these things don't need to be done in a request thread and aren't incredibly time-sensitive.  It seems to me that futures would be an ideal way of handling this.
>>
>> Thoughts?
>> _______________________________________________
>> Web-SIG mailing list
>> Web-SIG at python.org
>> Web SIG: http://www.python.org/sigs/web-sig
>> Unsubscribe: http://mail.python.org/mailman/options/web-sig/guido%40python.org
>>
>
>


From graham.dumpleton at gmail.com  Sat Jan  8 01:56:15 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sat, 8 Jan 2011 11:56:15 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <20110107155545.8AD763A40A8@sparrow.telecommunity.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
	<20110107155545.8AD763A40A8@sparrow.telecommunity.com>
Message-ID: <AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>

On 8 January 2011 02:55, P.J. Eby <pje at telecommunity.com> wrote:
> At 05:27 PM 1/7/2011 +1100, Graham Dumpleton wrote:
>>
>> Another thing though. For output changed to sys.stdout.buffer. For
>> input should we be using sys.stdin.buffer as well if want bytes?
>
> %&$*()&%!!! ?Sorry, still getting used to this whole Python 3 thing.
> ?(Honestly, I don't even use Python 2.6 for anything real yet.)
>
>
>> Good thing I tried running this. Did we all assume that someone else
>> was actually running it to check it? :-)
>
> Well, I only recently started changing the examples to actual Python 3, vs
> being the old Python 2 examples. ?Though, I'm not sure anybody ever ran the
> Python 2 ones. ?;-)

Latest CGI/WSGI bridge example extract from PEP 3333 seems to work
okay for my simple test.

So, if no more technical problems (vs cosmetic) that anyone else sees,
that is probably it and and we can toss this baby out the door.

Graham

From solipsis at pitrou.net  Sat Jan  8 04:36:52 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 8 Jan 2011 03:36:52 +0000 (UTC)
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <ig3erb$vll$1@dough.gmane.org> <ig3l9g$lqc$1@dough.gmane.org>
	<loom.20110106T122356-956@post.gmane.org>
	<ig4n1m$g6j$1@dough.gmane.org>
	<loom.20110106T185706-961@post.gmane.org>
	<ig6krp$dpi$1@dough.gmane.org>
	<loom.20110107T173907-709@post.gmane.org>
	<ig7t44$2tu$1@dough.gmane.org>
	<loom.20110107T221211-708@post.gmane.org>
	<ig84bj$7ff$1@dough.gmane.org>
Message-ID: <loom.20110108T041841-855@post.gmane.org>


Alice Bevan?McGregor <alice at ...> writes:
> 
> On 2011-01-07 13:21:36 -0800, Antoine Pitrou said:
> > Ok, so, WSGI doesn't "already involve generators". QED.
> 
> This can go around in circles; by allowing all forms of iterable, it 
> involves generators.  Geneators are a type of iterable.  QED right 
> back.  ;)

Please read back in context.

> There isn't any "compatibility language" sprinkled within the PEP.[...]
> 
> Using native strings where possible encourages compatibility, [snip]

The whole "native strings" thing *is* compatibility cruft. A Python 3 PEP would 
only need two string types: bytes and unicode (str).

> > Just because you "managed to write" some piece of code for a 
> > *particular* use case doesn't mean that cross-compatibility is a solved 
> > problem.
> 
> The particular use case happens to be PEP 444 as implemented using an 
> async and multi-process (some day multi-threaded) HTTP server, so I'm 
> not quite sure what you're getting at, here.

It's becoming to difficult to parse. You aren't sure yet what the async part of 
PEP 444 should look like but you have already implemented it?

> > If you think it's easy, then I'm sure the authors of various 3rd-party 
> > libs would welcome your help achieving it.
> 
> I helped proof a book about Python 3 compatibility and am giving a 
> presentation in March that contains information on Python 3 
> compatibility from the viewpoint of implementing the Marrow suite.

Well, I hope not too many people will waste time trying to write code
cross-compatible code rather than solely target Python 3. The whole point of
Python 3 is to make developers' life better, not worse.

> >> Python 2.x will be around for a long time.
> > 
> > And so will PEP 3333 and even PEP 333. People who value legacy 
> > compatibility will favour these old PEPs over your new one anyway. 
> > People who don't will progressively jump to 3.x.
> 
> Yup.  Not sure how this is really an issue.  PEP 444 is the /future/, 
> 333[3] is /now/ [-ish].

Please read back in context (instead of stripping it), *again*.



From alex.gronholm at nextday.fi  Sat Jan  8 04:45:00 2011
From: alex.gronholm at nextday.fi (=?UTF-8?B?QWxleCBHcsO2bmhvbG0=?=)
Date: Sat, 08 Jan 2011 05:45:00 +0200
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <loom.20110108T041841-855@post.gmane.org>
References: <ig3erb$vll$1@dough.gmane.org>
	<ig3l9g$lqc$1@dough.gmane.org>	<loom.20110106T122356-956@post.gmane.org>	<ig4n1m$g6j$1@dough.gmane.org>	<loom.20110106T185706-961@post.gmane.org>	<ig6krp$dpi$1@dough.gmane.org>	<loom.20110107T173907-709@post.gmane.org>	<ig7t44$2tu$1@dough.gmane.org>	<loom.20110107T221211-708@post.gmane.org>	<ig84bj$7ff$1@dough.gmane.org>
	<loom.20110108T041841-855@post.gmane.org>
Message-ID: <4D27DDBC.1040305@nextday.fi>

08.01.2011 05:36, Antoine Pitrou kirjoitti:
> Alice Bevan?McGregor<alice at ...>  writes:
>> On 2011-01-07 13:21:36 -0800, Antoine Pitrou said:
>>> Ok, so, WSGI doesn't "already involve generators". QED.
>> This can go around in circles; by allowing all forms of iterable, it
>> involves generators.  Geneators are a type of iterable.  QED right
>> back.  ;)
> Please read back in context.
>
>> There isn't any "compatibility language" sprinkled within the PEP.[...]
>>
>> Using native strings where possible encourages compatibility, [snip]
> The whole "native strings" thing *is* compatibility cruft. A Python 3 PEP would
> only need two string types: bytes and unicode (str).
>
>>> Just because you "managed to write" some piece of code for a
>>> *particular* use case doesn't mean that cross-compatibility is a solved
>>> problem.
>> The particular use case happens to be PEP 444 as implemented using an
>> async and multi-process (some day multi-threaded) HTTP server, so I'm
>> not quite sure what you're getting at, here.
> It's becoming to difficult to parse. You aren't sure yet what the async part of
> PEP 444 should look like but you have already implemented it?
We are still discussing the possible mechanics of PEP 444 with async 
support. There is nothing definite yet, and certainly no workable 
implementation yet either. Async support may or may not materialize in 
PEP 444, in another PEP or not at all based on the discussions on this 
list and on IRC.
>>> If you think it's easy, then I'm sure the authors of various 3rd-party
>>> libs would welcome your help achieving it.
>> I helped proof a book about Python 3 compatibility and am giving a
>> presentation in March that contains information on Python 3
>> compatibility from the viewpoint of implementing the Marrow suite.
> Well, I hope not too many people will waste time trying to write code
> cross-compatible code rather than solely target Python 3. The whole point of
> Python 3 is to make developers' life better, not worse.
>
>>>> Python 2.x will be around for a long time.
>>> And so will PEP 3333 and even PEP 333. People who value legacy
>>> compatibility will favour these old PEPs over your new one anyway.
>>> People who don't will progressively jump to 3.x.
>> Yup.  Not sure how this is really an issue.  PEP 444 is the /future/,
>> 333[3] is /now/ [-ish].
> Please read back in context (instead of stripping it), *again*.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From pje at telecommunity.com  Sat Jan  8 06:09:34 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 00:09:34 -0500
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
Message-ID: <20110108050939.2208C3A40A8@sparrow.telecommunity.com>

At 12:37 PM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>But is there really any problem with providing a unified method for 
>indication a suspend point?

Yes: a complexity burden that is paid by the many to serve the few -- 
or possibly non-existent.

I still haven't seen anything that suggests there is a large enough 
group of people who want a "portable" async API to justify 
inconveniencing everyone else in order to suit their needs, vs. 
simply having a different calling interface for that need.

If I could go back and change only ONE thing about WSGI 1, it would 
be the calling convention.  It was messed up from the start, 
specifically because I wasn't adamant enough about weighing the needs 
of the many enough against the needs of the few.  Only a few needed a 
push protocol (write()), and only a few even remotely cared about our 
minor nod to asynchrony (yielding empty strings to pause output).

If I'd been smart (or more to the point, prescient), I'd have just 
done a 3-tuple return value from the get-go, and said to hell with 
those other use cases, because everybody else is paying to carry a 
few people who aren't even going to use these features for real.  (As 
it happens, I thought write() would be needed in order to drive 
adoption, and it may well have been at one time.)

Anyway, with a new spec we have the benefit of hindsight: we know 
that, historically, nobody has actually cared enough to propose a 
full-blown async API who wasn't also trying to make their async 
server implementation work without needing threads.  Never in the 
history of the web-sig, AFAIK, has anyone come in and said, "hey, I 
want to have an async app that can run on any async framework."

Nobody blogs or twitters about how terrible it is that the async 
frameworks all have different APIs and that this makes their apps 
non-portable.  We see lots of complaints about not having a Python 3 
WSGI spec, but virtually none about WSGI being essentially synchronous.

I'm not saying there's zero audience for such a thing...  but then, 
at some point there was a non-zero audience for write() and for 
yielding empty strings.  ;-)

The big problem is this: if, as an app developer, you want this 
hypothetical portable async API, you either already have an app that 
is async or you don't.  If you do, then you already got married to 
some particular API and are happy with your choice -- or else you'd 
have bit the bullet and ported.

What you would not do, is come to the Web-SIG and ask for a spec to 
help you port, because you'd then *still have to port* to the new 
API...  unless of course you wanted it to look like the API you're 
already using...  in which case, why are you porting again, exactly?

Oh, you don't have an app...  okay, so *hypothetically*, if you had 
this API -- which, because you're not actually *using* an async API 
right now, you probably don't even know quite what you need -- 
hypothetically if you had this API you would write an app and then 
run it on multiple async frameworks...

See?  It just gets all the way to silly.  The only way you can 
actually get this far in the process seems to be if you are on the 
server side, thinking it would be really cool to make this thing 
because then surely you'll get users.

In practice, I can't imagine how you could write an app with 
substantial async functionality that was sanely portable across the 
major async frameworks, with the possible exception of the two that 
at least share some common code, paradigms, and API.  And even if you 
could, I can't imagine someone wanting to.

So far, you have yet to give a concrete example of an application 
that you personally (or anyone you know of) want to be able to run on 
two different servers.  You've spoken of hypothetical apps and 
hypothetical portability...  but not one concrete, "I want to run 
this under both Twisted and Eventlet" (or some other two 
frameworks/servers), "because of [actual, non-hypothetical rationale here]".

I don't deny that [actual non-hypothetical rationale] may exist 
somewhere, but until somebody shows up with a concrete case, I don't 
see a proposal getting much traction.  (The alternative would be if 
you pull a rabbit out of your hat and propose something that doesn't 
cost anybody anything to implement... but the fact that you're 
tossing the 3-tuple out in favor of yielding indicates you've got no 
such proposal ready at the present time.)

On the plus side, the "run this in a future after the request" 
concept has some legs, and I hope Timothy (or anybody) takes it and 
runs with it.  That has plenty of concrete use cases for portability 
-- every sufficiently-powerful web framework will want to either 
provide that feature, build other features on top of it, or both.

It's the "make the request itself async" part that's the hard sell 
here, and in need of some truly spectacular rationale in order to 
justify the ubiquitous costs it imposes.


From pje at telecommunity.com  Sat Jan  8 05:34:09 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 07 Jan 2011 23:34:09 -0500
Subject: [Web-SIG] PEP 444 Goals
In-Reply-To: <ig806f$ilk$1@dough.gmane.org>
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<20110107041816.8224E3A40A8@sparrow.telecommunity.com>
	<ig6lmh$hrj$1@dough.gmane.org>
	<20110107162815.607003A40A8@sparrow.telecommunity.com>
	<ig806f$ilk$1@dough.gmane.org>
Message-ID: <20110108051039.7EF073A40A8@sparrow.telecommunity.com>

At 01:22 PM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-07 08:28:15 -0800, P.J. Eby said:
>>At 01:17 AM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>>>On 2011-01-06 20:18:12 -0800, P.J. Eby said:
>>>>>:: Reduction of re-implementation / NIH syndrome 
>>>>>by>>>incorporating>the most common (1%) of features most 
>>>>>often>>>relegated to middleware>or functional helpers.
>>>>Note that nearly every application-friendly feature you add 
>>>>will>>increase the burden on both server developers and 
>>>>middleware>>developers, which ironically means that application 
>>>>developers>>actually end up with fewer options.
>>>Some things shouldn't have multiple options in the first place.  ;)
>>I meant that if a server doesn't implement the spec because of 
>>arequired feature, then the app developer doesn't have the option 
>>of using that feature anyway -- meaning that adding the feature to 
>>the spec didn't really help.
>
>I truly can not worry about non-conformant applications, middleware, 
>or servers and still keep my hair.

I said "if a server doesn't implement the *spec*", meaning, they 
choose not to support PEP 444 *at all*, not that they skip providing 
the feature.


>>Easy enough to write quick, say, 10-line utility functions that 
>>arecorrect middleware -- so that you could actually build 
>>yourapplication out of WSGI functions calling other WSGI-based functions.
>>The yielding thing wouldn't work for that at all.
>
>Handling a possible generator isn't that difficult.

That it's difficult at all means removes degree-of-difficulty as a 
strong motivation to switch.


>>So, in order to know what type each CGI variable is, you'll need a reference?
>
>Reference?  Re-read what I wrote.  Only URI-specific values utilize 
>an encoding reference variable in the environment; that's four 
>values out of the entire environ.  There is one, clearly defined 
>bytes value.  The rest are native strings, decoded using 
>latin1/iso-8859-1/"str-in-unicode" where native strings are unicode.

IOW, there are six specific facts someone needs to remember in order 
to know the type of a given CGI variable, over and above the mere 
fact that it's a CGI variable.  Hence, "reference".


From alex.gronholm at nextday.fi  Sat Jan  8 07:13:17 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sat, 08 Jan 2011 08:13:17 +0200
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
In-Reply-To: <20110108050939.2208C3A40A8@sparrow.telecommunity.com>
References: <20110108050939.2208C3A40A8@sparrow.telecommunity.com>
Message-ID: <4D28007D.30903@nextday.fi>

08.01.2011 07:09, P.J. Eby kirjoitti:
> At 12:37 PM 1/7/2011 -0800, Alice Bevan?McGregor wrote:
>> But is there really any problem with providing a unified method for 
>> indication a suspend point?
>
> Yes: a complexity burden that is paid by the many to serve the few -- 
> or possibly non-existent.
>
> I still haven't seen anything that suggests there is a large enough 
> group of people who want a "portable" async API to justify 
> inconveniencing everyone else in order to suit their needs, vs. simply 
> having a different calling interface for that need.
>
> If I could go back and change only ONE thing about WSGI 1, it would be 
> the calling convention.  It was messed up from the start, specifically 
> because I wasn't adamant enough about weighing the needs of the many 
> enough against the needs of the few.  Only a few needed a push 
> protocol (write()), and only a few even remotely cared about our minor 
> nod to asynchrony (yielding empty strings to pause output).
>
> If I'd been smart (or more to the point, prescient), I'd have just 
> done a 3-tuple return value from the get-go, and said to hell with 
> those other use cases, because everybody else is paying to carry a few 
> people who aren't even going to use these features for real.  (As it 
> happens, I thought write() would be needed in order to drive adoption, 
> and it may well have been at one time.)
>
> Anyway, with a new spec we have the benefit of hindsight: we know 
> that, historically, nobody has actually cared enough to propose a 
> full-blown async API who wasn't also trying to make their async server 
> implementation work without needing threads.  Never in the history of 
> the web-sig, AFAIK, has anyone come in and said, "hey, I want to have 
> an async app that can run on any async framework."
>
> Nobody blogs or twitters about how terrible it is that the async 
> frameworks all have different APIs and that this makes their apps 
> non-portable.  We see lots of complaints about not having a Python 3 
> WSGI spec, but virtually none about WSGI being essentially synchronous.
>
> I'm not saying there's zero audience for such a thing...  but then, at 
> some point there was a non-zero audience for write() and for yielding 
> empty strings.  ;-)
>
> The big problem is this: if, as an app developer, you want this 
> hypothetical portable async API, you either already have an app that 
> is async or you don't.  If you do, then you already got married to 
> some particular API and are happy with your choice -- or else you'd 
> have bit the bullet and ported.
>
> What you would not do, is come to the Web-SIG and ask for a spec to 
> help you port, because you'd then *still have to port* to the new 
> API...  unless of course you wanted it to look like the API you're 
> already using...  in which case, why are you porting again, exactly?
>
> Oh, you don't have an app...  okay, so *hypothetically*, if you had 
> this API -- which, because you're not actually *using* an async API 
> right now, you probably don't even know quite what you need -- 
> hypothetically if you had this API you would write an app and then run 
> it on multiple async frameworks...
>
> See?  It just gets all the way to silly.  The only way you can 
> actually get this far in the process seems to be if you are on the 
> server side, thinking it would be really cool to make this thing 
> because then surely you'll get users.
>
> In practice, I can't imagine how you could write an app with 
> substantial async functionality that was sanely portable across the 
> major async frameworks, with the possible exception of the two that at 
> least share some common code, paradigms, and API.  And even if you 
> could, I can't imagine someone wanting to.
>
> So far, you have yet to give a concrete example of an application that 
> you personally (or anyone you know of) want to be able to run on two 
> different servers.  You've spoken of hypothetical apps and 
> hypothetical portability...  but not one concrete, "I want to run this 
> under both Twisted and Eventlet" (or some other two 
> frameworks/servers), "because of [actual, non-hypothetical rationale 
> here]".
How do you suppose common async middleware could be implemented without 
a common async API? Today we have plenty of WSGI middleware, which would 
not be possible without a common API. You would have to make separate 
interfaces for every major framework and separately test against each of 
them instead of having a reasonable expectation that it will work 
uniformly across compliant frameworks. I would really love to see common 
middleware components that are usable on twisted, tornado etc. without 
modifications.

You seem to be under the impression that asynchronous applications only 
have some specialized uses. Asynchronous applications are no more 
limited in scope than synchronous ones are. It's just an alternative 
programming paradigm that has the potential of squeezing more 
performance out of a server. Note that I am in now way insisting that 
PEP 444 require async support; I'm only exploring that possibility. If 
we cannot figure out a way to make it easy for implementors to support, 
then I will push for a separate specification.
>
>
> I don't deny that [actual non-hypothetical rationale] may exist 
> somewhere, but until somebody shows up with a concrete case, I don't 
> see a proposal getting much traction.  (The alternative would be if 
> you pull a rabbit out of your hat and propose something that doesn't 
> cost anybody anything to implement... but the fact that you're tossing 
> the 3-tuple out in favor of yielding indicates you've got no such 
> proposal ready at the present time.)
>
> On the plus side, the "run this in a future after the request" concept 
> has some legs, and I hope Timothy (or anybody) takes it and runs with 
> it.  That has plenty of concrete use cases for portability -- every 
> sufficiently-powerful web framework will want to either provide that 
> feature, build other features on top of it, or both.
What exactly does "run this in a future after the request" mean? There 
seems to be some terminology confusion here.
>
>
> It's the "make the request itself async" part that's the hard sell 
> here, and in need of some truly spectacular rationale in order to 
> justify the ubiquitous costs it imposes.
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From alice at gothcandy.com  Sat Jan  8 07:13:07 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 22:13:07 -0800
Subject: [Web-SIG] PEP 444 Goals
References: <11174894.6.1294342577349.JavaMail.root@boaz>
	<20110107041816.8224E3A40A8@sparrow.telecommunity.com>
	<ig6lmh$hrj$1@dough.gmane.org>
	<20110107162815.607003A40A8@sparrow.telecommunity.com>
	<ig806f$ilk$1@dough.gmane.org>
	<20110108051039.7EF073A40A8@sparrow.telecommunity.com>
Message-ID: <ig8v9j$90u$1@dough.gmane.org>

On 2011-01-07 20:34:09 -0800, P.J. Eby said:
> That it [handling generators] is difficult at all means removes 
> degree-of-difficulty as a strong motivation to switch.

Agreed.  I will be following up with a more concrete idea (including 
p-code) to better describe what is currently in my brain.  (One half of 
which will be just as objectionable, the other half, with Alex 
Gr?nholm's input, far more reasonable.)

> IOW, there are six specific facts someone needs to remember in orderto 
> know the type of a given CGI variable, over and above the merefact that 
> it's a CGI variable.  Hence, "reference".

No, practically there is one.  If you are implementing a Python 3 
solution, a single value (original URI) is an instance of bytes, the 
rest are str.  If you are implementing a Python 2 solution, there's a 
single rule you need to remember: values derived from the URI 
(QUERY_STRING, PATH_INFO, etc.) are unicode, the rest are str.

Poloygot implementors are already accepting that they will need to 
include more in their headspace before writing a single line of code; 
knowing that "native string" differs between the two langauges is a 
fundamental concept nessicary for the act of writing polygot code.

	- Alice.



From alice at gothcandy.com  Sat Jan  8 07:20:07 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 22:20:07 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <20110108050939.2208C3A40A8@sparrow.telecommunity.com>
	<4D28007D.30903@nextday.fi>
Message-ID: <ig8vmn$a7v$1@dough.gmane.org>

On 2011-01-07 22:13:17 -0800, Alex Gr?nholm said:
> 08.01.2011 07:09, P.J. Eby wrote:
>> On the plus side, the "run this in a future after the request" concept 
>> has some legs... [snip]
> 
> What exactly does "run this in a future after the request" mean? There 
> seems to be some terminology confusion here.

I suspect he's referring to some of the notes on the "PEP 444 feature 
request - Futures executor" thread and several of my illustrated use 
cases, notably:

:: Image scaling (e.g. to multiple sizes) after uploading of an image 
to be scaled where the response (Congratulations, image uploded!) does 
not require the result of the scaling.

:: Content indexing which can also be performed after returning the 
success page.

The former would executor.submit() a number of scaling jobs, attach 
completion callbacks to perform some cleanup / database updating / 
etc., and return a response immediately.  The latter is a single 
executor submission that is entirely non-time-critical.

And likely other use cases as well.  This (inclusion of an executor 
tuned to the underlying server in the environment) is one thing I think 
we can (almost) all agree is a good idea.  :D  Discussion on that 
particular idea should be relegated to the feature request thread, 
though.

	- Alice.



From alice at gothcandy.com  Sat Jan  8 07:57:17 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 22:57:17 -0800
Subject: [Web-SIG] PEP 444 / WSGI 2 Async
References: <loom.20110108T041841-855@post.gmane.org>
Message-ID: <ig91sd$gtq$1@dough.gmane.org>

On 2011-01-07 13:21:36 -0800, Antoine Pitrou said:
> Ok, so, WSGI doesn't "already involve generators". QED.

Let me try this again.  With the understanding that:

:: PEP 333[3] and 444 define a response body as an iterable.
:: Thus WSGI involves iterables through definition.
:: A generator is a type of iterable.
:: Thus WSGI involves generators through the use of iterables.

The hypothetical redefinition of an application as a generator is not 
too far out to lunch, considering that WSGI _already involves 
generators_.  (And that the simple case, an application that does not 
utilize async, will require a single word be changed: s/return/yield)

Is that clearer?  The idea refered to below (and posted separately) 
involve this redefinition, which I understand fully will have a number 
of strong opponents.  Considering PEP 444 is a new spec (already 
breaking direct compatibility via the /already/ redefined return value) 
I hope people do not reject this out of hand but instead help explore 
the idea further.

On 2011-01-07 19:36:52 -0800, Antoine Pitrou said:
> Alice Bevan?McGregor <alice at ...> writes:
>> The particular use case happens to be PEP 444 as implemented using an 
>> async and multi-process (some day multi-threaded) HTTP server, so I'm 
>> not quite sure what you're getting at, here.
> 
> It's becoming to difficult to parse. You aren't sure yet what the async 
> part of PEP 444 should look like but you have already implemented it?

Marrow HTTPd (marrow.server.http) [1] is, internally, an asynchronous 
server.  It does not currently expose the reactor to the WSGI 
application via any interface whatsoever.  I am, however, working on 
some p-code examples (that I will post for discussion as mentioned 
above) which I can base a fork of m.s.http off of to experiment.

This means that, yes, I'm not sure how async will work in PEP 444 /in 
the end/, but I am at least attempting to explore the practical 
implications of the ideas thus far in a real codebase.  I'm "getting it 
done", even if it has to change or be scrapped.

>> I helped proof a book about Python 3 compatibility and am giving a 
>> presentation in March that contains information on Python 3 
>> compatibility from the viewpoint of implementing the Marrow suite.
> 
> Well, I hope not too many people will waste time trying to write code 
> cross-compatible code rather than solely target Python 3. The whole 
> point of Python 3 is to make developers' life better, not worse.

I agree, with one correction to your first point.  Application and 
framework developers should whole-heartedly embrase Python 3 and make 
full use of its many features, simplifications and clarifications.  
However, it is demonstrably not Insanely Difficult? to have compatible 
server and middleware implementations with the draft's definition of 
native string.  If server and middleware developers are willing to 
create polygot code, I'm not going to stop them.

Note that this type of compatibility is not mandated, and the use of 
native strings (with one well defined byte string exception) means that 
pure Python 3 programmers can be blissfully ignorant of the 
compatibility implications -- everything else is "unicode" (str), even 
if it's just "bytes-in-unicode" (latin1/iso-8859-1).  Pure Python 2 
programmers have only a small difference (for them) of the URI values 
being unicode; the remaining values are byte strings (str).

I would like to hear a technical reason why this (native strings) is a 
bad idea instead of vague "this will make things harder" -- it won't, 
at least, not measurably, and I have the proof as a working, 100% unit 
tested, performant, cross-compatible polygot HTTP/1.1-compliant server. 
 Written in several days worth of "full-time work" spread across weeks 
because this is a spare-time project; i.e. not a lot of literal work, 
nor "hard".

Hell, it has transformed from a crappy hack to experiment with HTTP 
into a complete (or very nearly so) implementation of PEP 444 in both 
of its current forms (published and draft) that is almost usable, 
ignoring the fact that PEP 444 is mutable, of course.

	- Alice.

[1] http://bit.ly/fLfamO



From alice at gothcandy.com  Sat Jan  8 08:07:56 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Fri, 7 Jan 2011 23:07:56 -0800
Subject: [Web-SIG] PEP 444 feature request - Futures executor
References: <4D27B252.9070505@nextday.fi>
Message-ID: <ig92gc$iun$1@dough.gmane.org>

On 2011-01-07 16:39:46 -0800, Alex Gr?nholm said:
> I believe allowing wsgi.executor to be either a process pool or a 
> thread pool is a recipe for disaster. I'm really not sure where to go 
> from here.

Two solutions:

:: Don't pass an executor to the job such that it can not schedule its 
own futures.

:: Mandate a thread pool, not process pool, to avoid deadlocking.

	- Alice.



From alice at gothcandy.com  Sat Jan  8 12:26:28 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 03:26:28 -0800
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
Message-ID: <ig9hl3$ads$1@dough.gmane.org>

Warning: this assumes we're running on bizzaro-world PEP 444 that 
mandates applications are generators.  Please do not dismiss this idea 
out of hand but give it a good look and maybe some feedback.  ;)

--

Howdy!

I've finished touching up the p-code illustrating my idea of using 
generators to implement async functionality within a WSGI application 
and middleware, including the idea of a wsgi2ref-supplied decorator to 
simplify middleware.

	https://gist.github.com/770743

There may be a few typos in there; I switched from the idea of passing 
back the returned value of the future to passing back the future itself 
in order to better handle exception handling (i.e. not requiring utter 
insanity in the middleware to determine the true source of an exception 
and the need to pass it along).

The second middleware demonstration (using a decorator) makes 
middleware look a lot more like an application: yielding futures, or a 
response, with the addition of yielding an application callable not 
explored in the first (long, but trivial) example.  I believe this 
should cover 99% of middleware use cases, including interactive 
debugging, request routing, etc. and the syntax isn't too bad, if you 
don't mind standardized decorators.

This should be implementable within the context of Marrow HTTPd 
(http://bit.ly/fLfamO) without too much difficulty.

As a side note, I'll be adding threading support to the server 
(actually, marrow.server, the underlying server/protocol abstraction 
m.s.http utilizes) using futures some time over the week-end by 
wrapping the async callback that calls the application with a call to 
an executor, making it immune to blocking, but I suspect the overhead 
will outweigh the benefit for speedy applications.

Testing multi-process vs. multi-threaded using 2 workers each and the 
prime calculation example, threading is 1.5x slower for CPU-intensive 
tasks under Python 2.7.  That's terrible.  It should be 2x; I have 2 
cores.  :/

	- Alice.



From alice at gothcandy.com  Sat Jan  8 14:39:44 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 05:39:44 -0800
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
References: <ig9hl3$ads$1@dough.gmane.org>
Message-ID: <ig9pf0$bkn$1@dough.gmane.org>

As a quick note, this proposal would signifigantly benefit from the 
simplified syntax offered by PEP 380 (Syntax for Delegating to a 
Subgenerator) [1] and possibly PEP 3152 (Cofunctions) [2].  The former 
simplifies delegation and exception passing, and the latter simplifies 
the async side of this.

Unfortunately, AFIK, both are affected by PEP 3003 (Python Language 
Moratorium) [3], which kinda sucks.

	- Alice.

[1] http://www.python.org/dev/peps/pep-0380/
[2] http://www.python.org/dev/peps/pep-3152/
[3] http://www.python.org/dev/peps/pep-3003/



From dstanek at dstanek.com  Sat Jan  8 15:08:57 2011
From: dstanek at dstanek.com (David Stanek)
Date: Sat, 8 Jan 2011 09:08:57 -0500
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <ig9hl3$ads$1@dough.gmane.org>
References: <ig9hl3$ads$1@dough.gmane.org>
Message-ID: <AANLkTi=FEhDk6QUqw5_+Qa8-HL-iAwqEbYC58NMP9Q50@mail.gmail.com>

On Sat, Jan 8, 2011 at 6:26 AM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
>
> I've finished touching up the p-code illustrating my idea of using
> generators to implement async functionality within a WSGI application and
> middleware, including the idea of a wsgi2ref-supplied decorator to simplify
> middleware.
>
> ? ? ? ?https://gist.github.com/770743
>

Under the new spec would I be forced to make my applications and
middleware this complicated? Where is the most up-to-date version of
pep444?

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek

From alice at gothcandy.com  Sat Jan  8 15:17:23 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 06:17:23 -0800
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
References: <ig9hl3$ads$1@dough.gmane.org>
	<AANLkTi=FEhDk6QUqw5_+Qa8-HL-iAwqEbYC58NMP9Q50@mail.gmail.com>
Message-ID: <ig9rlj$l4g$1@dough.gmane.org>

On 2011-01-08 06:08:57 -0800, David Stanek said:
> Under the new spec would I be forced to make my applications and 
> middleware this complicated?

An application that does not utilize futures (and thus this proposal 
for async) is different from the current draft as it is written [1] by 
only one word.  Replace 'return' with 'yield' and you're done.

Middleware is somewhat different (if using the decorator or PEP 380 
syntax) or substantially different (if not using either of the two 
mentioned simplifications) as middleware, by definition, needs to 
implement both server and application sides of the "WSGI conversation".

As a side benefit, this should further reduce the perceived mis-use of 
middleware [2,3] by the coercion (by implementation difficulty) of 
inappropriate middleware being reimplemented as functional calls.

> Where is the most up-to-date version of pep444?

I?m in the process right now of completing my transcription of [1] into 
[4].  Upon completion I will re-submit it for incorporation on the 
Python.org website.  (Still marked as draft, of course.)

	- Alice.

[1] https://github.com/GothAlice/wsgi2/blob/master/pep444.textile
[2] http://dirtsimple.org/2007/02/wsgi-middleware-considered-harmful.html
[3] http://mockit.blogspot.com/2009/07/its-all-wrong.html
[4] https://github.com/GothAlice/wsgi2/blob/master/pep-0444.rst



From dstanek at dstanek.com  Sat Jan  8 15:49:13 2011
From: dstanek at dstanek.com (David Stanek)
Date: Sat, 8 Jan 2011 09:49:13 -0500
Subject: [Web-SIG] WSGI-1 Warts
Message-ID: <AANLkTimEe4MxZ8C+jZ4f5aD3LiVr++YYf7T-8ZZz8wiY@mail.gmail.com>

Is there a list somewhere of the warts that need to be fixed? I found
some in this list's archive and on various blogs, but I was hoping to
find a more comprehensive grouping of them. This would help in
evaluating PEP 444.

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek

From alice at gothcandy.com  Sat Jan  8 16:03:28 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 07:03:28 -0800
Subject: [Web-SIG] WSGI-1 Warts
References: <AANLkTimEe4MxZ8C+jZ4f5aD3LiVr++YYf7T-8ZZz8wiY@mail.gmail.com>
Message-ID: <ig9uc0$po$1@dough.gmane.org>

On 2011-01-08 06:49:13 -0800, David Stanek said:

> Is there a list somewhere of the warts that need to be fixed? I found 
> some in this list's archive and on various blogs, but I was hoping to 
> find a more comprehensive grouping of them.

Graham Dumpleton, the author of Apache's mod_wsgi, has blogged 
extensively on the subject of WSGI faults.  So far all of the points he 
raises are IMHO valid, well described, and he offers solutions for 
most, if not all of them.  See:

	http://blog.dscpl.com.au/

Since not every post about WSGI is actually tagged WSGI, it's best just 
to hit "older posts" a couple of times and start reading.  ;)  
Utilizing RSS will reduce the amount of work that sounds like to a more 
manageable level.

	- Alice.



From alice at gothcandy.com  Sat Jan  8 16:09:30 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 07:09:30 -0800
Subject: [Web-SIG] PEP 444 feature request - Futures executor
References: <10086684.26.1294422432945.JavaMail.root@boaz>
Message-ID: <ig9una$2e5$1@dough.gmane.org>

On 2011-01-07 09:47:12 -0800, Timothy Farrell said:

> However, I'm requesting that servers _optionally_ provide 
> environ['wsgi.executor'] as a futures executor that applications can 
> use for the purpose of doing something after the response is fully sent 
> to the client.  This is feature request is designed to be concurrency 
> methodology agnostic.

Done.  (In terms of implementation, not updating PEP 444.)  :3

The Marrow server now implements a thread pool executor using the 
concurrent.futures module (or equiv. futures PyPi package).  The 
following are the commits; the changes will look bigger than they are 
due to cutting and pasting of several previously nested blocks of code 
into separate functions for use as callbacks.  100% unit test coverage 
is maintained (without errors), an example application is added, and 
the benchmark suite updated to support the definition of thread count.

	http://bit.ly/gUL33v
	http://bit.ly/gyVlgQ

Testing this yourself requires Git checkouts of the 
marrow.server/threading branch and marrow.server.http/threading branch, 
and likely the latest marrow.io from Git as well:

	https://github.com/pulp/marrow.io
	https://github.com/pulp/marrow.server/tree/threaded
	https://github.com/pulp/marrow.server.http/tree/threaded

This update has not been tested under Python 3.x yet; I'll do that 
shortly and push any fixes; I doubt there will be any.

On 2011-01-08 03:26:28 -0800, Alice Bevan?McGregor said in the [PEP 
444] Future- and Generator-Based Async Idea thread:

> As a side note, I'll be adding threading support to the server... but I 
> suspect the overhead will outweigh the benefit for speedy applications.

I was surprisingly quite wrong in this prediction.  The following is 
the output of a C25 pair of benchmarks, the first not threaded, the 
other with 30 threads  (enough so there would be no waiting).

	https://gist.github.com/770893

The difference is the loss of 60 RSecs out of 3280.  Note that the 
implementation I've devised can pass the concurrent.futures executor to 
the WSGI application (and, in fact, does), fufilling the requirements 
of this discussion.  :D

The use of callbacks internally to the HTTP protocol makes a huge 
difference in overhead, I guess.

	- Alice.



From dstanek at dstanek.com  Sat Jan  8 16:22:44 2011
From: dstanek at dstanek.com (David Stanek)
Date: Sat, 8 Jan 2011 10:22:44 -0500
Subject: [Web-SIG] WSGI-1 Warts
In-Reply-To: <ig9uc0$po$1@dough.gmane.org>
References: <AANLkTimEe4MxZ8C+jZ4f5aD3LiVr++YYf7T-8ZZz8wiY@mail.gmail.com>
	<ig9uc0$po$1@dough.gmane.org>
Message-ID: <AANLkTimnvrMTF4aXwgzmE-AdU00-KR3+_-4RdU_Us4EV@mail.gmail.com>

On Sat, Jan 8, 2011 at 10:03 AM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
> On 2011-01-08 06:49:13 -0800, David Stanek said:
>
>> Is there a list somewhere of the warts that need to be fixed? I found some
>> in this list's archive and on various blogs, but I was hoping to find a more
>> comprehensive grouping of them.
>
> Graham Dumpleton, the author of Apache's mod_wsgi, has blogged extensively
> on the subject of WSGI faults. ?So far all of the points he raises are IMHO
> valid, well described, and he offers solutions for most, if not all of them.
> ?See:
>
> ? ? ? ?http://blog.dscpl.com.au/
>
> Since not every post about WSGI is actually tagged WSGI, it's best just to
> hit "older posts" a couple of times and start reading. ?;) ?Utilizing RSS
> will reduce the amount of work that sounds like to a more manageable level.
>

I'm going to take some time this weekend to create a consolidated
list. I was hoping to find something like:

Issue: ....
Discussion: http://....
Summary of resolution: ...


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek

From alice at gothcandy.com  Sat Jan  8 16:35:42 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 07:35:42 -0800
Subject: [Web-SIG] WSGI-1 Warts
References: <AANLkTimEe4MxZ8C+jZ4f5aD3LiVr++YYf7T-8ZZz8wiY@mail.gmail.com>
	<ig9uc0$po$1@dough.gmane.org>
	<AANLkTimnvrMTF4aXwgzmE-AdU00-KR3+_-4RdU_Us4EV@mail.gmail.com>
Message-ID: <iga08e$9cj$1@dough.gmane.org>

On 2011-01-08 07:22:44 -0800, David Stanek said:

> I'm going to take some time this weekend to create a consolidated list. 
> I was hoping to find something like:
> 
> Issue: ....
> Discussion: http://....
> Summary of resolution: ...

I agree; that would be very, very nice to have, though it might be 
helpful (esp. considering the length some of these discussions go to 
and the mixing of ideas within single threads) to mirror the message 
nesting as a series of nested lists (if doing this in HTML) to more 
concisely collect posts vs. pointing to the head of a thread and having 
to go through literally everything.  Of course, that's more work, and 
should be restricted to threads that are, in fact, scattered or 
unfocused.

And that first sentance was waaay too long, indicating that I've now 
been up all night.  :(

	- Alice.



From pje at telecommunity.com  Sat Jan  8 18:00:18 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 12:00:18 -0500
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <ig9hl3$ads$1@dough.gmane.org>
References: <ig9hl3$ads$1@dough.gmane.org>
Message-ID: <20110108170024.B49933A40F5@sparrow.telecommunity.com>

At 03:26 AM 1/8/2011 -0800, Alice Bevan?McGregor wrote:
>Warning: this assumes we're running on bizzaro-world PEP 444 that 
>mandates applications are generators.  Please do not dismiss this 
>idea out of hand but give it a good look and maybe some feedback.  ;)

First-glance feedback: I'm impressed.  You may have something going 
here after all.  I just wish you'd sent this sooner.  ;-)

I can easily see why I didn't think of this myself: I hadn't shifted 
my thinking to accomodate for two important changes in the Python 
environment since the first WSGI spec, circa 2003-04:

1. Coroutines and decorators are ubiquitous and non-intrusive
2. WSGI has stdlib support, and in any event it is much easier to 
rely on non-stdlib packages

My major concern about the approach is still that it requires a fair 
amount of overhead on the part of both app developers and middleware 
developers, even if that overhead mostly consists of importing and 
decorating.  (More below.)


>The second middleware demonstration (using a decorator) makes 
>middleware look a lot more like an application: yielding futures, or 
>a response, with the addition of yielding an application callable 
>not explored in the first (long, but trivial) example.  I believe 
>this should cover 99% of middleware use cases, including interactive 
>debugging, request routing, etc. and the syntax isn't too bad, if 
>you don't mind standardized decorators.

If we assume that the implementation would be in a wsgi2ref for 
Python 3.3 and distributed standalone for 2.x, I think we can make 
something work.  (In the sense of practical to implement, not 
necessarily *desirable*.)

One of my goals is that it should be possible to write "async-naive" 
applications and middleware, so that people who don't care about 
async can ignore it.

On the application side, this is easy: a trivial decorator suffices 
to translate a return into a yield.

For middleware, it's not quite as simple, unless you have a pure 
ingress or egress filter, since you can't simply "call" the 
application.  However, a "context manager"-like pattern applies, 
wherein you could simply yield to calling a wrapped version of the application.

Hm.  This seems to pretty much generalize to a standard 
coroutine/trampoline pattern, where the server provides the 
trampoline, and can provide APIs in the environ to create waitable 
objects that can be yielded upward.

Actually, this is kind of like what I really wanted the futures PEP 
to be about.  And it also preserves composability nicely.

In fact, it doesn't actually need any middleware decorators, if the 
server provides the trampoline.

We would leave your "my_awesome_application" example intact (possibly 
apart from having a friendlier API for reading from wsgi.input), but 
change my_middleware as follows:

    def my_middleware(app):
        def wrapper(environ):
            # pre-response code here
            response = yield app(environ)
            # post-response code here
            yield altered_response
        return wrapper

That's it.  No decorators, no nothing.

The server-level trampoline is then just a function that looks 
something like this:

     def app_trampoline(coroutine, yielded):
         if [yielded is a future of some sort]:
             [arrange to invoke 'coroutine(result)' upon completion]
             [arrange to inovke 'coroutine(None, exc_info)' upon error]
             return "pause"
         elif [yielded is a response]:
             return "return"
         elif [yielded has send/throw methods]:
             return "call"  # tell the coroutine to call it
         else:
             raise TypeError

The trampoline function is used with a coroutine class like this:

     class Coroutine:

         def __init__(self, iterator, trampoline, callback):
             self.stack = [iterator]
             self.trampoline = trampoline
             self()

         def __call__(self, value=None, exc_info=()):
             stack = self.stack
             while stack:
                 try:
                     it = stack[-1]
                     if exc_info:
                         try:
                             rv = it.throw(*exc_info)
                         finally:
                             exc_info = ()
                     else:
                         rv = it.send(value)
                 except BaseException:
                     value = None
                     exc_info = sys.exc_info()
                     if exc_info[0] is StopIteration:
                         # pass return value up the stack
                         value, = exc_info[1].args or (None,)
                         exc_info = ()   # but not the error
                     stack.pop()
                 else:
                     switch = self.trampoline(self, rv)
                     if switch=="pause":
                         return
                     elif switch=="call":
                         stack.append(rv)  # Call subgenerator
                         value, exc_info = None, ()
                     elif switch=="return":
                         value, exc_info = rv, ()
                         stack.pop()

             # Coroutine is entirely finished
             self.callback(value)

And run by simply calling:

     Coroutine(app(environ), app_trampoline, process_response)

Where process_response() is a function receiving a three-tuple to 
process the actual result.

That's basically it.  The Coroutine class is 
server/framework-independent; the minimal trampoline function is the 
part the server author has to write.

The body iterator can follow a similar protocol, but the trampoline 
function is different:

     def body_trampoline(coroutine, yielded):
         if type(yielded) is bytes:
             if len(coroutine.stack)==1:  # only accept from 
outermost middleware
                 [send the bytes out]
                 [arrange to invoke coroutine() when send is completed]
                 return "pause"
             else:
                 return "return"
         if [yielded is a future of some sort]:
             [arrange to invoke 'coroutine(result)' upon completion]
             [arrange to inovke 'coroutine(None, exc_info)' upon error]
             return "pause"
         elif [yielded has send/throw methods]:
             return "call"  # tell the coroutine to call it
         else:
             raise TypeError

So, part of the server's "process_response" callback would look like:

     Coroutine(body_iter, body_trampoline, finish_response)


You can then implement response-processing middleware like this:

     def latinize_body(body_iter):
         while True:
             chunk = yield body_iter
             if chunk is None:
                 break
             else:
                 yield piglatin(yield body_iter)

     def piglatin(app):
         def wrapper(environ):
             s, h, b = yield app(environ)
             if [suitable for processing]:
                 yield s, h, latinize_body(b)
             else:
                 yield s, h, b  # skip body processing


My overall impression is still that there's something worth 
considering here, but there is still some ugly mental overheads 
involved for body-processing middleware, if we want to support 
pausing during the body iteration.  The latinize_body function above 
isn't exactly intuitively obvious, compared to a for loop, and it 
can't be replaced by one without using greenlets.

On the plus side, it can actually all be done without any decorators at all.

(The next interesting challenge would be to integrate this with 
Graham's proposal for adding cleanup handlers...)


From pje at telecommunity.com  Sat Jan  8 18:20:55 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 12:20:55 -0500
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <20110108170024.B49933A40F5@sparrow.telecommunity.com>
References: <ig9hl3$ads$1@dough.gmane.org>
	<20110108170024.B49933A40F5@sparrow.telecommunity.com>
Message-ID: <20110108172056.AAF2A3A40F5@sparrow.telecommunity.com>

I made a few errors in that massive post...

At 12:00 PM 1/8/2011 -0500, P.J. Eby wrote:
>My major concern about the approach is still that it requires a fair 
>amount of overhead on the part of both app developers and middleware 
>developers, even if that overhead mostly consists of importing and 
>decorating.  (More below.)

The above turned out to be happily wrong by the end of the post, 
since no decorators or imports are actually required for app and 
middleware developers.


>You can then implement response-processing middleware like this:
>
>     def latinize_body(body_iter):
>         while True:
>             chunk = yield body_iter
>             if chunk is None:
>                 break
>             else:
>                 yield piglatin(yield body_iter)

The last line above is incorrect; it should've been "yield 
piglatin(chunk)", i.e.:

     def latinize_body(body_iter):
         while True:
             chunk = yield body_iter
             if chunk is None:
                 break
             else:
                 yield piglatin(chunk)

It's still rather unintuitive, though.  There are also plenty of 
topics left to discuss, both of the substantial and bikeshedding varieties.

One big open question still in my mind is, are these middleware 
idioms any easier to get right than the WSGI 1 ones?  For things that 
don't process response bodies, the answer seems to be yes: you just 
stick in a "yield" and you're done.

For things that DO process response bodies, however, you have to have 
ugly loops like the one above.

I suppose it could be argued that, as unintuitive as that 
body-processing loop is, it's still orders of magnitude more 
intuitive than a piece of WSGI 1 middleware that has to handle both 
application yields and write()s!

I suppose my hesitance is due to the fact that it's not as simple as:

     return (piglatin(chunk) for chunk in body_iter)

Which is really the level of simplicity that I was looking 
for.  (IOW, all response-processing middleware pays in this 
slightly-added complexity to support the subset of apps and 
response-processing middleware that need to wait for events during 
body output.)


From pje at telecommunity.com  Sat Jan  8 18:53:37 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 12:53:37 -0500
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <ig9pf0$bkn$1@dough.gmane.org>
References: <ig9hl3$ads$1@dough.gmane.org>
 <ig9pf0$bkn$1@dough.gmane.org>
Message-ID: <20110108175406.569F63A40F5@sparrow.telecommunity.com>

At 05:39 AM 1/8/2011 -0800, Alice Bevan?McGregor wrote:
>As a quick note, this proposal would signifigantly benefit from the 
>simplified syntax offered by PEP 380 (Syntax for Delegating to a 
>Subgenerator) [1] and possibly PEP 3152 (Cofunctions) [2].  The 
>former simplifies delegation and exception passing, and the latter 
>simplifies the async side of this.
>
>Unfortunately, AFIK, both are affected by PEP 3003 (Python Language 
>Moratorium) [3], which kinda sucks.

Luckily, neither PEP is necessary, since we do not need to support 
arbitrary protocols for the "subgenerators" being called.  This makes 
it possible to simply "yield" instead of "yield from", and the 
trampoline functions take care of distinguishing a terminal 
("return") result from an intermediate one.

The Coroutine class I suggested, however, *does* accept explicit 
returns via "raise StopIteration(value)", so it is actually fully 
equivalent to supporting "yield from", as long as it's used with an 
appropriate trampoline function.

(In fact, the structure of the Coroutine class I proposed was stolen 
from an earlier Python-Dev post I did in an attempt to show why PEP 
380 was unnecessary for doing coroutines.  ;-) )

In effect, the only thing that PEP 380 would add here is the syntax 
sugar for 'raise StopIteration(value)', but you can do that with:

     def return_(value):
         raise StopIteration(value)

In any case, my suggestion doesn't need this for either apps or 
response bodies, since the type of data yielded suffices to indicate 
whether the value is a "return" or not.  You only need a subgenerator 
to raise StopIteration if you want to return something to your caller 
that *isn't* a response or body chunk.


From paul.joseph.davis at gmail.com  Sat Jan  8 19:24:56 2011
From: paul.joseph.davis at gmail.com (Paul Davis)
Date: Sat, 8 Jan 2011 13:24:56 -0500
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <ig9hl3$ads$1@dough.gmane.org>
References: <ig9hl3$ads$1@dough.gmane.org>
Message-ID: <AANLkTinViyKOTPmH3aaYO=b_TSb48t_nHXs9JYaCtqXx@mail.gmail.com>

On Sat, Jan 8, 2011 at 6:26 AM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
> Warning: this assumes we're running on bizzaro-world PEP 444 that mandates
> applications are generators. ?Please do not dismiss this idea out of hand
> but give it a good look and maybe some feedback. ?;)
>
> --
>
> Howdy!
>
> I've finished touching up the p-code illustrating my idea of using
> generators to implement async functionality within a WSGI application and
> middleware, including the idea of a wsgi2ref-supplied decorator to simplify
> middleware.
>
> ? ? ? ?https://gist.github.com/770743
>
> There may be a few typos in there; I switched from the idea of passing back
> the returned value of the future to passing back the future itself in order
> to better handle exception handling (i.e. not requiring utter insanity in
> the middleware to determine the true source of an exception and the need to
> pass it along).
>
> The second middleware demonstration (using a decorator) makes middleware
> look a lot more like an application: yielding futures, or a response, with
> the addition of yielding an application callable not explored in the first
> (long, but trivial) example. ?I believe this should cover 99% of middleware
> use cases, including interactive debugging, request routing, etc. and the
> syntax isn't too bad, if you don't mind standardized decorators.
>
> This should be implementable within the context of Marrow HTTPd
> (http://bit.ly/fLfamO) without too much difficulty.
>
> As a side note, I'll be adding threading support to the server (actually,
> marrow.server, the underlying server/protocol abstraction m.s.http utilizes)
> using futures some time over the week-end by wrapping the async callback
> that calls the application with a call to an executor, making it immune to
> blocking, but I suspect the overhead will outweigh the benefit for speedy
> applications.
>
> Testing multi-process vs. multi-threaded using 2 workers each and the prime
> calculation example, threading is 1.5x slower for CPU-intensive tasks under
> Python 2.7. ?That's terrible. ?It should be 2x; I have 2 cores. ?:/
>
> ? ? ? ?- Alice.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/paul.joseph.davis%40gmail.com
>

For contrast, I thought it might be beneficial to have a comparison
with an implementation that didn't use async might look like:

http://friendpaste.com/4lFbZsTpPGA9N9niyOt9PF

If your implementation requires that people change source code (yield
vs return) when they move code between sync and async servers, doesn't
that pretty much violate the main WSGI goal of portability?

IMO, the async middleware is quite more complex than the current state
of things with start_response. The ability to subtly miss invoking the
generator, or invoking it too many times and dropping part of a
response. Forcing every middleware to unwrap iterators and handle
their own StopExceptions is worrisome as well.

I can't decide if casting the complexity of the async middleware as a
side benefit of discouraging authors was a joke or not.

Either way this proposal reminds me quite a bit of Duff's device [1].
On its own Duff's device is quite amusing and could even be employed
in some situations to great effect. On the other hand, any WSGI spec
has to be understandable and implementable by people from all skill
ranges. If its a spec that only a handful of people comprehend, then I
fear its adoption would be significantly slowed in practice.


[1] http://en.wikipedia.org/wiki/Duff's_device

From pje at telecommunity.com  Sat Jan  8 20:57:36 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 14:57:36 -0500
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <AANLkTinViyKOTPmH3aaYO=b_TSb48t_nHXs9JYaCtqXx@mail.gmail.c
 om>
References: <ig9hl3$ads$1@dough.gmane.org>
	<AANLkTinViyKOTPmH3aaYO=b_TSb48t_nHXs9JYaCtqXx@mail.gmail.com>
Message-ID: <20110108195742.43CCF3A40F5@sparrow.telecommunity.com>

At 01:24 PM 1/8/2011 -0500, Paul Davis wrote:
>For contrast, I thought it might be beneficial to have a comparison
>with an implementation that didn't use async might look like:
>
>http://friendpaste.com/4lFbZsTpPGA9N9niyOt9PF

Compare your version with this one, that uses my revision of Alice's proposal:

def my_awesome_application(environ):
     # do stuff
     yield b'200 OK', [], ["Hello, World!"]

def my_middleware(app):
     def wrapper(environ):
         # maybe edit environ
         try:
             status, headers, body = yield app(environ)
             # maybe edit response:
             # body = latinize(body)
             yield status, headers, body
         except:
             # maybe handle error
         finally:
             # maybe release resources

def my_server(app, httpreq):
     environ = wsgi.make_environ(httpreq)

     def process_response(result):
         status, headers, body = result
         write_headers(httpreq, status, headers)
         Coroutine(body, body_trampoline, finish_response)

     def finish_response(result):
         # cleanup, if any

     Coroutine(app(environ), app_trampoline, process_response)


The primary differences are that the server needs to split some of 
its processing into separate routines, and response-processing done 
by middleware has to happen in a while loop rather than a for loop.


>If your implementation requires that people change source code (yield
>vs return) when they move code between sync and async servers, doesn't
>that pretty much violate the main WSGI goal of portability?

The idea here would be to have WSGI 2 use this protocol exclusively, 
not to have two different protocols.


>IMO, the async middleware is quite more complex than the current state
>of things with start_response.

Under the above proposal, it isn't, since you can't (only) do a for 
loop over the response body; you have to write a loop and a 
push-based handler as well.  In this case, it is reduced to just 
writing one loop.

I'm still not entirely convinced of the viability of the approach, 
but I'm no longer in the "that's just crazy talk" category regarding 
an async WSGI.  The cost is no longer crazy, but there's still some 
cost involved, and the use case rationale hasn't improved much.

OTOH, I can now conceive of actually *using* such an async API for 
something, and that's no small feat.  Before now, the idea held 
virtually zero interest for me.


>Either way this proposal reminds me quite a bit of Duff's device [1].
>On its own Duff's device is quite amusing and could even be employed
>in some situations to great effect. On the other hand, any WSGI spec
>has to be understandable and implementable by people from all skill
>ranges. If its a spec that only a handful of people comprehend, then I
>fear its adoption would be significantly slowed in practice.

Under my modification of Alice's proposal, nearly all of the 
complexity involved migrates to the server, mostly in the (shareable) 
Coroutine implementation.

For an async server, the "arrange for coroutine(result) to be called" 
operations are generally native to async APIs, so I'd expect them to 
find that simple to implement.  Synchronous servers just need to 
invoke the waited-on operation synchronously, then pass the value 
back into the coroutine.  (e.g. by returning "pause" from the 
trampoline, then calling coroutine(value, exc_info) to resume 
processing after the result is obtained.)



From pje at telecommunity.com  Sat Jan  8 22:16:52 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 16:16:52 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
Message-ID: <20110108211658.6DC163A4119@sparrow.telecommunity.com>

As a semi-proof-of-concept, I whipped these up:

   http://peak.telecommunity.com/DevCenter/AsyncWSGISketch

It's an expanded version of my Coroutine concept, updated with sample 
server code for both a synchronous server and an asynchronous 
one.  The synchronous "server" is really just a decorator that wraps 
a WSGI2 async app with futures support, and handles pauses by simply 
waiting for the future to finish.

The asynchronous server is a bit more hand-wavy, in that there are 
some bits (clearly marked) that will be server/framework 
dependent.  However, they should be straightforward for a specialist 
in any given async framework to implement.

What is *most* handwavy at the moment, however, is in the details of 
precisely what one is allowed to "yield to".  I've written the 
sketches dealing only with PEP 3148 futures, but sockets were also 
proposed, and IMO there should be simple support for obtaining data 
from wsgi.input.

However, even this part is pretty easy to extrapolate: both server 
examples just add more type-testing branches in their 
"base_trampoline()" function, copying and modifying the existing 
branches that deal with futures.

The entire result is surprisingly compact -- each server weighed in 
at about 40 lines, and the common Coroutine class used by both adds 
another 60-something lines.

In the limit case, it appears that any WSGI 1 server could provide an 
(emulated) async WSGI2 implementation, simply by wrapping WSGI2 apps 
with a finished version of the decorator in my sketch.

Or, since users could do it themselves, this would mean that WSGI2 
deployment wouldn't be dependent on all server implementers 
immediately turning out their own WSGI2 implementations.

True async API implementations would be more involved, of course -- 
using a WSGI2 decorator on say, Twisted's WSGI1 implementation, would 
give you no performance advantages vs. using Twisted's APIs 
directly.  But, as soon as someone wrote a Twisted-specific 
translation of my async-server sketch, such an app would be portable.

More discussion is still needed, but at this point I'm convinced the 
concept is *technically* feasible.  (Whether there's enough need in 
the "market" to make it worthwhile, is a separate question.)


From alice at gothcandy.com  Sun Jan  9 02:16:06 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 17:16:06 -0800
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
References: <ig9hl3$ads$1@dough.gmane.org>
	<20110108170024.B49933A40F5@sparrow.telecommunity.com>
Message-ID: <igb28m$pg0$1@dough.gmane.org>

On 2011-01-08 09:00:18 -0800, P.J. Eby said:

> (The next interesting challenge would be to integrate this withGraham's 
> proposal for adding cleanup handlers...)

class MyApplication(object):
    def __init__(self):
        pass # process startup code

    def __call__(self, environ):
        yield None # must be a generator
        pass # request code

    def __enter__(self):
        pass # request startup code

    def __exit(exc_type, exc_val, exc_tb):
        pass # request shutdown code -- regardless of exceptions

We could mandate context managers!  :D  (Which means you can still wrap 
a simple function in @contextmanager.)

	- Alice.



From alex.gronholm at nextday.fi  Sun Jan  9 02:22:44 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sun, 09 Jan 2011 03:22:44 +0200
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
Message-ID: <4D290DE4.1030400@nextday.fi>

08.01.2011 23:16, P.J. Eby kirjoitti:
> As a semi-proof-of-concept, I whipped these up:
>
>   http://peak.telecommunity.com/DevCenter/AsyncWSGISketch
>
> It's an expanded version of my Coroutine concept, updated with sample 
> server code for both a synchronous server and an asynchronous one.  
> The synchronous "server" is really just a decorator that wraps a WSGI2 
> async app with futures support, and handles pauses by simply waiting 
> for the future to finish.
>
> The asynchronous server is a bit more hand-wavy, in that there are 
> some bits (clearly marked) that will be server/framework dependent.  
> However, they should be straightforward for a specialist in any given 
> async framework to implement.
>
> What is *most* handwavy at the moment, however, is in the details of 
> precisely what one is allowed to "yield to".  I've written the 
> sketches dealing only with PEP 3148 futures, but sockets were also 
> proposed, and IMO there should be simple support for obtaining data 
> from wsgi.input.
>
> However, even this part is pretty easy to extrapolate: both server 
> examples just add more type-testing branches in their 
> "base_trampoline()" function, copying and modifying the existing 
> branches that deal with futures.
>
> The entire result is surprisingly compact -- each server weighed in at 
> about 40 lines, and the common Coroutine class used by both adds 
> another 60-something lines.
>
> In the limit case, it appears that any WSGI 1 server could provide an 
> (emulated) async WSGI2 implementation, simply by wrapping WSGI2 apps 
> with a finished version of the decorator in my sketch.
>
> Or, since users could do it themselves, this would mean that WSGI2 
> deployment wouldn't be dependent on all server implementers 
> immediately turning out their own WSGI2 implementations.
>
> True async API implementations would be more involved, of course -- 
> using a WSGI2 decorator on say, Twisted's WSGI1 implementation, would 
> give you no performance advantages vs. using Twisted's APIs directly.  
> But, as soon as someone wrote a Twisted-specific translation of my 
> async-server sketch, such an app would be portable.
>
> More discussion is still needed, but at this point I'm convinced the 
> concept is *technically* feasible.  (Whether there's enough need in 
> the "market" to make it worthwhile, is a separate question.)
I'm a bit unclear as to how this will work with async. How do you 
propose that an asynchronous application receives the request body?
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From alice at gothcandy.com  Sun Jan  9 03:15:00 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sat, 8 Jan 2011 18:15:00 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi>
Message-ID: <igb5n4$54u$1@dough.gmane.org>

On 2011-01-08 17:22:44 -0800, Alex Gr?nholm said:

>> On 2011-01-08 13:16:52 -0800, P.J. Eby said:
>> 
>> I've written the sketches dealing only with PEP 3148 futures, but 
>> sockets were also proposed, and IMO there should be simple support for 
>> obtaining data from wsgi.input.
> 
> I'm a bit unclear as to how this will work with async. How do you 
> propose that an asynchronous application receives the request body?

In my example https://gist.github.com/770743 (which has been simplified 
greatly by P.J. Eby in the "Future- and Generator-Based Async Idea" 
thread) for dealing with wsgi.input, I have:

    future = environ['wsgi.executor'].submit(environ['wsgi.input'].read, 4096)
    yield future

While ugly, if you were doing this, you'd likely:

	submit = environ['wsgi.executor'].submit
	input_ = environ['wsgi.input']

    future = yield submit(input_.read, 4096)
    data = future.

That's a bit nicer to read, and simplifies things if you need to make a 
number of async calls.

The idea here is that:

:: Your async server subclasses ThreadPoolExecutor.

:: The subclass overloads the submit method.

:: Your submit method detects bound methods on wsgi.input, sockets, and files.

:: If one of the above is detected, create a mock future that defines 
'fd' and 'operation' attributes or similar.

:: When yielding the mock future, your async reactor can detect 'fd' 
and do the appropriate thing for your async framework.  (Generally 
adding the fd to the appropriate select/epoll/kqueue readers/writers 
lists.)

:: When the condition is met, set_running_or_notify_cancel (when 
internally reading or writing data), set_result, saving the value, and 
return the future (filled with its data) back up to the application.

:: The application accepts the future instance as the return value of 
yield, and calls result across it to get the data.  (Obviously writes, 
if allowed, won't have data, but reads will.)

I hope that clearly identifies my idea on the subject.  Since async 
servers will /already/ be implementing their own executors, I don't see 
this as too crazy.

	- Alice.



From alex.gronholm at nextday.fi  Sun Jan  9 03:40:41 2011
From: alex.gronholm at nextday.fi (=?windows-1252?Q?Alex_Gr=F6nholm?=)
Date: Sun, 09 Jan 2011 04:40:41 +0200
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igb5n4$54u$1@dough.gmane.org>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>	<4D290DE4.1030400@nextday.fi>
	<igb5n4$54u$1@dough.gmane.org>
Message-ID: <4D292029.2080102@nextday.fi>

09.01.2011 04:15, Alice Bevan?McGregor kirjoitti:
> On 2011-01-08 17:22:44 -0800, Alex Gr?nholm said:
>
>>> On 2011-01-08 13:16:52 -0800, P.J. Eby said:
>>>
>>> I've written the sketches dealing only with PEP 3148 futures, but 
>>> sockets were also proposed, and IMO there should be simple support 
>>> for obtaining data from wsgi.input.
>>
>> I'm a bit unclear as to how this will work with async. How do you 
>> propose that an asynchronous application receives the request body?
>
> In my example https://gist.github.com/770743 (which has been 
> simplified greatly by P.J. Eby in the "Future- and Generator-Based 
> Async Idea" thread) for dealing with wsgi.input, I have:
>
> future = environ['wsgi.executor'].submit(environ['wsgi.input'].read, 
> 4096)
> yield future
>
> While ugly, if you were doing this, you'd likely:
>
> submit = environ['wsgi.executor'].submit
> input_ = environ['wsgi.input']
>
> future = yield submit(input_.read, 4096)
> data = future.
>
> That's a bit nicer to read, and simplifies things if you need to make 
> a number of async calls.
>
> The idea here is that:
>
> :: Your async server subclasses ThreadPoolExecutor.
>
> :: The subclass overloads the submit method.
>
> :: Your submit method detects bound methods on wsgi.input, sockets, 
> and files.
>
> :: If one of the above is detected, create a mock future that defines 
> 'fd' and 'operation' attributes or similar.
>
> :: When yielding the mock future, your async reactor can detect 'fd' 
> and do the appropriate thing for your async framework. (Generally 
> adding the fd to the appropriate select/epoll/kqueue readers/writers 
> lists.)
>
> :: When the condition is met, set_running_or_notify_cancel (when 
> internally reading or writing data), set_result, saving the value, and 
> return the future (filled with its data) back up to the application.
>
> :: The application accepts the future instance as the return value of 
> yield, and calls result across it to get the data. (Obviously writes, 
> if allowed, won't have data, but reads will.)
>
> I hope that clearly identifies my idea on the subject. Since async 
> servers will /already/ be implementing their own executors, I don't 
> see this as too crazy.
-1 on this. Those executors are meant for executing code in a thread 
pool. Mandating a magical socket operation filter here would 
considerably complicate server implementation.
>
>
> - Alice.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From pje at telecommunity.com  Sun Jan  9 04:34:41 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 22:34:41 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <4D292029.2080102@nextday.fi>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<4D292029.2080102@nextday.fi>
Message-ID: <20110109033446.911C93A4119@sparrow.telecommunity.com>

At 04:40 AM 1/9/2011 +0200, Alex Gr?nholm wrote:
>09.01.2011 04:15, Alice Bevan?McGregor kirjoitti:
>>I hope that clearly identifies my idea on the subject. Since async 
>>servers will /already/ be implementing their own executors, I don't 
>>see this as too crazy.
>-1 on this. Those executors are meant for executing code in a thread 
>pool. Mandating a magical socket operation filter here would 
>considerably complicate server implementation.

Actually, the *reverse* is true.  If you do it the way Alice 
proposes, my sketches don't get any more complex, because the 
filtering goes in the executor facade or submit function.

Truthfully, I don't really see the point of exposing the map() method 
(which is the only other executor method we'd expose), so it probably 
makes more sense to just offer a 'wsgi.submit' key...  which can be a 
function as follows:

       def submit(callable, *args, **kw):
           ob = getattr(callable, '__self__', None)
           if isinstance(ob, ServerProvidedSocket):  # could be an ABC
                future = MockFuture()
                if callable==ob.read:
                    # set up read callback to fire future
                elif callable==ob.write:
                    # set up write callback to fire future
                return future
           else:
               return real_executor.submit(callable, *args, **kw)

Granted, this might be a rather long function.  However, since it's 
essentially an optimization, a given server can decide how many 
functions can be shortcut in this way.  The spec may wish to offer a 
guarantee or recommendation for specific methods of certain 
stdlib-provided types (sockets in particular) and wsgi.input.

Personally, I do think it might be *better* to offer extended 
operations on wsgi.input that could be used via yield, e.g. "yield 
input.nb_read()".  But of course then the trampoline code has to 
recognize those values instead of futures.  Either way works, but 
somewhere there is going to be some type-testing (explicit or 
implicit) taking place to determine how to suspend and resume the app.

Note, too, that this complexity also only affects servers that want 
to offer a truly async API.  A synchronous server has no reason to 
pay particular attention to what's in a future, since it can't offer 
any performance improvement.

I do think that this sort of API discussion, though, is the most 
dangerous part of trying to do an async spec.  That is, I don't 
expect that everyone will spontaneously agree on the exact same 
API.  Alice's proposal (simply submitting object methods) has the 
advantage of severely limiting the scope of API discussions.  ;-)


From graham.dumpleton at gmail.com  Sun Jan  9 04:43:37 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Sun, 9 Jan 2011 14:43:37 +1100
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
In-Reply-To: <igb28m$pg0$1@dough.gmane.org>
References: <ig9hl3$ads$1@dough.gmane.org>
	<20110108170024.B49933A40F5@sparrow.telecommunity.com>
	<igb28m$pg0$1@dough.gmane.org>
Message-ID: <AANLkTikw1Z8xbz5M5PbkVKGDGt-MJj3TGOUXf9pOMyZz@mail.gmail.com>

On 9 January 2011 12:16, Alice Bevan?McGregor <alice at gothcandy.com> wrote:
> On 2011-01-08 09:00:18 -0800, P.J. Eby said:
>
>> (The next interesting challenge would be to integrate this withGraham's
>> proposal for adding cleanup handlers...)
>
> class MyApplication(object):
> ? def __init__(self):
> ? ? ? pass # process startup code
>
> ? def __call__(self, environ):
> ? ? ? yield None # must be a generator
> ? ? ? pass # request code
>
> ? def __enter__(self):
> ? ? ? pass # request startup code
>
> ? def __exit(exc_type, exc_val, exc_tb):
> ? ? ? pass # request shutdown code -- regardless of exceptions
>
> We could mandate context managers! ?:D ?(Which means you can still wrap a
> simple function in @contextmanager.)

Context managers don't solve the problem I am trying to address. The
'with' statement doesn't apply context managers to WSGI application
objects in way that is desirable and use of a decorator to achieve the
same means having to replace close() which is what am trying to avoid
because of extra complexity that causes for WSGI middleware just to
make sure wsgi.file_wrapper works. We want a world where it should
never be necessary for WSGI middleware, or proxy decorators, to have
to fudge up a generator and override the close() chain to add
cleanups.

Graham

> ? ? ? ?- Alice.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com
>

From pje at telecommunity.com  Sun Jan  9 04:45:46 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 08 Jan 2011 22:45:46 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igb5n4$54u$1@dough.gmane.org>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
Message-ID: <20110109034644.0454B3A4119@sparrow.telecommunity.com>

At 06:15 PM 1/8/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-08 17:22:44 -0800, Alex Gr?nholm said:
>>>On 2011-01-08 13:16:52 -0800, P.J. Eby said:
>>>I've written the sketches dealing only with PEP 3148 futures, but 
>>>sockets were also proposed, and IMO there should be simple support 
>>>for obtaining data from wsgi.input.
>>I'm a bit unclear as to how this will work with async. How do you 
>>propose that an asynchronous application receives the request body?
>
>In my example https://gist.github.com/770743 (which has been 
>simplified greatly by P.J. Eby in the "Future- and Generator-Based 
>Async Idea" thread) for dealing with wsgi.input, I have:
>
>    future = environ['wsgi.executor'].submit(environ['wsgi.input'].read, 4096)
>    yield future
>
>While ugly, if you were doing this, you'd likely:
>
>         submit = environ['wsgi.executor'].submit
>         input_ = environ['wsgi.input']
>
>    future = yield submit(input_.read, 4096)
>    data = future.

I don't quite understand the above -- in my sketch, the above would be:

     data = yield submit(input._read, 4096)

It looks like your original sketch wants to call .result() on the 
future, whereas in my version, the return value of yielding a future 
is the result (or an error is thrown if the result was an error).

Is there some reason I'm missing, for why you'd want to explicitly 
fetch the result in a separate step?

Meanwhile, thinking about Alex's question, ISTM that if WSGI 2 is 
asynchronous, then the wsgi.input object should probably just have 
read(), readline() etc. methods that simply return (possibly-mock) 
futures.  That's *much* better than having to do all that submit() 
crud just to read data from wsgi.input().

OTOH, if you want to use the cgi module to parse a form POST from the 
input, you're going to need to write an async version of it in that 
case, or else feed the entire operation to an executor...  but then 
the methods would need to be synchronous...  *argh*.

I'm starting to not like this idea at all.  Alex has actually 
pinpointed a very weak spot in the scheme, which is that if 
wsgi.input is synchronous, you destroy the asynchrony, but if it's 
asynchronous, you can't use it with any normal code that operates on a stream.

I don't see any immediate fixes for this problem, so I'll let it 
marinate in the back of my mind for a while.  This might be the 
achilles heel for the whole idea of a low-rent async WSGI.


From alex.gronholm at nextday.fi  Sun Jan  9 05:06:19 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sun, 09 Jan 2011 06:06:19 +0200
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <20110109034644.0454B3A4119@sparrow.telecommunity.com>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>	<4D290DE4.1030400@nextday.fi>
	<igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
Message-ID: <4D29343B.9000403@nextday.fi>

09.01.2011 05:45, P.J. Eby kirjoitti:
> At 06:15 PM 1/8/2011 -0800, Alice Bevan?McGregor wrote:
>> On 2011-01-08 17:22:44 -0800, Alex Gr?nholm said:
>>>> On 2011-01-08 13:16:52 -0800, P.J. Eby said:
>>>> I've written the sketches dealing only with PEP 3148 futures, but 
>>>> sockets were also proposed, and IMO there should be simple support 
>>>> for obtaining data from wsgi.input.
>>> I'm a bit unclear as to how this will work with async. How do you 
>>> propose that an asynchronous application receives the request body?
>>
>> In my example https://gist.github.com/770743 (which has been 
>> simplified greatly by P.J. Eby in the "Future- and Generator-Based 
>> Async Idea" thread) for dealing with wsgi.input, I have:
>>
>>    future = 
>> environ['wsgi.executor'].submit(environ['wsgi.input'].read, 4096)
>>    yield future
>>
>> While ugly, if you were doing this, you'd likely:
>>
>>         submit = environ['wsgi.executor'].submit
>>         input_ = environ['wsgi.input']
>>
>>    future = yield submit(input_.read, 4096)
>>    data = future.
>
> I don't quite understand the above -- in my sketch, the above would be:
>
>     data = yield submit(input._read, 4096)
>
> It looks like your original sketch wants to call .result() on the 
> future, whereas in my version, the return value of yielding a future 
> is the result (or an error is thrown if the result was an error).
I cooked up a simple do-nothing middleware example which Alice decorated 
with some comments:
https://gist.github.com/771398

A new feature here is that the application itself yields a (status, 
headers) tuple and then chunks of the body (or futures).
>
>
> Is there some reason I'm missing, for why you'd want to explicitly 
> fetch the result in a separate step?
>
> Meanwhile, thinking about Alex's question, ISTM that if WSGI 2 is 
> asynchronous, then the wsgi.input object should probably just have 
> read(), readline() etc. methods that simply return (possibly-mock) 
> futures.  That's *much* better than having to do all that submit() 
> crud just to read data from wsgi.input().
>
> OTOH, if you want to use the cgi module to parse a form POST from the 
> input, you're going to need to write an async version of it in that 
> case, or else feed the entire operation to an executor...  but then 
> the methods would need to be synchronous...  *argh*.
>
> I'm starting to not like this idea at all.  Alex has actually 
> pinpointed a very weak spot in the scheme, which is that if wsgi.input 
> is synchronous, you destroy the asynchrony, but if it's asynchronous, 
> you can't use it with any normal code that operates on a stream.
I liked the idea of having a separate async_read() method in wsgi.input, 
which would set the underlying socket in nonblocking mode and return a 
future. The event loop would watch the socket and read data into a 
buffer and trigger the callback when the given amount of data has been 
read. Conversely, .read() would set the socket in blocking mode. What 
kinds of problems would this cause?
>
>
> I don't see any immediate fixes for this problem, so I'll let it 
> marinate in the back of my mind for a while.  This might be the 
> achilles heel for the whole idea of a low-rent async WSGI.
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: 
> http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From alice at gothcandy.com  Sun Jan  9 12:01:12 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 03:01:12 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
Message-ID: <igc4ho$fol$1@dough.gmane.org>

On 2011-01-08 20:06:19 -0800, Alex Gr?nholm said:

> I liked the idea of having a separate async_read() method in 
> wsgi.input, which would set the underlying socket in nonblocking mode 
> and return a future. The event loop would watch the socket and read 
> data into a buffer and trigger the callback when the given amount of 
> data has been read. Conversely, .read() would set the socket in 
> blocking mode. What kinds of problems would this cause?

Manipulating the underlying socket is potentially dangerous 
(pipelining) and, in fact, not possible AFIK while being 
PEP444-compliant.  When the request body is fully consumed, additional 
attempts to read _must_ return empty strings.  Thus raw sockets are 
right out at a high level; internal to the reactor this may be 
possible, however.  It'd be interesting to adapt marrow.io to using 
futures in this way as an experiment.

OTOH, if you utilize callbacks extensively (as m.s.http does) you run 
into the problem of data passing.  Your application is called (wrapped 
in middleware), sets up some futures and callbacks, then returns.  No 
returned data.  Middleware just got shot in the foot.  The server, 
also, got shot in the foot.  How can it get a resopnse tuple back from 
a callback?  How can middleware be utilized?  That's a weird problem to 
wrap my head around.  Blocking the application pending the results of 
various socket operations is something that would have to be mandated 
to avoid this issue.  :/

Multiple in-flight reads would also be problematic; you may end up with 
buffer interleaving issues.  (e.g. job A reads 128 bytes at a time and 
has been requested to return 4KB, job B does the same... what happens 
to the data?)  Then you begin to involve locking...

Notice that my write_body method [1], writes using async, passing the 
iterable to the callback which is itself.  This is after-the-fact 
(after the request has been returned) and is A-OK, though would need to 
be updated heavily to support the ideas of async floating around right 
now.  I'm also extremely careful to never have multiple async callbacks 
pending (and thus never have muliple "jobs" for a single connection 
working at once).

	- Alice.

[1] 
https://github.com/pulp/marrow.server.http/blob/draft/marrow/server/http/protocol.py#L313-332 




From alice at gothcandy.com  Sun Jan  9 12:36:14 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 03:36:14 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110109033446.911C93A4119@sparrow.telecommunity.com>
Message-ID: <igc6je$nhn$1@dough.gmane.org>

On 2011-01-08 19:34:41 -0800, P.J. Eby said:

> At 04:40 AM 1/9/2011 +0200, Alex Gr?nholm wrote:
>> 09.01.2011 04:15, Alice Bevan?McGregor kirjoitti:
>>> I hope that clearly identifies my idea on the subject. Since 
>>> async>>servers will /already/ be implementing their own executors, I 
>>> don't>>see this as too crazy.
>> -1 on this. Those executors are meant for executing code in a 
>> thread>pool. Mandating a magical socket operation filter here 
>> would>considerably complicate server implementation.
> 
> Actually, the *reverse* is true.  If you do it the way Alice proposes, 
> my sketches don't get any more complex, because the filtering goes in 
> the executor facade or submit function.

Indeed; the executor is what then adds the file descriptor to the 
underlying server async reactor (select/epoll/kqueue/other).  In the 
case of the Marrow server, this would utilize a reactor callback (some 
might say "deferred") to update the Future instance with the data, 
setting completion status, executing callbacks, etc.  One might even be 
able to use a threading.Event (or whatever is the opposite of a lock) 
to wake up blocking .result() calls, even if not multi-threaded 
(greenthreads, etc.).

Of course, adding the file descriptor to a pure async reactor then 
.result() blocking on it from your application would result in a 
deadlock; the .result() would never complete as the reactor would never 
get a chance to perform the pending request.  (This is why Marrow 
requires threading be enabled globally before adding an executor to the 
environment; this requires rather explicit documentation.)  This 
problem is solved completely by yielding the future instance (pausing 
the application) to let the reactor do its thing.  (Yielding the future 
becomes a replacement for the blocking behaviour of future.result().)

Effectively what I propose adds emulation of threading on top of async 
by mutating an Executor.  (The Executor would be a mixed 
threading+async executor.)

I suggest bubbling a future back up the yield stack instead of the 
actual result to allow the application (or middleware, or whatever 
happened to yield the future) to capture exceptions generated by the 
future'd request.  Bubbling the future instance avoids excessive 
exception handling cruft in each middleware layer; and I see no real 
issue with this.  AFIK, you can use a shorthand (possibly wrapped in a 
try: block) if all you care about is the result:

    data = (yield my_future).result()

> Truthfully, I don't really see the point of exposing the map() method 
> (which is the only other executor method we'd expose), so it probably 
> makes more sense to just offer a 'wsgi.submit' key... which can be a 
> function as follows: [snip]

True; the executor itself could easily be hidden behind the filter.  In 
a multi-threaded environment, however, the map call poses no problem, 
and can be quite useful.  (E.g. with one of my use cases for inclusion 
of an executor in the environment: image scaling.)

> Granted, this might be a rather long function.  However, since it's 
> essentially an optimization, a given server can decide how many 
> functions can be shortcut in this way.  The spec may wish to offer a 
> guarantee or recommendation for specific methods of certain 
> stdlib-provided types (sockets in particular) and wsgi.input.

+1

> Personally, I do think it might be *better* to offer extended 
> operations on wsgi.input that could be used via yield, e.g. "yield 
> input.nb_read()".  But of course then the trampoline code has 
> torecognize those values instead of futures.

Because wsgi.input is provided by the server, and the executor is 
provided by the server, is there a reason why these extended functions 
couldn't return... futures?  :)

> Note, too, that this complexity also only affects servers that want to 
> offer a truly async API.  A synchronous server has no reason to pay 
> particular attention to what's in a future, since it can't offer any 
> performance improvement.

I feel a sync server and async server should provide the same API for 
accessing the input.  E.g. the application/middleware must be agnostic 
to the server in this regard.  This is why a little bit of magic goes a 
long way.  The following code would work on any WSGI2 stack that offers 
an executor (sync, async, or provided by middleware):

    data = (yield env['wsgi.submit'](env['wsgi.input'].read, 4096)).result()

In a sync server, the blocking read would execute in another thread.  
In an async one appropriate actions would be taken to request a socket 
read from the client.  Both cases pause the application pending the 
result.  (If you don't immediately yield the future the behaviour 
between servers is the same!)

> I do think that this sort of API discussion, though, is the most 
> dangerous part of trying to do an async spec.  That is, I don'texpect 
> that everyone will spontaneously agree on the exact same API.  Alice's 
> proposal (simply submitting object methods) has theadvantage of 
> severely limiting the scope of API discussions.  ;-)

Since each async server will either implement or utilize a specific 
async framework, each will offer its own "async-supported" featureset.  
What I mean is that all servers should make wsgi.input calls 
async-able, some would go further to make all socket calls async.  Some 
might go even further than that and define an API for external 
libraries (e.g. DBs) to be truly cooperatively async.  I do believe my 
solution is flexible enough for the majority of use cases, and where it 
isn't (i.e. would block) "abusing" futures in this way will allow an 
application to reasonalby fake async without killing async server (who 
are internally single-threaded) performance by delegating blocking 
calls.

I will have to experiment with determining the type of the class 
instance a method is bound to from the bound method itself; this is the 
crux of the implementation I suggest.  If you can't get that, the idea 
is pooched for anything but wsgi.input which the server would have a 
direct reference to anyway.

I hope the clarity of this post didn't degenerate too much over the few 
hours I had it open and noodling around.

	- Alice.



From alice at gothcandy.com  Sun Jan  9 13:25:11 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 04:25:11 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
Message-ID: <igc9f7$3qv$1@dough.gmane.org>

On 2011-01-08 13:16:52 -0800, P.J. Eby said:

> In the limit case, it appears that any WSGI 1 server could provide an 
> (emulated) async WSGI2 implementation, simply by wrapping WSGI2 apps 
> with a finished version of the decorator in my sketch.
> 
> Or, since users could do it themselves, this would mean that WSGI2 
> deployment wouldn't be dependent on all server implementers immediately 
> turning out their own WSGI2 implementations.

This, if you'll pardon my language, is bloody awesome.  :D  That would 
strongly drive adoption of WSGI2.  Note that adapting a WSGI1 
application to WSGI2 server would likewise be very handy, and I 
suspect, even easier to implement.

	- Alice.



From alice at gothcandy.com  Sun Jan  9 14:29:34 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 05:29:34 -0800
Subject: [Web-SIG] [PEP 444] Future- and Generator-Based Async Idea
References: <ig9hl3$ads$1@dough.gmane.org>
Message-ID: <igcd7u$itt$1@dough.gmane.org>

Here's what I've mutated Alex Gr?nholm's minimal middleware example 
into: (see the change history for the evolution of this)

	https://gist.github.com/771398

A complete functional (as in function, not working ;) async-capable 
middleware layer (that does nothing) is 12 lines.  That, I think is a 
reasonable amount of boilerplate.  Also, no decorators needed.  It's 
quite readable, even the way I've compressed it.

The class-based version is basically identical, but with added comments 
explaining the assumptions this example makes and demonstrating where 
the acutal middleware code can be implemented for simple middleware.

	- Alice.



From exarkun at twistedmatrix.com  Sun Jan  9 16:04:49 2011
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sun, 09 Jan 2011 15:04:49 -0000
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igc6je$nhn$1@dough.gmane.org>
References: <20110109033446.911C93A4119@sparrow.telecommunity.com>
	<igc6je$nhn$1@dough.gmane.org>
Message-ID: <20110109150449.1889.786099218.divmod.xquotient.27@localhost.localdomain>

On 11:36 am, alice at gothcandy.com wrote:
>On 2011-01-08 19:34:41 -0800, P.J. Eby said:
>>At 04:40 AM 1/9/2011 +0200, Alex Gr?nholm wrote:
>>>09.01.2011 04:15, Alice Bevan?McGregor kirjoitti:
>>>>I hope that clearly identifies my idea on the subject. Since 
>>>>async>>servers will /already/ be implementing their own executors, I 
>>>>don't>>see this as too crazy.
>>>-1 on this. Those executors are meant for executing code in a 
>>>thread>pool. Mandating a magical socket operation filter here 
>>>would>considerably complicate server implementation.
>>
>>Actually, the *reverse* is true.  If you do it the way Alice proposes, 
>>my sketches don't get any more complex, because the filtering goes in 
>>the executor facade or submit function.
>
>Indeed; the executor is what then adds the file descriptor to the 
>underlying server async reactor (select/epoll/kqueue/other).  In the 
>case of the Marrow server, this would utilize a reactor callback (some 
>might say "deferred") to

Don't say it if it's not true.  Deferreds aren't tied to a reactor, and 
Marrow doesn't appear to have anything called "deferred".  So this 
parallel to Twisted's Deferred is misleading and confusing.
>
>Since each async server will either implement or utilize a specific 
>async framework, each will offer its own "async-supported" featureset. 
>What I mean is that all servers should make wsgi.input calls async- 
>able, some would go further to make all socket calls async.  Some might 
>go even further than that and define an API for external libraries 
>(e.g. DBs) to be truly cooperatively async.

I think this effort would benefit from more thought on how exactly 
accessing this external library support will work.  If async wsgi is 
limited to performing a single read asynchronously, then it hardly seems 
compelling.

Jean-Paul

From alice at gothcandy.com  Sun Jan  9 16:57:01 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 07:57:01 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<4D292029.2080102@nextday.fi>
	<20110109033446.911C93A4119@sparrow.telecommunity.com>
	<igc6je$nhn$1@dough.gmane.org>
	<20110109150449.1889.786099218.divmod.xquotient.27@localhost.localdomain>
Message-ID: <igclsd$ng2$1@dough.gmane.org>

On 2011-01-09 07:04:49 -0800, 
exarkun at twistedmatrix.com said:

> Don't say it if it's not true.  Deferreds aren't tied to a reactor, and 
> Marrow doesn't appear to have anything called "deferred".  So this 
> parallel to Twisted's Deferred is misleading and confusing.

It was merely a comparison to the "you schedule something, attach some 
callbacks to it, and when it's finished your callbacks get executed" 
feature.  I did not mention Twisted; also:

:: defer - postpone: hold back to a later time; "let's postpone the exam"

:: deferred - postponed: put off until a later time; "surgery has been 
postponed"

Futures are very similar to deferreds with the one difference you 
mention: future instances are created by the executor/reactor and are 
(possibly) the internal representation instead of Twisted treating the 
Deferred as the executor in terms of registering calls.  In most other 
ways, they share the same goals, and similar methods, even.

Marrow's "deferred calls" code is buried in marrow.io, with IOStreams 
accepting callbacks as part of the standard read/write calls and 
registering these internally.  IOStream then performs read/writes 
across the raw sockets utilizing callbacks from the IOLoop reactor.  
When an IOStream meets its criteria (e.g. written all of the requested 
data, read a number of bytes >= the requested count, or read until a 
marker has appeared in the stream, e.g. \r\n) IOLoop then executes the 
callbacks registered with it, passing the data, if any.

I will likely expand this to include additional criteria and callback hooks.

IOStream, in this way, acts more like Twisted Deferreds than Futures.

> I think this effort would benefit from more thought on how exactly 
> accessing this external library support will work.  If async wsgi is 
> limited to performing a single read asynchronously, then it hardly 
> seems compelling.

There appears to be a misunderstanding over how futures work.  Please 
read PEP 3148 [1] carefully.  While there's not much there, here's the 
gist: the executor schedules the callable passed to submit.  If the 
"worker pool" is full, the underlying pooling mechanism will delay 
execution of the callable until a slot is freed.  Pool and slot are 
defined, by example only, as thread or process pools, but are not 
restricted to such.

(There are three relevant classes defined by concurrent.futures: 
Executor, ProcessPoolExecutor, and ThreadPoolExecutor.  Again, as long 
as you implement the Executor duck-typed interface, you're good to go 
and compliant to PEP 3148, regardless of underlying mechanics.)

If a "slot" is available at the moment of submission, the callable has 
a reasonable expectation of being immediately executed.  The 
future.result() method merely blocks awaiting completion of the already 
running, not yet running, or already completed future.  If already 
completed (a la the future sent back up to to the application after 
yielding it) the call to result is non-blocking / immediate.

Yielding the future is simply a way of safely "blocking" (usually done 
by calling .result() before the future is complete), not some absolute 
requirement for the future itself to run.  The future (and thus async 
socket calls et. al.) can, and should, be scheduled with the underlying 
async reactor in the call to submit().

	- Alice.

[1] http://www.python.org/dev/peps/pep-3148/



From alice at gothcandy.com  Sun Jan  9 17:28:39 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 08:28:39 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<4D292029.2080102@nextday.fi>
	<20110109033446.911C93A4119@sparrow.telecommunity.com>
	<igc6je$nhn$1@dough.gmane.org>
	<20110109150449.1889.786099218.divmod.xquotient.27@localhost.localdomain>
Message-ID: <igcnnn$vlq$1@dough.gmane.org>

On 2011-01-09 07:04:49 -0800, 
exarkun at twistedmatrix.com said:
> I think this effort would benefit from more thought on how exactly 
> accessing this external library support will work.  If async wsgi is 
> limited to performing a single read asynchronously, then it hardly 
> seems compelling.

Apologies if the last e-mail was too harsh; I'm about to go to bed, and 
it' been a long night/morning.  ;)

Here's a proposed solution: a generator API on top of futures.

If the async server implementing the executor can detect a generator 
being submitted, then:

:: The executor accepts the generator and begins iteration (passing the 
executor and the arguments supplied to submit).

:: The generator is expected to be /fast/.

:: The generator does work until it needs an operation over a file 
descriptor, at which point it yields the fd and the operation (say, 
'r', or 'w').

:: The executor schedules with the async reactor the generator to be 
re-called when the operation is possible.

:: The Future is considered complete when the generator raises 
GeneratorExit and the first argument is used as the return value of the 
Future.

Yielding a 2-tuple of readers/writers would work, too, and allow for 
more concurrent utilization of sockets, though I'm not sure of the use 
cases for this.  If so, the generator would be woken up when any of the 
readers or writers are available and sent() a 2-tuple of 
available_readers, available_writers.

The executor is passed along for any operations the generator can not 
accomplish safely without threads, and the executor, as it's running 
through the generator, will accomplish the same semantics as iterating 
the WSGI application: if a future instance is yielded, the generator is 
suspended until the future is complete, allowing heavy processing to be 
mixed with async calls in a fully async server.

The wsgi.input operations can be implemented this way, as can database 
operations and pretty much anything that uses sockets, pipes, or 
on-disk files.  In fact, the WSGI application -itself- could be called 
in this way (with the omission of the executor or a simple wrapper that 
saves the executor into the environ).

Just a quick thought before running off to bed.

	- Alice.



From pje at telecommunity.com  Sun Jan  9 18:03:38 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 09 Jan 2011 12:03:38 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <4D29343B.9000403@nextday.fi>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
Message-ID: <20110109170346.10F553A4065@sparrow.telecommunity.com>

At 06:06 AM 1/9/2011 +0200, Alex Gr?nholm wrote:
>A new feature here is that the application itself yields a (status, 
>headers) tuple and then chunks of the body (or futures).

Hm.  I'm not sure if I like that.  The typical app developer really 
shouldn't be yielding multiple body strings in the first place.  I 
much prefer that the canonical example of a WSGI app just return a 
list with a single bytestring -- preferably in a single statement for 
the entire return operation, whether it's a yield or a return.

IOW, I want it to look like the normal way to do thing is to just 
return the whole request at once, and use the additional difficulty 
of creating a second iterator to discourage people writing iterated 
bodies when they should just write everything to a BytesIO and be done with it.

Also, it makes middleware simpler: the last line can just yield the 
result of calling the app, or a modified version, i.e.:

     yield app(environ)

or:

     s, h, b = app(environ)
     # ... modify or replace s, h, b
     yield s, h, b

In your approach, the above samples have to be rewritten as:

     return app(environ)

or:

     result = app(environ)
     s, h = yield result
     # ... modify or replace s, h
     yield s, h

     for data in result:
          # modify b as we go
          yield result

Only that last bit doesn't actually work, because you have to be able 
to send future results back *into* the result.  Try actually making 
some code that runs on this protocol and yields to futures during the 
body iteration.

Really, this modified protocol can't work with a full async API the 
way my coroutine-based version does, AND the middleware is much more 
complicated.  In my version, your do-nothing middleware looks like this:


class NullMiddleware(object):
     def __init__(self, app):
         self.app = app

     def __call__(environ):
         # ACTION: pre-application environ mangling

         s, h, body = yield self.app(environ)

         # modify or replace s, h, body here

         yield s, h, body


If you want to actually process the body in some way, it looks like:

class NullMiddleware(object):

     def __init__(self, app):
         self.app = app

     def __call__(environ):
         # ACTION: pre-application environ mangling

         s, h, body = yield self.app(environ)

         # modify or replace s, h, body here

         yield s, h, self.process(body)

     def process(self, body_iter):
         while True:
             chunk = yield body_iter
             if chunk is None:
                 break
             # process/modify chunk here
             yield chunk

And that's still a lot simpler than your sketch.

Personally, I would write both of the above as:

     def null_middleware(app):

         def wrapped(environ):
             # ACTION: pre-application environ mangling
             s, h, body = yield app(environ)

             # modify or replace s, h, body here
             yield s, h, process(body)

         def process(body_iter):
             while True:
                 chunk = yield body_iter
                 if chunk is None:
                     break
                 # process/modify chunk here
                 yield chunk

         return wrapped

But that's just personal taste.  Even as a class, it's much easier to 
write.  The above middleware pattern works with the sketches I gave 
on the PEAK wiki, and I've now updated the wiki to include an example 
app and middleware for clarity.

Really, the only hole in this approach is dealing with applications 
that block.  The elephant in the room here is that while it's easy to 
write these example applications so they don't block, in practice 
people read files and do database queries and whatnot in their 
requests, and those APIs are generally synchronous.  So, unless they 
somehow fold their entire application into a future, it doesn't work.


>I liked the idea of having a separate async_read() method in 
>wsgi.input, which would set the underlying socket in nonblocking 
>mode and return a future. The event loop would watch the socket and 
>read data into a buffer and trigger the callback when the given 
>amount of data has been read. Conversely, .read() would set the 
>socket in blocking mode. What kinds of problems would this cause?

That you could never *call* the .read() method outside of a future, 
or else you would block the server, thereby obliterating the point of 
having the async API in the first place.


From pje at telecommunity.com  Sun Jan  9 18:26:19 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 09 Jan 2011 12:26:19 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igc9f7$3qv$1@dough.gmane.org>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<igc9f7$3qv$1@dough.gmane.org>
Message-ID: <20110109172624.EA8413A4065@sparrow.telecommunity.com>

At 04:25 AM 1/9/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-08 13:16:52 -0800, P.J. Eby said:
>
>>In the limit case, it appears that any WSGI 1 server could provide 
>>an (emulated) async WSGI2 implementation, simply by wrapping WSGI2 
>>apps with a finished version of the decorator in my sketch.
>>Or, since users could do it themselves, this would mean that WSGI2 
>>deployment wouldn't be dependent on all server implementers 
>>immediately turning out their own WSGI2 implementations.
>
>This, if you'll pardon my language, is bloody awesome.  :D  That 
>would strongly drive adoption of WSGI2.  Note that adapting a WSGI1 
>application to WSGI2 server would likewise be very handy, and I 
>suspect, even easier to implement.

I very much doubt that.  You'd need greenlets or a thread with a 
communication channel in order to support WSGI 1 apps that use write() calls.

By the way, I don't really see the point of the new sketches you're 
doing, as they aren't nearly as general as the one I've already done, 
but still have the same fundamental limitation: wsgi.input.

If wsgi.input offers any synchronous methods, then they must be used 
from a future and must somehow raise an error when called from within 
the application -- otherwise it would block, nullifying the point of 
having a generator-based API.

If it offers only asynchronous methods, OTOH, then you can't pass 
wsgi.input to any existing libraries (e.g. the cgi module).

The latter problem is the worse one, because it means that the 
translation of an app between my original WSGI2 API and the current 
sketch is no longer just "replace 'return' with 'yield'".

The only way this would work is if WSGI applications are still 
allowed to be written in a blocking style.  Greenlet-based frameworks 
would have no problem with this, of course, but servers like Twisted 
would still have to run WSGI apps in a worker thread pool, just 
because they *might* block.

If we're okay with this as a limitation, then adding _async method 
variants that return futures might work, and we can proceed from there.

Mostly, though, it seems to me that the need to be able to write 
blocking code does away with most of the benefit of trying to have a 
single API in the first place.  Either everyone ends up putting their 
whole app into a future, or else the server has to accept that the 
app could block... and put it into a future for them.  ;-)

So, the former case will be unacceptable to app developers who don't 
feel a need for async code, and the latter doesn't seem to offer 
anything to the developers of non-blocking servers.

(The exception to these conditions, of course, are greenlet-based 
servers, but they can run WSGI *1* apps in a non-blocking way, and so 
have no need for a new protocol.)


From alex.gronholm at nextday.fi  Sun Jan  9 19:09:28 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sun, 09 Jan 2011 20:09:28 +0200
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <20110109170346.10F553A4065@sparrow.telecommunity.com>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
Message-ID: <4D29F9D8.6030203@nextday.fi>

09.01.2011 19:03, P.J. Eby kirjoitti:
> At 06:06 AM 1/9/2011 +0200, Alex Gr?nholm wrote:
>> A new feature here is that the application itself yields a (status, 
>> headers) tuple and then chunks of the body (or futures).
>
> Hm.  I'm not sure if I like that.  The typical app developer really 
> shouldn't be yielding multiple body strings in the first place.  I 
> much prefer that the canonical example of a WSGI app just return a 
> list with a single bytestring -- preferably in a single statement for 
> the entire return operation, whether it's a yield or a return.
Uh, so don't yield multiple body strings then? How is that so difficult?
>
>
> IOW, I want it to look like the normal way to do thing is to just 
> return the whole request at once, and use the additional difficulty of 
> creating a second iterator to discourage people writing iterated 
> bodies when they should just write everything to a BytesIO and be done 
> with it.
I fail to understand why a second iterator is necessary when we can get 
away with just one.
>
>
> Also, it makes middleware simpler: the last line can just yield the 
> result of calling the app, or a modified version, i.e.:
>
>     yield app(environ)
>
> or:
>
>     s, h, b = app(environ)
>     # ... modify or replace s, h, b
>     yield s, h, b
Asynchronous applications may not be ready to send the status line as 
the first thing coming out of the generator. Consider an app that 
receives a file. The first thing coming out of the app is a future. The 
app needs to receive the entire file until it can determine what status 
line to send. Maybe there was an I/O error writing the file, so it needs 
to send a 500 response instead of 200. This is not possible with a body 
iterator, and if we are already iterating the application generator, I 
really don't understand why the body needs to be an iterator as well.
>
>
> In your approach, the above samples have to be rewritten as:
>
>     return app(environ)
>
> or:
>
>     result = app(environ)
>     s, h = yield result
>     # ... modify or replace s, h
>     yield s, h
>
>     for data in result:
>          # modify b as we go
>          yield result
>
> Only that last bit doesn't actually work, because you have to be able 
> to send future results back *into* the result.  Try actually making 
> some code that runs on this protocol and yields to futures during the 
> body iteration.
Did you miss the gist posted by myself (and improved by Alice)?
>
> Really, this modified protocol can't work with a full async API the 
> way my coroutine-based version does, AND the middleware is much more 
> complicated.  In my version, your do-nothing middleware looks like this:
>
>
> class NullMiddleware(object):
>     def __init__(self, app):
>         self.app = app
>
>     def __call__(environ):
>         # ACTION: pre-application environ mangling
>
>         s, h, body = yield self.app(environ)
>
>         # modify or replace s, h, body here
>
>         yield s, h, body
>
>
> If you want to actually process the body in some way, it looks like:
>
> class NullMiddleware(object):
>
>     def __init__(self, app):
>         self.app = app
>
>     def __call__(environ):
>         # ACTION: pre-application environ mangling
>
>         s, h, body = yield self.app(environ)
>
>         # modify or replace s, h, body here
>
>         yield s, h, self.process(body)
>
>     def process(self, body_iter):
>         while True:
>             chunk = yield body_iter
>             if chunk is None:
>                 break
>             # process/modify chunk here
>             yield chunk
>
> And that's still a lot simpler than your sketch.
>
> Personally, I would write both of the above as:
>
>     def null_middleware(app):
>
>         def wrapped(environ):
>             # ACTION: pre-application environ mangling
>             s, h, body = yield app(environ)
>
>             # modify or replace s, h, body here
>             yield s, h, process(body)
>
>         def process(body_iter):
>             while True:
>                 chunk = yield body_iter
>                 if chunk is None:
>                     break
>                 # process/modify chunk here
>                 yield chunk
>
>         return wrapped
>
> But that's just personal taste.  Even as a class, it's much easier to 
> write.  The above middleware pattern works with the sketches I gave on 
> the PEAK wiki, and I've now updated the wiki to include an example app 
> and middleware for clarity.
>
> Really, the only hole in this approach is dealing with applications 
> that block.  The elephant in the room here is that while it's easy to 
> write these example applications so they don't block, in practice 
> people read files and do database queries and whatnot in their 
> requests, and those APIs are generally synchronous.  So, unless they 
> somehow fold their entire application into a future, it doesn't work.
>
>
>> I liked the idea of having a separate async_read() method in 
>> wsgi.input, which would set the underlying socket in nonblocking mode 
>> and return a future. The event loop would watch the socket and read 
>> data into a buffer and trigger the callback when the given amount of 
>> data has been read. Conversely, .read() would set the socket in 
>> blocking mode. What kinds of problems would this cause?
>
> That you could never *call* the .read() method outside of a future, or 
> else you would block the server, thereby obliterating the point of 
> having the async API in the first place.
>
Outside of the application/middleware you mean? I hope there isn't any 
more confusion left about what a future is. The fact is that you cannot 
use synchronous API calls directly from an async app no matter what. 
Some workaround is always necessary.



From pje at telecommunity.com  Sun Jan  9 21:56:26 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 09 Jan 2011 15:56:26 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <4D29F9D8.6030203@nextday.fi>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
	<4D29F9D8.6030203@nextday.fi>
Message-ID: <20110109205633.CB4AB3A4065@sparrow.telecommunity.com>

At 08:09 PM 1/9/2011 +0200, Alex Gr?nholm wrote:
>Asynchronous applications may not be ready to send the status line 
>as the first thing coming out of the generator.

So?  In the sketches that are the subject of this thread, it doesn't 
have to be the first thing.  If the application yields a future 
first, it will be paused...  and so will the middleware.  When this 
line is executed in the middleware:

         status, headers, body = yield app(environ)

...the middleware is paused until the application actually yields its 
response tuple.

Specifically, this yield causes the app iterator to be pushed on the 
Coroutine object's .stack attribute, then iterated.  If the 
application yields a future, the server suspends the whole thing 
until it gets called back, at which point it .send()s the result back 
into the app iterator.

The app iterator then yields its response, which is tagged as a 
return value, so the app is popped off the .stack, and the response 
is sent via .send() into the middleware, which then proceeds as if 
nothing happened in the meantime.  It then yields *its* response, and 
whatever body iterator is given gets put into a second coroutine that 
proceeds similarly.

When the process_response() part of the middleware does a "yield 
body_iter", the body iterator is pushed, and the middleware is paused 
until the body iterator yields a chunk.  If the body yields a future, 
the whole process is suspended and resumed.  The middleware won't be 
resumed until the body yields another chunk, at which point it is 
resumed.  If it yields a chunk of its own, then that's passed up to 
any response-processing middleware further up the stack.

In contrast, middleware based on the 2+body protocol cannot process a 
body without embedding coroutine management into the middleware 
itself.   For example, you can't write a standalone body processor 
function, and reuse it inside of two pieces of middleware, without 
doing a bunch of send()/throw() logic to make it work.


>Outside of the application/middleware you mean? I hope there isn't 
>any more confusion left about what a future is. The fact is that you 
>cannot use synchronous API calls directly from an async app no 
>matter what. Some workaround is always necessary.

Which pretty much kills the whole idea as being a single, universal 
WSGI protocol, since most people don't care about async.


From alex.gronholm at nextday.fi  Sun Jan  9 22:15:41 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sun, 09 Jan 2011 23:15:41 +0200
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <20110109205633.CB4AB3A4065@sparrow.telecommunity.com>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
	<4D29F9D8.6030203@nextday.fi>
	<20110109205633.CB4AB3A4065@sparrow.telecommunity.com>
Message-ID: <4D2A257D.5000309@nextday.fi>

09.01.2011 22:56, P.J. Eby kirjoitti:
> At 08:09 PM 1/9/2011 +0200, Alex Gr?nholm wrote:
>> Asynchronous applications may not be ready to send the status line as 
>> the first thing coming out of the generator.
>
> So?  In the sketches that are the subject of this thread, it doesn't 
> have to be the first thing.  If the application yields a future first, 
> it will be paused...  and so will the middleware.  When this line is 
> executed in the middleware:
>
>         status, headers, body = yield app(environ)
>
> ...the middleware is paused until the application actually yields its 
> response tuple.
>
> Specifically, this yield causes the app iterator to be pushed on the 
> Coroutine object's .stack attribute, then iterated.  If the 
> application yields a future, the server suspends the whole thing until 
> it gets called back, at which point it .send()s the result back into 
> the app iterator.
>
> The app iterator then yields its response, which is tagged as a return 
> value, so the app is popped off the .stack, and the response is sent 
> via .send() into the middleware, which then proceeds as if nothing 
> happened in the meantime.  It then yields *its* response, and whatever 
> body iterator is given gets put into a second coroutine that proceeds 
> similarly.
>
> When the process_response() part of the middleware does a "yield 
> body_iter", the body iterator is pushed, and the middleware is paused 
> until the body iterator yields a chunk.  If the body yields a future, 
> the whole process is suspended and resumed.  The middleware won't be 
> resumed until the body yields another chunk, at which point it is 
> resumed.  If it yields a chunk of its own, then that's passed up to 
> any response-processing middleware further up the stack.
>
> In contrast, middleware based on the 2+body protocol cannot process a 
> body without embedding coroutine management into the middleware 
> itself.   For example, you can't write a standalone body processor 
> function, and reuse it inside of two pieces of middleware, without 
> doing a bunch of send()/throw() logic to make it work.
Some boilerplate code was necessary in WSGI 1 middleware too. Alice's 
cleaned up example didn't look too bad, and it would not require that 
Coroutine stack at all.

I think that at this point both sides need to present some code that 
really works, and those implementations could then be compared. The 
examples so far have been a bit too abstract to be fairly evaluated.
>
>
>> Outside of the application/middleware you mean? I hope there isn't 
>> any more confusion left about what a future is. The fact is that you 
>> cannot use synchronous API calls directly from an async app no matter 
>> what. Some workaround is always necessary.
>
> Which pretty much kills the whole idea as being a single, universal 
> WSGI protocol, since most people don't care about async.
I'm confused. Did you not know this? If so, why then were you at least 
initially receptive to the idea?
Personally I don't think that this is a big problem. Async apps will 
always have to take care not to block the reactor unreasonably long, and 
that is never going to change. Synchronous apps just need to follow the 
protocol, but beyond that they shouldn't have to care about the async 
side of things.
>
>


From alice at gothcandy.com  Mon Jan 10 01:39:03 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 16:39:03 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<igc9f7$3qv$1@dough.gmane.org>
	<20110109172624.EA8413A4065@sparrow.telecommunity.com>
Message-ID: <igdkf7$u3b$1@dough.gmane.org>

On 2011-01-09 09:26:19 -0800, P.J. Eby said:
> By the way, I don't really see the point of the new sketches you're doing...

I'm sorry.

> ...as they aren't nearly as general as the one I've already done, but 
> still have the same fundamental limitation: wsgi.input.

You missed the point entirely, then.

> If wsgi.input offers any synchronous methods...

Regardless of whether or not wsgi.input is implemented in an async way, 
wrap it in a future and eventually get around to yielding it.  Problem 
/solved/.  Identical APIs for both sync and async, and if you have an 
async server but haven't gotten around to implementing your own 
executor yet, wrapping the blocking read call in a future also solves 
the problem (albeit not in the most efficient way).

I.e. wrap every call to a wsgi.input method by passing it to wsgi.submit.

> ...then they must be used from a future and must some how raise an 
> error when called from within the application -- otherwise it would 
> block, nullifying the point ofhaving a generator-based API.

See above.  No extra errors, nothing really that insane.

> If it offers only asynchronous methods, OTOH, then you can't pass 
> wsgi.input to any existing libraries (e.g. the cgi module).

Describe to me how a function can be suspended (other than magical 
greenthreads) if it does not yield; if I knew this, maybe I wouldn't be 
so confused.

> The latter problem is the worse one, because it means that the 
> translation of an app between my original WSGI2 API and the current 
> sketch is no longer just "replace 'return' with 'yield'".

I've deviated from your sketch, obviously, and any semblance of 
yielding a 3-tuple.  Stop thinking of my example code as conforming to 
your ideas; it's a new idea, or, worst case, a narrowing of an idea 
into its simplest form.

> The only way this would work is if WSGI applications are still allowed 
> to be written in a blocking style.  Greenlet-based frameworks would 
> have no problem with this, of course, but servers like Twisted would 
> still have to run WSGI apps in a worker thread pool, just because they 
> *might* block.

Then that is not acceptable and "would not work".  The mechanics of 
yielding futures instances allows you to (in your server) implement the 
necessary async code however you wish while providing a uniform 
interface to both sync and async applications running on sync and async 
servers.  In fact, you would be able to safely run a sync application 
on an async server and vice-versa.  You can, on an async server:

:: Add a callback to the yielded future to re-schedule the application 
generator.

:: If using greenthreads, just block on future.result() then 
immediately wake up the application generator.

:: Do other things I can't think of because I'm still waking up.

The first solution is how Marrow HTTPd would operate.

> If we're okay with this as a limitation, then adding _async method 
> variants that return futures might work, and we can proceed from there.

That is not optimum, because now you have an optional API that 
applications who want to be compatible will need to detect and choose 
between.

> Mostly, though, it seems to me that the need to be able to write 
> blocking code does away with most of the benefit of trying to have a 
> single API in the first place.

You have artificially created this need, ignoring the semantics of 
using the server-specific executor to detect async-capable requests and 
the yield mechanics I suggested; which happens to be a single, coherent 
API across sync and async servers and applications.

	- Alice.



From alice at gothcandy.com  Mon Jan 10 02:06:28 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 17:06:28 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
Message-ID: <igdm2k$3oh$1@dough.gmane.org>

On 2011-01-09 09:03:38 -0800, P.J. Eby said:
> Hm.  I'm not sure if I like that.  The typical app developer really 
> shouldn't be yielding multiple body strings in the first place.

Wait; what?  So you want the app developer to load a 40MB talkcast MP3 
into memory before sending it?  You want to completely eliminate the 
ability to stream an HTML page to the client in chunks (e.g. <head> 
block, headers + search box, search results, advertisements, footer -- 
the exact thing Google does with every search result)?  That sounds 
like artificially restricting application developers, to me.

> I much prefer that the canonical example of a WSGI app just return a 
> list with a single bytestring...

Why is it wrapped in a list, then?

> IOW, I want it to look like the normal way to do thing is to just 
> return the whole request at once, and use the additional difficulty of 
> creating a second iterator to discourage people writing iterated bodies 
> when they should just write everything to a BytesIO and be done with it.

It sounds to me like your "should" doesn't cover an extremely large 
range of common use cases.

> In your approach, the above samples have to be rewritten as:
> 
>      return app(environ)
> 
> [snip]

My code does not use return.  At all.  Only yield.

> Try actually making some code that runs on this protocol and yields to 
> futures during the body iteration.

Sure.  I'll also implement my actual proposal of not having a separate 
body iterable.

> The above middleware pattern works with the sketches I gaveon the PEAK 
> wiki, and I've now updated the wiki to include an exampleapp and 
> middleware for clarity.

I'll need to re-read the code on your wiki; I find it incredibly 
difficult to grok, however, you can help me out a bit by answering a 
few questions about it: How does middleware trap exceptions raised by 
the application.  (Specifically how does the server pass the buck with 
exceptions?  And how does the exception get to the application to 
bubble out towards the server, through middleware, as it does now?)

> Really, the only hole in this approach is dealing with applications that block.

That's what the executor in the environ is for.  If you have image 
scaling or something else that will block you submit it.  All 
networking calls?  You submit them.

> The elephant in the room here is that while it's easy towrite these 
> example applications so they don't block, in practicepeople read files 
> and do database queries and what not in their requests, and those APIs 
> are generally synchronous.  So, unless they somehow fold their entire 
> application into a future, it doesn't work.

Actually, that's how multithreading support in marrow.server[.http] was 
implemented.  Overhead?  40-60 RSecs.  The option is provided for those 
who can do nothing about their application blocking, while still 
maintaining the internally async nature of the server.

> That you could never *call* the .read() method outside of a future,or 
> else you would block the server, thereby obliterating the point 
> ofhaving the async API in the first place.

See above re: your confusion over the calling semantics of wsgi.input 
in regards to my (and Alex's) proposal.  Specifically:

    data = (yield submit(wsgi_input.read, 4096)).result()

This would work on sync and async servers, and with sync and async 
applications, with no difference in the code.

	- Alice.



From alice at gothcandy.com  Mon Jan 10 02:09:36 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Sun, 9 Jan 2011 17:09:36 -0800
Subject: [Web-SIG] Server-side async API implementation sketches
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
	<igdm2k$3oh$1@dough.gmane.org>
Message-ID: <igdm8f$4e9$1@dough.gmane.org>

On 2011-01-09 17:06:28 -0800, Alice Bevan-McGregor said:
> On 2011-01-09 09:03:38 -0800, P.J. Eby said:
>> The elephant in the room here is that while it's easy towrite these
>> example applications so they don't block, in practicepeople read files
>> and do database queries and what not in their requests, and those APIs
>> are generally synchronous.  So, unless they somehow fold their entire
>> application into a future, it doesn't work.
> 
> Actually, that's how multithreading support in marrow.server[.http] was
> implemented.  Overhead?  40-60 RSecs.

Clarification here, that's less than 2% of total RSecs.

	- Alice.



From chris.dent at gmail.com  Mon Jan 10 10:48:23 2011
From: chris.dent at gmail.com (chris.dent at gmail.com)
Date: Mon, 10 Jan 2011 09:48:23 +0000 (GMT)
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igdm2k$3oh$1@dough.gmane.org>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
	<igdm2k$3oh$1@dough.gmane.org>
Message-ID: <alpine.OSX.1.10.1101100941070.48387@rust.local>

On Sun, 9 Jan 2011, Alice Bevan?McGregor wrote:

> On 2011-01-09 09:03:38 -0800, P.J. Eby said:
>> Hm.  I'm not sure if I like that.  The typical app developer really 
>> shouldn't be yielding multiple body strings in the first place.
>
> Wait; what?  So you want the app developer to load a 40MB talkcast MP3 into 
> memory before sending it?

My reaction too. I've read this elsewhere on this list too, in other
topics. A general statement that the correct way to make an
efficient WSGI (1) app is to return just one body string.

This runs contrary to everything I've ever understood about making
web apps that appear performant to the user: get the first byte out to
the browser as soon as possible.

This came up in discussions of wanting to have a cascading series of
generators (to save memory and improve responsiveness): store
generates data, serializers generates strings, handler generates
(sends out in chunks) the web page from those strings.

So, this is me saying: I'm in favor of a post-wsgi1 world where apps
are encouraged to be generators. To me they are just as useful in
sync and async contexts.

-- 
Chris Dent                                   http://burningchrome.com/
                                 [...]

From alice at gothcandy.com  Mon Jan 10 13:25:40 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Mon, 10 Jan 2011 04:25:40 -0800
Subject: [Web-SIG] Generator-Based Applications: Marrow HTTPd Example
Message-ID: <igets4$3lt$1@dough.gmane.org>

Howdy!

Here's a rewritten (and incomplete, but GET and HEAD requests work 
fine) marrow.server.http branch [1] that illustrates a simple 
application [2] and protocol implementation [3].  Most notably, examine 
the 'resume' method [4].

The 'basic' example yields a future instance and uses the data as the 
response body.

Note that this particular rewrite is not complete, nor has it been 
profiled and optimized; initial benchmarks (using the 'benchmark' 
example) show a reduction of ~600 RSecs from the 'draft' branch, which 
is substantial, but hasn't been traced to a particular segment of code 
or design decision yet.

The server is now -extremely- easy to read and follow, with all code 
acting in a linear way.  (Application worker threading has been removed 
from this branch as well; the server is once again purely async.)

	- Alice.

[1] https://github.com/pulp/marrow.server.http/tree/generator

[2] https://github.com/pulp/marrow.server.http/blob/generator/examples/basic.py

[3] 
https://github.com/pulp/marrow.server.http/blob/generator/marrow/server/http/protocol.py

[4] 

https://github.com/pulp/marrow.server.http/blob/generator/marrow/server/http/protocol.py#L177-226 




From mdipierro at cs.depaul.edu  Mon Jan 10 13:41:17 2011
From: mdipierro at cs.depaul.edu (Massimo Di Pierro)
Date: Mon, 10 Jan 2011 06:41:17 -0600
Subject: [Web-SIG] Generator-Based Applications: Marrow HTTPd Example
In-Reply-To: <igets4$3lt$1@dough.gmane.org>
References: <igets4$3lt$1@dough.gmane.org>
Message-ID: <DCB4DB01-B74A-42DB-A030-A06DF716EE69@cs.depaul.edu>

I like this a lot!

On Jan 10, 2011, at 6:25 AM, Alice Bevan?McGregor wrote:

> Howdy!
> 
> Here's a rewritten (and incomplete, but GET and HEAD requests work fine) marrow.server.http branch [1] that illustrates a simple application [2] and protocol implementation [3].  Most notably, examine the 'resume' method [4].
> 
> The 'basic' example yields a future instance and uses the data as the response body.
> 
> Note that this particular rewrite is not complete, nor has it been profiled and optimized; initial benchmarks (using the 'benchmark' example) show a reduction of ~600 RSecs from the 'draft' branch, which is substantial, but hasn't been traced to a particular segment of code or design decision yet.
> 
> The server is now -extremely- easy to read and follow, with all code acting in a linear way.  (Application worker threading has been removed from this branch as well; the server is once again purely async.)
> 
> 	- Alice.
> 
> [1] https://github.com/pulp/marrow.server.http/tree/generator
> 
> [2] https://github.com/pulp/marrow.server.http/blob/generator/examples/basic.py
> 
> [3] https://github.com/pulp/marrow.server.http/blob/generator/marrow/server/http/protocol.py
> 
> [4] 
> https://github.com/pulp/marrow.server.http/blob/generator/marrow/server/http/protocol.py#L177-226 
> 
> 
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/mdipierro%40cti.depaul.edu


From foom at fuhm.net  Mon Jan 10 15:22:12 2011
From: foom at fuhm.net (James Y Knight)
Date: Mon, 10 Jan 2011 09:22:12 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <alpine.OSX.1.10.1101100941070.48387@rust.local>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
	<igdm2k$3oh$1@dough.gmane.org>
	<alpine.OSX.1.10.1101100941070.48387@rust.local>
Message-ID: <45EDE0B9-AA0E-40A5-8640-66E7DBFDCECF@fuhm.net>


On Jan 10, 2011, at 4:48 AM, chris.dent at gmail.com wrote:

> My reaction too. I've read this elsewhere on this list too, in other
> topics. A general statement that the correct way to make an
> efficient WSGI (1) app is to return just one body string.
> 
> This runs contrary to everything I've ever understood about making
> web apps that appear performant to the user: get the first byte out to
> the browser as soon as possible.

Weellll. You want to get the earliest byte *which is required to display the page* out as soon as possible. The browser usually has to parse a whole lot of the response before it starts displaying anything useful.

And in order to do that, you really want to minimize the number of round-trip-times, which is heavily dependent upon the number of packets sent (not the amount of data!), when the data is small. Using a generator in WSGI forces the server to push out partial data as soon as possible, so it could end up using many more packets than if you buffered everything and sent it at once, and thus, will be slower.

As the buffering and streaming section of WSGI1 already says...:
> Generally speaking, applications will achieve the best throughput by buffering their (modestly-sized) output and sending it all at once. This is a common approach in existing frameworks such as Zope: the output is buffered in a StringIO or similar object, then transmitted all at once, along with the response headers.
> 
> [...]
> 
> For large files, however, or for specialized uses of HTTP streaming (such as multipart "server push"), an application may need to provide output in smaller blocks (e.g. to avoid loading a large file into memory). It's also sometimes the case that part of a response may be time-consuming to produce, but it would be useful to send ahead the portion of the response that precedes it.

James

From pje at telecommunity.com  Mon Jan 10 15:43:54 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 10 Jan 2011 09:43:54 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igdkf7$u3b$1@dough.gmane.org>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<igc9f7$3qv$1@dough.gmane.org>
	<20110109172624.EA8413A4065@sparrow.telecommunity.com>
	<igdkf7$u3b$1@dough.gmane.org>
Message-ID: <20110110144409.847B93A4065@sparrow.telecommunity.com>

At 04:39 PM 1/9/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-09 09:26:19 -0800, P.J. Eby said:
>>If wsgi.input offers any synchronous methods...
>
>Regardless of whether or not wsgi.input is implemented in an async 
>way, wrap it in a future and eventually get around to yielding 
>it.  Problem /solved/.

Not the API problem.  If I'm accustomed to writing synchronous code, 
the async version looks ridiculous.  Also, an existing WSGI web 
framework isn't going to be able to be ported to this API without 
putting it in a future.

My hope was for an API that would be a simple enough translation that 
*everybody* could be persuaded to use it, but having to use futures 
just to write a "normal" application simply isn't going to work for 
the core WSGI API.  As a separate "WSGI-A" profile, sure, it works fine.


>>If it offers only asynchronous methods, OTOH, then you can't pass 
>>wsgi.input to any existing libraries (e.g. the cgi module).
>
>Describe to me how a function can be suspended (other than magical 
>greenthreads) if it does not yield; if I knew this, maybe I wouldn't 
>be so confused.

I'm not sure what you're confused about.  I'm the one who forgot you 
have to read from wsgi.input in a blocking way to write a normal app.  ;-)

(Mainly, because I was so excited about the potential in your 
sketched API, and I got sucked into the process of implementing/improving it.)


>I've deviated from your sketch, obviously, and any semblance of 
>yielding a 3-tuple.  Stop thinking of my example code as conforming 
>to your ideas; it's a new idea, or, worst case, a narrowing of an 
>idea into its simplest form.

What I'm trying to point out is that you've missed two important API 
enhancements in my sketch, that make it so that app and middleware 
authors don't have to explicitly manage any generator methods or even 
future methods.


>  The mechanics of yielding futures instances allows you to (in your 
> server) implement the necessary async code however you wish while 
> providing a uniform interface to both sync and async applications 
> running on sync and async servers.  In fact, you would be able to 
> safely run a sync application on an async server and 
> vice-versa.  You can, on an async server:
>
>:: Add a callback to the yielded future to re-schedule the 
>application generator.
>
>:: If using greenthreads, just block on future.result() then 
>immediately wake up the application generator.
>
>:: Do other things I can't think of because I'm still waking up.

I am not sure why you're reiterating these things.  The sample code I 
posted shows precisely where you'd *do* them in a sync or async 
server.  That's not where the problem lies.


>That is not optimum, because now you have an optional API that 
>applications who want to be compatible will need to detect and choose between.

It wasn't supposed to be optional, but it's beside the point since 
the presence of a blocking API means the application can block.

The issue might be addressable by having an environment key like, 
'wsgi.canblock' (indicating whether the application is already in a 
separate thread/process), and a piece of middleware that simply 
spawns its child app to a future if wsgi.canblock isn't set.  Then 
people who write blocking applications could use the decorator.



>>Mostly, though, it seems to me that the need to be able to write 
>>blocking code does away with most of the benefit of trying to have 
>>a single API in the first place.
>
>You have artificially created this need, ignoring the semantics of 
>using the server-specific executor to detect async-capable requests 
>and the yield mechanics I suggested; which happens to be a single, 
>coherent API across sync and async servers and applications.

I haven't ignored them.  I'm simply representing the POV of existing 
WSGI apps and frameworks, which currently block, and are unlikely to 
be rewritten so as not to block.  I thought, briefly, that it was 
possible to make an API with a low-enough conceptual overhead to 
allow that porting to occur, and let my enthusiasm carry me away.

I was wrong, though: even the extremely minimalist version isn't 
going to be usable for ported code, which relegates the async version 
to a niche role.

I would note, though, that this is *still* better than my previous 
position, which was that there was no point making an async API *at all*.  ;-)


From pje at telecommunity.com  Mon Jan 10 16:12:02 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 10 Jan 2011 10:12:02 -0500
Subject: [Web-SIG] Server-side async API implementation sketches
In-Reply-To: <igdm2k$3oh$1@dough.gmane.org>
References: <20110108211658.6DC163A4119@sparrow.telecommunity.com>
	<4D290DE4.1030400@nextday.fi> <igb5n4$54u$1@dough.gmane.org>
	<20110109034644.0454B3A4119@sparrow.telecommunity.com>
	<4D29343B.9000403@nextday.fi>
	<20110109170346.10F553A4065@sparrow.telecommunity.com>
	<igdm2k$3oh$1@dough.gmane.org>
Message-ID: <20110110151207.540E53A4065@sparrow.telecommunity.com>

At 05:06 PM 1/9/2011 -0800, Alice Bevan?McGregor wrote:
>On 2011-01-09 09:03:38 -0800, P.J. Eby said:
>>Hm.  I'm not sure if I like that.  The typical app developer really 
>>shouldn't be yielding multiple body strings in the first place.
>
>Wait; what?  So you want the app developer to load a 40MB talkcast 
>MP3 into memory before sending it?

Statistically speaking, the "typical app" is producing a web page, 
made of HTML and severely limited in size by the short attention span 
of the human user reading it.  ;-)

Obviously, the spec should allow and support streaming.


>   You want to completely eliminate the ability to stream an HTML 
> page to the client in chunks (e.g. <head> block, headers + search 
> box, search results, advertisements, footer -- the exact thing 
> Google does with every search result)?  That sounds like 
> artificially restricting application developers, to me.

First, I don't want to eliminate it.   Second, Google is hardly the 
"typical app developer".  If you need the capability, it'll still be there.



>>In your approach, the above samples have to be rewritten as:
>>      return app(environ)
>>[snip]
>
>My code does not use return.  At all.  Only yield.

If you return the calling of a generator, then you pass the original 
generator through to the caller, and it is the equivalent of writing 
a loop in place that iterates over the subgenerator, only without the 
additional complexity of needing to send/throw.


>>The above middleware pattern works with the sketches I gaveon the 
>>PEAK wiki, and I've now updated the wiki to include an exampleapp 
>>and middleware for clarity.
>
>I'll need to re-read the code on your wiki; I find it incredibly 
>difficult to grok, however, you can help me out a bit by answering a 
>few questions about it: How does middleware trap exceptions raised 
>by the application.

With try/except around the "yield app(environ)" call (main app run), 
or with try/except around the "yield body_iter" call (body iterator run).


>  (Specifically how does the server pass the buck with 
> exceptions?  And how does the exception get to the application to 
> bubble out towards the server, through middleware, as it does now?)

All that is in the Coroutine class, which is a generator-based "green 
thread" implementation.

Remember how you were saying that your sketch would benefit from PEP 380?

The Coroutine class is a pure-Python implementation of PEP 380, minus 
the syntactic sugar.  It turns "yield" into "yield from" whenever the 
value you yield is itself a geniter.

So, if you pretend that "yield app(environ)" and "yield body_iter" 
are actually "yield from"s instead, then the mechanics should become clearer.

Coroutine runs a generator by sending or throwing into it.  It then 
takes the result (either a value or an exception) and decides where 
to send that.  If it's an object with send/throw methods, it pushes 
it on the stack, and passes None into it to start it running, thereby 
"calling" the subgenerator.  If it's an exception or a return value 
(e.g. StopIteration(value=None)), it pops the stack and propagates 
the exception or return value to calling generator.

If it's a future or some other object the server cares about, then 
the server can pause the coroutine (by returning 'routine.PAUSE' when 
the coroutine asks it what to do).

Coroutine accepts a trampoline function and a completion callback as 
parameters: the trampoline function inspects a value yielded by a 
generator and then tells the coroutine whether it should PAUSE, CALL, 
RETURN, RESUME, or RAISE in response to that particular 
yield.  RESUME is used for synchronous replies, where the yield 
returns immediately.  RETURN means pop the current generator off the 
stack and return a value to the calling generator.  RAISE raises an 
error immediately in the top-of-stack generator.  CALL pushes a 
geniter on the stack.

IOW, the Coroutine class lets you write servers with just a little 
glue code to tell it how you want the control to flow.  It's actually 
entirely independent of WSGI or any particular WSGI protocol...  I'm 
thinking that I should probably wrap it up into a PyPI package with 
some docs and tests, though I'm not sure when I'd get around to it.

(Heck, it's the sort of thing that probably ought to be in the stdlib 
-- certainly PEP 380 can be implemented in terms of it.)

Anyway, both the sync and async server examples have trampolines that 
detect futures and process them accordingly.  If you yield to a 
future, you get back its result -- either a value or an exception at 
the point where you yielded it.  You don't have to explicitly call 
.result() (in fact, you *can't*); it's already been called before 
control gets back to the place that yielded it.

IOW, in my sketch, yielding to a future looks like this:

     data = yield submit(wsgi_input.read, 4096)

without the '.result()' on the end.


From guido at python.org  Mon Jan 10 22:12:57 2011
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Jan 2011 13:12:57 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<AANLkTimAGrYFwXc=Eiwxg1J-9mHvgjNV76veMwfsTHka@mail.gmail.com>
	<4D227202.9090902@inteli-com.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
	<20110107155545.8AD763A40A8@sparrow.telecommunity.com>
	<AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>
Message-ID: <AANLkTimU-nD_+jk0-S0N9TMwo2-mVNpq1Ra17FDRuq=s@mail.gmail.com>

Ok, now that we've had a week of back and forth about this, let me
repeat my "threat". Unless more concerns are brought up in the next 24
hours, can PEP 3333 be accepted? It seems a lot of people are waiting
for a decision that enables implementers to go ahead and claim PEP
333[3] compatibility. PEP 444 can take longer.

--Guido

On Fri, Jan 7, 2011 at 4:56 PM, Graham Dumpleton
<graham.dumpleton at gmail.com> wrote:
> On 8 January 2011 02:55, P.J. Eby <pje at telecommunity.com> wrote:
>> At 05:27 PM 1/7/2011 +1100, Graham Dumpleton wrote:
>>>
>>> Another thing though. For output changed to sys.stdout.buffer. For
>>> input should we be using sys.stdin.buffer as well if want bytes?
>>
>> %&$*()&%!!! ?Sorry, still getting used to this whole Python 3 thing.
>> ?(Honestly, I don't even use Python 2.6 for anything real yet.)
>>
>>
>>> Good thing I tried running this. Did we all assume that someone else
>>> was actually running it to check it? :-)
>>
>> Well, I only recently started changing the examples to actual Python 3, vs
>> being the old Python 2 examples. ?Though, I'm not sure anybody ever ran the
>> Python 2 ones. ?;-)
>
> Latest CGI/WSGI bridge example extract from PEP 3333 seems to work
> okay for my simple test.
>
> So, if no more technical problems (vs cosmetic) that anyone else sees,
> that is probably it and and we can toss this baby out the door.
>
> Graham
>



-- 
--Guido van Rossum (python.org/~guido)

From alice at gothcandy.com  Mon Jan 10 22:33:52 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Mon, 10 Jan 2011 13:33:52 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
	<20110107155545.8AD763A40A8@sparrow.telecommunity.com>
	<AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>
	<AANLkTimU-nD_+jk0-S0N9TMwo2-mVNpq1Ra17FDRuq=s@mail.gmail.com>
Message-ID: <igfu00$p4o$1@dough.gmane.org>

On 2011-01-10 13:12:57 -0800, Guido van Rossum said:

> Ok, now that we've had a week of back and forth about this, let me 
> repeat my "threat". Unless more concerns are brought up in the next 24 
> hours, can PEP 3333 be accepted?

+9001 (> 9000)

> It seems a lot of people are waiting for a decision that enables 
> implementers to go ahead and claim PEP
> 333[3] compatibility.

Django, mod_wsgi, CherryPy, etc. all have solutions that would need 
AFIK minor tweaking before going "live", which would make adoption of 
PEP 3333 the fastest of any PEP I've ever seen. ;)

> PEP 444 can take longer.

Indeed it will!  :D

I have the conversion from Textile to ReST about half completed; I'll 
continue to poke it now that mailing list traffic seems to have died 
down and won't be consuming the majority of my Copious Spare Time?.  
ReST just doesn't jive with my neural net.  :/

	- Alice.



From alice at gothcandy.com  Tue Jan 11 00:59:32 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Mon, 10 Jan 2011 15:59:32 -0800
Subject: [Web-SIG] Generator-Based Applications: Marrow HTTPd Example
References: <igets4$3lt$1@dough.gmane.org>
Message-ID: <igg6h3$4o2$1@dough.gmane.org>

On 2011-01-10 04:25:40 -0800, Alice Bevan?McGregor said:

> Note that this particular rewrite is not complete, nor has it been 
> profiled and optimized; initial benchmarks (using the 'benchmark' 
> example) show a reduction of ~600 RSecs from the 'draft' branch, which 
> is substantial, but hasn't been traced to a particular segment of code 
> or design decision yet.

Ignore that number; I had some runaway processes eating up my CPU.  
That's what I get for going weeks or months between reboots.  ;)

The drop (benchmarking current 'draft' branch and 'generator' branch) 
is now ~200 RSecs (down from ~3.2 KRsecs).  Much more reasonable, and 
subject to enough stddev across runs to make the difference negligible 
at best.

*phew*

	- Alice.



From tfarrell at owassobible.org  Tue Jan 11 04:40:45 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Mon, 10 Jan 2011 21:40:45 -0600 (CST)
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <20110107201420.346A43A40A8@sparrow.telecommunity.com>
Message-ID: <1526394.51.1294717245711.JavaMail.root@boaz>

----- Original Message -----
From: "P.J. Eby" <pje at telecommunity.com>
To: "Timothy Farrell" <tfarrell at owassobible.org>, web-sig at python.org
Sent: Friday, January 7, 2011 2:14:20 PM
Subject: Re: [Web-SIG] PEP 444 feature request - Futures executor

> There are some other issues that might need to be addressed, like 
> maybe adding an attribute or two for the level of reliability 
> guaranteed by the executor, or allowing the app to request a given 
> reliability level.  Specifically, it might be important to distinguish between:

> * this will be run exactly once as long as the server doesn't crash
> * this will eventually be run once, even if the server suffers a 
> fatal error between now and then

> IOW, to indicate whether the thing being done is "transactional", so to speak.

I understand why this would be good (credit card transactions particularly), but how would this play our in the real world?  All servers will do their best to run the jobs given them.  

Are you suggesting that there would be a property of the executor that would change based on the load of the server or some other metric?  Say the server has 100 queued jobs and only 2 worker threads, would it then have a way of saying, "I'll get to this eventually, but I'm pretty swamped."?

Is that what you're getting at or something more like database transactions..."I guarantee that I won't stop halfway through this process."

Thanks,
-t

From tfarrell at owassobible.org  Tue Jan 11 15:01:13 2011
From: tfarrell at owassobible.org (Timothy Farrell)
Date: Tue, 11 Jan 2011 08:01:13 -0600 (CST)
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <AANLkTik+ddNMMtZBNXcF3-JxMbJJ7RCcLhocjpAd_6km@mail.gmail.com>
Message-ID: <22224755.4.1294754473033.JavaMail.root@boaz>

Quartz is certainly powerful, but I think it's outside the scope of something we want in a WSGI spec.  Is there a specific feature you're referring to?

----- Original Message -----
From: "Nam Nguyen" <bitsink at gmail.com>
To: "Timothy Farrell" <tfarrell at owassobible.org>
Cc: "P.J. Eby" <pje at telecommunity.com>, web-sig at python.org
Sent: Tuesday, January 11, 2011 2:28:55 AM
Subject: Re: [Web-SIG] PEP 444 feature request - Futures executor

On Tue, Jan 11, 2011 at 10:40 AM, Timothy Farrell
<tfarrell at owassobible.org> wrote:
> ----- Original Message -----
> From: "P.J. Eby" <pje at telecommunity.com>
> To: "Timothy Farrell" <tfarrell at owassobible.org>, web-sig at python.org
> Sent: Friday, January 7, 2011 2:14:20 PM
> Subject: Re: [Web-SIG] PEP 444 feature request - Futures executor
>
>> There are some other issues that might need to be addressed, like
>> maybe adding an attribute or two for the level of reliability
>> guaranteed by the executor, or allowing the app to request a given
>> reliability level. ?Specifically, it might be important to distinguish between:
>
>> * this will be run exactly once as long as the server doesn't crash
>> * this will eventually be run once, even if the server suffers a
>> fatal error between now and then
>
>> IOW, to indicate whether the thing being done is "transactional", so to speak.
>
> I understand why this would be good (credit card transactions particularly), but how would this play our in the real world? ?All servers will do their best to run the jobs given them.
>
> Are you suggesting that there would be a property of the executor that would change based on the load of the server or some other metric? ?Say the server has 100 queued jobs and only 2 worker threads, would it then have a way of saying, "I'll get to this eventually, but I'm pretty swamped."?
>
> Is that what you're getting at or something more like database transactions..."I guarantee that I won't stop halfway through this process."

Maybe in the same vein as Quartz (http://www.quartz-scheduler.org/) in
Java world.

Nam

From pje at telecommunity.com  Tue Jan 11 15:30:28 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 11 Jan 2011 09:30:28 -0500
Subject: [Web-SIG] PEP 444 feature request - Futures executor
In-Reply-To: <17844068.49.1294715482026.JavaMail.root@boaz>
References: <20110107201420.346A43A40A8@sparrow.telecommunity.com>
	<17844068.49.1294715482026.JavaMail.root@boaz>
Message-ID: <20110111143038.330983A4074@sparrow.telecommunity.com>

At 09:11 PM 1/10/2011 -0600, Timothy Farrell wrote:
>PJ,
>
>You seem to be old-hat at this so I'm looking for a little advise as 
>I draft this spec.  It seems a bad idea to me to just say 
>environ['wsgi.executor'] will be a wrapped futures executor because 
>the api handled by the executor can and likely will change over 
>time.  Am I write in thinking that a spec should be more specific in 
>saying that the executor object will have "these specific methods" 
>and so as future changes, the spec is not in danger of invalidation 
>due to the changes?

I'd actually just suggest something like:

     future = environ['wsgiorg.future'](func, *args, **kw)

(You need to use the wsgiorg.* namespace for extension proposals like 
this, btw.)


From alice at gothcandy.com  Tue Jan 11 20:11:03 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Tue, 11 Jan 2011 11:11:03 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
	<20110107155545.8AD763A40A8@sparrow.telecommunity.com>
	<AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>
	<AANLkTimU-nD_+jk0-S0N9TMwo2-mVNpq1Ra17FDRuq=s@mail.gmail.com>
Message-ID: <igia07$4qi$1@dough.gmane.org>

On 2011-01-10 13:12:57 -0800, Guido van Rossum said:

> Ok, now that we've had a week of back and forth about this, let me 
> repeat my "threat". Unless more concerns are brought up in the next 24 
> hours, can PEP 3333 be accepted? It seems a lot of people are waiting 
> for a decision that enables implementers to go ahead and claim PEP 
> 333[3] compatibility. PEP 444 can take longer.

Two hours to go...

	- Alice.



From alice at gothcandy.com  Wed Jan 12 23:34:22 2011
From: alice at gothcandy.com (=?utf-8?Q?Alice_Bevan=E2=80=93McGregor?=)
Date: Wed, 12 Jan 2011 14:34:22 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
	<20110107155545.8AD763A40A8@sparrow.telecommunity.com>
	<AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>
	<AANLkTimU-nD_+jk0-S0N9TMwo2-mVNpq1Ra17FDRuq=s@mail.gmail.com>
Message-ID: <igla9e$bbb$1@dough.gmane.org>

On 2011-01-10 13:12:57 -0800, Guido van Rossum said:
> Ok, now that we've had a week of back and forth about this, let me 
> repeat my "threat". Unless more concerns are brought up in the next 24 
> hours, can PEP 3333 be accepted? It seems a lot of people are waiting 
> for a decision that enables implementers to go ahead and claim PEP 
> 333[3] compatibility. PEP 444 can take longer.

With the lack of responses, can I assume this has been or will be 
shortly marked as "accepted"?

I look forward to updating WebCore with compatibility.

	? Alice.



From guido at python.org  Wed Jan 12 23:52:45 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 12 Jan 2011 14:52:45 -0800
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <igla9e$bbb$1@dough.gmane.org>
References: <AANLkTikYpOvAqPRB8Oa+qs+CuWjcP0iZSgUzL10gVWiK@mail.gmail.com>
	<20110104012922.D45143A4119@sparrow.telecommunity.com>
	<AANLkTikG6qiempnt4dsGCxvA-uVim2Zu2b91ZDcjvHSh@mail.gmail.com>
	<20110104154827.B2DB93A4890@sparrow.telecommunity.com>
	<AANLkTin7yfz1mTkO5Av+oW+fwdd35w9a2+UfzxveGzRq@mail.gmail.com>
	<AANLkTikLNVjXqLJAL4BaKTUz4syA8ZC+xdiHfQVG33X_@mail.gmail.com>
	<20110107043012.CC3EB3A40A8@sparrow.telecommunity.com>
	<D700971C-E7FB-49AA-B7FF-FECEB8B415E8@fuhm.net>
	<AANLkTikRjs3R65anbQw6VmGMQMnQjM94r=58+jYNviob@mail.gmail.com>
	<AANLkTinA6xNBG7=TbO-E4VnAjM9AnzNiLhN2465ub4=o@mail.gmail.com>
	<AANLkTimqiH_Jhro9WggWwvLhO39C0KNheh_HbJjJ0TBr@mail.gmail.com>
	<AANLkTi=LNp2u7CWGm67MKJxz=7-igLXkSruQSPKoofDu@mail.gmail.com>
	<AANLkTinJTegaBSe2-AJXfwtXXod6JoeBBeZHDNhA=Z5E@mail.gmail.com>
	<20110107155545.8AD763A40A8@sparrow.telecommunity.com>
	<AANLkTim4TMkX3f9dfxsG3AknhxWPuVnFm50nT3HZLyOy@mail.gmail.com>
	<AANLkTimU-nD_+jk0-S0N9TMwo2-mVNpq1Ra17FDRuq=s@mail.gmail.com>
	<igla9e$bbb$1@dough.gmane.org>
Message-ID: <AANLkTinpHUGr5AF_iEiG=0gSbT8SQ+hgnxJqAYRh8oUk@mail.gmail.com>

On Wed, Jan 12, 2011 at 2:34 PM, Alice Bevan?McGregor
<alice at gothcandy.com> wrote:
> On 2011-01-10 13:12:57 -0800, Guido van Rossum said:
>>
>> Ok, now that we've had a week of back and forth about this, let me repeat
>> my "threat". Unless more concerns are brought up in the next 24 hours, can
>> PEP 3333 be accepted? It seems a lot of people are waiting for a decision
>> that enables implementers to go ahead and claim PEP 333[3] compatibility.
>> PEP 444 can take longer.
>
> With the lack of responses, can I assume this has been or will be shortly
> marked as "accepted"?

Yep. Phillip, can you do the honors?

> I look forward to updating WebCore with compatibility.

-- 
--Guido van Rossum (python.org/~guido)

From pje at telecommunity.com  Thu Jan 13 02:02:12 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 12 Jan 2011 20:02:12 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
Message-ID: <20110113010224.8FDB33A4065@sparrow.telecommunity.com>

At 02:52 PM 1/12/2011 -0800, Guido van Rossum wrote:
>On Wed, Jan 12, 2011 at 2:34 PM, Alice Bevan?McGregor
><alice at gothcandy.com> wrote:
> > On 2011-01-10 13:12:57 -0800, Guido van Rossum said:
> >>
> >> Ok, now that we've had a week of back and forth about this, let me repeat
> >> my "threat". Unless more concerns are brought up in the next 24 hours, can
> >> PEP 3333 be accepted? It seems a lot of people are waiting for a decision
> >> that enables implementers to go ahead and claim PEP 333[3] compatibility.
> >> PEP 444 can take longer.
> >
> > With the lack of responses, can I assume this has been or will be shortly
> > marked as "accepted"?
>
>Yep. Phillip, can you do the honors?

Apparently not -- I went to check it in and found Raymond had already 
marked it "Final".  ;-)

(I'm not clear on whether there's a difference between "Final" and 
"Accepted" heredifference, but I assume that if we find some sort of 
actual error we can still fix it.)



From graham.dumpleton at gmail.com  Thu Jan 13 02:11:11 2011
From: graham.dumpleton at gmail.com (Graham Dumpleton)
Date: Thu, 13 Jan 2011 12:11:11 +1100
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <20110113010224.8FDB33A4065@sparrow.telecommunity.com>
References: <20110113010224.8FDB33A4065@sparrow.telecommunity.com>
Message-ID: <AANLkTikLPb2rDdcXD3iUsE=qz4DCRDJJpCQFrEdcfmDX@mail.gmail.com>

On 13 January 2011 12:02, P.J. Eby <pje at telecommunity.com> wrote:
> At 02:52 PM 1/12/2011 -0800, Guido van Rossum wrote:
>>
>> On Wed, Jan 12, 2011 at 2:34 PM, Alice Bevan?McGregor
>> <alice at gothcandy.com> wrote:
>> > On 2011-01-10 13:12:57 -0800, Guido van Rossum said:
>> >>
>> >> Ok, now that we've had a week of back and forth about this, let me
>> >> repeat
>> >> my "threat". Unless more concerns are brought up in the next 24 hours,
>> >> can
>> >> PEP 3333 be accepted? It seems a lot of people are waiting for a
>> >> decision
>> >> that enables implementers to go ahead and claim PEP 333[3]
>> >> compatibility.
>> >> PEP 444 can take longer.
>> >
>> > With the lack of responses, can I assume this has been or will be
>> > shortly
>> > marked as "accepted"?
>>
>> Yep. Phillip, can you do the honors?
>
> Apparently not -- I went to check it in and found Raymond had already marked
> it "Final". ?;-)
>
> (I'm not clear on whether there's a difference between "Final" and
> "Accepted" heredifference, but I assume that if we find some sort of actual
> error we can still fix it.)

You can partly blame me for that. They were talking about WSGI and
Python 3 on #python-dev and I mentioned that Guido had just blessed it
and made mistake of mentioning the word 'final' in the sentence not
knowing anything about what next status would be. Anyway, Raymond
decided to take it on themselves to update even though I said to leave
it to you. There response since is 'IIRC, informational peps go
straight to final upon acceptance.'. So, they seem to think that the
final status is correct.

Graham

From victor.stinner at haypocalc.com  Fri Jan 14 14:25:33 2011
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 14 Jan 2011 14:25:33 +0100
Subject: [Web-SIG] Issue #4953 (cgi) closed: please test it as much as
	possible
Message-ID: <1295011533.1559.4.camel@marge>

Hi,

I just closed issue #4953 just before Python 3.2 final: the CGI module
should now handle correctly binary files and Unicode. I also patched
urllib.parse.parse_qs() and urllib.parse.parse_qsl() to add encoding and
errors arguments.

Please test the CGI module (with Python 3.2) as much as possible,
especially with WSGI.

http://bugs.python.org/issue4953

Victor


From alex.gronholm at nextday.fi  Mon Jan 17 04:49:06 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Mon, 17 Jan 2011 05:49:06 +0200
Subject: [Web-SIG] PEP 444 and asynchronous support, continued
Message-ID: <4D33BC32.2010904@nextday.fi>

After a weekend of experimentation with several asynchronous frameworks 
including gevent, tornado and twisted (and writing one myself too), and 
these are my findings so far:

- asynchronous socket implementations vary wildly across different 
frameworks
- gevent is the fastest, tornado comes second while twisted is pretty slow
- twisted provides the most comprehensive support for implementing 
protocols, while the other two mostly just provide low level support for 
asynchronous sockets
- futures seem to have a significant overhead (from the thread 
synchronization)
- gevent provides the easiest programming interface with greenlets, 
since it pretty much lets you write asynchronous code as you would write 
it synchronously
- gevent could make use of the regular, synchronous PEP 444 API by 
monkey patching the socket library (through its import monkey; 
monkey.patch_socket() call)

The significance of this for the Python web standards effort is that 
providing an asynchronous API that works for the existing asynchronous 
frameworks does not seem feasible. I'd love to see a solution for this 
in the standard library, but gevent's monkey patching approach, while 
convenient for the developer, cannot obviously work there. Before an 
asynchronous WSGI API can be provided, this lower level problem needs to 
be solved first. The crucial question is: is it possible to provide 
gevent's level of convenience through the standard library, and if not, 
what is the next best solution? I'd like to hear everyone's thoughts on 
this (especially Guido's).

From glyph at twistedmatrix.com  Mon Jan 17 05:47:01 2011
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sun, 16 Jan 2011 23:47:01 -0500
Subject: [Web-SIG] PEP 444 and asynchronous support, continued
In-Reply-To: <4D33BC32.2010904@nextday.fi>
References: <4D33BC32.2010904@nextday.fi>
Message-ID: <FB409B5C-18A9-431B-93B8-2084C4DB0ACB@twistedmatrix.com>

On Jan 16, 2011, at 10:49 PM, Alex Gr?nholm wrote:

> After a weekend of experimentation with several asynchronous frameworks including gevent, tornado and twisted (and writing one myself too), and these are my findings so far:
> 
> - asynchronous socket implementations vary wildly across different frameworks

That's certainly true.

> - gevent is the fastest, tornado comes second while twisted is pretty slow

Fastest at... what?

If you have a WSGI benchmark for Twisted, could you contribute it in a form that we could use at <http://speed.twistedmatrix.com/> so that we can improve the situation?  Thanks.

> - futures seem to have a significant overhead (from the thread synchronization)

If there were some way to have tighter control over where the callbacks in add_done_callback were executed, thread synchronization might not be necessary.  The module as currently specified does need to have a bit of overhead to deal with that, but the general concept doesn't.

> The significance of this for the Python web standards effort is that providing an asynchronous API that works for the existing asynchronous frameworks does not seem feasible.

I don't see how that follows from anything you've said above.

> I'd love to see a solution for this in the standard library, but gevent's monkey patching approach, while convenient for the developer, cannot obviously work there.

gevent and eventlet don't need any special support from WSGI though.  It's basically its own special kind of multithreading, with explicit context-switches, but from the application developer's perspective it's almost exactly the same as working with threads.  The API can be the existing WSGI API.

Twisted and Tornado and Marrow (and Diesel, if that were a thing that still existed) do need explicit APIs though, and it seems to me that there might be some value in that.

For that matter, Eventlet can use Twisted as a networking engine, so actually you can already use Twisted asynchronously with WSGI that way.  The whole point of having an asynchronous WSGI standard is to allow applications to be written such that they can have explicitly-controlled event-driven concurrency, not abstracted-over context switches in a convenience wrapper.

> Before an asynchronous WSGI API can be provided, this lower level problem needs to be solved first.

I'm not even clear on what "lower level problem" you're talking about.  If you're talking about interoperability between event-driven frameworks, I see it the other way around: asynchronous WSGI is a good place to start working on interoperability, not a problem to solve later when the rest of the harder low-level things have somehow been unified.  (I'm pretty sure that's never going to happen.)

> The crucial question is: is it possible to provide gevent's level of convenience through the standard library, and if not, what is the next best solution? I'd like to hear everyone's thoughts on this (especially Guido's).

gevent and eventlet already have things that will monkey patch the socket module that the standard library uses (for example: <http://eventlet.net/doc/patching.html>), so ... yes?  And if this "level of convenience" is what you're aiming for (blocking calls with an efficient, non-threaded scheduler), again, you don't need async WSGI for that.


From alex.gronholm at nextday.fi  Mon Jan 17 05:59:08 2011
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Mon, 17 Jan 2011 06:59:08 +0200
Subject: [Web-SIG] PEP 444 and asynchronous support, continued
In-Reply-To: <FB409B5C-18A9-431B-93B8-2084C4DB0ACB@twistedmatrix.com>
References: <4D33BC32.2010904@nextday.fi>
	<FB409B5C-18A9-431B-93B8-2084C4DB0ACB@twistedmatrix.com>
Message-ID: <4D33CC9C.1070707@nextday.fi>

17.01.2011 06:47, Glyph Lefkowitz kirjoitti:
> On Jan 16, 2011, at 10:49 PM, Alex Gr?nholm wrote:
>
>> After a weekend of experimentation with several asynchronous frameworks including gevent, tornado and twisted (and writing one myself too), and these are my findings so far:
>>
>> - asynchronous socket implementations vary wildly across different frameworks
> That's certainly true.
>
>> - gevent is the fastest, tornado comes second while twisted is pretty slow
> Fastest at... what?
>
> If you have a WSGI benchmark for Twisted, could you contribute it in a form that we could use at<http://speed.twistedmatrix.com/>  so that we can improve the situation?  Thanks.
I'm already regretting saying anything about performance. Our tests were 
run with the Apache Benchmark (ab) against a "Hello World" type WSGI 
app. Certainly nothing special.
>> - futures seem to have a significant overhead (from the thread synchronization)
> If there were some way to have tighter control over where the callbacks in add_done_callback were executed, thread synchronization might not be necessary.  The module as currently specified does need to have a bit of overhead to deal with that, but the general concept doesn't.
Unfortunately you are wrong. Thread synchronization is not necessary for 
callbacks, but it is necessary for supporting the result() method, since 
other threads may be blocking on that call.
>> The significance of this for the Python web standards effort is that providing an asynchronous API that works for the existing asynchronous frameworks does not seem feasible.
> I don't see how that follows from anything you've said above.
Asynchronous apps (save for gevent and the likes) can't use the standard 
wsgi.input since reading would block the event loop. Therefore an 
alternative input has to be provided, right? How would that work then? 
If something, say, wsgi.async_input was to be provided, what would it 
return from .read()? Futures? Deferreds?
>> I'd love to see a solution for this in the standard library, but gevent's monkey patching approach, while convenient for the developer, cannot obviously work there.
> gevent and eventlet don't need any special support from WSGI though.  It's basically its own special kind of multithreading, with explicit context-switches, but from the application developer's perspective it's almost exactly the same as working with threads.  The API can be the existing WSGI API.
>
> Twisted and Tornado and Marrow (and Diesel, if that were a thing that still existed) do need explicit APIs though, and it seems to me that there might be some value in that.
Which leads to the problem I described above.
> For that matter, Eventlet can use Twisted as a networking engine, so actually you can already use Twisted asynchronously with WSGI that way.  The whole point of having an asynchronous WSGI standard is to allow applications to be written such that they can have explicitly-controlled event-driven concurrency, not abstracted-over context switches in a convenience wrapper.
It is my understanding that eventlet only runs on CPython. Am I mistaken?
>> Before an asynchronous WSGI API can be provided, this lower level problem needs to be solved first.
> I'm not even clear on what "lower level problem" you're talking about.  If you're talking about interoperability between event-driven frameworks, I see it the other way around: asynchronous WSGI is a good place to start working on interoperability, not a problem to solve later when the rest of the harder low-level things have somehow been unified.  (I'm pretty sure that's never going to happen.)
>
>> The crucial question is: is it possible to provide gevent's level of convenience through the standard library, and if not, what is the next best solution? I'd like to hear everyone's thoughts on this (especially Guido's).
> gevent and eventlet already have things that will monkey patch the socket module that the standard library uses (for example:<http://eventlet.net/doc/patching.html>), so ... yes?  And if this "level of convenience" is what you're aiming for (blocking calls with an efficient, non-threaded scheduler), again, you don't need async WSGI for that.
That's what I've been saying. But that only holds true for 
gevent/eventlet. Twisted, for one, needs explicit support unless, as you 
said, is used through eventlet.
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/alex.gronholm%40nextday.fi


From 8mayday at gmail.com  Thu Jan 20 12:17:03 2011
From: 8mayday at gmail.com (Andrey Popp)
Date: Thu, 20 Jan 2011 14:17:03 +0300
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <AANLkTikLPb2rDdcXD3iUsE=qz4DCRDJJpCQFrEdcfmDX@mail.gmail.com>
References: <20110113010224.8FDB33A4065@sparrow.telecommunity.com>
	<AANLkTikLPb2rDdcXD3iUsE=qz4DCRDJJpCQFrEdcfmDX@mail.gmail.com>
Message-ID: <D2DA130D-C23B-4D0E-AFD1-ADFD02FAE0B0@gmail.com>

Hello,

while PEP-3333 declared as "final" (not sure if it means the same as "accepted"), I would like to draw your attention at this bug report[1].

Personally, I'd like to fix PEP in the way it resolves mentioned bug. But if you disagree with me, I'd like at least to add note about statements like:

  "object A should be of type TA"

It's not common behavior for Python objects, and in some meaning it just breaks OOP in Python (see Liskov substitution principle[2]). Isn't this looks like unpythonic or at least as leaking abstraction?

[1]: http://bugs.python.org/issue10935
[2]: http://en.wikipedia.org/wiki/Liskov_substitution_principle


From foom at fuhm.net  Thu Jan 20 15:45:12 2011
From: foom at fuhm.net (James Y Knight)
Date: Thu, 20 Jan 2011 09:45:12 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <D2DA130D-C23B-4D0E-AFD1-ADFD02FAE0B0@gmail.com>
References: <20110113010224.8FDB33A4065@sparrow.telecommunity.com>
	<AANLkTikLPb2rDdcXD3iUsE=qz4DCRDJJpCQFrEdcfmDX@mail.gmail.com>
	<D2DA130D-C23B-4D0E-AFD1-ADFD02FAE0B0@gmail.com>
Message-ID: <23FDDB4C-2FA7-4669-BCA2-77154694CA36@fuhm.net>

On Jan 20, 2011, at 6:17 AM, Andrey Popp wrote:
> Hello,
> 
> while PEP-3333 declared as "final" (not sure if it means the same as "accepted"), I would like to draw your attention at this bug report[1].
> 
> Personally, I'd like to fix PEP in the way it resolves mentioned bug.

-1. I think the restriction to exact types is useful. It avoids the need to worry about running arbitrary unexpected user code in *Python* contexts. (as you pointed out, the C API ignores any overrides done in subclasses of str, and just accesses it as if it was the base class anyways...)

Furthermore, even were it not the case, the bug report is not timely: PEP 3333 is a minimal modification over PEP 333, which has been final for years.

> But if you disagree with me, I'd like at least to add note about statements like:
> 
>  "object A should be of type TA"
> 
> It's not common behavior for Python objects, and in some meaning it just breaks OOP in Python (see Liskov substitution principle[2]). Isn't this looks like unpythonic or at least as leaking abstraction?

It already states that for response_headers: "i.e. type(response_headers) is ListType". 

But I agree, a clarification could be added to the statement '''all objects referred to in this specification as "strings" must be of type str or StringType''' and '''For values referred to in this specification as "bytestrings" [...] the value must be of type bytes under Python 3, and str in earlier versions of Python'''. It's not 100% obvious that it really does mean "type(obj) is str/bytes".

James

From pje at telecommunity.com  Thu Jan 20 19:31:48 2011
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 20 Jan 2011 13:31:48 -0500
Subject: [Web-SIG] Declaring PEP 3333 accepted (was: PEP 444 != WSGI 2.0)
In-Reply-To: <23FDDB4C-2FA7-4669-BCA2-77154694CA36@fuhm.net>
References: <20110113010224.8FDB33A4065@sparrow.telecommunity.com>
	<AANLkTikLPb2rDdcXD3iUsE=qz4DCRDJJpCQFrEdcfmDX@mail.gmail.com>
	<D2DA130D-C23B-4D0E-AFD1-ADFD02FAE0B0@gmail.com>
	<23FDDB4C-2FA7-4669-BCA2-77154694CA36@fuhm.net>
Message-ID: <20110120183203.165B53A4065@sparrow.telecommunity.com>

At 09:45 AM 1/20/2011 -0500, James Y Knight wrote:
>But I agree, a clarification could be added to the statement '''all 
>objects referred to in this specification as "strings" must be of 
>type str or StringType''' and '''For values referred to in this 
>specification as "bytestrings" [...] the value must be of type bytes 
>under Python 3, and str in earlier versions of Python'''. It's not 
>100% obvious that it really does mean "type(obj) is str/bytes".

Feel free to write said clarification, check it in, and add glowing 
praise for your efforts to the acknowledgements section.  I will 
indeed appreciate it, so you won't even be lying.   ;-)

Or, you can just send me a patch, and receive slightly less 
praise.  Either way works for me, though.  ;-)