[Web-SIG] Draft 2: WSGI Response Upgrade Bridging

[PJ Eby]

On Fri, Oct 10, 2014 at 8:56 AM, Graham Dumpleton
<graham.dumpleton at gmail.com> wrote:
> So PJE, please step back and do not go rushing out to create a PEP. That is
> the worst thing you could do at this point and will only serve to deter
> people from the community contributing and so stifle proper discussion about
> this whole topic.

Huh?  Have you *read* the PEP?  The entire point of it is to provide a
basis for *experimenting* with new standards, not to "stifle
discussion" of them.  It's not even an *API*, for heavens' sake.  It's
just a description of how to upgrade to new standards from within
existing WSGI frameworks, without needing to tunnel responses and
without breaking subrequest middleware.

IOW, it's a WSGI *1.0* server extension protocol, and a fairly *minor*
one at that.  (Indeed, it's little more than an enhanced variation of
wsgi.file_wrapper!)  It's not any sort of competitor or alternative to
what Robert's working on; it's a *stepping stone* for what Robert's
working on.

In early discussion with Robert -- both here and on github -- it
became apparent to me that restricting post-WSGI specifications to
what can be achieved in WSGI 1 tunneling is a bad idea.  So I've
created a *bridging* specification, that allows post-WSGI APIs to be
accessed from within WSGI-based apps and frameworks.

That's *all*.

All of the things you've mentioned as being in scope for discussion,
are *still* in scope for discussion.  All *this* proposal does is show
how those things could be *accessed*, *today*, from inside existing
web apps and frameworks, once those new APIs exist.


> You have no more experience or mandate to be specifying a
> standard for this than anyone else.

If by "this" you're referring to HTTP/2 or some other new post-WSGI
API, then I agree with you.  But that's not what the PEP is about.


>  By creating a PEP though that gets
> perceived by many as meaning the discussion is over. This is exactly what
> you did for PEP 3333 and which caused previous discussion about improving
> WSGI to get shutdown.

That's an interesting perspective, but I don't see how it can be
reconciled with the facts.

First off, I didn't write a new PEP; I wrote up some of *your*
proposed clarifications for Python 3 as WSGI 1.0.1, which was intended
to add new clarifying text to PEP 333, *not* to create a new PEP.  It
was *Guido* who said it must be a new PEP, as you will see here:

  https://mail.python.org/pipermail/web-sig/2010-September/004691.html

and here (where he even says, "Don't see this as a new spec. See it as
a procedural issue."):

  https://mail.python.org/pipermail/web-sig/2010-September/004694.html

Second, I didn't make anybody stop discussing alternatives for moving
things forward.  Nobody *ever* said to stop working on a version 2 or
even 1.1, certainly not me.  See for example, this message, where I
agreed with Ian's POV that there was room for both PEP 333 fixes *and*
continued work on PEP 444:

  https://mail.python.org/pipermail/web-sig/2010-September/004662.html

Third, and finally, as far as I can tell from the record of the
discussion back then, it was you -- and *only* you -- who suggested
that the acceptance of PEP 3333 meant the discussion was *over*.
Indeed, on your blog you actually pushed back at Alice for bringing up
more PEP 444 discussion!

Nonetheless, discussion of PEP 444 and async APIs and such proceeded
well past the introduction of PEP 3333, even without its original
authors' participation.  And, ironically enough, your posts show up in
that discussion, bemoaning that Alice (the new PEP 444 champion) was
creating confusion by calling that proposal WSGI 2.0!


> The result was that the only thing that really got
> addressed in PEP 3333 was Python 3 compatibility and a lot of the other bits
> of the WSGI specification which are poorly defined, contradictory or
> restrictive and which cause WSGI server and application developers pain
> never got addressed. If that prior discussion hadn't been shutdown in that
> way, we could have been using a better defined and improved WSGI years ago
> already.

Those things didn't get addressed because *you* didn't take up the
lead -- a lead which I more than once mentioned you should take up.
For example, as I said in
https://mail.python.org/pipermail/web-sig/2010-September/004693.html :

> The full list of things Graham and others have asked for or
> recommended would indeed require a 1.1 version at minimum, and thus a
> new PEP.  But I really don't want to start down that road right now,
> and therefore hope that I can talk Graham or some other poor soul
> into shepherding a 1.1 PEP instead.  ;-)

You didn't, and haven't, taken up that slack.  What you've
consistently done is mutter and grumble on the sidelines about how
it's never going to happen and disclaim any responsibility for
actually writing a proposal because it's never going to go anywhere --
thereby *ensuring* that it's never going to go anywhere.  (And one key
reason I wrote the WSGI-RUB PEP is that I noticed I'd started doing
what *you* do: grumbling at Robert about his proposals, without taking
the time to write up my own, dumping the hard work on him instead of
getting my own hands dirty.)

PEPs don't magically arise from some mysterious group consensus.  They
happen because some poor shmuck does the work of *building* a
consensus around something they want to have happen, hammering out
agreements, and writing the thing up.  The only way the improvements
you want are ever going to happen are if you either lead the process
yourself, or get somebody else to do it for you.  If you want to ride
Robert's back and get him to do the dirty work, that's A-OK by me.  I
don't have a horse in *that* race, and haven't for *ten years*.

The PEP 444 discussion didn't stop because I did the dirty work of
turning some of your gripes into concrete specifications.  It stopped
because the poor schmucks who initially volunteered to do the heavy
lifting on PEP 444 were only doing it to get Python 3 sorted out, and
were sufficiently happy with the 1.0.1 clarifications to be glad of
dropping the *workload* involved...  a workload which you declined to
pick up, despite many people (myself included) *asking* for continued
discussion of PEP 444.

*PEPs* don't stifle discussions.  *Lack of volunteers* stifles
discussions.  Without somebody driving the process, discussions about
multiple substantive issues with a PEP tend to die a natural death as
everybody voices their opinion *once*...  and then shuts up.

What keeps a PEP moving is somebody taking those raw opinions and
pushing something forward from them, asking "so what about this, then?
 Will this work?  What do you think of that?"  Without that energy
being put *in*, nothing comes back out, except maybe the occasional
lengthy session of bikeshedding on the non-substantive parts of a
proposal.  Indeed, the more substantive the discussion, the fewer the
participants, and the harder it is to actually get things moving...
and the more likely the champion is to give up in the face of what
seems like overwhelming opposition.

So it's not suprising that Chris and Armin and Alice all gave up on
doing that: it's a lot of hard work, and *I'm* not volunteering for
it, either.

If you want Robert to do the work of shepherding a new post-WSGI PEP
under your guidance I am *all* in favor of it.  I've been trying to
get *you* off the sidelines on this thing for *years*.  Indeed, if
that is the only outcome of the work I did on the new RUB proposal,
then I am as happy to drop out of it as Chris and Armin were to drop
PEP 444.  (Frankly, some of my admonishments to Robert were based on
my expectation that you would continue to snipe from the sidelines and
avoid getting your hands dirty.)

Which is unfortunate, because AFAIK and IMO, *you* are the only person
currently active in the community who's both *actually* qualified to
ride herd on WSGI 1.1 *and* is an absolute "must-have" contributor for
the sucess of any true post-WSGI specification.  (Again, AFAIK and
IMO, not intended as a slight to anybody else whose qualifications and
contributions I'm presently unaware of.)

So the PEP I've written *isn't* an attempt to make such a post-WSGI
specification.  It's my attempt to build a bridge from the WSGI we
have, to whatever specification you and the rest of the Web-SIG come
up with *next*.  Indeed, it's intended as something of a "parting
gift", to address the development, deployment, and long-term migration
issues that *any* post-WSGI spec will have.

I'm tired of dealing with WSGI's limitations and corner cases and
quirks, and I don't want to have to spend a lot of time reviewing
post-WSGI specs to check for breakage on those quirks.  The point of
the RUB is to set the post-WSGI world entirely free of them, and it's
my gift to you and Robert and anybody else who wants to clean away the
mess and start over.

With it, you can imagine completely new APIs (like the generator-based
ones Robert was sketching), without needing to figure out how to make
the bloody thing work with WSGI 1.0 middleware.

Pre-empting that kind of free API design is the *last* thing I want to
have happen, which is precisely *why* I've put forward the RUB spec.
I don't *want* to spend a lot of time telling Robert all the things he
*can't* do, because of WSGI 1's limitations.

So instead, I've proposed something that will let him "have" whatever
sort of post-WSGI cake he wants, while still letting WSGI 1 code "eat"
it too.


> Right now, that you have created your own separate space

How is a Web-SIG thread a "separate space", let alone my "own" separate space?


> for writing up a specification which you are now
> trying to rush into a PEP comes across as you not really wanting to
> co-ordinate with Robert on this as a community effort with it instead
> appearing that you think you know better than anyone else and nothing anyone
> else says will be of value. In the face of that, it is hardly surprising
> that no one has really responded to what you have proposed.

Well, if that's the case, it's certainly an unfortunate
misunderstanding.  Because all I have *actually* done is described a
way that *everyone* can contribute to the development of new APIs,
without *first* needing everyone *else* to agree on those APIs and
modify their web frameworks to support them.  I'm trying to make it
possible for there to be *more* participation, not less.  Any J.
Random Developer with an idea or an itch to scratch should be able to
throw together an implementation of a post-WSGI API, and start using
it today from inside existing WSGI frameworks.  It's from just such
experiments that de facto -- and later, de jure -- standards can and
should arise.

Personally, I don't expect there to be much discussion of my proposal
right now because nobody is yet trying to *implement* any post-WSGI
specifications.  The RUB spec is mostly a stake in the ground, to say,
"Don't worry about WSGI 1 compatibility or tunneling; you can use any
API paradigm you want, and the community will still be able to make an
orderly transition to using it.  Here's how."

At this point, AFAIK, there are precisely two APIs that could have
benefited from the prior existence of WSGI-RUB: nghttp2 and
uwsgi.websockets.  (Which is why the examples in the spec are based on
them.)  And I'm not especially aware of anybody else writing new ones,
who would therefore be interested in it.  Frankly, I wrote the thing
to get it out of my head and to have a convenient place to point to
when anybody makes the same mistake I did, of trying to limit
post-WSGI API design to what can be safely shoehorned back through
WSGI 1 response middleware.

I don't expect much *detailed* discussion of WSGI-RUB, in other words,
until there's at least a strawman post-WSGI API proposal.  (Which,
IIUC from his last message, Robert is doing some experimenting
towards.)

Perhaps it is not clear from your cursory review of the existing
discussion, but both Robert and I *learned* some things from our
interaction.  And one of the things that *I* learned from that
interaction was that nitpicking from the sidelines about what Robert
couldn't do or how he should do it was not productive.

Which is why I've put forth a proposal that eliminates the need for
post-WSGI APIs to be nitpicked for WSGI 1 middleware compatibility.

And it's why I hope *you* will read that proposal, and have something
to say about its substance, because you are one of the people from
whom I have most eagerly awaited such feedback, be it good, bad, or
ugly.

But I would *much* prefer some feedback about its *substance*, to a
bunch of insinuations about whether I should've proposed it at all.