On Sep 19, 2004, at 12:30 PM, exarkun@divmod.com wrote:

twisted/web2 has an unacceptable dependency on nevow that will need to be discussed or removed before this subpackage can be added.

First off -- I consider this a rude thing to do, considering I am far from being unreachable. What was the emergency that prompted this removal at midnight on a saturday night? None? Yeah, that's what I thought.

On Sun, 19 Sep 2004 10:07:57 -0400, Itamar Shtull-Trauring itamar@itamarst.org wrote:

...

Why is this unacceptable? We depend on pyOpenSSL for TLS support, etc..

The logistical reasons:

There is no release of Nevow.

There is. Although it's old. This is a very good reason for not releasing twisted.web2. Fortunately, since we have decoupled releases now, it doesn't affect anything else.

None of the buildslaves have even an SVN trunk checkout installed, 

nor have any of them been asked to install one.

True.

Developers weren't told about this new dependency, so they may not 

have it installed.

twisted-web ml was. I don't expect developers that don't care about twisted web to install Nevow.

This leads to strangely failing tests, making it difficult to tell if outstanding changes that a developer wants to commit are broken.

I assume you mean "Could not import twisted.web2.test.test_web: Traceback: exceptions.ImportError, No module named nevow.". If so, "strangely failing tests" is quite a hyperbole.

The technical reasons:

Nevow imports Twisted.  Therefore, Twisted should not import Nevow.

Twisted is the networking framework and the web server framework.  

It is lower level than the application server and the templating system. The latter should depend on the former, if a dependency exists, not the reverse. _If_ it is desired for Twisted to gain functionality from a Nevow installed, Nevow must be careful to only import certain things from Twisted.

I would much prefer if Nevow gained functionality from Twisted and not the reverse, but if twisted.web/nevow developers desire things to work as they do currently in twisted.web2, at least the above issues need to be addressed. This thread should serve to resolve one of them (hopefully Twisted developers are now aware of the potential dependency ;).

Most of Twisted web2 does not/will not depend on Nevow. Generation of some server-generated pages does/will. Those tests should just be skipped if Nevow isn't installed. That they aren't is an issue, sure.

If you want to have any input on these technical matters I suggest participating on the twisted-web mailing list instead of just randomly removing code.

Please re-add it.

James

On Sun, 2004-09-19 at 22:35, exarkun@divmod.com wrote:

I did not come up with the idea to revert the web2 commit on the spot. Commits which introduce broken tests have always been fair game for reversion. Since the web2 tests were _new_, perhaps a different policy might apply. I'd certainly be willing to entertain arguments to that effect to decide what should be done in the future. I do not think reverting the commit was unreasonable here, though.

No, I don't think that a different policy should apply - revision history exists for a reason.

This point is worth emphasizing. We don't have a very rigorous process in place for development, but this has always been the case - trunk should be in a working state at all times. If you check in code that breaks the tests, it is at the discretion of any other developer to decide that your code is toast: it doesn't matter if you are available on IRC in real time. You can fix your breakages locally, without everyone else waiting for you.

...

[snip]

...

There is no release of Nevow.

There is. Although it's old. This is a very good reason for not releasing twisted.web2. Fortunately, since we have decoupled releases now, it doesn't affect anything else.

The first release of Twisted 2.0 will not be decoupled. It is important that we don't treat the releases as decoupled until we have actually successfully made at least one decoupled release of each project. I don't think this policy will change when project release cycles are separated though.

Even when trunk is in a completely working state, it is still difficult to get a release done right now, so I see no reason to accept tests which do not import, or tests which fail, in trunk. If you want to add a dependency on something new, this needs to be discussed before checking in.

...

Most of Twisted web2 does not/will not depend on Nevow. Generation of some server-generated pages does/will. Those tests should just be skipped if Nevow isn't installed. That they aren't is an issue, sure.

Do those server-side pages have simple, non-nevow fallbacks? If they can't, then perhaps we should be considering a merged release.

...

If you want to have any input on these technical matters I suggest participating on the twisted-web mailing list instead of just randomly removing code.

Personally I believe this is worth discussing generally, because these are not "web development" issues, although they do touch web code - I don't subscribe to twisted-web because I'm not interested in the particulars of HTTP rfc compliance or url argument parsing, but I most certainly *am* interested in circular dependencies threading through libraries which are distributed separately. There is perhaps a larger discussion here about the relationship between twisted-web and nevow, and Twisted and Divmod in general, that needs to happen on divmod-dev.

I am happy with my current level of avoidance of twisted web, discounting this most recent event. I hope that in the future web2 can be handled in such a way that it is not disruptive to those of us who are not interested in it.

Maybe I need to be a little more involved than I have been. I find the presence of both web and web2 in the same source tree a little upsetting - this harkens back the web.widgets / web.domtemplate / web.woven API disaster, which still haunts us.

Let's try to figure out which web server we are supporting and then actually support it, rather than having 9 different half-assed, half-supported implementations floating around.

On Sep 20, 2004, at 12:40 AM, Glyph Lefkowitz wrote:

Let's try to figure out which web server we are supporting and then actually support it, rather than having 9 different half-assed, half-supported implementations floating around.

Well, certainly nobody who wrote the original twisted.web code is supporting it, but that doesn't mean it's unusable. On the contrary, it's very usable, and there are probably lots and lots of applications which depend on all sorts of subtle and broken semantics it uses (people being far more interested in developing web related things than finger related things). I don't see any sane way to perform a complete rewrite with better semantics while still living in the old twisted.web namespace, especially given the miniscule amount of time most of the major developers of this project have to put against it. Developing backwards compatibility with the old APIs would be a death march, would never quite work anyway, and wouldn't really benefit anyone in particular.

We already figured out which web server we ("we" being those developers who actually care about the web) are going to be supporting. twisted.web2. web should have a deprecation warning in __init__ for a release, and then should be terminated with extreme prejudice.

As far as the nevow dependency, I have already suggested to James that twisted.web2 should work without it, but he didn't feel it was worth the effort. If it is worth the effort to someone else, I suggest they do it. My suggestion is simply not to have pages like directory listings and traceback renderings, but to have simple string templates which are returned in the case of nevow not being installed. This way you can use web2 with no dependencies to do a completely custom web server, but you're not going to get much out of the box.

dp

On Mon, 2004-09-20 at 10:49, Glyph Lefkowitz wrote:

...

We already figured out which web server we ("we" being those developers who actually care about the web) are going to be supporting. twisted.web2. web should have a deprecation warning in __init__ for a release, and then should be terminated with extreme prejudice.

I would terminate before providing a deprecation warning. If you want to provide a backwards-compatibility release for using web1 with the new reactor core, that's fine, but I don't see any reason to include web1 with the new download.

twisted.web will not be included. There will be two downloads ("Twisted Web 1 - Deprecated", "Twisted Web 2"). Most people will not download the first ever. The first will likely not have new releases unless we fine security issues. Moving it to branch and only doing releases from there may be fine (if our release infrastructure is up to it!) but I still think newweb should be twisted.web2.

Summary:

1. twisted.web2 remains web2. 2. Move twisted.web (and any other security bugfix only packages, I guess) to a "deprecated" branch.

Yes? No?

If yes, that just leaves the nevow/twisted.web2 merger question, which should probably be done on twisted-web mailing list.

On Mon, 2004-09-20 at 11:35, Christopher Armstrong wrote:

Don't bother moving twisted.web to a branch; the repository layout of trunk is (will be) decoupled from how releases are done, and since both will potentially be changed (e.g. for security) then I don't think it should be in a separate branch.

And +1 on keeping the name twisted.web2.

The release manager has spoken. I guess it's out of my hands now ;)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Urgh. I usually avoid getting involved in these discussions--not because I don't want to participate but because the ML style does not work very well with my brain. There's all these different parts of this proposal and some are being rejected by some people and modified by other people and it's very hard for me to figure out what the whole picture is.

In this case, I have to say something because this is pretty important to the product I'm developing. I just spent 6 weeks doing a conversion to Nevow because Woven isn't being supported any more. I do *not* want to do the same thing with Web. I've made modifications to Twisted trunk that I was hoping to get out of 2.0 for our official release, but if there's not going to be a web1 any more, that puts this project in jeopardy again because it means I may not be able to use 2.0.

So here's what I have to say: please *summarize*. I need to get a sense of:

1) Timeline.. when are changes being implemented, in what order, tied to what release etc. 2) How is the svn repo going to change? 3) What is going to be put into what release by whom? 4) What's the impact if you're currently using web1 and you want to be using web2 because 1 isn't supported any more? 5) What's the impact if you stay with web1?

If you haven't decided yet, go ahead and decide, but then summarize afterwards. :-)

Thanks..

C

Itamar Shtull-Trauring wrote: | On Mon, 2004-09-20 at 10:49, Glyph Lefkowitz wrote: | | |>>We already figured out which web server we ("we" being those developers |>>who actually care about the web) are going to be supporting. |>>twisted.web2. web should have a deprecation warning in __init__ for a |>>release, and then should be terminated with extreme prejudice. |> |>I would terminate before providing a deprecation warning. If you want |>to provide a backwards-compatibility release for using web1 with the new |>reactor core, that's fine, but I don't see any reason to include web1 |>with the new download. | | | twisted.web will not be included. There will be two downloads ("Twisted | Web 1 - Deprecated", "Twisted Web 2"). Most people will not download the | first ever. The first will likely not have new releases unless we fine | security issues. Moving it to branch and only doing releases from there | may be fine (if our release infrastructure is up to it!) but I still | think newweb should be twisted.web2. | | | Summary: | | 1. twisted.web2 remains web2. | 2. Move twisted.web (and any other security bugfix only packages, I | guess) to a "deprecated" branch. | | Yes? No? | | If yes, that just leaves the nevow/twisted.web2 merger question, which | should probably be done on twisted-web mailing list. | | | | _______________________________________________ | Twisted-Python mailing list | Twisted-Python@twistedmatrix.com | http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Mon, 2004-09-20 at 13:16, James Y Knight wrote:

I guess I'll keep the crossposting, since this may be of interest to those who do not read the twisted-web list.

Thanks james, that was a great summary of just about every web issue that I am concerned with. And thanks for the questions, MFen...

Unless anybody else objects I think this thread should be the last of the crossposting :)

Since people who care about web apparently don't read the twisted-web mailing list, I'm forwarding the relevant [portions of] messages to this mailing list, where they'll hopefully get read before any more responses are written on this topic.

James

From: James Y Knight foom@fuhm.net Date: September 14, 2004 10:19:14 PM EDT Subject: [Twisted-web] Twisted.web2

Itamar and I had dinner today and talked about new-web a bit. Here is what was discussed:

1. We need to keep compatibility with old Nevow resources
2. We ought to keep compatibility with old twisted.web resources
3. The new request API isn't the same as old request API.
4. new-web depends on Twisted 2.0. Nevow will be depending on new-web,

so the next Nevow will depend on twisted 2.0. Thus it cannot be released until Twisted 2.0. According to itamar the goal for next Twisted release is around halloween, although I hadn't seen that mentioned anywhere before. ;) 5) Rumors are Anthony is writing a non-sucky HTTP client. (is this true?). J. Simms (slyphon) mentioned something about writing one to me a few weeks ago but I didn't hear anything after. This should be coordinated so the API for the client and server is as similar as possible.

Conclusions for action:

1. new-web will be called twisted.web2. I will put what I've got into

svn trunk as twisted.web2 tomorrow, assuming no objections. This will allow easy backwards compatibility at the cost of a 2. 2) Nevow's resource traversal API must be moved to twisted.web2. This will probably have to be done by branching Nevow to remove those bits. Therefore, the resource traversal part of nevow should be mostly-frozen while this happens. 3) For distribution: twisted.web2 depends on Nevow (for static directory listings, and error pages), so will be distributed with it. Nevow recommends Twisted but does not require it for simple cases (e.g. flattening to a string from a CGI script), so there can also be a limited-functionality distribution of standalone Nevow.

Current status of new-web: the low-level HTTP protocol and header parsing is in passable shape. The high-level Resource API needs to be copied over from Nevow (again), because Nevow's API changed. Backwards compatibility adapters for the t.w.Request/nevow.inevow.IRequest and (possibly) nevow.inevow.IResource should be added.

I am not able to do everything myself. I will need help, especially from dp would be helpful for the Nevow transition, to make this happen. If we do it right, twisted.web2 might even be ready in time for the release of Twisted 2.0.

James

From: William Dode wilk-ml@flibuste.net Date: September 15, 2004 12:26:08 PM EDT Subject: Re: [Twisted-web] Twisted.web2

James Y Knight foom@fuhm.net writes:

...

3. For distribution: twisted.web2 depends on Nevow (for static

directory listings, and error pages), so will be distributed with it. Nevow recommends Twisted but does not require it for simple cases (e.g. flattening to a string from a CGI script), so there can also be a limited-functionality distribution of standalone Nevow.

It could be great if the server could not depend on any framework (i mean nevow).

From: James Y Knight foom@fuhm.net Date: September 15, 2004 1:49:26 PM EDT Subject: Re: [Twisted-web] Twisted.web2

On Sep 15, 2004, at 12:26 PM, William Dode wrote:

...

It could be great if the server could not depend on any framework (i mean nevow).

It will use nevow to render the pages the webserver generates, such as errors and directory listings. This does not mean you need to use nevow for your own pages. You can use whatever templating language you want (or none at all) to render your pages, assuming you can hook it up to twisted web somehow.

If you want to make a custom hacked-up copy of twisted.web without Nevow included, it will work, other than default server-generated pages. However, I don't see any reason to not include it by default.

From: Donovan Preston dp@ulaluma.com Date: September 15, 2004 2:08:35 PM EDT Subject: Re: [Twisted-web] Twisted.web2

On Sep 15, 2004, at 12:26 PM, William Dode wrote:

...

It could be great if the server could not depend on any framework (i mean nevow).

The server would depend on Nevow in order to generate the HTML for various automatically generated pages, such as directory listings. I suggested to James that having a standalone twisted.web server that didn't depend on Nevow and wasn't capable of generating any pages itself would be possible, but James didn't think it was worth the effort.

From: James Y Knight foom@fuhm.net Date: September 15, 2004 7:00:08 PM EDT Subject: [Twisted-web] Partial freeze of nevow coming up.

Here's the plan. The following modules will be removed from Nevow, and merged/copied/whatever into twisted.web2: appserver.py static.py half of context.py dirlist.py failure.py guard.py static.py vhost.py url.py, perhaps.

I want to create a nevow web2 integration branch, and while they are being removed from Nevow, and merged back into web2, I don't want the Nevow copy radically changing cause that just creates extra work.

So if there's any changes you want to make to any of those, please do it soon. The rest of Nevow can continue changing during the transition without problem.

Sidenote: Divmod needs to approve the relicensing of the code that moves into web2 from nevow under MIT license. Or else move all of Nevow under MIT license.

From: Donovan Preston dp@ulaluma.com Date: September 15, 2004 7:22:23 PM EDT Subject: Re: [Twisted-web] Partial freeze of nevow coming up.

...

Sidenote: Divmod needs to approve the relicensing of the code that moves into web2 from nevow under MIT license. Or else move all of Nevow under MIT license.

Moving Nevow to MIT license has already been approved. There is an issue in our RT tracker (which unfortunately nobody else can see) for switching to MIT license, but nobody has gotten time to do it. Anyone with commit access who has some spare time, feel free...

James Y Knight wrote:

...

5. Rumors are Anthony is writing a non-sucky HTTP client. (is this

true?). J. Simms (slyphon) mentioned something about writing one to me a few weeks ago but I didn't hear anything after. This should be coordinated so the API for the client and server is as similar as possible.

I have something that Works For Me. It makes sense to me (from an API point of view), and, more importantly for my current use-cases, works with urllib's Request objects, which means you can drop ClientForm on top of it directly, and it Just Works. This is good. It also works on top of a UDP transport, which the current http client does not. While HTTP-over-UDP is an Abomination In The Eyes of Great Cthulu, it's also used in some sick and warped protocols, which I need to support.

Anthony

On Sun, 2004-09-19 at 12:30, exarkun@divmod.com wrote:

Nevow imports Twisted.  Therefore, Twisted should not import Nevow.

Incorrect. If nevow imports twisted.internet and twisted.web2 imports nevow that is fine since it is a different project.

I think having twisted,web2 at least be semi-functional without nevow (i.e. importable and tests running) would be good enough for your concerns?