Hi,

Since I have a couple of projects building on nevow that I'd like to keep maintaining into the post-python2 world (and at least one of them contains a broader user base that I can't easily get to port templates and such), I've started porting nevow to python3. What's on https://github.com/msdemlei/nevow runs essentially all the unit tests that weren't broken on python2 except for those related to livepages.

I've not tried to port any python2 code to this now, though, since I find the current situation where lots of byte strings leak through from twisted.web quite unsatisfying. I've tried to formulate something like a preliminary policy in https://github.com/msdemlei/nevow/blob/master/doc/strings_and_bytes.rst, but I'm deeply unhappy about most of this.

Ideally, I'd like to fairly exclusively expose actual strings to nevow client code unless they actually deal with binary data (as in static.File, and of course request.write needs to be able to deal with this). This will probably mean isolating client code from twisted.web.Requests a bit more.

So:

(1) Is anyone else still interested in this? I'm very much interested in keeping Nevow alive, it's a the center of quite a few

Aloha On 01/23/2018 12:03 AM, Markus Demleitner wrote: projects of mine.

(2) Is anyone at all interested in the livepages stuff (I'm not, and dropping it would certainly make porting and maintaining the package a lot easier)? Definitely, Livepage is my bread and butter, although a rework of the Livepage part  might be be beneficial, I just never found the time to do it and the code just flawlessly does what it advertises. (3) If I went ahead with this and actually tried to get a python3-nevow into Debian (which I'd really, really like to see): How is everyone's feeling about keeping the name nevow even if serious functional changes (such as dropping livepages) were to come about? [What I care about most is keeping the templating system including the render_ and data_ methods]. I have no association to the name but ripping Nevow apart and still name it Nevow would be at least misleading.

Mahalo, Werner

Thanks,

Markus

_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web

Hi Glyph, On Tue, Jan 23, 2018 at 11:51:44PM -0800, Glyph wrote:

...

On Jan 23, 2018, at 2:03 AM, Markus Demleitner wrote: through from twisted.web quite unsatisfying. I've tried to formulate something like a preliminary policy in https://github.com/msdemlei/nevow/blob/master/doc/strings_and_bytes.rst, but I'm deeply unhappy about most of this.

Yeah, IRequest is an unfortunate interface. Work is ongoing to design a more thoughtful one and gradually migrate to it. I think https://github.com/wsanchez/txrequest https://github.com/wsanchez/txrequest is getting pretty complete, and from there the hope is that it will get integrated into Klein and then upstreamed into Twisted.

...which of course looks like a very long migration path, very likely stretching far beyond python2's EOL if my typical deployment is on Debian stable... I wonder if one couldn't arrange for inevow.IRequest to adapt to essentially this (perhaps with a bit more glue code to keep as compatible with existing code as possible).

...

Ideally, I'd like to fairly exclusively expose actual strings to

"actual" meaning text?

Yes: str.

...

(2) Is anyone at all interested in the livepages stuff (I'm not, and dropping it would certainly make porting and maintaining the package a lot easier)?

If you're going to drop livepage, is there any reason you're not just migrating to twisted.web.template? Is there any way t.w.t could be made into an easier migration path?

I have to admit I've not really investigated t.w.t in any detail; one thing that actually does matter -- because templates of my users will silently fail with it -- is the change of the XML namespace for the nevow elements and attributes. I suppose one could work around it on template load, though. Hm -- this might be an interesting alternative to keeping all of nevow. Of course, that still doesn't help against the insiduous breakage when suddenly code needs to deal with byte strings coming out of request.args.keys(), path components, and headers; that's a bit less of a direct concern for me since it's much less deployer-visible for my applications, but if I have to write some compatibility layer, why not have nevow be it?

...

(3) If I went ahead with this and actually tried to get a python3-nevow into Debian (which I'd really, really like to see): How is everyone's feeling about keeping the name nevow even if serious functional changes (such as dropping livepages) were to come about? [What I care about most is keeping the templating system including the render_ and data_ methods].

If you're going to do a fork, please come up with a new name. Using the existing one is confusing, it causes namespace conflicts (what happens when Nevow itself gets some upstream patches that port it to python 3 from someone else?) and it causes import conflicts.

Fair enough (a bit more on this in my response to Werner).

That said, if you are interested in porting Nevow the same way that Twisted is being ported (i.e. there's one set of sources, but `pip install` on python 3 only installs the modules ported thus far)

Ouch -- given my experiences while porting it would seem a major chore to keep python2 compatibity. But then I've not tried, and I may be wrong. If I actually wanted to follow https://twistedmatrix.com/trac/wiki/Plan/Python3 by the letter, that's certainly far beyond what I can reasonably commit in terms of resources.

then that would certainly be a way to invest the effort in just porting the parts that you need, and letting someone else port LivePage later :). I'd be happy to make you a maintainer if you're interested in doing this - not a lot of other activity on Nevow right now!

I'd certainly like to make the effort I invest as worthwhile as possible; but, in particular after the experience of porting, I feel I'm not nearly as familiar with nevow's codebase (and customs in the twisted ecosystem) as I should be to confidently work as maintainer. Before I commit to anything, let me try to work out if I can get nevow to the point where it doesn't cause extra porting pains in the python3 migration of dependent code. Thanks, Markus

On Wed, Jan 24, 2018, at 12:54 AM, Markus Demleitner wrote:

Hi Glyph,

On Tue, Jan 23, 2018 at 11:51:44PM -0800, Glyph wrote:

...

...

On Jan 23, 2018, at 2:03 AM, Markus Demleitner wrote: through from twisted.web quite unsatisfying. I've tried to formulate something like a preliminary policy in https://github.com/msdemlei/nevow/blob/master/doc/strings_and_bytes.rst, but I'm deeply unhappy about most of this.

Yeah, IRequest is an unfortunate interface. Work is ongoing to design a more thoughtful one and gradually migrate to it. I think https://github.com/wsanchez/txrequest https://github.com/wsanchez/txrequest is getting pretty complete, and from there the hope is that it will get integrated into Klein and then upstreamed into Twisted.

...which of course looks like a very long migration path, very likely stretching far beyond python2's EOL if my typical deployment is on Debian stable...

Shorter the more people we have working on it :-).

I wonder if one couldn't arrange for inevow.IRequest to adapt to essentially this (perhaps with a bit more glue code to keep as compatible with existing code as possible).

This would be great.

...

...

Ideally, I'd like to fairly exclusively expose actual strings to

"actual" meaning text?

Yes: str.

...

...

(2) Is anyone at all interested in the livepages stuff (I'm not, and dropping it would certainly make porting and maintaining the package a lot easier)?

If you're going to drop livepage, is there any reason you're not just migrating to twisted.web.template? Is there any way t.w.t could be made into an easier migration path?

I have to admit I've not really investigated t.w.t in any detail; one thing that actually does matter -- because templates of my users will silently fail with it -- is the change of the XML namespace for the nevow elements and attributes.

I suppose one could work around it on template load, though. Hm -- this might be an interesting alternative to keeping all of nevow.

I don't pretend to have a complete solution here - unfortunately I've ported very few projects *from* nevow *to* t.w.t so almost no work has gone into migration or compatibility. But I do think there's definitely something to investigate.

Of course, that still doesn't help against the insiduous breakage when suddenly code needs to deal with byte strings coming out of request.args.keys(), path components, and headers; that's a bit less of a direct concern for me since it's much less deployer-visible for my applications, but if I have to write some compatibility layer, why not have nevow be it?

This is largely about aligning as much effort as possible in one place :). Our goals are not exactly the same of course, you want Nevow+Py3, and most Klein interest right now is primarily around APIs rather than HTML, but this is exactly why extracting the request-processing fix to the lowest shared level is important :).

...

...

(3) If I went ahead with this and actually tried to get a python3-nevow into Debian (which I'd really, really like to see): How is everyone's feeling about keeping the name nevow even if serious functional changes (such as dropping livepages) were to come about? [What I care about most is keeping the templating system including the render_ and data_ methods].

If you're going to do a fork, please come up with a new name. Using the existing one is confusing, it causes namespace conflicts (what happens when Nevow itself gets some upstream patches that port it to python 3 from someone else?) and it causes import conflicts.

Fair enough (a bit more on this in my response to Werner).

I do hope we can figure out a way to work together! Although if a fork is necessary there will definitely be no hard feelings - the logistics on that are hard.

...

That said, if you are interested in porting Nevow the same way that Twisted is being ported (i.e. there's one set of sources, but `pip install` on python 3 only installs the modules ported thus far)

Ouch -- given my experiences while porting it would seem a major chore to keep python2 compatibity. But then I've not tried, and I may be wrong. If I actually wanted to follow https://twistedmatrix.com/trac/wiki/Plan/Python3 by the letter, that's certainly far beyond what I can reasonably commit in terms of resources.

Doing this at the very start of the project (Python2.5 + Python3.2) was agonizing; barely practical. But today (Python2.7 + Python3.4) is mostly quite pleasant.

...

then that would certainly be a way to invest the effort in just porting the parts that you need, and letting someone else port LivePage later :). I'd be happy to make you a maintainer if you're interested in doing this - not a lot of other activity on Nevow right now!

I'd certainly like to make the effort I invest as worthwhile as possible; but, in particular after the experience of porting, I feel I'm not nearly as familiar with nevow's codebase (and customs in the twisted ecosystem) as I should be to confidently work as maintainer.

There's really only one way to become familiar with these things - jump right in! :)

Before I commit to anything, let me try to work out if I can get nevow to the point where it doesn't cause extra porting pains in the python3 migration of dependent code.

I don't want to pressure you into any commitments you're uncomfortable with; as you said, just wanting to making the investment of your effort worthwhile. Sounds like your mind is in the right place though; let me know when you have completed your investigations :). -g

Hi Werner, On Thu, Jan 25, 2018 at 10:09:24AM -1000, Werner Thie wrote:

Fair enough, I assume you forked Nevow, is the code accessible publicly?

It's on https://github.com/msdemlei/nevow.git. -- Markus