[Twisted-Python] Deprecating subproject packaging
Hi, I think that Twisted should deprecate subproject packaging -- they aren't uploaded to pip, don't support Python 3, I don't think we have docs for using them anywhere, and when included in downstream they usually only end up in confusion as various imports fail but some succeed due to only some subprojects being installed. Eventuall removing them means we have less code (and only one setup.py + setup3.py), less packages, and simplifies things. It does mean that you can't *just* get Twisted Web, but Twisted Core is so big (internet, trial, amp, python, logger, cred, protocols -- even positioning is in there) that the size difference is marginal. For example, uncompressed Twisted (without the docs) is 14MB, while Core alone is 7MB. Assuming that you want to use Twisted Web, you'll need Twisted-Web and Twisted-Names (because not using Twisted Names in web.http.Request is deprecated), which is ~9MB. The difference in tarball size is even less -- Twisted without docs being 2.2MB and Twisted Core + Web + Names being 1.4MB. Plus, I am not sure that there is "subprojects" really, anymore -- it's all just Twisted. So, I think we should: - Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice). - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release. - Migrate Twisted's source dist to being done by "python setup.py sdist" instead of manually assembling the tarball. This means we can generate the requisite PKG-INFO metadata automatically. This should be done for the main tarball (which is uploaded to PyPI) in maybe 15.2, but not the subproject tarballs (Twisted Core + rest) which will remain the same until they are removed in 16.0. - Replace manually uploading these distributions to PyPI and use twine to automate it for 15.2, when Twisted's tarball has the needed PKG-INFO (from being generated by sdist). Twine doesn't support uploading .msis, but as we now have Python wheels on PyPI, I think we can do without uploading MSIs, and just have them on the website. Keeping the projects topfiles as they are I think is good -- although I am not sure if having a NEWS in each subproject is useful. Maybe we should just have the one NEWS file once 16.0 lands, since the other ones wouldn't be packaged, and are therefore less useful. As for downstream: - Apple includes all of Twisted, so I don't think this will affect them. - Ubuntu and Debian package the subprojects separately. I guess they would have to make dummy packages that depend on the monolithic python-twisted instead. - Fedora (so Red Hat too?) have obsoleted the subprojects and point them at python-twisted instead, so they don't package the subprojects. - FreeBSD packages the Twisted subprojects separately, so they would have to do the same as Debian to migrate. Overall, I don't think this adversely affects downstream in the long term -- it does what Red Hat did as of Fedora 21, and it'll end up being less packages overall. I think this is a good idea and am willing to put the work in to implement this. Does anyone object to this? Is there a use case for keeping the subprojects packagable that I don't know about? - Hawkie
I think this is a good idea and am willing to put the work in to implement this. Does anyone object to this? Is there a use case for keeping the subprojects packagable that I don't know about?
+10 1. The split is due to Twisted not fitting on actual diskettes in ancient times. 2. It adds a frustrating amount of complexity. —h
On Apr 22, 2015, at 04:04, HawkOwl <hawkowl@atleastfornow.net <mailto:hawkowl@atleastfornow.net>> wrote:
So, I think we should:
- Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice). - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release.
If anything, this seems like too generous a timeline to me. We have the deprecation process to preserve API stability. Do the subproject setup.py's expose any API surface? "from twisted.web import setup" already appears to be an ImportError to me. I don't think that the subproject release process has helped anyone in a long time. Is there any reason why we can't just drop it from the next release, other than "the change might not land in time"? One word of warning - do try to reach out and contact the package maintainers directly, as they may not follow this list closely, and for some reason they seem keen on separated release tarballs. They might have a compelling use-case (though I doubt it), but at least they should be notified so that unified packages show up promptly after the release where we first issue them. -glyph
On 22 Apr 2015, at 19:46, Glyph <glyph@twistedmatrix.com> wrote:
On Apr 22, 2015, at 04:04, HawkOwl <hawkowl@atleastfornow.net> wrote:
So, I think we should:
- Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice). - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release.
If anything, this seems like too generous a timeline to me. We have the deprecation process to preserve API stability. Do the subproject setup.py's expose any API surface? "from twisted.web import setup" already appears to be an ImportError to me. I don't think that the subproject release process has helped anyone in a long time. Is there any reason why we can't just drop it from the next release, other than "the change might not land in time"?
One word of warning - do try to reach out and contact the package maintainers directly, as they may not follow this list closely, and for some reason they seem keen on separated release tarballs. They might have a compelling use-case (though I doubt it), but at least they should be notified so that unified packages show up promptly after the release where we first issue them.
-glyph
The timeline generosity is simply so that downstream maintainers can get their stuff sorted before we drop it. I'm not sure how fast they move, and if all the major ones do it before that, I guess we can drop it early. Since there doesn't seem to be any "omg no Hawkie what are you doing" from either you or JP, and otherwise "yes" from people I've asked/on this list so far, I'm guessing that I'll take the initiative and reach out to the downstream maintainers with some degree of "this is what we're doing now unless there's a very good reason". - Hawkie
So, I think we should:
- Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice). - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release.
I am ok with deprecating this ... but I hope that this will not discourage the effort to decouple the Twisted subprojects. I would like to see the subproject developed as independent projects... this should help enforcing the decoupling and would speed up the test suite. Right now if you make a change in twisted.positioning, buildbot will run all tests :( -- Adi Roiban
Agreed that this is a good idea, though I'd still like to see NEWS broken out by "subproject" (or maybe we should say "topic" now?). I don't care whether it's in the same file or not, but when a new Twisted release comes out I find reviewing changes much easier if I can think about one "topic" at a time. Otherwise +100. I'm annoyed every time I have to install a downstream Twisted package and it isn't _all_ of Twisted. On Wed, Apr 22, 2015 at 8:28 AM, Adi Roiban <adi@roiban.ro> wrote: > >>> So, I think we should: > >>> > >>> - Deprecate the subproject setup.pys starting with 15.2 and stop > packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, > 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases > + 1 major release notice). > >>> - Remove the support code for subprojects in twisted.python._release + > the release scripts in the 16.0 release. > > I am ok with deprecating this ... but I hope that this will not > discourage the effort to decouple the Twisted subprojects. > > I would like to see the subproject developed as independent > projects... this should help enforcing the decoupling and would speed > up the test suite. > > Right now if you make a change in twisted.positioning, buildbot will > run all tests :( > > -- > Adi Roiban > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- -- Kevin Horn
On 23 Apr 2015, at 01:37, Kevin Horn <kevin.horn@gmail.com> wrote:
Agreed that this is a good idea, though I'd still like to see NEWS broken out by "subproject" (or maybe we should say "topic" now?). I don't care whether it's in the same file or not, but when a new Twisted release comes out I find reviewing changes much easier if I can think about one "topic" at a time.
Do you mean keeping the NEWS file as-is currently? I think that's a good idea. It's a nice, logical grouping. I personally think that it should pruned down a little -- Twisted Runner & Pair don't really need their own sections, and Twisted News & Conch could probably be rolled into the core changelist as well. Not sure, I guess we'll come to that when there's tickets in review for it. :) - Hawkie
On Wed, Apr 22, 2015 at 2:31 PM, HawkOwl <hawkowl@atleastfornow.net> wrote:
On 23 Apr 2015, at 01:37, Kevin Horn <kevin.horn@gmail.com> wrote:
Agreed that this is a good idea, though I'd still like to see NEWS broken out by "subproject" (or maybe we should say "topic" now?). I don't care whether it's in the same file or not, but when a new Twisted release comes out I find reviewing changes much easier if I can think about one "topic" at a time.
Do you mean keeping the NEWS file as-is currently? I think that's a good idea. It's a nice, logical grouping.
I personally think that it should pruned down a little -- Twisted Runner & Pair don't really need their own sections, and Twisted News & Conch could probably be rolled into the core changelist as well. Not sure, I guess we'll come to that when there's tickets in review for it. :)
Yes, pretty much this. Runner and Pair could maybe be rolled into Core, but I think News and Conch should remain separate. -- Kevin Horn
On Apr 22, 2015, at 09:28, Adi Roiban <adi@roiban.ro> wrote:
So, I think we should:
- Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice). - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release.
I am ok with deprecating this ... but I hope that this will not discourage the effort to decouple the Twisted subprojects.
There is not really any such effort underway. However, if anything, this change would make such an effort easier. If we want to release the projects separately, we should maintain them separately; the lesson of this "subproject" split is that what users want, to perceive that projects are more "lightweight", is a different "front door" to each project - different websites, different documentation indexes, different bug trackers - and not necessarily just a smaller download. I'm pretty sure that if we split everything up but the actual releases we'd have better luck convincing people twisted is not hard to understand. One immediate thing I think we should decouple, though, are our native dependencies. IOCPReactor should remain in Twisted, but iocpsupport should be a separate, optional dependency, available as an extra. Similarly, _initgroups, sendmsg, portmap, and raiser should all be optional dependencies, so that Twisted itself can be a pure wheel, and you can opt in to enhanced functionality by installing extras.
I would like to see the subproject developed as independent projects... this should help enforcing the decoupling and would speed up the test suite.
Speeding up the test suite should be accomplished by actually fixing and optimizing the tests, not by chopping big sections of it off. The amount of tests we have right now can run very quickly; but there are two immediate problems: 1. we are not leveraging twistd -j, and since we have no buildbots that run it, we keep making changes that break it for our test suite. We should introduce a builder that runs the tests in parallel. 2. many of the tests still return Deferreds when they should not, spin the reactor, wait for real timeouts, and so on. we should endeavor to reduce the number of real-IO tests as much as possible.
Right now if you make a change in twisted.positioning, buildbot will run all tests :(
Maybe we could speed up twisted.positioning's tests without sacrificing quality, but we would need to continue to run twisted.positioning's tests for every change to twisted core regardless, even if they were separate projects, or we would be reducing the amount of integration coverage received by twisted core. -glyph
On 23 Apr 2015, at 08:01, Glyph <glyph@twistedmatrix.com> wrote:
One immediate thing I think we should decouple, though, are our native dependencies. IOCPReactor should remain in Twisted, but iocpsupport should be a separate, optional dependency, available as an extra. Similarly, _initgroups, sendmsg, portmap, and raiser should all be optional dependencies, so that Twisted itself can be a pure wheel, and you can opt in to enhanced functionality by installing extras.
Also worth noting, pulling twisted.python.constants out into a separate package has a PR open so that part of this can become reality: https://github.com/twisted/constantly/pull/1 - Hawkie
On 08:04 am, hawkowl@atleastfornow.net wrote:
Hi,
I think that Twisted should deprecate subproject packaging
You probably won't hear much if any disagreement with this proposal on this list. The sub-projects were never for people who like and are into Twisted. They were a tool to help convince people who don't use Twisted yet that it's not an unapproachable monolith. We could discuss whether they have been effective in this regard but if you want to hear from people who *might* actually benefit from the split, you probably need to talk to people who aren't using Twisted. Jean-Paul
Hi, As discussed in the ML thread (http://twistedmatrix.com/pipermail/twisted-python/2015-April/029386.html), and with general "yes"ness, I think I should make it official -- Twisted subproject packages are going away. Downstream packagers that package Twisted as multiple subprojects (core+[mail,conch,names,etc]) should replace them with one Twisted package, from the source tarball released under `http://twistedmatrix.com/Releases/Twisted/{VERSION}/Twisted-{VERSION}.tar.bz2`. The roadmap for this is to completely drop the subproject packages at 16.0 at the latest, to give downstream packagers time to update. If the major distributions that package Twisted by using the subprojects[1] (FreeBSD, Debian, Gentoo) migrate to the one Twisted package before then, there is no use releasing the subproject packages and they will be dropped that release. - Hawkie [1] Fedora and Arch already just have the one python-twisted package.
participants (7)
-
Adi Roiban
-
exarkun@twistedmatrix.com
-
Glyph
-
HawkOwl
-
Hynek Schlawack
-
Kevin Horn
-
Phil Mayers