Unfortunately, I've been out of the loop on Twisted development for some time. During the course of the PyCon sprints, I've been catching up with what's been going on and trying to do some planning for future directions in development. Yesterday Itamar and I had our quarterly screaming argument about obscure details, and it didn't come to blows, which I believe means that spring is right around the corner! The major issue we've been discussing is splitting up Twisted into multiple packages. Twisted has been encountering some growing pains, due to a mismatch between the way it's developed and the way it's distributed. We (T.M.Labs) are really a community of developers working on several interrelated projects with different goals, stability, and release cycles, but we are distributing one monolithic system, with code of vastly different qualities in the same tarball and sometimes even in the same package. The primary problem that this leads to is revision coupling. This is probably old news to anyone trying to deploy a large system which relies on any part of Twisted besides the base reactor (and maybe twisted.web). Scenarios like this happen all the time: - Twisted v.N.M.0 is released. - Project P uses twisted components X and Y. (Generally X and Y are marked 'Unstable' or 'Semi-Stable', but are very useful or heavily marketed or otherwise convince unsuspecting users to trust their stability.) - Project Q uses twisted components Y and Z. - Project P v. 1 is released. - Project Q v. 1 is released. - Project P gives useful feedback to twisted developer A who then fixes several bugs in component X which are critical to the functioning of the as-yet-unreleased P v.2 - Twisted developer B working on project Q commits substantial "improvements" to component Y, which change the functionality in a way that breaks project X. - Radix prepares an alpha/bugfix release, v.N.M.1 At this point, Project Q is perfectly happy, because the most recent alpha works for their code. However, Project P is very unhappy, because there is no release which includes their bugfix but does NOT include code which forces them to upgrade some functionality in an a way which is incompatible with the previous micro-version release of Twisted which their users presumably already have installed. The release notes have to be very specific, and because nobody reads them anyway, a lot of confusion results. While there are problems with splitting Twisted into multiple projects, I think that the time has come to do so. Originally the feedback I heard from most users was that it was great to have lots of different packages available so readily; the "batteries included" philosophy does have a lot going for it. Now, most of what I'm hearing is complaints about the above scenario, not the least of which from my own employer :-). We are more often in the role of project Q than project P (although we have been in both more than once) and I'd prefer not to have a few projects who employ active Twisted developers control the release cycle based on their requirements. Because splitting the project involves moving some code around, we should take the opportunity to clean up another problem: our code has no effective separation between interface and implementation. Within the project this is fine, because we can easily refactor any uses of deprecated APIs. While we aren't going to break the existing package-structure interfaces (or even deprecate them, for the time being), this means that in the future, imports for twisted projects will probably look like: from twisted_core import reactor from twisted_web import Resource The way this will be implemented, a top-level "twisted_core" module (or possibly, package, but I would like as little nesting as possible here) will import explicitly public names from a _twisted_core implementation package. For the forseeable future, the twisted.* package hierarchy will continue to work for imports; however, considering how informally specified this interface has been, I imagine improved documentation will emerge for the newer interface packages. Personally I wanted to call these modules "twisted_web_1", and I also wanted to talk about versioning here, but amidst strenuous objections from Anthony Baxter and Itamar ST I will refrain :). Suffice it to say that I believe that regardless of many potential strategies available for versioning, a separate module for interface vs. implementation will make it possible to determine what is public and usable from a deployed application using the Python interactive prompt or reading the code. Since many packages don't have stability declarations anyway, I think this will result in many fewer unintentional uses of private code. The first package that we are going to try this on is twisted.web, because there is a major rewrite pending integration of all the fixes and redesign that has happened in the Nevow package. If this goes well, other packages will use the same naming conventions. In any case, we will be migrating packages out of the Twisted core. for example: lore, conch, manhole, and mail. As part of this process we hope to improve the website to reflect the name "Twisted" as primarily a part of "Twisted Matrix Labs", the group of developers working on these projects. Similar to tigris.org, TML is "a mid-sized open source community focused on building better tools for networked application development". Package maintainers: each project should have its own sub-site, so consider a project description and any other documentation you might want on that site. As far as our development process, we will be keeping all the code in one subversion repository and still running BuildBot over the entire system in a way similar to how we're doing it now. There are lots of possible improvements to the process, all the way up to an independent package management system using distutils: please, don't recommend any of these. We don't want to make a huge number of changes at the same time, and while this moving / renaming is fairly mechanical, it will touch almost every file in the world; we need to make sure it all works before we begin adding features based on it.