[Twisted-Python] Progress report on splitting packages
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
Since Chris didn't send one in. Hope this is accurate. News was split off, backwards compat code is in, site is up: http://projects.twistedmatrix.com/lowdown/ Remaining tasks: 1. Release automation and making a lowdown release. 2. Buildbot support. 3. Linking to projects from main site, and updating main site to explain all this. Many thanks to Chris and JP for all their work on this. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote:
Remaining tasks:
Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all need infrastructure for having separate packages. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Michal Pasternak wrote:
Is there anything automatable about this? I'd like to automate as much as possible in the release-twisted script (Twisted/sandbox/radix/release-twisted). Soon I'll be splitting that out into something like twisted.python.release with only the project-specific stuff in the "script". It'd be cool if you could implement a step for doing freebsd builds (or stuff facilitating a freebsd build, anyway). -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/fdaa75758deee2fce225653d708b9557.jpg?s=120&d=mm&r=g)
Christopher Armstrong [Mon, Apr 19, 2004 at 03:38:46PM -0400]:
I think I could add there automatic generation of a Makefiles for a given project (for those, who don't know anything about fbsd ports/netbsd pkgsrc, here's an example - ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc/net/py-twisted/ ) It could be really clean at the *BSD side - all Twisted projects share some settings in Makefile.common, each project has a separate directory and Makefile with its 'personalized' settings. I could also add to your release script generation of diffs against current official fbsd/nbsd package sources (from fbsd/nbsd cvs repo); then the package could be built (on a test machine), and if the package builds okay, the release script could submit a problem report with the diffs needed to update the package source to GNATS database (see http://netbsd.org/Misc/send-pr.html), so *BSD developers could pick that PR and update the official sources. Automated building of binary package in case of FreeBSD / NetBSD is, IMO, not worth it (at Twisted project side). Why? When the package source is in official CVS repo, both NetBSD and FreeBSD teams take care of building the binary package for you (so you don't have to build Twisted for NetBSD/atari, for example). Also (IMO) BSD users tend rather not to use binary package available from an opensource project's site, they will get the one from official BSD FTP or compile by their own, using Ports or pkgsrc. I hope, that this is the kind of automation you meant. In case it is not - more clues please. To start the work I need URL to the tarballs with source releases of "core" twisted and at least one twisted product. -- m
![](https://secure.gravatar.com/avatar/c565aef253600553567b390d70c5876a.jpg?s=120&d=mm&r=g)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 For Windows, as long as there's a setup.py this is probably a candidate for just running a buildslave that does bdist_wininst and sticks the result on a fileserver. That's predicated on having a buildslave, of course. The somewhat sophisticated installer around Twisted "proper" is mainly to provide documentation and icons and uninstall and other approachability enhancements. Many (most?) of these subprojects don't have a wide enough scope to need such niceties. However, even if they did, I couldn't provide packaging for all of them :-( C Itamar Shtull-Trauring wrote: | On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote: | | |>Remaining tasks: | | | Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all | need infrastructure for having separate packages. | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3-nr1 (Windows 2000) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFAhGfX3A5SrXAiHQcRAkKEAJ9OqnjQmilbD1beqWg/X/hPvpgMbQCfTqzB vFW0xQXejrEUJKNIkbZPYKY= =OUzb -----END PGP SIGNATURE-----
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Mon, Apr 19, 2004 at 01:48:27PM -0400, Itamar Shtull-Trauring wrote:
I'm curious -- why did we choose to change the name from twisted.news? And even though the name is changing, why is the new package "lowdown" and not "twisted.lowdown"? I'm also interested to see that NNTP protocol support has also migrated to lowdown -- I was expecting that protocol implementations were going to stay in Twisted (or perhaps a twisted.protocols project, if that gets split off), and just the apps (i.e. direct subpackages of twisted, e.g. news, mail, web, names, conch, ...) would be seperated. So, I assume that this is the policy that other splits should follow? If so, this needs to be documented. I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently. -Andrew.
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Andrew Bennetts wrote:
It was decided that doing package names like this would be too problematic. Unfortunately, Python isn't as cool as Java. The name was changed to lowdown because "news" is a bad top-level package name and "twistednews" and "tnews" and various permutations so forth were also lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha Ha!
Nah. These split-offs are going to contain protocols and minimal framework for implementing apps, along with maybe a very simple implementation of an actual app.
So, I assume that this is the policy that other splits should follow? If so, this needs to be documented.
Let it be known that the split-offs will contain relevant protocols.
I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc). It would probably be a good idea to outline what exactly WILL be split out, and what will be contained in the splits, now, though. I am not authoritative on this, but I can tell you what I plan on working on soon (In the order that I've thought of them): <twisted-package> -> <canonical project name> <details> twisted.news -> "Lowdown" done. contains nntp. twisted.flow -> "Flow" (?) This one's easy, no protocols or anything. twisted.conch -> "Conch" This will be named "Conch", and contain conch as well as the SSH protocol implemantations. twisted.lore -> "Lore" (?) Easy. twisted.words, twisted.im -> "Twisted Words" (?) This'll be a single package. The packagename will probably be 'words'. I'm not *sure*, but I expect IRC and the other IM/chat protocols will go into this package. twisted.mail -> ??? SMTP, POP3, IMAP, *some* of the stuff from current twisted.mail package. twisted.names -> "Twisted Names" (?) + DNS proto twisted.xish -> No idea. No idea. I imagine this will just be made a part of the Jabber package, whatever that is. twisted.runner -> ??? Yeah, might as well get rid of this one too. twisted.pair -> ??? I don't know anything about this package. twisted.trial -> ??? This one is going to have to require some thought, since *all* the other packages depend on it, but I think that people do want to split it out. twisted.web -> ??? HTTP protocol implementation, and twisted.web. It is still undecided whether Nevow will be included in this package. Discuss it on the twisted-web mailing list. twisted.protocols.ftp + slyphon's sandbox -> ??? Need to split this out too. Others... There has been talk of leaving Spread in Twisted core. That seems like a fine idea to me, but I'm not the one to debate that with. Also enterprise, manhole. Dunno if we'll split those out. Cred, internet, application, python, persisted are definitely staying. We will still have a 'protocols' package left with a few protocols that don't have any framework for them implemented. HTH! -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Mary Gardiner wrote:
It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Christopher Armstrong wrote:
Let me retract this adjective from my response. Twisted Matrix Laboratories is far from nebulous -- we're a tight-knit group of hackers, just as the web site says. However, TML must no longer be defined as "those who hack on Twisted", since Twisted is no longer the only project run by Twisted Matrix Laboratories (well, there were Twisted Java and Twisted Emacs and some others before, but those were never very popular). Now, TML must be defined as "those who hack on TML projects". >:-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 19, 2004, at 10:19 PM, Christopher Armstrong wrote:
We absolutely need a policy document. The most attractive thing about Twisted, to many developers, is the extremely consistent coding style, packaging style, and central point of distribution. A project developed by 'the labs' also implicitly depends on some other core packages, and we need policy about things like running regression tests with multiple versions of dependencies in order that those core packages continue to support their clients correctly to even maintain parity with our current development process's quality level. Some of the policy will seem obvious, but other packages don't follow it. For example, we should make sure that everything uses distutils as its build process.
![](https://secure.gravatar.com/avatar/3bef09da3292c944649ffc28673df870.jpg?s=120&d=mm&r=g)
On Tue, Apr 20, 2004, Glyph Lefkowitz wrote:
I had a thought briefly yesterday that got lost in my helpful (off-list) exchange with radix about whether policy could keep cross-projects tasks like docs (and testing I suppose) sane: a policy doc would be most helpful for newcomers to the Twisted community who've decided their itch is a new subproject, because they want to add support for a blah protocol. Reading a policy doc would be a lot faster than absorbing policy on IRC for an outsider. Or at least so it appears to me.
If people want a policy doc, send me notes towards it and I'll happily turn it into something doc-like or a series thereof. I know there's coding standards already. -Mary
![](https://secure.gravatar.com/avatar/4f1bdb13d00c0dc4355e24349d61e107.jpg?s=120&d=mm&r=g)
On Tuesday 20 April 2004 05:27 pm, Mary Gardiner wrote:
protocol. Reading a policy doc would be a lot faster than absorbing policy on IRC for an outsider. Or at least so it appears to me.
Using IRC as a sole source for such information is insane. Many potential contributors and users won't want anything to do with IRC, and #twisted isn't comfy to "outsiders." (It's not that people on #twisted aren't nice or helpful; it's just that the basic culture is... strange.) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote:
I get the name, I just wanted to know why exactly it was chosen over other names :) What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.
Ok :)
Well, when I say "document", a post to the mailing list would do just fine. Just somewhere public that can be referred to later, if necessary. -Andrew.
![](https://secure.gravatar.com/avatar/15fa47f2847592672210af8a25cd1f34.jpg?s=120&d=mm&r=g)
On Apr 19, 2004, at 10:17 PM, Andrew Bennetts wrote:
What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.
I'd like to hear more about this. I think it is valuable to Twisted's image to keep all the projects inside a top-level namespace. While these may be different projects/apps, they aren't completely separate: they're all built by the same group of people, for use with the Twisted framework. Also I hope the latest release of all twisted products will be available as one large tarball/windows installer for people who don't want to think about it. Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me. So, can it not be done technically? James
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 10:57, James Y Knight wrote:
I think so, yes. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/4f1bdb13d00c0dc4355e24349d61e107.jpg?s=120&d=mm&r=g)
On Tuesday 20 April 2004 11:29 am, Itamar Shtull-Trauring wrote:
I've been quiet here lately, but am interested in this discussion since I've been working on packaging issues for Zope 3. One thing that's interesting to note is that distutils actually allows installing subpackages directly. For example, it's possible to install twisted.news separately from other twisted.* packages (let's ignore dependencies for the moment). The only interesting dependency for twisted.news is the top-level twisted package. If that's a "pure container" package (it contains only a trivial __init__.py), then it only needs to be released and packaged once, and it won't create dependency issues. Some twisted.core package would be a dependency for practical issues, but distutils will grow some support for versionable dependencies like that (there's preliminary, undocumented support in the Python 2.4 CVS). My point is that there's no technical reason not to use the "twisted." prefix; there may be other reasons not to, but that would be something for the Twisted developers and maintainers to determine. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
![](https://secure.gravatar.com/avatar/15fa47f2847592672210af8a25cd1f34.jpg?s=120&d=mm&r=g)
On Apr 20, 2004, at 12:02 PM, Fred L. Drake, Jr. wrote:
If it's the case that this is possible, I'd say "best naming scheme" would be: twisted.core.*coremodules twisted.product twisted.otherproduct. However, given that twisted.core isn't going to exist, I'd be almost as happy with: twisted.*coremodules tmlabs.product tmlabs.otherproduct So consider that last my vote. James
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
Three suggestions so far: 1. Separate namespaces for each project. twisted.internet conch 2. Keep all projects and core under twisted. twisted.internet twisted.conch 3. Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html) twisted.internet t.conch or tmlabs.conch -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 11:53, Itamar Shtull-Trauring wrote:
For the record, this is the one I want, using 'tmlabs' for non-core projects. So we'd have twisted.internet, tmlabs.news, tmlabs.conch etc.. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Itamar Shtull-Trauring wrote:
I like this. With 'tmlabs'. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
I've got a quick question, how will twisted projects manage dependencies? If this isn't handled nicely, it could really reduce the value of twisted (nothing more frustrating than downloading N packages, each one after an error of the package not being found). On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote: | >>News was split off, backwards compat code is in, site is up: | >>http://projects.twistedmatrix.com/lowdown/ | > | > | >I'm curious -- why did we choose to change the name from twisted.news? | >And | >even though the name is changing, why is the new package "lowdown" and | >not | >"twisted.lowdown"? | | It was decided that doing package names like this would be too | problematic. Unfortunately, Python isn't as cool as Java. The name was | changed to lowdown because "news" is a bad top-level package name and | "twistednews" and "tnews" and various permutations so forth were also | lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha This makes very little sense to me. In my humble opinion, it should stay 'twisted.news'. Calling it 'lowdown' beacuse a top level package of 'news' is taken is really counterproductive, finding good names is hard enough. I do see the value in breaking twisted into modules, so that a subset of twisted could be used. However, I don't see why these items can be named 'twisted.XXXX', perhaps the 'core' distribution should just have place-holder '__init__.py' file... /twisted/news/__init__.py raise NotImplementedError("""\ Download this package from http://projects.twistedmatrix.com/news/ """) Best, Clark
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Clark C. Evans wrote:
I encourage you to contribute packages for your favorite OS. Windows and Debian will be covered. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/aadb127e483a6e92716c510c77745880.jpg?s=120&d=mm&r=g)
Clark C. Evans wrote:
I am a boring fart and I always though that names should be as explicit as possible. Am I right to think that all the code will be related to applications/application frameworks using twisted ? In that case something like twistedapp (please do not take that name as the ultimate recommendation) would make more sense. As well I would saws that the organization should be like twistedapp.news.lowdown twistedapp.web.nevow twistedapp.ssh.counch [...] I have no idea of what are flow, lore, and some other modules and it seems that some of you do not know neither what is xish neither ! Should it be called twistedapp.jabber.xish ??? So the protocol is obvious from the module name and it is easy to add more application of the same protocol without any name clash. At that point it is clear that nevow is a web application, even if the name itself does not give the information away. Thomas
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:
One problem with leaving it as twisted.news in future releases as Andrew suggests in a later email is that we lose our emphasis on the fact that it's a separate release with separate versions. Assume following scenario - we have lowdown 0.1 and lowdown 0.2, which are API incompatible. We want to be very clear that lowdown 0.1 can still be used on latest Twisted core, that you don't have to go in lockstep. So, looking at code - is twisted.foo part of core twisted and thus probably pretty stable or is it a separate package? What expectations can I have of API stability? Partially this is just Python conventions. Separate packages go in separate namespaces. Possibly Java's system is better, but if it doesn't really match our users' expectations then we're just going to confuse them if we use it. Then again, maybe we're just being stupid and we should stick to Java-style "twisted.foo", so "twisted.news" with a "twisted.news.nntp". Who else thinks that's a good idea?
Yeah. Jabber will probably be in the Words pacakge, so xish will end up there (unless people tell us it is generally applicable).
When Glyph and I did the original break out document (Glyph, could you forward it to the list?) we decided this ought to stay in core, because everything depends on it, including core :) Also it requires the reactor, internally. It's not worth breaking out. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/aadb127e483a6e92716c510c77745880.jpg?s=120&d=mm&r=g)
Itamar Shtull-Trauring wrote:
Agreed, but it is a short term thing as all new users will have never known the "old way".
It is no worse than currently where you have to read the doc. As well, even stable interface can change. Nothing is set in stone.
So you mean that os.path is not pythonic ? os.path is one of the first include that every python user do !
Big repeat ... I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, .. twisted.mail.pop3.byme twisted.mail.pop3.byyou twisted.nntp.lowdown twisted.nntp.cnn poweredbytwisted.mail.pop3.byme poweredbytwisted.mail.pop3.byyou poweredbytwisted.nntp.lowdown poweredbytwisted.nntp.cnn would be valid names You could even have a prefered implementation accessible as from twisted.nntp import default as nntp or the like Thomas
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 09:34, Thomas Mangin wrote:
Not going to happen. twisted.news... maybe. twisted.news.lowdown or whatever is not going to be how we do it. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 20, 2004, at 9:34 AM, Thomas Mangin wrote:
This makes no sense to me, and as itamar said is unlikely to be accepted. First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols. Second, twisted.words, for example, is explicitly multiprotocol. I suppose if we wanted to be boring, we could call it twisted.multiprotocolchatserver.ircgateway, but IMHO "words" is catchier.
![](https://secure.gravatar.com/avatar/aadb127e483a6e92716c510c77745880.jpg?s=120&d=mm&r=g)
Glyph Lefkowitz wrote:
well, I am glad that you are dismissing idea on the basis that you do not understand it (sorry for the irony), but as I did not spend great time or care to try to be understood, I can not complain too much.
First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols.
Yes, I know. Why are you telling me that ? Is it as I should have written : basename.(organisational_unit.)*(protocol.)+(implementation)+ with the ()* and ()* having a regex like meaning and the words basename, organinational_unit, ... having one of the possible listed values. basename in [twisted, tmlabs,whatever_is_agreed] organisational_unit in [mail, im,web] protocol in [nntp, ldap, oscar] implementation in [lowdown, my_fancy_implementation_name] It seems that by the time I wrote all that Clark have posted what I am trying to say.
I do not really care if a name is catchy or not , "explicit is better than implicit". A good name is short enough to not be a pain when you code and explicit. I do not that fancy should not be a criteria of selection when it comes to technology. I was trying to say the a logical organisation should help the understanding of the twisted package. It seems that there is a clear need to change the structure of the code as it is now too large and need to be structured in a modular fashion. But at the same time it seems that most of the people on the list are speaking of rules to make it harder for new user to learn twisted. You guys, know twisted by heart. I do not intend to. I intend to use it for _RAD_, so learning it should be _quick_ If you do not understand me, please ask Michael Schneider or Clark Evans which seems to think like me but are able to express themselves correctly. Thomas
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
Radix/Itamar, It's clear that y'all have put a good deal of thought into this, so please take my feedback as just meandering thoughts. On Tue, Apr 20, 2004 at 08:36:07AM -0400, Itamar Shtull-Trauring wrote: | One problem with leaving it as twisted.news in future releases as Andrew | suggests in a later email is that we lose our emphasis on the fact that | it's a separate release with separate versions. Assume following | scenario - we have lowdown 0.1 and lowdown 0.2, which are API | incompatible. We want to be very clear that lowdown 0.1 can still be | used on latest Twisted core, that you don't have to go in lockstep. I suppose that lockstep has caused release management problems? How is this going to solve the problem? (ie, are you sure it is a problem, and that the solution fixes it?) | So, looking at code - is twisted.foo part of core twisted and thus | probably pretty stable or is it a separate package? What expectations | can I have of API stability? | | Partially this is just Python conventions. Separate packages go in | separate namespaces. Possibly Java's system is better, but if it doesn't | really match our users' expectations then we're just going to confuse | them if we use it. This seems primarly a documentation problem, that is, it is solved with a README document in the package. Personally, I've never had issues with modules being stable or not. Perhaps a package manager could use a list of packages that are stable? ie, "python setup.py --with-experimental install" In any case, I think this package management issue needs to be addressed no matter what your naming convention is. | > twisted.xish -> No idea. | > No idea. I imagine this will just be made a part of the Jabber | > package, whatever that is. | | Yeah. Jabber will probably be in the Words pacakge, so xish will end up | there (unless people tell us it is generally applicable). So, it will be in jabber (which I don't need?) till at some time it turns out that someone needs it, and then it will move somewhere else. Bings, Clark
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 10:19, Clark C. Evans wrote:
We want to allow people to use latest version of Twisted with old versions of other packages. So if the package you use has a major API rewrite, you can still use the old version with the latest and coolest core Twisted. Lets say we rewrite twisted.web in a non-backwards compatible way and release it inside Twisted 1.4. If you need a bug fix to twisted.internet. that's in 1.4 but want to use the old twisted.web, you're screwed.
In any case, I think this package management issue needs to be addressed no matter what your naming convention is.
Which issue?
Well, "which I don't need" implies you want xish standalone, so maybe it should be from the start... -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
On Tue, Apr 20, 2004 at 10:35:07AM -0400, Itamar Shtull-Trauring wrote: | We want to allow people to use latest version of Twisted with old | versions of other packages. So if the package you use has a major API | rewrite, you can still use the old version with the latest and coolest | core Twisted. | | Lets say we rewrite twisted.web in a non-backwards compatible way and | release it inside Twisted 1.4. If you need a bug fix to | twisted.internet. that's in 1.4 but want to use the old twisted.web, | you're screwed. Itamar, looks like you have a really good use case there -- in particular, I'd like to upgrade twisted "core" without updating twisted.web till it is more stable and I've had time to update my code dependencies. If the 'twisted' modules could stay within twisted.* I would prefer this, but this, alas is up to the core people... ;) Clark
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 20, 2004, at 10:35 AM, Itamar Shtull-Trauring wrote:
In particular, this was a release management problem for Divmod, because we've been making changes to the imap4 code (which is in the "protocols" package), that our code depends on, but we can't release all of Twisted at once fast enough to roll out a minor change, because going through the full QA process for the whole system takes too long. We are trying to decouple things more so that a project which requires bugfixes in twisted.web doesn't accidentally suck in in-progress changes to the reactor as well.
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
On Tue, Apr 20, 2004 at 04:21:49PM -0400, Glyph Lefkowitz wrote: | >We want to allow people to use latest version of Twisted with old | >versions of other packages. So if the package you use has a major API | >rewrite, you can still use the old version with the latest and coolest | >core Twisted. | | In particular, this was a release management problem for Divmod, | because we've been making changes to the imap4 code (which is in the | "protocols" package), that our code depends on, but we can't release | all of Twisted at once fast enough to roll out a minor change, because | going through the full QA process for the whole system takes too long. | We are trying to decouple things more so that a project which requires | bugfixes in twisted.web doesn't accidentally suck in in-progress | changes to the reactor as well. This seems the driving reason. What ever choice is made, it should be there to support this need. My perspective: 1. I think it is best to stick with one top level package name. Splitting twisted.* into twisted and tmlabs or whatever dillutes the value of the overall product and creates unnecessary distinctions. 2. Overall, naming should be as boring as possible so that what is being implemented is clear, 'ssh' is better than 'consh'. Also, while more than one implementation is the 'open source' way of doing things, more than one way in the twisted project, imho, is not a great idea (it dillutes the marketing value) 3. If release management is causing this distinction, I suggest making up sub-packages: twisted.core.python .trial .internet twisted.web.client .server .http .ftp twisted.mail.smtp .pop .imap .client .server twisted.rpc.jelly .spread <other RPC stuff> twisted.im.jabber .aol .client .server twisted.database. <database stuff> twisted.libs.xml .flow (other othgogonal non-protocol libraries) 4. In particular, I don't see that "enterprise" and "jelly+spread" are any more 'core' than jabber... ;) Sorry for being so boring. Clark
![](https://secure.gravatar.com/avatar/44e3f98a2f3b7213d5f14b558a849dd2.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 13:36, Itamar Shtull-Trauring wrote:
On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:
[snip]
I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful. I don't know enough about the xish package as a whole to comment on other parts - I only used (possibly wrongly) what I needed from it to get the example working. Cheers, Matt -- Matt Goodall, Pollenation Internet Ltd w: http://www.pollenation.net e: matt@pollenation.net Any views expressed are my own and do not necessarily reflect the views of my employer.
![](https://secure.gravatar.com/avatar/44e3f98a2f3b7213d5f14b558a849dd2.jpg?s=120&d=mm&r=g)
On Fri, 2004-04-23 at 04:14, Dave Smith wrote:
Wow, thanks Dave :). iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt@pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer.
![](https://secure.gravatar.com/avatar/44e3f98a2f3b7213d5f14b558a849dd2.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-27 at 05:42, Dave Smith wrote:
Yep, that's the one. I don't know if xish already supports this, I just realised I missed it from the tests. Cheers, Matt -- Matt Goodall, Pollenation Internet Ltd w: http://www.pollenation.net e: matt@pollenation.net Any views expressed are my own and do not necessarily reflect the views of my employer.
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 19, 2004, at 9:58 PM, Christopher Armstrong wrote:
It was decided that doing package names like this would be too problematic.
When and by whom? :) At PyCon we talked about (and, IIRC, decided about) using the 'pkgutil' module to keep different subpackages in different locations. I originally wanted to move things to top-level packages but itamar convinced me that forcing a change in the import structure as part of the split would be a bad idea.
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Glyph Lefkowitz wrote:
Well, see newer posts to this list. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/4f1bdb13d00c0dc4355e24349d61e107.jpg?s=120&d=mm&r=g)
On Tuesday 20 April 2004 03:56 pm, Glyph Lefkowitz wrote:
I've convinced myself that this is inadvisable at this point (in general, not just for Twisted). There is a bug in pkgutil that renders it much more painful than it should be. See: http://www.python.org/sf/935117 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote:
Remaining tasks:
Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all need infrastructure for having separate packages. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Michal Pasternak wrote:
Is there anything automatable about this? I'd like to automate as much as possible in the release-twisted script (Twisted/sandbox/radix/release-twisted). Soon I'll be splitting that out into something like twisted.python.release with only the project-specific stuff in the "script". It'd be cool if you could implement a step for doing freebsd builds (or stuff facilitating a freebsd build, anyway). -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/fdaa75758deee2fce225653d708b9557.jpg?s=120&d=mm&r=g)
Christopher Armstrong [Mon, Apr 19, 2004 at 03:38:46PM -0400]:
I think I could add there automatic generation of a Makefiles for a given project (for those, who don't know anything about fbsd ports/netbsd pkgsrc, here's an example - ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc/net/py-twisted/ ) It could be really clean at the *BSD side - all Twisted projects share some settings in Makefile.common, each project has a separate directory and Makefile with its 'personalized' settings. I could also add to your release script generation of diffs against current official fbsd/nbsd package sources (from fbsd/nbsd cvs repo); then the package could be built (on a test machine), and if the package builds okay, the release script could submit a problem report with the diffs needed to update the package source to GNATS database (see http://netbsd.org/Misc/send-pr.html), so *BSD developers could pick that PR and update the official sources. Automated building of binary package in case of FreeBSD / NetBSD is, IMO, not worth it (at Twisted project side). Why? When the package source is in official CVS repo, both NetBSD and FreeBSD teams take care of building the binary package for you (so you don't have to build Twisted for NetBSD/atari, for example). Also (IMO) BSD users tend rather not to use binary package available from an opensource project's site, they will get the one from official BSD FTP or compile by their own, using Ports or pkgsrc. I hope, that this is the kind of automation you meant. In case it is not - more clues please. To start the work I need URL to the tarballs with source releases of "core" twisted and at least one twisted product. -- m
![](https://secure.gravatar.com/avatar/c565aef253600553567b390d70c5876a.jpg?s=120&d=mm&r=g)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 For Windows, as long as there's a setup.py this is probably a candidate for just running a buildslave that does bdist_wininst and sticks the result on a fileserver. That's predicated on having a buildslave, of course. The somewhat sophisticated installer around Twisted "proper" is mainly to provide documentation and icons and uninstall and other approachability enhancements. Many (most?) of these subprojects don't have a wide enough scope to need such niceties. However, even if they did, I couldn't provide packaging for all of them :-( C Itamar Shtull-Trauring wrote: | On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote: | | |>Remaining tasks: | | | Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all | need infrastructure for having separate packages. | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3-nr1 (Windows 2000) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFAhGfX3A5SrXAiHQcRAkKEAJ9OqnjQmilbD1beqWg/X/hPvpgMbQCfTqzB vFW0xQXejrEUJKNIkbZPYKY= =OUzb -----END PGP SIGNATURE-----
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Mon, Apr 19, 2004 at 01:48:27PM -0400, Itamar Shtull-Trauring wrote:
I'm curious -- why did we choose to change the name from twisted.news? And even though the name is changing, why is the new package "lowdown" and not "twisted.lowdown"? I'm also interested to see that NNTP protocol support has also migrated to lowdown -- I was expecting that protocol implementations were going to stay in Twisted (or perhaps a twisted.protocols project, if that gets split off), and just the apps (i.e. direct subpackages of twisted, e.g. news, mail, web, names, conch, ...) would be seperated. So, I assume that this is the policy that other splits should follow? If so, this needs to be documented. I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently. -Andrew.
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Andrew Bennetts wrote:
It was decided that doing package names like this would be too problematic. Unfortunately, Python isn't as cool as Java. The name was changed to lowdown because "news" is a bad top-level package name and "twistednews" and "tnews" and various permutations so forth were also lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha Ha!
Nah. These split-offs are going to contain protocols and minimal framework for implementing apps, along with maybe a very simple implementation of an actual app.
So, I assume that this is the policy that other splits should follow? If so, this needs to be documented.
Let it be known that the split-offs will contain relevant protocols.
I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc). It would probably be a good idea to outline what exactly WILL be split out, and what will be contained in the splits, now, though. I am not authoritative on this, but I can tell you what I plan on working on soon (In the order that I've thought of them): <twisted-package> -> <canonical project name> <details> twisted.news -> "Lowdown" done. contains nntp. twisted.flow -> "Flow" (?) This one's easy, no protocols or anything. twisted.conch -> "Conch" This will be named "Conch", and contain conch as well as the SSH protocol implemantations. twisted.lore -> "Lore" (?) Easy. twisted.words, twisted.im -> "Twisted Words" (?) This'll be a single package. The packagename will probably be 'words'. I'm not *sure*, but I expect IRC and the other IM/chat protocols will go into this package. twisted.mail -> ??? SMTP, POP3, IMAP, *some* of the stuff from current twisted.mail package. twisted.names -> "Twisted Names" (?) + DNS proto twisted.xish -> No idea. No idea. I imagine this will just be made a part of the Jabber package, whatever that is. twisted.runner -> ??? Yeah, might as well get rid of this one too. twisted.pair -> ??? I don't know anything about this package. twisted.trial -> ??? This one is going to have to require some thought, since *all* the other packages depend on it, but I think that people do want to split it out. twisted.web -> ??? HTTP protocol implementation, and twisted.web. It is still undecided whether Nevow will be included in this package. Discuss it on the twisted-web mailing list. twisted.protocols.ftp + slyphon's sandbox -> ??? Need to split this out too. Others... There has been talk of leaving Spread in Twisted core. That seems like a fine idea to me, but I'm not the one to debate that with. Also enterprise, manhole. Dunno if we'll split those out. Cred, internet, application, python, persisted are definitely staying. We will still have a 'protocols' package left with a few protocols that don't have any framework for them implemented. HTH! -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Mary Gardiner wrote:
It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Christopher Armstrong wrote:
Let me retract this adjective from my response. Twisted Matrix Laboratories is far from nebulous -- we're a tight-knit group of hackers, just as the web site says. However, TML must no longer be defined as "those who hack on Twisted", since Twisted is no longer the only project run by Twisted Matrix Laboratories (well, there were Twisted Java and Twisted Emacs and some others before, but those were never very popular). Now, TML must be defined as "those who hack on TML projects". >:-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 19, 2004, at 10:19 PM, Christopher Armstrong wrote:
We absolutely need a policy document. The most attractive thing about Twisted, to many developers, is the extremely consistent coding style, packaging style, and central point of distribution. A project developed by 'the labs' also implicitly depends on some other core packages, and we need policy about things like running regression tests with multiple versions of dependencies in order that those core packages continue to support their clients correctly to even maintain parity with our current development process's quality level. Some of the policy will seem obvious, but other packages don't follow it. For example, we should make sure that everything uses distutils as its build process.
![](https://secure.gravatar.com/avatar/3bef09da3292c944649ffc28673df870.jpg?s=120&d=mm&r=g)
On Tue, Apr 20, 2004, Glyph Lefkowitz wrote:
I had a thought briefly yesterday that got lost in my helpful (off-list) exchange with radix about whether policy could keep cross-projects tasks like docs (and testing I suppose) sane: a policy doc would be most helpful for newcomers to the Twisted community who've decided their itch is a new subproject, because they want to add support for a blah protocol. Reading a policy doc would be a lot faster than absorbing policy on IRC for an outsider. Or at least so it appears to me.
If people want a policy doc, send me notes towards it and I'll happily turn it into something doc-like or a series thereof. I know there's coding standards already. -Mary
![](https://secure.gravatar.com/avatar/4f1bdb13d00c0dc4355e24349d61e107.jpg?s=120&d=mm&r=g)
On Tuesday 20 April 2004 05:27 pm, Mary Gardiner wrote:
protocol. Reading a policy doc would be a lot faster than absorbing policy on IRC for an outsider. Or at least so it appears to me.
Using IRC as a sole source for such information is insane. Many potential contributors and users won't want anything to do with IRC, and #twisted isn't comfy to "outsiders." (It's not that people on #twisted aren't nice or helpful; it's just that the basic culture is... strange.) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote:
I get the name, I just wanted to know why exactly it was chosen over other names :) What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.
Ok :)
Well, when I say "document", a post to the mailing list would do just fine. Just somewhere public that can be referred to later, if necessary. -Andrew.
![](https://secure.gravatar.com/avatar/15fa47f2847592672210af8a25cd1f34.jpg?s=120&d=mm&r=g)
On Apr 19, 2004, at 10:17 PM, Andrew Bennetts wrote:
What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.
I'd like to hear more about this. I think it is valuable to Twisted's image to keep all the projects inside a top-level namespace. While these may be different projects/apps, they aren't completely separate: they're all built by the same group of people, for use with the Twisted framework. Also I hope the latest release of all twisted products will be available as one large tarball/windows installer for people who don't want to think about it. Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me. So, can it not be done technically? James
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 10:57, James Y Knight wrote:
I think so, yes. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/4f1bdb13d00c0dc4355e24349d61e107.jpg?s=120&d=mm&r=g)
On Tuesday 20 April 2004 11:29 am, Itamar Shtull-Trauring wrote:
I've been quiet here lately, but am interested in this discussion since I've been working on packaging issues for Zope 3. One thing that's interesting to note is that distutils actually allows installing subpackages directly. For example, it's possible to install twisted.news separately from other twisted.* packages (let's ignore dependencies for the moment). The only interesting dependency for twisted.news is the top-level twisted package. If that's a "pure container" package (it contains only a trivial __init__.py), then it only needs to be released and packaged once, and it won't create dependency issues. Some twisted.core package would be a dependency for practical issues, but distutils will grow some support for versionable dependencies like that (there's preliminary, undocumented support in the Python 2.4 CVS). My point is that there's no technical reason not to use the "twisted." prefix; there may be other reasons not to, but that would be something for the Twisted developers and maintainers to determine. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
![](https://secure.gravatar.com/avatar/15fa47f2847592672210af8a25cd1f34.jpg?s=120&d=mm&r=g)
On Apr 20, 2004, at 12:02 PM, Fred L. Drake, Jr. wrote:
If it's the case that this is possible, I'd say "best naming scheme" would be: twisted.core.*coremodules twisted.product twisted.otherproduct. However, given that twisted.core isn't going to exist, I'd be almost as happy with: twisted.*coremodules tmlabs.product tmlabs.otherproduct So consider that last my vote. James
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
Three suggestions so far: 1. Separate namespaces for each project. twisted.internet conch 2. Keep all projects and core under twisted. twisted.internet twisted.conch 3. Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html) twisted.internet t.conch or tmlabs.conch -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 11:53, Itamar Shtull-Trauring wrote:
For the record, this is the one I want, using 'tmlabs' for non-core projects. So we'd have twisted.internet, tmlabs.news, tmlabs.conch etc.. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Itamar Shtull-Trauring wrote:
I like this. With 'tmlabs'. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
I've got a quick question, how will twisted projects manage dependencies? If this isn't handled nicely, it could really reduce the value of twisted (nothing more frustrating than downloading N packages, each one after an error of the package not being found). On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote: | >>News was split off, backwards compat code is in, site is up: | >>http://projects.twistedmatrix.com/lowdown/ | > | > | >I'm curious -- why did we choose to change the name from twisted.news? | >And | >even though the name is changing, why is the new package "lowdown" and | >not | >"twisted.lowdown"? | | It was decided that doing package names like this would be too | problematic. Unfortunately, Python isn't as cool as Java. The name was | changed to lowdown because "news" is a bad top-level package name and | "twistednews" and "tnews" and various permutations so forth were also | lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha This makes very little sense to me. In my humble opinion, it should stay 'twisted.news'. Calling it 'lowdown' beacuse a top level package of 'news' is taken is really counterproductive, finding good names is hard enough. I do see the value in breaking twisted into modules, so that a subset of twisted could be used. However, I don't see why these items can be named 'twisted.XXXX', perhaps the 'core' distribution should just have place-holder '__init__.py' file... /twisted/news/__init__.py raise NotImplementedError("""\ Download this package from http://projects.twistedmatrix.com/news/ """) Best, Clark
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Clark C. Evans wrote:
I encourage you to contribute packages for your favorite OS. Windows and Debian will be covered. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/aadb127e483a6e92716c510c77745880.jpg?s=120&d=mm&r=g)
Clark C. Evans wrote:
I am a boring fart and I always though that names should be as explicit as possible. Am I right to think that all the code will be related to applications/application frameworks using twisted ? In that case something like twistedapp (please do not take that name as the ultimate recommendation) would make more sense. As well I would saws that the organization should be like twistedapp.news.lowdown twistedapp.web.nevow twistedapp.ssh.counch [...] I have no idea of what are flow, lore, and some other modules and it seems that some of you do not know neither what is xish neither ! Should it be called twistedapp.jabber.xish ??? So the protocol is obvious from the module name and it is easy to add more application of the same protocol without any name clash. At that point it is clear that nevow is a web application, even if the name itself does not give the information away. Thomas
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:
One problem with leaving it as twisted.news in future releases as Andrew suggests in a later email is that we lose our emphasis on the fact that it's a separate release with separate versions. Assume following scenario - we have lowdown 0.1 and lowdown 0.2, which are API incompatible. We want to be very clear that lowdown 0.1 can still be used on latest Twisted core, that you don't have to go in lockstep. So, looking at code - is twisted.foo part of core twisted and thus probably pretty stable or is it a separate package? What expectations can I have of API stability? Partially this is just Python conventions. Separate packages go in separate namespaces. Possibly Java's system is better, but if it doesn't really match our users' expectations then we're just going to confuse them if we use it. Then again, maybe we're just being stupid and we should stick to Java-style "twisted.foo", so "twisted.news" with a "twisted.news.nntp". Who else thinks that's a good idea?
Yeah. Jabber will probably be in the Words pacakge, so xish will end up there (unless people tell us it is generally applicable).
When Glyph and I did the original break out document (Glyph, could you forward it to the list?) we decided this ought to stay in core, because everything depends on it, including core :) Also it requires the reactor, internally. It's not worth breaking out. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/aadb127e483a6e92716c510c77745880.jpg?s=120&d=mm&r=g)
Itamar Shtull-Trauring wrote:
Agreed, but it is a short term thing as all new users will have never known the "old way".
It is no worse than currently where you have to read the doc. As well, even stable interface can change. Nothing is set in stone.
So you mean that os.path is not pythonic ? os.path is one of the first include that every python user do !
Big repeat ... I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, .. twisted.mail.pop3.byme twisted.mail.pop3.byyou twisted.nntp.lowdown twisted.nntp.cnn poweredbytwisted.mail.pop3.byme poweredbytwisted.mail.pop3.byyou poweredbytwisted.nntp.lowdown poweredbytwisted.nntp.cnn would be valid names You could even have a prefered implementation accessible as from twisted.nntp import default as nntp or the like Thomas
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 09:34, Thomas Mangin wrote:
Not going to happen. twisted.news... maybe. twisted.news.lowdown or whatever is not going to be how we do it. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 20, 2004, at 9:34 AM, Thomas Mangin wrote:
This makes no sense to me, and as itamar said is unlikely to be accepted. First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols. Second, twisted.words, for example, is explicitly multiprotocol. I suppose if we wanted to be boring, we could call it twisted.multiprotocolchatserver.ircgateway, but IMHO "words" is catchier.
![](https://secure.gravatar.com/avatar/aadb127e483a6e92716c510c77745880.jpg?s=120&d=mm&r=g)
Glyph Lefkowitz wrote:
well, I am glad that you are dismissing idea on the basis that you do not understand it (sorry for the irony), but as I did not spend great time or care to try to be understood, I can not complain too much.
First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols.
Yes, I know. Why are you telling me that ? Is it as I should have written : basename.(organisational_unit.)*(protocol.)+(implementation)+ with the ()* and ()* having a regex like meaning and the words basename, organinational_unit, ... having one of the possible listed values. basename in [twisted, tmlabs,whatever_is_agreed] organisational_unit in [mail, im,web] protocol in [nntp, ldap, oscar] implementation in [lowdown, my_fancy_implementation_name] It seems that by the time I wrote all that Clark have posted what I am trying to say.
I do not really care if a name is catchy or not , "explicit is better than implicit". A good name is short enough to not be a pain when you code and explicit. I do not that fancy should not be a criteria of selection when it comes to technology. I was trying to say the a logical organisation should help the understanding of the twisted package. It seems that there is a clear need to change the structure of the code as it is now too large and need to be structured in a modular fashion. But at the same time it seems that most of the people on the list are speaking of rules to make it harder for new user to learn twisted. You guys, know twisted by heart. I do not intend to. I intend to use it for _RAD_, so learning it should be _quick_ If you do not understand me, please ask Michael Schneider or Clark Evans which seems to think like me but are able to express themselves correctly. Thomas
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
Radix/Itamar, It's clear that y'all have put a good deal of thought into this, so please take my feedback as just meandering thoughts. On Tue, Apr 20, 2004 at 08:36:07AM -0400, Itamar Shtull-Trauring wrote: | One problem with leaving it as twisted.news in future releases as Andrew | suggests in a later email is that we lose our emphasis on the fact that | it's a separate release with separate versions. Assume following | scenario - we have lowdown 0.1 and lowdown 0.2, which are API | incompatible. We want to be very clear that lowdown 0.1 can still be | used on latest Twisted core, that you don't have to go in lockstep. I suppose that lockstep has caused release management problems? How is this going to solve the problem? (ie, are you sure it is a problem, and that the solution fixes it?) | So, looking at code - is twisted.foo part of core twisted and thus | probably pretty stable or is it a separate package? What expectations | can I have of API stability? | | Partially this is just Python conventions. Separate packages go in | separate namespaces. Possibly Java's system is better, but if it doesn't | really match our users' expectations then we're just going to confuse | them if we use it. This seems primarly a documentation problem, that is, it is solved with a README document in the package. Personally, I've never had issues with modules being stable or not. Perhaps a package manager could use a list of packages that are stable? ie, "python setup.py --with-experimental install" In any case, I think this package management issue needs to be addressed no matter what your naming convention is. | > twisted.xish -> No idea. | > No idea. I imagine this will just be made a part of the Jabber | > package, whatever that is. | | Yeah. Jabber will probably be in the Words pacakge, so xish will end up | there (unless people tell us it is generally applicable). So, it will be in jabber (which I don't need?) till at some time it turns out that someone needs it, and then it will move somewhere else. Bings, Clark
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 10:19, Clark C. Evans wrote:
We want to allow people to use latest version of Twisted with old versions of other packages. So if the package you use has a major API rewrite, you can still use the old version with the latest and coolest core Twisted. Lets say we rewrite twisted.web in a non-backwards compatible way and release it inside Twisted 1.4. If you need a bug fix to twisted.internet. that's in 1.4 but want to use the old twisted.web, you're screwed.
In any case, I think this package management issue needs to be addressed no matter what your naming convention is.
Which issue?
Well, "which I don't need" implies you want xish standalone, so maybe it should be from the start... -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
On Tue, Apr 20, 2004 at 10:35:07AM -0400, Itamar Shtull-Trauring wrote: | We want to allow people to use latest version of Twisted with old | versions of other packages. So if the package you use has a major API | rewrite, you can still use the old version with the latest and coolest | core Twisted. | | Lets say we rewrite twisted.web in a non-backwards compatible way and | release it inside Twisted 1.4. If you need a bug fix to | twisted.internet. that's in 1.4 but want to use the old twisted.web, | you're screwed. Itamar, looks like you have a really good use case there -- in particular, I'd like to upgrade twisted "core" without updating twisted.web till it is more stable and I've had time to update my code dependencies. If the 'twisted' modules could stay within twisted.* I would prefer this, but this, alas is up to the core people... ;) Clark
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 20, 2004, at 10:35 AM, Itamar Shtull-Trauring wrote:
In particular, this was a release management problem for Divmod, because we've been making changes to the imap4 code (which is in the "protocols" package), that our code depends on, but we can't release all of Twisted at once fast enough to roll out a minor change, because going through the full QA process for the whole system takes too long. We are trying to decouple things more so that a project which requires bugfixes in twisted.web doesn't accidentally suck in in-progress changes to the reactor as well.
![](https://secure.gravatar.com/avatar/8ca35506ac08cebd833ab53032896c0b.jpg?s=120&d=mm&r=g)
On Tue, Apr 20, 2004 at 04:21:49PM -0400, Glyph Lefkowitz wrote: | >We want to allow people to use latest version of Twisted with old | >versions of other packages. So if the package you use has a major API | >rewrite, you can still use the old version with the latest and coolest | >core Twisted. | | In particular, this was a release management problem for Divmod, | because we've been making changes to the imap4 code (which is in the | "protocols" package), that our code depends on, but we can't release | all of Twisted at once fast enough to roll out a minor change, because | going through the full QA process for the whole system takes too long. | We are trying to decouple things more so that a project which requires | bugfixes in twisted.web doesn't accidentally suck in in-progress | changes to the reactor as well. This seems the driving reason. What ever choice is made, it should be there to support this need. My perspective: 1. I think it is best to stick with one top level package name. Splitting twisted.* into twisted and tmlabs or whatever dillutes the value of the overall product and creates unnecessary distinctions. 2. Overall, naming should be as boring as possible so that what is being implemented is clear, 'ssh' is better than 'consh'. Also, while more than one implementation is the 'open source' way of doing things, more than one way in the twisted project, imho, is not a great idea (it dillutes the marketing value) 3. If release management is causing this distinction, I suggest making up sub-packages: twisted.core.python .trial .internet twisted.web.client .server .http .ftp twisted.mail.smtp .pop .imap .client .server twisted.rpc.jelly .spread <other RPC stuff> twisted.im.jabber .aol .client .server twisted.database. <database stuff> twisted.libs.xml .flow (other othgogonal non-protocol libraries) 4. In particular, I don't see that "enterprise" and "jelly+spread" are any more 'core' than jabber... ;) Sorry for being so boring. Clark
![](https://secure.gravatar.com/avatar/44e3f98a2f3b7213d5f14b558a849dd2.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-20 at 13:36, Itamar Shtull-Trauring wrote:
On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:
[snip]
I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful. I don't know enough about the xish package as a whole to comment on other parts - I only used (possibly wrongly) what I needed from it to get the example working. Cheers, Matt -- Matt Goodall, Pollenation Internet Ltd w: http://www.pollenation.net e: matt@pollenation.net Any views expressed are my own and do not necessarily reflect the views of my employer.
![](https://secure.gravatar.com/avatar/44e3f98a2f3b7213d5f14b558a849dd2.jpg?s=120&d=mm&r=g)
On Fri, 2004-04-23 at 04:14, Dave Smith wrote:
Wow, thanks Dave :). iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt@pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer.
![](https://secure.gravatar.com/avatar/44e3f98a2f3b7213d5f14b558a849dd2.jpg?s=120&d=mm&r=g)
On Tue, 2004-04-27 at 05:42, Dave Smith wrote:
Yep, that's the one. I don't know if xish already supports this, I just realised I missed it from the tests. Cheers, Matt -- Matt Goodall, Pollenation Internet Ltd w: http://www.pollenation.net e: matt@pollenation.net Any views expressed are my own and do not necessarily reflect the views of my employer.
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Apr 19, 2004, at 9:58 PM, Christopher Armstrong wrote:
It was decided that doing package names like this would be too problematic.
When and by whom? :) At PyCon we talked about (and, IIRC, decided about) using the 'pkgutil' module to keep different subpackages in different locations. I originally wanted to move things to top-level packages but itamar convinced me that forcing a change in the import structure as part of the split would be a bad idea.
![](https://secure.gravatar.com/avatar/3a7e70f3ef2ad1539da42afc85c8d09d.jpg?s=120&d=mm&r=g)
Glyph Lefkowitz wrote:
Well, see newer posts to this list. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
![](https://secure.gravatar.com/avatar/4f1bdb13d00c0dc4355e24349d61e107.jpg?s=120&d=mm&r=g)
On Tuesday 20 April 2004 03:56 pm, Glyph Lefkowitz wrote:
I've convinced myself that this is inadvisable at this point (in general, not just for Twisted). There is a bug in pkgutil that renders it much more painful than it should be. See: http://www.python.org/sf/935117 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
participants (14)
-
Andrew Bennetts
-
Christopher Armstrong
-
Clark C. Evans
-
Cory Dodt
-
Dave Smith
-
David Reid
-
Fred L. Drake, Jr.
-
Glyph Lefkowitz
-
Itamar Shtull-Trauring
-
James Y Knight
-
Mary Gardiner
-
Matt Goodall
-
Michal Pasternak
-
Thomas Mangin