Itamar Shtull-Trauring [Mon, Apr 19, 2004 at 02:03:51PM -0400]:

On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote:

...

Remaining tasks:

Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all need infrastructure for having separate packages.

I can create & test packages for FreeBSD Ports & NetBSD Packages Collection, just send me what to do. I can also provide binary packages for FreeBSD 5.2.1 at the moment.

Christopher Armstrong [Mon, Apr 19, 2004 at 03:38:46PM -0400]:

Michal Pasternak wrote:

...

I can create & test packages for FreeBSD Ports & NetBSD Packages Collection, just send me what to do. I can also provide binary packages for FreeBSD 5.2.1 at the moment.

Is there anything automatable about this? I'd like to automate as much as possible in the release-twisted script (Twisted/sandbox/radix/release-twisted).

I think I could add there automatic generation of a Makefiles for a given project (for those, who don't know anything about fbsd ports/netbsd pkgsrc, here's an example - ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc/net/py-twisted/ ) It could be really clean at the *BSD side - all Twisted projects share some settings in Makefile.common, each project has a separate directory and Makefile with its 'personalized' settings. I could also add to your release script generation of diffs against current official fbsd/nbsd package sources (from fbsd/nbsd cvs repo); then the package could be built (on a test machine), and if the package builds okay, the release script could submit a problem report with the diffs needed to update the package source to GNATS database (see http://netbsd.org/Misc/send-pr.html), so *BSD developers could pick that PR and update the official sources. Automated building of binary package in case of FreeBSD / NetBSD is, IMO, not worth it (at Twisted project side). Why? When the package source is in official CVS repo, both NetBSD and FreeBSD teams take care of building the binary package for you (so you don't have to build Twisted for NetBSD/atari, for example). Also (IMO) BSD users tend rather not to use binary package available from an opensource project's site, they will get the one from official BSD FTP or compile by their own, using Ports or pkgsrc. I hope, that this is the kind of automation you meant. In case it is not - more clues please. To start the work I need URL to the tarballs with source releases of "core" twisted and at least one twisted product. -- m

On Mon, Apr 19, 2004 at 01:48:27PM -0400, Itamar Shtull-Trauring wrote:

Since Chris didn't send one in. Hope this is accurate.

News was split off, backwards compat code is in, site is up: http://projects.twistedmatrix.com/lowdown/

I'm curious -- why did we choose to change the name from twisted.news? And even though the name is changing, why is the new package "lowdown" and not "twisted.lowdown"? I'm also interested to see that NNTP protocol support has also migrated to lowdown -- I was expecting that protocol implementations were going to stay in Twisted (or perhaps a twisted.protocols project, if that gets split off), and just the apps (i.e. direct subpackages of twisted, e.g. news, mail, web, names, conch, ...) would be seperated. So, I assume that this is the policy that other splits should follow? If so, this needs to be documented. I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently. -Andrew.

On Mon, Apr 19, 2004, Christopher Armstrong wrote:

Andrew Bennetts wrote:

...

I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.

I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc).

What about a document for people who in future, want to add new twisted subprojects, from scratch? Some of the same policy will be relevant to them, and that wouldn't be a transitory document. -Mary

Mary Gardiner wrote:

On Mon, Apr 19, 2004, Christopher Armstrong wrote:

...

Andrew Bennetts wrote:

...

I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.

I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc).

What about a document for people who in future, want to add new twisted subprojects, from scratch? Some of the same policy will be relevant to them, and that wouldn't be a transitory document.

It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/

I've got a quick question, how will twisted projects manage dependencies? If this isn't handled nicely, it could really reduce the value of twisted (nothing more frustrating than downloading N packages, each one after an error of the package not being found). On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote: | >>News was split off, backwards compat code is in, site is up: | >>http://projects.twistedmatrix.com/lowdown/ | > | > | >I'm curious -- why did we choose to change the name from twisted.news? | >And | >even though the name is changing, why is the new package "lowdown" and | >not | >"twisted.lowdown"? | | It was decided that doing package names like this would be too | problematic. Unfortunately, Python isn't as cool as Java. The name was | changed to lowdown because "news" is a bad top-level package name and | "twistednews" and "tnews" and various permutations so forth were also | lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha This makes very little sense to me. In my humble opinion, it should stay 'twisted.news'. Calling it 'lowdown' beacuse a top level package of 'news' is taken is really counterproductive, finding good names is hard enough. I do see the value in breaking twisted into modules, so that a subset of twisted could be used. However, I don't see why these items can be named 'twisted.XXXX', perhaps the 'core' distribution should just have place-holder '__init__.py' file... /twisted/news/__init__.py raise NotImplementedError("""\ Download this package from http://projects.twistedmatrix.com/news/ """) Best, Clark

Clark C. Evans wrote:

This makes very little sense to me. In my humble opinion, it should stay 'twisted.news'. Calling it 'lowdown' beacuse a top level package of 'news' is taken is really counterproductive, finding good names is hard enough. I do see the value in breaking twisted into modules, so that a subset of twisted could be used. However, I don't see why these items can be named 'twisted.XXXX', perhaps the 'core' distribution should just have place-holder '__init__.py' file...

/twisted/news/__init__.py

raise NotImplementedError("""\ Download this package from http://projects.twistedmatrix.com/news/ """)

I am a boring fart and I always though that names should be as explicit as possible. Am I right to think that all the code will be related to applications/application frameworks using twisted ? In that case something like twistedapp (please do not take that name as the ultimate recommendation) would make more sense. As well I would saws that the organization should be like twistedapp.news.lowdown twistedapp.web.nevow twistedapp.ssh.counch [...] I have no idea of what are flow, lore, and some other modules and it seems that some of you do not know neither what is xish neither ! Should it be called twistedapp.jabber.xish ??? So the protocol is obvious from the module name and it is easy to add more application of the same protocol without any name clash. At that point it is clear that nevow is a web application, even if the name itself does not give the information away. Thomas

Itamar Shtull-Trauring wrote:

One problem with leaving it as twisted.news in future releases as Andrew suggests in a later email is that we lose our emphasis on the fact that it's a separate release with separate versions. Assume following scenario - we have lowdown 0.1 and lowdown 0.2, which are API incompatible. We want to be very clear that lowdown 0.1 can still be used on latest Twisted core, that you don't have to go in lockstep.

Agreed, but it is a short term thing as all new users will have never known the "old way".

So, looking at code - is twisted.foo part of core twisted and thus probably pretty stable or is it a separate package? What expectations can I have of API stability?

It is no worse than currently where you have to read the doc. As well, even stable interface can change. Nothing is set in stone.

Partially this is just Python conventions. Separate packages go in separate namespaces. Possibly Java's system is better, but if it doesn't really match our users' expectations then we're just going to confuse them if we use it.

So you mean that os.path is not pythonic ? os.path is one of the first include that every python user do !

Then again, maybe we're just being stupid and we should stick to Java-style "twisted.foo", so "twisted.news" with a "twisted.news.nntp". Who else thinks that's a good idea?

Big repeat ... I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, .. twisted.mail.pop3.byme twisted.mail.pop3.byyou twisted.nntp.lowdown twisted.nntp.cnn poweredbytwisted.mail.pop3.byme poweredbytwisted.mail.pop3.byyou poweredbytwisted.nntp.lowdown poweredbytwisted.nntp.cnn would be valid names You could even have a prefered implementation accessible as from twisted.nntp import default as nntp or the like Thomas

Radix/Itamar, It's clear that y'all have put a good deal of thought into this, so please take my feedback as just meandering thoughts. On Tue, Apr 20, 2004 at 08:36:07AM -0400, Itamar Shtull-Trauring wrote: | One problem with leaving it as twisted.news in future releases as Andrew | suggests in a later email is that we lose our emphasis on the fact that | it's a separate release with separate versions. Assume following | scenario - we have lowdown 0.1 and lowdown 0.2, which are API | incompatible. We want to be very clear that lowdown 0.1 can still be | used on latest Twisted core, that you don't have to go in lockstep. I suppose that lockstep has caused release management problems? How is this going to solve the problem? (ie, are you sure it is a problem, and that the solution fixes it?) | So, looking at code - is twisted.foo part of core twisted and thus | probably pretty stable or is it a separate package? What expectations | can I have of API stability? | | Partially this is just Python conventions. Separate packages go in | separate namespaces. Possibly Java's system is better, but if it doesn't | really match our users' expectations then we're just going to confuse | them if we use it. This seems primarly a documentation problem, that is, it is solved with a README document in the package. Personally, I've never had issues with modules being stable or not. Perhaps a package manager could use a list of packages that are stable? ie, "python setup.py --with-experimental install" In any case, I think this package management issue needs to be addressed no matter what your naming convention is. | > twisted.xish -> No idea. | > No idea. I imagine this will just be made a part of the Jabber | > package, whatever that is. | | Yeah. Jabber will probably be in the Words pacakge, so xish will end up | there (unless people tell us it is generally applicable). So, it will be in jabber (which I don't need?) till at some time it turns out that someone needs it, and then it will move somewhere else. Bings, Clark

On Apr 19, 2004, at 10:17 PM, Andrew Bennetts wrote:

What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.

I'd like to hear more about this. I think it is valuable to Twisted's image to keep all the projects inside a top-level namespace. While these may be different projects/apps, they aren't completely separate: they're all built by the same group of people, for use with the Twisted framework. Also I hope the latest release of all twisted products will be available as one large tarball/windows installer for people who don't want to think about it. Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me. So, can it not be done technically? James

On Tue, Apr 20, 2004 at 10:35:07AM -0400, Itamar Shtull-Trauring wrote: | We want to allow people to use latest version of Twisted with old | versions of other packages. So if the package you use has a major API | rewrite, you can still use the old version with the latest and coolest | core Twisted. | | Lets say we rewrite twisted.web in a non-backwards compatible way and | release it inside Twisted 1.4. If you need a bug fix to | twisted.internet. that's in 1.4 but want to use the old twisted.web, | you're screwed. Itamar, looks like you have a really good use case there -- in particular, I'd like to upgrade twisted "core" without updating twisted.web till it is more stable and I've had time to update my code dependencies. If the 'twisted' modules could stay within twisted.* I would prefer this, but this, alas is up to the core people... ;) Clark

On Tuesday 20 April 2004 11:29 am, Itamar Shtull-Trauring wrote:

On Tue, 2004-04-20 at 10:57, James Y Knight wrote:

...

Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me.

So, can it not be done technically?

I think so, yes.

I've been quiet here lately, but am interested in this discussion since I've been working on packaging issues for Zope 3. One thing that's interesting to note is that distutils actually allows installing subpackages directly. For example, it's possible to install twisted.news separately from other twisted.* packages (let's ignore dependencies for the moment). The only interesting dependency for twisted.news is the top-level twisted package. If that's a "pure container" package (it contains only a trivial __init__.py), then it only needs to be released and packaged once, and it won't create dependency issues. Some twisted.core package would be a dependency for practical issues, but distutils will grow some support for versionable dependencies like that (there's preliminary, undocumented support in the Python 2.4 CVS). My point is that there's no technical reason not to use the "twisted." prefix; there may be other reasons not to, but that would be something for the Twisted developers and maintainers to determine. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

On Apr 20, 2004, at 12:02 PM, Fred L. Drake, Jr. wrote:

On Tuesday 20 April 2004 11:29 am, Itamar Shtull-Trauring wrote:

...

On Tue, 2004-04-20 at 10:57, James Y Knight wrote:

...

Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me.

So, can it not be done technically?

I think so, yes.

One thing that's interesting to note is that distutils actually allows installing subpackages directly. For example, it's possible to install twisted.news separately from other twisted.* packages (let's ignore dependencies for the moment).

If it's the case that this is possible, I'd say "best naming scheme" would be: twisted.core.*coremodules twisted.product twisted.otherproduct. However, given that twisted.core isn't going to exist, I'd be almost as happy with: twisted.*coremodules tmlabs.product tmlabs.otherproduct So consider that last my vote. James

On Tue, 2004-04-20 at 13:36, Itamar Shtull-Trauring wrote:

On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:

[snip]

...

twisted.xish -> No idea. No idea. I imagine this will just be made a part of the Jabber package, whatever that is.

Yeah. Jabber will probably be in the Words pacakge, so xish will end up there (unless people tell us it is generally applicable).

I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful. I don't know enough about the xish package as a whole to comment on other parts - I only used (possibly wrongly) what I needed from it to get the example working. Cheers, Matt -- Matt Goodall, Pollenation Internet Ltd w: http://www.pollenation.net e: matt@pollenation.net Any views expressed are my own and do not necessarily reflect the views of my employer.

On Apr 19, 2004, at 10:19 PM, Christopher Armstrong wrote:

It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous.

We absolutely need a policy document. The most attractive thing about Twisted, to many developers, is the extremely consistent coding style, packaging style, and central point of distribution. A project developed by 'the labs' also implicitly depends on some other core packages, and we need policy about things like running regression tests with multiple versions of dependencies in order that those core packages continue to support their clients correctly to even maintain parity with our current development process's quality level. Some of the policy will seem obvious, but other packages don't follow it. For example, we should make sure that everything uses distutils as its build process.

On Apr 20, 2004, at 9:34 AM, Thomas Mangin wrote:

I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, ..

This makes no sense to me, and as itamar said is unlikely to be accepted. First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols. Second, twisted.words, for example, is explicitly multiprotocol. I suppose if we wanted to be boring, we could call it twisted.multiprotocolchatserver.ircgateway, but IMHO "words" is catchier.

On Tuesday 20 April 2004 03:56 pm, Glyph Lefkowitz wrote:

When and by whom? :) At PyCon we talked about (and, IIRC, decided about) using the 'pkgutil' module to keep different subpackages in different locations.

I've convinced myself that this is inadvisable at this point (in general, not just for Twisted). There is a bug in pkgutil that renders it much more painful than it should be. See: http://www.python.org/sf/935117 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

On Tue, Apr 20, 2004, Itamar Shtull-Trauring wrote:

3. Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html)

twisted.internet t.conch or tmlabs.conch

I like 3. -Mary

On Tue, Apr 20, 2004 at 04:21:49PM -0400, Glyph Lefkowitz wrote: | >We want to allow people to use latest version of Twisted with old | >versions of other packages. So if the package you use has a major API | >rewrite, you can still use the old version with the latest and coolest | >core Twisted. | | In particular, this was a release management problem for Divmod, | because we've been making changes to the imap4 code (which is in the | "protocols" package), that our code depends on, but we can't release | all of Twisted at once fast enough to roll out a minor change, because | going through the full QA process for the whole system takes too long. | We are trying to decouple things more so that a project which requires | bugfixes in twisted.web doesn't accidentally suck in in-progress | changes to the reactor as well. This seems the driving reason. What ever choice is made, it should be there to support this need. My perspective: 1. I think it is best to stick with one top level package name. Splitting twisted.* into twisted and tmlabs or whatever dillutes the value of the overall product and creates unnecessary distinctions. 2. Overall, naming should be as boring as possible so that what is being implemented is clear, 'ssh' is better than 'consh'. Also, while more than one implementation is the 'open source' way of doing things, more than one way in the twisted project, imho, is not a great idea (it dillutes the marketing value) 3. If release management is causing this distinction, I suggest making up sub-packages: twisted.core.python .trial .internet twisted.web.client .server .http .ftp twisted.mail.smtp .pop .imap .client .server twisted.rpc.jelly .spread <other RPC stuff> twisted.im.jabber .aol .client .server twisted.database. <database stuff> twisted.libs.xml .flow (other othgogonal non-protocol libraries) 4. In particular, I don't see that "enterprise" and "jelly+spread" are any more 'core' than jabber... ;) Sorry for being so boring. Clark

Glyph Lefkowitz wrote:

...

I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, ..

This makes no sense to me, and as itamar said is unlikely to be accepted.

well, I am glad that you are dismissing idea on the basis that you do not understand it (sorry for the irony), but as I did not spend great time or care to try to be understood, I can not complain too much.

First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols.

Yes, I know. Why are you telling me that ? Is it as I should have written : basename.(organisational_unit.)*(protocol.)+(implementation)+ with the ()* and ()* having a regex like meaning and the words basename, organinational_unit, ... having one of the possible listed values. basename in [twisted, tmlabs,whatever_is_agreed] organisational_unit in [mail, im,web] protocol in [nntp, ldap, oscar] implementation in [lowdown, my_fancy_implementation_name] It seems that by the time I wrote all that Clark have posted what I am trying to say.

Second, twisted.words, for example, is explicitly multiprotocol. I suppose if we wanted to be boring, we could call it twisted.multiprotocolchatserver.ircgateway, but IMHO "words" is catchier.

I do not really care if a name is catchy or not , "explicit is better than implicit". A good name is short enough to not be a pain when you code and explicit. I do not that fancy should not be a criteria of selection when it comes to technology. I was trying to say the a logical organisation should help the understanding of the twisted package. It seems that there is a clear need to change the structure of the code as it is now too large and need to be structured in a modular fashion. But at the same time it seems that most of the people on the list are speaking of rules to make it harder for new user to learn twisted. You guys, know twisted by heart. I do not intend to. I intend to use it for _RAD_, so learning it should be _quick_ If you do not understand me, please ask Michael Schneider or Clark Evans which seems to think like me but are able to express themselves correctly. Thomas

On Fri, 2004-04-23 at 04:14, Dave Smith wrote:

On Apr 20, 2004, at 1:20 PM, Matt Goodall wrote:

...

I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful.

I'm working on adding the final bits of useful xpath functionality (added // support the other nite), and would be willing to have/maintain xish as it's own subproject. I just have to figure out what exactly that means... :)

Wow, thanks Dave :). iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt@pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer.

On Tue, 2004-04-27 at 05:42, Dave Smith wrote:

On Apr 26, 2004, at 5:00 AM, Matt Goodall wrote:

...

iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense.

You mean like /foo[@bar='abc']

Yep, that's the one. I don't know if xish already supports this, I just realised I missed it from the tests. Cheers, Matt -- Matt Goodall, Pollenation Internet Ltd w: http://www.pollenation.net e: matt@pollenation.net Any views expressed are my own and do not necessarily reflect the views of my employer.