[Twisted-Python] Progress report on splitting packages
Since Chris didn't send one in. Hope this is accurate.
News was split off, backwards compat code is in, site is up: http://projects.twistedmatrix.com/lowdown/
Remaining tasks: 1. Release automation and making a lowdown release. 2. Buildbot support. 3. Linking to projects from main site, and updating main site to explain all this.
Many thanks to Chris and JP for all their work on this.
On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote:
Remaining tasks:
Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all need infrastructure for having separate packages.
Itamar Shtull-Trauring [Mon, Apr 19, 2004 at 02:03:51PM -0400]:
On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote:
Remaining tasks:
Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all need infrastructure for having separate packages.
I can create & test packages for FreeBSD Ports & NetBSD Packages Collection, just send me what to do. I can also provide binary packages for FreeBSD 5.2.1 at the moment.
Michal Pasternak wrote:
I can create & test packages for FreeBSD Ports & NetBSD Packages Collection, just send me what to do. I can also provide binary packages for FreeBSD 5.2.1 at the moment.
Is there anything automatable about this? I'd like to automate as much as possible in the release-twisted script (Twisted/sandbox/radix/release-twisted). Soon I'll be splitting that out into something like twisted.python.release with only the project-specific stuff in the "script". It'd be cool if you could implement a step for doing freebsd builds (or stuff facilitating a freebsd build, anyway).
Christopher Armstrong [Mon, Apr 19, 2004 at 03:38:46PM -0400]:
Michal Pasternak wrote:
I can create & test packages for FreeBSD Ports & NetBSD Packages Collection, just send me what to do. I can also provide binary packages for FreeBSD 5.2.1 at the moment.
Is there anything automatable about this? I'd like to automate as much as possible in the release-twisted script (Twisted/sandbox/radix/release-twisted).
I think I could add there automatic generation of a Makefiles for a given project (for those, who don't know anything about fbsd ports/netbsd pkgsrc, here's an example - ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc/net/py-twisted/ ) It could be really clean at the *BSD side - all Twisted projects share some settings in Makefile.common, each project has a separate directory and Makefile with its 'personalized' settings.
I could also add to your release script generation of diffs against current official fbsd/nbsd package sources (from fbsd/nbsd cvs repo); then the package could be built (on a test machine), and if the package builds okay, the release script could submit a problem report with the diffs needed to update the package source to GNATS database (see http://netbsd.org/Misc/send-pr.html), so *BSD developers could pick that PR and update the official sources.
Automated building of binary package in case of FreeBSD / NetBSD is, IMO, not worth it (at Twisted project side). Why? When the package source is in official CVS repo, both NetBSD and FreeBSD teams take care of building the binary package for you (so you don't have to build Twisted for NetBSD/atari, for example). Also (IMO) BSD users tend rather not to use binary package available from an opensource project's site, they will get the one from official BSD FTP or compile by their own, using Ports or pkgsrc.
I hope, that this is the kind of automation you meant. In case it is not - more clues please.
To start the work I need URL to the tarballs with source releases of "core" twisted and at least one twisted product.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
For Windows, as long as there's a setup.py this is probably a candidate for just running a buildslave that does bdist_wininst and sticks the result on a fileserver. That's predicated on having a buildslave, of course.
The somewhat sophisticated installer around Twisted "proper" is mainly to provide documentation and icons and uninstall and other approachability enhancements.
Many (most?) of these subprojects don't have a wide enough scope to need such niceties. However, even if they did, I couldn't provide packaging for all of them :-(
C
Itamar Shtull-Trauring wrote:
| On Mon, 2004-04-19 at 13:48, Itamar Shtull-Trauring wrote: | | |>Remaining tasks: | | | Forgot to add, packaging. Windows, Debian, Gentoo, FreeBSD and so on all | need infrastructure for having separate packages. |
On Mon, Apr 19, 2004 at 01:48:27PM -0400, Itamar Shtull-Trauring wrote:
Since Chris didn't send one in. Hope this is accurate.
News was split off, backwards compat code is in, site is up: http://projects.twistedmatrix.com/lowdown/
I'm curious -- why did we choose to change the name from twisted.news? And even though the name is changing, why is the new package "lowdown" and not "twisted.lowdown"?
I'm also interested to see that NNTP protocol support has also migrated to lowdown -- I was expecting that protocol implementations were going to stay in Twisted (or perhaps a twisted.protocols project, if that gets split off), and just the apps (i.e. direct subpackages of twisted, e.g. news, mail, web, names, conch, ...) would be seperated.
So, I assume that this is the policy that other splits should follow? If so, this needs to be documented.
I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.
-Andrew.
Andrew Bennetts wrote:
On Mon, Apr 19, 2004 at 01:48:27PM -0400, Itamar Shtull-Trauring wrote:
Since Chris didn't send one in. Hope this is accurate.
News was split off, backwards compat code is in, site is up: http://projects.twistedmatrix.com/lowdown/
I'm curious -- why did we choose to change the name from twisted.news? And even though the name is changing, why is the new package "lowdown" and not "twisted.lowdown"?
It was decided that doing package names like this would be too problematic. Unfortunately, Python isn't as cool as Java. The name was changed to lowdown because "news" is a bad top-level package name and "twistednews" and "tnews" and various permutations so forth were also lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha Ha!
I'm also interested to see that NNTP protocol support has also migrated to lowdown -- I was expecting that protocol implementations were going to stay in Twisted (or perhaps a twisted.protocols project, if that gets split off), and just the apps (i.e. direct subpackages of twisted, e.g. news, mail, web, names, conch, ...) would be seperated.
Nah. These split-offs are going to contain protocols and minimal framework for implementing apps, along with maybe a very simple implementation of an actual app.
So, I assume that this is the policy that other splits should follow? If so, this needs to be documented.
Let it be known that the split-offs will contain relevant protocols.
I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.
I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc).
It would probably be a good idea to outline what exactly WILL be split out, and what will be contained in the splits, now, though. I am not authoritative on this, but I can tell you what I plan on working on soon (In the order that I've thought of them):
<twisted-package> -> <canonical project name> <details>
twisted.news -> "Lowdown" done. contains nntp.
twisted.flow -> "Flow" (?) This one's easy, no protocols or anything.
twisted.conch -> "Conch" This will be named "Conch", and contain conch as well as the SSH protocol implemantations.
twisted.lore -> "Lore" (?) Easy.
twisted.words, twisted.im -> "Twisted Words" (?) This'll be a single package. The packagename will probably be 'words'. I'm not *sure*, but I expect IRC and the other IM/chat protocols will go into this package.
twisted.mail -> ??? SMTP, POP3, IMAP, *some* of the stuff from current twisted.mail package.
twisted.names -> "Twisted Names" (?) + DNS proto
twisted.xish -> No idea. No idea. I imagine this will just be made a part of the Jabber package, whatever that is.
twisted.runner -> ??? Yeah, might as well get rid of this one too.
twisted.pair -> ??? I don't know anything about this package.
twisted.trial -> ??? This one is going to have to require some thought, since *all* the other packages depend on it, but I think that people do want to split it out.
twisted.web -> ??? HTTP protocol implementation, and twisted.web. It is still undecided whether Nevow will be included in this package. Discuss it on the twisted-web mailing list.
twisted.protocols.ftp + slyphon's sandbox -> ??? Need to split this out too.
Others...
There has been talk of leaving Spread in Twisted core. That seems like a fine idea to me, but I'm not the one to debate that with. Also enterprise, manhole. Dunno if we'll split those out. Cred, internet, application, python, persisted are definitely staying. We will still have a 'protocols' package left with a few protocols that don't have any framework for them implemented.
HTH!
On Mon, Apr 19, 2004, Christopher Armstrong wrote:
Andrew Bennetts wrote:
I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.
I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc).
What about a document for people who in future, want to add new twisted subprojects, from scratch? Some of the same policy will be relevant to them, and that wouldn't be a transitory document.
-Mary
Mary Gardiner wrote:
On Mon, Apr 19, 2004, Christopher Armstrong wrote:
Andrew Bennetts wrote:
I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.
I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc).
What about a document for people who in future, want to add new twisted subprojects, from scratch? Some of the same policy will be relevant to them, and that wouldn't be a transitory document.
It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous.
Christopher Armstrong wrote:
It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous.
Let me retract this adjective from my response. Twisted Matrix Laboratories is far from nebulous -- we're a tight-knit group of hackers, just as the web site says. However, TML must no longer be defined as "those who hack on Twisted", since Twisted is no longer the only project run by Twisted Matrix Laboratories (well, there were Twisted Java and Twisted Emacs and some others before, but those were never very popular). Now, TML must be defined as "those who hack on TML projects". >:-)
On Apr 19, 2004, at 10:19 PM, Christopher Armstrong wrote:
It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous.
We absolutely need a policy document. The most attractive thing about Twisted, to many developers, is the extremely consistent coding style, packaging style, and central point of distribution. A project developed by 'the labs' also implicitly depends on some other core packages, and we need policy about things like running regression tests with multiple versions of dependencies in order that those core packages continue to support their clients correctly to even maintain parity with our current development process's quality level.
Some of the policy will seem obvious, but other packages don't follow it. For example, we should make sure that everything uses distutils as its build process.
On Tue, Apr 20, 2004, Glyph Lefkowitz wrote:
On Apr 19, 2004, at 10:19 PM, Christopher Armstrong wrote:
It seems quite unnecessary, and I'm not convinced enough of its utility to actually put effort into it. There's not much special about a "Twisted Subproject"; AFAIC they're just projects that are developed by "Twisted Matrix Laboratories", which is rather nebulous.
We absolutely need a policy document.
I had a thought briefly yesterday that got lost in my helpful (off-list) exchange with radix about whether policy could keep cross-projects tasks like docs (and testing I suppose) sane: a policy doc would be most helpful for newcomers to the Twisted community who've decided their itch is a new subproject, because they want to add support for a blah protocol. Reading a policy doc would be a lot faster than absorbing policy on IRC for an outsider. Or at least so it appears to me.
Twisted, to many developers, is the extremely consistent coding style, packaging style, and central point of distribution. A project developed by 'the labs' also implicitly depends on some other core packages, and we need policy about things like running regression tests with multiple versions of dependencies in order that those core packages continue to support their clients correctly to even maintain parity with our current development process's quality level.
Some of the policy will seem obvious, but other packages don't follow it. For example, we should make sure that everything uses distutils as its build process.
If people want a policy doc, send me notes towards it and I'll happily turn it into something doc-like or a series thereof. I know there's coding standards already.
-Mary
On Tuesday 20 April 2004 05:27 pm, Mary Gardiner wrote:
protocol. Reading a policy doc would be a lot faster than absorbing policy on IRC for an outsider. Or at least so it appears to me.
Using IRC as a sole source for such information is insane. Many potential contributors and users won't want anything to do with IRC, and #twisted isn't comfy to "outsiders." (It's not that people on #twisted aren't nice or helpful; it's just that the basic culture is... strange.)
-Fred
On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote:
Andrew Bennetts wrote:
On Mon, Apr 19, 2004 at 01:48:27PM -0400, Itamar Shtull-Trauring wrote:
Since Chris didn't send one in. Hope this is accurate.
News was split off, backwards compat code is in, site is up: http://projects.twistedmatrix.com/lowdown/
I'm curious -- why did we choose to change the name from twisted.news? And even though the name is changing, why is the new package "lowdown" and not "twisted.lowdown"?
It was decided that doing package names like this would be too problematic. Unfortunately, Python isn't as cool as Java. The name was changed to lowdown because "news" is a bad top-level package name and "twistednews" and "tnews" and various permutations so forth were also lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha Ha!
I get the name, I just wanted to know why exactly it was chosen over other names :)
What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.
I'm also interested to see that NNTP protocol support has also migrated to lowdown -- I was expecting that protocol implementations were going to stay in Twisted (or perhaps a twisted.protocols project, if that gets split off), and just the apps (i.e. direct subpackages of twisted, e.g. news, mail, web, names, conch, ...) would be seperated.
Nah. These split-offs are going to contain protocols and minimal framework for implementing apps, along with maybe a very simple implementation of an actual app.
So, I assume that this is the policy that other splits should follow? If so, this needs to be documented.
Let it be known that the split-offs will contain relevant protocols.
Ok :)
I'd like to see the reasons for these decisions summarised in that document too -- I can guess at them, but it would be nice to know, and will help future splits of subprojects go smoothly and consistently.
I'm not interested in creating a "Split Document", because that's an inherently temporal thing. With the help of exarkun, there does exist a small file documenting the process that I must go through to do a split for a particular project, but it is not of general interest (no rationale, etc).
Well, when I say "document", a post to the mailing list would do just fine. Just somewhere public that can be referred to later, if necessary.
-Andrew.
On Apr 19, 2004, at 10:17 PM, Andrew Bennetts wrote:
What are the problems you allude to with keeping "twisted.news"? I was under the impression that Zope 3 is planning to go this route.
I'd like to hear more about this. I think it is valuable to Twisted's image to keep all the projects inside a top-level namespace. While these may be different projects/apps, they aren't completely separate: they're all built by the same group of people, for use with the Twisted framework. Also I hope the latest release of all twisted products will be available as one large tarball/windows installer for people who don't want to think about it.
Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me.
So, can it not be done technically?
James
On Tue, 2004-04-20 at 10:57, James Y Knight wrote:
Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me.
So, can it not be done technically?
I think so, yes.
On Tuesday 20 April 2004 11:29 am, Itamar Shtull-Trauring wrote:
On Tue, 2004-04-20 at 10:57, James Y Knight wrote:
Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me.
So, can it not be done technically?
I think so, yes.
I've been quiet here lately, but am interested in this discussion since I've been working on packaging issues for Zope 3.
One thing that's interesting to note is that distutils actually allows installing subpackages directly. For example, it's possible to install twisted.news separately from other twisted.* packages (let's ignore dependencies for the moment).
The only interesting dependency for twisted.news is the top-level twisted package. If that's a "pure container" package (it contains only a trivial __init__.py), then it only needs to be released and packaged once, and it won't create dependency issues. Some twisted.core package would be a dependency for practical issues, but distutils will grow some support for versionable dependencies like that (there's preliminary, undocumented support in the Python 2.4 CVS).
My point is that there's no technical reason not to use the "twisted." prefix; there may be other reasons not to, but that would be something for the Twisted developers and maintainers to determine.
-Fred
On Apr 20, 2004, at 12:02 PM, Fred L. Drake, Jr. wrote:
On Tuesday 20 April 2004 11:29 am, Itamar Shtull-Trauring wrote:
On Tue, 2004-04-20 at 10:57, James Y Knight wrote:
Using twisted.* would be better if twisted.core was in twisted.core but even without that the name sounds fine to me.
So, can it not be done technically?
I think so, yes.
One thing that's interesting to note is that distutils actually allows installing subpackages directly. For example, it's possible to install twisted.news separately from other twisted.* packages (let's ignore dependencies for the moment).
If it's the case that this is possible, I'd say "best naming scheme" would be: twisted.core.*coremodules twisted.product twisted.otherproduct.
However, given that twisted.core isn't going to exist, I'd be almost as happy with: twisted.*coremodules tmlabs.product tmlabs.otherproduct
So consider that last my vote.
James
Three suggestions so far:
1. Separate namespaces for each project.
twisted.internet conch
2. Keep all projects and core under twisted.
twisted.internet twisted.conch
3. Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html)
twisted.internet t.conch or tmlabs.conch
On Tue, 2004-04-20 at 11:53, Itamar Shtull-Trauring wrote:
- Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was
considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html)
twisted.internet t.conch or tmlabs.conch
For the record, this is the one I want, using 'tmlabs' for non-core projects. So we'd have twisted.internet, tmlabs.news, tmlabs.conch etc..
Itamar Shtull-Trauring wrote:
- Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was
considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html)
twisted.internet t.conch or tmlabs.conch
I like this. With 'tmlabs'.
On Tue, Apr 20, 2004, Itamar Shtull-Trauring wrote:
- Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was
considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html)
twisted.internet t.conch or tmlabs.conch
I like 3.
-Mary
I can get on board with 3
On Tue, 2004-04-20 at 08:53, Itamar Shtull-Trauring wrote:
Three suggestions so far:
- Separate namespaces for each project.
twisted.internet conch
- Keep all projects and core under twisted.
twisted.internet twisted.conch
- Separate namespace for projects, e.g. 't' or 'tmlabs' (Zope3 was
considering using 'z', though it like it won't happen in the end - http://mail.zope.org/pipermail/zope3-dev/2004-April/010469.html)
twisted.internet t.conch or tmlabs.conch
I've got a quick question, how will twisted projects manage dependencies? If this isn't handled nicely, it could really reduce the value of twisted (nothing more frustrating than downloading N packages, each one after an error of the package not being found).
On Mon, Apr 19, 2004 at 09:58:30PM -0400, Christopher Armstrong wrote: | >>News was split off, backwards compat code is in, site is up: | >>http://projects.twistedmatrix.com/lowdown/ | > | > | >I'm curious -- why did we choose to change the name from twisted.news? | >And | >even though the name is changing, why is the new package "lowdown" and | >not | >"twisted.lowdown"? | | It was decided that doing package names like this would be too | problematic. Unfortunately, Python isn't as cool as Java. The name was | changed to lowdown because "news" is a bad top-level package name and | "twistednews" and "tnews" and various permutations so forth were also | lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha
This makes very little sense to me. In my humble opinion, it should stay 'twisted.news'. Calling it 'lowdown' beacuse a top level package of 'news' is taken is really counterproductive, finding good names is hard enough. I do see the value in breaking twisted into modules, so that a subset of twisted could be used. However, I don't see why these items can be named 'twisted.XXXX', perhaps the 'core' distribution should just have place-holder '__init__.py' file...
/twisted/news/__init__.py
raise NotImplementedError("""\ Download this package from http://projects.twistedmatrix.com/news/ """)
Best,
Clark
Clark C. Evans wrote:
I've got a quick question, how will twisted projects manage dependencies? If this isn't handled nicely, it could really reduce the value of twisted (nothing more frustrating than downloading N packages, each one after an error of the package not being found).
I encourage you to contribute packages for your favorite OS. Windows and Debian will be covered.
Clark C. Evans wrote:
This makes very little sense to me. In my humble opinion, it should stay 'twisted.news'. Calling it 'lowdown' beacuse a top level package of 'news' is taken is really counterproductive, finding good names is hard enough. I do see the value in breaking twisted into modules, so that a subset of twisted could be used. However, I don't see why these items can be named 'twisted.XXXX', perhaps the 'core' distribution should just have place-holder '__init__.py' file...
/twisted/news/__init__.py
raise NotImplementedError("""\ Download this package from http://projects.twistedmatrix.com/news/ """)
I am a boring fart and I always though that names should be as explicit as possible.
Am I right to think that all the code will be related to applications/application frameworks using twisted ? In that case something like twistedapp (please do not take that name as the ultimate recommendation) would make more sense.
As well I would saws that the organization should be like twistedapp.news.lowdown twistedapp.web.nevow twistedapp.ssh.counch [...]
I have no idea of what are flow, lore, and some other modules and it seems that some of you do not know neither what is xish neither ! Should it be called twistedapp.jabber.xish ???
So the protocol is obvious from the module name and it is easy to add more application of the same protocol without any name clash. At that point it is clear that nevow is a web application, even if the name itself does not give the information away.
Thomas
On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:
It was decided that doing package names like this would be too problematic. Unfortunately, Python isn't as cool as Java. The name was changed to lowdown because "news" is a bad top-level package name and "twistednews" and "tnews" and various permutations so forth were also lame. Also, "Low Down" is cool. Get it? The LOW DOWN??? The NEWS??? Ha Ha!
One problem with leaving it as twisted.news in future releases as Andrew suggests in a later email is that we lose our emphasis on the fact that it's a separate release with separate versions. Assume following scenario - we have lowdown 0.1 and lowdown 0.2, which are API incompatible. We want to be very clear that lowdown 0.1 can still be used on latest Twisted core, that you don't have to go in lockstep.
So, looking at code - is twisted.foo part of core twisted and thus probably pretty stable or is it a separate package? What expectations can I have of API stability?
Partially this is just Python conventions. Separate packages go in separate namespaces. Possibly Java's system is better, but if it doesn't really match our users' expectations then we're just going to confuse them if we use it.
Then again, maybe we're just being stupid and we should stick to Java-style "twisted.foo", so "twisted.news" with a "twisted.news.nntp". Who else thinks that's a good idea?
twisted.xish -> No idea. No idea. I imagine this will just be made a part of the Jabber package, whatever that is.
Yeah. Jabber will probably be in the Words pacakge, so xish will end up there (unless people tell us it is generally applicable).
twisted.trial -> ??? This one is going to have to require some thought, since *all* the other packages depend on it, but I think that people do want to split it out.
When Glyph and I did the original break out document (Glyph, could you forward it to the list?) we decided this ought to stay in core, because everything depends on it, including core :) Also it requires the reactor, internally. It's not worth breaking out.
Itamar Shtull-Trauring wrote:
One problem with leaving it as twisted.news in future releases as Andrew suggests in a later email is that we lose our emphasis on the fact that it's a separate release with separate versions. Assume following scenario - we have lowdown 0.1 and lowdown 0.2, which are API incompatible. We want to be very clear that lowdown 0.1 can still be used on latest Twisted core, that you don't have to go in lockstep.
Agreed, but it is a short term thing as all new users will have never known the "old way".
So, looking at code - is twisted.foo part of core twisted and thus probably pretty stable or is it a separate package? What expectations can I have of API stability?
It is no worse than currently where you have to read the doc. As well, even stable interface can change. Nothing is set in stone.
Partially this is just Python conventions. Separate packages go in separate namespaces. Possibly Java's system is better, but if it doesn't really match our users' expectations then we're just going to confuse them if we use it.
So you mean that os.path is not pythonic ? os.path is one of the first include that every python user do !
Then again, maybe we're just being stupid and we should stick to Java-style "twisted.foo", so "twisted.news" with a "twisted.news.nntp". Who else thinks that's a good idea?
Big repeat ...
I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, ..
twisted.mail.pop3.byme twisted.mail.pop3.byyou twisted.nntp.lowdown twisted.nntp.cnn
poweredbytwisted.mail.pop3.byme poweredbytwisted.mail.pop3.byyou poweredbytwisted.nntp.lowdown poweredbytwisted.nntp.cnn
would be valid names
You could even have a prefered implementation accessible as from twisted.nntp import default as nntp or the like
Thomas
On Tue, 2004-04-20 at 09:34, Thomas Mangin wrote:
twisted.mail.pop3.byme twisted.mail.pop3.byyou twisted.nntp.lowdown twisted.nntp.cnn
Not going to happen. twisted.news... maybe. twisted.news.lowdown or whatever is not going to be how we do it.
On Apr 20, 2004, at 9:34 AM, Thomas Mangin wrote:
I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, ..
This makes no sense to me, and as itamar said is unlikely to be accepted.
First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols.
Second, twisted.words, for example, is explicitly multiprotocol. I suppose if we wanted to be boring, we could call it twisted.multiprotocolchatserver.ircgateway, but IMHO "words" is catchier.
Glyph Lefkowitz wrote:
I will repeat that I think that <basename>.<protocol>(.<implementation>)+ basename :: twisted, foobar, other protocol :: nntp, ldap, mail, ... implementation :: lowdown, smtp, pop3, ..
This makes no sense to me, and as itamar said is unlikely to be accepted.
well, I am glad that you are dismissing idea on the basis that you do not understand it (sorry for the irony), but as I did not spend great time or care to try to be understood, I can not complain too much.
First of all, nntp and ldap are the same kind of thing as smtp and pop3 - protocols.
Yes, I know. Why are you telling me that ? Is it as I should have written :
basename.(organisational_unit.)*(protocol.)+(implementation)+ with the ()* and ()* having a regex like meaning and the words basename, organinational_unit, ... having one of the possible listed values.
basename in [twisted, tmlabs,whatever_is_agreed] organisational_unit in [mail, im,web] protocol in [nntp, ldap, oscar] implementation in [lowdown, my_fancy_implementation_name]
It seems that by the time I wrote all that Clark have posted what I am trying to say.
Second, twisted.words, for example, is explicitly multiprotocol. I suppose if we wanted to be boring, we could call it twisted.multiprotocolchatserver.ircgateway, but IMHO "words" is catchier.
I do not really care if a name is catchy or not , "explicit is better than implicit". A good name is short enough to not be a pain when you code and explicit. I do not that fancy should not be a criteria of selection when it comes to technology.
I was trying to say the a logical organisation should help the understanding of the twisted package. It seems that there is a clear need to change the structure of the code as it is now too large and need to be structured in a modular fashion. But at the same time it seems that most of the people on the list are speaking of rules to make it harder for new user to learn twisted.
You guys, know twisted by heart. I do not intend to. I intend to use it for _RAD_, so learning it should be _quick_
If you do not understand me, please ask Michael Schneider or Clark Evans which seems to think like me but are able to express themselves correctly.
Thomas
Radix/Itamar,
It's clear that y'all have put a good deal of thought into this, so please take my feedback as just meandering thoughts.
On Tue, Apr 20, 2004 at 08:36:07AM -0400, Itamar Shtull-Trauring wrote: | One problem with leaving it as twisted.news in future releases as Andrew | suggests in a later email is that we lose our emphasis on the fact that | it's a separate release with separate versions. Assume following | scenario - we have lowdown 0.1 and lowdown 0.2, which are API | incompatible. We want to be very clear that lowdown 0.1 can still be | used on latest Twisted core, that you don't have to go in lockstep.
I suppose that lockstep has caused release management problems? How is this going to solve the problem? (ie, are you sure it is a problem, and that the solution fixes it?)
| So, looking at code - is twisted.foo part of core twisted and thus | probably pretty stable or is it a separate package? What expectations | can I have of API stability? | | Partially this is just Python conventions. Separate packages go in | separate namespaces. Possibly Java's system is better, but if it doesn't | really match our users' expectations then we're just going to confuse | them if we use it.
This seems primarly a documentation problem, that is, it is solved with a README document in the package. Personally, I've never had issues with modules being stable or not. Perhaps a package manager could use a list of packages that are stable? ie, "python setup.py --with-experimental install"
In any case, I think this package management issue needs to be addressed no matter what your naming convention is.
| > twisted.xish -> No idea. | > No idea. I imagine this will just be made a part of the Jabber | > package, whatever that is. | | Yeah. Jabber will probably be in the Words pacakge, so xish will end up | there (unless people tell us it is generally applicable).
So, it will be in jabber (which I don't need?) till at some time it turns out that someone needs it, and then it will move somewhere else.
Bings,
Clark
On Tue, 2004-04-20 at 10:19, Clark C. Evans wrote:
I suppose that lockstep has caused release management problems? How is this going to solve the problem? (ie, are you sure it is a problem, and that the solution fixes it?)
We want to allow people to use latest version of Twisted with old versions of other packages. So if the package you use has a major API rewrite, you can still use the old version with the latest and coolest core Twisted.
Lets say we rewrite twisted.web in a non-backwards compatible way and release it inside Twisted 1.4. If you need a bug fix to twisted.internet. that's in 1.4 but want to use the old twisted.web, you're screwed.
In any case, I think this package management issue needs to be addressed no matter what your naming convention is.
Which issue?
So, it will be in jabber (which I don't need?) till at some time it turns out that someone needs it, and then it will move somewhere else.
Well, "which I don't need" implies you want xish standalone, so maybe it should be from the start...
On Tue, Apr 20, 2004 at 10:35:07AM -0400, Itamar Shtull-Trauring wrote: | We want to allow people to use latest version of Twisted with old | versions of other packages. So if the package you use has a major API | rewrite, you can still use the old version with the latest and coolest | core Twisted. | | Lets say we rewrite twisted.web in a non-backwards compatible way and | release it inside Twisted 1.4. If you need a bug fix to | twisted.internet. that's in 1.4 but want to use the old twisted.web, | you're screwed.
Itamar, looks like you have a really good use case there -- in particular, I'd like to upgrade twisted "core" without updating twisted.web till it is more stable and I've had time to update my code dependencies. If the 'twisted' modules could stay within twisted.* I would prefer this, but this, alas is up to the core people... ;) Clark
On Apr 20, 2004, at 10:35 AM, Itamar Shtull-Trauring wrote:
We want to allow people to use latest version of Twisted with old versions of other packages. So if the package you use has a major API rewrite, you can still use the old version with the latest and coolest core Twisted.
In particular, this was a release management problem for Divmod, because we've been making changes to the imap4 code (which is in the "protocols" package), that our code depends on, but we can't release all of Twisted at once fast enough to roll out a minor change, because going through the full QA process for the whole system takes too long. We are trying to decouple things more so that a project which requires bugfixes in twisted.web doesn't accidentally suck in in-progress changes to the reactor as well.
On Tue, Apr 20, 2004 at 04:21:49PM -0400, Glyph Lefkowitz wrote: | >We want to allow people to use latest version of Twisted with old | >versions of other packages. So if the package you use has a major API | >rewrite, you can still use the old version with the latest and coolest | >core Twisted. | | In particular, this was a release management problem for Divmod, | because we've been making changes to the imap4 code (which is in the | "protocols" package), that our code depends on, but we can't release | all of Twisted at once fast enough to roll out a minor change, because | going through the full QA process for the whole system takes too long. | We are trying to decouple things more so that a project which requires | bugfixes in twisted.web doesn't accidentally suck in in-progress | changes to the reactor as well.
This seems the driving reason. What ever choice is made, it should be there to support this need.
My perspective:
1. I think it is best to stick with one top level package name. Splitting twisted.* into twisted and tmlabs or whatever dillutes the value of the overall product and creates unnecessary distinctions.
2. Overall, naming should be as boring as possible so that what is being implemented is clear, 'ssh' is better than 'consh'. Also, while more than one implementation is the 'open source' way of doing things, more than one way in the twisted project, imho, is not a great idea (it dillutes the marketing value)
3. If release management is causing this distinction, I suggest making up sub-packages:
twisted.core.python .trial .internet
twisted.web.client .server .http .ftp
twisted.mail.smtp .pop .imap .client .server
twisted.rpc.jelly .spread <other RPC stuff>
twisted.im.jabber .aol .client .server
twisted.database. <database stuff>
twisted.libs.xml .flow (other othgogonal non-protocol libraries)
4. In particular, I don't see that "enterprise" and "jelly+spread" are any more 'core' than jabber... ;)
Sorry for being so boring.
Clark
On Tue, 2004-04-20 at 13:36, Itamar Shtull-Trauring wrote:
On Mon, 2004-04-19 at 21:58, Christopher Armstrong wrote:
[snip]
twisted.xish -> No idea. No idea. I imagine this will just be made a part of the Jabber package, whatever that is.
Yeah. Jabber will probably be in the Words pacakge, so xish will end up there (unless people tell us it is generally applicable).
I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful.
I don't know enough about the xish package as a whole to comment on other parts - I only used (possibly wrongly) what I needed from it to get the example working.
Cheers, Matt
On Apr 20, 2004, at 1:20 PM, Matt Goodall wrote:
I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful.
I'm working on adding the final bits of useful xpath functionality (added // support the other nite), and would be willing to have/maintain xish as it's own subproject. I just have to figure out what exactly that means... :)
D.
On Fri, 2004-04-23 at 04:14, Dave Smith wrote:
On Apr 20, 2004, at 1:20 PM, Matt Goodall wrote:
I recently used xish.xpath instead of libxml's xpath in a Nevow example to avoid introducing the external dependency. I knew xish would be available to anyone interested in the example ... or so I thought ;-). The xish.xpath implementation is lacking a few useful bits of the xpath spec but other than that it's useful.
I'm working on adding the final bits of useful xpath functionality (added // support the other nite), and would be willing to have/maintain xish as it's own subproject. I just have to figure out what exactly that means... :)
Wow, thanks Dave :).
iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense.
Cheers, Matt
On Apr 26, 2004, at 5:00 AM, Matt Goodall wrote:
iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense.
You mean like /foo[@bar='abc']
D.
On Tue, 2004-04-27 at 05:42, Dave Smith wrote:
On Apr 26, 2004, at 5:00 AM, Matt Goodall wrote:
iirc, i missed one thing from my unittest that is useful - matching by attribute value. See the zvon.org's example 6 although I personally wouldn't bother with the normalize-space() nonsense.
You mean like /foo[@bar='abc']
Yep, that's the one. I don't know if xish already supports this, I just realised I missed it from the tests.
Cheers, Matt
On Apr 19, 2004, at 9:58 PM, Christopher Armstrong wrote:
It was decided that doing package names like this would be too problematic.
When and by whom? :) At PyCon we talked about (and, IIRC, decided about) using the 'pkgutil' module to keep different subpackages in different locations. I originally wanted to move things to top-level packages but itamar convinced me that forcing a change in the import structure as part of the split would be a bad idea.
Glyph Lefkowitz wrote:
On Apr 19, 2004, at 9:58 PM, Christopher Armstrong wrote:
It was decided that doing package names like this would be too problematic.
When and by whom? :) At PyCon we talked about (and, IIRC, decided about) using the 'pkgutil' module to keep different subpackages in different locations. I originally wanted to move things to top-level packages but itamar convinced me that forcing a change in the import structure as part of the split would be a bad idea.
Well, see newer posts to this list.
On Tuesday 20 April 2004 03:56 pm, Glyph Lefkowitz wrote:
When and by whom? :) At PyCon we talked about (and, IIRC, decided about) using the 'pkgutil' module to keep different subpackages in different locations.
I've convinced myself that this is inadvisable at this point (in general, not just for Twisted). There is a bug in pkgutil that renders it much more painful than it should be. See:
http://www.python.org/sf/935117
-Fred
participants (14)
-
Andrew Bennetts -
Christopher Armstrong -
Clark C. Evans -
Cory Dodt -
Dave Smith -
David Reid -
Fred L. Drake, Jr. -
Glyph Lefkowitz -
Itamar Shtull-Trauring -
James Y Knight -
Mary Gardiner -
Matt Goodall -
Michal Pasternak -
Thomas Mangin