Distutils at the PyCon 2004 sprints
I've not heard anything about either of the proposed Distutils sprints for next week. Just as a reminder, these are: - general distutils improvements and wart smashing (http://www.python.org/cgi-bin/moinmoin/DistUtils20) This is generally about avoiding some of the current warts rather than dealing with major new functionality, with the exception of the PEP 262 (http://www.python.org/peps/pep-0262.html) installation database. It's hard to get a good sense of what people's priorities are from the wiki page, but it would certainly be good to knock out some of the more long-standing warts. - dependency support (http://www.python.org/cgi-bin/moinmoin/DistutilsDependencies) This is Phillip Eby's proposal for adding dependency support to distutils, including automatic download and installation of required packages. I don't know what other people's reaction is to his proposal; aspects of it are certainly interesting and mesh well with the PEP 262 installation database. I'd really like to see some improvements happening in either of these areas. I don't know what it takes to get these sprints to go from proposed to concrete, or if we can just take space by squatting (unlikely to be well received ;). Who else will be at the sprints and willing to spend a day or two on distutils? In case there aren't enough outstanding distutils tasks, I'll be glad to propose a few more that I'm going to need to deal with anyway. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
At 05:22 PM 3/17/04 -0500, Fred L. Drake, Jr. wrote:
- dependency support (http://www.python.org/cgi-bin/moinmoin/DistutilsDependencies)
This is Phillip Eby's proposal for adding dependency support to distutils, including automatic download and installation of required packages.
FYI, the current implementation at http://cvs.eby-sarna.com/setuptools/ now includes a 'Require' class for specifying dependencies, and a 'find_packages()' function to automatically locate packages in a given directory (so you don't have to list them individually), e.g.: setup( name="setuptools", version="0.01", description="Distutils enhancements", author="Phillip J. Eby", author_email="peak@eby-sarna.com", license="PSF or ZPL", test_suite = 'setuptools.tests.test_suite', requires = [Require('Distutils','1.0.3','distutils')], packages = find_packages(), ) The 'Require' class takes a package name, version, and a module to check. It does not import the module (unless it's written in C), but rather uses a variety of sneaky tricks to do a version check. By default, the attribute looked for is '__version__' and the version class used for comparisons is distutils' StrictVersion. But you can override these defaults with keyword arguments. 'Require' objects can search a specified set of paths, so you can ensure that the target installation location is checked as well as the current sys.path. To handle packages without an explicit version attribute, you can set the requested version to 'None', and specify an attribute (i.e. class, function, or other global) name that must be defined by the module. There's a pretty extensive test suite that verifies all of these capabilities. The implementation also includes a skeleton 'depends' command that's automatically run prior to 'install' if there are any Require's passed to setup(). Right now that command just prints a message to say that it's being run, but sometime before the weekend I'd like to upgrade it to halt the installation with a list of unresolved dependencies, if applicable. I'll also add a 'homepage' keyword to 'Require', so that it can tell you where to go to manually find/download/install them. The current version also provides transparent support for building Pyrex extensions; if you have '.pyx' source files in an extension, the '.c' version will automatically be used instead if Pyrex is not available. Thus, you can easily distribute preprocessed '.c' versions for folks without Pyrex. These features are in addition to those described in my previous post at: http://mail.python.org/pipermail/distutils-sig/2004-March/003758.html Fred, do you think we should go ahead and move setuptools to cvs.zope.org? Actually, I guess first I should ask if you found its "features" mechanism useful for the sub-package dependency management you were prototyping for Zope X3. :)
On Wednesday 17 March 2004 05:57 pm, Phillip J. Eby wrote:
Fred, do you think we should go ahead and move setuptools to cvs.zope.org?
Yes, we can resolve this now. (This was the next email I needed to write today anyway. ;) Please check this in to cvs.zope.org as Packages/setuptools/. My preferred arrangement is to create a directory of that name, and add the package and any README.txt, setup.py, and such at that layer (see Packages/zpkgtools/ for an example of what I mean). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
At 06:06 PM 3/17/04 -0500, Fred L. Drake, Jr. wrote:
On Wednesday 17 March 2004 05:57 pm, Phillip J. Eby wrote:
Fred, do you think we should go ahead and move setuptools to cvs.zope.org?
Yes, we can resolve this now. (This was the next email I needed to write today anyway. ;) Please check this in to cvs.zope.org as Packages/setuptools/. My preferred arrangement is to create a directory of that name, and add the package and any README.txt, setup.py, and such at that layer (see Packages/zpkgtools/ for an example of what I mean).
Just to clarify, this would be equivalent to taking what's here: http://cvs.eby-sarna.com/setuptools/ and putting it here: http://cvs.zope.org/Packages/setuptools/ yes?
On Wednesday 17 March 2004 06:21 pm, Phillip J. Eby wrote:
Just to clarify, this would be equivalent to taking what's here:
http://cvs.eby-sarna.com/setuptools/
and putting it here:
Yep! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
At 07:16 PM 3/17/04 -0500, A.M. Kuchling wrote:
On Wed, Mar 17, 2004 at 05:57:39PM -0500, Phillip J. Eby wrote:
Fred, do you think we should go ahead and move setuptools to cvs.zope.org? Actually, I guess first I should ask if you found its
Shouldn't it go in the Python CVS tree, probably in nondist/sandbox?
--amk
Maybe so, but as a practical matter, I don't have commit access to the Python CVS. :) Also, isn't it customary for packages targeted for non-current versions of Python to live outside the Python CVS? E.g. logging, Optik, et al? Even if setuptools were adopted for 2.4, it would still need to be separately obtainable for 2.2 and 2.3 users.
[Phillip J. Eby]
Fred, do you think we should go ahead and move setuptools to cvs.zope.org?
[A.M. Kuchling]
Shouldn't it go in the Python CVS tree, probably in nondist/sandbox?
[Phillip]
Maybe so, but as a practical matter, I don't have commit access to the Python CVS. :)
And with an attitude like that, you're never gonna get it <wink>.
Also, isn't it customary for packages targeted for non-current versions of Python to live outside the Python CVS? E.g. logging, Optik, et al?
There's no non-accidental relation, as the nondist/ subtree has no relationship to Python releases.
Even if setuptools were adopted for 2.4, it would still need to be separately obtainable for 2.2 and 2.3 users.
The practical question is who y'all expect to *work* on this project. That you (Phillip) don't have commit access to the Python tree is an important point. If someone else who intends to work on it doesn't have commit access to Zope CVS, and doesn't want to sign the contributor thingie Zope Corp usually requires to get such access, then that's also an important point. Despite that Zope Corp funded initial work on the SpamBayes project, it started its life in the Python sandbox ("because" it was always going to be released under a PSF license), and we decided it was easiest for our contributors to set up a new SourceForge project for it. Aim to make life easiest for your likely primary contributors.
You know, it's really amazing how many more messages have been posted here on the subject of what CVS repository 'setuptools' should live in, than have commented on the design or implementation of setuptools itself. :) At 09:14 PM 3/17/04 -0500, Tim Peters wrote:
[Phillip J. Eby]
Fred, do you think we should go ahead and move setuptools to cvs.zope.org?
[A.M. Kuchling]
Shouldn't it go in the Python CVS tree, probably in nondist/sandbox?
[Phillip]
Maybe so, but as a practical matter, I don't have commit access to the Python CVS. :)
And with an attitude like that, you're never gonna get it <wink>.
Um, you mean my "practicality beats purity" attitude? :)
Also, isn't it customary for packages targeted for non-current versions of Python to live outside the Python CVS? E.g. logging, Optik, et al?
There's no non-accidental relation, as the nondist/ subtree has no relationship to Python releases.
Ah.
Even if setuptools were adopted for 2.4, it would still need to be separately obtainable for 2.2 and 2.3 users.
The practical question is who y'all expect to *work* on this project. That you (Phillip) don't have commit access to the Python tree is an important point. If someone else who intends to work on it doesn't have commit access to Zope CVS, and doesn't want to sign the contributor thingie Zope Corp usually requires to get such access, then that's also an important point.
Doesn't Python CVS access require a similar "contributor thingie", for the same reasons as Zope?
Despite that Zope Corp funded initial work on the SpamBayes project, it started its life in the Python sandbox ("because" it was always going to be released under a PSF license), and we decided it was easiest for our contributors to set up a new SourceForge project for it.
Aim to make life easiest for your likely primary contributors.
Fred and I are both zope.org CVS committers, and for the sprint I sort of assumed Fred and/or Jim would have "contributor thingies" on hand for people to sign if needed. :)
[Phillip J. Eby]
You know, it's really amazing how many more messages have been posted here on the subject of what CVS repository 'setuptools' should live in, than have commented on the design or implementation of setuptools itself. :)
But I understand CVS <0.9 wink>.
... Doesn't Python CVS access require a similar "contributor thingie",
In theory, but nobody has signed one yet (there's no form to sign yet, just a proposed form that doesn't actually make sense for various reasons -- which is why it hasn't moved beyond "proposed" status).
for the same reasons as Zope?
That's a different question, and as a Zope employee I'm not allowed to speculate about Zope Corp policies <heh>. The PSF wants a form for reasons that escape me now -- probably to ensure that there's no actionable question about the PSF's right to slap its own license on contributions, plus paying homage to various cover-your-ass legalistic superstitions "because everyone else does".
... Fred and I are both zope.org CVS committers, and for the sprint I sort of assumed Fred and/or Jim would have "contributor thingies" on hand for people to sign if needed. :)
You should really find out in advance whether contributors are willing to sign those. For example, if I weren't already a Zope employee, I wouldn't sign it on the spot-- it's too complicated for me to sign without paying a lawyer to review it first (I don't even know what half of it really means). BTW, if this code is intended to be released under a PSF license eventually, it probably won't help to have people sign something putting it under the ZPL first. Vice versa if it's intended to be released under the ZPL. SpamBayes moved to SourceForge partly because they didn't ask us to sign pages of licensing agreements just to use a CVS repository.
On Wed, Mar 17, 2004 at 10:14:27PM -0500, Tim Peters wrote:
... Doesn't Python CVS access require a similar "contributor thingie",
In theory, but nobody has signed one yet (there's no form to sign yet, just a proposed form that doesn't actually make sense for various reasons -- which is why it hasn't moved beyond "proposed" status).
Really..... I was asked to sign one and agree to be maintainer in order to get commit access for the Solaris and HP-UX bdist tools. Because they asked for "Employer" and because by employer is open source clueless, the PSF board (and I) agreed to pull the code.
for the same reasons as Zope?
That's a different question, and as a Zope employee I'm not allowed to speculate about Zope Corp policies <heh>. The PSF wants a form for reasons that escape me now -- probably to ensure that there's no actionable question about the PSF's right to slap its own license on contributions, plus paying homage to various cover-your-ass legalistic superstitions "because everyone else does".
Think "SCO". If Linus had contributor agreements from those SCO/Caldera employees Groklaw readers wouldn't have to be bit-dumpster diving to find emails that show their employer approved their participation. It's a fair thing to ask for reasonably significant contributions, I think.
... Fred and I are both zope.org CVS committers, and for the sprint I sort of assumed Fred and/or Jim would have "contributor thingies" on hand for people to sign if needed. :)
You should really find out in advance whether contributors are willing to sign those. For example, if I weren't already a Zope employee, I wouldn't sign it on the spot-- it's too complicated for me to sign without paying a lawyer to review it first (I don't even know what half of it really means).
The Zope one or the PSF one? The PSF one is basically "we agree that we both have copyright and can do whatever we want with the code." (Actually, I think there's 3 different options so you can choose what suits your needs best.)
BTW, if this code is intended to be released under a PSF license eventually, it probably won't help to have people sign something putting it under the ZPL first. Vice versa if it's intended to be released under the ZPL. SpamBayes moved to SourceForge partly because they didn't ask us to sign pages of licensing agreements just to use a CVS repository.
As much as I hate to say this, it's only going to get worse. A year ago, I never would have dreamed of asking for a copyright agreement if someone wanted to contribute to one of my projects (that I can't publish because my employer owns my every thought). Today, I probably would, if only to have a physical (non-digital) audit trail of contributors. At this point, it's like digging foxholes. We have to fortify our defenses in preparation for more "IP" (should be "intellectual rights" -- it is NOT _property!) attacks. I haven't read the ZPL in a while and I have no idea what they have to sign, however, if it's similar to the PSF agreement then once the maintainer has "joint copyright", they can shift the excercise of their rights from the PSF license to the ZPL to the GPL and back again as needed. None of the transitions would change pre-existing licenses, and at some transitions you could probably expect a fork since the other "joint copyright holders" retain full and equal rights as well. I don't mean to put a damper on things. It's just another challenge that open source community has to overcome. IANALBILWIHT mwa -- Mark W. Alexander slash@dotnetslash.net The contents of this message authored by Mark W. Alexander are released under the Creative Commons Attribution-NonCommercial license. Copyright of quoted materials are retained by the original author(s). http://creativecommons.org/licenses/by-nc/1.0/
[Tim, on PSF contributor agreements]
In theory, but nobody has signed one yet (there's no form to sign yet, just a proposed form that doesn't actually make sense for various reasons -- which is why it hasn't moved beyond "proposed" status).
[Mark W. Alexander]
Really..... I was asked to sign one
In http://www.python.org/sf/531901 Marc-Andre said The PSF will still require you to sign a contributor agreement for these addition, though, after these have been through the legal review phase. You didn't sign, and the forms still aren't through legal review.
and agree to be maintainer
That's a different issue -- that was a PEP 2 requirement.
in order to get commit access for the Solaris and HP-UX bdist tools.
Some form was needed just so that Marc-Andre could check in your stuff on your behalf. The issue was that the IP rights in your patch appeared to belong to your employer, and it was a substantial piece of work so it wasn't reasonable to overlook that. It doesn't really matter for short patches (an employer could try to sue over one of those, but wouldn't prevail). Most contributions to Python aren't entangled at all.
Because they asked for "Employer" and because by employer is open source clueless, the PSF board (and I) agreed to pull the code.
I'm on the PSF board, so I even know who your employer was <wink>.
... Think "SCO". If Linus had contributor agreements from those SCO/Caldera employees Groklaw readers wouldn't have to be bit-dumpster diving to find emails that show their employer approved their participation.
This isn't the place for an SCO debate, but no amount of paper can prevent a lawsuit. It can discourage one, but the PSF has other things in its favor discouraging lawsuits (see below).
It's a fair thing to ask for reasonably significant contributions, I think.
Yes, and the PSF is still trying to get a reasonable contributor form in place. The one copied from Zope Corp doesn't make sense for the PSF (as mentioned last time); for example, because Python's CVS is controlled by SourceForge (while Zope's is controlled by Zope), the words saying that the PSF may disable your account are simply absurd (the PSF cannot disable your SourceForge account); the bit about you agreeing that using your account password constitutes acceptance of the agreement is even sillier, because the PSF has no way of knowing whether you use your password (SourceForge might be able to tell -- we can't). An agreement that spouts nonsense stands poor chance of being upheld.
... The Zope one or the PSF one? The PSF one is basically "we agree that we both have copyright and can do whatever we want with the code." (Actually, I think there's 3 different options so you can choose what suits your needs best.)
The Zope one. The proposed PSF one was essentially copied from Zope's. There are at present no PSF forms available, just proposed forms that haven't passed legal review. Our attorney du jour happens to hate them, so to make progress we'll have to get a different attorney or different forms. ...
I don't mean to put a damper on things. It's just another challenge that open source community has to overcome.
Or ignore <0.5 wink>. There's potentially money in going after Linux. Nobody is going to make a dime going after the PSF, and because the PSF is a public charity (under US tax law), its primary assets can only end up in the hands of government agencies or other public charities. When the March of Dimes starts suing Open Source public charities, or SCO is recognized by the IRS as a government agency, then I'll start to worry about the PSF -- before then, the worst they can do is drive the PSF bankrupt. We want to discourage that too, but can't really prevent it if someone with a few spare million lawyer dollars is determined to make it happen.
On Thu, Mar 18, 2004 at 01:21:49AM -0500, Tim Peters wrote:
Some form was needed just so that Marc-Andre could check in your stuff on your behalf. The issue was that the IP rights in your patch appeared to belong to your employer, and it was a substantial piece of work so it wasn't reasonable to overlook that. It doesn't really matter for short patches (an employer could try to sue over one of those, but wouldn't prevail). Most contributions to Python aren't entangled at all.
Because they asked for "Employer" and because by employer is open source clueless, the PSF board (and I) agreed to pull the code.
I'm on the PSF board, so I even know who your employer was <wink>.
if was==still_is: # will be true until new_offer >= exisiting_recompense I'm sure I have your sympathies ;) FWIW, I have since managed to get Mark Lutz in for on-site training and Python is now the "official" language of our (worldwide!) division (although I still have to thrash some perl-mongers soundly to make them understand that other people have to be able to follow and maintain their code). Progress is being made. It's just soooo daaaaarrrrn sssllooooooowwwww......
This isn't the place for an SCO debate, but no amount of paper can prevent a lawsuit. It can discourage one, but the PSF has other things in its favor discouraging lawsuits (see below).
Discouragement is good.
It's a fair thing to ask for reasonably significant contributions, I think.
Yes, and the PSF is still trying to get a reasonable contributor form in place.
I understand and completely support the PSF's stance. The only suggestion I have would be to place a more prominent notice about "significant contributions" requiring an agreement. I only found out about it after the fact, and was seriously p.o.'ed _at_myself_ for placing the PSF in that position.
I don't mean to put a damper on things. It's just another challenge that open source community has to overcome.
Or ignore <0.5 wink>. There's potentially money in going after Linux. Nobody is going to make a dime going after the PSF, and because the PSF is a public charity (under US tax law), its primary assets can only end up in the hands of government agencies or other public charities. When the March of Dimes starts suing Open Source public charities, or SCO is recognized by the IRS as a government agency, then I'll start to worry about the PSF -- before then, the worst they can do is drive the PSF bankrupt. We want to discourage that too, but can't really prevent it if someone with a few spare million lawyer dollars is determined to make it happen.
This is very informative! Large open source projects have a great case for becoming public charities based on the educational aspect alone. (This is how our local LUG is pursuing 501(3c) status). For Big Corporation X to somehow "attack" a public charity would be very bad PR. Pass my best wishes and thanks again on to the board. mwa -- Mark W. Alexander slash@dotnetslash.net The contents of this message authored by Mark W. Alexander are released under the Creative Commons Attribution-NonCommercial license. Copyright of quoted materials are retained by the original author(s). http://creativecommons.org/licenses/by-nc/1.0/
On Wed, Mar 17, 2004 at 05:57:39PM -0500, Phillip J. Eby wrote:
Fred, do you think we should go ahead and move setuptools to cvs.zope.org? Actually, I guess first I should ask if you found its
At 07:16 PM 3/17/04 -0500, A.M. Kuchling wrote:
Shouldn't it go in the Python CVS tree, probably in nondist/sandbox?
This would also be fine by me. Whether the code is in the zope.org or python.sf.net repositories is immaterial to Zope's immediate needs (I'm fairly certain of this); whether it's covered by the PSF license or the ZPL2 is also irrelevant. (If Phillip commits it to the zope.org repository, all such versions will be available by the ZPL2. I think the python.sf.net repository is more flexible, but that's not important.) On Wednesday 17 March 2004 07:27 pm, Phillip J. Eby wrote:
Maybe so, but as a practical matter, I don't have commit access to the Python CVS. :)
As Tim suggests, this is something that can be dealt with.
Also, isn't it customary for packages targeted for non-current versions of Python to live outside the Python CVS? E.g. logging, Optik, et al? Even if setuptools were adopted for 2.4, it would still need to be separately obtainable for 2.2 and 2.3 users.
Your examples were developed for current versions of Python when they were initially developed; they've remained available for older versions as well. Distutils itself is in Python's CVS and is maintained for older versions at all (there's a top-level module you can check out and get a separate distutils distribution). So I'd say that *if* setuptools is being developed as a potential addition to Python's standard library, it's quite reasonable for it to be in Python's CVS. My own thinking is that its more of a testbed for enhancements to distutils. It may be that its also useful as an add-on package that works with older versions of distutils as well (in which case, the Python CVS is still a good place to maintain it). As it stands, the code is Phillip's, and he can do with it as he pleases. I won't object to using the zope.org CVS, though the python.sf.net CVS may prove better in the long term, especially if setuptools should maintain an identity of it's own. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
At 11:10 PM 3/17/04 -0500, Fred L. Drake, Jr. wrote:
At 07:16 PM 3/17/04 -0500, A.M. Kuchling wrote:
Shouldn't it go in the Python CVS tree, probably in nondist/sandbox?
This would also be fine by me. Whether the code is in the zope.org or python.sf.net repositories is immaterial to Zope's immediate needs (I'm fairly certain of this); whether it's covered by the PSF license or the ZPL2 is also irrelevant. (If Phillip commits it to the zope.org repository, all such versions will be available by the ZPL2. I think the python.sf.net repository is more flexible, but that's not important.)
On Wednesday 17 March 2004 07:27 pm, Phillip J. Eby wrote:
Maybe so, but as a practical matter, I don't have commit access to the Python CVS. :)
As Tim suggests, this is something that can be dealt with.
Okay, then I'm somewhat more inclined to lean towards the Python CVS, although naturally the most convenient thing for my packages is to leave it in the Eby-Sarna CVS.
Your examples were developed for current versions of Python when they were initially developed; they've remained available for older versions as well. Distutils itself is in Python's CVS and is maintained for older versions at all (there's a top-level module you can check out and get a separate distutils distribution). So I'd say that *if* setuptools is being developed as a potential addition to Python's standard library, it's quite reasonable for it to be in Python's CVS. My own thinking is that its more of a testbed for enhancements to distutils. It may be that its also useful as an add-on package that works with older versions of distutils as well (in which case, the Python CVS is still a good place to maintain it).
Yes, it's intended to be both a testbed for future distutils enhancements, and an immediately usable "field upgrade" for developers that want to use its features.
As it stands, the code is Phillip's, and he can do with it as he pleases. I won't object to using the zope.org CVS, though the python.sf.net CVS may prove better in the long term, especially if setuptools should maintain an identity of it's own.
Okay, so I guess the Python CVS would be fine, although I have never used SourceForge's CVS facilities as a developer before, so it may take me a while to get my setup figured out. If all else fails, you guys can always send me the sprint results in the form of a patch against the Eby-Sarna CVS. :)
Phillip J. Eby wrote:
At 11:10 PM 3/17/04 -0500, Fred L. Drake, Jr. wrote:
At 07:16 PM 3/17/04 -0500, A.M. Kuchling wrote:
Shouldn't it go in the Python CVS tree, probably in nondist/sandbox?
This would also be fine by me. Whether the code is in the zope.org or python.sf.net repositories is immaterial to Zope's immediate needs (I'm fairly certain of this); whether it's covered by the PSF license or the ZPL2 is also irrelevant. (If Phillip commits it to the zope.org repository, all such versions will be available by the ZPL2. I think the python.sf.net repository is more flexible, but that's not important.)
Note: if you ever want your code to go into the Python distribution you'll hve to make sure that you own the copyright on it and have full rights and title to the code. The Zope contributor agreement as it stands causes rights and title to be shared between Zope Corp and yourself (whatever that means). In the end it's probably better for legal reasons to keep Zope Corp out of this and leave the code in your CVS or in a separate SourcForge project CVS (this is the way that many other contributors of larger chunks of code have done it and it's the most flexible). Your decision, really :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 18 2004)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2004, Oxford, UK 28 days left EuroPython 2004, Göteborg, Sweden 80 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
participants (6)
-
A.M. Kuchling -
Fred L. Drake, Jr. -
M.-A. Lemburg -
Mark W. Alexander -
Phillip J. Eby -
Tim Peters