[Distutils] how do I add some functionality to my setup.py which is testable and modular?

[Zooko Wilcox-O'Hearn]

Dear Tarek Ziadé and fellowship of the packaging:

Please pay attention to this issue.  If we can extend our build  
processes in a modular, re-usable, and testable way then this could  
contribute to the success of distutils/setuptools/Distribute.

In particular, I want Nevow to be automatically installable as a  
dependency (using, for now, setuptools, but presumably the same will  
apply to Distribute), and one of the issues is still outstanding and  
Glyph refuses to accept my patch which fixes it because the fix isn't  
testable.

http://divmod.org/trac/ticket/2699

I want Distribute-compatible build extensions to be accepted by  
Glyph.  This requires that they be testable.  Please help me  
understand how to write build extensions which are testable.

Regards,

Zooko

On Sunday,2009-08-16, at 14:16 , Zooko Wilcox-O'Hearn wrote:

> Folks:
>
> I have a recurring problem with distutils/setuptools/Distribute,  
> which is that I don't know how to extend the functionality of  
> setup.py and make the new functionality be testable and modular.
>
> Here's one specific example that I currently have a lot of  
> experience with: I'd like to generate a version number from  
> revision control history.  Don't get hung up on the desired  
> functionality though -- if you think that generating version  
> numbers is best done a different way, or if you don't care about  
> generating version numbers, then please just mentally insert some  
> other extension of your build functionality that you do care about.
>
> I'm familiar with three different ways to implement an extension  
> like this.
>
> The first is what Nevow does [1], which is to write in setup.py  
> "from nevow import __version__".  Well, this works in the case that  
> setup.py is being executed as the command-line script in a fresh,  
> empty Python interpreter, but it fails in the case that the Python  
> interpreter has already been running for a while, has already  
> imported nevow (a *different* nevow from a different location on  
> the local filesystem), and is now importing *this* nevow's setup  
> module in order to build this nevow.  This happens with setuptools  
> and py2exe.  Distribute v0.6 includes a patch [2] to fix this, but  
> I'm not sure the patch is right (it involves 'for m in  
> various_modules: del sys.modules[m]' and it doesn't seem to fix all  
> cases).  PJE says "Thou shalt not import yourself when trying to  
> build yourself" [3].  Glyph says "Then how do I test this  
> code?" [4].  I say "Ugh, I don't know. Let's look at the other two  
> ways to do it."
>
> The second is what I do in some of my packages such as pycryptopp  
> [5]. Just take that functionality that you want to add to all of  
> your packages and cut and paste the code into each of your  
> setup.py's, then edit it a little to reflect the correct name of  
> the current package.  This sucks because you're cutting and pasting  
> code, because your setup.py gets bigger and hairier the more  
> functionality your build system has, and because, again, you can't  
> test it.
>
> The third is what I do in Tahoe-LAFS [6].  I moved the  
> functionality in question into a separate Python package, in this  
> case named "darcsver" [7], and used setuptools's plugin system to  
> add a command named "darcsver" which initializes the  
> distribution.metadata.version attribute correctly.  Then I had to  
> add a bunch of aliases to my setup.cfg [8] saying "If you're going  
> to build, first darcsver, and if you're going to install, first  
> darcsver, and ...".  This sort of works, except that yesterday my  
> programming partner Brian Warner informed me [9] that he expected  
> the "python ./setup.py --version" command-line to output the  
> version number.  Argh!  There is no way to configure in my  
> setup.cfg "If you're going to --version, first darcsver.".
>
> So it appears to me that none of these techniques are both modular/ 
> testable and compatible with distutils/setuptools/Distribute.  What  
> are we to do?
>
> Regards,
>
> Zooko
>
> [1] http://www.divmod.org/trac/browser/trunk/Nevow/setup.py?rev=17531
> [2] http://bugs.python.org/setuptools/issue20
> [3] http://bugs.python.org/setuptools/msg139
> [4] http://www.divmod.org/trac/ticket/2699#comment:20
> [5] http://allmydata.org/trac/pycryptopp/browser/setup.py?rev=661#L181
> [6] http://allmydata.org/trac/tahoe/browser/setup.py?rev=4036#L97
> [7] http://allmydata.org/trac/darcsver
> [8] http://allmydata.org/trac/tahoe/browser/setup.cfg?rev=3996#L46
> [9] http://allmydata.org/trac/darcsver/ticket/6
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig