Mailman 3 February 2010 - Distutils-SIG

Re: [Distutils] RFC: Barely minimal CI + tests + packaging in Python WAS: Guidelines for where to put tests & how to package them
by P.J. Eby 26 Feb '10

26 Feb '10

At 11:19 AM 2/26/2010 -0500, Olemis Lang wrote: >On Fri, Feb 26, 2010 at 10:52 AM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 10:16 AM 2/26/2010 -0500, Olemis Lang wrote: > >> > >> Well not as happily as I thought in first place. The fact is that > >> `setuptools` test command creates the runner by calling the class > >> specified in `test_runner` arg but supplies no parameters. In this > >> case I'd like to specify `xmlrunner:XMLTestRunner` (is that the > >> correct syntax ?) to obtain a JUnit XML report, but I need to specify > >> the path where those reports will be outputted, so that the CI tool > >> (Hudson in this case) can publish'em and generate the nice graphs we > >> all love . Considering the current implementation, I still need to > >> override `run_tests` method as a whole, and repeat 99% of what is > >> implemented in there, in order to get the exact runner I need > >> > >> :( > >> > >> My suggestion (hint, $0.02 ... whatever ;o) is > >> > >> 1. use a factory method (e.g `create_runner` ) returning an > >> instance of test runner. This method could be overridden by > >> subclasses if further options or whatever needs to be considered > >> to instanciate the runner > > > > I'm confused. Why don't you just set test_runner to point to your factory > > function, then? That's what nose does. (Well, in its case it's the > > test_loader, but same basic idea.) > > > >Well in theory it is possible (and did not think about that in the >first place, thnx ;o) but : > > 1. How could I know about extra command line > params (without parsing them once again of course ;o) ? > 2. In the case of the build script, I'm writing a setup script rather than > a framework (e.g. nose). > 2.1 The XML runner is not always used, but only when running at > the CI buildenv (I supposed I could just use > `-r xmlrunner:XMLTestRunner` but I need an extra arg) > How could I specify output path in cmdline ? An environment variable would solve these problems, at least if you have a decent shell. ;-) > 2.2 So I suppose I should write a function to do that but > - Under tests pkg? ... (Unit | functional | ...) tests >don't need it > - In setup.py itself ? ... What should I supply in to -r arg ? If you are running setup.py from the command line (as opposed to programmatically via easy_install), then it's __main__:whatever. (It won't work if setup.py isn't the "true" __main__, though, e.g., if the script is being execfile'd.) > How could I specify output path in cmdline ? > 3. In the case of `dutest` well ... that could help (depending on > the answer to 1 ;o) > 3.1 How could I reuse features already stored in `Distribution` > (especially values set after finalize_xxx ) ? You can't, unless you... > 4. (Hypothetical so far, so I'm prepared to receive tomatoes ;o) > What if I want to reuse (subclass) `test` command ? Which is perfectly allowable. You'll just have to duplicate that one method. Of course, if you write a suitable patch to refactor it to call another method, I'll happily put it in the trunk, since you're working from trunk to get the test_runner feature anyway. ("Suitable" = tested by you, doesn't break or remove existing functionality, & other "I'll know it if I see it" criteria ;-) )

1 0

Re: [Distutils] RFC: Barely minimal CI + tests + packaging in Python WAS: Guidelines for where to put tests & how to package them
by P.J. Eby 26 Feb '10

26 Feb '10

At 10:16 AM 2/26/2010 -0500, Olemis Lang wrote: >Well not as happily as I thought in first place. The fact is that >`setuptools` test command creates the runner by calling the class >specified in `test_runner` arg but supplies no parameters. In this >case I'd like to specify `xmlrunner:XMLTestRunner` (is that the >correct syntax ?) to obtain a JUnit XML report, but I need to specify >the path where those reports will be outputted, so that the CI tool >(Hudson in this case) can publish'em and generate the nice graphs we >all love . Considering the current implementation, I still need to >override `run_tests` method as a whole, and repeat 99% of what is >implemented in there, in order to get the exact runner I need > >:( > >My suggestion (hint, $0.02 ... whatever ;o) is > > 1. use a factory method (e.g `create_runner` ) returning an > instance of test runner. This method could be overridden by > subclasses if further options or whatever needs to be considered > to instanciate the runner I'm confused. Why don't you just set test_runner to point to your factory function, then? That's what nose does. (Well, in its case it's the test_loader, but same basic idea.)

2 1

Re: [Distutils] RFC: Barely minimal CI + tests + packaging in Python WAS: Guidelines for where to put tests & how to package them
by P.J. Eby 26 Feb '10

26 Feb '10

At 12:10 PM 2/25/2010 -0500, Olemis Lang wrote: >Thnx for your reply ! > >On Thu, Feb 25, 2010 at 12:00 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 10:41 AM 2/25/2010 -0500, Olemis Lang wrote: > >> > >> PS: BTW, how could I trigger easy_install(ation) at a given point > >> while implementing a distutils command, and let the command perform > >> further actions if deps are installed correctly ? > > > > Setuptools' "Distribution" object has a method for fetching dependencies. > > See setuptools' "test" command for an example. (This doesn't install the > > dependencies globally, just drops eggs into the build directory. But > > they're there and available to be reused for installation in a later phase, > > under normal circumstances.) > > > >So this means that setuptools `test` command already retrieves dependencies ? Yes - it also retrieves any testing-specific dependencies (per the "tests_require" setup() argument), so that if you use another testing library like py.test or nose, it can be used instead. >If so, then I shouldn't care about that, because I'd only need to >override `run_tests` method in order to do what I want to do (i.e. use >another test runner ;o) In that case, you may want to simply use the setuptools trunk version (setuptools==dev), which supports a test_runner setup() argument to the test command. ;-)

2 2

zc.buildout - order of extends processing
by Martin Aspeli 26 Feb '10

26 Feb '10

Hi, In my quest to make re-usable buildout components, I've started using a pattern like this: In a server-specific buildout file (e.g. prod-web-master.cfg) I have e.g.: [buildout] extends = buildout.d/base.cfg buildout.d/zope.cfg buildout.d/relstorage.cfg buildout.d/haproxy.cfg [ports] haproxy = 8100 [eggs] main = my.package ... Each file in buildout.cfg adds additional functionality, usually by defining new parts and doing e.g. parts += haproxy-build haproxy-config [haproxy-build] recipe = zc.recipe.cmmi ... Sometimes, they also extend some shared variables, e.g. [eggs] main += RelStorage With this type of setup, the order in which configuration is loaded and processed becomes important, in particular for += and -= type operations and straightforward overrides. For example, I may have a default ${ports:haproxy} defined in haproxy.cfg, but overridden in prod-web-slave.cfg because the slave node wants it to run on a different port, say. I can't find much documentation about how this is supposed to work. I've had situations where I've been surprised by the order in which things are processed, in particular if I extend one file that in turn extends another. So, a few questions - Is the order well-defined? e.g. is it always doing a depth-first or breath-first inclusion? Is the load order the same as the execution order? - Do settings in the top-level config (the one run with bin/buildout -c <file>) always override extended ones? - More generally, if A extends B and B extends C, does configuration in B always take precedence over configuration in C with the same part/option? Does a += in B always extend the options set in C? - If I have multiple files all extending a common base, e.g. A extends B and C, and both B and C extend a common D, when does the common base (D) get processed? Cheers, Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

1 0

how to use mr.developer?
by }--o 24 Feb '10

24 Feb '10

hi, I'm just experimenting with distutils, and would like to move my django apps under it. One of my apps uses django-sugar that should be checked out from github. bin/buildout check out my django-sugar repo under src/, and creates an egg-link under develop-eggs, but it's not added to sys.path for the bin/* binaries. Is there a simple way to do it? I've seen the extra_paths variable, but then I still have to write each ${buildout:auto-checkout} line again. this is my buildout.cfg file: [buildout] parts = python django-1.1 develop = . eggs = django-contacts django-l10n django-registration eggs-directory = /home/nagyv/.buildout/eggs find-links = ${buildout:eggs-directory} download-cache = /home/nagyv/.buildout/dlcache extensions = mr.developer sources = sources auto-checkout = uni-form sugar [python] recipe = zc.recipe.egg interpreter = python eggs = ${buildout:eggs} [django-1.1] recipe = djangorecipe version = 1.1.1 project = contacts projectegg = contacts settings = testsettings test = contacts testrunner = test-1.1 eggs = ${buildout:eggs} [sources] uni-form = git git://github.com/nagyv/django-uni-form.git sugar = git git://github.com/nagyv/django-sugar.git thanks for your help! Viktor

2 1

how to use mr.developer?
by }--o 24 Feb '10

24 Feb '10

hi, I'm just experimenting with distutils, and would like to move my django apps under it. One of my apps uses django-sugar that should be checked out from github. bin/buildout check out my django-sugar repo under src/, and creates an egg-link under develop-eggs, but it's not added to sys.path for the bin/* binaries. Is there a simple way to do it? I've seen the extra_paths variable, but then I still have to write each ${buildout:auto-checkout} line again. this is my buildout.cfg file: [buildout] parts = python django-1.1 develop = . eggs = django-contacts django-l10n django-registration eggs-directory = /home/nagyv/.buildout/eggs find-links = ${buildout:eggs-directory} download-cache = /home/nagyv/.buildout/dlcache extensions = mr.developer sources = sources auto-checkout = uni-form sugar [python] recipe = zc.recipe.egg interpreter = python eggs = ${buildout:eggs} [django-1.1] recipe = djangorecipe version = 1.1.1 project = contacts projectegg = contacts settings = testsettings test = contacts testrunner = test-1.1 eggs = ${buildout:eggs} [sources] uni-form = git git://github.com/nagyv/django-uni-form.git sugar = git git://github.com/nagyv/django-sugar.git thanks for your help! Viktor

2 1

setuptools and additional scripts handling
by Manlio Perillo 24 Feb '10

24 Feb '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. I'm starting to use setuptools for my projects and I have a few questions. In my web applications, usually I have some scripts that needs to be installed as cron scripts. I would like to install them using setuptools, since it is important that the correct python interpreter is used (I use virtualenv), and setuptools does this automatically.. Is it possible to declare a custom directory where some of the scripts needs to be installed? Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt6nGQACgkQscQJ24LbaUT8QACgg2tkF6vJWWTiBoFnWtr6tb1X C7sAnjzu/l/ytRMD44STkTiiVJ/uIxmy =hSnP -----END PGP SIGNATURE-----

4 9

Re: [Distutils] from distutils.core import DistutilsOptionError
by P.J. Eby 24 Feb '10

24 Feb '10

At 06:05 PM 2/23/2010 -0500, Tarek Ziadé wrote: >Or do you mean that you consider the exception classes located in >Distutils to be a common need >for people that write setup.py scripts ? A setup script may want to explicitly throw (or catch) distutils errors, and having only one place to import these things from makes distutils easier to use. I don't consider it a *common* need, but I do consider it part of distutils' "core API", in the sense that if your code is either invoking distutils or being invoked by it, you may need to throw or catch those errors. In general, everything that's a *distutils-defined* symbol (not types, os, etc.) in distutils.core, I thought was intended to be imported from there, as they are all things that could be considered part of the "core API". (And apparently, at least Thomas thought the same thing.) (If I'd been writing distutils, I'd have named the module distutils.api rather than distutils.core, and explicitly created an __all__ list for it, but it's pretty much the same thing.)

2 1

Re: [Distutils] from distutils.core import DistutilsOptionError
by P.J. Eby 24 Feb '10

24 Feb '10

At 05:32 PM 2/23/2010 -0500, Tarek Ziadé wrote: >2/ the work done in 2.7 so far was following the usual deprecation >process for relocated >APIs or removed APIs, but nothing for misplaced import statements. We >did discuss this in the past and said it was fine this way. Btw, imports from distutils.core *are* API; its docstring describes it as "The only module that needs to be imported to use the Distutils". It's supposed to be a one-stop shop for importing common things needed by setup scripts, including the errors, Extension, Command, Distribution, etc. So, the unused imports in distutils.core are not "misplaced import statements", nor is there a good reason (AFAICT) for changing the API to require the average setup script author to remember which module to import things from. There should be only one obvious way to do it, and right now, the one obvious way to import things in a setup script is from distutils.core.

2 1

from distutils.core import DistutilsOptionError
by Thomas Heller 23 Feb '10

23 Feb '10

One of my setup scripts contains: from distutils.core import DistutilsOptionError which doesn't work anymore with Python trunk. Bug in Python, or in my setup script? -- Thanks, Thomas

3 3

Distutils-SIG February 2010