'python setup.py test' in develop mode?
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight. I'm working on a new tool called pkgme[1]. It uses distribute and recommends using virtualenv for development. After activating the virtualenv you can do 'python setup.py develop' and do convenient in-place editing so that your changes to the source show up immediately in the running environment. So far so good. However, when I want to run the test suite, it does the equivalent of a 'python setup.py build' and then runs the tests out of the build/lib.{platform}-{version} directory. This actually screws with our tests because (at least currently) the tests are looking for shell scripts and other programs that live in our tree, but are not part of the Python packaging, so don't get built or installed by setup.py. When we do filesystem introspection to try to find the scripts to exec them, it will work when running in develop mode, but fail in the tests, since __file__ points into build/lib.* instead of the source tree. So I'm wondering if other folks have hit similar problems and what best practices can be recommended to deal with this problem. I can think of various hacks around the problem, but don't want to reinvent the wheel. Or do you think that 'python setup.py test' should use the in-place tree and this is Just A Bug? Cheers, -Barry [1] https://launchpad.net/pkgme
At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote:
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight.
I'm working on a new tool called pkgme[1]. It uses distribute and recommends using virtualenv for development. After activating the virtualenv you can do 'python setup.py develop' and do convenient in-place editing so that your changes to the source show up immediately in the running environment. So far so good.
However, when I want to run the test suite, it does the equivalent of a 'python setup.py build' and then runs the tests out of the build/lib.{platform}-{version} directory.
Really? That sounds wrong. "python setup.py test" should run the tests against the *source* tree, with in-place extension builds.
2010-12-18 00:16:52 P.J. Eby napisał(a):
At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote:
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight.
I'm working on a new tool called pkgme[1]. It uses distribute and recommends using virtualenv for development. After activating the virtualenv you can do 'python setup.py develop' and do convenient in-place editing so that your changes to the source show up immediately in the running environment. So far so good.
However, when I want to run the test suite, it does the equivalent of a 'python setup.py build' and then runs the tests out of the build/lib.{platform}-{version} directory.
Really? That sounds wrong. "python setup.py test" should run the tests against the *source* tree, with in-place extension builds.
Modules are sometimes transformed by 2to3 during building (transformed versions are placed in build/lib*). Running tests (with Python 3) against the source tree would cause syntax errors etc. -- Arfrever Frehtes Taifersar Arahesis
At 04:49 AM 12/18/2010 +0100, Arfrever Frehtes Taifersar Arahesis wrote:
2010-12-18 00:16:52 P.J. Eby napisaÅ(a):
At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote:
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight.
I'm working on a new tool called pkgme[1]. It uses distribute and recommends using virtualenv for development. After activating the virtualenv you can do 'python setup.py develop' and do convenient in-place editing so that your changes to the source show up immediately in the running environment. So far so good.
However, when I want to run the test suite, it does the equivalent of a 'python setup.py build' and then runs the tests out of the build/lib.{platform}-{version} directory.
Really? That sounds wrong. "python setup.py test" should run the tests against the *source* tree, with in-place extension builds.
Modules are sometimes transformed by 2to3 during building (transformed versions are placed in build/lib*). Running tests (with Python 3) against the source tree would cause syntax errors etc.
Ah, so this issue is specific to distribute, then, not setuptools.
On Dec 18, 2010, at 04:49 AM, Arfrever Frehtes Taifersar Arahesis wrote:
2010-12-18 00:16:52 P.J. Eby napisał(a):
At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote:
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight.
I'm working on a new tool called pkgme[1]. It uses distribute and recommends using virtualenv for development. After activating the virtualenv you can do 'python setup.py develop' and do convenient in-place editing so that your changes to the source show up immediately in the running environment. So far so good.
However, when I want to run the test suite, it does the equivalent of a 'python setup.py build' and then runs the tests out of the build/lib.{platform}-{version} directory.
Really? That sounds wrong. "python setup.py test" should run the tests against the *source* tree, with in-place extension builds.
Modules are sometimes transformed by 2to3 during building (transformed versions are placed in build/lib*). Running tests (with Python 3) against the source tree would cause syntax errors etc.
Okay, I get that there are build-specific details that could cause problems running the tests out of the source tree. 'setup.py develop' would have similar problems, but that's much more of an isolated environment. By that I mean, it's mostly there for the developer while 'setup.py test' is a generic interface that 3rd parties would use. E.g. many Debian packages run 'setup.py test' as part of the build process (and this against multiple versions of Python, including 2.X and 3.X). Still, it would be nice if there were a 'setup.py test --develop' mode or some such. Usually when I'm working on a package, I don't care about build details, and I'm happy to do the moral equivalent of 'make clean' when I'm doing testing with 2to3 and such. In any case, thanks for pointing out that a 'setup.py test' using in-tree code is not the default case, and wouldn't make sense to be the default. I've worked around the issue in our specific case. -Barry
On Fri, Dec 17, 2010 at 05:26:47PM -0500, Barry Warsaw wrote:
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight.
I've a set of rules that serve me well. Rule #1: $VCS checkout and 'make test' should run the tests. This means I've a Makefile that creates the virtualenv/bootstraps a buildout, with all my test dependencies, and then runs the actual test runner script (which may be setup.py test, or it may be something different). (Likewise, 'make run' should start a local copy of the application, if this is a runnable application. But that's unrelated to testing.) <snip>
Or do you think that 'python setup.py test' should use the in-place tree and this is Just A Bug?
I don't know if it's a bug or not. I always assumed it worked as designed: perhaps trying to verify that your setuptools metadata is correctly specified and no modules are going to be missing when the package is installed? Actually, isn't running the tests from the build/ directory required when you want to support both Python 2.x and 3.x via 2to3, and you want to run the tests with Python 3? (I haven't done that personally, but I think Distribute explicitly supports this.) FWIW I found this tests-against-build-dir behaviour mildly irritating, but mostly irrelevant: I always specify my own test suite loader in setup.py (for various reasons, e.g. doctests), and my test loader ignores build/ and uses the in-place tree. I tend to support 'python setup.py test' as a convenience for other people. I dislike the copious build output it always produces at the beginning, and the time wasted to do the build. This might change when I start paying more attention to Python 3. -- Bumper sticker: No radio - Already stolen.
On Dec 18, 2010, at 04:24 AM, Marius Gedminas wrote:
On Fri, Dec 17, 2010 at 05:26:47PM -0500, Barry Warsaw wrote:
Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight.
I've a set of rules that serve me well.
Rule #1: $VCS checkout and 'make test' should run the tests. This means I've a Makefile that creates the virtualenv/bootstraps a buildout, with all my test dependencies, and then runs the actual test runner script (which may be setup.py test, or it may be something different).
(Likewise, 'make run' should start a local copy of the application, if this is a runnable application. But that's unrelated to testing.)
I think these are fine, but certainly with tests, I'd like 'make test' to be a thin wrapper around 'setup.py test' *in the general case*. Or IOW, I'd like to see 'setup.py test' be an established standard convention for Python packages. Not required of course, but I do think most simple packages can hook into it very easily, and should!
FWIW I found this tests-against-build-dir behaviour mildly irritating, but mostly irrelevant: I always specify my own test suite loader in setup.py (for various reasons, e.g. doctests), and my test loader ignores build/ and uses the in-place tree.
I think the fundamental problem in our case is that these extra artifacts live in-tree but are not in Python package directories. They also aren't installed, so we probably need to add rules to get these installed properly, and then the auto-discovery would be able to find them without caring about the build directories. For now, I do some path hacking to get it working, but it's an ugly solution.
I tend to support 'python setup.py test' as a convenience for other people. I dislike the copious build output it always produces at the beginning, and the time wasted to do the build. This might change when I start paying more attention to Python 3.
A reduced output command would be useful, but I do like the interface a lot. 'python setup.py test' is nice and consistent, so IMO should be promoted as a best practice. -Barry
participants (4)
-
Arfrever Frehtes Taifersar Arahesis -
Barry Warsaw -
Marius Gedminas -
P.J. Eby