Hi Holger, I haven't really explored the integrated testing support in devpi-server so can't comment on that specifically but I can add some general observations: I always bundle my tests inside my Python packages so that they are published to PyPI (and as such they would also be included in wheels if I was publishing those) however I seem to be in the tiny minority in the larger Python community on this point :-) Also, running tests against an installed sdist seems like added value to me, because lots of people get setup.py files wrong on the first few iterations, and this would point out bugs in their installation procedures. On the other hand you could be opening a can of worms, so maybe make this an optional thing? :-) Regards, - Peter Odding PS. I'm loving devpi-server 2.x, the new web interface is also very nice, so thanks a lot to you and Florian for your effort on devpi-* On 29 September 2014 12:41, holger krekel wrote:

Hi all,

Currently you can't use "devpi test" with wheel files because they don't contain the tests unless you put all tests inside the software archive. To get the tests we could require that there is an "sdist" archive file containing the "tox.ini" file and the tests. Thus "devpi test X" would potentially download two files, a platform specific one and the sdist to configure and perform testing.

Apart from running the tests against the sdist we probably also want to run the tests against the installed sdist. Not sure i grasp all implications yet (also UI wise) but does anyone of you have preliminary thoughts on the issue?

best, holger

-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.

-- *Peter Odding* Team lead operational IT *E* peter....@paylogic.com *T* +31(0)652715946 *W* www.paylogic.com

Hey Holger!

I always bundle my tests inside my Python packages so that they are

...

published to PyPI (and as such they would also be included in wheels if I was publishing those) however I seem to be in the tiny minority in the larger Python community on this point :-)

Not sure it's a minority. I used to inline tests as well but for various reasons now put them outside. With wheels you won't be able to inline at least a tox.ini easily anymore.

Fair enough, maybe my impression is just wrong, or it could be that the general attitude about this point is changing.

...

me, because lots of people get setup.py files wrong on the first few iterations, and this would point out bugs in their installation

Also, running tests against an installed sdist seems like added value to procedures.

...

On the other hand you could be opening a can of worms, so maybe make this an optional thing? :-)

To get an "optional can of worms"? :)

That's a good point, an optional can of worms is still a can of worms :-). I was more thinking along the lines of giving people the option to test one way or the other, so that they can opt in to or opt out from testing installed sdists because they know whether this kind of testing will work for their project, given the way they've set up their testing infrastructure. Regards, - Peter

On 29 September 2014 12:41, holger krekel wrote:

...

...

Hi all,

Currently you can't use "devpi test" with wheel files because they don't contain the tests unless you put all tests inside the software archive. To get the tests we could require that there is an "sdist" archive file containing the "tox.ini" file and the tests. Thus "devpi test X" would potentially download two files, a platform specific one and the sdist to configure and perform testing.

Apart from running the tests against the sdist we probably also want to run the tests against the installed sdist. Not sure i grasp all implications yet (also UI wise) but does anyone of you have preliminary thoughts on the issue?

best, holger

Hey Holger, The problem with wheels is that there is no way to "devpi test" them

because currently the command's implementation assumes that a package contains a tox.ini and the tests. Even if a wheel could contain inlined tests there'd still not be no tox.ini. What we need is a canonical way to discover a tox.ini and the tests from a wheel. Using the "sdist" for that seems like a straight forward choice.

Forgetting about wheels for just a moment, does this require people to include the tox.ini in their source distribution archives using MANIFEST.in? I guess so because I see that "devpi test" works (exclusively?) by downloading the sdist and testing based on the downloaded artifact. I see now why wheels are so different from source distributions in this context; wheels contain only what will be installed by users (basically a binary distribution instead of a source distribution) while source distributions can contain additional files like a readme and tox.ini file. Of course there are ways to include data files in binary distributions, but that could get nasty. I guess given all of this, using the source distribution to discover the tox.ini file seems like the most sane choice. Are you actually using "devpi test" or "tox", btw?

I've been a happy user of tox for a long time now, but I've only recently started investigating devpi's support for automated test suites, which explains why it took me a while to get up to speed in this discussion :-) Regards, - Peter