Problem with distribute and converting tests for python3
Hello, Firstly thanks to those working on distribute, I had been using setuptools but was getting annoyed with it slowly developing (eg. not having python3 support, etc), so distribute seems like it is what I am looking for. Any way I am using distribute for one of my packages now (I wanted to try and support python3) but have noticed something which doesn't work for me. I have my main package sources in one directory and then tests in another directory (it makes it easy to separate them out when creating a binary package). Now this works fine for python2.x as I can include the tests in a source distribution by using MANIFEST.in, but it doesn't work for python3 as distribute doesn't seem to run 2to3 on them (although they are run by the distribute/setuptools test command in the setup.py script). When I looked at the distribute documentation the example there has the tests in a subpackage of the main package. So my question is, am I structuring my source distribution wrongly, is there a better way to do this which would work with distribute? If I were to structure my source distribution like the distribute example then how do I ensure unit tests aren't included in a binary distribution? Is there anyway I can get distribute to pick up my tests as I currently have my source distribution structured? Michael Whapples
2009/10/11 Michael Whapples <mwhapples@aim.com>:
Hello, Firstly thanks to those working on distribute, I had been using setuptools but was getting annoyed with it slowly developing (eg. not having python3 support, etc), so distribute seems like it is what I am looking for.
Any way I am using distribute for one of my packages now (I wanted to try and support python3) but have noticed something which doesn't work for me. I have my main package sources in one directory and then tests in another directory (it makes it easy to separate them out when creating a binary package). Now this works fine for python2.x as I can include the tests in a source distribution by using MANIFEST.in, but it doesn't work for python3 as distribute doesn't seem to run 2to3 on them (although they are run by the distribute/setuptools test command in the setup.py script).
You have to tell setup that these files are source files, somehow. By just including them in MANIFEST.in but not in any source package they'll get treated as resources. Perhaps if you included it as a apckage or py_module? It is true that Distribute picks up files in test/test*.py for inclusion in source distributions, which I didn't know until now. These will not be included in the binary distribution, and hence not converted with 2to3 either, so that will not work with 2to3 support. That needs to be documented or fixed somehow, although I don't really know how...
If I were to structure my source distribution like the distribute example then how do I ensure unit tests aren't included in a binary distribution?
Well... what is the actual use case for doing that? There's no problem with distributing the tests (in fact, I would say it's a good idea). -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
My problem with having unit tests included in a binary distribution is that they take up space and have no way to be run, so can certainly be called a waste of space. OK, may be they could be run by a user importing the correct modules, etc, I meant no simple way (eg. as done in a source package with the command $ setup.py test ). Also anyway what would the reason be for a user to run tests on the binary module, I (or another developer in the cases of modified versions) should have run the tests before distributing the binary package. Source distributions are a different matter in my mind, someone can modify it and so having tests is useful. OK, to possible solutions: 1. Distribute offers the keyword argument "convert_2to3_doctests", may be it could have one to give test directory, and then .py files in that could be converted. 2. As setuptools automatically includes test/test*.py, why can't distribute actually compile these files as well (I question whether tests in a test directory should be prefixed with test, but I don't have a really strong view should people disagree). May be even both of those could be combined, so if a test directory is not specified then test/test*.py is the default (if such a directory exists). As a note: my views of structuring a source distribution and not including tests in binary distributions may be influenced by my java programming, hope that helps explain where I may be coming from. Michael Whapples On 11/10/09 21:43, Lennart Regebro wrote:
2009/10/11 Michael Whapples<mwhapples@aim.com>:
Hello, Firstly thanks to those working on distribute, I had been using setuptools but was getting annoyed with it slowly developing (eg. not having python3 support, etc), so distribute seems like it is what I am looking for.
Any way I am using distribute for one of my packages now (I wanted to try and support python3) but have noticed something which doesn't work for me. I have my main package sources in one directory and then tests in another directory (it makes it easy to separate them out when creating a binary package). Now this works fine for python2.x as I can include the tests in a source distribution by using MANIFEST.in, but it doesn't work for python3 as distribute doesn't seem to run 2to3 on them (although they are run by the distribute/setuptools test command in the setup.py script).
You have to tell setup that these files are source files, somehow. By just including them in MANIFEST.in but not in any source package they'll get treated as resources. Perhaps if you included it as a apckage or py_module?
It is true that Distribute picks up files in test/test*.py for inclusion in source distributions, which I didn't know until now. These will not be included in the binary distribution, and hence not converted with 2to3 either, so that will not work with 2to3 support. That needs to be documented or fixed somehow, although I don't really know how...
If I were to structure my source distribution like the distribute example then how do I ensure unit tests aren't included in a binary distribution?
Well... what is the actual use case for doing that? There's no problem with distributing the tests (in fact, I would say it's a good idea).
2009/10/11 Michael Whapples <mwhapples@aim.com>:
My problem with having unit tests included in a binary distribution is that they take up space and have no way to be run, so can certainly be called a waste of space. OK, may be they could be run by a user importing the correct modules, etc, I meant no simple way (eg. as done in a source package with the command $ setup.py test
True, you need some test runner like nose or zope.tetsing or so. But then it works.
). Also anyway what would the reason be for a user to run tests on the binary module, I (or another developer in the cases of modified versions) should have run the tests before distributing the binary package.
Yes, but if something breaks it can be good to run the tests locally to see if something in the environments breaks them. Also tests can be useful as a sort of documentation.
OK, to possible solutions: 1. Distribute offers the keyword argument "convert_2to3_doctests", may be it could have one to give test directory, and then .py files in that could be converted.
Well, the test directory is test/test*.py.
2. As setuptools automatically includes test/test*.py, why can't distribute actually compile these files as well (I question whether tests in a test directory should be prefixed with test, but I don't have a really strong view should people disagree).
The problem is that these files doesn't end up in the binary distribution and hence 2to3 can't be run on them. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On 12/10/09 05:24, Lennart Regebro wrote:
2009/10/11 Michael Whapples<mwhapples@aim.com>:
[...]
). Also anyway what would the reason be for a user to run tests on the binary module, I (or another developer in the cases of modified versions) should have run the tests before distributing the binary package.
Yes, but if something breaks it can be good to run the tests locally to see if something in the environments breaks them. Also tests can be useful as a sort of documentation. In that case I would ask them to download the source distribution and run the tests there, they probably will want the source distribution as I may need to ask them to apply a patch and test it out (they have the environment I didn't foresee/have).
I question how unit tests in a binary package are like a sort of documentation as the user will not be able to see the test source code as it will all be in .pyc files. Doctests are a different matter, but I am on about unit tests.
OK, to possible solutions: 1. Distribute offers the keyword argument "convert_2to3_doctests", may be it could have one to give test directory, and then .py files in that could be converted.
Well, the test directory is test/test*.py.
2. As setuptools automatically includes test/test*.py, why can't distribute actually compile these files as well (I question whether tests in a test directory should be prefixed with test, but I don't have a really strong view should people disagree).
The problem is that these files doesn't end up in the binary distribution and hence 2to3 can't be run on them. Bad wording on my part, I said compile when I meant convert, so point two should be "As distribute will include test/test*.py, why can't distribute run 2to3 on these files and so convert my tests and so then be able to run tests from the test command". This probably would need distribute to create a separate directory (eg. test_3k) and then run tests from the new location to achieve this.
Michael Whapples
2009/10/12 Michael Whapples <mwhapples@aim.com>:
In that case I would ask them to download the source distribution and run the tests there, they probably will want the source distribution as I may need to ask them to apply a patch and test it out (they have the environment I didn't foresee/have).
Well, that's one step extra, but fine.
I question how unit tests in a binary package are like a sort of documentation as the user will not be able to see the test source code as it will all be in .pyc files.
A binary distribution will include the .py files as well. "Binary" is different from "source" in that extensions are already compiled. You don't need to make binary distributions for anything else than Windows, and then only if you have C-extensions. Modules without c-extensions are best distributed as a source distribution.
The problem is that these files doesn't end up in the binary distribution and hence 2to3 can't be run on them.
Bad wording on my part, I said compile when I meant convert, so point two should be "As distribute will include test/test*.py, why can't distribute run 2to3 on these files and so convert my tests and so then be able to run tests from the test command". This probably would need distribute to create a separate directory (eg. test_3k) and then run tests from the new location to achieve this.
No, that wasn't bad wording on your part, I understood perfectly, I think. The problem is that these files doesn't end up in the binary distribution and hence 2to3 isn't run on them as the 2to3 conversion is done in build_py. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
OK, I didn't realise the .py files are included in a binary distribution, I understood it to be that the .pyc file was enough and so had followed through that the .py file wouldn't be included in a binary distribution like it is typically done in java (java tends not to include the .java source files in a binary .jar file, only the .class files). I did warn you that I may be making some assumptions/hold views due to my java programming. This leads to some more general questions about python and distributing modules (eg. why do we need binary distributions (bdist_egg, bdist, etc) when a source distribution should be fine and do all needed, why does python distribute source and compiled files in a binary distribution (what does the source files add in such a case), etc). Anyway back to the actual topic. My main question of the possible solution is, why can't we have a way of specifying extra directories to run 2to3 on? Tests are an example (well at least the way I view tests) of when this could be useful, it may be useful in other cases (not thought of an example). Michael Whapples On 12/10/09 18:16, Lennart Regebro wrote:
2009/10/12 Michael Whapples<mwhapples@aim.com>:
In that case I would ask them to download the source distribution and run the tests there, they probably will want the source distribution as I may need to ask them to apply a patch and test it out (they have the environment I didn't foresee/have).
Well, that's one step extra, but fine.
I question how unit tests in a binary package are like a sort of documentation as the user will not be able to see the test source code as it will all be in .pyc files.
A binary distribution will include the .py files as well. "Binary" is different from "source" in that extensions are already compiled. You don't need to make binary distributions for anything else than Windows, and then only if you have C-extensions. Modules without c-extensions are best distributed as a source distribution.
The problem is that these files doesn't end up in the binary distribution and hence 2to3 can't be run on them.
Bad wording on my part, I said compile when I meant convert, so point two should be "As distribute will include test/test*.py, why can't distribute run 2to3 on these files and so convert my tests and so then be able to run tests from the test command". This probably would need distribute to create a separate directory (eg. test_3k) and then run tests from the new location to achieve this.
No, that wasn't bad wording on your part, I understood perfectly, I think. The problem is that these files doesn't end up in the binary distribution and hence 2to3 isn't run on them as the 2to3 conversion is done in build_py.
Michael Whapples wrote:
OK, I didn't realise the .py files are included in a binary distribution, I understood it to be that the .pyc file was enough and so had followed through that the .py file wouldn't be included in a binary distribution like it is typically done in java (java tends not to include the .java source files in a binary .jar file, only the .class files). I did warn you that I may be making some assumptions/hold views due to my java programming. This leads to some more general questions about python and distributing modules (eg. why do we need binary distributions (bdist_egg, bdist, etc) when a source distribution should be fine and do all needed, why does python distribute source and compiled files in a binary distribution (what does the source files add in such a case), etc).
Anyway back to the actual topic. My main question of the possible solution is, why can't we have a way of specifying extra directories to run 2to3 on? Tests are an example (well at least the way I view tests) of when this could be useful, it may be useful in other cases (not thought of an example).
Michael Whapples On 12/10/09 18:16, Lennart Regebro wrote:
2009/10/12 Michael Whapples<mwhapples@aim.com>:
In that case I would ask them to download the source distribution and run the tests there, they probably will want the source distribution as I may need to ask them to apply a patch and test it out (they have the environment I didn't foresee/have). Well, that's one step extra, but fine.
I question how unit tests in a binary package are like a sort of documentation as the user will not be able to see the test source code as it will all be in .pyc files. A binary distribution will include the .py files as well. "Binary" is different from "source" in that extensions are already compiled. You don't need to make binary distributions for anything else than Windows, and then only if you have C-extensions. Modules without c-extensions are best distributed as a source distribution.
The problem is that these files doesn't end up in the binary distribution and hence 2to3 can't be run on them. Bad wording on my part, I said compile when I meant convert, so point two should be "As distribute will include test/test*.py, why can't distribute run 2to3 on these files and so convert my tests and so then be able to run tests from the test command". This probably would need distribute to create a separate directory (eg. test_3k) and then run tests from the new location to achieve this. No, that wasn't bad wording on your part, I understood perfectly, I think. The problem is that these files doesn't end up in the binary distribution and hence 2to3 isn't run on them as the 2to3 conversion is done in build_py.
Binary distribution is a packaging type such as an rpm or a deb file usually for a particular distro. Not about whether source or various compilation products are or are not included in these binary packages. You can write an install that requires no build and and no source and just moves precompiled files into position. Regards, Gerry
2009/10/12 Michael Whapples <mwhapples@aim.com>:
This leads to some more general questions about python and distributing modules (eg. why do we need binary distributions (bdist_egg, bdist, etc) when a source distribution should be fine and do all needed
You don't. As mentioned, binary distributions are generally not needed, unless you have C-extensions that need compiling, and then usually only on Windows, as Windows doesn't include a compiler.
why does python distribute source and compiled files in a binary distribution (what does the source files add in such a case), etc).
Well, why not? Excluding them serves no purpose. And including them means you can read the source, debug, etc.
Anyway back to the actual topic. My main question of the possible solution is, why can't we have a way of specifying extra directories to run 2to3 on?
It's a pretty special case. I agree that for the test/test*.py case we should find some sort of solution for backwards compatibility reason, but otherwise I'm afraid I don't really see the point. There is already support for including several source directories, it seems to me that this is just that, another source directory, in this case with tests. I haven't tried that with 2to3, but if that doesn't work, that's definitely a bug. In your case, if you absolutely want to have your code like this, I'd guess subclassing the test command to run 2to3 on the tests first would be the best way.
Tests are an example (well at least the way I view tests) of when this could be useful, it may be useful in other cases (not thought of an example).
-- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
participants (3)
-
Gerry Reno -
Lennart Regebro -
Michael Whapples