distribute 0.6.10 and convert_2to3_doctests
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I have a Python package called 'munepy' https://launchpad.net/munepy which provides yet another flavor of enums. I'm working on the code for various reasons and I thought I'd take the opportunity to learn how to support both Python 2 and 3 from the same code base.
I've updated the code so that it supports Python 2.6 at a minimum, and made it 'python -3' clean. I've switched it from using setuptools to using distribute. I was looking at this page:
http://packages.python.org/distribute/python3.html
and it seems very cool that distribute can help make my life easier by allowing me to support both Python 2 and 3 from the same code base. I set use_2to3=True in my setup.py and indeed
% python3 setup.py test
(On Ubuntu 9.10) runs 2to3 over my .py files. However, my doctests are in separate .txt files and I cannot seem to get the right incantation for convert_2to3_doctests.
In the root of my source directory, my doctest lives at munepy/docs/README.txt, so I put this in my setup.py:
... use_2to3 = True, convert_2to3_doctests = [ 'munepy/docs/README.txt', ], ...
but I never see that the README.txt is ever 'fixed'. Indeed, the test fails because of a syntax error when the doctest tries to print something using Python 2 syntax. I'm clearly not using this setup argument correctly, but I can't tell where I'm going wrong.
How do you use convert_2to3_doctests?
TIA, - -Barry
On Thu, Jan 7, 2010 at 00:17, Barry Warsaw barry@python.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I have a Python package called 'munepy' https://launchpad.net/munepy which provides yet another flavor of enums. I'm working on the code for various reasons and I thought I'd take the opportunity to learn how to support both Python 2 and 3 from the same code base.
I've updated the code so that it supports Python 2.6 at a minimum, and made it 'python -3' clean. I've switched it from using setuptools to using distribute. I was looking at this page:
http://packages.python.org/distribute/python3.html
and it seems very cool that distribute can help make my life easier by allowing me to support both Python 2 and 3 from the same code base. I set use_2to3=True in my setup.py and indeed
% python3 setup.py test
(On Ubuntu 9.10) runs 2to3 over my .py files. However, my doctests are in separate .txt files and I cannot seem to get the right incantation for convert_2to3_doctests.
In the root of my source directory, my doctest lives at munepy/docs/README.txt, so I put this in my setup.py:
... use_2to3 = True, convert_2to3_doctests = [ 'munepy/docs/README.txt', ], ...
but I never see that the README.txt is ever 'fixed'. Indeed, the test fails because of a syntax error when the doctest tries to print something using Python 2 syntax. I'm clearly not using this setup argument correctly, but I can't tell where I'm going wrong.
How do you use convert_2to3_doctests?
Exactly like this. Note, however, that Distribute doesn't know that the file isn't already converted. It will look at the timestamps of the files, notice it hasn't changed, and do nothing. So you need to delete the target file in build/ and rerun the tests to have it run the conversion.
On Jan 7, 2010, at 6:09 AM, Lennart Regebro wrote:
... use_2to3 = True, convert_2to3_doctests = [ 'munepy/docs/README.txt', ], ...
but I never see that the README.txt is ever 'fixed'. Indeed, the test fails because of a syntax error when the doctest tries to print something using Python 2 syntax. I'm clearly not using this setup argument correctly, but I can't tell where I'm going wrong.
How do you use convert_2to3_doctests?
Exactly like this. Note, however, that Distribute doesn't know that the file isn't already converted. It will look at the timestamps of the files, notice it hasn't changed, and do nothing. So you need to delete the target file in build/ and rerun the tests to have it run the conversion.
Hi Lennart, thanks for the response. However, I don't think this is quite it. I read this in the docs and deleted the entire build directory. Then when I re-run the tests I can see that the .py files get "fixed" but it never tries to fix the README.txt.
-Barry
On Thu, Jan 7, 2010 at 12:31, Barry Warsaw barry@python.org wrote:
Hi Lennart, thanks for the response. However, I don't think this is quite it. I read this in the docs and deleted the entire build directory. Then when I re-run the tests I can see that the .py files get "fixed" but it never tries to fix the README.txt.
I just checked out munepy, and tried it, and with me the problem is that it doesn't even copy it. That's because Distribute sees any non-python file as "Package data", which means you have to set include_package_data = True in setup().
There is a bug here, or several. I suspect we should try to copy doctest files in this case, even if you don't have include_package_data, and also Distribute should complain if the specified doctest file doesn't exist or isn't being copied.
But in any case, once I add "include_package_data = True" it works for me, the file does get copied and converted.
The next problem is that the tests seem to be run on the original, and not on the build-copy. And why that is, I don't know, and I have to debug that, which I can't do right now. I'll try tonight or tomorrow. It's most likely a bug in Distribute.
On Jan 07, 2010, at 02:34 PM, Lennart Regebro wrote:
I just checked out munepy, and tried it, and with me the problem is that it doesn't even copy it. That's because Distribute sees any non-python file as "Package data", which means you have to set include_package_data = True in setup().
Ah yes, of course. Thanks, that's definitely a bug in my setup.py. I've fixed this and pushed up the latest revisions. I will upload the package to pypi soon.
There is a bug here, or several. I suspect we should try to copy doctest files in this case, even if you don't have include_package_data, and also Distribute should complain if the specified doctest file doesn't exist or isn't being copied.
Either way seems fine. It would be nice to have more automatic support of doctests in general, but I'm also okay with Distribute just complaining and making me fix my setup.py.
But in any case, once I add "include_package_data = True" it works for me, the file does get copied and converted.
The next problem is that the tests seem to be run on the original, and not on the build-copy. And why that is, I don't know, and I have to debug that, which I can't do right now. I'll try tonight or tomorrow. It's most likely a bug in Distribute.
Thanks. I wonder if my test_documentation.py is doing evil things that confuse Distribute?
I really think getting this right will provide a very powerful incentive to Python package authors, both to switch to Distribute and to support Python 3. I'm very excited to see this functionality.
-Barry
On Thu, Jan 7, 2010 at 14:34, Lennart Regebro regebro@gmail.com wrote:
The next problem is that the tests seem to be run on the original, and not on the build-copy. And why that is, I don't know, and I have to debug that, which I can't do right now. I'll try tonight or tomorrow. It's most likely a bug in Distribute.
Well, kinda. :)
The problem is this code in setup.py:
from munepy import __version__
That means munepy is already imported, so when it then looks for the tests to run, it will import it from the already imported munepy module. Changing it to
from munepy import __version__ del sys.modules['munepy']
Goes around the problem, but it's admittedly not a pretty fix. I'm sure others will be bitten by this too. I'm not sure what a good fix would be, except possibly removing all modules specified in setup.py from sys.modules before running any commands.
On Jan 26, 2010, at 05:53 PM, Lennart Regebro wrote:
On Thu, Jan 7, 2010 at 14:34, Lennart Regebro regebro@gmail.com wrote:
The next problem is that the tests seem to be run on the original, and not on the build-copy. And why that is, I don't know, and I have to debug that, which I can't do right now. I'll try tonight or tomorrow. It's most likely a bug in Distribute.
Well, kinda. :)
The problem is this code in setup.py:
from munepy import __version__
That means munepy is already imported, so when it then looks for the tests to run, it will import it from the already imported munepy
Ah. We've run into similar problems in other contexts. On the face of it, I think it would be wonderful to be able to set the setup.py version number this way, but in practice it seems too problematic. I think this ends up biting namespace packages fairly hard too.
module. Changing it to
from munepy import __version__ del sys.modules['munepy']
Goes around the problem, but it's admittedly not a pretty fix. I'm sure others will be bitten by this too. I'm not sure what a good fix would be, except possibly removing all modules specified in setup.py from sys.modules before running any commands.
I'm going to make the following change to setup.py instead:
=== modified file 'setup.py' --- setup.py 2010-01-08 21:59:23 +0000 +++ setup.py 2010-01-29 20:55:12 +0000 @@ -18,9 +18,22 @@ import distribute_setup distribute_setup.use_setuptools()
+import re import sys
-from munepy import __version__ +# Do not import __version__ from munepy. Doing so causes problems with +# doctests running under 2to3. +with open('munepy/__init__.py') as fp: + for line in fp: + if line.startswith('__version__'): + mo = re.search(r'\d+.\d+.\d', line) + assert mo, 'No valid __version__ string found' + __version__ = mo.group(0) + break + else: + raise AssertionError('No __version__ assignment found') + + from setuptools import setup, find_packages
However, it would be nice if setuptools/distribute supported something like this out of the box. The important thing is to have exactly one place to set the package's version number.
I've had to make a few other changes to the code to get it to pass under 2to3, but I don't think those are on topic for this mailing list, so I'll skip them.
Thanks for looking into this! -Barry
On Fri, Jan 29, 2010 at 10:05 PM, Barry Warsaw barry@python.org wrote: [..]
However, it would be nice if setuptools/distribute supported something like this out of the box. The important thing is to have exactly one place to set the package's version number.
For metadata fields like "version", one option I am working on in Distutils itself is to have a complementary section in the static setup.cfg file, where you can set some fields:
[setup] name=foo version=1.9.8
Once setup() is run, the Distribution class will look at this file, to complete the options provided by code;
Of course this supposes that the version is not calculated by some code (like in your solution). But I think a plain, non-development version, can be static.
Tarek
On Fri, Jan 29, 2010 at 22:27, Tarek Ziadé ziade.tarek@gmail.com wrote:
On Fri, Jan 29, 2010 at 10:05 PM, Barry Warsaw barry@python.org wrote: [..]
However, it would be nice if setuptools/distribute supported something like this out of the box. The important thing is to have exactly one place to set the package's version number.
For metadata fields like "version", one option I am working on in Distutils itself is to have a complementary section in the static setup.cfg file, where you can set some fields:
[setup] name=foo version=1.9.8
Once setup() is run, the Distribution class will look at this file, to complete the options provided by code;
Of course this supposes that the version is not calculated by some code (like in your solution). But I think a plain, non-development version, can be static.
What you want is then a simple way of getting this version number into the python code. Any recommendations for that?
On Fri, Jan 29, 2010 at 10:29 PM, Lennart Regebro regebro@gmail.com wrote:
On Fri, Jan 29, 2010 at 22:27, Tarek Ziadé ziade.tarek@gmail.com wrote:
On Fri, Jan 29, 2010 at 10:05 PM, Barry Warsaw barry@python.org wrote: [..]
However, it would be nice if setuptools/distribute supported something like this out of the box. The important thing is to have exactly one place to set the package's version number.
For metadata fields like "version", one option I am working on in Distutils itself is to have a complementary section in the static setup.cfg file, where you can set some fields:
[setup] name=foo version=1.9.8
Once setup() is run, the Distribution class will look at this file, to complete the options provided by code;
Of course this supposes that the version is not calculated by some code (like in your solution). But I think a plain, non-development version, can be static.
What you want is then a simple way of getting this version number into the python code. Any recommendations for that?
It depends on what code. Let's say that we want to have one and only one place for the "version" value.
There are two type of code in a Distutils-based project. The one that is used in setup.py to create distribution, compile extensions (I'll call it the "build code"), and the one that is installed on the target system.
For the build code, if Distutils has the capability to get metadata from setup.cfg, then it would be just a matter of publishing an API for getting it as well in there.
Now, if the project is installed, setup.[py/cfg] are gone, and the only place where the version is located is the PKG-INFO file. Distribute/Setuptools provides and APIs to read this PKG-INFO file and get it already. IOW, in the packages and modules of the project, this API can be used.
The only concern I have is that the development mode (e.g. getting the version in the packages and modules of the project when it's not installed) supposes that the "develop" command was called (so that the PKG-INFO file is locally available).
But I would definitely completely isolate setup.py code from the project' packages/modules to avoid problems.
Tarek
On Jan 29, 2010, at 10:27 PM, Tarek Ziadé wrote:
For metadata fields like "version", one option I am working on in Distutils itself is to have a complementary section in the static setup.cfg file, where you can set some fields:
[setup] name=foo version=1.9.8
Once setup() is run, the Distribution class will look at this file, to complete the options provided by code;
Of course this supposes that the version is not calculated by some code (like in your solution). But I think a plain, non-development version, can be static.
Actually, my version number is a static assignment in __init__.py:
__version__ = '2.0.2'
and I just manually bump that with every release. I'd be just as happy to bump it in setup.cfg (maybe more!) as long as there was an API that I could add to __init__.py to assign it to my module's namespace, e.g. in mypkg/__init__.py:
from distribute.resources import get_version __version__ = get_version('mypkg')
I don't much care how that's spelled. The important things are:
* One place to bump version numbers * Available from setup.py without importing from the package * Available from the package's namespace
-Barry
On Fri, Jan 29, 2010 at 10:54 PM, Barry Warsaw barry@python.org wrote: [..]
- One place to bump version numbers
- Available from setup.py without importing from the package
So yes, setup.cfg would work here.
- Available from the package's namespace
Yes, that's how Jinja does already for example, using Setuptools's pkg_resources :
__version__ = __import__('pkg_resources').get_distribution('Jinja2').version
see http://dev.pocoo.org/projects/jinja/browser/jinja2/__init__.py
Notice that this pkg_resources browsing feature would be in Distutils once PEP 376 is accepted, so until then, I could probably backport in Distribute the setup.cfg feature.
Tarek
On Jan 29, 2010, at 11:03 PM, Tarek Ziadé wrote:
Yes, that's how Jinja does already for example, using Setuptools's pkg_resources :
__version__ = __import__('pkg_resources').get_distribution('Jinja2').version
And that's different yet again from what PJE suggests. This is screaming for a blessed API to be pushed into the stdlib.
(BTW, why use __import__() there?)
see http://dev.pocoo.org/projects/jinja/browser/jinja2/__init__.py
Notice that this pkg_resources browsing feature would be in Distutils once PEP 376 is accepted, so until then, I could probably backport in Distribute the setup.cfg feature.
That would be cool. -Barry
At 07:08 PM 1/30/2010 -0500, Barry Warsaw wrote:
On Jan 29, 2010, at 11:03 PM, Tarek Ziadé wrote:
Yes, that's how Jinja does already for example, using Setuptools's pkg_resources :
__version__ = __import__('pkg_resources').get_distribution('Jinja2').version
And that's different yet again from what PJE suggests.
Not really; just a different way of spelling the same thing. In either case, you're looking at the .version attribute of the Distribution instance.
This is screaming for a blessed API to be pushed into the stdlib.
(BTW, why use __import__() there?)
It lets you save an extra line to import pkg_resources; that's really the only difference.
At 07:08 PM 1/30/2010 -0500, Barry Warsaw wrote:
On Jan 29, 2010, at 11:03 PM, Tarek Ziadé wrote:
Yes, that's how Jinja does already for example, using Setuptools's pkg_resources :
__version__ = __import__('pkg_resources').get_distribution('Jinja2').version
And that's different yet again from what PJE suggests.
*shrug* A mere stylistic difference. My spelling's just a shorter way to do the same thing, while at the same time asserting the package's requirements. But there's nothing *wrong* with the above way of doing it.
Personally, I think that querying a package's version is generally a wrongheaded idea in the first place. If you need a specific version, asserting this via your project's metadata or a require() line in your script is the way to go. If you are querying API existence, hasattr() is the way to go.
That kind of leaves working around bugs in a specific version or two as the only sensible use case for a version check... in which case, you're still better off just querying the installed version rather than asking the package for a __version__.
This is screaming for a blessed API to be pushed into the stdlib.
(BTW, why use __import__() there?)
To make it a one-liner, I would guess, in the case where you're not doing anything else with pkg_resources.
Hi Tarek,
For metadata fields like "version", one option I am working on in Distutils itself is to have a complementary section in the static setup.cfg file, where you can set some fields:
[setup] name=foo version=1.9.8
Interesting.
Well it was discussed on the mailing list extensively last year and I even wrote a PEP proposal that got blocked.. :-)
If I understand correctly, these parameters in the setup.cfg file would be similar to those passed into the setup() function to distutils.
I'm asking this question, knowing that the answer is probably that it was before your time. But it doesn't hurt asking.
Do you have a test suite of parameters for the setup() function in distutils?
Whichever way I look at this problem I keep coming back to the issue that unless there is a test suite (TDD style) that can be used as inputs, there's no viable way to come up with equivilent functionality through a metadata based system.
To me, there is simply too many combinations of parameters used in funny ways to obtain the results that the developers are after.
Does a Parameter testsuite for the setup() function exist for distutils ? If not, should we be considering building one? (for distutils)
David
At 04:05 PM 1/29/2010 -0500, Barry Warsaw wrote:
The important thing is to have exactly one place to set the package's version number.
Put it in setup.py, then. If you absolutely must have a __version__ at runtime, use this to extract it from the installation metadata:
__version__ = pkg_resources.require('MyProject')[0].version
Mostly, though, I don't bother with having a __version__ in my modules or packages any more, since you can just do the above if you want to check the installed version of something.
"P.J. Eby" pje@telecommunity.com writes:
At 04:05 PM 1/29/2010 -0500, Barry Warsaw wrote:
The important thing is to have exactly one place to set the package's version number.
Put it in setup.py, then. If you absolutely must have a __version__ at runtime, use this to extract it from the installation metadata:
__version__ = pkg_resources.require('MyProject')[0].versionMostly, though, I don't bother with having a __version__ in my modules or packages any more, since you can just do the above if you want to check the installed version of something.
That assumes that the only things that will need to query the package's version are Python modules. That's often not the case, especially when there are other tools (e.g. shell programs, make files, or any program not written in Python) that are part of the same package.
Better would be to have a *non-executable* data file containing the version string and other such package meta-data. “Query the metadata” should not necessarily imply “parse or execute a bunch of Python code”.
On Jan 29, 2010, at 7:32 PM, Ben Finney wrote:
"P.J. Eby" pje@telecommunity.com writes:
At 04:05 PM 1/29/2010 -0500, Barry Warsaw wrote:
The important thing is to have exactly one place to set the package's version number.
Put it in setup.py, then. If you absolutely must have a __version__ at runtime, use this to extract it from the installation metadata:
__version__ = pkg_resources.require('MyProject')[0].version
Mostly, though, I don't bother with having a __version__ in my modules or packages any more, since you can just do the above if you want to check the installed version of something.
That assumes that the only things that will need to query the package's version are Python modules. That's often not the case, especially when there are other tools (e.g. shell programs, make files, or any program not written in Python) that are part of the same package.
Better would be to have a *non-executable* data file containing the version string and other such package meta-data. “Query the metadata” should not necessarily imply “parse or execute a bunch of Python code”.
I'd love to have a standard, documented way to set/query module versions.
I actually started this e-mail about a week ago when I went looking for a "standard" way to version something the other day and, after I couldn't find a definitive answer by Googling, went looking in the stdlib for guidance.
I started with two core modules, sys and distutils, and tried just doing the same things to both modules:
import sys
sys.version
'2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) \n[GCC 4.0.1 (Apple Inc. build 5493)]'
sys.version_info
(2, 6, 4, 'final', 0)
sys.__version__
Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute '__version__'
#--------------------------------------------
import distutils
distutils.version
<module 'distutils.version' from '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/version.pyc'>
distutils.version_info
Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'version_info'
distutils.__version__
'2.6.4'
The standard library, and modules in PyPI, are all over the place on it so maybe a completely new API/method is in order.
S
At 11:32 AM 1/30/2010 +1100, Ben Finney wrote:
That assumes that the only things that will need to query the package's version are Python modules. That's often not the case, especially when there are other tools (e.g. shell programs, make files, or any program not written in Python) that are part of the same package. Better would be to have a *non-executable* data file containing the version string and other such package meta-data. âQuery the metadataâ should not necessarily imply âparse or execute a bunch of Python codeâ.
Which is precisely why that way's better; pkg_resources is just parsing the info from a PKG-INFO file -- or more commonly, just parsing a file or directory *name*, without even opening a file. So other tools can certainly do the same.
On Jan 29, 2010, at 06:53 PM, P.J. Eby wrote:
At 04:05 PM 1/29/2010 -0500, Barry Warsaw wrote:
The important thing is to have exactly one place to set the package's version number.
Put it in setup.py, then. If you absolutely must have a __version__ at runtime, use this to extract it from the installation metadata:
__version__ = pkg_resources.require('MyProject')[0].versionMostly, though, I don't bother with having a __version__ in my modules or packages any more, since you can just do the above if you want to check the installed version of something.
That's good to know. What about adding an API to make that even simpler:
pkg_resources.get_version('MyProject')
?
-Barry
Teilnehmer (7)
-
Barry Warsaw -
Ben Finney -
David Lyon -
Lennart Regebro -
P.J. Eby -
ssteinerX＠gmail.com -
Tarek Ziadé