[setuptools] When .py files are "data", strange things happen
I want setuptools to install a bunch of sample scripts in a package-relative directory. Each end user will later run a script that copies these to a directory relative to his or her home directory. However, strange things happen to data files that have a .py extension. 1) Saying "include_package_data = True" was not enough to get them included in the egg. The .txt files were included, but the .py files were magically excluded. 2) I overcame the above problem with the following setup parameter: package_data = { # Include all .py files in the data directories, they # are excluded by default. 'cpif.data': ['lander/*.py', 'samples/*.py'], } But now I get a bogus .pyc file generated for each of these sample scripts. These bogus .pyc files get included in the egg. 3) I tried to overcome problem #2 above with the following setup parameter: exclude_package_data = { # The example programs should not be shipped with .pyc files # XXX This is not working, the .pyc files are being created and # included anyway. 'cpif.data': ['lander/*.pyc', 'samples/*.pyc'], } But this didn't work. Putting the following lines in my MANIFEST.in file also didn't work: exclude cpif/data/lander/*.pyc exclude cpif/data/samples/*.pyc I suppose I could turn off .pyc file generation, but that's not what I want. I just want to treat certain .py files as data files. The "data" .py files are in a subdirectory of a python package, but the directory in which they directly reside does not contain an __init__.py file, so they are not directly importable. So I don't think .pyc files should be generated for them in any case. Thanks in advance for ideas/sympathy/fixes, etc. -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/
On 1/5/06, David Handy <david@handysoftware.com> wrote:
I just want to treat certain .py files as data files. The "data" .py files are in a subdirectory of a python package, but the directory in which they directly reside does not contain an __init__.py file, so they are not directly importable. So I don't think .pyc files should be generated for them in any case.
Do they *have* to end in .py? I've seen apps that end in .cfg or .dat and happly execfile them into their program space. This is not to say that setuptools shouldn't be improved. I'm just curious as to the choice of the extension. (syntax highlighting concerns can be alleviated for vim and emacs by using a comment). -- -jeff
At 10:54 PM 1/4/2006 -0500, David Handy wrote:
I want setuptools to install a bunch of sample scripts in a package-relative directory. Each end user will later run a script that copies these to a directory relative to his or her home directory.
However, strange things happen to data files that have a .py extension.
1) Saying "include_package_data = True" was not enough to get them included in the egg. The .txt files were included, but the .py files were magically excluded.
2) I overcame the above problem with the following setup parameter:
package_data = { # Include all .py files in the data directories, they # are excluded by default. 'cpif.data': ['lander/*.py', 'samples/*.py'], }
But now I get a bogus .pyc file generated for each of these sample scripts. These bogus .pyc files get included in the egg.
If I understand your use case correctly, I don't see how that hurts anything. The simplest way to solve your problem, however, is to put the samples in a subdirectory of your project's .egg-info directory, and then use the metadata APIs to access them. .py files under .egg-info do not get compiled. Meanwhile, it's not really practical to have .py files inside a package be included, but not compiled. So your three choices are: * Give the files a different extension (e.g. .py.orig or .py.in or .cfg or whatever), which will then trivially work with include_package_data=True * Include them explicitly as .py files, as shown above (and then ignore the irrelevant .pyc files) * Put them in a subdirectory of the project's .egg-info, and then use the pkg_resources metadata APIs at runtime to get access.
On Thu, Jan 05, 2006 at 12:07:39PM -0500, Phillip J. Eby wrote:
At 10:54 PM 1/4/2006 -0500, David Handy wrote:
I want setuptools to install a bunch of sample scripts in a package-relative directory. Each end user will later run a script that copies these to a directory relative to his or her home directory.
However, strange things happen to data files that have a .py extension.
1) Saying "include_package_data = True" was not enough to get them included in the egg. The .txt files were included, but the .py files were magically excluded.
2) I overcame the above problem with the following setup parameter:
package_data = { # Include all .py files in the data directories, they # are excluded by default. 'cpif.data': ['lander/*.py', 'samples/*.py'], }
But now I get a bogus .pyc file generated for each of these sample scripts. These bogus .pyc files get included in the egg.
If I understand your use case correctly, I don't see how that hurts anything.
My project is not the typical software component - it is a kit for beginning programmers. They are following the directions in my book. The .py "data" files I refer to are a whole bunch of simple sample programs that users are supposed to refer to as they go through my book. Really, they're documentation more than code. Most of my users won't navigate to the package directory itself, but instead will work with copies of those files in their "MyPrograms" directory. But those who do poke around (like I always do), and know a little bit of Python, would wonder, "why are these single file programs, that are never imported, compiled to .pyc files anyway? Am I missing something?" I didn't want the answer to be "they're compiled because the author didn't try to learn how to use setuptools properly." :) But it's not a huge issue.
The simplest way to solve your problem, however, is to put the samples in a subdirectory of your project's .egg-info directory, and then use the metadata APIs to access them. .py files under .egg-info do not get compiled.
Meanwhile, it's not really practical to have .py files inside a package be included, but not compiled. So your three choices are:
* Give the files a different extension (e.g. .py.orig or .py.in or .cfg or whatever), which will then trivially work with include_package_data=True
I prefer to keep the names intact to make the packaging itself as transparent as possible.
* Include them explicitly as .py files, as shown above (and then ignore the irrelevant .pyc files)
That's the path of least resistance right now. The welcome script that copies the files to the user's MyPrograms directory will ignore .pyc files.
* Put them in a subdirectory of the project's .egg-info, and then use the pkg_resources metadata APIs at runtime to get access.
I'll probably do this eventually. Right now I am not installing setuptools with my package, just using it to create the installer. Thanks for the advice! -- David Handy Computer Programming is Fun! Beginning Computer Programming with Python http://www.handysoftware.com/cpif/
participants (3)
-
David Handy -
Jeff Pitman -
Phillip J. Eby