Greg Ward wrote:
I've just fixed the behaviour of the "install_data" command: the default directory for installating data files is now just the installation base, which is usually sys.prefix. Thus, if you put
data_files = ["my_data"]
in your setup script, you will wind up installing (eg.) /usr/local/my_data -- definitely the wrong thing on Unix, and probably wrong on Windows too. (Except for applications that have their own prefix, but that's not really dealt with yet.)
This is the Right Thing now because of the change to "install_data" that lets you specify where to put data files; the above should be spelled
data_files = [("share", ["my_data"])]
which will install /usr/local/share/my_data.
Of course, this still doesn't solve the problem of, "How does my application/module know where Distutils installed this data file if the user did some wild funky custom installation?". Oh well, better than nothing.
Would you like to have a complete new concept, which could solve all your problems? Here it is. I would like to introduce a class 'Data_Files' similar to the 'Extension' class. It should accept following parameters: * A basis directory to which all other filenames will become relative. base_dir : 'install_data','install_lib','install_headers' or any other path which is defined in 'install' * A directory under this base dir like the 'share' dir in Greg's example above. copy_to : a path name * a files list as usual files : ['foo','bar', ...] * a list of templates as in MANIFEST.in, which is used to manipulate the files list template : a list or a ';' separeted string * a flag which switches the preserving of paths on or off ( file 'foo/bar' would copied to 'base_dir/foo/bar', instead of 'base_dir/bar' ) preserve_path : 0 or 1 How to use this? Consider you had a distutils.cfg template which you want to install. There are now different ways to so: data_files = [ # The old way (the patch doesn't break old code.): ("lib/python1.5/site-packages/distutils",["distutils/distutils.cfg"]), # this is not portable with win32 # the same using a different base dir Data_Files( base_dir='install_lib', files=["distutils/distutils.cfg"], copy_to='distutils' ), # much better, but you still have to specify the target directory # without target directory Data_Files( base_dir='install_lib', files=["distutils/distutils.cfg"], preserve_path=1, ), # the file has already the rght path so we can use this # and now the most sophisticated solution: Data_Files( base_dir='install_lib', template="recursive-include distutils *.cfg", preserve_path=1, ), I think this can solve all our problems.
Of course, this still doesn't solve the problem of, "How does my application/module know where Distutils installed this data file if the user did some wild funky custom installation?". You only have to create a config file with the path of the other files and put this config file in your packages directory. (see the example above)
Maybe you are not convinced, so here a more complex example. It is taken
from a distutils setup.py file for PyOpenGL. (One of the packages I use
to test distutils.) It installs some examples in its package directory,
these contain data files and some images.
Using the old way looks like this:
# non python files of examples
data_files = [
('lib/python1.5/site-packages/OpenGL/Demo/dek',[
"OpenGL/Demo/dek/README",
"OpenGL/Demo/dek/image.ppm"
]),
('lib/python1.5/site-packages/OpenGL/Demo/dek/OglSurface',[
"OpenGL/Demo/dek/OglSurface/1crn.face",
"OpenGL/Demo/dek/OglSurface/1crn.pdb",
"OpenGL/Demo/dek/OglSurface/1crn.vert",
"OpenGL/Demo/dek/OglSurface/1crn.xyzr",
"OpenGL/Demo/dek/OglSurface/README",
"OpenGL/Demo/dek/OglSurface/test.ppm",
]),
('lib/python1.5/site-packages/OpenGL/Demo/srenner',[
"OpenGL/Demo/srenner/README",
]),
('lib/python1.5/site-packages/OpenGL/Demo/srenner/Images',
glob (os.path.join
("OpenGL/Demo/srenner/Images","*.ppm"))
),
],
And the new way is much simpler (and more portable):
# non python files of examples
data_files = [
Data_Files(
base_dir='install_lib',
# py-files shouldn't be installed with install_data
template=["graft OpenGL","exclude *.py*"] ,
preserve_path=1,
)
],
What does the patch change in distutils?
First I created the class 'Data_Files', which is currently located
in install_data, maybe it should get its own file as Extension did.
There were also some changes in install_data to work with this new
class.
Then I had to take the template proccesing out of the sdist command.
It is now a seperate class in sdist.py. This was necessary because
I want to use it in install_data.
And finally the patch creates a distutils.cfg template and changes
distutils setup.py, so it would install this template.
Do you like it so, or is there anything to change? (Maybe some
names could get better names, but this not a big problem.)
kind regards
Rene Liebscher
diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/install_data.py distutils/distutils/command/install_data.py
--- distutils.orig/distutils/command/install_data.py Mon Jun 26 09:34:37 2000
+++ distutils/distutils/command/install_data.py Mon Jun 26 12:51:52 2000
@@ -7,10 +7,59 @@
__revision__ = "$Id: install_data.py,v 1.10 2000/06/24 17:36:24 gward Exp $"
-import os
-from types import StringType
+import os,sys,string
+from types import StringType,TupleType,ListType
from distutils.core import Command
from distutils.util import change_root
+from distutils.command.sdist import Template_Processor
+
+class Data_Files:
+ """ container for list of data files.
+ supports alternate base_dirs e.g. 'install_lib','install_header',...
+ supports a directory where to copy files
+ supports templates as in MANIFEST.in
+ supports preserving of paths in filenames eg. foo/xyz is copied to base_dir/foo/xyz
+ """
+
+ def __init__(self,base_dir=None,files=None,copy_to=None,template=None,preserve_path=0):
+ self.base_dir = base_dir
+ self.files = files
+ self.copy_to = copy_to
+ self.template = template
+ self.preserve_path = preserve_path
+ self.finalized = 0
+
+ def warn (self, msg):
+ sys.stderr.write ("warning: %s: %s\n" %
+ ("install_data", msg))
+
+ def debug_print (self, msg):
+ """Print 'msg' to stdout if the global DEBUG (taken from the
+ DISTUTILS_DEBUG environment variable) flag is true.
+ """
+ from distutils.core import DEBUG
+ if DEBUG:
+ print msg
+
+
+ def finalize(self):
+ """ complete the files list by processing the given template """
+ if self.finalized:
+ return
+ if self.files == None:
+ self.files = []
+ if self.template != None:
+ if type(self.template) == StringType:
+ self.template = string.split(self.template,";")
+ template_processor = Template_Processor(self.files)
+ template_processor.set_warnings_function(self.warn)
+ template_processor.set_debug_print_function(self.debug_print)
+ for line in self.template:
+ template_processor.process_line(string.strip(line))
+ self.finalized = 1
+
+# end class Data_Files
+
class install_data (Command):
@@ -32,34 +81,86 @@
def finalize_options (self):
self.set_undefined_options('install',
- ('install_data', 'install_dir'),
- ('root', 'root'),
- )
-
+ ('install_data', 'install_dir'),
+ ('root', 'root'),
+ )
+
+ def check_data(self,d):
+ """ check if data are in new format, if not create a suitable object.
+ returns finalized data object
+ """
+ if not isinstance(d, Data_Files):
+ self.warn(("old-style data files list found "
+ "-- please convert to Data_Files instance"))
+ if type(d) is TupleType:
+ if len(d) != 2 or not (type(d[1]) is ListType):
+ raise DistutilsSetupError, \
+ ("each element of 'data_files' option must be an "
+ "Data File instance, a string or 2-tuple (string,[strings])")
+ d = Data_Files(copy_to=d[0],files=d[1])
+ else:
+ if not (type(d) is StringType):
+ raise DistutilsSetupError, \
+ ("each element of 'data_files' option must be an "
+ "Data File instance, a string or 2-tuple (string,[strings])")
+ d = Data_Files(files=[d])
+ d.finalize()
+ return d
+
+
def run (self):
- self.mkpath(self.install_dir)
- for f in self.data_files:
- if type(f) == StringType:
- # it's a simple file, so copy it
- self.warn("setup script did not provide a directory for "
- "'%s' -- installing right in '%s'" %
- (f, self.install_dir))
- out = self.copy_file(f, self.install_dir)
- self.outfiles.append(out)
+ self.outfiles = []
+ install_cmd = self.get_finalized_command('install')
+
+ for d in self.data_files:
+ d = self.check_data(d)
+
+ install_dir = self.install_dir
+ # alternative base dir given => overwrite install_dir
+ if d.base_dir != None:
+ install_dir = getattr(install_cmd,d.base_dir)
+
+ # copy to an other directory
+ if d.copy_to != None:
+ if not os.path.isabs(d.copy_to):
+ # relatiev path to install_dir
+ dir = os.path.join(install_dir, d.copy_to)
+ elif install_cmd.root:
+ # absolute path and alternative root set
+ dir = change_root(self.root,d.copy_to)
+ else:
+ # absolute path
+ dir = d.copy_to
else:
- # it's a tuple with path to install to and a list of files
- dir = f[0]
- if not os.path.isabs(dir):
- dir = os.path.join(self.install_dir, dir)
- elif self.root:
- dir = change_root(self.root, dir)
- self.mkpath(dir)
- for data in f[1]:
- out = self.copy_file(data, dir)
- self.outfiles.append(out)
+ # simply copy to install_dir
+ dir = install_dir
+
+ dir=os.path.normpath(dir)
+ # create path
+ self.mkpath(dir)
+
+ # copy all files
+ for f in d.files:
+ if d.copy_to == None:
+ self.warn("setup script did not provide a directory for "
+ "'%s' -- installing right in '%s'" %
+ (f, install_dir))
+ if d.preserve_path:
+ # preserve path in filename
+ self.mkpath(os.path.dirname(os.path.join(dir,f)))
+ out = self.copy_file(f, os.path.join(dir,f))
+ else:
+ out = self.copy_file(f, dir)
+ self.outfiles.append(out)
+
+ return self.outfiles
def get_inputs (self):
- return self.data_files or []
-
+ inputs = []
+ for d in self.data_files:
+ d = self.check_data(d)
+ inputs.append(d.files)
+ return inputs
+
def get_outputs (self):
return self.outfiles
diff -BurN --minimal --exclude=*.pyc distutils.orig/distutils/command/sdist.py distutils/distutils/command/sdist.py
--- distutils.orig/distutils/command/sdist.py Mon Jun 26 09:34:37 2000
+++ distutils/distutils/command/sdist.py Mon Jun 26 12:39:08 2000
@@ -34,6 +34,260 @@
"List of available source distribution formats:")
+class Template_Processor:
+
+ def __init__(self, files_list=[]):
+ self.files=files_list
+ self.all_files = findall ()
+ self.template_warn=self.dummy_function
+ self.debug_print=self.dummy_function
+
+ def set_warnings_function(self, warnings_function):
+ self.template_warn=warnings_function
+
+ def set_debug_print_function(self, debug_print_function):
+ self.debug_print=debug_print_function
+
+ def dummy_function(self,warning):
+ pass
+
+ def search_dir (self, dir, pattern=None):
+ """Recursively find files under 'dir' matching 'pattern' (a string
+ containing a Unix-style glob pattern). If 'pattern' is None, find
+ all files under 'dir'. Return the list of found filenames.
+ """
+ allfiles = findall (dir)
+ if pattern is None:
+ return allfiles
+
+ pattern_re = translate_pattern (pattern)
+ files = []
+ for file in allfiles:
+ if pattern_re.match (os.path.basename (file)):
+ files.append (file)
+
+ return files
+
+ # search_dir ()
+
+ def recursive_exclude_pattern (self, dir, pattern=None):
+ """Remove filenames from 'self.files' that are under 'dir' and
+ whose basenames match 'pattern'.
+ """
+ self.debug_print("recursive_exclude_pattern: dir=%s, pattern=%s" %
+ (dir, pattern))
+ if pattern is None:
+ pattern_re = None
+ else:
+ pattern_re = translate_pattern (pattern)
+
+ for i in range (len (self.files)-1, -1, -1):
+ (cur_dir, cur_base) = os.path.split (self.files[i])
+ if (cur_dir == dir and
+ (pattern_re is None or pattern_re.match (cur_base))):
+ self.debug_print("removing %s" % self.files[i])
+ del self.files[i]
+
+
+ def process_line (self, line):
+ """Read and parse a line of a template file. Process all file
+ specifications (include and exclude) in the template and
+ update 'self.files' accordingly (filenames may be added to
+ or removed from 'self.files' based on the template).
+ """
+ assert self.files is not None and type (self.files) is ListType
+
+
+ words = string.split (line)
+ action = words[0]
+
+ # First, check that the right number of words are present
+ # for the given action (which is the first word)
+ if action in ('include','exclude',
+ 'global-include','global-exclude'):
+ if len (words) < 2:
+ self.template_warn \
+ ("invalid manifest template line: " +
+ "'%s' expects <pattern1> <pattern2> ..." %
+ action)
+ return
+
+ pattern_list = map(convert_path, words[1:])
+
+ elif action in ('recursive-include','recursive-exclude'):
+ if len (words) < 3:
+ self.template_warn \
+ ("invalid manifest template line: " +
+ "'%s' expects <dir> <pattern1> <pattern2> ..." %
+ action)
+ return
+
+ dir = convert_path(words[1])
+ pattern_list = map (convert_path, words[2:])
+
+ elif action in ('graft','prune'):
+ if len (words) != 2:
+ self.template_warn \
+ ("invalid manifest template line: " +
+ "'%s' expects a single