Mailman 3 July 2001 - Distutils-SIG

Package DB: strawman PEP
by Andrew Kuchling July 17, 2001

July 17, 2001

It seems time to bite the bullet and actually begin designing and implementing a database of installed packages. As a strawman to get a focused discussion started, here's a draft of a PEP, with lots of XXX's in it. Followups to the Distutils SIG, please. --amk PEP: XXX Title: A Database of Installed Python Packages Version: $Revision: 1.1 $ Author: A.M. Kuchling <akuchlin(a)mems-exchange.org> Type: Standards Track Created: 08-Jul-2001 Status: Draft Post-History: Introduction This PEP describes a format for a database of Python packages installed on a system. Requirements We need a way to figure out what packages, and what versions of those packages, are installed on a system. We want to provide features similar to CPAN, APT, or RPM. Required use cases that should be supported are: * Is package X on a system? * What version of package X is installed? * Where can the new version of package X be found? XXX Does this mean "a home page where the user can go and find a download link", or "a place where a program can find the newest version?" Perhaps both... * What files did package X put on my system? * What package did the file x/y/z.py come from? * Has anyone modified x/y/z.py locally? Database Location The database lives in a bunch of files under <prefix>/lib/python<version>/install/. This location will be called INSTALLDB through the remainder of this PEP. XXX is that a good location? What effect does platform-dependent code vs. platform-independent code have on this? The structure of the database is deliberately kept simple; each file in this directory or its subdirectories (if any) describes a single package. The rationale for scanning subdirectories is that we can move to a directory-based indexing scheme if the package directory contains too many entries. That is, instead of $INSTALLDB/Numeric, we could switch to $INSTALLDB/N/Nu/Numeric or some similar scheme. XXX how much do we care about performance? Do we really need to use an anydbm file or something similar? XXX is the actual filename important? Let's say the installation data for PIL is in the file INSTALLDB/Numeric. Is this OK? When we want to figure out if Numeric is installed, do we want to open a single file, or have to scan them all? Note that for human-interface purposes, we'll often have to scan all the packages anyway, for a case-insensitive or keyword search. Database Contents Each file in $INSTALLDB or its subdirectories describes a single package, and has the following contents: An initial line listing the sections in this file, separated by whitespace. Currently this will always be 'PKG-INFO FILES'. This is for future-proofing; if we add a new section, for example to list documentation files, then we'd add a DOCS section and list it in the contents. Sections are always separated by blank lines. XXX too simple? [PKG-INFO section] An initial set of RFC-822 headers containing the package information for a file, as described in PEP 241, "Metadata for Python Software Packages". A blank line indicating the end of the PKG-INFO section. An entry for each file installed by the package. XXX Are .pyc and .pyo files in this list? What about compiled .so files? AMK thinks "no" and "yes", respectively. Each file's entry is a single tab-delimited line that contains the following fields: XXX should each file entry be all on one line and tab-delimited? More RFC-822 headers? AMK thinks tab-delimited seems sufficent. * The file's size * XXX do we need to store permissions? The owner/group? * An MD5 digest of the file, written in hex. (XXX All 16 bytes of the digest seems unnecessary; first 8 bytes only, maybe? Is a zlib.crc32() hash sufficient?) * The file's full path, as installed on the system. (XXX should it be relative to sys.prefix, or sys.prefix + '/lib/python<version>?' If so, full paths are still needed; consider a package that installs a startup script such as /etc/init.d/zope) * XXX some sort of type indicator, to indicate whether this is a Python module, binary module, documentation file, config file? Do we need this? A package that uses the Distutils for installation will automatically update the database. Packages that roll their own installation XXX what's the relationship between this database and the RPM or DPKG database? I'm tempted to make the Python database completely optional; a distributor can preserve the interface of the package management tool and replace it with their own wrapper on top of their own package manager. (XXX but how would the Distutils know that, and not bother to update the Python database?) Deliverables Patches to the Distutils that 1) implement a InstallationDatabase class, 2) Update the database when a new package is installed. 3) a simple package management tool, features to be added to this PEP. (Or a separate PEP?) References [1] Michael Muller's patch (posted to the Distutils-SIG around 28 Dec 1999) generates a list of installed files. Acknowledgements Ideas for this PEP originally came from postings by Greg Ward, Fred Drake, Mats Wichmann, and others. Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil End:

6 7

RE: [Distutils] Distutils enhancement - set variables from the co mmand line
by Moore, Paul July 17, 2001

July 17, 2001

>From: M.-A. Lemburg [mailto:mal@lemburg.com] > Paul Moore wrote: >> My suggestion would be to allow the "global_opts" or "cmdX_opts" to contain >> arguments of the form key=value. These can be exposed to setup.py as a >> dictionary, say distutils.args. >> >> What do people think? > > Doesn't setup.cfg provide enough means for the user to customize the > distutils behaviour ? Not really. As far as I can see from the documentation, it allows the user to set existing command line options. It doesn't allow the user to supply arbitrary configuration information. The example I hit this with was installing a module which depended on the JPEG library. On Windows, there is no "standard" location where such a library would be installed. It is not unlikely, however, that the user might have built the library just to install this extension - so a reasonable default might be ..\jpeg-6b. So I envision setup.py having something like: JPEG_LOCN = distutils.args.get('JPEG_LOCN', '../jpeg-6b') The user can then either run python setup.py build or python setup.py build JPEG_LOCN=C:\LocalLibs\JPEG depending on his setup. Paul.

3 2

RE: [Distutils] Why doesn't distutils respect the INCLUDE/LIB en vironment variable (MSVC)?
by Moore, Paul July 16, 2001

July 16, 2001

From: M.-A. Lemburg [mailto:mal@lemburg.com] > I don't know about the convention for looking up parameters > on Windows: do env vars override settings in the registry > or the other way around ? I just checked. It's actually worse than I thought. The environment variables are the *only* thing that affect the command-line tools. The registry is purely for the IDE, I guess. So what distutils is doing, rather than being unhelpful, is actually wrong... It *is* possible to install Visual C++ in such a way that the environment variables are not set by default, leaving the user to run a BAT file to set them up before the command line tools are usable. It's possible that the distutils approach is to cater for such users, by registering the appropriate variables even if VCVARS32.BAT has not been run. > Depending on what the standard procedure is on Windows, I'd > suggest to either prepend or append the env var settings to the > LIB/INCLUDE values found through the registry. Based on the above, I'd suggest not grubbing through the registry at all - just leave things as the user has them. Or to handle the case where the user hasn't set the environment variables, maybe check whether INCLUDE is set - if not, either prompt the user (probably better, as VCVARS32.BAT definitely does the right thing), or grub through the registry in that case only. A bare minimum patch (which just leaves the vars if set, otherwise works as at present) could be (UNTESTED): --- msvccompiler.py.orig Thu Apr 19 10:24:24 2001 +++ msvccompiler.py Mon Jul 16 14:02:05 2001 @@ -155,8 +155,11 @@ def set_path_env_var (name, version_number): """Set environment variable 'name' to an MSVC path type value obtained from 'get_msvc_paths()'. This is equivalent to a SET command prior - to execution of spawned commands.""" + to execution of spawned commands. If the environment variable is + already set, leave it alone.""" + if os.environ.has_key(name) + return p = get_msvc_paths (name, version_number) if p: os.environ[name] = string.join (p,';') Paul.

1 0

Why doesn't distutils respect the INCLUDE/LIB environment variable (MSVC)?
by Paul Moore July 16, 2001

July 16, 2001

I'm trying to build an extension (Imaging) which depends on a number of external libraries (JPEG, Zlib, Tk). Rather than hard code the locations of the libraries in the setup.py, I set my INCLUDE and LIB environment variables so that MSVC picks up the libraries automatically. Unfortunately, this doesn't seem to work properly, as distutils ignores these variables! Checking in the source, I see that this is deliberate (see get_msvc_paths and set_path_env_var in msvccompiler.py). I don't believe that this decision is correct - it leaves the user no way, short of modifying the registry manually, to add include/library directories to the standard paths. So manually hacking setup.py becomes a necessity. Can I suggest that msvccompiler.py *adds* the current entries to INCLUDE and LIB, rather than replacing those variables completely. Or at least, that another way of including include/library directories external to setup.py be available... Thanks, Paul.

2 1

Distutils enhancement - set variables from the command line
by Paul Moore July 16, 2001

July 16, 2001

It would sometimes be nice for the user to be able to customise the setup process without having to edit setup.py. The obvious case is to allow the user to override library directories, where setup.py can assume the "normal" case, but still allow for non-standard locations. One relatively simple approach would be to write setup.py to read a configuration file, and get such options from there. But an alternative option would be to allow setup.py to have access to (part of) the command line typed by the user. My suggestion would be to allow the "global_opts" or "cmdX_opts" to contain arguments of the form key=value. These can be exposed to setup.py as a dictionary, say distutils.args. What do people think? Paul.

2 1

大连-2001年国际绿色食品与人类健康博览会(Agro Annual Meeting China 2001)
by wwwjessie＠21cn.com July 16, 2001

July 16, 2001

2001年中国国际农业科技年会国际绿色食品与人类健康博览会及学术讨论会展期：　2001年9月4日-7日地点：　大连星海会展中心主办：　中华人民共和国农业部　中国科学技术协会　大连市人民政府承办：　中国绿色食品发展中心　中国农学会　中国绿色食品协会　大连市农业局　大连星海会展中心网络服务提供商：壹食品中国网 http://www.ifood1.com <http://www.ifood1.com/index.asp?fr=distutils-sig@python.org> → 通过壹食品中国网报名参展：九折优惠(比如现有每个 3M X 3M 的标准展位原价RMB4500，通过我们只需付RMB4050)，报名截止日期2001年7月20日 <http://green2001.ifood1.com/from1.asp> → 欢迎免费注册 <http://www.ifood1.com/signup/sevagreem.asp> 成为公司会员。 7月20日前注册，您将在7月25日前通过电子邮件方式免费获得30条采购信息。如果您不想收到我们的邮件，请联系我们 <mailto:unsubscribe@ifood1.com> ，我们以后将不再发邮件给您。查询： sales(a)ifood1.com <mailto:sales@ifood1.com> 　　电话：0755-3786309　销售部沈小姐杜先生回执（请传真：0755-3239047或发电子邮件： sales(a)ifood1.com <mailto:sales@ifood1.com> ） □ 本公司有意通过壹食品中国网参展　　 □ 本公司拟进一步了解该博览会，请与我们联系公司名称：______________________________________ 联系人：_____________________________________ 电话：_____________________________________ 传真：_____________________________________ E-mail：_____________________________________

1 0

Building an intermediate library
by Paul Moore July 15, 2001

July 15, 2001

I have an extension which I am trying to build using distutils. There are a series of .c files in a subdirectory, and a couple of driver C files. The subdirectory files should, ideally, be compiled and then built into a temporary .lib file, which is then linked with the 2 individual driver files to create two .pyd extensions. I can do the build by listing all of the subdirectory .c files in the build instructions for the 2 .pyds, but if I do this, the compiler (MSVC) doesn't seem to eliminate unused object files. So either I need to list exactly the right .c files for each extension, or I build the intermediate .lib. The problem is, I can't work out if it is possible to persuade distutils to build an intermediate library like this. Is it possible? If so, can anyone point me to any suitable examples of what to do? (Or alternatively, point me at the appropriate parts of the source, and appropriate example setup.py files, to allow me to work out how to do this for myself.) Thanks, Paul.

2 3

Possible solution for PEP250 and bdist_wininst
by Thomas Heller July 15, 2001

July 15, 2001

I have a possible solution for this problem. (I'll use the name INSTALLPATH for installation directory stored in the registry under the key HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<version>\InstallPath). The bdist_wininst installer at _install_ time sets the PYTHONHOME environment variable to INSTALLPATH, then loads the python dll and retrieves the 'extinstallpath' attribute from the sys module: wwsprintf(buffer, "PYTHONHOME=%s", INSTALLPATH); _putenv(buffer); Py_SetProgramName(modulename); Py_Initialize(); pextinstallpath = PySys_GetObject("extinstallpath"); Py_Finalize(); If this is successful, the (string contents of) pextinstallpath is appended to INSTALLPATH, and that will be the directory where the package will be installed. If unsuccessful, INSTALLPATH will be used as before. I'm unsure about the change to site.py, but this should work: diff -c -r1.26 site.py *** site.py 2001/03/23 17:53:49 1.26 --- site.py 2001/07/13 15:32:27 *************** *** 140,153 **** "python" + sys.version[:3], "site-packages"), makepath(prefix, "lib", "site-python")] - elif os.sep == ':': - sitedirs = [makepath(prefix, "lib", "site-packages")] else: ! sitedirs = [prefix] for sitedir in sitedirs: if os.path.isdir(sitedir): addsitedir(sitedir) # Define new built-ins 'quit' and 'exit'. # These are simply strings that display a hint on how to exit. if os.sep == ':': --- 140,154 ---- "python" + sys.version[:3], "site-packages"), makepath(prefix, "lib", "site-python")] else: ! sitedirs = [prefix, os.path.join(prefix, "lib", "site-packages")] for sitedir in sitedirs: if os.path.isdir(sitedir): addsitedir(sitedir) + if os.sep == '\\': + sys.extinstallpath = os.path.join(sys.prefix, "lib", "site-packages") + # Define new built-ins 'quit' and 'exit'. # These are simply strings that display a hint on how to exit. if os.sep == ':': If anyone cares, I can post the diffs for the bdist_wininst sources. Thomas

2 4

Re: [Mingw-users] How to get an import lib from a DLL
by Paul Moore July 15, 2001

July 15, 2001

On Sun, 15 Jul 2001 00:06:53 +0200, you wrote: >In the distutils section of the python documentation > >http://www.python.org/doc/current/inst/tweak-flags.html#SECTION0003120000000 >00000000 > >you'll find a link to this page [...] Got it! Thanks - I was looking in the "Distributing Python Modules" document, when I should have been looking in "Installing Python Modules". Thanks for the pointer, I have it sorted now. Paul.

1 0

Building extensions with mingw
by Paul Moore July 14, 2001

July 14, 2001

I have ActiveState Python 2.1 (which I assume was built with MSVC), and I am trying to build an extension using the mingw compiler. When I try to build (python setup.py build -c mingw32) I get a whole load of undefined references to _imp__Py... symbols. Presumably, I'm missing some sort of supporting import library to tell mingw about Python - the question is, how do I get one? Or am I being naive in assuming that I can use mingw to build extensions for a Python built with MSVC? Any suggestions would be gratefully accepted... Paul.

3 2