[Python-checkins] r72680 - peps/trunk/pep-0376.txt
tarek.ziade
python-checkins at python.org
Sat May 16 18:02:06 CEST 2009
Author: tarek.ziade
Date: Sat May 16 18:02:06 2009
New Revision: 72680
Log:
changes after feedback from python-dev
Modified:
peps/trunk/pep-0376.txt
Modified: peps/trunk/pep-0376.txt
==============================================================================
--- peps/trunk/pep-0376.txt (original)
+++ peps/trunk/pep-0376.txt Sat May 16 18:02:06 2009
@@ -22,28 +22,41 @@
Definitions
===========
-A **project** is a Python application composed of one or many Python packages.
-It is distributed using a `setup.py` script with Distutils and/or Setuptools.
-
-Once installed, one or several **packages** are added in Python's site-packages.
+A **project** is a Python application composed of one or several files, which can
+be Python modules, extensions or data. It is distributed using a `setup.py` script
+with Distutils and/or Setuptools. The `setup.py` script indicates where each
+elements should be installed.
+
+Once installed, the elements are located in various places in the system, like:
+
+- in Python's site-packages (Python modules, Python modules organized into packages,
+ Extensions, etc.)
+- in Python's `include` directory.
+- in Python's `bin` or `Script` directory.
+- etc.
Rationale
=========
There are two problems right now in the way projects are installed in Python:
-- There are too many ways to install a project in Python.
+- There are too many ways to do it.
- There is no API to get the metadata of installed projects.
How projects are installed
--------------------------
-Right now, when a project is installed in Python, every package its contains
-is installed in the `site-packages` directory with the Distutils `install`
-command.
+Right now, when a project is installed in Python, every elements its contains
+is installed in various directories.
+
+The pure Python code for instance is installed in the `purelib` directory,
+which is located in the Python installation in `lib\python2.6\site-packages`
+for example under unix-like systems or Mac OS X, and in `Lib/site-packages`
+under Windows. This is done with the Distutils `install` command, which calls
+various subcommands.
The `install_egg_info` subcommand is called during this process, in order to
-create an `.egg-info` file in the `site-packages` directory.
+create an `.egg-info` file in the `purelib` directory.
For example, if the `zlib` project (which contains one package) is installed,
two elements will be installed in `site-packages`::
@@ -51,8 +64,8 @@
- zlib
- zlib-2.5.2-py2.4.egg-info
-Where `zlib` is the package, and `zlib-2.5.2-py2.4.egg-info` is
-a file containing the package metadata as described in PEP 314.
+Where `zlib` is a Python package, and `zlib-2.5.2-py2.4.egg-info` is
+a file containing the project metadata as described in PEP 314 [#pep314]_.
This file corresponds to the file called `PKG-INFO`, built by
the `sdist` command.
@@ -63,11 +76,13 @@
- `easy_install` creates an `EGG-INFO` directory inside an `.egg` directory,
and adds a `PKG-INFO` file inside this directory. The `.egg` directory
- contains in that case the packages of the project.
-
-- `pip` creates an `.egg-info` directory inside the site-packages directory
- and adds a `PKG-INFO` file inside it. Packages are installed in
- site-packages directory in a regular way.
+ contains in that case all the elements of the project that are supposed to
+ be installed in `site-packages`, and is placed in the `site-packages`
+ directory.
+
+- `pip` creates an `.egg-info` directory inside the `site-packages` directory
+ and adds a `PKG-INFO` file inside it. Elements of the project are then
+ installed in various places like Distutils does.
They both add other files in the `EGG-INFO` or `.egg-info` directory, and
create or modify `.pth` files.
@@ -76,16 +91,19 @@
---------------------
Distutils doesn't provide any `uninstall` command. If you want to uninstall
-a project, you have to be a power user and remove the various package
-directories from the right `site-packages` directory, then look over the right
-`pth` files. And this method differs, depending on the tools you are using.
-
-The worst issue is that you depend on the way the packager created his package.
-When you call `python setup.py install`, it will not be installed the same way
-depending on the tool used by the packager (mainly Distutils or Setuptools).
+a project, you have to be a power user and remove the various elements that
+were installed. Then look over the `.pth` file to clean them if necessary.
+
+And the process differs, depending on the tools you have used to install the
+project, and if the project's `setup.py` uses Distutils or Setuptools.
-But there's common behavior: files are copied in your installation.
-And there's a way to keep track of theses file, so to remove them.
+Under some circumstances, you might not be able to know for sure that you
+have removed everything, or that you didn't break another project by
+removing a file that was shared among the two projects.
+
+But there's common behavior: when you install a project, files are copied
+in your system. And there's a way to keep track of theses files, so to remove
+them.
What this PEP proposes
----------------------
@@ -101,51 +119,58 @@
=============================
The first change would be to make `.egg-info` a directory and let it
-hold the `PKG-INFO` file built by the `write_pkg_file` method.
+hold the `PKG-INFO` file built by the `write_pkg_file` method of
+the `Distribution` class in Distutils.
-This change will not impact Python itself, because this file is not
-used anywhere yet in the standard library. So there's no need of
-deprecation.
+This change will not impact Python itself, because `egg-info` files are not
+used anywhere yet in the standard library besides Distutils.
Although it will impact the `setuptools` and `pip` projects, but given
-the fact that they already work with a directory that contains a
-`PKG-INFO` file, the change will be small.
+the fact that they already work with a directory that contains a `PKG-INFO`
+file, the change will have no deep consequences.
-For example, if the `zlib` package is installed, two elements
-will be installed in `site-packages`::
+For example, if the `zlib` package is installed, the elements that
+will be installed in `site-packages` will become::
- zlib
- zlib-2.5.2.egg-info/
PKG-INFO
-The Python version will also be removed from the .egg-info directory
-name. To be able to implement this change, the impacted code in Distutils
-is the `install_egg_info` command, and the various third-party projects.
+The Python version will also be removed from the `.egg-info` directory
+name.
Adding a RECORD in the .egg-info directory
==========================================
A `RECORD` file will be added inside the `.egg-info` directory at installation
-time.
+time. The `RECORD` file will hold the list of installed files. These correspond
+to the files listed by the `record` option of the `install` command, and will
+always be generated. This will allow uninstallation, as explained later in this
+PEP. This RECORD file is inspired from PEP 262 FILES [#pep262]_.
+
+The RECORD format
+-----------------
+
+The `RECORD` file is composed of records, one line per installed file.
+Each record is composed of three elements separated by a `;` character:
+
+- the file's full **path**
-- the `RECORD` file will hold the list of installed files. These
- correspond to the files listed by the `record` option of the `install`
- command, and will always be generated. This will allow uninstallation, as
- explained later in this PEP.
-
-The `install` command will record by default installed files in the
-RECORD file, using these rules:
-
-- if the installed file is located in a directory in `site-packages`,
- it will be a '/'-separated relative path, no matter what is the target
- system. This makes this information cross-compatible and allows simple
- installation to be relocatable.
+ - if the installed file is located in a directory in `site-packages`,
+ it will be a '/'-separated relative path, no matter what is the target
+ system. This makes this information cross-compatible and allows simple
+ installation to be relocatable.
-- if the installed file is located elsewhere in the system, a
- '/'-separated absolute path is used.
+ - if the installed file is located elsewhere in the system, a
+ '/'-separated absolute path is used.
-This will require changing the way the `install` command writes the record
-file, so the old `record` behavior will be deprecated.
+- the **MD5** hash of the file, encoded in hex. Notice that `pyc` and `pyo`
+ generated files will not have a hash.
+
+- the file's size in bytes
+
+Example
+-------
Back to our `zlib` example, we will have::
@@ -154,6 +179,21 @@
PKG-INFO
RECORD
+And the RECORD file will contain::
+
+ zlib/include/zconf.h;b690274f621402dda63bf11ba5373bf2;9544
+ zlib/include/zlib.h;9c4b84aff68aa55f2e9bf70481b94333;66188
+ zlib/lib/libz.a;e6d43fb94292411909404b07d0692d46;91128
+ zlib/share/man/man3/zlib.3;785dc03452f0508ff0678fba2457e0ba;4486
+ zlib-2.5.2.egg-info/PKG-INFO;6fe57de576d749536082d8e205b77748;195
+ zlib-2.5.2.egg-info/RECORD
+
+Notice that:
+
+- the `RECORD` file can't contain a hash of itself and is just mentioned here
+- `zlib` and `zlib-2.5.2.egg-info` are located in `site-packages` so the file
+ paths are relative to it.
+
New functions in pkgutil
========================
@@ -178,16 +218,31 @@
Uses `get_egg_info` to get the `PKG-INFO` file, and returns a
`DistributionMetadata` instance that contains the metadata.
- This will require a small change in `DistributionMetadata` (see #4908).
-- get_egg_info_file(project_name, filename) -> file object or None
+- get_files(project_name) -> iterator of (path, hash, size, other_projects)
+
+ Uses `get_egg_info` to get the `RECORD` file, and returns an iterator.
+
+ Each returned element is a tuple `(path, hash, size, other_projects)` where
+ ``path`, ``hash``, ``size`` are the values found in the RECORD file.
- Uses `get_egg_info` and gets any file inside the directory,
- pointed by filename.
+ `other_projects` is a tuple containing the name of the projects that are
+ also referring to this file in their own RECORD file (same path).
+
+ If `other_projects` is empty, it means that the file is only referred by the
+ current project. In other words, it can be removed if the project is removed.
+
+- get_egg_info_file(project_name, path) -> file object or None
+
+ Uses `get_egg_info` and gets any element inside the directory,
+ pointed by its relative path. `get_egg_info_file` will perform
+ an `os.path.join` on `get_egg_info(project_name)` and `path` to build the
+ whole path.
Let's use it with our `zlib` example::
- >>> from pkgutil import get_egg_info, get_metadata, get_egg_info_file
+ >>> from pkgutil import (get_egg_info, get_metadata, get_egg_info_file,
+ ... get_files)
>>> get_egg_info('zlib')
'/opt/local/lib/python2.6/site-packages/zlib-2.5.2.egg-info'
>>> metadata = get_metadata('zlib')
@@ -197,6 +252,16 @@
some
...
files
+ >>> for path, hash, size, other_projects in get_files('zlib'):
+ ... print '%s %s %d %s' % (path, hash, size, ','.join(other_projects))
+ ...
+ zlib/include/zconf.h b690274f621402dda63bf11ba5373bf2 9544
+ zlib/include/zlib.h 9c4b84aff68aa55f2e9bf70481b94333 66188
+ zlib/lib/libz.a e6d43fb94292411909404b07d0692d46 91128
+ zlib/share/man/man3/zlib.3 785dc03452f0508ff0678fba2457e0ba 4486
+ zlib-2.5.2.egg-info/PKG-INFO 6fe57de576d749536082d8e205b77748 195
+ zlib-2.5.2.egg-info/RECORD None None
+
Adding an Uninstall function
============================
@@ -204,14 +269,13 @@
Distutils provides a very basic way to install a project, which is running
the `install` command over the `setup.py` script of the distribution.
-Distutils will provide a very basic `uninstall` command that will remove
-all files listed in the `RECORD` file of a project, as long as they are not
-mentioned in another `RECORD` file and as long as the package is installed
-using the standard described earlier.
-
-This command will be added in ``distutils.util`` and will take the name
-of the project to uninstall as its argument. A call to uninstall will return a
-list of uninstalled files::
+Distutils will provide a very basic ``uninstall`` function, that will be added
+in ``distutils.util`` and will take the name of the project to uninstall as
+its argument. ``uninstall`` will use ``pkgutil.get_files`` and remove all
+unique files, as long as their hash didn't change. Then it will remove
+directories where it removed the last elements.
+
+``uninstall`` will return a list of uninstalled files::
>>> from distutils.util import uninstall
>>> uninstall('zlib')
@@ -220,23 +284,28 @@
If the project is not found, a ``DistutilsUninstallError`` will be raised.
-To make it a reference API for third-party projects that wish to provide
-an `uninstall feature`. The ``uninstall`` function can also be invoked with a
-second callable argument, that will be invoked for each file to be removed.
-If this callable returns `True`, the file will be removed.
+To make it a reference API for third-party projects that wish to control
+how `uninstall` works, a second callable argument can be used. It will be
+called for each file that is removed. If the callable returns `True`, the
+file will be removed. If it returns False, it will be left alone.
Examples::
>>> def _remove_and_log(path):
... logging.info('Removing %s' % path)
... return True
+ ...
>>> uninstall('zlib', _remove_and_log)
>>> def _dry_run(path):
... logging.info('Removing %s (dry run)' % path)
... return False
+ ...
>>> uninstall('zlib', _dry_run)
+Of course, a third-party tool can use ``pkgutil.get_files`` for a maximum
+control, to implement their own uninstall feature.
+
Backward compatibility and roadmap
==================================
@@ -244,10 +313,20 @@
version of Distutils, and will also work with existing third-party tools.
Although, a backport of the new Distutils for 2.5, 2.6, 3.0 and 3.1 will be
-provided so people can benefit from the new features.
+provided so people can benefit from these new features.
The plan is to integrate them for Python 2.7 and Python 3.2
+References
+==========
+
+.. [#pep262]
+ http://www.python.org/dev/peps/pep-0262
+
+.. [#pep314]
+ http://www.python.org/dev/peps/pep-0314
+
+
Aknowledgments
==============
More information about the Python-checkins
mailing list