I'm trying to define some meaningful terms for talking about eggs and what
they contain, as I prepare for some refactoring to make the egg runtime's
class and attribute names be more consistent. Currently, the terminology
driving those names has been kind of vague and handwavy; I'd like to get a
little more precise.
So, the following is an attempt at working out a coherent vocabulary,
conceptual framework, and architectural overview of "Pluggable
Distributions" (i.e., eggs). My plan is to refactor pkg_resources,
setuptools.packge_index, and possibly some other parts of setuptools to
match the more formal concept hierarchy laid out here.
I'd appreciate any feedback, but it needs to be fairly soon as there should
ideally be only one "great renaming", marking the transition of setuptools
from 0.5 to 0.6, with a stable API for extenders going forward.
Pluggable Distributions of Python Software
==========================================
A "Distribution" is a collection of files that represent a "Release" of a
"Project" as of a particular point in time, denoted by a
"Version". Releases may have zero or more "Requirements", which indicate
what releases of another project the release requires in order to
function. A Requirement names the other project, expresses some criteria
as to what releases of that project are acceptable, and lists any "Extras"
that the requiring release may need from that project. (An Extra is an
optional feature of a Release, that can only be used if its additional
Requirements are satisfied.)
Notice, by the way, that this definition of Distribution is broad enough to
include directories containing Python packages or modules, not just "built
distributions" created by the distutils. For example, the directory
containing the Python standard library is a "distribution" by this
definition, and so are the directories you edit your project's code in! In
other words, every copy of a project's code is a "distribution", even if
you don't take any special steps to make it one.
Not all distributions are "importable", however. An "importable"
distribution is one whose file or directory name can be referenced on
sys.path, to allow importing modules or packages from that
distribution. So, simple directories of code (and zipfiles with the
correct internal layout) are "importable distributions", but most of the
distributions built by the distutils (source archives and binary
installers) are not.
A "Project" is a library, framework, script, application, or collection of
data or other files relevant to Python. "Projects" must have unique names,
in order to tell them apart. Currently, PyPI is useful as a way of
registering project names for uniqueness, because the 'name' argument to
distutils 'setup()' command is used to identify the project on PyPI, as
well as to generate Distributions' file names.
A "Pluggable Distribution" or "Pluggable", is an importable distribution
that satisfies these important additional properties:
1. Its project name and distribution format can be unambiguously
determined from file or directory names, without actually examining any
file contents. (Most distutils distribution formats cannot guarantee this,
because they do not place any restrictions on project name strings, and
thus allow ambiguity as to what part of their filenames is the project
name, and what part is the version.)
2. A pluggable distribution contains metadata identifying its
release's version, requirements, extras, and any additional requirements
needed to implement those extras. It may also contain other metadata
specific to an application or framework, to support integrating the
pluggable's project with that application or framework.
Distributions that satisfy these two properties are thus "pluggable",
because they can be automatically discovered and "activated" (by adding
them to sys.path), then used for importing Python modules or accessing
other resource files and directories that are part of the distributed project.
The Working Set
---------------
The collection of distributions that are currently activated is called a
Working Set. Note that a Working Set can contain any importable
distribution, not just pluggable ones. For example, the Python standard
library is an importable distribution that will usually be part of the
Working Set, even though it is not pluggable. Similarly, when you are
doing development work on a project, the files you are editing are also a
Distribution. (And, with a little attention to the directory names used,
and including some additional metadata, such a "development distribution"
can be made pluggable as well.)
When Python runs a program, that program must have all its requirements met
by importable distributions in the working set. Initially, a Python
program's Working Set consists only of the importable distributions
(whether pluggable or not) listed in sys.path, such as the directory
containing the program's __main__ script, and the directories containing
the standard library and site-packages. If these are the only
distributions that the program requires, then of course that program can run.
However, if some of the requirements are not satisfied by the working set,
this can lead to errors that may be hard to diagnose. So, if a Python
program were made part of a Project, and the project explicitly defines its
Requirements, which are then expressed as part of a Pluggable Distribution,
then a runtime facility could automatically attempt to locate suitable
pluggables and add them to the working set, or at least give a more
specific error message if a requirement can't be satisfied.
The Environment
---------------
A set of directories that may be searched for pluggable distributions is
called an Environment. By default, the Environment consists of all
existing directories on sys.path, plus any distribution sources registered
with the runtime.
Given an Environment, and a Requirement to be satisfied, our proposed
runtime facility would search the environment for pluggable distributions
that satisfy the requirement (and the requirements of those distributions,
recursively), such that it returns a list of distributions to be added to
the working set, or raises a DependencyNotFound error.
Note that a Working Set should not contain multiple distributions for the
same project, so the runtime system must not propose to add a pluggable
distribution to a Working Set if that set already contains a pluggable for
the same project. If a project's requirements can't be met without adding
a conflicting pluggable to the working set, a VersionConflict error is
raised. (Unlike a working set, an Environment may contain more than one
pluggable for a given project, because these are simply distributions that
are *available* to be activated.)
Python Eggs
-----------
"Python Eggs" are distributions in specific formats that implement the
concept of a "Pluggable Distribution". An egg may be a zipfile or
directory whose name ends with '.egg', that contains Python modules or
packages, plus an 'EGG-INFO' subdirectory containing metadata. An egg may
also be a directory containing one or more 'ProjectName.egg-info'
subdirectories with metadata.
The latter form is primarily intended to add discoverability to
distributions that -- for whatever reason -- cannot be restructured to the
primary egg format. For example, by placing appropriate .egg-info
directories in site-packages, one could document what distributions are
already installed in that directory. While this would not make those
releases capable of being individually activated, it does allow the runtime
system to be aware that any requirements for those projects are already
met, and to know that it should not attempt to add any other releases of
those projects to the working set.
The last form of egg is a '.egg-link' file. These exist to support
symbolic linking on platforms that do not natively support symbolic links
(e.g. Windows). These consist simply of a single line indicating the
location of a directory that contains either an EGG-INFO or
ProjectName.egg-info subdirectory. This format will be used by project
management utilities to add an in-development distribution to the
development Environment.
Initialization, Development, and Deployment
-------------------------------------------
Pluggable distributions can be manually made part of the working set by
modifying sys.path. This can be done via PYTHONPATH, .pth files, or direct
code manipulation. However, it is generally more useful to put
distributions in the working set by automatically locating them in an
appropriate Environment.
The default Environment is the directories already on sys.path, so simply
placing pluggable distributions in those directories suffices to make them
available for adding to the working set.
But *something* must add them to the working set, even if it is just to
designate the project the current program is part of, so that its
dependencies can be automatically resolved and added to the working
set. This means that either a program's start scripts must invoke the
runtime facility and make this initial request, or there must be some
automatic means by which this is accomplished.
The EasyInstall program accomplishes this by creating wrapper scripts when
a distribution is installed. The wrapper scripts know what project the
"real" script is part of, and so can ensure that the right working set is
active when the scripts run. The scripts' author does not need to invoke
the runtime facility directly, nor do they even need to be aware that it
exists.
For development, however, one does not generally want to have to "install"
scripts that one is actively editing. So, future versions of the runtime
facility will have an option to automatically create wrapper scripts that
invoke the in-development versions of the scripts, rather than versions
installed in eggs. This will allow developers to continue to write scripts
without embedding any project or version information in them.
Essentially, for development purposes, there will be a tool to "install" an
in-development distribution to a given Environment, using a symlink or
.egg-link file to include the distribution, and generating wrapper scripts
to invoke any "main program" scripts in the project. Thus, a user's
development Environment can include one or more projects whose source code
he or she is editing, as well as any number of built distributions. He or
she can then also build source or binary distributions of their project for
deployment, whenever it is necessary or convenient to do so.
