Hi all,
We realized that actually as far as we could tell, it wouldn't be that
hard at this point to clean up how sdists work so that it would be
possible to migrate away from distutils. So we wrote up a little draft
proposal.
The main question is, does this approach seem sound?
-n
---
PEP: ??
Title: Standard interface for interacting with source trees
and source distributions
Version: $Revision$
Last-Modified: $Date$
Author: Nathaniel J. Smith <njs(a)pobox.com>
Thomas Kluyver <takowl(a)gmail.com>
Status: Draft
Type: Standards-Track
Content-Type: text/x-rst
Created: 30-Sep-2015
Post-History:
Discussions-To: <distutils-sig(a)python.org>
Abstract
========
Distutils delenda est.
Extended abstract
=================
While ``distutils`` / ``setuptools`` have taken us a long way, they
suffer from three serious problems: (a) they're missing important
features like autoconfiguration and usable build-time dependency
declaration, (b) extending them is quirky, complicated, and fragile,
(c) you are forced to use them anyway, because they provide the
standard interface for installing python packages expected by both
users and installation tools like ``pip``.
Previous efforts (e.g. distutils2 or setuptools itself) have attempted
to solve problems (a) and/or (b). We propose to solve (c).
The goal of this PEP is get distutils-sig out of the business of being
a gatekeeper for Python build systems. If you want to use distutils,
great; if you want to use something else, then the more the merrier.
The difficulty of interfacing with distutils means that there aren't
many such systems right now, but to give a sense of what we're
thinking about see `flit <https://github.com/takluyver/flit>`_ or
`bento
<https://cournape.github.io/Bento/>`_. Fortunately, wheels have now
solved many of the hard problems here -- e.g. it's no longer necessary
that a build system also know about every possible installation
configuration -- so pretty much all we really need from a build system
is that it have some way to spit out standard-compliant wheels.
We therefore propose a new, relatively minimal interface for
installation tools like ``pip`` to interact with package source trees
and source distributions.
Synopsis and rationale
======================
To limit the scope of our design, we adopt several principles.
First, we distinguish between a *source tree* (e.g., a VCS checkout)
and a *source distribution* (e.g., an official snapshot release like
``lxml-3.4.4.zip``).
There isn't a whole lot that *source trees* can be assumed to have in
common. About all you know is that they can -- via some more or less
Rube-Goldbergian process -- produce one or more binary distributions.
In particular, you *cannot* tell via simple static inspection:
- What version number will be attached to the resulting packages (e.g.
it might be determined programmatically by consulting VCS metadata --
I have here a build of numpy version "1.11.0.dev0+4a9ad17")
- What build- or run-time dependencies are required (e.g. these may
depend on arbitrarily complex configuration settings that are
determined via a mix of manual settings and auto-probing)
- Or even how many distinct binary distributions will be produced
(e.g. a source distribution may always produce wheel A, but only
produce wheel B when built on Unix-like systems).
Therefore, when dealing with source trees, our goal is just to provide
a standard UX for the core operations that are commonly performed on
other people's packages; anything fancier and more developer-centric
we leave at the discretion of individual package developers. So our
source trees just provide some simple hooks to let a tool like
``pip``:
- query for build dependencies
- run a build, producing wheels as output
- set up the current source tree so that it can be placed on
``sys.path`` in "develop mode"
and that's it. We teach users that the standard way to install a
package from a VCS checkout is now ``pip install .`` instead of
``python setup.py install``. (This is already a good idea anyway --
e.g., pip can do reliable uninstall / upgrades.)
Next, we note that pretty much all the operations that you might want
to perform on a *source distribution* are also operations that you
might want to perform on a source tree, and via the same UX. The only
thing you do with source distributions that you don't do with source
trees is, well, distribute them. There's all kind of metadata you
could imagine including in a source distribution, but each piece of
metadata puts an increased burden on source distribution generation
tools, and most operations will still have to work without this
metadata. So we only include extra metadata in source distributions if
it helps solve specific problems that are unique to distribution. If
you want wheel-style metadata, get a wheel and look at it -- they're
great and getting better.
Therefore, our source distributions are basically just source trees +
a mechanism for signing.
Finally: we explicitly do *not* have any concept of "depending on a
source distribution". As in other systems like Debian, dependencies
are always phrased in terms of binary distributions (wheels), and when
a user runs something like ``pip install <package>``, then the
long-run plan is that <package> and all its transitive dependencies
should be available as wheels in a package index. But this is not yet
realistic, so as a transitional / backwards-compatibility measure, we
provide a simple mechanism for ``pip install <package>`` to handle
cases where <package> is provided only as a source distribution.
Source trees
============
We retroactively declare the legacy source tree format involving
``setup.py`` to be "version 0". We don't try to specify it further;
its de facto specification is encoded in the source code of
``distutils``, ``setuptools``, ``pip``, and other tools.
A version 1-or-greater format source tree can be identified by the
presence of a file ``_pypackage/_pypackage.cfg``.
If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present,
then we have a version 1+ source tree, i.e., ``setup.py`` is ignored.
This is necessary because we anticipate that version 1+ source trees
may want to contain a ``setup.py`` file for backwards compatibility,
e.g.::
#!/usr/bin/env python
import sys
print("Don't call setup.py directly!")
print("Use 'pip install .' instead!")
print("(You might have to upgrade pip first.)")
sys.exit(1)
In the current version of the specification, the one file
``_pypackage/_pypackage.cfg`` is where pretty much all the action is
(though see below). The motivation for putting it into a subdirectory
is that:
- the way of all standards is that cruft accumulates over time, so
this way we pre-emptively have a place to put it,
- real-world projects often accumulate build system cruft as well, so
we might as well provide one obvious place to put it too.
Of course this then creates the possibility of collisions between
standard files and user files, and trying to teach arbitrary users not
to scatter files around willy-nilly never works, so we adopt the
convention that names starting with an underscore are reserved for
official use, and non-underscored names are available for
idiosyncratic use by individual projects.
The alternative would be to simply place the main configuration file
at the top-level, create the subdirectory only when specifically
needed (most trees won't need it), and let users worry about finding
their own place for their cruft. Not sure which is the best approach.
Plus we can have a nice bikeshed about the names in general (FIXME).
_pypackage.cfg
--------------
The ``_pypackage.cfg`` file contains various settings. Another good
bike-shed topic is which file format to use for storing these (FIXME),
but for purposes of this draft I'll write examples using `toml
<https://github.com/toml-lang/toml>`_, because you'll instantly be
able to understand the semantics, it has similar expressivity to JSON
while being more human-friendly (e.g., it supports comments and
multi-line strings), it's better-specified than ConfigParser, and it's
much simpler than YAML. Rust's package manager uses toml for similar
purposes.
Here's an example ``_pypackage/_pypackage.cfg``::
# Version of the "pypackage format" that this file uses.
# Optional. If not present then 1 is assumed.
# All version changes indicate incompatible changes; backwards
# compatible changes are indicated by just having extra stuff in
# the file.
version = 1
[build]
# An inline requirements file. Optional.
# (FIXME: I guess this means we need a spec for requirements files?)
requirements = """
mybuildtool >= 2.1
special_windows_tool ; sys_platform == "win32"
"""
# The path to an out-of-line requirements file. Optional.
requirements-file = "build-requirements.txt"
# A hook that will be called to query build requirements. Optional.
requirements-dynamic = "mybuildtool:get_requirements"
# A hook that will be called to build wheels. Required.
build-wheels = "mybuildtool:do_build"
# A hook that will be called to do an in-place build (see below).
# Optional.
build-in-place = "mybuildtool:do_inplace_build"
# The "x" namespace is reserved for third-party extensions.
# To use x.foo you should own the name "foo" on pypi.
[x.mybuildtool]
spam = ["spam", "spam", "spam"]
All paths are relative to the ``_pypackage/`` directory (so e.g. the
build.requirements-file value above refers to a file named
``_pypackage/build-requirements.txt``).
A *hook* is a Python object that is looked up using the same rules as
traditional setuptools entry_points: a dotted module name, followed by
a colon, followed by a dotted name that is looked up within that
module. *Running a hook* means: first, find or create a python
interpreter which is executing in the current venv, whose working
directory is set to the ``_pypackage/`` directory, and which has the
``_pypackage/`` directory on ``sys.path``. Then, inside this
interpreter, look up the hook object, and call it, with arguments as
specified below.
A build command like ``pip wheel <source tree>`` performs the following steps:
1) Validate the ``_pypackage.cfg`` version number.
2) Create an empty virtualenv / venv, that matches the environment
that the installer is targeting (e.g. if you want wheels for CPython
3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows
venv).
3) If the build.requirements key is present, then in this venv run the
equivalent of ``pip install -r <a file containing its value>``, using
whatever index settings are currently in effect.
4) If the build.requirements-file key is present, then in this venv
run the equivalent of ``pip install -r <the named file>``, using
whatever index settings are currently in effect.
5) If the build.requirements-dynamic key is present, then in this venv
run the hook with no arguments, capture its stdout, and pipe it into
``pip install -r -``, using whatever index settings are currently in
effect. If the hook raises an exception, then abort the build with an
error.
Note: because these steps are performed in sequence, the
build.requirements-dynamic hook is allowed to use packages that are
listed in build.requirements or build.requirements-file.
6) In this venv, run the build.build-wheels hook. This should be a
Python function which takes one argument.
This argument is an arbitrary dictionary intended to contain
user-specified configuration, specified via some install-tool-specific
mechanism. The intention is that tools like ``pip`` should provide
some way for users to specify key/value settings that will be passed
in here, analogous to the legacy ``--install-option`` and
``--global-option`` arguments.
To make it easier for packages to transition from version 0 to
version 1 sdists, we suggest that ``pip`` and other tools that have
such existing option-setting interfaces SHOULD map them to entries in
this dictionary when -- e.g.::
pip --global-option=a --install-option=b --install-option=c
could produce a dict like::
{"--global-option": ["a"], "--install-option": ["b", "c"]}
The hook's return value is a list of pathnames relative to the
scratch directory. Each entry names a wheel file created by this
build.
Errors are signaled by raising an exception.
When performing an in-place build (e.g. for ``pip install -e .``),
then the same steps are followed, except that instead of the
build.build-wheels hook, we call the build.build-in-place hook, and
instead of returning a list of wheel files, it returns the name of a
directory that should be placed onto ``sys.path`` (usually this will
be the source tree itself, but may not be, e.g. if a build system
wants to enforce a rule where the source is always kept pristine then
it could symlink the .py files into a build directory, place the
extension modules and dist-info there, and return that). This
directory must contain importable versions of the code in the source
tree, along with appropriate .dist-info directories.
(FIXME: in-place builds are useful but intrinsically kinda broken --
e.g. extensions / source / metadata can all easily get out of sync --
so while I think this paragraph provides a reasonable hack that
preserves current functionality, maybe we should defer specifying them
to until after we've thought through the issues more?)
When working with source trees, build tools like ``pip`` are
encouraged to cache and re-use virtualenvs for performance.
Other contents of _pypackage/
-----------------------------
_RECORD, _RECORD.jws, _RECORD.p7s: see below.
_x/<pypi name>/: reserved for use by tools (e.g.
_x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/)
Source distributions
====================
A *source distribution* is a file in a well-known archive format such
as zip or tar.gz, which contains a single directory, and this
directory is a source tree (in the sense defined in the previous
section).
The ``_pypackage/`` directory in a source distribution SHOULD also
contain a _RECORD file, as defined in PEP 427, and MAY also contain
_RECORD.jws and/or _RECORD.p7s signature files.
For official releases, source distributions SHOULD be named as
``<package>-<version>.<ext>``, and the directory they contain SHOULD
be named ``<package>-<version>``, and building this source tree SHOULD
produce a wheel named ``<package>-<version>-<compatibility tag>.whl``
(though it may produce other wheels as well).
(FIXME: maybe we should add that if you want your sdist on PyPI then
you MUST include a proper _RECORD file and use the proper naming
convention?)
Integration tools like ``pip`` SHOULD take advantage of this
convention by applying the following heuristic: when seeking a package
<package>, if no appropriate wheel can be found, but an sdist named
<package>-<version>.<ext> is found, then:
1) build the sdist
2) add the resulting wheels to the package search space
3) retry the original operation
This handles a variety of simple and complex cases -- for example, if
we need a package 'foo', and we find foo-1.0.zip which builds foo.whl
and bar.whl, and foo.whl depends on bar.whl, then everything will work
out. There remain other cases that are not handled, e.g. if we start
out searching for bar.whl we will never discover foo-1.0.zip. We take
the perspective that this is nonetheless sufficient for a transitional
heuristic, and anyone who runs into this problem should just upload
wheels already. If this turns out to be inadequate in practice, then
it will be addressed by future extensions.
Examples
========
**Example 1:** While we assume that installation tools will have to
continue supporting version 0 sdists for the indefinite future, it's a
useful check to make sure that our new format can continue to support
packages using distutils / setuptools as their build system. We assume
that a future version ``pip`` will take its existing knowledge of
distutils internals and expose them as the appropriate hooks, and then
existing distutils / setuptools packages can be ported forward by
using the following ``_pypackage/_pypackage.cfg``::
[build]
requirements = """
pip >= whatever
wheel
"""
# Applies monkeypatches, then does 'setup.py dist_info' and
# extracts the setup_requires
requirements-dynamic = "pip.pypackage_hooks:setup_requirements"
# Applies monkeypatches, then does 'setup.py wheel'
build-wheels = "pip.pypackage_hooks:build_wheels"
# Applies monkeypatches, then does:
# setup.py dist_info && setup.py build_ext -i
build-in-place = "pip.pypackage_hooks:build_in_place"
This is also useful for any other installation tools that may want to
support version 0 sdists without having to implement bug-for-bug
compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is
present, they can use this as a default.
**Example 2:** For packages using numpy.distutils. This is identical
to the distutils / setuptools example above, except that numpy is
moved into the list of static build requirements. Right now, most
projects using numpy.distutils don't bother trying to declare this
dependency, and instead simply error out if numpy is not already
installed. This is because currently the only way to declare a build
dependency is via the ``setup_requires`` argument to the ``setup``
function, and in this case the ``setup`` function is
``numpy.distutils.setup``, which... obviously doesn't work very well.
Drop this ``_pypackage.cfg`` into an existing project like this and it
will become robustly pip-installable with no further changes::
[build]
requirements = """
numpy
pip >= whatever
wheel
"""
requirements-dynamic = "pip.pypackage_hooks:setup_requirements"
build-wheels = "pip.pypackage_hooks:build_wheels"
build-in-place = "pip.pypackage_hooks:build_in_place"
**Example 3:** `flit <https://github.com/takluyver/flit>`_ is a tool
designed to make distributing simple packages simple, but it currently
has no support for sdists, and for convenience includes its own
installation code that's redundant with that in pip. These 4 lines of
boilerplate make any flit-using source tree pip-installable, and lets
flit get out of the package installation business::
[build]
requirements = "flit"
build-wheels = "flit.pypackage_hooks:build_wheels"
build-in-place = "flit.pypackage_hooks:build_in_place"
FAQ
===
**Why is it version 1 instead of version 2?** Because the legacy sdist
format is barely a format at all, and to `remind us to keep things
simple <https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effe…>`_.
**What about cross-compilation?** Standardizing an interface for
cross-compilation seems premature given how complicated the
configuration required can be, the lack of an existing de facto
standard, and the authors of this PEP's inexperience with
cross-compilation. This would be a great target for future extensions,
though. In the mean time, there's no requirement that
``_pypackage/_pypackage.cfg`` contain the *only* entry points to a
project's build system -- packages that want to support
cross-compilation can still do so, they'll just need to include a
README explaining how to do it.
**PEP 426 says that the new sdist format will support automatically
creating policy-compliant .deb/.rpm packages. What happened to that?**
Step 1: enhance the wheel format as necessary so that a wheel can be
automatically converted into a policy-compliant .deb/.rpm package (see
PEP 491). Step 2: make it possible to automatically turn sdists into
wheels (this PEP). Step 3: we're done.
**What about automatically running tests?** Arguably this is another
thing that should be pushed off to wheel metadata instead of sdist
metadata: it's good practice to include tests inside your built
distribution so that end-users can test their install (and see above
re: our focus here being on stuff that end-users want to do, not
dedicated package developers), there are lots of packages that have to
be built before they can be tested anyway (e.g. because of binary
extensions), and in any case it's good practice to test against an
installed version in order to make sure your install code works
properly. But even if we do want this in sdist, then it's hardly
urgent (e.g. there is no ``pip test`` that people will miss), so we
defer that for a future extension to avoid blocking the core
functionality.
--
Nathaniel J. Smith -- http://vorpus.org
Hi all,
I'm finding it impossible to keep track of that other thread, and I
guess I'm probably not the only one, so I figured I'd try splitting a
few of the more specific discussions out :-).
One thing that seems to be a key issue, but where I remain very
confused, is the question of what pip actually needs from an sdist.
(Not PyPI, just pip or other package install tools.)
Right now, IIUC, there are three times that pip install touches
sdist-related metadata:
1) it uses the name+version that are embedded in the sdist filename to
select an sdist from an index like PyPI
2) after unpacking this sdist it then calls 'setup.py egg_info' to get
the full metadata for the wheel (or wheel equivalent) that this sdist
will eventually produce. Specifically what it does with this is
extract the setup_requires and install_requires fields, and uses them
to go find other wheels/sdists that also need to be installed
3) eventually it actually builds the package, and this produces a
wheel (or wheel equivalent) that has its own metadata (which often
matches the metadata from egg_info in step (2), but not always)
Is that a correct description of current behavior? Is there anything
that pip ever looks at besides name, version, dependencies?
Paul says that this is broken, and that pip gets lots of bug reports
that "can be traced back to needing to run setup.py egg-info to get
metadata" [1]. Since AFAICT the only metadata that pip actually
touches is name, version, and dependencies, and it already knows the
name and version before it runs egg_info, I assume that what this
means is that it's crucial for pip to have static access to dependency
information? OTOH in another email Paul says that name and version are
the minimum he wants [2], so maybe I'm reading too much into this :-).
>From the discussion so far, it sounds like the particularly crucial
question is whether pip needs to statically know dependencies before
building a wheel.
Trying to reason through from first principles, I can't see any reason
why it would.
It would be somewhat convenient if sdists did list their binary
dependencies: if that were the case, then pip could take a strictly
phased approach:
1) solve the complete dependency graph to find a set of packages to
install / remove
2) for all the packages-to-be-installed that are sdists, turn them into wheels
3) install all the wheels
OTOH if sdists have only name and version statically, but not
dependency information, then you need to do something like:
1) create a fake dependency graph that contains accurate information
for all known wheels, and for each sdist add a fake node that has the
right name and version number but pretends not to have any
dependencies.
2) solve this graph to find the set of packages to install / remove
3) if any of the packages-to-be-installed are sdists, then fetch them,
run egg_info or build them or whatever to get their real dependencies,
add these to the graph, and go to step 1
4) else, we have wheels for everything; install them.
(This works because dependencies are constraints -- adding
dependencies can only reduce the space of possible solutions, never
enlarge it. Also, because by the time we decide to fetch and build any
sdists, we already know that we're very likely to want to install
them, so the performance penalty for building packages we turn out not
to want is not high. And, crucially, we know that there exists some
set of dependency metadata which would convince us to install these
sdists, and dependency metadata is under the package author's control,
so we already have established a trust route to the author of this
package -- if they don't declare any dependencies, then we'll be
installing and running arbitrary code of theirs, so running arbitrary
code to check their dependencies doesn't require any additional
trust.)
But there's often a large difference between what we work out from
first principles and how things actually work :-). Is there anything
I'm missing in the analysis above? Do the relevant pip maintainers
even read this mailing list? :-)
-n
[1] https://mail.python.org/pipermail/distutils-sig/2015-October/026960.html
[2] https://mail.python.org/pipermail/distutils-sig/2015-October/026942.html
--
Nathaniel J. Smith -- http://vorpus.org
Hi all,
Again trying to split out some more focused discussion from the big
thread about sdists...
One big theme there has been the problem of "sources of truth": e.g.
in current sdists, there is a PKG-INFO file that has lots of static
metadata in it, but because the "real" version of that metadata is in
setup.py, everyone ignores PKG-INFO.
A clear desideratum for a new sdist format is that we avoid this
problem, by having static metadata that is actually trustworthy. I see
two fundamentally different strategies that we might use to accomplish
this. In time honored mailing list tradition, these are of course the
one that I hear other people advocating and the one that I like ;-).
The first strategy is: sdists and the wheels they generate logically
share the same metadata; so, we need some mechanism to enforce that
whatever static metadata is in the sdist will match the metadata in
the resulting wheel. (The wheel might potentially have additional
metadata beyond what is in the sdist, but anything that overlaps has
to match.) An open question is what this mechanism will look like --
if everyone used distutils/setuptools, then we could write the code in
distutils/setuptools so that when it generated wheel metadata, it
always copied it directly out of the sdist metadata (when present).
But not everyone will use distutils/setuptools, because distutils
delenda est. So we need some mechanism to statically analyze an
arbitrary build system and prove things about the data it outputs.
Which sounds... undecideable. Or we could have some kind of
after-the-fact enforcement mechanism, where tools like pip are
required -- as the last step when building a wheel from an sdist -- to
double-check that all the metadata matches, and if it doesn't then
they produce a hard error and refuse to continue. But even this
wouldn't necessarily guarantee that PyPI can trust the metadata, since
PyPI is not going to run this enforcement mechanism...
The second strategy is: put static metadata in both sdists and wheels,
but treat them as logically distinct things: the static metadata in
sdists is the source of truth for information *about that sdist*
(sdist name, sdist version, sdist description, sdist authors, etc.),
and the static metadata in wheels is the source of truth for
information about that wheel, but we think of these as distinct things
and don't pretend that we can statically guarantee that they will
match. I mean, in practice, they basically always will match. But IMO
making this distinction in our minds leads to clearer thinking. When
PyPI needs to know the name/version/description for an sdist, it can
still do that; and since we've lowered our ambitions to only finding
the sdist name instead of the wheel name, it can actually do it
reliably in a totally static way, without having to run arbitrary code
to validate this. OTOH pip will always have to be prepared to handle
the possibility of mismatch between what it was expecting based on the
sdist metadata and what it actually got after building it, so we might
as well acknowledge that in our mental model.
One potential advantage of this approach is that we might be able to
talk ourselves into trusting the existing PKG-INFO as providing static
metadata about the sdist, and thus PyPI at least could start trusting
it for things like the "description" field, and if we define a new
sdist format then it would be possible to generate its static metadata
from current setup.py files (e.g. by modifying setuptools's sdist
command). Contrast this with the other approach, where getting any
kind of static source-of-truth would require rewriting almost all
existing setup.py files.
The challenge, of course, is that there are a few places where pip
actually does need to know something about wheels based on examining
an sdist -- in particular name and version and (controversially)
dependencies. But this can/should be addressed explicitly, e.g. by
writing down a special rule about the name and version fields.
-n
--
Nathaniel J. Smith -- http://vorpus.org
On Tue, Oct 6, 2015 at 7:46 PM, Ionel Cristian Mărieș
<contact(a)ionelmc.ro> wrote:
>
> On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins <robertc(a)robertcollins.net>
> wrote:
>>
>>
>> Hangon, there's clearly a *huge* gap in understanding here.
>>
>> pbr does *not* modify *anyones* setup.py output unless its enabled.
>
>
> Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr
> doing weird stuff to distributions by abusing said entrypoints.
>
> For reference: https://bugs.launchpad.net/pbr/+bug/1483067
>
> There's nothing special about pbr here. It's not like it's the first package
> doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like
> pdbpp, but you don't put that in production.
Starting a sub-thread since issues with pbr are orthogonal to the
original disucssion.
But one point I'd like to raise about this is that when I originally
designed d2to1, on which a chunk of pbr is based, it was *explicitly*
designed to never be installed in site-packages (with the exception of
downstream packaging systems which can do what they want and are more
controlled). This is exactly because I knew different packages might
have dependencies on different versions of d2to1 as features are
added, and that if some version is installed in site-packages it can
lead to VersionConflict issues (this is in part exacerbated by a
bug/misfeature in setuptools--I fixed that bug a while ago but the fix
had to be rolled back due to a regression [1]).
So TL;DR unless you know what you're doing, d2to1 should *never* be
"installed"--it was only meant to be used with setup_requires, where
the appropriate d2to1 used in building/installing a package is only
temporarily enabled on sys.path via a temporary egg install. If some
project is making it a *runtime* requirement that's a mistake.
I don't know what features pbr has grown that might make someone want
it to be a runtime dependency (the only entry-points I noticed were
for adding egg-info writers but that should only be needed at
build-time too...), but maybe something like that should be split off
as a separate module or something...
Best,
Erik
[1] https://bitbucket.org/tarek/distribute/pull-requests/20/fixes-and-adds-a-re…
This is a follow up to the thread "Where should I put tests when packaging python modules?"
I have never used tox up to now. But reading the mails of the thread, it seems
that tox is the current state of the art.
This leads me to the conclusion that the sample project should use tox.
Any reasons not to do it?
Regards,
Thomas Güttler
--
http://www.thomas-guettler.de/
On Sun, Oct 4, 2015 at 1:02 PM, Paul Moore <p.f.moore(a)gmail.com> wrote:
[...]
> A
> common bug report for pip is users finding that their installs fail,
> because setup.py requires numpy to be installed in order to run, and
> yet pip is running setup.py egg-info precisely to find out what the
> requirements are. We tell the user that the setup.py is written
> incorrectly, and they should install numpy and retry the install, but
> it's not a good user experience. And from a selfish point of view,
> users blame *pip* for the consequences of a project whose code to
> generate the metadata is buggy. Those bug reports are a drain on the
> time of the pip developers, as well as a frustrating experience for
> the users.
[...]
This is mostly orthogonal to the other discussion, but it's a problem
relevant to distutils-sig where I think there's a lot of confusion, so
I just wanted to try and clarify what's happening here.
It's true that almost no packages accurately declare their
dependencies on numpy. This is for two reasons:
First, for packages that build-depend on numpy (e.g. they contain
extensions that need numpy's header files), then distutils' design
makes things tricky, b/c you have to call np.get_include() to find the
headers and pass that to setup(). But this isn't the main issue --
solving this requires nasty hacks, but at least the more prominent
projects are gradually gaining those hacks.
The main issue is that people really, really, REALLY hate the
recursive behavior of 'pip install -U'. They hate it *so much* --
especially the way it has a habit of suddenly trying to rebuild numpy
when all you were trying to do was upgrade some little pure-python
package -- that they actively refuse to accurately report their
project's dependencies, because install -U can't recurse over
dependencies that it can't see.
E.g., here's two setup.py files for extremely mainstream projects,
that both report accurate setup_requires and install_requires on numpy
if and only if numpy is not already installed; if it is installed then
they pretend that they don't need it:
https://github.com/scipy/scipy/blob/master/setup.py#L200-L208https://github.com/statsmodels/statsmodels/blob/master/setup.py#L96-L156
Obviously this is terrible -- it means that the wheels for these
projects end up with incorrect dependencies (usually! it's kinda
random!), and even if you install from source then whether future
upgrades will work correctly depends on arbitrary details about how
your virtualenv was configured when you did the original install.
But people care about this so much that you actually have prominent
developers going around and filing bugs on packages complaining that
they provide accurate metadata and that this is a bug, they should lie
instead. E.g.:
https://github.com/pydata/patsy/issues/5
I'm not sure I agree with this conclusion, but the users have spoken.
AFAICT the only way to fix this problem and start getting packages
with accurate metadata is for pip to gain a non-recursive upgrade
mode.
The bug for that is
https://github.com/pypa/pip/issues/59
AFAICT from reading that thread, work on this has stalled out because
of the following reasoning:
1) Everyone agrees that pip should have 'upgrade $PKG' and
'upgrade-all' commands, and 'install -U' should be deprecated/removed.
2) But implementing 'upgrade-all' is tricky and dangerous without
first fixing pip's dependency resolver.
3) Therefore we can't add 'upgrade' or 'upgrade-all' until after we
fix pip's dependency resolver.
I feel like there's a certain logical gap between (2) and (3)... we
could defer 'upgrade-all' until later but start supporting 'upgrade
$PKG' right now, couldn't we? (It'd be implemented as the equivalent
of 'pip install $PKG=$LATEST_VERSION', which is not scary looking at
all.)
-n
--
Nathaniel J. Smith -- http://vorpus.org
Hi, I built a package of using stdeb, any idea if these are acceptable to debian + ubuntu, and where to start / who to contact to get a package included in those distros ? S++
Hi
I'm a totally new to Python and I wanted to learn it. I have installed
Python IDLE 3.5 successfully.
Now I want to use Theano library in that IDE and I'm confused how to use
that.
Just have some questions like
Do I need to install additional packages to use Theano on 3.5 ?
Please kindly help.
Thank you
-------------------------------
Vivekanand Manchikanti