New submission from Tomas Dabasinskas:
In my setup.py I'm using dependency_links where I specify locations of packages I've built using setup.py sdist. When I run python setup.py develop to download, build and install those packages, I'm getting an error (please see attached)
If I go ahead and run:
(pantheon)[tomas@tomo-laptop pantheon-frontend-0.2]$ python setup.py -q bdist_egg --dist-dir /tmp/easy_install-cRdc3D/pantheon-frontend-0.2/egg-dist-tmp-hIIz0x
it works fine...
I've tried updating setuptools, but it's still the same
$ pip install setuptools --upgrade
Using cached setuptools-18.3.2-py2.py3-none-any.whl
Installing collected packages: setuptools
Found existing installation: setuptools 12.0.5
Successfully uninstalled setuptools-12.0.5
Successfully installed setuptools-18.3.2
title: Setuptools fails with MemoryError when using dependency_links
Added file: http://bugs.python.org/setuptools/file175/log
Setuptools tracker <setuptools(a)bugs.python.org>
Now that PEP 470 is accepted, I don't forsee any more changes to the API that
installers are expected to use when talking to PyPI. Given that there are
multiple implementations of PyPI and multiple clients talking to PyPI I wanted
to document the API that any implementation (PyPI included) is expected to
I've gone ahead and made myself the BDFL-Delegate for this PEP since this is
just documenting the current (once PEP 470 is fully implemented) behavior and
it isn't adding, changing, or removing anything. I did want to post it here for
others to see so that people can check my work but unless someone points
something out or objects to it I plan to accept this PEP in a week.
You can see this PEP online at https://www.python.org/dev/peps/pep-0503/ or I
have reproduced it inline below.
There are many implementations of a Python package repository and many tools
that consume them. Of these, the cannonical implementation that defines what
the "simple" repository API looks like is the implementation that powers
PyPI. This document will specify that API, documenting what the correct
behavior for any implementation of the simple repository API.
A repository that implements the simple API is defined by its base url, this is
the top level URL that all additional URLS are below. The API is named the
"simple" repository due to fact that PyPI's base URL is
.. note:: All subsequent URLs in this document will be relative to this base
URL (so given PyPI's URL, an URL of ``/foo/`` would be
Within a repository, the root URL (``/``) **MUST** be a valid HTML5 page with a
single anchor element per project in the repository. The text of the anchor tag
**MUST** be the normalized name of the project and the href attribute **MUST**
link to the URL for that particular project. As an example::
Below the root URL is another URL for each individual project contained within
a repository. The format of this URL is ``/<project>/`` where the ``<project>``
is replaced by the normalized name for that project, so a project named
"HolyGrail" would have an URL like ``/holygrail/``. This URL must response with
a valid HTML5 page with a single anchor element per file for the project. The
text of the anchor tag **MUST** be the filename of the file and the href
attribute **MUST** be an URL that links to the location of the file for
download. The URL **SHOULD** include a hash in the form of an URL fragment with
the following syntax: ``#<hashname>=<hashvalue>``, where ``<hashname>`` is the
lowercase name of the hash function (such as ``sha256``) and ``<hashvalue>`` is
the hex encoded digest.
In addition to the above, the following constraints are placed on the API:
* All URLs **MUST** end with a ``/`` and the repository **SHOULD** redirect the
URLs without a ``/`` to add a ``/`` to the end.
* There is no constraints on where the files must be hosted relative to the
* There may be any other HTML elements on the API pages as long as the required
anchor elements exist.
* Repositories **MAY** redirect unnormalized URLs to the cannonical normalized
URL (e.g. ``/Foobar/`` may redirect to ``/foobar/``), however clients
**MUST NOT** rely on this redirection and **MUST** request the normalized
* Repositories **SHOULD** choose a hash function from one of the ones
guarenteed to be available via the ``hashlib`` module in the Python standard
library (currently ``md5``, ``sha1``, ``sha224``, ``sha256``, ``sha384``,
``sha512``). The current recommendation is to use ``sha256``.
This PEP references the concept of a "normalized" project name. As per PEP 426
the only valid characters in a name are the ASCII alphabet, ASCII numbers,
``.``, ``-``, and ``_``. The name should be lowercased with all runs of the
characters ``.``, ``-``, or ``_`` replaced with a single ``-`` character. This
can be implemented in Python with the ``re`` module::
return re.sub(r"[-_.]+", "-", name).lower()
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I recently uploaded version 2.4.0 of SCons.
For some reason pip wasn't installing 2.4.0 but was pulling 2.3.6 so I
figured I'd delete the release and re-upload.
Then I get the following errors:
Submitting dist/scons-2.4.0.tar.gz to https://pypi.python.org/pypi
Upload failed (400): This filename has previously been used, you should use
a different version.
error: Upload failed (400): This filename has previously been used, you
should use a different version.
Do I really need to cut a whole new release (to change version #) to
re-upload to pypi?
I've recently been working on adding SOABI support for Python 2.x and other
pieces needed to get wheels w/ C extensions for Linux working. Here's the
work for wheels:
Based on that, I've added support for those wheels to pip here:
As mentioned in the wheels PR, there are some questions and decisions made
that I need guidance on:
- On Linux, the distro name/version (as determined by
platform.linux_distribution()) will be appended to the platform string,
e.g. linux_x86_64_ubuntu_14_04. This is going to be necessary to make a
reasonable attempt at wheel compatibility in PyPI. But this may violate PEP
- By default, wheels will be built using the most specific platform
information. In practice, I build our wheels using Debian Squeeze in
Docker and therefore they should work on most currently "supported"
Linuxes, but allowing such wheels to PyPI could still be dangerous because
forward compatibility is not always guaranteed (e.g. if a SO version/name
changes, or a C lib API method changes in a non-backward compatible way but
the SO version/name does not change). That said, I'd be happy to make a
much more generalized version of our docker-build system that'd allow
package authors to easily/rapidly build distro/version-specific wheels for
many of the popular Linux distros. We can assume that a wheel built on a
vanilla install of e.g. Ubuntu 14.04 will work on any other installation of
14.04 (this is what the distro vendors promise, anyway).
- I attempt to set the SOABI if the SOABI config var is unset, this is for
Python 2, but will also be done even on Python 3. Maybe that is the wrong
decision (or maybe SOABI is guaranteed to be set on Python 3).
- Do any other implementations define SOABI? PyPy does not, I did not test
others. What should we do with these?
Because the project I work for relies heavily on large number of
packages, some of which have complicated build-time dependencies, we have
always provided them as eggs and monkeypatched platform support back in to
pkg_resources. Now that the PyPA has settled on wheels as the preferred
binary packaging format, I am pretty heavily motivated to do the work to
work out all the issues with this implementation.
I am getting the below error while I am trying to install atfork package
from pip repositories. I have done a thorough google search but am not able
to find and appropriate solution for it.Installation of SSL packages and
ssl package from python too do not solve the mystry.
# pip install --allow-external atfork atforkCollecting atfork
Could not find a version that satisfies the requirement atfork (from
Some insecure and unverifiable files were ignored (use --allow-unverified
atfork to allow).
No matching distribution found for atfork
It says no matching distribution, but at pypi repo I am able to see it.
Sorry for the necropost :) "Life" has been happening and I wasn't
monitoring the discussion of the dependency PEP for a while. I have been
reviewing the discussion here. I thought I'd clear up a few things from my
Regarding practical examples:
Here is Lex Hider demonstrating some of the issues he faced. I don't think
he ran into BLAS issues as such, but it shows how much work can be required
to install some packages. At some point I should take a transcript of this
and write it up as a post, and adjust the PEP to use this as a real-world
motivating argument to remove argument about whether the underlying problem
of external dependencies is a 'real' problem or not. I fully agree it is
important to have a genuine example, and that would be a big improvement.
Regarding the connection to yum/deb/conda etc, my initial thought was
actually to specify package names rather than header files, but I was
convinced that supplying the missing header files was ultimately more
useful. This is, in part, because library names are more consistent than
package names across systems. On the other hand, I was still uncertain
about what happens if two libraries have the same name but different
functionality, you might want to refer to the package name instead.
Regarding whether this is a maintenance nightmare for authors who need to
understand a complex set of deploy environments. I don't really think so.
Typically, an author will have one 'primary' environment, often a Linux
platform, which will tend to 'just work' because the author is working in
that framework. Often, it's only a small number of dependencies that need
to be declared, and it's about giving package authors a 'release valve' to
manually handle exceptions where there are package issues.
Regarding 'boil the ocean' -- I think that's only true when this PEP is
taken to be trying to "solve all dependency interactions". In fact, it's
more about letting authors handle common, known issues on specific
platforms. That's my view anyway. I know if I had this for my packages,
just having one or two of these declared would handle a lot of problems.
On Sun, Sep 13, 2015 at 4:27 AM, PJ Eby <pje(a)telecommunity.com> wrote:
> On Thu, Sep 10, 2015 at 11:34 AM, Daniel Holth <dholth(a)gmail.com> wrote:
> > That is too bad. markerlib was added to pkg_resources as _markerlib in
> > It is used for .dist-info metadata as present in wheel. Then, only to
> > implement markers to setup.py or .egg-info style metadata, pkg_resources
> > gains its inline markers implementation in 2013, including its own
> > definitions of the key/values used in environment marker comparisons.
> > switched to underscores in revision aa74cf234684 in the inline
> > implementation and in revision 1fc8a2d94f9f for setuptools' vendored
> > _markerlib.
> To clarify: the Distribute fork of setuptools added _markerlib, but
> Distribute didn't support older version of Python as the official
> setuptools 0.6 did, which is why I added the inline version there.
Several years later, setuptools no longer supports <2.6 (as I understand
it, please correct me if I've missed something), and Jython now supports
everything we need to be able to drop the inline version.
> There was also active discussion at the time about changing the
> markers spec, as Nick was working on PEP 426. Previously, there were
> two other PEPs, 345 and 390, with their own not-quite compatible
> specs, and more recently there is now a PEP 496.
I knew about 345 but 390 is new to me.
*reads* Oh I see -
describe something very similar.
Thanks for that!
> So, there has never really been a stable standard for environment
> markers; all of the previous specs have had various ambiguities,
> underspecification, or excessive lenience. My hope at the time of the
> PEP 426 discussion was that we could define a *strict* grammar in
> place of a loose pseudo-grammar so that the spec would be robust to
> multiple implementations rather than forcing everyone to copy the
> quirks of one particular implementation (e.g. markerlib).
> Unfortunately, even PEP 496 is still a little underspecified: it
> doesn't even say what kinds of string literals are allowed, address
> encodings or character sets, etc. Is r"foo" a legal string
I'm not sure what you mean by this. But I think my confusion is based on a
point that I'll get to in a second..
Can you use double-quotes? Backslash escaping?
> Adjacent string literal concatentation? But it's better than the
> previous versions, since the mini-language is no longer an
> ambiguously-defined subset of Python and thus can no longer be parsed
> using Python's built-in grammar, and probably not its lexer either.
Actually, the draft implementation I put at  uses the ast module to
parse the markers.
At the time I wrote that I was under a misapprehension about the status of
markerlib, so I hadn't read it (not even the version vendored into
setuptools). After I wrote that draft I ended up looking at
setuptools/_markerlib/markers.py and realised that I'd written something
Hence my confusion when reading "is r'foo' a legal string?" and so on - I
hadn't even considered details like that because, although it's not longer
described that way in the PEP, I'd still used the python parser for the
I'm not sure what to do here; going back to defining it as a "subset of
Python" seems like a backward step, but I'm struggling to see what benefit
we'd get from the effort of more rigorously defining the grammar. Perhaps
that's my inexperience showing though - I'm happy to take guidance from
people who have been thinking about this for more than the 3 months or so
since I started looking at it.
 My current plan is to drop my implementation in setuptools and instead
work on https://bitbucket.org/dholth/markerlib; once PEP-496 is accepted
and markerlib implements it, we can rework setuptools (and pip, and
anything else that needs to read markers) with the new version.
> > I would prefer to see _markerlib._VARS used everywhere, the inline
> > pkg_resources implementation deleted, markerlib improved if necessary,
> > no backwards compatibility with unspecified environment marker
> variables. My
> > hunch is that no one needs the backwards compatibility.
> It may have changed since when I added markers to the official
> setuptools, but I intentionally did not document the marker feature;
> it was experimental and added mainly to support setuptools' internal
> use case of including SSL backports on older Python versions to
> support easy_install checking SSL certs. So, anybody using it was
> (and perhaps is; I haven't looked lately) using an undocumented
> experimental feature rather than an officially supported one.
I had a small novel written here, but it was pointless. In summary, that
you're correct to say that everyone using them markres an experimental
feature (from PEP-426) rather than the officially documented one (in
> In any case, if compatibility is broken, it should probably be done by
> switching to the much-stricter PEP 496, rather than one of its
> even-more-ambiguous predecessors.
> Distutils-SIG maillist - Distutils-SIG(a)python.org
I’m working on specifying dependencies for a project (IPython) that are
dependent on the Python implementation - we want to depend on a package on
CPython, but not on PyPy. I see from PEPs 345
<https://www.python.org/dev/peps/pep-0345/> and 426
<https://www.python.org/dev/peps/pep-0426/#environment-markers> and 496
<https://www.python.org/dev/peps/pep-0496/> that this should be available
as platform_python_version environment marker, but when I try to use this
in setup.py, it fails, claiming this is an invalid marker (setuptools
18.3.1). I discovered that pkg_resources has its own implementation of
which isn’t consistent with any PEPs describing them. pip uses markerlib,
which does seem to implement PEP 345 correctly. The relevant difference in
this case is that pkg_resources misspells platform_python_implementation as
python_implementation, but it is not the only one. Due to the inconsistent
implementations, I don’t think there’s a way to use this environment marker
anywhere. It seems like the whole concept of environment markers is
experimental, and it would be premature to adopt them for any packages in
production. Is this the case?
I found that when I run pip install ., the pkg_resources version is used,
and it will balk at the correct platform_python_version as invalid.
However, when I build a wheel and try to install it with pip install
ipython-...whl, the pip version is used, and it balks at pkg_resources
incorrect python_implementation. This conflict makes it impossible to use
the marker, as far as I can tell. Has anyone been able to use the python
implementation environment marker?
I have a PR
to setuptools to fix what seems to be some inconsistency with the PEPs
while preserving the misspelling for backward-compatibility, but it’s not
clear which metadata/env marker PEPs setuptools and/or pip are meant to
support at this point. It’s also unclear why pkg_resources has its own
implementation, instead of all participants using the shared _markerlib,
which would at least avoid inconsistency.