Distutils-SIG September 2015

distutils-sig@python.org

40 participants
42 discussions

No Python 3.5 option for Python version on files upload form

by Wolfgang Maier

You still cannot select Python 3.5 as the Python Version for wheels and other files uploaded to pypi over the web interface (see pypi issue #341).

5 years, 11 months

[issue162] Setuptools fails with MemoryError when using dependency_links

by Tomas Dabasinskas

New submission from Tomas Dabasinskas: In my setup.py I'm using dependency_links where I specify locations of packages I've built using setup.py sdist. When I run python setup.py develop to download, build and install those packages, I'm getting an error (please see attached) If I go ahead and run: (pantheon)[tomas@tomo-laptop pantheon-frontend-0.2]$ python setup.py -q bdist_egg --dist-dir /tmp/easy_install-cRdc3D/pantheon-frontend-0.2/egg-dist-tmp-hIIz0x it works fine... I've tried updating setuptools, but it's still the same $ pip install setuptools --upgrade Collecting setuptools Using cached setuptools-18.3.2-py2.py3-none-any.whl Installing collected packages: setuptools Found existing installation: setuptools 12.0.5 Uninstalling setuptools-12.0.5: Successfully uninstalled setuptools-12.0.5 Successfully installed setuptools-18.3.2 ---------- files: log messages: 750 nosy: tomas priority: bug status: unread title: Setuptools fails with MemoryError when using dependency_links Added file: http://bugs.python.org/setuptools/file175/log _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue162> _______________________________________________

5 years, 12 months

PEP 503 - Simple Repository API

by Donald Stufft

Now that PEP 470 is accepted, I don't forsee any more changes to the API that installers are expected to use when talking to PyPI. Given that there are multiple implementations of PyPI and multiple clients talking to PyPI I wanted to document the API that any implementation (PyPI included) is expected to adhere too. I've gone ahead and made myself the BDFL-Delegate for this PEP since this is just documenting the current (once PEP 470 is fully implemented) behavior and it isn't adding, changing, or removing anything. I did want to post it here for others to see so that people can check my work but unless someone points something out or objects to it I plan to accept this PEP in a week. You can see this PEP online at https://www.python.org/dev/peps/pep-0503/ or I have reproduced it inline below. ----------------------------------------------- Abstract ======== There are many implementations of a Python package repository and many tools that consume them. Of these, the cannonical implementation that defines what the "simple" repository API looks like is the implementation that powers PyPI. This document will specify that API, documenting what the correct behavior for any implementation of the simple repository API. Specification ============= A repository that implements the simple API is defined by its base url, this is the top level URL that all additional URLS are below. The API is named the "simple" repository due to fact that PyPI's base URL is ``https://pypi.python.org/simple/``. .. note:: All subsequent URLs in this document will be relative to this base URL (so given PyPI's URL, an URL of ``/foo/`` would be ``https://pypi.python.org/simple/foo/``). Within a repository, the root URL (``/``) **MUST** be a valid HTML5 page with a single anchor element per project in the repository. The text of the anchor tag **MUST** be the normalized name of the project and the href attribute **MUST** link to the URL for that particular project. As an example:: <!DOCTYPE html> <html> <body> <a href="/frob/">frob</a> <a href="/spamspamspam/">spamspamspam</a> </body> </html> Below the root URL is another URL for each individual project contained within a repository. The format of this URL is ``/<project>/`` where the ``<project>`` is replaced by the normalized name for that project, so a project named "HolyGrail" would have an URL like ``/holygrail/``. This URL must response with a valid HTML5 page with a single anchor element per file for the project. The text of the anchor tag **MUST** be the filename of the file and the href attribute **MUST** be an URL that links to the location of the file for download. The URL **SHOULD** include a hash in the form of an URL fragment with the following syntax: ``#<hashname>=<hashvalue>``, where ``<hashname>`` is the lowercase name of the hash function (such as ``sha256``) and ``<hashvalue>`` is the hex encoded digest. In addition to the above, the following constraints are placed on the API: * All URLs **MUST** end with a ``/`` and the repository **SHOULD** redirect the URLs without a ``/`` to add a ``/`` to the end. * There is no constraints on where the files must be hosted relative to the repository. * There may be any other HTML elements on the API pages as long as the required anchor elements exist. * Repositories **MAY** redirect unnormalized URLs to the cannonical normalized URL (e.g. ``/Foobar/`` may redirect to ``/foobar/``), however clients **MUST NOT** rely on this redirection and **MUST** request the normalized URL. * Repositories **SHOULD** choose a hash function from one of the ones guarenteed to be available via the ``hashlib`` module in the Python standard library (currently ``md5``, ``sha1``, ``sha224``, ``sha256``, ``sha384``, ``sha512``). The current recommendation is to use ``sha256``. Normalized Names ---------------- This PEP references the concept of a "normalized" project name. As per PEP 426 the only valid characters in a name are the ASCII alphabet, ASCII numbers, ``.``, ``-``, and ``_``. The name should be lowercased with all runs of the characters ``.``, ``-``, or ``_`` replaced with a single ``-`` character. This can be implemented in Python with the ``re`` module:: import re def normalize(name): return re.sub(r"[-_.]+", "-", name).lower() ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

5 years, 12 months

Can't re-upload package?

by Bill Deegan

Greetings, I recently uploaded version 2.4.0 of SCons. For some reason pip wasn't installing 2.4.0 but was pulling 2.3.6 so I figured I'd delete the release and re-upload. Then I get the following errors: Submitting dist/scons-2.4.0.tar.gz to https://pypi.python.org/pypi Upload failed (400): This filename has previously been used, you should use a different version. error: Upload failed (400): This filename has previously been used, you should use a different version. Do I really need to cut a whole new release (to change version #) to re-upload to pypi? -Bill

6 years

Working toward Linux wheel support

by Nate Coraor

Hi all, I've recently been working on adding SOABI support for Python 2.x and other pieces needed to get wheels w/ C extensions for Linux working. Here's the work for wheels: https://bitbucket.org/pypa/wheel/pull-request/54/ Based on that, I've added support for those wheels to pip here: https://github.com/natefoo/pip/tree/linux-wheels As mentioned in the wheels PR, there are some questions and decisions made that I need guidance on: - On Linux, the distro name/version (as determined by platform.linux_distribution()) will be appended to the platform string, e.g. linux_x86_64_ubuntu_14_04. This is going to be necessary to make a reasonable attempt at wheel compatibility in PyPI. But this may violate PEP 425. - By default, wheels will be built using the most specific platform information. In practice, I build our wheels[1] using Debian Squeeze in Docker and therefore they should work on most currently "supported" Linuxes, but allowing such wheels to PyPI could still be dangerous because forward compatibility is not always guaranteed (e.g. if a SO version/name changes, or a C lib API method changes in a non-backward compatible way but the SO version/name does not change). That said, I'd be happy to make a much more generalized version of our docker-build[2] system that'd allow package authors to easily/rapidly build distro/version-specific wheels for many of the popular Linux distros. We can assume that a wheel built on a vanilla install of e.g. Ubuntu 14.04 will work on any other installation of 14.04 (this is what the distro vendors promise, anyway). - I attempt to set the SOABI if the SOABI config var is unset, this is for Python 2, but will also be done even on Python 3. Maybe that is the wrong decision (or maybe SOABI is guaranteed to be set on Python 3). - Do any other implementations define SOABI? PyPy does not, I did not test others. What should we do with these? Because the project I work for[3] relies heavily on large number of packages, some of which have complicated build-time dependencies, we have always provided them as eggs and monkeypatched platform support back in to pkg_resources. Now that the PyPA has settled on wheels as the preferred binary packaging format, I am pretty heavily motivated to do the work to work out all the issues with this implementation. Thanks, --nate [1] https://wheels.galaxyproject.org/ [2] https://github.com/galaxyproject/docker-build/ [3] https://galaxyproject.org/

6 years

wheel 0.25.0 released with Python 3.5 support

by Daniel Holth

With Nate's help, I've released wheel 0.25.0, which includes contributions from several contributors, and most notably fixes a Python 3.5 issue. Enjoy! Daniel Holth

6 years

Fwd: error while installing using pip.

by OmPs

Hi all, I am getting the below error while I am trying to install atfork package from pip repositories. I have done a thorough google search but am not able to find and appropriate solution for it.Installation of SSL packages and ssl package from python too do not solve the mystry. # pip install --allow-external atfork atforkCollecting atfork Could not find a version that satisfies the requirement atfork (from versions: ) Some insecure and unverifiable files were ignored (use --allow-unverified atfork to allow). No matching distribution found for atfork It says no matching distribution, but at pypi repo I am able to see it. https://pypi.python.org/pypi/atfork/0.1.2

6 years

Re: [Distutils] PEP for dependencies on libraries like BLAS

by Tennessee Leeuwenburg

Hi all, Sorry for the necropost :) "Life" has been happening and I wasn't monitoring the discussion of the dependency PEP for a while. I have been reviewing the discussion here. I thought I'd clear up a few things from my perspective. Regarding practical examples: https://youtu.be/Fqknoni5aX0?list=PLs4CJRBY5F1KMMpoEWMuBRvHvjBJeAJoS&t=371 Here is Lex Hider demonstrating some of the issues he faced. I don't think he ran into BLAS issues as such, but it shows how much work can be required to install some packages. At some point I should take a transcript of this and write it up as a post, and adjust the PEP to use this as a real-world motivating argument to remove argument about whether the underlying problem of external dependencies is a 'real' problem or not. I fully agree it is important to have a genuine example, and that would be a big improvement. Regarding the connection to yum/deb/conda etc, my initial thought was actually to specify package names rather than header files, but I was convinced that supplying the missing header files was ultimately more useful. This is, in part, because library names are more consistent than package names across systems. On the other hand, I was still uncertain about what happens if two libraries have the same name but different functionality, you might want to refer to the package name instead. Regarding whether this is a maintenance nightmare for authors who need to understand a complex set of deploy environments. I don't really think so. Typically, an author will have one 'primary' environment, often a Linux platform, which will tend to 'just work' because the author is working in that framework. Often, it's only a small number of dependencies that need to be declared, and it's about giving package authors a 'release valve' to manually handle exceptions where there are package issues. Regarding 'boil the ocean' -- I think that's only true when this PEP is taken to be trying to "solve all dependency interactions". In fact, it's more about letting authors handle common, known issues on specific platforms. That's my view anyway. I know if I had this for my packages, just having one or two of these declared would handle a lot of problems.

6 years

The future of env markers (was Re: platform_python_implementation not implemented.)

by James Polley

On Sun, Sep 13, 2015 at 4:27 AM, PJ Eby <pje(a)telecommunity.com> wrote: > On Thu, Sep 10, 2015 at 11:34 AM, Daniel Holth <dholth(a)gmail.com> wrote: > > That is too bad. markerlib was added to pkg_resources as _markerlib in > 2012. > > It is used for .dist-info metadata as present in wheel. Then, only to > > implement markers to setup.py or .egg-info style metadata, pkg_resources > > gains its inline markers implementation in 2013, including its own > > definitions of the key/values used in environment marker comparisons. > Dots > > switched to underscores in revision aa74cf234684 in the inline > > implementation and in revision 1fc8a2d94f9f for setuptools' vendored > > _markerlib. > > To clarify: the Distribute fork of setuptools added _markerlib, but > Distribute didn't support older version of Python as the official > setuptools 0.6 did, which is why I added the inline version there. > Several years later, setuptools no longer supports <2.6 (as I understand it, please correct me if I've missed something), and Jython now supports everything we need to be able to drop the inline version. > There was also active discussion at the time about changing the > markers spec, as Nick was working on PEP 426. Previously, there were > two other PEPs, 345 and 390, with their own not-quite compatible > specs, and more recently there is now a PEP 496. > I knew about 345 but 390 is new to me. *reads* Oh I see - https://www.python.org/dev/peps/pep-0390/#context-dependant-sections does describe something very similar. Thanks for that! > > So, there has never really been a stable standard for environment > markers; all of the previous specs have had various ambiguities, > underspecification, or excessive lenience. My hope at the time of the > PEP 426 discussion was that we could define a *strict* grammar in > place of a loose pseudo-grammar so that the spec would be robust to > multiple implementations rather than forcing everyone to copy the > quirks of one particular implementation (e.g. markerlib). > > Unfortunately, even PEP 496 is still a little underspecified: it > doesn't even say what kinds of string literals are allowed, address > encodings or character sets, etc. Is r"foo" a legal string > expression? I'm not sure what you mean by this. But I think my confusion is based on a point that I'll get to in a second.. Can you use double-quotes? Backslash escaping? > Adjacent string literal concatentation? But it's better than the > previous versions, since the mini-language is no longer an > ambiguously-defined subset of Python and thus can no longer be parsed > using Python's built-in grammar, and probably not its lexer either. > Actually, the draft implementation I put at [1] uses the ast module to parse the markers. At the time I wrote that I was under a misapprehension about the status of markerlib, so I hadn't read it (not even the version vendored into setuptools). After I wrote that draft I ended up looking at setuptools/_markerlib/markers.py[2] and realised that I'd written something very similar[3] Hence my confusion when reading "is r'foo' a legal string?" and so on - I hadn't even considered details like that because, although it's not longer described that way in the PEP, I'd still used the python parser for the implementation. I'm not sure what to do here; going back to defining it as a "subset of Python" seems like a backward step, but I'm struggling to see what benefit we'd get from the effort of more rigorously defining the grammar. Perhaps that's my inexperience showing though - I'm happy to take guidance from people who have been thinking about this for more than the 3 months or so since I started looking at it. [1] https://bitbucket.org/pypa/setuptools/pull-requests/141/implementation-of-p… [2] https://bitbucket.org/pypa/setuptools/src/121794c652a32aed899515b16227f13c2… [3] My current plan is to drop my implementation in setuptools and instead work on https://bitbucket.org/dholth/markerlib; once PEP-496 is accepted and markerlib implements it, we can rework setuptools (and pip, and anything else that needs to read markers) with the new version. > > > I would prefer to see _markerlib._VARS used everywhere, the inline > > pkg_resources implementation deleted, markerlib improved if necessary, > and > > no backwards compatibility with unspecified environment marker > variables. My > > hunch is that no one needs the backwards compatibility. > > It may have changed since when I added markers to the official > setuptools, but I intentionally did not document the marker feature; > it was experimental and added mainly to support setuptools' internal > use case of including SSL backports on older Python versions to > support easy_install checking SSL certs. So, anybody using it was > (and perhaps is; I haven't looked lately) using an undocumented > experimental feature rather than an officially supported one. > I had a small novel written here, but it was pointless. In summary, that you're correct to say that everyone using them markres an experimental feature (from PEP-426) rather than the officially documented one (in PEP-345) > > In any case, if compatibility is broken, it should probably be done by > switching to the much-stricter PEP 496, rather than one of its > even-more-ambiguous predecessors. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig >

6 years

platform_python_implementation not implemented.

by MinRK

Hello, I’m working on specifying dependencies for a project (IPython) that are dependent on the Python implementation - we want to depend on a package on CPython, but not on PyPy. I see from PEPs 345 <https://www.python.org/dev/peps/pep-0345/> and 426 <https://www.python.org/dev/peps/pep-0426/#environment-markers> and 496 <https://www.python.org/dev/peps/pep-0496/> that this should be available as platform_python_version environment marker, but when I try to use this in setup.py, it fails, claiming this is an invalid marker (setuptools 18.3.1). I discovered that pkg_resources has its own implementation of environment markers <https://bitbucket.org/pypa/setuptools/src/7a705b610abb1177ca169311c4ee261f3…>, which isn’t consistent with any PEPs describing them. pip uses markerlib, which does seem to implement PEP 345 correctly. The relevant difference in this case is that pkg_resources misspells platform_python_implementation as python_implementation, but it is not the only one. Due to the inconsistent implementations, I don’t think there’s a way to use this environment marker anywhere. It seems like the whole concept of environment markers is experimental, and it would be premature to adopt them for any packages in production. Is this the case? I found that when I run pip install ., the pkg_resources version is used, and it will balk at the correct platform_python_version as invalid. However, when I build a wheel and try to install it with pip install ipython-...whl, the pip version is used, and it balks at pkg_resources incorrect python_implementation. This conflict makes it impossible to use the marker, as far as I can tell. Has anyone been able to use the python implementation environment marker? I have a PR <https://bitbucket.org/pypa/setuptools/pull-requests/147/make-environment-ma…> to setuptools to fix what seems to be some inconsistency with the PEPs while preserving the misspelling for backward-compatibility, but it’s not clear which metadata/env marker PEPs setuptools and/or pip are meant to support at this point. It’s also unclear why pkg_resources has its own implementation, instead of all participants using the shared _markerlib, which would at least avoid inconsistency. Thanks, -MinRK

6 years

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG September 2015