Mailman 3 October 2015 - Distutils-SIG

Meeting info re: sdists
by Nathaniel Smith 14 Oct '15

14 Oct '15

Hangouts URL: https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a Shared document for notes: https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkX… -n -- Nathaniel J. Smith -- http://vorpus.org

7 18

rst not rendered on pypi
by Nico Schlömer 14 Oct '15

14 Oct '15

Hi everyone, I'm trying to get my README.rst [1] display properly on my PyPy project [2], but for some reason it's displayed as raw text. The file was created from a README.md [3] via ``` pandoc README.md -o README.rst ``` Any hints on what may have been going wrong? Cheers, Nico [1] http://chunk.io/f/f4385eda77de4ed78f8eba8cc8d8c8b6 [2] https://pypi.python.org/pypi/voropy [3] https://github.com/nschloe/voropy/blob/master/README.md

4 4

Towards a simple and standard sdist format that isn't intertwined with distutils
by Nathaniel Smith 12 Oct '15

12 Oct '15

Hi all, We realized that actually as far as we could tell, it wouldn't be that hard at this point to clean up how sdists work so that it would be possible to migrate away from distutils. So we wrote up a little draft proposal. The main question is, does this approach seem sound? -n --- PEP: ?? Title: Standard interface for interacting with source trees and source distributions Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com> Thomas Kluyver <takowl(a)gmail.com> Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: Discussions-To: <distutils-sig(a)python.org> Abstract ======== Distutils delenda est. Extended abstract ================= While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like autoconfiguration and usable build-time dependency declaration, (b) extending them is quirky, complicated, and fragile, (c) you are forced to use them anyway, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). We propose to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then the more the merrier. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit <https://github.com/takluyver/flit>`_ or `bento <https://cournape.github.io/Bento/>`_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. Synopsis and rationale ====================== To limit the scope of our design, we adopt several principles. First, we distinguish between a *source tree* (e.g., a VCS checkout) and a *source distribution* (e.g., an official snapshot release like ``lxml-3.4.4.zip``). There isn't a whole lot that *source trees* can be assumed to have in common. About all you know is that they can -- via some more or less Rube-Goldbergian process -- produce one or more binary distributions. In particular, you *cannot* tell via simple static inspection: - What version number will be attached to the resulting packages (e.g. it might be determined programmatically by consulting VCS metadata -- I have here a build of numpy version "1.11.0.dev0+4a9ad17") - What build- or run-time dependencies are required (e.g. these may depend on arbitrarily complex configuration settings that are determined via a mix of manual settings and auto-probing) - Or even how many distinct binary distributions will be produced (e.g. a source distribution may always produce wheel A, but only produce wheel B when built on Unix-like systems). Therefore, when dealing with source trees, our goal is just to provide a standard UX for the core operations that are commonly performed on other people's packages; anything fancier and more developer-centric we leave at the discretion of individual package developers. So our source trees just provide some simple hooks to let a tool like ``pip``: - query for build dependencies - run a build, producing wheels as output - set up the current source tree so that it can be placed on ``sys.path`` in "develop mode" and that's it. We teach users that the standard way to install a package from a VCS checkout is now ``pip install .`` instead of ``python setup.py install``. (This is already a good idea anyway -- e.g., pip can do reliable uninstall / upgrades.) Next, we note that pretty much all the operations that you might want to perform on a *source distribution* are also operations that you might want to perform on a source tree, and via the same UX. The only thing you do with source distributions that you don't do with source trees is, well, distribute them. There's all kind of metadata you could imagine including in a source distribution, but each piece of metadata puts an increased burden on source distribution generation tools, and most operations will still have to work without this metadata. So we only include extra metadata in source distributions if it helps solve specific problems that are unique to distribution. If you want wheel-style metadata, get a wheel and look at it -- they're great and getting better. Therefore, our source distributions are basically just source trees + a mechanism for signing. Finally: we explicitly do *not* have any concept of "depending on a source distribution". As in other systems like Debian, dependencies are always phrased in terms of binary distributions (wheels), and when a user runs something like ``pip install <package>``, then the long-run plan is that <package> and all its transitive dependencies should be available as wheels in a package index. But this is not yet realistic, so as a transitional / backwards-compatibility measure, we provide a simple mechanism for ``pip install <package>`` to handle cases where <package> is provided only as a source distribution. Source trees ============ We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code of ``distutils``, ``setuptools``, ``pip``, and other tools. A version 1-or-greater format source tree can be identified by the presence of a file ``_pypackage/_pypackage.cfg``. If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. This is necessary because we anticipate that version 1+ source trees may want to contain a ``setup.py`` file for backwards compatibility, e.g.:: #!/usr/bin/env python import sys print("Don't call setup.py directly!") print("Use 'pip install .' instead!") print("(You might have to upgrade pip first.)") sys.exit(1) In the current version of the specification, the one file ``_pypackage/_pypackage.cfg`` is where pretty much all the action is (though see below). The motivation for putting it into a subdirectory is that: - the way of all standards is that cruft accumulates over time, so this way we pre-emptively have a place to put it, - real-world projects often accumulate build system cruft as well, so we might as well provide one obvious place to put it too. Of course this then creates the possibility of collisions between standard files and user files, and trying to teach arbitrary users not to scatter files around willy-nilly never works, so we adopt the convention that names starting with an underscore are reserved for official use, and non-underscored names are available for idiosyncratic use by individual projects. The alternative would be to simply place the main configuration file at the top-level, create the subdirectory only when specifically needed (most trees won't need it), and let users worry about finding their own place for their cruft. Not sure which is the best approach. Plus we can have a nice bikeshed about the names in general (FIXME). _pypackage.cfg -------------- The ``_pypackage.cfg`` file contains various settings. Another good bike-shed topic is which file format to use for storing these (FIXME), but for purposes of this draft I'll write examples using `toml <https://github.com/toml-lang/toml>`_, because you'll instantly be able to understand the semantics, it has similar expressivity to JSON while being more human-friendly (e.g., it supports comments and multi-line strings), it's better-specified than ConfigParser, and it's much simpler than YAML. Rust's package manager uses toml for similar purposes. Here's an example ``_pypackage/_pypackage.cfg``:: # Version of the "pypackage format" that this file uses. # Optional. If not present then 1 is assumed. # All version changes indicate incompatible changes; backwards # compatible changes are indicated by just having extra stuff in # the file. version = 1 [build] # An inline requirements file. Optional. # (FIXME: I guess this means we need a spec for requirements files?) requirements = """ mybuildtool >= 2.1 special_windows_tool ; sys_platform == "win32" """ # The path to an out-of-line requirements file. Optional. requirements-file = "build-requirements.txt" # A hook that will be called to query build requirements. Optional. requirements-dynamic = "mybuildtool:get_requirements" # A hook that will be called to build wheels. Required. build-wheels = "mybuildtool:do_build" # A hook that will be called to do an in-place build (see below). # Optional. build-in-place = "mybuildtool:do_inplace_build" # The "x" namespace is reserved for third-party extensions. # To use x.foo you should own the name "foo" on pypi. [x.mybuildtool] spam = ["spam", "spam", "spam"] All paths are relative to the ``_pypackage/`` directory (so e.g. the build.requirements-file value above refers to a file named ``_pypackage/build-requirements.txt``). A *hook* is a Python object that is looked up using the same rules as traditional setuptools entry_points: a dotted module name, followed by a colon, followed by a dotted name that is looked up within that module. *Running a hook* means: first, find or create a python interpreter which is executing in the current venv, whose working directory is set to the ``_pypackage/`` directory, and which has the ``_pypackage/`` directory on ``sys.path``. Then, inside this interpreter, look up the hook object, and call it, with arguments as specified below. A build command like ``pip wheel <source tree>`` performs the following steps: 1) Validate the ``_pypackage.cfg`` version number. 2) Create an empty virtualenv / venv, that matches the environment that the installer is targeting (e.g. if you want wheels for CPython 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows venv). 3) If the build.requirements key is present, then in this venv run the equivalent of ``pip install -r <a file containing its value>``, using whatever index settings are currently in effect. 4) If the build.requirements-file key is present, then in this venv run the equivalent of ``pip install -r <the named file>``, using whatever index settings are currently in effect. 5) If the build.requirements-dynamic key is present, then in this venv run the hook with no arguments, capture its stdout, and pipe it into ``pip install -r -``, using whatever index settings are currently in effect. If the hook raises an exception, then abort the build with an error. Note: because these steps are performed in sequence, the build.requirements-dynamic hook is allowed to use packages that are listed in build.requirements or build.requirements-file. 6) In this venv, run the build.build-wheels hook. This should be a Python function which takes one argument. This argument is an arbitrary dictionary intended to contain user-specified configuration, specified via some install-tool-specific mechanism. The intention is that tools like ``pip`` should provide some way for users to specify key/value settings that will be passed in here, analogous to the legacy ``--install-option`` and ``--global-option`` arguments. To make it easier for packages to transition from version 0 to version 1 sdists, we suggest that ``pip`` and other tools that have such existing option-setting interfaces SHOULD map them to entries in this dictionary when -- e.g.:: pip --global-option=a --install-option=b --install-option=c could produce a dict like:: {"--global-option": ["a"], "--install-option": ["b", "c"]} The hook's return value is a list of pathnames relative to the scratch directory. Each entry names a wheel file created by this build. Errors are signaled by raising an exception. When performing an in-place build (e.g. for ``pip install -e .``), then the same steps are followed, except that instead of the build.build-wheels hook, we call the build.build-in-place hook, and instead of returning a list of wheel files, it returns the name of a directory that should be placed onto ``sys.path`` (usually this will be the source tree itself, but may not be, e.g. if a build system wants to enforce a rule where the source is always kept pristine then it could symlink the .py files into a build directory, place the extension modules and dist-info there, and return that). This directory must contain importable versions of the code in the source tree, along with appropriate .dist-info directories. (FIXME: in-place builds are useful but intrinsically kinda broken -- e.g. extensions / source / metadata can all easily get out of sync -- so while I think this paragraph provides a reasonable hack that preserves current functionality, maybe we should defer specifying them to until after we've thought through the issues more?) When working with source trees, build tools like ``pip`` are encouraged to cache and re-use virtualenvs for performance. Other contents of _pypackage/ ----------------------------- _RECORD, _RECORD.jws, _RECORD.p7s: see below. _x/<pypi name>/: reserved for use by tools (e.g. _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) Source distributions ==================== A *source distribution* is a file in a well-known archive format such as zip or tar.gz, which contains a single directory, and this directory is a source tree (in the sense defined in the previous section). The ``_pypackage/`` directory in a source distribution SHOULD also contain a _RECORD file, as defined in PEP 427, and MAY also contain _RECORD.jws and/or _RECORD.p7s signature files. For official releases, source distributions SHOULD be named as ``<package>-<version>.<ext>``, and the directory they contain SHOULD be named ``<package>-<version>``, and building this source tree SHOULD produce a wheel named ``<package>-<version>-<compatibility tag>.whl`` (though it may produce other wheels as well). (FIXME: maybe we should add that if you want your sdist on PyPI then you MUST include a proper _RECORD file and use the proper naming convention?) Integration tools like ``pip`` SHOULD take advantage of this convention by applying the following heuristic: when seeking a package <package>, if no appropriate wheel can be found, but an sdist named <package>-<version>.<ext> is found, then: 1) build the sdist 2) add the resulting wheels to the package search space 3) retry the original operation This handles a variety of simple and complex cases -- for example, if we need a package 'foo', and we find foo-1.0.zip which builds foo.whl and bar.whl, and foo.whl depends on bar.whl, then everything will work out. There remain other cases that are not handled, e.g. if we start out searching for bar.whl we will never discover foo-1.0.zip. We take the perspective that this is nonetheless sufficient for a transitional heuristic, and anyone who runs into this problem should just upload wheels already. If this turns out to be inadequate in practice, then it will be addressed by future extensions. Examples ======== **Example 1:** While we assume that installation tools will have to continue supporting version 0 sdists for the indefinite future, it's a useful check to make sure that our new format can continue to support packages using distutils / setuptools as their build system. We assume that a future version ``pip`` will take its existing knowledge of distutils internals and expose them as the appropriate hooks, and then existing distutils / setuptools packages can be ported forward by using the following ``_pypackage/_pypackage.cfg``:: [build] requirements = """ pip >= whatever wheel """ # Applies monkeypatches, then does 'setup.py dist_info' and # extracts the setup_requires requirements-dynamic = "pip.pypackage_hooks:setup_requirements" # Applies monkeypatches, then does 'setup.py wheel' build-wheels = "pip.pypackage_hooks:build_wheels" # Applies monkeypatches, then does: # setup.py dist_info && setup.py build_ext -i build-in-place = "pip.pypackage_hooks:build_in_place" This is also useful for any other installation tools that may want to support version 0 sdists without having to implement bug-for-bug compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is present, they can use this as a default. **Example 2:** For packages using numpy.distutils. This is identical to the distutils / setuptools example above, except that numpy is moved into the list of static build requirements. Right now, most projects using numpy.distutils don't bother trying to declare this dependency, and instead simply error out if numpy is not already installed. This is because currently the only way to declare a build dependency is via the ``setup_requires`` argument to the ``setup`` function, and in this case the ``setup`` function is ``numpy.distutils.setup``, which... obviously doesn't work very well. Drop this ``_pypackage.cfg`` into an existing project like this and it will become robustly pip-installable with no further changes:: [build] requirements = """ numpy pip >= whatever wheel """ requirements-dynamic = "pip.pypackage_hooks:setup_requirements" build-wheels = "pip.pypackage_hooks:build_wheels" build-in-place = "pip.pypackage_hooks:build_in_place" **Example 3:** `flit <https://github.com/takluyver/flit>`_ is a tool designed to make distributing simple packages simple, but it currently has no support for sdists, and for convenience includes its own installation code that's redundant with that in pip. These 4 lines of boilerplate make any flit-using source tree pip-installable, and lets flit get out of the package installation business:: [build] requirements = "flit" build-wheels = "flit.pypackage_hooks:build_wheels" build-in-place = "flit.pypackage_hooks:build_in_place" FAQ === **Why is it version 1 instead of version 2?** Because the legacy sdist format is barely a format at all, and to `remind us to keep things simple <https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effe…>`_. **What about cross-compilation?** Standardizing an interface for cross-compilation seems premature given how complicated the configuration required can be, the lack of an existing de facto standard, and the authors of this PEP's inexperience with cross-compilation. This would be a great target for future extensions, though. In the mean time, there's no requirement that ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a project's build system -- packages that want to support cross-compilation can still do so, they'll just need to include a README explaining how to do it. **PEP 426 says that the new sdist format will support automatically creating policy-compliant .deb/.rpm packages. What happened to that?** Step 1: enhance the wheel format as necessary so that a wheel can be automatically converted into a policy-compliant .deb/.rpm package (see PEP 491). Step 2: make it possible to automatically turn sdists into wheels (this PEP). Step 3: we're done. **What about automatically running tests?** Arguably this is another thing that should be pushed off to wheel metadata instead of sdist metadata: it's good practice to include tests inside your built distribution so that end-users can test their install (and see above re: our focus here being on stuff that end-users want to do, not dedicated package developers), there are lots of packages that have to be built before they can be tested anyway (e.g. because of binary extensions), and in any case it's good practice to test against an installed version in order to make sure your install code works properly. But even if we do want this in sdist, then it's hardly urgent (e.g. there is no ``pip test`` that people will miss), so we defer that for a future extension to avoid blocking the core functionality. -- Nathaniel J. Smith -- http://vorpus.org

19 128

What metadata does pip actually need about sdists?
by Nathaniel Smith 12 Oct '15

12 Oct '15

Hi all, I'm finding it impossible to keep track of that other thread, and I guess I'm probably not the only one, so I figured I'd try splitting a few of the more specific discussions out :-). One thing that seems to be a key issue, but where I remain very confused, is the question of what pip actually needs from an sdist. (Not PyPI, just pip or other package install tools.) Right now, IIUC, there are three times that pip install touches sdist-related metadata: 1) it uses the name+version that are embedded in the sdist filename to select an sdist from an index like PyPI 2) after unpacking this sdist it then calls 'setup.py egg_info' to get the full metadata for the wheel (or wheel equivalent) that this sdist will eventually produce. Specifically what it does with this is extract the setup_requires and install_requires fields, and uses them to go find other wheels/sdists that also need to be installed 3) eventually it actually builds the package, and this produces a wheel (or wheel equivalent) that has its own metadata (which often matches the metadata from egg_info in step (2), but not always) Is that a correct description of current behavior? Is there anything that pip ever looks at besides name, version, dependencies? Paul says that this is broken, and that pip gets lots of bug reports that "can be traced back to needing to run setup.py egg-info to get metadata" [1]. Since AFAICT the only metadata that pip actually touches is name, version, and dependencies, and it already knows the name and version before it runs egg_info, I assume that what this means is that it's crucial for pip to have static access to dependency information? OTOH in another email Paul says that name and version are the minimum he wants [2], so maybe I'm reading too much into this :-). >From the discussion so far, it sounds like the particularly crucial question is whether pip needs to statically know dependencies before building a wheel. Trying to reason through from first principles, I can't see any reason why it would. It would be somewhat convenient if sdists did list their binary dependencies: if that were the case, then pip could take a strictly phased approach: 1) solve the complete dependency graph to find a set of packages to install / remove 2) for all the packages-to-be-installed that are sdists, turn them into wheels 3) install all the wheels OTOH if sdists have only name and version statically, but not dependency information, then you need to do something like: 1) create a fake dependency graph that contains accurate information for all known wheels, and for each sdist add a fake node that has the right name and version number but pretends not to have any dependencies. 2) solve this graph to find the set of packages to install / remove 3) if any of the packages-to-be-installed are sdists, then fetch them, run egg_info or build them or whatever to get their real dependencies, add these to the graph, and go to step 1 4) else, we have wheels for everything; install them. (This works because dependencies are constraints -- adding dependencies can only reduce the space of possible solutions, never enlarge it. Also, because by the time we decide to fetch and build any sdists, we already know that we're very likely to want to install them, so the performance penalty for building packages we turn out not to want is not high. And, crucially, we know that there exists some set of dependency metadata which would convince us to install these sdists, and dependency metadata is under the package author's control, so we already have established a trust route to the author of this package -- if they don't declare any dependencies, then we'll be installing and running arbitrary code of theirs, so running arbitrary code to check their dependencies doesn't require any additional trust.) But there's often a large difference between what we work out from first principles and how things actually work :-). Is there anything I'm missing in the analysis above? Do the relevant pip maintainers even read this mailing list? :-) -n [1] https://mail.python.org/pipermail/distutils-sig/2015-October/026960.html [2] https://mail.python.org/pipermail/distutils-sig/2015-October/026942.html -- Nathaniel J. Smith -- http://vorpus.org

5 15

Sources of truth
by Nathaniel Smith 12 Oct '15

12 Oct '15

Hi all, Again trying to split out some more focused discussion from the big thread about sdists... One big theme there has been the problem of "sources of truth": e.g. in current sdists, there is a PKG-INFO file that has lots of static metadata in it, but because the "real" version of that metadata is in setup.py, everyone ignores PKG-INFO. A clear desideratum for a new sdist format is that we avoid this problem, by having static metadata that is actually trustworthy. I see two fundamentally different strategies that we might use to accomplish this. In time honored mailing list tradition, these are of course the one that I hear other people advocating and the one that I like ;-). The first strategy is: sdists and the wheels they generate logically share the same metadata; so, we need some mechanism to enforce that whatever static metadata is in the sdist will match the metadata in the resulting wheel. (The wheel might potentially have additional metadata beyond what is in the sdist, but anything that overlaps has to match.) An open question is what this mechanism will look like -- if everyone used distutils/setuptools, then we could write the code in distutils/setuptools so that when it generated wheel metadata, it always copied it directly out of the sdist metadata (when present). But not everyone will use distutils/setuptools, because distutils delenda est. So we need some mechanism to statically analyze an arbitrary build system and prove things about the data it outputs. Which sounds... undecideable. Or we could have some kind of after-the-fact enforcement mechanism, where tools like pip are required -- as the last step when building a wheel from an sdist -- to double-check that all the metadata matches, and if it doesn't then they produce a hard error and refuse to continue. But even this wouldn't necessarily guarantee that PyPI can trust the metadata, since PyPI is not going to run this enforcement mechanism... The second strategy is: put static metadata in both sdists and wheels, but treat them as logically distinct things: the static metadata in sdists is the source of truth for information *about that sdist* (sdist name, sdist version, sdist description, sdist authors, etc.), and the static metadata in wheels is the source of truth for information about that wheel, but we think of these as distinct things and don't pretend that we can statically guarantee that they will match. I mean, in practice, they basically always will match. But IMO making this distinction in our minds leads to clearer thinking. When PyPI needs to know the name/version/description for an sdist, it can still do that; and since we've lowered our ambitions to only finding the sdist name instead of the wheel name, it can actually do it reliably in a totally static way, without having to run arbitrary code to validate this. OTOH pip will always have to be prepared to handle the possibility of mismatch between what it was expecting based on the sdist metadata and what it actually got after building it, so we might as well acknowledge that in our mental model. One potential advantage of this approach is that we might be able to talk ourselves into trusting the existing PKG-INFO as providing static metadata about the sdist, and thus PyPI at least could start trusting it for things like the "description" field, and if we define a new sdist format then it would be possible to generate its static metadata from current setup.py files (e.g. by modifying setuptools's sdist command). Contrast this with the other approach, where getting any kind of static source-of-truth would require rewriting almost all existing setup.py files. The challenge, of course, is that there are a few places where pip actually does need to know something about wheels based on examining an sdist -- in particular name and version and (controversially) dependencies. But this can/should be addressed explicitly, e.g. by writing down a special rule about the name and version fields. -n -- Nathaniel J. Smith -- http://vorpus.org

3 3

Re: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?)
by Erik Bray 12 Oct '15

12 Oct '15

On Tue, Oct 6, 2015 at 7:46 PM, Ionel Cristian Mărieș <contact(a)ionelmc.ro> wrote: > > On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins <robertc(a)robertcollins.net> > wrote: >> >> >> Hangon, there's clearly a *huge* gap in understanding here. >> >> pbr does *not* modify *anyones* setup.py output unless its enabled. > > > Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr > doing weird stuff to distributions by abusing said entrypoints. > > For reference: https://bugs.launchpad.net/pbr/+bug/1483067 > > There's nothing special about pbr here. It's not like it's the first package > doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like > pdbpp, but you don't put that in production. Starting a sub-thread since issues with pbr are orthogonal to the original disucssion. But one point I'd like to raise about this is that when I originally designed d2to1, on which a chunk of pbr is based, it was *explicitly* designed to never be installed in site-packages (with the exception of downstream packaging systems which can do what they want and are more controlled). This is exactly because I knew different packages might have dependencies on different versions of d2to1 as features are added, and that if some version is installed in site-packages it can lead to VersionConflict issues (this is in part exacerbated by a bug/misfeature in setuptools--I fixed that bug a while ago but the fix had to be rolled back due to a regression [1]). So TL;DR unless you know what you're doing, d2to1 should *never* be "installed"--it was only meant to be used with setup_requires, where the appropriate d2to1 used in building/installing a package is only temporarily enabled on sys.path via a temporary egg install. If some project is making it a *runtime* requirement that's a mistake. I don't know what features pbr has grown that might make someone want it to be a runtime dependency (the only entry-points I noticed were for adding egg-info writers but that should only be needed at build-time too...), but maybe something like that should be split off as a separate module or something... Best, Erik [1] https://bitbucket.org/tarek/distribute/pull-requests/20/fixes-and-adds-a-re…

4 4

sampleproject: use tox?
by Thomas Güttler 11 Oct '15

11 Oct '15

This is a follow up to the thread "Where should I put tests when packaging python modules?" I have never used tox up to now. But reading the mails of the thread, it seems that tox is the current state of the art. This leads me to the conclusion that the sample project should use tox. Any reasons not to do it? Regards, Thomas Güttler -- http://www.thomas-guettler.de/

8 10

Bad setup.py dependencies on numpy, and 'pip upgrade' (was: Towards a simple and standard sdist format that isn't intertwined with distutils)
by Nathaniel Smith 11 Oct '15

11 Oct '15

On Sun, Oct 4, 2015 at 1:02 PM, Paul Moore <p.f.moore(a)gmail.com> wrote: [...] > A > common bug report for pip is users finding that their installs fail, > because setup.py requires numpy to be installed in order to run, and > yet pip is running setup.py egg-info precisely to find out what the > requirements are. We tell the user that the setup.py is written > incorrectly, and they should install numpy and retry the install, but > it's not a good user experience. And from a selfish point of view, > users blame *pip* for the consequences of a project whose code to > generate the metadata is buggy. Those bug reports are a drain on the > time of the pip developers, as well as a frustrating experience for > the users. [...] This is mostly orthogonal to the other discussion, but it's a problem relevant to distutils-sig where I think there's a lot of confusion, so I just wanted to try and clarify what's happening here. It's true that almost no packages accurately declare their dependencies on numpy. This is for two reasons: First, for packages that build-depend on numpy (e.g. they contain extensions that need numpy's header files), then distutils' design makes things tricky, b/c you have to call np.get_include() to find the headers and pass that to setup(). But this isn't the main issue -- solving this requires nasty hacks, but at least the more prominent projects are gradually gaining those hacks. The main issue is that people really, really, REALLY hate the recursive behavior of 'pip install -U'. They hate it *so much* -- especially the way it has a habit of suddenly trying to rebuild numpy when all you were trying to do was upgrade some little pure-python package -- that they actively refuse to accurately report their project's dependencies, because install -U can't recurse over dependencies that it can't see. E.g., here's two setup.py files for extremely mainstream projects, that both report accurate setup_requires and install_requires on numpy if and only if numpy is not already installed; if it is installed then they pretend that they don't need it: https://github.com/scipy/scipy/blob/master/setup.py#L200-L208 https://github.com/statsmodels/statsmodels/blob/master/setup.py#L96-L156 Obviously this is terrible -- it means that the wheels for these projects end up with incorrect dependencies (usually! it's kinda random!), and even if you install from source then whether future upgrades will work correctly depends on arbitrary details about how your virtualenv was configured when you did the original install. But people care about this so much that you actually have prominent developers going around and filing bugs on packages complaining that they provide accurate metadata and that this is a bug, they should lie instead. E.g.: https://github.com/pydata/patsy/issues/5 I'm not sure I agree with this conclusion, but the users have spoken. AFAICT the only way to fix this problem and start getting packages with accurate metadata is for pip to gain a non-recursive upgrade mode. The bug for that is https://github.com/pypa/pip/issues/59 AFAICT from reading that thread, work on this has stalled out because of the following reasoning: 1) Everyone agrees that pip should have 'upgrade $PKG' and 'upgrade-all' commands, and 'install -U' should be deprecated/removed. 2) But implementing 'upgrade-all' is tricky and dangerous without first fixing pip's dependency resolver. 3) Therefore we can't add 'upgrade' or 'upgrade-all' until after we fix pip's dependency resolver. I feel like there's a certain logical gap between (2) and (3)... we could defer 'upgrade-all' until later but start supporting 'upgrade $PKG' right now, couldn't we? (It'd be implemented as the equivalent of 'pip install $PKG=$LATEST_VERSION', which is not scary looking at all.) -n -- Nathaniel J. Smith -- http://vorpus.org

2 1

Uploading stdeb built debs ?
by Stuart Axon 10 Oct '15

10 Oct '15

Hi, I built a package of using stdeb, any idea if these are acceptable to debian + ubuntu, and where to start / who to contact to get a package included in those distros ? S++

2 1

Help regarding Python Theano on IDLE 3.5<Immediate>
by vivek manchikanti 06 Oct '15

06 Oct '15

Hi I'm a totally new to Python and I wanted to learn it. I have installed Python IDLE 3.5 successfully. Now I want to use Theano library in that IDE and I'm confused how to use that. Just have some questions like Do I need to install additional packages to use Theano on 3.5 ? Please kindly help. Thank you ------------------------------- Vivekanand Manchikanti

2 1