From p at 2015.forums.dobrogost.net Thu Oct 1 16:41:22 2015 From: p at 2015.forums.dobrogost.net (Piotr Dobrogost) Date: Thu, 1 Oct 2015 16:41:22 +0200 Subject: [Distutils] =?utf-8?q?Unicode_in_the_=22long=5Fdescription=22_?= =?utf-8?q?=E2=80=93_is_this_possible_now=3F?= Message-ID: Hi! I'd like to know if anything changed since 2011 with regard to using unicode as a value of "long_description" argument to setuptools/distutils setup function? The problem was described by Carl Meyer in 2011 here https://github.com/pypa/virtualenv/issues/201#issuecomment-3145690 ? "The initial error is on reading from the file containing non-ASCII (docs/news.txt in our case). This part is easily fixable by switching from open(...) to codecs.open(..., encoding="UTF-8"). But then you get an encoding error on the output side. The usual way to fix this would be to explicitly encode the string to bytes in the desired encoding before output, e.g. long_description = long_description.encode("UTF-8"). We can't do that for long-description in setup.py, though, because distutils rfc822-escapes it internally, and that requires string-only methods (split). So long-description must be a string (unicode), it can't be encoded bytes. Attempts to change the default encoding of sys.stdout are pointless, as in the installation case long-description isn't written to stdout, it's written to a file." Btw, in what situation does long_description get written to sys.stdout? Regards, Piotr Dobrogost From guettliml at thomas-guettler.de Thu Oct 1 19:52:04 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 1 Oct 2015 19:52:04 +0200 Subject: [Distutils] README.rst vs DESCRIPTION.rst In-Reply-To: <5607C303.1040201@thomas-guettler.de> References: <5607C303.1040201@thomas-guettler.de> Message-ID: <560D72C4.5060004@thomas-guettler.de> one week has passed. I check the feedback. Most people want one README.rst. I see the file was already updated: https://github.com/pypa/sampleproject/commit/e7de7861963274595c5b3b0d6942e35ce5a733bc Issue solved. Thank you Carl Meyer! Regards, Thomas G?ttler Am 27.09.2015 um 12:20 schrieb Thomas G?ttler: > Zen of Python: > > There should be one-- and preferably only one --obvious way to do it. > > I would like to find a default for the description file of a python package. > > The sampleproject uses DESCRIPTION.rst > https://github.com/pypa/sampleproject/blob/master/setup.py > > But I guess it is more common to use README.rst. > For example django uses this file name. > > Any good reason to **not** use README.rst but a different > name like DESCRIPTION.rst? > > Of course anyone can use the name he wants. I just want an agreement > for the name to make life easier for newcomers. > > I will check this mail thread in a week or two and write > a pull request to https://github.com/pypa/sampleproject/blob/master/setup.py > if there is an agreement. > > If an agreement was found, which other documents should be updated? > > Regards, > Thomas G?ttler > > > > > -- http://www.thomas-guettler.de/ From njs at pobox.com Fri Oct 2 06:53:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 1 Oct 2015 21:53:39 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils Message-ID: Hi all, We realized that actually as far as we could tell, it wouldn't be that hard at this point to clean up how sdists work so that it would be possible to migrate away from distutils. So we wrote up a little draft proposal. The main question is, does this approach seem sound? -n --- PEP: ?? Title: Standard interface for interacting with source trees and source distributions Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith Thomas Kluyver Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: Discussions-To: Abstract ======== Distutils delenda est. Extended abstract ================= While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like autoconfiguration and usable build-time dependency declaration, (b) extending them is quirky, complicated, and fragile, (c) you are forced to use them anyway, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). We propose to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then the more the merrier. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit `_ or `bento `_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. Synopsis and rationale ====================== To limit the scope of our design, we adopt several principles. First, we distinguish between a *source tree* (e.g., a VCS checkout) and a *source distribution* (e.g., an official snapshot release like ``lxml-3.4.4.zip``). There isn't a whole lot that *source trees* can be assumed to have in common. About all you know is that they can -- via some more or less Rube-Goldbergian process -- produce one or more binary distributions. In particular, you *cannot* tell via simple static inspection: - What version number will be attached to the resulting packages (e.g. it might be determined programmatically by consulting VCS metadata -- I have here a build of numpy version "1.11.0.dev0+4a9ad17") - What build- or run-time dependencies are required (e.g. these may depend on arbitrarily complex configuration settings that are determined via a mix of manual settings and auto-probing) - Or even how many distinct binary distributions will be produced (e.g. a source distribution may always produce wheel A, but only produce wheel B when built on Unix-like systems). Therefore, when dealing with source trees, our goal is just to provide a standard UX for the core operations that are commonly performed on other people's packages; anything fancier and more developer-centric we leave at the discretion of individual package developers. So our source trees just provide some simple hooks to let a tool like ``pip``: - query for build dependencies - run a build, producing wheels as output - set up the current source tree so that it can be placed on ``sys.path`` in "develop mode" and that's it. We teach users that the standard way to install a package from a VCS checkout is now ``pip install .`` instead of ``python setup.py install``. (This is already a good idea anyway -- e.g., pip can do reliable uninstall / upgrades.) Next, we note that pretty much all the operations that you might want to perform on a *source distribution* are also operations that you might want to perform on a source tree, and via the same UX. The only thing you do with source distributions that you don't do with source trees is, well, distribute them. There's all kind of metadata you could imagine including in a source distribution, but each piece of metadata puts an increased burden on source distribution generation tools, and most operations will still have to work without this metadata. So we only include extra metadata in source distributions if it helps solve specific problems that are unique to distribution. If you want wheel-style metadata, get a wheel and look at it -- they're great and getting better. Therefore, our source distributions are basically just source trees + a mechanism for signing. Finally: we explicitly do *not* have any concept of "depending on a source distribution". As in other systems like Debian, dependencies are always phrased in terms of binary distributions (wheels), and when a user runs something like ``pip install ``, then the long-run plan is that and all its transitive dependencies should be available as wheels in a package index. But this is not yet realistic, so as a transitional / backwards-compatibility measure, we provide a simple mechanism for ``pip install `` to handle cases where is provided only as a source distribution. Source trees ============ We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code of ``distutils``, ``setuptools``, ``pip``, and other tools. A version 1-or-greater format source tree can be identified by the presence of a file ``_pypackage/_pypackage.cfg``. If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. This is necessary because we anticipate that version 1+ source trees may want to contain a ``setup.py`` file for backwards compatibility, e.g.:: #!/usr/bin/env python import sys print("Don't call setup.py directly!") print("Use 'pip install .' instead!") print("(You might have to upgrade pip first.)") sys.exit(1) In the current version of the specification, the one file ``_pypackage/_pypackage.cfg`` is where pretty much all the action is (though see below). The motivation for putting it into a subdirectory is that: - the way of all standards is that cruft accumulates over time, so this way we pre-emptively have a place to put it, - real-world projects often accumulate build system cruft as well, so we might as well provide one obvious place to put it too. Of course this then creates the possibility of collisions between standard files and user files, and trying to teach arbitrary users not to scatter files around willy-nilly never works, so we adopt the convention that names starting with an underscore are reserved for official use, and non-underscored names are available for idiosyncratic use by individual projects. The alternative would be to simply place the main configuration file at the top-level, create the subdirectory only when specifically needed (most trees won't need it), and let users worry about finding their own place for their cruft. Not sure which is the best approach. Plus we can have a nice bikeshed about the names in general (FIXME). _pypackage.cfg -------------- The ``_pypackage.cfg`` file contains various settings. Another good bike-shed topic is which file format to use for storing these (FIXME), but for purposes of this draft I'll write examples using `toml `_, because you'll instantly be able to understand the semantics, it has similar expressivity to JSON while being more human-friendly (e.g., it supports comments and multi-line strings), it's better-specified than ConfigParser, and it's much simpler than YAML. Rust's package manager uses toml for similar purposes. Here's an example ``_pypackage/_pypackage.cfg``:: # Version of the "pypackage format" that this file uses. # Optional. If not present then 1 is assumed. # All version changes indicate incompatible changes; backwards # compatible changes are indicated by just having extra stuff in # the file. version = 1 [build] # An inline requirements file. Optional. # (FIXME: I guess this means we need a spec for requirements files?) requirements = """ mybuildtool >= 2.1 special_windows_tool ; sys_platform == "win32" """ # The path to an out-of-line requirements file. Optional. requirements-file = "build-requirements.txt" # A hook that will be called to query build requirements. Optional. requirements-dynamic = "mybuildtool:get_requirements" # A hook that will be called to build wheels. Required. build-wheels = "mybuildtool:do_build" # A hook that will be called to do an in-place build (see below). # Optional. build-in-place = "mybuildtool:do_inplace_build" # The "x" namespace is reserved for third-party extensions. # To use x.foo you should own the name "foo" on pypi. [x.mybuildtool] spam = ["spam", "spam", "spam"] All paths are relative to the ``_pypackage/`` directory (so e.g. the build.requirements-file value above refers to a file named ``_pypackage/build-requirements.txt``). A *hook* is a Python object that is looked up using the same rules as traditional setuptools entry_points: a dotted module name, followed by a colon, followed by a dotted name that is looked up within that module. *Running a hook* means: first, find or create a python interpreter which is executing in the current venv, whose working directory is set to the ``_pypackage/`` directory, and which has the ``_pypackage/`` directory on ``sys.path``. Then, inside this interpreter, look up the hook object, and call it, with arguments as specified below. A build command like ``pip wheel `` performs the following steps: 1) Validate the ``_pypackage.cfg`` version number. 2) Create an empty virtualenv / venv, that matches the environment that the installer is targeting (e.g. if you want wheels for CPython 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows venv). 3) If the build.requirements key is present, then in this venv run the equivalent of ``pip install -r ``, using whatever index settings are currently in effect. 4) If the build.requirements-file key is present, then in this venv run the equivalent of ``pip install -r ``, using whatever index settings are currently in effect. 5) If the build.requirements-dynamic key is present, then in this venv run the hook with no arguments, capture its stdout, and pipe it into ``pip install -r -``, using whatever index settings are currently in effect. If the hook raises an exception, then abort the build with an error. Note: because these steps are performed in sequence, the build.requirements-dynamic hook is allowed to use packages that are listed in build.requirements or build.requirements-file. 6) In this venv, run the build.build-wheels hook. This should be a Python function which takes one argument. This argument is an arbitrary dictionary intended to contain user-specified configuration, specified via some install-tool-specific mechanism. The intention is that tools like ``pip`` should provide some way for users to specify key/value settings that will be passed in here, analogous to the legacy ``--install-option`` and ``--global-option`` arguments. To make it easier for packages to transition from version 0 to version 1 sdists, we suggest that ``pip`` and other tools that have such existing option-setting interfaces SHOULD map them to entries in this dictionary when -- e.g.:: pip --global-option=a --install-option=b --install-option=c could produce a dict like:: {"--global-option": ["a"], "--install-option": ["b", "c"]} The hook's return value is a list of pathnames relative to the scratch directory. Each entry names a wheel file created by this build. Errors are signaled by raising an exception. When performing an in-place build (e.g. for ``pip install -e .``), then the same steps are followed, except that instead of the build.build-wheels hook, we call the build.build-in-place hook, and instead of returning a list of wheel files, it returns the name of a directory that should be placed onto ``sys.path`` (usually this will be the source tree itself, but may not be, e.g. if a build system wants to enforce a rule where the source is always kept pristine then it could symlink the .py files into a build directory, place the extension modules and dist-info there, and return that). This directory must contain importable versions of the code in the source tree, along with appropriate .dist-info directories. (FIXME: in-place builds are useful but intrinsically kinda broken -- e.g. extensions / source / metadata can all easily get out of sync -- so while I think this paragraph provides a reasonable hack that preserves current functionality, maybe we should defer specifying them to until after we've thought through the issues more?) When working with source trees, build tools like ``pip`` are encouraged to cache and re-use virtualenvs for performance. Other contents of _pypackage/ ----------------------------- _RECORD, _RECORD.jws, _RECORD.p7s: see below. _x//: reserved for use by tools (e.g. _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) Source distributions ==================== A *source distribution* is a file in a well-known archive format such as zip or tar.gz, which contains a single directory, and this directory is a source tree (in the sense defined in the previous section). The ``_pypackage/`` directory in a source distribution SHOULD also contain a _RECORD file, as defined in PEP 427, and MAY also contain _RECORD.jws and/or _RECORD.p7s signature files. For official releases, source distributions SHOULD be named as ``-.``, and the directory they contain SHOULD be named ``-``, and building this source tree SHOULD produce a wheel named ``--.whl`` (though it may produce other wheels as well). (FIXME: maybe we should add that if you want your sdist on PyPI then you MUST include a proper _RECORD file and use the proper naming convention?) Integration tools like ``pip`` SHOULD take advantage of this convention by applying the following heuristic: when seeking a package , if no appropriate wheel can be found, but an sdist named -. is found, then: 1) build the sdist 2) add the resulting wheels to the package search space 3) retry the original operation This handles a variety of simple and complex cases -- for example, if we need a package 'foo', and we find foo-1.0.zip which builds foo.whl and bar.whl, and foo.whl depends on bar.whl, then everything will work out. There remain other cases that are not handled, e.g. if we start out searching for bar.whl we will never discover foo-1.0.zip. We take the perspective that this is nonetheless sufficient for a transitional heuristic, and anyone who runs into this problem should just upload wheels already. If this turns out to be inadequate in practice, then it will be addressed by future extensions. Examples ======== **Example 1:** While we assume that installation tools will have to continue supporting version 0 sdists for the indefinite future, it's a useful check to make sure that our new format can continue to support packages using distutils / setuptools as their build system. We assume that a future version ``pip`` will take its existing knowledge of distutils internals and expose them as the appropriate hooks, and then existing distutils / setuptools packages can be ported forward by using the following ``_pypackage/_pypackage.cfg``:: [build] requirements = """ pip >= whatever wheel """ # Applies monkeypatches, then does 'setup.py dist_info' and # extracts the setup_requires requirements-dynamic = "pip.pypackage_hooks:setup_requirements" # Applies monkeypatches, then does 'setup.py wheel' build-wheels = "pip.pypackage_hooks:build_wheels" # Applies monkeypatches, then does: # setup.py dist_info && setup.py build_ext -i build-in-place = "pip.pypackage_hooks:build_in_place" This is also useful for any other installation tools that may want to support version 0 sdists without having to implement bug-for-bug compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is present, they can use this as a default. **Example 2:** For packages using numpy.distutils. This is identical to the distutils / setuptools example above, except that numpy is moved into the list of static build requirements. Right now, most projects using numpy.distutils don't bother trying to declare this dependency, and instead simply error out if numpy is not already installed. This is because currently the only way to declare a build dependency is via the ``setup_requires`` argument to the ``setup`` function, and in this case the ``setup`` function is ``numpy.distutils.setup``, which... obviously doesn't work very well. Drop this ``_pypackage.cfg`` into an existing project like this and it will become robustly pip-installable with no further changes:: [build] requirements = """ numpy pip >= whatever wheel """ requirements-dynamic = "pip.pypackage_hooks:setup_requirements" build-wheels = "pip.pypackage_hooks:build_wheels" build-in-place = "pip.pypackage_hooks:build_in_place" **Example 3:** `flit `_ is a tool designed to make distributing simple packages simple, but it currently has no support for sdists, and for convenience includes its own installation code that's redundant with that in pip. These 4 lines of boilerplate make any flit-using source tree pip-installable, and lets flit get out of the package installation business:: [build] requirements = "flit" build-wheels = "flit.pypackage_hooks:build_wheels" build-in-place = "flit.pypackage_hooks:build_in_place" FAQ === **Why is it version 1 instead of version 2?** Because the legacy sdist format is barely a format at all, and to `remind us to keep things simple `_. **What about cross-compilation?** Standardizing an interface for cross-compilation seems premature given how complicated the configuration required can be, the lack of an existing de facto standard, and the authors of this PEP's inexperience with cross-compilation. This would be a great target for future extensions, though. In the mean time, there's no requirement that ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a project's build system -- packages that want to support cross-compilation can still do so, they'll just need to include a README explaining how to do it. **PEP 426 says that the new sdist format will support automatically creating policy-compliant .deb/.rpm packages. What happened to that?** Step 1: enhance the wheel format as necessary so that a wheel can be automatically converted into a policy-compliant .deb/.rpm package (see PEP 491). Step 2: make it possible to automatically turn sdists into wheels (this PEP). Step 3: we're done. **What about automatically running tests?** Arguably this is another thing that should be pushed off to wheel metadata instead of sdist metadata: it's good practice to include tests inside your built distribution so that end-users can test their install (and see above re: our focus here being on stuff that end-users want to do, not dedicated package developers), there are lots of packages that have to be built before they can be tested anyway (e.g. because of binary extensions), and in any case it's good practice to test against an installed version in order to make sure your install code works properly. But even if we do want this in sdist, then it's hardly urgent (e.g. there is no ``pip test`` that people will miss), so we defer that for a future extension to avoid blocking the core functionality. -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Fri Oct 2 09:12:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 00:12:19 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 1, 2015 10:45 PM, "Marcus Smith" wrote: > > Can you clarify the relationship to PEP426 metadata? > There's no standard for metadata in here other than what's required to run a build hook. Right -- the idea is that discretely installable binary packages (i.e. wheels) are a different sort of thing than a source tree / source distribution, and need different metadata. PEP 426 is then taken as a draft of the next metadata spec *for wheels*, and this is a draft "simplest thing that could possibly work" metadata spec for source trees / source distributions. The "synopsis and rationale" section provides more motivation for this approach. > Does that imply you would have each build tool enforce their own convention for where metadata is found? I think you mean: "does that imply that each build tool would have its own way of determining what metadata to attach to the wheels it generates?", and the answer to that is yes -- already right now distutils does it via kwargs passed to the setup() function, flit does it via entries in its flit.ini file (though for simplicity it might later move this to an extension section in _pypackage.cfg -- and note that this purely declarative style is only possible because flit is designed to make simple things easy and complicated things someone else's problem), and in general it might require arbitrary code to be executed. I expect that some conventions will probably get sorted out once we have experimental build systems competing with each other and figuring out what works and what doesn't, but distutils-sig has historically had poor luck with trying to design this kind of thing a priori. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 13:58:50 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 07:58:50 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > We realized that actually as far as we could tell, it wouldn't > be that > hard at this point to clean up how sdists work so that it would be > possible to migrate away from distutils. So we wrote up a little > draft > proposal. > > The main question is, does this approach seem sound? I've just read over your proposal, but I've also just woken up so I might be a little slow still! After reading what you have, I don't think that this proposal is the right way to go about improving sdists. The first thing that immediately stood out to me, is that it's recommending that downstream redistributors like Debian, Fedora, etc utilize Wheels instead of the sdist to build their packages from. However, that is not really going to fly with most (all?) of the downstream redistributors. Debian for instance has policy that requires the use of building all of it's packages from Source, not from anything else and Wheels are not a source package. While it can theoretically work for pure python packages, it quickly devolves into a mess when you factor in packages that have any C code what so ever. Overall, this feels more like a sidegrade than an upgrade. One major theme throughout of the PEP is that we're going to push to rely heavily on wheels as the primary format of installation. While that works well for things like Debian, I don't think it's going to work as wheel for us. If we were only distributing pure python packages, then yes absolutely, however given that we are not, we have to worry about ABI issues. Given that there is so many different environments that a particular package might be installed into, all with different ABIs we have to assume that installing from source is still going to be a primary path for end users to install and that we are never going to have a world where we can assume a Wheel in a repository. One of the problems with the current system, is that we have no mechanism by which to determine dependencies of a source distribution without downloading the file and executing some potentially untrusted code. This makes dependency resolution harder and much much slower than if we could read that information statically from a source distribution. This PEP doesn't offer anything in the way of solving this problem. To a similar tune, this PEP also doesn't make it possible to really get at any other metadata without executing software. This makes it pratically impossible to safely inspect an unknown or untrusted package to determine what it is and to get information about it. Right now PyPI relies on the uploading tool to send that information alongside of the file it is uploading, but honestly what it should be doing is extracting that information from within the file. This is sort of possible right now since distutils and setuptools both create a static metadata file within the source distribution, but we don't rely on that within PyPI because that information may or may not be accurate and may or may not exist. However the twine uploading tool *does* rely on that, and this PEP would break the ability for twine to upload a package without executing arbitrary code. Overall, I don't think that this really solves most of the foundational problems with the current format. Largely it feels that what it achieves is shuffling around some logic (you need to create a hook that you reference from within a .cfg file instead of creating a setuptools extension or so) but without fixing most of the problems. The largest benefit I see to switching to this right now is that it would enable us to have build time dependencies that were controlled by pip rather than installed implicitly via the execution of the setup.py. That doesn't feel like a big enough benefit to me to do a mass shakeup of what we recommend and tell people to do. Having people adjust and change and do something new requires effort, and we need something to justify that effort to other people and I don't think that this PEP has something we can really use to justify that effort. I *do* think that there is a core of some ideas here that are valuable, and in fact are similar to some ideas I've had. The main flaw I see here is that it doesn't really fix sdists, it takes a solution that would work for VCS checkouts and then reuses it for sdists. In my mind, the supported flow for package installation would be: ? ? VCS/Bare Directory -> Source Distribution -> Wheel This would (eventually) be the only path that was supported for installation but you could "enter" the path at any stage. For example, if there is a Wheel already available, then you jump right on at the end and just install that, if there is a sdist available then pip first builds it into a wheel and then installs that, etc. I think your PEP is something like what the VCS/Bare Directory to sdist tooling could look like, but I don't think it's what the sdist to wheel path should look like.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Fri Oct 2 14:38:42 2015 From: cournape at gmail.com (David Cournapeau) Date: Fri, 2 Oct 2015 13:38:42 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 12:58 PM, Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > > We realized that actually as far as we could tell, it wouldn't > > be that > > hard at this point to clean up how sdists work so that it would be > > possible to migrate away from distutils. So we wrote up a little > > draft > > proposal. > > > > The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might > be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. Debian for instance > has > policy that requires the use of building all of it's packages from Source, > not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a > mess > when you factor in packages that have any C code what so ever. > > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on > wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that > we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, > all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never > going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism > by > which to determine dependencies of a source distribution without > downloading > the file and executing some potentially untrusted code. This makes > dependency > resolution harder and much much slower than if we could read that > information > statically from a source distribution. This PEP doesn't offer anything in > the > way of solving this problem. > > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine > what > it is and to get information about it. Right now PyPI relies on the > uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from > within the > file. This is sort of possible right now since distutils and setuptools > both > create a static metadata file within the source distribution, but we don't > rely > on that within PyPI because that information may or may not be accurate > and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. > > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference > from > within a .cfg file instead of creating a setuptools extension or so) but > without fixing most of the problems. The largest benefit I see to > switching to > this right now is that it would enable us to have build time dependencies > that > were controlled by pip rather than installed implicitly via the execution > of > the setup.py. That doesn't feel like a big enough benefit to me to do a > mass > shakeup of what we recommend and tell people to do. Having people adjust > and > change and do something new requires effort, and we need something to > justify > that effort to other people and I don't think that this PEP has something > we > can really use to justify that effort. > > I *do* think that there is a core of some ideas here that are valuable, > and in > fact are similar to some ideas I've had. The main flaw I see here is that > it > doesn't really fix sdists, it takes a solution that would work for VCS > checkouts and then reuses it for sdists. In my mind, the supported flow for > package installation would be: > > VCS/Bare Directory -> Source Distribution -> Wheel > > This would (eventually) be the only path that was supported for > installation > but you could "enter" the path at any stage. For example, if there is a > Wheel > already available, then you jump right on at the end and just install > that, if > there is a sdist available then pip first builds it into a wheel and then > installs that, etc. > > I think your PEP is something like what the VCS/Bare Directory to sdist > tooling > could look like, but I don't think it's what the sdist to wheel path should > look like. > A major feature of the proposal is to allow alternative build/packaging tools. If that proposal is not acceptable in its current form, how would you envision interoperability between pip and new systems. For example, is it realistic to encode which commands and options a setup.py would need to support to be pip-installable (without the setup.py using distutils) ? David -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 15:02:53 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 09:02:53 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 8:38:45 AM, David Cournapeau (cournape at gmail.com) wrote: > > A major feature of the proposal is to allow alternative build/packaging > tools. > > If that proposal is not acceptable in its current form, how would you > envision interoperability between pip and new systems. For example, is it > realistic to encode which commands and options a setup.py would need to > support to be pip-installable (without the setup.py using distutils) ? > I think it depends if you mean in the short term or the long term and what exactly you envision as the scope of an alternative build/packaging tool. In the short term we have two real options that I can think of off the top of my head. One is as you mentioned, define the interface to setup.py that any build tool is expected to follow. However I?m not sure if that?s a great idea because you end up having a bootstrapping problem since pip won?t know that it needs X available to execute a particular setup.py (a common problem with numpy.distutils, as I know you?re aware!). We could do a minimal extension and add another defacto-ish standard of allowing pip and setuptools to process additional setup_requires like arguments from a setup.cfg to solve that problem though. The flip side to this is that since it involves new capabilities in pip/setuptools/any other installer is that it you?ll have several years until you can depend on setup.cfg based setup_requires from being able to be depended on. Another short term option is to simply say that using something that isn?t distutils/setuptools isn?t supported, but that if you want to do something else, you should rely on setuptools ability to be extended. This means you?d need a very minimal ``setup.py`` [1], but that you could then do pretty much anything you wanted, since setuptools lets you override a good bit of it?s own logic, pbr uses this to have completely static metadata [2]. This option would just work and require no changes in pip or setuptools so then you could start depending on it right away and it would just work, the downside of course is that you?re tied to extending setuptools and the APIs it provides rather than being able to more freely do what you want, but the ability to do that in setuptools is pretty extensive. Longer term, I think the answer is sdist 2.0 which has proper metadata inside of it (name, version, dependencies, etc) but which also includes a hook like this PEP has to specify the build system that should be used to build a wheel out of this source distribution. [1] Example for pbr:?https://github.com/testing-cabal/mock/blob/master/setup.py [2]?https://github.com/testing-cabal/mock/blob/master/setup.cfg ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Fri Oct 2 07:45:51 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 1 Oct 2015 22:45:51 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Can you clarify the relationship to PEP426 metadata? There's no standard for metadata in here other than what's required to run a build hook. Does that imply you would have each build tool enforce their own convention for where metadata is found? On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith wrote: > Hi all, > > We realized that actually as far as we could tell, it wouldn't be that > hard at this point to clean up how sdists work so that it would be > possible to migrate away from distutils. So we wrote up a little draft > proposal. > > The main question is, does this approach seem sound? > > -n > > --- > > PEP: ?? > Title: Standard interface for interacting with source trees > and source distributions > Version: $Revision$ > Last-Modified: $Date$ > Author: Nathaniel J. Smith > Thomas Kluyver > Status: Draft > Type: Standards-Track > Content-Type: text/x-rst > Created: 30-Sep-2015 > Post-History: > Discussions-To: > > Abstract > ======== > > Distutils delenda est. > > > Extended abstract > ================= > > While ``distutils`` / ``setuptools`` have taken us a long way, they > suffer from three serious problems: (a) they're missing important > features like autoconfiguration and usable build-time dependency > declaration, (b) extending them is quirky, complicated, and fragile, > (c) you are forced to use them anyway, because they provide the > standard interface for installing python packages expected by both > users and installation tools like ``pip``. > > Previous efforts (e.g. distutils2 or setuptools itself) have attempted > to solve problems (a) and/or (b). We propose to solve (c). > > The goal of this PEP is get distutils-sig out of the business of being > a gatekeeper for Python build systems. If you want to use distutils, > great; if you want to use something else, then the more the merrier. > The difficulty of interfacing with distutils means that there aren't > many such systems right now, but to give a sense of what we're > thinking about see `flit `_ or > `bento > `_. Fortunately, wheels have now > solved many of the hard problems here -- e.g. it's no longer necessary > that a build system also know about every possible installation > configuration -- so pretty much all we really need from a build system > is that it have some way to spit out standard-compliant wheels. > > We therefore propose a new, relatively minimal interface for > installation tools like ``pip`` to interact with package source trees > and source distributions. > > > Synopsis and rationale > ====================== > > To limit the scope of our design, we adopt several principles. > > First, we distinguish between a *source tree* (e.g., a VCS checkout) > and a *source distribution* (e.g., an official snapshot release like > ``lxml-3.4.4.zip``). > > There isn't a whole lot that *source trees* can be assumed to have in > common. About all you know is that they can -- via some more or less > Rube-Goldbergian process -- produce one or more binary distributions. > In particular, you *cannot* tell via simple static inspection: > - What version number will be attached to the resulting packages (e.g. > it might be determined programmatically by consulting VCS metadata -- > I have here a build of numpy version "1.11.0.dev0+4a9ad17") > - What build- or run-time dependencies are required (e.g. these may > depend on arbitrarily complex configuration settings that are > determined via a mix of manual settings and auto-probing) > - Or even how many distinct binary distributions will be produced > (e.g. a source distribution may always produce wheel A, but only > produce wheel B when built on Unix-like systems). > > Therefore, when dealing with source trees, our goal is just to provide > a standard UX for the core operations that are commonly performed on > other people's packages; anything fancier and more developer-centric > we leave at the discretion of individual package developers. So our > source trees just provide some simple hooks to let a tool like > ``pip``: > > - query for build dependencies > - run a build, producing wheels as output > - set up the current source tree so that it can be placed on > ``sys.path`` in "develop mode" > > and that's it. We teach users that the standard way to install a > package from a VCS checkout is now ``pip install .`` instead of > ``python setup.py install``. (This is already a good idea anyway -- > e.g., pip can do reliable uninstall / upgrades.) > > Next, we note that pretty much all the operations that you might want > to perform on a *source distribution* are also operations that you > might want to perform on a source tree, and via the same UX. The only > thing you do with source distributions that you don't do with source > trees is, well, distribute them. There's all kind of metadata you > could imagine including in a source distribution, but each piece of > metadata puts an increased burden on source distribution generation > tools, and most operations will still have to work without this > metadata. So we only include extra metadata in source distributions if > it helps solve specific problems that are unique to distribution. If > you want wheel-style metadata, get a wheel and look at it -- they're > great and getting better. > > Therefore, our source distributions are basically just source trees + > a mechanism for signing. > > Finally: we explicitly do *not* have any concept of "depending on a > source distribution". As in other systems like Debian, dependencies > are always phrased in terms of binary distributions (wheels), and when > a user runs something like ``pip install ``, then the > long-run plan is that and all its transitive dependencies > should be available as wheels in a package index. But this is not yet > realistic, so as a transitional / backwards-compatibility measure, we > provide a simple mechanism for ``pip install `` to handle > cases where is provided only as a source distribution. > > > Source trees > ============ > > We retroactively declare the legacy source tree format involving > ``setup.py`` to be "version 0". We don't try to specify it further; > its de facto specification is encoded in the source code of > ``distutils``, ``setuptools``, ``pip``, and other tools. > > A version 1-or-greater format source tree can be identified by the > presence of a file ``_pypackage/_pypackage.cfg``. > > If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, > then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. > This is necessary because we anticipate that version 1+ source trees > may want to contain a ``setup.py`` file for backwards compatibility, > e.g.:: > > #!/usr/bin/env python > import sys > print("Don't call setup.py directly!") > print("Use 'pip install .' instead!") > print("(You might have to upgrade pip first.)") > sys.exit(1) > > In the current version of the specification, the one file > ``_pypackage/_pypackage.cfg`` is where pretty much all the action is > (though see below). The motivation for putting it into a subdirectory > is that: > - the way of all standards is that cruft accumulates over time, so > this way we pre-emptively have a place to put it, > - real-world projects often accumulate build system cruft as well, so > we might as well provide one obvious place to put it too. > > Of course this then creates the possibility of collisions between > standard files and user files, and trying to teach arbitrary users not > to scatter files around willy-nilly never works, so we adopt the > convention that names starting with an underscore are reserved for > official use, and non-underscored names are available for > idiosyncratic use by individual projects. > > The alternative would be to simply place the main configuration file > at the top-level, create the subdirectory only when specifically > needed (most trees won't need it), and let users worry about finding > their own place for their cruft. Not sure which is the best approach. > Plus we can have a nice bikeshed about the names in general (FIXME). > > _pypackage.cfg > -------------- > > The ``_pypackage.cfg`` file contains various settings. Another good > bike-shed topic is which file format to use for storing these (FIXME), > but for purposes of this draft I'll write examples using `toml > `_, because you'll instantly be > able to understand the semantics, it has similar expressivity to JSON > while being more human-friendly (e.g., it supports comments and > multi-line strings), it's better-specified than ConfigParser, and it's > much simpler than YAML. Rust's package manager uses toml for similar > purposes. > > Here's an example ``_pypackage/_pypackage.cfg``:: > > # Version of the "pypackage format" that this file uses. > # Optional. If not present then 1 is assumed. > # All version changes indicate incompatible changes; backwards > # compatible changes are indicated by just having extra stuff in > # the file. > version = 1 > > [build] > # An inline requirements file. Optional. > # (FIXME: I guess this means we need a spec for requirements files?) > requirements = """ > mybuildtool >= 2.1 > special_windows_tool ; sys_platform == "win32" > """ > # The path to an out-of-line requirements file. Optional. > requirements-file = "build-requirements.txt" > # A hook that will be called to query build requirements. Optional. > requirements-dynamic = "mybuildtool:get_requirements" > > # A hook that will be called to build wheels. Required. > build-wheels = "mybuildtool:do_build" > > # A hook that will be called to do an in-place build (see below). > # Optional. > build-in-place = "mybuildtool:do_inplace_build" > > # The "x" namespace is reserved for third-party extensions. > # To use x.foo you should own the name "foo" on pypi. > [x.mybuildtool] > spam = ["spam", "spam", "spam"] > > All paths are relative to the ``_pypackage/`` directory (so e.g. the > build.requirements-file value above refers to a file named > ``_pypackage/build-requirements.txt``). > > A *hook* is a Python object that is looked up using the same rules as > traditional setuptools entry_points: a dotted module name, followed by > a colon, followed by a dotted name that is looked up within that > module. *Running a hook* means: first, find or create a python > interpreter which is executing in the current venv, whose working > directory is set to the ``_pypackage/`` directory, and which has the > ``_pypackage/`` directory on ``sys.path``. Then, inside this > interpreter, look up the hook object, and call it, with arguments as > specified below. > > A build command like ``pip wheel `` performs the following > steps: > > 1) Validate the ``_pypackage.cfg`` version number. > > 2) Create an empty virtualenv / venv, that matches the environment > that the installer is targeting (e.g. if you want wheels for CPython > 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows > venv). > > 3) If the build.requirements key is present, then in this venv run the > equivalent of ``pip install -r ``, using > whatever index settings are currently in effect. > > 4) If the build.requirements-file key is present, then in this venv > run the equivalent of ``pip install -r ``, using > whatever index settings are currently in effect. > > 5) If the build.requirements-dynamic key is present, then in this venv > run the hook with no arguments, capture its stdout, and pipe it into > ``pip install -r -``, using whatever index settings are currently in > effect. If the hook raises an exception, then abort the build with an > error. > > Note: because these steps are performed in sequence, the > build.requirements-dynamic hook is allowed to use packages that are > listed in build.requirements or build.requirements-file. > > 6) In this venv, run the build.build-wheels hook. This should be a > Python function which takes one argument. > > This argument is an arbitrary dictionary intended to contain > user-specified configuration, specified via some install-tool-specific > mechanism. The intention is that tools like ``pip`` should provide > some way for users to specify key/value settings that will be passed > in here, analogous to the legacy ``--install-option`` and > ``--global-option`` arguments. > > To make it easier for packages to transition from version 0 to > version 1 sdists, we suggest that ``pip`` and other tools that have > such existing option-setting interfaces SHOULD map them to entries in > this dictionary when -- e.g.:: > > pip --global-option=a --install-option=b --install-option=c > > could produce a dict like:: > > {"--global-option": ["a"], "--install-option": ["b", "c"]} > > The hook's return value is a list of pathnames relative to the > scratch directory. Each entry names a wheel file created by this > build. > > Errors are signaled by raising an exception. > > When performing an in-place build (e.g. for ``pip install -e .``), > then the same steps are followed, except that instead of the > build.build-wheels hook, we call the build.build-in-place hook, and > instead of returning a list of wheel files, it returns the name of a > directory that should be placed onto ``sys.path`` (usually this will > be the source tree itself, but may not be, e.g. if a build system > wants to enforce a rule where the source is always kept pristine then > it could symlink the .py files into a build directory, place the > extension modules and dist-info there, and return that). This > directory must contain importable versions of the code in the source > tree, along with appropriate .dist-info directories. > > (FIXME: in-place builds are useful but intrinsically kinda broken -- > e.g. extensions / source / metadata can all easily get out of sync -- > so while I think this paragraph provides a reasonable hack that > preserves current functionality, maybe we should defer specifying them > to until after we've thought through the issues more?) > > When working with source trees, build tools like ``pip`` are > encouraged to cache and re-use virtualenvs for performance. > > > Other contents of _pypackage/ > ----------------------------- > > _RECORD, _RECORD.jws, _RECORD.p7s: see below. > > _x//: reserved for use by tools (e.g. > _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) > > > Source distributions > ==================== > > A *source distribution* is a file in a well-known archive format such > as zip or tar.gz, which contains a single directory, and this > directory is a source tree (in the sense defined in the previous > section). > > The ``_pypackage/`` directory in a source distribution SHOULD also > contain a _RECORD file, as defined in PEP 427, and MAY also contain > _RECORD.jws and/or _RECORD.p7s signature files. > > For official releases, source distributions SHOULD be named as > ``-.``, and the directory they contain SHOULD > be named ``-``, and building this source tree SHOULD > produce a wheel named ``--.whl`` > (though it may produce other wheels as well). > > (FIXME: maybe we should add that if you want your sdist on PyPI then > you MUST include a proper _RECORD file and use the proper naming > convention?) > > Integration tools like ``pip`` SHOULD take advantage of this > convention by applying the following heuristic: when seeking a package > , if no appropriate wheel can be found, but an sdist named > -. is found, then: > > 1) build the sdist > 2) add the resulting wheels to the package search space > 3) retry the original operation > > This handles a variety of simple and complex cases -- for example, if > we need a package 'foo', and we find foo-1.0.zip which builds foo.whl > and bar.whl, and foo.whl depends on bar.whl, then everything will work > out. There remain other cases that are not handled, e.g. if we start > out searching for bar.whl we will never discover foo-1.0.zip. We take > the perspective that this is nonetheless sufficient for a transitional > heuristic, and anyone who runs into this problem should just upload > wheels already. If this turns out to be inadequate in practice, then > it will be addressed by future extensions. > > > Examples > ======== > > **Example 1:** While we assume that installation tools will have to > continue supporting version 0 sdists for the indefinite future, it's a > useful check to make sure that our new format can continue to support > packages using distutils / setuptools as their build system. We assume > that a future version ``pip`` will take its existing knowledge of > distutils internals and expose them as the appropriate hooks, and then > existing distutils / setuptools packages can be ported forward by > using the following ``_pypackage/_pypackage.cfg``:: > > [build] > requirements = """ > pip >= whatever > wheel > """ > # Applies monkeypatches, then does 'setup.py dist_info' and > # extracts the setup_requires > requirements-dynamic = "pip.pypackage_hooks:setup_requirements" > # Applies monkeypatches, then does 'setup.py wheel' > build-wheels = "pip.pypackage_hooks:build_wheels" > # Applies monkeypatches, then does: > # setup.py dist_info && setup.py build_ext -i > build-in-place = "pip.pypackage_hooks:build_in_place" > > This is also useful for any other installation tools that may want to > support version 0 sdists without having to implement bug-for-bug > compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is > present, they can use this as a default. > > **Example 2:** For packages using numpy.distutils. This is identical > to the distutils / setuptools example above, except that numpy is > moved into the list of static build requirements. Right now, most > projects using numpy.distutils don't bother trying to declare this > dependency, and instead simply error out if numpy is not already > installed. This is because currently the only way to declare a build > dependency is via the ``setup_requires`` argument to the ``setup`` > function, and in this case the ``setup`` function is > ``numpy.distutils.setup``, which... obviously doesn't work very well. > Drop this ``_pypackage.cfg`` into an existing project like this and it > will become robustly pip-installable with no further changes:: > > [build] > requirements = """ > numpy > pip >= whatever > wheel > """ > requirements-dynamic = "pip.pypackage_hooks:setup_requirements" > build-wheels = "pip.pypackage_hooks:build_wheels" > build-in-place = "pip.pypackage_hooks:build_in_place" > > **Example 3:** `flit `_ is a tool > designed to make distributing simple packages simple, but it currently > has no support for sdists, and for convenience includes its own > installation code that's redundant with that in pip. These 4 lines of > boilerplate make any flit-using source tree pip-installable, and lets > flit get out of the package installation business:: > > [build] > requirements = "flit" > build-wheels = "flit.pypackage_hooks:build_wheels" > build-in-place = "flit.pypackage_hooks:build_in_place" > > > FAQ > === > > **Why is it version 1 instead of version 2?** Because the legacy sdist > format is barely a format at all, and to `remind us to keep things > simple < > https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect > >`_. > > **What about cross-compilation?** Standardizing an interface for > cross-compilation seems premature given how complicated the > configuration required can be, the lack of an existing de facto > standard, and the authors of this PEP's inexperience with > cross-compilation. This would be a great target for future extensions, > though. In the mean time, there's no requirement that > ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a > project's build system -- packages that want to support > cross-compilation can still do so, they'll just need to include a > README explaining how to do it. > > **PEP 426 says that the new sdist format will support automatically > creating policy-compliant .deb/.rpm packages. What happened to that?** > Step 1: enhance the wheel format as necessary so that a wheel can be > automatically converted into a policy-compliant .deb/.rpm package (see > PEP 491). Step 2: make it possible to automatically turn sdists into > wheels (this PEP). Step 3: we're done. > > **What about automatically running tests?** Arguably this is another > thing that should be pushed off to wheel metadata instead of sdist > metadata: it's good practice to include tests inside your built > distribution so that end-users can test their install (and see above > re: our focus here being on stuff that end-users want to do, not > dedicated package developers), there are lots of packages that have to > be built before they can be tested anyway (e.g. because of binary > extensions), and in any case it's good practice to test against an > installed version in order to make sure your install code works > properly. But even if we do want this in sdist, then it's hardly > urgent (e.g. there is no ``pip test`` that people will miss), so we > defer that for a future extension to avoid blocking the core > functionality. > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Oct 2 16:17:52 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 14:17:52 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Thank you for your work on this. We have to kill distutils to make progress in packaging. On Fri, Oct 2, 2015 at 10:12 AM Daniel Holth wrote: > One way to do sdist 2.0 would be to have the package-1.0.dist-info > directory in there (most sdists contain setuptools metadata) and to have a > flag static-metadata=1 in setup.cfg asserting that setup.py [if present] > does not alter the list of dependencies. > > In the old MEBS design the package could suggest a build system, but pip > would invoke a list of build plugins to inspect the directory and return > True if they were able to build the package. This would allow for ignoring > the package's suggested build system. Instead of defining a command line > interface for setup.py MEBS would define a set of methods on the build > plugin. > > I thought Robert Collins had a working setup-requires implementation > already? I have a worse but backwards compatible one too at > https://bitbucket.org/dholth/setup-requires/src/tip/setup.py > > On Fri, Oct 2, 2015 at 9:42 AM Marcus Smith wrote: > >> Can you clarify the relationship to PEP426 metadata? >> There's no standard for metadata in here other than what's required to >> run a build hook. >> Does that imply you would have each build tool enforce their own >> convention for where metadata is found? >> >> On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith wrote: >> >>> Hi all, >>> >>> We realized that actually as far as we could tell, it wouldn't be that >>> hard at this point to clean up how sdists work so that it would be >>> possible to migrate away from distutils. So we wrote up a little draft >>> proposal. >>> >>> The main question is, does this approach seem sound? >>> >>> -n >>> >>> --- >>> >>> PEP: ?? >>> Title: Standard interface for interacting with source trees >>> and source distributions >>> Version: $Revision$ >>> Last-Modified: $Date$ >>> Author: Nathaniel J. Smith >>> Thomas Kluyver >>> Status: Draft >>> Type: Standards-Track >>> Content-Type: text/x-rst >>> Created: 30-Sep-2015 >>> Post-History: >>> Discussions-To: >>> >>> Abstract >>> ======== >>> >>> Distutils delenda est. >>> >>> >>> Extended abstract >>> ================= >>> >>> While ``distutils`` / ``setuptools`` have taken us a long way, they >>> suffer from three serious problems: (a) they're missing important >>> features like autoconfiguration and usable build-time dependency >>> declaration, (b) extending them is quirky, complicated, and fragile, >>> (c) you are forced to use them anyway, because they provide the >>> standard interface for installing python packages expected by both >>> users and installation tools like ``pip``. >>> >>> Previous efforts (e.g. distutils2 or setuptools itself) have attempted >>> to solve problems (a) and/or (b). We propose to solve (c). >>> >>> The goal of this PEP is get distutils-sig out of the business of being >>> a gatekeeper for Python build systems. If you want to use distutils, >>> great; if you want to use something else, then the more the merrier. >>> The difficulty of interfacing with distutils means that there aren't >>> many such systems right now, but to give a sense of what we're >>> thinking about see `flit `_ or >>> `bento >>> `_. Fortunately, wheels have now >>> solved many of the hard problems here -- e.g. it's no longer necessary >>> that a build system also know about every possible installation >>> configuration -- so pretty much all we really need from a build system >>> is that it have some way to spit out standard-compliant wheels. >>> >>> We therefore propose a new, relatively minimal interface for >>> installation tools like ``pip`` to interact with package source trees >>> and source distributions. >>> >>> >>> Synopsis and rationale >>> ====================== >>> >>> To limit the scope of our design, we adopt several principles. >>> >>> First, we distinguish between a *source tree* (e.g., a VCS checkout) >>> and a *source distribution* (e.g., an official snapshot release like >>> ``lxml-3.4.4.zip``). >>> >>> There isn't a whole lot that *source trees* can be assumed to have in >>> common. About all you know is that they can -- via some more or less >>> Rube-Goldbergian process -- produce one or more binary distributions. >>> In particular, you *cannot* tell via simple static inspection: >>> - What version number will be attached to the resulting packages (e.g. >>> it might be determined programmatically by consulting VCS metadata -- >>> I have here a build of numpy version "1.11.0.dev0+4a9ad17") >>> - What build- or run-time dependencies are required (e.g. these may >>> depend on arbitrarily complex configuration settings that are >>> determined via a mix of manual settings and auto-probing) >>> - Or even how many distinct binary distributions will be produced >>> (e.g. a source distribution may always produce wheel A, but only >>> produce wheel B when built on Unix-like systems). >>> >>> Therefore, when dealing with source trees, our goal is just to provide >>> a standard UX for the core operations that are commonly performed on >>> other people's packages; anything fancier and more developer-centric >>> we leave at the discretion of individual package developers. So our >>> source trees just provide some simple hooks to let a tool like >>> ``pip``: >>> >>> - query for build dependencies >>> - run a build, producing wheels as output >>> - set up the current source tree so that it can be placed on >>> ``sys.path`` in "develop mode" >>> >>> and that's it. We teach users that the standard way to install a >>> package from a VCS checkout is now ``pip install .`` instead of >>> ``python setup.py install``. (This is already a good idea anyway -- >>> e.g., pip can do reliable uninstall / upgrades.) >>> >>> Next, we note that pretty much all the operations that you might want >>> to perform on a *source distribution* are also operations that you >>> might want to perform on a source tree, and via the same UX. The only >>> thing you do with source distributions that you don't do with source >>> trees is, well, distribute them. There's all kind of metadata you >>> could imagine including in a source distribution, but each piece of >>> metadata puts an increased burden on source distribution generation >>> tools, and most operations will still have to work without this >>> metadata. So we only include extra metadata in source distributions if >>> it helps solve specific problems that are unique to distribution. If >>> you want wheel-style metadata, get a wheel and look at it -- they're >>> great and getting better. >>> >>> Therefore, our source distributions are basically just source trees + >>> a mechanism for signing. >>> >>> Finally: we explicitly do *not* have any concept of "depending on a >>> source distribution". As in other systems like Debian, dependencies >>> are always phrased in terms of binary distributions (wheels), and when >>> a user runs something like ``pip install ``, then the >>> long-run plan is that and all its transitive dependencies >>> should be available as wheels in a package index. But this is not yet >>> realistic, so as a transitional / backwards-compatibility measure, we >>> provide a simple mechanism for ``pip install `` to handle >>> cases where is provided only as a source distribution. >>> >>> >>> Source trees >>> ============ >>> >>> We retroactively declare the legacy source tree format involving >>> ``setup.py`` to be "version 0". We don't try to specify it further; >>> its de facto specification is encoded in the source code of >>> ``distutils``, ``setuptools``, ``pip``, and other tools. >>> >>> A version 1-or-greater format source tree can be identified by the >>> presence of a file ``_pypackage/_pypackage.cfg``. >>> >>> If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, >>> then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. >>> This is necessary because we anticipate that version 1+ source trees >>> may want to contain a ``setup.py`` file for backwards compatibility, >>> e.g.:: >>> >>> #!/usr/bin/env python >>> import sys >>> print("Don't call setup.py directly!") >>> print("Use 'pip install .' instead!") >>> print("(You might have to upgrade pip first.)") >>> sys.exit(1) >>> >>> In the current version of the specification, the one file >>> ``_pypackage/_pypackage.cfg`` is where pretty much all the action is >>> (though see below). The motivation for putting it into a subdirectory >>> is that: >>> - the way of all standards is that cruft accumulates over time, so >>> this way we pre-emptively have a place to put it, >>> - real-world projects often accumulate build system cruft as well, so >>> we might as well provide one obvious place to put it too. >>> >>> Of course this then creates the possibility of collisions between >>> standard files and user files, and trying to teach arbitrary users not >>> to scatter files around willy-nilly never works, so we adopt the >>> convention that names starting with an underscore are reserved for >>> official use, and non-underscored names are available for >>> idiosyncratic use by individual projects. >>> >>> The alternative would be to simply place the main configuration file >>> at the top-level, create the subdirectory only when specifically >>> needed (most trees won't need it), and let users worry about finding >>> their own place for their cruft. Not sure which is the best approach. >>> Plus we can have a nice bikeshed about the names in general (FIXME). >>> >>> _pypackage.cfg >>> -------------- >>> >>> The ``_pypackage.cfg`` file contains various settings. Another good >>> bike-shed topic is which file format to use for storing these (FIXME), >>> but for purposes of this draft I'll write examples using `toml >>> `_, because you'll instantly be >>> able to understand the semantics, it has similar expressivity to JSON >>> while being more human-friendly (e.g., it supports comments and >>> multi-line strings), it's better-specified than ConfigParser, and it's >>> much simpler than YAML. Rust's package manager uses toml for similar >>> purposes. >>> >>> Here's an example ``_pypackage/_pypackage.cfg``:: >>> >>> # Version of the "pypackage format" that this file uses. >>> # Optional. If not present then 1 is assumed. >>> # All version changes indicate incompatible changes; backwards >>> # compatible changes are indicated by just having extra stuff in >>> # the file. >>> version = 1 >>> >>> [build] >>> # An inline requirements file. Optional. >>> # (FIXME: I guess this means we need a spec for requirements files?) >>> requirements = """ >>> mybuildtool >= 2.1 >>> special_windows_tool ; sys_platform == "win32" >>> """ >>> # The path to an out-of-line requirements file. Optional. >>> requirements-file = "build-requirements.txt" >>> # A hook that will be called to query build requirements. Optional. >>> requirements-dynamic = "mybuildtool:get_requirements" >>> >>> # A hook that will be called to build wheels. Required. >>> build-wheels = "mybuildtool:do_build" >>> >>> # A hook that will be called to do an in-place build (see below). >>> # Optional. >>> build-in-place = "mybuildtool:do_inplace_build" >>> >>> # The "x" namespace is reserved for third-party extensions. >>> # To use x.foo you should own the name "foo" on pypi. >>> [x.mybuildtool] >>> spam = ["spam", "spam", "spam"] >>> >>> All paths are relative to the ``_pypackage/`` directory (so e.g. the >>> build.requirements-file value above refers to a file named >>> ``_pypackage/build-requirements.txt``). >>> >>> A *hook* is a Python object that is looked up using the same rules as >>> traditional setuptools entry_points: a dotted module name, followed by >>> a colon, followed by a dotted name that is looked up within that >>> module. *Running a hook* means: first, find or create a python >>> interpreter which is executing in the current venv, whose working >>> directory is set to the ``_pypackage/`` directory, and which has the >>> ``_pypackage/`` directory on ``sys.path``. Then, inside this >>> interpreter, look up the hook object, and call it, with arguments as >>> specified below. >>> >>> A build command like ``pip wheel `` performs the following >>> steps: >>> >>> 1) Validate the ``_pypackage.cfg`` version number. >>> >>> 2) Create an empty virtualenv / venv, that matches the environment >>> that the installer is targeting (e.g. if you want wheels for CPython >>> 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows >>> venv). >>> >>> 3) If the build.requirements key is present, then in this venv run the >>> equivalent of ``pip install -r ``, using >>> whatever index settings are currently in effect. >>> >>> 4) If the build.requirements-file key is present, then in this venv >>> run the equivalent of ``pip install -r ``, using >>> whatever index settings are currently in effect. >>> >>> 5) If the build.requirements-dynamic key is present, then in this venv >>> run the hook with no arguments, capture its stdout, and pipe it into >>> ``pip install -r -``, using whatever index settings are currently in >>> effect. If the hook raises an exception, then abort the build with an >>> error. >>> >>> Note: because these steps are performed in sequence, the >>> build.requirements-dynamic hook is allowed to use packages that are >>> listed in build.requirements or build.requirements-file. >>> >>> 6) In this venv, run the build.build-wheels hook. This should be a >>> Python function which takes one argument. >>> >>> This argument is an arbitrary dictionary intended to contain >>> user-specified configuration, specified via some install-tool-specific >>> mechanism. The intention is that tools like ``pip`` should provide >>> some way for users to specify key/value settings that will be passed >>> in here, analogous to the legacy ``--install-option`` and >>> ``--global-option`` arguments. >>> >>> To make it easier for packages to transition from version 0 to >>> version 1 sdists, we suggest that ``pip`` and other tools that have >>> such existing option-setting interfaces SHOULD map them to entries in >>> this dictionary when -- e.g.:: >>> >>> pip --global-option=a --install-option=b --install-option=c >>> >>> could produce a dict like:: >>> >>> {"--global-option": ["a"], "--install-option": ["b", "c"]} >>> >>> The hook's return value is a list of pathnames relative to the >>> scratch directory. Each entry names a wheel file created by this >>> build. >>> >>> Errors are signaled by raising an exception. >>> >>> When performing an in-place build (e.g. for ``pip install -e .``), >>> then the same steps are followed, except that instead of the >>> build.build-wheels hook, we call the build.build-in-place hook, and >>> instead of returning a list of wheel files, it returns the name of a >>> directory that should be placed onto ``sys.path`` (usually this will >>> be the source tree itself, but may not be, e.g. if a build system >>> wants to enforce a rule where the source is always kept pristine then >>> it could symlink the .py files into a build directory, place the >>> extension modules and dist-info there, and return that). This >>> directory must contain importable versions of the code in the source >>> tree, along with appropriate .dist-info directories. >>> >>> (FIXME: in-place builds are useful but intrinsically kinda broken -- >>> e.g. extensions / source / metadata can all easily get out of sync -- >>> so while I think this paragraph provides a reasonable hack that >>> preserves current functionality, maybe we should defer specifying them >>> to until after we've thought through the issues more?) >>> >>> When working with source trees, build tools like ``pip`` are >>> encouraged to cache and re-use virtualenvs for performance. >>> >>> >>> Other contents of _pypackage/ >>> ----------------------------- >>> >>> _RECORD, _RECORD.jws, _RECORD.p7s: see below. >>> >>> _x//: reserved for use by tools (e.g. >>> _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) >>> >>> >>> Source distributions >>> ==================== >>> >>> A *source distribution* is a file in a well-known archive format such >>> as zip or tar.gz, which contains a single directory, and this >>> directory is a source tree (in the sense defined in the previous >>> section). >>> >>> The ``_pypackage/`` directory in a source distribution SHOULD also >>> contain a _RECORD file, as defined in PEP 427, and MAY also contain >>> _RECORD.jws and/or _RECORD.p7s signature files. >>> >>> For official releases, source distributions SHOULD be named as >>> ``-.``, and the directory they contain SHOULD >>> be named ``-``, and building this source tree SHOULD >>> produce a wheel named ``--.whl`` >>> (though it may produce other wheels as well). >>> >>> (FIXME: maybe we should add that if you want your sdist on PyPI then >>> you MUST include a proper _RECORD file and use the proper naming >>> convention?) >>> >>> Integration tools like ``pip`` SHOULD take advantage of this >>> convention by applying the following heuristic: when seeking a package >>> , if no appropriate wheel can be found, but an sdist named >>> -. is found, then: >>> >>> 1) build the sdist >>> 2) add the resulting wheels to the package search space >>> 3) retry the original operation >>> >>> This handles a variety of simple and complex cases -- for example, if >>> we need a package 'foo', and we find foo-1.0.zip which builds foo.whl >>> and bar.whl, and foo.whl depends on bar.whl, then everything will work >>> out. There remain other cases that are not handled, e.g. if we start >>> out searching for bar.whl we will never discover foo-1.0.zip. We take >>> the perspective that this is nonetheless sufficient for a transitional >>> heuristic, and anyone who runs into this problem should just upload >>> wheels already. If this turns out to be inadequate in practice, then >>> it will be addressed by future extensions. >>> >>> >>> Examples >>> ======== >>> >>> **Example 1:** While we assume that installation tools will have to >>> continue supporting version 0 sdists for the indefinite future, it's a >>> useful check to make sure that our new format can continue to support >>> packages using distutils / setuptools as their build system. We assume >>> that a future version ``pip`` will take its existing knowledge of >>> distutils internals and expose them as the appropriate hooks, and then >>> existing distutils / setuptools packages can be ported forward by >>> using the following ``_pypackage/_pypackage.cfg``:: >>> >>> [build] >>> requirements = """ >>> pip >= whatever >>> wheel >>> """ >>> # Applies monkeypatches, then does 'setup.py dist_info' and >>> # extracts the setup_requires >>> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >>> # Applies monkeypatches, then does 'setup.py wheel' >>> build-wheels = "pip.pypackage_hooks:build_wheels" >>> # Applies monkeypatches, then does: >>> # setup.py dist_info && setup.py build_ext -i >>> build-in-place = "pip.pypackage_hooks:build_in_place" >>> >>> This is also useful for any other installation tools that may want to >>> support version 0 sdists without having to implement bug-for-bug >>> compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is >>> present, they can use this as a default. >>> >>> **Example 2:** For packages using numpy.distutils. This is identical >>> to the distutils / setuptools example above, except that numpy is >>> moved into the list of static build requirements. Right now, most >>> projects using numpy.distutils don't bother trying to declare this >>> dependency, and instead simply error out if numpy is not already >>> installed. This is because currently the only way to declare a build >>> dependency is via the ``setup_requires`` argument to the ``setup`` >>> function, and in this case the ``setup`` function is >>> ``numpy.distutils.setup``, which... obviously doesn't work very well. >>> Drop this ``_pypackage.cfg`` into an existing project like this and it >>> will become robustly pip-installable with no further changes:: >>> >>> [build] >>> requirements = """ >>> numpy >>> pip >= whatever >>> wheel >>> """ >>> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >>> build-wheels = "pip.pypackage_hooks:build_wheels" >>> build-in-place = "pip.pypackage_hooks:build_in_place" >>> >>> **Example 3:** `flit `_ is a tool >>> designed to make distributing simple packages simple, but it currently >>> has no support for sdists, and for convenience includes its own >>> installation code that's redundant with that in pip. These 4 lines of >>> boilerplate make any flit-using source tree pip-installable, and lets >>> flit get out of the package installation business:: >>> >>> [build] >>> requirements = "flit" >>> build-wheels = "flit.pypackage_hooks:build_wheels" >>> build-in-place = "flit.pypackage_hooks:build_in_place" >>> >>> >>> FAQ >>> === >>> >>> **Why is it version 1 instead of version 2?** Because the legacy sdist >>> format is barely a format at all, and to `remind us to keep things >>> simple < >>> https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect >>> >`_. >>> >>> **What about cross-compilation?** Standardizing an interface for >>> cross-compilation seems premature given how complicated the >>> configuration required can be, the lack of an existing de facto >>> standard, and the authors of this PEP's inexperience with >>> cross-compilation. This would be a great target for future extensions, >>> though. In the mean time, there's no requirement that >>> ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a >>> project's build system -- packages that want to support >>> cross-compilation can still do so, they'll just need to include a >>> README explaining how to do it. >>> >>> **PEP 426 says that the new sdist format will support automatically >>> creating policy-compliant .deb/.rpm packages. What happened to that?** >>> Step 1: enhance the wheel format as necessary so that a wheel can be >>> automatically converted into a policy-compliant .deb/.rpm package (see >>> PEP 491). Step 2: make it possible to automatically turn sdists into >>> wheels (this PEP). Step 3: we're done. >>> >>> **What about automatically running tests?** Arguably this is another >>> thing that should be pushed off to wheel metadata instead of sdist >>> metadata: it's good practice to include tests inside your built >>> distribution so that end-users can test their install (and see above >>> re: our focus here being on stuff that end-users want to do, not >>> dedicated package developers), there are lots of packages that have to >>> be built before they can be tested anyway (e.g. because of binary >>> extensions), and in any case it's good practice to test against an >>> installed version in order to make sure your install code works >>> properly. But even if we do want this in sdist, then it's hardly >>> urgent (e.g. there is no ``pip test`` that people will miss), so we >>> defer that for a future extension to avoid blocking the core >>> functionality. >>> >>> -- >>> Nathaniel J. Smith -- http://vorpus.org >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Oct 2 16:12:27 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 14:12:27 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: One way to do sdist 2.0 would be to have the package-1.0.dist-info directory in there (most sdists contain setuptools metadata) and to have a flag static-metadata=1 in setup.cfg asserting that setup.py [if present] does not alter the list of dependencies. In the old MEBS design the package could suggest a build system, but pip would invoke a list of build plugins to inspect the directory and return True if they were able to build the package. This would allow for ignoring the package's suggested build system. Instead of defining a command line interface for setup.py MEBS would define a set of methods on the build plugin. I thought Robert Collins had a working setup-requires implementation already? I have a worse but backwards compatible one too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py On Fri, Oct 2, 2015 at 9:42 AM Marcus Smith wrote: > Can you clarify the relationship to PEP426 metadata? > There's no standard for metadata in here other than what's required to run > a build hook. > Does that imply you would have each build tool enforce their own > convention for where metadata is found? > > On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith wrote: > >> Hi all, >> >> We realized that actually as far as we could tell, it wouldn't be that >> hard at this point to clean up how sdists work so that it would be >> possible to migrate away from distutils. So we wrote up a little draft >> proposal. >> >> The main question is, does this approach seem sound? >> >> -n >> >> --- >> >> PEP: ?? >> Title: Standard interface for interacting with source trees >> and source distributions >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Nathaniel J. Smith >> Thomas Kluyver >> Status: Draft >> Type: Standards-Track >> Content-Type: text/x-rst >> Created: 30-Sep-2015 >> Post-History: >> Discussions-To: >> >> Abstract >> ======== >> >> Distutils delenda est. >> >> >> Extended abstract >> ================= >> >> While ``distutils`` / ``setuptools`` have taken us a long way, they >> suffer from three serious problems: (a) they're missing important >> features like autoconfiguration and usable build-time dependency >> declaration, (b) extending them is quirky, complicated, and fragile, >> (c) you are forced to use them anyway, because they provide the >> standard interface for installing python packages expected by both >> users and installation tools like ``pip``. >> >> Previous efforts (e.g. distutils2 or setuptools itself) have attempted >> to solve problems (a) and/or (b). We propose to solve (c). >> >> The goal of this PEP is get distutils-sig out of the business of being >> a gatekeeper for Python build systems. If you want to use distutils, >> great; if you want to use something else, then the more the merrier. >> The difficulty of interfacing with distutils means that there aren't >> many such systems right now, but to give a sense of what we're >> thinking about see `flit `_ or >> `bento >> `_. Fortunately, wheels have now >> solved many of the hard problems here -- e.g. it's no longer necessary >> that a build system also know about every possible installation >> configuration -- so pretty much all we really need from a build system >> is that it have some way to spit out standard-compliant wheels. >> >> We therefore propose a new, relatively minimal interface for >> installation tools like ``pip`` to interact with package source trees >> and source distributions. >> >> >> Synopsis and rationale >> ====================== >> >> To limit the scope of our design, we adopt several principles. >> >> First, we distinguish between a *source tree* (e.g., a VCS checkout) >> and a *source distribution* (e.g., an official snapshot release like >> ``lxml-3.4.4.zip``). >> >> There isn't a whole lot that *source trees* can be assumed to have in >> common. About all you know is that they can -- via some more or less >> Rube-Goldbergian process -- produce one or more binary distributions. >> In particular, you *cannot* tell via simple static inspection: >> - What version number will be attached to the resulting packages (e.g. >> it might be determined programmatically by consulting VCS metadata -- >> I have here a build of numpy version "1.11.0.dev0+4a9ad17") >> - What build- or run-time dependencies are required (e.g. these may >> depend on arbitrarily complex configuration settings that are >> determined via a mix of manual settings and auto-probing) >> - Or even how many distinct binary distributions will be produced >> (e.g. a source distribution may always produce wheel A, but only >> produce wheel B when built on Unix-like systems). >> >> Therefore, when dealing with source trees, our goal is just to provide >> a standard UX for the core operations that are commonly performed on >> other people's packages; anything fancier and more developer-centric >> we leave at the discretion of individual package developers. So our >> source trees just provide some simple hooks to let a tool like >> ``pip``: >> >> - query for build dependencies >> - run a build, producing wheels as output >> - set up the current source tree so that it can be placed on >> ``sys.path`` in "develop mode" >> >> and that's it. We teach users that the standard way to install a >> package from a VCS checkout is now ``pip install .`` instead of >> ``python setup.py install``. (This is already a good idea anyway -- >> e.g., pip can do reliable uninstall / upgrades.) >> >> Next, we note that pretty much all the operations that you might want >> to perform on a *source distribution* are also operations that you >> might want to perform on a source tree, and via the same UX. The only >> thing you do with source distributions that you don't do with source >> trees is, well, distribute them. There's all kind of metadata you >> could imagine including in a source distribution, but each piece of >> metadata puts an increased burden on source distribution generation >> tools, and most operations will still have to work without this >> metadata. So we only include extra metadata in source distributions if >> it helps solve specific problems that are unique to distribution. If >> you want wheel-style metadata, get a wheel and look at it -- they're >> great and getting better. >> >> Therefore, our source distributions are basically just source trees + >> a mechanism for signing. >> >> Finally: we explicitly do *not* have any concept of "depending on a >> source distribution". As in other systems like Debian, dependencies >> are always phrased in terms of binary distributions (wheels), and when >> a user runs something like ``pip install ``, then the >> long-run plan is that and all its transitive dependencies >> should be available as wheels in a package index. But this is not yet >> realistic, so as a transitional / backwards-compatibility measure, we >> provide a simple mechanism for ``pip install `` to handle >> cases where is provided only as a source distribution. >> >> >> Source trees >> ============ >> >> We retroactively declare the legacy source tree format involving >> ``setup.py`` to be "version 0". We don't try to specify it further; >> its de facto specification is encoded in the source code of >> ``distutils``, ``setuptools``, ``pip``, and other tools. >> >> A version 1-or-greater format source tree can be identified by the >> presence of a file ``_pypackage/_pypackage.cfg``. >> >> If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, >> then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. >> This is necessary because we anticipate that version 1+ source trees >> may want to contain a ``setup.py`` file for backwards compatibility, >> e.g.:: >> >> #!/usr/bin/env python >> import sys >> print("Don't call setup.py directly!") >> print("Use 'pip install .' instead!") >> print("(You might have to upgrade pip first.)") >> sys.exit(1) >> >> In the current version of the specification, the one file >> ``_pypackage/_pypackage.cfg`` is where pretty much all the action is >> (though see below). The motivation for putting it into a subdirectory >> is that: >> - the way of all standards is that cruft accumulates over time, so >> this way we pre-emptively have a place to put it, >> - real-world projects often accumulate build system cruft as well, so >> we might as well provide one obvious place to put it too. >> >> Of course this then creates the possibility of collisions between >> standard files and user files, and trying to teach arbitrary users not >> to scatter files around willy-nilly never works, so we adopt the >> convention that names starting with an underscore are reserved for >> official use, and non-underscored names are available for >> idiosyncratic use by individual projects. >> >> The alternative would be to simply place the main configuration file >> at the top-level, create the subdirectory only when specifically >> needed (most trees won't need it), and let users worry about finding >> their own place for their cruft. Not sure which is the best approach. >> Plus we can have a nice bikeshed about the names in general (FIXME). >> >> _pypackage.cfg >> -------------- >> >> The ``_pypackage.cfg`` file contains various settings. Another good >> bike-shed topic is which file format to use for storing these (FIXME), >> but for purposes of this draft I'll write examples using `toml >> `_, because you'll instantly be >> able to understand the semantics, it has similar expressivity to JSON >> while being more human-friendly (e.g., it supports comments and >> multi-line strings), it's better-specified than ConfigParser, and it's >> much simpler than YAML. Rust's package manager uses toml for similar >> purposes. >> >> Here's an example ``_pypackage/_pypackage.cfg``:: >> >> # Version of the "pypackage format" that this file uses. >> # Optional. If not present then 1 is assumed. >> # All version changes indicate incompatible changes; backwards >> # compatible changes are indicated by just having extra stuff in >> # the file. >> version = 1 >> >> [build] >> # An inline requirements file. Optional. >> # (FIXME: I guess this means we need a spec for requirements files?) >> requirements = """ >> mybuildtool >= 2.1 >> special_windows_tool ; sys_platform == "win32" >> """ >> # The path to an out-of-line requirements file. Optional. >> requirements-file = "build-requirements.txt" >> # A hook that will be called to query build requirements. Optional. >> requirements-dynamic = "mybuildtool:get_requirements" >> >> # A hook that will be called to build wheels. Required. >> build-wheels = "mybuildtool:do_build" >> >> # A hook that will be called to do an in-place build (see below). >> # Optional. >> build-in-place = "mybuildtool:do_inplace_build" >> >> # The "x" namespace is reserved for third-party extensions. >> # To use x.foo you should own the name "foo" on pypi. >> [x.mybuildtool] >> spam = ["spam", "spam", "spam"] >> >> All paths are relative to the ``_pypackage/`` directory (so e.g. the >> build.requirements-file value above refers to a file named >> ``_pypackage/build-requirements.txt``). >> >> A *hook* is a Python object that is looked up using the same rules as >> traditional setuptools entry_points: a dotted module name, followed by >> a colon, followed by a dotted name that is looked up within that >> module. *Running a hook* means: first, find or create a python >> interpreter which is executing in the current venv, whose working >> directory is set to the ``_pypackage/`` directory, and which has the >> ``_pypackage/`` directory on ``sys.path``. Then, inside this >> interpreter, look up the hook object, and call it, with arguments as >> specified below. >> >> A build command like ``pip wheel `` performs the following >> steps: >> >> 1) Validate the ``_pypackage.cfg`` version number. >> >> 2) Create an empty virtualenv / venv, that matches the environment >> that the installer is targeting (e.g. if you want wheels for CPython >> 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows >> venv). >> >> 3) If the build.requirements key is present, then in this venv run the >> equivalent of ``pip install -r ``, using >> whatever index settings are currently in effect. >> >> 4) If the build.requirements-file key is present, then in this venv >> run the equivalent of ``pip install -r ``, using >> whatever index settings are currently in effect. >> >> 5) If the build.requirements-dynamic key is present, then in this venv >> run the hook with no arguments, capture its stdout, and pipe it into >> ``pip install -r -``, using whatever index settings are currently in >> effect. If the hook raises an exception, then abort the build with an >> error. >> >> Note: because these steps are performed in sequence, the >> build.requirements-dynamic hook is allowed to use packages that are >> listed in build.requirements or build.requirements-file. >> >> 6) In this venv, run the build.build-wheels hook. This should be a >> Python function which takes one argument. >> >> This argument is an arbitrary dictionary intended to contain >> user-specified configuration, specified via some install-tool-specific >> mechanism. The intention is that tools like ``pip`` should provide >> some way for users to specify key/value settings that will be passed >> in here, analogous to the legacy ``--install-option`` and >> ``--global-option`` arguments. >> >> To make it easier for packages to transition from version 0 to >> version 1 sdists, we suggest that ``pip`` and other tools that have >> such existing option-setting interfaces SHOULD map them to entries in >> this dictionary when -- e.g.:: >> >> pip --global-option=a --install-option=b --install-option=c >> >> could produce a dict like:: >> >> {"--global-option": ["a"], "--install-option": ["b", "c"]} >> >> The hook's return value is a list of pathnames relative to the >> scratch directory. Each entry names a wheel file created by this >> build. >> >> Errors are signaled by raising an exception. >> >> When performing an in-place build (e.g. for ``pip install -e .``), >> then the same steps are followed, except that instead of the >> build.build-wheels hook, we call the build.build-in-place hook, and >> instead of returning a list of wheel files, it returns the name of a >> directory that should be placed onto ``sys.path`` (usually this will >> be the source tree itself, but may not be, e.g. if a build system >> wants to enforce a rule where the source is always kept pristine then >> it could symlink the .py files into a build directory, place the >> extension modules and dist-info there, and return that). This >> directory must contain importable versions of the code in the source >> tree, along with appropriate .dist-info directories. >> >> (FIXME: in-place builds are useful but intrinsically kinda broken -- >> e.g. extensions / source / metadata can all easily get out of sync -- >> so while I think this paragraph provides a reasonable hack that >> preserves current functionality, maybe we should defer specifying them >> to until after we've thought through the issues more?) >> >> When working with source trees, build tools like ``pip`` are >> encouraged to cache and re-use virtualenvs for performance. >> >> >> Other contents of _pypackage/ >> ----------------------------- >> >> _RECORD, _RECORD.jws, _RECORD.p7s: see below. >> >> _x//: reserved for use by tools (e.g. >> _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) >> >> >> Source distributions >> ==================== >> >> A *source distribution* is a file in a well-known archive format such >> as zip or tar.gz, which contains a single directory, and this >> directory is a source tree (in the sense defined in the previous >> section). >> >> The ``_pypackage/`` directory in a source distribution SHOULD also >> contain a _RECORD file, as defined in PEP 427, and MAY also contain >> _RECORD.jws and/or _RECORD.p7s signature files. >> >> For official releases, source distributions SHOULD be named as >> ``-.``, and the directory they contain SHOULD >> be named ``-``, and building this source tree SHOULD >> produce a wheel named ``--.whl`` >> (though it may produce other wheels as well). >> >> (FIXME: maybe we should add that if you want your sdist on PyPI then >> you MUST include a proper _RECORD file and use the proper naming >> convention?) >> >> Integration tools like ``pip`` SHOULD take advantage of this >> convention by applying the following heuristic: when seeking a package >> , if no appropriate wheel can be found, but an sdist named >> -. is found, then: >> >> 1) build the sdist >> 2) add the resulting wheels to the package search space >> 3) retry the original operation >> >> This handles a variety of simple and complex cases -- for example, if >> we need a package 'foo', and we find foo-1.0.zip which builds foo.whl >> and bar.whl, and foo.whl depends on bar.whl, then everything will work >> out. There remain other cases that are not handled, e.g. if we start >> out searching for bar.whl we will never discover foo-1.0.zip. We take >> the perspective that this is nonetheless sufficient for a transitional >> heuristic, and anyone who runs into this problem should just upload >> wheels already. If this turns out to be inadequate in practice, then >> it will be addressed by future extensions. >> >> >> Examples >> ======== >> >> **Example 1:** While we assume that installation tools will have to >> continue supporting version 0 sdists for the indefinite future, it's a >> useful check to make sure that our new format can continue to support >> packages using distutils / setuptools as their build system. We assume >> that a future version ``pip`` will take its existing knowledge of >> distutils internals and expose them as the appropriate hooks, and then >> existing distutils / setuptools packages can be ported forward by >> using the following ``_pypackage/_pypackage.cfg``:: >> >> [build] >> requirements = """ >> pip >= whatever >> wheel >> """ >> # Applies monkeypatches, then does 'setup.py dist_info' and >> # extracts the setup_requires >> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >> # Applies monkeypatches, then does 'setup.py wheel' >> build-wheels = "pip.pypackage_hooks:build_wheels" >> # Applies monkeypatches, then does: >> # setup.py dist_info && setup.py build_ext -i >> build-in-place = "pip.pypackage_hooks:build_in_place" >> >> This is also useful for any other installation tools that may want to >> support version 0 sdists without having to implement bug-for-bug >> compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is >> present, they can use this as a default. >> >> **Example 2:** For packages using numpy.distutils. This is identical >> to the distutils / setuptools example above, except that numpy is >> moved into the list of static build requirements. Right now, most >> projects using numpy.distutils don't bother trying to declare this >> dependency, and instead simply error out if numpy is not already >> installed. This is because currently the only way to declare a build >> dependency is via the ``setup_requires`` argument to the ``setup`` >> function, and in this case the ``setup`` function is >> ``numpy.distutils.setup``, which... obviously doesn't work very well. >> Drop this ``_pypackage.cfg`` into an existing project like this and it >> will become robustly pip-installable with no further changes:: >> >> [build] >> requirements = """ >> numpy >> pip >= whatever >> wheel >> """ >> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >> build-wheels = "pip.pypackage_hooks:build_wheels" >> build-in-place = "pip.pypackage_hooks:build_in_place" >> >> **Example 3:** `flit `_ is a tool >> designed to make distributing simple packages simple, but it currently >> has no support for sdists, and for convenience includes its own >> installation code that's redundant with that in pip. These 4 lines of >> boilerplate make any flit-using source tree pip-installable, and lets >> flit get out of the package installation business:: >> >> [build] >> requirements = "flit" >> build-wheels = "flit.pypackage_hooks:build_wheels" >> build-in-place = "flit.pypackage_hooks:build_in_place" >> >> >> FAQ >> === >> >> **Why is it version 1 instead of version 2?** Because the legacy sdist >> format is barely a format at all, and to `remind us to keep things >> simple < >> https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect >> >`_. >> >> **What about cross-compilation?** Standardizing an interface for >> cross-compilation seems premature given how complicated the >> configuration required can be, the lack of an existing de facto >> standard, and the authors of this PEP's inexperience with >> cross-compilation. This would be a great target for future extensions, >> though. In the mean time, there's no requirement that >> ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a >> project's build system -- packages that want to support >> cross-compilation can still do so, they'll just need to include a >> README explaining how to do it. >> >> **PEP 426 says that the new sdist format will support automatically >> creating policy-compliant .deb/.rpm packages. What happened to that?** >> Step 1: enhance the wheel format as necessary so that a wheel can be >> automatically converted into a policy-compliant .deb/.rpm package (see >> PEP 491). Step 2: make it possible to automatically turn sdists into >> wheels (this PEP). Step 3: we're done. >> >> **What about automatically running tests?** Arguably this is another >> thing that should be pushed off to wheel metadata instead of sdist >> metadata: it's good practice to include tests inside your built >> distribution so that end-users can test their install (and see above >> re: our focus here being on stuff that end-users want to do, not >> dedicated package developers), there are lots of packages that have to >> be built before they can be tested anyway (e.g. because of binary >> extensions), and in any case it's good practice to test against an >> installed version in order to make sure your install code works >> properly. But even if we do want this in sdist, then it's hardly >> urgent (e.g. there is no ``pip test`` that people will miss), so we >> defer that for a future extension to avoid blocking the core >> functionality. >> >> -- >> Nathaniel J. Smith -- http://vorpus.org >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Fri Oct 2 15:52:27 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 2 Oct 2015 14:52:27 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 12:58, Donald Stufft wrote: > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. > I can't now find that it in the draft text, but I don't think this is a problem. This proposal means there's a standardised way to turn a source tree into wheels. So downstream distributors can download an sdist - or even a tarball of a VCS tag, if they're being strict about it - build wheels from that using the config in this proposal, and then transform the wheels into their own package format. > Longer term, I think the answer is sdist 2.0 which has proper metadata inside of it (name, version, dependencies, etc) but which also includes a hook like this PEP has to specify the build system I hadn't heard of this before - is it something that's being worked on? Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Oct 2 18:30:42 2015 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 Oct 2015 09:30:42 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: > > > So downstream distributors can download an sdist - or even a tarball of a > VCS tag, if they're being strict about it - build wheels from that using > the config in this proposal, and then transform the wheels into their own > package format. > this has wheel itself being the interoperability standard. the going idea is that there would be a metadata artifact ("pydist.json" in PEP426), that provides the standard across different formats. > Longer term, I think the answer is sdist 2.0 which has proper metadata > inside of it (name, version, dependencies, etc) but which also includes a > hook like this PEP has to specify the build system > > I hadn't heard of this before - is it something that's being worked on? > I haven't seen any PEP's for it yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Fri Oct 2 19:38:47 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 2 Oct 2015 13:38:47 -0400 Subject: [Distutils] intelcompiler class in distutils In-Reply-To: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> References: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> Message-ID: On Wed, Sep 30, 2015 at 3:17 PM, Hogan, Christopher wrote: > Hello, > > > > We noticed there are classes in distutils for specific compilers like > Borland, Cygwin, MSVC, etc., and are interested in creating a class for the > Intel compiler. Is this something that the python developers/community > would like to see, or would such a patch have a low chance for acceptance? Hi, I don't have any weight to add one way or the other, but I have encountered a need for this in the past (though it was a long time ago and I can't recall the context). It might be tricky to get it accepted into distutils in the stdlib, if only because it would have to come with some promise of continued maintenance. And also getting anything added to distutils these days is hard (though last I recall the freeze may be liftable for non-backwards-compat-breaking new features where needed?) That said, you might have better luck adding it to setuptools instead, or a completely separate third-party package. Adding support for a new compiler that is *not* directly in the distutils package requires a bit of hackery though. Incidentally, d2to1 [1] actually supports packages that include their own compiler class, and allows them to use that compiler for builds. I don't know that this feature has ever actually been used, but my point in bringing it up is just to demonstrate that it can be done. And you can look at that code for an example of how to add a new compiler without it being directly in the distutils module (even if it's a bit of a hack). Best, Erik [1] https://github.com/embray/d2to1/blob/master/d2to1/util.py#L330 From brett at python.org Fri Oct 2 20:41:07 2015 From: brett at python.org (Brett Cannon) Date: Fri, 02 Oct 2015 18:41:07 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, 2 Oct 2015 at 05:08 Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > > We realized that actually as far as we could tell, it wouldn't > > be that > > hard at this point to clean up how sdists work so that it would be > > possible to migrate away from distutils. So we wrote up a little > > draft > > proposal. > > > > The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might > be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. Debian for instance > has > policy that requires the use of building all of it's packages from Source, > not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a > mess > when you factor in packages that have any C code what so ever. > So wouldn't they then download the sdist, build a wheel as an intermediate, and then generate the .deb file? I mean as long as people upload an sdist for those that want to build from source and a wheel for convenience -- which is probably what most people providing wheels do anyway -- then I don't see the problem. > > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on > wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that > we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, > all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never > going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism > by > which to determine dependencies of a source distribution without > downloading > the file and executing some potentially untrusted code. This makes > dependency > resolution harder and much much slower than if we could read that > information > statically from a source distribution. This PEP doesn't offer anything in > the > way of solving this problem. > Isn't that what the requirements and requirements-file fields in the _pypackage file provide? Only if you use that requirements-dynamic would it require execcuting arbitrary code to gather dependency information, or am I missing something? > > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine > what > it is and to get information about it. Right now PyPI relies on the > uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from > within the > file. This is sort of possible right now since distutils and setuptools > both > create a static metadata file within the source distribution, but we don't > rely > on that within PyPI because that information may or may not be accurate > and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. > Isn't that only if you use the dynamic fields? > > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference > from > within a .cfg file instead of creating a setuptools extension or so) but > without fixing most of the problems. The largest benefit I see to > switching to > this right now is that it would enable us to have build time dependencies > that > were controlled by pip rather than installed implicitly via the execution > of > the setup.py. That doesn't feel like a big enough benefit to me to do a > mass > shakeup of what we recommend and tell people to do. Having people adjust > and > change and do something new requires effort, and we need something to > justify > that effort to other people and I don't think that this PEP has something > we > can really use to justify that effort. > >From my naive perspective this proposal seems to help push forward a decoupling of building using distutils/setuptools as the only way you can properly build Python projects (which is what I think we are all after) and will hopefully eventually free pip up to simply do orchestration. > > I *do* think that there is a core of some ideas here that are valuable, > and in > fact are similar to some ideas I've had. The main flaw I see here is that > it > doesn't really fix sdists, it takes a solution that would work for VCS > checkouts and then reuses it for sdists. In my mind, the supported flow for > package installation would be: > > VCS/Bare Directory -> Source Distribution -> Wheel > > This would (eventually) be the only path that was supported for > installation > but you could "enter" the path at any stage. For example, if there is a > Wheel > already available, then you jump right on at the end and just install > that, if > there is a sdist available then pip first builds it into a wheel and then > installs that, etc. > > I think your PEP is something like what the VCS/Bare Directory to sdist > tooling > could look like, but I don't think it's what the sdist to wheel path should > look like. > Is there another proposal I'm unaware for the sdist -> wheel step that is build tool-agnostic? I'm all for going with the best solution but there has to be an actual alternative to compare against and I don't know of any others right now and this proposal does seem to move things forward in a reasonable fashion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Oct 2 21:02:44 2015 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 Oct 2015 12:02:44 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: > >> The first thing that immediately stood out to me, is that it's >> recommending >> that downstream redistributors like Debian, Fedora, etc utilize Wheels >> instead >> of the sdist to build their packages from. However, that is not really >> going to >> fly with most (all?) of the downstream redistributors. Debian for >> instance has >> policy that requires the use of building all of it's packages from >> Source, not >> from anything else and Wheels are not a source package. While it can >> theoretically work for pure python packages, it quickly devolves into a >> mess >> when you factor in packages that have any C code what so ever. >> > > So wouldn't they then download the sdist, build a wheel as an > intermediate, and then generate the .deb file? > the new goal I think was to have standardized metadata immediately available in an sdist, and get away from the model, that you had to run a build step, before you had a metadata artifact. so here, you'd have to build a wheel (potentially one with binary extensions) just to know what the metadata is? that doesn't sound right. > I mean as long as people upload an sdist for those that want to build from > source and a wheel for convenience -- which is probably what most people > providing wheels do anyway -- then I don't see the problem. > > >> >> Overall, this feels more like a sidegrade than an upgrade. One major theme >> throughout of the PEP is that we're going to push to rely heavily on >> wheels as >> the primary format of installation. While that works well for things like >> Debian, I don't think it's going to work as wheel for us. If we were only >> distributing pure python packages, then yes absolutely, however given >> that we >> are not, we have to worry about ABI issues. Given that there is so many >> different environments that a particular package might be installed into, >> all >> with different ABIs we have to assume that installing from source is still >> going to be a primary path for end users to install and that we are never >> going >> to have a world where we can assume a Wheel in a repository. >> >> One of the problems with the current system, is that we have no mechanism >> by >> which to determine dependencies of a source distribution without >> downloading >> the file and executing some potentially untrusted code. This makes >> dependency >> resolution harder and much much slower than if we could read that >> information >> statically from a source distribution. This PEP doesn't offer anything in >> the >> way of solving this problem. >> > > Isn't that what the requirements and requirements-file fields in the > _pypackage file provide? Only if you use that requirements-dynamic would it > require execcuting arbitrary code to gather dependency information, or am I > missing something? > those are just requirements to run the build hook, not run time dependencies. > Is there another proposal I'm unaware for the sdist -> wheel step that is > build tool-agnostic? > PEP426 talks about it some https://www.python.org/dev/peps/pep-0426/#metabuild-system -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 21:09:30 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 15:09:30 -0400 Subject: [Distutils] intelcompiler class in distutils In-Reply-To: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> References: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> Message-ID: On October 2, 2015 at 9:42:22 AM, Hogan, Christopher (christopher.hogan at intel.com) wrote: > Hello, > > We noticed there are classes in distutils for specific compilers like Borland, Cygwin, > MSVC, etc., and are interested in creating a class for the Intel compiler. Is this something > that the python developers/community would like to see, or would such a patch have a low > chance for acceptance? > Is the Intel Compiler Free? (or OSS, but Free is the main thing). I think as long as there is a way for people to get a copy of the compiler to run tests and stuff against without spending $$$ there is very little reason not to add another compiler class. If it costs money always, then IDK what the policy for that is, I personally still wouldn?t have a problem as long as there was a commitment from intel to support it, but it?s not my decision either. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Fri Oct 2 21:45:25 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 15:45:25 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 2:41:20 PM, Brett Cannon (brett at python.org) wrote: > > Is there another proposal I'm unaware for the sdist -> wheel > step that is build tool-agnostic? I'm all for going with the best > solution but there has to be an actual alternative to compare > against and I don't know of any others right now and this proposal > does seem to move things forward in a reasonable fashion. As Marcus said, it was touched on in PEP 426 but there isn?t a fully fleshed out PEP (or even a sort of fleshed out PEP) but the high level idea isn?t super hard, take PEP 426 metadata + some sdist specific metadata and stick it in a tarball or something and then you have sdist 2.0. Part of that sdist specific metadata would be describing in some way how to build a wheel out of this sdist (rather than just assuming distutils/setuptools). I think we need to push back against partial solutions to problems that we have without at least some semblance of a plan for how to solve the rest of the problems we have, or to explicitly declare them unsolvable. That doesn't mean that we need to turn every thing into an overengineered mess where we try to solve every problem ever, but we need to be careful that we minimize churn where possible. An example just recently where we had this was we accepted PEP 438 2 years ago as a partial solution to try and "move things forward" and then it turned out that solution was, well partial, and when we removed the partial solution for a full solution we had the folks who decided to rely on the partial solution upset. IOW we need to balance between avoiding churn and making a change "worth" it, and in my opinion this idea here doesn't solve enough problems to make it worth it.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Fri Oct 2 21:48:47 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 19:48:47 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: The MEBS idea is inspired by heroku buildpacks where you just ask a list of tools whether they can build something. https://devcenter.heroku.com/articles/buildpacks . The idea would be that pip would use MEBS instead of its setup.py-focused builder. The first available MEBS plugin would notice setup.py and do what pip does now (force setuptools, build in a subprocess). You should know about flit https://github.com/takluyver/flit and Bento http://cournape.github.io/Bento/ which have their own lightweight metadata formats, which are transformed into standard Python formats by the respective tools. requires.txt is popular but I'm not a fan of it, it seems like it was invented by people who didn't want to have a proper setup.py for their project. We have to come up with something simpler than setup.py if we want to get some of the people who don't understand how to write setup.py. Ideally any required new user-editable "which build system" metadata could be boiled down to a single line in setup.cfg. There would be 3 stages: VCS checkout (minimal metadata, a "generate machine readable metadata" step equivalent to "setup.py egg_info") -> new sdist (PEP 376 style static metadata that can be trusted) -> wheel. (How pip builds a package from source: 1. download sdist; .egg-info directory is almost always present 2. run setup.py egg_info to get dependencies, because the static one is not reliable, because too many requirements lists have 'if' statements 3. compile) For all the talk about static metadata, the build script in general needs to remain a Turing-complete script. Build systems everywhere are programs to build other programs. I really like your idea about returning a list of built artifacts. Python packaging is strictly 1:1 source package -> output package but rpm, deb, can generate many packages from a single source package. I don't think we have to worry that much about Debian & RHEL. They will get over it if setup.py is no longer there. Change brings work but stagnation brings death. On Fri, Oct 2, 2015 at 2:41 PM Brett Cannon wrote: > On Fri, 2 Oct 2015 at 05:08 Donald Stufft wrote: > >> On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: >> > > We realized that actually as far as we could tell, it wouldn't >> > be that >> > hard at this point to clean up how sdists work so that it would be >> > possible to migrate away from distutils. So we wrote up a little >> > draft >> > proposal. >> > >> > The main question is, does this approach seem sound? >> >> I've just read over your proposal, but I've also just woken up so I might >> be >> a little slow still! After reading what you have, I don't think that this >> proposal is the right way to go about improving sdists. >> >> The first thing that immediately stood out to me, is that it's >> recommending >> that downstream redistributors like Debian, Fedora, etc utilize Wheels >> instead >> of the sdist to build their packages from. However, that is not really >> going to >> fly with most (all?) of the downstream redistributors. Debian for >> instance has >> policy that requires the use of building all of it's packages from >> Source, not >> from anything else and Wheels are not a source package. While it can >> theoretically work for pure python packages, it quickly devolves into a >> mess >> when you factor in packages that have any C code what so ever. >> > > So wouldn't they then download the sdist, build a wheel as an > intermediate, and then generate the .deb file? I mean as long as people > upload an sdist for those that want to build from source and a wheel for > convenience -- which is probably what most people providing wheels do > anyway -- then I don't see the problem. > > >> >> Overall, this feels more like a sidegrade than an upgrade. One major theme >> throughout of the PEP is that we're going to push to rely heavily on >> wheels as >> the primary format of installation. While that works well for things like >> Debian, I don't think it's going to work as wheel for us. If we were only >> distributing pure python packages, then yes absolutely, however given >> that we >> are not, we have to worry about ABI issues. Given that there is so many >> different environments that a particular package might be installed into, >> all >> with different ABIs we have to assume that installing from source is still >> going to be a primary path for end users to install and that we are never >> going >> to have a world where we can assume a Wheel in a repository. >> >> One of the problems with the current system, is that we have no mechanism >> by >> which to determine dependencies of a source distribution without >> downloading >> the file and executing some potentially untrusted code. This makes >> dependency >> resolution harder and much much slower than if we could read that >> information >> statically from a source distribution. This PEP doesn't offer anything in >> the >> way of solving this problem. >> > > Isn't that what the requirements and requirements-file fields in the > _pypackage file provide? Only if you use that requirements-dynamic would it > require execcuting arbitrary code to gather dependency information, or am I > missing something? > > >> >> To a similar tune, this PEP also doesn't make it possible to really get at >> any other metadata without executing software. This makes it pratically >> impossible to safely inspect an unknown or untrusted package to determine >> what >> it is and to get information about it. Right now PyPI relies on the >> uploading >> tool to send that information alongside of the file it is uploading, but >> honestly what it should be doing is extracting that information from >> within the >> file. This is sort of possible right now since distutils and setuptools >> both >> create a static metadata file within the source distribution, but we >> don't rely >> on that within PyPI because that information may or may not be accurate >> and may >> or may not exist. However the twine uploading tool *does* rely on that, >> and >> this PEP would break the ability for twine to upload a package without >> executing arbitrary code. >> > > Isn't that only if you use the dynamic fields? > > >> >> Overall, I don't think that this really solves most of the foundational >> problems with the current format. Largely it feels that what it achieves >> is >> shuffling around some logic (you need to create a hook that you reference >> from >> within a .cfg file instead of creating a setuptools extension or so) but >> without fixing most of the problems. The largest benefit I see to >> switching to >> this right now is that it would enable us to have build time dependencies >> that >> were controlled by pip rather than installed implicitly via the execution >> of >> the setup.py. That doesn't feel like a big enough benefit to me to do a >> mass >> shakeup of what we recommend and tell people to do. Having people adjust >> and >> change and do something new requires effort, and we need something to >> justify >> that effort to other people and I don't think that this PEP has something >> we >> can really use to justify that effort. >> > > From my naive perspective this proposal seems to help push forward a > decoupling of building using distutils/setuptools as the only way you can > properly build Python projects (which is what I think we are all after) and > will hopefully eventually free pip up to simply do orchestration. > > >> >> I *do* think that there is a core of some ideas here that are valuable, >> and in >> fact are similar to some ideas I've had. The main flaw I see here is that >> it >> doesn't really fix sdists, it takes a solution that would work for VCS >> checkouts and then reuses it for sdists. In my mind, the supported flow >> for >> package installation would be: >> >> VCS/Bare Directory -> Source Distribution -> Wheel >> >> This would (eventually) be the only path that was supported for >> installation >> but you could "enter" the path at any stage. For example, if there is a >> Wheel >> already available, then you jump right on at the end and just install >> that, if >> there is a sdist available then pip first builds it into a wheel and then >> installs that, etc. >> >> I think your PEP is something like what the VCS/Bare Directory to sdist >> tooling >> could look like, but I don't think it's what the sdist to wheel path >> should >> look like. >> > > Is there another proposal I'm unaware for the sdist -> wheel step that is > build tool-agnostic? I'm all for going with the best solution but there has > to be an actual alternative to compare against and I don't know of any > others right now and this proposal does seem to move things forward in a > reasonable fashion. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 2 22:03:07 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 21:03:07 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 20:02, Marcus Smith wrote: >> So wouldn't they then download the sdist, build a wheel as an >> intermediate, and then generate the .deb file? > > the new goal I think was to have standardized metadata immediately available > in an sdist, and get away from the model, that you had to run a build step, > before you had a metadata artifact. > so here, you'd have to build a wheel (potentially one with binary > extensions) just to know what the metadata is? that doesn't sound right. I'm uncomfortable with the fact that the proposed sdist format has more or less no metadata of its own (even the filename format is only a recommendation) so (for example) if someone does "pip install foo==1.0" I don't see how pip can find a suitable sdist, if no wheel is available. I would rather see an sdist format that can be introspected *without* running code or a build tool. Installers and packaging tools like pip need to be able to do that - one of the biggest issues with pip's current sdist handling is that it can't make any meaningful decisions before building at least the egg-info. Ultimately the question for me is at what point do we require packaging tools like pip (and ad-hoc distribution analysis scripts - I write a lot of those!) to run code from the package in order to continue? I'd like to be able to know, at a minimum, the package name and version, as those are needed to make decisions on whether there is a conflict with an already installed version. Basically, why aren't you using a PEP 426-compatible metadata format in the sdist (there's no reason not to, you just have to mandate that tools used to build sdists generate that form of metadata file)? You could also say that source trees SHOULD store metadata in the _pypackage directory (in an appropriate defined format, maybe one more suited to human editing than JSON) and tools that work on source trees (build tools, things that create sdists) SHOULD use that data as the definitive source of metadata, rather than using their own configuration. I don't see a problem with allowing source trees to have some flexibility, but sdists are tool-generated and so could easily be required to contain static metadata in a standard format. >> Is there another proposal I'm unaware for the sdist -> wheel step that is >> build tool-agnostic? > > PEP426 talks about it some > https://www.python.org/dev/peps/pep-0426/#metabuild-system While the metabuild system is a good long-term goal, I'll take something that's being developed now over a great idea that no-one has time to work on... Wheel came about because Daniel just got on with it. Having said that, I very strongly prefer a sdist proposal that's compatible with PEP 426 (at least to the extent that tools like wheel already support it). Throwing away all of the work already done on PEP 426 doesn't seem like a good plan. Paul From p.f.moore at gmail.com Fri Oct 2 22:14:28 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 21:14:28 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 20:45, Donald Stufft wrote: > n October 2, 2015 at 2:41:20 PM, Brett Cannon (brett at python.org) wrote: >> > Is there another proposal I'm unaware for the sdist -> wheel >> step that is build tool-agnostic? I'm all for going with the best >> solution but there has to be an actual alternative to compare >> against and I don't know of any others right now and this proposal >> does seem to move things forward in a reasonable fashion. > > As Marcus said, it was touched on in PEP 426 but there isn?t a fully fleshed > out PEP (or even a sort of fleshed out PEP) but the high level idea isn?t super > hard, take PEP 426 metadata + some sdist specific metadata and stick it in a > tarball or something and then you have sdist 2.0. Part of that sdist specific > metadata would be describing in some way how to build a wheel out of this sdist > (rather than just assuming distutils/setuptools). > > I think we need to push back against partial solutions to problems that we > have without at least some semblance of a plan for how to solve the rest of > the problems we have, or to explicitly declare them unsolvable. That doesn't > mean that we need to turn every thing into an overengineered mess where we try > to solve every problem ever, but we need to be careful that we minimize churn > where possible. An example just recently where we had this was we accepted > PEP 438 2 years ago as a partial solution to try and "move things forward" and > then it turned out that solution was, well partial, and when we removed the > partial solution for a full solution we had the folks who decided to rely on > the partial solution upset. > > IOW we need to balance between avoiding churn and making a change "worth" it, > and in my opinion this idea here doesn't solve enough problems to make it > worth it. That's a fair point. But I don't see any reason Nathaniel's proposal *couldn't* be that solution. I'd want to see the sdist format required to include static metadata, and the metadata format to be PEP 426, but neither of those seem incompatible with the ideas behind the proposal. Maybe I'm missing something massive, but I don't see a *huge* gap between this proposal and the basic ideas behind the metabuild concept - the sdist should define how to build it (in terms of required packages/tools for the build, and commands/hooks to call to do the specific build steps - make a wheel, do an in-place install). The biggest sticking point would be if Nathaniel is adamantly opposed to static metadata. Is there a good enumeration anywhere of the problems sdist 2.0 / the metabuild system needs to solve? I can think of: 1. Metadata needs to be discoverable without running code. 2. Packages should be able to specify the tools needed to build them. 3. Installers should only interact with sdists through well-defined entry points. 4. Build processes required are (a) create a wheel, (b) do a "develop" style in-place installation. 5. (Maybe) There needs to be better means of handling build errors. The proposal here seems to at least move towards better solutions for 2, 3, and 4. Paul From njs at pobox.com Fri Oct 2 22:19:56 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 13:19:56 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 4:58 AM, Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: >> > We realized that actually as far as we could tell, it wouldn't >> be that >> hard at this point to clean up how sdists work so that it would be >> possible to migrate away from distutils. So we wrote up a little >> draft >> proposal. >> >> The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels instead > of the sdist to build their packages from. However, that is not really going to > fly with most (all?) of the downstream redistributors. Debian for instance has > policy that requires the use of building all of it's packages from Source, not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a mess > when you factor in packages that have any C code what so ever. I think this was addressed downthread -- the idea would be that Debian would build from sdist, with a two step process: convert sdist to wheels, repack wheels into binary .deb. > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism by > which to determine dependencies of a source distribution without downloading > the file and executing some potentially untrusted code. This makes dependency > resolution harder and much much slower than if we could read that information > statically from a source distribution. This PEP doesn't offer anything in the > way of solving this problem. What are the "dependencies of a source distribution"? Do you mean the runtime dependencies of the wheels that will be built from a source distribution? If you need that metadata to be statically in the sdist, then you might as well give up now because it's simply impossible. As the very simplest example, every package that uses the numpy C API gets a runtime dependency on "numpy >= [whatever version happened to be installed on the *build* machine]". There are plenty of more complex examples too (e.g. ones that involve build/configure-time decisions about whether to rely on particular system libraries, or build/configure-time decisions about whether particular packages should even be built). For comparison, here's the Debian source package metadata: https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles Note that the only mandatory fields are format version / package name / package version / maintainer / checksums. The closest they come to making promises about the built packages are the Package-List and Binary fields which provide a optional hint about what binary packages will be built, and are allowed to contain lies (e.g. they explicitly don't guarantee that all the binary packages named will actually be produced on every architecture). The only kind of dependencies that a source package can declare are build-depends. > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine what > it is and to get information about it. Right now PyPI relies on the uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from within the > file. This is sort of possible right now since distutils and setuptools both > create a static metadata file within the source distribution, but we don't rely > on that within PyPI because that information may or may not be accurate and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. Okay, what metadata do you need? We certainly could put name / version kind of stuff in there. We left it out because we weren't sure what was necessary and it's easy to add later, but anything that's needed by twine fits neatly into the existing text saying that we should "include extra metadata in source distributions if it helps solve specific problems that are unique to distribution" -- twine uploads definitely count. > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference from > within a .cfg file instead of creating a setuptools extension or so) but numpy.distutils is the biggest distutils/setuptools extension around, and everyone involved in maintaining it wants to kill it with fire :-). That's a problem... > without fixing most of the problems. The largest benefit I see to switching to > this right now is that it would enable us to have build time dependencies that > were controlled by pip rather than installed implicitly via the execution of > the setup.py. Yes, this problem means that literally every numerical python package currently has a broken setup.py. > That doesn't feel like a big enough benefit to me to do a mass > shakeup of what we recommend and tell people to do. Having people adjust and > change and do something new requires effort, and we need something to justify > that effort to other people and I don't think that this PEP has something we > can really use to justify that effort. The end-user adjustment is teaching people to switch to always using pip to install packages -- this seems like something we will certainly do sooner or later, so we might as well get started. And it's already actually the right thing to do -- if you use 'setup.py install' then you get a timebomb in your venv where later upgrades may leave you with a broken package :-(. (This is orthogonal to the actual PEP.) In the long run, the idea that every package has to contain code that knows how to implement installation in very possible configuration (--user? --single-version-externally-managed?) is clearly broken, and teaching people to use 'pip install' is obviously the only sensible alternative. -n -- Nathaniel J. Smith -- http://vorpus.org From dholth at gmail.com Fri Oct 2 22:24:25 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 20:24:25 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: We need to embrace partial solutions and the fine folks who propose them so the whole packaging ecosystem can have some progress. PEP 438 may not be a good analogue to adding a new sdist format since the latter only adds new things that you can do. A new sdist format will inconvenience a much more limited set of people, mainly the pip authors and the OS package maintainers. Sorry but the section of the PEP that prefixes filenames with _ has distracted the discussion away from the general idea. Instead of multiple hooks why not a single object exposed through an entry point that has several optional methods? NO [build] requirements = "flit" build-wheels = "flit.pypackage_hooks:build_wheels" build-in-place = "flit.pypackage_hooks:build_in_place" YES [build] build-system=flit class ABuildHook: def build_wheels(self): ... entry_points = {'new.sdist.build.hooks': ['flit= some_module:ABuildHook']} On Fri, Oct 2, 2015 at 3:48 PM Daniel Holth wrote: > The MEBS idea is inspired by heroku buildpacks where you just ask a list > of tools whether they can build something. > https://devcenter.heroku.com/articles/buildpacks . The idea would be that > pip would use MEBS instead of its setup.py-focused builder. The first > available MEBS plugin would notice setup.py and do what pip does now (force > setuptools, build in a subprocess). > > You should know about flit https://github.com/takluyver/flit and Bento > http://cournape.github.io/Bento/ which have their own lightweight > metadata formats, which are transformed into standard Python formats by the > respective tools. > > requires.txt is popular but I'm not a fan of it, it seems like it was > invented by people who didn't want to have a proper setup.py for their > project. > > We have to come up with something simpler than setup.py if we want to get > some of the people who don't understand how to write setup.py. Ideally > any required new user-editable "which build system" metadata could be > boiled down to a single line in setup.cfg. There would be 3 stages: VCS > checkout (minimal metadata, a "generate machine readable metadata" step > equivalent to "setup.py egg_info") -> new sdist (PEP 376 style static > metadata that can be trusted) -> wheel. > > (How pip builds a package from source: 1. download sdist; .egg-info > directory is almost always present 2. run setup.py egg_info to get > dependencies, because the static one is not reliable, because too many > requirements lists have 'if' statements 3. compile) > > For all the talk about static metadata, the build script in general needs > to remain a Turing-complete script. Build systems everywhere are programs > to build other programs. > > I really like your idea about returning a list of built artifacts. Python > packaging is strictly 1:1 source package -> output package but rpm, deb, > can generate many packages from a single source package. > > I don't think we have to worry that much about Debian & RHEL. They will > get over it if setup.py is no longer there. Change brings work but > stagnation brings death. > > On Fri, Oct 2, 2015 at 2:41 PM Brett Cannon wrote: > >> On Fri, 2 Oct 2015 at 05:08 Donald Stufft wrote: >> >>> On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) >>> wrote: >>> > > We realized that actually as far as we could tell, it wouldn't >>> > be that >>> > hard at this point to clean up how sdists work so that it would be >>> > possible to migrate away from distutils. So we wrote up a little >>> > draft >>> > proposal. >>> > >>> > The main question is, does this approach seem sound? >>> >>> I've just read over your proposal, but I've also just woken up so I >>> might be >>> a little slow still! After reading what you have, I don't think that this >>> proposal is the right way to go about improving sdists. >>> >>> The first thing that immediately stood out to me, is that it's >>> recommending >>> that downstream redistributors like Debian, Fedora, etc utilize Wheels >>> instead >>> of the sdist to build their packages from. However, that is not really >>> going to >>> fly with most (all?) of the downstream redistributors. Debian for >>> instance has >>> policy that requires the use of building all of it's packages from >>> Source, not >>> from anything else and Wheels are not a source package. While it can >>> theoretically work for pure python packages, it quickly devolves into a >>> mess >>> when you factor in packages that have any C code what so ever. >>> >> >> So wouldn't they then download the sdist, build a wheel as an >> intermediate, and then generate the .deb file? I mean as long as people >> upload an sdist for those that want to build from source and a wheel for >> convenience -- which is probably what most people providing wheels do >> anyway -- then I don't see the problem. >> >> >>> >>> Overall, this feels more like a sidegrade than an upgrade. One major >>> theme >>> throughout of the PEP is that we're going to push to rely heavily on >>> wheels as >>> the primary format of installation. While that works well for things like >>> Debian, I don't think it's going to work as wheel for us. If we were only >>> distributing pure python packages, then yes absolutely, however given >>> that we >>> are not, we have to worry about ABI issues. Given that there is so many >>> different environments that a particular package might be installed >>> into, all >>> with different ABIs we have to assume that installing from source is >>> still >>> going to be a primary path for end users to install and that we are >>> never going >>> to have a world where we can assume a Wheel in a repository. >>> >>> One of the problems with the current system, is that we have no >>> mechanism by >>> which to determine dependencies of a source distribution without >>> downloading >>> the file and executing some potentially untrusted code. This makes >>> dependency >>> resolution harder and much much slower than if we could read that >>> information >>> statically from a source distribution. This PEP doesn't offer anything >>> in the >>> way of solving this problem. >>> >> >> Isn't that what the requirements and requirements-file fields in the >> _pypackage file provide? Only if you use that requirements-dynamic would it >> require execcuting arbitrary code to gather dependency information, or am I >> missing something? >> >> >>> >>> To a similar tune, this PEP also doesn't make it possible to really get >>> at >>> any other metadata without executing software. This makes it pratically >>> impossible to safely inspect an unknown or untrusted package to >>> determine what >>> it is and to get information about it. Right now PyPI relies on the >>> uploading >>> tool to send that information alongside of the file it is uploading, but >>> honestly what it should be doing is extracting that information from >>> within the >>> file. This is sort of possible right now since distutils and setuptools >>> both >>> create a static metadata file within the source distribution, but we >>> don't rely >>> on that within PyPI because that information may or may not be accurate >>> and may >>> or may not exist. However the twine uploading tool *does* rely on that, >>> and >>> this PEP would break the ability for twine to upload a package without >>> executing arbitrary code. >>> >> >> Isn't that only if you use the dynamic fields? >> >> >>> >>> Overall, I don't think that this really solves most of the foundational >>> problems with the current format. Largely it feels that what it achieves >>> is >>> shuffling around some logic (you need to create a hook that you >>> reference from >>> within a .cfg file instead of creating a setuptools extension or so) but >>> without fixing most of the problems. The largest benefit I see to >>> switching to >>> this right now is that it would enable us to have build time >>> dependencies that >>> were controlled by pip rather than installed implicitly via the >>> execution of >>> the setup.py. That doesn't feel like a big enough benefit to me to do a >>> mass >>> shakeup of what we recommend and tell people to do. Having people adjust >>> and >>> change and do something new requires effort, and we need something to >>> justify >>> that effort to other people and I don't think that this PEP has >>> something we >>> can really use to justify that effort. >>> >> >> From my naive perspective this proposal seems to help push forward a >> decoupling of building using distutils/setuptools as the only way you can >> properly build Python projects (which is what I think we are all after) and >> will hopefully eventually free pip up to simply do orchestration. >> >> >>> >>> I *do* think that there is a core of some ideas here that are valuable, >>> and in >>> fact are similar to some ideas I've had. The main flaw I see here is >>> that it >>> doesn't really fix sdists, it takes a solution that would work for VCS >>> checkouts and then reuses it for sdists. In my mind, the supported flow >>> for >>> package installation would be: >>> >>> VCS/Bare Directory -> Source Distribution -> Wheel >>> >>> This would (eventually) be the only path that was supported for >>> installation >>> but you could "enter" the path at any stage. For example, if there is a >>> Wheel >>> already available, then you jump right on at the end and just install >>> that, if >>> there is a sdist available then pip first builds it into a wheel and then >>> installs that, etc. >>> >>> I think your PEP is something like what the VCS/Bare Directory to sdist >>> tooling >>> could look like, but I don't think it's what the sdist to wheel path >>> should >>> look like. >>> >> >> Is there another proposal I'm unaware for the sdist -> wheel step that is >> build tool-agnostic? I'm all for going with the best solution but there has >> to be an actual alternative to compare against and I don't know of any >> others right now and this proposal does seem to move things forward in a >> reasonable fashion. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 22:25:14 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 16:25:14 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: <3A3FFB47-34F3-4F46-8B7F-FDFD184BFE8A@stufft.io> I'm speaking to the proposal as currently written. It's not completely off base for what I think a solution could be. I think part of the problem though is we don't have all the building blocks figured out and standardized yet. PEP426 has stalled (I plan to pick it up once Warehouse is deployed but someone else could do that) and we should probably get the environment markers sorted out because they are going to be even more important for a static sdist. I think that the current proposal conflates a bcs checkout with a sdist too much. As Paul said, sdists are generated and are not generally for human consumption or creation. We should strictly define what it looks like l, but have pluggable build systems. I don't think we need anything more complex than the ability for a sdist to say that it gets built using X hook. Give that hook a standard API and then any tool can be a first class build tool. Sent from my iPhone > On Oct 2, 2015, at 4:14 PM, Paul Moore wrote: > > That's a fair point. But I don't see any reason Nathaniel's proposal > *couldn't* be that solution. I'd want to see the sdist format required > to include static metadata, and the metadata format to be PEP 426, but > neither of those seem incompatible with the ideas behind the proposal. > > Maybe I'm missing something massive, but I don't see a *huge* gap > between this proposal and the basic ideas behind the metabuild concept From njs at pobox.com Fri Oct 2 22:27:49 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 13:27:49 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 1:24 PM, Daniel Holth wrote: > Instead of multiple hooks why not a single object exposed through an entry > point that has several optional methods? > > NO > > [build] > requirements = "flit" > build-wheels = "flit.pypackage_hooks:build_wheels" > build-in-place = "flit.pypackage_hooks:build_in_place" > > YES > > [build] > build-system=flit > > class ABuildHook: > def build_wheels(self): ... > > entry_points = {'new.sdist.build.hooks': ['flit= some_module:ABuildHook']} Mostly because this rules out the possibility of shipping the build hook inside the package being built, which seems like a useful option to leave open (e.g. some people will probably want to vendor their build system for more-or-less legitimate reasons). Notice that the _pypackage directory is added to sys.path before resolving hooks. -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Fri Oct 2 22:42:13 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 21:42:13 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 21:19, Nathaniel Smith wrote: >> One of the problems with the current system, is that we have no mechanism by >> which to determine dependencies of a source distribution without downloading >> the file and executing some potentially untrusted code. This makes dependency >> resolution harder and much much slower than if we could read that information >> statically from a source distribution. This PEP doesn't offer anything in the >> way of solving this problem. > > What are the "dependencies of a source distribution"? Do you mean the > runtime dependencies of the wheels that will be built from a source > distribution? > > If you need that metadata to be statically in the sdist, then you > might as well give up now because it's simply impossible. > > As the very simplest example, every package that uses the numpy C API > gets a runtime dependency on "numpy >= [whatever version happened to > be installed on the *build* machine]". There are plenty of more > complex examples too (e.g. ones that involve build/configure-time > decisions about whether to rely on particular system libraries, or > build/configure-time decisions about whether particular packages > should even be built). I'm really not at all clear what you're saying here. It's quite possible that those of us who don't understand the complexities of the scientific/numpy world are missing something important, but if so it would be useful if you could spell out the problems in detail. >From my point of view, it's not a source distribution or a binary distribution that depends on something (numpy or whatever) - it's the *project*. If project foo needs numpy to work, it depends on numpy. If it depends on features in numpy 1.9, it depends on numpy>=1.9. Optional dependencies are covered by extras, and environment specific dependencies are covered by environment markers.[1] That remains true for all wheels that are built from that project, for whatever platform using whatever tools. It should also be true for the source distribution, precisely *because* it's independent of the build process. I can understand that a binary wheel may need a certain set of libraries installed - but that's about the platform tags that are part of the wheel definition, not about dependencies. Platform tags are an ongoing discussion, and a good example of a partial solution that needs to be extended, certainly, but they aren't really relevant in any way that I can see to how the build chain works. You seem to be saying that wheels need a dependency on "the version of numpy they were built against". That sounds to me like a binary compatibility requirement that platform tags are intended to cover. It may well be a requirement that platform tags need significant enhancement (maybe even redesign) to cover, but it's not a dependency in the sense that pip and the packaging PEPs use the term. And if my understanding is correct, I'm against trying to fit that information into a dependency simply to work around the current limitations of the platform tag mechanism. I'm all in favour of new initiatives to make progress in areas that are currently stalled (we definitely need people willing to contribute) but we really don't have the resources to throw away the progress we've already made. Even if some of the packaging PEPs are still works in progress, what is there represents an investment we need to build on, not bypass. Paul [1] If extras and environment markers don't cover the needs of scientific modules, we need some input into their design from the scientific community. But again, let's not throw away the work that's already done. From chris.barker at noaa.gov Fri Oct 2 23:03:33 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 14:03:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: I'm not sure if I understand what Nathaniel is getting at, but... > As the very simplest example, every package that uses the numpy C API > > gets a runtime dependency on "numpy >= [whatever version happened to > > be installed on the *build* machine]". > From my point of view, it's not a source distribution or a binary > distribution that depends on something (numpy or whatever) - it's the > *project*. If project foo needs numpy to work, it depends on numpy. If > it depends on features in numpy 1.9, it depends on numpy>=1.9. > Here is the gap (I'm a bit confused about what a "project" is -- so I"ll use the term "package", meaning a python package). A given package might depend on numpy, as you say, and it may work with all numpy versions 1.6 to 1.9. Fine, so we specify that in install_requires. And this shodl be the dependency in the sdist, too. If the package is pur python, this is fine and done. But if the package has some extensions code that used the numpy C API ( a very common occurrence), then when it is built, it will only work (reliably) with the version of numpy it was built with. So the project itself, and the sdist depend on numpy >=1.6, but a build binary wheel depends on numpy == 1.7 (for instance). Which requires a binary (wheel) dependency that is somewhat different than the source dependency. In a way, this is a lot like a python package that may work fine on py2.7 to py3.5, but a binary wheel is for py3.4 specifically (yes?) Note that conda, being developed originally for scipy, has packages like: gsw-3.0.3-np19py27_0.tar.bz2 so this binary depends specifically on py2.7 and numpy1.9 but once you get beyond python_numpy, you cold get a REALLY BIG matrix of possible version combnations! nevertheless, I think it would be really helpful to have the concept of a binary dependency that is distinct from the "package" dependency. Maybe down to "this wheel depends on this other particular wheel" - not only version, but also wheel specific. That would open the door to making wheels for binary non-python dependencies like libpng: then other wheels would depend on the libpng wheel, but different wheels built from the same package might have those dependencies handled differently, particularly on different platforms. I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that > needs to be extended, certainly, but they aren't really relevant in > any way that I can see to how the build chain works. > I haven't followed the discussion well, but I really think platform tags are never going to be specific enough to handle these use-cases And these really ARE dependencies, even if they aren't pure-python ones... And if my > understanding is correct, I'm against trying to fit that information > into a dependency simply to work around the current limitations of the > platform tag mechanism. > I can't imagine how you could extend platform tags to cover all of this -- but maybe I'm being unimaginative... BTW, numpy is nearly unique here -- I can't think of any other package that is a python package with a C API that is widely used -- most other cases are either a python package OR a regular old compiled lib. Which is probably why conda can get away with essentially special casing it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 23:30:29 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 17:30:29 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 4:20:00 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Fri, Oct 2, 2015 at 4:58 AM, Donald Stufft wrote: > > > > One of the problems with the current system, is that we have no mechanism by > > which to determine dependencies of a source distribution without downloading > > the file and executing some potentially untrusted code. This makes dependency > > resolution harder and much much slower than if we could read that information > > statically from a source distribution. This PEP doesn't offer anything in the > > way of solving this problem. > > What are the "dependencies of a source distribution"? Do you mean the > runtime dependencies of the wheels that will be built from a source > distribution? > > If you need that metadata to be statically in the sdist, then you > might as well give up now because it's simply impossible. I don?t believe this is impossible. > > As the very simplest example, every package that uses the numpy C API > gets a runtime dependency on "numpy >= [whatever version happened to > be installed on the *build* machine]?. A quick, off the cuff idea here is to allow additional ABI declarations and stop trying to use the same system for API and ABI. A source distribution can't have ABI dependencies, only wheels can and an installation has to be valid for both API and any relevant ABI requirements. > There are plenty of more > complex examples too (e.g. ones that involve build/configure-time > decisions about whether to rely on particular system libraries, or > build/configure-time decisions about whether particular packages > should even be built). I don't think build/configure-time decisions are great ideas as it's near impossible to actually depend on them. For example, take Pillow, Pillow will conditionally compile against libraries that enable it to much around with PNGs. However, if I *need* Pillow with PNG support, I don't have any mechanism to declare that. If instead, builds were *not* conditional and Pillow instead split it's PNG capabilities out into it's own package called say, Pillow-PNG which also did not conditionally compile against anything, but unconditionally did, then we could add in something like having Pillow declare a "weak" dependency on Pillow-PNG where we attempt to get it by default if possible, but we will skip installing it if we can't locate/build it. If you combine this with Extras, you could then easily make it so that people can depend on particular conditional features by doing something like ``Pillow[PNG]`` in their dependency metadata. > > For comparison, here's the Debian source package metadata: > https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles > Note that the only mandatory fields are format version / package name > / package version / maintainer / checksums. The closest they come to > making promises about the built packages are the Package-List and > Binary fields which provide a optional hint about what binary packages > will be built, and are allowed to contain lies (e.g. they explicitly > don't guarantee that all the binary packages named will actually be > produced on every architecture). The only kind of dependencies that a > source package can declare are build-depends. Debian doesn't really have "source packages" like we do, but inside of the debian/ directory is the control file which lists all of the dependency information (or explicitly lists a placeholder where something can't be statically declared). > > > To a similar tune, this PEP also doesn't make it possible to really get at > > any other metadata without executing software. This makes it pratically > > impossible to safely inspect an unknown or untrusted package to determine what > > it is and to get information about it. Right now PyPI relies on the uploading > > tool to send that information alongside of the file it is uploading, but > > honestly what it should be doing is extracting that information from within the > > file. This is sort of possible right now since distutils and setuptools both > > create a static metadata file within the source distribution, but we don't rely > > on that within PyPI because that information may or may not be accurate and may > > or may not exist. However the twine uploading tool *does* rely on that, and > > this PEP would break the ability for twine to upload a package without > > executing arbitrary code. > > Okay, what metadata do you need? We certainly could put name / version > kind of stuff in there. We left it out because we weren't sure what > was necessary and it's easy to add later, but anything that's needed > by twine fits neatly into the existing text saying that we should > "include extra metadata in source distributions if it helps solve > specific problems that are unique to distribution" -- twine uploads > definitely count. Everything that isn't specific to a built wheel. Look at the previously accepted metadata specs as well as PEP 426. If you're not including a field that was included in one of those, there should be a rationale for why that field is no longer being included. > > > Overall, I don't think that this really solves most of the foundational > > problems with the current format. Largely it feels that what it achieves is > > shuffling around some logic (you need to create a hook that you reference from > > within a .cfg file instead of creating a setuptools extension or so) but > > numpy.distutils is the biggest distutils/setuptools extension around, > and everyone involved in maintaining it wants to kill it with fire > :-). That's a problem? Well, it's not really what I call setuptools extension because it doesn't use the extension points in setuptools to do it's work. It expects you to just ``import numpy.distutils`` at the top of your ``setup.py`` and use that, which means that it breaks things like pip because we don't have a way to know that we need to install numpy first. > > > without fixing most of the problems. The largest benefit I see to switching to > > this right now is that it would enable us to have build time dependencies that > > were controlled by pip rather than installed implicitly via the execution of > > the setup.py. > > Yes, this problem means that literally every numerical python package > currently has a broken setup.py. Because numpy.distutils wasn't written to plug into setuptools. If it had been they wouldn't be.? > > > That doesn't feel like a big enough benefit to me to do a mass > > shakeup of what we recommend and tell people to do. Having people adjust and > > change and do something new requires effort, and we need something to justify > > that effort to other people and I don't think that this PEP has something we > > can really use to justify that effort. > > The end-user adjustment is teaching people to switch to always using > pip to install packages -- this seems like something we will certainly > do sooner or later, so we might as well get started. > > And it's already actually the right thing to do -- if you use > 'setup.py install' then you get a timebomb in your venv where later > upgrades may leave you with a broken package :-(. (This is orthogonal > to the actual PEP.) In the long run, the idea that every package has > to contain code that knows how to implement installation in very > possible configuration (--user? --single-version-externally-managed?) > is clearly broken, and teaching people to use 'pip install' is > obviously the only sensible alternative. > Sorry, ambigous end-user in my original statement. I don't mean the end-end-users (e.g. the people executing ``pip install``), I mean the packagers. I'm going to re-read the originall proposal and try to point out more actionable feedback shortly. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Fri Oct 2 23:36:00 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 17:36:00 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 4:24:38 PM, Daniel Holth (dholth at gmail.com) wrote: > > We need to embrace partial solutions and the fine folks who propose > them so the whole packaging ecosystem can have some progress. > PEP 438 may not be a good analogue to adding a new sdist format since > the latter only adds new things that you can do. A new sdist format > will inconvenience a much more limited set of people, mainly > the pip authors and the OS package maintainers. Packaging formats are a bit like HTTP, "move fast and break things" isn't super great because anytime you add a new format, you have to support that *forever* (or long enough to basically be forever). It's unlike a software package where you can, for example, deprecate and remove something and if someone is still using that feature they just continue to use an older version of your library. Packagers have no control over what version of the tools people are using and they are discouraged (or even disallowed) to go back in time and correct their older packages. Being conservative in what we accept as an accepted standard and implement in the main tools is a good thing, being liberal in experiments and what we try and having people try out partial solutions and figure out what works and what doesn't without making it a standard or putting it in pip is also a good thing. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Fri Oct 2 23:37:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 14:37:29 -0700 Subject: [Distutils] pip install from source dir?? Message-ID: I can't seem to find a way to pip install from a source dir -- not a tarball. Why would I want to do that? In this case, it's because I'm trying to build conda packages for python packages that have compiled C code, and are not using setuptools. On Windows, you need to use setuptools in order to use the "MS compiler for Python 2.7". So a straight "setup.py build" fails. But pip injects setuptools, so that "pip install" does work. But conda has already unpacked the source distribution, so I'd like to point pip at that, and can't find a way to do that (I could have sworn it was possible...) Do I have to build an sdist, and then point pip at that?? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 23:39:05 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 17:39:05 -0400 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: ``pip install path/to/directory/with/setup.py/in/it/`` On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) wrote: > I can't seem to find a way to pip install from a source dir -- not a > tarball. > > Why would I want to do that? > > In this case, it's because I'm trying to build conda packages for python > packages that have compiled C code, and are not using setuptools. > > On Windows, you need to use setuptools in order to use the "MS compiler for > Python 2.7". So a straight "setup.py build" fails. > > But pip injects setuptools, so that "pip install" does work. But conda has > already unpacked the source distribution, so I'd like to point pip at that, > and can't find a way to do that (I could have sworn it was possible...) > > Do I have to build an sdist, and then point pip at that?? > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From njs at pobox.com Fri Oct 2 23:45:55 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 14:45:55 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 1:03 PM, Paul Moore wrote: > On 2 October 2015 at 20:02, Marcus Smith wrote: >>> So wouldn't they then download the sdist, build a wheel as an >>> intermediate, and then generate the .deb file? >> >> the new goal I think was to have standardized metadata immediately available >> in an sdist, and get away from the model, that you had to run a build step, >> before you had a metadata artifact. >> so here, you'd have to build a wheel (potentially one with binary >> extensions) just to know what the metadata is? that doesn't sound right. > > I'm uncomfortable with the fact that the proposed sdist format has > more or less no metadata of its own (even the filename format is only > a recommendation) so (for example) if someone does "pip install > foo==1.0" I don't see how pip can find a suitable sdist, if no wheel > is available. About the filename thing: The reason that the draft makes the inclusion of package/version info a SHOULD instead of a MUST is that regardless of what the spec says, all decent installation tools are going to support doing things like curl https://github.com/numpy/numpy/archive/master.zip -O numpy-master.zip pip install numpy-master.zip So we can either handle that by saying that "numpy-master.zip" is an sdist, just not one that we would allow on PyPI (which is what the current draft does), or we could handle it by saying that numpy-master.zip is almost-but-not-quite an sdist, and handling it is a commonly supported extension to the standard. Doesn't really matter that much either way -- just a matter of terminology. Either way the sdists are PyPI are obviously going to be named -.. For sdists that do have a name/version: it's not really crucial to the proposal that name/version are only in the filename -- they could be repeated inside the file as well. Given that the version number in particular is something that usually would need to be computed at sdist-build-time (from a __version__.py or whatever -- it's very common that the source-of-truth for version numbers is not static), then leaving it out of the static metadata is nice because it makes sdist-building-code much simpler -- 90% of the time you could just keep the static metadata file instead of having to rewrite it for each sdist. But that's just an engineering trade-off, it's not crucial to the concept. > I would rather see an sdist format that can be introspected *without* > running code or a build tool. Installers and packaging tools like pip > need to be able to do that - one of the biggest issues with pip's > current sdist handling is that it can't make any meaningful decisions > before building at least the egg-info. Another way to look at this is to say that pip's current handling is proof that the build-to-get-metadata strategy is viable :-). It would indeed be nice if this weren't necessary, but the python packaging ecosystem has a long history of trying to make simplifying assumptions that turn out to bite us later... I think this is one of those. Note that for making installation decisions, name + version aren't enough: you also need full dependency information. And dependency information is definitely not fixed at sdist-creation-time. > Ultimately the question for me is at what point do we require > packaging tools like pip (and ad-hoc distribution analysis scripts - I > write a lot of those!) to run code from the package in order to > continue? I'd like to be able to know, at a minimum, the package name > and version, as those are needed to make decisions on whether there is > a conflict with an already installed version. > > Basically, why aren't you using a PEP 426-compatible metadata format > in the sdist (there's no reason not to, you just have to mandate that > tools used to build sdists generate that form of metadata file)? You > could also say that source trees SHOULD store metadata in the > _pypackage directory (in an appropriate defined format, maybe one more > suited to human editing than JSON) and tools that work on source trees > (build tools, things that create sdists) SHOULD use that data as the > definitive source of metadata, rather than using their own > configuration. I don't see a problem with allowing source trees to > have some flexibility, but sdists are tool-generated and so could > easily be required to contain static metadata in a standard format. > >>> Is there another proposal I'm unaware for the sdist -> wheel step that is >>> build tool-agnostic? >> >> PEP426 talks about it some >> https://www.python.org/dev/peps/pep-0426/#metabuild-system > > While the metabuild system is a good long-term goal, I'll take > something that's being developed now over a great idea that no-one has > time to work on... Wheel came about because Daniel just got on with > it. > > Having said that, I very strongly prefer a sdist proposal that's > compatible with PEP 426 (at least to the extent that tools like wheel > already support it). Throwing away all of the work already done on PEP > 426 doesn't seem like a good plan. Nothing is being thrown away -- the proposal is just that sdists and wheels are different things, so we should think of PEP 426 as wheel metadata, rather than all metadata. -n -- Nathaniel J. Smith -- http://vorpus.org From chris.barker at noaa.gov Fri Oct 2 23:49:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 14:49:14 -0700 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:39 PM, Donald Stufft wrote: > ``pip install path/to/directory/with/setup.py/in/it/`` WOW!! Sorry for being such an idiot -- I tried that, I really did -- a couple times, and it failed. but now it works fine, just like you'd expect. I have no idea what was doing before. a small doc patch might be in order -- I actually couldn't find that anywhere. Though looking now, I see: pip install [options] [-e] ... I must have gotten that tied to the -e option in my head, and didn't notice that the -e is optional... Sorry for the noise: Thanks, -CHB > On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) > wrote: > > I can't seem to find a way to pip install from a source dir -- not a > > tarball. > > > > Why would I want to do that? > > > > In this case, it's because I'm trying to build conda packages for python > > packages that have compiled C code, and are not using setuptools. > > > > On Windows, you need to use setuptools in order to use the "MS compiler > for > > Python 2.7". So a straight "setup.py build" fails. > > > > But pip injects setuptools, so that "pip install" does work. But conda > has > > already unpacked the source distribution, so I'd like to point pip at > that, > > and can't find a way to do that (I could have sworn it was possible...) > > > > Do I have to build an sdist, and then point pip at that?? > > > > -CHB > > > > > > > > -- > > > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker at noaa.gov > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Oct 3 00:00:48 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 15:00:48 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:45 PM, Nathaniel Smith wrote: > > I would rather see an sdist format that can be introspected *without* > > running code or a build tool. indeed -- this has come up a lot on this list, for binary dists, too, of course. but "build script as turing complete" [someone posted that in this thread...] requirement is there, too... but maybe we can get a long way with convention, without changing the tools. A setup.py is typically a bunc of stuff that builds up the setup, and then a call to setup.py, passing in teh objects created to various paramters. But if we tried to establish a convention that your setup.py would do: a_bunch_of_stuff_that_bulit_up_a_dict_of_options then setup(**setup_options_dict) then the_options_dict could be introspected without actually createing a seteup object. and in the easy cases, the_options_dict could be completely declarative, and maybe even stored in another file. and in the complex cases, it could still have all the parts that could be pre-declared declarative, so setup.py would be: setup_options_dict = load_options_dict("setup_options.txt") [do assorted complex stuff to edit/add to setup_options_dict] setup(**setup_options) would this move us toward an as-declarative-as-possible setup.py? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 3 00:15:38 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 15:15:38 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 1:42 PM, Paul Moore wrote: > On 2 October 2015 at 21:19, Nathaniel Smith wrote: >>> One of the problems with the current system, is that we have no mechanism by >>> which to determine dependencies of a source distribution without downloading >>> the file and executing some potentially untrusted code. This makes dependency >>> resolution harder and much much slower than if we could read that information >>> statically from a source distribution. This PEP doesn't offer anything in the >>> way of solving this problem. >> >> What are the "dependencies of a source distribution"? Do you mean the >> runtime dependencies of the wheels that will be built from a source >> distribution? >> >> If you need that metadata to be statically in the sdist, then you >> might as well give up now because it's simply impossible. >> >> As the very simplest example, every package that uses the numpy C API >> gets a runtime dependency on "numpy >= [whatever version happened to >> be installed on the *build* machine]". There are plenty of more >> complex examples too (e.g. ones that involve build/configure-time >> decisions about whether to rely on particular system libraries, or >> build/configure-time decisions about whether particular packages >> should even be built). > > I'm really not at all clear what you're saying here. It's quite > possible that those of us who don't understand the complexities of the > scientific/numpy world are missing something important, but if so it > would be useful if you could spell out the problems in detail. > > From my point of view, it's not a source distribution or a binary > distribution that depends on something (numpy or whatever) - it's the > *project*. If project foo needs numpy to work, it depends on numpy. If > it depends on features in numpy 1.9, it depends on numpy>=1.9. > Optional dependencies are covered by extras, and environment specific > dependencies are covered by environment markers.[1] That remains true > for all wheels that are built from that project, for whatever platform > using whatever tools. It should also be true for the source > distribution, precisely *because* it's independent of the build > process. "Project" is a pretty messy concept. Obviously in simple cases there's a one-to-one mapping between project <-> wheel <-> importable package, but this breaks down quickly in edge cases. Consider a project that provides builds multiple wheels out of the same source tree. You obviously can't expect that all of these packages will have the same dependencies. This situation is not common today for Python packages, but the only reason for that is that distutils makes it really hard to do -- it's extremely common in other package ecosystems, and the advantages are obvious. E.g., maybe numpy.distutils should be split into a separately installable package from numpy -- there's no technical reason that this should mean we are now forced to move the code for it into its own VCS repository. > I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that > needs to be extended, certainly, but they aren't really relevant in > any way that I can see to how the build chain works. (I assume that by "platform tags" you mean what PEP 426 calls "environment markers".) Environment markers are really useful for extending the set of cases that can be handled by a single architecture-dependent wheel. And they're a good fit for that environment, given that wheels can't contain arbitrary code. But they're certainly never going to be adequate to provide a single static description of every possible build configuration of every possible project. And installing an sdist already requires arbitrary code execution, so it doesn't make sense to try to build some elaborate system to avoid arbitrary code execution just for the dependency specification. You're right that in a perfect future world numpy C API related dependencies would be handling by some separate ABI-tracking mechanism similar to how the CPython ABI is tracked, so here are some other examples of why environment markers are inadequate: In the future it will almost certainly be possible to build numpy in two different configurations: one where it expects to find BLAS inside a wheel distributed for this purpose (e.g. this is necessary to provide high-quality windows wheels), and one where it expects to find BLAS installed on the system. This decision will *not* be tied to the platform, but be selectable at build time. E.g., on OS X there is a system-provided BLAS library, but it has some issues. So the default wheels on PyPI will probably act like windows and depend on a BLAS-package that we control, but there will also be individual users who prefer to build numpy in the configuration where it uses the system BLAS, so we definitely need to support both options on OS X. Now the problem: There will never be a single environment marker that you can stick into a wheel or sdist that says "we depend on the 'pyblas' package if the system is OS X (ok) and the user set this flag in this configuration file during the build process (wait wut)". Similarly, I think someone was saying in a discussion recently that lxml supports being built either in a mode where it requires libxml be available on the system, or else it can be statically linked. Even if in the future we start having metadata that lets us describe dependencies on external system libraries, it's never going to be the case that we can put the *same* dependency metadata into wheels that are built using these two configurations. > You seem to be saying that wheels need a dependency on "the version of > numpy they were built against". That sounds to me like a binary > compatibility requirement that platform tags are intended to cover. It > may well be a requirement that platform tags need significant > enhancement (maybe even redesign) to cover, but it's not a dependency > in the sense that pip and the packaging PEPs use the term. And if my > understanding is correct, I'm against trying to fit that information > into a dependency simply to work around the current limitations of the > platform tag mechanism. > > I'm all in favour of new initiatives to make progress in areas that > are currently stalled (we definitely need people willing to > contribute) but we really don't have the resources to throw away the > progress we've already made. Even if some of the packaging PEPs are > still works in progress, what is there represents an investment we > need to build on, not bypass. > > Paul > > [1] If extras and environment markers don't cover the needs of > scientific modules, we need some input into their design from the > scientific community. But again, let's not throw away the work that's > already done. As far as sdists go, you can either cover 90% of the cases by building increasingly elaborate metadata formats, or you can cover 100% of the cases by keeping things simple... -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sat Oct 3 00:17:57 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 15:17:57 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:36 PM, Donald Stufft wrote: > On October 2, 2015 at 4:24:38 PM, Daniel Holth (dholth at gmail.com) wrote: >> > We need to embrace partial solutions and the fine folks who propose >> them so the whole packaging ecosystem can have some progress. >> PEP 438 may not be a good analogue to adding a new sdist format since >> the latter only adds new things that you can do. A new sdist format >> will inconvenience a much more limited set of people, mainly >> the pip authors and the OS package maintainers. > > Packaging formats are a bit like HTTP, "move fast and break things" isn't super > great because anytime you add a new format, you have to support that *forever* > (or long enough to basically be forever). Right: this is why it's important for me to make the case that putting full PEP 426 metadata in sdists is not just temporarily inconvenient, but actually conceptually the wrong thing to do. -n -- Nathaniel J. Smith -- http://vorpus.org From wes.turner at gmail.com Sat Oct 3 00:26:07 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 2 Oct 2015 17:26:07 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 2, 2015 5:18 PM, "Nathaniel Smith" wrote: > > On Fri, Oct 2, 2015 at 2:36 PM, Donald Stufft wrote: > > On October 2, 2015 at 4:24:38 PM, Daniel Holth (dholth at gmail.com) wrote: > >> > We need to embrace partial solutions and the fine folks who propose > >> them so the whole packaging ecosystem can have some progress. > >> PEP 438 may not be a good analogue to adding a new sdist format since > >> the latter only adds new things that you can do. A new sdist format > >> will inconvenience a much more limited set of people, mainly > >> the pip authors and the OS package maintainers. > > > > Packaging formats are a bit like HTTP, "move fast and break things" isn't super > > great because anytime you add a new format, you have to support that *forever* > > (or long enough to basically be forever). > > Right: this is why it's important for me to make the case that putting > full PEP 426 metadata in sdists is not just temporarily inconvenient, > but actually conceptually the wrong thing to do. pydist.jsonld would be a helpful metadata file to add to an sdist, as well URIs to dependencies with rule/constraints in the reified edges drawn from e.g. - setup.py - requirements.txt - requirements.lock/versions/freeze.txt - requirements.peep.txt - requirements-dev/test/docs.txt - [versions.cfg] > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Oct 3 00:26:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 23:26:47 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 22:45, Nathaniel Smith wrote: >> I'm uncomfortable with the fact that the proposed sdist format has >> more or less no metadata of its own (even the filename format is only >> a recommendation) so (for example) if someone does "pip install >> foo==1.0" I don't see how pip can find a suitable sdist, if no wheel >> is available. > > About the filename thing: > > The reason that the draft makes the inclusion of package/version info > a SHOULD instead of a MUST is that regardless of what the spec says, > all decent installation tools are going to support doing things like > > curl https://github.com/numpy/numpy/archive/master.zip -O numpy-master.zip > pip install numpy-master.zip > > So we can either handle that by saying that "numpy-master.zip" is an > sdist, just not one that we would allow on PyPI (which is what the > current draft does), or we could handle it by saying that > numpy-master.zip is almost-but-not-quite an sdist, and handling it is > a commonly supported extension to the standard. Doesn't really matter > that much either way -- just a matter of terminology. Either way the > sdists are PyPI are obviously going to be named > -.. OK, that's a good point, and I never felt it was crucial that the name/version be encoded in the filename. But having them in some form of static metadata should be mandatory. Your _pypackage.cfg doesn't contain the package name or version, so how would I get them without running code? That's my real point. > Given that the version number in > particular is something that usually would need to be computed at > sdist-build-time (from a __version__.py or whatever -- it's very > common that the source-of-truth for version numbers is not static), > then leaving it out of the static metadata is nice because it makes > sdist-building-code much simpler -- 90% of the time you could just > keep the static metadata file instead of having to rewrite it for each > sdist. But that's just an engineering trade-off, it's not crucial to > the concept. I'm willing to allow for it being non-static in the source tree, but not in the sdist. >> I would rather see an sdist format that can be introspected *without* >> running code or a build tool. Installers and packaging tools like pip >> need to be able to do that - one of the biggest issues with pip's >> current sdist handling is that it can't make any meaningful decisions >> before building at least the egg-info. > > Another way to look at this is to say that pip's current handling is > proof that the build-to-get-metadata strategy is viable :-). Not if you look at the bug reports for pip that can be traced back to needing to run setup.py egg-info to get metadata, or other variations on not having static introspectable metadata in sdists. Paul From erik.m.bray at gmail.com Sat Oct 3 00:37:00 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 2 Oct 2015 18:37:00 -0400 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 5:39 PM, Donald Stufft wrote: > ``pip install path/to/directory/with/setup.py/in/it/`` In particular I install most of my packages from source these days by cd'ing into the source and $ pip install . That's all it takes. Works with -e too. Erik > On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) wrote: >> I can't seem to find a way to pip install from a source dir -- not a >> tarball. >> >> Why would I want to do that? >> >> In this case, it's because I'm trying to build conda packages for python >> packages that have compiled C code, and are not using setuptools. >> >> On Windows, you need to use setuptools in order to use the "MS compiler for >> Python 2.7". So a straight "setup.py build" fails. >> >> But pip injects setuptools, so that "pip install" does work. But conda has >> already unpacked the source distribution, so I'd like to point pip at that, >> and can't find a way to do that (I could have sworn it was possible...) >> >> Do I have to build an sdist, and then point pip at that?? >> >> -CHB >> >> >> From njs at pobox.com Sat Oct 3 00:47:15 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 15:47:15 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 3:26 PM, Paul Moore wrote: > On 2 October 2015 at 22:45, Nathaniel Smith wrote: >>> I'm uncomfortable with the fact that the proposed sdist format has >>> more or less no metadata of its own (even the filename format is only >>> a recommendation) so (for example) if someone does "pip install >>> foo==1.0" I don't see how pip can find a suitable sdist, if no wheel >>> is available. >> >> About the filename thing: >> >> The reason that the draft makes the inclusion of package/version info >> a SHOULD instead of a MUST is that regardless of what the spec says, >> all decent installation tools are going to support doing things like >> >> curl https://github.com/numpy/numpy/archive/master.zip -O numpy-master.zip >> pip install numpy-master.zip >> >> So we can either handle that by saying that "numpy-master.zip" is an >> sdist, just not one that we would allow on PyPI (which is what the >> current draft does), or we could handle it by saying that >> numpy-master.zip is almost-but-not-quite an sdist, and handling it is >> a commonly supported extension to the standard. Doesn't really matter >> that much either way -- just a matter of terminology. Either way the >> sdists are PyPI are obviously going to be named >> -.. > > OK, that's a good point, and I never felt it was crucial that the > name/version be encoded in the filename. But having them in some form > of static metadata should be mandatory. Your _pypackage.cfg doesn't > contain the package name or version, so how would I get them without > running code? That's my real point. Well, first, it's just not possible for a devel snapshot like numpy-master.zip or a VCS checkout to contain static version metadata, since the actual version of the generated wheels *will* be determined by running arbitrary code (e.g. 'git rev-parse HEAD'). So we're only talking about tagged/released source trees. Then the argument would be, what are you going to do with that name/version information? If the answer is "decide to install it", then (a) if you want to support installation from VCS snapshots (and you do) then your tool already has to support running arbitrary code to get the version number, and (b) installing the package will also certainly require running arbitrary code even if you have a nice official-release sdist, so. OTOH the twine upload case that Donald mentioned is a good example of an operation that might actually want some metadata from release sdists specifically :-). I'm not opposed to adding it if there's a clear use case, I just don't think we should try to shove every piece of wheel metadata into the sdist without a clear understanding of how they make sense and solve a problem *for sdists*. [...] >>> I would rather see an sdist format that can be introspected *without* >>> running code or a build tool. Installers and packaging tools like pip >>> need to be able to do that - one of the biggest issues with pip's >>> current sdist handling is that it can't make any meaningful decisions >>> before building at least the egg-info. >> >> Another way to look at this is to say that pip's current handling is >> proof that the build-to-get-metadata strategy is viable :-). > > Not if you look at the bug reports for pip that can be traced back to > needing to run setup.py egg-info to get metadata, or other variations > on not having static introspectable metadata in sdists. That sounds interesting! Can you elaborate? Links? I know that one unpleasant aspect of the current design is that the split between egg-info and actual building creates the possibility for time-of-definition-to-time-of-use bugs, where the final wheel hopefully matches what egg-info said it would, but in practice there could be skew. (Of course this is true in any system which represents this information in more than one place -- e.g. in sdist metadata and also in wheel metadata -- but right now it's particularly bad in cases where you can't actually get all the arguments you want to pass to setup() without running some code, but the code you need to run needs to be fetched via setup_requires=..., so you have to lie to setup() during the egg-info operation and hope that everything will work out in the end.) This is the motivation for the draft PEP to dropping egg-info as a separate operation. But there are certainly people who know more about the internal details of what pip needs than I do, and I'd love to hear more. -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Sat Oct 3 00:52:40 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 23:52:40 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 23:15, Nathaniel Smith wrote: > "Project" is a pretty messy concept. Obviously in simple cases there's > a one-to-one mapping between project <-> wheel <-> importable package, > but this breaks down quickly in edge cases. I mistakenly used "project" in an attempt to avoid confusion resulting from me using the word "distribution" as a more general term than the way you were using "source distribution" or "binary distribution". Clearly I failed and made things more confusing. I use the term "distribution" in the sense used here https://packaging.python.org/en/latest/glossary/#term-distribution-package. Note that this is in contrast to the terms "source distribution" and "binary distribution" or "built distribution" in the same page. Sorry for confusing things. I'll stick to the terminology as in the PUG glossary from now on. > Consider a project that provides builds multiple wheels out of the > same source tree. You obviously can't expect that all of these > packages will have the same dependencies. Correct. But a distribution can and should (I believe) have the same dependencies for all of the source and built distributions derived from it. > This situation is not common today for Python packages, but the only > reason for that is that distutils makes it really hard to do -- it's > extremely common in other package ecosystems, and the advantages are > obvious. E.g., maybe numpy.distutils should be split into a separately > installable package from numpy -- there's no technical reason that > this should mean we are now forced to move the code for it into its > own VCS repository. I'm lost here, I'm afraid. Could you rephrase this in terms of the definitions from the PUG glossary? It sounds to me like the VCS repository is the project, which contains multiple distributions. I don't see how that's particularly hard. Each distribution just has its own subdirectory (and setup.py) in the VCS repository... > (I assume that by "platform tags" you mean what PEP 426 calls > "environment markers".) Nope, I mean as defined in PEP 425. The platform tag is part of the compatibility tag. Maybe I meant the ABI tag, I don't really follow the distinctions. > Environment markers are really useful for extending the set of cases > that can be handled by a single architecture-dependent wheel. And > they're a good fit for that environment, given that wheels can't > contain arbitrary code. > > But they're certainly never going to be adequate to provide a single > static description of every possible build configuration of every > possible project. And installing an sdist already requires arbitrary > code execution, so it doesn't make sense to try to build some > elaborate system to avoid arbitrary code execution just for the > dependency specification. > > You're right that in a perfect future world numpy C API related > dependencies would be handling by some separate ABI-tracking mechanism > similar to how the CPython ABI is tracked, so here are some other > examples of why environment markers are inadequate: > > In the future it will almost certainly be possible to build numpy in > two different configurations: one where it expects to find BLAS inside > a wheel distributed for this purpose (e.g. this is necessary to > provide high-quality windows wheels), and one where it expects to find > BLAS installed on the system. This decision will *not* be tied to the > platform, but be selectable at build time. E.g., on OS X there is a > system-provided BLAS library, but it has some issues. So the default > wheels on PyPI will probably act like windows and depend on a > BLAS-package that we control, but there will also be individual users > who prefer to build numpy in the configuration where it uses the > system BLAS, so we definitely need to support both options on OS X. > Now the problem: There will never be a single environment marker that > you can stick into a wheel or sdist that says "we depend on the > 'pyblas' package if the system is OS X (ok) and the user set this flag > in this configuration file during the build process (wait wut)". > > Similarly, I think someone was saying in a discussion recently that > lxml supports being built either in a mode where it requires libxml be > available on the system, or else it can be statically linked. Even if > in the future we start having metadata that lets us describe > dependencies on external system libraries, it's never going to be the > case that we can put the *same* dependency metadata into wheels that > are built using these two configurations. This is precisely the very complex issue that's being discussed under the banner of extending compatibility tags in a way that gives a viable but practical way of distinguishing binary wheels. You can either see that as a discussion about "expanding compatibility tags" or "finding something better than compatibility tags". I don't have much of a stake in that discussion, as the current compatibility tags suit my needs fine, as a Windows user. The issues seem to be around Linux and possibly some of the complexities around binary dependencies for numerical libraries. But the key point here is that I see the solution for this as being about distinguishing the "right" wheel for the target environment. It's not about anything that should reach back to sdists. Maybe a solution will involve a PEP 426 metadata enhancement that adds metadata that's only valid in binary distributions and not in source distributions, but that's fine by me. But it won't replace the existing dependency data, which *is* valid at the sdist level. At least as far as I can see - I'm willing to be enlightened. But your argument seems to be that sdist-level dependency information should be omitted because more detailed ABI compatibility data *might* be needed at the wheel level for some packages. I don't agree with that - we still need the existing metadata, even if more might be required in specialist cases. >> [1] If extras and environment markers don't cover the needs of >> scientific modules, we need some input into their design from the >> scientific community. But again, let's not throw away the work that's >> already done. > > As far as sdists go, you can either cover 90% of the cases by building > increasingly elaborate metadata formats, or you can cover 100% of the > cases by keeping things simple... But your argument seems to be that having metadata generated from package build code is "simpler". My strong opinion, based on what I've seen of the problems caused by having metadata in an "exectable setup.py", is that static metadata is far simpler. I don't believe that the cost of changing to a new system can be justified *without* getting the benefits of static metadata. Paul From donald at stufft.io Sat Oct 3 01:04:38 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 19:04:38 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > Distutils delenda est. > I think that you should drop (from this PEP) the handling of a VCS/arbitrary directories and focus solely on creating a format for source distributions. A source distribution can (and should) be fairly strict and well defined exactly where all of the files go, what files exist and don't exist, and things of that nature (more on this later). In addition, the metadata files should be optimizes for machines to read and parse them first, for humans to read them second, and humans to write them not at all. Given the Python standard library, your metadata inside of the source distribution should (probably) be JSON. This is another reason why this should focus on the source distribution as well, because the file you put into VCS needs to be able to be written by humans. Metadata 2.0 should probably get finished before or as part of a new sdist format happening. I fell off towards the end of that and it appears that it got a lot more complex since I last looked at it. It probably needs more work. The filename should be strictly defined similarly to what Wheels have, probably something like {name}-{version}.{ext}, and like wheel it should mandate that any - characters inside of any of the fields should be escaped to a _ so that we can unambigiously parse it. It should not support arbitrary filenames because they are (almost never) actually sdists. In another email you mentioned something like the tarballs that github produces, but those are not source distributions, they are vcs exports and shouldn't be covered by this PEP. I don't believe that Python should develop anything like the Debian ability to have a single source "package" create multiple binary packages. The metadata of the Wheel *must* strictly match the metadata of the sdist (except for things that are Wheel specific). This includes things like name, version, etc. Trying to go down this path I think will make things a lot more complicated since we have a segmented archive where people have to claim particular names, otherwise how do you prevent me from registering the name "foobar" on PyPI and saying it produces the "Django" wheel? Since I think this should only deal with source distributions, then the primary thing we need is an operation that will take an unpacked source distribution that is currently sitting on the filesystem and turn it into a wheel located in a specific location. The layout for a source distribution should be specified, I think something like: ? ? . ? ? ??? meta ? ? ? ? ??? DESCRIPTION.rst ? ? ? ? ??? FORMAT-VERSION ? ? ? ? ??? LICENSE.txt ? ? ? ? ??? METADATA.json ? ? ??? src ? ? ? ? ??? my-cool-build-tool.cfg ? ? ? ? ??? mypackage ? ? ? ? ? ? ??? __init__.py I don't particularly care about the exact names, but this layout gives us two top level directories (and only two), one is a place where all of the source distribution metadata goes, and one is a src directory where all of the files for the project should go, including any relevant configuration for the build tool in use by the project. Having two directories like this eliminates the need to worry about naming collisions between the metadata files and the project itself. We should probably give this a new name instead of "sdist" and give it a dedicated extension. Perhaps we should call them "source wheels" and have the extension be something like .swhl or .src.whl. This means we don't need to worry about making the same artifact compatible with both the legacy toolchain and a toolchain that supports "source wheels". We should also probably specify a particular container format to be used for a .whl/.src.whl. It probably makes sense to simply use zip since that is what wheels use and it supports different compression algorithms internally. We probably want to at least suggest limiting compression algorithms used to Deflate and None, if not mandate that one of those two are used. We should include absolutely as much metadata as part of the static metadata inside the sdist as we can. I don't think there is any case to be made for things like name, version, summary, description, classifiers, license, keywords, contact information (author/maintainers), project URLs, etc are Wheel specific. I think there are other things which are arguably able to be specified in the sdist, but I'd need to fiddle with it to be sure. Basically any metadata that isn't included as static information will not be able to be displayed on PyPI. The metada should directly include the specifiers inside of it and shouldn't propagate the meme that pip's requirements.txt format is anything but a way to recreate a specific environment with pip. Build requirements cannot be dynamic. We don't need a "build in place" hook, you don't build source distributions in place, you build wheels with them. Another PEP that handles a VCS/non sdist directory can add things like building in place. I don't think there's ever going to be a world where pip depends on virtualenv or pyvenv. The PEP shouldn't prescribe how the tool installs the build deps or executes the build hook, though I think it should mandate that it is called with a compatible Python to the Wheel that is desired to be produced. Cross compiling can be handled later. Possibly we might want to make the hooks calling an executable instead of doing something that involves importing the hook and calling it. This would make it easier to prevent the build tool from monkeypatching the installation tool *and* make it easier for downstream redistributors to use it. If you're interested, I'm happy to directly collaborate on this PEP if it's in a github repository somewhere or something. There's an interoptability repo you can use or your own or whatever. Or you can tell me to go pound sand too and I'll just comment on posts to the ML. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Sat Oct 3 01:38:38 2015 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 Oct 2015 16:38:38 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: > > > If you're interested, I'm happy to directly collaborate on this PEP if > it's in > a github repository somewhere or something. There's an interoptability repo > btw, the repo he's talking about is here: https://github.com/pypa/interoperability-peps it has a convention about where to add pep ideas that have no number yet -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Oct 3 01:58:41 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 2 Oct 2015 18:58:41 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 2, 2015 5:53 PM, "Paul Moore" wrote: > > On 2 October 2015 at 23:15, Nathaniel Smith wrote: > > "Project" is a pretty messy concept. Obviously in simple cases there's > > a one-to-one mapping between project <-> wheel <-> importable package, > > but this breaks down quickly in edge cases. > > I mistakenly used "project" in an attempt to avoid confusion resulting > from me using the word "distribution" as a more general term than the > way you were using "source distribution" or "binary distribution". > Clearly I failed and made things more confusing. > > I use the term "distribution" in the sense used here > https://packaging.python.org/en/latest/glossary/#term-distribution-package . > Note that this is in contrast to the terms "source distribution" and > "binary distribution" or "built distribution" in the same page. > > Sorry for confusing things. I'll stick to the terminology as in the > PUG glossary from now on. > > > Consider a project that provides builds multiple wheels out of the > > same source tree. You obviously can't expect that all of these > > packages will have the same dependencies. > > Correct. But a distribution can and should (I believe) have the same > dependencies for all of the source and built distributions derived > from it. > > > This situation is not common today for Python packages, but the only > > reason for that is that distutils makes it really hard to do -- it's > > extremely common in other package ecosystems, and the advantages are > > obvious. E.g., maybe numpy.distutils should be split into a separately > > installable package from numpy -- there's no technical reason that > > this should mean we are now forced to move the code for it into its > > own VCS repository. > > I'm lost here, I'm afraid. Could you rephrase this in terms of the > definitions from the PUG glossary? It sounds to me like the VCS > repository is the project, which contains multiple distributions. I > don't see how that's particularly hard. Each distribution just has its > own subdirectory (and setup.py) in the VCS repository... > > > (I assume that by "platform tags" you mean what PEP 426 calls > > "environment markers".) > > Nope, I mean as defined in PEP 425. The platform tag is part of the > compatibility tag. Maybe I meant the ABI tag, I don't really follow > the distinctions. > > > Environment markers are really useful for extending the set of cases > > that can be handled by a single architecture-dependent wheel. And > > they're a good fit for that environment, given that wheels can't > > contain arbitrary code. > > > > But they're certainly never going to be adequate to provide a single > > static description of every possible build configuration of every > > possible project. And installing an sdist already requires arbitrary > > code execution, so it doesn't make sense to try to build some > > elaborate system to avoid arbitrary code execution just for the > > dependency specification. > > > > You're right that in a perfect future world numpy C API related > > dependencies would be handling by some separate ABI-tracking mechanism > > similar to how the CPython ABI is tracked, so here are some other > > examples of why environment markers are inadequate: > > > > In the future it will almost certainly be possible to build numpy in > > two different configurations: one where it expects to find BLAS inside > > a wheel distributed for this purpose (e.g. this is necessary to > > provide high-quality windows wheels), and one where it expects to find > > BLAS installed on the system. This decision will *not* be tied to the > > platform, but be selectable at build time. E.g., on OS X there is a > > system-provided BLAS library, but it has some issues. So the default > > wheels on PyPI will probably act like windows and depend on a > > BLAS-package that we control, but there will also be individual users > > who prefer to build numpy in the configuration where it uses the > > system BLAS, so we definitely need to support both options on OS X. > > Now the problem: There will never be a single environment marker that > > you can stick into a wheel or sdist that says "we depend on the > > 'pyblas' package if the system is OS X (ok) and the user set this flag > > in this configuration file during the build process (wait wut)". > > > > Similarly, I think someone was saying in a discussion recently that > > lxml supports being built either in a mode where it requires libxml be > > available on the system, or else it can be statically linked. Even if > > in the future we start having metadata that lets us describe > > dependencies on external system libraries, it's never going to be the > > case that we can put the *same* dependency metadata into wheels that > > are built using these two configurations. > > This is precisely the very complex issue that's being discussed under > the banner of extending compatibility tags in a way that gives a > viable but practical way of distinguishing binary wheels. You can > either see that as a discussion about "expanding compatibility tags" > or "finding something better than compatibility tags". I don't have > much of a stake in that discussion, as the current compatibility tags > suit my needs fine, as a Windows user. The issues seem to be around > Linux and possibly some of the complexities around binary dependencies > for numerical libraries. > > But the key point here is that I see the solution for this as being > about distinguishing the "right" wheel for the target environment. > It's not about anything that should reach back to sdists. Maybe a > solution will involve a PEP 426 metadata enhancement that adds > metadata that's only valid in binary distributions and not in source > distributions, but that's fine by me. But it won't replace the > existing dependency data, which *is* valid at the sdist level. this would be good to discuss here: PEP 426: Define a JSON-LD context as part of the proposal #31 https://github.com/pypa/interoperability-peps/issues/31 > > At least as far as I can see - I'm willing to be enlightened. But your > argument seems to be that sdist-level dependency information should be > omitted because more detailed ABI compatibility data *might* be needed > at the wheel level for some packages. I don't agree with that - we > still need the existing metadata, even if more might be required in > specialist cases. if sys.platform: extras_require = > > > >> [1] If extras and environment markers don't cover the needs of > >> scientific modules, we need some input into their design from the > >> scientific community. But again, let's not throw away the work that's > >> already done. > > > > As far as sdists go, you can either cover 90% of the cases by building > > increasingly elaborate metadata formats, or you can cover 100% of the > > cases by keeping things simple... everything in one (composed) 'pydist.jsonld' > > But your argument seems to be that having metadata generated from > package build code is "simpler". My strong opinion, based on what I've > seen of the problems caused by having metadata in an "exectable > setup.py", is that static metadata is far simpler. static JSONLD is easily indexable with warehouse (Postgresql), elasticsearch, triplestores > > I don't believe that the cost of changing to a new system can be > justified *without* getting the benefits of static metadata. external benefits of canonical package URIs + schema.org/Action > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 3 03:03:30 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 18:03:30 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:30 PM, Donald Stufft wrote: > On October 2, 2015 at 4:20:00 PM, Nathaniel Smith (njs at pobox.com) wrote: [...] >> There are plenty of more >> complex examples too (e.g. ones that involve build/configure-time >> decisions about whether to rely on particular system libraries, or >> build/configure-time decisions about whether particular packages >> should even be built). > > I don't think build/configure-time decisions are great ideas as it's near > impossible to actually depend on them. For example, take Pillow, Pillow will > conditionally compile against libraries that enable it to much around with > PNGs. However, if I *need* Pillow with PNG support, I don't have any mechanism > to declare that. If instead, builds were *not* conditional and Pillow instead > split it's PNG capabilities out into it's own package called say, Pillow-PNG > which also did not conditionally compile against anything, but unconditionally > did, then we could add in something like having Pillow declare a "weak" > dependency on Pillow-PNG where we attempt to get it by default if possible, but > we will skip installing it if we can't locate/build it. If you combine this > with Extras, you could then easily make it so that people can depend on > particular conditional features by doing something like ``Pillow[PNG]`` in > their dependency metadata. While I agree with the sentiment here, I don't think we can simply unconditionally rule out build/configure-time decisions. I gave an example in the other subthread of a numpy wheel, which depending on build configuration might depend implicitly on the system BLAS, might have BLAS statically linked, or might depend explicitly on a "BLAS wheel". (And note that when configured to use a "BLAS wheel" then this would actually be a build-dependency, not just a runtime-dependency.) As far as downstream users are concerned, all of these numpy wheels export exactly the same API -- how numpy finds BLAS is just an internal detail. So in this case the problems your paragraph above is worrying about just don't arise. And numpy absolutely will need the option to be built in these different ways. >> >> For comparison, here's the Debian source package metadata: >> https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles >> Note that the only mandatory fields are format version / package name >> / package version / maintainer / checksums. The closest they come to >> making promises about the built packages are the Package-List and >> Binary fields which provide a optional hint about what binary packages >> will be built, and are allowed to contain lies (e.g. they explicitly >> don't guarantee that all the binary packages named will actually be >> produced on every architecture). The only kind of dependencies that a >> source package can declare are build-depends. > > Debian doesn't really have "source packages" like we do, but inside of the > debian/ directory is the control file which lists all of the dependency > information (or explicitly lists a placeholder where something can't be > statically declared). Someone who is more of an expert on debian packaging can correct me if I'm wrong, but I'm 99% sure that this is incorrect, and in an important way. The actual interface between a build tool like dpkg-buildpackage and a source package is: (a) the .dsc file, with the required fields I mentioned, (b) the debian/rules file, which is an opaque executable that can be called to perform standard operations like "build" and "clean" -- basically the moral equivalent of the hooks in our sdist proposal. The debian/control file does have a conventional format, but this is just convention -- most-or-all debian/control scripts all use the same set of tools to work with this file, and expect it to be in the same place. But if, say, Debian decides that they need a new kind of placeholder to handle a situation that hasn't arisen before, then there's no need to change the definition of a source package: you just add support for the new placeholder to the tools that work with this file, and then packages that want to make use of the new placeholder just have to Build-Depend on the latest version of those tools. This is the idea motivating the sdist PEP's design: you can't specify all of a source distribution's metadata statically, and then given that you'll be specifying at least part of the metadata dynamically, you want to do it in a way that you can change without having to do a PEP and update pip etc. >> >> > To a similar tune, this PEP also doesn't make it possible to really get at >> > any other metadata without executing software. This makes it pratically >> > impossible to safely inspect an unknown or untrusted package to determine what >> > it is and to get information about it. Right now PyPI relies on the uploading >> > tool to send that information alongside of the file it is uploading, but >> > honestly what it should be doing is extracting that information from within the >> > file. This is sort of possible right now since distutils and setuptools both >> > create a static metadata file within the source distribution, but we don't rely >> > on that within PyPI because that information may or may not be accurate and may >> > or may not exist. However the twine uploading tool *does* rely on that, and >> > this PEP would break the ability for twine to upload a package without >> > executing arbitrary code. >> >> Okay, what metadata do you need? We certainly could put name / version >> kind of stuff in there. We left it out because we weren't sure what >> was necessary and it's easy to add later, but anything that's needed >> by twine fits neatly into the existing text saying that we should >> "include extra metadata in source distributions if it helps solve >> specific problems that are unique to distribution" -- twine uploads >> definitely count. > > Everything that isn't specific to a built wheel. Look at the previously > accepted metadata specs as well as PEP 426. If you're not including a field > that was included in one of those, there should be a rationale for why that > field is no longer being included. The default rationale is just "let's keep our options open" -- it's much easier to add than to subtract later. In particular I hesitate a little bit to just drop in everything from PEP 426 and friends, because previous specs haven't really thought through the distinction between sdists and wheels -- e.g. if an sdist generates two wheels, they probably won't have the same name, description, trove classifiers, etc. They may not even have the same version (e.g. if two different tools with existing numbering schemes get merged into a single distribution -- esp. if one of them needs an epoch marker). So it may well make sense to have an "sdist description field", but it's not immediately obvious that it's identical to a wheel's description field. I mean, in practice it's probably no big deal -- a description is some text for human readers, whatever, it's useful and it'll be fine. But given that we can trivially add more fields to the pypackage.cfg file later, and that current sdists don't have any of this metadata, I just don't want to risk blocking progress on one axis (enabling better build systems) while waiting to achieve maximal progress on another mostly-orthogonal axis (having nice metadata in sdists for tools like twine to take advantage of). Bottom line: If after further discussion we reach the point where the only thing blocking this is the addition of name and description and trove classifier fields, then of course we'll just add those to the PEP :-). -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sat Oct 3 03:06:32 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 18:06:32 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 3:52 PM, Paul Moore wrote: > On 2 October 2015 at 23:15, Nathaniel Smith wrote: [...] >> This situation is not common today for Python packages, but the only >> reason for that is that distutils makes it really hard to do -- it's >> extremely common in other package ecosystems, and the advantages are >> obvious. E.g., maybe numpy.distutils should be split into a separately >> installable package from numpy -- there's no technical reason that >> this should mean we are now forced to move the code for it into its >> own VCS repository. > > I'm lost here, I'm afraid. Could you rephrase this in terms of the > definitions from the PUG glossary? It sounds to me like the VCS > repository is the project, which contains multiple distributions. I > don't see how that's particularly hard. Each distribution just has its > own subdirectory (and setup.py) in the VCS repository... The problem is that projects tend to release the whole project together, rather than releasing individual subdirectories, and that usually you can't just rip those subdirectories out of the parent project and expect to build them on their own, because there's shared infrastructure for build, configuration, static libraries with utility code that get built once and then linked into each distribution... having a VCS checkout = one wheel rule blocks a lot of otherwise sensible project arrangements and forces awkward technical workarounds. But if you allow one VCS checkout to produce multiple wheels, then I can't see how you can avoid having one sdist produce multiple wheels. >> (I assume that by "platform tags" you mean what PEP 426 calls >> "environment markers".) > > Nope, I mean as defined in PEP 425. The platform tag is part of the > compatibility tag. Maybe I meant the ABI tag, I don't really follow > the distinctions. > >> Environment markers are really useful for extending the set of cases >> that can be handled by a single architecture-dependent wheel. And >> they're a good fit for that environment, given that wheels can't >> contain arbitrary code. >> >> But they're certainly never going to be adequate to provide a single >> static description of every possible build configuration of every >> possible project. And installing an sdist already requires arbitrary >> code execution, so it doesn't make sense to try to build some >> elaborate system to avoid arbitrary code execution just for the >> dependency specification. >> >> You're right that in a perfect future world numpy C API related >> dependencies would be handling by some separate ABI-tracking mechanism >> similar to how the CPython ABI is tracked, so here are some other >> examples of why environment markers are inadequate: >> >> In the future it will almost certainly be possible to build numpy in >> two different configurations: one where it expects to find BLAS inside >> a wheel distributed for this purpose (e.g. this is necessary to >> provide high-quality windows wheels), and one where it expects to find >> BLAS installed on the system. This decision will *not* be tied to the >> platform, but be selectable at build time. E.g., on OS X there is a >> system-provided BLAS library, but it has some issues. So the default >> wheels on PyPI will probably act like windows and depend on a >> BLAS-package that we control, but there will also be individual users >> who prefer to build numpy in the configuration where it uses the >> system BLAS, so we definitely need to support both options on OS X. >> Now the problem: There will never be a single environment marker that >> you can stick into a wheel or sdist that says "we depend on the >> 'pyblas' package if the system is OS X (ok) and the user set this flag >> in this configuration file during the build process (wait wut)". >> >> Similarly, I think someone was saying in a discussion recently that >> lxml supports being built either in a mode where it requires libxml be >> available on the system, or else it can be statically linked. Even if >> in the future we start having metadata that lets us describe >> dependencies on external system libraries, it's never going to be the >> case that we can put the *same* dependency metadata into wheels that >> are built using these two configurations. > > This is precisely the very complex issue that's being discussed under > the banner of extending compatibility tags in a way that gives a > viable but practical way of distinguishing binary wheels. You can > either see that as a discussion about "expanding compatibility tags" > or "finding something better than compatibility tags". I don't have > much of a stake in that discussion, as the current compatibility tags > suit my needs fine, as a Windows user. The issues seem to be around > Linux and possibly some of the complexities around binary dependencies > for numerical libraries. > > But the key point here is that I see the solution for this as being > about distinguishing the "right" wheel for the target environment. > It's not about anything that should reach back to sdists. Maybe a > solution will involve a PEP 426 metadata enhancement that adds > metadata that's only valid in binary distributions and not in source > distributions, but that's fine by me. But it won't replace the > existing dependency data, which *is* valid at the sdist level. Okay. I think this is the key question for me: you want sdist's to contain rich static metadata, presumably because you want to do something with that metadata -- you say that not having it causes problems. The obvious thing that pip might want to use that metadata for is so that it can look at an sdist and know whether the wheel it builds from that sdist will be useful in solving its current dependency goals. But to answer this question, you actually do need to know what the compatibility tag will be. So: what problems does it solve for pip to get access to static information about some, *but not all* of the eventual wheel's metadata? (It's also the case that in the numpy example I gave above, it isn't just the compatibility tag that can vary between wheels built from the same sdist, it's really truly the actual runtime dependency metadata that varies. But even if we ignore that, I think my question still stands.) -n -- Nathaniel J. Smith -- http://vorpus.org From chris.barker at noaa.gov Sat Oct 3 03:27:23 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 18:27:23 -0700 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 3:37 PM, Erik Bray wrote: > On Fri, Oct 2, 2015 at 5:39 PM, Donald Stufft wrote: > > ``pip install path/to/directory/with/setup.py/in/it/`` > > In particular I install most of my packages from source these days by > cd'ing into the source and > > $ pip install . > > That's all it takes. Works with -e too. > yup -- plugged this into a conda recipe and it works fine. in fact, I'm going to try to do all of my python recipes that way. Actually: $PYTHON -m pip install ./ $PYTHON is the python in conda's current build environment -- this will assure it used the right pip. Thanks all, this is working great. -CHB > Erik > > > On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) > wrote: > >> I can't seem to find a way to pip install from a source dir -- not a > >> tarball. > >> > >> Why would I want to do that? > >> > >> In this case, it's because I'm trying to build conda packages for python > >> packages that have compiled C code, and are not using setuptools. > >> > >> On Windows, you need to use setuptools in order to use the "MS compiler > for > >> Python 2.7". So a straight "setup.py build" fails. > >> > >> But pip injects setuptools, so that "pip install" does work. But conda > has > >> already unpacked the source distribution, so I'd like to point pip at > that, > >> and can't find a way to do that (I could have sworn it was possible...) > >> > >> Do I have to build an sdist, and then point pip at that?? > >> > >> -CHB > >> > >> > >> > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 3 04:27:33 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 19:27:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Hi Donald, Thanks for taking the time to make such detailed comments! Thoughts below. On Fri, Oct 2, 2015 at 4:04 PM, Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: >> >> Distutils delenda est. >> > > I think that you should drop (from this PEP) the handling of a VCS/arbitrary > directories and focus solely on creating a format for source distributions. A > source distribution can (and should) be fairly strict and well defined exactly > where all of the files go, what files exist and don't exist, and things of that > nature (more on this later). Hmm. Okay, I think this really helps clarify our key point of difference! For me, an important requirement is that there continue to be a single standard command that end-users can use to install a VCS checkout. This is a really important usability property -- everyone knows "setup.py install". Unfortunately we can't keep "setup.py install" given its entanglement with distutils and the desire to split building and installation, so the obvious answer is that this should become 'pip install ', and from that everything else follows. Having a standard way to install from a VCS checkout is also useful for things like requirements files... and in fact it's required by our current standards. PEP 440 has this as an example of a valid dependency specification: pip @ git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d So I'm extremely reluctant to give up on standardizing how to handle VCS checkouts. And if we're going to have a standard for that, then would sure be nice if we could share the work between this standard and the one for sdists, given how similar they are. [...] > I don't believe that Python should develop anything like the Debian ability to > have a single source "package" create multiple binary packages. The metadata of > the Wheel *must* strictly match the metadata of the sdist (except for things > that are Wheel specific). This includes things like name, version, etc. Trying > to go down this path I think will make things a lot more complicated since we > have a segmented archive where people have to claim particular names, otherwise > how do you prevent me from registering the name "foobar" on PyPI and saying it > produces the "Django" wheel? What prevents it in the current draft is that there's no way for foobar to say any such thing :-). If you ask for Django, then the only sdist it will look at is the one in the Django segment. This is an intentionally limited solution, based on the intuition that multiple wheels from a single sdist will tend to be a relatively rare case, when they do occur then there will generally be one "main" wheel that people will want to depend on, and that people should be uploading wheels anyway rather than relying on sdists. (Part of the intuition for the last part is that we also have a not-terribly-secret-conspiracy here for writing a PEP to get Linux wheels onto PyPI and at least achieve feature parity with Windows / OS X. Obviously there will always be weird platforms -- iOS and FreeBSD and Linux-without-glibc and ... -- but this should dramatically reduce the frequency with which people need sdist dependencies.) If that proves inadequate, then the obvious extension would be to add some metadata to the sdist similar to Debian's, where an sdist has a list of all the wheels that it (might) produce when built, PyPI would grow an API by which pip-or-whoever could query for all sdists that claim to be able to produce wheel X, and at the same time PyPI would start enforcing the rule that if you want to upload an sdist that claims to produce wheel X then you have to own the name X. (After all, you need to own that name anyway so you can upload the wheels.) Or alternatively people could just split up their packages, like would be required by your proposal anyway :-). So I sorta doubt it will be a problem in practice, but even if becomes one then it won't be hard to fix. (And to be clear, the multiple-wheels-from-one-sdist thing is not a primary goal of this proposal -- the main reason we put it in is that once you've given up on having static wheel metadata inside the sdist then supporting multiple-wheels-from-one-sdist is trivial, so you might as well do it, esp. since it's a case that does seem to come up with some regularity in real life and you don't want to make people fight with their tools when it's unnecessary.) > Since I think this should only deal with source distributions, then the primary > thing we need is an operation that will take an unpacked source distribution > that is currently sitting on the filesystem and turn it into a wheel located > in a specific location. > > The layout for a source distribution should be specified, I think something > like: > > . > ??? meta > ? ??? DESCRIPTION.rst > ? ??? FORMAT-VERSION > ? ??? LICENSE.txt > ? ??? METADATA.json > ??? src > ??? my-cool-build-tool.cfg > ??? mypackage > ??? __init__.py > > I don't particularly care about the exact names, but this layout gives us two > top level directories (and only two), one is a place where all of the source > distribution metadata goes, and one is a src directory where all of the files > for the project should go, including any relevant configuration for the build > tool in use by the project. Having two directories like this eliminates the > need to worry about naming collisions between the metadata files and the > project itself. > > We should probably give this a new name instead of "sdist" and give it a > dedicated extension. Perhaps we should call them "source wheels" and have the > extension be something like .swhl or .src.whl. This means we don't need to > worry about making the same artifact compatible with both the legacy toolchain > and a toolchain that supports "source wheels". > > We should also probably specify a particular container format to be used for > a .whl/.src.whl. It probably makes sense to simply use zip since that is what > wheels use and it supports different compression algorithms internally. We > probably want to at least suggest limiting compression algorithms used to > Deflate and None, if not mandate that one of those two are used. > > We should include absolutely as much metadata as part of the static metadata > inside the sdist as we can. I don't think there is any case to be made for > things like name, version, summary, description, classifiers, license, > keywords, contact information (author/maintainers), project URLs, etc are > Wheel specific. I think there are other things which are arguably able to be > specified in the sdist, but I'd need to fiddle with it to be sure. Basically > any metadata that isn't included as static information will not be able to be > displayed on PyPI. I feel like this idea of "source wheels" makes some sense if we want something that looks like a wheel, but without the ABI compatibility issues of wheels. I'm uncertain how well it can be made to work in practice, or how urgent it is once we have a 95% solution in place for linux wheels, but it's certainly an interesting idea. To me it feels rather different from a traditional sdist, and obviously there's still the problem of having a standard way to build from a VCS checkout. It might even make sense to have standard methods to go: VCS checkout -> sdist -> (wheels and/or source wheels) ? > The metada should directly include the specifiers inside of it and shouldn't > propagate the meme that pip's requirements.txt format is anything but a way > to recreate a specific environment with pip. Yeah, there's a big question mark next to the requirements.txt stuff in the draft PEP, because something more standard and structured would certainly be nice. But requirements.txt is wildly popular, and for a good reason -- it provides a simple terse syntax that does what people want. (By comparison, the PEP 426 JSON syntax for requirements with extras and environment specifiers is extremely cumbersome yet less featureful.) And semantically, what we want here is a way to say "to build *this* I need an environment that looks like *this*", which is pretty close to what requirements.txt is actually designed for. So I dunno -- instead of fighting the meme maybe we should embrace it :-). But obviously this is a tangent to the main questions. [...] > I don't think there's ever going to be a world where pip depends on virtualenv > or pyvenv. Huh, really? Can you elaborate on why not? The standard doesn't have to require the use of clean build environments (I was thinking of the text in the standard as applying with the "as if rule" -- a valid sdist is one that can be built the way described, if you have some way that will work to build such sdists then your way is valid too). But using clean environments by default is really the only way that we're going to get a world where most packages have accurate build requirements. -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Sat Oct 3 13:51:28 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Oct 2015 12:51:28 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 3 October 2015 at 02:03, Nathaniel Smith wrote: > In particular I hesitate a little bit to just drop in everything from > PEP 426 and friends, because previous specs haven't really thought > through the distinction between sdists and wheels -- e.g. if an sdist > generates two wheels, they probably won't have the same name, > description, trove classifiers, etc. They may not even have the same > version (e.g. if two different tools with existing numbering schemes > get merged into a single distribution -- esp. if one of them needs an > epoch marker). So it may well make sense to have an "sdist description > field", but it's not immediately obvious that it's identical to a > wheel's description field. I can only assume you're using the term sdist differently from the way it is normally used (as in the PUG glossary), because for me the idea that a sdist generates multiple wheels makes no sense. If I do "pip wheel " I expect to get a single wheel that can be installed to give the same results as "pip install " would, The wheel install will work on my build machine, and on any machine where the wheel's compatibility metadata (the compatibility tags, currently) says it's valid. The above behaviour is key to pip's mechanism for auto-generating and caching wheels when doing an install, so I don't see how it could easily be discarded. If what you're calling a "sdist" doesn't work like this, maybe you should invent a new term, so that people don't get confused? If it *does* work like this, I don't see what you mean by a sdist generating two wheels. Paul From dholth at gmail.com Sat Oct 3 18:45:31 2015 From: dholth at gmail.com (Daniel Holth) Date: Sat, 03 Oct 2015 16:45:31 +0000 Subject: [Distutils] Did https://mail.python.org/pipermail/python-list/2008-June/467441.html get an answer? In-Reply-To: References: Message-ID: In wheel we had to jump through some hoops to do rfc822 style documents with utf-8, a task which can also be accomplished more easily with newer versions of the standard Python 3 email library (but not the versions we had to support at the time). https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/pkginfo.py?at=default&fileviewer=file-view-default On Mon, Sep 21, 2015 at 9:44 PM Robert Collins wrote: > On 22 September 2015 at 13:07, Daniel Holth wrote: > > It's a bug if bdist_wheel generates anything other than utf-8 METADATA. > > Cool. And PKG-INFO ? > > -Rob > > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Oct 3 19:30:57 2015 From: brett at python.org (Brett Cannon) Date: Sat, 03 Oct 2015 17:30:57 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote: > On 3 October 2015 at 02:03, Nathaniel Smith wrote: > > In particular I hesitate a little bit to just drop in everything from > > PEP 426 and friends, because previous specs haven't really thought > > through the distinction between sdists and wheels -- e.g. if an sdist > > generates two wheels, they probably won't have the same name, > > description, trove classifiers, etc. They may not even have the same > > version (e.g. if two different tools with existing numbering schemes > > get merged into a single distribution -- esp. if one of them needs an > > epoch marker). So it may well make sense to have an "sdist description > > field", but it's not immediately obvious that it's identical to a > > wheel's description field. > > I can only assume you're using the term sdist differently from the way > it is normally used (as in the PUG glossary), because for me the idea > that a sdist generates multiple wheels makes no sense. > > If I do "pip wheel " I expect to get a single wheel that > can be installed to give the same results as "pip install > " would, The wheel install will work on my build machine, > and on any machine where the wheel's compatibility metadata (the > compatibility tags, currently) says it's valid. > > The above behaviour is key to pip's mechanism for auto-generating and > caching wheels when doing an install, so I don't see how it could > easily be discarded. > > If what you're calling a "sdist" doesn't work like this, maybe you > should invent a new term, so that people don't get confused? If it > *does* work like this, I don't see what you mean by a sdist generating > two wheels. > I think sdist is getting a bit overloaded in this discussion. From my understanding of what Paul and Donald are after, the process should be: VCS -> sdist/source wheel -> binary wheel. Here, "VCS" is literally a git/hg clone of some source tree. A "sdist/source wheel" is carefully constructed zip file that contains all the source code from the VCS necessary to build a binary wheel for a project as long as the proper dependencies exist (e.g., proper version of NumPy, BLAS in one of its various forms, etc.). The binary wheel is obviously the final artifact that can just be flat-out loaded by Python without any work. So Paul doesn't see sdist/source wheels producing more than one binary wheel because in its own way an sdist/source wheel is a "compiled" artifact of select source code whose only purpose is to generate a binary wheel for a single project. So while a VCS clone might have multiple subprojects, each project should generate a single sdist/source wheel. Now this isn't to say that an sdist/source wheel won't generate different *versions* of a binary wheel based on whether e.g. BLAS is system-linked, statically linked, or dynamically loaded from a Linux binary wheel. But the key point is that the sdist/source wheel still **only** makes one kind of project. >From this perspective I don't see Nathaniel's desire for installing from a VCS as something other than requiring a step to bundle up the source code into an sdist/source wheel that pip knows how to handle. But I think what Paul and Donald are saying is pip doesn't want to have anything to do with the VCS -> sdist/source wheel step and that is entirely up to the project to manage through whatever tooling they choose. I also view the sdist//source wheel as almost a mini-VCS checkout since it is just a controlled view of the source code with a bunch of helpful, static metadata for pip to use to execute whatever build steps are needed to get to a binary wheel. Or I'm totally wrong. =) But for me I actually like the idea of an sdist/source wheel being explained as "a blob of source in a structured way that can produce a binary wheel for the package" and a binary wheel as "a blob of bytes for a package that Python can import" and I'm totally fine having C extensions not just shuttle around a blind zip but a structured zip in the form of an sdist/source wheel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Oct 3 19:36:15 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 3 Oct 2015 13:36:15 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 3, 2015 at 1:31:48 PM, Brett Cannon (brett at python.org) wrote: > On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote: > > > On 3 October 2015 at 02:03, Nathaniel Smith wrote: > > > In particular I hesitate a little bit to just drop in everything from > > > PEP 426 and friends, because previous specs haven't really thought > > > through the distinction between sdists and wheels -- e.g. if an sdist > > > generates two wheels, they probably won't have the same name, > > > description, trove classifiers, etc. They may not even have the same > > > version (e.g. if two different tools with existing numbering schemes > > > get merged into a single distribution -- esp. if one of them needs an > > > epoch marker). So it may well make sense to have an "sdist description > > > field", but it's not immediately obvious that it's identical to a > > > wheel's description field. > > > > I can only assume you're using the term sdist differently from the way > > it is normally used (as in the PUG glossary), because for me the idea > > that a sdist generates multiple wheels makes no sense. > > > > If I do "pip wheel " I expect to get a single wheel that > > can be installed to give the same results as "pip install > > " would, The wheel install will work on my build machine, > > and on any machine where the wheel's compatibility metadata (the > > compatibility tags, currently) says it's valid. > > > > The above behaviour is key to pip's mechanism for auto-generating and > > caching wheels when doing an install, so I don't see how it could > > easily be discarded. > > > > If what you're calling a "sdist" doesn't work like this, maybe you > > should invent a new term, so that people don't get confused? If it > > *does* work like this, I don't see what you mean by a sdist generating > > two wheels. > > > > I think sdist is getting a bit overloaded in this discussion. From my > understanding of what Paul and Donald are after, the process should be: > > VCS -> sdist/source wheel -> binary wheel. > > Here, "VCS" is literally a git/hg clone of some source tree. A > "sdist/source wheel" is carefully constructed zip file that contains all > the source code from the VCS necessary to build a binary wheel for a > project as long as the proper dependencies exist (e.g., proper version of > NumPy, BLAS in one of its various forms, etc.). The binary wheel is > obviously the final artifact that can just be flat-out loaded by Python > without any work. > > So Paul doesn't see sdist/source wheels producing more than one binary > wheel because in its own way an sdist/source wheel is a "compiled" artifact > of select source code whose only purpose is to generate a binary wheel for > a single project. So while a VCS clone might have multiple subprojects, > each project should generate a single sdist/source wheel. > > Now this isn't to say that an sdist/source wheel won't generate different > *versions* of a binary wheel based on whether e.g. BLAS is system-linked, > statically linked, or dynamically loaded from a Linux binary wheel. But the > key point is that the sdist/source wheel still **only** makes one kind of > project. > > From this perspective I don't see Nathaniel's desire for installing from a > VCS as something other than requiring a step to bundle up the source code > into an sdist/source wheel that pip knows how to handle. But I think what > Paul and Donald are saying is pip doesn't want to have anything to do with > the VCS -> sdist/source wheel step and that is entirely up to the project > to manage through whatever tooling they choose. I also view the > sdist//source wheel as almost a mini-VCS checkout since it is just a > controlled view of the source code with a bunch of helpful, static metadata > for pip to use to execute whatever build steps are needed to get to a > binary wheel. > > Or I'm totally wrong. =) But for me I actually like the idea of an > sdist/source wheel being explained as "a blob of source in a structured way > that can produce a binary wheel for the package" and a binary wheel as "a > blob of bytes for a package that Python can import" and I'm totally fine > having C extensions not just shuttle around a blind zip but a structured > zip in the form of an sdist/source wheel. This is basically accurate, the only thing is that I think that we do need an answer for handling the VCS side of things, but I think that we should defer it until after this PEP. Right now I think the PEP is trying to tackle two different problems at once because on the surface they look similar but IMO a VCS/arbitrary directory and a sdist are two entirely different things that are only similar on the surface. So Yes the path would be: VCS -> sdist (aka Source Wheel or w/e) -> Wheel -> Install \ ?\-> "In Place" install (aka Development Install) Right now, we have the Wheel -> Install part, and I'd like for this PEP to focus on the sdist -> Wheel part, and then a future PEP focus on the VCS part. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sat Oct 3 19:50:38 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 3 Oct 2015 13:50:38 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 10:27:36 PM, Nathaniel Smith (njs at pobox.com) wrote: > > So I'm extremely reluctant to give up on standardizing how to handle > VCS checkouts. And if we're going to have a standard for that, then > would sure be nice if we could share the work between this standard > and the one for sdists, given how similar they are. Mentioned in another email, but I don't think we should give up on the VCS handling, I just think we should defer it to another PEP. I think this PEP is suffering from trying to use the same mechanism for VCS and sdists when they are different things and have different considerations. The Python packaging toolchain has a long history of suffering because it tried to reuse the same thing across multiple "phases" of the packaging life cycle. The problem with that is different phases have different needs, and when you're trying to use the same thing across multiple you suffer from something that is a sort of "Jack of All Trades, Master of None" kind of thing. > > What prevents it in the current draft is that there's no way for > foobar to say any such thing :-). If you ask for Django, then the only > sdist it will look at is the one in the Django segment. This is an > intentionally limited solution, based on the intuition that multiple > wheels from a single sdist will tend to be a relatively rare case, > when they do occur then there will generally be one "main" wheel that > people will want to depend on, and that people should be uploading > wheels anyway rather than relying on sdists. There *must* be a 1:1 mapping in name/version between sdist and wheels. This assumption is baked into basically every layer of the toolchain. > > I feel like this idea of "source wheels" makes some sense if we want > something that looks like a wheel, but without the ABI compatibility > issues of wheels. I'm uncertain how well it can be made to work in > practice, or how urgent it is once we have a 95% solution in place for > linux wheels, but it's certainly an interesting idea. To me it feels > rather different from a traditional sdist, and obviously there's still > the problem of having a standard way to build from a VCS checkout. I feel like you have some sort of "a sdist is jsut a tarball of a VCS" mentality and I don't think that idea of a sdist is generally useful. > > I don't think there's ever going to be a world where pip depends on virtualenv > > or pyvenv. > > Huh, really? Can you elaborate on why not? The standard doesn't have > to require the use of clean build environments (I was thinking of the > text in the standard as applying with the "as if rule" -- a valid > sdist is one that can be built the way described, if you have some way > that will work to build such sdists then your way is valid too). But > using clean environments by default is really the only way that we're > going to get a world where most packages have accurate build > requirements. > We'll most likely use a clean environment by explicitly emptying out the sys.path or something similar. We won't depend on virtual environments because it is a layering violation. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Sat Oct 3 20:18:01 2015 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 3 Oct 2015 13:18:01 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 3, 2015 12:36 PM, "Donald Stufft" wrote: > > > > On October 3, 2015 at 1:31:48 PM, Brett Cannon (brett at python.org) wrote: > > On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote: > > > > > On 3 October 2015 at 02:03, Nathaniel Smith wrote: > > > > In particular I hesitate a little bit to just drop in everything from > > > > PEP 426 and friends, because previous specs haven't really thought > > > > through the distinction between sdists and wheels -- e.g. if an sdist > > > > generates two wheels, they probably won't have the same name, > > > > description, trove classifiers, etc. They may not even have the same > > > > version (e.g. if two different tools with existing numbering schemes > > > > get merged into a single distribution -- esp. if one of them needs an > > > > epoch marker). So it may well make sense to have an "sdist description > > > > field", but it's not immediately obvious that it's identical to a > > > > wheel's description field. > > > > > > I can only assume you're using the term sdist differently from the way > > > it is normally used (as in the PUG glossary), because for me the idea > > > that a sdist generates multiple wheels makes no sense. > > > > > > If I do "pip wheel " I expect to get a single wheel that > > > can be installed to give the same results as "pip install > > > " would, The wheel install will work on my build machine, > > > and on any machine where the wheel's compatibility metadata (the > > > compatibility tags, currently) says it's valid. > > > > > > The above behaviour is key to pip's mechanism for auto-generating and > > > caching wheels when doing an install, so I don't see how it could > > > easily be discarded. > > > > > > If what you're calling a "sdist" doesn't work like this, maybe you > > > should invent a new term, so that people don't get confused? If it > > > *does* work like this, I don't see what you mean by a sdist generating > > > two wheels. > > > > > > > I think sdist is getting a bit overloaded in this discussion. From my > > understanding of what Paul and Donald are after, the process should be: > > > > VCS -> sdist/source wheel -> binary wheel. > > > > Here, "VCS" is literally a git/hg clone of some source tree. A > > "sdist/source wheel" is carefully constructed zip file that contains all > > the source code from the VCS necessary to build a binary wheel for a > > project as long as the proper dependencies exist (e.g., proper version of > > NumPy, BLAS in one of its various forms, etc.). The binary wheel is > > obviously the final artifact that can just be flat-out loaded by Python > > without any work. > > > > So Paul doesn't see sdist/source wheels producing more than one binary > > wheel because in its own way an sdist/source wheel is a "compiled" artifact > > of select source code whose only purpose is to generate a binary wheel for > > a single project. So while a VCS clone might have multiple subprojects, > > each project should generate a single sdist/source wheel. > > > > Now this isn't to say that an sdist/source wheel won't generate different > > *versions* of a binary wheel based on whether e.g. BLAS is system-linked, > > statically linked, or dynamically loaded from a Linux binary wheel. But the > > key point is that the sdist/source wheel still **only** makes one kind of > > project. > > > > From this perspective I don't see Nathaniel's desire for installing from a > > VCS as something other than requiring a step to bundle up the source code > > into an sdist/source wheel that pip knows how to handle. But I think what > > Paul and Donald are saying is pip doesn't want to have anything to do with > > the VCS -> sdist/source wheel step and that is entirely up to the project > > to manage through whatever tooling they choose. I also view the > > sdist//source wheel as almost a mini-VCS checkout since it is just a > > controlled view of the source code with a bunch of helpful, static metadata > > for pip to use to execute whatever build steps are needed to get to a > > binary wheel. > > > > Or I'm totally wrong. =) But for me I actually like the idea of an > > sdist/source wheel being explained as "a blob of source in a structured way > > that can produce a binary wheel for the package" and a binary wheel as "a > > blob of bytes for a package that Python can import" and I'm totally fine > > having C extensions not just shuttle around a blind zip but a structured > > zip in the form of an sdist/source wheel. > > This is basically accurate, the only thing is that I think that we do need an > answer for handling the VCS side of things, but I think that we should defer > it until after this PEP. Right now I think the PEP is trying to tackle two > different problems at once because on the surface they look similar but IMO > a VCS/arbitrary directory and a sdist are two entirely different things that > are only similar on the surface. > > So Yes the path would be: > > VCS -> sdist (aka Source Wheel or w/e) -> Wheel -> Install > \ > \-> "In Place" install (aka Development Install) > > Right now, we have the Wheel -> Install part, and I'd like for this PEP to > focus on the sdist -> Wheel part, and then a future PEP focus on the VCS part. thanks! is there a diagram of this somewhere? Is this basically a FSM with URIs for each resource and transformation, where more specific attributes are added (e.g. to pydist.jsonld)? [PEP 426 JSONLD] differences between VCS and sdist: * MANIFEST.in * setup.py * egg-info metadata PEP 426 JSONLD: https://github.com/pypa/interoperability-peps/issues/31 > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From setuptools at bugs.python.org Sun Oct 4 06:45:46 2015 From: setuptools at bugs.python.org (Stephen) Date: Sun, 04 Oct 2015 04:45:46 +0000 Subject: [Distutils] [issue163] Resolved Message-ID: <1443933946.17.0.650887817144.issue163@psf.upfronthosting.co.za> New submission from Stephen: Resolved ---------- files: 4588238e23891u2uy12.html messages: 754 nosy: stephenedu priority: wish status: resolved title: Resolved Added file: http://bugs.python.org/setuptools/file176/4588238e23891u2uy12.html _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- Play Slots, Roulette, and Poker Online

Play Slots, Roulette, and Poker Online

Australian casinos and gaming online slot machines

If you'd like to perform massive money obtain a individual stand, if you afford after that it do not become in case you concept the planet. You're able to search for podiums wherever clean generally stands at, others simply possess a workplace. As soon as you change the right tactic and obtain knowledgeable about the operating of gambling, then you can probably play for higher stakes. If you too are interested it's better to have a around through the well-known directories which come up with the massive set of 21 websites and their wide opinions. Instead, associates could possibly be necessary to enter selected limitations for them to get into the bonuses. Just as it's having some other game, you'll find methods as you are able to produce to be able to boost your likelihood of profitable at Pay Gown. The magic to be able to disguise behind a screen to hide your feelings been employed by effectively for several people of the game nowadays.

The most effective online casino on the web today boast good design, good games, and several financial selections to choose from. It's one of the most liked outdoor recreation that have been happening for a long time. It's likewise good for squeeze with fingers that flop nicely, like suited connectors, in case you are called by way of a great palm. That could be perfect give, no real matter what the turn and water cards were. Guaranteed, it may for some people; but, knowing everything you are performing beforehand, it makes your experience a much easier and much more enjoyable one.

The game will start when every one of the people have previously positioned the wagers around the betting circle. Another good way to get forward in casinos is always to just utilize some typically common sense. Several poker sites include major competitions on Sundays that attract a great number of internet positives. Sure, it could to some people; but, once you learn that which you are undertaking beforehand, it creates your experience a and much more fulfilling one.

Nevertheless, the final motion is going to be based on the supplier through the complete length of the bets. These bonuses usually are paid-as a percentage of the primary deposit that is made into their fresh gamers consideration. The volume of negative defeats is relatively generally increased inside the online game because of this of what some individuals state the game to be rigged or set.

However, with all the limitations of engineering in those days, it absolutely was quite comprehensible if immediate perform was unable to produce. But to become a champion on a regular basis, it takes more than only capabilities and coaching. They have specialist and live sellers who provide you with a tailored experience-you won't forget. We invested a couple of hours suspended while in the lazy stream on Sunday evening and observed it quite enjoyable.

Like all other races of activities, is a neighborhood card game that requires talent and probability. But, if you'd like in on the true activity, and start using household some good cash, you're likely to should deposit your consideration. This removes the confusion and dilemma of being forced to initially convert an unknown currency to at least one's personal before making a bet. In this manner you can filter down the machines that can advantage anyone essentially the most and figure out how to keep away from the fool machines. You may make usage of any opinions and reviews that have been manufactured concerning the casino and it would additionally enable when the presents any form of promotions for gamers, specially people who play consistently. In Australian baccarat, the ball player must produce a couple of decisions before coping the cards - these entail determining the positioning for profitable around the following palm, and price the worth of the choice.

From njs at pobox.com Sun Oct 4 20:22:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 4 Oct 2015 11:22:47 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Sat, Oct 3, 2015 at 10:50 AM, Donald Stufft wrote: > On October 2, 2015 at 10:27:36 PM, Nathaniel Smith (njs at pobox.com) wrote: >> >> So I'm extremely reluctant to give up on standardizing how to handle >> VCS checkouts. And if we're going to have a standard for that, then >> would sure be nice if we could share the work between this standard >> and the one for sdists, given how similar they are. > > Mentioned in another email, but I don't think we should give up on the VCS > handling, I just think we should defer it to another PEP. I think this PEP is > suffering from trying to use the same mechanism for VCS and sdists when they > are different things and have different considerations. The Python packaging > toolchain has a long history of suffering because it tried to reuse the same > thing across multiple "phases" of the packaging life cycle. The problem with > that is different phases have different needs, and when you're trying to use > the same thing across multiple you suffer from something that is a sort of > "Jack of All Trades, Master of None" kind of thing. I guess to make progress in this conversation I need some more detailed explanations. I totally get that there's a long history of thought and conversations behind the various assertions here like "a sdist is fundamentally different from a VCS checkout", "there must be a 1-1 mapping between sdists and wheels", "pip needs sdists that have full wheel metadata in static form", and I'm barging in from the outside with no context, but I literally have no idea why the specific design features you're asking for are desirable or even viable. Right now if I were to try and write the PEP you're asking for, then the rationale section would just be "because Donald said so" over and over :-). I couldn't write the motivation section, because I don't know any problems that the PEP you're describing would fix for me as a package author (which doesn't mean they don't exist, but!). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Oct 4 22:02:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 4 Oct 2015 21:02:44 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Let me see if I can help clarify, so it's not just Donald who says so :-) It does feel as if we're trying to explain a lot of things that "everybody knows". Clearly not everybody knows, as you don't, but what we're trying to clarify here is the de facto realities of how sdists work, and how people expect them to work. Unfortunately, there's an awful lot of things in the packaging ecosystem that are defined by existing practice, and traditionally haven't been formally documented. I'm sure it feels as if we're just repeatedly saying "it has to be like that" - but in truth, it's more that what we're saying is the definition of a sdist, as established by existing practice. I wish we could point you to a formal definition of the requirements, but unfortunately they've never been written down. With luck, one of the outcomes here will be that someone will record what a sdist is - but we need to reflect current reality, and not end up reusing the term "sdist" to mean something different from what people currently use it for. On 4 October 2015 at 19:22, Nathaniel Smith wrote: > "a sdist is fundamentally different from a VCS checkout", Specifically, a sdist is built by the packaging tools - at the moment, by "setup.py sdist", but in future by whatever tool(s) may replace distutils/setuptools. So a sdist has a defined format, and we can mandate certain things about it. In particular, we can require files to be present which are in tool-friendly formats, because the tools will build them. On the other hand, a VCS checkout is fundamentally built by a human, for use by humans. File formats need to be human-editable, we have to be prepared to work with constraints imposed by workflows and processes *other* than Python packaging tools. So we have much less ability to dictate the format. Your proposal mandates a single directory "owned" by the packaging ecosystem, which follows the git/hg/subversion model, so it's lightweight and low-risk. But you still cant realistically ask the user to maintain package data in (for example) a JSON file in that directory. > "there must be a 1-1 mapping between sdists and wheels", The fundamental reason is one I know I've mentioned here before - pip implements "pip install " by first building a wheel and then installing it. If a sdist generates two wheels, how will pip know which one to install? Also, users expect "pip wheel " to produce the wheel corresponding to the sdist. You're proposing to change that expectation - the onus is on you to justify that change. You need to consider backward compatibility in the wider sense here too - right now, there *is* a one-to-one mapping between a sdist and a wheel. If you want to change that you need to justify it, it's not enough just to claim that no-one has come up with a persuasive argument to keep things as they are. Change is not a bad thing, and "because we've always done it that way" is not a good argument, but change needs to be justified even so. > "pip needs sdists that have full wheel metadata in static form" I assume here you're now OK with the distinction between a sdist and a VCS checkout? If you still think we're saying that pip needs static metadata in *VCS checkouts* then please review the comments already made about the difference between a sdist and a VCS checkout. But basically, a sdist is a tool-generated archive that captures the state of the project and allows for *reproducible* builds of that project. If your understanding of what a sdist is differs from this, we need to stop and agree on terminology before going any further. I will concede that https://packaging.python.org/en/latest/glossary/ doesn't mention the point that a sdist needs to provide reproducible builds. But that's certainly how sdists are used at present, and how people expect them to work. Certainly, if I lost the wheel I'd built from a sdist, I'd expect to just rebuild it from the sdist and get the same wheel. Pip needs metadata to do dependency resolution. This includes project name, version, and dependency information. We could debate about whether *full* metadata is needed, but I'm not sure what the point is. Once you are recording the stuff that pip needs, why *not* record everything? There are other tools (and ad-hoc scripts) that would benefit from having the full metadata, so why would you make it harder for them to work? You claim that you want to keep your options open - but to me, it's more important to leave the *user's* options open. If we don't provide certain values, a user who needs that data has to propose a change to the format, wait for it to be implemented, and even then they can't rely on it until all projects move to the new format. Better to just require everything from the start, then users can get at whatever they need. As far as why the metadata should be static, the current sdist format does actually include static metadata, in the PKG-INFO file. So again we have a case where it's up to you to justify the backward compatibility break. But it's a little less clear-cut here, because you are proposing a new sdist format, so you've already argued for a break with the old format. Also the old format is not typically introspected, it's just used to unpack and run setup.py. So you can reasonably argue that the current state of affairs is irrelevant. However, we're talking here about whether the metadata should be statically available, or dynamically generated. The key point here is that dynamic metadata requires the tool (pip, my one-off script, whatever) to *run arbitrary code* in order to get the metadata. OK, with signing we can ensure that it's *trusted* code, but it still could do anything the project author wanted, and we can make no assumptions about what it does. That makes a tool's job much harder. A common bug report for pip is users finding that their installs fail, because setup.py requires numpy to be installed in order to run, and yet pip is running setup.py egg-info precisely to find out what the requirements are. We tell the user that the setup.py is written incorrectly, and they should install numpy and retry the install, but it's not a good user experience. And from a selfish point of view, users blame *pip* for the consequences of a project whose code to generate the metadata is buggy. Those bug reports are a drain on the time of the pip developers, as well as a frustrating experience for the users. If you want to argue that a VCS checkout, or development directory, needs to generate metatata dynamically, I won't argue. That's fine. But the sdist is a tool-generated snapshot of a *specific* release of a project (maybe "the release I made at 1:15 today for my test build", but still a specific build) and it should be perfectly possible to capture the dynamically generated metadata from the VCS checkout and store it in the sdist when it is built. If you feel that there is metadata that cannot be stored statically in the sdist, could you please give a specific example? But do remember that a sdist is intended as a *snapshot* of a VCS checkout that can be used to reproducibly build the project - so "the version number needs to include the time of the build" isn't a valid example. Paul From donald at stufft.io Sun Oct 4 22:45:05 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 4 Oct 2015 16:45:05 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 4, 2015 at 2:22:51 PM, Nathaniel Smith (njs at pobox.com) wrote: > > I guess to make progress in this conversation I need some more > detailed explanations. I totally get that there's a long history > of thought and conversations behind the various assertions > here like "a sdist is fundamentally different from a VCS checkout", > "there must be a 1-1 mapping between sdists and wheels", "pip > needs sdists that have full wheel metadata in static form", and > I'm barging in from the outside with no context, but I literally > have no idea why the specific design features you're asking for > are desirable or even viable. Right now if I were to try and write > the PEP you're asking for, then the rationale section would just > be "because Donald said so" over and over :-). I couldn't write > the motivation section, because I don't know any problems that > the PEP you're describing would fix for me as a package author > (which doesn't mean they don't exist, but!). I don't mind going into more details! I'll do the things you specifically mentioned and then if there is other things, feel free to bring them up too. I should also mention, that these are my opinions from my experiences with the toolchain and ecosystem, others may agree or disagree with me. I have strong opinions, but that doesn't make them immutable laws of the universe, although "because Donald said so" sounds like a pretty good answer to me ;) "a sdist is fundamentally different from a VCS checkout" This one I have a hard time trying to explain. They are focused on different things. With an sdist you need to have a project name, a version, a list of files, things like that. The use cases and needs for each "phase" are different. For instance, in a VCS checkout you can derrive the list of files or the version by asking the VCS but a sdist doesn't have a VCS so it has to have that baked into it. A more C centric example, is that you often times have something like autogen.sh in a C project's VCS, but you don't have the output of that checked into the VCS, however when you prepare a tarball for distribution you run autogen.sh and then include the output there. There are other differences too, in a VCS we don't really need the ability to statically read any metadata except for build dependencies and how to invoke the build tool. Most everything else can be dynamically configured because you're not distributing that. However in a sdist, we need as much of the metadata to be static as possible. Something like PyPI needs to be able to inspect any of the files uploaded to it (sdist, wheels, etc) for certain information and anything that can't be statically and safely read from it might as well not even exist as far as PyPI is concerned. We currently have the situation where we have a single file that is used for all phases of the process, dev (``setup.py develop`` & ``setup.py sdist``), building of a wheel (``setup.py bdist_wheel``) and even installation sometimes (``setup.py install``). Throughout this there are a lot of common problems where some author tried to optimize their ``setup.py`` for their development use cases and broke it for the other cases. An example of this is version handling, where it's not unusual for someone's first forray into attempting to deduplication version involves importing their thing (which works fine on their machine) and passing it into the setup kwargs. This simple thing would generally work just fine if the output of ``setup.py sdist`` produced static metadata and ``setup.py`` was no longer being used. This also becomes expressed in what interfaces you give to the toolchain at each "phase". It's important for something inside of a VCS checkout to be able to be written by human beings. This leads to wanting to use formats like INI (which is ugly) or something like TOML or YAML or some other nice, human friendly format. These formats are great for humans to write and for humans to read but are not particularly great as data interchange formats. Looking at something like JSON, msgpack, etc are far better for data interchange for computers to talk to other computers, but are not great for humans to write, edit, or even really read in many cases. If we go back to distutils2, you can see this effect happening there, they had two similar keywords arguments in their setup.cfg statements, description and description-file, these both did the same things, but just pulled from different sources (inline or via a file) forcing every tool in the chain to have to support both of these options even though it could have easily made an sdist that was distinct from the VCS code and simplified code there. I see the blurring of lines between the various phases of a package one of the fundamental flaws of distutils and setuptools. "there must be a 1-1 mapping between sdists and wheels" This has technical and social reasons. In the techincal side, the 1-1 mapping between sdists and wheels (and all other bdists) is an assumption baked into all of the tools. From PyPI's enforcement mechanisms, to pip's caching, to things like devpi and the such breaking this assumption will break a lot of code. This is all code and code is not immutable so we could of course change that, however we wouldn't be able to rely on the fact that we've fixed that assumption for many years (probably at least 5 at the earliest, 10+ is more likely). The social side is a bit more interesting though. In Debian, end users almost *never* actually interact with source packages and in near 100% of the time they are interacting soley with built packages (in fact, unlike Python, you have to manually build a deb before you can even attempt to install something). There really aren't "source packages" in Debian, just sources that happen to produce a Debian package. In Python land, a source package is still a package and people have expectations around that, I think people would be very confused if a sdist "foo-1.0.tar.gz" could produce a wheel "bar-3.0.whl". In addition, systems like Debian don't really try to protect against a malicious DD at all. Things like "prevent foo from claiming to be bar" are enforced via societal conventions and the fact that it is not an open repo and there are gatekeepers keeping everything in place. On the flip side, we let anyone upload to PyPI and rely on things like ACLs to secure things. This means that we need to know ahead of time what names a package is going to produce. The simpliest mechanism for this is to enforce a 1:1 mapping between sdist and wheel because that is an immutable property and easy to understand. I could possibly envision something that allowed this, but it would require a project to explicitly declare up front what names it will produce, and require registering those names with PyPI before you could upload a sdist that could produce those named wheels. Ultimately, I don't think the very minor benefits are worth the additional complexity and pain of trying to adapt all of the tooling and human expectations to this. "pip needs sdists that have full wheel metadata in static form" I think I could come around to the idea that some metadata doesn't make sense for a sdist, and that it really needs to be a part of wheels but not a part of sdist. I think that the argument needs to be made in the other direction though, we should assume that all metadata will be included as part of the sdist and then make an argument for why each particular piece of metadata is Wheel specific not specific to a particular version of a project. Things like name, version, description, classifiers, etc are easily able to be classified into specific to a particular (name, version) tuple. Other things like "Python ABI" are easily able to be classified into specific to a particular wheel. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From njs at pobox.com Mon Oct 5 08:29:31 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 4 Oct 2015 23:29:31 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Sat, Oct 3, 2015 at 10:50 AM, Donald Stufft wrote: > I feel like you have some sort of "a sdist is jsut a tarball of a VCS" > mentality and I don't think that idea of a sdist is generally useful. Hmm, so, between thinking this over more on the plane today and reading your and Paul's great replies, I think I see where a lot of this disjunction might be arising. I'll try to reply to those in more detail later, but first let me try to lay this out and see if it makes things clearer. First, let's drop the word "sdist", it's confusing. I'm starting from the long history and conventions around how people make what I'll call "source releases" (and in a few paragraphs will contrast with "source wheels"). 'Everyone knows' that when you release a new version of some package (in pretty much any language), then one key step is to put together a file called -.. And 'everyone knows' that if you see a file that follows this naming convention, and you download it, then what you'll find inside is: a single directory called -/, and inside this directory will be something that's almost like a VCS checkout -- it'll probably contain a README, source files in convenient places to be edited or grepped, etc. The differences from a VCS checkout (if any) will be little convenience stuff -- like ./autogen.sh will have been run already, or there will be an extra file containing a fixed version number instead of it being autodetected, or -DNDEBUG will be in the default CFLAGS, or Cython files will have been pre-translated to C -- but fundamentally it will be similar to a VCS checkout, and switching back and forth between them won't be too jarring. 95% of the time there will be a standard way to build the thing ('./configure && make && make install', or 'python setup.py install', or similar). And these kind of source releases have a rich ecosystem around them and serve a wide range of uses: they provide a low-tech archival record (while VCS's come and go), they end up in deb and rpm "original source" bundles, they get downloaded by users and built by hand (maybe with weird configury on top, like a hack to enable cross-compilation) or poked around in by hand, etc. etc. When sdists were originally designed, then "source releases" is what the designers were thinking about. Then, easy_install came along, and pulled off a clever hack where when you asked for some package by name, then it would try to automagically go out and track down any relevant source releases and build them all. And it works great, except when it doesn't. And creates massive headaches for everyone trying to work on python packaging afterwards, because source releases were not designed to be used this way. My hypothesis is that the requirements that were confusing me are based around the idea that an sdist should be something designed to slot into this particular use case: i.e., something that pip can automatically grab and work with while solving a dependency resolution problem. Therefore it really needs to have a static name, and static version number, and static dependencies, and must produce exactly one binary wheel that shares all that metadata. Let's call this a "source wheel" -- what we're really looking for here is a way to ship extension modules inside something that acts like an architecture-neutral "none-any" wheel. So: the email that started this thread was a proposal for how to standardize the format of "source releases", and Donald's counter was a proposal for how to standardize the format of "source wheels". Does that sound right? If so, then some follow-up thoughts: 1) If we design a source wheel format, then I am 100% in favor of the suggestion of giving it a unique extension like "swhl". I'm still a young whippersnapper compared to some, but I've been downloading files named -. for 20 years, and AFAICR every one of those files unpacked to make a single directory that was laid out like a VCS checkout. Obviously we can change what goes inside, but we should change the naming convention at the same time because otherwise we're just going to confuse people. 2) I think there's a strong case to be made that Python actually needs standards for *both* source releases and source wheels. There's certainly no logical contradiction -- they're conceptually different things. It sounds like we all agree that "pip" should continue to have a way to build and install an arbitrary VCS checkout, and extending that standard to cover building and installing a classic "source release" would be... almost difficult *not* to do. And I think that there will continue to be a clear need for source releases even in a world where source wheels exist, because of all those places where source releases get used that aren't automagic-easy_install/pip-builds. For example, most pure Python packages (the ones that already make "none-any" wheels) have no need at all for source wheels, but they still need source releases to serve as archival snapshots. And more complex packages that need build-time configuration (e.g. numpy) will continue to require source releases that can be configured to build wheels that have a variety of different properties (e.g., different dependency metadata), so they can't get by with source wheels alone -- but you can imagine that such projects might reasonably *in addition* provide a source wheel that locks down the same default configuration that gets used for their uploaded binary wheel builds, and is designed for pip to use when trying to resolve dependencies on platforms where a regular binary wheel is unavailable. Pictorially, this world would look like: VCS checkout -> source release \ \ --------------------+--> in-place install | +--> wheels -> install | +--> source wheels -> wheels -> install 3) It sounds like we all agree that - 'pip install ' should work - that there is some crucial metadata that VCS checkouts won't be able to provide without running arbitrary code (e.g. dependencies and version numbers) - that what metadata they do provide (e.g., which arbitrary code to run) should be specified in a human-friendly configuration file Given this, in the big picture it sounds like the only really essentially controversial things about the original proposal might be: - that 'pip install ' should work the same as 'pip install ' (does anyone actually disagree?) - the 1 paragraph describing the "transitional plan" allowing pip to automatically install from these source releases *as part of a dependency resolution plan* (as opposed to when a VCS-checkout-or-equivalent is explicitly given as an install target). Which honestly I don't like either, it's just there as a backcompat measure so that these source releases don't create a regression versus existing sdists -- note that one of the goals of the design was that current sdists could be upgraded into this format by dropping in a single static file (or by an install tool "virtually" dropping in this file when it encounters an old-style sdist -- so you don't need to keep around code to handle both cases separately). Does that sound right? (Other features of the original proposal include stuff like the lack of trivial metadata like "name" and "description", and the support for generating multiple wheels from one directory. I am explicitly calling these "inessential".) -n -- Nathaniel J. Smith -- http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Oct 5 11:03:21 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 10:03:21 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 5 October 2015 at 07:29, Nathaniel Smith wrote: > First, let's drop the word "sdist", it's confusing. I'll read your full reply later, when I have the time, but please note we can't drop the term sdist - it's a well known concept in packaging. Having said that, I'm happy if you want to restate your proposal in terms of a new concept that's not a sdist (you'll need to come up with a suitable term, and make it clear that it's distinct from a sdist when you formalise the proposal, but that's OK for now). My immediate thought is that I'm against a proposal that throws out the sdist concept in favour of something new, as there's a lot of reworking that would need to be done to achieve that, and I don't know who's going to do that. So I'd need convincing that the proposal is practical. For example, PyPI would need to be able to host these new things, and distinguish between them and sdists. But I'll hold off on detailed comments until I've read your full email. Paul From p.f.moore at gmail.com Mon Oct 5 13:28:48 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 12:28:48 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: OK, I've had a better read of your email now. Responses inline. On 5 October 2015 at 07:29, Nathaniel Smith wrote: > First, let's drop the word "sdist", it's confusing. We can't (see below for details). We can deprecate the sdist concept, if that's what you want to propose. From what I gather, you're proposing deprecating it in favour of a "source wheel" concept. I don't have a huge issue with that other than that I don't see the necessity - the sdist concept pretty much covers what you want, except maybe that it's not clear enough to people outside the packaging community how it differs from a VCS checkout. > I'm starting from the long history and conventions around how people make > what I'll call "source releases" (and in a few paragraphs will contrast with > "source wheels"). 'Everyone knows' that when you release a new version of > some package (in pretty much any language), then one key step is to put > together a file called -.. And 'everyone > knows' that if you see a file that follows this naming convention, and you > download it, then what you'll find inside is: a single directory called > -/, and inside this directory will be something that's > almost like a VCS checkout -- it'll probably contain a README, source files > in convenient places to be edited or grepped, etc. The differences from a > VCS checkout (if any) will be little convenience stuff -- like ./autogen.sh > will have been run already, or there will be an extra file containing a > fixed version number instead of it being autodetected, or -DNDEBUG will be > in the default CFLAGS, or Cython files will have been pre-translated to C -- > but fundamentally it will be similar to a VCS checkout, and switching back > and forth between them won't be too jarring. 95% of the time there will be a > standard way to build the thing ('./configure && make && make install', or > 'python setup.py install', or similar). Breaking at this point, because that's frankly *not* the reality in the Python packaging world (at least not nowadays - I'm not clear to what extent you're just talking about history and background here, although your reference to Cython makes me think you're talking in terms of current practice). It may look like that, but there are some fundamental differences. First and foremost, nobody zips up and publishes their VCS checkout in the way you describe. (At least not if they are using the standard tools - distutils and setuptools). Instead, they create a "sdist" using the "python setup.py sdist" command. I'm sorry, but I'm going to carry on using the "sdist" term here, because I'm describing current practice and sdists *are* current practice. The difference is between a sdist and what you call a "source release" is subtle, precisely because the current sdist format is a bit of a mess, but the key point is that all sdists are created by a standard process, and conform to a standard naming convention and layout. The packaging tools rely on being able to make that assumption, in all sorts of ways which we're doing our best to clarify as part of this thread, but which honestly have been a little bit implicit up to this point. Further muddying the water is the fact that as you say, pip needs to be able to build from a VCS checkout (a directory on the user's local system) and we have code in pip that does that - mostly by assuming that you can treat a VCS checkout as an unpacked sdist, but there are hacks we need to do to make that work (we run setup.py egg-info to get the metadata we need, for example, which has implications as we only get that data at a later stage than we have it in the sdist case) and differences in functionality (develop mode). At this point I'm not saying that things have to be this way, or even that "make a source release however you choose as long as it follows these conventions" isn't a viable way forward, but I do think we need to agree on our picture of how things are now, or we'll continue talking past each other. > And these kind of source releases > have a rich ecosystem around them and serve a wide range of uses: they > provide a low-tech archival record (while VCS's come and go), they end up in > deb and rpm "original source" bundles, they get downloaded by users and > built by hand (maybe with weird configury on top, like a hack to enable > cross-compilation) or poked around in by hand, etc. etc. When sdists were > originally designed, then "source releases" is what the designers were > thinking about. This, on the other hand, I suspect is not far from the truth. When sdists were designed, they were a convenience for bundling the stuff needed to do setup.py install later, possibly on a different machine. But that's a long time ago, and not really relevant now. For better or worse. Unless you are suggesting that we go all the way back to that original point? Which you may be, but that means discarding the work that's been done based on the sdist concept since then. Which leads nicely on to... > Then, easy_install came along, and pulled off a clever hack where when you > asked for some package by name, then it would try to automagically go out > and track down any relevant source releases and build them all. And it works > great, except when it doesn't. And creates massive headaches for everyone > trying to work on python packaging afterwards, because source releases were > not designed to be used this way. > > My hypothesis is that the requirements that were confusing me are based > around the idea that an sdist should be something designed to slot into this > particular use case: i.e., something that pip can automatically grab and > work with while solving a dependency resolution problem. Therefore it really > needs to have a static name, and static version number, and static > dependencies, and must produce exactly one binary wheel that shares all that > metadata. Anyone who knows my history will know that I'm the last person to defend setuptools' hacks, but you hit the nail on the head above. When it works, it works great (meh, "sufficiently well" :-)) And pip *needs* to do static dependency resolution. We have enough bug reports and feature requests asking that we improve the dependency resolution process that nobody is going to be happy with anything that doesn't allow for at least as much static information as we currently have, and ultimately more. > Let's call this a "source wheel" -- what we're really looking for > here is a way to ship extension modules inside something that acts like an > architecture-neutral "none-any" wheel. I don't understand this statement. What do extension modules matter here? We need to be able to ship sources in a form that can participate in dependency resolution (and any other potential discovery processes that may turn up in future) without having to run code to do so. The reasons for this are: 1. Running code is a performance overhead, and possibly even a security risk (even trusted code may behave in ways you didn't anticipate). We want to do as little as possible of that as we can, and in particular we want to discard invalid candidate files without running any of their code. 2. Running code introduces the possibility of that code failing. We don't want end users to have installs fail because code in distributions we're going to discard is buggy. 3. Repositories like PyPI need to present project metadata, for both human and tool consumption - they can only do this if it's available statically. You seem to be thinking that binary wheels are sufficient for this for pure-Python code. Look at it the other way - we discard sdists from the dependency calculations whenever there's an equivalent binary wheel available. That's always for non-any wheels, but less often for architecture-dependent wheels. But in order to know that the wheel is equivalent, we need to match it with the sdist - so the sdist needs the metadata you're trying to argue against providing... > So: the email that started this thread was a proposal for how to standardize > the format of "source releases", and Donald's counter was a proposal for how > to standardize the format of "source wheels". Does that sound right? Well, essentially yes, although I read it as your original email being a proposal for a new format to replace sdists, and Donald's and my counter is that there's already a been a certain amount of thinking and design gone into how we move from the current ad-hoc sdist format to a better defined and specified "next version", so how does your proposal affect that? It seems that your answer is that you want to bypass that and offer an alternative. Is that fair? For the record, I don't like the term "source wheel" and would prefer to stick with "sdist" if appropriate, or choose a term that doesn't include the word "wheel" otherwise (as wheels seem to me to be strongly, and beneficially, linked in people's minds to the concept of a binary release format). > If so, then some follow-up thoughts: > > 1) If we design a source wheel format, then I am 100% in favor of the > suggestion of giving it a unique extension like "swhl". I'm still a young > whippersnapper compared to some, but I've been downloading files named > -. for 20 years, and AFAICR every one of > those files unpacked to make a single directory that was laid out like a VCS > checkout. Obviously we can change what goes inside, but we should change the > naming convention at the same time because otherwise we're just going to > confuse people. I have no problem with making it clear that "sdist version 2" or "source wheel" is not the same as a packed VCS checkout. I don't see the need for a new term, I'd be happy with "-.sdist" as the name. I'd also like to emphasize strongly that PyPI only hosts sdists, and *not* source releases - that source releases are typically only seen in Python in the form of a VCS checkout or development directory. (There's an implication there that we need to explore, that pip won't necessarily gain the ability to be pointed at a non-sdist format packed "source release" archive, and download it and process it. That's not a given, but I'd like to be sure we are happy with the potential re-introduction of confusion over the distinction between a sdist/source wheel and a source release that would result). > 2) I think there's a strong case to be made that Python actually needs > standards for *both* source releases and source wheels. There's certainly no > logical contradiction -- they're conceptually different things. It sounds > like we all agree that "pip" should continue to have a way to build and > install an arbitrary VCS checkout, and extending that standard to cover > building and installing a classic "source release" would be... almost > difficult *not* to do. As noted above, I'm happy for that discussion to occur. But I'm *not* sure the case is as strong as you think. Technically, it's certainly not too hard, but the social issues are what concern me. How will we explain to someone that they can't upload their file to PyPI because it's a "source release" not a "source wheel"? What is the implication on people's workflow? How would we explain why people might want to make "source releases" *at all*? Personally, I can only see a need in my personal experience for a VCS url that people can clone, a packaged source artifact that I can upload to PyPI for automatic consumption, and (binary) wheels. That second item is a "source wheel" - not a "source release". > And I think that there will continue to be a clear need for source releases > even in a world where source wheels exist, because of all those places where > source releases get used that aren't automagic-easy_install/pip-builds. For > example, most pure Python packages (the ones that already make "none-any" > wheels) have no need at all for source wheels, but they still need source > releases to serve as archival snapshots. And more complex packages that need > build-time configuration (e.g. numpy) will continue to require source > releases that can be configured to build wheels that have a variety of > different properties (e.g., different dependency metadata), so they can't > get by with source wheels alone -- but you can imagine that such projects > might reasonably *in addition* provide a source wheel that locks down the > same default configuration that gets used for their uploaded binary wheel > builds, and is designed for pip to use when trying to resolve dependencies > on platforms where a regular binary wheel is unavailable. > > Pictorially, this world would look like: > > VCS checkout -> source release > \ \ > --------------------+--> in-place install > | > +--> wheels -> install > | > +--> source wheels -> wheels -> install I don't see the need for source releases that you do. That's likely because I don't deal with the sorts of complex projects you do, though, so I'm not dismissing the issue. As I say, my objections are mostly non-technical. I do think you should consider how to document "what a source release is intended to achieve" in a way that explains it to people who don't need the complexity it adds - and with the explicit goal of making sure that you dissuade people who *don't* need source releases from thinking they do. > 3) It sounds like we all agree that > - 'pip install ' should work Yes. > - that there is some crucial metadata that VCS checkouts won't be able to > provide without running arbitrary code (e.g. dependencies and version > numbers) I'm still resisting this one, although I can live with "Nathanial tells me so" :-) > - that what metadata they do provide (e.g., which arbitrary code to run) > should be specified in a human-friendly configuration file I don't agree to that one particularly, in the sense that I don't really care. I'd be happy with a system that said something like that for a VCS checkout, pip expects "setup.py egg-info" and "setup.py sdist" to work, and produce respectively a set of static metadata in a known location, and a properly formatted "source wheel"/sdist file. Non-distutils build tools can write a wrapper setup.py that works however they prefer. (That's roughly what we have now, BTW). > Given this, in the big picture it sounds like the only really essentially > controversial things about the original proposal might be: > - that 'pip install ' should work the same as > 'pip install ' (does anyone actually disagree?) Yes, to the extent that I want to ensure it's clearly documented how, and why, this differs from a sdist/source wheel. But that's not a technical issue. > - the 1 paragraph describing the "transitional plan" allowing pip to > automatically install from these source releases *as part of a dependency > resolution plan* (as opposed to when a VCS-checkout-or-equivalent is > explicitly given as an install target). Which honestly I don't like either, > it's just there as a backcompat measure so that these source releases don't > create a regression versus existing sdists -- note that one of the goals of > the design was that current sdists could be upgraded into this format by > dropping in a single static file (or by an install tool "virtually" dropping > in this file when it encounters an old-style sdist -- so you don't need to > keep around code to handle both cases separately). Given that these source releases won't be hosted on PyPI (as the proposal currently stands) there's no real need for this - all you need to say is that you can point pip at any old URL and take your chances :-) > Does that sound right? Not really. For me the big controversy is whether we move forward from where we are with sdists, or we ignore the current sdist mechanism and start over. A key further question, which I don't think has been stated explicitly until I started this email, is what formats will be supported for hosting on PyPI. I am against hosting formats that don't support static metadata, such as your "source distribution", as I don't see how PyPI would be able to publish the metadata if it weren't static. And following on from that, we need to agree whether the key formats should be required to have a static version. I'm OK with a VCS checkout having a dynamically generated version, that's part of the "all bets are off" contract over such things (if you don't generate a version that reflects every change, you get to deal with the consequences) but I don't think that's a reasonable thing to allow in "published" formats. > (Other features of the original proposal include stuff like the lack of > trivial metadata like "name" and "description", and the support for > generating multiple wheels from one directory. I am explicitly calling these > "inessential".) Not sure what you mean by lack of name/description being "inessential". The double negative confuses me. Do you mean you're OK with requiring them? Fair enough. For multiple wheels, I'd tend to consider the opposite to be true - it's not that the capability is non-essential, but rather that in published formats (source wheel and later in the chain) it's essential that one source generates one target. Paul From donald at stufft.io Mon Oct 5 14:44:04 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 08:44:04 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 2:29:35 AM, Nathaniel Smith (njs at pobox.com) wrote: > > Does that sound right? Off the bat, I'm perfectly fine with `pip install archive-made-from-vcs.tar.gz` working the same as `pip install git+https://github.com/foo/bar.git``. The fact that one is a VCS and one isn't is immaterial to me that's just a transport mechanism for an arbitrary collection of files and directories. These arbitrary collections of file and directories can only provide a very minimal amount of information about the thing that is contained inside of them, and indeed there may even be multiple things inside of this rooted within different sub directories of the top level arbitrary collection of files. I agree with Paul though, I don't see any way this arbitrary collection of files/directories can be distributed on PyPI. PyPI is not just a place to dump whatever random archives you want for a package, it's specifically a place for Python package files (sdists, wheels, and eggs right now for the most part). It sounds like maybe you're just looking for a way to make it so that pip no longer makes these "arbitrary files/directories" installs work by treating them? like unpacked sdists and instead follows some supported path. If so, that is reasonable to me (and something I planned on getting around to). If that is what you're shooting for, I think it got confused by trying to mix in the sdist concept, as our sdists and wheels are not really for human consumption any more. I don't think that it makes sense for pip to go directly from a VCS[1] to a Wheel in the grand scheme of things. Right now we kind of do it, but that's because we just treat them like an unpacked sdist [2], long term though I don't think that is the correct way to do things. We (I?) want to minimize the different installation "paths" that can be taken and the main variable then when you do a ``pip install`` is how far long that path we already are. My ideal path looks something like this: ? ? VCS -> Source Wheel [3] -> Wheel -> Installed ? ? \-> Inplace Installation [4] So in that regards, I think that the only things (in my ideal world) people should be uploading to PyPI are source wheels (mandatory? [5]) and binary wheels (optional?). I don't know if downstream would prefer to use a source wheel or an arbitrary collection of files that may or may not be in a tarball, but I'm focused primarily on the needs of our toolchain, while still trying to make sure we don't end up in a situation that hurts downstream as well. I also don't think it makes sense for pip to ever really install these items, no matter what the transport mechanism is (VCS, tarball, unpacked on disk) without being explicitly pointed to by an URL or file path. There's obviously some backwards compatability issues here, because we can't just stop fetching .tar.gz links or anything, but I think the expectation should be that these items are only ever directly installed, not installed as part of the dependency resolution process. In that vein, they don't really participate in the dependency resolution process either (we'll install their dependencies and what not, but we'll assume that since you're pointing us to an explicit archive that we aren't going to resolve that particular dependency to anything other than what you've explicitly given us). If we're only installing these items when explicitly being given them by a direct URL or path, then a lot of the worries about non static metadata no longer exist, because as a user you're explicitly opting into installing something that isn't a real package, but is a VCS install of something that could be a package. Which we'll fetch, turn into a source wheel, and then turn that into a binary wheel (or do an inplace install). I also don't think it makes sense for these VCS installs to directly support outputting multiple different source wheels, but instead rely on the fact pip lets you say "install this artifact, but first CD into a particular directory". So you'd essentially just structure the filesystem of your VCS install to make independent little mini projects that could be independently packaged into their own VCS installs, but are all contained within some larger thing and we need to install a specific sub directory. I'm happy to be convinced otherwise, but given that this is a sort of edge case that a project will need this and we already have the subdirectory support I think it's simpler to just leverage that. Given my desire to reduce the number of installation paths we actually support, I think that trying to standardize this concept of a VCS install depends first on standardizing the concept of a source wheel (or sdist 2.0). This is because, in my mind, the main two things you can do with a VCS install, from pip's point of view, is do an in place installation or create a source wheel. I also agree with Paul that we should not be adding new formats to PyPI that do not support static metadata. What metadata in specific makes sense for that particular format (like ABI doesn't make sense for a source wheel/sdist) is going to be format dependent, but there should never be a "run this bit of code to find out X" situation. The closest we should get is "that type of metadata doesn't make sense for X, you need to build X into a Y for it". PyPI needs static metadata [6], and static metadata makes other tools easier to operate as well. [1] I'm going to just call these VCS installs, but I mean any install that is ? ? not a fully formed Python package and is instead just an arbitrary ? ? collection of files and directories where we can have a minimal amount of ? ? control over the structure. [2] This goes back to my belief that one of the "original sins" of distutils ? ? and setuptools was blurring the lines between the different phases in the ? ? life cycle of a package. [3] Unlike Paul, I think a Source Wheel is actually a decent name for this ? ? concept. It's similar to .rpm and .src.rpm in that world, and I think it ? ? makes it more obvious that this item isn't an installable item in it's own ? ? right, that it exists in order to produce binary wheels. However, this ? ? concept is currently being handled by the sdist "format", however ? ? ambigiously defined that currently is. I also think it makes it a bit ? ? easier to get rid of the ambigous "package", since we can just call them ? ? all "wheels" which is easier to say than "distribution". [4] Even though I want to reduce the number of paths we take, I don't think ? ? we'll ever be able to reasonably get rid of the inplace installation path. ? ? There will probably need to be some back and forth about exactly which ? ? parts of the inplace install the installer should be responsible for and ? ? which parts the build tool is responsible for. [5] Mandatory in the sense of, if you're going to upload any files to PyPI you ? ? must upload one of these, not mandatory in the absolute sense. In a future ? ? world, ideally the only thing people will need to upload is a source wheel ? ? and we'll have a build farm that will take that and automatically produce ? ? binary wheels from them. [6] In reality, we don't have static metadata today, and we get by. This is ? ? mostly because we force uploads to include static metadata alongside the ? ? upload and we present that instead. In the future I want to move us to a ? ? situation where you *just* upload the file, and PyPI inspects the file for ? ? all of the metadata it needs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Mon Oct 5 15:11:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 14:11:56 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 5 October 2015 at 13:44, Donald Stufft wrote: > [3] Unlike Paul, I think a Source Wheel is actually a decent name for this > concept. It's similar to .rpm and .src.rpm in that world, and I think it > makes it more obvious that this item isn't an installable item in it's own > right, that it exists in order to produce binary wheels. However, this > concept is currently being handled by the sdist "format", however > ambigiously defined that currently is. I also think it makes it a bit > easier to get rid of the ambigous "package", since we can just call them > all "wheels" which is easier to say than "distribution". Being a Windows user, I hadn't caught the parallel between Wheel-Source wheel and RPM-Source RPM (is there also a similar deb-source deb pair?) But if the concept works for people with a Linux background, I'm OK with it. (My main concern is that end users commonly make requests to projects saying "please provide wheels". I don't want that nice simple concept to get confused, if we can avoid it). Paul From solipsis at pitrou.net Mon Oct 5 15:18:37 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 15:18:37 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils References: Message-ID: <20151005151837.35894286@fsol> On Mon, 5 Oct 2015 14:11:56 +0100 Paul Moore wrote: > > Being a Windows user, I hadn't caught the parallel between > Wheel-Source wheel and RPM-Source RPM (is there also a similar > deb-source deb pair?) > > But if the concept works for people with a Linux background, I'm OK with it. The "source RPM" concept is actually confusing since, IIUC, a "source RPM" is nothing like a normal RPM (i.e. you can't install it using the "rpm" tool). It should actually be called a "RPM source" (again, IIUC). > (My main concern is that end users commonly make requests to projects > saying "please provide wheels". I don't want that nice simple concept > to get confused, if we can avoid it). +1 Regards Antoine. From solipsis at pitrou.net Mon Oct 5 15:22:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 15:22:01 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils References: Message-ID: <20151005152201.4433e9a6@fsol> On Mon, 5 Oct 2015 08:44:04 -0400 Donald Stufft wrote: > > I don't think that it makes sense for pip to go directly from a VCS[1] to a > Wheel in the grand scheme of things. Right now we kind of do it, but that's > because we just treat them like an unpacked sdist [2], long term though I don't > think that is the correct way to do things. We (I?) want to minimize the > different installation "paths" that can be taken and the main variable then > when you do a ``pip install`` is how far long that path we already are. My > ideal path looks something like this: > > ? ? VCS -> Source Wheel [3] -> Wheel -> Installed > ? ? \-> Inplace Installation [4] A valid use case may be to do an in-place installation from a sdist (although you may question the sanity of doing development from a source tree which isn't VCS-backed :-)). In any case, sdists being apt for human consumption is an important feature IMHO. Regards Antoine. From donald at stufft.io Mon Oct 5 15:25:39 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:25:39 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 9:11:59 AM, Paul Moore (p.f.moore at gmail.com) wrote: > On 5 October 2015 at 13:44, Donald Stufft wrote: > > [3] Unlike Paul, I think a Source Wheel is actually a decent name for this > > concept. It's similar to .rpm and .src.rpm in that world, and I think it > > makes it more obvious that this item isn't an installable item in it's own > > right, that it exists in order to produce binary wheels. However, this > > concept is currently being handled by the sdist "format", however > > ambigiously defined that currently is. I also think it makes it a bit > > easier to get rid of the ambigous "package", since we can just call them > > all "wheels" which is easier to say than "distribution". > > Being a Windows user, I hadn't caught the parallel between > Wheel-Source wheel and RPM-Source RPM (is there also a similar > deb-source deb pair?) > > But if the concept works for people with a Linux background, I'm OK with it. > > (My main concern is that end users commonly make requests to projects > saying "please provide wheels". I don't want that nice simple concept > to get confused, if we can avoid it). > > Paul > I'm not dead set on the source wheel name or anything either fwiw. I liked the parallels between that and .src.rpm and .rpm and I think the "sdist" name is kind of a mouthful to say. The debian analog to a source rpm is just a "source package", but that's really just a directory that you can run the debian build tools in, it's not really a package format like source RPMs are. Source RPMs have an "install" concept, but it's not really like pip's install, it just unpacks the file into place so that you can later build it. The main thing I wanted was something that wasn't a mouthful to say and that had a dedicated extension so we don't end up where the filename looks incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). I liked the idea of tying it into Wheels to keep the "new guard" of packaging formats related to each other, but I'm also happy to use a different term. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Mon Oct 5 15:26:30 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 05 Oct 2015 13:26:30 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Axle? On Mon, Oct 5, 2015 at 9:25 AM Donald Stufft wrote: > On October 5, 2015 at 9:11:59 AM, Paul Moore (p.f.moore at gmail.com) wrote: > > On 5 October 2015 at 13:44, Donald Stufft wrote: > > > [3] Unlike Paul, I think a Source Wheel is actually a decent name for > this > > > concept. It's similar to .rpm and .src.rpm in that world, and I think > it > > > makes it more obvious that this item isn't an installable item in it's > own > > > right, that it exists in order to produce binary wheels. However, this > > > concept is currently being handled by the sdist "format", however > > > ambigiously defined that currently is. I also think it makes it a bit > > > easier to get rid of the ambigous "package", since we can just call > them > > > all "wheels" which is easier to say than "distribution". > > > > Being a Windows user, I hadn't caught the parallel between > > Wheel-Source wheel and RPM-Source RPM (is there also a similar > > deb-source deb pair?) > > > > But if the concept works for people with a Linux background, I'm OK with > it. > > > > (My main concern is that end users commonly make requests to projects > > saying "please provide wheels". I don't want that nice simple concept > > to get confused, if we can avoid it). > > > > Paul > > > > I'm not dead set on the source wheel name or anything either fwiw. I liked > the > parallels between that and .src.rpm and .rpm and I think the "sdist" name > is > kind of a mouthful to say. The debian analog to a source rpm is just a > "source > package", but that's really just a directory that you can run the debian > build > tools in, it's not really a package format like source RPMs are. Source > RPMs > have an "install" concept, but it's not really like pip's install, it just > unpacks the file into place so that you can later build it. > > The main thing I wanted was something that wasn't a mouthful to say and > that > had a dedicated extension so we don't end up where the filename looks > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). I liked the > idea > of tying it into Wheels to keep the "new guard" of packaging formats > related to > each other, but I'm also happy to use a different term. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 5 15:28:48 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:28:48 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005152201.4433e9a6@fsol> References: <20151005152201.4433e9a6@fsol> Message-ID: On October 5, 2015 at 9:25:24 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Mon, 5 Oct 2015 08:44:04 -0400 > Donald Stufft wrote: > > > > I don't think that it makes sense for pip to go directly from a VCS[1] to a > > Wheel in the grand scheme of things. Right now we kind of do it, but that's > > because we just treat them like an unpacked sdist [2], long term though I don't > > think that is the correct way to do things. We (I?) want to minimize the > > different installation "paths" that can be taken and the main variable then > > when you do a ``pip install`` is how far long that path we already are. My > > ideal path looks something like this: > > > > VCS -> Source Wheel [3] -> Wheel -> Installed > > \-> Inplace Installation [4] > > A valid use case may be to do an in-place installation from a sdist > (although you may question the sanity of doing development from a > source tree which isn't VCS-backed :-)). > > In any case, sdists being apt for human consumption is an important > feature IMHO. > I don't think so. Remember in my footnote I mentioned that I wasn't using VCS to mean literally "something checked into a VCS", but rather the more traditional (outside of Python) "source release" concept. This could be a simple tarball that just has the proper, for human consumption, files in it or it could be a VCS checkout, or it could just be some files sitting on disk. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Mon Oct 5 15:25:14 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 05 Oct 2015 13:25:14 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: The OP asks for a Python callable interface to pip instead of setup.py's command line interface. That could be accomplished now by figuring out all of the arguments that pip will send to setup.py (setup.py egg_info and setup.py bdist_wheel)?, and then by writing a setup.py emulator that implements those commands by invoking the discovered callables, but it's hard to know exactly what that command line interface is. It has been a packaging TODO to write the list of arguments pip can send to setup.py down... Suppose someone from the pip team wrote the generic setup.py. It would implement the command line interface, import a script called, say, build.py, and invoke Python callables to do egg_info and bdist_wheel. Then the flit author could implement a couple of functions instead of having to reverse-engineer a command line interface. This would be an improvement because build system authors do not know how to extend setuptools or reverse engineer the necessary setup.py command line interface. On Mon, Oct 5, 2015 at 7:28 AM Paul Moore wrote: > OK, I've had a better read of your email now. Responses inline. > > On 5 October 2015 at 07:29, Nathaniel Smith wrote: > > First, let's drop the word "sdist", it's confusing. > > We can't (see below for details). We can deprecate the sdist concept, > if that's what you want to propose. From what I gather, you're > proposing deprecating it in favour of a "source wheel" concept. I > don't have a huge issue with that other than that I don't see the > necessity - the sdist concept pretty much covers what you want, except > maybe that it's not clear enough to people outside the packaging > community how it differs from a VCS checkout. > > > I'm starting from the long history and conventions around how people make > > what I'll call "source releases" (and in a few paragraphs will contrast > with > > "source wheels"). 'Everyone knows' that when you release a new version of > > some package (in pretty much any language), then one key step is to put > > together a file called -.. And > 'everyone > > knows' that if you see a file that follows this naming convention, and > you > > download it, then what you'll find inside is: a single directory called > > -/, and inside this directory will be something that's > > almost like a VCS checkout -- it'll probably contain a README, source > files > > in convenient places to be edited or grepped, etc. The differences from a > > VCS checkout (if any) will be little convenience stuff -- like > ./autogen.sh > > will have been run already, or there will be an extra file containing a > > fixed version number instead of it being autodetected, or -DNDEBUG will > be > > in the default CFLAGS, or Cython files will have been pre-translated to > C -- > > but fundamentally it will be similar to a VCS checkout, and switching > back > > and forth between them won't be too jarring. 95% of the time there will > be a > > standard way to build the thing ('./configure && make && make install', > or > > 'python setup.py install', or similar). > > Breaking at this point, because that's frankly *not* the reality in > the Python packaging world (at least not nowadays - I'm not clear to > what extent you're just talking about history and background here, > although your reference to Cython makes me think you're talking in > terms of current practice). It may look like that, but there are some > fundamental differences. > > First and foremost, nobody zips up and publishes their VCS checkout in > the way you describe. (At least not if they are using the standard > tools - distutils and setuptools). Instead, they create a "sdist" > using the "python setup.py sdist" command. I'm sorry, but I'm going to > carry on using the "sdist" term here, because I'm describing current > practice and sdists *are* current practice. > > The difference is between a sdist and what you call a "source release" > is subtle, precisely because the current sdist format is a bit of a > mess, but the key point is that all sdists are created by a standard > process, and conform to a standard naming convention and layout. The > packaging tools rely on being able to make that assumption, in all > sorts of ways which we're doing our best to clarify as part of this > thread, but which honestly have been a little bit implicit up to this > point. > > Further muddying the water is the fact that as you say, pip needs to > be able to build from a VCS checkout (a directory on the user's local > system) and we have code in pip that does that - mostly by assuming > that you can treat a VCS checkout as an unpacked sdist, but there are > hacks we need to do to make that work (we run setup.py egg-info to get > the metadata we need, for example, which has implications as we only > get that data at a later stage than we have it in the sdist case) and > differences in functionality (develop mode). > > At this point I'm not saying that things have to be this way, or even > that "make a source release however you choose as long as it follows > these conventions" isn't a viable way forward, but I do think we need > to agree on our picture of how things are now, or we'll continue > talking past each other. > > > And these kind of source releases > > have a rich ecosystem around them and serve a wide range of uses: they > > provide a low-tech archival record (while VCS's come and go), they end > up in > > deb and rpm "original source" bundles, they get downloaded by users and > > built by hand (maybe with weird configury on top, like a hack to enable > > cross-compilation) or poked around in by hand, etc. etc. When sdists were > > originally designed, then "source releases" is what the designers were > > thinking about. > > This, on the other hand, I suspect is not far from the truth. When > sdists were designed, they were a convenience for bundling the stuff > needed to do setup.py install later, possibly on a different machine. > > But that's a long time ago, and not really relevant now. For better or > worse. Unless you are suggesting that we go all the way back to that > original point? Which you may be, but that means discarding the work > that's been done based on the sdist concept since then. Which leads > nicely on to... > > > Then, easy_install came along, and pulled off a clever hack where when > you > > asked for some package by name, then it would try to automagically go out > > and track down any relevant source releases and build them all. And it > works > > great, except when it doesn't. And creates massive headaches for everyone > > trying to work on python packaging afterwards, because source releases > were > > not designed to be used this way. > > > > My hypothesis is that the requirements that were confusing me are based > > around the idea that an sdist should be something designed to slot into > this > > particular use case: i.e., something that pip can automatically grab and > > work with while solving a dependency resolution problem. Therefore it > really > > needs to have a static name, and static version number, and static > > dependencies, and must produce exactly one binary wheel that shares all > that > > metadata. > > Anyone who knows my history will know that I'm the last person to > defend setuptools' hacks, but you hit the nail on the head above. When > it works, it works great (meh, "sufficiently well" :-)) > > And pip *needs* to do static dependency resolution. We have enough bug > reports and feature requests asking that we improve the dependency > resolution process that nobody is going to be happy with anything that > doesn't allow for at least as much static information as we currently > have, and ultimately more. > > > Let's call this a "source wheel" -- what we're really looking for > > here is a way to ship extension modules inside something that acts like > an > > architecture-neutral "none-any" wheel. > > I don't understand this statement. What do extension modules matter > here? We need to be able to ship sources in a form that can > participate in dependency resolution (and any other potential > discovery processes that may turn up in future) without having to run > code to do so. The reasons for this are: > > 1. Running code is a performance overhead, and possibly even a > security risk (even trusted code may behave in ways you didn't > anticipate). We want to do as little as possible of that as we can, > and in particular we want to discard invalid candidate files without > running any of their code. > 2. Running code introduces the possibility of that code failing. We > don't want end users to have installs fail because code in > distributions we're going to discard is buggy. > 3. Repositories like PyPI need to present project metadata, for both > human and tool consumption - they can only do this if it's available > statically. > > You seem to be thinking that binary wheels are sufficient for this for > pure-Python code. Look at it the other way - we discard sdists from > the dependency calculations whenever there's an equivalent binary > wheel available. That's always for non-any wheels, but less often for > architecture-dependent wheels. But in order to know that the wheel is > equivalent, we need to match it with the sdist - so the sdist needs > the metadata you're trying to argue against providing... > > > So: the email that started this thread was a proposal for how to > standardize > > the format of "source releases", and Donald's counter was a proposal for > how > > to standardize the format of "source wheels". Does that sound right? > > Well, essentially yes, although I read it as your original email being > a proposal for a new format to replace sdists, and Donald's and my > counter is that there's already a been a certain amount of thinking > and design gone into how we move from the current ad-hoc sdist format > to a better defined and specified "next version", so how does your > proposal affect that? > > It seems that your answer is that you want to bypass that and offer an > alternative. Is that fair? > > For the record, I don't like the term "source wheel" and would prefer > to stick with "sdist" if appropriate, or choose a term that doesn't > include the word "wheel" otherwise (as wheels seem to me to be > strongly, and beneficially, linked in people's minds to the concept of > a binary release format). > > > If so, then some follow-up thoughts: > > > > 1) If we design a source wheel format, then I am 100% in favor of the > > suggestion of giving it a unique extension like "swhl". I'm still a young > > whippersnapper compared to some, but I've been downloading files named > > -. for 20 years, and AFAICR every one of > > those files unpacked to make a single directory that was laid out like a > VCS > > checkout. Obviously we can change what goes inside, but we should change > the > > naming convention at the same time because otherwise we're just going to > > confuse people. > > I have no problem with making it clear that "sdist version 2" or > "source wheel" is not the same as a packed VCS checkout. I don't see > the need for a new term, I'd be happy with "-.sdist" > as the name. I'd also like to emphasize strongly that PyPI only hosts > sdists, and *not* source releases - that source releases are typically > only seen in Python in the form of a VCS checkout or development > directory. > > (There's an implication there that we need to explore, that pip won't > necessarily gain the ability to be pointed at a non-sdist format > packed "source release" archive, and download it and process it. > That's not a given, but I'd like to be sure we are happy with the > potential re-introduction of confusion over the distinction between a > sdist/source wheel and a source release that would result). > > > 2) I think there's a strong case to be made that Python actually needs > > standards for *both* source releases and source wheels. There's > certainly no > > logical contradiction -- they're conceptually different things. It sounds > > like we all agree that "pip" should continue to have a way to build and > > install an arbitrary VCS checkout, and extending that standard to cover > > building and installing a classic "source release" would be... almost > > difficult *not* to do. > > As noted above, I'm happy for that discussion to occur. But I'm *not* > sure the case is as strong as you think. Technically, it's certainly > not too hard, but the social issues are what concern me. How will we > explain to someone that they can't upload their file to PyPI because > it's a "source release" not a "source wheel"? What is the implication > on people's workflow? How would we explain why people might want to > make "source releases" *at all*? Personally, I can only see a need in > my personal experience for a VCS url that people can clone, a packaged > source artifact that I can upload to PyPI for automatic consumption, > and (binary) wheels. That second item is a "source wheel" - not a > "source release". > > > And I think that there will continue to be a clear need for source > releases > > even in a world where source wheels exist, because of all those places > where > > source releases get used that aren't automagic-easy_install/pip-builds. > For > > example, most pure Python packages (the ones that already make "none-any" > > wheels) have no need at all for source wheels, but they still need source > > releases to serve as archival snapshots. And more complex packages that > need > > build-time configuration (e.g. numpy) will continue to require source > > releases that can be configured to build wheels that have a variety of > > different properties (e.g., different dependency metadata), so they can't > > get by with source wheels alone -- but you can imagine that such projects > > might reasonably *in addition* provide a source wheel that locks down the > > same default configuration that gets used for their uploaded binary wheel > > builds, and is designed for pip to use when trying to resolve > dependencies > > on platforms where a regular binary wheel is unavailable. > > > > Pictorially, this world would look like: > > > > VCS checkout -> source release > > \ \ > > --------------------+--> in-place install > > | > > +--> wheels -> install > > | > > +--> source wheels -> wheels -> install > > I don't see the need for source releases that you do. That's likely > because I don't deal with the sorts of complex projects you do, > though, so I'm not dismissing the issue. As I say, my objections are > mostly non-technical. I do think you should consider how to document > "what a source release is intended to achieve" in a way that explains > it to people who don't need the complexity it adds - and with the > explicit goal of making sure that you dissuade people who *don't* need > source releases from thinking they do. > > > 3) It sounds like we all agree that > > - 'pip install ' should work > > Yes. > > > - that there is some crucial metadata that VCS checkouts won't be able > to > > provide without running arbitrary code (e.g. dependencies and version > > numbers) > > I'm still resisting this one, although I can live with "Nathanial > tells me so" :-) > > > - that what metadata they do provide (e.g., which arbitrary code to > run) > > should be specified in a human-friendly configuration file > > I don't agree to that one particularly, in the sense that I don't > really care. I'd be happy with a system that said something like that > for a VCS checkout, pip expects "setup.py egg-info" and "setup.py > sdist" to work, and produce respectively a set of static metadata in a > known location, and a properly formatted "source wheel"/sdist file. > Non-distutils build tools can write a wrapper setup.py that works > however they prefer. (That's roughly what we have now, BTW). > > > Given this, in the big picture it sounds like the only really essentially > > controversial things about the original proposal might be: > > - that 'pip install ' should work the same as > > 'pip install ' (does anyone actually disagree?) > > Yes, to the extent that I want to ensure it's clearly documented how, > and why, this differs from a sdist/source wheel. But that's not a > technical issue. > > > - the 1 paragraph describing the "transitional plan" allowing pip to > > automatically install from these source releases *as part of a dependency > > resolution plan* (as opposed to when a VCS-checkout-or-equivalent is > > explicitly given as an install target). Which honestly I don't like > either, > > it's just there as a backcompat measure so that these source releases > don't > > create a regression versus existing sdists -- note that one of the goals > of > > the design was that current sdists could be upgraded into this format by > > dropping in a single static file (or by an install tool "virtually" > dropping > > in this file when it encounters an old-style sdist -- so you don't need > to > > keep around code to handle both cases separately). > > Given that these source releases won't be hosted on PyPI (as the > proposal currently stands) there's no real need for this - all you > need to say is that you can point pip at any old URL and take your > chances :-) > > > Does that sound right? > > Not really. For me the big controversy is whether we move forward from > where we are with sdists, or we ignore the current sdist mechanism and > start over. > > A key further question, which I don't think has been stated explicitly > until I started this email, is what formats will be supported for > hosting on PyPI. I am against hosting formats that don't support > static metadata, such as your "source distribution", as I don't see > how PyPI would be able to publish the metadata if it weren't static. > > And following on from that, we need to agree whether the key formats > should be required to have a static version. I'm OK with a VCS > checkout having a dynamically generated version, that's part of the > "all bets are off" contract over such things (if you don't generate a > version that reflects every change, you get to deal with the > consequences) but I don't think that's a reasonable thing to allow in > "published" formats. > > > (Other features of the original proposal include stuff like the lack of > > trivial metadata like "name" and "description", and the support for > > generating multiple wheels from one directory. I am explicitly calling > these > > "inessential".) > > Not sure what you mean by lack of name/description being > "inessential". The double negative confuses me. Do you mean you're OK > with requiring them? Fair enough. > > For multiple wheels, I'd tend to consider the opposite to be true - > it's not that the capability is non-essential, but rather that in > published formats (source wheel and later in the chain) it's essential > that one source generates one target. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Oct 5 15:39:10 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 15:39:10 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> Message-ID: <20151005153910.7f921992@fsol> On Mon, 5 Oct 2015 09:28:48 -0400 Donald Stufft wrote: > > > > In any case, sdists being apt for human consumption is an important > > feature IMHO. > > > > I don't think so. Remember in my footnote I mentioned that I wasn't using VCS > to mean literally "something checked into a VCS", but rather the more > traditional (outside of Python) "source release" concept. This could be a > simple tarball that just has the proper, for human consumption, files in it > or it could be a VCS checkout, or it could just be some files sitting on disk. But why use two different formats for "source release" and "sdists"? Currently sdists fit the assumptions for a source release, why introduce some complexity and have the users deal with separate concepts (with all the confusion that will inevitably ensue)? Regards Antoine. From p.f.moore at gmail.com Mon Oct 5 15:41:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 14:41:25 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 5 October 2015 at 14:25, Donald Stufft wrote: > The main thing I wanted was something that wasn't a mouthful to say and that > had a dedicated extension so we don't end up where the filename looks > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). You find "Source Wheel" easier to say / type than "sdist"? (I type it more than say it, but typing sdist is easier, and when I say it I usually go for "ess-dist, or suh-dist"). And ".sdist" seems like a perfectly good suffix to me. And the bikeshed should be a light purple colour :-) Paul From wes.turner at gmail.com Mon Oct 5 15:45:27 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 5 Oct 2015 08:45:27 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 5, 2015 8:41 AM, "Paul Moore" wrote: > > On 5 October 2015 at 14:25, Donald Stufft wrote: > > The main thing I wanted was something that wasn't a mouthful to say and that > > had a dedicated extension so we don't end up where the filename looks > > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). > > You find "Source Wheel" easier to say / type than "sdist"? (I type it > more than say it, but typing sdist is easier, and when I say it I > usually go for "ess-dist, or suh-dist"). And ".sdist" seems like a > perfectly good suffix to me. How about something like .sdist.whl.zip ? There are already many tools with e.g. ZIP MIME extensions so that one can open the file and browse the archive with a file explorer without a platform specific file type association install step > > And the bikeshed should be a light purple colour :-) > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 5 15:51:05 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:51:05 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005153910.7f921992@fsol> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> Message-ID: <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Because the status quo, which is a single format, sucks. I don't think changing it so we invoke a Python function instead of a script is going to make it not suck. The fundamental reason it sucks (in my mind) is that we have a single format trying to do way too many things and we need to break that out into smaller chunks. You can see these needs are different I think by looking at how what Nathaniel wants differs from what me and Paul want. He wants something that will make the human side easier and will support different tools, we want something that pip can consume more reasonably. Trying to do too much with a singular format just means it sucks for all the uses cases instead of being great for one use case. I also don't think it will be confusing. They'll associate the VCS thing (a source release) as something focused on development for most everyone. Most people won't explicitly make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with manually producing binary wheels or producing source releases other than what's checked into vcs will be something that I suspect only advanced users will do. Sent from my iPhone > On Oct 5, 2015, at 9:39 AM, Antoine Pitrou wrote: > > On Mon, 5 Oct 2015 09:28:48 -0400 > Donald Stufft wrote: >>> >>> In any case, sdists being apt for human consumption is an important >>> feature IMHO. >> >> I don't think so. Remember in my footnote I mentioned that I wasn't using VCS >> to mean literally "something checked into a VCS", but rather the more >> traditional (outside of Python) "source release" concept. This could be a >> simple tarball that just has the proper, for human consumption, files in it >> or it could be a VCS checkout, or it could just be some files sitting on disk. > > But why use two different formats for "source release" and "sdists"? > Currently sdists fit the assumptions for a source release, why > introduce some complexity and have the users deal with separate > concepts (with all the confusion that will inevitably ensue)? > > Regards > > Antoine. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Mon Oct 5 15:54:09 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:54:09 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 9:45:29 AM, Wes Turner (wes.turner at gmail.com) wrote: > > > How about something like > .sdist.whl.zip ? > > > There are already many tools with e.g. ZIP MIME extensions > so that one can open the file and browse the archive with a file > explorer > without a platform specific file type association install step >? -1 on using a .zip extension. I don?t really want people opening these up to inspect them unless they know what they are doing. For most people, they should just think of them as blobs of stuff. +0 on .sdist.whl instead of .swhl or .src.whl. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Mon Oct 5 16:01:30 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 10:01:30 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 9:41:28 AM, Paul Moore (p.f.moore at gmail.com) wrote: > On 5 October 2015 at 14:25, Donald Stufft wrote: > > The main thing I wanted was something that wasn't a mouthful to say and that > > had a dedicated extension so we don't end up where the filename looks > > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). > > You find "Source Wheel" easier to say / type than "sdist"? (I type it > more than say it, but typing sdist is easier, and when I say it I > usually go for "ess-dist, or suh-dist"). And ".sdist" seems like a > perfectly good suffix to me. Yea I do, but I have a stutter and "d" (particularly trying to transition to it from the "s") and "st" are hard for me to get the sounds out anyways (ironic given my name contains both a D and a St!). I don't know how hard it is for someone else to say, probably slightly harder but not by much. This is why I also hate the "distribution" term, I have a hard time saying the whole word without getting stuck in the "stribut" part. Anyways, name is the least important part to me as long as we get a dedicated extension out of it :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Mon Oct 5 16:52:02 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 16:52:02 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: <20151005165202.0709e341@fsol> On Mon, 5 Oct 2015 09:51:05 -0400 Donald Stufft wrote: > > You can see these needs are different I think by looking at how what Nathaniel wants differs from what me and Paul want. He wants something that will make the human side easier and will support different tools, we want something that pip can consume more reasonably. Trying to do too much with a singular format just means it sucks for all the uses cases instead of being great for one use case. That doesn't seem to follow. You can have regular human-compatible content and a few machine-compatible files besides (perhaps in a dedicated subdirectory). I don't see how that "sucks". > I also don't think it will be confusing. They'll associate the VCS thing (a source release) as something focused on development for most everyone. Most people won't explicitly make one and nobody will be uploading it to PyPI. Well, what is the point of standardizing the concept of source releases if nobody produces them? Regards Antoine. From donald at stufft.io Mon Oct 5 17:10:53 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 11:10:53 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005165202.0709e341@fsol> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> <20151005165202.0709e341@fsol> Message-ID: On October 5, 2015 at 10:52:22 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Mon, 5 Oct 2015 09:51:05 -0400 > Donald Stufft wrote: > > > > You can see these needs are different I think by looking at how what Nathaniel wants differs > from what me and Paul want. He wants something that will make the human side easier and > will support different tools, we want something that pip can consume more reasonably. > Trying to do too much with a singular format just means it sucks for all the uses cases instead > of being great for one use case. > > That doesn't seem to follow. You can have regular human-compatible > content and a few machine-compatible files besides (perhaps in a > dedicated subdirectory). I don't see how that "sucks?. Machine compatible needs things to be as tightly specified as possible, human compatible needs it to be as loosely specified as possible. In addition, the original proposal had things like "version" being dynamically computed because it might change based on something in the directory (like if the VCS gets another commit). For a machine format, we want all metadata to be static, for something designed for humans to interact with and much around with the code we want something that is more dynamic. If you have two different files one for machines and one for humans, what happens if they disagree? There should be a single source of truth in any one artifact for all of the information. We currently have the situation you describe, a PKG-INFO that is static metadata and a setup.py that is aimed towards humans... but nothing uses the PKG-INFO because the source of truth is the setup.py and the PKG-INFO might be lies. So we have a lot of effort and hacks going in to make something that is best designed for human interaction to be used for machines. > > > I also don't think it will be confusing. They'll associate the VCS thing (a source release) > as something focused on development for most everyone. Most people won't explicitly > make one and nobody will be uploading it to PyPI. > > Well, what is the point of standardizing the concept of source releases > if nobody produces them? > Well, a VCS commit will be a form of one, just not one that people explicitly distribute. I don't think it makes a sense to standard this idea of a "source release" as something different other than what we need to bootstrap a VCS based build as well. So we need to support it for VCSs anyways, and divorcing the standard from VCSs makes it more robust and allows the idea of a .tar.gz based "VCS" build anyways. I also didn't say nobody would produce them. I said most people would have no need for explicit source releases (particularly in a tarball form) and wouldn't produce them. If I thought *nobody* would produce them, then I wouldn't be willing to have them standardized. I think that they might be useful for some projects that have more advanced needs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From fungi at yuggoth.org Mon Oct 5 17:14:39 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 5 Oct 2015 15:14:39 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005153910.7f921992@fsol> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> Message-ID: <20151005151439.GM4731@yuggoth.org> On 2015-10-05 15:39:10 +0200 (+0200), Antoine Pitrou wrote: [...] > But why use two different formats for "source release" and "sdists"? > Currently sdists fit the assumptions for a source release, why > introduce some complexity and have the users deal with separate > concepts (with all the confusion that will inevitably ensue)? An sdist is an installable package which just happens to _look_ a lot like a source release tarball, but trying to pretend that downstream packagers will want to use it as such leads to a variety of pain points in the upstream/downstream relationship. For better or worse a lot of distros don't want generated files in upstream source code releases, since they need to confirm that they also ship the necessary tooling to regenerate any required files and that the generated files they ship match what their packaged tooling produces. While this similarity was probably seen as a "Good Thing [TM]" initially (hence standardizing on a .tar.gz extension), over time both the generated content of a typical sdist and the concern most distros have over shipping upstream-generated files has increased to the point where they really need to be viewed as separate and distinct release artifacts now. -- Jeremy Stanley From dholth at gmail.com Mon Oct 5 18:05:41 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 05 Oct 2015 16:05:41 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: sdists work well most of the time. Some problems 1 - hard to depend on something setup.py must import 2 - hard to use non-distutils build system due to the first problem and a poorly defined command line interface 3 - have to regenerate metadata to get dependencies Assume we are not ever going to be able to remove support for pip install git+https:// or the current sdist. So you need new-sdist for 3, but could add 1 & 2 within the current framework by using an elaborate generic setup.py or by adding a measurable amount of complexity to pip. Perhaps the underlying code for 1 & 2 is needed to implement features desired in new-sdist. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 5 18:08:07 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 12:08:07 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 12:05:53 PM, Daniel Holth (dholth at gmail.com) wrote: > sdists work well most of the time. Some problems > > 1 - hard to depend on something setup.py must import > 2 - hard to use non-distutils build system due to the first problem and a > poorly defined command line interface > 3 - have to regenerate metadata to get dependencies > > Assume we are not ever going to be able to remove support for pip install > git+https:// or the current sdist. > > So you need new-sdist for 3, but could add 1 & 2 within the current > framework by using an elaborate generic setup.py or by adding a measurable > amount of complexity to pip. Perhaps the underlying code for 1 & 2 is > needed to implement features desired in new-sdist. > I'm not opposed to layering things ontop of the current sdist format in the interim to solve problems that currently exist until a better solution can be done. That's not what this PEP idea was though, it was creating a whole new sdist. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From guettliml at thomas-guettler.de Tue Oct 6 09:07:46 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Tue, 6 Oct 2015 09:07:46 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? Message-ID: <56137342.8090201@thomas-guettler.de> Hi, Where should I put tests when packaging python modules? I want a "cowpath", an "obvious way" Dear experts, please decide: inside the module like this answer: http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules XOR outside the module like this: https://github.com/pypa/sampleproject/tree/master/tests I think there is no need to hurry. Let's wait one week, and then check which one is preferred. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From glyph at twistedmatrix.com Tue Oct 6 09:20:36 2015 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 6 Oct 2015 00:20:36 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <56137342.8090201@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> Message-ID: <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> > On Oct 6, 2015, at 12:07 AM, Thomas G?ttler wrote: > > Hi, > > Where should I put tests when packaging python modules? > > I want a "cowpath", an "obvious way" > > Dear experts, please decide: > > inside the module like this answer: > > http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules > > XOR > > outside the module like this: > > https://github.com/pypa/sampleproject/tree/master/tests > > I think there is no need to hurry. Let's wait one week, > and then check which one is preferred. > > Regards, > Thomas G?ttler Inside the package. If you put your tests outside your package, then you can't install the tests for two packages simultaneously, because everyone's tests are just in the top-level package "tests". This tends to infest the whole package, since then tests import things from each other using 'from tests import ...'. This is recommended by the hitchhiker's guide, and seconded by . -glyph From ben+python at benfinney.id.au Tue Oct 6 09:38:33 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 06 Oct 2015 18:38:33 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> Message-ID: <85egh81m6e.fsf@benfinney.id.au> Thomas G?ttler writes: > Where should I put tests when packaging python modules? When packaging them? The same place they go when not packaging them :-) > I want a "cowpath", an "obvious way" For me, the obvious way is to have: > outside the module like this: > https://github.com/pypa/sampleproject/tree/master/tests and have ?setup.py? ensure the tests are distributed with the source package, but not installed. > I think there is no need to hurry. Let's wait one week, and then check > which one is preferred. More important than which is preferred, we should use the one which is best (regardless how popular it may be). So instead of just hearing votes, we should examine reasoned argument in favour of good options. -- \ ?Visitors are expected to complain at the office between the | `\ hours of 9 and 11 a.m. daily.? ?hotel, Athens | _o__) | Ben Finney From solipsis at pitrou.net Tue Oct 6 09:51:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 09:51:01 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> Message-ID: <20151006095101.031c8a56@fsol> On Tue, 6 Oct 2015 09:07:46 +0200 Thomas G?ttler wrote: > > Dear experts, please decide: > > inside the module like this answer: > > http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules They should be inside the module. That way, you can check an installed module is ok by running e.g. "python -m mypackage.tests". Any other choice makes testing installed modules more cumbersome. > outside the module like this: > > https://github.com/pypa/sampleproject/tree/master/tests There is no actual reason to do that except win a couple kilobytes if you are distributing your package on floppy disks for consumption on Z80-based machines with 64KB RAM. Even Python *itself* puts its test suite inside the standard library, not outside it (though some Linux distros may strip it away). Try "python -m test.regrtest" (again, this may fail if your distro decided to ship the test suite in a separate package). The PyP"A" should definitely fix its sample project to reflect good practices. Regards Antoine. From p.f.moore at gmail.com Tue Oct 6 10:17:22 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Oct 2015 09:17:22 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006095101.031c8a56@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On 6 October 2015 at 08:51, Antoine Pitrou wrote: > They should be inside the module. That way, you can check an installed > module is ok by running e.g. "python -m mypackage.tests". Any other > choice makes testing installed modules more cumbersome. One inconvenience with this is that if you use an external testing framework like nose or pytest, you either need to make your project depend on it, or you need to document that "python -m mypackage.tests" has additional dependencies that are not installed by default. With an external tests directory, the testing framework is just another "development requirement". It's only a minor point, conceded. > The PyP"A" should definitely fix its sample project to reflect good > practices. +1 on the sample project following recommended guidelines. When I originally wrote that project, the consensus (such as it was) was in favour of tests outside the installed project. But that may have simply reflected the group of people who responded at the time. It does however have the advantage that it's consistent with how other PyPA projects like pip and virtualenv structure their tests. The big problem is that I don't think there *is* a consensus on best practice. Both options have their supporters. It would also be very easy to take the view that the PyPA sample project should omit the test directory altogether, as it's a sample for the *packaging* guide, and development processes like testing are out of scope (that's why we don't include a documentation directory, or recommend sphinx, for example). Personally, I think if we go to that extreme, the sample project becomes useless as a "real world" template, but maybe people wanting a starter project template should be directed to projects such as cookiecutter instead. Paul From wes.turner at gmail.com Tue Oct 6 10:25:17 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Oct 2015 03:25:17 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Oct 6, 2015 3:17 AM, "Paul Moore" wrote: > > On 6 October 2015 at 08:51, Antoine Pitrou wrote: > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > One inconvenience with this is that if you use an external testing > framework like nose or pytest, you either need to make your project > depend on it, or you need to document that "python -m mypackage.tests" > has additional dependencies that are not installed by default. > > With an external tests directory, the testing framework is just > another "development requirement". > > It's only a minor point, conceded. otherwise try/except imports - [ ] @skipif in stdlib would be helpful > > > The PyP"A" should definitely fix its sample project to reflect good > > practices. > > +1 on the sample project following recommended guidelines. When I > originally wrote that project, the consensus (such as it was) was in > favour of tests outside the installed project. But that may have > simply reflected the group of people who responded at the time. It > does however have the advantage that it's consistent with how other > PyPA projects like pip and virtualenv structure their tests. many of these cookiecutter Python project templates also define e.g. tox.ini and .travis.yml: https://github.com/audreyr/cookiecutter-pypackage/blob/master/README.rst#similar-cookiecutter-templates > > The big problem is that I don't think there *is* a consensus on best > practice. Both options have their supporters. > > It would also be very easy to take the view that the PyPA sample > project should omit the test directory altogether, as it's a sample > for the *packaging* guide, and development processes like testing are > out of scope (that's why we don't include a documentation directory, > or recommend sphinx, for example). Personally, I think if we go to > that extreme, the sample project becomes useless as a "real world" > template, but maybe people wanting a starter project template should > be directed to projects such as cookiecutter instead. I think it wise to encourage TDD. https://github.com/audreyr/cookiecutter/blob/master/README.rst#python > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 10:44:15 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 10:44:15 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006104415.7fb554a2@fsol> On Tue, 6 Oct 2015 09:17:22 +0100 Paul Moore wrote: > On 6 October 2015 at 08:51, Antoine Pitrou wrote: > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > One inconvenience with this is that if you use an external testing > framework like nose or pytest, you either need to make your project > depend on it, or you need to document that "python -m mypackage.tests" > has additional dependencies that are not installed by default. > > With an external tests directory, the testing framework is just > another "development requirement". Doesn't / didn't setuptools have something called test_requires? > It would also be very easy to take the view that the PyPA sample > project should omit the test directory altogether, as it's a sample > for the *packaging* guide, and development processes like testing are > out of scope (that's why we don't include a documentation directory, > or recommend sphinx, for example). That sounds like the best course to me. Regards Antoine. From contact at ionelmc.ro Tue Oct 6 10:44:44 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 11:44:44 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On Tue, Oct 6, 2015 at 10:20 AM, Glyph Lefkowitz wrote: > If you put your tests outside your package, then you can't install the > tests for two packages simultaneously, because everyone's tests are just in > the top-level package "tests". This tends to infest the whole package, > since then tests import things from each other using 'from tests import > ...'. This is recommended by the hitchhiker's guide, and seconded by < > http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html > >. > ?I don't want to be harsh here but arguments would be way more interesting to discuss, as opposed to giving links to JPC's outdated packaging guide. He's an exceptional developer but many of the things inside that article are out of fashion (he disagrees about using console_scripts entrypoints?, doesn't understand what src is for and so on). I don't think anyone would ever intentionally put a "tests" package in site-packages, why would you mention that? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Tue Oct 6 10:30:00 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 11:30:00 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006095101.031c8a56@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > They should be inside the module. That way, you can check an installed > module is ok by running e.g. "python -m mypackage.tests". Any other > choice makes testing installed modules more cumbersome. > ?Does that really make sense? I haven't heard of any user actually running tests? that way. To be honest I haven't ever ran Python's own tests suite as part of a user installation. ?I've seen some projects that lump up lots of test data and crazy files in their packages tests and that created install issues? (Pelican is one example, pretty sure there are others). On the other hand, if the user really wants to run the tests he can just get the sources (that would naturally include everything)? Seems odd to suggest something is a best practice without giving any clue of how test dependencies would be managed. Just because CPython does it doesn't mean libraries should. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Tue Oct 6 11:18:03 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 06 Oct 2015 20:18:03 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <85612k1hkk.fsf@benfinney.id.au> Ionel Cristian M?rie? writes: > I've seen some projects that lump up lots of test data and crazy files > in their packages tests and that created install issues. > > On the other hand, if the user really wants to run the tests he can > just get the sources (that would naturally include everything)? Yes, this is a sensible approach: * The source package contains all the source files a developer would use to make further changes and test them. * The package for installation contains only those files useful run-time users, plus metadata (e.g. copyright information). I highly recommend it, and I would like the PyPA to also recommend the above approach. It does, though, require the acknowledgement of a separate *build* step in the development-and-release process. The build step is prior to the packaging step, and it generates run-time files from the collection of source files. That separation of a discrete build step is crucial for many good practices: generate documentation, integration testing, OS packaging, etc. -- \ ?The restriction of knowledge to an elite group destroys the | `\ spirit of society and leads to its intellectual | _o__) impoverishment.? ?Albert Einstein | Ben Finney From cournape at gmail.com Tue Oct 6 11:24:30 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 10:24:30 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Tue, Oct 6, 2015 at 9:30 AM, Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou > wrote: > >> They should be inside the module. That way, you can check an installed >> module is ok by running e.g. "python -m mypackage.tests". Any other >> choice makes testing installed modules more cumbersome. >> > > ?Does that really make sense? I haven't heard of any user actually running > tests? > that way. To be honest I haven't ever ran Python's own tests suite as part > of a user installation. > It makes a lot of sense for downstream packagers. Allowing testing installed packages is also the simplest way to enable testing on target machines which are different from the build machines. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Oct 6 11:51:44 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Oct 2015 04:51:44 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Oct 6, 2015 4:24 AM, "David Cournapeau" wrote: > > > > On Tue, Oct 6, 2015 at 9:30 AM, Ionel Cristian M?rie? wrote: >> >> >> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: >>> >>> They should be inside the module. That way, you can check an installed >>> module is ok by running e.g. "python -m mypackage.tests". Any other >>> choice makes testing installed modules more cumbersome. >> >> >> ?Does that really make sense? I haven't heard of any user actually running tests? >> that way. To be honest I haven't ever ran Python's own tests suite as part >> of a user installation. > > > It makes a lot of sense for downstream packagers. Allowing testing installed packages is also the simplest way to enable testing on target machines which are different from the build machines. self-testable programs are really ideal (e.g POST power-on self test) relevant recent topical discussion of e.g CRC and an optional '-t' preemptive CLI parameter: https://github.com/audreyr/cookiecutter-pypackage/pull/52 > > David > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 11:54:25 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 05:54:25 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On October 6, 2015 at 3:21:04 AM, Glyph Lefkowitz (glyph at twistedmatrix.com) wrote: > > Inside the package. > > If you put your tests outside your package, then you can't install > the tests for two packages simultaneously, because everyone's > tests are just in the top-level package "tests". This tends to > infest the whole package, since then tests import things from > each other using 'from tests import ...'. This is recommended > by the hitchhiker's guide, and seconded by .? I dislike putting tests inside the package. The supposed benefit is that anyone can run the tests at anytime, but I don't find that actually true because it means (as someone else pointed out) that you either have to depend on all your test dependencies or that there is already an additional step to install them. If you're going to have to locate and install the test dependencies, then you might as well fetch the tarball with tests as well. Someone suggested setuptools test_requires, but that only functions when you have a setup.py available and you execute ``setup.py test``. It does not help you at all once the package is installed and the sdist is gone. I also don't think people actually run the tests when they are installed in any significant number, at least I've never once in my life done it or even had a desire to do it. Some projects have test suites which are significantly large too. The PyCA cryptography project for instance, has to ship it's vectors as an additional package to reduce the size of the final build product. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Tue Oct 6 12:18:30 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 11:18:30 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On Tue, Oct 6, 2015 at 10:54 AM, Donald Stufft wrote: > On October 6, 2015 at 3:21:04 AM, Glyph Lefkowitz (glyph at twistedmatrix.com) > wrote: > > > Inside the package. > > > > If you put your tests outside your package, then you can't install > > the tests for two packages simultaneously, because everyone's > > tests are just in the top-level package "tests". This tends to > > infest the whole package, since then tests import things from > > each other using 'from tests import ...'. This is recommended > > by the hitchhiker's guide, and seconded by . > > I dislike putting tests inside the package. > > The supposed benefit is that anyone can run the tests at anytime, but I > don't > find that actually true because it means (as someone else pointed out) > that you > either have to depend on all your test dependencies or that there is > already an > additional step to install them. If you're going to have to locate and > install > the test dependencies, then you might as well fetch the tarball with tests > as > well. > > Someone suggested setuptools test_requires, but that only functions when > you > have a setup.py available and you execute ``setup.py test``. It does not > help > you at all once the package is installed and the sdist is gone. > > I also don't think people actually run the tests when they are installed > in any > significant number, at least I've never once in my life done it or even > had a > desire to do it. > The significant number is not so relevant if you buy the argument that it is useful to downstream packagers: it may be a few users, but those are crucial. I also forgot to mention that the ability to test something without building is crucial when you want to distribute binaries. David > Some projects have test suites which are significantly large too. The PyCA > cryptography project for instance, has to ship it's vectors as an > additional > package to reduce the size of the final build product. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 12:21:49 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 06:21:49 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com) wrote: > > The significant number is not so relevant if you buy the argument > that it is useful to downstream packagers: it may be a few users, > but those are crucial. > > I also forgot to mention that the ability to test something without > building is crucial when you want to distribute binaries. Is it actually useful to them? None of the Linux downstreams I know of have ever mentioned a preference for it. As far as I know, the only preference they've ever expressed to me is that the tests are included in the sdist. FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, so it apparently doesn't find it useful either. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Tue Oct 6 12:33:07 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 11:33:07 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On Tue, Oct 6, 2015 at 11:21 AM, Donald Stufft wrote: > On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com) > wrote: > > > The significant number is not so relevant if you buy the argument > > that it is useful to downstream packagers: it may be a few users, > > but those are crucial. > > > > I also forgot to mention that the ability to test something without > > building is crucial when you want to distribute binaries. > > Is it actually useful to them? None of the Linux downstreams I know of have > ever mentioned a preference for it. As far as I know, the only preference > they've ever expressed to me is that the tests are included in the sdist. > It is at least useful to me, and I am packaging quite a few binaries. > > FreeBSD relies on ``python setup.py test`` as it's preferred test > invocation, > so it apparently doesn't find it useful either. > I would like to hear their rationale before guessing. It is hard for me to imagine they would not rather test the binaries rather than from sources. Something as simple as making sure you have not forgotten runtime dependencies becomes much easier this way. David > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 12:43:18 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 06:43:18 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On October 6, 2015 at 6:33:10 AM, David Cournapeau (cournape at gmail.com) wrote: > On Tue, Oct 6, 2015 at 11:21 AM, Donald Stufft wrote: > > > On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com) > > wrote: > > > > The significant number is not so relevant if you buy the argument > > > that it is useful to downstream packagers: it may be a few users, > > > but those are crucial. > > > > > > I also forgot to mention that the ability to test something without > > > building is crucial when you want to distribute binaries. > > > > Is it actually useful to them? None of the Linux downstreams I know of have > > ever mentioned a preference for it. As far as I know, the only preference > > they've ever expressed to me is that the tests are included in the sdist. > > > > It is at least useful to me, and I am packaging quite a few binaries. > > > > > > FreeBSD relies on ``python setup.py test`` as it's preferred test > > invocation, > > so it apparently doesn't find it useful either. > > > > I would like to hear their rationale before guessing. It is hard for me to > imagine they would not rather test the binaries rather than from sources. > Something as simple as making sure you have not forgotten runtime > dependencies becomes much easier this way. > I'm able to test runtime dependencies just fine without needing to put my tests inside of the package. To the extent anyone can actually test runtime dependencies in a test framework without actually depending on your test tools at runtime [1]. I really don't think either way is "better" to be honest. I think any attempts that one way is better than the other relies on nebulous edge cases that don't really happen much in reality. You'll tell me that you've found it useful to run tests against an installed distribution without having to fetch the original tarball, I'll tell you that I've found it useful to run a newer version of the test suite against an older installed copy of the library. In practice, I think that you'll be able to manage finding the tarball and running those tests against your installed distribution while I'd be able to manage to copy just the tests I care about running out of the test suite and run them manually. IOW, I don't think it really matters and people should just do whatever they want. [1] By this I mean, you'll never detect if you're missing a runtime dependency ? ? on something that your test dependencies include as part of their ? ? dependencies, since installing your test tools to run your tests would then ? ? trigger this missing dependency to be installed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From contact at ionelmc.ro Tue Oct 6 12:47:51 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 13:47:51 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Tue, Oct 6, 2015 at 12:51 PM, Wes Turner wrote: > self-testable programs are really ideal (e.g POST power-on self test) > relevant recent topical discussion of e.g > CRC and an optional '-t' preemptive CLI parameter: > https://github.com/audreyr/cookiecutter-pypackage/pull/52 > ?I would be interesting ?to talk about what's worth including in a "self-test" feature. Most suites aren't suitable for including as whole. You don't want to include integration (functional) tests for sure :-) There's also the ever-unanswered question of how to deal with test dependencies. Some will think that it's ok to always install them, but then again, after seeing how `mock` depends on `pbr` - and how `pbr` just decides to alter your `setup.py sdist/bdist_*` output without being asked or invoked has made me very wary of this practice. How I have to make sure my python installs have certain versions of `pbr` or no `pbr` at all, every time I want to build a package :-( Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 13:00:18 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 13:00:18 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006130018.33a5760f@fsol> On Tue, 6 Oct 2015 11:30:00 +0300 Ionel Cristian M?rie? wrote: > On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > > > ?Does that really make sense? I haven't heard of any user actually running > tests? > that way. To be honest I haven't ever ran Python's own tests suite as part > of a user installation. There are several situations besides the "downstream packagers" use case mentioned somewhere else: * One of your users report a weird issue, you can ask them to run the test suite on their installation to check that nominal behaviour of the package is ok on their machine. If you don't ship the test suite, you have to ask them to do extra manual steps in order to do this verification, which can be cumbersome and delay proper response to the issue. * Your package requires non-Python data files for proper functioning, and you want to check the installation procedure puts them in the right place. The natural way to do that is to run the test suite on the installed package. Really, "ship the test suite" should be the norm and not shipping it should be the exception (if e.g. testing needs large data files). Regards Antoine. From donald at stufft.io Tue Oct 6 13:07:31 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 07:07:31 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006130018.33a5760f@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> Message-ID: On October 6, 2015 at 7:00:41 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Tue, 6 Oct 2015 11:30:00 +0300 > Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > > > > > They should be inside the module. That way, you can check an installed > > > module is ok by running e.g. "python -m mypackage.tests". Any other > > > choice makes testing installed modules more cumbersome. > > > > > > > ?Does that really make sense? I haven't heard of any user actually running > > tests > > that way. To be honest I haven't ever ran Python's own tests suite as part > > of a user installation. > > There are several situations besides the "downstream packagers" use case > mentioned somewhere else: > > * One of your users report a weird issue, you can ask them to run the > test suite on their installation to check that nominal behaviour of > the package is ok on their machine. If you don't ship the test suite, > you have to ask them to do extra manual steps in order to do this > verification, which can be cumbersome and delay proper response to > the issue. I've never, in my entire life, been asked to run someone's test suite to validate a bug nor have I asked someone to run a test suite to validate a bug. This is still ignoring the problems of test dependencies as well so you'll still need to ask them to install some number of dependencies, and I think it's fairly trivial to ask someone to download a tarball, untar it, and run two commands. > > * Your package requires non-Python data files for proper functioning, > and you want to check the installation procedure puts them in the > right place. The natural way to do that is to run the test suite on > the installed package. You're confusing "ships the test suite as part of the package" with "runs the test suite on the installed package". The two aren't really related, you can run the tests against an installed package trivially in either situation. > > Really, "ship the test suite" should be the norm and not shipping it > should be the exception (if e.g. testing needs large data files). > > Regards > > Antoine. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Tue Oct 6 14:13:25 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 14:13:25 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> Message-ID: <20151006141325.125514c7@fsol> On Tue, 6 Oct 2015 07:07:31 -0400 Donald Stufft wrote: > > I've never, in my entire life [...] Can I suggest your entire life is an anecdotal data point here? > This is still ignoring the problems of test dependencies Only if your tests have dependencies that runtime doesn't have. > as well so you'll > still need to ask them to install some number of dependencies, and I think it's > fairly trivial to ask someone to download a tarball, untar it, and run two > commands. Any number of things can be described as trivial depending on the skillset and patience of the user. When users report a bug, they are not expecting to be asked to download and "untar" stuff. Not every user is a programmer. > You're confusing "ships the test suite as part of the package" with "runs the > test suite on the installed package". The two aren't really related, you can > run the tests against an installed package trivially in either situation. It's not trivial, because if you aren't careful you'll be running them against the tarball / checkout instead (because of Python munging the PYTHONPATH behind your back, for example), and this can go unnoticed for a long time. By contrast, if you don't need a tarball / checkout to run them, you can guarantee they are run against the installed location. Regards Antoine. From contact at ionelmc.ro Tue Oct 6 14:34:38 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 15:34:38 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006141325.125514c7@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou wrote: > On Tue, 6 Oct 2015 07:07:31 -0400 > Donald Stufft wrote: > > > > I've never, in my entire life [...] > > Can I suggest your entire life is an anecdotal data point here? > ?Make that two anecdotal data points :-)? Any number of things can be described as trivial depending on the > skillset and patience of the user. When users report a bug, they are > not expecting to be asked to download and "untar" stuff. Not every > user is a programmer. > ? But seriously now, your arguments are also anecdotal. Lets not pretend we're objective here. That sort of attitude is disingenuous and will quickly devolve this discussion to mere ad hominems. ? > It's not trivial, because if you aren't careful you'll be running them > against the tarball / checkout instead (because of Python munging the > PYTHONPATH behind your back, for example), and this can go unnoticed for > a long time. > ?This is a flaw of the project layout really. If you don't isolate your sources from the import paths then you're probably testing the wrong way. In other words, you're probably not testing the installed code. Very few test runners change the current working directory by default [1], so it's better to just get a better project layout. pyca/cryptography is a good example.? ?[1] trial is the only one that I know of, and it's hardly popular for testing anything but projects that use Twisted?. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 14:51:10 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 14:51:10 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: <20151006145110.3e56ea96@fsol> On Tue, 6 Oct 2015 15:34:38 +0300 Ionel Cristian M?rie? wrote: > > Very few > test runners change the current working directory by default [1], so it's > better to just get a better project layout. pyca/cryptography > is a good example.? The "src" convention is actually terrible when working with Python code, since suddenly you can't experiment easily on a VCS checkout, you have to do extra steps and/or write helper scripts for it. The fact that few Python projects, including amongst the most popular projects, use that convention mean it's really not considered a good practice, nor convenient. Regards Antoine. From vivekmanchikanti at gmail.com Tue Oct 6 04:02:04 2015 From: vivekmanchikanti at gmail.com (vivek manchikanti) Date: Mon, 5 Oct 2015 21:02:04 -0500 Subject: [Distutils] Help regarding Python Theano on IDLE 3.5 Message-ID: Hi I'm a totally new to Python and I wanted to learn it. I have installed Python IDLE 3.5 successfully. Now I want to use Theano library in that IDE and I'm confused how to use that. Just have some questions like Do I need to install additional packages to use Theano on 3.5 ? Please kindly help. Thank you ------------------------------- Vivekanand Manchikanti -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 14:57:12 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 08:57:12 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006145110.3e56ea96@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On October 6, 2015 at 8:51:30 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Tue, 6 Oct 2015 15:34:38 +0300 > Ionel Cristian M?rie? wrote: > > > > Very few > > test runners change the current working directory by default [1], so it's > > better to just get a better project layout. pyca/cryptography > > is a good example. > > The "src" convention is actually terrible when working with Python > code, since suddenly you can't experiment easily on a VCS checkout, you > have to do extra steps and/or write helper scripts for it. Without doing it, you have very little assurances you?re actually testing against the installed project and not the project that's sitting in curdir. This is why pyca/cryptography does it, attempting to run the copy in . won't do anything but raise an exception since the .so won't be built. It doesn't really make experimenting in a VCS any harder, since all you need to do first is run ``pip install -e .`` and it will do a development install and add the src/ directory to sys.path. > > The fact that few Python projects, including amongst the most popular > projects, use that convention mean it's really not considered a good > practice, nor convenient. > > Of course, the same argument can be made for installing tests, since it's not very common. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tim at tim-smith.us Tue Oct 6 15:05:56 2015 From: tim at tim-smith.us (Tim Smith) Date: Tue, 6 Oct 2015 06:05:56 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? Message-ID: On Tue, Oct 6, 2015 at 5:35 AM, wrote: > On Tue, 6 Oct 2015 11:30:00 +0300 > Ionel Cristian M?rie? wrote: >> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: >> >> > They should be inside the module. That way, you can check an installed >> > module is ok by running e.g. "python -m mypackage.tests". Any other >> > choice makes testing installed modules more cumbersome. >> > >> >> ?Does that really make sense? I haven't heard of any user actually running >> tests that way. Numeric computing may be a special case because of its sensitivity to compiler/BLAS interactions but numpy and scipy support e.g. `numpy.test()` and Homebrew has received bug reports from people running them and getting unexpected results so I believe this is practiced and useful in some domains. Tim From contact at ionelmc.ro Tue Oct 6 15:00:12 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 16:00:12 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006145110.3e56ea96@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On Tue, Oct 6, 2015 at 3:51 PM, Antoine Pitrou wrote: > The "src" convention is actually terrible when working with Python > code, since suddenly you can't experiment easily on a VCS checkout, you > have to do extra steps and/or write helper scripts for it. > > The fact that few Python projects, including amongst the most popular > projects, use that convention mean it's really not considered a good > practice, nor convenient. > ?Convenience over correctness - I can understand that. Using virtualenvs (or tox) is not always something people want. People get attached to certain workflows, that's fine with me.? But lets not confuse that with what's right. Just because it's popular doesn't mean it anywhere close to correct. It means it works, once in a while people hit some pitfalls, suffer for it but continue the same way. The same people then complain about "terrible" packaging experience in Python. I think we should look at this more meticulously, not solely through the perspective of what's popular. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 15:07:29 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 15:07:29 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: <20151006150729.3e0c3f38@fsol> On Tue, 6 Oct 2015 08:57:12 -0400 Donald Stufft wrote: > > It doesn't really make experimenting in a VCS any harder, since all you need to > do first is run ``pip install -e .`` and it will do a development install and > add the src/ directory to sys.path. That means you're suddently polluting your Python install with a development package. So either you create a dedicated virtualenv (more command-line boilerplate, including each time you switch from a project another) or you risk messing with other package installs. Regards Antoine. From donald at stufft.io Tue Oct 6 15:33:03 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 09:33:03 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006150729.3e0c3f38@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> <20151006150729.3e0c3f38@fsol> Message-ID: On October 6, 2015 at 9:08:12 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Tue, 6 Oct 2015 08:57:12 -0400 > Donald Stufft wrote: > > > > It doesn't really make experimenting in a VCS any harder, since all you need to > > do first is run ``pip install -e .`` and it will do a development install and > > add the src/ directory to sys.path. > > That means you're suddently polluting your Python install with a > development package. So either you create a dedicated virtualenv (more > command-line boilerplate, including each time you switch from a project > another) or you risk messing with other package installs. > Unless your project has zero dependencies you?ll want to use a dedicated virtual environment anyways. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Tue Oct 6 15:51:54 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 14:51:54 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On Tue, Oct 6, 2015 at 1:57 PM, Donald Stufft wrote: > > > On October 6, 2015 at 8:51:30 AM, Antoine Pitrou (solipsis at pitrou.net) > wrote: > > On Tue, 6 Oct 2015 15:34:38 +0300 > > Ionel Cristian M?rie? wrote: > > > > > > Very few > > > test runners change the current working directory by default [1], so > it's > > > better to just get a better project layout. pyca/cryptography > > > is a good example. > > > > The "src" convention is actually terrible when working with Python > > code, since suddenly you can't experiment easily on a VCS checkout, you > > have to do extra steps and/or write helper scripts for it. > > Without doing it, you have very little assurances you?re actually testing > against the installed project and not the project that's sitting in curdir. > This is why pyca/cryptography does it, attempting to run the copy in . > won't > do anything but raise an exception since the .so won't be built. > > It doesn't really make experimenting in a VCS any harder, since all you > need to > do first is run ``pip install -e .`` and it will do a development install > and > add the src/ directory to sys.path. > > > > > The fact that few Python projects, including amongst the most popular > > projects, use that convention mean it's really not considered a good > > practice, nor convenient. > > > > > > Of course, the same argument can be made for installing tests, since it's > not > very common. > So we can actually get some data here :) At Enthought, we support around 400 packages. More than half of them are python packages, and we can make the reasonable assumption than the vast majority of those packages are fairly popular. More precisely, if I install all our supported packages on Linux: - I count ~ 308 packages by making the assumption that one directory in site-packages is a package (wrong but decent approximation) - I count as `with tests` a package with at least one directory called test in say package With those assumptions, I count 46 % packages with tests installed. So it is not "not very common". Granted, we are biased toward scientific packages which to include tests in the package, but we also package popular web/etc... packages. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Oct 6 16:10:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Oct 2015 15:10:59 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On 6 October 2015 at 14:51, David Cournapeau wrote: >> Of course, the same argument can be made for installing tests, since it's >> not >> very common. > > > So we can actually get some data here :) > > At Enthought, we support around 400 packages. More than half of them are > python packages, and we can make the reasonable assumption than the vast > majority of those packages are fairly popular. > > More precisely, if I install all our supported packages on Linux: > - I count ~ 308 packages by making the assumption that one directory in > site-packages is a package (wrong but decent approximation) > - I count as `with tests` a package with at least one directory called test > in say package > > With those assumptions, I count 46 % packages with tests installed. So it is > not "not very common". Granted, we are biased toward scientific packages > which to include tests in the package, but we also package popular > web/etc... packages. Interesting. Prompted by this, I did a check on my Python 3.4 installation. 84 packages, of which 21 have a "tests" subdirectory containing __init__.py and 4 have a "test" subdirectory containing __init__.py. That's 25%, which is certainly higher than I would have guessed. Of course, the implication is that 75% (or 54% in David's case) use test suites *not* installed with the package :-) But I think it's fair to say that installing the tests with the package is a common enough model, and it clearly does have some benefits. To answer the OP's question, I stand by my original impression, which is that "it depends" :-) I don't want the sample project to *not* include tests, so as far as that is concerned, I'm going with "the status quo wins there" and suggesting there's no reason to change. Paul From njs at pobox.com Tue Oct 6 16:12:40 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 6 Oct 2015 07:12:40 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: Message-ID: On Oct 6, 2015 6:06 AM, "Tim Smith" wrote: > > On Tue, Oct 6, 2015 at 5:35 AM, wrote: > > On Tue, 6 Oct 2015 11:30:00 +0300 > > Ionel Cristian M?rie? wrote: > >> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > >> > >> > They should be inside the module. That way, you can check an installed > >> > module is ok by running e.g. "python -m mypackage.tests". Any other > >> > choice makes testing installed modules more cumbersome. > >> > > >> > >> ?Does that really make sense? I haven't heard of any user actually running > >> tests that way. > > Numeric computing may be a special case because of its sensitivity to > compiler/BLAS interactions but numpy and scipy support e.g. > `numpy.test()` and Homebrew has received bug reports from people > running them and getting unexpected results so I believe this is > practiced and useful in some domains. Yes, and numpy upstream gets bug / support requests on a totally regular basis (weekly-ish?) from end-users who have somehow managed to wedge up their install and either they noticed because they ran the installed test suite, or else they are reporting some bizarre results and we say "can you run python -m 'import numpy; numpy.test()'?". My impression is that there's a non-trivial subset of users who install numpy from binaries and then immediately run the test suite as a matter of course, and as a developer this is something that I'm happy to encourage (yay more test coverage). Adding another step where they have to download something extra would definitely discourage this. I can easily believe that there are other parts of the python world where this is uncommon, but if some people are saying "I've never seen this and it isn't useful" and others are saying "I see this all the time and it's very useful", then this isn't really anecdote versus contradictory anecdote. The conclusion is that we're talking about a thing that happens at least sometimes and is useful at least sometimes. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Oct 6 16:20:41 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 6 Oct 2015 07:20:41 -0700 Subject: [Distutils] Help regarding Python Theano on IDLE 3.5 In-Reply-To: References: Message-ID: On Oct 6, 2015 5:55 AM, "vivek manchikanti" wrote: > > Hi > > I'm a totally new to Python and I wanted to learn it. I have installed Python IDLE 3.5 successfully. > > Now I want to use Theano library in that IDE and I'm confused how to use that. > Just have some questions like > Do I need to install additional packages to use Theano on 3.5 ? > Please kindly help. Hi Vivek, This mailing list is for people talking about how to make it easier to build and install python packages; I doubt there's anyone here who knows much about theano. You'll probably have better luck finding someone who can help with your questions if you ask on stackoverflow or the theano-users group: http://www.deeplearning.net/software/theano/#how-to-seek-help Good luck! -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Oct 6 16:21:35 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 6 Oct 2015 14:21:35 +0000 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006142135.GS4731@yuggoth.org> On 2015-10-06 13:47:51 +0300 (+0300), Ionel Cristian M?rie? wrote: [...] > `pbr` just decides to alter your `setup.py sdist/bdist_*` output > without being asked or invoked [...] Assuming you're talking about https://launchpad.net/bugs/1483067 then it was fixed in 1.7.0. If you're still seeing it in later releases, a detailed bug report would be most appreciated since that's not at all the intent. -- Jeremy Stanley From solipsis at pitrou.net Tue Oct 6 16:24:21 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 16:24:21 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> <20151006150729.3e0c3f38@fsol> Message-ID: <20151006162421.5e6a847b@fsol> On Tue, 6 Oct 2015 09:33:03 -0400 Donald Stufft wrote: > On October 6, 2015 at 9:08:12 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > > On Tue, 6 Oct 2015 08:57:12 -0400 > > Donald Stufft wrote: > > > > > > It doesn't really make experimenting in a VCS any harder, since all you need to > > > do first is run ``pip install -e .`` and it will do a development install and > > > add the src/ directory to sys.path. > > > > That means you're suddently polluting your Python install with a > > development package. So either you create a dedicated virtualenv (more > > command-line boilerplate, including each time you switch from a project > > another) or you risk messing with other package installs. > > Unless your project has zero dependencies you?ll want to use a dedicated virtual environment anyways. Not necessarily, you can share that environment with other projects. Regards Antoine. From fungi at yuggoth.org Tue Oct 6 16:27:22 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 6 Oct 2015 14:27:22 +0000 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: <20151006142722.GT4731@yuggoth.org> On 2015-10-06 15:10:59 +0100 (+0100), Paul Moore wrote: [...] > That's 25%, which is certainly higher than I would have > guessed. Of course, the implication is that 75% (or 54% in David's > case) use test suites *not* installed with the package :-) [...] It seems rather optimistic to assume that 100% of projects have test suites. More accurately, 75% either have no tests _or_ keep them outside the package. ;) -- Jeremy Stanley From p.f.moore at gmail.com Tue Oct 6 16:39:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Oct 2015 15:39:25 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006142722.GT4731@yuggoth.org> References: <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> <20151006142722.GT4731@yuggoth.org> Message-ID: On 6 October 2015 at 15:27, Jeremy Stanley wrote: > It seems rather optimistic to assume that 100% of projects have test > suites. More accurately, 75% either have no tests _or_ keep them > outside the package. ;) I'm having a happy day today - don't spoil it... :-) Paul From fred at fdrake.net Tue Oct 6 17:26:29 2015 From: fred at fdrake.net (Fred Drake) Date: Tue, 6 Oct 2015 11:26:29 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau wrote: > More precisely, if I install all our supported packages on Linux: > - I count ~ 308 packages by making the assumption that one directory in > site-packages is a package (wrong but decent approximation) > - I count as `with tests` a package with at least one directory called test > in say package > > With those assumptions, I count 46 % packages with tests installed. So it is > not "not very common". Granted, we are biased toward scientific packages > which to include tests in the package, but we also package popular > web/etc... packages. For what it's worth, the Zope community has long been including tests inside packages (regardless of whether that's right or wrong). How the dependencies issue has been addressed has varied over time and depending on the authors of each package; sometimes tests_require is used, other times there's a [test] extra (as setuptools is universally accepted in the Zope community), and other times the test dependencies are just made package dependencies. Since zc.buildout is widely used in the Zope community, getting the right dependencies for the right processes has never been a significant issue. (And yes, most Zope packages use a src/ directory. Not to isolate the tests and the implementation, but to isolate the code from other files that may be present in the package.) -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From erik.m.bray at gmail.com Tue Oct 6 17:33:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 6 Oct 2015 11:33:07 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 8:34 AM, Ionel Cristian M?rie? wrote: > On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou wrote: >> >> On Tue, 6 Oct 2015 07:07:31 -0400 >> Donald Stufft wrote: >> > >> > I've never, in my entire life [...] >> >> Can I suggest your entire life is an anecdotal data point here? > > > Make that two anecdotal data points :-) > >> Any number of things can be described as trivial depending on the >> skillset and patience of the user. When users report a bug, they are >> not expecting to be asked to download and "untar" stuff. Not every >> user is a programmer. > > But seriously now, your arguments are also anecdotal. Lets not pretend we're > objective here. That sort of attitude is disingenuous and will quickly > devolve this discussion to mere ad hominems. Okay, though, so maybe if there is nothing to offer here but anecdata then maybe we should stop acting like there's "one right way here". I have projects that install their test suite and test dependencies because it is frequently useful to ask users to run a self-test (and users themselves want to be able to do it for a variety of reasons). There are other projects where it doesn't make sense, and those don't have to install the tests (I still think in those cases that the tests should live in the package instead of outside it, but simply not installed). In any case, let's not get trapped into endless circular discussions about what is correct, period, and instead consider individual use cases--not dismissing individual projects' or peoples' experiences and needs--and discuss what the most appropriate action is for those uses cases. Python projects are not monolithic in their audiences (and that includes developer audiences and user audiences). Erik From contact at ionelmc.ro Tue Oct 6 18:04:54 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 19:04:54 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 6:33 PM, Erik Bray wrote: > ?Okay, though, so maybe if there is nothing to offer here but anecdata > then maybe we should stop acting like there's "one right way here". I > have projects that install their test suite and test dependencies > because it is frequently useful to ask users to run a self-test (and > users themselves want to be able to do it for a variety of reasons). > ? > > There are other projects where it doesn't make sense, and those don't > have to install the tests (I still think in those cases that the tests > should live in the package instead of outside it, but simply not > installed). ?To be honest, I like the idea of providing the tests? in the installed package. Using the test suite as a debugging/diagnostic tool is obviously desirable. It's just that it's unpractical for general use. Another anecdote I know, but humour these two concerns: * How to deal with dependencies? ** Should we use extras? Installing test deps has the disadvantage of version conflicts. Unless we make some sort of "virtualenv-for-tests" tool? ** Should we vendor the deps? But isn't that maybe too hard to do (or plain wrong for other reasons)? ** Should we avoid having deps? That can be too limiting in some situations, unittest is very bare compared to pytest or nose. ?* What sort of tests should be included? Integration tests are special case, if they need external services, temporary storage or what not. Would we then need to have clear separation for different types of tests? I'm not saying tests inside package is bad at all. But to flaunt it around as a "best practice" requires at least some recommendations for the two concerns illustrated above, and probably better tooling. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Tue Oct 6 22:54:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 6 Oct 2015 16:54:07 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 12:04 PM, Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 6:33 PM, Erik Bray wrote: >> >> Okay, though, so maybe if there is nothing to offer here but anecdata >> then maybe we should stop acting like there's "one right way here". I >> have projects that install their test suite and test dependencies >> because it is frequently useful to ask users to run a self-test (and >> users themselves want to be able to do it for a variety of reasons). >> >> There are other projects where it doesn't make sense, and those don't >> have to install the tests (I still think in those cases that the tests >> should live in the package instead of outside it, but simply not >> installed). > > > To be honest, I like the idea of providing the tests in the installed > package. Using the test suite as a debugging/diagnostic tool is obviously > desirable. It's just that it's unpractical for general use. Another anecdote > I know, but humour these two concerns: Those are all fair questions, and I can't say I have great answers for all of them. But that's a fair point that if installable tests *are* recommended as a practice then they should be addressed. Before continuing on there are really three options being discussed here: 1) Tests totally outside the package. 2) Tests in the package but not installed (only in source dists / VCS checkout) 3) Tests in the package and some or all installed. Skimming back through the rest of the thread I don't see too much support for 1). The only argument against it is the need for specifying dependencies, etc., which really only impacts developers so long as the tests aren't *installed*, I think. But there's also the question of what kinds of tests we're talking about. I think unit tests should live in the .tests for a library. Other kinds of tests I don't have a strong opinion about. So in any case if we did recommend putting tests in a subpackage we could still do so without making a decree as to whether or not they should be installed. Maybe by default suggest that they not be installed, and only have them installable if the developer has a plan for addressing the questions below. I think each of these questions may have different answers depending on the needs of different projects. > * How to deal with dependencies? What dependencies exactly? Dependencies just needed for running the tests (as opposed to running the code being tested? Some of my code has optional dependencies, and the tests for that code are only run if those optional dependencies are satisfied, but that's not what we're talking about right?) > ** Should we use extras? Installing test deps has the disadvantage of > version conflicts. Unless we make some sort of "virtualenv-for-tests" tool? I'm rather fond of mktmpenv in virtualenv-wrapper, but I admit that may be out of scope here... Not to point to using easy_install as best practice, but something resembling the way setup_requires could actually work here--dependencies are downloaded into a temp dir and installed there as eggs; added to sys.path. No need for something as heavy-weight as a virtualenv. This still has some potential for VersionConflict errors if not handled carefully though. But I might try out something like this and see where I get with it, because I think it would be useful. > ** Should we vendor the deps? But isn't that maybe too hard to do (or plain > wrong for other reasons)? > ** Should we avoid having deps? That can be too limiting in some situations, > unittest is very bare compared to pytest or nose. py.test comes with a command to generate a bundled copy of py.test for vendoring: https://pytest.org/latest/goodpractises.html#deprecated-create-a-pytest-standalone-script (for reasons I've lost track of it's apparently deprecated now though?) I've used this with quite a bit of success to support installed tests. It allows me to ship a version of py.test that works with my tests along with the package, and there is no issue with version conflicts or anything. In principle the same trick could be used to bundle other dependencies. In practice it's a little more complicated because downstream packagers don't like this, but it's easy to remove the bundled py.test and use the one provided by the system instead (as in on Debian). In that case we do have to work with the downstream packagers to make sure all the tests are working for them. I don't think this will be be much of an issue for the average pure-Python package. > * What sort of tests should be included? Integration tests are special case, > if they need external services, temporary storage or what not. Would we then > need to have clear separation for different types of tests? I try to separate out tests like this and disable them by default. I think the answer to that question is going to vary a lot on a case by case basis, but it is a good question to pose to anyone considering having installed tests. > I'm not saying tests inside package is bad at all. But to flaunt it around > as a "best practice" requires at least some recommendations for the two > concerns illustrated above, and probably better tooling. Fair point, I agree! Best, Erik From barry at python.org Tue Oct 6 23:19:22 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:19:22 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: <20151006171922.0a53de51@anarchist.wooz.org> On Oct 06, 2015, at 05:54 AM, Donald Stufft wrote: >I dislike putting tests inside the package. I'm a big fan of putting the tests inside the package. I've often looked at a package's tests to get a better understanding of something that was unclear for the documentation, or didn't work the way I expected. Having the tests there in the installed package makes it easier to refer to. I also find that with tox+nose2 (my preferred one-two punch for testing), it makes it quite easy to find and run the full test suite or individual tests based on a regexp pattern. I also like the symmetry of having a docs/ directory for doctests and a tests/ directory for unittests. For complex packages with lots of subpackages, I have lots of tests/ directories, so that the unitests are near to the code they test. This way the source tree gets organized for free without additional complexity in an outside-the-package tests tree. YMMV, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Oct 6 23:21:27 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:21:27 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: <20151006172127.5dd9867a@anarchist.wooz.org> On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, >so it apparently doesn't find it useful either. Oh how I wish there was a standard way to *declare* how to run the test suite, such that all our automated tools (or the humans :) didn't have to guess. At least in Debuntu though, we can pretty much make any of the usual ways work during package build. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Oct 6 23:31:28 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:31:28 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: <20151006173128.7d712b62@anarchist.wooz.org> On Oct 06, 2015, at 11:33 AM, David Cournapeau wrote: >I would like to hear their rationale before guessing. It is hard for me to >imagine they would not rather test the binaries rather than from sources. >Something as simple as making sure you have not forgotten runtime >dependencies becomes much easier this way. In Debian we can do both. It's usually good practice to run the package's test suite during the build process, on the unbuilt source tree. That doesn't work for all packages though (tox comes to mind as a recent example), so we *also* have a way to run the test suite on a built-and-installed version of the Debian binary package. I usually try to at least do an import test in this phase, so for some like tox, I'll do a more extensive test. In Ubuntu, failing the built-and-installed test (a.k.a. autopkgtest or DEP-8) will prevent a package from getting promoted from the -proposed channel to the release channel, which usually shields end users from seeing broken packages. Debian doesn't have this gateway in place yet. There is a way to run those on a local test build, so that's pretty nice. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Oct 6 23:40:35 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:40:35 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006174035.7557d23d@anarchist.wooz.org> On Oct 06, 2015, at 09:51 AM, Antoine Pitrou wrote: >The PyP"A" should definitely fix its sample project to reflect good >practices. Here's my own sample project. There are actually two git branches, one with an extension module and one with pure Python. https://gitlab.com/warsaw/stupid Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Tue Oct 6 23:41:55 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 17:41:55 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006171922.0a53de51@anarchist.wooz.org> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> Message-ID: On October 6, 2015 at 5:20:03 PM, Barry Warsaw (barry at python.org) wrote: > On Oct 06, 2015, at 05:54 AM, Donald Stufft wrote: > > >I dislike putting tests inside the package. > > I'm a big fan of putting the tests inside the package. I've often looked at a > package's tests to get a better understanding of something that was unclear > for the documentation, or didn't work the way I expected. Having the tests > there in the installed package makes it easier to refer to. I also find that > with tox+nose2 (my preferred one-two punch for testing), it makes it quite > easy to find and run the full test suite or individual tests based on a regexp > pattern. I also like the symmetry of having a docs/ directory for doctests > and a tests/ directory for unittests. > > For complex packages with lots of subpackages, I have lots of tests/ > directories, so that the unitests are near to the code they test. This way > the source tree gets organized for free without additional complexity in an > outside-the-package tests tree. > I?m not sure I understand what you?re advocating here, it sounds like you want your tests at something like mycoolproject/tests so that they are importable from mycoolproject.tests? but then you talk about symmetry with docs/ and tests/ which sounds more like you have top level directories for tests/ docs/ and then mycoolproject/. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From barry at python.org Tue Oct 6 23:42:11 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:42:11 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006104415.7fb554a2@fsol> Message-ID: <20151006174211.78ae9bc2@anarchist.wooz.org> On Oct 06, 2015, at 10:44 AM, Antoine Pitrou wrote: >Doesn't / didn't setuptools have something called test_requires? Since I pretty much use tox everywhere these days, I've taken to defining test requirements in my tox.ini rather than my setup.py. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From contact at ionelmc.ro Tue Oct 6 23:47:31 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 00:47:31 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 11:54 PM, Erik Bray wrote: > Skimming back through the rest of the thread I don't see too much > support for 1). The only argument against it is the need for > specifying dependencies, etc., which really only impacts developers so > long as the tests aren't *installed*, I think. But there's also the > question of what kinds of tests we're talking about. I think unit > tests should live in the .tests for a library. Other > kinds of tests I don't have a strong opinion about. > ?I think there's some confusion here. The pain point with "inside" tests is exactly the dependencies. And by dependencies I mean test dependencies, we're talking about tests here :-)? If you install two packages with "inside" tests, then how do you deal with the version conflict of test dependencies? That's the big elephant in the room everyone is ignoring :) If you have to bend over backwards (to install the test dependencies) in order to run the installed tests then what's the point of installing them at all? It's far more safer to ask the user to just checkout the sources and run the tests from there. Why completely mess up user's site-packages just because you want to have this weird `python -mfoobar.tests` workflow? I like the idea, I really do. But it's not for everyone. I strongly feel that only projects that don't have any test dependencies should install the tests, or provide the tests inside the package. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Oct 6 23:46:24 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:46:24 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <85612k1hkk.fsf@benfinney.id.au> Message-ID: <20151006174624.62f2bf9f@anarchist.wooz.org> On Oct 06, 2015, at 08:18 PM, Ben Finney wrote: >Yes, this is a sensible approach: > >* The source package contains all the source files a developer would use > to make further changes and test them. > >* The package for installation contains only those files useful run-time > users, plus metadata (e.g. copyright information). I generally don't both stripping out in-package tests when building binary packages for Debian. First because it's more work for (IMHO) dubious value and second because I think installed test files are actually useful sometimes. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ben+python at benfinney.id.au Tue Oct 6 23:51:46 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 07 Oct 2015 08:51:46 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> Message-ID: <85mvvvzmvh.fsf@benfinney.id.au> Barry Warsaw writes: > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > >FreeBSD relies on ``python setup.py test`` as it's preferred test > >invocation > > Oh how I wish there was a standard way to *declare* how to run the > test suite, such that all our automated tools (or the humans :) didn't > have to guess. I think the above describes the standard way of declaring the test runner: The ?setup.py test? command. Now, I lament that more Python projects don't *conform to* that standard, but at least it exists. -- \ ?I have never made but one prayer to God, a very short one: ?O | `\ Lord, make my enemies ridiculous!? And God granted it.? | _o__) ?Voltaire | Ben Finney From ben+python at benfinney.id.au Tue Oct 6 23:54:37 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 07 Oct 2015 08:54:37 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> Message-ID: <85h9m3zmqq.fsf@benfinney.id.au> Barry Warsaw writes: > I'm a big fan of putting the tests inside the package. I've often > looked at a package's tests to get a better understanding of something > that was unclear for the documentation, or didn't work the way I > expected. Having the tests there in the installed package makes it > easier to refer to. That doesn't follow, or I'm not understanding you. If you have the tests in the source package, as is being advocated, you have the tests available for reference. So the *relative location* of the tests, within that source tree, doesn't argue for what you're saying. Are you arguing the separate point of whether tests should be *installed* with the package? -- \ ?We now have access to so much information that we can find | `\ support for any prejudice or opinion.? ?David Suzuki, 2008-06-27 | _o__) | Ben Finney From solipsis at pitrou.net Wed Oct 7 00:04:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2015 00:04:01 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: <20151007000401.48fb497c@fsol> On Wed, 7 Oct 2015 00:47:31 +0300 Ionel Cristian M?rie? wrote: > On Tue, Oct 6, 2015 at 11:54 PM, Erik Bray wrote: > > > Skimming back through the rest of the thread I don't see too much > > support for 1). The only argument against it is the need for > > specifying dependencies, etc., which really only impacts developers so > > long as the tests aren't *installed*, I think. But there's also the > > question of what kinds of tests we're talking about. I think unit > > tests should live in the .tests for a library. Other > > kinds of tests I don't have a strong opinion about. > > > > ?I think there's some confusion here. The pain point with "inside" tests is > exactly the dependencies. Is it your personal experience or some theoretical argument you're making? > If you install two packages with "inside" tests, then how do you deal with > the version conflict of test dependencies? Well, how do you deal with the version conflict of non-test dependencies? How are tests supposed to be a problem here, while they usually have so few dependencies of their own? > If you have to bend over backwards (to install the test dependencies) While some packages may have non-trivial test dependencies, usual practice is for test suites to require the exact same dependencies as the rest of the package, + perhaps a test runner. Since we're talking about good practice for the average package, it's not very useful to point out that 0.1% of PyPI packages may have excruciatingly annoying test dependencies. > Why completely mess up user's site-packages > just because you want to have this weird `python -mfoobar.tests` workflow? Did you have such an experience or are you making this up for the sake of the argument? And just because you are not used to a "workflow" doesn't make it "weird" in any case. Python itself uses such a workflow ("python -m test"). Regards Antoine. From barry at python.org Wed Oct 7 00:08:37 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 18:08:37 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: <20151006180837.5233d549@anarchist.wooz.org> On Oct 07, 2015, at 08:51 AM, Ben Finney wrote: >I think the above describes the standard way of declaring the test >runner: The ?setup.py test? command. > >Now, I lament that more Python projects don't *conform to* that >standard, but at least it exists. It's *a* standard but not *the* standard, just from pure observation. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From contact at ionelmc.ro Wed Oct 7 00:08:40 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 01:08:40 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <85mvvvzmvh.fsf@benfinney.id.au> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney wrote: > I think the above describes the standard way of declaring the test > runner: The ?setup.py test? command. > > Now, I lament that more Python projects don't *conform to* that > standard, but at least it exists. > ?There's a very simple answer to that: easy_install (that's what `setup.py test` will use to install deps). It has several design issue wrt how packages are installed and how dependencies are managed. Lets not use `setup.py test`. It's either bad or useless. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Oct 7 00:16:15 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 18:16:15 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> Message-ID: <20151006181615.53f69fdd@anarchist.wooz.org> On Oct 06, 2015, at 05:41 PM, Donald Stufft wrote: >I?m not sure I understand what you?re advocating here, it sounds like you >want your tests at something like mycoolproject/tests so that they are >importable from mycoolproject.tests? but then you talk about symmetry with >docs/ and tests/ which sounds more like you have top level directories for >tests/ docs/ and then mycoolproject/. Ah, sorry for being unclear. I put tests in myproj/tests so yes they are importable via myproj.tests. I also put __init__.py's in my docs/ directory so the directory, but not the individual .rst doc files are also importable via myproj.docs. I do this because of the handy nose2 plugin I cargo cult around that allows me to run individual tests or doctests with a command line switch. If I have subpackages, the pattern repeats, so e.g. myproj/subpkg1/tests -> import myproj.subpkg1.tests myproj/subpkg2/docs -> import myproj.subpkg2.docs Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From carl at oddbird.net Wed Oct 7 00:16:41 2015 From: carl at oddbird.net (Carl Meyer) Date: Tue, 6 Oct 2015 16:16:41 -0600 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151007000401.48fb497c@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> Message-ID: <56144849.3070107@oddbird.net> On 10/06/2015 04:04 PM, Antoine Pitrou wrote: [snip] > ...How are tests supposed to be a problem here, while they > usually have so few dependencies of their own? > >> If you have to bend over backwards (to install the test dependencies) > > While some packages may have non-trivial test dependencies, usual > practice is for test suites to require the exact same dependencies as > the rest of the package, + perhaps a test runner. > > Since we're talking about good practice for the average package, it's > not very useful to point out that 0.1% of PyPI packages may have > excruciatingly annoying test dependencies. I think this discussion could probably do with fewer unsupported assertions about what is "usual" -- it's clear that experiences in different parts of the community vary widely. Speaking personally and anecdotally, I maintain 15 or so projects on PyPI, and every single one of them has at least three or four test-only dependencies; not just a test runner, but also testing utilities of one kind or another (e.g. the mock backport for Python 2). So my personal percentage of "packages with more than one test-only dependency" is not 0.1%, it's 100%. I don't have any idea what the real percentage is on PyPI, and wouldn't hazard a guess. I'm fairly sure you don't know either. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 00:19:34 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 18:19:34 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> <85h9m3zmqq.fsf@benfinney.id.au> Message-ID: <20151006181934.410b0395@anarchist.wooz.org> On Oct 07, 2015, at 08:54 AM, Ben Finney wrote: >Barry Warsaw writes: > >> I'm a big fan of putting the tests inside the package. I've often >> looked at a package's tests to get a better understanding of something >> that was unclear for the documentation, or didn't work the way I >> expected. Having the tests there in the installed package makes it >> easier to refer to. > >That doesn't follow, or I'm not understanding you. > >If you have the tests in the source package, as is being advocated, you >have the tests available for reference. So the *relative location* of >the tests, within that source tree, doesn't argue for what you're >saying. Since I'm not sure I follow that, I'll answer the question you asked: >Are you arguing the separate point of whether tests should be >*installed* with the package? Yes. We've had this conversation before in the context of Debian package sponsorship. I know and respect that you disagree. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From solipsis at pitrou.net Wed Oct 7 00:34:14 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2015 00:34:14 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> <56144849.3070107@oddbird.net> Message-ID: <20151007003414.36cfa9d7@fsol> On Tue, 6 Oct 2015 16:16:41 -0600 Carl Meyer wrote: > On 10/06/2015 04:04 PM, Antoine Pitrou wrote: > [snip] > > ...How are tests supposed to be a problem here, while they > > usually have so few dependencies of their own? > > > >> If you have to bend over backwards (to install the test dependencies) > > > > While some packages may have non-trivial test dependencies, usual > > practice is for test suites to require the exact same dependencies as > > the rest of the package, + perhaps a test runner. > > > > Since we're talking about good practice for the average package, it's > > not very useful to point out that 0.1% of PyPI packages may have > > excruciatingly annoying test dependencies. > > I think this discussion could probably do with fewer unsupported > assertions about what is "usual" -- it's clear that experiences in > different parts of the community vary widely. > > Speaking personally and anecdotally, I maintain 15 or so projects on > PyPI, and every single one of them has at least three or four test-only > dependencies; not just a test runner, but also testing utilities of one > kind or another (e.g. the mock backport for Python 2). They're still trivial dependencies, though. Usually small or medium-sized pure Python packages with a rather stable API (especially stdlib backports, of course). I don't see how they could cause the kind of mess the OP claimed they would. So I'd still like to know what "bend over backwards" is supposed to mean here. Regards Antoine. From contact at ionelmc.ro Wed Oct 7 00:44:34 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 01:44:34 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151007000401.48fb497c@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> Message-ID: On Wed, Oct 7, 2015 at 1:04 AM, Antoine Pitrou wrote: > > ?I think there's some confusion here. The pain point with "inside" tests > is > > exactly the dependencies. > > Is it your personal experience or some theoretical argument you're > making? > I've published about 27 packages that have tests on PyPI. Out of those only 5 could run purely on stdlib unittest.? That leaves me with 22 packages that need test tools like pytest/nose and assorted plugins. > > If you install two packages with "inside" tests, then how do you deal > with > > the version conflict of test dependencies? > > Well, how do you deal with the version conflict of non-test > dependencies? How are tests supposed to be a problem here, while they > usually have so few dependencies of their own? > ?It's double the trouble to find compatible releases. Current tooling don't resolve conflicts automatically.? Maybe it's something handled better in Conda for all I know but I don't use that. > > If you have to bend over backwards (to install the test dependencies) > > While some packages may have non-trivial test dependencies, usual > practice is for test suites to require the exact same dependencies as > the rest of the package, + perhaps a test runner. > Can't relate to that - definitely not `usual practice` from my perspective.? Since we're talking about good practice for the average package, it's > not very useful to point out that 0.1% of PyPI packages may have > excruciatingly annoying test dependencies. > ?Already feeling guilty ... I hope you're finally happy now :-) ? > > Why completely mess up user's site-packages > > just because you want to have this weird `python -mfoobar.tests` > workflow? > > Did you have such an experience or are you making this up for the sake > of the argument? > ?I got burnt by the pbr issue [1] once (cause mock has it as a run-time dependency). I don't normally use `mock` but in the circle of hell I live in someone else depended on it.? ?I don't look forward to that happening again, and I don't want to litter my site-packages with useless test stuff. I already have too much stuff in there. > And just because you are not used to a "workflow" doesn't make it > "weird" in any case. Python itself uses such a workflow ("python -m > test"). ?It's weird in the sense that you have to do all these gymnastics to get the test dependencies right? ?before running that. As I previously stated, I like the idea of `python -mfoobar.test` - just that dependencies and scope make it weird and impractival for *general* use. ?[1] https://bugs.launchpad.net/pbr/+bug/1483067? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Wed Oct 7 00:46:35 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 07 Oct 2015 09:46:35 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> <85h9m3zmqq.fsf@benfinney.id.au> <20151006181934.410b0395@anarchist.wooz.org> Message-ID: <857fmzzkc4.fsf@benfinney.id.au> Barry Warsaw writes: > On Oct 07, 2015, at 08:54 AM, Ben Finney wrote: > > >Barry Warsaw writes: > > > >> I'm a big fan of putting the tests inside the package. I've often > >> looked at a package's tests to get a better understanding of something > >> that was unclear for the documentation, or didn't work the way I > >> expected. Having the tests there in the installed package makes it > >> easier to refer to. > > > >That doesn't follow, or I'm not understanding you. > >[?] > > > >Are you arguing the separate point of whether tests should be > >*installed* with the package? > > Yes. We've had this conversation before in the context of Debian package > sponsorship. I know and respect that you disagree. Okay. That's quite an orthogonal dimension, though, to the *relative location* of tests within the source tree. So ?I'm a big fan of putting tests inside the [Python] package [directory]? can't be motivated by ?Having the tests there in the installed package?. The two aren't related, AFAICT. -- \ ?There was a point to this story, but it has temporarily | `\ escaped the chronicler's mind.? ?Douglas Adams | _o__) | Ben Finney From solipsis at pitrou.net Wed Oct 7 00:55:22 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2015 00:55:22 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> Message-ID: <20151007005522.6e10aa0f@fsol> On Wed, 7 Oct 2015 01:44:34 +0300 Ionel Cristian M?rie? wrote: > > That leaves me with 22 packages that need test tools like pytest/nose and > assorted plugins. > [...] > > ?It's double the trouble to find compatible releases. Hmm, are you saying py.test / nose and assorted plugins break APIs often? I would be a bit surprised, but I don't use them nowadays. Modern unittest is quite capable. > > And just because you are not used to a "workflow" doesn't make it > > "weird" in any case. Python itself uses such a workflow ("python -m > > test"). > > ?It's weird in the sense that you have to do all these gymnastics to get > the test dependencies right? Well, "getting the dependencies right" must be done, whether you run the tests from their installed location or from the source tree. In both cases, you can probably get the package management system to automate package downloads and installs. Rehards Antoine. From robertc at robertcollins.net Wed Oct 7 01:23:13 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 7 Oct 2015 12:23:13 +1300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On 6 October 2015 at 23:47, Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 12:51 PM, Wes Turner wrote: >> >> self-testable programs are really ideal (e.g POST power-on self test) >> relevant recent topical discussion of e.g >> CRC and an optional '-t' preemptive CLI parameter: >> https://github.com/audreyr/cookiecutter-pypackage/pull/52 > > > I would be interesting to talk about what's worth including in a "self-test" > feature. > > Most suites aren't suitable for including as whole. You don't want to > include integration (functional) tests for sure :-) > > There's also the ever-unanswered question of how to deal with test > dependencies. Some will think that it's ok to always install them, but then > again, after seeing how `mock` depends on `pbr` - and how `pbr` just decides > to alter your `setup.py sdist/bdist_*` output without being asked or invoked > has made me very wary of this practice. How I have to make sure my python > installs have certain versions of `pbr` or no `pbr` at all, every time I > want to build a package :-( Hangon, there's clearly a *huge* gap in understanding here. pbr does *not* modify *anyones* setup.py output unless its enabled. ***setuptools*** enables all its plugins unconditionally (because entrypoints, yay), and then pbr has to explicitly *opt-out* of doing anything. Which *we do*. There was a bug where a new thing added didn't have this opt-out, and its since been fixed. There's a separate related bug that the opt-out checking code isn't quite robust enough, and thats causing some havoc, but there's a patch up to fix it and as soon as we have a reliable test we'll be landing it and cutting a release. If there was a way to plug into setuptools where pbr code wasn't called on random packages, and we didn't have to manually opt-out, why that would be brilliant. [Note: I'm not saying setuptools is buggy - it chose this interface, and thats fine, but the consequence is bugs like this that we have to fix]. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 7 01:32:43 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 7 Oct 2015 12:32:43 +1300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <56137342.8090201@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> Message-ID: On 6 October 2015 at 20:07, Thomas G?ttler wrote: > Hi, > > Where should I put tests when packaging python modules? > > I want a "cowpath", an "obvious way" > > Dear experts, please decide: > > inside the module like this answer: > > http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules > > XOR > > outside the module like this: > > https://github.com/pypa/sampleproject/tree/master/tests > > I think there is no need to hurry. Let's wait one week, > and then check which one is preferred. My preference is to have the tests clearly namespaced to the project. That can be done one of three ways: ./projectpackage/[zero-or-more-dirs-deep]/tests ./projectname_tests/ ./tests with a runner configuration to add appropriate metadata The namespacing lets me gather up test output from multiple projects and not have it confused (nor have to artificially separate out what tests ran in what test process). This is mainly useful when taking a big-data mindset to your tests, so not relevant so much to 'heres my 1k loc project, enjoy world' scenarios. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From contact at ionelmc.ro Wed Oct 7 01:46:28 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 02:46:28 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins wrote: > > Hangon, there's clearly a *huge* gap in understanding here. > > pbr does *not* modify *anyones* setup.py output unless its enabled. ?Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr doing weird stuff to distributions by abusing said entrypoints. For reference: https://bugs.launchpad.net/pbr/+bug/1483067 ? There's nothing special about pbr here. It's not like it's the first package doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like pdbpp, but you don't put that in production. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 7 01:52:34 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 19:52:34 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On October 6, 2015 at 7:47:16 PM, Ionel Cristian M?rie? (contact at ionelmc.ro) wrote: > > ?Unless it's >=1.7.0. You can't blame setuptools having entrypoints > for pbr doing weird stuff to distributions by abusing said entry points.? Relax. pbr had a bug, it was acknowledged as a bug and subsequently fixed. The design of setuptools entry points made it an easy bug to have happen than if setuptools entry points were designed differently. Using the entry points given to it by setuptools is hardly abusing anything. ? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From guettliml at thomas-guettler.de Wed Oct 7 07:12:22 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 7 Oct 2015 07:12:22 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: <5614A9B6.7030506@thomas-guettler.de> Am 07.10.2015 um 00:08 schrieb Ionel Cristian M?rie?: > On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney > wrote: > >> I think the above describes the standard way of declaring the test >> runner: The ?setup.py test? command. >> >> Now, I lament that more Python projects don't *conform to* that >> standard, but at least it exists. >> > > ?There's a very simple answer to that: easy_install (that's what `setup.py > test` will use to install deps). It has several design issue wrt how > packages are installed and how dependencies are managed. > > Lets not use `setup.py test`. It's either bad or useless. Sorry, I am not an expert in the area "packaging". I don't understand what you say. I thought "easy_install" is a very old and deprecated method. Why not use `setup.py test`? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Wed Oct 7 07:38:58 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 7 Oct 2015 07:38:58 +0200 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: <5614AFF2.1090604@thomas-guettler.de> Am 06.10.2015 um 17:33 schrieb Erik Bray: > On Tue, Oct 6, 2015 at 8:34 AM, Ionel Cristian M?rie? > wrote: >> On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou wrote: >>> >>> On Tue, 6 Oct 2015 07:07:31 -0400 >>> Donald Stufft wrote: >>>> >>>> I've never, in my entire life [...] >>> >>> Can I suggest your entire life is an anecdotal data point here? >> >> >> Make that two anecdotal data points :-) >> >>> Any number of things can be described as trivial depending on the >>> skillset and patience of the user. When users report a bug, they are >>> not expecting to be asked to download and "untar" stuff. Not every >>> user is a programmer. >> >> But seriously now, your arguments are also anecdotal. Lets not pretend we're >> objective here. That sort of attitude is disingenuous and will quickly >> devolve this discussion to mere ad hominems. > > Okay, though, so maybe if there is nothing to offer here but anecdata > then maybe we should stop acting like there's "one right way here". I > have projects that install their test suite and test dependencies > because it is frequently useful to ask users to run a self-test (and > users themselves want to be able to do it for a variety of reasons). > > There are other projects where it doesn't make sense, and those don't > have to install the tests (I still think in those cases that the tests > should live in the package instead of outside it, but simply not > installed). > > In any case, let's not get trapped into endless circular discussions > about what is correct, period, and instead consider individual use > cases--not dismissing individual projects' or peoples' experiences and > needs--and discuss what the most appropriate action is for those uses > cases. Python projects are not monolithic in their audiences (and > that includes developer audiences and user audiences). Yes, there is not generic "one right way here". Yes, let's consider individual use cases. My use case are the docs for new comers: - https://github.com/pypa/sampleproject - https://packaging.python.org/en/latest/distributing/ That's why started the thread. New comers don't have the experience you have. New comers want to get their coding done. They need a simple advice: "If unsure, then do X" And for this question, I want a "one right way here". Up to now it looks like there is no consensus. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From marius at gedmin.as Wed Oct 7 07:35:08 2015 From: marius at gedmin.as (Marius Gedminas) Date: Wed, 7 Oct 2015 08:35:08 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006172127.5dd9867a@anarchist.wooz.org> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> Message-ID: <20151007053508.GA29553@platonas> On Tue, Oct 06, 2015 at 05:21:27PM -0400, Barry Warsaw wrote: > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, > >so it apparently doesn't find it useful either. > > Oh how I wish there was a standard way to *declare* how to run the test suite, > such that all our automated tools (or the humans :) didn't have to guess. I have hopes for 'tox.ini' becoming the standard way to test a Python project. Marius Gedminas -- "Actually, the Singularity seems rather useful in the entire work avoidance field. "I _could_ write up that report now but if I put it off, I may well become a weakly godlike entity, at which point not only will I be able to type faster but my comments will be more on-target." - James Nicoll -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 173 bytes Desc: Digital signature URL: From chris.barker at noaa.gov Wed Oct 7 08:11:47 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 6 Oct 2015 23:11:47 -0700 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: <5614AFF2.1090604@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <5614AFF2.1090604@thomas-guettler.de> Message-ID: On Tue, Oct 6, 2015 at 10:38 PM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > Yes, there is not generic "one right way here". > > Yes, let's consider individual use cases. > > My use case are the docs for new comers: > > - https://github.com/pypa/sampleproject > - https://packaging.python.org/en/latest/distributing/ > > That's why started the thread. > unfortunately, that isn't a use-case -- every newcomer has a different use case. I was happy to see this thread, because I thought maybe I"d learn what i should teach my students - new to python. But alas - there clearly really is no consensus. What i've told newbies in the past is somethig like: """ if you want your user to be able to install you package, and then run something like: import my_package my_package.test() then put your tests inside the package. If you are fine with only being able to run the tests from the source tree -- then put your tests outside the package. """ but really, newbies have no idea how to make this decsion. Maybe we could come up with a decision tree for this -- some guidance for knowing what to do, when? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 7 13:11:43 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Oct 2015 06:11:43 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151007053508.GA29553@platonas> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <20151007053508.GA29553@platonas> Message-ID: On Oct 7, 2015 12:44 AM, "Marius Gedminas" wrote: > > On Tue, Oct 06, 2015 at 05:21:27PM -0400, Barry Warsaw wrote: > > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > > > >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, > > >so it apparently doesn't find it useful either. > > > > Oh how I wish there was a standard way to *declare* how to run the test suite, > > such that all our automated tools (or the humans :) didn't have to guess. make test > > I have hopes for 'tox.ini' becoming the standard way to test a Python > project. * https://tox.readthedocs.org/en/latest/config.html * https://github.com/docker/docker-registry/blob/master/tox.ini #flake8 * dox = docker + tox | PyPI: https://pypi.python.org/pypi/dox | Src: https://git.openstack.org/cgit/stackforge/dox/tree/dox.yml * docker-compose.yml | Docs: https://docs.docker.com/compose/ | Docs: https://github.com/docker/compose/blob/master/docs/yml.md * https://github.com/kelseyhightower/kubernetes-docker-files/blob/master/docker-compose.yml * https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/pods.md#alternatives-considered * https://github.com/docker/docker/issues/8781 ( pods ( containers ) ) * http://docs.buildbot.net/latest/tutorial/docker.html * http://docs.buildbot.net/current/tutorial/docker.html#building-and-running-buildbot tox.ini often is not sufficient: * [Makefile: make test/tox] * setup.py * tox.ini * docker/platform-ver/Dockerfile * [dox.yml] * [docker-compose.yml] * [CI config] * http://docs.buildbot.net/current/manual/configuration.html * jenkins-kubernetes, jenkins-mesos > > Marius Gedminas > -- > "Actually, the Singularity seems rather useful in the entire work avoidance > field. "I _could_ write up that report now but if I put it off, I may well > become a weakly godlike entity, at which point not only will I be able to > type faster but my comments will be more on-target." - James Nicoll > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 7 13:58:06 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 14:58:06 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <5614A9B6.7030506@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 8:12 AM, Thomas G?ttler wrote: > I thought "easy_install" is a very old and deprecated method. > ?Indeed it is. That why people put all sorts of custom "test" commands in their setup.py to work around the deficiencies of the "test? ?" command setuptools provides.? So we end up with lots of variations of "how to use pytest to run tests via `setup.py test`", "how to use pip to install deps, instead of what `setup.py test` normally does" and so on. If you're gonna implement a test runner in your setup.py you might as well use a supported and well maintained tool: tox. > Why not use `setup.py test`? > ?Because: 1. There's Tox?, which does exactly that, and more. It's maintained. It gets features. 2. The "test" command will install the "test_requires" dependencies as eggs. You will end up with multiple versions of the same eggs right in your source checkout. 3. The "test" command will install the "test_requires" dependencies with easy_install. That means wheels cannot be used. 4. Because the builtin "test" command is so bare people tend to implement a custom one. Everyone does something slightly different, and slightly buggy. 5. There's no established tooling that relies on `setup.py test`. There isn't even a test result protocol like TAP [1] for it. Why use something so limited and outdated if there's no practical advantage? [1] https://testanything.org/ Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 7 14:18:25 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 08:18:25 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On October 7, 2015 at 7:58:55 AM, Ionel Cristian M?rie? (contact at ionelmc.ro) wrote: > On Wed, Oct 7, 2015 at 8:12 AM, Thomas G?ttler > > wrote: > > Why not use `setup.py test`? > > > > Because: > > 1. There's Tox?, which does exactly that, and more. It's maintained. It > gets features. tox and setup.py test are not really equivalent.?There?s no way (to my knowledge) to test the item outside of a virtual environment. This is important for?downstreams who want to test that the package build and the tests successfully are executed in their environment, not within some virtual environment. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Wed Oct 7 14:20:36 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Oct 2015 07:20:36 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Oct 7, 2015 6:58 AM, "Ionel Cristian M?rie?" wrote: > > > On Wed, Oct 7, 2015 at 8:12 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: >> >> I thought "easy_install" is a very old and deprecated method. > > > ?Indeed it is. That why people put all sorts of custom "test" commands in their setup.py to work around the deficiencies of the "test? > ?" command setuptools provides.? So we end up with lots of variations of "how to use pytest to run tests via `setup.py test`", "how to use pip to install deps, instead of what `setup.py test` normally does" and so on. > > If you're gonna implement a test runner in your setup.py you might as well use a supported and well maintained tool: tox. > >> >> Why not use `setup.py test`? > > > ?Because: > > 1. There's Tox?, which does exactly that, and more. It's maintained. It gets features. Tox rocks. * detox can run concurrent processes: https://pypi.python.org/pypi/detox/ * TIL timeit.default_timer measures **wall time** by default and not CPU time: concurrent test timings are likely different from linear tests run on an machine with load > 2. The "test" command will install the "test_requires" dependencies as eggs. You will end up with multiple versions of the same eggs right in your source checkout. * is there no way around this? * is this required / spec'd / fixable? > 3. The "test" command will install the "test_requires" dependencies with easy_install. That means wheels cannot be used. would it be possible to add this to wheel? as if, after package deployment, in-situ tests are no longer relevant. (I think it wise to encourage TDD here) > 4. Because the builtin "test" command is so bare people tend to implement a custom one. Everyone does something slightly different, and slightly buggy. * README.rst test invocation examples (all, subset, one) * Makefile (make test; [vim] :make) * python setup.py nosetests http://nose.readthedocs.org/en/latest/api/commands.html * python setup.py [test] https://pytest.org/latest/goodpractises.html#integrating-with-setuptools-python-setup-py-test > 5. There's no established tooling that relies on `setup.py test`. There isn't even a test result protocol like TAP [1] for it. Why use something so limited and outdated if there's no practical advantage? * xUnit XML: https://westurner.org/wiki/awesome-python-testing#xunit-xml ``` xUnit XML? https://en.wikipedia.org/wiki/XUnit https://nose.readthedocs.org/en/latest/plugins/xunit.html http://nosexunit.sourceforge.net/ https://pytest.org/latest/usage.html#creating-junitxml-format-files https://github.com/xmlrunner/unittest-xml-reporting https://github.com/zandev/shunit2/compare/master...jeremycarroll:master ``` * TAP protocol > > [1] https://testanything.org/ > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 7 15:42:05 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 16:42:05 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 3:18 PM, Donald Stufft wrote: > tox and setup.py test are not really equivalent. There?s no way (to my > knowledge) to test the item outside of a virtual environment. This is > important for downstreams who want to test that the package build and the > tests successfully are executed in their environment, not within some > virtual environment. ?Hmmmm ... you're right. But making Tox not use virtualenvs is not impossible - much alike? to how Detox is working, we could have a "Tax" (just made that up) that just skips making any virtualenv. It's a matter of making two subclasses and a console_scripts entrypoint (I think). I think it's a good name: ``use Tax instead of Tox if you wanna "tax" your global site-packages`` :-) We only need someone to do it. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 7 15:49:52 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 16:49:52 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 3:20 PM, Wes Turner wrote: > > 2. The "test" command will install the "test_requires" dependencies as > eggs. You will end up with multiple versions of the same eggs right in your > source checkout. > > * is there no way around this? > * is this required / spec'd / fixable? > ?It's not that bad now, recent setuptools put the eggs in a ".eggs" dir - so it's not as messy as before. ? > > 3. The "test" command will install the "test_requires" dependencies with > easy_install. That means wheels cannot be used. > > would it be possible to add this to wheel? > It's up to the maintainers of wheel/setuptools to figure this one out (or not) I think. Either way, you should search through the distutils-sig archives for clues/intentions, eg: https://mail.python.org/pipermail/distutils-sig/2014-December/thread.html#25482 ? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Oct 7 17:13:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 7 Oct 2015 11:13:07 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Tue, Oct 6, 2015 at 6:08 PM, Ionel Cristian M?rie? wrote: > > On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney > wrote: >> >> I think the above describes the standard way of declaring the test >> runner: The ?setup.py test? command. >> >> Now, I lament that more Python projects don't *conform to* that >> standard, but at least it exists. > > > There's a very simple answer to that: easy_install (that's what `setup.py > test` will use to install deps). It has several design issue wrt how > packages are installed and how dependencies are managed. > > Lets not use `setup.py test`. It's either bad or useless. Says who? Many of the projects I'm involved in use `setup.py test` exclusively and for good reason--they all have C and/or Cython extension modules that need to be built for the tests to even run. Only setup.py knows about those extension modules and how to find and build them. Using `setup.py test` ensures that everything required to run the package (including runtime dependencies) is built and ready, and then the tests can start. Without it, we would have to tell developers to go through a build process first and then make sure they're running the tests on the built code. `setup.py test` makes it a no-brainer. For pure Python packages I think it's less important and can usually rely on "just run 'nose', or 'py.test'" (or "tox" but that's true regardless of how the tests are invoked outside of tox). Best, Erik From contact at ionelmc.ro Wed Oct 7 17:31:26 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 18:31:26 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 6:13 PM, Erik Bray wrote: > > Lets not use `setup.py test`. It's either bad or useless. > > Says who? Many of the projects I'm involved in use `setup.py test` > exclusively and for good reason--they all have C and/or Cython > extension modules that need to be built for the tests to even run. > Only setup.py knows about those extension modules and how to find and > build them. Using `setup.py test` ensures that everything required to > run the package (including runtime dependencies) is built and ready, ?Well ok, then it's not useless. :-) For pure Python packages I think it's less important and can usually > rely on "just run 'nose', or 'py.test'" (or "tox" but that's true > regardless of how the tests are invoked outside of tox). > That implies you would be testing code that you didn't install. That allows preventable mistakes, like publishing releases on PyPI that don't actually work, or do not even install at all (because you didn't test that). `setup.py test` doesn't really allow you to fully test that part, but Tox does. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Oct 7 17:32:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 7 Oct 2015 11:32:07 -0400 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) Message-ID: On Tue, Oct 6, 2015 at 7:46 PM, Ionel Cristian M?rie? wrote: > > On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins > wrote: >> >> >> Hangon, there's clearly a *huge* gap in understanding here. >> >> pbr does *not* modify *anyones* setup.py output unless its enabled. > > > Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr > doing weird stuff to distributions by abusing said entrypoints. > > For reference: https://bugs.launchpad.net/pbr/+bug/1483067 > > There's nothing special about pbr here. It's not like it's the first package > doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like > pdbpp, but you don't put that in production. Starting a sub-thread since issues with pbr are orthogonal to the original disucssion. But one point I'd like to raise about this is that when I originally designed d2to1, on which a chunk of pbr is based, it was *explicitly* designed to never be installed in site-packages (with the exception of downstream packaging systems which can do what they want and are more controlled). This is exactly because I knew different packages might have dependencies on different versions of d2to1 as features are added, and that if some version is installed in site-packages it can lead to VersionConflict issues (this is in part exacerbated by a bug/misfeature in setuptools--I fixed that bug a while ago but the fix had to be rolled back due to a regression [1]). So TL;DR unless you know what you're doing, d2to1 should *never* be "installed"--it was only meant to be used with setup_requires, where the appropriate d2to1 used in building/installing a package is only temporarily enabled on sys.path via a temporary egg install. If some project is making it a *runtime* requirement that's a mistake. I don't know what features pbr has grown that might make someone want it to be a runtime dependency (the only entry-points I noticed were for adding egg-info writers but that should only be needed at build-time too...), but maybe something like that should be split off as a separate module or something... Best, Erik [1] https://bitbucket.org/tarek/distribute/pull-requests/20/fixes-and-adds-a-regression-test-for-323/diff From erik.m.bray at gmail.com Wed Oct 7 17:37:45 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 7 Oct 2015 11:37:45 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 11:31 AM, Ionel Cristian M?rie? wrote: > > On Wed, Oct 7, 2015 at 6:13 PM, Erik Bray wrote: >> >> > Lets not use `setup.py test`. It's either bad or useless. >> >> Says who? Many of the projects I'm involved in use `setup.py test` >> exclusively and for good reason--they all have C and/or Cython >> extension modules that need to be built for the tests to even run. >> Only setup.py knows about those extension modules and how to find and >> build them. Using `setup.py test` ensures that everything required to >> run the package (including runtime dependencies) is built and ready, > > > Well ok, then it's not useless. :-) > >> For pure Python packages I think it's less important and can usually >> rely on "just run 'nose', or 'py.test'" (or "tox" but that's true >> regardless of how the tests are invoked outside of tox). > > > That implies you would be testing code that you didn't install. That allows > preventable mistakes, like publishing releases on PyPI that don't actually > work, or do not even install at all (because you didn't test that). > `setup.py test` doesn't really allow you to fully test that part, but Tox > does. Which, incidentally, is a great reason for installable tests :) Running in the source tree is great for development. But when preparing a release it's great to be able to create an sdist, install that into a virtualenv, and run `package.test()` or `python -m package.tests` or whatever. Occasionally catches problems with the source dist if nothing else. Best, Erik From contact at ionelmc.ro Wed Oct 7 18:29:26 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 19:29:26 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 6:37 PM, Erik Bray wrote: > Which, incidentally, is a great reason for installable tests :) > Not really. Doesn't matter where you have the tests. It matters where you have the code being tested. Tests being installed is a mere consequence of the location of tests. ? > Running in the source tree is great for development. But when > preparing a release it's great to be able to create an sdist, install > that into a virtualenv, and run `package.test()` or `python -m > package.tests` or whatever. Occasionally catches problems with the > source dist if nothing else. > ?? ?As I said, I like the idea. It's just that it's not feasible right now. Lets go over the issues again: * Tests too bulky (pyca/cryptography) * Tests can't be installed at all: ? https://github.com/getpelican/pelican/issues/1409? * Not clear how to install test dependencies. tests_require? extras? no deps? What about version conflicts and way too many deps being installed. Dependencies are like cars, they are very useful but too many of them create problems. * Real problems like standardized test output or run protocol are not solved at all. Little benefit of doing it like this if you can't build good CI tools around this. * Workflows are under-specified. User are not guided to make quality releases on PyPI. ?Maybe we should have a PEP that would specify/propose some concrete solutions to all those? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 7 19:27:28 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 10:27:28 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Mon, Oct 5, 2015 at 6:51 AM, Donald Stufft wrote: [...] > I also don't think it will be confusing. They'll associate the VCS thing (a source release) as something focused on development for most everyone. Most people won't explicitly make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with manually producing binary wheels or producing source releases other than what's checked into vcs will be something that I suspect only advanced users will do. Of course people will make source releases, and should be able to upload them to PyPI. The end goal is that *pip* will not use source releases, but PyPI is not just there for pip. If it was, it wouldn't even show package descriptions :-). There are projects on PyPI right now, today, that have no way to generate sdists and will never have any need for "source wheels" (because they don't use distutils and they build "none-any" wheels directly from their source). It should still be possible for them to upload source releases for all the other reasons that having source releases is useful: they form a permanent record of the whole project state (including potentially docs, tests, working notes, etc. that don't make it into the wheels), human users may well want to download those archives, Debian may prefer to use that as their orig.tar.gz, etc. etc. And on the other end of the complexity scale, there are projects like numpy where it's not clear to me whether they'll ever be able to support "source wheels", and even if they do they'll still need source releases to support user configuration at build time. -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Wed Oct 7 19:51:44 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 13:51:44 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 7, 2015 at 1:27:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Mon, Oct 5, 2015 at 6:51 AM, Donald Stufft wrote: > [...] > > I also don't think it will be confusing. They'll associate the VCS thing (a source release) > as something focused on development for most everyone. Most people won't explicitly > make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces > a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with > manually producing binary wheels or producing source releases other than what's checked > into vcs will be something that I suspect only advanced users will do. > > Of course people will make source releases, and should be able to > upload them to PyPI. The end goal is that *pip* will not use source > releases, but PyPI is not just there for pip. If it was, it wouldn't > even show package descriptions :-). > > There are projects on PyPI right now, today, that have no way to > generate sdists and will never have any need for "source wheels" > (because they don't use distutils and they build "none-any" wheels > directly from their source). It should still be possible for them to > upload source releases for all the other reasons that having source > releases is useful: they form a permanent record of the whole project > state (including potentially docs, tests, working notes, etc. that > don't make it into the wheels), human users may well want to download > those archives, Debian may prefer to use that as their orig.tar.gz, > etc. etc. > > And on the other end of the complexity scale, there are projects like > numpy where it's not clear to me whether they'll ever be able to > support "source wheels", and even if they do they'll still need source > releases to support user configuration at build time. We must have different ideas of what a source release vs source wheel would look like, because I'm having a hard time squaring what you've said here with what it looks like in my head. In my head, source releases (outside of the VCS use case) will be rare and only for very complex packages that are doing very complex things. Source wheels will be something that will be semi mandatory to being a well behaved citizen (for Debian and such to download) and binary wheels will be something that you'll want to have but aren't required. I don't see any reason why source wheels wouldn't include docs, tests, and other misc files. I picture building a binary wheel directly being something similar to using fpm to build binary .deb packages directly, totally possible but unadvised. Having talked to folks who deal with Debian/Fedora packages, they won't accept a binary wheel as the input source and (given how I explained it to them) they are excited about the concept of source wheels and moving away from dynamic metadata and towards static metadata. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Wed Oct 7 20:14:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 19:14:20 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 18:27, Nathaniel Smith wrote: > There are projects on PyPI right now, today, that have no way to > generate sdists and will never have any need for "source wheels" I think I'm as confused by what you're saying here as Donald is. Could you give a few examples of such projects? I'd like to go & take a look at them and try to understand what they are doing that is so incompatible with what Donald and I are thinking of as a"source wheel". Paul From donald at stufft.io Wed Oct 7 20:23:02 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 14:23:02 -0400 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On October 7, 2015 at 11:32:32 AM, Erik Bray (erik.m.bray at gmail.com) wrote: > > I don't know what features pbr has grown that might make someone > want > it to be a runtime dependency (the only entry-points I noticed > were > for adding egg-info writers but that should only be needed at > build-time too...), but maybe something like that should be > split off > as a separate module or something? It has runtime utilities for dealing with versions. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From oscar.j.benjamin at gmail.com Wed Oct 7 20:30:46 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 07 Oct 2015 18:30:46 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, 7 Oct 2015 18:51 Donald Stufft wrote: On October 7, 2015 at 1:27:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Mon, Oct 5, 2015 at 6:51 AM, Donald Stufft wrote: > [...] > > I also don't think it will be confusing. They'll associate the VCS thing (a source release) > as something focused on development for most everyone. Most people won't explicitly > make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces > a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with > manually producing binary wheels or producing source releases other than what's checked > into vcs will be something that I suspect only advanced users will do. > > Of course people will make source releases, and should be able to > upload them to PyPI. The end goal is that *pip* will not use source > releases, but PyPI is not just there for pip. If it was, it wouldn't > even show package descriptions :-). > > There are projects on PyPI right now, today, that have no way to > generate sdists and will never have any need for "source wheels" > (because they don't use distutils and they build "none-any" wheels > directly from their source). It should still be possible for them to > upload source releases for all the other reasons that having source > releases is useful: they form a permanent record of the whole project > state (including potentially docs, tests, working notes, etc. that > don't make it into the wheels), human users may well want to download > those archives, Debian may prefer to use that as their orig.tar.gz, > etc. etc. > > And on the other end of the complexity scale, there are projects like > numpy where it's not clear to me whether they'll ever be able to > support "source wheels", and even if they do they'll still need source > releases to support user configuration at build time. We must have different ideas of what a source release vs source wheel would look like, because I'm having a hard time squaring what you've said here with what it looks like in my head. In my head, source releases (outside of the VCS use case) will be rare and only for very complex packages that are doing very complex things. Source wheels will be something that will be semi mandatory to being a well behaved citizen (for Debian and such to download) and binary wheels will be something that you'll want to have but aren't required. I don't see any reason why source wheels wouldn't include docs, tests, and other misc files. I picture building a binary wheel directly being something similar to using fpm to build binary .deb packages directly, totally possible but unadvised. Having talked to folks who deal with Debian/Fedora packages, they won't accept a binary wheel as the input source and (given how I explained it to them) they are excited about the concept of source wheels and moving away from dynamic metadata and towards static metadata. Your idea of an sdist as something that has fully static build/runtime dependency metadata and a one to one correspondence with binary wheels is not a usable format when releasing the code for e.g. numpy 1.10. It's fine to say that pip/PyPI should work with the source in some other distribution format and numpy could produce that but it means that the standard tarball release needs to be supported some how separately. Numpy should be able to use PyPI in order to host the tarball even if pip ignores the file. If numpy released only source wheels then there would be more than one source wheel for each release corresponding to e.g. the different ways that numpy is linked. There still needs to be a way to release a single file representing the code for the release as a whole. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 7 20:42:00 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 14:42:00 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 7, 2015 at 2:31:03 PM, Oscar Benjamin (oscar.j.benjamin at gmail.com) wrote: > > > Your idea of an sdist as something that has fully static build/runtime > dependency metadata and a one to one correspondence with binary > wheels is not a usable format when releasing the code for e.g. > numpy 1.10. It's fine to say that pip/PyPI should work with the > source in some other distribution format and numpy could produce > that but it means that the standard tarball release needs to be > supported some how separately. Numpy should be able to use PyPI > in order to host the tarball even if pip ignores the file. > > > If numpy released only source wheels then there would be more > than one source wheel for each release corresponding to e.g. > the different ways that numpy is linked. There still needs to > be a way to release a single file representing the code for the > release as a whole. > Can you expand on this please? I've never used numpy for anything serious and I'm trying to figure out why and what parts of what I'm thinking of wouldn't work for it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From barry at python.org Wed Oct 7 21:04:26 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 15:04:26 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: <20151007150426.436ae7fd@anarchist.wooz.org> On Oct 07, 2015, at 08:18 AM, Donald Stufft wrote: >tox and setup.py test are not really equivalent.?There?s no way (to my >knowledge) to test the item outside of a virtual environment. This is >important for?downstreams who want to test that the package build and the >tests successfully are executed in their environment, not within some virtual >environment. I usually do not use tox to test a package when building it for Debian. It's pretty easy to extract the actual command used to run the test suit from the tox.ini and that's what I put in the debian/rules file. It can make things build a little more reliably, but also eliminates a build dependency on tox. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 21:13:14 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 15:13:14 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <20151007053508.GA29553@platonas> Message-ID: <20151007151314.35bf4c3e@anarchist.wooz.org> On Oct 07, 2015, at 08:35 AM, Marius Gedminas wrote: >I have hopes for 'tox.ini' becoming the standard way to test a Python >project. As do I, modulo outliers of course. I'd like to see 90% of PyPI packages have a tox.ini. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 21:14:58 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 15:14:58 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> <85h9m3zmqq.fsf@benfinney.id.au> <20151006181934.410b0395@anarchist.wooz.org> <857fmzzkc4.fsf@benfinney.id.au> Message-ID: <20151007151458.61cf001b@anarchist.wooz.org> On Oct 07, 2015, at 09:46 AM, Ben Finney wrote: >So ?I'm a big fan of putting tests inside the [Python] package >[directory]? can't be motivated by ?Having the tests there in the >installed package?. The two aren't related, AFAICT. It makes it easier for sure. When the tests are inside the package, nothing special has to be done; you just install the package and the tests subdirectories come along for the ride. If the tests are outside the package then you first have to figure out where they're going to go when they're installed, and then do something special to get them there. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Wed Oct 7 21:36:12 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 07 Oct 2015 19:36:12 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, 7 Oct 2015 19:42 Donald Stufft wrote: On October 7, 2015 at 2:31:03 PM, Oscar Benjamin (oscar.j.benjamin at gmail.com) wrote: > > > Your idea of an sdist as something that has fully static build/runtime > dependency metadata and a one to one correspondence with binary > wheels is not a usable format when releasing the code for e.g. > numpy 1.10. It's fine to say that pip/PyPI should work with the > source in some other distribution format and numpy could produce > that but it means that the standard tarball release needs to be > supported some how separately. Numpy should be able to use PyPI > in order to host the tarball even if pip ignores the file. > > > If numpy released only source wheels then there would be more > than one source wheel for each release corresponding to e.g. > the different ways that numpy is linked. There still needs to > be a way to release a single file representing the code for the > release as a whole. > Can you expand on this please? I've never used numpy for anything serious and I'm trying to figure out why and what parts of what I'm thinking of wouldn't work for it. Currently I can take the code from the numpy release and compile it in different incompatible ways. For example I could make a wheel that bundles a BLAS library. Or I could make a wheel that expects to use a system BLAS library that should be installed separately somehow or I could build a wheel against pyopenblas and make a wheel that depends on pyopenblas. Or I could link a BLAS library statically into numpy. A numpy release supports being compiled and linked in many different ways and will continue to do so regardless of any decisions made by PYPA. What that means is that there is not a one to one correspondence between a numpy release and a binary wheel. If there must be a one to one correspondence between a source wheel and a binary wheel then it follows that there cannot be a one to one correspondence between the source release and a source wheel. Of course numpy could say that they will only upload one particular source wheel and binary wheel to PyPI but people need to be able to use the source release in many different ways. So only releasing a source wheel that maps one to one to a particular way of compiling numpy is not an acceptable way for numpy to release its code. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 7 22:28:42 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 21:28:42 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 20:36, Oscar Benjamin wrote: > Currently I can take the code from the numpy release and compile it in > different incompatible ways. For example I could make a wheel that bundles a > BLAS library. Or I could make a wheel that expects to use a system BLAS > library that should be installed separately somehow or I could build a wheel > against pyopenblas and make a wheel that depends on pyopenblas. Or I could > link a BLAS library statically into numpy. > > A numpy release supports being compiled and linked in many different ways > and will continue to do so regardless of any decisions made by PYPA. What > that means is that there is not a one to one correspondence between a numpy > release and a binary wheel. If there must be a one to one correspondence > between a source wheel and a binary wheel then it follows that there cannot > be a one to one correspondence between the source release and a source > wheel. > > Of course numpy could say that they will only upload one particular source > wheel and binary wheel to PyPI but people need to be able to use the source > release in many different ways. So only releasing a source wheel that maps > one to one to a particular way of compiling numpy is not an acceptable way > for numpy to release its code. The disconnect here seems to be that I view all of those wheels as being numpy 1.9.X wheels (or whatever). They differ in terms of compatibility details, but they are all wheels for the same project/version. So there's no problem with them all being built from the same source wheel. I also have no problem with it being possible to configure the build differently from a single source wheel, to generate all those wheels. The configuration isn't metadata, it's "just" settings for the build. Of course, there *is* an unsolved issue here, which is how we manage compatibility for wheels at the level needed for numpy. But I thought the discussion on that was ongoing? I'm concerned that this proposal is actually about bypassing that discussion, and instead trying to treat incompatibly linked wheels as "different" in terms of project metadata, which I think is the wrong way of handling things. I note that Christoph Gohlke's numpy builds are tagged with a "+mkl" local version modifier - that's presumably intended to mark the fact that they are built with an incompatible runtime - but that's a misuse of local versions (and I've found it causes niggling issues with how pip recognises upgrades, etc). So, in summary: Your points above don't seem to me to in any way preclude having a single numpy source wheel, and a number of (mutually incompatible, but the same in terms of project and version) binary wheels. Paul From contact at ionelmc.ro Wed Oct 7 22:34:19 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 23:34:19 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 4:42 PM, Ionel Cristian M?rie? wrote: > On Wed, Oct 7, 2015 at 3:18 PM, Donald Stufft wrote: > >> tox and setup.py test are not really equivalent. There?s no way (to my >> knowledge) to test the item outside of a virtual environment. This is >> important for downstreams who want to test that the package build and the >> tests successfully are executed in their environment, not within some >> virtual environment. > > > ?Hmmmm ... you're right. But making Tox not use virtualenvs is not > impossible - much alike? to how Detox is working, we could have a "Tax" > (just made that up) that just skips making any virtualenv. It's a matter of > making two subclasses and a console_scripts entrypoint (I think). I think > it's a good name: ``use Tax instead of Tox if you wanna "tax" your global > site-packages`` :-) > ?Just for kicks, I verified this, it's not hard at all: https://pypi.python.org/pypi/tax Barry may want to look at it, in case he has too many tox.ini files to copy-paste from :-) Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 7 23:28:50 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 14:28:50 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, Oct 7, 2015 at 1:28 PM, Paul Moore wrote: > The disconnect here seems to be that I view all of those wheels as > being numpy 1.9.X wheels (or whatever). They differ in terms of > compatibility details, but they are all wheels for the same > project/version. So there's no problem with them all being built from > the same source wheel. I also have no problem with it being possible > to configure the build differently from a single source wheel, to > generate all those wheels. The configuration isn't metadata, it's > "just" settings for the build. But the different builds for the different configurations end up with different metadata. If I'm understanding right, the whole point of "source wheels" is that they have all the static metadata that pip needs in order to make decisions, and this has to match the resulting wheels -- right? The way I'm imagining it is that there are multiple levels of metadata staticness: package name, author, description, ... static in: VCS checkouts, source releases, source wheels, wheels package version static in: source releases, source wheels, wheels package dependencies static in: source wheels, wheels environment tag static in: wheels > Of course, there *is* an unsolved issue here, which is how we manage > compatibility for wheels at the level needed for numpy. But I thought > the discussion on that was ongoing? I'm concerned that this proposal > is actually about bypassing that discussion, and instead trying to > treat incompatibly linked wheels as "different" in terms of project > metadata, which I think is the wrong way of handling things. I note > that Christoph Gohlke's numpy builds are tagged with a "+mkl" local > version modifier - that's presumably intended to mark the fact that > they are built with an incompatible runtime - but that's a misuse of > local versions (and I've found it causes niggling issues with how pip > recognises upgrades, etc). Yeah, that's not a good long term solution -- it needs to be moved into the metadata (probably by creating an MKL wheel and then making the numpy wheel depend on it). That's exactly the problem :-) > So, in summary: Your points above don't seem to me to in any way > preclude having a single numpy source wheel, and a number of (mutually > incompatible, but the same in terms of project and version) binary > wheels. Maybe I have misunderstood: does it actually help pip at all to have static access to name and version, but not to anything else? I've been assuming not, but I don't think anyone's pointed to any examples yet of the problems that pip is encountering due to the lack of static metadata -- would this actually be enough to solve them? -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Wed Oct 7 23:41:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 22:41:56 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 22:28, Nathaniel Smith wrote: > Maybe I have misunderstood: does it actually help pip at all to have > static access to name and version, but not to anything else? I've been > assuming not, but I don't think anyone's pointed to any examples yet > of the problems that pip is encountering due to the lack of static > metadata -- would this actually be enough to solve them? The principle I am working on is that *all* metadata in a source wheel should be statically available - that's not just for pip, but for all other consumers, including distro packagers. What's not set in stone is precisely what (subsets of) metadata are appropriate for source wheels as opposed to (binary) wheels. So I'd counter your question with the converse - what metadata specifically are you unwilling to include statically in source wheels? My feeling is that there isn't anything you'd be unwilling to include that I'd consider as "source wheel metadata". Possibly the nearest we'd have to an issue is over allowing the build process to *add* dependencies to a binary wheel (e.g. a some builds depend on currently-hypothetical MKL wheel, which provides needed DLLs). I don't in principle object to that, but I'd like to see a fleshed out proposal on how wheels containing just DLLs (as opposed to Python packages) would work in practice - until we have a mechanism for building/distributing such wheels, I think it's premature to worry about specifying dependencies. But whatever comes out of this, the Metadata 2.0 spec should ultimately be updated to note which metadata is mandated in source wheels, and which in binary wheels only. Paul From njs at pobox.com Wed Oct 7 23:53:46 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 14:53:46 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, Oct 7, 2015 at 11:14 AM, Paul Moore wrote: > On 7 October 2015 at 18:27, Nathaniel Smith wrote: >> There are projects on PyPI right now, today, that have no way to >> generate sdists and will never have any need for "source wheels" > > I think I'm as confused by what you're saying here as Donald is. Could > you give a few examples of such projects? I'd like to go & take a look > at them and try to understand what they are doing that is so > incompatible with what Donald and I are thinking of as a"source > wheel". An example would be flit itself: https://github.com/takluyver/flit https://pypi.python.org/pypi/flit It's not that you couldn't support a "source wheel" here, it's just that forcing them to go checkout -> source wheel -> wheel would be adding pointless hassle while accomplishing nothing useful. pip would never actually touch the source wheel, and for the remaining use cases for source distribution, a classic "source release" that's basically a tarball of a VCS checkout + static version number would be more familiar and useful. -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Wed Oct 7 23:58:45 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 17:58:45 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > Yeah, that's not a good long term solution -- it needs to be moved > into the metadata (probably by creating an MKL wheel and then > making > the numpy wheel depend on it). That's exactly the problem :-) >? Are you available on IRC or for a video call or something? I feel like there's something foundational from both sides that we're each missing here and it'd be easier to just hash it out in real time rather than lobbying random emails coming from places of confusion (at least on my side). I'm not sure if Paul (or anyone else!) would want to jump in on it too, though I feel like probably if it's me and you then the two "sides" will probably be reasonably well represented so if more folks don't want to join that's probably OK too, particularly since we wouldn't be making any actual decisions there :D ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Thu Oct 8 00:25:36 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 23:25:36 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 22:53, Nathaniel Smith wrote: >> I think I'm as confused by what you're saying here as Donald is. Could >> you give a few examples of such projects? I'd like to go & take a look >> at them and try to understand what they are doing that is so >> incompatible with what Donald and I are thinking of as a"source >> wheel". > > An example would be flit itself: > https://github.com/takluyver/flit > https://pypi.python.org/pypi/flit > > It's not that you couldn't support a "source wheel" here, it's just > that forcing them to go checkout -> source wheel -> wheel would be > adding pointless hassle while accomplishing nothing useful. pip would > never actually touch the source wheel, and for the remaining use cases > for source distribution, a classic "source release" that's basically a > tarball of a VCS checkout + static version number would be more > familiar and useful. I'm not sure I follow. If you have a binary wheel of flit, "pip install flit" won't need a source wheel, certainly (that's just as true for flit as for something complex like numpy). But distro packages would still want a source wheel to build their packages. If you mean that flit itself wouldn't use a source wheel, then while that may well be true, it's hardly relevant - whether flit chooses to use a source wheel is its own choice. But I'd hope flit *could* use a source wheel, as otherwise I couldn't use it to build wheels for other projects which want to use it and distribute source wheels. Should any such exist - this is pretty hypothetical at this point, and so not likely to be very productive. I am inclined to think that we're basically in agreement, we're just confused over terminology, and/or worrying about hypothetical cases. Would it help if I said that the *only* distinction between "source release" and source wheel that I care about is that in a source wheel the metadata must be static? We can discuss what metadata precisely, and we can thrash out other differences that might make more use of the fact that conceptually a "source release" is for humans to work with whereas a source wheel is for tools to consume, but those are details. I'm not clear if you think I have some more complicated picture than that, but really I don't [1]. Paul [1] I'd like a source wheel to have a defined format, but even that's not a killer. A zipfile with 2 directories "metadata" containing machine readable static metadata, and "source" with the complete contents of a source release, would do me. Of course when you build, if the metadata the build produces doesn't match the static data, that's a bug in the project packaging and we'd want to guard against it (it's the main reason the static data in the current sdist format is useless, that we can't rely on it :-() We can thrash this sort of stuff out, though. From robertc at robertcollins.net Thu Oct 8 00:39:11 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 8 Oct 2015 11:39:11 +1300 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On 8 October 2015 at 04:32, Erik Bray wrote: > Starting a sub-thread since issues with pbr are orthogonal to the > original disucssion. > > But one point I'd like to raise about this is that when I originally > designed d2to1, on which a chunk of pbr is based, it was *explicitly* > designed to never be installed in site-packages (with the exception of > downstream packaging systems which can do what they want and are more > controlled). This is exactly because I knew different packages might > have dependencies on different versions of d2to1 as features are > added, and that if some version is installed in site-packages it can > lead to VersionConflict issues (this is in part exacerbated by a > bug/misfeature in setuptools--I fixed that bug a while ago but the fix > had to be rolled back due to a regression [1]). So yes - that principle makes a lot of sense. There are two factors for pbr. a) as Donald mentions, there's a runtime API too - but we could indeed split that out into a separate package, if it would help things. b) more importantly, in OpenStack infra we can't use easy-install - its inconsistency with pip and lack of updated handling for wheels, HTTPS, separate configuration - consistently cause headaches every single time it happens. So many years ago we put in place pre-installation of all known build-time dependencies - we just install them statically, because we find the effort required to make them be compatible is less than the headaches from easy-install. As such pbr has a hard API requirement: thou shalt be backwards compatible. Clearly b) can hit VersionConflicts if pbr (and any other build dependencies like setuptools itself) are out of date, but that is easily handled for automated environments (pip install -U pip setuptools wheel pbr && echo YAY), and it errors cleanly enough for hand-use by folk that its a decent enough tradeoff in our experience. > I don't know what features pbr has grown that might make someone want > it to be a runtime dependency (the only entry-points I noticed were > for adding egg-info writers but that should only be needed at > build-time too...), but maybe something like that should be split off > as a separate module or something... Its all about easy-install. This is why I put a proof of concept static-setup-requires thing together for pip (and its on our teams roadmap to deliver a production version of it via patches to all the pypa projects, we're just not at that point yet - the resolver is first, and we have to finish rolling out constraints within OpenStack before that gets large timeslices). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Thu Oct 8 06:47:38 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 21:47:38 -0700 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On Wed, Oct 7, 2015 at 3:39 PM, Robert Collins wrote: [...] > Its all about easy-install. This is why I put a proof of concept > static-setup-requires thing together for pip (and its on our teams > roadmap to deliver a production version of it via patches to all the > pypa projects, we're just not at that point yet - the resolver is > first, and we have to finish rolling out constraints within OpenStack > before that gets large timeslices). I remembered Nick saying something about this at PyCon, but I couldn't find anything when I looked -- could you point me to the PoC? -n -- Nathaniel J. Smith -- http://vorpus.org From oscar.j.benjamin at gmail.com Thu Oct 8 12:18:22 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 11:18:22 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 22:41, Paul Moore wrote: > On 7 October 2015 at 22:28, Nathaniel Smith wrote: >> Maybe I have misunderstood: does it actually help pip at all to have >> static access to name and version, but not to anything else? I've been >> assuming not, but I don't think anyone's pointed to any examples yet >> of the problems that pip is encountering due to the lack of static >> metadata -- would this actually be enough to solve them? > > The principle I am working on is that *all* metadata in a source wheel > should be statically available - that's not just for pip, but for all > other consumers, including distro packagers. What's not set in stone > is precisely what (subsets of) metadata are appropriate for source > wheels as opposed to (binary) wheels. A concrete example would be whether or not the numpy source wheel depends on pyopenblas. Depending on how numpy is built the binary wheel may or may not depend on pyopenblas. It doesn't make any sense to say that the numpy source release depends on pyopenblas so what should be the dependencies of the source wheel? One possibility which I think is what Nathaniel is getting at is that there is a source release and then that could be used to generate different possible source wheels each of which would correspond to a particular configuration of numpy. Each source wheel would correspond to one binary wheel and have all static metadata but there still needs to be a separate source release that is used to generate the different source wheels. The step that turns a source wheel into a binary wheel would be analogous to the ./configure step in a typical makefile project. ./configure is used to specify the options corresponding to all the different ways of compiling and installing the project. After running ./configure the command "make" is unparametrised and performs the actual compilation: this step is analogous to converting a source wheel to a binary wheel. I think this satisfies all of the requirements for static metadata and one-to-one correspondence of source wheels and binary wheels. If numpy followed this then I imagine that there would be a single source wheel on PyPI corresponding to the one configuration that would be used consistently there. However numpy still needs to separately release the code in a form that is also usable in all of the many other contexts that it is already used. IOW they will need to continue to issue source releases in more or less the same form as today. It makes sense for PyPI to host the source release archives on the project page even if pip will simply ignore them. -- Oscar From p.f.moore at gmail.com Thu Oct 8 13:46:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Oct 2015 12:46:30 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 11:18, Oscar Benjamin wrote: > On 7 October 2015 at 22:41, Paul Moore wrote: >> On 7 October 2015 at 22:28, Nathaniel Smith wrote: >>> Maybe I have misunderstood: does it actually help pip at all to have >>> static access to name and version, but not to anything else? I've been >>> assuming not, but I don't think anyone's pointed to any examples yet >>> of the problems that pip is encountering due to the lack of static >>> metadata -- would this actually be enough to solve them? >> >> The principle I am working on is that *all* metadata in a source wheel >> should be statically available - that's not just for pip, but for all >> other consumers, including distro packagers. What's not set in stone >> is precisely what (subsets of) metadata are appropriate for source >> wheels as opposed to (binary) wheels. > > A concrete example would be whether or not the numpy source wheel > depends on pyopenblas. Depending on how numpy is built the binary > wheel may or may not depend on pyopenblas. It doesn't make any sense > to say that the numpy source release depends on pyopenblas so what > should be the dependencies of the source wheel? Well, I said this previously but I don't have any objections to the idea that binary wheels have additional dependencies - so the source wheel doesn't depend on pyopenblas but the binary does. But as I understand it, this is currently theoretical - there isn't yet any pyopenblas validate these speculations against? I say this not because I think the approach is invalid, but because I think there are probably a lot of untested questions that need answering. Let's expand the scenario a bit. The user (presumably) still just says "python -m pip install numpy". What happens then? 1. Assume there's a binary wheel that's compatible with the user's platform. 1a. If there are multiple compatible binary wheels, pip chooses the "most compatible" so we're safe to assume there's only one. [1] 2. Looking at the dependencies, say it depends on pyopenblas. So pip needs to install pyopenblas. 2a. If there's a compatible wheel for pyopenblas, pip installs that too. 2b. If there's no compatible pyopenblas wheel, pip falls back to a source wheel, builds it, and uses that. If the build fails, the whole numpy install fails. 3. If there's no compatible numpy binary wheel, pip gets the source wheel and builds it. There's no user interaction possible here [2], so the build uses whatever defaults the numpy build process identifies as "most appropriate" for the user's platform. This may be simply a lowest common denominator, or it may do some form of introspection of the user's system to get the best possible build. Either way, a wheel is generated that's known to work on the user's system, so there should be no additional dependencies injected at this point, and pip will use that wheel directly. The only constraint here is that a binary numpy wheel built with the default options on a given machine from a numpy source wheel cannot have extra dependencies that aren't known to be already satisfied by the user's system, because by the time pip generates a wheel from the source wheel, it's finished doing dependency resolution so any new dependencies won't get checked. I don't see it as a problem for any hypothetical new build system to conform to this constraint - by default a built wheel must work on the system it's built on. All it means is that to build binaries with additional dependencies must be done manually, supplying options describing your intent. [1] Dependencies are *not* considered as part of the compatibility matching, so it's correct that this step happens before the dependency checks. Maybe you're assuming that if there are two wheels, one depending on pyopenblas and one not, then if the user doesn't have pyopenblas installed the wheel that doesn't depend on it will be used? But that's not how pip works. [2] When pip runs installs, it does so non-interactively. Whatever command pip uses to build a wheel ("python setup.py bdist_wheel" at the moment) must run without user interaction and produce a wheel that is compatible with the user's environment. So unless I'm mistaken about what you're saying, I don't see any issue here. Unless you're saying that you're not willing to work under some of the constraints I describe above - but in that case, you need pip's compatibility matching, dependency resolution, or automated wheel build processes to change. That's fine but to move the discussion forwards, we'd then need to understand (and agree with) whatever changes you need in pip. At the moment, I'm not aware that anyone has asked for substantive changes to pip's behaviour in these areas as part of this proposal. > One possibility which I think is what Nathaniel is getting at is that > there is a source release and then that could be used to generate > different possible source wheels each of which would correspond to a > particular configuration of numpy. Each source wheel would correspond > to one binary wheel and have all static metadata but there still needs > to be a separate source release that is used to generate the different > source wheels. That's possible, but what would these multiple source wheels be called? They couldn't all be called "numpy" as how would the user say which one they wanted? Pip can't decide. They can't be called numpy and distinguished by versions, as then how would you decide whether "numpy with openblas" is "newer" or "older" than "numpy with MKL"? That's the issue with Christoph Gohlke's current means of versioning his MKL builds. So you're looking at multiple PyPI projects, one for each "flavour" of numpy. Or you're looking at changes to how PyPI and pip define a "project". Neither of those options sound particularly straightforward to me. > The step that turns a source wheel into a binary wheel would be > analogous to the ./configure step in a typical makefile project. > ./configure is used to specify the options corresponding to all the > different ways of compiling and installing the project. After running > ./configure the command "make" is unparametrised and performs the > actual compilation: this step is analogous to converting a source > wheel to a binary wheel. But the Python (PyPI/pip) model is different from the autoconf "typical makefile project" model. There's no configure step. If you're proposing that we add one, then that's a pretty major change in structure and would have some fairly wide-ranging impacts (on PyPI and pip, and also on 3rd party projects like bandersnatch and devpi). I don't think we're even close to understanding how we'd manage such a change. > I think this satisfies all of the requirements for static metadata and > one-to-one correspondence of source wheels and binary wheels. If numpy > followed this then I imagine that there would be a single source wheel > on PyPI corresponding to the one configuration that would be used > consistently there. However numpy still needs to separately release > the code in a form that is also usable in all of the many other > contexts that it is already used. IOW they will need to continue to > issue source releases in more or less the same form as today. It makes > sense for PyPI to host the source release archives on the project page > even if pip will simply ignore them. So you're talking about numpy only supporting one configuration via PyPI, and expecting any other configurations to be made available only via other channels? I guess you could do that, but I hope you won't. It feels to me like giving up before we've properly tried to understand the issues. Paul From contact at ionelmc.ro Thu Oct 8 14:05:14 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 8 Oct 2015 15:05:14 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Thu, Oct 8, 2015 at 1:18 PM, Oscar Benjamin wrote: > I think this satisfies all of the requirements for static metadata and > one-to-one correspondence of source wheels and binary wheels. If numpy > followed this then I imagine that there would be a single source wheel > on PyPI corresponding to the one configuration that would be used > consistently there. However numpy still needs to separately release > the code in a form that is also usable in all of the many other > contexts that it is already used. > ?Can't that configuration just be the build defaults?? There would be a single source but with some preset build configuration. People with different needs can just override those. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Oct 8 14:39:11 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 13:39:11 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 12:46, Paul Moore wrote: > On 8 October 2015 at 11:18, Oscar Benjamin wrote: >> >> A concrete example would be whether or not the numpy source wheel >> depends on pyopenblas. Depending on how numpy is built the binary >> wheel may or may not depend on pyopenblas. It doesn't make any sense >> to say that the numpy source release depends on pyopenblas so what >> should be the dependencies of the source wheel? > > Well, I said this previously but I don't have any objections to the > idea that binary wheels have additional dependencies - so the source > wheel doesn't depend on pyopenblas but the binary does. Okay, I guess I'm confused by what you mean when you say that a source wheel (or sdist) should have a "one-to-one" correspondence with a binary wheel. > But as I understand it, this is currently theoretical - there isn't > yet any pyopenblas validate these speculations against? I don't think pyopenblas is ready yet but it is being developed with specifically this scenario in mind. > So unless I'm mistaken about what you're saying, I don't see any issue > here. Unless you're saying that you're not willing to work under some > of the constraints I describe above As an aside : I'm not a contributor to numpy. I just use it a lot and teach people how to use it (which is where the packaging problems come in). > - but in that case, you need pip's > compatibility matching, dependency resolution, or automated wheel > build processes to change. That's fine but to move the discussion > forwards, we'd then need to understand (and agree with) whatever > changes you need in pip. At the moment, I'm not aware that anyone has > asked for substantive changes to pip's behaviour in these areas as > part of this proposal. I don't think anyone is suggesting significant changes to pip's dependency resolution. Compatibility matching does need improvement IMO. Also the automated build process does need to be changed - specifically we need build-requires so that third party build tools can work. I didn't think improving the build process was controversial... >> I think this satisfies all of the requirements for static metadata and >> one-to-one correspondence of source wheels and binary wheels. If numpy >> followed this then I imagine that there would be a single source wheel >> on PyPI corresponding to the one configuration that would be used >> consistently there. However numpy still needs to separately release >> the code in a form that is also usable in all of the many other >> contexts that it is already used. IOW they will need to continue to >> issue source releases in more or less the same form as today. It makes >> sense for PyPI to host the source release archives on the project page >> even if pip will simply ignore them. > > So you're talking about numpy only supporting one configuration via > PyPI, and expecting any other configurations to be made available only > via other channels? I guess you could do that, but I hope you won't. > It feels to me like giving up before we've properly tried to > understand the issues. Okay so again I'm not a numpy dev. Numpy already supports being used in lots of setups that are not via pypi. Apart from Cristoph's builds you have all kinds of people building on all kinds of OSes and linking with different BLAS libraries in different ways. Some people will compile numpy statically with CPython. If you follow the discussions about numpy development it's clear that the numpy devs don't know all of the ways that numpy is built and used. Clearly pip/PyPI cannot be used to statically link numpy with CPython or for all of the different (often non-redistributable) BLAS libraries so numpy will support some setups that are not possible through pip. That's fine, I don't see the problem with that. At the moment an sdist is the same thing is a source release. If you propose to change it so that projects should upload source wheels and then make source wheels something tightly defined (e.g. a zip file containing exactly two directories setup to build for one particular configuration) then there needs to be a separate way to simply release the code in traditional format as is done now. -- Oscar From oscar.j.benjamin at gmail.com Thu Oct 8 14:47:28 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 13:47:28 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 13:05, Ionel Cristian M?rie? wrote: > On Thu, Oct 8, 2015 at 1:18 PM, Oscar Benjamin > wrote: >> >> I think this satisfies all of the requirements for static metadata and >> one-to-one correspondence of source wheels and binary wheels. If numpy >> followed this then I imagine that there would be a single source wheel >> on PyPI corresponding to the one configuration that would be used >> consistently there. However numpy still needs to separately release >> the code in a form that is also usable in all of the many other >> contexts that it is already used. > > Can't that configuration just be the build defaults? There would be a single > source but with some preset build configuration. People with different needs > can just override those. Yeah, I guess so. Maybe I'm just not understanding what the "one-to-one" correspondence is supposed to mean. Earlier in the thread it was said to be important because of wheel caching etc. but if it's possible to configure different builds then it's not really one-to-one. -- Oscar From donald at stufft.io Thu Oct 8 15:01:59 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Oct 2015 09:01:59 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 8, 2015 at 8:48:16 AM, Oscar Benjamin (oscar.j.benjamin at gmail.com) wrote: > On 8 October 2015 at 13:05, Ionel Cristian M?rie? wrote: > > On Thu, Oct 8, 2015 at 1:18 PM, Oscar Benjamin > > wrote: > >> > >> I think this satisfies all of the requirements for static metadata and > >> one-to-one correspondence of source wheels and binary wheels. If numpy > >> followed this then I imagine that there would be a single source wheel > >> on PyPI corresponding to the one configuration that would be used > >> consistently there. However numpy still needs to separately release > >> the code in a form that is also usable in all of the many other > >> contexts that it is already used. > > > > Can't that configuration just be the build defaults? There would be a single > > source but with some preset build configuration. People with different needs > > can just override those. > > Yeah, I guess so. Maybe I'm just not understanding what the > "one-to-one" correspondence is supposed to mean. Earlier in the thread > it was said to be important because of wheel caching etc. but if it's > possible to configure different builds then it's not really > one-to-one. > One of the features in the original PEP was the ability to produce multiple different Wheels from the same source release much like how Debian does. e.g. numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions that would differ from the name/version of the original sdist thing that was being proposed. That won't work with our packaging toolchain, the idea that their a singular name/version for one sdist (and then for the wheels that produces) is pretty heavily baked into the entire toolchain. That?s what I meant by wheels and sdists being 1:1. As far as static metadata goes, I think that one of my earlier messages tried to get across the idea that if there is a good reason for something to be dynamic then we can possibly do that, but that the current PEP went too far and made (well kept) *everything* dynamic. My point was that we should assume static for all metadata and then make exceptions for the data that we can't assume that for, but each case should be properly documented with motivations? or *why* that can't be static. That will give everyone else the ability to see the use case, and figure out if that?s a use case we want to support, if we like how the PEP is supporting it, or if there is possibly some other feature we could add instead that would still support that, while offering the static-ness that we desire. Or if not, at least we'll have it documented as to *why* it needs to be dynamic. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Thu Oct 8 15:20:41 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Oct 2015 14:20:41 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 13:39, Oscar Benjamin wrote: > On 8 October 2015 at 12:46, Paul Moore wrote: >> On 8 October 2015 at 11:18, Oscar Benjamin wrote: >>> >>> A concrete example would be whether or not the numpy source wheel >>> depends on pyopenblas. Depending on how numpy is built the binary >>> wheel may or may not depend on pyopenblas. It doesn't make any sense >>> to say that the numpy source release depends on pyopenblas so what >>> should be the dependencies of the source wheel? >> >> Well, I said this previously but I don't have any objections to the >> idea that binary wheels have additional dependencies - so the source >> wheel doesn't depend on pyopenblas but the binary does. > > Okay, I guess I'm confused by what you mean when you say that a source > wheel (or sdist) should have a "one-to-one" correspondence with a > binary wheel. OK, let me try to clarify: 1. The identifying data is name and version. There should only ever be *one* source wheel with a given name/version. 2. It's possible to have multiple binary wheels derived from a source wheel, but they must all have the same name/version as the source wheel, and only one will ever be considered as the "right" one for a given system. (I.e., compatibility matching takes place and a single best match is selected). The key things here is that binary wheels that differ only in compatibility tags are considered "the same" in this sense (think equivalence class, if you have a mathematical background). Things that may have muddied the water: 1. When I say "all metadata in a source wheel must be static". Technically, tools (pip) don't care about anything other than name and version. The rest is mostly for things like publishing on PyPI and distro packaging, which is not a "core" use. 2. The whole pyopenblas question, which as I understand it is part of the effort to work around the limitations of the compatibility tag mechanism by making binary library dependencies into package dependencies. That approach looks promising, but it moves some of the "what is the appropriate wheel for this environment" question away from compatibility tags and into dependency checking. Which is a non-trivial change to the model. I'd like to see the "binary dependency as package dependency" approach explored properly, but it's completely orthogonal to the question of how source wheels should work. You'd have the same issues with the current sdist format. I'd suggest that the work around this area be separated out from the revamped source distribution discussion. It could probably even be prototyped without getting involved with numpy - how about creating a test project that linked either statically or dynamically to a dummy external library, and then sorting out the wheel issues with that (project testproj depends on project testlib if it's built to link dynamically, and doesn't if it's built statically)? You could do that without anyone needing to understand about BLAS, or numpy, and get much better informed feedback from the wider distutils-sig than you currently get (because you keep needing to explain numpy build processes every time anyone asks a question). For the purposes of this discussion, however, I'd suggest that we assume that building a binary wheel from a source wheel *doesn't* add any new dependencies. Apart from the pyopenblas case, I don't know of any other case where that would happen. (If making that restriction makes the whole discussion trivial, then that's sort of my point - I don't think there *is* a big debate to be had outside of the pyopenblas question). Paul From contact at ionelmc.ro Thu Oct 8 15:34:08 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 8 Oct 2015 16:34:08 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Thu, Oct 8, 2015 at 4:01 PM, Donald Stufft wrote: > One of the features in the original PEP was the ability to produce multiple > different Wheels from the same source release much like how Debian does. > e.g. > numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and > numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions > that > would differ from the name/version of the original sdist thing that was > being > proposed. > ?Sorry if this sounds obtuse but isn't that useless overspecialization? They can just publish `numpy-mlk` and `numpy-?thatblas` or whatever on PyPI, and that will even work better when it comes to dependencies. I mean, if you build something for `numpy-mkl` then it wouldn't work on a `numpy-otherblas` anyway right? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Oct 8 15:51:23 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 14:51:23 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 14:34, Ionel Cristian M?rie? wrote: > > On Thu, Oct 8, 2015 at 4:01 PM, Donald Stufft wrote: >> >> One of the features in the original PEP was the ability to produce >> multiple >> different Wheels from the same source release much like how Debian does. >> e.g. >> numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and >> numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions >> that >> would differ from the name/version of the original sdist thing that was >> being >> proposed. > > > Sorry if this sounds obtuse but isn't that useless overspecialization? They > can just publish `numpy-mlk` and `numpy-thatblas` or whatever on PyPI, and > that will even work better when it comes to dependencies. > I mean, if you > build something for `numpy-mkl` then it wouldn't work on a `numpy-otherblas` > anyway right? It depends. If you're using numpy from pure Python code the difference between mkl and otherblas is probably irrelevant. So in most cases you'd want to be able to depend just on "numpy" but in some cases you'd need to be more specific. Perhaps you could solve that with "provides"... Really though it's probably best to keep the set of binaries on PyPI internally consistent and not try to represent everything. My point earlier was that regardless of what goes on PyPI as the official numpy wheel there will be many people using the numpy code in other ways. If pip is not the only consumer of a source release then it's not really reasonable to dictate (and redesign in a less human-friendly way) its layout purely for pip's benefit. -- Oscar From contact at ionelmc.ro Thu Oct 8 16:16:07 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 8 Oct 2015 17:16:07 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Thu, Oct 8, 2015 at 4:51 PM, Oscar Benjamin wrote: > It depends. If you're using numpy from pure Python code the difference > between mkl and otherblas is probably irrelevant. So in most cases > you'd want to be able to depend just on "numpy" but in some cases > you'd need to be more specific. Perhaps you could solve that with > "provides"... > > Really though it's probably best to keep the set of binaries on PyPI > internally consistent and not try to represent everything. My point > earlier was that regardless of what goes on PyPI as the official numpy > wheel there will be many people using the numpy code in other ways. If > pip is not the only consumer of a source release then it's not really > reasonable to dictate (and redesign in a less human-friendly way) its > layout purely for pip's benefit. > ?Yes indeed. But then shouldn't we talk about proper dependency resolution, compatible releases, meta packages and stuff like that? Unless I completely misunderstood the discussion here (quite probable :-) then this whole multiple source distributions idea is more like a workaround. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Oct 8 16:43:44 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Oct 2015 09:43:44 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 9:23 AM, "Ionel Cristian M?rie?" wrote: > > > On Thu, Oct 8, 2015 at 4:51 PM, Oscar Benjamin wrote: >> >> It depends. If you're using numpy from pure Python code the difference >> between mkl and otherblas is probably irrelevant. So in most cases >> you'd want to be able to depend just on "numpy" but in some cases >> you'd need to be more specific. Perhaps you could solve that with >> "provides"... >> >> Really though it's probably best to keep the set of binaries on PyPI >> internally consistent and not try to represent everything. My point >> earlier was that regardless of what goes on PyPI as the official numpy >> wheel there will be many people using the numpy code in other ways. If >> pip is not the only consumer of a source release then it's not really >> reasonable to dictate (and redesign in a less human-friendly way) its >> layout purely for pip's benefit. > > > ?Yes indeed. But then shouldn't we talk about proper dependency resolution, compatible releases, meta packages and stuff like that? Unless I completely misunderstood the discussion here (quite probable :-) then this whole multiple source distributions idea is more like a workaround. so, because install_requires and extras_require are computed at [egg-info] time, an sdist['s metadata] is/can/maybe technically different on different platforms, no? because of things like if sys.platform in [...]: INSTALL_REQUIRES.extend([...]) > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Oct 8 16:54:43 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Oct 2015 09:54:43 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 8:34 AM, "Ionel Cristian M?rie?" wrote: > > > On Thu, Oct 8, 2015 at 4:01 PM, Donald Stufft wrote: >> >> One of the features in the original PEP was the ability to produce multiple >> different Wheels from the same source release much like how Debian does. e.g. >> numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and >> numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions that >> would differ from the name/version of the original sdist thing that was being >> proposed. > > > ?Sorry if this sounds obtuse but isn't that useless overspecialization? They can just publish `numpy-mlk` and `numpy-?thatblas` or whatever on PyPI, and that will even work better when it comes to dependencies. I mean, if you build something for `numpy-mkl` then it wouldn't work on a `numpy-otherblas` anyway right? from a reproducibility standpoint, when I run e.g. pip freeze, how (without e.g. numpy-mkl, numpy-blas3k) do I reinstall the same package? * { } URI params, post-#frag=ment encoding? * {+1} prefix-[suffix] how do I install the same version of the package that [you had / this Jupyter notebook has (version_information, watermark)] on [your / their] machine? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 8 17:14:33 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Oct 2015 08:14:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > > Yeah, that's not a good long term solution -- it needs to be moved > > into the metadata (probably by creating an MKL wheel and then > > making > > the numpy wheel depend on it). That's exactly the problem :-) > > > > Are you available on IRC or for a video call or something? I feel like there's > something foundational from both sides that we're each missing here and it'd be > easier to just hash it out in real time rather than lobbying random emails > coming from places of confusion (at least on my side). > > I'm not sure if Paul (or anyone else!) would want to jump in on it too, though > I feel like probably if it's me and you then the two "sides" will probably be > reasonably well represented so if more folks don't want to join that's probably > OK too, particularly since we wouldn't be making any actual decisions there :D This does sound like it would be a good idea -- couldn't hurt, anyway :-). I'll contact you offlist. If anyone else wants to join in, email me... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Oct 8 17:50:42 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 8 Oct 2015 08:50:42 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: > > But the different builds for the different configurations end up with > different metadata. If I'm understanding right, the whole point of > "source wheels" is that they have all the static metadata that pip > needs in order to make decisions, and this has to match the resulting > wheels -- right? > I think we're largely talking about variances in "external" non-python system dependencies (and their build settings). PEP426 currently doesn't cover this in the core metadata, so as it stands, any 2.0 sdist couldn't exhaust these build variances in it's core metadata. There has been some discussion on how to represent external dependencies. In brief, I think the going idea is that it would be through extensions ( https://www.python.org/dev/peps/pep-0426/#metadata-extensions), not in the core python metadata, and the various groups (distro folks, science folks, etc..) would implement these themselves to fulfill their needs... Assuming they did implement such an extension, it would exist in the sdist, and for cases like numpy likely support some notion of "build options", and hence allow for a 1 to many mapping between sdist and binary wheels. Marcus > The way I'm imagining it is that there are multiple levels of metadata > staticness: > > package name, author, description, ... > static in: VCS checkouts, source releases, source wheels, wheels > package version > static in: source releases, source wheels, wheels > package dependencies > static in: source wheels, wheels > environment tag > static in: wheels > > > Of course, there *is* an unsolved issue here, which is how we manage > > compatibility for wheels at the level needed for numpy. But I thought > > the discussion on that was ongoing? I'm concerned that this proposal > > is actually about bypassing that discussion, and instead trying to > > treat incompatibly linked wheels as "different" in terms of project > > metadata, which I think is the wrong way of handling things. I note > > that Christoph Gohlke's numpy builds are tagged with a "+mkl" local > > version modifier - that's presumably intended to mark the fact that > > they are built with an incompatible runtime - but that's a misuse of > > local versions (and I've found it causes niggling issues with how pip > > recognises upgrades, etc). > > Yeah, that's not a good long term solution -- it needs to be moved > into the metadata (probably by creating an MKL wheel and then making > the numpy wheel depend on it). That's exactly the problem :-) > > > So, in summary: Your points above don't seem to me to in any way > > preclude having a single numpy source wheel, and a number of (mutually > > incompatible, but the same in terms of project and version) binary > > wheels. > > Maybe I have misunderstood: does it actually help pip at all to have > static access to name and version, but not to anything else? I've been > assuming not, but I don't think anyone's pointed to any examples yet > of the problems that pip is encountering due to the lack of static > metadata -- would this actually be enough to solve them? > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 8 18:33:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Oct 2015 09:33:47 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: > > On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > > > Yeah, that's not a good long term solution -- it needs to be moved > > > into the metadata (probably by creating an MKL wheel and then > > > making > > > the numpy wheel depend on it). That's exactly the problem :-) > > > > > > > Are you available on IRC or for a video call or something? I feel like there's > > something foundational from both sides that we're each missing here and it'd be > > easier to just hash it out in real time rather than lobbying random emails > > coming from places of confusion (at least on my side). > > > > I'm not sure if Paul (or anyone else!) would want to jump in on it too, though > > I feel like probably if it's me and you then the two "sides" will probably be > > reasonably well represented so if more folks don't want to join that's probably > > OK too, particularly since we wouldn't be making any actual decisions there :D > > This does sound like it would be a good idea -- couldn't hurt, anyway :-). I'll contact you offlist. If anyone else wants to join in, email me... Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / 2 pm New York / 7 pm London. Since there's been at least some interest, we'll do it as a Google hangout and send a link around in case anyone wants to listen in, and of course summarize back to the list. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 8 21:58:38 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Oct 2015 12:58:38 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 09:33, "Nathaniel Smith" wrote: > > On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: > > > > On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > > > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > > > > Yeah, that's not a good long term solution -- it needs to be moved > > > > into the metadata (probably by creating an MKL wheel and then > > > > making > > > > the numpy wheel depend on it). That's exactly the problem :-) > > > > > > > > > > Are you available on IRC or for a video call or something? I feel like there's > > > something foundational from both sides that we're each missing here and it'd be > > > easier to just hash it out in real time rather than lobbying random emails > > > coming from places of confusion (at least on my side). > > > > > > I'm not sure if Paul (or anyone else!) would want to jump in on it too, though > > > I feel like probably if it's me and you then the two "sides" will probably be > > > reasonably well represented so if more folks don't want to join that's probably > > > OK too, particularly since we wouldn't be making any actual decisions there :D > > > > This does sound like it would be a good idea -- couldn't hurt, anyway :-). I'll contact you offlist. If anyone else wants to join in, email me... > > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / 2 pm New York / 7 pm London. Since there's been at least some interest, we'll do it as a Google hangout and send a link around in case anyone wants to listen in, and of course summarize back to the list. Correction: happening *Monday Oct 12* at 11 am California / 2 pm New York / 7 pm London. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Thu Oct 8 22:05:16 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 8 Oct 2015 22:05:16 +0200 Subject: [Distutils] sampleproject: use tox? Message-ID: <5616CC7C.4060601@thomas-guettler.de> This is a follow up to the thread "Where should I put tests when packaging python modules?" I have never used tox up to now. But reading the mails of the thread, it seems that tox is the current state of the art. This leads me to the conclusion that the sample project should use tox. Any reasons not to do it? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From chris.jerdonek at gmail.com Thu Oct 8 22:40:36 2015 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 8 Oct 2015 13:40:36 -0700 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <5616CC7C.4060601@thomas-guettler.de> References: <5616CC7C.4060601@thomas-guettler.de> Message-ID: On Thursday, October 8, 2015, Thomas G?ttler wrote: > This is a follow up to the thread "Where should I put tests when packaging > python modules?" > > I have never used tox up to now. But reading the mails of the thread, it > seems > that tox is the current state of the art. > > This leads me to the conclusion that the sample project should use tox. > > Any reasons not to do it? In my experience, tox is most useful when developing something that needs to work (and be tested) against multiple Python versions, environments, etc. For projects where the environment is fixed (e.g. an internal application), this isn't necessarily needed. So tox may be an unnecessary addition in those cases. --Chris > > Regards, > Thomas G?ttler > > > > > -- > http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Oct 8 23:33:06 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2015 23:33:06 +0200 Subject: [Distutils] sampleproject: use tox? References: <5616CC7C.4060601@thomas-guettler.de> Message-ID: <20151008233306.279700b6@fsol> On Thu, 8 Oct 2015 22:05:16 +0200 Thomas G?ttler wrote: > This is a follow up to the thread "Where should I put tests when packaging python modules?" > > I have never used tox up to now. But reading the mails of the thread, it seems > that tox is the current state of the art. I don't see why that would be the case. tox is certainly "the state of the art" in certain circles, but many projects don't mandate it at all or even don't use it at all and rely on other mechanisms (e.g. CI with a rich configuration matrix). Regards Antoine. From robertc at robertcollins.net Fri Oct 9 06:29:58 2015 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 9 Oct 2015 17:29:58 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: Toss me an invite too - I'm very interested in this, for all that I havent' kibbitzed on the thread yet :) On 9 October 2015 at 05:33, Nathaniel Smith wrote: > On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: >> >> On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: >> > >> > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: >> > > > Yeah, that's not a good long term solution -- it needs to be moved >> > > into the metadata (probably by creating an MKL wheel and then >> > > making >> > > the numpy wheel depend on it). That's exactly the problem :-) >> > > >> > >> > Are you available on IRC or for a video call or something? I feel like >> > there's >> > something foundational from both sides that we're each missing here and >> > it'd be >> > easier to just hash it out in real time rather than lobbying random >> > emails >> > coming from places of confusion (at least on my side). >> > >> > I'm not sure if Paul (or anyone else!) would want to jump in on it too, >> > though >> > I feel like probably if it's me and you then the two "sides" will >> > probably be >> > reasonably well represented so if more folks don't want to join that's >> > probably >> > OK too, particularly since we wouldn't be making any actual decisions >> > there :D >> >> This does sound like it would be a good idea -- couldn't hurt, anyway :-). >> I'll contact you offlist. If anyone else wants to join in, email me... > > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / 2 pm > New York / 7 pm London. Since there's been at least some interest, we'll do > it as a Google hangout and send a link around in case anyone wants to listen > in, and of course summarize back to the list. > > -n > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Robert Collins Distinguished Technologist HP Converged Cloud From opensource at ronnypfannschmidt.de Fri Oct 9 08:57:32 2015 From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Fri, 09 Oct 2015 08:57:32 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: I'd also like to join Am 9. Oktober 2015 06:29:58 MESZ, schrieb Robert Collins : >Toss me an invite too - I'm very interested in this, for all that I >havent' kibbitzed on the thread yet :) > >On 9 October 2015 at 05:33, Nathaniel Smith wrote: >> On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: >>> >>> On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: >>> > >>> > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) >wrote: >>> > > > Yeah, that's not a good long term solution -- it needs to be >moved >>> > > into the metadata (probably by creating an MKL wheel and then >>> > > making >>> > > the numpy wheel depend on it). That's exactly the problem :-) >>> > > >>> > >>> > Are you available on IRC or for a video call or something? I feel >like >>> > there's >>> > something foundational from both sides that we're each missing >here and >>> > it'd be >>> > easier to just hash it out in real time rather than lobbying >random >>> > emails >>> > coming from places of confusion (at least on my side). >>> > >>> > I'm not sure if Paul (or anyone else!) would want to jump in on it >too, >>> > though >>> > I feel like probably if it's me and you then the two "sides" will >>> > probably be >>> > reasonably well represented so if more folks don't want to join >that's >>> > probably >>> > OK too, particularly since we wouldn't be making any actual >decisions >>> > there :D >>> >>> This does sound like it would be a good idea -- couldn't hurt, >anyway :-). >>> I'll contact you offlist. If anyone else wants to join in, email >me... >> >> Looks like this is happening tomorrow (Fri Oct 9) at 11 am California >/ 2 pm >> New York / 7 pm London. Since there's been at least some interest, >we'll do >> it as a Google hangout and send a link around in case anyone wants to >listen >> in, and of course summarize back to the list. >> >> -n >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> MFG Ronny From guettliml at thomas-guettler.de Fri Oct 9 11:32:03 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 9 Oct 2015 11:32:03 +0200 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <20151008233306.279700b6@fsol> References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> Message-ID: <56178993.7070802@thomas-guettler.de> Am 08.10.2015 um 23:33 schrieb Antoine Pitrou: > On Thu, 8 Oct 2015 22:05:16 +0200 > Thomas G?ttler wrote: >> This is a follow up to the thread "Where should I put tests when packaging python modules?" >> >> I have never used tox up to now. But reading the mails of the thread, it seems >> that tox is the current state of the art. > > I don't see why that would be the case. tox is certainly "the state of > the art" in certain circles, but many projects don't mandate it at all > or even don't use it at all and rely on other mechanisms (e.g. CI with > a rich configuration matrix). My question was: Any reasons not to use tox in the sample project? I read your answer like this: Many projects don't mandate it at all or even don't use it at all and rely on other mechanisms Yes, it is correct that many projects is a different way. I search a simple guideline for new comers to set up a python project including testing. "Since many projects don't mandate it at all ..." I don't know if this a good argument against it. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From p.f.moore at gmail.com Fri Oct 9 11:43:52 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 10:43:52 +0100 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <56178993.7070802@thomas-guettler.de> References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On 9 October 2015 at 10:32, Thomas G?ttler wrote: > My question was: Any reasons not to use tox in the sample project? 1. Because the sample project is intended to explain how to *package* a project, not how to test it, or document it, or any of a number of other non-packaging things. 2. Because tox is not universally agreed as best practice, and the sample project is intended to demonstrate agreed best practice. > I search a simple guideline for new comers to set up > a python project including testing. The sample project is not intended for that purpose. It's a good starting point for something like that, but that's not its purpose. Paul PS I sympathise with your wish for a "do this to set up your first project" beginner's guide. I've wanted one of those myself for a long time now. But I don't believe such a thing exists, unfortunately. From contact at ionelmc.ro Fri Oct 9 12:11:18 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Fri, 9 Oct 2015 13:11:18 +0300 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore wrote: > > The sample project is not intended for that purpose. It's a good > starting point for something like that, but that's not its purpose. > > Paul > > PS I sympathise with your wish for a "do this to set up your first > project" beginner's guide. I've wanted one of those myself for a long > time now. But I don't believe such a thing exists, unfortunately. ?Should we at least give some pointers as to what people should look at for a fully fledged project? Like a lists with examples/templates covering all the variants. Eg: "PyPA's sample projects only covers the packaging aspect of a project. For more exhaustive examples take a look at cookiecutter templates, barry's sample, ogrisel's appveyor demo, this and that etc etc" How about something like that? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 9 12:23:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 11:23:25 +0100 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: Well, https://packaging.python.org/en/latest/distributing/ already says """ The section does not aim to cover best practices for Python project development as a whole. For example, it does not provide guidance or tool recommendations for version control, documentation, or testing """ It may be good to add a footnote with your "For more exhaustive examples..." comment - but it would need links, and good luck on getting any sort of agreement on what constitutes good examples :-) Paul On 9 October 2015 at 11:11, Ionel Cristian M?rie? wrote: > > On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore wrote: >> >> >> The sample project is not intended for that purpose. It's a good >> starting point for something like that, but that's not its purpose. >> >> Paul >> >> PS I sympathise with your wish for a "do this to set up your first >> project" beginner's guide. I've wanted one of those myself for a long >> time now. But I don't believe such a thing exists, unfortunately. > > > Should we at least give some pointers as to what people should look at for a > fully fledged project? Like a lists with examples/templates covering all the > variants. Eg: > > "PyPA's sample projects only covers the packaging aspect of a project. For > more exhaustive examples take a look at cookiecutter templates, barry's > sample, ogrisel's appveyor demo, this and that etc etc" > > How about something like that? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro From dw+distutils-sig at hmmz.org Wed Oct 7 01:14:24 2015 From: dw+distutils-sig at hmmz.org (David Wilson) Date: Tue, 6 Oct 2015 23:14:24 +0000 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006095101.031c8a56@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006231424.GA30512@k3> On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: > They should be inside the module. That way, you can check an installed > module is ok by running e.g. "python -m mypackage.tests". Any other > choice makes testing installed modules more cumbersome. As Donald mentioned, this doesn't work in the general case since many packages ship quite substantial test data around that often doesn't end up installed, and in other cases since the package requires significant fixture setup or external resources (e.g. running SQLAlchemy tests without a working database server would be meaningless). The option of always shipping test data as a standard part of a package in a vein attempt to always ensure it can be tested (which is not always likely given the SQLAlchemy example above) strikes me as incredibly wasteful, not from some oh-precious-bytes standpoint, but from the perspective of distributing a Python application of any size where the effects of always shipping half-configured test suites has increased the resulting distribution size potentially by 3 or 4x. https://github.com/bennoleslie/pexif is the first hit on Google for a module I thought would need some test data. It's actually quite minimally tested, yet already the tests + data are 3.6x the size of the module itself. I appreciate arguments for inlining tests alongside a package in order to allow reuse of the suite's functionality by consuming applications' test suites, but as above, in the general case this simply isn't something that will always work and can be relied on by default. Is there perhaps a third option that was absent from the original post? e.g. organizing tests in a separate, optional, potentially pip-installable package. > > > outside the module like this: > > > > https://github.com/pypa/sampleproject/tree/master/tests > > There is no actual reason to do that except win a couple kilobytes if > you are distributing your package on floppy disks for consumption on > Z80-based machines with 64KB RAM. > > Even Python *itself* puts its test suite inside the standard library, > not outside it (though some Linux distros may strip it away). > Try "python -m test.regrtest" (again, this may fail if your distro > decided to ship the test suite in a separate package). > > The PyP"A" should definitely fix its sample project to reflect good > practices. > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From chris.barker at noaa.gov Fri Oct 9 18:05:31 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 09:05:31 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: How many people can join a hangout? we may be bumping up against that limit :-) -CHB On Thu, Oct 8, 2015 at 12:58 PM, Nathaniel Smith wrote: > On Oct 8, 2015 09:33, "Nathaniel Smith" wrote: > > > > On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: > > > > > > On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > > > > > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) > wrote: > > > > > > Yeah, that's not a good long term solution -- it needs to be > moved > > > > > into the metadata (probably by creating an MKL wheel and then > > > > > making > > > > > the numpy wheel depend on it). That's exactly the problem :-) > > > > > > > > > > > > > Are you available on IRC or for a video call or something? I feel > like there's > > > > something foundational from both sides that we're each missing here > and it'd be > > > > easier to just hash it out in real time rather than lobbying random > emails > > > > coming from places of confusion (at least on my side). > > > > > > > > I'm not sure if Paul (or anyone else!) would want to jump in on it > too, though > > > > I feel like probably if it's me and you then the two "sides" will > probably be > > > > reasonably well represented so if more folks don't want to join > that's probably > > > > OK too, particularly since we wouldn't be making any actual > decisions there :D > > > > > > This does sound like it would be a good idea -- couldn't hurt, anyway > :-). I'll contact you offlist. If anyone else wants to join in, email me... > > > > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / > 2 pm New York / 7 pm London. Since there's been at least some interest, > we'll do it as a Google hangout and send a link around in case anyone wants > to listen in, and of course summarize back to the list. > > Correction: happening *Monday Oct 12* at 11 am California / 2 pm New York > / 7 pm London. > > -n > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Fri Oct 9 18:16:13 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Fri, 9 Oct 2015 19:16:13 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Fri, Oct 9, 2015 at 7:05 PM, Chris Barker wrote: > How many people can join a hangout? we may be bumping up against that > limit :-) ?AFAIK there's no limit on the number of people that can listen in. Also, Hangouts can record video (the "On Air" thing).? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 19:04:00 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 10:04:00 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: I"ve been following this thread and gotten a bit lost. But I do care, because I'm a heavy numpy user, and also because I was involved for years in building pacakges for OS-X, and currently need distribute some of my own stuff that has semi-ugly C lib dependencies (Not as ugly as BLAS.,though :-) ) In the last couple years I gave up on PyPi and pip (and went with Anaconda and conda)-- that ecosystem simply doesn't currently support my use-cases. But it would be nice to go back, and it looks like there are some idea on the table that will make that possible. But could someone clarify a thing or two for me?? 1) what in the world is a "source wheel"? And how is it different than an sdist (other than maybe in a different file format. 2) Is it indeed "OK" with the current PEPs and tools for different binary wheels to have different dependencies? This would be the example of, for instance the Matplotlib binary wheel for Windows depends on a py_zlib, whereas the binary wheel for OS-X relies on the the system lib, and therefor does not have that dependency? (and has anyone worked out the linking issues so that that would all work with virtualenv and friends...) if (2) then it seems the issue is what to do with the same package on the same platform having potentially different dependencies -- i.e. numpy w/ mkl and numpy w/some_other_blas. In that case, I think that this could completely explode into m**n possible wheels if we try to accommodate it in a fully flexible manner -- so making it a name-thing (like was proposed here), makes sense to me -- numpy_mkl and numpy_openblas are, as far as pip is concerned, different packages. I think this is OK, as we probably only want some small subset of possible build up on PyPi anyway (only one?). But it does get a bit tricky if you want to put up a package that depends on "pure" numpy -- i.e. it doesn't care which BLAS the numpy it uses, but it DOES need numpy. This could this be Accomidated byt have an "or" option for dependencies: numpy_mkl>=1.9 or numpy_openblas >= 1.9 However, that would mean that the author of that wheel would need to know all of the wheels that might be available up front - less than ideal. Which seems to point more to having an optional "binary_build" component to the name. Not sure what syntax is available, but the idea would be that: "numpy|mkl" and "numpy|openblas" would both match "numpy" which, of course would require changes to the whole stack... and off the top of my head, I'm wondering whether having one "binary_build' flag would be enough, or if we'd find that we wanted n options, and n*m combinations, and this would all blow up. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 9 19:18:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 18:18:19 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 9 October 2015 at 18:04, Chris Barker wrote: > 1) what in the world is a "source wheel"? And how is it different than an > sdist (other than maybe in a different file format. A "source wheel" is the proposed name for a to-be-defined replacement for sdists. For now, you can think of "source wheel" and "sdist" as the same. > 2) Is it indeed "OK" with the current PEPs and tools for different binary > wheels to have different dependencies? This would be the example of, for > instance the Matplotlib binary wheel for Windows depends on a py_zlib, > whereas the binary wheel for OS-X relies on the the system lib, and therefor > does not have that dependency? > (and has anyone worked out the linking issues so that that would all work > with virtualenv and friends...) It's not *currently* OK for different binary wheels to have different dependencies. At least I don't think it is. It's basically not something that as far as I'm aware anyone has ever considered an option up till now, and so it's quite likely that there are assumptions baked into the tools that would break if different builds of (a given version of) a package had different dependencies. One of the proposed approaches to binary dependencies (the one I've been referring to as "the pyopenblas approach") is based on the idea that different wheels could have different dependency metadata. I've tried to enumerate the questions that need to be looked at if we were to go down that route, but to my knowledge, no-one has yet either tested how well things would work if this happened, or audited the code for problematic assumptions. So it's a possibility for the future, but like many things it depends on someone doing the work to make it happen (and as far as I know it's mostly the numpy community who have the need, so I assume someone in that community would need to move this forward). Paul From p.f.moore at gmail.com Fri Oct 9 19:28:09 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 18:28:09 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 9 October 2015 at 18:04, Chris Barker wrote: > Which seems to point more to having an optional "binary_build" component to > the name. Not sure what syntax is available, but the idea would be that: > > "numpy|mkl" > > and > > "numpy|openblas" > > would both match "numpy" > > which, of course would require changes to the whole stack... This sounds more like it's an extension of the wheel "compatibility tag" approach. That's another option I can imagine being worth looking at. Basically, wheels currently encode in their names the Python version, ABI and architecture(s) they work on. That's sufficient for simple uses, but for more complicated scenarios you need more (such as, "this wheel is only valid if library FOO is present). It's not impossible to extend the compatibility tag mechanism to include such things, but there are a number of issues that would need to be thrashed out (spot the common theme here? :-)) Specifically, it's horribly cumbersome to encode everything in the filename, so some more scalable mechanism is needed. Also, you need a notion of compatibility order (is wheel X "more compatible" with your system than wheel Y?). And of course, compatibility is about "does it work with what's there?" which doesn't allow for the possibility of downloading and installing a dependency (that's what dependency resolution does, compatibility checking is at a later stage in the process). So again this is a possible solution, but needs someone to work on the details. ... or thinking again, maybe you mean having multiple packages (numpy_mkl, numpy_openblas, ...) all of which satisfy a "numpy" requirement? That's definitely viable, the Metadata 2.0 spec allows for one package "providing" an implementation of another (https://www.python.org/dev/peps/pep-0426/#provides). But that part of Metadata 2.0 isn't implemented yet - it needs some people with time to work on it again, and it will probably be implemented alongside "sdist 2.0", which is what we've been calling "when we get round to reworking/rethinking the sdist format". Paul Paul From guettliml at thomas-guettler.de Fri Oct 9 19:44:23 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 9 Oct 2015 19:44:23 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006231424.GA30512@k3> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> Message-ID: <5617FCF7.9000504@thomas-guettler.de> Am 07.10.2015 um 01:14 schrieb David Wilson: > On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: > >> They should be inside the module. That way, you can check an installed >> module is ok by running e.g. "python -m mypackage.tests". Any other >> choice makes testing installed modules more cumbersome. > > As Donald mentioned, this doesn't work in the general case since many > packages ship quite substantial test data around that often doesn't end > up installed, and in other cases since the package requires significant > fixture setup or external resources (e.g. running SQLAlchemy tests > without a working database server would be meaningless). Should work with a temporary sqlite db. > The option of always shipping test data as a standard part of a package > in a vein attempt to always ensure it can be tested (which is not always > likely given the SQLAlchemy example above) strikes me as incredibly > wasteful, not from some oh-precious-bytes standpoint, but from the > perspective of distributing a Python application of any size where the > effects of always shipping half-configured test suites has increased the > resulting distribution size potentially by 3 or 4x. > > https://github.com/bennoleslie/pexif is the first hit on Google for a > module I thought would need some test data. It's actually quite > minimally tested, yet already the tests + data are 3.6x the size of the > module itself. > > > I appreciate arguments for inlining tests alongside a package in order > to allow reuse of the suite's functionality by consuming applications' > test suites, but as above, in the general case this simply isn't > something that will always work and can be relied on by default. > > > Is there perhaps a third option that was absent from the original post? > e.g. organizing tests in a separate, optional, potentially > pip-installable package. Yes, this third way is plausible. I guess there is even a fourth way. The question remains: If a new comer asks you "How to package my python code and its tests?", there should be one default answer which works for 80% of all cases. I think the confusion gets worse by creating new public accessible repos which explain "Hey that's my way to package stupid simple python code". Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From brett at python.org Fri Oct 9 19:51:45 2015 From: brett at python.org (Brett Cannon) Date: Fri, 09 Oct 2015 17:51:45 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: Listening in to an On Air session is unbounded, but direct participants are capped at 10. On Fri, 9 Oct 2015 at 09:17 Ionel Cristian M?rie? wrote: > > On Fri, Oct 9, 2015 at 7:05 PM, Chris Barker > wrote: > >> How many people can join a hangout? we may be bumping up against that >> limit :-) > > > ?AFAIK there's no limit on the number of people that can listen in. Also, > Hangouts can record video (the "On Air" thing).? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Oct 9 20:00:57 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 9 Oct 2015 12:00:57 -0600 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: <561800D9.8050503@oddbird.net> On 10/09/2015 11:18 AM, Paul Moore wrote: > On 9 October 2015 at 18:04, Chris Barker wrote: >> 1) what in the world is a "source wheel"? And how is it different than an >> sdist (other than maybe in a different file format. > > A "source wheel" is the proposed name for a to-be-defined replacement > for sdists. For now, you can think of "source wheel" and "sdist" as > the same. > >> 2) Is it indeed "OK" with the current PEPs and tools for different binary >> wheels to have different dependencies? This would be the example of, for >> instance the Matplotlib binary wheel for Windows depends on a py_zlib, >> whereas the binary wheel for OS-X relies on the the system lib, and therefor >> does not have that dependency? >> (and has anyone worked out the linking issues so that that would all work >> with virtualenv and friends...) > > It's not *currently* OK for different binary wheels to have different > dependencies. At least I don't think it is. It's basically not > something that as far as I'm aware anyone has ever considered an > option up till now, and so it's quite likely that there are > assumptions baked into the tools that would break if different builds > of (a given version of) a package had different dependencies. AFAIK this is actually just fine currently, it's just not considered ideal for a hopeful future static-metadata world. Today, in the all-metadata-is-dynamic world that we actually live in, the tools aren't able to make any assumptions at all, and just have to download whatever best-match wheel or sdist they find for a given package/version requirement, unpack it, and see what it says about its own metadata (including dependencies). So different binary wheels of the same package at the same version having different dependencies works just fine. It's not like any of the tooling would actually know the difference, since they will only deal with one of those wheels at a given time. But of course this dynamic-metadata world prevents the tools from doing all kinds of useful things (like actually building dependency graphs in advance so they can do proper dependency conflict resolution), which is why we want to move towards static metadata. And it's in that transition that issues like "are different wheels for the same project at the same version allowed to have different dependencies" become relevant. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Fri Oct 9 20:28:28 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 09 Oct 2015 18:28:28 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <561800D9.8050503@oddbird.net> References: <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> <561800D9.8050503@oddbird.net> Message-ID: On Fri, 9 Oct 2015 19:01 Carl Meyer wrote: On 10/09/2015 11:18 AM, Paul Moore wrote: > On 9 October 2015 at 18:04, Chris Barker wrote: >> 1) what in the world is a "source wheel"? And how is it different than an >> sdist (other than maybe in a different file format. > > A "source wheel" is the proposed name for a to-be-defined replacement > for sdists. For now, you can think of "source wheel" and "sdist" as > the same. > >> 2) Is it indeed "OK" with the current PEPs and tools for different binary >> wheels to have different dependencies? This would be the example of, for >> instance the Matplotlib binary wheel for Windows depends on a py_zlib, >> whereas the binary wheel for OS-X relies on the the system lib, and therefor >> does not have that dependency? >> (and has anyone worked out the linking issues so that that would all work >> with virtualenv and friends...) > > It's not *currently* OK for different binary wheels to have different > dependencies. At least I don't think it is. It's basically not > something that as far as I'm aware anyone has ever considered an > option up till now, and so it's quite likely that there are > assumptions baked into the tools that would break if different builds > of (a given version of) a package had different dependencies. AFAIK this is actually just fine currently, it's just not considered ideal for a hopeful future static-metadata world. Why would it need dynamic metadata for the windows matplotlib wheel to have different metadata from the OSX matplotlib wheel? The platform Windows/OSX is static and each wheel declares its own dependencies statically but differently. Am I missing something? -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 20:29:53 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:29:53 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <5617FCF7.9000504@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <5617FCF7.9000504@thomas-guettler.de> Message-ID: On Fri, Oct 9, 2015 at 10:44 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > The question remains: If a new comer asks you "How to package my > python code and its tests?", there should be one default answer > which works for 80% of all cases. > Should be, maybe -- but clearly there is no consensus as to what the "one" answer shoudl be. But I _think_ there are two answers: 1) inside the package, so it can be installed and the tests run on the installed version. 2) outside the package, so that potentially large or complex test requirements are not required for installation. So the intro docs can lay out those two options, with a bit of text that helps a newbie make a decision. from above, -- I think that EITHER option would work fine for 80% of cases. which makes me think -- why not pick one as the default while clearly documenting the other. And despite the fact that I have always used option (2) in my work, I think, if there is going to be one in the simple example, it should be (1) -- it's a fine way to get started, and users can move their tests outside of the package later if they start getting big. As a package grows to that extent, it will likely need other structural changes anyway. -Chris > I think the confusion gets worse by creating new public accessible > repos which explain "Hey that's my way to package stupid simple > python code". > > Regards, > Thomas G?ttler > > > > > > -- > http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Oct 9 20:35:19 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 9 Oct 2015 12:35:19 -0600 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <561800D9.8050503@oddbird.net> Message-ID: <561808E7.2020505@oddbird.net> On 10/09/2015 12:28 PM, Oscar Benjamin wrote: > Why would it need dynamic metadata for the windows matplotlib wheel to > have different metadata from the OSX matplotlib wheel? The platform > Windows/OSX is static and each wheel declares its own dependencies > statically but differently. Am I missing something? I didn't say that required dynamic metadata (wheel metadata is already static). I just said that it works fine currently, and that it becomes an open question with the move towards static metadata in both source and binary releases, because we have to answer questions like "what information beyond just package/version makes up a complete node in a dependency graph." Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Fri Oct 9 20:44:49 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 09 Oct 2015 18:44:49 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <561808E7.2020505@oddbird.net> References: <561800D9.8050503@oddbird.net> <561808E7.2020505@oddbird.net> Message-ID: On Fri, 9 Oct 2015 19:35 Carl Meyer wrote: On 10/09/2015 12:28 PM, Oscar Benjamin wrote: > Why would it need dynamic metadata for the windows matplotlib wheel to > have different metadata from the OSX matplotlib wheel? The platform > Windows/OSX is static and each wheel declares its own dependencies > statically but differently. Am I missing something? I didn't say that required dynamic metadata (wheel metadata is already static). I just said that it works fine currently, and that it becomes an open question with the move towards static metadata in both source and binary releases, because we have to answer questions like "what information beyond just package/version makes up a complete node in a dependency graph." Assuming it's tied to the operating system it doesn't matter surely. When pip runs on Windows it can ignore dependencies that apply to other platforms so I don't see how this case makes it more complex. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Oct 9 20:48:05 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 9 Oct 2015 12:48:05 -0600 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <561800D9.8050503@oddbird.net> <561808E7.2020505@oddbird.net> Message-ID: <56180BE5.7080708@oddbird.net> On 10/09/2015 12:44 PM, Oscar Benjamin wrote: > > On Fri, 9 Oct 2015 19:35 Carl Meyer > wrote: > > On 10/09/2015 12:28 PM, Oscar Benjamin wrote: > > Why would it need dynamic metadata for the windows matplotlib wheel to > > have different metadata from the OSX matplotlib wheel? The platform > > Windows/OSX is static and each wheel declares its own dependencies > > statically but differently. Am I missing something? > > I didn't say that required dynamic metadata (wheel metadata is already > static). I just said that it works fine currently, and that it becomes > an open question with the move towards static metadata in both source > and binary releases, because we have to answer questions like "what > information beyond just package/version makes up a complete node in a > dependency graph." > > > > Assuming it's tied to the operating system it doesn't matter surely. > When pip runs on Windows it can ignore dependencies that apply to other > platforms so I don't see how this case makes it more complex. Sure. "Assuming it's tied to the operating system" is an assumption that can't be made here, though, if I understand the examples that have already been given repeatedly regarding numpy and BLAS-linking. If "OS" were actually sufficient to distinguish all cases, then the existing wheel platform tags would already be an adequate solution to this problem. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From chris.barker at noaa.gov Fri Oct 9 20:47:28 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:47:28 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Fri, Oct 9, 2015 at 10:28 AM, Paul Moore wrote: > ... or thinking again, maybe you mean having multiple packages > (numpy_mkl, numpy_openblas, ...) all of which satisfy a "numpy" > requirement? yes, that is EXACTLY what I meant. the idea here is that if you build a package that requires both numpy with a particular BLAS, then you'd do: eg: numpy|mkl >= 1.9 but if you build one that only required numpy, and doesn't care which BLAS it's using, then you simply do: numpy >= 1.9 So pip, when asked to install said package, would look and see if any package called numpy|* was installed (the ther right version). if so, it would move along. If not, then it would go look on PyPi for "numpy" -- and this is where it gets tricky -- which numpy| should it use? At this stage, it wouldn't matter, anyone would do. But say it installed numpy|mkl. Then next the user goes to install a package that depends on numpy|openblas -- now pip goes and looks, and find numpy|mkl, but not numpy|openblas -- so it need to go install numpy|openblas. which overwrites numpy|mkl, which is still OK. Until the user goes to install somethign that DOES depend on numpy|mkl. No we are stuck. But is this any different that two packages that depend on two different specific versions of the same package? But all this is making me think that the way this could be handled is by numpy NOT building a BLAS into the package at all. But rather, having the package depend on another package that provides the BLAS, so: numpy on PyPi would depend on this theoretical py_openblas any package that depends on only numpy would be easy. any package that depends on openblas would then depend on py_openblas. So the question is: if I compile my third party package against numpy and mkl, it would depend on numpy and py_mkl. If I went to installed this on a system that had a numpy that depends on openblas, pip would install py_mkl (having already installed numpy and py_openblas). And I'd have a numpy extension calling into a different BLAS than numpy itself is calling into -- would that cause any problems? Python would be linked to two libs with the same names -- would that cause conflict? I'm way out of my depth here! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 20:49:33 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:49:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> <561800D9.8050503@oddbird.net> Message-ID: On Fri, Oct 9, 2015 at 11:28 AM, Oscar Benjamin wrote: > Why would it need dynamic metadata for the windows matplotlib wheel to > have different metadata from the OSX matplotlib wheel? The platform > Windows/OSX is static and each wheel declares its own dependencies > statically but differently. Am I missing something? > I think the metadata can be static for the binary wheels, yes. but the dependencies would be different for the two wheels. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 20:53:08 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:53:08 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <561800D9.8050503@oddbird.net> <561808E7.2020505@oddbird.net> Message-ID: On Fri, Oct 9, 2015 at 11:44 AM, Oscar Benjamin wrote: > Assuming it's tied to the operating system it doesn't matter surely. When > pip runs on Windows it can ignore dependencies that apply to other > platforms so I don't see how this case makes it more complex. > does pip currently support platfrom-specific dependencies? That would solve part of the problem. but yes, there are dependencies that are a function of how the wheel was built, beyond what platform it is on. So those could be static in binary wheel, but not static in the sdist (or source wheel, or...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Fri Oct 9 21:20:44 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 9 Oct 2015 15:20:44 -0400 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <5614AFF2.1090604@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 2:11 AM, Chris Barker wrote: > On Tue, Oct 6, 2015 at 10:38 PM, Thomas G?ttler > wrote: >> >> Yes, there is not generic "one right way here". >> >> Yes, let's consider individual use cases. >> >> My use case are the docs for new comers: >> >> - https://github.com/pypa/sampleproject >> - https://packaging.python.org/en/latest/distributing/ >> >> That's why started the thread. > > > unfortunately, that isn't a use-case -- every newcomer has a different use > case. Indeed--I've helped newcomers whose very first attempt at packaging Python code includes Cython code for simulations. I think even for newcomers what we should be providing is not a "this is the way to do it" because then they get confused when that way doesn't work for them. Better, in the long term (and I'm happy to contribute to such an effort if it will help) is to provide a sort of Choose Your Own Adventure story. It can't all go on one page because that would be a mess, but a sort of "If you need to do this, read this. If you need to do this, read this. Now if you need to include some data files that are installed in your package read on, because there's really only one right way to do that. But now you have some options if you want to include tests: ..." > I was happy to see this thread, because I thought maybe I"d learn what i > should teach my students - new to python. > > But alas - there clearly really is no consensus. > > What i've told newbies in the past is somethig like: > > """ > if you want your user to be able to install you package, and then run > something like: > > import my_package > my_package.test() > > then put your tests inside the package. > > If you are fine with only being able to run the tests from the source tree > -- then put your tests outside the package. > """ > > but really, newbies have no idea how to make this decsion. > > Maybe we could come up with a decision tree for this -- some guidance for > knowing what to do, when? Exactly. I think it could even be fun :) How could we get started to add something like this to the packaging docs? Erik From mmericke at gmail.com Fri Oct 9 17:05:54 2015 From: mmericke at gmail.com (Michael Merickel) Date: Fri, 9 Oct 2015 10:05:54 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006231424.GA30512@k3> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> Message-ID: At the pylons project we've had a history of keeping our tests inside the packages. However, keeping them outside has proven to be nicer in some projects as well. 1) It reduces the size of the binary wheels that do not need to package the tests. 2) It still allows you to run the tests on an arbitrary installation if you want to by pulling down the repo and running them against the version installed in the virtualenv. Distributing the tests to every single installation is definitely not a requirement in order to be able to run them for people who want to run them, and decoupling them can really help with that. Projects that are shipping tests inside the package may have them removed upstream and then it is very difficult to run the suite against some binary distribution. On Tue, Oct 6, 2015 at 6:14 PM, David Wilson wrote: > On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: > > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > As Donald mentioned, this doesn't work in the general case since many > packages ship quite substantial test data around that often doesn't end > up installed, and in other cases since the package requires significant > fixture setup or external resources (e.g. running SQLAlchemy tests > without a working database server would be meaningless). > > The option of always shipping test data as a standard part of a package > in a vein attempt to always ensure it can be tested (which is not always > likely given the SQLAlchemy example above) strikes me as incredibly > wasteful, not from some oh-precious-bytes standpoint, but from the > perspective of distributing a Python application of any size where the > effects of always shipping half-configured test suites has increased the > resulting distribution size potentially by 3 or 4x. > > https://github.com/bennoleslie/pexif is the first hit on Google for a > module I thought would need some test data. It's actually quite > minimally tested, yet already the tests + data are 3.6x the size of the > module itself. > > > I appreciate arguments for inlining tests alongside a package in order > to allow reuse of the suite's functionality by consuming applications' > test suites, but as above, in the general case this simply isn't > something that will always work and can be relied on by default. > > > Is there perhaps a third option that was absent from the original post? > e.g. organizing tests in a separate, optional, potentially > pip-installable package. > > > > > > > outside the module like this: > > > > > > https://github.com/pypa/sampleproject/tree/master/tests > > > > There is no actual reason to do that except win a couple kilobytes if > > you are distributing your package on floppy disks for consumption on > > Z80-based machines with 64KB RAM. > > > > Even Python *itself* puts its test suite inside the standard library, > > not outside it (though some Linux distros may strip it away). > > Try "python -m test.regrtest" (again, this may fail if your distro > > decided to ship the test suite in a separate package). > > > > The PyP"A" should definitely fix its sample project to reflect good > > practices. > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuaxo2 at yahoo.com Sat Oct 10 08:48:59 2015 From: stuaxo2 at yahoo.com (Stuart Axon) Date: Sat, 10 Oct 2015 06:48:59 +0000 (UTC) Subject: [Distutils] Uploading stdeb built debs ? References: <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> Hi,? ? I built a package of using stdeb, any idea if these are acceptable to debian + ubuntu, and where to start / who to contact to get a package included in those distros ??S++ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wichert at wiggy.net Sat Oct 10 10:47:07 2015 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 10 Oct 2015 09:47:07 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> Message-ID: <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> Also having test code in the package can be very painful if you use tools like venusian which scan and try to import all Python files. Sent from my iPhone > On 09 Oct 2015, at 16:05, Michael Merickel wrote: > > At the pylons project we've had a history of keeping our tests inside the packages. However, keeping them outside has proven to be nicer in some projects as well. 1) It reduces the size of the binary wheels that do not need to package the tests. 2) It still allows you to run the tests on an arbitrary installation if you want to by pulling down the repo and running them against the version installed in the virtualenv. Distributing the tests to every single installation is definitely not a requirement in order to be able to run them for people who want to run them, and decoupling them can really help with that. Projects that are shipping tests inside the package may have them removed upstream and then it is very difficult to run the suite against some binary distribution. > >> On Tue, Oct 6, 2015 at 6:14 PM, David Wilson wrote: >> On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: >> >> > They should be inside the module. That way, you can check an installed >> > module is ok by running e.g. "python -m mypackage.tests". Any other >> > choice makes testing installed modules more cumbersome. >> >> As Donald mentioned, this doesn't work in the general case since many >> packages ship quite substantial test data around that often doesn't end >> up installed, and in other cases since the package requires significant >> fixture setup or external resources (e.g. running SQLAlchemy tests >> without a working database server would be meaningless). >> >> The option of always shipping test data as a standard part of a package >> in a vein attempt to always ensure it can be tested (which is not always >> likely given the SQLAlchemy example above) strikes me as incredibly >> wasteful, not from some oh-precious-bytes standpoint, but from the >> perspective of distributing a Python application of any size where the >> effects of always shipping half-configured test suites has increased the >> resulting distribution size potentially by 3 or 4x. >> >> https://github.com/bennoleslie/pexif is the first hit on Google for a >> module I thought would need some test data. It's actually quite >> minimally tested, yet already the tests + data are 3.6x the size of the >> module itself. >> >> >> I appreciate arguments for inlining tests alongside a package in order >> to allow reuse of the suite's functionality by consuming applications' >> test suites, but as above, in the general case this simply isn't >> something that will always work and can be relied on by default. >> >> >> Is there perhaps a third option that was absent from the original post? >> e.g. organizing tests in a separate, optional, potentially >> pip-installable package. >> >> >> > >> > > outside the module like this: >> > > >> > > https://github.com/pypa/sampleproject/tree/master/tests >> > >> > There is no actual reason to do that except win a couple kilobytes if >> > you are distributing your package on floppy disks for consumption on >> > Z80-based machines with 64KB RAM. >> > >> > Even Python *itself* puts its test suite inside the standard library, >> > not outside it (though some Linux distros may strip it away). >> > Try "python -m test.regrtest" (again, this may fail if your distro >> > decided to ship the test suite in a separate package). >> > >> > The PyP"A" should definitely fix its sample project to reflect good >> > practices. >> > >> > Regards >> > >> > Antoine. >> > >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Sat Oct 10 14:06:01 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Sat, 10 Oct 2015 14:06:01 +0200 Subject: [Distutils] venusian pain was: Where should I put tests when packaging python modules? In-Reply-To: <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> Message-ID: <5618FF29.104@thomas-guettler.de> Am 10.10.2015 um 10:47 schrieb Wichert Akkerman: > Also having test code in the package can be very painful if you use tools like venusian which scan and try to import all Python files. Hi Wichert Akkerman, can you please explain this pain? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From ncoghlan at gmail.com Sat Oct 10 16:11:02 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2015 00:11:02 +1000 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On 9 October 2015 at 20:11, Ionel Cristian M?rie? wrote: > > On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore wrote: >> >> >> The sample project is not intended for that purpose. It's a good >> starting point for something like that, but that's not its purpose. >> >> Paul >> >> PS I sympathise with your wish for a "do this to set up your first >> project" beginner's guide. I've wanted one of those myself for a long >> time now. But I don't believe such a thing exists, unfortunately. > > Should we at least give some pointers as to what people should look at for a > fully fledged project? Like a lists with examples/templates covering all the > variants. Eg: > > "PyPA's sample projects only covers the packaging aspect of a project. For > more exhaustive examples take a look at cookiecutter templates, barry's > sample, ogrisel's appveyor demo, this and that etc etc" > > How about something like that? This unfortunately isn't a good idea, at least for folks aiming to publish their first project. While it's well-intentioned, the problem with it is that it can be *tremendously* demotivating for folks that aren't already familiar with all those additional niceties, as it hits newcomers over the head with how much they still have to learn. Testing in particular is problematic, as writing tests involves writing programs about programs. That means you need to already understand a fair bit about programs *and* about how programs fail, to appreciate why it may make sense to write a second program to make sure the first program is behaving as you expect. Folks that already have that knowledge aren't going to need the packaging guide to tell them how to do it, while folks that don't have that knowledge are going to be confused by a digression on testing and documentation when they're wanting to learn about packaging and distribution. That said, what might be interesting is to have a section in the packaging user guide that talks about "The second release", as *that's* where the value of things like testing ("How do you know that everything you published the first time still works?") and semantic versioning ("How do your users know whether or not your API is stable?") start to make more sense. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 10 16:22:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2015 00:22:04 +1000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 10 October 2015 at 04:47, Chris Barker wrote: > So the question is: if I compile my third party package against numpy and > mkl, it would depend on numpy and py_mkl. > > If I went to installed this on a system that had a numpy that depends on > openblas, pip would install py_mkl (having already installed numpy and > py_openblas). And I'd have a numpy extension calling into a different BLAS > than numpy itself is calling into -- would that cause any problems? Python > would be linked to two libs with the same names -- would that cause > conflict? I'm way out of my depth here! Yes, you'd get weird binary ABI problems in that situation. There's a reason centrally built Linux distros and cross-platform distros like conda exist - we pick the common ABIs that all the components we build use, and enforce them in the build system. For "built by anyone" ABIs, the best you can hope for is to detect-and-report fundamental conflicts, as you can't solve the general case. The approach I proposed for metadata 2.0 is introducing an environmental constraints extension that caused things to fail at install time if the installer detected a binary incompatibility: https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension The equivalent in a pre-metadata 2.0 world would be yet-another-file in the dist-info directory. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From contact at ionelmc.ro Sat Oct 10 16:55:07 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 10 Oct 2015 17:55:07 +0300 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On Sat, Oct 10, 2015 at 5:11 PM, Nick Coghlan wrote: > This unfortunately isn't a good idea, at least for folks aiming to > publish their first project. While it's well-intentioned, the problem > with it is that it can be *tremendously* demotivating for folks that > aren't already familiar with all those additional niceties, as it hits > newcomers over the head with how much they still have to learn. > Testing in particular is problematic, as writing tests involves > writing programs about programs. That means you need to already > understand a fair bit about programs *and* about how programs fail, to > appreciate why it may make sense to write a second program to make > sure the first program is behaving as you expect. Folks that already > have that knowledge aren't going to need the packaging guide to tell > them how to do it, while folks that don't have that knowledge are > going to be confused by a digression on testing and documentation when > they're wanting to learn about packaging and distribution. > ?This begs the conception of yet another opinionated guide for X, Y and Z in Python. I believe we got plenty of those, and what encouraged their creation is exactly this sort of thinking. The problem is that none of them are really comprehensive, well maintained or authoritative enough (we wouldn't have this discussion if there was). They may have been "well-intentioned" but they only create confusion now ... What if there was something official-ish that takes the packaging guide further, for people that want more than just packaging? It seems that there is some demand. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Oct 10 17:18:30 2015 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 10 Oct 2015 10:18:30 -0500 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On Oct 10, 2015 9:55 AM, "Ionel Cristian M?rie?" wrote: > > > On Sat, Oct 10, 2015 at 5:11 PM, Nick Coghlan wrote: >> >> This unfortunately isn't a good idea, at least for folks aiming to >> publish their first project. While it's well-intentioned, the problem >> with it is that it can be *tremendously* demotivating for folks that >> aren't already familiar with all those additional niceties, as it hits >> newcomers over the head with how much they still have to learn. >> Testing in particular is problematic, as writing tests involves >> writing programs about programs. That means you need to already >> understand a fair bit about programs *and* about how programs fail, to >> appreciate why it may make sense to write a second program to make >> sure the first program is behaving as you expect. Folks that already >> have that knowledge aren't going to need the packaging guide to tell >> them how to do it, while folks that don't have that knowledge are >> going to be confused by a digression on testing and documentation when >> they're wanting to learn about packaging and distribution. > > > ?This begs the conception of yet another opinionated guide for X, Y and Z in Python. I believe we got plenty of those, and what encouraged their creation is exactly this sort of thinking. > > The problem is that none of them are really comprehensive, well maintained or authoritative enough (we wouldn't have this discussion if there was). They may have been "well-intentioned" but they only create confusion now ... > > What if there was something official-ish that takes the packaging guide further, for people that want more than just packaging? It seems that there is some demand. references for such a CreativeWork * http://docs.python-guide.org/en/latest/scenarios/ci/ * https://en.wikipedia.org/wiki/Continuous_integration#Best_practices * http://docs.python-guide.org/en/latest/writing/documentation/ * https://docs.python.org/devguide/documenting.html#sections * http://docs.python-guide.org/en/latest/writing/tests/ * https://packaging.python.org/en/latest/glossary/ * /peps /projects * https://westurner.org/tools/#python-packages * more links above (in this thread) * dox (docker + tox), cookiecutter- * https://westurner.org/tools/#python * https://westurner.org/wiki/awesome-python-testing > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wichert at wiggy.net Sat Oct 10 21:47:41 2015 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 10 Oct 2015 20:47:41 +0100 Subject: [Distutils] venusian pain was: Where should I put tests when packaging python modules? In-Reply-To: <5618FF29.104@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> <5618FF29.104@thomas-guettler.de> Message-ID: <8711BE71-F721-46F0-A30E-097E098547C6@wiggy.net> > On 10 Oct 2015, at 13:06, Thomas G?ttler wrote: > > Am 10.10.2015 um 10:47 schrieb Wichert Akkerman: >> Also having test code in the package can be very painful if you use tools like venusian which scan and try to import all Python files. > > > Hi Wichert Akkerman, > > can you please explain this pain? Importing tests often leads to problems for two reasons: 1) tests try to import things that are not normally installed (mock, pytest, redis-mock, etc.) which breaks application startup, and 2) some tests have import side-effects which can be fine for testing purposes, but should never trigger during normal usage. Wichert. From lac at openend.se Sat Oct 10 21:53:31 2015 From: lac at openend.se (Laura Creighton) Date: Sat, 10 Oct 2015 21:53:31 +0200 Subject: [Distutils] warning about potential problem for wheels Message-ID: <201510101953.t9AJrVwx007005@fido.openend.se> (note, I currently don't have mail delivery on for distutils. I could change this, but right now I don't think I have a lot to contribute. This is just a warning). If you have old windows hardware, which does not support SSE2, and windows 7, you can build your own python 3.5. This will work. But wheels won't, you will need to build them from source as well. see: http://bugs.python.org/issue25361 This means that wheels could start failing. It would be good if the wheels themselves could detect this problem and protest in a reasonable fashion, but I have no idea if this is possible. In any case, I thought you needed to know. Laura From oscar.j.benjamin at gmail.com Sat Oct 10 23:52:58 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 10 Oct 2015 21:52:58 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510101953.t9AJrVwx007005@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> Message-ID: On Sat, 10 Oct 2015 20:53 Laura Creighton wrote: (note, I currently don't have mail delivery on for distutils. I could change this, but right now I don't think I have a lot to contribute. This is just a warning). If you have old windows hardware, which does not support SSE2, and windows 7, you can build your own python 3.5. This will work. But wheels won't, you will need to build them from source as well. see: http:// bugs.python.org /issue25361 This means that wheels could start failing. It would be good if the wheels themselves could detect this problem and protest in a reasonable fashion, but I have no idea if this is possible. In any case, I thought you needed to know. There is no way for wheels to do this. A wheel is just a zip file with a standardised layout. Pip just extracts the zip, reads the metadata and copies the files to the appropriate locations. The metadata has no way to describe the fact that it the wheel contains SSE2 dependent binaries. The standard tools used to create wheels don't know anything about the contents of the compiled binaries so they don't really have a way to detect that the wheel depends on SSE2. Really this is just a case of an unsupported platform. It's unfortunate that CPython doesn't properly support this hardware but I think it's reasonable that if you have to build your interpreter from source then you have to build your extension modules as well. I'm not sure of a robust solution to detecting the problem at install time. Extension module authors can only really guarantee that their Windows binaries are compatible with standard released binaries. So if someone builds their own interpreter using different compiler options then there's no real way for pip or the extension module author to know if the binaries will be compatible. So either pip rejects all binaries for a non standard interpreter build or it installs them and hopes for the best. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Sun Oct 11 00:37:17 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 11 Oct 2015 00:37:17 +0200 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> Message-ID: <201510102237.t9AMbH5e018456@fido.openend.se> In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: >Really this is just a case of an unsupported platform. It's unfortunate >that CPython doesn't properly support this hardware but I think it's >reasonable that if you have to build your interpreter from source then you >have to build your extension modules as well. Alas that there is no easy way to detect. The situation I am imagining is where the administrators of a school build pythons for the students to run on their obsolete hardware, and then the poor students don't understand why pip doesn't work. But I suppose we will just get to deal with that problem when and if it happens. Laura From p.f.moore at gmail.com Sun Oct 11 00:47:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Oct 2015 23:47:58 +0100 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510102237.t9AMbH5e018456@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: You could, I suppose, patch your custom Python to *not* treat wheels built for cp35-none-win32 (or cp35-none-win_amd64, as appropriate) as compatible with that build (that requires a patch to the tag matching code in pip). You then wouldn't be able to use any binary wheels on PyPI, which is a bit heavy handed but probably necessary. But that's probably way more complex than is justified... Paul From oscar.j.benjamin at gmail.com Sun Oct 11 01:09:17 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 10 Oct 2015 23:09:17 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510102237.t9AMbH5e018456@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: On Sat, 10 Oct 2015 23:37 Laura Creighton wrote: In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: >Really this is just a case of an unsupported platform. It's unfortunate >that CPython doesn't properly support this hardware but I think it's >reasonable that if you have to build your interpreter from source then you >have to build your extension modules as well. Alas that there is no easy way to detect. The situation I am imagining is where the administrators of a school build pythons for the students to run on their obsolete hardware, and then the poor students don't understand why pip doesn't work. But I suppose we will just get to deal with that problem when and if it happens. Does it sound plausible to you that a school would build their own Pythons? I only know a few schools and I'd be very surprised if this happened at one of them but I guess there's a lot of schools in the world... The administrators at my daughter's school don't even understand how to put text into an email let alone install compilers and build Python! -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 11 01:10:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 16:10:39 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510102237.t9AMbH5e018456@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > >Really this is just a case of an unsupported platform. It's unfortunate > >that CPython doesn't properly support this hardware but I think it's > >reasonable that if you have to build your interpreter from source then you > >have to build your extension modules as well. > > Alas that there is no easy way to detect. The situation I am > imagining is where the administrators of a school build pythons for > the students to run on their obsolete hardware, and then the poor > students don't understand why pip doesn't work. But I suppose we > will just get to deal with that problem when and if it happens. In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: - ~0.5% of Firefox crashes are on machines that are missing SSE2 - <0.1% of machines with Steam installed are missing SSE2 I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sun Oct 11 01:24:23 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 10 Oct 2015 19:24:23 -0400 Subject: [Distutils] Uploading stdeb built debs ? References: <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20151010192423.416e35b3@anarchist.wooz.org> On Oct 10, 2015, at 06:48 AM, Stuart Axon via Distutils-SIG wrote: >Hi,? ? I built a package of using stdeb, any idea if these are acceptable to >debian + ubuntu, and where to start / who to contact to get a package >included in those distros ??S++ You can contact debian-python at lists.debian.org Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From njs at pobox.com Sun Oct 11 02:22:12 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 17:22:12 -0700 Subject: [Distutils] Bad setup.py dependencies on numpy, and 'pip upgrade' (was: Towards a simple and standard sdist format that isn't intertwined with distutils) Message-ID: On Sun, Oct 4, 2015 at 1:02 PM, Paul Moore wrote: [...] > A > common bug report for pip is users finding that their installs fail, > because setup.py requires numpy to be installed in order to run, and > yet pip is running setup.py egg-info precisely to find out what the > requirements are. We tell the user that the setup.py is written > incorrectly, and they should install numpy and retry the install, but > it's not a good user experience. And from a selfish point of view, > users blame *pip* for the consequences of a project whose code to > generate the metadata is buggy. Those bug reports are a drain on the > time of the pip developers, as well as a frustrating experience for > the users. [...] This is mostly orthogonal to the other discussion, but it's a problem relevant to distutils-sig where I think there's a lot of confusion, so I just wanted to try and clarify what's happening here. It's true that almost no packages accurately declare their dependencies on numpy. This is for two reasons: First, for packages that build-depend on numpy (e.g. they contain extensions that need numpy's header files), then distutils' design makes things tricky, b/c you have to call np.get_include() to find the headers and pass that to setup(). But this isn't the main issue -- solving this requires nasty hacks, but at least the more prominent projects are gradually gaining those hacks. The main issue is that people really, really, REALLY hate the recursive behavior of 'pip install -U'. They hate it *so much* -- especially the way it has a habit of suddenly trying to rebuild numpy when all you were trying to do was upgrade some little pure-python package -- that they actively refuse to accurately report their project's dependencies, because install -U can't recurse over dependencies that it can't see. E.g., here's two setup.py files for extremely mainstream projects, that both report accurate setup_requires and install_requires on numpy if and only if numpy is not already installed; if it is installed then they pretend that they don't need it: https://github.com/scipy/scipy/blob/master/setup.py#L200-L208 https://github.com/statsmodels/statsmodels/blob/master/setup.py#L96-L156 Obviously this is terrible -- it means that the wheels for these projects end up with incorrect dependencies (usually! it's kinda random!), and even if you install from source then whether future upgrades will work correctly depends on arbitrary details about how your virtualenv was configured when you did the original install. But people care about this so much that you actually have prominent developers going around and filing bugs on packages complaining that they provide accurate metadata and that this is a bug, they should lie instead. E.g.: https://github.com/pydata/patsy/issues/5 I'm not sure I agree with this conclusion, but the users have spoken. AFAICT the only way to fix this problem and start getting packages with accurate metadata is for pip to gain a non-recursive upgrade mode. The bug for that is https://github.com/pypa/pip/issues/59 AFAICT from reading that thread, work on this has stalled out because of the following reasoning: 1) Everyone agrees that pip should have 'upgrade $PKG' and 'upgrade-all' commands, and 'install -U' should be deprecated/removed. 2) But implementing 'upgrade-all' is tricky and dangerous without first fixing pip's dependency resolver. 3) Therefore we can't add 'upgrade' or 'upgrade-all' until after we fix pip's dependency resolver. I feel like there's a certain logical gap between (2) and (3)... we could defer 'upgrade-all' until later but start supporting 'upgrade $PKG' right now, couldn't we? (It'd be implemented as the equivalent of 'pip install $PKG=$LATEST_VERSION', which is not scary looking at all.) -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Sun Oct 11 02:28:40 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 Oct 2015 20:28:40 -0400 Subject: [Distutils] Bad setup.py dependencies on numpy, and 'pip upgrade' (was: Towards a simple and standard sdist format that isn't intertwined with distutils) In-Reply-To: References: Message-ID: On October 10, 2015 at 8:22:16 PM, Nathaniel Smith (njs at pobox.com) wrote: The bug for that is? https://github.com/pypa/pip/issues/59? AFAICT from reading that thread, work on this has stalled out because? of the following reasoning:? 1) Everyone agrees that pip should have 'upgrade $PKG' and? 'upgrade-all' commands, and 'install -U' should be deprecated/removed.? 2) But implementing 'upgrade-all' is tricky and dangerous without? first fixing pip's dependency resolver.? 3) Therefore we can't add 'upgrade' or 'upgrade-all' until after we? fix pip's dependency resolver.? I feel like there's a certain logical gap between (2) and (3)... we? could defer 'upgrade-all' until later but start supporting 'upgrade? $PKG' right now, couldn't we? (It'd be implemented as the equivalent? of 'pip install $PKG=$LATEST_VERSION', which is not scary looking at? all.)? I don?t see any reason that ``pip upgrade`` can?t be implemented prior to ``pip upgrade-all`` (or ``pip upgrade ?all``). It?s not adding a new thing that can cause problems without a dependency solver, it?s just moving the existing one and switching it so it?s non recursive by default. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 11 05:26:50 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 20:26:50 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Fri, Oct 9, 2015 at 10:04 AM, Chris Barker wrote: > 1) what in the world is a "source wheel"? And how is it different than an > sdist (other than maybe in a different file format. It seemed like people had different ideas about what a platonic "sdist" even is and this was causing us to talk past each other. So I proposed the terms "source release" and "source wheel" as two somewhat abstract concepts to let us to state positions like "I think an sdist should fill the role of a source wheel, and that there's no need for source releases", or "I think source wheels and source releases are different and important, so we need one spec for each". I'm not sure how much it helped :-) This is the email that suggested distinguishing between the terms: https://mail.python.org/pipermail/distutils-sig/2015-October/026981.html and the "source wheel" term in particular was stolen from Donald's earlier email: https://mail.python.org/pipermail/distutils-sig/2015-October/026964.html -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sun Oct 11 06:01:48 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 21:01:48 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, Oct 7, 2015 at 2:41 PM, Paul Moore wrote: [...] > Possibly the nearest > we'd have to an issue is over allowing the build process to *add* > dependencies to a binary wheel (e.g. a some builds depend on > currently-hypothetical MKL wheel, which provides needed DLLs). I don't > in principle object to that, but I'd like to see a fleshed out > proposal on how wheels containing just DLLs (as opposed to Python > packages) would work in practice - until we have a mechanism for > building/distributing such wheels, I think it's premature to worry > about specifying dependencies. Just to answer this: There's no formal proposal because it doesn't involve distutils-sig -- it's something that we already know how to do and are working on implementing :-). The only reason it doesn't already exist on PyPI is that Windows is the main target, and toolchain problems mean that we aren't yet able to build numpy or scipy wheels on Windows at all. (Long story...) The basic idea though is just, we make a "python package" with trivial structure: pyopenblas/ __init__.py openblas.dll include/ ... Usage: # in downstream package setup.py Extension(..., include_dirs=... + pyopenblas.get_include(), linker_arguments=... + pyopenblas.get_linker_arguments(), ...) # in downstream package __init__.py import pyopenblas pyopenblas.enable() Implementation: pyopenblas/__init__.py contains some code like: DIR = os.path.dirname(__file__) def get_include(): return [os.path.join(DIR, "include")] def get_linker_arguments(): return ["-L" + DIR, "-lpyopenblas"] def enable(): # Platform specific code to let the runtime linker find libopenblas.so if WINDOWS: # ctypes magic to preload the DLL elif LINUX: # modify os.environ["LD_LIBRARY_PATH"] else: ... -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sun Oct 11 06:31:27 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 21:31:27 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? Message-ID: Hi all, I'm finding it impossible to keep track of that other thread, and I guess I'm probably not the only one, so I figured I'd try splitting a few of the more specific discussions out :-). One thing that seems to be a key issue, but where I remain very confused, is the question of what pip actually needs from an sdist. (Not PyPI, just pip or other package install tools.) Right now, IIUC, there are three times that pip install touches sdist-related metadata: 1) it uses the name+version that are embedded in the sdist filename to select an sdist from an index like PyPI 2) after unpacking this sdist it then calls 'setup.py egg_info' to get the full metadata for the wheel (or wheel equivalent) that this sdist will eventually produce. Specifically what it does with this is extract the setup_requires and install_requires fields, and uses them to go find other wheels/sdists that also need to be installed 3) eventually it actually builds the package, and this produces a wheel (or wheel equivalent) that has its own metadata (which often matches the metadata from egg_info in step (2), but not always) Is that a correct description of current behavior? Is there anything that pip ever looks at besides name, version, dependencies? Paul says that this is broken, and that pip gets lots of bug reports that "can be traced back to needing to run setup.py egg-info to get metadata" [1]. Since AFAICT the only metadata that pip actually touches is name, version, and dependencies, and it already knows the name and version before it runs egg_info, I assume that what this means is that it's crucial for pip to have static access to dependency information? OTOH in another email Paul says that name and version are the minimum he wants [2], so maybe I'm reading too much into this :-). >From the discussion so far, it sounds like the particularly crucial question is whether pip needs to statically know dependencies before building a wheel. Trying to reason through from first principles, I can't see any reason why it would. It would be somewhat convenient if sdists did list their binary dependencies: if that were the case, then pip could take a strictly phased approach: 1) solve the complete dependency graph to find a set of packages to install / remove 2) for all the packages-to-be-installed that are sdists, turn them into wheels 3) install all the wheels OTOH if sdists have only name and version statically, but not dependency information, then you need to do something like: 1) create a fake dependency graph that contains accurate information for all known wheels, and for each sdist add a fake node that has the right name and version number but pretends not to have any dependencies. 2) solve this graph to find the set of packages to install / remove 3) if any of the packages-to-be-installed are sdists, then fetch them, run egg_info or build them or whatever to get their real dependencies, add these to the graph, and go to step 1 4) else, we have wheels for everything; install them. (This works because dependencies are constraints -- adding dependencies can only reduce the space of possible solutions, never enlarge it. Also, because by the time we decide to fetch and build any sdists, we already know that we're very likely to want to install them, so the performance penalty for building packages we turn out not to want is not high. And, crucially, we know that there exists some set of dependency metadata which would convince us to install these sdists, and dependency metadata is under the package author's control, so we already have established a trust route to the author of this package -- if they don't declare any dependencies, then we'll be installing and running arbitrary code of theirs, so running arbitrary code to check their dependencies doesn't require any additional trust.) But there's often a large difference between what we work out from first principles and how things actually work :-). Is there anything I'm missing in the analysis above? Do the relevant pip maintainers even read this mailing list? :-) -n [1] https://mail.python.org/pipermail/distutils-sig/2015-October/026960.html [2] https://mail.python.org/pipermail/distutils-sig/2015-October/026942.html -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Sun Oct 11 13:03:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2015 12:03:44 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 11 October 2015 at 05:01, Nathaniel Smith wrote: > There's no formal proposal because it doesn't involve distutils-sig -- > it's something that we already know how to do and are working on > implementing :-). The only reason it doesn't already exist on PyPI is > that Windows is the main target, and toolchain problems mean that we > aren't yet able to build numpy or scipy wheels on Windows at all. > (Long story...) > > The basic idea though is just, we make a "python package" with trivial > structure: Cool, thanks for the clarification - I hadn't realised it was as straightforward as this. Paul From p.f.moore at gmail.com Sun Oct 11 13:49:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2015 12:49:19 +0100 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 11 October 2015 at 05:31, Nathaniel Smith wrote: > Do the relevant pip maintainers > even read this mailing list? :-) Donald and I are pip maintainers (not the only ones, but probably the ones who make the most noise on this list :-)). The comments I've made that you quoted are based mostly on how I recall pip works, it's unfortunately been some time since I looked into pip's dependency resolution and install code (it's a bit of a pit of snakes) and I haven't had time to check the details when commenting on this thread, so I don't claim my comments are authoritative. I think your description of the behaviour is right. As regards what pip could do, technically you are of course correct (it's possible, it just needs someone willing to make the code changes). I don't know, however, if the approach you're proposing fits with how we currently envisage pip developing. Specifically, my personal goal [1] is that we get to a point where pip can do all of the dependency resolution steps, getting to a point where it knows exactly what it will install, *before* it starts downloading packages, running build steps, etc. (And I don't see that sitting well with your proposal). The reason I want that is so that I would like to be able to see what pip plans on doing, and audit it before it begins (a "pip install --dry-run" type of step). Typically, this lets me avoid getting part-way through an install only to find that I need a project that doesn't have a wheel and I can't build from the sdist. That's a real problem I deal with a lot - at the moment I work with "pip list --outdated", check the listing to see what provides wheels, and update just those packages individually. But that's messy and manual, and not always 100% effective - and I'd like to be able to do better. I also don't think it's reasonable to focus solely on what *pip* requires. Just as important (IMO) is allowing PyPI to provide rich metadata. In particular, I write a lot of programs that query PyPI for package information. There are a lot of useful scripts I would like to be able to write that need to query PyPI for the dependency details of a package, and that simply isn't possible at the moment. Most of these are ad-hoc or personal-use scripts, and a lot of things I simply haven't done because I can't. So it's hard to quantify the benefits. But currently, the wonderful Metadata 2.0 ideal future promises me static dependency data on PyPI. That will only happen if we get enough people interested in implementing it, but I live in hope. If I understand your proposals, they are closing off that option, and explicitly stating that we'll never have dependency metadata on PyPI for projects that don't publish wheels. Maybe you're right, and it'll never be possible. But I don't want to give up that dream without being sure it's necessary - which is why I push back so hard on the idea. Paul [1] Which is not backed up by me having the time to write the code, so the only relevance here is that I'm inclined to support proposals that work towards that goal. From steve.dower at python.org Sun Oct 11 16:17:42 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 11 Oct 2015 07:17:42 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Nathaniel Smith" Sent: ?10/?10/?2015 16:11 To: "Laura Creighton" Cc: "distutils-sig" Subject: Re: [Distutils] warning about potential problem for wheels On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > >Really this is just a case of an unsupported platform. It's unfortunate > >that CPython doesn't properly support this hardware but I think it's > >reasonable that if you have to build your interpreter from source then you > >have to build your extension modules as well. > > Alas that there is no easy way to detect. The situation I am > imagining is where the administrators of a school build pythons for > the students to run on their obsolete hardware, and then the poor > students don't understand why pip doesn't work. But I suppose we > will just get to deal with that problem when and if it happens. In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: - ~0.5% of Firefox crashes are on machines that are missing SSE2 - <0.1% of machines with Steam installed are missing SSE2 I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Oct 11 16:31:24 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 Oct 2015 10:31:24 -0400 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? Sent from my iPhone > On Oct 11, 2015, at 10:17 AM, Steve Dower wrote: > > An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). > > That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). > > Cheers, > Steve > > Top-posted from my Windows Phone > From: Nathaniel Smith > Sent: ?10/?10/?2015 16:11 > To: Laura Creighton > Cc: distutils-sig > Subject: Re: [Distutils] warning about potential problem for wheels > > On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > > > >Really this is just a case of an unsupported platform. It's unfortunate > > >that CPython doesn't properly support this hardware but I think it's > > >reasonable that if you have to build your interpreter from source then you > > >have to build your extension modules as well. > > > > Alas that there is no easy way to detect. The situation I am > > imagining is where the administrators of a school build pythons for > > the students to run on their obsolete hardware, and then the poor > > students don't understand why pip doesn't work. But I suppose we > > will just get to deal with that problem when and if it happens. > > In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: > - ~0.5% of Firefox crashes are on machines that are missing SSE2 > - <0.1% of machines with Steam installed are missing SSE2 > > I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) > > -n > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Sun Oct 11 17:20:55 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 11 Oct 2015 15:20:55 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: On Sun, 11 Oct 2015 15:31 Donald Stufft wrote: Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? It should be fine either way as long as the CPU can handle the particular instructions used. X86 is backward compatible like that so unless the compiler does something funny when the SSE option is enabled it should be fine. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sun Oct 11 17:07:30 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 11 Oct 2015 08:07:30 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: Yes. SSE never makes it into the module ABI, it's just an implementation concern. (I believe it may be used across function/module boundaries internally when the compiler can prove that it's inaccessible from the outside - a more space effective form of inlining.) There will be some 3.5.0-built wheels that won't work on some machines running 3.5.1 without SSE, and some wheels will see a performance decrease when built with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones weren't expecting the perf increase anyway, so compatibility is likely there biggest concern. This does only affect 32-bit builds, so now I'm thinking about the possibility of treating those as highly compatible while the 64-bit ones get better performance treatment, though I'm not sure how that could actually play out. It may help remove some of the questions about which one to use though. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Donald Stufft" Sent: ?10/?11/?2015 7:31 To: "Steve Dower" Cc: "distutils-sig" ; "Laura Creighton" Subject: Re: [Distutils] warning about potential problem for wheels Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? Sent from my iPhone On Oct 11, 2015, at 10:17 AM, Steve Dower wrote: An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). Cheers, Steve Top-posted from my Windows Phone From: Nathaniel Smith Sent: ?10/?10/?2015 16:11 To: Laura Creighton Cc: distutils-sig Subject: Re: [Distutils] warning about potential problem for wheels On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > >Really this is just a case of an unsupported platform. It's unfortunate > >that CPython doesn't properly support this hardware but I think it's > >reasonable that if you have to build your interpreter from source then you > >have to build your extension modules as well. > > Alas that there is no easy way to detect. The situation I am > imagining is where the administrators of a school build pythons for > the students to run on their obsolete hardware, and then the poor > students don't understand why pip doesn't work. But I suppose we > will just get to deal with that problem when and if it happens. In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: - ~0.5% of Firefox crashes are on machines that are missing SSE2 - <0.1% of machines with Steam installed are missing SSE2 I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) -n _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Oct 11 17:49:58 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 Oct 2015 11:49:58 -0400 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: <06B5B098-7A7F-4766-BAD8-9EAEC8EE766A@stufft.io> Okay. Thanks (and Oscar too). I'm not really that great at C and compiled stuff so I wasn't sure if it would have any negative effects like that. Seems like a reasonable change. Sent from my iPhone > On Oct 11, 2015, at 11:07 AM, Steve Dower wrote: > > Yes. SSE never makes it into the module ABI, it's just an implementation concern. (I believe it may be used across function/module boundaries internally when the compiler can prove that it's inaccessible from the outside - a more space effective form of inlining.) > > There will be some 3.5.0-built wheels that won't work on some machines running 3.5.1 without SSE, and some wheels will see a performance decrease when built with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones weren't expecting the perf increase anyway, so compatibility is likely there biggest concern. > > This does only affect 32-bit builds, so now I'm thinking about the possibility of treating those as highly compatible while the 64-bit ones get better performance treatment, though I'm not sure how that could actually play out. It may help remove some of the questions about which one to use though. > > Cheers, > Steve > > Top-posted from my Windows Phone > From: Donald Stufft > Sent: ?10/?11/?2015 7:31 > To: Steve Dower > Cc: distutils-sig; Laura Creighton > Subject: Re: [Distutils] warning about potential problem for wheels > > Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? > > Sent from my iPhone > >> On Oct 11, 2015, at 10:17 AM, Steve Dower wrote: >> >> An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). >> >> That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). >> >> Cheers, >> Steve >> >> Top-posted from my Windows Phone >> From: Nathaniel Smith >> Sent: ?10/?10/?2015 16:11 >> To: Laura Creighton >> Cc: distutils-sig >> Subject: Re: [Distutils] warning about potential problem for wheels >> >> On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: >> > >> > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: >> > >> > >Really this is just a case of an unsupported platform. It's unfortunate >> > >that CPython doesn't properly support this hardware but I think it's >> > >reasonable that if you have to build your interpreter from source then you >> > >have to build your extension modules as well. >> > >> > Alas that there is no easy way to detect. The situation I am >> > imagining is where the administrators of a school build pythons for >> > the students to run on their obsolete hardware, and then the poor >> > students don't understand why pip doesn't work. But I suppose we >> > will just get to deal with that problem when and if it happens. >> >> In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: >> - ~0.5% of Firefox crashes are on machines that are missing SSE2 >> - <0.1% of machines with Steam installed are missing SSE2 >> >> I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) >> >> -n >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Oct 11 18:44:07 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2015 18:44:07 +0200 Subject: [Distutils] warning about potential problem for wheels References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: <20151011184407.22503ba9@fsol> On Sun, 11 Oct 2015 08:07:30 -0700 Steve Dower wrote: > > This does only affect 32-bit builds, so now I'm thinking about the > possibility of treating those as highly compatible while the 64-bit > ones get better performance treatment, though I'm not sure how that > could actually play out. It may help remove some of the questions > about which one to use though. That sounds reasonable to me. I don't know Windows very much, but are there still many people using 32-bit Windows these days (on x86, I mean)? Regards Antoine. From oscar.j.benjamin at gmail.com Sun Oct 11 19:43:14 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 11 Oct 2015 17:43:14 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <20151011184407.22503ba9@fsol> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: On Sun, 11 Oct 2015 17:44 Antoine Pitrou wrote: On Sun, 11 Oct 2015 08:07:30 -0700 Steve Dower wrote: > > This does only affect 32-bit builds, so now I'm thinking about the > possibility of treating those as highly compatible while the 64-bit > ones get better performance treatment, though I'm not sure how that > could actually play out. It may help remove some of the questions > about which one to use though. That sounds reasonable to me. I don't know Windows very much, but are there still many people using 32-bit Windows these days (on x86, I mean)? I don't know but I think it makes sense to follow Windows' lead. So if 3.5 supports Vista and Vista doesn't require SSE2 then CPython shouldn't either. If 3.6 or whatever drops support for Vista and if Windows 7 requires SSE2 then CPython can require it too. I assume this what happens with the OSX binaries. Note that my recently retired computer was 64 bit and had SSE but didn't have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers are going to stop adding new instructions. So this general issue isn't limited to 32 bit hardware and won't be solved by special casing that. I think it makes sense to have a general policy for architectures that will be supported by the official build in future. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Oct 11 19:48:51 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 10:48:51 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > 2) after unpacking this sdist it then calls 'setup.py egg_info' to get > the full metadata for the wheel I wouldn't say "get the full metadata for the wheel". it's not a wheel yet. `egg_info` run so we can use the pkg_resources api to find the dependencies. > Specifically what it does with this is > extract the setup_requires and install_requires fields specifically, we call `requires` on pkg_resources distribution objects https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583 > 3) eventually it actually builds the package, and this produces a > wheel (or wheel equivalent) that has its own metadata (which often > matches the metadata from egg_info in step (2), but not always) > "not always"? not following your point they're 2 different formats, but should contain the same essential information. here's the wheel code that does the conversion https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90 > name and version before it runs egg_info, I assume that what this > means is that it's crucial for pip to have static access to dependency > information? yes > It would be somewhat convenient if sdists did list their binary > dependencies: not sure about your insertion of "binary" here. pip is concerned with finding python project dependencies (i.e. name and version constraints) in the sdist and then based on the current install environment, it will further constrain the wheels chosen based on architecture and python implementation. and to be perfectly clear, none of this deals with non-python OS/Distro requirements. > 3) if any of the packages-to-be-installed are sdists, then fetch them, > run egg_info or build them or whatever to get their real dependencies, > add these to the graph, and go to step 1 > this is the pain we don't want in the future. since we're simply talking about name/version constraints (not platform/python), It's hard to conceive that we'd agree on an sdist spec that didn't include that. > Do the relevant pip maintainers > even read this mailing list? :-) > I try : ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Oct 11 20:12:51 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 11:12:51 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > since we're simply talking about name/version constraints (not platform/python) let me clarify that... since I imagine someone might bring up markers in response... markers allow you to vary your name/version dependencies by environment (for example, by platform and python version), but to be clear, it's still not a "binary" dependency declaration... i.e. the dependency is still declared by name and a version specifier alone. On Sun, Oct 11, 2015 at 10:48 AM, Marcus Smith wrote: > > >> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get >> the full metadata for the wheel > > > I wouldn't say "get the full metadata for the wheel". it's not a wheel > yet. > `egg_info` run so we can use the pkg_resources api to find the > dependencies. > > > >> Specifically what it does with this is >> extract the setup_requires and install_requires fields > > > specifically, we call `requires` on pkg_resources distribution objects > https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583 > > > >> 3) eventually it actually builds the package, and this produces a >> wheel (or wheel equivalent) that has its own metadata (which often >> matches the metadata from egg_info in step (2), but not always) >> > > "not always"? not following your point > they're 2 different formats, but should contain the same essential > information. > here's the wheel code that does the conversion > > https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90 > > > >> name and version before it runs egg_info, I assume that what this >> means is that it's crucial for pip to have static access to dependency >> information? > > > yes > > > >> It would be somewhat convenient if sdists did list their binary >> dependencies: > > > not sure about your insertion of "binary" here. > pip is concerned with finding python project dependencies (i.e. name and > version constraints) in the sdist > and then based on the current install environment, it will further > constrain the wheels chosen based on architecture and python implementation. > and to be perfectly clear, none of this deals with non-python OS/Distro > requirements. > > > >> 3) if any of the packages-to-be-installed are sdists, then fetch them, >> run egg_info or build them or whatever to get their real dependencies, >> add these to the graph, and go to step 1 >> > > this is the pain we don't want in the future. > > since we're simply talking about name/version constraints (not > platform/python), It's hard to conceive that we'd agree on an sdist spec > that didn't include that. > > > >> Do the relevant pip maintainers >> even read this mailing list? :-) >> > > I try : ) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sun Oct 11 20:18:21 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 11 Oct 2015 11:18:21 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: The general rule is the minimum requirements of all supported operating systems, so in this case requiring SSE is a bug. Note that this doesn't prevent us from having SSE optimized code paths, as long as there is a fallback to one that is more compatible. And it's also not binding on redistributors or package builders, who are free to add more restrictive requirements. As far as identifying these sort of requirements in packages, I don't think they really make sense as part of a platform tag as a non-SSE build is fully compatible with an SSE build, but as part of version ordering it would be nice to be able to prefer certain tagged builds without excluding ones without the tag (e.g. if my platform is win32_sse2 and there's only a win32 tagged build, I still want it - how we formalize this is probably very complex though. Also, my platform tag is probably 100 chars long with everything I may want to represent...) Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Oscar Benjamin" Sent: ?10/?11/?2015 10:43 To: "Antoine Pitrou" ; "Distutils-Sig at python.org" Subject: Re: [Distutils] warning about potential problem for wheels On Sun, 11 Oct 2015 17:44 Antoine Pitrou wrote: On Sun, 11 Oct 2015 08:07:30 -0700 Steve Dower wrote: > > This does only affect 32-bit builds, so now I'm thinking about the > possibility of treating those as highly compatible while the 64-bit > ones get better performance treatment, though I'm not sure how that > could actually play out. It may help remove some of the questions > about which one to use though. That sounds reasonable to me. I don't know Windows very much, but are there still many people using 32-bit Windows these days (on x86, I mean)? I don't know but I think it makes sense to follow Windows' lead. So if 3.5 supports Vista and Vista doesn't require SSE2 then CPython shouldn't either. If 3.6 or whatever drops support for Vista and if Windows 7 requires SSE2 then CPython can require it too. I assume this what happens with the OSX binaries. Note that my recently retired computer was 64 bit and had SSE but didn't have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers are going to stop adding new instructions. So this general issue isn't limited to 32 bit hardware and won't be solved by special casing that. I think it makes sense to have a general policy for architectures that will be supported by the official build in future. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Oct 11 21:22:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2015 20:22:47 +0100 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 11 October 2015 at 19:37, Robert Collins wrote: > 'pip install --only-binary :all: THINGTOINSTALL' will do exactly what > you want I think in that it will only download wheels. ooh, I didn't spot that getting added! Many thanks for that. Paul From robertc at robertcollins.net Sun Oct 11 22:13:53 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 09:13:53 +1300 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 12 October 2015 at 08:22, Paul Moore wrote: > On 11 October 2015 at 19:37, Robert Collins wrote: >> 'pip install --only-binary :all: THINGTOINSTALL' will do exactly what >> you want I think in that it will only download wheels. > > ooh, I didn't spot that getting added! Many thanks for that. > > Paul You're welcome :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Sun Oct 11 23:53:03 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 10:53:03 +1300 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <5616CC7C.4060601@thomas-guettler.de> References: <5616CC7C.4060601@thomas-guettler.de> Message-ID: On 9 October 2015 at 09:05, Thomas G?ttler wrote: > This is a follow up to the thread "Where should I put tests when packaging python modules?" > > I have never used tox up to now. But reading the mails of the thread, it seems > that tox is the current state of the art. > > This leads me to the conclusion that the sample project should use tox. > > Any reasons not to do it? Yes, as already said - the sample project is limited in scope to packaging. But in the larger context of 'setting up a Python project from scratch - many folk have written opinionated guides - and if you were writing one of those, even if tox isn't consensus opinion, I'd say use it, so that new folk don't need to read a chunk of prose about choosing the right-thing-for-them. The challenge with education is making the learning curve approachable :) -rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Sun Oct 11 20:37:59 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 07:37:59 +1300 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 12 October 2015 at 00:49, Paul Moore wrote: > On 11 October 2015 at 05:31, Nathaniel Smith wrote: >> Do the relevant pip maintainers >> even read this mailing list? :-) > > Donald and I are pip maintainers (not the only ones, but probably the > ones who make the most noise on this list :-)). The comments I've made > that you quoted are based mostly on how I recall pip works, it's > unfortunately been some time since I looked into pip's dependency > resolution and install code (it's a bit of a pit of snakes) and I > haven't had time to check the details when commenting on this thread, > so I don't claim my comments are authoritative. > > I think your description of the behaviour is right. I have this on my to-read-threads list, just haven't had time to do it justice yet. FWIW I am *very* familiar with the dependency/install code at the moment, having overhauled it for: - wheel caching - working proof of concept static-dependencies-in-metadata handling - working proof of concept full resolver And various related bits. > ...The reason I want that is so that I would like to be able > to see what pip plans on doing, and audit it before it begins (a "pip > install --dry-run" type of step). Typically, this lets me avoid > getting part-way through an install only to find that I need a project > that doesn't have a wheel and I can't build from the sdist. That's a > real problem I deal with a lot - at the moment I work with "pip list > --outdated", check the listing to see what provides wheels, and update > just those packages individually. But that's messy and manual, and not > always 100% effective - and I'd like to be able to do better. 'pip install --only-binary :all: THINGTOINSTALL' will do exactly what you want I think in that it will only download wheels. -Rob From robertc at robertcollins.net Mon Oct 12 02:02:10 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 13:02:10 +1300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> Message-ID: On 10 October 2015 at 21:47, Wichert Akkerman wrote: > Also having test code in the package can be very painful if you use tools > like venusian which scan and try to import all Python files. Even if tests are not in-package, such tools need to cope with extras in general: any optionally-installed dependencies will have this issue. Venusian offers a callback on import errors - http://venusian.readthedocs.org/en/latest/#onerror-scan-callback. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 12 02:19:26 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 13:19:26 +1300 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On 8 October 2015 at 17:47, Nathaniel Smith wrote: > On Wed, Oct 7, 2015 at 3:39 PM, Robert Collins > wrote: > [...] >> Its all about easy-install. This is why I put a proof of concept >> static-setup-requires thing together for pip (and its on our teams >> roadmap to deliver a production version of it via patches to all the >> pypa projects, we're just not at that point yet - the resolver is >> first, and we have to finish rolling out constraints within OpenStack >> before that gets large timeslices). > > I remembered Nick saying something about this at PyCon, but I couldn't > find anything when I looked -- could you point me to the PoC? https://github.com/rbtcollins/pip/tree/issue-988 - resolver - its a little horked right now as I haven't fully addressed the constraints changes in the merge with it, but its readable. https://github.com/rbtcollins/pip/tree/declarative - declarative commits POC. Totally stale code but also worth reading (and not hard to update when the time comes) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 12 06:06:45 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 17:06:45 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: EWOW, huge thread. I've read nearly all of it but in order not to make it massively worse, I'm going to reply to all the points I think need raising in one mail :). Top level thoughts here, more point fashion with only rough editing below the fold. I realise many things - like the issue between different wheels of the same package consuming different numpy abis - have been touched on, but AFAICT they are entirely orthogonal to the proposal, which was to solve 'be able to use arbitrary build systems and still install with pip'. Of the actual problems with using arbitrary build systems, 99% of them seem to boil down to 'setup-requires isn't introspectable by pip (https://github.com/pypa/pip/issues/1820 ). - If it was, then alternative build systems could be depended on reasonably; and the mooted thunk from setuptools CLI to arbitrary build system would be viable. It is, in principle a matter of one patch to teach pip *a* way to do this (and then any and all build systems that want to can utilise it). https://github.com/rbtcollins/pip/tree/declarative is a POC I did - my next steps on that were to discuss the right ecosystem stuff for it - e.g. should pip consume it via setuptools, or should pip support it as *the way* and other systems including setuptools can choose to use it? A related but separate thing is being able to *exclusively* install things without setuptools present - I've filed https://github.com/pypa/pip/issues/3175 about that, but I think its -much- lower priority than reliably enabling third party build tools. -Rob ---- " solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. " Actually pip still punts a *lot* here - we have bypasses to let things like C compiler flags be set during wheel build, and when thats done we don't cache the wheels (or even try to build wheels). " While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: ... (c) you are forced to use them anyway, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``." I don't understand the claim of (c) here - its entirely possible to write a package that doesn't use setuptools and have it do the right thing - pip uses a subprocess to drive package installation, and the interface is documented. The interface might be fugly as, but it exists and works. It is missing setup-requires handling, but so is setup.py itself. The only thing we'd really need to do AFAICT is make our setuptools monkeypatching thunk handle setuptools not being installed (which would be a sensible thing to Just Do anyhow). " - query for build dependencies - run a build, producing wheels as output - set up the current source tree so that it can be placed on ``sys.path`` in "develop mode" " So we have that already. setup.py egg-info, setup.py bdist_wheel, setup.py develop. "A version 1-or-greater format source tree can be identified by the presence of a file ``_pypackage/_pypackage.cfg``. " I really don't like this. Its going to be with us forever, and its intrusive (its visible), and so far isn't shown to be fixing anything. "to scatter files around willy-nilly never works, so we adopt the convention that names starting with an underscore are reserved for official use, and non-underscored names are available for idiosyncratic use by individual projects." I can see the motivation here, but is it really solving a problem we have? On the specifics of the format: I don't want to kibbitz over strawman aspects at this point. Having the extension mechanism be both pip specific and in Python means that we're going to face significant adoption issues: the former because pip is not by any means the only thing around - and some distros have until very recently been actively hostile to pip (which in turn means we need to wait a decade or two for them to age-out and stop being used). The latter because we'll face all the headaches of running arbitrary untrusted code and dealing with two deps with different versions of the same hook and so on: I think its an intrinsically unsafe design. @dstufft "problem with numpy.distutils, as I know you?re aware!). We could do a minimal extension and add another defacto-ish standard of allowing pip and setuptools to process additional setup_requires like arguments from a setup.cfg to solve that problem though. The flip side to this is that since it involves new capabilities in pip/setuptools/any other installer is that it you?ll have several years until you can depend on setup.cfg based setup_requires from being able to be depended on. " Well. For *any* proposal that involves modifying pip, we have to assume that all existing things keep working, and that anyone wanting to utilise the new thing will have to either a) include a local compatibility thunk, or b) error when being used from a too-old toolchain. I don't think that should really be a factor in design since its intrinsic to the quagmire. "Longer term, I think the answer is sdist 2.0 which has proper metadata inside of it (name, version, dependencies, etc) but which also includes a hook like this PEP has to specify the build system that should be used to build a wheel out of this source distribution." Any reason that can't just be setup.cfg ? @Daniel "I thought Robert Collins had a working setup-requires implementation already? I have a worse but backwards compatible one too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py" - https://github.com/rbtcollins/pip/tree/declarative - I'll be updating that probably early next year at this rate - after issue-988 anyhow. The issue with your approach is that pip doesn't handle having concurrent installs done well - and in fact it will end up locking its environment somehow. @Paul " I can understand that a binary wheel may need a certain set of libraries installed - but that's about the platform tags that are part of the wheel definition, not about dependencies. Platform tags are an ongoing discussion, and a good example of a partial solution that" - thats where the draft PEP tennessee and I start is aimed - at making those libraries be metadata, not platform tags. @Chris " A given package might depend on numpy, as you say, and it may work with all numpy versions 1.6 to 1.9. Fine, so we specify that in install_requires. And this shodl be the dependency in the sdist, too. If the package is pur python, this is fine and done. But if the package has some extensions code that used the numpy C API ( a very common occurrence), then when it is built, it will only work (reliably) with the version of numpy it was built with. So the project itself, and the sdist depend on numpy >=1.6, but a build binary wheel depends on numpy == 1.7 (for instance). Which requires a binary (wheel) dependency that is somewhat different than the source dependency. " - so yes, that is where bdist_wheel should be creating different metadata for that wheel. The issue that arises is that we need unique file names so that they can coexist on PyPI or local archives - which is where wheel tags come in. I'd be in favour of not using semantic tags for this - rather hash the deps or something and just make a unique file name. Use actual metadata for metadata. @Nathaniel "I know that one unpleasant aspect of the current design is that the split between egg-info and actual building creates the possibility for time-of-definition-to-time-of-use bugs, where the final wheel hopefully matches what egg-info said it would, but in practice there could be skew. (Of course this is true in any system which represents" - actually see https://bugs.launchpad.net/pbr/+bug/1502692 for a bug where this 'skew' is desirable: for older environments we want tailored deps with no markers, for anything supporting markers we want them - so the wheel will have markers and egg_info won't. @Nathaniel " (Part of the intuition for the last part is that we also have a not-terribly-secret-conspiracy here for writing a PEP to get Linux wheels onto PyPI and at least achieve feature parity with Windows / OS X. Obviously there will always be weird platforms -- iOS and FreeBSD and Linux-without-glibc and ... -- but this should dramatically reduce the frequency with which people need sdist dependencies.)" - I think a distinction between sdist and binary names for dependencies would be a terrible mistake. It will raise complexity for reasoning and describing things without solving any concrete problem that I can see. @Nathaniel "I guess to make progress in this conversation I need some more detailed explanations. I totally get that there's a long history of thought and conversations behind the various assertions here like "a sdist is fundamentally different from a VCS checkout", "there must be a 1-1 mapping between sdists and wheels", "pip needs sdists that have full wheel metadata in static form", and I'm barging in from the outside with no context, but I literally have no idea why the specific design features you're asking for are desirable or even viable. Right now if I were to try and write the PEP you're asking for, then the rationale section would just be "because Donald said so" over and over :-). I couldn't write the motivation section, because I don't know any problems that the PEP you're describing would fix for me as a package author (which doesn't mean they don't exist, but!)." -- VCS trees are (generally) by-humans for humans. They are the primary source of data and can do thinks like inferring versions from commit data. sdists are derived from the VCS tree and can include extra data (such as statically defined version data). Wheels are derived from a tree on disk and can (today) be built from either VCS trees or sdists. I'm not sure that forcing an sdist step is beneficial - the egg-info step we have today is basically that without the cost of compressing and decompressing potentially large trees for no reason. @Jeremy "An sdist is an installable package which just happens to _look_ a lot like a source release tarball, but trying to pretend that downstream packagers will want to use it as such leads to a variety of pain points in the upstream/downstream relationship. For better or worse a lot of distros don't want generated files in upstream source code releases, since they need to confirm that they also ship the necessary tooling to regenerate any required files and that the generated files they ship match what their packaged tooling produces." - Well, pbr doesn't work if you just tar up or git export your VCS tree: it requires the chance to add metadata. And while distros have whinged about pbr in a number of contexts, that hasn't been one so far. Downstreams are pretty used to receiving tarballs with generated files in them - as long as they *have the option* to recreate those, so the source material isn't lost. [And for version data, 'grab from git' is a valid answer there']. OTOH perhaps ftpmaster just hasn't noticed and we're about to get a bug report ;) From robertc at robertcollins.net Mon Oct 12 06:37:36 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 17:37:36 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 17:06, Robert Collins wrote: > EWOW, huge thread. > > I've read nearly all of it but in order not to make it massively > worse, I'm going to reply to all the points I think need raising in > one mail :). And a bugfix :) - I didn't link to the docs for the build system interface we have today - https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 06:53:36 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 21:53:36 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 4:49 AM, Paul Moore wrote: [...] > As regards what pip could do, technically you are of course correct > (it's possible, it just needs someone willing to make the code > changes). I don't know, however, if the approach you're proposing fits > with how we currently envisage pip developing. Specifically, my > personal goal [1] is that we get to a point where pip can do all of > the dependency resolution steps, getting to a point where it knows > exactly what it will install, *before* it starts downloading packages, > running build steps, etc. Thanks for stating this so clearly. Unfortunately I just don't see any way this can possibly be achieved. It seems to unambiguously rule out the various "compromise" proposals from the other thread (e.g., your suggestion that packages would have to specify most dependencies statically, but would have the option of adding some extra ones at build time, would not accomplish the goal stated above). To accomplish this, it really is necessary that we be able to *exactly* predict the full dependency set (and also environment specifiers and potential external system requirements and ... -- anything that possibly affects whether a wheel is installable) before we download any packages or build any wheels. But as discussed at length in the other thread, it's a fact of life that the same source release may be configured in different ways that create different resulting dependencies. NumPy is one example of this, but it's hardly unusual -- pretty much any optional dependency on a C library works like this. And these kinds of issues will presumably only get more complicated if we start adding more complex dependency structures (e.g. "provides", per-package abi compatibility tracking, ...). Are you aware of any other systems that have accomplished anything like this? From a quick skim, it looks like .deb, .rpm, gentoo, freebsd ports, and ruby gems all allow for arbitrary code execution inside "source" packages when specifying dependencies, and none of the systems I looked at have the property that you're looking for. This doesn't prove that it's impossible, but... I do see one clear path to accomplish what you want: 1) enable linux wheels on pypi 2) build an autobuilder infrastructure for pypi 3) now that 95% of packages have wheels, flip the switch so that pip ignores sdists when auto-installing dependencies This strategy at least has the advantage that it only requires we do things that have been done before and we know are possible :-). And the alternative is -- what? As far as pip goes, it sounds like we all agree that there's a perfectly sound algorithm for solving the installation problem without access to static dependencies (i.e., wrapping the solve-then-build cycle in a loop); it's just that this leads to a bit more complicated code in pip. But pip clearly has to implement and debug this code no matter what, because we are committed to handling traditional sdists for some years yet. It seems like the best we can hope for is that if we impose these constraints on wheels, and we somehow manage to make this work at all (which seems to be a research problem), then we might eventually be able to replace some working code with some slightly simpler working code. (Plus whatever other benefits there are of having static dependency metadata on pypi, like interesting global static analyses of the package ecosystem.) I know "debian packagers think source wheels with static metadata would be great" was cited in the other thread as an example of an advantage, but note that for numpy, the dependencies in the configuration that debian will want to use are exactly *not* the dependencies in the configuration we'll put on pypi, because the pypi version wants to be self-contained inside the pypi ecosystem but the debian version wants to rely as much as possible on debian-distributed libraries. So even if numpy has a "source wheel" that has static dependency metadata, then this will be exactly what debian *can't* use; they'll need a traditional non-static-metadata source release instead. I dunno -- I'm sure there must exist some other ways forward that don't require dropping the dream of static dependencies. At one extreme, we had a birds-of-a-feature at SciPy this year on "the future of numpy", and the most vocal audience contingent was in favor of numpy simply dropping upstream support for pip/wheels/pypi entirely and requiring all downstream packages/users to switch to conda or building by hand. It sounds like a terrible idea to me. But I would find it easier to believe in the pypi/pip ecosystem if there were some concrete plan for how this all-static world was actually going to work, and that it wasn't just chasing rainbows. -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 12 07:05:01 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 22:05:01 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:48 AM, Marcus Smith wrote: > >> >> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get >> the full metadata for the wheel > > > I wouldn't say "get the full metadata for the wheel". it's not a wheel yet. > `egg_info` run so we can use the pkg_resources api to find the dependencies. > > >> >> Specifically what it does with this is >> extract the setup_requires and install_requires fields > > > specifically, we call `requires` on pkg_resources distribution objects > https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583 > > >> >> 3) eventually it actually builds the package, and this produces a >> wheel (or wheel equivalent) that has its own metadata (which often >> matches the metadata from egg_info in step (2), but not always) > > > "not always"? not following your point > they're 2 different formats, but should contain the same essential > information. > here's the wheel code that does the conversion > https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90 I just meant that due to the fact that you're running two chunks of arbitrary code, there's no way to statically guarantee that the metadata produced in the two calls to 'setup.py' is going to match. In the message here: https://mail.python.org/pipermail/distutils-sig/2015-October/027161.html I linked to some packages whose setup.py's actually report different dependency information in the egg_info and build phases. I'm not saying this is a good or bad or even particularly relevant thing, just observing that it's a fact about how the current system works :-). (And this is part of why my draft PEP that set all this off just has a single atomic "build a wheel" operation instead of splitting this into two phases.) > > >> >> name and version before it runs egg_info, I assume that what this >> means is that it's crucial for pip to have static access to dependency >> information? > > > yes > > >> >> It would be somewhat convenient if sdists did list their binary >> dependencies: > > > not sure about your insertion of "binary" here. > pip is concerned with finding python project dependencies (i.e. name and > version constraints) in the sdist > and then based on the current install environment, it will further constrain > the wheels chosen based on architecture and python implementation. > and to be perfectly clear, none of this deals with non-python OS/Distro > requirements. I didn't mean anything in particular by "binary", sorry if that threw you off. It is probably because I am still finding it useful to think of source packages and binary packages as logically distinct, i.e., a source package is a black box that can produce a binary package and when I have a dependency it names a binary package that I want (which can be obtained either by downloading it or by finding a source package that will produce that binary package and building it). >> 3) if any of the packages-to-be-installed are sdists, then fetch them, >> run egg_info or build them or whatever to get their real dependencies, >> add these to the graph, and go to step 1 > > > this is the pain we don't want in the future. > > since we're simply talking about name/version constraints (not > platform/python), It's hard to conceive that we'd agree on an sdist spec > that didn't include that. > > >> >> Do the relevant pip maintainers >> even read this mailing list? :-) > > > I try : ) Sorry, I realized that may have come across as much more negative than I intended :-). I just realized at the end of typing this whole thing that I wasn't even sure if I was sending it to the right place :-). -- Nathaniel J. Smith -- http://vorpus.org From qwcode at gmail.com Mon Oct 12 07:12:05 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 22:12:05 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > > > it's a fact of life that the same > source release may be configured in different ways that create > different resulting dependencies. NumPy is one example of this, but > it's hardly unusual I've tried to respond to this point twice.. but let me try again. : ) If I'm confused, please help me. aren't the "different resulting dependencies" non-python, non-pypi system dependencies? the dependencies the sdist would cover are the python/pypi dependencies, and they don't usually vary based on building extensions as for representing non-python distro dependencies and build variances, I mentioned before that PEP426 extensions might be used to cover that, but that's still an open question I think.. --Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Oct 12 07:32:10 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 22:32:10 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:12 PM, Marcus Smith wrote: >> >> it's a fact of life that the same >> source release may be configured in different ways that create >> different resulting dependencies. NumPy is one example of this, but >> it's hardly unusual > > > I've tried to respond to this point twice.. but let me try again. : ) > If I'm confused, please help me. > > aren't the "different resulting dependencies" non-python, non-pypi system > dependencies? > > the dependencies the sdist would cover are the python/pypi dependencies, and > they don't usually vary based on building extensions > > as for representing non-python distro dependencies and build variances, I > mentioned before that PEP426 extensions might be used to cover that, but > that's still an open question I think.. Ah, sorry, there was some discussion of this buried deep inside the giant thread :-) For NumPy/SciPy/scikit-learn/etc. we have the problem that we need to somehow distribute the standard linear algebra libraries like BLAS and LAPACK. For e.g. debian packages, or a self-built version on Linux, these are generally treated as distro dependencies -- the user is responsible for making sure that they're (somehow) installed, and then we link to them directly. For wheels, though, we can't assume this -- not just because there's no spec for distro dependencies, but because even if there would then it'd probably be fragile (no guarantee that different distros have compatible ABIs) and it certainly wouldn't work on Windows or OS X. So instead, the current plan is that we're going to drop the libraries inside a wheel and upload it to PyPI: https://mail.python.org/pipermail/distutils-sig/2015-October/027164.html and our wheels on pypi will have an install_requires on this special library wheel. Compared to PEP426-extensions idea, this has the advantages that (a) it doesn't require a PEP, (b) there's no question about whether it will work :-). Another example of where this might arise is if a package [1] wants to optionally use dynd as its array backend instead of numpy [2], so it would need to optionally link to the dynd C API instead of or in addition to the numpy API. But dynd is a young and rapidly changing project, so you might want to support this as an experimental build configuration while still uploading "official" wheels that depend on numpy only. But the configuration that does link to dynd would still want to fetch dynd using python-level dependencies [3]. Does that help? -n [1] This isn't terribly hypothetical, pandas is one of the most popular data processing libraries in the world and they have an experimental branch doing exactly this: https://github.com/pydata/pandas/issues/8643 [2] https://github.com/libdynd/libdynd [3] https://pypi.python.org/pypi/dynd -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 12 07:36:37 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 22:36:37 -0700 Subject: [Distutils] Sources of truth Message-ID: Hi all, Again trying to split out some more focused discussion from the big thread about sdists... One big theme there has been the problem of "sources of truth": e.g. in current sdists, there is a PKG-INFO file that has lots of static metadata in it, but because the "real" version of that metadata is in setup.py, everyone ignores PKG-INFO. A clear desideratum for a new sdist format is that we avoid this problem, by having static metadata that is actually trustworthy. I see two fundamentally different strategies that we might use to accomplish this. In time honored mailing list tradition, these are of course the one that I hear other people advocating and the one that I like ;-). The first strategy is: sdists and the wheels they generate logically share the same metadata; so, we need some mechanism to enforce that whatever static metadata is in the sdist will match the metadata in the resulting wheel. (The wheel might potentially have additional metadata beyond what is in the sdist, but anything that overlaps has to match.) An open question is what this mechanism will look like -- if everyone used distutils/setuptools, then we could write the code in distutils/setuptools so that when it generated wheel metadata, it always copied it directly out of the sdist metadata (when present). But not everyone will use distutils/setuptools, because distutils delenda est. So we need some mechanism to statically analyze an arbitrary build system and prove things about the data it outputs. Which sounds... undecideable. Or we could have some kind of after-the-fact enforcement mechanism, where tools like pip are required -- as the last step when building a wheel from an sdist -- to double-check that all the metadata matches, and if it doesn't then they produce a hard error and refuse to continue. But even this wouldn't necessarily guarantee that PyPI can trust the metadata, since PyPI is not going to run this enforcement mechanism... The second strategy is: put static metadata in both sdists and wheels, but treat them as logically distinct things: the static metadata in sdists is the source of truth for information *about that sdist* (sdist name, sdist version, sdist description, sdist authors, etc.), and the static metadata in wheels is the source of truth for information about that wheel, but we think of these as distinct things and don't pretend that we can statically guarantee that they will match. I mean, in practice, they basically always will match. But IMO making this distinction in our minds leads to clearer thinking. When PyPI needs to know the name/version/description for an sdist, it can still do that; and since we've lowered our ambitions to only finding the sdist name instead of the wheel name, it can actually do it reliably in a totally static way, without having to run arbitrary code to validate this. OTOH pip will always have to be prepared to handle the possibility of mismatch between what it was expecting based on the sdist metadata and what it actually got after building it, so we might as well acknowledge that in our mental model. One potential advantage of this approach is that we might be able to talk ourselves into trusting the existing PKG-INFO as providing static metadata about the sdist, and thus PyPI at least could start trusting it for things like the "description" field, and if we define a new sdist format then it would be possible to generate its static metadata from current setup.py files (e.g. by modifying setuptools's sdist command). Contrast this with the other approach, where getting any kind of static source-of-truth would require rewriting almost all existing setup.py files. The challenge, of course, is that there are a few places where pip actually does need to know something about wheels based on examining an sdist -- in particular name and version and (controversially) dependencies. But this can/should be addressed explicitly, e.g. by writing down a special rule about the name and version fields. -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Mon Oct 12 07:49:47 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Oct 2015 01:49:47 -0400 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On October 12, 2015 at 12:54:07 AM, Nathaniel Smith (njs at pobox.com) wrote: I dunno -- I'm sure there must exist some other ways forward that? don't require dropping the dream of static dependencies. At one? extreme, we had a birds-of-a-feature at SciPy this year on "the future? of numpy", and the most vocal audience contingent was in favor of? numpy simply dropping upstream support for pip/wheels/pypi entirely? and requiring all downstream packages/users to switch to conda or? building by hand. It sounds like a terrible idea to me. But I would? find it easier to believe in the pypi/pip ecosystem if there were some? concrete plan for how this all-static world was actually going to? work, and that it wasn't just chasing rainbows.? FTR, I plan on making some sort of auto builder for PyPI so it?s possible that we can get pip to a point where almost all things it downloads are binary wheels and we don?t need to worry too much about needing to optimize the sdist case. I also think that part of the problem with egg-info and setup.y?s and pip?s attempted use of them is just down to the setup.py interface being pretty horrible and setup.py?s serving too many masters where it needs to be both VCS entry point, sdist metadata entry point, installation entry point, and wheel building entry point. I also think that it would be a terrible idea to have the science stack leave the ?standard? Python packaging ecosystem and go their own way and I think it?d make the science packages essentially useless to the bulk of the non science users. I think a lot of the chasing rainbows like stuff comes mostly from: We have some desires from our experiences but we haven?t yet taken the time to fully flesh out what the impact of those desires are, nor are any of us science stack users (or contributors) to my knowledge, so we don?t deal with the complexities of that much [1]. One possible idea that I?ve thought of here, which may or may not be a good idea: Require packages to declare up front conditional dependencies (I?m assuming the list of dependencies that a project *could* have is both finite and known ahead of time) and let them give groups of these dependencies a name. I?m thinking something similar to setuptools extras where you might be able to put a list of dependencies to a named group. The build interface could include a way for the thing that?s calling the build tool to say ?I require the feature represented by this named group of dependencies?[2], and then the build tool can hard fail if it detects it can?t be build in a way that requires those dependencies at runtime. When the final build is done, it could put into the Wheel a list of all of the additional named groups of dependencies it built with. The runtime dependencies of a wheel would then be the combination of all of those named groups of dependencies + the typical install_requires dependencies. This could possibly even be presented nicely on PyPI as a sort of ?here are the extra features you can get with this library, and what that does to it?s dependencies?. Would something like that solve Numpy?s dependency needs? Or is it the case that you don?t know ahead of time what the entire list of the dependency specifiers could be (or that it would be unreasonable to have to declare them all up front?). I think I recall someone saying that something might depend on something like ?Numpy >= 1.0? in their sdist, but once it?s been built then they?ll need to depend on something like ?Numpy >= $VERSION_IT_WAS_BUILT_AGAINST?. If this is something that?s needed, then we might not be able to satisfy this particular thing. I think I should point out too, that I?m not dead set on having static dependency information inside of a sdist/source wheel/whatever. What I am dead set on having is that all of the metadata *inside* of the sdist/source wheel/whatever should be static and it should include as much as possible which isn?t specific to a particular wheel. This means that things like name, version, summary, description, project URLs, etc. these are obviously (to me anyways) not going to be specific to a wheel and should be kept static inside of the sdist (and then copied over to resulting wheel as static as well [3]) and you simply can?t get information out of a sdist that is inherent to wheels. Obvious (again, to me) examples of data like that are things like build number, ABI the wheel was compiled against, etc. Is the list of runtime dependencies one of the things that are specific to a particular wheel? I don?t know, it?s not obvious to me but maybe it is. [4] [1] That being said, I?d love it if someone who does deal with this things would become more involved with the ?standard? ecosystem so we can have experts who deal with that side of things as well, because I do think it?s an important use case. [2] And probably: ?Even if you can do the feature suggested by this named group of dependencies, I don?t want it? [3] This rule should be setup so that we can have an assertion put into place that this data will remain the exact same when building a wheel from a sdist. [4] I think there?s possibly some confusion in what is causing problems. I think that the entirety of ``setup.py`` is full of problems with executing in different environments and isn?t very well designed to enable robust packages. A better designed interface would likely resolve a good number of the problems that pip currently has either way. It is true however that needing to download the sdist prior to resolving dependencies makes things a lot slower, but I also think that doing it correctly is more important than doing it quickly. We could possibly resolve some of this by pushing more people to publish Wheels, expanding the cases where Wheels are possible to be uploaded, and creating a wheel builder service. These things could make it possible to reduce the situations where you *need* to download + build locally to do a dependency resolution. I think that something like the pip wheel building cache can reduce the need for this as well, since we?ll already have built wheels available locally in the wheel cache that we can query for dependency information.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Mon Oct 12 07:55:03 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 22:55:03 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > > > So instead, the current plan is that we're going > to drop the libraries inside a wheel and upload it to PyPI: > aha... ok, now it's clearer where you're coming from. but using what platform in the wheel tag? linux wheels are blocked currently regardless of the tagging though, I'm not so sure about using PyPI like this to hold all these "python" wrapper packages for binary extensions -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Oct 12 08:14:43 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 23:14:43 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:55 PM, Marcus Smith wrote: >> >> So instead, the current plan is that we're going >> to drop the libraries inside a wheel and upload it to PyPI: > > aha... ok, now it's clearer where you're coming from. > but using what platform in the wheel tag? > linux wheels are blocked currently Windows is the first goal, likely OS X after that (OS X provides a bit more in the base system so distributing these external dependencies isn't as urgent, but it would be good eventually), and yeah, doing this for Linux would obviously require unblocking Linux wheels. (We do have a plan for unblocking them, but the critical person is a bit busy so I'm not sure how long it will be before it can be unveiled :-). It's not really a secret, the plan is just: take advantage of Continuum and Enthought's hard-won knowledge from distributing hundreds of thousands of pre-built python package binaries on Linux to define a named "common Linux platform" abi tag that is known to be abi-compatible with >99.99% of Linux installs out there. This doesn't prevent the use of other abi tags -- e.g. distro-specific ones -- or the development of more detailed metadata about "distro dependencies", and wouldn't require that every single Linux box be willing to install wheels with this abi tag, but it would at least bring Linux support up to parity with the other platforms.) > regardless of the tagging though, I'm not so sure about using PyPI like this > to hold all these "python" wrapper packages for binary extensions If someone else invents a better solution before this is implemented then of course we'd switch to that instead... -n -- Nathaniel J. Smith -- http://vorpus.org From robertc at robertcollins.net Mon Oct 12 08:00:51 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 19:00:51 +1300 Subject: [Distutils] Sources of truth In-Reply-To: References: Message-ID: On 12 October 2015 at 18:36, Nathaniel Smith wrote: > Hi all, > > Again trying to split out some more focused discussion from the big > thread about sdists... > > One big theme there has been the problem of "sources of truth": e.g. > in current sdists, there is a PKG-INFO file that has lots of static > metadata in it, but because the "real" version of that metadata is in > setup.py, everyone ignores PKG-INFO. > > A clear desideratum for a new sdist format is that we avoid this > problem, by having static metadata that is actually trustworthy. I see > two fundamentally different strategies that we might use to accomplish > this. In time honored mailing list tradition, these are of course the > one that I hear other people advocating and the one that I like ;-). > The first strategy is: sdists and the wheels they generate logically > share the same metadata; so, we need some mechanism to enforce that This is false: they don't share the same metadata. Some portions are the same, but deps, supported platforms, those will differ (and perhaps more than that). In particular, an sdist doesn't have a dependency on an ABI, and a wheel doesn't have a dependency on an API. Some APIs are ABIs (approximately true for all pure Python packages, for instance), but some are not (numpy). > The second strategy is: put static metadata in both sdists and wheels, > but treat them as logically distinct things: the static metadata in > sdists is the source of truth for information *about that sdist* > (sdist name, sdist version, sdist description, sdist authors, etc.), > and the static metadata in wheels is the source of truth for > information about that wheel, but we think of these as distinct things > and don't pretend that we can statically guarantee that they will > match. I mean, in practice, they basically always will match. The analgous current data won't match for pbr using packages when we fix https://bugs.launchpad.net/pbr/+bug/1502692 (older pip's don't support PEP-426 environment markers, but don't error when they are used either, leading to silent failure to install dependencies). Now, you might say 'hey, but the new shiny will support markers from day one'. Well the problem is backwards compat: we're going to have future things that change, and the more we split things out the more the changes are likely to need skewed results like this approach to deal with it. ... > the sdist name instead of the wheel name, it can actually do it but the sdist and the wheel have to have the same name- or do you mean the filename on disk, vs the distribution name? > reliably in a totally static way, without having to run arbitrary code > to validate this. OTOH pip will always have to be prepared to handle > the possibility of mismatch between what it was expecting based on the > sdist metadata and what it actually got after building it, so we might > as well acknowledge that in our mental model. > > One potential advantage of this approach is that we might be able to > talk ourselves into trusting the existing PKG-INFO as providing static > metadata about the sdist, and thus PyPI at least could start trusting > it for things like the "description" field, and if we define a new The challenge is the 40K broken packages up there on PyPI. Basically pip has a bugfix for any of: sdists built using distutils sdists built using random build systems that don't understand what an sdist is (e.g. automake) sdists built using versions of setuptools that had a bug in this area There is no corrective mechanism for broken packages other than route-around-it-while-you-ask-the-author-to-upload-a-fix. So I think to tackle the 'please trust the metadata in the sdist' problem, one needs to have a graceful ramp-up of that trust with robust backoff mechanisms that don't involve 50% of PyPI users hating on that one old project in the corner everyone has a dep on but that is actually moribund and not doing uploads. I can imagine several such routes, including a crowdsourced blacklist - but its going to be (like we're dealing with with the automatic wheel cache already) years of bug reports until things age out. > sdist format then it would be possible to generate its static metadata > from current setup.py files (e.g. by modifying setuptools's sdist > command). Contrast this with the other approach, where getting any > kind of static source-of-truth would require rewriting almost all > existing setup.py files. We already generate static metadata from current setup.py files: setup.py egg_info does precisely that. There, bug fixed ;). > The challenge, of course, is that there are a few places where pip > actually does need to know something about wheels based on examining > an sdist -- in particular name and version and (controversially) > dependencies. But this can/should be addressed explicitly, e.g. by > writing down a special rule about the name and version fields. I'm sorry, I don't follow. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 09:23:46 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 00:23:46 -0700 Subject: [Distutils] Sources of truth In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 11:00 PM, Robert Collins wrote: > On 12 October 2015 at 18:36, Nathaniel Smith wrote: [...] >> the sdist name instead of the wheel name, it can actually do it > > but the sdist and the wheel have to have the same name- or do you mean > the filename on disk, vs the distribution name? I mean the distribution name - there's no way to guarantee that building foo-1.0.zip won't spit out bar-7.4.whl, where by "no way" I mean "it's literally undecideable". I mean, if someone actually did this it would be super weird and we would all shun them, but our code and specs still need to be prepared for the possibility. IIUC this is why PyPI can't trust PKG-INFO: 99.9% of the time the metadata in PKG-INFO matches what you will get when you run setup.py, but right now PyPI wants to know what setup.py will do, and there's no way to know if it will be the same as what PKG-INFO says, so it just doesn't trust PKG-INFO. OTOH if we redefine PyPI's goal as being, figure out what's in PKG-INFO (or whatever replaces it), and declare that it's okay (for PyPI's purposes) if that doesn't match what the build system will eventually do, then that's a viable way forward. >> reliably in a totally static way, without having to run arbitrary code >> to validate this. OTOH pip will always have to be prepared to handle >> the possibility of mismatch between what it was expecting based on the >> sdist metadata and what it actually got after building it, so we might >> as well acknowledge that in our mental model. >> >> One potential advantage of this approach is that we might be able to >> talk ourselves into trusting the existing PKG-INFO as providing static >> metadata about the sdist, and thus PyPI at least could start trusting >> it for things like the "description" field, and if we define a new > > The challenge is the 40K broken packages up there on PyPI. Basically > pip has a bugfix for any of: > sdists built using distutils > sdists built using random build systems that don't understand what an > sdist is (e.g. automake) > sdists built using versions of setuptools that had a bug in this area > > There is no corrective mechanism for broken packages other than > route-around-it-while-you-ask-the-author-to-upload-a-fix. IIUC what PyPI wants to do with PKG-INFO is read out stuff like the description and trove classifiers fields. Are there really 40K sdists on PyPI that have PKG-INFO files and where those files contain incorrect descriptions and so forth? I mean, obviously someone would have to check :-) But it seems unlikely, since almost everyone uploads by running 'sdist upload' or twine or something similarly automated. > So I think to tackle the 'please trust the metadata in the sdist' > problem, one needs to have a graceful ramp-up of that trust with > robust backoff mechanisms that don't involve 50% of PyPI users hating > on that one old project in the corner everyone has a dep on but that > is actually moribund and not doing uploads. I can imagine several such > routes, including a crowdsourced blacklist - but its going to be (like > we're dealing with with the automatic wheel cache already) years of > bug reports until things age out. > >> sdist format then it would be possible to generate its static metadata >> from current setup.py files (e.g. by modifying setuptools's sdist >> command). Contrast this with the other approach, where getting any >> kind of static source-of-truth would require rewriting almost all >> existing setup.py files. > > We already generate static metadata from current setup.py files: > setup.py egg_info does precisely that. There, bug fixed ;). I'm pretty sure that merely making it so 'setup.py sdist' created a file that contained the output from egg_info would not solve the current problem. That's pretty much exactly what the existing PKG-INFO *is*, isn't it? Yet apparently no-one trusts it. >> The challenge, of course, is that there are a few places where pip >> actually does need to know something about wheels based on examining >> an sdist -- in particular name and version and (controversially) >> dependencies. But this can/should be addressed explicitly, e.g. by >> writing down a special rule about the name and version fields. > > I'm sorry, I don't follow. E.g., we can document that if you have a sdist foo-1.0, then pip and similar tools will expect this to generate a foo-1.0 wheel (but be prepared to do something sensible if this doesn't happen, like give an error message or whatever). That's really all pip needs, right? -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 12 10:24:35 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 01:24:35 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:49 PM, Donald Stufft wrote: > FTR, I plan on making some sort of auto builder for PyPI so it?s possible > that we can get pip to a point where almost all things it downloads are > binary wheels and we don?t need to worry too much about needing to optimize > the sdist case. That would be 1000% awesomesauce. > I also think that part of the problem with egg-info and > setup.y?s and pip?s attempted use of them is just down to the setup.py > interface being pretty horrible and setup.py?s serving too many masters > where it needs to be both VCS entry point, sdist metadata entry point, > installation entry point, and wheel building entry point. Yeah, I'm glad that Robert linked https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface because I hadn't seen it, and it's great that it's documented, but what reading that documentation mostly convinced me is that I definitely could not write a new implementation of that interface and get it right without significant pain and trial-and-error :-). I mean... you have to support both --single-version-externally-managed enabled and disabled? develop has to know the right arcane magic to get the directory added to sys.path? (how do you know where in sys.path to add the magic?) what is egg_info even supposed to do -- the linked to docs don't actually define what egg metadata looks like, and IIRC it's not standardized, is it? (as opposed to the dist-info used in wheels) Like, obviously all that should be needed to implement a new build system is the ability to generate a wheel, right? Wheels are well-documented and one honking great idea. My current best guess at where we want to end up with sdists is: - there's a static file like the _pypackage/_pypackage.cfg in my original proposal (ignore the name, we can change it, I don't care) - it's written in some simple well-specified language like json or toml (if you look at the spec, toml is basically json encoded in an INI-style syntax -- it's definitely nowhere near as complex as yaml) - it includes static metadata for name, version (required for pypi uploads, optional otherwise to accomodate VCS checkouts), description, long_description, authors, URLs, trove, etc. etc., but not install_depends - it also includes the setup-dependencies / setup-dependencies-dynamic / build-wheel / build-in-place entries/hooks from my original proposal (or the moral equivalent) - it lives inside the single top-level directory next to the source code (instead of having a meta/ + source/ split, because that way it works for VCS checkouts too and AFAICT there aren't any other advantages to this split. We can afford to steal a single name -- everyone use used to having a setup.py or Makefile or Gemfile or whatever cluttering up the top-level of their source, and this is just a replacement for that.) That gives you static metadata for PyPI and solves the setup_depends problem while simultaneously cleaning up the "fugly" setup.py interface so that it becomes easy to write your own build system. (I think instead of just arguing about tests, we should have big debates here about which of the 3-4 highly-regarded competing build systems we should be recommending to newbies. But this will only happen if we make designing a new build system something that dozens of people do and a few take seriously, which requires making it accessible.) I *think* the above interface also makes it possible to write a shim in either direction (a setup.py that speaks the current "fugly" interface to old versions of pip while using the new files to actually do its work, or a new-style sdist that delegates all the actual work to setup.py), and even for setuptools to start generating sdists that are simultaneously both old-style and new-style for compatibility. So I think we can at least solve the metadata-for-PyPI problem, the setup_requires problem, and the let-a-thousand-build-systems-bloom problems now, without solving the static-install_requires-in-sdists problem (but also without creating any particular obstacles to solving it in the future, if it's even possible). And in the mean time we can keep working on autobuilders and linux wheels that will remove much of the demand for automagic sdist building -- it's difficult to make predictions, especially about the future, so who knows, maybe in the end we'll decide we just don't care that much about static-install_requires-in-sdists. > I also think that it would be a terrible idea to have the science stack > leave the ?standard? Python packaging ecosystem and go their own way and I > think it?d make the science packages essentially useless to the bulk of the > non science users. I think a lot of the chasing rainbows like stuff comes > mostly from: We have some desires from our experiences but we haven?t yet > taken the time to fully flesh out what the impact of those desires are, nor > are any of us science stack users (or contributors) to my knowledge, so we > don?t deal with the complexities of that much [1]. > > One possible idea that I?ve thought of here, which may or may not be a good > idea: > > Require packages to declare up front conditional dependencies (I?m assuming > the list of dependencies that a project *could* have is both finite and > known ahead of time) and let them give groups of these dependencies a name. > I?m thinking something similar to setuptools extras where you might be able > to put a list of dependencies to a named group. The build interface could > include a way for the thing that?s calling the build tool to say ?I require > the feature represented by this named group of dependencies?[2], and then > the build tool can hard fail if it detects it can?t be build in a way that > requires those dependencies at runtime. When the final build is done, it > could put into the Wheel a list of all of the additional named groups of > dependencies it built with. The runtime dependencies of a wheel would then > be the combination of all of those named groups of dependencies + the > typical install_requires dependencies. This could possibly even be presented > nicely on PyPI as a sort of ?here are the extra features you can get with > this library, and what that does to it?s dependencies?. > > Would something like that solve Numpy?s dependency needs? The answer is, I don't know! I am not at all confident in my ability to anticipate every possible difficult case, and NumPy is really not the only complicated library out there, it just gets most of the press. :-) Your idea definitely sounds like something worth exploring as part of the general debate around things like "Provides:" and extras, but I'd be hesitant to commit to locking something in right now. Also: > Or is it the case > that you don?t know ahead of time what the entire list of the dependency > specifiers could be (or that it would be unreasonable to have to declare > them all up front?). I think I recall someone saying that something might > depend on something like ?Numpy >= 1.0? in their sdist, but once it?s been > built then they?ll need to depend on something like ?Numpy >= > $VERSION_IT_WAS_BUILT_AGAINST?. If this is something that?s needed, then we > might not be able to satisfy this particular thing. Yes, it is true right now that packages that use the NumPy C API should have a wheel dependency on Numpy >= $VERSION_IT_WAS_BUILT_AGAINST. We'd like it if wheels/pip had even more flexible metadata in this regard that could express things like "package A needs numpy ABI version 3, and numpy 1.8, 1.9, and 1.10 say that they can provide ABI version 3, but 1.11 and 1.7 don't", but we shouldn't worry about the exact details of that kind of system right now -- I'm thinking any more flexible solution to the NumPy ABI problem would be at least as problematic for pip's sdist resolution rules as the current situation, and it sounds like the current situation is already bad enough to rule out a lot of cases. -n -- Nathaniel J. Smith -- http://vorpus.org From robertc at robertcollins.net Mon Oct 12 10:49:10 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 21:49:10 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 19:23, Ralf Gommers wrote: > > > On Mon, Oct 12, 2015 at 6:37 AM, Robert Collins > wrote: >> >> On 12 October 2015 at 17:06, Robert Collins >> wrote: >> > EWOW, huge thread. >> > >> > I've read nearly all of it but in order not to make it massively >> > worse, I'm going to reply to all the points I think need raising in >> > one mail :). >> >> And a bugfix :) - I didn't link to the docs for the build system >> interface we have today - >> >> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > > > From that link: > """ > In order for pip to install a package from source, setup.py must implement > the following commands: > ... > The install command should implement the complete process of installing the > package to the target directory XXX. > """ > That just sounds so wrong. You want the build system to build, not install. Right so - with the automatic wheel cache we added, that is only used if building a wheel failed. So you can consider it to be legacy cruft - the preferred interface is build-a-wheel then install-that-wheel. > And if "install" actually means "build to a tempdir so pip can copy it over > it to its final location", then how does that address something like > installing docs to a different dir than the package itself? Well, how do we install docs to a different dir with wheels? If we've got an answer for that, I think we're in a good position to figure something out for install (even if that is 'fix your package so it can build wheels). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Mon Oct 12 11:53:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Oct 2015 10:53:33 +0100 Subject: [Distutils] Sources of truth In-Reply-To: References: Message-ID: On 12 October 2015 at 08:23, Nathaniel Smith wrote: > I mean the distribution name - there's no way to guarantee that > building foo-1.0.zip won't spit out bar-7.4.whl, where by "no way" I > mean "it's literally undecideable". I mean, if someone actually did > this it would be super weird and we would all shun them, but our code > and specs still need to be prepared for the possibility. But the whole point of a spec is to *make* that guarantee. Pip doesn't expect to ever work with random code that can do literally anything, that's silly. Pip works with code that conforms to a spec. At the moment, the spec is "it does what setuptools/distutils does", which is a lousy spec because (a) nobody else can implement the spec without reading the huge mess that is distutils/setuptools, (b) it's too broad (we don't *really* need every corner case that distutils covers) and (c) distutils/setuptools don't satisfy the needs of a number of communities, notably the scientific community. So we want to define a more usable spec for the future. Pip will always have to deal with backward compatibility, and all of the hacks that implies, but we can at least say "if a project declares that it follows the new spec, we can rely on certain things". One of those things we can rely on is that building foo-1.0.zip won't spit out bar-7.4.whl - that would be part of the contract of that spec. The debate is over on the one hand, we want to be able to rely on when writing packaging tools, vs what flexibility we want to have when writing build tools. But it's not over ignoring the other use cases, it's about agreeing a *contract* that satisfies the needs of *both* packaging tools and build tools. At the moment, the dial is 100% over to the packaging side - builders have zero flexibility, the rule is "use distutils and setuptools". That doesn't feel like zero flexibility, because distutils/setuptools let you do a lot of things. But when you do finally hit the limits you're stopped cold. It also is a bad choice for packaging tools, because distutils and setuptools are a dreadful API for automation. So nobody wins. Two suggestions that have been around for a long while that give more flexibility to build tools while at the same time giving packaging tools a better interface to work with are: 1. Declarative setup - you have to put your metadata in file X in format Y, and it's static. But when you do, your build tool can do whatever it wants as long as it spits out binaries that confirm to your declared metadata. The downside to this is that it's totally static, and developers don't like that ("what if I want to generate my version number from my VCS tags?") 2. A fixed API - we document a set of commands that packaging tools can use and build tools have to provide. This is where we talk about documenting the distutils commands that pip relies on ("setup.py egg-info", "setup.py install", "setup.py develop"...) This one usually falls down because nobody likes the idea of writing a "dummy" setup.py that translates the interface, and because no-one has done anything more with this idea than suggest documenting the setuptools commands that pip *currently* uses, even though these are probably a lousy API design. It also does very little for packaging tools *other* than pip (for example, it does nothing for PyPI, which cannot accept an API that requires running user-supplied code). The key drivers here have been about defining something that packaging tools can use effectively, and build tools can cope with being required to provide. While not constraining *either* type of tool beyond the minimum needed for the contract. Paul From p.f.moore at gmail.com Mon Oct 12 12:06:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Oct 2015 11:06:30 +0100 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 12 October 2015 at 07:14, Nathaniel Smith wrote: > On Sun, Oct 11, 2015 at 10:55 PM, Marcus Smith wrote: >>> >>> So instead, the current plan is that we're going >>> to drop the libraries inside a wheel and upload it to PyPI: >> >> aha... ok, now it's clearer where you're coming from. >> but using what platform in the wheel tag? >> linux wheels are blocked currently > > Windows is the first goal, likely OS X after that (OS X provides a bit > more in the base system so distributing these external dependencies > isn't as urgent, but it would be good eventually), and yeah, doing > this for Linux would obviously require unblocking Linux wheels. I'm running out of time to keep up with this mega-thread, so I'm starting to only skim at this point. Apologies if I've missed something that has already been said as a result... My view is that if you distribute a library (e.g., openblas) in a wheel, then it's essentially a "Python dependency" at that point, insofar as it can participate in the standard package dependency process in exactly the same way as (say) numpy itself does. So if you use that approach, I no longer see *any* reason why you can't work within the existing dependency process. Is that right? > We do have a plan for unblocking them, but the critical person is a > bit busy so I'm not sure how long it will be before it can be unveiled > :-). Obviously, as Marcus said, you need Linux wheels unblocked, and that means being able to specify compatibility cleanly, but as you say here, you're looking at that. Paul From robertc at robertcollins.net Mon Oct 12 12:01:53 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 23:01:53 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 21:15, Thomas Kluyver wrote: > On 12 October 2015 at 05:37, Robert Collins > wrote: >> >> And a bugfix :) - I didn't link to the docs for the build system >> interface we have today - >> >> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > > > I'm happy to see that the command line interface pip requires from setup.py > is now documented. But the first thing it describes is the egg_info command, > and the description of what that involves is basically 'whatever setuptools > does'. Egg info is not any kind of standard, AFAIK - unlike dist info. One https://www.python.org/dev/peps/pep-0314/ describes PKG_INFO, which can in principle include dependency data. I don't know why they're not being fully populated - I suspect its another case of forward-looking-spec-not-quite-right-and-thus-never-used. Someone with more history on Python packaging will have to fill that in. One option would be to push on that. > of my main goals in writing flit is to build wheels without involving > setuptools at all, so I'm probably never going to implement that. Well thats fair enough. but: pip uses pkg_resources to find out whats installed and what it depends on, and that pulls out of egg-info on the file system for non-wheel-installs (dist-info for wheel installs). So, if that isn't well enough documented, it would be entirely reasonable to make a PEP to capture it so that you can emit it with confidence. There's already buy-in from everyone in the ecosystem to use PEPs to document interoperability impacting changes across these tools, so I can't imagine it being a contentious proposal. > I also don't want to have a setup.py in the VCS at all, because it's an > invitation for people to run setup.py whatever, and then file bug reports > that it doesn't do exactly what setuptools/distutils would do. Ok, so I understand that frustration. You're going to get those through pip too though - right now there's a giant escape clause in pip where arbitrary options are passed through to setup.py, because we don't have semantic modelling of all the things that people want to do. And until thats achieved - and the options aged out - folk are going to use those options and then be surprised when flit using packages don't behave the same way. > This is what I like about Nathaniel's proposal. By looking for a new file, > it clearly indicates that this is a new style Python source tree, without > potential confusion over what setup.py does, and it lets us specify a simple > build system interface based on standardised pieces like wheels rather than > messing around with the details of setuptools. The problem with 'simple' is that we've got a rich (not just old: fully featured) interface. Building something that isn't complected out from behind that is both valuable and hard. An 85% solution or 95% solution will retain the trap-doors that let folk tickle the system from the outside, and thats going to lead back to the same bug reports you don't want IMO :/. I'm not against a spec that avoids the need for each tree to have a setup.py - I just don't think its going to be the end of the story in any deep fashion :). Here's a draft one I'd be happy with (with no boilerplate or anything). For commands we can define a small number environment variables like e.g. OUTPUT_PATH and PYTHON have those supplied when invoking the command. Python packaging config in $root/pypa.yaml Defined keys: ---- version: # not needed yet, since its a new file, defaults to 1. setup-requires: - requirement - requirement[extra] - requirement:markers - requirement[extra]:markers build-tool: # basic command that will spit a hunk of json back to the caller defining the # commands to use with the build tool. ---- Build tool output json (in yaml format for ease of reading): version: # not needed yet, defaults to 1 egg-info: # comand to run to generate an egg-info directory # only used if dist-info is not defined dist-info: # command to run to generate a dist-info directory wheel: # command to build a wheel develop: # command to do an in-place installation install: # command to do a direct installation # only used if wheeling failed provided-by: # the requirement that provides the build system. # this is used to facilitate caching the build tool output # so that we don't need to probe everytime. (e.g. if the # same dependency version is present in two cases # we can presume the output - I threw this in as an # obvious tweak for large dependency chains but it is # probably not worth it for typical 10-20 package things -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ralf.gommers at gmail.com Mon Oct 12 08:23:03 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 12 Oct 2015 08:23:03 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Mon, Oct 12, 2015 at 6:37 AM, Robert Collins wrote: > On 12 October 2015 at 17:06, Robert Collins > wrote: > > EWOW, huge thread. > > > > I've read nearly all of it but in order not to make it massively > > worse, I'm going to reply to all the points I think need raising in > > one mail :). > > And a bugfix :) - I didn't link to the docs for the build system > interface we have today - > https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > >From that link: """ In order for pip to install a package from source, setup.py must implement the following commands: ... The install command should implement the complete process of installing the package to the target directory XXX. """ That just sounds so wrong. You want the build system to build, not install. And if "install" actually means "build to a tempdir so pip can copy it over it to its final location", then how does that address something like installing docs to a different dir than the package itself? +1 for your main point of focusing more on enabling other build systems though. Ralf > -Rob > > > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Mon Oct 12 10:15:00 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 12 Oct 2015 09:15:00 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 05:37, Robert Collins wrote: > And a bugfix :) - I didn't link to the docs for the build system > interface we have today - > https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > I'm happy to see that the command line interface pip requires from setup.py is now documented. But the first thing it describes is the egg_info command, and the description of what that involves is basically 'whatever setuptools does'. Egg info is not any kind of standard, AFAIK - unlike dist info. One of my main goals in writing flit is to build wheels without involving setuptools at all, so I'm probably never going to implement that. I also don't want to have a setup.py in the VCS at all, because it's an invitation for people to run setup.py whatever, and then file bug reports that it doesn't do exactly what setuptools/distutils would do. This is what I like about Nathaniel's proposal. By looking for a new file, it clearly indicates that this is a new style Python source tree, without potential confusion over what setup.py does, and it lets us specify a simple build system interface based on standardised pieces like wheels rather than messing around with the details of setuptools. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Oct 12 14:52:12 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Oct 2015 12:52:12 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: I could be wrong but if egg-info actually generated a dist-info directory it would probably still work. On Mon, Oct 12, 2015 at 8:50 AM Thomas Kluyver wrote: > On 12 October 2015 at 05:37, Robert Collins > wrote: > >> And a bugfix :) - I didn't link to the docs for the build system >> interface we have today - >> >> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface >> > > I'm happy to see that the command line interface pip requires from > setup.py is now documented. But the first thing it describes is the > egg_info command, and the description of what that involves is basically > 'whatever setuptools does'. Egg info is not any kind of standard, AFAIK - > unlike dist info. One of my main goals in writing flit is to build wheels > without involving setuptools at all, so I'm probably never going to > implement that. > > I also don't want to have a setup.py in the VCS at all, because it's an > invitation for people to run setup.py whatever, and then file bug reports > that it doesn't do exactly what setuptools/distutils would do. > > This is what I like about Nathaniel's proposal. By looking for a new file, > it clearly indicates that this is a new style Python source tree, without > potential confusion over what setup.py does, and it lets us specify a > simple build system interface based on standardised pieces like wheels > rather than messing around with the details of setuptools. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Oct 12 15:12:46 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 12 Oct 2015 08:12:46 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 11, 2015 11:07 PM, "Robert Collins" wrote: > > EWOW, huge thread. > > I've read nearly all of it but in order not to make it massively > worse, I'm going to reply to all the points I think need raising in > one mail :). > > Top level thoughts here, more point fashion with only rough editing > below the fold. > > I realise many things - like the issue between different wheels of the > same package consuming different numpy abis - have been touched on, > but AFAICT they are entirely orthogonal to the proposal, which was to > solve 'be able to use arbitrary build systems and still install with > pip'. > > Of the actual problems with using arbitrary build systems, 99% of them > seem to boil down to 'setup-requires isn't introspectable by pip > (https://github.com/pypa/pip/issues/1820 ). - If it was, then > alternative build systems could be depended on reasonably; and the > mooted thunk from setuptools CLI to arbitrary build system would be > viable. > > It is, in principle a matter of one patch to teach pip *a* way to do > this (and then any and all build systems that want to can utilise it). > https://github.com/rbtcollins/pip/tree/declarative is a POC I did - my > next steps on that were to discuss the right ecosystem stuff for it - > e.g. should pip consume it via setuptools, or should pip support it as > *the way* and other systems including setuptools can choose to use it? as a standard RDF graph representation, JSON-LD would be uniquely portable here. "PEP 426: Define a JSON-LD context as part of the proposal" https://github.com/pypa/interoperability-peps/issues/31 > > A related but separate thing is being able to *exclusively* install > things without setuptools present - I've filed > https://github.com/pypa/pip/issues/3175 about that, but I think its > -much- lower priority than reliably enabling third party build tools. peep may not need setuptools? * SHA256 * --no-deps * https://pypi.python.org/pypi/peep * wheels > > -Rob > > ---- > > > " > solved many of the hard problems here -- e.g. it's no longer necessary > that a build system also know about every possible installation > configuration -- so pretty much all we really need from a build system > is that it have some way to spit out standard-compliant wheels. > " > > Actually pip still punts a *lot* here - we have bypasses to let things > like C compiler flags be set during wheel build, and when thats done > we don't cache the wheels (or even try to build wheels). > > " > While ``distutils`` / ``setuptools`` have taken us a long way, they > suffer from three serious problems: ... > (c) you are forced to use them anyway, because they provide the > standard interface for installing python packages expected by both > users and installation tools like ``pip``." > > I don't understand the claim of (c) here - its entirely possible to > write a package that doesn't use setuptools and have it do the right > thing - pip uses a subprocess to drive package installation, and the > interface is documented. The interface might be fugly as, but it > exists and works. It is missing setup-requires handling, but so is > setup.py itself. The only thing we'd really need to do AFAICT is make > our setuptools monkeypatching thunk handle setuptools not being > installed (which would be a sensible thing to Just Do anyhow). > > " > - query for build dependencies > - run a build, producing wheels as output > - set up the current source tree so that it can be placed on > ``sys.path`` in "develop mode" > " > > So we have that already. setup.py egg-info, setup.py bdist_wheel, > setup.py develop. > > "A version 1-or-greater format source tree can be identified by the > presence of a file ``_pypackage/_pypackage.cfg``. > " > > I really don't like this. Its going to be with us forever, and its > intrusive (its visible), and so far isn't shown to be fixing anything. > > > "to scatter files around willy-nilly never works, so we adopt the > convention that names starting with an underscore are reserved for > official use, and non-underscored names are available for > idiosyncratic use by individual projects." > > I can see the motivation here, but is it really solving a problem we have? > > > On the specifics of the format: I don't want to kibbitz over strawman > aspects at this point. > > Having the extension mechanism be both pip specific and in Python > means that we're going to face significant adoption issues: the former > because pip is not by any means the only thing around - and some > distros have until very recently been actively hostile to pip (which > in turn means we need to wait a decade or two for them to age-out and > stop being used). The latter because we'll face all the headaches of > running arbitrary untrusted code and dealing with two deps with > different versions of the same hook and so on: I think its an > intrinsically unsafe design. > > @dstufft "problem with numpy.distutils, as I know you?re aware!). We > could do a minimal extension and add another defacto-ish standard of > allowing pip and setuptools to process additional setup_requires like > arguments from a setup.cfg to solve that problem though. The flip side > to this is that since it involves new capabilities in > pip/setuptools/any other installer is that it you?ll have several > years until you can depend on setup.cfg based setup_requires from > being able to be depended on. > " > > Well. For *any* proposal that involves modifying pip, we have to > assume that all existing things keep working, and that anyone wanting > to utilise the new thing will have to either a) include a local > compatibility thunk, or b) error when being used from a too-old > toolchain. I don't think that should really be a factor in design > since its intrinsic to the quagmire. > > "Longer term, I think the answer is sdist 2.0 which has proper > metadata inside of it (name, version, dependencies, etc) but which > also includes a hook like this PEP has to specify the build system > that should be used to build a wheel out of this source distribution." a composed JSON-LD document indicating provenance (who, what, when) for each part of the build chain [VCS archive, egg-info, sdist, wheel, bdist] pydist.jsonld? > > Any reason that can't just be setup.cfg ? > > @Daniel "I thought Robert Collins had a working setup-requires > implementation already? I have a worse but backwards compatible one > too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py" - > https://github.com/rbtcollins/pip/tree/declarative - I'll be updating > that probably early next year at this rate - after issue-988 anyhow. > The issue with your approach is that pip doesn't handle having > concurrent installs done well - and in fact it will end up locking its > environment somehow. > > @Paul " > I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that" - > thats where the draft PEP tennessee and I start is aimed - at making > those libraries be metadata, not platform tags. > > @Chris " > A given package might depend on numpy, as you say, and it may work > with all numpy versions 1.6 to 1.9. Fine, so we specify that in > install_requires. And this shodl be the dependency in the sdist, too. > If the package is pur python, this is fine and done. > > But if the package has some extensions code that used the numpy C API > ( a very common occurrence), then when it is built, it will only work > (reliably) with the version of numpy it was built with. > > So the project itself, and the sdist depend on numpy >=1.6, but a > build binary wheel depends on numpy == 1.7 (for instance). > > Which requires a binary (wheel) dependency that is somewhat different > than the source dependency. > " - so yes, that is where bdist_wheel should be creating different > metadata for that wheel. The issue that arises is that we need unique > file names so that they can coexist on PyPI or local archives - which > is where wheel tags come in. I'd be in favour of not using semantic > tags for this - rather hash the deps or something and just make a > unique file name. Use actual metadata for metadata. > > @Nathaniel "I know that one unpleasant aspect of the current design is that the > split between egg-info and actual building creates the possibility for > time-of-definition-to-time-of-use bugs, where the final wheel > hopefully matches what egg-info said it would, but in practice there > could be skew. (Of course this is true in any system which represents" > - actually see https://bugs.launchpad.net/pbr/+bug/1502692 for a bug > where this 'skew' is desirable: for older environments we want > tailored deps with no markers, for anything supporting markers we want > them - so the wheel will have markers and egg_info won't. > > @Nathaniel " > (Part of the intuition for the last part is that we also have a > not-terribly-secret-conspiracy here for writing a PEP to get Linux > wheels onto PyPI and at least achieve feature parity with Windows / OS > X. Obviously there will always be weird platforms -- iOS and FreeBSD > and Linux-without-glibc and ... -- but this should dramatically reduce > the frequency with which people need sdist dependencies.)" - I think a > distinction between sdist and binary names for dependencies would be a > terrible mistake. It will raise complexity for reasoning and > describing things without solving any concrete problem that I can see. > > @Nathaniel "I guess to make progress in this conversation I need some > more detailed explanations. I totally get that there's a long history > of thought and conversations behind the various assertions here like > "a sdist is fundamentally different from a VCS checkout", "there must > be a 1-1 mapping between sdists and wheels", "pip needs sdists that > have full wheel metadata in static form", and I'm barging in from the > outside with no context, but I literally have no idea why the specific > design features you're asking for are desirable or even viable. Right > now if I were to try and write the PEP you're asking for, then the > rationale section would just be "because Donald said so" over and over > :-). I couldn't write the motivation section, because I don't know any > problems that the PEP you're describing would fix for me as a package > author (which doesn't mean they don't exist, but!)." -- VCS trees are > (generally) by-humans for humans. They are the primary source of data > and can do thinks like inferring versions from commit data. sdists are > derived from the VCS tree and can include extra data (such as > statically defined version data). Wheels are derived from a tree on > disk and can (today) be built from either VCS trees or sdists. I'm not > sure that forcing an sdist step is beneficial - the egg-info step we > have today is basically that without the cost of compressing and > decompressing potentially large trees for no reason. > > @Jeremy "An sdist is an installable package which just happens to _look_ a > lot like a source release tarball, but trying to pretend that > downstream packagers will want to use it as such leads to a variety > of pain points in the upstream/downstream relationship. For better > or worse a lot of distros don't want generated files in upstream > source code releases, since they need to confirm that they also ship > the necessary tooling to regenerate any required files and that the > generated files they ship match what their packaged tooling > produces." - Well, pbr doesn't work if you just tar up or git export > your VCS tree: it requires the chance to add metadata. And while > distros have whinged about pbr in a number of contexts, that hasn't > been one so far. Downstreams are pretty used to receiving tarballs > with generated files in them - as long as they *have the option* to > recreate those, so the source material isn't lost. [And for version > data, 'grab from git' is a valid answer there']. OTOH perhaps > ftpmaster just hasn't noticed and we're about to get a bug report ;) another interesting use case for [not-] pip: https://github.com/mitsuhiko/pipsi > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Mon Oct 12 15:33:03 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 12 Oct 2015 14:33:03 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 11:01, Robert Collins wrote: > Python packaging config in $root/pypa.yaml > > Defined keys: > ---- > ... > build-tool: > # basic command that will spit a hunk of json back to the caller > defining the > # commands to use with the build tool. > ---- > > Build tool output json (in yaml format for ease of reading): > I would be quite happy with something along the lines of this proposal, though I'd bikeshed about some of the details. I like the idea of the source tree having a single reference to the build tool, and the build tool describing itself to pip. I'd probably use references to Python functions/modules rather than specifying shell-style commands, though; it seems like there's less to go wrong that way. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Oct 12 19:11:10 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 13 Oct 2015 06:11:10 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 13 October 2015 at 01:52, Daniel Holth wrote: > I could be wrong but if egg-info actually generated a dist-info directory it > would probably still work. I'd worry about fallout - since pip doesn't need to change (much) either way, I'd be inclined to be specific and allow setuptools to merely define the bootstrap code and opt in to later changes without pressure. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 12 19:16:53 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 13 Oct 2015 06:16:53 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 13 October 2015 at 02:33, Thomas Kluyver wrote: > On 12 October 2015 at 11:01, Robert Collins > wrote: >> >> Python packaging config in $root/pypa.yaml >> >> Defined keys: >> ---- >> ... >> build-tool: >> # basic command that will spit a hunk of json back to the caller >> defining the >> # commands to use with the build tool. >> ---- >> >> Build tool output json (in yaml format for ease of reading): > > > I would be quite happy with something along the lines of this proposal, > though I'd bikeshed about some of the details. I like the idea of the source > tree having a single reference to the build tool, and the build tool > describing itself to pip. I'd probably use references to Python > functions/modules rather than specifying shell-style commands, though; it > seems like there's less to go wrong that way. One of the fundamental things that emerged during the review of the design of my static setup-requires implementation in pip was that setuptools behaviour of not installing setup requirements into the target environment was deliberate design: it permits the use of different, mutually incompatible, versions of a given setup requirement by packages in the same dependency tree. E.g. imagine A and B both use setuptools-vcs, and setuptools-vcs does an incompatible 2.0 release. When A upgrades to that and B hasn't, if B install-requires A, pip installing B needs to install both those setuptools-vcs versions transiently, not permanently. (Even if one version is already installed, the build-time actions around the other of A|B need to have the other version installed). [My branch of pip doesn't do this - its one of the differences between proof of concept and production ready] So until we solve the problems related to unloading something loaded into Python and loading a different version in and all the related pain that can occur - I think using Python function calls is a non-starter. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 19:23:53 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 10:23:53 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 12, 2015 10:16 AM, "Robert Collins" wrote: > > On 13 October 2015 at 02:33, Thomas Kluyver wrote: > > On 12 October 2015 at 11:01, Robert Collins > > wrote: > >> > >> Python packaging config in $root/pypa.yaml > >> > >> Defined keys: > >> ---- > >> ... > >> build-tool: > >> # basic command that will spit a hunk of json back to the caller > >> defining the > >> # commands to use with the build tool. > >> ---- > >> > >> Build tool output json (in yaml format for ease of reading): > > > > > > I would be quite happy with something along the lines of this proposal, > > though I'd bikeshed about some of the details. I like the idea of the source > > tree having a single reference to the build tool, and the build tool > > describing itself to pip. I'd probably use references to Python > > functions/modules rather than specifying shell-style commands, though; it > > seems like there's less to go wrong that way. > > One of the fundamental things that emerged during the review of the > design of my static setup-requires implementation in pip was that > setuptools behaviour of not installing setup requirements into the > target environment was deliberate design: it permits the use of > different, mutually incompatible, versions of a given setup > requirement by packages in the same dependency tree. E.g. imagine A > and B both use setuptools-vcs, and setuptools-vcs does an incompatible > 2.0 release. When A upgrades to that and B hasn't, if B > install-requires A, pip installing B needs to install both those > setuptools-vcs versions transiently, not permanently. (Even if one > version is already installed, the build-time actions around the other > of A|B need to have the other version installed). [My branch of pip > doesn't do this - its one of the differences between proof of concept > and production ready] > > So until we solve the problems related to unloading something loaded > into Python and loading a different version in and all the related > pain that can occur - I think using Python function calls is a > non-starter. I don't see the contradiction here. If you look at the original draft PEP then it exactly specifies that builds get isolated environments, and build tools are supposed to spawn a child and then have that child do a function call using whatever mechanism they prefer. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Oct 12 19:47:33 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 13 Oct 2015 06:47:33 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 13 October 2015 at 06:23, Nathaniel Smith wrote: > On Oct 12, 2015 10:16 AM, "Robert Collins" > wrote: ... >> So until we solve the problems related to unloading something loaded >> into Python and loading a different version in and all the related >> pain that can occur - I think using Python function calls is a >> non-starter. > > I don't see the contradiction here. If you look at the original draft PEP > then it exactly specifies that builds get isolated environments, and build > tools are supposed to spawn a child and then have that child do a function > call using whatever mechanism they prefer. Ok, so here's a worked example to let us debug the disconnect. Given: A at 1.0: setup-requires: S~=1.0 install-requires: B B at 1.0: setup-requires: S~=2.0 S at 1.0: no dependencies at all. S at 2.0: no dependencies at all. and no binaries of A or B... then: pip install A will do the following (key bits related to this proposal only, happy path only): - download the A at 1.0 sdist - read pypa.yaml and read the setup-requires + build-tool keys - download S at 1.0 and prepare a built version of it - place S at 1.0 into PYTHONPATH and the built versions bin into PATH - run the build-tool command to determine how to use it - run the resulting wheel-build command to build a wheel for A at 1.0 - read the wheel metadata in to determine A's install-requires - download the B at 1.0 sdist - read pypa.yaml and read the setup-requires + build-tool keys - download S at 2.0 and prepare a built version of it - place S at 2.0 into PYTHONPATH and the built versions bin into PATH - run the build-tool command to determine how to use it - run the resulting wheel-build command to build a wheel for B at 1.0 - read the wheel metadata in to determine B's install-requires - install the B wheel into the target environment - install the A wheel into the target environment Note the two places where PYTHONPATH and PATH need to be something other than the environment of pip itself. pip may not be installed in the target environment (or may be a different version than the pip installed there). I don't understand how you propose to have S at 1.0 and S at 2.0 co-exist in pip's Python process. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 20:01:14 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 11:01:14 -0700 Subject: [Distutils] Meeting info re: sdists Message-ID: Hangouts URL: https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a Shared document for notes: https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Tue Oct 13 18:09:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Oct 2015 09:09:19 -0700 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: Hi all, I was going to send around a summary from this meeting, but I seem to have come down with the flu last night. So if someone else wants to do it while it's fresh in our minds, that'd be great, or else I'll try to get to it next week or so. Cheers, -n On Oct 12, 2015 11:01 AM, "Nathaniel Smith" wrote: > Hangouts URL: > https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a > > Shared document for notes: > > https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Oct 13 18:59:05 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 13 Oct 2015 09:59:05 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: On Sun, Oct 11, 2015 at 11:18 AM, Steve Dower wrote: > And it's also not binding on redistributors or package builders, who are > free to add more restrictive requirements. > not binding, but presumably a simple: setup.py build with distutils (or setuptools) will NOT use SSE2 in this case? That's the beauty of distutils of course, that is builds extensions with all the same settings as the host python. So even is SSE2 provides little for Python itself, in the usual case, we'll see performance hits n compiled extensions that are not compiled by particularly smart people. Also -- this thread started with "I can build Python3 myself, but now I can't install arbitrary binary wheels" -- if the goal is to fix that, then we don't WANT people tweaking things to build their wheels with SSE2 -- it would just get uglier. I certanly thought the "official" policy was that binary wheels put up on PyPi be (fully) compatible with the python.org build(s). What I'm getting at is that a decision to not use SSE2 in the main python build is a decision to not use it for any PyPi packages -- which could be a Bad Thing. > As far as identifying these sort of requirements in packages, I don't > think they really make sense as part of a platform tag as a non-SSE build > is fully compatible with an SSE build, but as part of version ordering it > would be nice to be able to prefer certain tagged builds without excluding > ones without the tag (e.g. if my platform is win32_sse2 and there's only a > win32 tagged build, I still want it - how we formalize this is probably > very complex though. > yeah, that's the trick with trying to shoehorn dependencies liek this into platform tags. But even if so do we WANT to now be supporting 4 Windows platforms? win32 win32_sse2 win64 win64_sse2 and then, what about sse3, etc??? My thoughts -- pick one, and either: * SSE2 is a requirement for python3.5 on Windows - then schools and the like can build their own Python3.5, AND their own package repository - on the one hand, institutions should be able to do this - on the other hand, the same institutions that NEED to do this (nes with old hardware),. are probably short on other resources, like admins savvy enough to be setting up build systems... * SSE2 is NOT used for the Windows builds. - then all packages on PyPi should also not use SSE2 - Other organizations, like Anaconda, Enthought, etc can build their own python and dependent packages can be build against that. - At least these folks DO have the resources to support that. -CHB > Also, my platform tag is probably 100 chars long with everything I may > want to represent...) > > Cheers, > Steve > > Top-posted from my Windows Phone > ------------------------------ > From: Oscar Benjamin > Sent: ?10/?11/?2015 10:43 > To: Antoine Pitrou ; Distutils-Sig at python.org > Subject: Re: [Distutils] warning about potential problem for wheels > > On Sun, 11 Oct 2015 17:44 Antoine Pitrou wrote: > > On Sun, 11 Oct 2015 08:07:30 -0700 > Steve Dower wrote: > > > > This does only affect 32-bit builds, so now I'm thinking about the > > possibility of treating those as highly compatible while the 64-bit > > ones get better performance treatment, though I'm not sure how that > > could actually play out. It may help remove some of the questions > > about which one to use though. > > That sounds reasonable to me. I don't know Windows very much, but are > there still many people using 32-bit Windows these days (on x86, I > mean)? > > > > I don't know but I think it makes sense to follow Windows' lead. So if 3.5 > supports Vista and Vista doesn't require SSE2 then CPython shouldn't > either. If 3.6 or whatever drops support for Vista and if Windows 7 > requires SSE2 then CPython can require it too. I assume this what happens > with the OSX binaries. > > Note that my recently retired computer was 64 bit and had SSE but didn't > have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after > SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers > are going to stop adding new instructions. So this general issue isn't > limited to 32 bit hardware and won't be solved by special casing that. I > think it makes sense to have a general policy for architectures that will > be supported by the official build in future. > > -- > Oscar > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 13 19:12:35 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 13 Oct 2015 13:12:35 -0400 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: On October 13, 2015 at 1:07:44 PM, Chris Barker (chris.barker at noaa.gov) wrote: yeah, that's the trick with trying to shoehorn dependencies liek this into platform tags. But even if so do we WANT to now be supporting 4 Windows platforms? win32 win32_sse2 win64 win64_sse2 and then, what about sse3, etc??? Longer term, more platforms (particularly more platforms that really only differ by some compiler flags) shouldn?t be a very big deal, because my dream for PyPI is that authors upload a source package and PyPI builds all their wheels for them (optionally of course). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 00:31:19 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 11:31:19 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 05:09, Nathaniel Smith wrote: > Hi all, > > I was going to send around a summary from this meeting, but I seem to have > come down with the flu last night. So if someone else wants to do it while > it's fresh in our minds, that'd be great, or else I'll try to get to it next > week or so. > > Cheers, > -n > > On Oct 12, 2015 11:01 AM, "Nathaniel Smith" wrote: >> >> Hangouts URL: >> https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a >> >> Shared document for notes: >> >> https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing So that ^ is worth a read for any interested party. The summary: We agreed on the following design constraints(MUSTS): * Graceful forward compat (old pip either can install, or fails cleanly, on new VCS and sdists, pypi can be changed ) * Backwards compat (new pip can install both old and new VCS/sdists, new pypi can handle old and new sdists) * Setup-requires static to the point the build system can be installed by pip * Setup-requirements need to be isolated: must be able to use different incompatible versions of the same setup-requirement during one install run. * Describing how to use the build system has to be user friendly. [e.g. json configs are out] And the following design goals(SHOULDs): * Be able to use things like flit cleanly within pip without needing a setup.py in-tree. * Long term - reduce variation in code paths * Long term - reduce amount of dynamism? in install pipeline? * We may never be able to remove it entirely, but being able to not run egg-info in the common case should be achievable * Things that have reason to change (deps) are more reasonable to be dynamic (even with PEP-426 markers there are exceptions) * project description, version etc in sdist have no reason to be dynamic * Files in VCS should not need to be different in sdist - ever We had broad agreement on the basic design: * There will be a file that describes how to install the build tool, and how pip or other systems should invoke it * It will be minimal: this file is strictly additive to the system, no overlap with existing things (even setup-requires - because setup-requires presumes the buildtool (setuptools) is already installed) I *think* we had broad agreement that its not desirable, overall, to try and skip steps like metadata generation - because with a resolver we'd end up potentially doing things like building dozens of versions of scipy just to extract metadata about dependencies. I like the term 'bootstrap-requires' -because we're bootstrapping the ability to use the build tool. The build tool interface needs to be sufficient that one could write a shim to existing setuptools in it. Previous posts have described what that might look like, but to reprise my sketch it had the following entries schema: #was version egg-info: dist-info: wheel: develop: install: provided-by: We have a design decision to make. We could make bootstrap-requires be a superset of setup-requires. That is, projects should put anything that is functionally a setup-requires thing into bootstrap-requires. This had the ad-hoc effect of dictating to setuptools how static setup-requires would be expressed. I don't tlike that. Alternatively we can make the interface permit querying for things to install before running any of the other things in the build system interface: e.g. one way would be build-requires: # command to query for package specific build dependencies # returns a PEP-426 JSON file, only the keys https://www.python.org/dev/peps/pep-0426/#metadata-version and "build_requires" are consulted. Alternatives here include: - just generating {dist.egg}-info twice - once to get setup_requires, once to get everything once thats resolved - drop dist-info and egg-info and use PEP-426 for those calls as well - don't use pep-426 at all, just use requires.txt format which is already well established - Use a pep-345 file I think pep-426 is ok since we're already using it within wheels (metadata.json). I'm a little nervous about dropping egg-info/dist-info - but hell, lets try it (since its common to have a proof of concept before accepting peps, we'll know it works for setuptools since that should be in the proof of concept). That would give a build tool interface of: schema: #was version build-requires: # command to get a pep-426 json blob on stdout with build_requires populated dist-info: # command to get a pep-426 json blob on stdout with everything populated wheel: # build a wheel develop: # do a develop install install: # do a direct install provided-by: # allows caching the build tool interface We need to define the exact environment that the build tool will be run in: will it inherit already installed things from the install target? Will it be isolated or hermetic? Being able to fix things at a distance is important - and remembering we have a lot of legacy paths, that means a greenfield approach isn't very viable. Separately we need to define how-and-when dist-info will be called in the case of sdists which have valid static metadata. Thats sufficiently different we think it should be a separate PEP. Go forward: Nathaniel to write two PEPs: * bootstrap-requirements and build system interface - enables multiple build systems * usable static metadata in sdists - allow the common case to do less work by ensuring valid metadata in sdists -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From chris.barker at noaa.gov Wed Oct 14 01:31:32 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 13 Oct 2015 16:31:32 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: <-1380413487201379509@unknownmsgid> But even if so do we WANT to now be supporting 4 Windows platforms? win32 win32_sse2 win64 win64_sse2 and then, what about sse3, etc??? Longer term, more platforms (particularly more platforms that really only differ by some compiler flags) shouldn?t be a very big deal, because my dream for PyPI is that authors upload a source package and PyPI builds all their wheels for them (optionally of course). That would be pretty nice. I'm still a bit wary though -- the more platforms you have, the more you have to test and debug.... Testing, off course would be part of the automated build. But debugging could get painful ... CHB ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 14 01:39:28 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 13 Oct 2015 18:39:28 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 13, 2015 6:01 PM, "Robert Collins" wrote: > > On 14 October 2015 at 05:09, Nathaniel Smith wrote: > > Hi all, > > > > I was going to send around a summary from this meeting, but I seem to have > > come down with the flu last night. So if someone else wants to do it while > > it's fresh in our minds, that'd be great, or else I'll try to get to it next > > week or so. > > > > Cheers, > > -n > > > > On Oct 12, 2015 11:01 AM, "Nathaniel Smith" wrote: > >> > >> Hangouts URL: > >> https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a > >> > >> Shared document for notes: > >> > >> https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing > > So that ^ is worth a read for any interested party. > > The summary: > > We agreed on the following design constraints(MUSTS): > > * Graceful forward compat (old pip either can install, or fails > cleanly, on new VCS and sdists, pypi can be changed ) > * Backwards compat (new pip can install both old and new VCS/sdists, > new pypi can handle old and new sdists) > * Setup-requires static to the point the build system can be installed by pip > * Setup-requirements need to be isolated: must be able to use > different incompatible versions of the same setup-requirement during > one install run. > * Describing how to use the build system has to be user friendly. > [e.g. json configs are out] (.cfg | .yml) > JSONLD Should these build settings all be included in e.g. PEP 426 + JSONLD metadata.jsonld? - for backward compat, there would then be both metadata.json and metadata.jsonld (because of normative JSONLD form and PEP 426+) .cfg (ConfigParse): (section, key, value) .yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF triple graph (of linked typed packages with attributes, generated from [...])) > > And the following design goals(SHOULDs): > > * Be able to use things like flit cleanly within pip without needing a > setup.py in-tree. > * Long term - reduce variation in code paths > * Long term - reduce amount of dynamism? in install pipeline? * OS Packaging compat (--prefix) * Conda solves for this. (static files with package-level checksums) > * We may never be able to remove it entirely, but being able to not > run egg-info in the common case should be achievable > * Things that have reason to change (deps) are more reasonable to be > dynamic (even with PEP-426 markers there are exceptions) > * project description, version etc in sdist have no reason to be dynamic > * Files in VCS should not need to be different in sdist - ever > > > We had broad agreement on the basic design: > * There will be a file that describes how to install the build tool, > and how pip or other systems should invoke it > * It will be minimal: this file is strictly additive to the system, no > overlap with existing things (even setup-requires - because > setup-requires presumes the buildtool (setuptools) is already > installed) > > I *think* we had broad agreement that its not desirable, overall, to > try and skip steps like metadata generation - because with a resolver > we'd end up potentially doing things like building dozens of versions > of scipy just to extract metadata about dependencies. > > I like the term 'bootstrap-requires' -because we're bootstrapping the > ability to use the build tool. > > The build tool interface needs to be sufficient that one could write a > shim to existing setuptools in it. Previous posts have described what > that might look like, but to reprise my sketch it had the following > entries > schema: #was version > egg-info: > dist-info: > wheel: > develop: > install: > provided-by: > > We have a design decision to make. > > We could make bootstrap-requires be a superset of setup-requires. That > is, projects should put anything that is functionally a setup-requires > thing into bootstrap-requires. This had the ad-hoc effect of dictating > to setuptools how static setup-requires would be expressed. I don't > tlike that. > > Alternatively we can make the interface permit querying for things to > install before running any of the other things in the build system > interface: e.g. one way would be > > build-requires: > # command to query for package specific build dependencies > # returns a PEP-426 JSON file, only the keys > https://www.python.org/dev/peps/pep-0426/#metadata-version and > "build_requires" are consulted. > > Alternatives here include: > - just generating {dist.egg}-info twice - once to get setup_requires, > once to get everything once thats resolved > - drop dist-info and egg-info and use PEP-426 for those calls as well > - don't use pep-426 at all, just use requires.txt format which is > already well established > - Use a pep-345 file > > I think pep-426 is ok since we're already using it within wheels > (metadata.json). I'm a little nervous about dropping > egg-info/dist-info - but hell, lets try it (since its common to have a > proof of concept before accepting peps, we'll know it works for > setuptools since that should be in the proof of concept). That would > give a build tool interface of: > > schema: #was version > build-requires: # command to get a pep-426 json blob on stdout with > build_requires populated > dist-info: # command to get a pep-426 json blob on stdout with > everything populated > wheel: # build a wheel > develop: # do a develop install > install: # do a direct install > provided-by: # allows caching the build tool interface > > We need to define the exact environment that the build tool will be > run in: will it inherit already installed things from the install > target? Will it be isolated or hermetic? Being able to fix things at a > distance is important - and remembering we have a lot of legacy paths, > that means a greenfield approach isn't very viable. > > Separately we need to define how-and-when dist-info will be called in > the case of sdists which have valid static metadata. Thats > sufficiently different we think it should be a separate PEP. > > Go forward: > > Nathaniel to write two PEPs: > * bootstrap-requirements and build system interface - enables > multiple build systems > * usable static metadata in sdists - allow the common case to do less > work by ensuring valid metadata in sdists > > -Rob > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Oct 14 02:25:21 2015 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 13 Oct 2015 17:25:21 -0700 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: thanks for the summary! * Things that have reason to change (deps) are more reasonable to be > dynamic (even with PEP-426 markers there are exceptions) > as we know, for *many* cases, run-time deps aren't dynamic. is there a consensus for those cases? exist in the sdist metadata? or no? or maybe with the "auto-building" Donald mentioned, we'd rely solely on wheel "built" metadata as the vehicle for dependency resolution also, did Nathaniel's idea of uploading c libraries into PyPI as python wrapper projects get discussed? --Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 02:46:11 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 13:46:11 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 13:25, Marcus Smith wrote: > > > thanks for the summary! > > >> * Things that have reason to change (deps) are more reasonable to be >> dynamic (even with PEP-426 markers there are exceptions) > > > as we know, for *many* cases, run-time deps aren't dynamic. > is there a consensus for those cases? exist in the sdist metadata? or no? The plan we hashed out (this would be the new PEP on static metadata in sdists) - pip etc to lint and error if they encounter a malformed dist-info in an sdist - then start putting dist-info, not egg-info into sdists - for any field if the field is entirely absent, we'll get it by running the dist-info build-tool command I described in the other mails Concretely: {'build_requires': []} -> no build requirements {} -> get build requirements by running the build system > or maybe with the "auto-building" Donald mentioned, we'd rely solely on > wheel "built" metadata as the vehicle for dependency resolution I don't believe thats feasible in a performant fashion. > also, did Nathaniel's idea of uploading c libraries into PyPI as python > wrapper projects get discussed? Nope :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 14 02:50:14 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 13:50:14 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 12:39, Wes Turner wrote: > .... > (.cfg | .yml) > JSONLD > > Should these build settings all be included in e.g. PEP 426 + JSONLD > metadata.jsonld? > - for backward compat, there would then be both metadata.json and > metadata.jsonld (because of normative JSONLD form and PEP 426+) I'm sorry, I have literally no idea what you're talking about. I know you have a thing about JSONLD, but I don't understand how or why it is relevant to these design discussions, which are 99% fine tuning backwards/forwards compat needs around the existing Python code thats out there and will be interacting with what we produce. If you want to shift the needle here and get discussions talking about things with JSONLD rather than PEP-426 (or some future leaner and less enterprisy version) ... then for me at least, I need more clear communication, less assumption that I have *any* overlap with where you're coming from, and at least some connection to the problems we're working on in your mails. > .cfg (ConfigParse): (section, key, value) > > .yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF > triple graph (of linked typed packages with attributes, generated from > [...])) Still not following. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From wes.turner at gmail.com Wed Oct 14 04:04:16 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 13 Oct 2015 21:04:16 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > > On 14 October 2015 at 12:39, Wes Turner wrote: > > > .... > > (.cfg | .yml) > JSONLD > > > > Should these build settings all be included in e.g. PEP 426 + JSONLD > > metadata.jsonld? > > - for backward compat, there would then be both metadata.json and > > metadata.jsonld (because of normative JSONLD form and PEP 426+) > > I'm sorry, I have literally no idea what you're talking about. I know > you have a thing about JSONLD, but I don't understand how or why it is > relevant to these design discussions, which are 99% fine tuning > backwards/forwards compat needs around the existing Python code thats > out there and will be interacting with what we produce. > > If you want to shift the needle here and get discussions talking about > things with JSONLD rather than PEP-426 (or some future leaner and less > enterprisy version) ... then for me at least, I need more clear > communication, less assumption that I have *any* overlap with where > you're coming from, and at least some connection to the problems we're > working on in your mails. > > > .cfg (ConfigParse): (section, key, value) > > > > .yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF > > triple graph (of linked typed packages with attributes, generated from > > [...])) > > Still not following. * egg-info and dist-info should generate JSONLD * sdist, bdist, and wheel should update **append to** the same JSONLD metadata document * if there are additional parameters which would be necessary to reproduce a package(s), those should be included in the package metadata ... the *provenance* of these build artifacts would then be represented in the package metadata document is this specific to sdist? nope, sorry, now is the time. > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 04:40:41 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 15:40:41 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 15:04, Wes Turner wrote: > > On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > * egg-info and dist-info should generate JSONLD The .egg-info and .dist-info directories are existing defined formats, I don't see any way that they can be retroactively defined as being JSONLD. I understand you have a JSONLD spec being discussed - this would be a successor to PEP-426, and Nick has put the breaks on all such things until we *catch up* with the already designed and defined work - so I'm not sure where your spec fits in things: but even assuming its all approved, we still can't change the meaning of existing in the wild artifacts. > * sdist, bdist, and wheel should update **append to** the same JSONLD > metadata document Why? > * if there are additional parameters which would be necessary to reproduce a > package(s), those should be included in the package metadata Why? > ... the *provenance* of these build artifacts would then be represented in > the package metadata document I hate to sound like a broken record, but why? We could list the hashes of all the headers the C compiler consulted in generating a numpy extension module, but what problem are we solving. > is this specific to sdist? nope, sorry, now is the time. The time for what? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From wes.turner at gmail.com Wed Oct 14 05:05:13 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 13 Oct 2015 22:05:13 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 13, 2015 9:40 PM, "Robert Collins" wrote: > > On 14 October 2015 at 15:04, Wes Turner wrote: > > > > On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > > > * egg-info and dist-info should generate JSONLD > > The .egg-info and .dist-info directories are existing defined formats, > I don't see any way that they can be retroactively defined as being > JSONLD. I understand you have a JSONLD spec being discussed - this > would be a successor to PEP-426, and Nick has put the breaks on all > such things until we *catch up* with the already designed and defined > work - so I'm not sure where your spec fits in things: but even > assuming its all approved, we still can't change the meaning of > existing in the wild artifacts. > > > * sdist, bdist, and wheel should update **append to** the same JSONLD > > metadata document > > Why? So that the project artifacts are traceable through the build process; in order to (identify sources of variance) [Requirements Traceability, 5Ws] This is called "provenance" in real metadata circles. [W3C PROV]. > > > * if there are additional parameters which would be necessary to reproduce a > > package(s), those should be included in the package metadata > > Why? Because otherwise one has no idea what they are working with. Because, for reproducibility, it's optimal to record all inputs as structured data. Why would we be throwing this process knowledge away? dash-delimited filenames to indicate a phantom ABI parameter? > > > ... the *provenance* of these build artifacts would then be represented in > > the package metadata document > > I hate to sound like a broken record, but why? We could list the > hashes of all the headers the C compiler consulted in generating a > numpy extension module, but what problem are we solving. This could be useful; we currently must use actual OS packaging to verify per-file on-disk checksums (e.g. debsums). e.g. we actually currently have no idea if the uncompressed archive files differ from the checksummed archive, unless we re-download and compare with an archive manifest; because there is not yet a manifest containing (path, checksum, [perms], [fsattrs]) > > > is this specific to sdist? nope, sorry, now is the time. > > The time for what? > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 14 10:33:17 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Oct 2015 09:33:17 +0100 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 01:46, Robert Collins wrote: > On 14 October 2015 at 13:25, Marcus Smith wrote: >> >> >> thanks for the summary! Agreed. >> >>> * Things that have reason to change (deps) are more reasonable to be >>> dynamic (even with PEP-426 markers there are exceptions) >> >> >> as we know, for *many* cases, run-time deps aren't dynamic. >> is there a consensus for those cases? exist in the sdist metadata? or no? > > The plan we hashed out (this would be the new PEP on static metadata in sdists) > - pip etc to lint and error if they encounter a malformed dist-info in an sdist > - then start putting dist-info, not egg-info into sdists > - for any field if the field is entirely absent, we'll get it by > running the dist-info build-tool command I described in the other > mails > > Concretely: > {'build_requires': []} -> no build requirements > {} -> get build requirements by running the build system One use case that I don't think is covered here is publishing dependency metadata via PyPI. I believe distlib needs this for some functions (at the moment, I think it uses an externally hosted set of package dependency data that's maintained by Vinay), and I know there have been a number of utility scripts I've needed to write that I simply haven't been able to because doing so would involve a significant amount of extra work downloading and running package code. If there are dependencies that are only detectable at wheel build time, then so be it (I'm still looking for an example, but it's clear that the potential is there) but I'd like some way of getting at whatever dependency information a wheel (source or binary) provides via the PyPI JSON API - and I'd like an assurance that if dependency information *can* be determined statically, it will be. Paul From robertc at robertcollins.net Wed Oct 14 10:50:06 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 21:50:06 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 21:33, Paul Moore wrote: > On 14 October 2015 at 01:46, Robert Collins wrote: ... >> Concretely: >> {'build_requires': []} -> no build requirements >> {} -> get build requirements by running the build system > > One use case that I don't think is covered here is publishing > dependency metadata via PyPI. I believe distlib needs this for some > functions (at the moment, I think it uses an externally hosted set of > package dependency data that's maintained by Vinay), and I know there > have been a number of utility scripts I've needed to write that I > simply haven't been able to because doing so would involve a > significant amount of extra work downloading and running package code. Where thats static (expressable for all installs using PEP-426 conditions) we will be able to do that for sdists. Where its dependent on the build environment, we can't do that today (but we already can for wheels). For illustration, consider a numpy extension, the version of numpy present in the build environment determines some of the install-requires. The sdist can build against a much wider range of numpy versions than the resulting binary can run against. > If there are dependencies that are only detectable at wheel build > time, then so be it (I'm still looking for an example, but it's clear > that the potential is there) but I'd like some way of getting at > whatever dependency information a wheel (source or binary) provides > via the PyPI JSON API - and I'd like an assurance that if dependency > information *can* be determined statically, it will be. That is exactly the intent. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Wed Oct 14 11:32:21 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Oct 2015 10:32:21 +0100 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 09:50, Robert Collins wrote: > For illustration, consider a numpy extension, the version of numpy > present in the build environment determines some of the > install-requires. The sdist can build against a much wider range of > numpy versions than the resulting binary can run against. For my purposes, having the sdist depend on "numpy" and the wheel depend on a specific version of numpy would be fine. In general, for the uses I have in mind I'm not really interested in versions, mostly just "package X depends on package Y". Think of things like scripts to build a local package repository with everything needed to install a particular set of packages. I'm not worried about getting a minimal set of files, just getting all files available for each package needed is good enough. (I know "pip wheel" is more or less what I need for this specific case, my actual requirements tend to be a bit more fiddly). >> If there are dependencies that are only detectable at wheel build >> time, then so be it (I'm still looking for an example, but it's clear >> that the potential is there) but I'd like some way of getting at >> whatever dependency information a wheel (source or binary) provides >> via the PyPI JSON API - and I'd like an assurance that if dependency >> information *can* be determined statically, it will be. > > That is exactly the intent. Excellent, thanks. Paul From solipsis at pitrou.net Wed Oct 14 12:40:48 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Oct 2015 12:40:48 +0200 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: <20151014124048.2cea5551@fsol> On Tue, 13 Oct 2015 09:59:05 -0700 Chris Barker wrote: > > So even is SSE2 provides little for Python itself, in the usual case, we'll > see performance hits n compiled extensions that are not compiled by > particularly smart people. Since the question is only for 32-bit builds, does this even matter? 32-bit builds on x86 generally bring you poorer performance by themselves, as the CPU has only access to 8 general-purpose registers (instead of 16) in that mode. If you want better performance (or want to efficiently address 3+GB of RAM, which is an extremely common situation nowadays), you probably want 64-bit builds. Regards Antoine. From nico.schloemer at gmail.com Tue Oct 13 19:03:34 2015 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Tue, 13 Oct 2015 17:03:34 +0000 Subject: [Distutils] rst not rendered on pypi Message-ID: Hi everyone, I'm trying to get my README.rst [1] display properly on my PyPy project [2], but for some reason it's displayed as raw text. The file was created from a README.md [3] via ``` pandoc README.md -o README.rst ``` Any hints on what may have been going wrong? Cheers, Nico [1] http://chunk.io/f/f4385eda77de4ed78f8eba8cc8d8c8b6 [2] https://pypi.python.org/pypi/voropy [3] https://github.com/nschloe/voropy/blob/master/README.md -------------- next part -------------- An HTML attachment was scrubbed... URL: From schettino72 at gmail.com Wed Oct 14 14:03:04 2015 From: schettino72 at gmail.com (Eduardo Schettino) Date: Wed, 14 Oct 2015 20:03:04 +0800 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 1:03 AM, Nico Schl?mer wrote: > Hi everyone, > > I'm trying to get my README.rst [1] display properly on my PyPy project > [2], but for some reason it's displayed as raw text. The file was created > from a README.md [3] via > ``` > pandoc README.md -o README.rst > ``` > Any hints on what may have been going wrong? > > You can check it using: https://pypi.python.org/pypi/readme -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 14 13:43:10 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 14 Oct 2015 14:43:10 +0300 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: This looks suspicious: *A twice-folded M??bius strip, created with VoroPy's ``moebius_tri -i 2 out.e``. Visualization with `ParaView `__.* Regardless, you should use https://pypi.python.org/pypi/readme to check it (that's what PyPI uses for rendering). Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Tue, Oct 13, 2015 at 8:03 PM, Nico Schl?mer wrote: > Hi everyone, > > I'm trying to get my README.rst [1] display properly on my PyPy project > [2], but for some reason it's displayed as raw text. The file was created > from a README.md [3] via > ``` > pandoc README.md -o README.rst > ``` > Any hints on what may have been going wrong? > > Cheers, > Nico > > > [1] http://chunk.io/f/f4385eda77de4ed78f8eba8cc8d8c8b6 > [2] https://pypi.python.org/pypi/voropy > [3] https://github.com/nschloe/voropy/blob/master/README.md > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico.schloemer at gmail.com Wed Oct 14 16:19:22 2015 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Wed, 14 Oct 2015 14:19:22 +0000 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: Exactly what I needed. Thanks! --Nico On Wed, Oct 14, 2015 at 2:03 PM Eduardo Schettino wrote: > > On Wed, Oct 14, 2015 at 1:03 AM, Nico Schl?mer > wrote: > >> Hi everyone, >> >> I'm trying to get my README.rst [1] display properly on my PyPy project >> [2], but for some reason it's displayed as raw text. The file was created >> from a README.md [3] via >> ``` >> pandoc README.md -o README.rst >> ``` >> Any hints on what may have been going wrong? >> >> You can check it using: https://pypi.python.org/pypi/readme > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From randy at thesyrings.us Wed Oct 14 16:29:37 2015 From: randy at thesyrings.us (Randy Syring) Date: Wed, 14 Oct 2015 10:29:37 -0400 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: <561E66D1.9060907@thesyrings.us> I've also found restview (https://pypi.python.org/pypi/restview) very easy to use. I believe it uses readme behind the scenes (at least, it's included as a dependency in restviews setup.py). *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 10/14/2015 10:19 AM, Nico Schl?mer wrote: > Exactly what I needed. Thanks! > > --Nico > > On Wed, Oct 14, 2015 at 2:03 PM Eduardo Schettino > > wrote: > > > On Wed, Oct 14, 2015 at 1:03 AM, Nico Schl?mer > > wrote: > > Hi everyone, > > I'm trying to get my README.rst [1] display properly on my > PyPy project [2], but for some reason it's displayed as raw > text. The file was created from a README.md [3] via > ``` > pandoc README.md -o README.rst > ``` > Any hints on what may have been going wrong? > > You can check it using: https://pypi.python.org/pypi/readme > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 14 16:35:28 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Oct 2015 00:35:28 +1000 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 12:40, Robert Collins wrote: > On 14 October 2015 at 15:04, Wes Turner wrote: >> >> On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > >> * egg-info and dist-info should generate JSONLD > > The .egg-info and .dist-info directories are existing defined formats, > I don't see any way that they can be retroactively defined as being > JSONLD. I understand you have a JSONLD spec being discussed - this > would be a successor to PEP-426, and Nick has put the breaks on all > such things until we *catch up* with the already designed and defined > work - so I'm not sure where your spec fits in things: but even > assuming its all approved, we still can't change the meaning of > existing in the wild artifacts. I finally found time to investigate JSON-LD a while back, and I think there's potentially value in having the formal specification of our metadata formats be in the form of a JSON-LD context definition: https://github.com/pypa/interoperability-peps/issues/31 However, doing that doesn't actually solve any immediate problems for us, as none of our tools are JSON-LD aware, they're all based on ad hoc JSON processing. Thus it's firmly in the "might be nice to do" category for me, rather than "we need to do this as an urgent priority". One thing we do need to be careful of is the fact that PEP 426 is still a draft spec - if there are things we want to formalise *now* in a JSON format aimed at the *current* capabilities of the ecosystem, we may want to push some of the ideas like metadata extensions and the finer granularity of dependencies out to a more speculative metadata 3.0 spec, and descope 2.0 to a "only incremental additions to current capabilities, wrapped up in a better defined format" effort. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Wed Oct 14 18:25:43 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 09:25:43 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <20151014124048.2cea5551@fsol> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> <20151014124048.2cea5551@fsol> Message-ID: On Wed, Oct 14, 2015 at 3:40 AM, Antoine Pitrou wrote: > On Tue, 13 Oct 2015 09:59:05 -0700 > Chris Barker wrote: > > > > So even is SSE2 provides little for Python itself, in the usual case, > we'll > > see performance hits n compiled extensions that are not compiled by > > particularly smart people. > > Since the question is only for 32-bit builds, IS that the case: """ Note that my recently retired computer was 64 bit and had SSE but didn't have SSE2 (I'm fairly sure - CPU was some budget AMD model) """ granted, such machines are probably really really rare, but maybe it does matter for 64 bit, too? > does this even matter? > 32-bit builds on x86 generally bring you poorer performance by > themselves, If a user has a 32 bit machine, they have no choice -- we could argue that anyone for whom performance matters probably isn't running an old, cheap machine, but still... > If you want better performance (or want > to efficiently address 3+GB of RAM, which is an extremely common > situation nowadays), you probably want 64-bit builds. > I thought that wasn't the case (well it certainly is if you want to access that memory), but for raw performance, 64 bit is pushing a lot more memory, in the form of larger pointers, around -- so is sometimes slower. I'm sure it depends on the use-case, as it always does.... Also a note -- I'm lost on the details, but the most recent numpy release required SSE2: """ * Compiling with msvc9 or msvc10 for 32 bit Windows now requires SSE2. This was the easiest fix for what looked to be some miscompiled code when SSE2 was not used. If you need to compile for 32 bit Windows systems without SSE2 support, mingw32 should still work. """ and that's not py3.5 (msvc9 is the py2.7 compiler, and msvc10 is py3.4, yes???) -Chris > > Regards