Mailman 3 - Distutils-SIG

package management - common storage while keeping the versions straight
by David Mathog July 7, 2020

July 7, 2020

Hi all, First post here. I have a cluster where the common software is NFS shared from the file server to other nodes. All the python packages are kept in a directory which is referenced by PYTHONPATH. The good part of that is that there is just one copy of each package-version. The bad part, as you have all no doubt guessed, is that python by itself is really bad at specifying and loading a set of particular library versions (see below), so upgrading one program will break another due to conflicting installed versions. Hence the common use of virtualenv's. But as far as I can tell each virtualenv installs a copy of each package-version it needs, resulting in multiple copies of the same package-version for common packages on the same disk. What I am after is some method of keeping exactly one copy of each package-version in the common area (ie, one might find foo-1.2, foo-1.7, and foo-2.3 there), while also presenting only the one version of each (let's say foo-1.7) to a particular installed program. On linux it might do that by making soft links to the common PYTHONPATH area from another directory for which it sets PYTHONPATH for the application. Finally, this has to be usable by any account which has read execute access to the main directory. Does such a beast exist? If so, please point me to it! The limitations of python version handling to which I refer above can be illustrated for "scanpy-scripts"'s dependencies. Given all the needed libraries in one place (plus incompatible versions) the right set can be loaded (and verified) like this: export PYTHONPATH=/path/to_common_area python3 __requires__ = ['scipy <1.3.0,>=1.2.0', 'anndata <0.6.20', 'loompy <3.0.0,>=2.00', 'h5py <2.10'] import pkg_resources import scipy import anndata import loompy import h5py import scanpy print(scipy.__version__) print(anndata.__version__) print(loompy.__version__) print(h5py.__version__) print(scanpy.__version__) quit() which emits exactly the versions scanpy-scripts needs: 1.2.3 0.6.19 2.0.17 2.9.0 1.4.3 However, adding , 'scanpy <1.4.4,>=1.4.2' at the end of __requires__ makes the whole thing fail at import pkg_resources with (many lines deleted) 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (scipy 1.2.3 (/home/common/lib/python3.6/site-packages/scipy-1.2.3-py3.6-linux-x86_64.egg), Requirement.parse('scipy>=1.3.1'), {'umap-learn'}) even though the scanpy it loaded in the first case was within the desired range. Moreover, specifying the desired versions as parameters to import pkg_resources does not work at all since pkg_resources only keeps the highest version of each package it finds when imported. (A limitation that never made the least bit of sense to me.) The test system is CentOS 8 with python 3.6.8. Thanks, David Mathog

6 18

Development installation with pyproject.toml/poetry?
by fredrik＠averpil.com July 1, 2020

July 1, 2020

Hi, I have converted my setup.py to pyproject.toml and use poetry to build/manage deps. I maintain packages which also have dependency packages and I frequently use development installations locally ("pip install -e ../my-dep") so to develop dependencies at the same time as my main packages. Q: With a pyproject.toml, can I enable some sort of development installation without also maintaining a setup.py - or is there a plan/PEP for enabling this without a setup.py? Right now, I have a setup.py which needs to stipulate a number of things which is already in my pyproject.toml. In particular, I find it a little bothering that I need to specify the entry_points/console_scripts in both the pyproject.toml and in setup.py. Kind regards, Fredrik

4 3

setup.py with a different name
by Gabriel Becedillas June 30, 2020

June 30, 2020

Hi, I'm trying to build a python package but my setup.py is actually named pkg1_setup.py (this is because I have multiple packages in the same repo). The package built is *almost* fine, except that setup.py is not included in the package built. I understand that the *normal* structure would be to have something like this: * setup.py * mypackage\... but before refactoring the repository structure to accommodate that I'd like to validate that there is no other way of doing that. I tried using the script_name parameter to setuptools.setup but had no luck. Any suggestions? Thanks a lot

2 1

ANN: distlib 0.3.1 released on PyPI
by Vinay Sajip June 28, 2020

June 28, 2020

I've recently released version 0.3.1 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #132: Added documentation to help with relative interpreter paths. Thanks to Paul Kienzle for the patch. * Fixed #134: Allowed specifying a different target Python version when generating scripts. * Fixed #135: Exposed the ``enquote_executable`` function previously implemented as an internal function. * Addressed #138: Improved metadata support for newer metadata versions. Thanks to James Tocknell for the patch. * Changed the output of flags in entry point definitions in wheels. Thanks to frostming (明希) for the patch. * Stopped writing JSON metadata. Only old-style metadata is written now. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Note: Mailman3 might mishandle some of the links below. In that case, just copy and paste the links into your browser address bar - that should work. Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.1/ [2] https://distlib.readthedocs.io/en/0.3.1/ [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

Re: package management - common storage while keeping the versions straight
by Jason Madden June 25, 2020

June 25, 2020

On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a particular installed program. > On linux it might do that by making soft links to the common > PYTHONPATH area from another directory for which it sets PYTHONPATH > for the application. Finally, this has to be usable by any account > which has read execute access to the main directory. > > Does such a beast exist? If so, please point me to it! zc.buildout[1] and zc.recipe.egg[2] can do something very much like this. zc.buildout tries hard to maintain reproducibility and isolation, and one of the ways it does this is by keeping each package in its own .egg directory. It then generates the entry-point scripts and a REPL with ``sys.path`` explicitly set to reference exactly the versions specified. There's no virtualenv-like activation step that puts all those scripts on the path, though, so one must either do that manually or just invoke the generated scripts directly. For example, here's a buildout configuration that specifies a shared directory to store eggs in. It also has some parts using zc.recipe.egg, one that will use zope.interface 4 and one that will use zope.interface 5 (egg specifications can be arbitrarily complex, of course, dependencies are followed, etc): [buildout] eggs-directory = /<path>/buildout-eggs abi-tag-eggs = true parts = old-interface new-interface new-interface-plus [old-interface] recipe = zc.recipe.egg eggs = zope.interface == 4.0 interpreter = old-py [new-interface] recipe = zc.recipe.egg eggs = zope.interface == 5.1 interpreter = new-py [new-interface-plus] recipe = zc.recipe.egg eggs = zope.interface == 5.1 zope.component interpreter = new-py-plus After running `buildout`, I have three REPL files (if zope.interface had defined any entry point scripts, I could also have had those generated): $ head bin/new-py #!<path/to/python> import sys sys.path[0:0] = [ '/<path>/buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg', '/<path/to/python>/site-packages', ] $ head bin/old-py #!<path/to/python> import sys sys.path[0:0] = [ '/<path>/buildout-eggs/pypy_73/zope.interface-4.0.0-py2.7.egg', '/<path/to/python>/site-packages', ] $ head bin/new-py-plus #!<path/to/python> import sys sys.path[0:0] = [ '/<path>/buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg', '/<path>/buildout-eggs/pypy_73/zope.component-4.6.1-py2.7.egg', '/<path/to/python>/site-packages', ] I've got a collection of zope.interface eggs referenced from a variety of different buildouts (at least at one point in time) and from a variety of different Python implementations, but only ever one copy of each: $ ls -ld buildout-eggs/*/zope.interface* Permissions Size User Date Modified Name drwxr-xr-x - jmadden 2017-05-04 06:53 buildout-eggs/pypy_41/zope.interface-4.4.0-py2.7.egg/ drwxr-xr-x - jmadden 2017-05-04 07:00 buildout-eggs/cp27m/zope.interface-4.4.0-py2.7-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-05-09 17:55 buildout-eggs/cp34m/zope.interface-4.4.0-py3.4-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-06-08 10:20 buildout-eggs/cp27m/zope.interface-4.4.1-py2.7-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-07-11 10:07 buildout-eggs/cp36m/zope.interface-4.4.2-py3.6-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-12-08 11:10 buildout-eggs/cp27m/zope.interface-3.6.7-py2.7-macosx-10.13-x86_64.egg/ drwxr-xr-x - jmadden 2018-05-07 11:05 buildout-eggs/cp27m/zope.interface-4.1.3-py2.7-macosx-10.13-x86_64.egg/ drwxr-xr-x - jmadden 2020-03-13 07:53 buildout-eggs/cp27m/zope.interface-4.6.0-py2.7-macosx-10.15-x86_64.egg/ drwxr-xr-x - jmadden 2020-04-08 07:32 buildout-eggs/cp38/zope.interface-5.1.0-py3.8-macosx-10.15-x86_64.egg/ drwxr-xr-x - jmadden 2020-05-17 09:30 buildout-eggs/cp27m/zope.interface-5.1.0-py2.7.egg/ drwxr-xr-x - jmadden 2020-06-11 07:42 buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg/ drwxr-xr-x - jmadden 2020-06-25 11:58 buildout-eggs/pypy_73/zope.interface-4.0.0-py2.7.egg/ ~Jason [1] http://www.buildout.org/ ; I recommend version 3, which uses pip to find eggs [2] https://pypi.org/project/zc.recipe.egg/

1 0

Twine 3.2.0 released
by Brian Rutledge June 24, 2020

June 24, 2020

https://pypi.org/project/twine/3.2.0/ Changelog: https://twine.readthedocs.io/en/latest/changelog.html This is my first release as a Twine maintainer. Thanks to: - Sumana Harihareswara for getting me started, and nudging me along the way - Ian Stapleton Cordasco, Jason R. Coombs, and Dustin Ingram for code review and guidance in general - Devesh Kumar Singh, Felipe Mulinari Rocha Campos, and Vikram Jayanthi for substantial contributions to this release

2 1

[PEP 508] Clarification of URL References
by w_minchin＠hotmail.com June 23, 2020

June 23, 2020

PEP 508 allows for pointing to required packages using URL's. This is the except: A minimal URL based lookup:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346… However, there is very little documentation, blog posts, etc on how to use beyond the PEP itself, and so I struggled to understand it. I incorrectly assumed that *pip*, at the start of the line, had something to do with the given package being *installed via pip*, rather than *the name of the package being installed*. Therefore, is it possible to list another example line in the PEP to make it clear that the line doesn't have to start with "pip" to work? For further reading, my original issue, and the discovered solution --> https://github.com/jazzband/pip-tools/issues/1056 The explanation there that was helpful for me (that I didn't get from reading the PEP) was: > I believe the issue is in the name @ url pattern. It should be project_name @ vcs_url, no eggs param needed in this case by the way, but the project_name should be the same as package name in vcs_url/setup.py. [source](https://github.com/jazzband/pip-tools/issues/1056#issuecomment-5829… c.f. the issue I raised on GitHub, where I was suggested to post here --> https://github.com/python/peps/issues/1444 Thanks.

2 2

Who is approving PyPI size limit exceptions these days?
by Steve Dower June 18, 2020

June 18, 2020

These issues seem to be awaiting approval: https://github.com/pypa/pypi-support/labels/limit%20request Most pressing (for me ;) ) is my colleagues at https://github.com/pypa/pypi-support/issues/457, who are already dealing with confused users not finding the packages they expect. Thanks, Steve

1 0

(no subject)
by Raregem Augustine June 16, 2020

June 16, 2020

Hello! I have tried several times and it has not been working to install external packages in Python. I continue to get ERROR: Could not find a version that satisfies the requirement SomePackage (from versions: non) ERROR: No matching distribution found for SomePackage. Please, how can I fix this. Thank you.

2 1

Installing a module
by Rishi June 16, 2020

June 16, 2020

2 1