Mailman 3 June 2020 - Distutils-SIG

Failure: Error
by oliviajohn123123＠hotmail.com 21 Sep '20

21 Sep '20

facing same NOt found issue the issue code is "C:\Users\jaraco\projects\distutils_namespace\myns.projB\test\testBasic.py", can you guide me for this ? https://thelifefoundry.com/hotmail/

10 9

Improving Communication
by Donald Stufft 30 Jul '20

30 Jul '20

Currently in the packaging space, we have a number of avenues for communication, which are: - distutils-sig - pypa-dev - virtualenv-users - Other project specific mailing lists - IRC - gitter - Various issue trackers spread across multiple platforms. - Probably more places I’m not remembering. The result of this is that all discussion ends up being super fractured amongst the various places. Sometimes that is exactly what you want (for instance, someone who is working on the wheel specs probably doesn’t care about deprecation policy and internal module renaming in pip) and sometimes that ends up being the opposite of what you want (for instance, when you’re describing something that touches PyPI, setuptools, flit, pip, etc all at once). Theoretically the idea is that distutils-sig is where cross project reaching stuff goes, IRC/gitter is where real time discussion goes, and the various project mailing lists and issue trackers are where the project specific bits go. The problem is that often times doesn’t actually happen in practice except for the largest and most obvious of changes. I think our current “communications stack” kind of sucks, and I’d love to figure out a better way for us to handle this that solves the sort of weird “independent but related” set of projects we have here. From my POV, a list of our major problems are: * Discussion gets fractured across a variety of platforms and locations, which can make it difficult to actually keep up with what’s going on but also to know how to loop in someone relevant if their input would be valuable. You have to more or less simultaneously know someone’s email, Github username, IRC nick, bitbucket username, etc to be able to bring threads of discussion to people’s attention. * It’s not always clear to users where a discussion should go, often times they’ll come to one location and need to get redirected to another location. If any discussion did happen in the incorrect location, it tends to need to get restarted in the new location (and depending on the specific platform, it may be impossible to actually redirect everyone over to the proper location, so you again, end up fractured with the discussion happening in two places). * A lot of the technology in this stack is particularly old, and lacks a lot of the modern day affordances that newer things have. An example is being able to edit a discussion post to fix typos that can hinder the ability of others to actually understand whats being talked about. In your typical mailing list or IRC there’s no mechanism by which you can edit an already sent message, so your only option is to either let the problem ride and hope it doesn’t trip up too many people, or send an additional message to correct the error. However these show up as additional, later messages which someone might not even see until they’ve already been thoroughly confused by the first message (since people tend to read email/IRC in a linear fashion). - There is a lot of things in this one, other things are things like being able to control in a more fine grained manner what email you’re going to get. - Holy crap, formatting and typography to make things actually readable and not a big block of plaintext. * We don’t have a clear way for users to get help, leaving users to treat random issues, discussion areas, etc as a support forum, rather than some place that’s actually optimized for that. Some of this ties back into some of the earlier things too, where it’s hard to actually redirect discussions These aren’t *new* problems, and often times the existing members of a community are the least effected becasue they’ve already spent effort learning the ins and outs and also curating a (typically custom) workflow that they’ve grown accustomed too. The problem with that is that often times that means that new users are left out, and the community gets smaller and smaller as time goes on as people leave and aren’t replaced with new blood, because they’re driven off but the issues with the stack. A big part of the place this is coming from, is me sitting back and realizing that I tend to be drawn towards pulling discussions into Github issues rather than onto the varying mailing lists, not because that’s always the most appropriate place for it, but because it’s the least painful place in terms of features and functionality. I figure if I’m doing that, when I already have a significant investment in setting up tooling and being involved here, that others (and particularly new users) are likely feeling the same way. - Donald

9 12

pip install very slow
by Stuart McGraw 21 Jul '20

21 Jul '20

Hello, First time trying to use Python's package tools and I'm attempting to use pip to install from a local project: python3 -m pip install --user --upgrade --no-deps ./ I have a simple directory structure very similar to that recommended in https://packaging.python.org/tutorials/packaging-projects/ project/ data/ mypackage/ __init__.py data/ *.csv module1.py module2.py tmpl/ *.jinja setup.py The project/data/ directory contains several GB of stuff I use during development. It is not supposed to be part of the package and it is not mentioned in setup.py. The problem is when I run the install command above it takes many minutes to finish. When I move the project/data/ sub-directory out of the project/ directory, it takes a few seconds. I thought perhaps that "packages=setuptools.find_packages()" in my setup.py file might be the culprit but replacing in with "packages=['mypackage']" made no difference. I'm not sure why pip seems to be roaming through project/data/ (if that's what's happening) but is there some way to get it not to? My apologies if this is covered in the docs somewhere; I'm still going though them but didn't see anything with an initial scan. If it helps, here is my setup.py file: setuptools.setup ( name="mypackage", version="...", author="Me", author_email="...", description="...", long_description="...", long_description_content_type="text/markdown", url="...", packages=['mypackage'], package_data={'mypackage': ['data/*.csv', 'tmpl/*.jinja',]}, zip_safe = False, classifiers=[...], python_requires='>=3.6', install_requires = ['jinja2',], ) Thanks for any enlightenment.

4 5

FASTEN and dependency analysis at call graph level
by Sumana Harihareswara 16 Jul '20

16 Jul '20

Last week I heard for the first time about the research project FASTEN https://www.fasten-project.eu/. "FASTEN stands for Fine-Grained Analysis of Software Ecosystems as Networks." > instead of analyzing dependencies at the package level, we will analyze them at the call graph level! This will allow us to be super precise when we are tracking dependencies, when we do change impact analysis, when we recommend clients to update packages etc. It will also open the door to new sophisticated applications, e.g. licensing compliance, dependency risk profiling and data-driven API evolution. That's from the blog post by Georgios Gousios, the PI, at http://www.gousios.gr/blog/Introducing-Fasten.html . More info: https://www.fasten-project.eu/view/Main/Introduction https://www.fasten-project.eu/view/Main/Overview https://www.fasten-project.eu/view/Main/Contacts And people who are interested in dynamically and statically analyzing call graphs in Python may be interested in "Graph Schema and its representation" in https://www.fasten-project.eu/view/Main/Deliverables . I've sent a note to FASTEN inviting the team to come talk about their project here on distutils-sig, because FASTEN's site says they aim to eventually integrate into PyPI -- I'm not 100% sure whether that means "create a service that people can use WITH PyPI" or "get FASTEN's work incorporated into pypi.org". -- Sumana Harihareswara pip project manager on contract with Python Software Foundation Changeset Consulting https://changeset.nyc

3 2

package management - common storage while keeping the versions straight
by David Mathog 08 Jul '20

08 Jul '20

Hi all, First post here. I have a cluster where the common software is NFS shared from the file server to other nodes. All the python packages are kept in a directory which is referenced by PYTHONPATH. The good part of that is that there is just one copy of each package-version. The bad part, as you have all no doubt guessed, is that python by itself is really bad at specifying and loading a set of particular library versions (see below), so upgrading one program will break another due to conflicting installed versions. Hence the common use of virtualenv's. But as far as I can tell each virtualenv installs a copy of each package-version it needs, resulting in multiple copies of the same package-version for common packages on the same disk. What I am after is some method of keeping exactly one copy of each package-version in the common area (ie, one might find foo-1.2, foo-1.7, and foo-2.3 there), while also presenting only the one version of each (let's say foo-1.7) to a particular installed program. On linux it might do that by making soft links to the common PYTHONPATH area from another directory for which it sets PYTHONPATH for the application. Finally, this has to be usable by any account which has read execute access to the main directory. Does such a beast exist? If so, please point me to it! The limitations of python version handling to which I refer above can be illustrated for "scanpy-scripts"'s dependencies. Given all the needed libraries in one place (plus incompatible versions) the right set can be loaded (and verified) like this: export PYTHONPATH=/path/to_common_area python3 __requires__ = ['scipy <1.3.0,>=1.2.0', 'anndata <0.6.20', 'loompy <3.0.0,>=2.00', 'h5py <2.10'] import pkg_resources import scipy import anndata import loompy import h5py import scanpy print(scipy.__version__) print(anndata.__version__) print(loompy.__version__) print(h5py.__version__) print(scanpy.__version__) quit() which emits exactly the versions scanpy-scripts needs: 1.2.3 0.6.19 2.0.17 2.9.0 1.4.3 However, adding , 'scanpy <1.4.4,>=1.4.2' at the end of __requires__ makes the whole thing fail at import pkg_resources with (many lines deleted) 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (scipy 1.2.3 (/home/common/lib/python3.6/site-packages/scipy-1.2.3-py3.6-linux-x86_64.egg), Requirement.parse('scipy>=1.3.1'), {'umap-learn'}) even though the scanpy it loaded in the first case was within the desired range. Moreover, specifying the desired versions as parameters to import pkg_resources does not work at all since pkg_resources only keeps the highest version of each package it finds when imported. (A limitation that never made the least bit of sense to me.) The test system is CentOS 8 with python 3.6.8. Thanks, David Mathog

6 18

setup.py with a different name
by Gabriel Becedillas 30 Jun '20

30 Jun '20

Hi, I'm trying to build a python package but my setup.py is actually named pkg1_setup.py (this is because I have multiple packages in the same repo). The package built is *almost* fine, except that setup.py is not included in the package built. I understand that the *normal* structure would be to have something like this: * setup.py * mypackage\... but before refactoring the repository structure to accommodate that I'd like to validate that there is no other way of doing that. I tried using the script_name parameter to setuptools.setup but had no luck. Any suggestions? Thanks a lot

2 1

ANN: distlib 0.3.1 released on PyPI
by Vinay Sajip 28 Jun '20

28 Jun '20

I've recently released version 0.3.1 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #132: Added documentation to help with relative interpreter paths. Thanks to Paul Kienzle for the patch. * Fixed #134: Allowed specifying a different target Python version when generating scripts. * Fixed #135: Exposed the ``enquote_executable`` function previously implemented as an internal function. * Addressed #138: Improved metadata support for newer metadata versions. Thanks to James Tocknell for the patch. * Changed the output of flags in entry point definitions in wheels. Thanks to frostming (明希) for the patch. * Stopped writing JSON metadata. Only old-style metadata is written now. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Note: Mailman3 might mishandle some of the links below. In that case, just copy and paste the links into your browser address bar - that should work. Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.3.1/ [2] https://distlib.readthedocs.io/en/0.3.1/ [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

Re: package management - common storage while keeping the versions straight
by Jason Madden 25 Jun '20

25 Jun '20

On Tue, 2020-06-23 at 15:51 -0700, David Mathog wrote: > What I am after is some method of keeping exactly one copy of each > package-version in the common area (ie, one might find foo-1.2, > foo-1.7, and foo-2.3 there), while also presenting only the one > version of each (let's say foo-1.7) to a particular installed program. > On linux it might do that by making soft links to the common > PYTHONPATH area from another directory for which it sets PYTHONPATH > for the application. Finally, this has to be usable by any account > which has read execute access to the main directory. > > Does such a beast exist? If so, please point me to it! zc.buildout[1] and zc.recipe.egg[2] can do something very much like this. zc.buildout tries hard to maintain reproducibility and isolation, and one of the ways it does this is by keeping each package in its own .egg directory. It then generates the entry-point scripts and a REPL with ``sys.path`` explicitly set to reference exactly the versions specified. There's no virtualenv-like activation step that puts all those scripts on the path, though, so one must either do that manually or just invoke the generated scripts directly. For example, here's a buildout configuration that specifies a shared directory to store eggs in. It also has some parts using zc.recipe.egg, one that will use zope.interface 4 and one that will use zope.interface 5 (egg specifications can be arbitrarily complex, of course, dependencies are followed, etc): [buildout] eggs-directory = /<path>/buildout-eggs abi-tag-eggs = true parts = old-interface new-interface new-interface-plus [old-interface] recipe = zc.recipe.egg eggs = zope.interface == 4.0 interpreter = old-py [new-interface] recipe = zc.recipe.egg eggs = zope.interface == 5.1 interpreter = new-py [new-interface-plus] recipe = zc.recipe.egg eggs = zope.interface == 5.1 zope.component interpreter = new-py-plus After running `buildout`, I have three REPL files (if zope.interface had defined any entry point scripts, I could also have had those generated): $ head bin/new-py #!<path/to/python> import sys sys.path[0:0] = [ '/<path>/buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg', '/<path/to/python>/site-packages', ] $ head bin/old-py #!<path/to/python> import sys sys.path[0:0] = [ '/<path>/buildout-eggs/pypy_73/zope.interface-4.0.0-py2.7.egg', '/<path/to/python>/site-packages', ] $ head bin/new-py-plus #!<path/to/python> import sys sys.path[0:0] = [ '/<path>/buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg', '/<path>/buildout-eggs/pypy_73/zope.component-4.6.1-py2.7.egg', '/<path/to/python>/site-packages', ] I've got a collection of zope.interface eggs referenced from a variety of different buildouts (at least at one point in time) and from a variety of different Python implementations, but only ever one copy of each: $ ls -ld buildout-eggs/*/zope.interface* Permissions Size User Date Modified Name drwxr-xr-x - jmadden 2017-05-04 06:53 buildout-eggs/pypy_41/zope.interface-4.4.0-py2.7.egg/ drwxr-xr-x - jmadden 2017-05-04 07:00 buildout-eggs/cp27m/zope.interface-4.4.0-py2.7-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-05-09 17:55 buildout-eggs/cp34m/zope.interface-4.4.0-py3.4-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-06-08 10:20 buildout-eggs/cp27m/zope.interface-4.4.1-py2.7-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-07-11 10:07 buildout-eggs/cp36m/zope.interface-4.4.2-py3.6-macosx-10.12-x86_64.egg/ drwxr-xr-x - jmadden 2017-12-08 11:10 buildout-eggs/cp27m/zope.interface-3.6.7-py2.7-macosx-10.13-x86_64.egg/ drwxr-xr-x - jmadden 2018-05-07 11:05 buildout-eggs/cp27m/zope.interface-4.1.3-py2.7-macosx-10.13-x86_64.egg/ drwxr-xr-x - jmadden 2020-03-13 07:53 buildout-eggs/cp27m/zope.interface-4.6.0-py2.7-macosx-10.15-x86_64.egg/ drwxr-xr-x - jmadden 2020-04-08 07:32 buildout-eggs/cp38/zope.interface-5.1.0-py3.8-macosx-10.15-x86_64.egg/ drwxr-xr-x - jmadden 2020-05-17 09:30 buildout-eggs/cp27m/zope.interface-5.1.0-py2.7.egg/ drwxr-xr-x - jmadden 2020-06-11 07:42 buildout-eggs/pypy_73/zope.interface-5.1.0-py2.7.egg/ drwxr-xr-x - jmadden 2020-06-25 11:58 buildout-eggs/pypy_73/zope.interface-4.0.0-py2.7.egg/ ~Jason [1] http://www.buildout.org/ ; I recommend version 3, which uses pip to find eggs [2] https://pypi.org/project/zc.recipe.egg/

1 0

Twine 3.2.0 released
by Brian Rutledge 24 Jun '20

24 Jun '20

https://pypi.org/project/twine/3.2.0/ Changelog: https://twine.readthedocs.io/en/latest/changelog.html This is my first release as a Twine maintainer. Thanks to: - Sumana Harihareswara for getting me started, and nudging me along the way - Ian Stapleton Cordasco, Jason R. Coombs, and Dustin Ingram for code review and guidance in general - Devesh Kumar Singh, Felipe Mulinari Rocha Campos, and Vikram Jayanthi for substantial contributions to this release

2 1

[PEP 508] Clarification of URL References
by w_minchin＠hotmail.com 23 Jun '20

23 Jun '20

PEP 508 allows for pointing to required packages using URL's. This is the except: A minimal URL based lookup:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346… However, there is very little documentation, blog posts, etc on how to use beyond the PEP itself, and so I struggled to understand it. I incorrectly assumed that *pip*, at the start of the line, had something to do with the given package being *installed via pip*, rather than *the name of the package being installed*. Therefore, is it possible to list another example line in the PEP to make it clear that the line doesn't have to start with "pip" to work? For further reading, my original issue, and the discovered solution --> https://github.com/jazzband/pip-tools/issues/1056 The explanation there that was helpful for me (that I didn't get from reading the PEP) was: > I believe the issue is in the name @ url pattern. It should be project_name @ vcs_url, no eggs param needed in this case by the way, but the project_name should be the same as package name in vcs_url/setup.py. [source](https://github.com/jazzband/pip-tools/issues/1056#issuecomment-5829… c.f. the issue I raised on GitHub, where I was suggested to post here --> https://github.com/python/peps/issues/1444 Thanks.

2 2

Distutils-SIG June 2020