Distutils-SIG July 2020

distutils-sig@python.org

28 participants
16 discussions

Archive this list & redirect conversation elsewhere?
by Pradyun Gedam 09 May '21

09 May '21

TL;DR: OK to archive this mailing list? Reply by Aug 30th. Below: Context, Proposal, Reasoning, and timeline. *Context*: For multiple years now, we've been advising users to use setuptools and to move away from using distutils. There has been a long standing plan [^1] to vendor distutils into setuptools to allow it to evolve independently of the Python standard library and to allow for removal of distutils from the Python standard library. Recently, setuptools adopted the distutils [^2] which, as far as I can tell, starts the long process of removing distutils from the Python standard library. PEP 517 [^3] also removed the special status of distutils/setuptools as the only pipeline that can be used for generating distributions for Python projects. For some time now, "distutils" has not been a "primary" tool in the broader Python Packaging ecosystem, with setuptools being an overall superior tool that also has a good interoperability story with the other Python packaging tooling. This is acknowledged in the description of this mailing list as: > Now, it's better described as the "packaging interoperability SIG", > where issues that cut across different parts of the Python packaging > ecosystem get discussed and resolved. However, this mailing list is no longer serving this stated role either, with the Packaging category on discuss.python.org becoming the primary location for packaging tool interoperability discussions. Over the last year, the Packaging category on discuss.python.org had 841 active topics, with only 40 topics with 3 or fewer responses. [^5] In the last 100 days, the Packaging category on discuss.python.org has had 91 active topics. More than 10 PEPs have been discussed in the Packaging category on discuss.python.org in the last 100 days. Over the last year, distutils-sig had ~109 active threads, with (based on a quick skim) most having 3 or fewer responses/posters. [^4] In the last 100 days, distutils-sig has had 32 active threads (at least 7 of these have the same subject as another thread with Re:/Fwd: added). There has been only 1 PEP-related feedback discussion on distutils-sig in the last year. Most of the other threads are user support requests or announcements. *Proposal*: I suggest that, one month from now, we stop posting to this list (distutils-sig(a)python.org) and archive it. *Reasoning*: I think we do not use this mailing list for its dedicated purpose, and it does not serve any secondary function that isn't better served by a different communication channel already. (1) this mailing list is no longer the primary location for interoperability discussions (2) we have better channels for user support requests (such as issue trackers of various projects, packaging-problems etc.) (3) we have other channels for making announcements (such as the pypi-announce mailing list, the PyPA twitter account, discuss.python.org etc.) *Timeline*: Here's what I suggest, and what I will carry out if there is no objection. In one month, on August 30th, I would verify that no one has argued here for why this mailing list should not be closed/archived. Or, even if a few people have objected to closing the list, I would check for rough consensus, especially of people who are doing SOMETHING productive having to do with Python Packaging (such as maintainers of Python packages, maintainers of Python Packaging tooling, folks running key infrastructure, etc.). Then, I would request the list administrators to post a final message to this list (marking its close and suggesting that people use discuss.python.org instead) and, to archive this mailing list. This would leave archives available at their current URLs, so links, browsing and search would work. And finally, I would look through relevant documentation within PyPA repositories to see what needs updating (READMEs and so on pointing to the old list), and submit pull requests. I appreciate the work folks here have done to carry forward Python packaging over the past 21 years of this mailing list. I don't mean to diminish that or to insult anyone here. I want to help us out, and I think closing this list will help focus our energy better. But I am open to hearing that I am wrong. Realizing-that-distutils-sig-is-older-than-me-ly, Pradyun Gedam [^1]: https://www.pypa.io/en/latest/roadmap/#vendor-distutils-into-setuptools [^2]: https://github.com/pypa/setuptools/pull/2143 [^3]: https://www.python.org/dev/peps/pep-0517/ [^4]: https://mail.python.org/archives/list/distutils-sig@python.org/latest?count… [^5]: https://discuss.python.org/c/packaging/14/l/top/yearly?order=posts

11 19

Announcement: pip 20.2 release!
by Pradyun Gedam 23 Oct '20

23 Oct '20

On behalf of the PyPA, I am pleased to announce that we have just released pip 20.2, a new version of pip. You can install it by running python -m pip install --upgrade pip. The highlights for this release are: - The beta of the next-generation dependency resolver is available - Faster installations from wheel files - Improved handling of wheels containing non-ASCII file contents - Faster pip list using parallelized network operations - Installed packages now contain metadata about whether they were directly requested by the user (PEP 376’s REQUESTED file) The new dependency resolver is *off by default* because it is *not yet ready for everyday use*. The new dependency resolver is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the --use-feature=2020-resolver flag. Please see our guide on how to test and migrate, and how to report issues <https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-res…>. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3 (in October 2020). This release also partially optimizes pip’s network usage during installation (as part of a Google Summer of Code project by McSinyx <https://summerofcode.withgoogle.com/projects/#5428041779511296>). Please test it with pip install --use-feature=fast-deps ... and report bugs to the issue tracker <https://github.com/pypa/pip/issues/new?template=bug-report.md>. This functionality is *still experimental* and *not ready for everyday use*. You can find more details (including deprecations and removals) in the changelog <https://pip.pypa.io/en/stable/news/>. As with all pip releases, a significant amount of the work was contributed by pip’s user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Specific thanks go to Mozilla (through its Mozilla Open Source Support <https://www.mozilla.org/en-US/moss/> Awards) and to the Chan Zuckerberg Initiative <https://chanzuckerberg.com/eoss/> DAF, an advised fund of Silicon Valley Community Foundation, for their funding that enabled substantial work on the new resolver.

2 2

Failure: Error
by oliviajohn123123＠hotmail.com 21 Sep '20

21 Sep '20

facing same NOt found issue the issue code is "C:\Users\jaraco\projects\distutils_namespace\myns.projB\test\testBasic.py", can you guide me for this ? https://thelifefoundry.com/hotmail/

10 9

Improving Communication
by Donald Stufft 30 Jul '20

30 Jul '20

Currently in the packaging space, we have a number of avenues for communication, which are: - distutils-sig - pypa-dev - virtualenv-users - Other project specific mailing lists - IRC - gitter - Various issue trackers spread across multiple platforms. - Probably more places I’m not remembering. The result of this is that all discussion ends up being super fractured amongst the various places. Sometimes that is exactly what you want (for instance, someone who is working on the wheel specs probably doesn’t care about deprecation policy and internal module renaming in pip) and sometimes that ends up being the opposite of what you want (for instance, when you’re describing something that touches PyPI, setuptools, flit, pip, etc all at once). Theoretically the idea is that distutils-sig is where cross project reaching stuff goes, IRC/gitter is where real time discussion goes, and the various project mailing lists and issue trackers are where the project specific bits go. The problem is that often times doesn’t actually happen in practice except for the largest and most obvious of changes. I think our current “communications stack” kind of sucks, and I’d love to figure out a better way for us to handle this that solves the sort of weird “independent but related” set of projects we have here. From my POV, a list of our major problems are: * Discussion gets fractured across a variety of platforms and locations, which can make it difficult to actually keep up with what’s going on but also to know how to loop in someone relevant if their input would be valuable. You have to more or less simultaneously know someone’s email, Github username, IRC nick, bitbucket username, etc to be able to bring threads of discussion to people’s attention. * It’s not always clear to users where a discussion should go, often times they’ll come to one location and need to get redirected to another location. If any discussion did happen in the incorrect location, it tends to need to get restarted in the new location (and depending on the specific platform, it may be impossible to actually redirect everyone over to the proper location, so you again, end up fractured with the discussion happening in two places). * A lot of the technology in this stack is particularly old, and lacks a lot of the modern day affordances that newer things have. An example is being able to edit a discussion post to fix typos that can hinder the ability of others to actually understand whats being talked about. In your typical mailing list or IRC there’s no mechanism by which you can edit an already sent message, so your only option is to either let the problem ride and hope it doesn’t trip up too many people, or send an additional message to correct the error. However these show up as additional, later messages which someone might not even see until they’ve already been thoroughly confused by the first message (since people tend to read email/IRC in a linear fashion). - There is a lot of things in this one, other things are things like being able to control in a more fine grained manner what email you’re going to get. - Holy crap, formatting and typography to make things actually readable and not a big block of plaintext. * We don’t have a clear way for users to get help, leaving users to treat random issues, discussion areas, etc as a support forum, rather than some place that’s actually optimized for that. Some of this ties back into some of the earlier things too, where it’s hard to actually redirect discussions These aren’t *new* problems, and often times the existing members of a community are the least effected becasue they’ve already spent effort learning the ins and outs and also curating a (typically custom) workflow that they’ve grown accustomed too. The problem with that is that often times that means that new users are left out, and the community gets smaller and smaller as time goes on as people leave and aren’t replaced with new blood, because they’re driven off but the issues with the stack. A big part of the place this is coming from, is me sitting back and realizing that I tend to be drawn towards pulling discussions into Github issues rather than onto the varying mailing lists, not because that’s always the most appropriate place for it, but because it’s the least painful place in terms of features and functionality. I figure if I’m doing that, when I already have a significant investment in setting up tooling and being involved here, that others (and particularly new users) are likely feeling the same way. - Donald

9 12

The Error NOT FOUND On My Project
by xelekid611＠mado34.com 25 Jul '20

25 Jul '20

I am facing the issue of not found "C:\Users\jaraco\projects\distutils_namespace\myns.projB\test\testBasic.py", can anyone guide me for this ? https://apkpalace.net/

1 0

Re: Fwd: Re: Use of "python" shebang an installation error?
by Filipe Laíns 22 Jul '20

22 Jul '20

On Tue, 2020-07-21 at 14:24 -0700, David Mathog wrote: > On Tue, Jul 21, 2020 at 2:03 PM Filipe Laíns <lains(a)archlinux.org> > wrote: > > On Tue, 2020-07-21 at 13:51 -0700, David Mathog wrote: > > > biopython-1.77, for instance, when installed into a virtualenv > > > with > > > pip3, has many of these shebangs: > > > > > > #!/usr/bin/env python > > > > > Is it being installed as an egg or wheel? Can you check what > > happens > > when you pass --use-pep517 to the pip install command (to force > > installing as a wheel)? > > > > It is wheel, it creates a directory > > biopython/lib/python3.6/site-packages/biopython-1.77.dist-info > > when installed either of these ways: > > pip3 install biopython > pip3 install biopython --use-pep517 > > Either way it has the same shebang in the files listed a few posts > up. > > Thanks, > > David Mathog You've fallen off-list again. The shebang handling for wheels is very well defined[1]. It looks like those shebangs come from biopython so it's their bug. No file a python package is supposed to be executed directly, so they shouldn't really have shebangs. The files that are superposed to be executed directly should be defined as scripts and have a #!python shebang so that the wheel installer can do its job properly. So unfortunately I think in most of the packages you are having issues with, the bug comes from the package upstream :/ [1] https://www.python.org/dev/peps/pep-0427/#recommended-installer-features Filipe Laíns

1 0

Fwd: Re: Use of "python" shebang an installation error?
by David Mathog 22 Jul '20

22 Jul '20

(oops, had to resend, forgot to change the destination to <distutils-sig(a)python.org>) On Mon, Jul 20, 2020 at 12:38 PM John Thorvald Wodder II <jwodder(a)gmail.com> wrote: > > On 2020 Jul 20, at 15:25, David Mathog <dmathog(a)gmail.com> wrote: > > Lately I have been working on a CentOS 8 machine, and it has "python2" > > and "python3", but no "python". Many packages install scripts with a > > shebang like: > > > > #!/usr/bin/env python > > > > and those do not work on this OS. Seems like rather a large missing > > dependency which goes by without triggering a fatal error. > > How exactly are these packages getting installed? Last time I checked, both pip and setuptools automatically set the shebang in scripts (both console_script entry points and scripts declared with the "scripts" argument to `setup()`) to use the path of the running Python interpreter. Are these packages installed using your system package manager? If so, you should take the problem up with its maintainers. Good point, I have been installing so many packages I get confused about which installer was used for which package. It turned out that many (but not all) of the files which contained #!/usr/bin/env python shebangs were installed using standard OS level tools (cmake, configure, make and the like). Example package, hisat2. I guess there isn't much choice for those but to scan the directories for python scripts and fix the shebangs. Installs that are initially into venvs and used pip3 are still an issue. Example: python3 -m venv johnnydep cd johnnydep grep -r '/usr/bin/env python$' . #finds: ./lib/python3.6/site-packages/pip/_vendor/appdirs.py:#!/usr/bin/env python ./lib/python3.6/site-packages/pip/_vendor/chardet/cli/chardetect.py:#!/usr/bin/env python ./lib/python3.6/site-packages/pip/_vendor/requests/certs.py:#!/usr/bin/env python ./lib/python3.6/site-packages/pkg_resources/_vendor/appdirs.py:#!/usr/bin/env python ./lib/python3.6/site-packages/johnnydep/pipper.py:#!/usr/bin/env python cd bin ls -1 | grep python lrwxrwxrwx. 1 modules modules 7 Jul 20 14:09 python -> python3 lrwxrwxrwx. 1 modules modules 16 Jul 20 14:09 python3 -> /usr/bin/python3 source activate pip3 install johnnydep head -1 johnnydep #!/home/common/lib/python3.6/Envs/johnnydep/bin/python #same for "tabulate" and all other shebangs in bin. cd .. grep -r '/usr/bin/env python$' . #same as before grep -r '/home/common/lib/python3.6/Envs/johnnydep/bin/python' . #just the files in the bin directory. It looks like none of the "#!/usr/bin/env python" shebangs within the venv are going to be used after the install, so perhaps those are harmless. The shebangs like #!/home/common/lib/python3.6/Envs/johnnydep/bin/python are OK within the venv, but once they are "devirtualized" they become a problem. That was a known problem though - my devirtualizer code already patches all of the ones in the bin directory. I have not seen any elsewhere (yet) within the venv, but there is probably no rule that keeps them from appearing in "share" or elsewhere. The "python" in use in the venv is just a symbolic link to "python3" which is itself a symbolic link to the actual program "/usr/bin/python3". It is constructed that way based on "python -m venv" which uses pieces which come from the CentOS 8 python3-libs-3.6.8-23.el8.x86_64 RPM. Is there some requirement that a venv have a "python"? Odd that RedHat (and so CentOS) provide a "python" there, but not in the OS itself. Thanks, David Mathog

6 8

Fwd: Re: Use of "python" shebang an installation error?
by David Mathog 21 Jul '20

21 Jul '20

(oops, had to resend, forgot to change the destination to <distutils-sig(a)python.org>) biopython-1.77, for instance, when installed into a virtualenv with pip3, has many of these shebangs: #!/usr/bin/env python And they are all over the place. They are: ./site-packages/Bio/bgzf.py: ./site-packages/Bio/PDB/parse_pdb_header.py: ./site-packages/Bio/PDB/PDBList.py: ./site-packages/Bio/Restriction/__init__.py: ./site-packages/Bio/Restriction/Restriction.py: ./site-packages/Bio/Restriction/PrintFormat.py: ./site-packages/Bio/Restriction/Restriction_Dictionary.py: ./site-packages/Bio/Wise/__init__.py: ./site-packages/Bio/Wise/psw.py: ./site-packages/Bio/Wise/dnal.py: ./site-packages/Bio/UniProt/GOA.py: ./site-packages/Bio/SeqUtils/__init__.py: Regards, David Mathog

2 1

Fwd: Re: Use of "python" shebang an installation error?
by David Mathog 21 Jul '20

21 Jul '20

On Mon, Jul 20, 2020 at 4:40 PM John Thorvald Wodder II <jwodder(a)gmail.com> wrote: > > First of all, your last two messages only went to me, not to the list. The mailing list doesn't set Reply-To on messages or the like, so you have to manually set "To: distutils-sig(a)python.org" when replying. Aargh, right. I use gmail for my home mail, and since I'm stuck working at home, that is what I used here. Gmail likes to hide, well, pretty much everything. I will repost those responses. > > As to your e-mail, though, are any of those files even meant to be executed? They're not in bin/; they just appear to be regular source files that some developer slapped a shebang on. That in a sense is the issue. I don't know, you don't know, maybe the developer knows (if he/she still remembers). I really don't want to do the work to dig through the code for every package I install to determine if a shebang is used or not. Yet if I don't figure this out some end user will run a script (one of a hundred in some package I installed for their use) which will blow up because of this issue. The best I can do now is run pdvctrl reshebang $TARGET_DIR or pdvctrl reshebang $ROOT_DIR... and fix them up after the fact. (pdvctrl from python_devirtualizer here: https://sourceforge.net/projects/python-devirtualizer/). Even then it usually has to guess that "python" means "python3" and not "python2", and sometimes it guesses wrong. Today's version of that recurring issue: https://github.com/lastz/lastz/issues/30 Regards, David Mathog

1 0

pip install very slow
by Stuart McGraw 21 Jul '20

21 Jul '20

Hello, First time trying to use Python's package tools and I'm attempting to use pip to install from a local project: python3 -m pip install --user --upgrade --no-deps ./ I have a simple directory structure very similar to that recommended in https://packaging.python.org/tutorials/packaging-projects/ project/ data/ mypackage/ __init__.py data/ *.csv module1.py module2.py tmpl/ *.jinja setup.py The project/data/ directory contains several GB of stuff I use during development. It is not supposed to be part of the package and it is not mentioned in setup.py. The problem is when I run the install command above it takes many minutes to finish. When I move the project/data/ sub-directory out of the project/ directory, it takes a few seconds. I thought perhaps that "packages=setuptools.find_packages()" in my setup.py file might be the culprit but replacing in with "packages=['mypackage']" made no difference. I'm not sure why pip seems to be roaming through project/data/ (if that's what's happening) but is there some way to get it not to? My apologies if this is covered in the docs somewhere; I'm still going though them but didn't see anything with an initial scan. If it helps, here is my setup.py file: setuptools.setup ( name="mypackage", version="...", author="Me", author_email="...", description="...", long_description="...", long_description_content_type="text/markdown", url="...", packages=['mypackage'], package_data={'mypackage': ['data/*.csv', 'tmpl/*.jinja',]}, zip_safe = False, classifiers=[...], python_requires='>=3.6', install_requires = ['jinja2',], ) Thanks for any enlightenment.

4 5

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG July 2020