Distutils-SIG December 2020

distutils-sig@python.org

14 participants
8 discussions

Archive this list & redirect conversation elsewhere?
by Pradyun Gedam 09 May '21

09 May '21

TL;DR: OK to archive this mailing list? Reply by Aug 30th. Below: Context, Proposal, Reasoning, and timeline. *Context*: For multiple years now, we've been advising users to use setuptools and to move away from using distutils. There has been a long standing plan [^1] to vendor distutils into setuptools to allow it to evolve independently of the Python standard library and to allow for removal of distutils from the Python standard library. Recently, setuptools adopted the distutils [^2] which, as far as I can tell, starts the long process of removing distutils from the Python standard library. PEP 517 [^3] also removed the special status of distutils/setuptools as the only pipeline that can be used for generating distributions for Python projects. For some time now, "distutils" has not been a "primary" tool in the broader Python Packaging ecosystem, with setuptools being an overall superior tool that also has a good interoperability story with the other Python packaging tooling. This is acknowledged in the description of this mailing list as: > Now, it's better described as the "packaging interoperability SIG", > where issues that cut across different parts of the Python packaging > ecosystem get discussed and resolved. However, this mailing list is no longer serving this stated role either, with the Packaging category on discuss.python.org becoming the primary location for packaging tool interoperability discussions. Over the last year, the Packaging category on discuss.python.org had 841 active topics, with only 40 topics with 3 or fewer responses. [^5] In the last 100 days, the Packaging category on discuss.python.org has had 91 active topics. More than 10 PEPs have been discussed in the Packaging category on discuss.python.org in the last 100 days. Over the last year, distutils-sig had ~109 active threads, with (based on a quick skim) most having 3 or fewer responses/posters. [^4] In the last 100 days, distutils-sig has had 32 active threads (at least 7 of these have the same subject as another thread with Re:/Fwd: added). There has been only 1 PEP-related feedback discussion on distutils-sig in the last year. Most of the other threads are user support requests or announcements. *Proposal*: I suggest that, one month from now, we stop posting to this list (distutils-sig(a)python.org) and archive it. *Reasoning*: I think we do not use this mailing list for its dedicated purpose, and it does not serve any secondary function that isn't better served by a different communication channel already. (1) this mailing list is no longer the primary location for interoperability discussions (2) we have better channels for user support requests (such as issue trackers of various projects, packaging-problems etc.) (3) we have other channels for making announcements (such as the pypi-announce mailing list, the PyPA twitter account, discuss.python.org etc.) *Timeline*: Here's what I suggest, and what I will carry out if there is no objection. In one month, on August 30th, I would verify that no one has argued here for why this mailing list should not be closed/archived. Or, even if a few people have objected to closing the list, I would check for rough consensus, especially of people who are doing SOMETHING productive having to do with Python Packaging (such as maintainers of Python packages, maintainers of Python Packaging tooling, folks running key infrastructure, etc.). Then, I would request the list administrators to post a final message to this list (marking its close and suggesting that people use discuss.python.org instead) and, to archive this mailing list. This would leave archives available at their current URLs, so links, browsing and search would work. And finally, I would look through relevant documentation within PyPA repositories to see what needs updating (READMEs and so on pointing to the old list), and submit pull requests. I appreciate the work folks here have done to carry forward Python packaging over the past 21 years of this mailing list. I don't mean to diminish that or to insult anyone here. I want to help us out, and I think closing this list will help focus our energy better. But I am open to hearing that I am wrong. Realizing-that-distutils-sig-is-older-than-me-ly, Pradyun Gedam [^1]: https://www.pypa.io/en/latest/roadmap/#vendor-distutils-into-setuptools [^2]: https://github.com/pypa/setuptools/pull/2143 [^3]: https://www.python.org/dev/peps/pep-0517/ [^4]: https://mail.python.org/archives/list/distutils-sig@python.org/latest?count… [^5]: https://discuss.python.org/c/packaging/14/l/top/yearly?order=posts

11 19

Making setup.py run external command to build
by Julian Smith 02 Apr '21

02 Apr '21

Hello I have a project with a fairly involved build process where we generate C++ via a python-clang-based code analyser, generates Python bindings for the resulting C++ API with SWIG, and finally compile and link to create various .so's and .py files. I'd like to package things up with a setup.py script, but from what i have read, it looks like distutils expects to compile, link and run swig itself, using distutils.core.setup()'s ext_modules arg? Is there a way to write setup.py so that it instead runs an external command (e.g. 'cd foo && make all') when it is asked to build? Thanks for any help, - Jules -- http://op59.net

7 19

Twine 3.3.0 released
by Brian Rutledge 24 Dec '20

24 Dec '20

https://pypi.org/project/twine/3.3.0/ Changelog (now via towncrier): https://twine.readthedocs.io/en/latest/changelog.html Notable improvements include more `twine upload --verbose` output, and a `--strict` option for `twine check`.

1 0

Python-Setuptools EOL Versions?
by Symphoni Bush - NOAA Affiliate 16 Dec '20

16 Dec '20

Good afternoon, I am trying to find out if there are any end-of-life versions for Python-Setuptools, and if so, when do these versions typically become EOL/unsupported? If this information is publicly stored anywhere please let me know. Thanks.

1 0

AWS Glue: Easy_Install usecase
by simon.jones＠axtria.com 15 Dec '20

15 Dec '20

Hi everyone, https://setuptools.readthedocs.io/en/latest/deprecated/easy_install.html instructed me that the mailing list would like to know any use cases where Easy_install is felt needed and I believe I have one. and I do not see any posts about this specific use case. AWS Glue is a serverless runtime environment offered by AWS, usually it runs Spark jobs but Jan 22, 2019 they released Python Shells to run quick snippet codes. they have a fairly comprehensive list of preinstalled libraries and allow you to specify wheels and eggs for anything additional stored on an S3 that you would need, with one limitation - no C-extensions, Cython wheels cannot be installed. for my use-case I needed to access some SFTPs but Paramiko depends on cryptography for SSH shenanigans and cryptography (along with its dependencies cffi, PyNaCL, and bcrypt) are CPython. Turns out if you install at runtime there is no issue - it's just with initial setup that has the limitation. of course you then have to install every time that a process runs during paid runtime, rather than startup but better that then not at all. I tried 2 approaches to install at runtime: subprocess.call(['pip' ...]) and setuptools.command.easy_install.main() subprocess.call to pip installs pure python very nicely, but every time it hit a CPython wheel/tarball - CFFI, PyNaCl, bcrypt, cryptography, and paramiko - took 6-7 minutes EACH just to install (files were provided locally, downloaded from the S3 from previous attempt seeing what would happen if you just gave it CPython ) In a separate run setuptools...easy_install processed everything in 2 seconds. Somewhere between pkg_resources.getdistribution('paramiko').activate(), importing them, and setting host keys lead to Main process starting a little less than 2 minutes later. Required use-case? Not technically Fringe use-case? Fairly Appreciated that this is still around? Very much, 2-minute startup runtime is appreciated over upwards of half an hour on every single run.

1 0

pip 20.3 release (new resolver as default)
by Sumana Harihareswara 13 Dec '20

13 Dec '20

On behalf of the PyPA and the pip team, I am pleased to announce that we have just released pip 20.3, a new version of pip. You can install it by running `python -m pip install --upgrade pip`. [Cross-posted to https://discuss.python.org/t/announcement-pip-20-3-release/5948 which will be easier to read in a web browser and to link to.] This is an important and disruptive release -- we [explained why in a blog post last year](https://pyfound.blogspot.com/2019/12/moss-czi-support-pip.html). We even made [a video about it](https://www.youtube.com/watch?v=B4GQCBBsuNU). ## Highlights * **DISRUPTION**: Switch to the new dependency resolver by default. (#9019) Watch out for changes in handling editable installs, constraints files, and more: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-res… * **DEPRECATION**: Deprecate support for Python 3.5 (to be removed in pip 21.0) (#8181) * **DEPRECATION**: pip freeze will stop filtering the pip, setuptools, distribute and wheel packages from pip freeze output in a future version. To keep the previous behavior, users should use the new `--exclude` option. (#4256) * Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files. * Support for PEP 600: Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions. (#9077) * Documentation improvements: Resolver migration guide, quickstart guide, and new documentation theme. * Add support for MacOS Big Sur compatibility tags. (#9138) The new resolver is now *on by default*. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please see [our guide on how to test and migrate, and how to report issues](https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020). You can use the deprecated (old) resolver, using the flag `--use-deprecated=legacy-resolver`, until we remove it in the pip 21.0 release in January 2021. You can find more details (including deprecations and removals) [in the changelog](https://pip.pypa.io/en/stable/news/). ## User experience Command-line output for this version of pip, and documentation to help with errors, is significantly better, because you worked with our experts to test and improve it. [Contribute to our user experience work: sign up to become a member of the UX Studies group](https://bit.ly/pip-ux-studies) (after you join, we'll notify you about future UX surveys and interviews). ## What to expect in 20.1 We aim to release pip 20.1 in January 2021, per our [usual release cadence](https://pip.pypa.io/en/latest/development/release-process/#release-cadence). You can expect: * Removal of [Python 2.7](https://pip.pypa.io/en/latest/development/release-process/#python-2-support) and 3.5 support * Further improvements in the new resolver * Removal of legacy resolver support ## Thanks As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Specific thanks go to Mozilla (through its [Mozilla Open Source Support](https://www.mozilla.org/en-US/moss/) Awards) and to the [Chan Zuckerberg Initiative](https://chanzuckerberg.com/eoss/) DAF, an advised fund of Silicon Valley Community Foundation, for their funding that enabled substantial work on the new resolver. That funding went to [Simply Secure](https://simplysecure.org/) (specifically Georgia Bullen, Bernard Tyers, Nicole Harris, Ngọc Triệu, and Karissa McKelvey), [Changeset Consulting](https://changeset.nyc/) (Sumana Harihareswara), [Atos](https://www.atos.net) (Paul F. Moore), [Tzu-ping Chung](https://uranusjr.com), [Pradyun Gedam](https://pradyunsg.me/), and Ilan Schnell. Thanks also to Ernest W. Durbin III at the Python Software Foundation for liaising with the project. -- Sumana Harihareswara, pip project manager Changeset Consulting https://changeset.nyc

2 2

Installation Instructions for Python?
by Cort, Brian 11 Dec '20

11 Dec '20

Hello, We were trying to create an automated installation process. As part of this, our software deployment team has requested the installation instructions, which I was unable to locate. Can you help by pointing me towards them? Thank you, Brian Brian Cort | Application Analyst Kroger Technology: Point of Sale | The Kroger Co. office: 503.797.3229 cell: 971.293.6244 [cid:image001.png@01D6CD4D.01E06F70] For more POS related information, please feel free to see our publicly accessible Wiki/Confluence Site: https://confluence.kroger.com/confluence/x/BySXBQ ________________________________ This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is confidential and protected by law from unauthorized disclosure. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

4 3

Critical problem in PyCharm caused by the removal of "--build-dir" in 2020.3
by Mikhail Golubev 01 Dec '20

01 Dec '20

Hi there! I'm a member of the PyCharm team, and we faced a rather critical issue with the release of pip 2020.3 (https://youtrack.jetbrains.com/issue/PY-45712). It turned out that we actually still used the "--build-dir" option in our internal scripts for installing packages through pip, and we completely missed the fact that it was deprecated. To make the matter worse, a few releases back we implemented automatic upgrading of packaging tools in all virtual environments created in the IDE, so now it's impossible to install anything in a new project unless pip is explicitly downgraded first. Judging by how pip now operates regarding building of packages, this option can be safely dropped in our wrappers, but we still need some time to prepare and test patches for a few previous releases and upcoming 2020.3. Is it possible to give us some time to soften the blow for the users? The second question is about the behavior of this option. It appears that we initially started using it because in the past packages were not built in a temporary directory by default. Could you please point me to the exact version of pip where it changed? -- I couldn't find it in a changelog. It would help us decide whether we need to keep some compatibility layer for interpreters with an old version of pip installed. Thanks a lot!

4 4

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG December 2020