Distutils-SIG September 2018

distutils-sig@python.org

42 participants
29 discussions

by Alex Grönholm

This release contains the following significant changes: * Wheel signature generation and verification commands have been removed * The wheel installation features have been removed * The documentation has been rewritten to conform to the PyPA guidelines * The ability to add multiple license files to the wheel has been added (the most common ones are included by default) There were also a few bug fixes as well. If you are left missing the removed features, let me know on the issue tracker.

2 years, 11 months

setuptools configuration in pyproject.toml

by Bernat Gabor

I'm aware this might be a controversial subject, so let's have the initial discussion about it here first for full disclosure and see what people think about it. Should setuptools support pyproject.toml as configuration source or not (alternative to setup.cfg which it already does - https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-set… )? The main benefit of having this would be to decrease configuration files and have build dependencies and other types of dependencies in one location. Furthermore many other packaging projects (flit, poetry) already do define their dependencies inside pyproject.toml; so would create one unified location where to look for such in the future. The counter-argument is that "a big part of pyproject.toml was keeping that file clean" and would furthermore increase the size of that file down the line. So what do people think? Should we encourage or discourage to have a single python project file? I'm personally supporting build/code quality tools supporting pyproject.toml as their main configuration file. Thanks

2 years, 11 months

Stepping away from Twine maintainership

by Sumana Harihareswara

Quick note to thank Ian Stapleton Cordasco and Thea Flowers for their work maintaining Twine! I realized I don't have time to help maintain it right now so I'm stepping away from that, and am grateful for their work, including new releases this month: https://pypi.org/project/twine/#history And thanks to Dustin Ingram for all his recent work on Twine as well. As he said https://twitter.com/di_codes/status/1044358639081975813 : > New twine subcommand: $ twine check dist/* > Use it to verify that the README for your package is valid and will be rendered correctly on PyPI. > Between that and Markdown support, there's no excuse for mis-rendered PyPI descriptions anymore! More details: https://packaging.python.org/guides/making-a-pypi-friendly-readme/#validati… -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

2 years, 11 months

New virtualenv maintainer

by Donald Stufft

We don’t typically bother to make big announcements on distutils-sig for maintainers of projects, however I’m making an exception in this case because of the recent threads on virtualenv maintenance. The current core team of virtualenv agrees that it is an important package, and deserves additional maintenance, but have somewhat stepped back from it in recent times for a variety of reasons (many ephemeral in nature). We’ve been happy to have a number of people step forward to offer help in maintaining the project, and after some thought and discussion, we’ve decided to extend an invitation of a commit bit to Bernat Gabor, and I’m happy to say that he has graciously accepted, and is now part of the team. For those unaware, Bernat is currently a developer for the tox project, which is one of the major consumers of virtualenv which made him a great fit for maintainership since it gives another real world perspective (consuming virtualenv as an API, instead of as a tool) that the current team currently lacked. So Welcome Bernat!

2 years, 12 months

Re: Distlib vs Packaging (Was: disable building wheel for a package)

by Tzu-ping Chung

Pipenv’s resolver does not use troove classifiers as far as I am aware. I could be missing something though, the current resolver implementation is not the best code base to reason with. Can you provide some ideas on this, Dan? TP > On 20/9, 2018, at 15:09, Bernat Gabor <gaborjbernat(a)gmail.com> wrote: > > One of such differences is pipenv apparently trying to work from the troove classifiers for version compatibility instead of the existing python_requires (see https://github.com/tox-dev/tox/pull/1005 <https://github.com/tox-dev/tox/pull/1005>). Why? Point in case how people start to think pipenv is pip. At this point, pip is a completely different beast than pip I think. > > Such check at best would fall under the ``twine check`` command at PyPi upload time, not at install time. > > On Thu, Sep 20, 2018 at 8:03 AM Tzu-ping Chung <uranusjr(a)gmail.com <mailto:uranusjr@gmail.com>> wrote: > > >> On 20/9, 2018, at 13:22, Chris Jerdonek <chris.jerdonek(a)gmail.com <mailto:chris.jerdonek@gmail.com>> wrote: >> >> On Wed, Sep 19, 2018 at 8:54 PM, Dan Ryan <dan(a)danryan.co <mailto:dan@danryan.co>> wrote: >> I should clarify that we have already implemented a number of these as >> libraries over the last several months (and I am super familiar with pip's >> internals by now and I'm sure TP is getting there as well). More on this >> below >> ... >> We are super cognizant of that aspect as I am pretty sure we are hitting >> this wall in a full (nearly) pip-free reimplementation of all of the pipenv >> internals from the ground up, including wheel building/installation, but we >> basically had to start by calling pip directly, then slowly reimplement each >> aspect of the underlying logic using various elements in distlib/setuptools >> or rebuilding those. >> >> Is the hope or game plan then for pipenv not to have to depend on pip? This is partly what I was trying to learn in my email to this list a month ago (on Aug. 20, with subject: "pipenv and pip"): >> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2… <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2…> >> >> Based on the replies, I wasn't getting that impression at the time (though I don't remember getting a clear answer), but maybe things have changed since then. > > The resolution side of Pipenv really needs a Python API, and also cannot really > use the CLI because it needs something slightly different than pip’s high-level > logic (Nick mentioned this briefly). If we can’t use pip internals, then yes, the plan > is to not depend on pip. The hope is we can share those internals with pip (either > following the same standards, or using the same implementation), hence my series > of questions. > > The installation side of Pipenv will continue to use pip directly, at least for a while > more even after the resolution side breaks away, since “pip install” is adequate > enough for our purposes. There are some possible improvements if there is a > lower-layer library (e.g. to avoid pip startup overhead), but that is far less important. > > >> >> It should certainly be a lot easier for pipenv to move fast since there is no legacy base of users to maintain compatibility with. However, I worry about the fracturing this will cause. In creating these libraries, from the pip tracker it doesn't look like any effort is going into refactoring pip to make use of them. This relates to the point I made earlier today about how there won't be an easy way to cut pip over to using a new library unless an effort is made from the beginning. Thus, it's looking like things could be on track to split the user and maintainer base in two, with pip bearing the legacy burden and perhaps not seeing the improvements. Are we okay with that future? > > I’m afraid the new implementation will still need to deal with compatibility issues. > Users expect Pipenv to work exactly as pip, and get very angry if it does not, > especially when they see it is under the PyPA organisation on GitHub. The last > time Pipenv tries to explain it does whatever arbitrary things it does, we get > labelled as “toxic” (there are other issues in play, but this is IMO the ultimate > cause). Whether the image is fair or not, I would most definitely want to avoid > similar incidents from happening again. > > I think Pipenv would be okay to maintain a different (from scratch) implementation > than pip’s, but it would still need to do (almost) exactly what pip is doing, unless > we can have people (pip maintainers or otherwise) backing the differences. Whether > pip uses the new implementation or not wouldn’t change the requirement :( > > > TP > > >> >> --Chris >> >> >> Since you mentioned following along, here's what we're working on right now: >> >> https://github.com/sarugaku/requirementslib <https://github.com/sarugaku/requirementslib> -- abstraction layer for parsing >> and converting various requirements formats >> (pipfile/requirements.txt/command line/InstallRequirement) and moving >> between all of them >> https://github.com/sarugaku/resolvelib <https://github.com/sarugaku/resolvelib> -- directed acyclic graph library for >> handling dependency resolution (not yet being used in pipenv) >> https://github.com/sarugaku/passa <https://github.com/sarugaku/passa> -- dependency resolver/installer/pipfile >> manager (bulk of the logic we have been talking about is in here right now) >> -- I think we will probably split this back out into multiple other smaller >> libraries or something based on the discussion >> https://github.com/sarugaku/plette <https://github.com/sarugaku/plette> -- this is a rewrite of pipfile with some >> additional logic / validation >> https://github.com/sarugaku/shellingham <https://github.com/sarugaku/shellingham> -- this is a shell detection library >> made up of some tooling we built in pipenv for environment detection >> https://github.com/sarugaku/pythonfinder <https://github.com/sarugaku/pythonfinder> -- this is a library for finding >> python (pep 514 compliant) by version and for finding any other executables >> (cross platform) >> https://github.com/sarugaku/virtenv <https://github.com/sarugaku/virtenv> -- python api for virtualenv creation >> >> Happy to provide access or take advice as needed on any of those. Thanks >> all for the receptiveness and collaboration >> >> Dan Ryan >> gh: @techalchemy // e: dan(a)danryan.co <mailto:dan@danryan.co> >> >> From: Donald Stufft [mailto:donald@stufft.io <mailto:donald@stufft.io>] >> Sent: Wednesday, September 19, 2018 1:52 PM >> To: Tzu-ping Chung >> Cc: Distutils >> Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel >> for a package) >> >> My general recommendation if you want a Python implementation/interface for >> something pip does, is: >> >> - Open an issue on the pip repository to document your intent and to make >> sure that there is nobody there who is against having that functionality >> split out. This might also give a chance for people with familiarity in that >> API to mention pain points that you can solve in a new API. We can also >> probably give you a good sense if the thing you want in a library is >> something that probably has multiple things that are dependent on getting >> split out first (for instance, if you said you wanted a library for >> installing wheels, we'd probably tell you that there is a dependency on PEP >> 425 tags, pip locations, maybe other that need resolved first) and also >> whether this is something that should have a PEP first or not. Getting some >> rough agreement on the plan to split X thing out before you start is overall >> a good thing. >> >> - Create or update a PEP if required, and get it into the provisional state. >> >> - Make the library, either as a PR to packaging or as it's own independent >> library. If there are questions that come up while creating that library/PR >> that have to do with specific pip behaviors, go back to that original issue >> and ask for clarification etc. Ideally at some point you'll open a PR on pip >> that uses the new library (my suggestion is to not bundle the library in the >> initial PR, and just import it normally so that the PR diff doesn't include >> the full bundled library until there's agreement on it). If there's another >> tool (pipenv, whatever) that is looking to use that same functionality, open >> a WIP PR there too that switches it to using that. Use feedback and what you >> learn from trying to integrate in those libraries to influence back the >> design of the API itself. >> >> Creating a PEP and creating the library and the PRs can happen in parallel, >> but at least for pip if something deserves a PEP, we're not going to merge a >> PR until that PEP is generally agreed on. However it can be supremely useful >> to have them all going at the same time, because you run into things that >> you didn't really notice until you went to actually implement it. >> >> My other big suggestion would be to e careful about how much you bite off at >> one time. Pip's internal code base is not the greatest, so pulling out >> smaller chunks at a time rather than trying to start right off pulling out a >> big topic is more likely to meet with success. Be cognizant of what the >> dependencies are for the feature you want to implement, because if it has >> dependencies, you'll need to pull them out first before you can pull it out >> OR you'll need to design the API to invert those dependencies so they get >> passed in instead. >> >> I personally would be happy to at a minimum participate on any issue where >> someone was trying to split out some functionality from pip into a re-usable >> library if not follow the develop of that library directly to help guide it >> more closely. My hope for pip is that it ends up being the glue around a >> bunch of these libraries, and that it doesn't implement most of the stuff >> itself anymore. >> -- >> Distutils-SIG mailing list -- distutils-sig(a)python.org <mailto:distutils-sig@python.org> >> To unsubscribe send an email to distutils-sig-leave(a)python.org <mailto:distutils-sig-leave@python.org> >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/> >> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…> > -- > Distutils-SIG mailing list -- distutils-sig(a)python.org <mailto:distutils-sig@python.org> > To unsubscribe send an email to distutils-sig-leave(a)python.org <mailto:distutils-sig-leave@python.org> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/> > Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…>

2 years, 12 months

Opinions on requiring younger glibc in manylinux1 wheel?

by Antoine Pitrou

Hi, According to recent messages, it seems manylinux2010 won't be ready soon. However, the baseline software in manylinux1 is becoming very old. As an example, a popular C++ library (Abseil - https://abseil.io/) requires a more recent glibc (see https://github.com/abseil/abseil-cpp/commit/add89fd0e4bfd7d874bb55b67f4e13b…). What do you think of publishing manylinux1 wheels that would require a more recent glibc? This is being discussed currently for the pyarrow package. Regards Antoine.

2 years, 12 months

Adopting virtualenv package maintenance

by Sorin Sbarnea

As it seems that virtualenv package is in need of some maintenance effort, focused mostly on doing reviews, closing or merging them and eventually doing a new release once a month. I know that virtualenv is in deprecation mode as its would be no longer needed when Python2 will no longer be used. The reality is that Python 2.x will still be in production after January 1st, 2020 because there are deployed products with LTS contracts which will need some time to get updated to newer versions that use py3. This automatically translates to the need to have a working virtualenv for testing them. I am part of the OpenStack team and I am sure that, even if I like it or not, I would have to deal with some amount of py2 even after the magic date. The current situation with virtualenv is pretty bad because there are lots of open pull-requests which are not reviewed or merged, mostly because there is nobody available to do that boring extra work. I had few changes that were improving the CI testing of virtualenv which soon will be one year old,... most of them without any feedback. Even finding whom to ping by email or irc was a challenge as I got two responses: no response at all or someone else telling me that they are not maintainers of the virtualenv package. Example https://groups.google.com/forum/#!topic/pypa-dev/YMVsRbNoVpg For these reasons I would like to become a maintainer for virtualenv, preferably working with two others on keeping it alive for a couple of years till we could organize a big wake ceremony for it. It would be preferable if two others would join the maintenance "taskforce" because merging a change should almost always involve at least two reviewers. While I cannot make any guarantees regarding dealing with all reported bugs, I can commit on assuring that there are no PRs that are not reviewed for longer than 30 days (aiming for one week). Now there are ~75 open PRs. I have being doing open source for a long time and I respect all the time and efforth put by project maintainers and at the same time I always tried to do my best dealign with incoming PRs because if someone spended his time trying to make a contribution that is passing CI, they probably deserve at least a review. https://github.com/pypa/virtualenv/pulls Thanks Sorin Sbarnea @ssbarnea on irc/github/...

2 years, 12 months

SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register

by Wes Turner

Should C extensions that compile all add `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the risk of Spectre variant 2 (which does indeed affect user space applications as well as kernels)? [1] https://github.com/speed47/spectre-meltdown-checker/issues/119#issuecomment… [2] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) [3] https://en.wikipedia.org/wiki/Speculative_Store_Bypass#Speculative_executio…

3 years

pip installing scripts into another virtualenv

by Alex Becker

As part of a package management tool <https://github.com/alexbecker/dotlock>, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess), into a different virtualenv than the virtualenv my python process is running in (if any). I have this *mostly* working with: pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under. Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)? Sincerely, Alex Becker

3 years

PEP 518 and the pyproject.toml format

by Melvyn Sopacua

Hello, First post, so short introduction: I'm mostly working on Django projects, currently for 3yourminD in Berlin. Working with python / Django for 6 going on 7 years. Also experienced in PHP, some Js and build systems like docker. We're currently migrating everything to docker and pip / requirements.txt isn't cutting it anymore and I've tasked myself with looking for or building an alternative tool. This weekend I did quite a bit of research into the packaging tools and came accross PEP 518. What it doesn't address is that there now is a file resvered that declares very little. It does not declare and define sections to serve as a deployment file for a project and apparently it is unclear to tool builders what the one section is does define is supposed to be used for. Case in point: poetry doesn't add itself to the build-system table when writing the py-project.toml file. Which, as I read it, it should. Ideally, every tool.${name} should be in the build system? So, can we improve the file specification and introduce sections that allows a project (not a library) to specify how it should be built? Right now, this is all off-loaded to the tools that use the file and one cannot rely on anything. It also makes migration from one tool to another difficult and cooperation between tools hard. Or is this intentional: is this supposed to be tied to specific tools as build system are generally tightly coupled? And should we not change that? It would help if the PEP had some explanation of the choices made to reserve a file that partially sets out to replace (all?) other files, but does not do anything to accomplish that. I realize that PEP-517 defines the API and leaves the implementation up to the tool, but that doesn't necessarily mean the py- project file should not name / reserve tables that the PEP 517 API should use. I'm considering writing a tool that focuses solely on the deployment and dependency maintenance of a project and would like to not "vendor lock-in" my users, with even those dependency lists, because I have to use the tool namespace and can fill it every way I see fit and so will others. So ideally, we would reserve dependency tables: [dependencies.$name] Reserved names: standard, dev, test Tools can add more names which they can then interpret themselves: [dependencies.stage] django-anonymizer = "^0.5" And test tables and probably one or two more that come to mind. Is there any interest from the team to improve the file format and lock down such tables? I'd love to do the initial proposal if that helps move things forward. -- Melvyn Sopacua

3 years

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG September 2018