Mailman 3 - Distutils-SIG

Announcing Bandersnatch 3.0(.1) - now asyncio based!
by Cooper Ry Lees 30 Oct '18

30 Oct '18

Hi All, Bandersnatch is the static PyPI mirroring software following PEP381. It allows you to have a local full static copy of PyPI (no dynamic APIs). We've finally cut 3.0 (with a .1 to change URL to GitHub on PyPI) which has fixed a few bugs and moved to asyncio thread with requests. My original plan was to depreciate XMLRPC and move to full asyncio code base, on top of aiohttp, but we didn't settle on a new API on PyPI to replace XMLRPC. https://pypi.org/project/bandersnatch/ <https://pypi.org/project/bandersnatch/> https://github.com/pypa/bandersnatch/ <https://github.com/pypa/bandersnatch/> Please upgrade and test and let me know of any bugs and we'll roll out fixes. Noted Changes from latest 2.2.1: - Move to asyncio executors around request calls Fixes #81 (on BitBucket) - Use platform.uname() to support Windows Fixes #19 (Windows still has a few more bugs until full support) - Add bandersnatch verify subcommand to re-download + delete unneeded packages Fixes #8 + many follow on Issues during testing - Thanks - electricworry & tau3 for testing + fixes! - Introduce much more Lint checks (black, isort, mypy) other than flake8 - Thanks @asottile - Make tox run lint checks + print out test coverage - Add whitelist + blacklist plugins - Thanks @dwighthubbard - Add generated documentation - Thanks @dwighthubbard - Move to requiring Python >= 3.6.1 Fixes #66 Cheers all who reported bugs and PRs - Keep them coming! Cooper

1 0

Bloomberg packaging sprint info
by Chris Jerdonek 27 Oct '18

27 Oct '18

Hi, Regarding the Bloomberg packaging sprint on Oct. 27-28-- https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… Are there ways that people can participate who won't be there in person (e.g. in addition to the tracker)? I won't be there but can probably dedicate some time to write and review pip patches. Also, what PyPA members have signed up to participate? Thanks, --Chris

2 2

(no subject)
by Matthew Woodin 26 Oct '18

26 Oct '18

1 0

Announcing Wheelodex
by John Thorvald Wodder II 19 Oct '18

19 Oct '18

I've created a website that is likely to be of interest to people here: Wheelodex <https://www.wheelodex.org>, a site for browsing the metadata of wheels on PyPI. It allows you to find out what projects a wheel depends on, what other projects depend on a given project, what commands & other entry points a wheel defines, what files are in a wheel, etc. You can even search for wheels containing a given module or file, or browse a list of all commands & other entry points defined by wheels. There's also a basic API for getting wheel data as JSON: <https://www.wheelodex.org/json-api/>. I'm open to suggestions on what else to do with the data. I'm also open to suggestions on how to make the interface look less sucky. -- John Wodder

6 5

Re: [Distutils] [Python-ideas] Pypi private repo's
by Alex Walters 17 Oct '18

17 Oct '18

I am fairly sure if you give the PyPA that suggestion, they will just deflate at the thought of the workload. Besides, we already offer private repos for free, several ways ranging from devpi to python -m SimpleHTTPServer in a specially created directory. From: Python-ideas <python-ideas-bounces+tritium-list=sdamon.com(a)python.org> On Behalf Of Nick Humrich Sent: Wednesday, April 4, 2018 12:26 PM To: python-ideas(a)python.org Subject: [Python-ideas] Pypi private repo's I am sure this has been discussed before, and this might not even be the best place for this discussion, but I just wanted to make sure this has been thought about. What if pypi.org <http://pypi.org> supported private repos at a cost, similar to npm? This would be able to help support the cost of pypi, and hopefully make it better/more reliable, thus in turn improving the python community. If this discussion should happen somewhere else, let me know. Nick

5 5

manylinux1 guidelines for zlib?
by Antoine Pitrou 15 Oct '18

15 Oct '18

Hello, Surprisingly, the manylinux1 spec doesn't seem to include the zlib in the list of known-to-be-available libraries (are there GNU/Linux systems out there without a zlib installed?). Since I'm assuming several packages already had a need for that, is there a recommended way to link in the zlib as part of a manylinux1 wheel? Would you recommend static linking with a private version, or dynamic linking? Regards Antoine.

4 6

History of python packaging
by Константин 15 Oct '18

15 Oct '18

Hello! My name is Konstantin and I am going to participate in the PyPA event [1] next week. To understand the packaging landscape better, I've decided to create an article for myself generally, but I think it could be interesting for others as well because I've found that https://www.pypa.io/en/latest/history/ is not enough for me to understand how packaging works in general. If anyone has an interest, I would really appreciate any comments on it, either technical or just stylistic. https://medium.com/@knikitin/the-brief-history-of-package-distribution-in-p… , before I will publish it for a wider audience. Thank you! [1] https://generalassemb.ly/education/bloomberg-open-source-weekend-pypa/londo…

3 3

setuptools API compatibility
by brian＠sweetapp.com 11 Oct '18

11 Oct '18

Hi, Does setuptools make any API stability guarantees i.e. should all setup.py files that work with setuptools version X also work with setuptools version Y where Y > X? I'm asking because the App Engine Python 3.7 runtime keeps it's version of setuptools up-to-date and I'm wondering if that is problematic due to API compatibility i.e. that package installation might fail due to an incompatibility introduced in setuptools. Cheers, Brian

5 5

[poposal for preparing a pep] sharing package installations between virtualenvs
by RonnyPfannschmidt 08 Oct '18

08 Oct '18

hi everyone, i just did a little spring cleaning, and while i cleaned up my ~/Projects folder - the vast amount of that was virtualenvs with roughly the same packages installed - each about 500 mb for like 2-5 mb of code this seems excessively wasteful - my impression is that ~98% of the files seem to be duplicate. since the impending eol of python2 and the multi python version capabilities of python3 compiled code caches, i'd like to create a proposal for a shared folder structure $LIB/python/shared-packages/ should be a root folder in which unpacked wheels would reside at a initial iteration this solution would suffer from the expanding python path issue since each package would have a toplevel sys.path entry. on order to later on speed this up a simple api can be provided that makes python aware of the metatadata locations and the map toplevel/namespace names directly to import locations for example like shared-packages.pth import sys;sys.path.insert(0, "$LIB/python/shared-packages/shared_packages_support-1.0-py3.whl/"); import shared_packages_support;sys.path.pop(0);shared_packages_support.enable() shared_packages.json { "pytest": "pytest-3.8.8-py2.py3.whl"; } in the aftermath, when shared packages support and fast import/metadata for shared packages where implemented, the sizes of most virtualenvs would shrink significantly -- Ronny

5 6

Send
by youtube.com/ [whatItSaysHere] 06 Oct '18

06 Oct '18

1 0

world-wirde
by youtube studio 05 Oct '18

05 Oct '18

Good

1 0

pip 18.1 has been released!
by Pradyun Gedam 05 Oct '18

05 Oct '18

On behalf of the PyPA, I am pleased to announce that pip 18.1 has just been released. To install pip 18.1, you can run:: python -m pip install --upgrade pip or use get-pip as described in https://pip.pypa.io/en/latest/installing. Note that if you are using a version of pip supplied by your distribution vendor, vendor-supplied upgrades will be available in due course. The highlights of this release are: - Python 3.7 is now supported - Dependency Links support is now scheduled for removal in pip 19.0 (the next pip release, scheduled in January 2019). - Plaform specific options can now be used with the --target option, to enable certain workflows. - Much more helpful error messages on invalid configuration files - Many bug fixes and minor improvements Thanks to everyone who put so much effort into the new release. Many of the contributions came from community members, whether in the form of code, participation in design discussions and/or bug reports. The pip development team is extremely grateful to everyone in the community for their contributions. Thanks, Pradyun

1 0

Re: disable building wheel for a package
by Jeroen Demeyer 04 Oct '18

04 Oct '18

On 2018-09-14 12:55, Alex Grönholm wrote: > I'm curious: what data does it attempt to install and where? Have you > created a ticket for this somewhere? The OP mentioned absolute paths. However, it really sounds like a bad idea to hard-code an absolute installation path. Let's consider it a feature that wheel doesn't support that. See https://github.com/pypa/wheel/issues/92

16 76

ANN: distlib 0.2.8 released on PyPI
by Vinay Sajip 02 Oct '18

02 Oct '18

I've recently released version 0.2.8 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #107: Updated documentation on testing to include information on setting PYTHONHASHSEED. * Fixed #108: Updated metadata scan to look for the METADATA file as well as the JSON formats. * Fixed #109: Removed existing files (which might have been symlinks) before overwriting. * Fixed #111: Avoided unnecessary newlines in script preambles, which caused problems with detecting encoding declarations. Thanks to Wim Glenn for the report and patch. * Fixed #112: Handled wheel tags and platform-dependent downloads correctly in SimpleScrapingLocator. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements,please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.org/project/distlib/0.2.8/[2] https://goo.gl/tVzKUc[3] https://bitbucket.org/pypa/distlib/issues/new

1 0

breakages in travis-ci and ci.appveyor
by Robin Becker 02 Oct '18

02 Oct '18

I am using Matthew Brett's manylinux builder setup (https://github.com/matthew-brett/multibuild) for reportlab and just started to see this > File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/wheeltools.py", line 14, in <module> > from wheel.util import urlsafe_b64encode, open_for_csv, native # type: ignore > ImportError: cannot import name 'open_for_csv' at the same time my appveyor builds have started to fail with this > c:\Python%PYTHON%\scripts\pip.exe --version > pip 18.0 from c:\python34\lib\site-packages\pip (python 3.4) > c:\Python%PYTHON%\scripts\pip.exe install wheel > Collecting wheel > Using cached https://files.pythonhosted.org/packages/b3/bb/42354ce8c08f66ae0cd0f4a841f40… > Installing collected packages: wheel > Successfully installed wheel-0.32.0 > c:\Python%PYTHON%\python.exe -W ignore setup.py bdist_wheel > ================================================ > Attempting build of _rl_accel & pyHnj ................ > File "c:\Python34\lib\site-packages\wheel\bdist_wheel.py", line 237, in run > self.egg2dist(self.egginfo_dir, distinfo_dir) > File "c:\Python34\lib\site-packages\wheel\bdist_wheel.py", line 344, in egg2dist > pkg_info = pkginfo_to_metadata(egginfo_path, pkginfo_path) > File "c:\Python34\lib\site-packages\wheel\metadata.py", line 88, in pkginfo_to_metadata > for key, value in generate_requirements({extra: reqs}): > File "c:\Python34\lib\site-packages\wheel\metadata.py", line 67, in generate_requirements > for new_req in convert_requirements(depends): > File "c:\Python34\lib\site-packages\wheel\metadata.py", line 36, in convert_requirements > spec = requires_to_requires_dist(parsed_requirement) > File "c:\Python34\lib\site-packages\wheel\metadata.py", line 21, in requires_to_requires_dist > if requirement.url: > AttributeError: 'Requirement' object has no attribute 'url' > Command exited with code 1 I assume these are related to wheel changes just announced. Certainly the multibuild docker was updated to use 0.32.0 I'm not sure if I have to do something or how I can fix this. -- Robin Becker

3 4

Notes from python core sprint on workflow tooling
by Nathaniel Smith 01 Oct '18

01 Oct '18

Now that the basic wheels/pip/PyPI infrastructure is mostly functional, there's been a lot of interest in improving higher-level project workflow. We have a lot of powerful tools for this – virtualenv, pyenv, conda, tox, pipenv, poetry, ... – and more in development, like PEP 582 [1], which adds a support for project-local packages directories (`__pypackages__/`) directly to the interpreter. But to me it feels like right now, Python workflow tools are like the blind men and the elephant [2]. Each group sees one part of the problem, and so we end up with one set of people building legs, another a trunk, a third some ears... and there's no overall plan for how they can fit together. For example, PEP 582 is trying to solve the problem that virtualenv is really hard to use for beginners just starting out [3]. This is a serious problem! But I don't want a solution that *only* works for beginners starting out, so that once they get a little more sophisticated they have to throw it out and learn something new from scratch. So I think now might be a time for a bit of top-down design. **I want a picture of the elephant.** If we had that, maybe we could see how all these different ideas could be put together into a coherent whole. So at the Python core sprint a few weeks ago, I dragged some interested parties [4] into a room with a whiteboard [5], and we made a start at it. And now I'm writing it up to share with you all. This is very much a draft, intended as a seed for discussion, not a conclusion. [1] https://www.python.org/dev/peps/pep-0582/ [2] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant [3] https://www.python.org/dev/peps/pep-0582/#motivation [4] I won't try to list names, because I know I'll forget someone, and I don't know if everyone would agree with everything I wrote there. But thank you all! [5] https://photos.app.goo.gl/4HfY8P3ESPNi9oLMA, including special guest appearance by Kushal's elbow # The idealized lifecycle of a Python project ## 1. Beginner Everyone starts out as a rank beginner. This may be the first time they have programmed at all. At this stage, users want to: - install *one* thing to get started (e.g. python itself) - write and run simple scripts (standalone .py files) - run a REPL - install and use PyPI packages like requests or numpy - install and use tools like jupyter - their IDE should also be able to find these packages/tools Over time, they'll probably end up with multiple scripts, and maybe want to organize them into subdirectories. The above should all work from subdirectories. ## 2. Sharing with others Now we have a neat little script. Or maybe we've made a pretty jupyter notebook that computes some crucial business analytics. We want to share it with our friends or coworkers. We still need the features above; and now we also care about: - version control - some way for our friend to reconstruct, on their computer: - the same PyPI packages that we were using - the same tools that we were using - the ways we invoked those tools This last point is important: as projects grow in complexity, and are used by a wider audience, they often end up with fairly complex tool specifications that have to be shared among a team. For example: - to run tests: in an environment that has pytest, pytest-cov, and pytest-trio installed, and with our project working directory on PYTHONPATH, run `pytest -Werror --cov ...` - to format code: in an environment using python 3.6 or later, that has black installed, run `black -l 79 *.py my-util-directory/*.py` This kind of tool specification also puts us in a good position to set up CI when we reach that point. At this point our project can grow in a few different directions. ## 3a. Deployable webapp This adds the requirement to "deploy". I think this is mostly covered by the set-up-an-environment-to-run-a-command functionality already described? I'm not super familiar with this, but it's pipenv's core target, and pipenv doesn't have much more than that, so I assume that's about right... ## 3b. Reusable library For this we also need to: - Build sdists and wheels - Which means: pyproject.toml, and some way to invoke it - Install our library into our environments - Including dependency locking (best practice is to not pin dependencies in wheel metadata, but to pin all dependencies in CI; so there needs to be some way to track those separately, but integrated enough that it's not a huge ceremony to add or change a dependency) ## 3c. Reusable standalone app I think this is pretty much like the "Reusable library", except that it'd be nice to have better tools to build/distribute standalone applications. But if we had them, we could invoke them the same way as we invoke other build systems? # How do existing tools/proposals fit into this picture? pyenv, virtualenv, and conda all solve parts of the "create an environment" problem, but consider the other aspects out-of-scope. tox solves the problem of keeping a shared record of how to run a bunch of different tools in the appropriate environments, but doesn't handle pinning or procuring appropriate python versions, and requires a separate bootstrapping step to install tox. `__pypackages__` (if implemented) makes it very easy for beginners to use PyPI packages in their own scripts and from the REPL; in particular, it would be part of python, so it meets the "install *one* thing" criterion. But, it doesn't provide any way to run tools. (There's no way to put `__pypackages__/bin` on PATH.) It doesn't allow scripts to be organized into subdirectories. (For security reasons, we can't have the python interpreter going off walking the filesystem looking for `__pypackages__/`, so the PEP specifies that `__pypackages__/` has to be in the same directory as the script that uses it.) There's no way to share your `__pypackages__` environment with a friend. So... it seems like a something that people would outgrow very quickly. pipenv and poetry are interesting. Their basic strategy is to say, there is a top-level command that acts as your entry point to performing workflow actions on on a python project (`pipenv` or `poetry`, respectively). And this strategy at least in principle can solve the problems that `__pypackages__/` runs into. In particular, it doesn't rely on `$PATH`, so it can run tools; and because it's a dedicated project management tool, it can go looking for the project marker file. # A fantastic elephant So if our idealized user had an idealized tool, what would that look like? They'll be interacting with Python through a dedicated tool, similar to pipenv or poetry. In my little fantasy here I'll call it `pyp`, because (a) I want to be neutral, (b) 6 characters is too long. To get this tool, either they install Python (via python.org download, apt, homebrew, whatever), and the tool is automatically included. Or else, they install the tool directly, and it has the ability to install Python interpreters when needed. Once they have the tool, they start by making a new directory for their project (this way they're ready to switch to version control later). Then they somehow mark this directory as being a "python project root". I guess the UI would be something like `pyp new <name>` and it just does it for you, but we have to figure out what this creates on disk. We need some sort of marker file. Files that currently serve this kind of role include tox.ini, Pipfile, pyproject.toml, __pypackages__, ... But only one of these is a standard thing we're already committed to sticking with, so, pyproject.toml it is. Let's make it the marker for any python project, not just redistributable libraries. (And if we do grow up into a redistributable library, then we're already prepared.) In the initial default configuration, there's a single default environment. You can install things with `pyp install ...` or `pyp uninstall ...`, and it tracks the requested packages in some standardized way in pyproject.toml, and also pins specific versions somewhere (could be pyproject.toml again I guess, or poetry's pyproject.lock would work too). This way when we decide to share our project later, our friends can recreate our environment on their system. However, there's also the capability to configure multiple custom execution environments, including python version and installed packages. And the capability to configure new aliases like `pyp test` or `pyp reformat`, which run some specified command in a specified environment. Since the install/locking metadata is all standardized, you can even switch between competing tools, and integrate with third-party tools like pyup.io. For redistributable libraries, we also need some way to get the wheel metadata and the workflow metadata to play nicely together. Maybe this means that we need a standardized install-requires field in pyproject.toml, so that build backends and workflow tools have a shared source of truth? # What's wrong with pipenv? Since pipenv is the tool that those of us in the room were most familiar with, that comes closest to matching this vision, we brainstormed a list of complaints about it. Some of these are more reasonable than others. - Not ambitious enough. This is a fuzzy sort of thing, but perception matters, and it's right there in the name: it's a tool to use pip, to manage an environment. If we're reconceiving this as the grand unified entryway to all of Python, then the name starts to feel pretty weird. The whole thing where it's only intended to work for webapp-style projects would have to change. - Uses Pipfile as a project marker instead of pyproject.toml. - Not shipped with Python. (Obviously not pipenv's fault, but nonetheless.) - Environments should be stored in project directory, not off in $HOME somewhere. (Not sure what this is about, but some of the folks present were quite insistent.) - Environments should be relocatable. - Hardcoded to only support "default" and "dev" environments, which is insufficient. - No mechanism for sharing prespecified commands like "run tests" or "reformat". - Can't install Python. (There's... really no reason we *couldn't* distribute pre-built Python interpreters on PyPI? between the python.org installers and the manylinux image, we're already building redistributable run-anywhere binaries for the most popular platforms on every Python release; we just aren't zipping them up and putting them on PyPI.) -n -- Nathaniel J. Smith -- https://vorpus.org

7 19

Wheel 0.32 released
by Alex Grönholm 29 Sep '18

29 Sep '18

This release contains the following significant changes: * Wheel signature generation and verification commands have been removed * The wheel installation features have been removed * The documentation has been rewritten to conform to the PyPA guidelines * The ability to add multiple license files to the wheel has been added (the most common ones are included by default) There were also a few bug fixes as well. If you are left missing the removed features, let me know on the issue tracker.

1 0

setuptools configuration in pyproject.toml
by Bernat Gabor 28 Sep '18

28 Sep '18

I'm aware this might be a controversial subject, so let's have the initial discussion about it here first for full disclosure and see what people think about it. Should setuptools support pyproject.toml as configuration source or not (alternative to setup.cfg which it already does - https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-set… )? The main benefit of having this would be to decrease configuration files and have build dependencies and other types of dependencies in one location. Furthermore many other packaging projects (flit, poetry) already do define their dependencies inside pyproject.toml; so would create one unified location where to look for such in the future. The counter-argument is that "a big part of pyproject.toml was keeping that file clean" and would furthermore increase the size of that file down the line. So what do people think? Should we encourage or discourage to have a single python project file? I'm personally supporting build/code quality tools supporting pyproject.toml as their main configuration file. Thanks

12 17

Stepping away from Twine maintainership
by Sumana Harihareswara 25 Sep '18

25 Sep '18

Quick note to thank Ian Stapleton Cordasco and Thea Flowers for their work maintaining Twine! I realized I don't have time to help maintain it right now so I'm stepping away from that, and am grateful for their work, including new releases this month: https://pypi.org/project/twine/#history And thanks to Dustin Ingram for all his recent work on Twine as well. As he said https://twitter.com/di_codes/status/1044358639081975813 : > New twine subcommand: $ twine check dist/* > Use it to verify that the README for your package is valid and will be rendered correctly on PyPI. > Between that and Markdown support, there's no excuse for mis-rendered PyPI descriptions anymore! More details: https://packaging.python.org/guides/making-a-pypi-friendly-readme/#validati… -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

New virtualenv maintainer
by Donald Stufft 20 Sep '18

20 Sep '18

We don’t typically bother to make big announcements on distutils-sig for maintainers of projects, however I’m making an exception in this case because of the recent threads on virtualenv maintenance. The current core team of virtualenv agrees that it is an important package, and deserves additional maintenance, but have somewhat stepped back from it in recent times for a variety of reasons (many ephemeral in nature). We’ve been happy to have a number of people step forward to offer help in maintaining the project, and after some thought and discussion, we’ve decided to extend an invitation of a commit bit to Bernat Gabor, and I’m happy to say that he has graciously accepted, and is now part of the team. For those unaware, Bernat is currently a developer for the tox project, which is one of the major consumers of virtualenv which made him a great fit for maintainership since it gives another real world perspective (consuming virtualenv as an API, instead of as a tool) that the current team currently lacked. So Welcome Bernat!

5 4

Re: Distlib vs Packaging (Was: disable building wheel for a package)
by Tzu-ping Chung 20 Sep '18

20 Sep '18

Pipenv’s resolver does not use troove classifiers as far as I am aware. I could be missing something though, the current resolver implementation is not the best code base to reason with. Can you provide some ideas on this, Dan? TP > On 20/9, 2018, at 15:09, Bernat Gabor <gaborjbernat(a)gmail.com> wrote: > > One of such differences is pipenv apparently trying to work from the troove classifiers for version compatibility instead of the existing python_requires (see https://github.com/tox-dev/tox/pull/1005 <https://github.com/tox-dev/tox/pull/1005>). Why? Point in case how people start to think pipenv is pip. At this point, pip is a completely different beast than pip I think. > > Such check at best would fall under the ``twine check`` command at PyPi upload time, not at install time. > > On Thu, Sep 20, 2018 at 8:03 AM Tzu-ping Chung <uranusjr(a)gmail.com <mailto:uranusjr@gmail.com>> wrote: > > >> On 20/9, 2018, at 13:22, Chris Jerdonek <chris.jerdonek(a)gmail.com <mailto:chris.jerdonek@gmail.com>> wrote: >> >> On Wed, Sep 19, 2018 at 8:54 PM, Dan Ryan <dan(a)danryan.co <mailto:dan@danryan.co>> wrote: >> I should clarify that we have already implemented a number of these as >> libraries over the last several months (and I am super familiar with pip's >> internals by now and I'm sure TP is getting there as well). More on this >> below >> ... >> We are super cognizant of that aspect as I am pretty sure we are hitting >> this wall in a full (nearly) pip-free reimplementation of all of the pipenv >> internals from the ground up, including wheel building/installation, but we >> basically had to start by calling pip directly, then slowly reimplement each >> aspect of the underlying logic using various elements in distlib/setuptools >> or rebuilding those. >> >> Is the hope or game plan then for pipenv not to have to depend on pip? This is partly what I was trying to learn in my email to this list a month ago (on Aug. 20, with subject: "pipenv and pip"): >> https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2… <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2…> >> >> Based on the replies, I wasn't getting that impression at the time (though I don't remember getting a clear answer), but maybe things have changed since then. > > The resolution side of Pipenv really needs a Python API, and also cannot really > use the CLI because it needs something slightly different than pip’s high-level > logic (Nick mentioned this briefly). If we can’t use pip internals, then yes, the plan > is to not depend on pip. The hope is we can share those internals with pip (either > following the same standards, or using the same implementation), hence my series > of questions. > > The installation side of Pipenv will continue to use pip directly, at least for a while > more even after the resolution side breaks away, since “pip install” is adequate > enough for our purposes. There are some possible improvements if there is a > lower-layer library (e.g. to avoid pip startup overhead), but that is far less important. > > >> >> It should certainly be a lot easier for pipenv to move fast since there is no legacy base of users to maintain compatibility with. However, I worry about the fracturing this will cause. In creating these libraries, from the pip tracker it doesn't look like any effort is going into refactoring pip to make use of them. This relates to the point I made earlier today about how there won't be an easy way to cut pip over to using a new library unless an effort is made from the beginning. Thus, it's looking like things could be on track to split the user and maintainer base in two, with pip bearing the legacy burden and perhaps not seeing the improvements. Are we okay with that future? > > I’m afraid the new implementation will still need to deal with compatibility issues. > Users expect Pipenv to work exactly as pip, and get very angry if it does not, > especially when they see it is under the PyPA organisation on GitHub. The last > time Pipenv tries to explain it does whatever arbitrary things it does, we get > labelled as “toxic” (there are other issues in play, but this is IMO the ultimate > cause). Whether the image is fair or not, I would most definitely want to avoid > similar incidents from happening again. > > I think Pipenv would be okay to maintain a different (from scratch) implementation > than pip’s, but it would still need to do (almost) exactly what pip is doing, unless > we can have people (pip maintainers or otherwise) backing the differences. Whether > pip uses the new implementation or not wouldn’t change the requirement :( > > > TP > > >> >> --Chris >> >> >> Since you mentioned following along, here's what we're working on right now: >> >> https://github.com/sarugaku/requirementslib <https://github.com/sarugaku/requirementslib> -- abstraction layer for parsing >> and converting various requirements formats >> (pipfile/requirements.txt/command line/InstallRequirement) and moving >> between all of them >> https://github.com/sarugaku/resolvelib <https://github.com/sarugaku/resolvelib> -- directed acyclic graph library for >> handling dependency resolution (not yet being used in pipenv) >> https://github.com/sarugaku/passa <https://github.com/sarugaku/passa> -- dependency resolver/installer/pipfile >> manager (bulk of the logic we have been talking about is in here right now) >> -- I think we will probably split this back out into multiple other smaller >> libraries or something based on the discussion >> https://github.com/sarugaku/plette <https://github.com/sarugaku/plette> -- this is a rewrite of pipfile with some >> additional logic / validation >> https://github.com/sarugaku/shellingham <https://github.com/sarugaku/shellingham> -- this is a shell detection library >> made up of some tooling we built in pipenv for environment detection >> https://github.com/sarugaku/pythonfinder <https://github.com/sarugaku/pythonfinder> -- this is a library for finding >> python (pep 514 compliant) by version and for finding any other executables >> (cross platform) >> https://github.com/sarugaku/virtenv <https://github.com/sarugaku/virtenv> -- python api for virtualenv creation >> >> Happy to provide access or take advice as needed on any of those. Thanks >> all for the receptiveness and collaboration >> >> Dan Ryan >> gh: @techalchemy // e: dan(a)danryan.co <mailto:dan@danryan.co> >> >> From: Donald Stufft [mailto:donald@stufft.io <mailto:donald@stufft.io>] >> Sent: Wednesday, September 19, 2018 1:52 PM >> To: Tzu-ping Chung >> Cc: Distutils >> Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel >> for a package) >> >> My general recommendation if you want a Python implementation/interface for >> something pip does, is: >> >> - Open an issue on the pip repository to document your intent and to make >> sure that there is nobody there who is against having that functionality >> split out. This might also give a chance for people with familiarity in that >> API to mention pain points that you can solve in a new API. We can also >> probably give you a good sense if the thing you want in a library is >> something that probably has multiple things that are dependent on getting >> split out first (for instance, if you said you wanted a library for >> installing wheels, we'd probably tell you that there is a dependency on PEP >> 425 tags, pip locations, maybe other that need resolved first) and also >> whether this is something that should have a PEP first or not. Getting some >> rough agreement on the plan to split X thing out before you start is overall >> a good thing. >> >> - Create or update a PEP if required, and get it into the provisional state. >> >> - Make the library, either as a PR to packaging or as it's own independent >> library. If there are questions that come up while creating that library/PR >> that have to do with specific pip behaviors, go back to that original issue >> and ask for clarification etc. Ideally at some point you'll open a PR on pip >> that uses the new library (my suggestion is to not bundle the library in the >> initial PR, and just import it normally so that the PR diff doesn't include >> the full bundled library until there's agreement on it). If there's another >> tool (pipenv, whatever) that is looking to use that same functionality, open >> a WIP PR there too that switches it to using that. Use feedback and what you >> learn from trying to integrate in those libraries to influence back the >> design of the API itself. >> >> Creating a PEP and creating the library and the PRs can happen in parallel, >> but at least for pip if something deserves a PEP, we're not going to merge a >> PR until that PEP is generally agreed on. However it can be supremely useful >> to have them all going at the same time, because you run into things that >> you didn't really notice until you went to actually implement it. >> >> My other big suggestion would be to e careful about how much you bite off at >> one time. Pip's internal code base is not the greatest, so pulling out >> smaller chunks at a time rather than trying to start right off pulling out a >> big topic is more likely to meet with success. Be cognizant of what the >> dependencies are for the feature you want to implement, because if it has >> dependencies, you'll need to pull them out first before you can pull it out >> OR you'll need to design the API to invert those dependencies so they get >> passed in instead. >> >> I personally would be happy to at a minimum participate on any issue where >> someone was trying to split out some functionality from pip into a re-usable >> library if not follow the develop of that library directly to help guide it >> more closely. My hope for pip is that it ends up being the glue around a >> bunch of these libraries, and that it doesn't implement most of the stuff >> itself anymore. >> -- >> Distutils-SIG mailing list -- distutils-sig(a)python.org <mailto:distutils-sig@python.org> >> To unsubscribe send an email to distutils-sig-leave(a)python.org <mailto:distutils-sig-leave@python.org> >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/> >> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…> > -- > Distutils-SIG mailing list -- distutils-sig(a)python.org <mailto:distutils-sig@python.org> > To unsubscribe send an email to distutils-sig-leave(a)python.org <mailto:distutils-sig-leave@python.org> > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/> > Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… <https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…>

1 0

Opinions on requiring younger glibc in manylinux1 wheel?
by Antoine Pitrou 19 Sep '18

19 Sep '18

Hi, According to recent messages, it seems manylinux2010 won't be ready soon. However, the baseline software in manylinux1 is becoming very old. As an example, a popular C++ library (Abseil - https://abseil.io/) requires a more recent glibc (see https://github.com/abseil/abseil-cpp/commit/add89fd0e4bfd7d874bb55b67f4e13b…). What do you think of publishing manylinux1 wheels that would require a more recent glibc? This is being discussed currently for the pyarrow package. Regards Antoine.

9 17

Adopting virtualenv package maintenance
by Sorin Sbarnea 19 Sep '18

19 Sep '18

As it seems that virtualenv package is in need of some maintenance effort, focused mostly on doing reviews, closing or merging them and eventually doing a new release once a month. I know that virtualenv is in deprecation mode as its would be no longer needed when Python2 will no longer be used. The reality is that Python 2.x will still be in production after January 1st, 2020 because there are deployed products with LTS contracts which will need some time to get updated to newer versions that use py3. This automatically translates to the need to have a working virtualenv for testing them. I am part of the OpenStack team and I am sure that, even if I like it or not, I would have to deal with some amount of py2 even after the magic date. The current situation with virtualenv is pretty bad because there are lots of open pull-requests which are not reviewed or merged, mostly because there is nobody available to do that boring extra work. I had few changes that were improving the CI testing of virtualenv which soon will be one year old,... most of them without any feedback. Even finding whom to ping by email or irc was a challenge as I got two responses: no response at all or someone else telling me that they are not maintainers of the virtualenv package. Example https://groups.google.com/forum/#!topic/pypa-dev/YMVsRbNoVpg For these reasons I would like to become a maintainer for virtualenv, preferably working with two others on keeping it alive for a couple of years till we could organize a big wake ceremony for it. It would be preferable if two others would join the maintenance "taskforce" because merging a change should almost always involve at least two reviewers. While I cannot make any guarantees regarding dealing with all reported bugs, I can commit on assuring that there are no PRs that are not reviewed for longer than 30 days (aiming for one week). Now there are ~75 open PRs. I have being doing open source for a long time and I respect all the time and efforth put by project maintainers and at the same time I always tried to do my best dealign with incoming PRs because if someone spended his time trying to make a contribution that is passing CI, they probably deserve at least a review. https://github.com/pypa/virtualenv/pulls Thanks Sorin Sbarnea @ssbarnea on irc/github/...

11 21

SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register
by Wes Turner 17 Sep '18

17 Sep '18

Should C extensions that compile all add `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the risk of Spectre variant 2 (which does indeed affect user space applications as well as kernels)? [1] https://github.com/speed47/spectre-meltdown-checker/issues/119#issuecomment… [2] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) [3] https://en.wikipedia.org/wiki/Speculative_Store_Bypass#Speculative_executio…

3 6

pip installing scripts into another virtualenv
by Alex Becker 16 Sep '18

16 Sep '18

As part of a package management tool <https://github.com/alexbecker/dotlock>, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess), into a different virtualenv than the virtualenv my python process is running in (if any). I have this *mostly* working with: pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under. Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)? Sincerely, Alex Becker

6 7

PEP 518 and the pyproject.toml format
by Melvyn Sopacua 16 Sep '18

16 Sep '18

Hello, First post, so short introduction: I'm mostly working on Django projects, currently for 3yourminD in Berlin. Working with python / Django for 6 going on 7 years. Also experienced in PHP, some Js and build systems like docker. We're currently migrating everything to docker and pip / requirements.txt isn't cutting it anymore and I've tasked myself with looking for or building an alternative tool. This weekend I did quite a bit of research into the packaging tools and came accross PEP 518. What it doesn't address is that there now is a file resvered that declares very little. It does not declare and define sections to serve as a deployment file for a project and apparently it is unclear to tool builders what the one section is does define is supposed to be used for. Case in point: poetry doesn't add itself to the build-system table when writing the py-project.toml file. Which, as I read it, it should. Ideally, every tool.${name} should be in the build system? So, can we improve the file specification and introduce sections that allows a project (not a library) to specify how it should be built? Right now, this is all off-loaded to the tools that use the file and one cannot rely on anything. It also makes migration from one tool to another difficult and cooperation between tools hard. Or is this intentional: is this supposed to be tied to specific tools as build system are generally tightly coupled? And should we not change that? It would help if the PEP had some explanation of the choices made to reserve a file that partially sets out to replace (all?) other files, but does not do anything to accomplish that. I realize that PEP-517 defines the API and leaves the implementation up to the tool, but that doesn't necessarily mean the py- project file should not name / reserve tables that the PEP 517 API should use. I'm considering writing a tool that focuses solely on the deployment and dependency maintenance of a project and would like to not "vendor lock-in" my users, with even those dependency lists, because I have to use the tool namespace and can fill it every way I see fit and so will others. So ideally, we would reserve dependency tables: [dependencies.$name] Reserved names: standard, dev, test Tools can add more names which they can then interpret themselves: [dependencies.stage] django-anonymizer = "^0.5" And test tables and probably one or two more that come to mind. Is there any interest from the team to improve the file format and lock down such tables? I'd love to do the initial proposal if that helps move things forward. -- Melvyn Sopacua

4 5

Re: disable building wheel for a package
by Jeroen Demeyer 15 Sep '18

15 Sep '18

On 2018-09-14 18:36, Paul Moore wrote: > And actually, https://packaging.python.org/guides/distributing-packages-using-setuptools/… > is a reasonably clear explanation of the current behaviour I actually rewrote that paragraph recently, so I'm glad that you find it reasonably clear :-) But indeed, user installations should be mentioned.

2 1

Re: disable building wheel for a package
by Jeroen Demeyer 14 Sep '18

14 Sep '18

On 2018-09-14 16:39, Paul Moore wrote: > My understanding (and I'm not an expert here, so hopefully someone > else will confirm or correct) is that yes, the data directory is > installed to "the installation root", which is $VIRTUAL_ENV for a > virtualenv, and "something else" for non-virtualenvs (I think it's / > on Unix and sys.prefix on Windows, no idea what happens for user > installs). More precisely: it's always sys.prefix for non-user installs (which is "/usr" not "/" on my system). For user installs, it's site.USER_BASE which is "/home/jdemeyer/.local" for me. I think that this works great, I see nothing to fix here (except being more explicit that absolute paths don't work). Jeroen.

3 2

Re: disable building wheel for a package
by Jeroen Demeyer 14 Sep '18

14 Sep '18

On 2018-09-14 15:49, sashk wrote: > Do I understand correctly, that when using relative paths in the > data_files and installing package into virtual environment, > installation prefix is sys.prefix and it is the same as $VIRTUAL_ENV? Indeed, unless you specifically configure things in a non-default way. So you just need to replace absolute paths by paths relative to $VIRTUAL_ENV and your problem will be solved.

1 0

disable building wheel for a package
by sashk 14 Sep '18

14 Sep '18

4 5

Environment markers for GPU/CUDA availibility
by Dustin Ingram 11 Sep '18

11 Sep '18

Hi all, trying to pull together a few separate discussions into a single thread here. The main issue is that currently PEP 508 does not provide environment markers for GPU/CUDA availability, which leads to problems for projects that want to provide distributions for environments with and without GPU support. As far as I can tell, there's been multiple suggestions to bring this issue to distutils-sig, but no one has actually done it. Relevant issues: (closed) "How should Python packages depending on TensorFlow structure their requirements?" https://github.com/tensorflow/tensorflow/issues/7166 (closed) "Adding gpu or cuda specification in PEP 508" https://github.com/python/peps/issues/581 (closed) "More support for conditional installation" https://github.com/pypa/pipenv/issues/1353 (no response) "Adding gpu or cuda markers in PEP 508" https://github.com/pypa/interoperability-peps/issues/68 There is now a third-party project which attempts to amend this for tensorflow (https://github.com/akatrevorjay/tensorflow-auto-detect) but this approach is somewhat fragile (depends on version numbers being in sync), doesn't directly scale to all similar projects, and would require maintainers for a given project to maintain _three_ separate projects, instead of just one. I'm not intimately familiar with PEP 508, so my questions for this list: * Is the demand sufficient to justify supporting this use case? * Is it possible to add support for GPU Environment markers? * If so, what would need to be done? * If implemented, what should the transition look like for projects like tensorflow? Thanks! D.

10 22

Should an sdist/MANIFEST.in include docs and tests?
by Segev Finer 11 Sep '18

11 Sep '18

What should really be included in an sdist via MANIFEST.in? Besides the obvious files required for a functioning package (Anything not caught by the default rules required for the package to function that is) and obviously LICENSE.txt and similar. A package's source tree, more often than not, includes other files such as documentation, tests, examples, a random assortment of other text files, etc. Should docs & tests, in particular, be included in an sdist via MANIFEST.in? Should other files be added too? Or maybe the sdist should be kept to a minimum? This is not clearly discussed in the packaging guide: https://packaging.python.org/guides/distributing-packages-using-setuptools/…. The sampleproject (https://github.com/pypa/sampleproject) does seem to include tests (Well a no-op test) and doesn't include them in the sdist.

9 12

Oct 27-28: Bloomberg sponsoring packaging sprint
by Sumana Harihareswara 10 Sep '18

10 Sep '18

The weekend of October 27-28, simultaneously in London, UK and New York City, USA, Bloomberg will host a Python packaging and distribution tools event. Please mark your calendars! If you live in North America or Europe and would need assistance to attend this as a mentor/helper, watch for more details in July. If you live outside of the US or UK and would need an invitation letter to get a visa to travel to one of these sprints, please write to Kevin P. Fleming at Bloomberg, kpfleming AT bloomberg DOT net, and he'll start setting you up. Details: Thanks to Bloomberg for their generosity. They're already a Platinum PSF sponsor, and they'll host this, pay for a maintainers'/mentors' dinner the night before, provide clusters of cloud virtual machines for the attendees to use, and book and pay for some contributors' lodging and travel. This'll be an opportunity to advance Python packaging/distro tools, teach new contributors (including many Bloomberg employees), and yeah, if you want to get to know Bloomberg for career reasons, that too. :) We hope mentors can arrive Thursday night 25 Oct, do prep, setup, and dinner on Friday, then participate Sat-Sun, then leave Sunday evening or Monday. We'll be putting more details on these lists (distutils-sig and pypa-dev) and at https://wiki.python.org/psf/PackagingSprints . Thanks to Bloomberg folks Mario Corchero and Henry Kleynhans in London and Kevin P. Fleming in New York City for coordinating this, and thanks especially to Mario and to Paul Ganssle for suggesting it! -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

2 2

Three clarification questions about PEP 425 and PyPy3
by Brett Cannon 04 Sep '18

04 Sep '18

I noticed that for PyPy3, the tag triples considered compatible were (roughly; trimmed out the long list of macOS versions): [('pp360', 'pypy3_60', 'macosx_10_13_x86_64'), ('pp360', 'none', 'macosx_10_13_x86_64'), ('py3', 'none', 'macosx_10_13_x86_64'), ('pp360', 'none', 'any'), ('pp3', 'none', 'any'), ('py360', 'none', 'any'), ('py3', 'none', 'any')] Now the first question I have is about ('pp3', 'none', 'any'). Is this meant to be a generic thing for any interpreter of the interpreter implementation and major version, or is this special to CPython and PyPy3? Question two is why isn't there a ('py35', 'none', 'any') or ('py34', 'none', 'any') and older to py30 after py3 like there is for CPython? Seems like if they are just source then they should be compatible as much as CPython. Question three is why isn't there a ('py35', 'none', 'macosx_10_13_x86_64') for PyPy3 or CPython 3.7? I can't figure out what a Python- and platform-specific wheel but agnostic to API wouldn't ever work? And I'm assuming ('py360', 'none', 'any'), isn't legitimate since that makes no sense. ;)

8 35

(no subject)
by youtube.com/[whatItSaysHere] [whatItSaysHere] 04 Sep '18

04 Sep '18

null

1 0

Editable requirement parsing in pip
by Tzu-ping Chung 31 Aug '18

31 Aug '18

Hi, I was verifying my Pipenv resolver work, and found a quirk in pip when handling -e requirements. With a requirements.txt: # Works as expected colorama ; os_name == "nt" # Works as expected ./colorama ; os_name == "nt" # Drops markers; always installs disregarding the OS (!) -e ./colorama ; os_name == "nt" This by itself is okay, since it doesn’t really make sense to me to specify a requirement as editable without intending it to always work, although somehow surprising and (AFAICT) undocumented. The behavior in the command line is a little more preplexing: # Works as expected $ pip install "./colorama ; os_name == 'nt'" # Does not work $ pip install -e "./colorama ; os_name == 'nt'" ./colorama ; os_name == "nt" should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+ # This either $ pip install "-e ./colorama ; os_name == 'nt'" ./colorama ; os_name == "nt" should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+ Since pip’s documentation says a requirements.txt “is just a list of pip install arguments placed in a file” [1], this is surprising to me. [1]: https://pip.pypa.io/en/stable/user_guide/#id1 So my questions are * Is this behaviour intentional, or is it an oversight? * Is there specification or documentation on how pip is supposed to work? Given that PEP 508 (although pip does not actually use its URL lookup syntax) allows requirements to specify markers in all cases, I would expect pip to allow and honour them in all cases. Or, if that’s not the case (since editable is not really a standardised thing), I’d expect pip to at least be consistent with itself :) -- Tzu-ping Chung (@uranusjr) uranusjr(a)gmail.com https://uranusjr.com

1 0

Clarification on string arguments in PEP 517 hooks
by Paul Moore 28 Aug '18

28 Aug '18

I'm in the process of implementing PEP 517 for pip, and I've hit a question. I'm pretty sure I know the answer, but I want to be clear, as whatever the answer is will require some fixing up. The various hooks take directory paths as arguments, and typically return a filename (e.g., build_wheel). The returned filename is always explicitly noted as being *a unicode string*. However, argumnents (metadata_directory in build_wheel/prepare_metadata_for_build_wheel) are *not* explicitly mandated to be Unicode. My assumption is that the intent is that *all* strings, whether arguments or return values, must be Unicode. The reason that I've hit this is that the current setuptools PEP 517 backend passes the metadata_directory direct to distutils which, on Python 2, checks for a (non-Unicode) string type and errors if Unicode is supplied. Fixing that is going to be annoyingly tricky, so I want to be sure that's the requirement before I start making fixes to setuptools. Paul

3 4

AofzaX.TH
by youtube.com/[whatItSaysHere] [whatItSaysHere] 25 Aug '18

25 Aug '18

Amazing Thailand

1 0

(no subject)
by bunkeryou 24 Aug '18

24 Aug '18

ส่งจากสมาร์ทโฟน Samsung Galaxy ของฉัน

1 0

Re: pipenv and pip
by Tzu-ping Chung 21 Aug '18

21 Aug '18

Hi, Dan and I had been doing most of the maintenance work for Pipenv recently, and as Dan mentioned, we have been working on some related projects that poke into pip internals significantly, so I feel I should voice some opinions. I have significantly less experience messing with pip than Dan, and might be able to offer a slightly different perspective. Pipenv mainly interacts with pip for two things: install/uninstall/upgrade packages, and to gain information about a package (what versions are available, what dependencies does a particular version has, etc.). For the former case, we are currently using it with subprocesses, and it is likely the intended way of interaction. I have to say, however, that the experience is not flawless. pip has a significant startup time, and does not offer chances for interaction once it is started on running, so we really don’t have a good way to, for example, provide installation progress bar for the user, unless we parse pip’s stdout directly. These are not essential to Pipenv’s functionality, however, so they are more like an annoyance rather than glaring problems. The other thing Pipenv uses pip for—getting package information—is more troubling (to me, personally). Pipenv has a slightly different need from pip regarding dependency resolution. pip can (and does) freely drop dependencies that does not match the current environment, but Pipenv needs to generate a lock file for an abstract platform that works for, say, both macOS and Windows. This means pip’s resolver is not useful for us, and we need to implement our own. Our own resolver, however, still needs to know about packages it gets, and we are left with two choices: a. try re-implement the same logic, or b. use pip internals to cobble something together. We tried to go for a. for a while, but as you’d easily imagine, our own implementation is buggy, cannot handle edge cases nearly as well, and fielded a lot of complaints along the lines of “I can do this in pip, why can’t I do the same in Pipenv”. One example is how package artifacts are discovered. At my own first glance, I thought to myself this wouldn’t be that hard—we have a simple API, and the naming conventions are there, so as long as we specify sources in Pipfile (we do), we should be able to discover them no problem. I couldn’t be more wrong. There are find_links, dependency_links, pip.conf for the user, for the machine, all sorts of things, and for everything quirk in pip we don’t replicate 100%, issues are filed urging use to fix it. In the end we gave up and use pip’s internal PackageFinder instead. This is a big problem going forward, and we are fully aware of that. The strategy we are taking at the moment is to try to limit the surface area of pip internals usage. Dan mentioned we have been building a resolver for Pipenv[1], and we took the chance to work toward centralising things interfacing with pip internals. Those are still internals, of course, but we now have a relatively good idea what we actually need from pip, and I’d be extremely happy if some parts of pip can come out as standalone with official blessing. The things I am particularly interested in (since they would be beneficial for Pipenv) are: * VcsSupport * PackageFinder * WheelBuilder (and everything that comes with it like the wheel cache, preparer, unpack_url, etc.) Sorry for the very long post, but I want to get everything out so it might be easier to paint a complete picture of the state we are currently in. [1]: https://github.com/sarugaku/passa <https://github.com/sarugaku/passa> Yours, TP -- Tzu-ping Chung (@uranusjr) uranusjr(a)gmail.com https://uranusjr.com > On 21/8, 2018, at 00:00, distutils-sig-request(a)python.org wrote: > > Send Distutils-SIG mailing list submissions to > distutils-sig(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > or, via email, send a message with subject or body 'help' to > distutils-sig-request(a)python.org > > You can reach the person managing the list at > distutils-sig-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Distutils-SIG digest..."Today's Topics: > > 1. Re: pipenv and pip (Dan Ryan) > 2. Re: pipenv and pip (Dan Ryan) > > From: Dan Ryan <dan(a)danryan.co> > Subject: [Distutils] Re: pipenv and pip > Date: 20 August 2018 at 22:04:11 GMT+8 > To: Chris Jerdonek <chris.jerdonek(a)gmail.com> > Cc: distutils sig <distutils-sig(a)python.org> > > > The truth is that it’s basically impossible to gauge bugs in pip vs bugs in our patches to it which are often a lot more likely — reproductions of edge cases can be impossible but there are specific things I know we broke (like parsing certain kinds of extras, previously) — mostly bugs land in pips issue tracker before we report them or we will direct people there. We have like 2 active maintainers and we are maintaining like 15 pipenv related projects so we normally just point people at pip rather than file an issue. I am usually on irc as well if needed, and I often ask for clarification there > > Dan Ryan // pipenv maintainer > gh: @techalchemy > >> On Aug 20, 2018, at 4:32 AM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: >> >> Thanks. Is the state of affairs as you described them what you're >> planning for the future as well, or do you anticipate any changes >> worthy of note? >> >> Also, are any of the bugs filed in pipenv's tracker due to bugs or >> rough spots in pip -- is there a way to find those, like by using a >> label? It would be good to be able to know about those so pip can >> improve and become more useful. It doesn't seem like any bugs have >> been filed in pip's tracker in the past year by any of pipenv's top >> contributors. That seems a bit surprising to me given pipenv's heavy >> reliance on pip (together with the fact that I know pip has its share >> of issues), or is there another way you have of communicating >> regarding things that interconnect with pip? >> >> Thanks, >> --Chris >> >> >> >>> On Mon, Aug 20, 2018 at 12:51 AM, Dan Ryan <dan(a)danryan.co> wrote: >>> Sure I can grab that— we patch pip because we use some internals to handle resolution and we have some bugs around that currently. They aren’t upstreamed because they aren’t actually present in pip, only in pipenv. Pipenv crosses back and forth across the virtualenv boundary during the process. Pipenv relies on piptools and vendors a patched version of pip to ensure consistency as well as to provide a few hacks around querying the index. We do have a bit of reimplementation around some kinds of logic, with the largest overlap being in parsing of requirements. >>> >>> As we handle some resolution, which isn’t really something pip does, there is no cli interface to achieve this. I maintain a library (as of last week) which provides compatibility shims between pip versions 8-current. It is a good idea to use the cli, but we already spend enough resources forking subprocesses into the background that it is a lot more efficient to use the internals, which I track quite closely. The preference toward cli interaction is largely to allow internal api breakage which we don’t mind. >>> >>> For the most part, we have open channels of communication as necessary. We rely as heavily as we can on pip, packaging, and setuptools to connect the dots, retrieve package info, etc. >>> >>> Dan Ryan // pipenv maintainer >>> gh: @techalchemy >>> >>>> On Aug 20, 2018, at 2:41 AM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: >>>> >>>> Hi, >>>> >>>> Can someone explain to me the relationship between pipenv and pip, >>>> from the perspective of pipenv's maintainers? >>>> >>>> For example, does pipenv currently reimplement anything that pip tries >>>> to do, or does it simply call out to pip through the CLI or through >>>> its internal API's? Does it have any preferences or future plans in >>>> this regard? How about upstreaming to pip fixes or things that would >>>> help pipenv? >>>> >>>> I've been contributing to pip more lately, and I had a look at >>>> pipenv's repository for the first time today. >>>> https://github.com/pypa/pipenv >>>> >>>> Given that pip's code was recently made internal, I was a bit >>>> surprised to see that pipenv vendors and patches pip: >>>> https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip >>>> Before I had always assumed that pipenv used pip's CLI (because that's >>>> what pip says you should do). >>>> >>>> I also noticed that some bugs in pipenv's tracker seem closely related >>>> to pip's behavior, but I don't recall seeing any bugs or PR's in pip's >>>> tracker reported from pipenv maintainers. >>>> >>>> Without knowing a whole lot more than what I've stated, one concern I >>>> have is around fragmentation, duplication of work, and repeating >>>> mistakes (or introducing new ones) if a lot of work is going into >>>> pipenv without coordinating with pip. Is this in any way similar to >>>> the beginning of what happened with distutils, setuptools, and >>>> distribute that we are still recovering from? >>>> >>>> --Chris >>>> -- >>>> Distutils-SIG mailing list -- distutils-sig(a)python.org >>>> To unsubscribe send an email to distutils-sig-leave(a)python.org >>>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >>>> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… > > > > From: Dan Ryan <dan(a)danryan.co> > Subject: [Distutils] Re: pipenv and pip > Date: 20 August 2018 at 22:09:23 GMT+8 > To: Paul Moore <p.f.moore(a)gmail.com> > Cc: Distutils <distutils-sig(a)python.org> > > > How would I (said library) maintain compatibility? I’m pretty clever. The shim library doesn’t actually do anything besides provide import paths. If I shim something that didn’t exist, it shims None for any pip versions where it doesn’t exist. So for example if you are running pip 9 and you import RequirementTracker from the shims library you just import None > > Dan Ryan // pipenv maintainer > gh: @techalchemy > >> On Aug 20, 2018, at 8:15 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: >> >>> On Mon, 20 Aug 2018 at 12:25, Wes Turner <wes.turner(a)gmail.com> wrote: >>> >>> On Monday, August 20, 2018, Paul Moore <p.f.moore(a)gmail.com> wrote: >> >>>> I know "security by obscurity" doesn't work, but I'm happier if >>>> details of this library *aren't* widely known - it's not something I'd >>>> want to encourage people using, nor is it supported by pip, as it's >>>> basically a direct interface into pip's internal functions, papering >>>> over the name changes that we did in pip 10 specifically to dissuade >>>> people from doing this. >>> >>> >>> If someone was committing to identifying useful API methods, parameters, and return values; >>> writing a ~PEP; >>> implementing said API; >>> and maintaining backwards compatible shims for some reason; >>> would something like `pip.api` be an appropriate namespace? >>> (now that we're on version 18 with a faster release cycle)? >> >> I'm not quite sure I know what you mean here. The key point is that >> pip 18.0 might have an internal function pip._internal.xxx, and in pip >> 18.1 there's no such function, and the functionality doesn't even >> exist any more. How would a 3rd party project maintain backwards >> compatible shims in the face of that? Agreed it's not likely in >> practice - but we're not going to guarantee it. >> >> To be honest I don't see the point of discussing pip's internal API. >> It's just that - internal. I'd rather discuss useful (general) >> packaging libraries, that tools can build on - pip can vendor those >> and act as (just) another consumer, rather than getting into debates >> about support and internal APIs. >> >> Paul >> -- >> Distutils-SIG mailing list -- distutils-sig(a)python.org >> To unsubscribe send an email to distutils-sig-leave(a)python.org >> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ >> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… > > > -- > Distutils-SIG mailing list -- distutils-sig(a)python.org > To unsubscribe send an email to distutils-sig-leave(a)python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/

6 9

Accessing extras during install command
by Joni Orponen 21 Aug '18

21 Aug '18

Hello, I'd like to run different external programs during the install command which are provided by (potentially) installed extras, but I cannot find where the currently selected extras are stored. The only thing I quickly found by going through the distutils and setuptools codebases is an old TODO from easy_install indicating it might not have been implemented yet: https://github.com/pypa/setuptools/blob/afba2d89c4436451ad4d7e19228584246a9… I can hack this by try excepting on ImportError, but is there a more elegant (and intended) way? -- Joni Orponen

2 1

pipenv and pip
by Chris Jerdonek 20 Aug '18

20 Aug '18

Hi, Can someone explain to me the relationship between pipenv and pip, from the perspective of pipenv's maintainers? For example, does pipenv currently reimplement anything that pip tries to do, or does it simply call out to pip through the CLI or through its internal API's? Does it have any preferences or future plans in this regard? How about upstreaming to pip fixes or things that would help pipenv? I've been contributing to pip more lately, and I had a look at pipenv's repository for the first time today. https://github.com/pypa/pipenv Given that pip's code was recently made internal, I was a bit surprised to see that pipenv vendors and patches pip: https://github.com/pypa/pipenv/tree/master/pipenv/patched/notpip Before I had always assumed that pipenv used pip's CLI (because that's what pip says you should do). I also noticed that some bugs in pipenv's tracker seem closely related to pip's behavior, but I don't recall seeing any bugs or PR's in pip's tracker reported from pipenv maintainers. Without knowing a whole lot more than what I've stated, one concern I have is around fragmentation, duplication of work, and repeating mistakes (or introducing new ones) if a lot of work is going into pipenv without coordinating with pip. Is this in any way similar to the beginning of what happened with distutils, setuptools, and distribute that we are still recovering from? --Chris

5 13

Re: Cannot upload distribution archives using twine using functioning username and login for pypi.org
by Chung Tzu-ping 16 Aug '18

16 Aug '18

According to the documentation[1], TestPyPI is a complete independent instance from PyPI, and they don’t share databases at all. You need to register a separate account there. [1]: https://packaging.python.org/guides/using-testpypi/ -- Tzu-ping Chung (@uranusjr) uranusjr(a)gmail.com Sent from my iPhone > On 16 Aug 2018, at 23:55, tom(a)tombaker.org wrote: > > When I follow the instructions at > https://packaging.python.org/tutorials/packaging-projects/ > to upload my distribution archives with > > $ twine upload --repository-url https://test.pypi.org/legacy/ dist/*` > > I get prompted for my username and password, but the upload aborts with > `HTTPError: 403 Client Error` (see below) even though I can successfully log > into https://pypi.org using the same username and password. > > Can anyone advise? > > Tom > > ``` > Uploading distributions to https://test.pypi.org/legacy/ > Enter your username: tombaker > Enter your password: > Uploading mklists-0.1.1-py3-none-any.whl > 100%|*************************************************************************| 10.6k/10.6k [00:00<00:00, 15.8kB/s] > HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://test.pypi.org/legacy/ > ```

1 0

Cannot upload distribution archives using twine using functioning username and login for pypi.org
by tom＠tombaker.org 16 Aug '18

16 Aug '18

When I follow the instructions at https://packaging.python.org/tutorials/packaging-projects/ to upload my distribution archives with $ twine upload --repository-url https://test.pypi.org/legacy/ dist/*` I get prompted for my username and password, but the upload aborts with `HTTPError: 403 Client Error` (see below) even though I can successfully log into https://pypi.org using the same username and password. Can anyone advise? Tom ``` Uploading distributions to https://test.pypi.org/legacy/ Enter your username: tombaker Enter your password: Uploading mklists-0.1.1-py3-none-any.whl 100%|*************************************************************************| 10.6k/10.6k [00:00<00:00, 15.8kB/s] HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://test.pypi.org/legacy/ ```

3 2

Is ensurepip still a thing?
by Chris Barker 12 Aug '18

12 Aug '18

I'm updating some instructions for my students, in which the first thing I do is have them run ensurepip: $ python3 -m ensurepip --upgrade which resulted in: $ python3 -m ensurepip --upgrade Looking in links: /var/folders/ym/tj87fc850yd6526nbrn14rxm0000gn/T/tmpwc8nd6oj Requirement already up-to-date: setuptools in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (39.0.1) Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (10.0.1) (this is after a brand-new python 3.7 install on OS-X from python.org) All good. But then I use pip, and get (after a successful install): You are using pip version 10.0.1, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Huh? shouldn't ensurepip have updated it for me already?? Or should I simply suggest the pip install --upgrade pip command and not bother with ensurepip anymore? BTW -- shouldn't that be: python3 -m pip install --upgrade pip to make sure they get the "right" pip? KInda hard to keep up with the latest ... Thanks, -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

6 8

Re: Packaging Advice for EFF's Certbot
by Brad Warren 06 Aug '18

06 Aug '18

>> On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> >> However, there *are* folks that have been working on allowing >> applications to be defined primarily as Python projects, and then have >> the creation of wrapper native installers be a pushbutton exercise, >> rather than requiring careful human handholding. > > But it sounds like they also want to be able to install/remove/upgrade > *parts* of the Python project, for their plugin support. That’s correct. We currently have 18 official plugins for Certbot with plans to add more and a few dozen third-party plugins. > Do any of these tools allow that? This is a good question. If we went with something like dh-virtualenv or packaged virtualenvs with fpm, would we be able to have separate packages for our plugins that installed things in the same virtualenv? I haven’t looked into this yet, but I wouldn’t expect this to work. > That's the thing that really made me think about conda. My biggest concern with conda right now is I believe we (or our users) would be on their own for setting up a cron job or systemd timer for renewal. Is this correct? > On Jul 24, 2018, at 11:20 PM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: > > Just to be clear, I wasn't meaning to promote or recommend the Docker > option I described. Sure! After reading your 2nd email describing how this would work in more detail, I think this would require a pretty major rewrite to how Certbot currently works. Given the other downsides, I’m not sure this is the best approach for us, but I appreciate you throwing out the idea regardless just in case it was! > On Jul 26, 2018, at 3:20 AM, Ben Finney via Distutils-SIG <distutils-sig(a)python.org> wrote: > Just focus on Certbot, and cheer from the sidelines as the OS distributions > do the work of third party packages. > > Yes, that's a different set of problems (for example, keeping Certbot > compatible with those versions of the libraries the OS repositories > provide). We’ve done the work to maintain compatibility with the older versions of our dependencies available in the official OS repos where we are packaged. The source of our problems with official OS packaging is described below and in the Google Doc linked in my initial email. > On Jul 28, 2018, at 8:53 AM, matthew(a)woodcraft.me.uk wrote: > > I wouldn't be too put off by the idea of Debian politics. Certbot should be a good fit for stable-updates: We thought so too and getting updates like this was our main packaging plan when we launched in 2015. Unfortunately, it hasn’t gone well and is the main reason we’re seeking our own solution. Perhaps we did something incorrectly, but as Nathaniel pointed out, Certbot is broken in Debian Stretch and has been since January. The same and many other problems affect the packages in Ubuntu Xenial. We’ve also struggled to find people to help maintain our PPA for older, non-EOL’d versions of Ubuntu. If anyone reading this would like to help us solve these problems or know someone who would, please reach out off-list. While these packages exist in OS repos, some users will continue to use them regardless of the alternative packaging approach we take. Unless the current issues are resolved and we’re confident new ones in the future will be fixed quickly as well, I think we need to offer alternative packaging so we can provide our users some means of getting a working version of Certbot. > On Jul 28, 2018, at 12:00 PM, Wes Turner <wes.turner(a)gmail.com> wrote: > > Took a minute more to read the gdoc link. Wes Turner, while I don’t have any specific questions or comments right now, thank you very much for all the ideas and links.

3 3

Re: Proposal for timestamps of installed files
by Jeroen Demeyer 06 Aug '18

06 Aug '18

On 2018-08-06 19:49, Chris Jerdonek wrote: > Does this proposal mean that .pyc files would need to be regenerated > after installing from a wheel (because the timestamp inside the .pyc > file would no longer match the .py file from which it came)? No, I specifically mentioned that the timestamps of .py files would be preserved.

1 0

Proposal for timestamps of installed files
by Jeroen Demeyer 06 Aug '18

06 Aug '18

Proposal: change timestamps of files installed from a wheel to the time of installation, except for .py files. **Existing situation** Let's consider two ways of installing a Python package: installation from a source tarball and installation from a wheel, in both cases using "pip install". The installation roughly works as follows: (A) Create a temporary directory for pip to work in, with a filename like /tmp/pip-6N_2Kx-build (B) Extract the archive (sources or wheel) somewhere in the temporary directory (C) Build the package in the temporary directory (in both cases, this involves creating .pyc files for the .py files) (D) Copy the installation from the temporary directory to the final installation directory (typically somewhere in site-packages) (E) Delete the temporary directory Both (B) and (D) involve a copy operation of some sort, so the question is: what to do with timestamps? In both cases, (D) currently preserves the timestamp of the files. For (B), timestamps are preserved when extracting a tarball but not when extracting a wheel. **Proposal** Change step (D) for wheel installs to NOT preserve timestamps. It might be safer to make sure that all installed files have the same timestamp, so this would mean: give all installed files the timestamp of a fixed time, namely the time when step (D) was started in the installation process. Important exception: for .py files, do preserve the timestamp. I think that (B) should preserve timestamps. However, that is a different issue which should be fixed independently. **The problem** For a source installation, the timestamps are preserved in both (B) and (D), so the timestamp of the installed files can be much older than the time of installation. This is a problem for dependency checking: suppose that I am the developer of a package MYPKG which has a dependency on the C API of a package OTHERPKG. The C API of OTHERPKG is defined by .h files which are installed by OTHERPKG. The package MYPKG correctly declares this dependency using the "depends" keyword of the Extension objects. Now, when I install a different version of OTHERPKG, I want that distutils notices that the dependency has been changed and automatically rebuilds the Extensions with that dependency. This requires that the timestamp of the installed .h files is more recent than the timestamp of the compiled extension. In other words, the timestamp of installed files should be the time of installation. A similar situation happens for a Cython API exposed by installed .pxd (and possibly also .h) files. **Comparison with automake** Automake (which is probably the most widely used build system for open source/free software) installs files with timestamp the time of installation. So, my proposal is doing what automake already does. **The strange case of .pyc files** As noted in https://github.com/pypa/pip/issues/5648 it is important that the timestamp of .py files are preserved after the .pyc file is created. This is because Python checks the exact timestamp of the .py file to determine whether it is up to date. This explains the exception for .py files in the proposal. **Backwards compatibility** In other words: won't this break packages which rely on timestamps to be preserved? When installing a wheel, timestamps from the wheel are already not preserved in step (B). So any package which would rely on preserving timestamps would already be broken when installed as wheel. So this proposal can only break stuff which depends on timestamps from step (C), which happens to be the case for .py files. In my proposal, I'm only suggesting to make changes for wheel installs. I do think that the change makes sense for installs from source, but that will be irrelevant after PEP 517 is implemented anyway. Jeroen.

2 1

Re: PEP 541 Regarding emd
by Robert Collins 05 Aug '18

05 Aug '18

On 4 August 2018 at 03:34, Chathika Gunaratne <Chathika.Gunaratne(a)ucf.edu> wrote: > Dear Admin, > > > > I am writing to you regarding the Package Index name emd. I am a PhD student > at the University of Central Florida and for my dissertation, I have > recently developed a Python software, Evolutionary Model Discovery, which I > am assembling into a package, EMD. https://github.com/chathika/EMD > > > > However, I noticed that there is already a package named emd on pypi.org. > This package seems abandoned with just one release in February 2012 and no > contact information about the author, > https://pypi.org/project/emd/1.0/#history . > > > > I read about PEP 541 https://www.python.org/dev/peps/pep-0541/ and was > curious as to if it is still possible that I can acquire the package name > EMD. I wanted to make sure of this before I went ahead with the name EMD for > my package, please. Generally speaking its not whether a package has been updated that matters, its whether the package is in use: if its published in debian/suse/fedora, if its getting downloads, then taking over the name is going to have negative consequences. That said, this has had 0 downloads per the bigtable statistics (0 for all recorded time) so in this case that seems unlikely. -Rob

2 1

packaging guide and private packages index
by David Cournapeau 04 Aug '18

04 Aug '18

Hi, I see that there is almost no mention of private packages index in the packaging guide, and no recommendation on what to use. Currently googling for private packages mostly return obsolete (and not very practical) recommendations based on dependency links. In 2018, what would be the recommended practices for teams that develop private packages that depend on each other ? thank you, David

9 9

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 14:02, Thomas Kluyver wrote: > On Sat, Aug 4, 2018, at 9:34 AM, Paul Moore wrote: >> Whether timestamps are >> preserved by the wheel building process depends on the build system - >> so the question boils down to "does setup.py bdist_wheel preserve >> timestamps?" in the case of the setuptools backend - which is really a >> question for the wheel project. In the more general case, you'd have >> to ask the same question of flit, and any other backends you cared >> about. > > IIRC, Flit will preserve the timestamps of the files when you build a wheel For the record: my post wasn't about *building* a wheel, but about *installing* a wheel.

3 2

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 00:02, Chris Jerdonek wrote: > I'm not sure how relevant it is, but this issue was recently filed on > pip's issue tracker ("Reproducible installs"): > https://github.com/pypa/pip/issues/5648 This seems to be about preserving timestamps when extracting wheel files. > There is also an older (closed) pip issue that might be relevant > ("Preserving timestamps on copy"): > https://github.com/pypa/pip/issues/3201 This is a different issue about preserving timestamps from the source directory to the temporary build directory. So both are different issues, and I agree with both: during the source extraction and build process, you want to preserve timestamps as much as possible. But for the installation, you do NOT want to preserve timestamps.

4 4

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 13:16, Paul Moore wrote: > Can you give a > specific example of an end to end process where the packaging > toolchain's current behaviour gives demonstrably the wrong result? Yes I can. I will work out a detailed proposal about timestamps in the build/install process in general. But please give me some time to write it up.

2 1

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

The fact that installs from wheels don't preserve timestamps is a very good argument that it's OK to NOT preserve timestamps in general. Any package which would rely on preserving timestamps would already be broken when installing using a wheel. By the way, initially I thought that this was a distutils issue, but it's clear now that it's really a pip issue. Jeroen.

2 1

Re: Timestamp of installed files
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 10:34, Paul Moore wrote: > Jeroen seemed to say he agreed with this, but > I'm not sure I see how that matches his stated requirement for > installs to not preserve timestamps... The way how pip currently works (I assume that this stays the case) is that it uses a temporary build directory for everything, for example /tmp/pip-build-uvfWuR So concretely for a wheel install: (A) Extract the wheel in build directory (B) Generate .pyc files in build directory (C) Copy files from build directory to final installation directory in site-packages. The way I understand it, https://github.com/pypa/pip/issues/5648 is about (A) while my proposal is about (C). So I think that timestamps should be preserved in (A) but not for (C). Jeroen.

2 1

Timestamp of installed files
by Jeroen Demeyer 03 Aug '18

03 Aug '18

Hello, I would like to draw attention to https://bugs.python.org/issue32773 Currently, distutils (and everything that uses it, such as setuptools and pip) installs files with the same timestamp as in the sources (I'm only discussing copied files, not generated/compiled files). In other words, it preserves the timestamp when it copies the file from source to installation directory. There is a comment in the distutils sources # XXX copy_file by default preserves atime and mtime. IMHO this is # the right thing to do, but perhaps it should be an option -- in # particular, a site administrator might want installed files to # reflect the time of installation rather than the last # modification time before the installed release. It just says "IMHO, this is the right thing to do" without any justification. Now IMHO, preserving timestamps is NOT the right thing to do: the timestamp of installed files should be the time of installation. Autotools (which is probably the most-used installer for open source/free software projects) does that. The reason why preserving timestamps is bad is because it breaks dependency checking: when you install something, you typically want to rebuild other things depending on it. For .py files, this isn't relevant (nothing depends on them), but it is relevant for .h files and certain Cython files. So I suggested in bpo-32773 to fix this. The main open question is: to what extent could this break backwards compatibility? Personally, I cannot imagine a case where it is important that timestamps are preserved. If it's decided that this should become optional, it should be done on a per-project basis (and not by the user installing the package). For example, I would want all my projects to not preserve timestamps. Suggestions on how to do this (a new argument to setup()?) are welcome. Jeroen PS: if you're interested in the history, this goes back to commit 13ae1c8ff81befcfd0b0ece98ef471cd504642d8 Author: Greg Ward <gward(a)python.net> Date: Mon Mar 22 14:55:25 1999 +0000 First checkin of real Distutils command modules. So yes, I want to change a 19-year old "feature".

2 1

PEP 541 Regarding emd
by Chathika Gunaratne 03 Aug '18

03 Aug '18

Dear Admin, I am writing to you regarding the Package Index name emd. I am a PhD student at the University of Central Florida and for my dissertation, I have recently developed a Python software, Evolutionary Model Discovery, which I am assembling into a package, EMD. https://github.com/chathika/EMD However, I noticed that there is already a package named emd on pypi.org. This package seems abandoned with just one release in February 2012 and no contact information about the author, https://pypi.org/project/emd/1.0/#history . I read about PEP 541 https://www.python.org/dev/peps/pep-0541/ and was curious as to if it is still possible that I can acquire the package name EMD. I wanted to make sure of this before I went ahead with the name EMD for my package, please. Looking forward to your reply. Best Regards, Chathika Gunaratne Graduate Research Assistant, Complex Adaptive Systems Lab University of Central Florida, Room 314, Engineering II, 12800 Pegasus Dr., Orlando, FL 32816-2993

1 0

Discussion on handling various User Agents in Linehaul/BigQuery on PyPI
by Donald Stufft 03 Aug '18

03 Aug '18

Hey there! I’m trying to sort out what people are expecting behavior wise from the BigQuery statistics for PyPI. I’ve opened up an issue on Linehaul (the daemon that handles processing those statistics) at https://github.com/pypa/linehaul/issues/44 <https://github.com/pypa/linehaul/issues/44>. If anyone has any opinion on what we do here, please feel free to weigh in on that issue.

1 0

PEP 541
by faresana Ahmade 01 Aug '18

01 Aug '18

abou5alil1990(a)gmail.com

1 0

Pip install Issue
by Monika.singhal＠anthem.com 31 Jul '18

31 Jul '18

Hi I am seeing issue with installing any package using pip. Could you please assist? Below is the error I am getting. (/home/ab63996/test) [ab63996@dwbddisc1r2e ~]$ conda list # packages in environment at /home/ab63996/test: # ca-certificates 2018.03.07 0 certifi 2018.4.16 py27_0 libgcc-ng 7.2.0 hdf63c60_3 openssl 1.0.2o h20670df_0 pip 10.0.1 py27_0 python 2.7.12 1 readline 6.2 2 setuptools 39.2.0 py27_0 sqlite 3.13.0 0 tk 8.5.18 0 wheel 0.31.1 py27_0 zlib 1.2.11 ha838bed_2 (/home/ab63996/test) $ pip install --trusted-host=files.pythonhosted.org spyder --cert /opt/cloudera/security/x509/cert.pem Collecting spyder Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl (Caused by ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))) You are using pip version 10.0.1, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. (/home/ab63996/test) [ab63996@dwbddisc1r2e ~]$

3 2

Re: Issue with using pip install
by Wayne Werner 31 Jul '18

31 Jul '18

Are you using an older version of Python? On Mon, Jul 30, 2018, 5:53 PM Singhal, Monika <Monika.Singhal(a)anthem.com> wrote: > I am setting issue installing any package using pip. I am getting below > error. Do you have any idea on what could be causing the issue? > > > > You should consider upgrading via the 'pip install --upgrade pip' command. > > (/home/ab63996/test) [ab63996@dwbddisc1r2e ~]$ pip install > --trusted-host=files.pythonhosted.org spyder --cert > /opt/cloudera/security/x509/cert.pem > > Collecting spyder > > Retrying (Retry(total=4, connect=None, read=None, redirect=None, > status=None)) after connection broken by 'ProtocolError('Connection > aborted.', error(104, 'Connection reset by peer'))': > /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl > > Retrying (Retry(total=3, connect=None, read=None, redirect=None, > status=None)) after connection broken by 'ProtocolError('Connection > aborted.', error(104, 'Connection reset by peer'))': > /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl > > Retrying (Retry(total=2, connect=None, read=None, redirect=None, > status=None)) after connection broken by 'ProtocolError('Connection > aborted.', error(104, 'Connection reset by peer'))': > /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl > > Retrying (Retry(total=1, connect=None, read=None, redirect=None, > status=None)) after connection broken by 'ProtocolError('Connection > aborted.', error(104, 'Connection reset by peer'))': > /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl > > Retrying (Retry(total=0, connect=None, read=None, redirect=None, > status=None)) after connection broken by 'ProtocolError('Connection > aborted.', error(104, 'Connection reset by peer'))': > /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl > > Could not install packages due to an EnvironmentError: > HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries > exceeded with url: > /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl > (Caused by ProtocolError('Connection aborted.', error(104, 'Connection > reset by peer'))) > > > > You are using pip version 10.0.1, however version 18.0 is available. > > You should consider upgrading via the 'pip install --upgrade pip' command. > > (/home/ab63996/test) [ab63996@dwbddisc1r2e ~]$ conda list > > # packages in environment at /home/ab63996/test: > > # > > ca-certificates 2018.03.07 0 > > certifi 2018.4.16 py27_0 > > libgcc-ng 7.2.0 hdf63c60_3 > > openssl 1.0.2o h20670df_0 > > pip 10.0.1 py27_0 > > python 2.7.12 1 > > readline 6.2 2 > > setuptools 39.2.0 py27_0 > > sqlite 3.13.0 0 > > tk 8.5.18 0 > > wheel 0.31.1 py27_0 > > zlib 1.2.11 ha838bed_2 > > > > > > > > > > > > > > > > *CONFIDENTIALITY NOTICE:* This e-mail message, including any attachments, > is > for the sole use of the intended recipient(s) and may contain confidential > and privileged information or may otherwise be protected by law. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply e-mail > and destroy all copies of the original message and any attachment thereto. > -- > Distutils-SIG mailing list -- distutils-sig(a)python.org > To unsubscribe send an email to distutils-sig-leave(a)python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… >

2 1

Issue with using pip install
by Singhal, Monika 30 Jul '18

30 Jul '18

I am setting issue installing any package using pip. I am getting below error. Do you have any idea on what could be causing the issue? You should consider upgrading via the 'pip install --upgrade pip' command. (/home/ab63996/test) [ab63996@dwbddisc1r2e ~]$ pip install --trusted-host=files.pythonhosted.org spyder --cert /opt/cloudera/security/x509/cert.pem Collecting spyder Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl (Caused by ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))) You are using pip version 10.0.1, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. (/home/ab63996/test) [ab63996@dwbddisc1r2e ~]$ conda list # packages in environment at /home/ab63996/test: # ca-certificates 2018.03.07 0 certifi 2018.4.16 py27_0 libgcc-ng 7.2.0 hdf63c60_3 openssl 1.0.2o h20670df_0 pip 10.0.1 py27_0 python 2.7.12 1 readline 6.2 2 setuptools 39.2.0 py27_0 sqlite 3.13.0 0 tk 8.5.18 0 wheel 0.31.1 py27_0 zlib 1.2.11 ha838bed_2 CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information or may otherwise be protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and any attachment thereto.

1 0

Issue with installing python packages using pip
by Monika.singhal＠anthem.com 30 Jul '18

30 Jul '18

Hi, I am seeing issue installing python package using pip in my virtual environment. I am getting below error. Could you please assist? I have python version as 2.7.12. # packages in environment at /home/ab63996/test: # ca-certificates 2018.03.07 0 certifi 2018.4.16 py27_0 libgcc-ng 7.2.0 hdf63c60_3 openssl 1.0.2o h20670df_0 pip 10.0.1 py27_0 python 2.7.12 1 readline 6.2 2 setuptools 39.2.0 py27_0 sqlite 3.13.0 0 tk 8.5.18 0 wheel 0.31.1 py27_0 zlib 1.2.11 ha838bed_2 (/home/ab63996/test) [$ pip install --trusted-host=files.pythonhosted.org spyder --cert /opt/cloudera/security/x509/cert.pem Collecting spyder Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/db/96/59727db154f253ae81a1491481fb53bff0ebf1a6a03d2258e6eb8d57cb66/spyder-3.3.0-py2-none-any.whl (Caused by ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))) You are using pip version 10.0.1, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

1 0

Re: Packaging Advice for EFF's Certbot
by Brad Warren 28 Jul '18

28 Jul '18

Wow! Thanks for all the feedback everyone. > On Jul 23, 2018, at 6:52 PM, Chris Jerdonek <chris.jerdonek(a)gmail.com> wrote: > > 1) Can you list in the document the non-Python dependencies and what > they're used for to give people an idea? Done! I added this to the “Certbot Background” section at the top of the document. > 2) I don't know how Certbot is architected, but would it be possible > to put the "meat" of Certbot inside a Docker container (ideally > including most of the non-Python dependencies), and then have a much > lighter-weight Python application (with fewer dependencies) running > outside of Docker, and that calls the application inside the Docker > container when it needs to do stuff? Maybe. Currently each method of performing the domain validation required to get a certificate and the support for installing the certificate in different servers is implemented by a plugin to Certbot. These plugins implement a class with a certain interface which it registers as a setuptools entry_point. At runtime, Certbot picks the class type desired by the user, creates an instance of it, and calls its methods as necessary to obtain a certificate, install it, etc. Most of these plugins make use of code in Certbot’s modules that implement common functionality to simplify plugin development. This architecture seems like the opposite of what you’re proposing as Certbot is calling the plugin as needed rather than the other way around, but we’re certainly open to the possibility of changing things to make it easier to distribute Certbot. Are you aware of another significantly sized project that is structured in this way? > On Jul 23, 2018, at 7:04 PM, Eli Ribble via Distutils-SIG <distutils-sig(a)python.org> wrote: > > I think I'd separate out certbot installed by the package manager which is just a simple bootstrapper and the certbot installed in the virtualenv which does real work. This is an interesting idea. By making our own .deb and .rpm packages, we can handle things like installing non-Python dependencies and adding cron jobs or systemd timers in the conventional, system specific way rather than trying to use our single certbot-auto script that tries to work everywhere. It also gives us the option to distribute this Certbot bootstrapper separately on less popular OSes with the caveat that those users will have to install dependencies and set up automating certificate renewal themselves. Thanks for the suggestion! > On Jul 23, 2018, at 7:54 PM, Michael Sarahan <msarahan(a)anaconda.com> wrote: > > > How will separately distributed plugins work? > > Having that layout makes plugins easy, because the paths are both standardized and relocatable (assume relative paths most of the time). Do you know if our approach to using setuptools entry_points to find plugins will work? This is described at https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discove… <https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discove…>. > > I work for Anaconda, and am also a core member of Conda-forge. If you have any questions or just want to talk about how conda/conda-forge may or may not meet your needs, please feel free to reach out. Thanks a lot on this offer. I may take you up on that as I continue to dig into our options and experiment. > On Jul 23, 2018, at 11:10 PM, James Bennett <ubernostrum(a)gmail.com> wrote: > > On Mon, Jul 23, 2018 at 8:17 PM, Alex Walters <tritium-list(a)sdamon.com <mailto:tritium-list@sdamon.com>> wrote: > As a user of certbot, docker, conda, nix, and guix are non-starters. I'm not depending on those tools for my production server (and while docker may be a dependency for some people, that is hardly universal). Adding heavyweight technical dependencies are problematic if your goal is to get everyone using your software. You're better off with cx_freeze or pyinstaller binaries downloaded from a website or a PPA-like-system to add to system package managers, which are not perfect solutions either. > > I would emphasize this point. Not wanting to install a lot of extra software to use Certbot is certainly fair and we’d obviously prefer our packaging solution to be as lightweight as possible. Thanks for bringing this up as a consideration. With that said, our current approaches aren’t working for us. We’re a small development team and continuing to maintain our custom certbot-auto installer which installs dependencies through your OS package manager and creates a virtual environment tucked away in /opt is a significant drain on our resources. If there’s existing tools or projects which can make this easier for us, we’d like to consider them so we can focus our efforts on Certbot and its features rather than packaging and distribution. Our concerns with cx_freeze/pyinstaller and maintaining our own native packaging repos are listed in the Google Doc. If you have information which may help here, please share! Barring someone suggesting something we haven’t considered, maintaining our own native package repos is one of the three options we’re still considering and we haven’t heard many complaints about certbot-auto (when it works). If we go with something that requires significantly more resources though, we’ll definitely take the fact it does this into consideration and would likely offer a lighter weight (but more manual) installation process for sysadmins who prefer it. > The real question here is what use case you're targeting. Our main use case has been the long tail of individuals or small teams of sysadmins who maintain servers and need or want help and automation around maintaining a secure TLS configuration. Making Certbot available to anyone else who wants to use it is just icing on the cake. > On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > However, there *are* folks that have been working on allowing > applications to be defined primarily as Python projects, and then have > the creation of wrapper native installers be a pushbutton exercise, > rather than requiring careful human handholding. This is definitely our preferred approach to building native packages right now. To be honest, no one on my team has any experience building .debs and .rpms and we’re happy to learn what we need to if we go with this approach, but the more reliable automation around the process we can use the better. > Regardless, given the design constraints and the target audience, it's > unlikely to be possible to avoid offering a two-level solution, where > the lower platform dependent layer sets up any services and cron jobs > required, as well as establishing a Python virtual environment in > /opt, and then the bulk of certbot, as well as any installed plugins, > run in that venv (plugins would still be managed with pip, just > installed into the certbot venv rather than globally). Outside of FPM’s support for packaging a virtual environment which Freddy Rietdijk mentioned, we really hadn’t considered this approach. I think it’s an interesting idea and could work well for us. I gave a couple more thoughts on this approach in my reply to Eli Ribble who suggested the same thing. > For actually building and publishing those platform dependent pieces, > openSUSE's Open Build Service is the most complete offering I'm aware > of: https://openbuildservice.org/ (Fedora's COPR is good for targeting > RPM based distros, and Ubuntu offer their PPA system, but OBS allows > creation of multiple respository formats). Thanks a lot for this and the other pointers in your post Nick. It’s very helpful!

10 12

pip Install for Python 2.7.6
by Ayesha Anwar 26 Jul '18

26 Jul '18

Hi I am Ayesha Anwar student of MS in Remote Sensing and GIS, National University of Sciences and Technology. I am working on my final year project and I am using Python 2.7.6 version on windows 7 (32). I could not use any other python version due to my model requirements. I need to install pip for this python version, I tried both pip files(available on net) to install pip for this version (see attachment) but its not working for this version Kindly send me pip.py file compatible for this version I would be greatfull to you for this kind help. Kind Regards -- Ayesha Anwar MS-15 (RS-GIS) National University of Science and Technology Islamabad, Pakistan Linkedin: Ayesha Anwar https://pk.linkedin.com/in/ayeshaanwar

3 2

Summary of PyPI overhaul in new LWN article
by Sumana Harihareswara 26 Jul '18

26 Jul '18

Today, LWN published my new article "A new package index for Python". https://lwn.net/Articles/751458/ In it, I discuss security, policy, UX and developer experience changes in the 15+ years since PyPI's founding, new features (and deprecated old features) in Warehouse, and future plans. Plus: screenshots! If you aren't already an LWN subscriber, you can use this subscriber link for the next week to read the article despite the LWN paywall. https://lwn.net/SubscriberLink/751458/81b2759e7025d6b9/ This summary should help occasional Python programmers -- and frequent Pythonists who don't follow packaging/distro discussions closely -- understand why a new application is necessary, what's new, what features are going away, and what to expect in the near future. I also hope it catches the attention of downstreams that ought to migrate. -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc

4 9

Export status for setuptools (18.0.1)
by Contact.CCT 26 Jul '18

26 Jul '18

Hi Thanks for maintaining setuptools. Do you know whether it is Not controlled, or if it has an Export control classification number (ECCN) in compliance with US export control laws ? (Bureau of Industry and Security website : http://www.bis.doc.gov/) Thanks for your time. Best regards. THALES Group

2 1

Packaging Advice for EFF's Certbot
by Brad Warren 24 Jul '18

24 Jul '18

Hi! I work at the Electronic Frontier Foundation on Certbot which is the most popular end user application for obtaining and installing SSL/TLS certificates from Let’s Encrypt. Over the past few years, distributing Certbot has been one of our development team's biggest challenges and we’re currently rethinking how we do so. It was suggested to me that I post to this list to see if anyone was interested in offering advice for how we should approach this. Of course, Certbot is written entirely in Python. If you’re interested, I wrote up a bit of background and what we’re currently thinking at https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7A… <https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7A…>. Feel free to reach out to me on or off list or on IRC at bmw on Freenode. Thanks so much for any help. Best, Brad Warren Senior Staff Technologist Electronic Frontier Foundation

10 9

Re: Make an ordered list of sdists to be installed?
by Tzu-ping Chung 23 Jul '18

23 Jul '18

I just happened to be looking into this area, and may be able to offer some help. Some background information (optional reading): https://github.com/pypa/pipenv/issues/2596#issuecomment-405656806 Sébastien (@sdispater) is the author of Poetry, including the resolver, called Mixology. As he mentioned in the issue, he recently rewrote the resolver, and had yet to decide when and how to extract if from other inner workings of Poetry. The old version of Mixology (before the rewrite), however, is still available on PyPI, and is good enough for this particular use with some additional work. I made a quick, very inefficient implementation, available as a GitHub Gist: https://gist.github.com/uranusjr/a7a9f20c6e43810bd19f0c73e9617182 Hopefully this would help with your need 😊 TP From: distutils-sig-request(a)python.org Sent: 24 July 2018 00:00 To: distutils-sig(a)python.org I don’t know the details, but I did read that Poetry has a sophisticated dependency resolver. https://github.com/sdispater/poetry I don’t know if there is a way to access the resolver independently of the tool, but perhaps it would provide a handy reference. 2018년 7월 23일 (월) 오전 5:49, Thomas Kluyver <thomas(a)kluyver.me.uk>님이 작성: Hi all, Do we know of any tool that can, given the name of one or more packages, follow dependency chains and produce a list of packages in the order they need to be installed, assuming every package needed will be built from source? Running "pip download --no-binary :all: ipython" gets me a set of sdists to be installed, but I lose any information about the order. I assume some packages will fail to build if their dependencies are not installed first, so the order is significant. Pip appears to keep track of the ordering internally: if I run "pip install --no-binary :all: ipython", all the dependencies are downloaded, and then the collected packages are installed starting from those with no dependencies and finishing with the package I requested. But I don't know of any way to get this information out of pip. Is there an option that I'm overlooking? Or some other tool that can do this? The use case I'm thinking about is to automatically generate instructions for a build system which separates the downloading and installing steps, so for each step it expects one or more URLs to download, along with instructions for how to install that piece. The installation steps shouldn't download further data. I could work around the issue by telling it to download all the sdists in a single step and then install in one shot with --no-index and --find-links. But it's more elegant - and better for caching - if we can install each package as a single step. Thanks, Thomas -- Distutils-SIG mailing list -- distutils-sig(a)python.org To unsubscribe send an email to distutils-sig-leave(a)python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…

1 0

Make an ordered list of sdists to be installed?
by Thomas Kluyver 23 Jul '18

23 Jul '18

Hi all, Do we know of any tool that can, given the name of one or more packages, follow dependency chains and produce a list of packages in the order they need to be installed, assuming every package needed will be built from source? Running "pip download --no-binary :all: ipython" gets me a set of sdists to be installed, but I lose any information about the order. I assume some packages will fail to build if their dependencies are not installed first, so the order is significant. Pip appears to keep track of the ordering internally: if I run "pip install --no-binary :all: ipython", all the dependencies are downloaded, and then the collected packages are installed starting from those with no dependencies and finishing with the package I requested. But I don't know of any way to get this information out of pip. Is there an option that I'm overlooking? Or some other tool that can do this? The use case I'm thinking about is to automatically generate instructions for a build system which separates the downloading and installing steps, so for each step it expects one or more URLs to download, along with instructions for how to install that piece. The installation steps shouldn't download further data. I could work around the issue by telling it to download all the sdists in a single step and then install in one shot with --no-index and --find-links. But it's more elegant - and better for caching - if we can install each package as a single step. Thanks, Thomas

4 3

pip 18.0 has been released!
by Pradyun Gedam 22 Jul '18

22 Jul '18

On behalf of the PyPA, I am pleased to announce that pip 18.0 has just been released. To install pip 18.0, you can run python -m pip install --upgrade pip or use get-pip as described in https://pip.pypa.io/en/latest/installing. Note that if you are using a version of pip supplied by your distribution vendor, vendor-supplied upgrades will be available in due course (or you can use pip 18 in a virtual environment). This is the first pip release since adopting 3 month release cadence and a Calendar based versioning scheme (also known as CalVer). In simpler words, there will be a new pip release every 3 months unless there are no changes since the previous release. More details such as release months can be found in pip's development documentation. The highlights of this release are: - Python 3.3 is no longer supported - if you need pip on Python 3.3, you should stay on pip 10, which is the last version to support Python 3.3. - Complete PEP 518 support - includes support for installation of build dependencies from source and Unicode support on Python 2 and Windows - New --prefer-binary flag, to prefer older wheels over newer sdists - Many bug fixes and minor improvements Thanks to everyone who put so much effort into the new release. Many of the contributions came from community members, whether in the form of code, participation in design discussions and/or bug reports. The pip development team is extremely grateful to everyone in the community for their contributions. Thanks, Pradyun

4 4

[Distutils]PEP 518 - pyproject.toml with no build-system.requires
by Pradyun Gedam 21 Jul '18

21 Jul '18

Hey everyone! In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table). In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier). There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior. Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case. I can think of at least 2 options for behavior when build-system.requires is missing: 1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"]. 2. Making the build-system table mandatory in pyproject.toml. I personally think (2) would be fine -- "Explicit is better than implicit." It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though. Thoughts on this? Cheers, Pradyun

10 69

[Distutils]Need help regarding Python and DB2 connectivity
by Shivam Satija 21 Jul '18

21 Jul '18

Hi, I am trying to connect IBM DB2 with python. I am using Python 3.6.0 and want to know if the DB2 connectivity is even possible with this version of python? I have read at various places that only versions 2.x and 3.4 are supported. Is this the case? I am not able to run basic commands like import ibm_db on python 3.6. Thanks, Shivam

3 2

[Distutils]Approaching wheel 1.0
by alex.gronholm＠nextday.fi 20 Jul '18

20 Jul '18

Today I pushed 29 new commits to the pypa/wheel repository, closing around 15 issues. This is a major overhaul/refactoring of the wheel project. Here's the short list of major changes: * The documentation has been rewritten to conform to PyPA guidelines (h ttps://gist.github.com/qwcode/8431828) * Wheel signing and verifying features were stripped out, as they were scarcely used and would have potentially presented a significant maintenance burden * The wheel install feature was also stripped out, largely from the same reason as above * Added a new command, "wheel pack" that will create a new wheel file from a previously unpacked wheel directory, regenerating the RECORD file in the process * Added a new metadata key, "license_files" which can be used to include multiple license files in wheels (this deprecates the "license_file" key) There are still a few issues to work out before I'm comfortable releasing 1.0.0 final. For that, I would like to enlist the help of the Python packaging community, as I may not have the necessary expertise: * Py_UNICODE_SIZE / WITH_PYMALLOC / Py_DEBUG on Windows: https://github .com/pypa/wheel/issues/171 * Supporting tools like milksnake by allowing platform specific wheel tags with ABI set to "none": * https://github.com/pypa/wheel/issues/125 * https://github.com/pypa/wheel/issues/153 * https://github.com/pypa/wheel/issues/175 * https://github.com/pypa/wheel/issues/185 * Automatically detecting C extensions that use the limited ABI and tagging wheels without an explicit --py-limited-abi: https://github.com /pypa/wheel/issues/198 * bdist_wheel not cleaning build directory (really a distutils problem, but can wheel reasonably do something here?) https://github.com/pypa/wh eel/issues/117 I mostly need help defining the concrete steps to take to resolve these issues.

2 1

[Distutils]Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers
by Cooper Ry Lees 19 Jul '18

19 Jul '18

Hi distutils, Today Facebook Open Sourced a competitor to PEX and other zip based distribution methods for Python (and potentially other languages). Basically it's claim to fame is start up time for large modules being similar to regular on file system modules due to extracting on read via SquashFS mounted executables. For more information read our blog post: - https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-s… <https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-s…> The bdist_xar 'wheel' like plugin is also now in PyPI: https://pypi.org/project/xar/ <https://pypi.org/project/xar/> (https://github.com/facebookincubator/xar/ <https://github.com/facebookincubator/xar/>). Main reason I wanted to post here was I'd love to reach out to our OS maintainers on list (e.g. Mr Stinner) and talk about getting the components of XAR into the OS's package repos. The main components that make sense are: - xarexec_fuse: https://github.com/facebookincubator/xar/ <https://github.com/facebookincubator/xar/> - squashfuse: a newer version with squashfuse_ll (and optionally zstd support) https://github.com/vasi/squashfuse <https://github.com/vasi/squashfuse> - squashfs-tools: a newer version with zstd support (made optional I guess) Once we had this, it would make building XARs as easy as building wheels, even more so when we can define OS level dependencies for it! Please feel free to reach out to me directly with any questions etc. - Also want to note, Sumanah suggested I floated this here, I apologize if this is a misuse of the list. If so, please ignore. I was torn wether or not this is appropriate. Thanks, Cooper

6 9

[Distutils]how can I install packages of python 3.7 on window 10
by HIMANSHU SATYWALI 19 Jul '18

19 Jul '18

Sent from Mail for Windows 10ws 10

1 0

[Distutils]Re: The Wheel specification and compatibility tags on Windows
by Cosimo Lupo 17 Jul '18

17 Jul '18

I would like to revive this 5 year old thread and see if we can stir things up a bit. Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel that includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the list of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish. Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore: > 2. How should tools determine which ABIs a given Python supports? > (This is the get_supported function in wheel and distlib). The "Use" > section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) > gives a Linux-based example, but nothing normative and nothing that is > understandable to a Windows user. And from Vinay Sajip > For Windows, we (eventually) need some low-level sysconfig-supported way > to get the ABI information in an analogous way to how it happens on POSIX: and > because that's not currently there, distlib doesn't provide any ABI information > on Windows other than "none". Other related links: https://github.com/pypa/pip/issues/4445 https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support? Thank you, Cosimo Lupo -- Cosimo Lupo

6 8

[Distutils]Package maintainers: verify your PyPI account email address
by Sumana Harihareswara 17 Jul '18

17 Jul '18

Heads-up: if you have a pypi.org or test.pypi.org account, please verify your email address: https://pypi.org/manage/account/ https://test.pypi.org/manage/account/ Reason: > We have a problem with a bit of our data, namely that due to historical reasons we have a fair amount of users in the database that do not have a verified primary email address. The side effect of this is that we're currently sending emails to email addresses that we have not had verified. This is a bad situation to be in, because in order to keep our bounce/spam rate low, we should be confirming all email addresses before sending email to them. In addition the way our bounce handling code works is it un-verifies the email address, which the intent was to stop sending email to it until the user has reverified their email address. > > In total there are about 193k user accounts with a unverified email address for their primary address, and 44k that do have a verified email address for their primary account. > > So we need to come up with a strategy to resolve this, because it's pretty important that we don't send email to unverified addresses. (quoting from Donald Stufft's explanation https://github.com/pypa/warehouse/issues/3632 which goes on to detail the step-by-step plan) Package maintainers should especially do this soon. As of a few days ago, any user whose primary email address is unverified can't upload a file: https://github.com/pypa/warehouse/pull/4292 Please forward to other package maintainers. After we iron out any issues I figure we'll email the announce list https://mail.python.org/mm3/mailman3/lists/pypi-announce.python.org/ . -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

[Distutils](no subject)
by phung237337＠gmail.com 16 Jul '18

16 Jul '18

Được gửi từ Yahoo Mail trên Android

1 0

[Distutils]ssl error in installing packages
by Nitin Thakur 16 Jul '18

16 Jul '18

SSL-CERTIFICATE-VERIFICATION FAILED while installing packages using pip command. Sent from Mail for Windows 10

3 2

[Distutils]pypi/twine complains about license
by Robin Becker 13 Jul '18

13 Jul '18

After release of Python-3.7 I wanted to upload to pypi a newly built version of a C-extension which already has been migrated to the new site. $ twine --version twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.18.1, setuptools: 36.2.0, requests-toolbelt: 0.8.0, tqdm: 4.14.0) $ twine upload *.whl Uploading distributions to https://upload.pypi.org/legacy/ Uploading pyRXP-2.1.1-cp37-cp37m-manylinux1_i686.whl 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 104K/104K [00:00<00:00, 141KB/s] HTTPError: 400 Client Error: Invalid value for classifiers. Error: 'License :: OSI Approved :: ReportLab BSD derived' is not a valid choice for this field for url: https://upload.pypi.org/legacy/ 1) I think it is completely wrong for twine/pypi to fail to upload because of the license field. The license is derived from BSD and the same string is present in the previously uploaded versions of this package. What are valid licenses? Presumably pypi is now a gatekeeper for the license police. 2) I looked in vain on the new pypi.org site for a manual upload mechanism. Is this now frowned on? 3) I was able to upload the same package several times without error; does this mean I am overwriting the file? -- Robin Becker

12 25

[Distutils]Provisional PEPs can now actually be marked as Provisional
by Nick Coghlan 08 Jul '18

08 Jul '18

Hi folks, Some time ago, we adjusted the distutils-sig PEP approval process to include a "Provisional" status, where we approved changes for inclusion in the reference implementations (typically pip and setuptools), but still reserved the right to make adjustments if practical experience showed that we'd missed something critical. Earlier today, I merged the amendment to PEP 1 that makes that part of the Python Enhancement Proposal process in general, and explicitly marked the affected PEPs as Provisional. For distutils-sig, that update affects both of the pyproject.toml PEPs (although I expect PEP 518 isn't too far off being marked as Final - the question of the precise meaning of a missing build-system table when pyproject.toml is present seems to be the only underspecified point that anyone has found during the Provisional period). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 2

[Distutils]Handing over default BDFL-Delegate responsibilities for packaging interoperability PEPs to Paul Moore
by Nick Coghlan 08 Jul '18

08 Jul '18

Hi folks, Since 2013, I've been the default BDFL-Delegate for packaging interoperability PEPs. In that time, the Python packaging ecosystem has moved forward in a lot of different areas, with pip being shipped by default with CPython, the wheel binary packaging format reaching ever-increasing heights of popularity, the cross-distro manylinux ABI compatibility specification being developed, the new pyproject.toml based sdist format being defined, the PSF's Packaging Working Group being formed, the Python Packaging User Guide being developed, and various aspects of the packaging metadata being enhanced to improve the general user experience of the Python packaging ecosystem. The role of the BDFL-Delegate in that process is partly about making arbitrary decisions when arbitrary decisions need to be made ("The bikeshed shall be green!"), but also about helping to guide discussions in productive directions, as well as determining when more complex PEP level proposals have reached a sufficient level of consensus that it makes sense to provisionally accept them and move on to publishing reference implementations. While it's been a fascinating ~5 years, I've decided that it's time for me to hand over those responsibilities to another PyPA contributor. With Guido's approval, I've asked Paul Moore if he'd be willing to take on the role, and Paul has graciously accepted the additional responsibility. Paul's a long term pip contributor, and also a CPython core developer, with a lot of practical experience in getting Python (and Python packaging) to work well in Windows environments. He's also a familiar, calm, and constructive presence in design discussions within distutils-sig, pip and other PyPA projects, which is an important characteristic when taking on BDFL-Delegate responsibilities. I'd like to personally thank Paul for being willing to take on this task, and I look forward to many more productive design discussions! Cheers, Nick. P.S. I'm not stepping down from Python packaging related activities entirely, as I'll still be involved in Python Packaging User Guide and pipenv maintenance, and will continue as a member of the PSF's Packaging Working Group. However, the final sign-off for packaging interoperability PEPs will now rest with Paul or someone else that he appoints, rather than with me :) -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 3

[Distutils][ANN] Python 3.7 now available in the manylinux docker images
by Nathaniel Smith 25 Jun '18

25 Jun '18

Hi all, Thanks to Emanuele Gaifas [1], the manylinux docker images now include Python 3.7 (currently the rc1 release), so if you want you can now start building and uploading wheels for 3.7 ahead of its expected release on June 27 [2]. -n [1] https://github.com/pypa/manylinux/pull/196 [2] https://www.python.org/dev/peps/pep-0537/ -- Nathaniel J. Smith -- https://vorpus.org

4 5

[Distutils]Possible change to PEP 517: look up the backend as $BACKEND.__build_backend__?
by Nathaniel Smith 24 Jun '18

24 Jun '18

Hi all, I had a thought for something that might be a simple way to improve dev experience with custom build backends. A PEP 517 build backend is a Python object that has some special methods on it. And the way a project picks which object to use, is via pyproject.toml: [build-system] build-backend = "module1.module2:object" Currently, this means that the build backend is the Python object: module1.module2.object Here's my idea: what if change it, so that the above config is interpreted as meaning that the build backend is the Python object: module1.module2.object.__build_backend__ (I.e., we tack a "__build_backend__" on the end before looking it up.) Why does this matter? Well, with the current system, if you want to use flit [1] as your build backend, you have to write: build-backend = "flit.buildapi" And if you want to use intreehooks [2],you have to writ: build-backend = "intreehooks:loader" These names are slightly awkward, because these projects don't want to just jam all the PEP 517 methods directly onto the top-level module object, so they each have to invent some ad hoc sub-object to put the methods on. And then that's exposed to all their users as a bit of random cruft you have to copy-paste. The idea of __build_backend__ is that these projects could rename the 'buildapi' and 'loader' objects to be '__build_backend__' instead, and then users could write: build-backend = "flit" build-backend = "intreehooks" build-backend = "setuptools" and it just feels nicer. Right now PEP 517 is still marked provisional, and pip hasn't shipped support yet, so I think changing this is still pretty easy. (It would mean a small amount of work for projects like flit that have already implemented backends.) What do you think? (Thomas, I'd love your thoughts in particular :-).) -n [1] https://github.com/takluyver/flit/ [2] https://github.com/takluyver/intreehooks -- Nathaniel J. Smith -- https://vorpus.org

3 5

Dependency resolution in pip
by alex.gronholm＠nextday.fi 14 Jun '18

14 Jun '18

I recently stumbled into a worrying problem with pip. I found out that doing "pip install pusher requests" installs urllib3 v1.23 as a dependency even though requests specifically restricts the version to lower than 1.23. Then if instead I do "pip install requests pusher" it installs urllib3 v1.22 as expected. As I recall, pip has long had a problem with combining version specifiers and extras when the same target has been required from multiple sources. What I wanted to ask was, is this a simple bug, or a larger unresolved design problem? Should pip also take into consideration the requirements from existing installed packages so pip won't end up installing upgrades they're incompatible with?

3 2

Custom installation commands in setup.py
by Vinay Sajip 14 Jun '18

14 Jun '18

Are custom installation commands in setup.py no longer respected by setuptools? For example, the pybind11 project has a custom InstallHeaders command class in its setup.py, which is passed to the setup() call. When setup is imported from setuptools, the custom command class never gets invoked. When setup is imported from distutils.core, the custom command class is invoked. What's the reason for the disparity - can someone please enlighten me? Regards, Vinay Sajip

1 0

PyPI not showing latest version?
by Vinay Sajip 13 Jun '18

13 Jun '18

I just uploaded python-gnupg 0.4.3 to PyPI using Twine. Search still shows the previous version: https://pypi.org/search/?q=python-gnupg => 0.4.2 However, clicking on the link brings up the page for the latest version: https://pypi.org/project/python-gnupg/ => 0.4.3 But pip install is also wrongly picking up 0.4.2. What's the expected delay between uploading a new version and having it be available via pip? I would have expected more or less immediately. All systems are showing as operational. Regards, Vinay Sajip

4 4

Discussion/direction for platform/abi options being added to pip's "install" and "wheel" subcommands
by Loren Carvalho 08 Jun '18

08 Jun '18

Hello distutils-sig! Apologies if this is the wrong mailing list, Sumanah suggested I write to you fine folks to get some feedback/direction on my pip PR. Here's the link: https://github.com/pypa/pip/pull/5404 There's a bit of discussion in that PR thread, but also in a couple of Issue threads, most notably https://github.com/pypa/pip/issues/5355 (where I first proposed adding the dist options to install) and https://github.com/pypa/pip/issues/5453 (where some other folks seem to be requesting similar functionality). Granted there's a lot to ingest, but the discussion in the PR has seemed to stagnate and I desperately would like some direction on how to proceed... or IF to proceed, if it's ultimately decided that this is not something pip maintainers want to do, I totally respect that. I summarize the status quo in my latest comment on the PR, but to reiterate here: my PR adds `--platform`, `--abi`, `--python-version` and `--implementation` as valid arguments to `install` (formerly these options were only available on `download`). They are only usable when invoking `install` with `--target`, per pfmoore's suggestion. Some other folks have suggested that this limitation (the `--target` one) is unnecessary and that merely using the dist options alone is plenty-explicit about intent. The same folks also would like to see the dist options on the `wheel` subcommand. I personally have no qualms with their suggestion NOR do i have qualms with pfmoore's suggestion, so I am looking for guidance. Thanks so much in advance, I appreciate your time. -- Loren <3

2 1

EuroPython financial aid deadline: 5 June
by Sumana Harihareswara 03 Jun '18

03 Jun '18

EuroPython 2018 will be in Edinburgh, Scotland, UK, from July 23-29, and will include packaging/distribution sprints: https://wiki.python.org/psf/PackagingSprints and a devpi help desk, among other sessions: https://ep2018.europython.eu/en/events/sessions/ Financial aid is available, for free tickets, travel costs, and accommodation. Apply by 5 June: https://ep2018.europython.eu/en/registration/financial-aid/ (I'm not involved in organizing EuroPython -- just signal-boosting.) -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

sudo pip install: install pip files into /usr/local on Linux?
by Victor Stinner 01 Jun '18

01 Jun '18

Hi, pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified. I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed. Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip". Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user). Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local? I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues! Issues with the current Fedora implementation: (1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path: https://bugzilla.redhat.com/show_bug.cgi?id=1532287 (2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr. https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24 The implementation is made of a single patch on the Python site module: https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-user-… -- There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local: (3) Priority issue between PATH and PYTHONPATH directories. When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used? Obvisouly, users can replace these two environment variables... (4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version. For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local. Fedora works around issue (4) with a downstream patch on pip: https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow-pip1… -- I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution. Victor

9 23

Excluding specific Python modules from wheels
by Michal Petrucha 30 May '18

30 May '18

Hello everyone, I've been looking for a way to ensure that certain modules don't end up in a wheel, while the rest of the package they reside in does. If I only cared about sdist, I could add a MANIFEST.in, in which I'd exclude those specific files, however, unfortunately, MANIFEST.in has no effect on bdists (at least of the wheel kind). The use case is that our application auto-generates a parser and lexer with ply, and that parser might not work with different versions of ply. Since we don't have a whole lot of control over what version users have installed in their environments, we'd like to generate those modules in the target environment. I took a deep dive into distutils and setuptools, and as far as I can see, any Python modules residing inside a package listed in the packages argument to setup() are included in the distribution unconditionally. Searching this mailing list only reveals a short thread from nine years ago [1] without any solution... For now, the easiest hacky solution for me is to add a couple of os.remove calls to setup.py, but I'm not a big fan of setup.py messing with the source tree. And as a follow-up question, is there any post-installation hook that we could use to trigger regeneration of those files? Any ideas? Michal [1]: https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/…

3 2

appveyor updateb broken by changes to pip
by Robin Becker 18 May '18

18 May '18

Not exactly sure if this is the issue, but my appveyor test script contains testenv\scripts\pip.exe install --cache-dir=dist --find-links file:dist reportlab where in turn reportlab setup.py contained install_requires=['pillow>=2.4.0']. After an upgrade sometime between Apr 26 and May 18 this started to fail because pip could no longer find a pillow>=2.4. > "=============================== Python27 TEST START" > 941c:\Python%PYTHON%\python.exe -mvirtualenv testenv > 942New python executable in C:\projects\reportlab\testenv\Scripts\python.exe > 943Installing setuptools, pip, wheel...done. > 944testenv\scripts\pip.exe install --cache-dir=dist --find-links file:dist reportlab > 945Looking in links: file:dist > 946Collecting reportlab > 947Collecting pillow>=2.4.0 (from reportlab) > 948 Could not find a version that satisfies the requirement pillow>=2.4.0 (from reportlab) (from versions: ) > 949No matching distribution found for pillow>=2.4.0 (from reportlab) > 950Command exited with code 1 > 951myenv\scripts\python.exe -mrl_ci_tools email --subject="!!!!! APPVEYOR %APPVEYOR_PROJECT_NAME% build failed" --faddr=rl-wheelbuilder --body="see https://ci.appveyor.com/project/MrRLBitBucket/%PROJECT_NAME%/history" I had assumed that pip would fail to find pillow in the dist folder and then move on to somewhere more useful eg pypi, but with pip 10.0.1 this seems not to happen. It's easy to fix with a separate install of the, but what is the correct way to indicate dependencies which will allow installation from my freshly built wheel? -- Robin Becker

1 0

Manual PIP upgrade for TLS 1.2 support fails.
by edlipsongm＠gmail.com 17 May '18

17 May '18

I'm trying to install python 3.5.5 on RHEL without root. I used the referenced --prefix to identify where the install is (per webpage). This worked in 3.5.4. PIP installs, but fails to install other modules. Searching for a solution I found a reference to an external way to upgrade PIP to a version where this problem is supposed to be resolved. I obtainde get_pip.py and ran it, but it too fails (below) for TLS issues. Policy says I have to obscure anything related to my company, hence the paths have been edited in this email. Are there any other solutions? bash-3.2$ which python <installpathObscuredbyPolicy>/Python-3.5.5/python -bash-3.2$ python <homepathObscuredbyPolicy >/python/get-pip.py Collecting pip /tmp/tmp9qehrweu/pip.zip/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)': /simple/pip/ Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)) - skipping Could not find a version that satisfies the requirement pip (from versions: ) No matching distribution found for pip Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:719)'),)) - skipping

1 0

requirements.txt or not requirements.txt?
by Carles Sala Cladellas 15 May '18

15 May '18

Hello here! TL;DR: Should I use requirements.txt, or should I have my dependencies only listed inside setup.py? Long version: The two most common ways to have dependencies listed in a python package are either having them listed both in setup.py and requirements.txt files, or having them only listed in requirements.txt and then import them using some `parse_requirements` methods. In the projects which I am currently working on there are five lists of requirements: - requirements.txt - requirements_test.txt - requirements_dev.txt - install_requires inside setup.py - tests_require inside setup.py I wanted to remove duplicities, so I thought about removing the setup.py lists and parse the dependencies from the requirements*.txt files. But then I came across this, https://stackoverflow.com/a/28842733, which recommends using a `dev` entry in `extras_require` and then using `pip install -e .[dev]` to install them, which could be easily extended to also have a `pip install -e .[test]`. I find this a very clean option, since it reduces the amount of files the repository and makes setup.py and makes setup.py self-contained. Are any known drawbacks to this approach? Is it a good way to go? Regards, -- Carles Sala Cladellas carles(a)pythiac.com / +34 666 278 679 https://www.pythiac.com Pythia Consulting - Data Science as a Service

7 7

Python 3.7 binary wheels
by Lele Gaifax 11 May '18

11 May '18

Hi all, Python 3.7 is steadily approaching its final release and I start wondering a) when will be the right time to start uploading 3.7 binary wheels on PyPI? b) if/when manylinux2010 happens, does that mean that I should build both manylinux1 and manylinux2010 variants? Thank you in advance, ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele(a)metapensiero.it | -- Fortunato Depero, 1929.

4 6

Build a module of SWIG interfaces in a single invocation
by Clifford Simon 11 May '18

11 May '18

Dear Distutils SIG, I am asking about the best way to build a module, when the sources are SWIG interface files (*.i). SWIG is supposed to generate the Python scripts. I can build the module by running build_ext and then build_py, but I cannot build the module with build because it runs build_py and build_ext in the wrong order, since the Python scripts don't exist until build_ext has run. I have asked this on StackOverflow, here: https://stackoverflow.com/questions/50239473/building-a-module-with-setupto…, but my question attracted very little interest there. Thanks for the help, Clifford Simon

1 0

This list will soon be migrating to Mailman 3
by Nick Coghlan 10 May '18

10 May '18

Hi folks, The recent discussion of communication workflows prompted me to investigate exactly what would be involved in migrating the list to Mailman 3 (with its native web gateway and other account management improvements), and I'm happy to report that there don't appear to be any blockers to our initiating the migration. I'm currently planning to send the migration request to postmaster(a)python.org later this week (probably Thursday evening), but don't have an ETA for how long the actual migration may take after that (as I believe distutils-sig will be one of the larger python.org list migrations undertaken so far). Accordingly, this is an initial heads up as to exactly what the change will mean: 1. If you only ever access the mailing list via email, and never tinker with your subscription settings or look up threads in the list archives, then you shouldn't notice any real changes other than some of the headers on mails from the list changing a bit (such as the new Archived-At header appearing on each message). 2. If you do use the website to change your subscription settings or look up threads in the list archives, or have wished you had a more web-forum-like interface for accessing the list, then the rest of this email is likely to be of interest :) == Changes to subscription management (and list moderation) == Mailman 3 relies on a more conventional user account management model than the historically list-centric model in Mailman 2. This means that either before or after the migration, folks that want to modify their subscription settings will need to go to https://mail.python.org/mm3/ and register for an account. If you use GitLab, GitHub, Google, or Facebook, then you can go straight to https://mail.python.org/mm3/accounts/login/, select one of those options, and grant the required access to look up your email address. (At least for GitHub, the request will come from Mark Sapiro's developer key, and I believe that's the case for the other services as well) If your address on the linked service matches your subscription address, then you're done. If it doesn't match, then you can head to https://mail.python.org/mm3/accounts/email/ to register more addresses (and hence link any related subscriptions to your account). If you don't use any of those services, or simply don't want to use them with mail.python.org, then head to https://mail.python.org/mm3/accounts/signup/ to create a conventional username-and-password based account. Regardless of how you sign up, the primary authentication mechanism is access to the relevant email address - the old MM2 plain text email password isn't used at all. After the migration, the current https://mail.python.org/mailman/listinfo/distutils-sig URL will automatically redirect to https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ For a working example of what that will look like, see https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ == Changes to list archiving (and the native web gateway) == After the migration, the current list archive at https://mail.python.org/pipermail/distutils-sig/ will remain in place in order to preserve existing links, but will no longer be updated with new messages. That page will also be updated with a link to the new archiver/web gateway page. That page will be at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/, and not only features stable and predictable URLs for each post, but also includes a native web gateway, allowing folks to both create new threads and reply to existing ones using the web page, without needing to explicitly subscribe to the list first. Again, core-workflow provides an example of what that will look like in practice, with the new archive at https://mail.python.org/mm3/archives/list/core-workflow@python.org/, and the post-migration legacy archive at https://mail.python.org/pipermail/core-workflow/. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 19

org-mode README file formats
by Diane Trout 09 May '18

09 May '18

Hi, I was building a package where I had a README.org file, which setuptools couldn't find. It listed .md as a valid format, so I was wondering if org-mode was sufficiently plain text to be added to the list of accepted README file formats? For what it's worth, github-markup can handle .org files https://github.com/github/markup Diane

6 9

Packaging/Warehouse sprint at PyCon 2018
by Sumana Harihareswara 08 May '18

08 May '18

In case you're planning your PyCon Cleveland travel: we are planning to hold a Warehouse/packaging sprint at PyCon (the sprints are Monday, May 14th - Thursday, May 17th 2018). We welcome package maintainers, backend and frontend web developers, infrastructure administrators, technical writers, and testers to help us make the new PyPI, and the packaging ecosystem more generally, as usable and robust as possible. I took the liberty of updating https://us.pycon.org/2018/community/sprints/ to say so. Once we're closer to the sprints I'll work on a more detailed list of things we'll work on in Cleveland. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 4

(Final) PyPI/Warehouse weekly report: legacy is shut down
by Sumana Harihareswara 02 May '18

02 May '18

As I announced yesterday[1], here and on the pypi-announce[2] and general Python announcement[3] lists, we have shut down legacy.pypi.org, on schedule. (See the notes from our final weekly call[4], a screenshot of all our closed milestones[5], a screenshot of the hit ratio for Legacy going to 0%[6], and Ernest pouring out a toast to the old codebase[7].) This is the last weekly report you'll get from me on this project, as the MOSS funding has nearly run out (we set aside a little for me to run the PyCon sprint and for Nicole to run the EuroPython sprint). Thanks so much to Mozilla's Open Source Support program for the award[8] that enabled this work[9]. And thanks to the PSF and its Packaging Working Group[10] for facilitating it. Highlights from the last week: The podcast Talk Python To Me released an episode interviewing Dustin Ingram, Nicole Harris, and Ernest W. Durbin III about Warehouse -- you can listen[11] or read the transcript[12]. And the Python Bytes podcast had a short chat about Warehouse[13] as well. Ernest sunset Legacy[14], fixed a subsequent outage[15] (my fault for putting a hostname in the title of a blog post!), updated a cabotage setting[16], updated CDN configuration[17], and fixed another service disruption[18]. And he improved search for XML-RPC endpoint users[19]. Since we got 1700+ responses to the "buy a feature" survey[20], we took down the banner[21] -- Nicole notes that the data is really useful and will really help with redesigning the project detail page! She also fixed modal alignment[22] and table alignment[23] in IE11. Dustin replaced our Twisted usage with gunicorn[24] and fixed an edge case concerning identical canonical versions of a release[25], and Dustin and Ernest made old pypi.python.org links for files, display actions[26], search and browse actions[27] redirect appropriately. And Dustin merged "Support XML-RPC multicall"[28] and then "Skip tweens for XML-RPC multicall subrequests"[29] then "Deprecate XML-RPC MultiCall"[30] and I think we've all had sequences like that in our lives. Laura Hampton and I ran a Warehouse sprint night[31] in New York City (giving participants several tasks at varying difficulty levels[32]), where Corey Girard helped us make profile pages display "you" versus a username more logically[33] -- thanks, Corey! -- and Kshitij Chawla found a setup issue[34]. And the team found some more developer experience snags and got to fixing them: PyPUG instructions[35], the README[36], Docker instructions[37], discoverability for the architecture overview[38]. We are slowing down a bit on pull request review and issue response as our dedicated time on Warehouse comes to a close, but we still did a lot of review and replying. Thanks to the volunteers who got pull requests merged in the past week: * nixjdm, who added description_content_type to the JSON API[39] * cheungnj, who improved how we display the "last released" date on a project[40] * aalmazan, who fixed how we handle tab cycling inside active modals[41] * alex, who fixed a pytest argument[42] * kpayson64, who updated wheel types Warehouse supports[43] (see the followup conversation, on whether PyPI should allow Linux wheel uploads for ARM[44]) Special shoutout to GitHub user jdufresne[45] who has submitted a bunch of pull requests to various projects, including setuptools[46], updating their URLs from pypi.python.org to pypi.org (example[47]). I've done some similar issue-opening (example[48]). And thanks to Donald Stufft for helping with the infrastructure changeover[49]! You can help by: * updating the distutils docs[50] to reflect how PyPI currently works * giving yeraydiazdiaz feedback on this approach to automated frontend testing[51] * keeping an eye on Warehouse pull requests and reviewing[52] them * telling hiring managers you know to consider hiring Ernest[53] and giving him paid time to work on PyPI * finding us at PyCon North America[54] and giving us friendly feedback Dustin, Ernest, Laura, Nicole and I will continue volunteering a few hours per week around here, just as many of us did before the project. We're all grateful we got to work together and make this happen, and hope to have further paid opportunities to dedicate time to this infrastructure and its symbiotic community. -- Sumana Harihareswara (basically my last note as) Warehouse/PyPI project manager PyPA member Packaging Working Group member Changeset Consulting -- open to new client engagements starting in June/Julysh(a)changeset.nyc Links: 1. https://mail.python.org/mm3/archives/list/pypi-announce@python.org/thread/2… 2. https://mail.python.org/mm3/archives/list/pypi-announce@python.org/thread/2… 3. https://mail.python.org/pipermail/python-announce-list/2018-April/011916.ht… 4. https://wiki.python.org/psf/PackagingWG/2018-04-30-Warehouse 5. https://twitter.com/di_codes/status/990982434572439554 6. https://twitter.com/EWDurbin/status/990959524768112640 7. https://twitter.com/EWDurbin/status/990966616803422208 8. https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html 9. https://blog.mozilla.org/blog/2018/01/23/moss-q4-supporting-python-ecosyste… 10. https://wiki.python.org/psf/PackagingWG 11. https://talkpython.fm/episodes/show/159/inside-the-new-pypi-launch 12. https://talkpython.fm/episodes/transcript/159/inside-the-new-pypi-launch 13. https://pythonbytes.fm/episodes/show/75/pypi.org-officially-launches 14. https://status.python.org/incidents/ptvp1wnn0jmq 15. https://status.python.org/incidents/ttdz2bj310xw 16. https://github.com/cabotage/cabotage-app/commit/6e7eafce737726a04bd6ce08968… 17. https://status.python.org/incidents/w9zkk6k75xrc 18. https://status.python.org/incidents/0gyxs265qp6b 19. https://github.com/pypa/warehouse/pull/3827 20. https://twitter.com/nlhkabu/status/988856279526465537 21. https://github.com/pypa/warehouse/pull/3854 22. https://github.com/pypa/warehouse/pull/3841 23. https://github.com/pypa/warehouse/pull/3839 24. https://github.com/pypa/warehouse/pull/3865 25. https://github.com/pypa/warehouse/pull/3822 26. https://github.com/pypa/warehouse/pull/3856 27. https://github.com/pypa/warehouse/pull/3800 28. https://github.com/pypa/warehouse/pull/3778/files 29. https://github.com/pypa/warehouse/pull/3811/files 30. https://github.com/pypa/warehouse/pull/3821/files 31. https://www.meetup.com/nycpython/events/249722192/ 32. https://docs.google.com/presentation/d/1JHTvs58DaHJzha8aN8oz_IYaiyZlupgioAB… 33. https://github.com/pypa/warehouse/pull/3837 34. https://github.com/pypa/warehouse/issues/3836 35. https://github.com/pypa/python-packaging-user-guide/pull/471 36. https://github.com/pypa/warehouse/pull/3863 37. https://github.com/pypa/warehouse/pull/3871 38. https://github.com/pypa/warehouse/pull/3864 39. https://github.com/pypa/warehouse/pull/3820 40. https://github.com/pypa/warehouse/pull/3481 41. https://github.com/pypa/warehouse/pull/3619 42. https://github.com/pypa/warehouse/pull/3805 43. https://github.com/pypa/warehouse/pull/2010 44. https://github.com/pypa/warehouse/issues/3668 45. https://github.com/jdufresne 46. https://github.com/pypa/setuptools/pull/1340 47. https://github.com/joke2k/faker/pull/750 48. https://github.com/ralic/Lein_Reagent/issues/1 49. https://github.com/python/pypi-infra/pull/24 50. https://docs.python.org/3.6/distutils/packageindex.html#package-index 51. https://github.com/pypa/warehouse/pull/3450 52. https://warehouse.readthedocs.io/development/reviewing-patches/ 53. https://ernest.ly/blog/working 54. https://wiki.python.org/psf/PackagingSprints

3 2

Fwd: [pypi-announce] legacy.pypi.org shut down, please use pypi.org
by Sumana Harihareswara 30 Apr '18

30 Apr '18

Roadmap's updated https://wiki.python.org/psf/WarehouseRoadmap . We'd love your help for the next chapter, the post-legacy-shutdown tasks: https://github.com/pypa/warehouse/milestone/12 -- Sumana Harihareswara PyPI/Warehouse project manager Changeset Consulting https://changeset.nyc -------- Forwarded Message -------- Subject: [pypi-announce] legacy.pypi.org shut down, please use pypi.org Date: Mon, 30 Apr 2018 15:25:50 -0000 From: sh(a)changeset.nyc Reply-To: distutils-sig(a)python.org To: pypi-announce(a)python.org We have sunset the original Python Package Index service, which was temporarily deployed at https://legacy.pypi.org . The new PyPI is at https://pypi.org . Browser and API calls to pypi.python.org will continue to redirect to pypi.org . If you have been using legacy.pypi.org directly, please start using pypi.org : https://warehouse.readthedocs.io/api-reference/integration-guide/#migrating… If there is a feature that the new codebase does not support, you should file an issue at https://github.com/pypa/warehouse/issues as soon as possible. If you use JFrog Artifactory, please make sure you're running the latest version. Please see the guidance from JFrog https://jfrog.com/knowledge-base/why-am-i-not-able-to-connect-to-pypi-pytho… and full discussion of the issue https://github.com/pypa/warehouse/issues/3275 . Maintenance report on the sunsetting: https://status.python.org/incidents/ptvp1wnn0jmq Historical context and future plans: https://lwn.net/Articles/751458/ Sincerely, Sumana Harihareswara on behalf of the PyPI team _______________________________________________ pypi-announce mailing list pypi-announce(a)python.org https://mail.python.org/mm3/mailman3/lists/pypi-announce.python.org/

1 0

help needed on pinflow account
by Changshu Liu 30 Apr '18

30 Apr '18

I apologize if this email spams you, but this is the only email address I can find from http:// <goog_582715680>pypa.io. And we are in an situation that need your help. We have a former employee who created user/project on pypi named as p*inflow, *which is inactive for 2+ years. - https://pypi.org/user/pinflow/ - https://pypi.org/project/pinflow/ And now we want to reuse this name to open source a project. We know the email access of this user, but both password for this email and pypi account are lost during the transition period. Is there any way to recover/reset the account's password so that we can update new package? Thanks in advance and this will give us huge value. (please little r to me if you can help and I can share the email address of pinflow user) Thanks, Changshu

1 0

Re: [Distutils] A problem with setuptools, virtualenv and numpy
by Christian Marquardt 29 Apr '18

29 Apr '18

Installing numpy with setupegg.py does indeed install numpy as an egg, but it does not solve the problem I describe here (tested with numpy-1.3.0rc1, and the rest as described below). If numpy is listed as a dependency for another package which is installed into a virtual environment, setuptools tries to download and install numpy instead of just adjusting the easy_install.pth file. Hmm. I tried to install my packages without a virtual environment, and that works ok. So it must be a problem in the interaction between setuptools and virtualenv... I'll cross post this in the virtualenv mailing list... Many thanks, Christian. ----- "David Cournapeau" <david(a)ar.media.kyoto-u.ac.jp> wrote: > Christian Marquardt wrote: > > Hello, > > > > I've run into the following problem: > > > > - I have numpy (v1.3.0b1) successfully installed in my system's > > site-packages (for Python 2.5.2). > > - I have setup a virtual environment (with virtualenv 1.3.3) which > > uses the systems site-packages; in particular, I can load numpy > > when working in that environment. > > - I have a (namespaced) package which requires numpy (via > > "install_requires = ['numpy']" in the setup() of setuptools > > (v0.6c9, as coming with virtualenv), and try to build that in > > the virtual environment using the usual python setup.py install. > > - When processing the requirements for the packages, setuptools > > downloads a fresh copy from numpy 1.3.0b1 from Pypi and tries > > to install it - although numpy is already available in the > system > > wide site-packages directory. > > > > To me, that's a bug, but I would like to know if it is a bug in > numpy > > (which might not support setuptools in the correct way) or > setuptools > > which tries to install an already existing package. > > > > If you want setuptools 'support', I think you should install numpy > through the setupegg.py script. > > cheers, > > David -- Dr. Christian Marquardt Email: christian(a)marquardt.sc Wilhelm-Leuschner-Str. 27 Tel.: +49 (0) 6151 95 13 776 64293 Darmstadt Mobile: +49 (0) 179 290 84 74 Germany Fax: +49 (0) 6151 95 13 885

1 0

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2