Mailman 3 - devpi-dev

Fwd: [pypi-announce] upgrade to pip 20.2 -- plus changes coming in 20.3
by Sumana Harihareswara 31 Jul '20

31 Jul '20

A new pip is out. Please see below, upgrade, and let us know if devpi users start to have trouble. In particular, we need your feedback on the beta of the new dependency resolver, because we want to make it the default in the October release. We have a low-traffic, announce-only list to announce major changes to pip/PyPI -- subscribe at https://mail.python.org/mailman3/lists/pypi-announce.python.org/ . best, Sumana Harihareswara, pip project manager -------- Forwarded Message -------- Subject: [pypi-announce] upgrade to pip 20.2 -- plus changes coming in 20.3 Date: Thu, 30 Jul 2020 11:24:58 -0400 From: Sumana Harihareswara <sh(a)changeset.nyc> Reply-To: distutils-sig(a)python.org Organization: Changeset Consulting To: pypi-announce(a)python.org On behalf of the Python Packaging Authority, I am pleased to announce the release of pip 20.2. Please upgrade for speed improvements, bug fixes, and better logging. You can install it by running python -m pip install --upgrade pip. We make major releases each quarter, so this is the first new release since 20.1 in April. NOTICE: This release includes the beta of the next-generation dependency resolver. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the `--use-feature=2020-resolver` flag. Please see our guide on how to test and migrate, and how to report issues <https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-res…>. The new dependency resolver is *off by default* because it is *not yet ready for everyday use*. For release highlights and thank-yous, please see <https://blog.python.org/2020/07/upgrade-pip-20-2-changes-20-3.html> . The full changelog is at <https://pip.pypa.io/en/stable/news/>. Future: We plan to make pip's next quarterly release, 20.3, in October 2020. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3. -- Sumana Harihareswara project manager for pip, on contract with Python Software Foundation Changeset Consulting, https://changeset.nyc _______________________________________________ pypi-announce mailing list -- pypi-announce(a)python.org To unsubscribe send an email to pypi-announce-leave(a)python.org https://mail.python.org/mailman3/lists/pypi-announce.python.org/ -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

2 2

Announcement: pip 20.1b1 beta release
by Sumana Harihareswara 21 Apr '20

21 Apr '20

On behalf of the PyPA, I am pleased to announce that a beta release of pip, pip 20.1b1, has been released. The highlights for this release are: * Significant speedups when building local directories, by changing behavior to perform in-place builds, instead of copying to temporary directories. * Significant speedups in `pip list --outdated`, by parallelizing network access. This is the first instance of parallel code within pip's codebase. * A new `pip cache` command, which makes it possible to introspect and manage pip's cache directory. * Better `pip freeze` for packages installed from direct URLs, enabled by the implementation of PEP 610. We would be grateful for all the testing that users could do to ensure that, when pip 20.1 is released, it's as solid as we can make it. You can upgrade to this beta with `python -m pip install -U --pre pip`. This release also contains an alpha version of pip's next generation resolver. It is **off by default** because it is **unstable and not ready for everyday use**. If you're curious about this, please visit this GitHub issue about the resolver, what doesn't work yet, and what kind of testing would help us out https://github.com/pypa/pip/issues/8099 . The full changelog is available. https://pip.pypa.io/en/latest/news/ As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Specific thanks go to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for their support that enabled the work on the new resolver. -- Sumana Harihareswara pip project manager under contract with Python Software Foundation Changeset Consulting https://changeset.nyc

1 0

PEP 458: Secure transport independent download integrity for PyPI packages
by Sumana Harihareswara 12 Feb '20

12 Feb '20

Hi, devpi folks! I figure you might want to take a look at the PyPI security PEP currently being discussed, since I could imagine devpi wanting to also add TUF metadata handling for packages, and in case there are interoperability concerns/questions. https://discuss.python.org/t/pep-458-surviving-a-compromise-of-pypi/2648/ The PEP authors are revising the proposed summary, title, etc., per https://github.com/secure-systems-lab/peps/blob/c13384a4fac6822626abb7e09ab… : > Attacks on software repositories are common, even in organizations with very good security practices__. The resulting repository compromise allows an attacker to edit all files stored on the repository and sign these files using any keys stored on the repository (online keys). In many signing schemes (like TLS), this access allows the attacker to replace files on the repository and make it look like these files are coming from PyPI. Without a way to revoke and replace the trusted private key, it is very challenging to recover from a repository compromise. In addition to the dangers of repository compromise, software repositories are vulnerable to an attacker on the network (MITM) intercepting and changing files. These and other attacks on software repositories are detailed here__. This PEP aims to protect users of PyPI from compromises of the integrity, consistency and freshness properties of PyPI packages, and enhances compromise resilience, by mitigating key risk and providing mechanisms to recover from a compromise of PyPI or its signing keys. In addition to protecting direct users of PyPI, this PEP aims to provide similar protection for users of PyPI mirrors. > To provide compromise resilient protection of PyPI, this PEP proposes the use of The Update Framework [2]_ (TUF). ..... > This PEP describes changes to the PyPI infrastructure that are needed to ensure that users get valid packages from PyPI. ... > __ https://github.com/theupdateframework/pip/wiki/Attacks-on-software-reposito… > __ https://theupdateframework.github.io/security.html Discussion should probably be directed to the Discourse thread at discuss.python.org ; this is just a heads-up. -- Sumana Harihareswara Changeset Consulting sh(a)changeset.nyc

1 1

Clean the index after purging old versions
by Patrick Mézard 15 Jan '20

15 Jan '20

Hello, I have a devpi server where we have recently deleted a lot of old package versions, using devpi-client. The data directory now looks like: 7.6 GiB [##########] /.indices 956.5 MiB [# ] .sqlite 461.4 MiB [ ] /+files Is this expected? Is there any command to run to rebuild/purge the index? Thanks, -- Patrick Mézard

2 2

Strange errors: "While handling ... project does not exist"
by Nicholas Williams 08 Jan '20

08 Jan '20

We're starting to roll out our new Devpi installation to projects now that we've tested it extensively. We have one master and three mirrors. For the purposes of this email, the mirrors are load-balanced behind the domain name "devpi.example.com," and the master is available at " primary-devpi-server.example.com." We configured our Pipenv and Pip installations to use this index URL: https://devpi.example.com/org_name/stable/+simple/ That URL will use the mirrors. The "org_name/stable" index extends root/pypi and also includes our internal projects. Everything appears to be working on the client. No errors, installs are going off without a hitch. However, the Devpi server on the master machine (not the mirrors) logs a ton of errors like the following: [req78237] [Rtx14602] while handling https://primary-devpi-server.example.com/root/pypi/+simple/internal_project… : The project internal_project_name does not exist. What is the cause of these errors? Is this something that should concern us, or something that we can ignore? "internal_project_name" installs just fine, but for some reason yields those errors on the master server. Thanks, Nick

2 6

Need help with reverse proxy config and Devi-Web
by Nicholas Williams 19 Dec '19

19 Dec '19

I'm having trouble with a reverse-proxy config and could use some assistance. We have the following setup: LOAD BALANCER Accepts requests via the official hostname and HTTPS (port 443) Forwards requests to Devpi Nginx server Sends headers X-Forwaded-Port, X-Forwarded-Host, X-Forwarded-Proto matching the official hostname, HTTPS, and 443. DEVPI NGINX SERVER Accepts requests via port 80 Serves requests for +f files directly (works perfectly) Forwards remaining requests to Devpi Python server DOES NOT override headers X-Forwaded-Port, X-Forwarded-Host, X-Forwarded-Proto (I have those proxy_set_header values from the recommended nginx.conf commented out so that Nginx doesn't override them) DOES NOT send header X-Outside-Url (I also have that commented out) The behavior is that all links, CSS tags, and JavaScript tags point to http://localhost/... instead of https://the.correct.domain.name/... What do I need to change to make Devpi properly use the X-Forwaded-Port, X-Forwarded-Host, X-Forwarded-Proto headers coming from the load balancer? Thanks, Nick

2 2

devpi-client 5.1.0 and devpi-pr 1.0.0 released
by Florian Schulze 31 Oct '19

31 Oct '19

Hi! I released devpi-client 5.1.0 and devpi-pr 1.0.0. For devpi-client keyring support for devpi push to external indexes like pypi.org was added and it has some bugfixes. https://pypi.org/project/devpi-client/ With devpi-pr a pull request workflow can be added to devpi-server and devpi-client. https://pypi.org/project/devpi-pr/ Regards, Florian Schulze

1 0

devpi-server 5.2.0 and devpi-web 4.0.0 released
by Florian Schulze 19 Oct '19

19 Oct '19

Hi! I released devpi-server 5.2.0 and devpi-web 4.0.0. See the changelogs at: https://pypi.org/project/devpi-server/ https://pypi.org/project/devpi-web/ Regards, Florian Schulze

1 0

Upcoming releases
by Florian Schulze 02 Oct '19

02 Oct '19

Hi! I uploaded development releases for testing: https://m.devpi.net/fschulze/dev/devpi-web/4.0.0.dev1 https://m.devpi.net/fschulze/dev/devpi-server/5.2.0.dev0 These releases are drop-in updates. For devpi-web you might have to run ``--recreate-search-index`` in case you want to go back to a previous version. Besides that there are no changes preventing a downgrade. But it's best to test devpi-web on a replica which can be rebuilt if necessary. Speaking of rebuilding replicas: With the ``--replica-file-search-path`` option and the ``--hard-links`` option it is possible to vastly cut down the replication time when you already have the files on disk. Either create the new replica in a new folder, or rename the old server folder to create the new replica in the same place. Then run the new replica with ``--replica-file-search-path old_location/+files --hard-links`` and while replicating the new replica will look for existing files in the specified location and create hard links if it finds one and it's checksum matches. As the version number already indicates, devpi-web has the biggest changes, namely dropping Python 2.7 support, a different way to index projects and several changes to templates which might require updates in themes: 4.0.0.dev1 (2019-09-18) ======================= Deprecations and Removals ------------------------- - Dropped support for Python 2.7. Features -------- - project.pt: implement #319 - add link to latest version and summary of latest version to project view - doc.pt: implement #339: when viewing older documentation, show link to latest and stable documentation. - implement #346: add support for 'latest' and 'stable' versions for project version URLs - version.pt, style.css: implement #347 - show navigation link when newer version is available - The search indexing is now performed in a separate thread. Indexing prioritizes private indexes, then projects downloaded from mirrors and lastly never downloaded projects from mirrors. - On server start a refresh run for the search index is started automatically. In addition to that the indexing is done in smaller batches, which allows resuming of indexing the large pypi project lists. - In many cases re-indexing is now avoided. - version.pt, common.js: all list like metadata is now collapsed to ~3 lines instead of only classifiers. - The ``--recreate-search-index`` option now just removes the whole index. Upon server restart the index is rebuilt automatically in the background. - With devpi-server 5.1.1 or newer, indexing of mirrors will not trigger a refresh of the simple links for projects which have downloaded releases. This allows quicker indexing for new replicas and in case the search index is rebuilt using ``--recreate-search-index``. - root.pt: added new view for user Bug Fixes --------- - Pass query string on to documentation iframe, this fixes Sphinx search among other things. Other Changes ------------- - macros.pt, version.pt, style.css: moved separators to css. - The minimum supported version of devpi-server is now 5.1.0. - The interface of ``update_projects`` for search indexers has changed. The list of projects is now a list of ``ProjectIndexingInfo`` objects instead of ``dict``. To get the dictionary as before, call ``preprocess_project`` on the object. 5.2.0.dev0 (2019-09-18) ======================= Features -------- - Metrics for the sqlite storage cache are exposed in the JSON output of the +status view. Bug Fixes --------- - fix #545: provide proper error message when there is an exception during push to an external repository. - Fix possible race condition when writing files. - Fix possible assertion error if importing multiple changes in a replica fails in the middle and fetching a single change set is tried as fallback. - For plugins the ``offline`` attribute of mirror stage instances now works independently of the ``--offline-mode`` command line option. This only applies to ``_perstage`` methods. - Files created in a transaction are written directly to temporary files instead of being kept in memory until commit. - Unnecessary database writes where the final value didn't change are now prevented. Other Changes ------------- - The timeout when fetching the list of remote projects for a mirror index is set to a minimum of 30s by default and to 60s when running as replica. Other fetches of mirrors still use the timeout specified via ``--request-timeout``.

1 1

mirror_whitelist inheritance
by Florian Schulze 01 Oct '19

01 Oct '19

Hi! There currently is an issue about mirror_whitelist that needs input from anyone who uses it together with inheritance (more than just root/pypi in bases). https://github.com/devpi/devpi/issues/725 Regards, Florian Schulze

1 0