Mailman 3 - devpi-dev

Questions about high availability and replication
by Nicholas Williams Oct. 1, 2019

Oct. 1, 2019

Hello, My organization is exploring replacing our current, patched-together "wheelhouse" with a proper Devpi server mirroring PyPi and hosting our internal packages. Because this Devpi architecture will be accessed by hundreds of developers and over 100 build server workers, it needs to be highly available (able to handle a lot of traffic). So this leads me to several questions (five major questions and a few sub-questions, to be more accurate): - Is anyone familiar with a threshold of … [View More]diminishing returns when it comes to the "threads" server setting? At what point, if any, does increasing this number do no good? I assume that # of "threads" roughly equates to # of simultaneous connections that single server is capable of, perhaps minus one or two. Is this a fair assumption? - How many "devpi-server" instances can access a single "serverdir?" Is it just one, and bad things will happen if a second "devpi-server" tries to access it (and, if I need multiple servers, I should use master-replica replication)? Or can multiple "devpi-server" instances access a single "serverdir" to serve as multiple hot masters? - When employing master-replica replication, I'm wondering if the following configuration is sensible, or if there are better approaches: 1) One (or possibly more, if allowed) master, 2) Two or more replicas replicating off the master, 3) An HTTPS load balancer that sends all traffic to the replicas (not the master? or include the master in the load balancing?). In this case, what happens if someone uses the https://load-balanced-devpi-url/ URL to publish a new package? Does a replica it lands on just forward that up to the master? Or does the user get an error? Or do worse things happen? - When employing master-replica replication, I noticed in the documentation this word of caution: "You probably want to make sure that before starting the replica and/or master that you also have the Web interface and search plugin installed. You cannot install it later." To be clear, I already have "devpi-web" installed (and a theme, too), but this note confuses (and slightly concerns) me, so I'd like to understand it better. Why would you be unable to install "devpi-web" after starting a master or replica? What would it break? Why wouldn't this work? Or do you just mean that you would have to shut down the servers to install "devpi-web" (which is understandable)? Is this different from non-replication environments (can you install "devpi-web" after starting a non-replicating server), and if so, why? I'm asking for details about this because I'm curious if it has potential consequences (stability, etc.) that extend beyond just the "devpi-web" plugin. - When creating a new index, could someone elaborate a bit more on the "mirror_whitelist" setting? What is the difference between not setting it at all (default/implicit) and explicitly setting it to "*"? What does "*" actually mean? When you upload a new package to the index that conflicts with a package in its base index, is the resulting behavior that the uploaded package (or perhaps just the versions you upload) ALWAYS overrides the one in the base index, regardless of the "mirror_whitelist" setting? My goal here is to have an index that uses "root/pypi" as its base and also hosts our internal packages, so that we can point Pipenv to one URL and install everything from there. Is there a more sensible approach than that which I am planning on taking? Thanks in advance for helping me make the right decisions in our upcoming deployment. Nick Williams [View Less]

2 5

Is it possible to upload external test results for a package?
by con-f-use＠gmx.net Sept. 19, 2019

Sept. 19, 2019

Salutations, another quick question about tox testing results: Given I have `results.json` file from a tox run, can I upload that to a devpi index without running the tests again? Background is that we build our packages in our CI system, test them without uploading them to an index, then if the tests succeed, we only upload the wheels to our devpi index. The way I have read the devpi documentation, displaying test results on the index always requires to upload a source distribution to … [View More]

2 3

mirror_whitelist negation
by con-f-use＠gmx.net Sept. 19, 2019

Sept. 19, 2019

Hi, quick question: If I want to whitelist everything but a few packages, is that possible with devpi indexes and if so what is the syntax? E.g. i'd expect: `mirror_whitelist=['*', '!somepack', '!mypackage']` would whitelist things like `pip` and `setuptools` but not the two explicitly forbidden packages named somepack and mypackage. Cheers and thx, Christoph

2 5

devpi-constrained
by Florian Schulze Aug. 5, 2019

Aug. 5, 2019

Hi! I released devpi-constrained. It's a releases filter for devpi-server. See https://pypi.org/project/devpi-constrained/ for details. Regards, Florian Schulze

1 0

PyCon AU Sprints
by Graham Williamson Aug. 5, 2019

Aug. 5, 2019

Hi, Apologies for this being last minute, but I'm currently attending the PyConAU dev sprints taking place today and tomorrow, 05Aug-06Aug UTC+10. I plan on working on the following: devpi-ldap * My existing PR for the ldap auth plugin (https://github.com/devpi/devpi-ldap/pull/43), it needs refinement. * Adding info on bootstrapping an LDAP server. devpi * Adding contributing information to the documentation. If there's time, I'm might try cherry-picking a few other things from the issues … [View More]

3 3

PR-RFC request events
by Florian Schulze Aug. 1, 2019

Aug. 1, 2019

I created a PR for a new request events hook at https://github.com/devpi/devpi/pull/692 The problem with the current hooks devpiserver_stage_created, devpiserver_on_changed_versiondata and devpiserver_on_upload is that they are database events. That means when data is imported or replicated, they are called again. That is also why they don't have access to the request and thus infos like the logged in user. The devpiserver_on_upload_sync is called from within the request, but it may be … [View More]

1 0

ACL setup when using restrict-modify
by Louis Opter July 22, 2019

July 22, 2019

Hello, Thank you for the project, quick question: I see that restrict-modify actually only allows changes made by the principals listed there. Which kinda means that you are either root or you aren't, should restrict-modify also allow index_create/user_modify for authenticated users? Thank you, -- Louis Opter

2 4

Internal PyPi repo use case
by Jessie Lin July 17, 2019

July 17, 2019

Hello Devpi deverlopers, We're looking for a solution for internal PyPi server. We like developers/data Scientists to use the packages only from the internal repo, which is approved by legal and scanned for security purposes. Can we let admins to download from external PyPi, and other users only download from internal repo only? If the package is not in the local repo, the developers need to discuss with Admin first. From reading the doc it seems Devpi can be a through cache. Is this … [View More]

3 3

Re: serving devpi-web over https
by Michel Albert July 12, 2019

July 12, 2019

I am having the exact same issue. I've set up devpi to be served inside a kubernetes cluster and use "edge-termination" for SSL traffic. This means that the traffic goes from the browser to K8S as HTTPS which forwards the traffic *unencrypted* to devpi. All links to resources like CSS and JS files are rendered as "http" on the returned document which causes the browser to block them. On a wild guess I assume that devpi auto-detects the protocol? If that is the case then is makes sense that … [View More]

4 5

Releases
by Florian Schulze July 1, 2019

July 1, 2019

Hi! Last Friday I released devpi-client 5.0.0, devpi-server 5.0.0 and devpi-web 3.5.2. Check out the PYPI pages for change logs: https://pypi.org/project/devpi-client/ https://pypi.org/project/devpi-server/ https://pypi.org/project/devpi-web/ Regards, Florian Schulze

1 0