Re: [devpi-dev] changing pypi polling/changelog logic

Hi Holger, if I have understood you correctly, your change is only targeting the pypi to master update mechanism. According to our experience inconsistencies between master and replica are just as common as inconsistencies between pypi and master (e.g., see https://bitbucket.org/hpk42/devpi/issue/238) So, couldn't the same idea you proposed here also be applied to the replica updates? Regards, Stephan ________________________________________ From: holger krekel Sent: Tuesday, June 9, 2015 8:03 PM To: Erb, Stephan Cc: holger krekel; devp...@googlegroups.com Subject: Re: [devpi-dev] changing pypi polling/changelog logic Hi Stephan, On Tue, Jun 09, 2015 at 13:17 +0000, Erb, Stephan wrote:

Hi Holger,

could you please elaborate in how far these changes affect the master-replica setup? Over the last months we have seen several inconsistencies where a replica was having a different state than the master. This sounds similar to the problem between the master and PyPi.

The master replica setup is not affected, nor is latency. pypi files will show up on the master and their replicas without big delays usually (unless the replica is erroring, or was disconnected and is catching up).

Regarding changelog thread: Would be great if this one could be kept around. We are using master and replicas in a high-availability setup where we hide the master and multiple replicas behind a load balancer. Given this setup, the inconsistency window may not become too large, i.e., our users expect to upload a package to the master and then be able to install from any of the replicas a few seconds later.

Uploading private packages is unaffected by the change proposed below. The suggested change only targets the Master/pypi.python.org mirroring setup so that we don't need to rely on the changelog-API calls to pypi.python.org but would determine ourselves when we re-check with pypi.python.org. hope that clarifies things, holger

Best Regards, Stephan ________________________________________ From: devp...@googlegroups.com on behalf of holger krekel Sent: Wednesday, June 3, 2015 4:39 PM To: devpi-dev Subject: [devpi-dev] changing pypi polling/changelog logic

Hi all,

the issue of not-up-to-date pypi packages on /root/pypi keeps repeating. see e.g. https://bitbucket.org/hpk42/devpi/issue/59/add-option-to-devpi-server-to-for... We've added an "refresh" button and a "devpi refresh" client side command but i think we need to find a more general solution.

In addition there are people who would like to mirror in packages from non-pypi sources, see e.g. https://bitbucket.org/hpk42/devpi/issue/59/add-option-to-devpi-server-to-for...

And some would like to see WSGI deployment https://bitbucket.org/hpk42/devpi/issue/155/devpi-should-support-generic-wsg... which is currently problematic because of the changelog thread.

So I propose roughly the following:

- make root/pypi (or in general mirror index types) grow an URL field (https://pypi.python.org/simple by default) from which simple pages are polled

- if a mirror page is accessed and has recent enough information (say 30 minutes by default) use the cached info, otherwise refetch/refresh and update cached information if there are changes. If there is old information and the old site can not be reached, simply use the old information, emit a warning in the log.

- the current changelog thread is not started by default but can be enabled with a new command line option. I guess for most production settings it is not important to have real-time updates.

- allow to set the mirror-url so that non-pypi pages can be used.

- existing "refresh" buttons and "devpi refresh" commands continue to force a refresh irrespective of cache settings.

this should solve all above mentioned issues and provide a more solid overall experience.

thoughts, comments?

holger

-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.

-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.

-- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu