Hi Holger,
could you please elaborate in how far these changes affect the master-replica setup? Over the last months we have seen several inconsistencies where a replica was having a different state than the master. This sounds similar to the problem between the master and PyPi.
Regarding changelog thread: Would be great if this one could be kept around. We are using master and replicas in a high-availability setup where we hide the master and multiple replicas behind a load balancer. Given this setup, the inconsistency window may not become too large, i.e., our users expect to upload a package to the master and then be able to install from any of the replicas a few seconds later.
Best Regards,
Stephan
________________________________________
From: devp...@googlegroups.com on behalf of holger krekel
Sent: Wednesday, June 3, 2015 4:39 PM
To: devpi-dev
Subject: [devpi-dev] changing pypi polling/changelog logic
Hi all,
the issue of not-up-to-date pypi packages on /root/pypi keeps repeating.
see e.g. https://bitbucket.org/hpk42/devpi/issue/59/add-option-to-devpi-server-to-for...
We've added an "refresh" button and a "devpi refresh" client side
command but i think we need to find a more general solution.
In addition there are people who would like to mirror in packages
from non-pypi sources, see e.g. https://bitbucket.org/hpk42/devpi/issue/59/add-option-to-devpi-server-to-for...
And some would like to see WSGI deployment https://bitbucket.org/hpk42/devpi/issue/155/devpi-should-support-generic-wsg... which is currently problematic
because of the changelog thread.
So I propose roughly the following:
- make root/pypi (or in general mirror index types) grow an URL field
(https://pypi.python.org/simple by default) from which simple
pages are polled
- if a mirror page is accessed and has recent enough information
(say 30 minutes by default) use the cached info, otherwise
refetch/refresh and update cached information if there are changes.
If there is old information and the old site can not be reached,
simply use the old information, emit a warning in the log.
- the current changelog thread is not started by default but can be enabled
with a new command line option. I guess for most production settings
it is not important to have real-time updates.
- allow to set the mirror-url so that non-pypi pages can be used.
- existing "refresh" buttons and "devpi refresh" commands continue
to force a refresh irrespective of cache settings.
this should solve all above mentioned issues and provide a more solid
overall experience.
thoughts, comments?
holger
--
You received this message because you are subscribed to the Google Groups "devpi-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com.
To post to this group, send email to devp...@googlegroups.com.
Visit this group at http://groups.google.com/group/devpi-dev.
For more options, visit https://groups.google.com/d/optout.