Hi all,
the issue of not-up-to-date pypi packages on /root/pypi keeps repeating.
see e.g. https://bitbucket.org/hpk42/devpi/issue/59/add-option-to-devpi-server-to-fo…
We've added an "refresh" button and a "devpi refresh" client side
command but i think we need to find a more general solution.
In addition there are people who would like to mirror in packages
from non-pypi sources, see e.g. https://bitbucket.org/hpk42/devpi/issue/59/add-option-to-devpi-server-to-fo…
And some would like to see WSGI deployment https://bitbucket.org/hpk42/devpi/issue/155/devpi-should-support-generic-ws… which is currently problematic
because of the changelog thread.
So I propose roughly the following:
- make root/pypi (or in general mirror index types) grow an URL field
(https://pypi.python.org/simple by default) from which simple
pages are polled
- if a mirror page is accessed and has recent enough information
(say 30 minutes by default) use the cached info, otherwise
refetch/refresh and update cached information if there are changes.
If there is old information and the old site can not be reached,
simply use the old information, emit a warning in the log.
- the current changelog thread is not started by default but can be enabled
with a new command line option. I guess for most production settings
it is not important to have real-time updates.
- allow to set the mirror-url so that non-pypi pages can be used.
- existing "refresh" buttons and "devpi refresh" commands continue
to force a refresh irrespective of cache settings.
this should solve all above mentioned issues and provide a more solid
overall experience.
thoughts, comments?
holger
On Wed, Jun 03, 2015 at 16:48 +0200, Florian Schulze wrote:
> >So I propose roughly the following:
>
> ...
>
> >- if a mirror page is accessed and has recent enough information
> >(say 30 minutes by default) use the cached info, otherwise
> >refetch/refresh and update cached information if there are changes.
> >If there is old information and the old site can not be reached,
> >simply use the old information, emit a warning in the log.
>
> If the background thread is running, the refresh should be triggered
> in the background as well. That way if I don't have internet, I
> don't have to wait for the timeout all the time I'm installing
> something.
Do you mean that if a simple page is more than 30 minutes old we would
start a background refresh task but otherwise serve the cached
information?
If you are offline then trying to get from pypi.python.org should
immediately fail and not introduce a significant hold up for serving the
cached information (to be verified of course). So i'd hope to be able
to avoid any background thread. If in doubt we could increase the
default timeout and/or set it to "0" meaning you have to manually
refresh always.
holger
> Otherwise I can't think of anything speaking against your points.
>
> Regards,
> Florian Schulze
>
--
about me: http://holgerkrekel.net/about-me/
contracting: http://merlinux.eu
