Hi Florian, Yesterday i spent some time on top of the devpi-server-3.0.0 branch to move our storage to use normalized projectnames instead of "real" names (let's call them "display names" -- Donald uses this also for warehouse IIRC). One area of complications was in project existence checks and i got the impression we might even get rid of needing a up-front full simple list. In any case i think the normalization effort should simplify (and speed up) the code for mirroring and would conflict with a parallel effort to introduce generalized mirroring. Therefore i suggest to first land normalization before tackling mirroring. I hope to finish it soon but probably not before mid next week. cheers, holger On Fri, Nov 13, 2015 at 12:55 +0100, Florian Schulze wrote:
Hi!
We want to support external mirrors other than PyPI.
tl;dr The basics are mostly clear. Migration from a mirror could use input. Mirroring of devpi indexes needs more thought.
I'd say we only support mirrors that provide a PyPI simple view where each project has one folder. For "findlinks" like private package storages which have all packages in one folder, I recommend to use CheesePrism and mirror that. I don't think we could support that lazily, as each file needs to be inspected for metadata, so we would have to download everything.
When creating a mirror index, you provide the root url of the simple view (like https://pypi.python.org/simple/) and you can set the cache expiry time (defaults to 1800 seconds). The index settings are "mirror_url" and "mirror_cache_expiry".
Mirrors are lazy, exactly like root/pypi is at the moment.
To enable migration, we add support to "push" from a mirror into a regular index (also useful to selectively get pypi packages into an index) and look into "bulk pushing". Any thoughts on this?
We should rename "pypi_whitelist" into "mirror_whitelist" or something like that. We will do a 3.0.0 anyway and can switch this during export/import.
Should we add a setting that exempts a mirror from the whitelisting, or should we just get people to migrate to a regular index to avoid the need to whitelist packages?
What about mirroring a devpi index? I see 3 scenarios: 1. Just mirror +simple of the index. + works now - problem is, that all inherited indexes and pypi would be mirrored as well. 2. Mirror a view which excludes mirrors. 3. Mirror a view that excludes any inherited indexes.
I think both 2 and 3 have valid use cases. We need good names for the URL, I didn't come up with good ones yet (is there one word for "not inheriting").
We could later optimize devpi mirroring in various ways. For example with a serial header and with a notification protocol, though I'm not sure more than the serial is really necessary for most use cases.
Regards, Florian Schulze
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
-- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu