Hi, I just did some work moving bandersnatch a bit further towards a packaged release. The recent changesets contain changes that require some manual adjustments for existing installations: * run `bin/python bootstrap.py` and `bin/buildout` again because I switched to buildout 2 * you need to update your cronjobs because the commands have been renamed to 'bin/bandersnatch mirror' and 'bin/bandersnatch update-stats' * check the default config file: it now includes the parameters for statistic updates I think I don't need any further breakage for mirror administrators in the near future and will be able to make a PIP-installable release in the next days. If you find any further bugs, please tell me on bitbucket. Also, if enough people start gittipping me, then I'll start adding more mirrors on international locations as I can afford them. Cheers, Christian
Christian Theune <ct@gocept.com> writes:
Hi,
I just did some work moving bandersnatch a bit further towards a packaged release.
The recent changesets contain changes that require some manual adjustments for existing installations:
* run `bin/python bootstrap.py` and `bin/buildout` again because I switched to buildout 2 * you need to update your cronjobs because the commands have been renamed to 'bin/bandersnatch mirror' and 'bin/bandersnatch update-stats' * check the default config file: it now includes the parameters for statistic updates
I think I don't need any further breakage for mirror administrators in the near future and will be able to make a PIP-installable release in the next days.
If you find any further bugs, please tell me on bitbucket.
does bandersnatch support some kind of "selective mirroring"? e.g. only mirror sdists or only a certain list of packages?
Hi, On 2013-04-06 19:56:19 +0000, Ralf Schmitt said:
does bandersnatch support some kind of "selective mirroring"? e.g. only mirror sdists or only a certain list of packages?
At the moment it does not. My primary goal is currently to provide a stable solution to reliably run mirrors in the sense of PEP 381. The internal mechanisms of PEP 381 and the way that clients can authenticate the mirrored packages is not compatible with a partial mirror as the index pages are not generated but taken byte-by-byte so they can be verified with PyPIs cryptographic signature on them. I can see that this would be useful. However, my current knowledge is that the way to authenticate the packages will change in the "near" future: Richard Jones mentioned that "the update framework" would be interesting for this. I do not know yet what the implications for the mirrors will be, but I don't want to invest too much time into complexity that I need to rework in the near future. Christian
Christian Theune <ct@gocept.com> writes:
Hi,
On 2013-04-06 19:56:19 +0000, Ralf Schmitt said:
does bandersnatch support some kind of "selective mirroring"? e.g. only mirror sdists or only a certain list of packages?
At the moment it does not. My primary goal is currently to provide a stable solution to reliably run mirrors in the sense of PEP 381.
The internal mechanisms of PEP 381 and the way that clients can authenticate the mirrored packages is not compatible with a partial mirror as the index pages are not generated but taken byte-by-byte so they can be verified with PyPIs cryptographic signature on them.
I can see that this would be useful. However, my current knowledge is that the way to authenticate the packages will change in the "near" future: Richard Jones mentioned that "the update framework" would be interesting for this. I do not know yet what the implications for the mirrors will be, but I don't want to invest too much time into complexity that I need to rework in the near future.
thanks for the explanation! -- cheers ralf
On 04/07/2013 09:36 AM, Christian Theune wrote:
I can see that this would be useful. However, my current knowledge is that the way to authenticate the packages will change in the "near" future: Richard Jones mentioned that "the update framework" would be interesting for this. I do not know yet what the implications for the mirrors will be, but I don't want to invest too much time into complexity that I need to rework in the near future.
Indeed, we are working on this as I write this. I am obtaining some results on how large TUF metadata would be in describing PyPI. I will email some results very soon.
On Sun, Apr 7, 2013 at 9:36 AM, Christian Theune <ct@gocept.com> wrote:
The internal mechanisms of PEP 381 and the way that clients can authenticate the mirrored packages is not compatible with a partial mirror as the index pages are not generated but taken byte-by-byte so they can be verified with PyPIs cryptographic signature on them.
FWIW, those pages use relative URLs for PyPI-hosted packages, which means you could serve those as a redirect in order to implement partial mirroring, at least in theory. ;-)
On 2013-04-08 19:43:07 +0000, PJ Eby said:
On Sun, Apr 7, 2013 at 9:36 AM, Christian Theune <ct@gocept.com> wrote:
The internal mechanisms of PEP 381 and the way that clients can authenticate the mirrored packages is not compatible with a partial mirror as the index pages are not generated but taken byte-by-byte so they can be verified with PyPIs cryptographic signature on them.
FWIW, those pages use relative URLs for PyPI-hosted packages, which means you could serve those as a redirect in order to implement partial mirroring, at least in theory. ;-)
Heh. I did not see this coming, but I like it. :) The simplest solution on a static mirror that I can see for this would be (with nginx) a "missing files" rule that redirects to PyPI. However, that would be bad form as that would cause *any* 404-triggering request on a mirror to hit the upstream server - bad for bots. I'll let this remain science fiction for now. :) Christian
On Tue, Apr 9, 2013 at 2:41 AM, Christian Theune <ct@gocept.com> wrote:
The simplest solution on a static mirror that I can see for this would be (with nginx) a "missing files" rule that redirects to PyPI. However, that would be bad form as that would cause *any* 404-triggering request on a mirror to hit the upstream server - bad for bots.
Even if you limited it to just the /packages subtree? I guess I'm not sure what scenario results in bot-havoc for a partial mirror. I mean, wouldn't partial mirrors be basically private?
On 2013-04-10 03:14:37 +0000, PJ Eby said:
On Tue, Apr 9, 2013 at 2:41 AM, Christian Theune <ct@gocept.com> wrote:
The simplest solution on a static mirror that I can see for this would be (with nginx) a "missing files" rule that redirects to PyPI. However, that would be bad form as that would cause *any* 404-triggering request on a mirror to hit the upstream server - bad for bots.
Even if you limited it to just the /packages subtree? I guess I'm not sure what scenario results in bot-havoc for a partial mirror. I mean, wouldn't partial mirrors be basically private?
Right, that's what I expect in most cases, too. I see projects setting up project-specificy PyPI-style index pages so I can see benefit for that kind of application of not mirroring PyPI completely but keeping it public. Christian
participants (4)
-
Christian Theune -
PJ Eby -
Ralf Schmitt -
Trishank Karthik Kuppusamy