Preloading the PyPI cache for offline use
I want to have a devpi instance I can use for offline use. The server will be in offline mode 99% of the time, and only online when I make a special effort. So I want a way to "preload" the PyPI cache so that packages I might need for offline use are available. I'd expect to maybe go online once a week or so (possibly less, depends on how much development activity I'm doing at the time), and refresh the cache with latest versions.
What's the best way of doing this? One option would be to do something like "pip download -r requirements.txt" to refresh all the packages in the given requirements file. I guess that would probably work, but it feels a bit clunky. And it doesn't really give me a way of doing anything more complex (for example, removing older versions of packages that I no longer need to have cached). Is there a better way to manage the cache?
Thanks, Paul
On 2 Feb 2016, at 12:23, Paul Moore wrote:
I want to have a devpi instance I can use for offline use. The server will be in offline mode 99% of the time, and only online when I make a special effort. So I want a way to "preload" the PyPI cache so that packages I might need for offline use are available. I'd expect to maybe go online once a week or so (possibly less, depends on how much development activity I'm doing at the time), and refresh the cache with latest versions.
What's the best way of doing this? One option would be to do something like "pip download -r requirements.txt" to refresh all the packages in the given requirements file. I guess that would probably work, but it feels a bit clunky. And it doesn't really give me a way of doing anything more complex (for example, removing older versions of packages that I no longer need to have cached). Is there a better way to manage the cache?
Currently there is no better way. You might be able to write a small script or modify pip to only request the HEAD of the download URL. That way the devpi-server will download the package, but you don't have to download it from devpi. I think there is no way currently to remove old versions besides going through the files directly. A plugin to devpi-server using some internals could implement that though.
Regards, Florian Schulze
On Tue, Feb 02, 2016 at 14:10 +0100, Florian Schulze wrote:
On 2 Feb 2016, at 12:23, Paul Moore wrote:
I want to have a devpi instance I can use for offline use. The server will be in offline mode 99% of the time, and only online when I make a special effort. So I want a way to "preload" the PyPI cache so that packages I might need for offline use are available. I'd expect to maybe go online once a week or so (possibly less, depends on how much development activity I'm doing at the time), and refresh the cache with latest versions.
What's the best way of doing this? One option would be to do something like "pip download -r requirements.txt" to refresh all the packages in the given requirements file. I guess that would probably work, but it feels a bit clunky. And it doesn't really give me a way of doing anything more complex (for example, removing older versions of packages that I no longer need to have cached). Is there a better way to manage the cache?
Currently there is no better way. You might be able to write a small script or modify pip to only request the HEAD of the download URL. That way the devpi-server will download the package, but you don't have to download it from devpi.
If you run in this shallow mode you need to have all dependencies listed, otherwise you'll miss some and your cache is not filled enough.
I think there is no way currently to remove old versions besides going through the files directly. A plugin to devpi-server using some internals could implement that though.
I think more precise cache handling didn't come up so far because most people including me don't seem to worry too much disk space usage. Do you have an idea how you'd like to remove cached packages UI-wise? hand-selecting them? Removing all but newest versions for each package?
I guess one could try to make "devpi remove" work for mirror indexes -- it accepts version specs like pytest<2.6 etc. but currently only works on private indices.
holger
Regards, Florian Schulze
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at https://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
On 2 February 2016 at 19:03, holger krekel hol...@merlinux.eu wrote:
I think there is no way currently to remove old versions besides going through the files directly. A plugin to devpi-server using some internals could implement that though.
I think more precise cache handling didn't come up so far because most people including me don't seem to worry too much disk space usage. Do you have an idea how you'd like to remove cached packages UI-wise? hand-selecting them? Removing all but newest versions for each package?
I guess one could try to make "devpi remove" work for mirror indexes -- it accepts version specs like pytest<2.6 etc. but currently only works on private indices.
I wasn't really thinking it was something I'd need for anything specific, I'd just sort of imagined there might be some sort of "manage the cache" command I'd missed. I was thinking in terms of something like a "devpi cache" command with subcommands for various things
devpi cache list -- lists what's in the cache devpi cache remove PACKAGE [--outdated] -- remove all files for PACKAGE (or all but the latest)
I was thinking that such a command would also include some sort of "devpi cache add" subcommand to populate the cache (hence my original question).
But this isn't a case of "I would need these things", more that I'd assumed there was some form of cache management that I'd missed (and if so, this is how it'd look).
I'm OK with just doing "pip download" via the cache to pre-seed it. If I want a more "controlled" mirror I'm probably better off just doing "pip download" direct to get the files locally, and using --find-links to let pip find them. Something like devpi is probably more complex than I need if I'm going to do things that manually, anyway.
Thanks, Paul
participants (3)
-
Florian Schulze -
holger krekel -
Paul Moore