Hello Holger,

No, I haven't tried bandersnatch.
I think devpi is perfect for my workflow and I'm not willing to try other things.

Sorry for not being very clear on my intentions.
My fault. I made the question more complicated than it should be.

In a nutshell, I just wanted a full PyPI mirror, but I'm not sure how I should load all those 38K packages into devpi cache.
I guess that I should make /root/pypi volatile and I should upload package by package onto it.
Does it make sense?

I'm basically confused about how devpi decides (or detects) that eventually a package must be updated from PyPI since I've uploaded it by hand. I'm not sure if this idea of uploading by hand would work well or would eventually make devpii confused about when new updates should be downloaded from PyPI.

Thanks   

Richard Gomes
http://rgomes.info
http://www.linkedin.com/in/rgomes
mobile: +44(77)9955-6813
inum: +883(5100)0800-9804
sip:r...@ippi.fr

On 24/01/14 20:05, holger krekel wrote:
On Fri, Jan 24, 2014 at 04:02 -0800, Richard Gomes wrote:
I've downloaded all packages from PyPI and I'd like to create a local 
mirror using devpi.
If you want to have a full non-lazy mirror, did you consider using
bandersnatch?

I had the following idea:

     1. create a new index say /root/pypimirror based on /root/pypi

     2. upload the entire folder containing 38,000+ packages onto 
/root/pypimirror
This might fail if your system uses a 32K limit on directory entries.
I've just fixed it for /root/pypi (not released yet) and i can also
fix it for private indices.

     3. eventually setting /root/pypimirrot to NotVolatile (if this can be 
done, somehow)
You can change index volatility any time.

    /myuser/myindex
        +-- /root/pypimirror
                +-- /root/pypi

Another idea would be creating a "parallel" index to /root/pypi and 
exposing a third index which derives from both.

    /myuser/myindex
        +-- /root/pypi
        +-- /root/pypimirror


Does this idea make sense? Is there a better way of doing it, in particular 
without having to download everything again from PyPI?
Could you state more clearly what you want to achieve in the first
place?  I can see a number of possible motivations but would like
to understand your particular ones.  (Did i mention that i was always
very bad in school at understanding textual questions in math questions
although others seemed to be able to guess the correct meaning of the 
question? :)

I have another concern: performance. Since 38,000+ packages implies on 
large directories in the file system... do you think that devpi will have 
troubles managing such amount of packages?
see above.  /root/pypi handles >32K fine on trunk.  And private indices
can also be made to do so. (So far there wasn't a use case for >32K 
private packages -- let's see if we have one here)

best,
holger



Thanks a lot,

-- Richard

-- 
You received this message because you are subscribed to the Google Groups "devpi-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com.
To post to this group, send email to devp...@googlegroups.com.
Visit this group at http://groups.google.com/group/devpi-dev.
For more options, visit https://groups.google.com/groups/opt_out.