Hello Holger, I understood the point about triggering all index pages. But I guess that it would download everything [again] from PyPI, correct? So, in order to avoid another wasteful full download, cos I've already done that... I'm trying to upload packages from the a local folder. Unless you tell me it is not going to work, I'd like to persist on this route. You know... I can take this chance to gain some mileage with devpi :) First thing would be making /root/pypi volatile (if I understood properly!). # let's play with /root/test first, just to see how it works (py276)rgomes@pypi:/srv/pypi$ devpi index /root/test volatile=False /root/test changing volatile: False http://pypi.localdomain:8080/root/test: type=stage bases=root/pypi volatile=False uploadtrigger_jenkins=None acl_upload=root [87112 refs] (py276)rgomes@pypi:/srv/pypi$ devpi index /root/test volatile=True /root/test changing volatile: True http://pypi.localdomain:8080/root/test: type=stage bases=root/pypi volatile=True uploadtrigger_jenkins=None acl_upload=root [87112 refs] # now let's try with /root/pypi (py276)rgomes@pypi:/srv/pypi$ devpi index /root/pypi volatile=True /root/pypi changing volatile: True http://pypi.localdomain:8080/root/pypi: type=mirror bases= volatile=False Traceback (most recent call last): File "/home/rgomes/.virtualenvs/py276/bin/devpi", line 9, in <module> load_entry_point('devpi-common==1.2', 'console_scripts', 'devpi')() File "/home/rgomes/.virtualenvs/py276/lib/python2.7/site-packages/devpi/main.py", line 29, in main return method(hub, hub.args) File "/home/rgomes/.virtualenvs/py276/lib/python2.7/site-packages/devpi/index.py", line 65, in main return index_modify(hub, url, kvdict) File "/home/rgomes/.virtualenvs/py276/lib/python2.7/site-packages/devpi/index.py", line 14, in index_modify index_show(hub, url) File "/home/rgomes/.virtualenvs/py276/lib/python2.7/site-packages/devpi/index.py", line 36, in index_show ixconfig["uploadtrigger_jenkins"],)) KeyError: 'uploadtrigger_jenkins' [87112 refs] Hum... not very good :( I've opened issue #82 https://bitbucket.org/hpk42/devpi/issue/82/cannot-make-root-pypi-volatile Meanwhile, I will play with another index and let you know how it goes. Thanks Richard Gomes http://rgomes.info http://www.linkedin.com/in/rgomes mobile: +44(77)9955-6813 inum <http://www.inum.net/>: +883(5100)0800-9804 sip:r...@ippi.fr On 25/01/14 09:29, holger krekel wrote:
Hi Richard,
On Fri, Jan 24, 2014 at 21:51 +0000, Richard Gomes wrote:
Hello Holger,
No, I haven't tried bandersnatch. I think devpi is perfect for my workflow and I'm not willing to try other things. ok.
Sorry for not being very clear on my intentions. My fault. I made the question more complicated than it should be.
In a nutshell, I just wanted a full PyPI mirror, but I'm not sure how I should load all those 38K packages into devpi cache. I guess that I should make /root/pypi volatile and I should upload package by package onto it. Does it make sense? There are 38K projects but many more archive files. If you want a full mirror, that's around 40Gbytes of storage (and the according network traffic).
What i think makes more sense is to go for triggering all index pages by iterating over all projects and maybe get and thereby cache the first (highest version) archive files. You can get some ideas how to do that in the repository at server/extra/compare_devpi_server.py e.g. through the getnames() function.
I'm basically confused about how devpi decides (or detects) that eventually a package must be updated from PyPI since I've uploaded it by hand. I'm not sure if this idea of uploading by hand would work well or would eventually make devpii confused about when new updates should be downloaded from PyPI. One you touched/retrieved all project index pages, devpi-server will auto-update all index pages for those index pages.
HTH, holger
http://rgomes.info http://www.linkedin.com/in/rgomes mobile: +44(77)9955-6813 inum <http://www.inum.net/>: +883(5100)0800-9804 sip:r...@ippi.fr
On 24/01/14 20:05, holger krekel wrote:
On Fri, Jan 24, 2014 at 04:02 -0800, Richard Gomes wrote:
I've downloaded all packages from PyPI and I'd like to create a local mirror using devpi. If you want to have a full non-lazy mirror, did you consider using bandersnatch?
I had the following idea:
1. create a new index say /root/pypimirror based on /root/pypi
2. upload the entire folder containing 38,000+ packages onto /root/pypimirror This might fail if your system uses a 32K limit on directory entries. I've just fixed it for /root/pypi (not released yet) and i can also fix it for private indices.
3. eventually setting /root/pypimirrot to NotVolatile (if this can be done, somehow)
You can change index volatility any time.
/myuser/myindex +-- /root/pypimirror +-- /root/pypi
Another idea would be creating a "parallel" index to /root/pypi and exposing a third index which derives from both.
/myuser/myindex +-- /root/pypi +-- /root/pypimirror
Does this idea make sense? Is there a better way of doing it, in particular without having to download everything again from PyPI?
Could you state more clearly what you want to achieve in the first place? I can see a number of possible motivations but would like to understand your particular ones. (Did i mention that i was always very bad in school at understanding textual questions in math questions although others seemed to be able to guess the correct meaning of the question? :)
I have another concern: performance. Since 38,000+ packages implies on large directories in the file system... do you think that devpi will have troubles managing such amount of packages? see above. /root/pypi handles >32K fine on trunk. And private indices can also be made to do so. (So far there wasn't a use case for >32K private packages -- let's see if we have one here)
best, holger
Thanks a lot,
-- Richard
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/groups/opt_out.