nailed-down versions by having custom index
Hi all, I'd like to do a bit of experimenting with a custom index with only the right packages in just the right versions. Some possible advantages I'm hoping to see for my usecase: - Isolation from one of various servers being down (pypi, sourceforge). - Guaranteed versions even when not using buildout. (Stuff needs "python setup.py install" in the system python, apparently, for arcgis/arcview on windows to be able to use it). Personally, I'd just let everyone use buildout, but I think I won't be able to get that working with that "install it in site-packages" requirement. (Ideas welcome, of course). Question: how to maintain such a custom index? iirc chrism has a script to build an apache-served folder structure from a bunch of eggs/tarballs, so just having the tarballs is enough. Probably just putting those tarballs in one location is enough anyway. But how to get the right collection of tarballs in there? Is that a by-hand approach? Can you automate it? Question 2: I thought you could only have one index. So installing something extra from pypi seems out. How do people handle that? I'm totally used to buildout and its [versions] list and buildout.dumppickedversions and KGS lists, so I need to reset my thinking on this a little. I need some input :-) Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Programmer/advisor at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets"
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Reinout van Rees wrote:
Hi all,
I'd like to do a bit of experimenting with a custom index with only the right packages in just the right versions. Some possible advantages I'm hoping to see for my usecase:
- Isolation from one of various servers being down (pypi, sourceforge).
- Guaranteed versions even when not using buildout. (Stuff needs "python setup.py install" in the system python, apparently, for arcgis/arcview on windows to be able to use it).
Personally, I'd just let everyone use buildout, but I think I won't be able to get that working with that "install it in site-packages" requirement. (Ideas welcome, of course).
Question: how to maintain such a custom index?
iirc chrism has a script to build an apache-served folder structure from a bunch of eggs/tarballs, so just having the tarballs is enough. Probably just putting those tarballs in one location is enough anyway.
You need the 'simple' bookkeeping files: the attached script builds them from that directory full of tarballs. Run it via: $ cd /path/to/directory/full/of/tarballs $ /path/to/python /path/to/makeindex.py *.{gz,tgz,zip,egg} # move aside previous 'simple' first $ mv index simple
But how to get the right collection of tarballs in there? Is that a by-hand approach? Can you automate it?
You might look at compoze: it is designed to allow fetching sources and building an index based on the current workingset: http://svn.repoze.org/compoze/trunk/
Question 2: I thought you could only have one index. So installing something extra from pypi seems out. How do people handle that?
Fetch it from pypi and put it in your index.
I'm totally used to buildout and its [versions] list and buildout.dumppickedversions and KGS lists, so I need to reset my thinking on this a little. I need some input :-)
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktomEYACgkQ+gerLs4ltQ43DwCgkPpva6OsQ6xf/en8B2mSVIvv RDgAnRG1pHEreyJmTk/5c6Z2BM4zYV7J =tkVM -----END PGP SIGNATURE-----
On 02/02/2010 10:25 PM, Tres Seaver wrote:
But how to get the right collection of tarballs in there? Is that a by-hand approach? Can you automate it?
You might look at compoze: it is designed to allow fetching sources and building an index based on the current workingset:
Thanks, I'll try that one out. He he, a README in structuredtext (not reST) format if I'm not mistaken. I still like that format :-)
Question 2: I thought you could only have one index. So installing something extra from pypi seems out. How do people handle that?
Fetch it from pypi and put it in your index.
My fear is that I end up with quite a lot of project-specific custom indexes if I want to keep it rebuildable. Multiply that with a quite a number of project releases (so: new indexes) and you're looking at a big bucket of bits. How's that working out in practice for you? If you have a base "product/framework/whatever" (a bunch of libraries, partly your own and partly from pypi), I can see it is useful to have that in such a custom index. But multiple in-house websites will use that *and* add their own extra needed libraries to the mix. => So you need to add a new index for that, duplicating the "product/framework/whatever" release and adding your own extra eggs? Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Programmer/advisor at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets"
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Reinout, Reinout van Rees wrote:
I'd like to do a bit of experimenting with a custom index with only the right packages in just the right versions. Some possible advantages I'm hoping to see for my usecase:
I use chishop for this with success: http://github.com/ask/chishop It emulates PyPI's API, so you can get the packages in there easily enough by adding your chishop url in .pypirc and (using Python 2.6) "python setup.py sdist upload -r my_server".
Question 2: I thought you could only have one index. So installing something extra from pypi seems out. How do people handle that?
I use pip (and pip requirements files) for installation, and pip has both the -i/--index-url option (for replacing PyPI entirely) and the - --extra-index-url option (for layering another index in addition to PyPI). I don't know how one would do that with other tools. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFLaJtU1j/fhc23WEARAlDzAJ9oxm78s54v0BoeyFseSzNRBczACQCgtzrV W7zFxpPR/7SayS0RmWzGsJU= =oS95 -----END PGP SIGNATURE-----
On 2010-02-02, at 1:38 PM, Carl Meyer wrote:
Question 2: I thought you could only have one index. So installing something extra from pypi seems out. How do people handle that?
I use pip (and pip requirements files) for installation, and pip has both the -i/--index-url option (for replacing PyPI entirely) and the - --extra-index-url option (for layering another index in addition to PyPI). I don't know how one would do that with other tools.
You can pass multiple `-i' arguments to pip. $ pip -i http://internal/packages/ -i http://pypi.python.org/simple install internalpackage Not sure what the `--extra-index-url' is for, then. -srid
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sridhar Ratnakumar wrote:
You can pass multiple `-i' arguments to pip.
$ pip -i http://internal/packages/ -i http://pypi.python.org/simple install internalpackage
Not sure what the `--extra-index-url' is for, then.
So you don't have to manually specify the PyPI index URL if you want to add to it instead of replacing it. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFLai+V1j/fhc23WEARAjXqAJ9n9zx8AFeYaxBr7TxNlcYEuNZx/wCfZSbc rJs1YLLEySIpFFybCli+9A8= =V5K0 -----END PGP SIGNATURE-----
At 09:44 PM 2/2/2010 +0100, Reinout van Rees wrote:
Question: how to maintain such a custom index?
Create an Apache-served folder structure such that for each project you want to serve, indexroot/project/ is a directory listing of downloadables. Alternately, make a indexroot/project/index.html for that project, that that links directly to your download targets (which you need not directly serve).
But how to get the right collection of tarballs in there? Is that a by-hand approach? Can you automate it?
Presumably, if you have something particular in mind, you could write a script to do it. Perhaps you could even use buildout to do it in some fashion. ;-)
Question 2: I thought you could only have one index. So installing something extra from pypi seems out.
You can always add a mod_rewrite rule in your index root, that checks for the nonexistence of a projectname, and then automatically redirects to the PyPI index for that page. easy_install will not be confused by this - it will treat the PyPI page as if it had been served by your index. (But of course it will correctly interpret any relative links found as relative to the PyPI page, rather than as relative to the URL it tried to fetch from your index. If you don't understand any of this, don't worry... it Just Works.)
On 02/03/2010 12:31 AM, P.J. Eby wrote:
Question 2: I thought you could only have one index. So installing something extra from pypi seems out.
You can always add a mod_rewrite rule in your index root, that checks for the nonexistence of a projectname, and then automatically redirects to the PyPI index for that page. easy_install will not be confused by this - it will treat the PyPI page as if it had been served by your index. (But of course it will correctly interpret any relative links found as relative to the PyPI page, rather than as relative to the URL it tried to fetch from your index. If you don't understand any of this, don't worry... it Just Works.)
Hey, great idea. For another employer I did something similar which looked up the "missing" package in a local pypi cache/proxy thingy. Redirecting... Hm :-) I *think* I'll use this redirect trick in combination with a local sdistmaker-made simple index [1]. Then I'll just use my customary buildout techniques to get repeatable builds. For the arcgis-wants-it-in-sitepackages-on-windows problem I'll have to fix up something else. I suspect a custom index (with Tres' compoze to manage it) fits that bill just fine. Ok, I'm off setting stuff up. Reinout [1]: http://pypi.python.org/pypi/tha.sdistmaker -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Programmer/advisor at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets"
Reinout van Rees wrote:
Hi all,
Hi Reinout, [..]
Question: how to maintain such a custom index?
http://pypi.python.org/pypi/haufe.eggserver might be another option, Raphael
iirc chrism has a script to build an apache-served folder structure from a bunch of eggs/tarballs, so just having the tarballs is enough. Probably just putting those tarballs in one location is enough anyway.
But how to get the right collection of tarballs in there? Is that a by-hand approach? Can you automate it?
Question 2: I thought you could only have one index. So installing something extra from pypi seems out. How do people handle that?
I'm totally used to buildout and its [versions] list and buildout.dumppickedversions and KGS lists, so I need to reset my thinking on this a little. I need some input :-)
Reinout
participants (6)
-
Carl Meyer -
P.J. Eby -
Raphael Ritz -
Reinout van Rees -
Sridhar Ratnakumar -
Tres Seaver