[buildout] private eggs and egg repositories
Hi All,
I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
Has anyone done this before?
Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?
Also, does anyone have any software that makes a good egg repository?
Does anyone have anything that supports the equivalent of:
python setup.py upload
...but to a local repository?
cheers,
Chris
On 15.11.2008 14:28 Uhr, Chris Withers wrote:
Hi All,
I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
http://pypi.python.org/pypi/haufe.eggserver
-aj
Andreas Jung wrote:
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
So you basically add the egg-server url as a find-links option?
If you have a find-links in ~/.buildout/default.cfg, will that override or be combined with one in a buildout.cfg?
cheers,
Chris
On 15.11.2008 15:26 Uhr, Chris Withers wrote:
Andreas Jung wrote:
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
So you basically add the egg-server url as a find-links option?
Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.
If you have a find-links in ~/.buildout/default.cfg, will that override or be combined with one in a buildout.cfg?
We have the URL of our internal eggserver within our buildout.cfg files.
Andreas
Andreas Jung wrote:
So you basically add the egg-server url as a find-links option?
Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.
Out on interest, how does buildout handle password-protected indexes?
cheers,
Chris
On 17.11.2008 8:11 Uhr, Chris Withers wrote:
Andreas Jung wrote:
So you basically add the egg-server url as a find-links option?
Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.
Out on interest, how does buildout handle password-protected indexes?
Unsupported - we trust our internal and external developers.
Andreas
Andreas Jung wrote:
Out on interest, how does buildout handle password-protected indexes?
Unsupported - we trust our internal and external developers.
Okay, but surely that means you can only expose that packaging server to a very limited set of people? If you can upload and download without restriction, then at most you can only expose it to an intranet of machines that need packages (what do you do if they're on more than one site with no linking vpn?) and even for developers, I guess they must have to be attached to some vpn to upload packages? Still, how do you stop clients that should only be reading packages (which I'm guessing is the majority) from uploading rogue packages?
cheers,
Chris
On 17.11.2008 8:17 Uhr, Chris Withers wrote:
Andreas Jung wrote:
Out on interest, how does buildout handle password-protected indexes?
Unsupported - we trust our internal and external developers.
Okay, but surely that means you can only expose that packaging server to a very limited set of people? If you can upload and download without restriction, then at most you can only expose it to an intranet of machines that need packages (what do you do if they're on more than one site with no linking vpn?) and even for developers, I guess they must have to be attached to some vpn to upload packages? Still, how do you stop clients that should only be reading packages (which I'm guessing is the majority) from uploading rogue packages?
The scope of haufe.eggserver is basically for internal development and deployment only. So here security does not matter. Eggbasket obviously provides support for restricting uploads on a per package basis as PyPI does. However I did not get Eggbasket running.
Andreas
Andreas Jung wrote:
The scope of haufe.eggserver is basically for internal development and deployment only.
Even in that case, how do you shield it from people who could upload rogue packages?
Eggbasket obviously provides support for restricting uploads on a per package basis as PyPI does. However I did not get Eggbasket running.
Which Eggbasket are you referring to? I found one from the grok guys, one which claimed to be a re-implementation of your egg server, and I have a nagging feeling that ChrisM or Tres had something of that name too...
cheers,
Chris
On Mon, Nov 17, 2008 at 8:13 AM, Andreas Jung lists@zopyx.com wrote:
On 17.11.2008 8:11 Uhr, Chris Withers wrote:
Andreas Jung wrote:
So you basically add the egg-server url as a find-links option?
Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.
Out on interest, how does buildout handle password-protected indexes?
Unsupported - we trust our internal and external developers.
Not quite true, you can.
-> use lovely.httprequest and you're set (it does authentication for any pypi compatible server as long as you add a .httpauth file)
Cheers
On Mon, Nov 17, 2008 at 9:04 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
On Mon, Nov 17, 2008 at 8:13 AM, Andreas Jung lists@zopyx.com wrote:
On 17.11.2008 8:11 Uhr, Chris Withers wrote:
Andreas Jung wrote:
So you basically add the egg-server url as a find-links option?
Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.
Out on interest, how does buildout handle password-protected indexes?
Unsupported - we trust our internal and external developers.
Not quite true, you can.
-> use lovely.httprequest and you're set (it does authentication for any pypi compatible server as long as you add a .httpauth file)
Cheers
oups sorry, typo -> lovely.buildouthttp
http://pypi.python.org/pypi/lovely.buildouthttp
Chris Withers wrote:
Andreas Jung wrote:
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
So you basically add the egg-server url as a find-links option?
Actually, find-links won't replace the default location that easy_install/zc.buildout will go to when looking for eggs. It will only *enhance* the list of potentially installable packages.
If you want your buildout or easy_install to talk to a private package index *only*, you have to use the 'index' option (-i for easy_install), e.g.:
[buildout] index = http://your/index # can also be file:///...
If you have a find-links in ~/.buildout/default.cfg, will that override or be combined with one in a buildout.cfg?
It will be overridden if buildout.cfg specifies a find-links option as well. buildout.cfg can append to it, however:
[buildout] ... find-links += some/more/stuff/here
On Nov 15, 2008, at 8:28 AM, Chris Withers wrote:
Hi All,
I'm getting a growing number of customer-specific and non-open- source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
Has anyone done this before?
Yes
Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?
Just set up a flat directory somewhere and tell buildout (or any other setuptools-based software, like easy_install) to add it to its list of find links.
A repository doesn't have to be a formal index.
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in:
extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private
This provides restricted access with authentication using ssh keys.
Jim
-- Jim Fulton Zope Corporation
Jim Fulton wrote:
Has anyone done this before?
Yes
I had a feeling that might be the case ;-)
Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?
Just set up a flat directory somewhere and tell buildout (or any other setuptools-based software, like easy_install) to add it to its list of find links.
Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in:
extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private
This provides restricted access with authentication using ssh keys.
Cool. Out of interest, what ftp server do you use for this and how do you build/upload the eggs to sftp://private.zope.com/private?
cheers,
Chris
On Nov 15, 2008, at 9:22 AM, Chris Withers wrote: ...
Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?
find-links are searched before the index
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys.
Cool. Out of interest, what ftp server do you use for this
The one built in to open ssh.
and how do you build/upload the eggs to sftp://private.zope.com/private?
We build using the setup sdist command and the scp the result to the server.
Jim
-- Jim Fulton Zope Corporation
Jim Fulton wrote:
Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?
find-links are searched before the index
Are all find-links searched, regardless of whether they're in ~/.buildout/default.cfg or a buildout.cfg? (ie: what happens if both have find-links in them?)
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys.
Cool. Out of interest, what ftp server do you use for this
The one built in to open ssh.
So I assume the clients just have to be able to have their public key on the sftp server? Do you use specific keys for sftp or do the clients just have shell access on the server?
and how do you build/upload the eggs to sftp://private.zope.com/private?
We build using the setup sdist command and the scp the result to the server.
*nods*
cheers,
Chris
On Nov 15, 2008, at 9:32 AM, Chris Withers wrote:
Jim Fulton wrote:
Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?
find-links are searched before the index
Are all find-links searched, regardless of whether they're in ~/.buildout/default.cfg or a buildout.cfg? (ie: what happens if both have find-links in them?)
For any option, if an option is defined in default.cfg and buildout.cfg, then the one in buildout.cfg overrides the one in default.cfg. There is no implicit merging.
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys.
Cool. Out of interest, what ftp server do you use for this
The one built in to open ssh.
So I assume the clients just have to be able to have their public key on the sftp server?
Yes. sftp is simply a file-transfer protocol implemented on top of ssh.
Do you use specific keys for sftp
No
or do the clients just have shell access on the server?
They don't necessarily have shell access. ssh lets you restrict what people can do. One downside of sftp is that it lets the user access any file on the system they have permissions for, so, for example, they can read any world-readable file.
Jim
-- Jim Fulton Zope Corporation
Jim Fulton wrote:
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in:
extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private
Incidentally, the docs at http://pypi.python.org/pypi/zc.buildoutsftp contain a rather crucial bug by saying extension rather than extensions ;-)
Also slight annoyance that pycrypto doesn't have a win32 egg :-( How did you get around this?
Chris
Chris Withers wrote:
Also slight annoyance that pycrypto doesn't have a win32 egg :-( How did you get around this?
Well, no answer from Jim or pycrypto's maintainer. I eventually found these:
http://www.voidspace.org.uk/python/modules.shtml#pycrypto
Manually installing first lets zc.buildoutsftp install, however, whenever I run buildout:
- I still get the following warning whenever I run buildout:
Download error: unknown url type: sftp -- Some packages may not be found!
Is that expected?
- pycrypto likes to be pretty verbose:
paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) failed. paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) successful! paramiko.transport: Secsh channel 1 opened. paramiko.transport.sftp: [chan 1] Opened sftp connection (server version 3)
Any way to shut this up?
- most seriously, buildout now seems to hang when it's almost done. I'm guessing this is the same windows-specific behaviour I mentioned before (seems to be to do with spawned processes) although this one is onyl escaped from with a Ctrl-C. Anyone else seen this?
cheers,
Chris
On Nov 19, 2008, at 11:03 AM, Chris Withers wrote:
Chris Withers wrote:
Also slight annoyance that pycrypto doesn't have a win32 egg :-( How did you get around this?
Well, no answer from Jim or pycrypto's maintainer. I eventually found these:
I didn't notice that message.
http://www.voidspace.org.uk/python/modules.shtml#pycrypto
Manually installing first lets zc.buildoutsftp install, however, whenever I run buildout:
- I still get the following warning whenever I run buildout:
Download error: unknown url type: sftp -- Some packages may not be found!
Is that expected?
Yes. It's looking for the latest version of the extension, which it needs to interpret sftp links. :)
- pycrypto likes to be pretty verbose:
paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) failed. paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) successful! paramiko.transport: Secsh channel 1 opened. paramiko.transport.sftp: [chan 1] Opened sftp connection (server version 3)
Any way to shut this up?
You can make buildout quieter, which just raises the overall logging threshold.
It might be interesting to see if the buildoutsftp extension could set the logging level of the paramiko logger to be different but relative to the top-level logging level.
- most seriously, buildout now seems to hang when it's almost done.
I'm guessing this is the same windows-specific behaviour I mentioned before (seems to be to do with spawned processes) although this one is onyl escaped from with a Ctrl-C. Anyone else seen this?
No. But a colleague reports hanging on resource-challenged VMs. There might be a resource issue affecting you.
One lamosity of buildoutsftp is that it makes a separate ssh connection for each fetch it does and it doesn't clean them up. One of these days, I need to go back and implement a connection pool for it.
Jim
-- Jim Fulton Zope Corporation
Jim Fulton wrote:
Well, no answer from Jim or pycrypto's maintainer. I eventually found these:
I didn't notice that message.
No worries, so I take it you guys do just install pycrypto manually on Windows?
- I still get the following warning whenever I run buildout:
Download error: unknown url type: sftp -- Some packages may not be found!
Is that expected?
Yes. It's looking for the latest version of the extension, which it needs to interpret sftp links. :)
That's a shame :-S I guess it's a bit chicken'n'egg, pardon the pun... So, safe to ignore though, right?
It might be interesting to see if the buildoutsftp extension could set the logging level of the paramiko logger to be different but relative to the top-level logging level.
I guess just bumping up the log level on that specific logger should be pretty easy and not really have any side effects.
- most seriously, buildout now seems to hang when it's almost done.
I'm guessing this is the same windows-specific behaviour I mentioned before (seems to be to do with spawned processes) although this one is onyl escaped from with a Ctrl-C. Anyone else seen this?
No. But a colleague reports hanging on resource-challenged VMs. There might be a resource issue affecting you.
It's a "real machine" with not a lot else going on so I'd be surprised :-S
cheers,
Chris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Chris Withers wrote:
Hi All,
I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.
Has anyone done this before?
Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?
Also, does anyone have any software that makes a good egg repository?
Does anyone have anything that supports the equivalent of:
python setup.py upload
...but to a local repository?
The attached script makes an index from a directory full of distributions. We use it to create the indexes on dist.repoze.org, as well as for customer-project-specific indexes.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
Chris Withers wrote:
I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.
Well, the solution I'm currently going with actually turned out to be rather simple:
- upload the eggs into a /eggs folder in the customer's svn repository (maybe clunky, but it's the easiest way I have of getting a password protected, served over https and mirrored between production and dr setuptools-compatible index)
- add the following to the buildouts:
[buildout] extensions=lovely.buildouthttp find-links=https://server/svn/eggs/
- put the appropriate entry in ~/.buildout/.httpauth
Hope this helps someone else in the same situation...
cheers,
Chris
participants (6)
-
Andreas Jung -
Chris Withers -
Jim Fulton -
Philipp von Weitershausen -
Tarek Ziadé -
Tres Seaver