Mailman 3 [buildout] private eggs and egg repositories - Distutils-SIG

[buildout] private eggs and egg repositories

Chris Withers

15 Nov 2008 15 Nov '08

1:28 p.m.

Hi All, I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale. What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it. Has anyone done this before? Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"? Also, does anyone have any software that makes a good egg repository? Does anyone have anything that supports the equivalent of: python setup.py upload ...but to a local repository? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Show replies by date

Andreas Jung

15 Nov 15 Nov

1:31 p.m.

On 15.11.2008 14:28 Uhr, Chris Withers wrote:

...

Hi All,

I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.

What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.

http://pypi.python.org/pypi/haufe.eggserver -aj

Chris Withers

2:26 p.m.

Andreas Jung wrote:

...

...
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.

http://pypi.python.org/pypi/haufe.eggserver

So you basically add the egg-server url as a find-links option? If you have a find-links in ~/.buildout/default.cfg, will that override or be combined with one in a buildout.cfg? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Andreas Jung

2:57 p.m.

On 15.11.2008 15:26 Uhr, Chris Withers wrote:

...

Andreas Jung wrote:

...
...
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.

http://pypi.python.org/pypi/haufe.eggserver

So you basically add the egg-server url as a find-links option?

Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.

...

If you have a find-links in ~/.buildout/default.cfg, will that override or be combined with one in a buildout.cfg?

We have the URL of our internal eggserver within our buildout.cfg files. Andreas

Chris Withers

17 Nov 17 Nov

7:11 a.m.

Andreas Jung wrote:

...

...
So you basically add the egg-server url as a find-links option?

Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.

Out on interest, how does buildout handle password-protected indexes? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Andreas Jung

7:13 a.m.

On 17.11.2008 8:11 Uhr, Chris Withers wrote:

...

Andreas Jung wrote:

...
...
So you basically add the egg-server url as a find-links option?

Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.

Out on interest, how does buildout handle password-protected indexes?

Unsupported - we trust our internal and external developers. Andreas

Chris Withers

7:17 a.m.

Andreas Jung wrote:

...

...
Out on interest, how does buildout handle password-protected indexes?

Unsupported - we trust our internal and external developers.

Okay, but surely that means you can only expose that packaging server to a very limited set of people? If you can upload and download without restriction, then at most you can only expose it to an intranet of machines that need packages (what do you do if they're on more than one site with no linking vpn?) and even for developers, I guess they must have to be attached to some vpn to upload packages? Still, how do you stop clients that should only be reading packages (which I'm guessing is the majority) from uploading rogue packages? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Andreas Jung

7:24 a.m.

On 17.11.2008 8:17 Uhr, Chris Withers wrote:

...

Andreas Jung wrote:

...
...
Out on interest, how does buildout handle password-protected indexes?

Unsupported - we trust our internal and external developers.

Okay, but surely that means you can only expose that packaging server to a very limited set of people? If you can upload and download without restriction, then at most you can only expose it to an intranet of machines that need packages (what do you do if they're on more than one site with no linking vpn?) and even for developers, I guess they must have to be attached to some vpn to upload packages? Still, how do you stop clients that should only be reading packages (which I'm guessing is the majority) from uploading rogue packages?

The scope of haufe.eggserver is basically for internal development and deployment only. So here security does not matter. Eggbasket obviously provides support for restricting uploads on a per package basis as PyPI does. However I did not get Eggbasket running. Andreas

Chris Withers

7:31 a.m.

Andreas Jung wrote:

...

The scope of haufe.eggserver is basically for internal development and deployment only.

Even in that case, how do you shield it from people who could upload rogue packages?

...

Eggbasket obviously provides support for restricting uploads on a per package basis as PyPI does. However I did not get Eggbasket running.

Which Eggbasket are you referring to? I found one from the grok guys, one which claimed to be a re-implementation of your egg server, and I have a nagging feeling that ChrisM or Tres had something of that name too... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Tarek Ziadé

8:04 a.m.

On Mon, Nov 17, 2008 at 8:13 AM, Andreas Jung wrote:

...

On 17.11.2008 8:11 Uhr, Chris Withers wrote:

...
Andreas Jung wrote:

...
...
So you basically add the egg-server url as a find-links option?

Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.

Out on interest, how does buildout handle password-protected indexes?

Unsupported - we trust our internal and external developers.

Not quite true, you can. -> use lovely.httprequest and you're set (it does authentication for any pypi compatible server as long as you add a .httpauth file) Cheers

Tarek Ziadé

8:05 a.m.

On Mon, Nov 17, 2008 at 9:04 AM, Tarek Ziadé wrote:

...

On Mon, Nov 17, 2008 at 8:13 AM, Andreas Jung wrote:

...
On 17.11.2008 8:11 Uhr, Chris Withers wrote:

...
Andreas Jung wrote:

...
...
So you basically add the egg-server url as a find-links option?

Basically yes, we create an index page based on the content right now through cron - I will implement this functionality into haufe.eggserver soon.

Out on interest, how does buildout handle password-protected indexes?

Unsupported - we trust our internal and external developers.

Not quite true, you can.

-> use lovely.httprequest and you're set (it does authentication for any pypi compatible server as long as you add a .httpauth file)

Cheers

oups sorry, typo -> lovely.buildouthttp http://pypi.python.org/pypi/lovely.buildouthttp -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

Philipp von Weitershausen

15 Nov 15 Nov

3:11 p.m.

Chris Withers wrote:

...

Andreas Jung wrote:

...
...
What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.

http://pypi.python.org/pypi/haufe.eggserver

So you basically add the egg-server url as a find-links option?

Actually, find-links won't replace the default location that easy_install/zc.buildout will go to when looking for eggs. It will only *enhance* the list of potentially installable packages. If you want your buildout or easy_install to talk to a private package index *only*, you have to use the 'index' option (-i for easy_install), e.g.: [buildout] index = http://your/index # can also be file:///...

...

If you have a find-links in ~/.buildout/default.cfg, will that override or be combined with one in a buildout.cfg?

It will be overridden if buildout.cfg specifies a find-links option as well. buildout.cfg can append to it, however: [buildout] ... find-links += some/more/stuff/here

Jim Fulton

2:20 p.m.

On Nov 15, 2008, at 8:28 AM, Chris Withers wrote:

...

Hi All,

I'm getting a growing number of customer-specific and non-open- source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.

What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.

Has anyone done this before?

Yes

...

Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?

Just set up a flat directory somewhere and tell buildout (or any other setuptools-based software, like easy_install) to add it to its list of find links. A repository doesn't have to be a formal index. We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys. Jim -- Jim Fulton Zope Corporation

Chris Withers

2:22 p.m.

Jim Fulton wrote:

...

...
Has anyone done this before?

Yes

I had a feeling that might be the case ;-)

...

...
Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?

Just set up a flat directory somewhere and tell buildout (or any other setuptools-based software, like easy_install) to add it to its list of find links.

Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?

...

We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in:

extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private

This provides restricted access with authentication using ssh keys.

Cool. Out of interest, what ftp server do you use for this and how do you build/upload the eggs to sftp://private.zope.com/private? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Jim Fulton

2:26 p.m.

On Nov 15, 2008, at 9:22 AM, Chris Withers wrote: ...

...

Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?

find-links are searched before the index

...

...
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys.

Cool. Out of interest, what ftp server do you use for this

The one built in to open ssh.

...

and how do you build/upload the eggs to sftp://private.zope.com/private?

We build using the setup sdist command and the scp the result to the server. Jim -- Jim Fulton Zope Corporation

Chris Withers

2:32 p.m.

Jim Fulton wrote:

...

...
Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used?

find-links are searched before the index

Are all find-links searched, regardless of whether they're in ~/.buildout/default.cfg or a buildout.cfg? (ie: what happens if both have find-links in them?)

...

...
...
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys.

Cool. Out of interest, what ftp server do you use for this

The one built in to open ssh.

So I assume the clients just have to be able to have their public key on the sftp server? Do you use specific keys for sftp or do the clients just have shell access on the server?

...

...
and how do you build/upload the eggs to sftp://private.zope.com/private?

We build using the setup sdist command and the scp the result to the server.

*nods* cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Jim Fulton

2:43 p.m.

On Nov 15, 2008, at 9:32 AM, Chris Withers wrote:

...

Jim Fulton wrote:

...
...
Does find-links take preference over the repository? I thought *all* find-links *and* the repository were checked and the best match used? find-links are searched before the index

Are all find-links searched, regardless of whether they're in ~/.buildout/default.cfg or a buildout.cfg? (ie: what happens if both have find-links in them?)

For any option, if an option is defined in default.cfg and buildout.cfg, then the one in buildout.cfg overrides the one in default.cfg. There is no implicit merging.

...

...
...
...
We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in: extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private This provides restricted access with authentication using ssh keys.

Cool. Out of interest, what ftp server do you use for this The one built in to open ssh.

So I assume the clients just have to be able to have their public key on the sftp server?

Yes. sftp is simply a file-transfer protocol implemented on top of ssh.

...

Do you use specific keys for sftp

...

or do the clients just have shell access on the server?

They don't necessarily have shell access. ssh lets you restrict what people can do. One downside of sftp is that it lets the user access any file on the system they have permissions for, so, for example, they can read any world-readable file. Jim -- Jim Fulton Zope Corporation

Chris Withers

17 Nov 17 Nov

5:34 p.m.

Jim Fulton wrote:

...

We simply set up an sftp server and point buildout at it using the buildoutsftp buildout extension and sftp urls, as in:

extensions = zc.buildoutsftp find-links = sftp://private.zope.com/private

Incidentally, the docs at http://pypi.python.org/pypi/zc.buildoutsftp contain a rather crucial bug by saying extension rather than extensions ;-) Also slight annoyance that pycrypto doesn't have a win32 egg :-( How did you get around this? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Chris Withers

19 Nov 19 Nov

4:03 p.m.

Chris Withers wrote:

...

Also slight annoyance that pycrypto doesn't have a win32 egg :-( How did you get around this?

Well, no answer from Jim or pycrypto's maintainer. I eventually found these: http://www.voidspace.org.uk/python/modules.shtml#pycrypto Manually installing first lets zc.buildoutsftp install, however, whenever I run buildout: - I still get the following warning whenever I run buildout: Download error: unknown url type: sftp -- Some packages may not be found! Is that expected? - pycrypto likes to be pretty verbose: paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) failed. paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) successful! paramiko.transport: Secsh channel 1 opened. paramiko.transport.sftp: [chan 1] Opened sftp connection (server version 3) Any way to shut this up? - most seriously, buildout now seems to hang when it's almost done. I'm guessing this is the same windows-specific behaviour I mentioned before (seems to be to do with spawned processes) although this one is onyl escaped from with a Ctrl-C. Anyone else seen this? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Jim Fulton

5:49 p.m.

On Nov 19, 2008, at 11:03 AM, Chris Withers wrote:

...

Chris Withers wrote:

...
Also slight annoyance that pycrypto doesn't have a win32 egg :-( How did you get around this?

Well, no answer from Jim or pycrypto's maintainer. I eventually found these:

I didn't notice that message.

...

http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Manually installing first lets zc.buildoutsftp install, however, whenever I run buildout:

- I still get the following warning whenever I run buildout:

Download error: unknown url type: sftp -- Some packages may not be found!

Is that expected?

Yes. It's looking for the latest version of the extension, which it needs to interpret sftp links. :)

...

- pycrypto likes to be pretty verbose:

paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) failed. paramiko.transport: Connected (version 2.0, client OpenSSH_3.9p1) paramiko.transport: Authentication (publickey) successful! paramiko.transport: Secsh channel 1 opened. paramiko.transport.sftp: [chan 1] Opened sftp connection (server version 3)

Any way to shut this up?

You can make buildout quieter, which just raises the overall logging threshold. It might be interesting to see if the buildoutsftp extension could set the logging level of the paramiko logger to be different but relative to the top-level logging level.

...

- most seriously, buildout now seems to hang when it's almost done. I'm guessing this is the same windows-specific behaviour I mentioned before (seems to be to do with spawned processes) although this one is onyl escaped from with a Ctrl-C. Anyone else seen this?

No. But a colleague reports hanging on resource-challenged VMs. There might be a resource issue affecting you. One lamosity of buildoutsftp is that it makes a separate ssh connection for each fetch it does and it doesn't clean them up. One of these days, I need to go back and implement a connection pool for it. Jim -- Jim Fulton Zope Corporation

Chris Withers

5:59 p.m.

Jim Fulton wrote:

...

...
Well, no answer from Jim or pycrypto's maintainer. I eventually found these:

I didn't notice that message.

No worries, so I take it you guys do just install pycrypto manually on Windows?

...

...
- I still get the following warning whenever I run buildout:

Download error: unknown url type: sftp -- Some packages may not be found!

Is that expected?

Yes. It's looking for the latest version of the extension, which it needs to interpret sftp links. :)

That's a shame :-S I guess it's a bit chicken'n'egg, pardon the pun... So, safe to ignore though, right?

...

It might be interesting to see if the buildoutsftp extension could set the logging level of the paramiko logger to be different but relative to the top-level logging level.

I guess just bumping up the log level on that specific logger should be pretty easy and not really have any side effects.

...

...
- most seriously, buildout now seems to hang when it's almost done. I'm guessing this is the same windows-specific behaviour I mentioned before (seems to be to do with spawned processes) although this one is onyl escaped from with a Ctrl-C. Anyone else seen this?

No. But a colleague reports hanging on resource-challenged VMs. There might be a resource issue affecting you.

It's a "real machine" with not a lot else going on so I'd be surprised :-S cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Tres Seaver

15 Nov 15 Nov

8:11 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:

...

Hi All,

I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.

What I'd like to do is have a "local egg repository" that I can put these eggs in and then just point buildout at it.

Has anyone done this before?

Failing that, is there any way in buildout to say "use this egg repository in favour of pypi, and only look at pypi if the egg isn't found in the local repository"?

Also, does anyone have any software that makes a good egg repository?

Does anyone have anything that supports the equivalent of:

python setup.py upload

...but to a local repository?

The attached script makes an index from a directory full of distributions. We use it to create the indexes on dist.repoze.org, as well as for customer-project-specific indexes. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJHyzZ+gerLs4ltQ4RAj6uAJwI9460ChTKOaU1eYd+u/sverl8qACfWUjB ofH4v/hcrFECvuf/19XFFsg= =dulB -----END PGP SIGNATURE-----

Chris Withers

20 Nov 20 Nov

8:06 a.m.

New subject: [buildout] private eggs and egg repositories - my current solution

Chris Withers wrote:

...

I'm getting a growing number of customer-specific and non-open-source eggs. I used to just put these in as svn:externals and then add the path in as a develop egg, however, I'm not sure this is going to scale.

Well, the solution I'm currently going with actually turned out to be rather simple: - upload the eggs into a /eggs folder in the customer's svn repository (maybe clunky, but it's the easiest way I have of getting a password protected, served over https and mirrored between production and dr setuptools-compatible index) - add the following to the buildouts: [buildout] extensions=lovely.buildouthttp find-links=https://server/svn/eggs/ - put the appropriate entry in ~/.buildout/.httpauth Hope this helps someone else in the same situation... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

5635

Age (days ago)

5640

Last active (days ago)

List overview

Download

22 comments

6 participants

participants (6)

Andreas Jung
Chris Withers
Jim Fulton
Philipp von Weitershausen
Tarek Ziadé
Tres Seaver

[buildout] private eggs and egg repositories

tags

participants (6)