Mailman 3 [buildout] extending a .cfg supplied by another package - Distutils-SIG

newer
How to customize egg_info behavior

[buildout] extending a .cfg supplied by another package

Chris Withers

12 May 2010 12 May '10

2:17 p.m.

Hi All, I'd like to be able to do: [buildout] extends=some.package:aconfig.cfg ...kinda like you can with templates in BFG, which I believe uses pkg_tools to do the hard work. What're the chances of that happening? cheers, Chris

Show replies by date

Jim Fulton

12 May 12 May

9:51 p.m.

On Wed, May 12, 2010 at 8:17 AM, Chris Withers wrote:

...

Hi All,

I'd like to be able to do:

[buildout] extends=some.package:aconfig.cfg

That's an interesting idea.

...

...kinda like you can with templates in BFG, which I believe uses pkg_tools to do the hard work.

What're the chances of that happening?

Low, in the short term. A good first step would be to work out a somewhat detailed proposal including: - Descriptions of specific situations where this would solve a problem, with specific examples of how the proposed meachanism would be applied. - A syntax proposal. It has to address distinguishing file paths from urls from these new things. I suspect these should be expressed as URLs, e.g.: project://some.project/aconfig.cfg I'll note that I suspect you can do this now using a buildout extension that installs a new URL handler in urtllib2. Jim -- Jim Fulton

Wolfgang Schnerring

14 May 14 May

9:47 a.m.

* Jim Fulton  [2010-05-12 21:51]:
> Chris Withers  wrote:
> > I'd like to be able to do:
> > [buildout]
> > extends=some.package:aconfig.cfg
> - Descriptions of specific situations where this would solve a
>   problem, with specific examples of how the proposed meachanism would
>   be applied.

I don't know what Chris had in mind, but one use case for me is
versioning the buildout configuration and keeping it in sync with the
package I'm buildouting. For example, say the way a package is deployed
changes from its version 1.5 to 2.0, so some buildout parts need to work
differently than before, e. g. because the appserver needs to be
configured differently now.

I'm not aware of a good way to deal with that currently, other than
using an SCM tag for the buildout config files. But this is cumbersome,
as a) it's an additional thing you need to do when cutting a release and
b) you need to take care to first figure out and then check out the
matching buildout config version for the package version you want to
deploy.

If I could put the relevant buildout configuration into the egg itself,
that would handle this issue nicely.

> - A syntax proposal.  It has to address distinguishing file paths from
>   urls from these new things. I suspect these should be expressed as
>   URLs, e.g.:
>      project://some.project/aconfig.cfg

I'd propose egg://some.dotted.name/config.cfg

> I'll note that I suspect you can do this now using a buildout
> extension that installs a new URL handler in urtllib2.

That's an interesting approach. I'd definitely want to use buildout's
egg version handling/pinning for the "extends egg", but yeah, maybe
that'd be possible that way, too.

Wolfgang

Tres Seaver

4:10 p.m.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wolfgang Schnerring wrote:
> * Jim Fulton  [2010-05-12 21:51]:
>> Chris Withers  wrote:
>>> I'd like to be able to do:
>>> [buildout]
>>> extends=some.package:aconfig.cfg
>> - Descriptions of specific situations where this would solve a
>>   problem, with specific examples of how the proposed meachanism would
>>   be applied.
> 
> I don't know what Chris had in mind, but one use case for me is
> versioning the buildout configuration and keeping it in sync with the
> package I'm buildouting. For example, say the way a package is deployed
> changes from its version 1.5 to 2.0, so some buildout parts need to work
> differently than before, e. g. because the appserver needs to be
> configured differently now.
> 
> I'm not aware of a good way to deal with that currently, other than
> using an SCM tag for the buildout config files. But this is cumbersome,
> as a) it's an additional thing you need to do when cutting a release and
> b) you need to take care to first figure out and then check out the
> matching buildout config version for the package version you want to
> deploy.
> 
> If I could put the relevant buildout configuration into the egg itself,
> that would handle this issue nicely.
> 
>> - A syntax proposal.  It has to address distinguishing file paths from
>>   urls from these new things. I suspect these should be expressed as
>>   URLs, e.g.:
>>      project://some.project/aconfig.cfg
> 
> I'd propose egg://some.dotted.name/config.cfg

I think egg:some.dotted.name#config.cfg would be clearer:  the
"hierarchical" URL form is misleading.  We could even adopt the
convention that such config files needed to be registered as entry
points under some given section, e.g.:

  setup(
      ...
      entry_points = { 'zc.buildout.cfg':
       {'buildout.cfg':  './buildout.cfg',
        'develop.cfg': './develop.cfg'},
      }
  )

perhaps with a fallback to searching in the root directory of the egg
for unregistered files.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver@palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvtWcwACgkQ+gerLs4ltQ42wgCgmUfJoETb3D+Ogckby4WIVols
N+wAn0I94tV91vB821sTOdtoxjDxw4M/
=0jjq
-----END PGP SIGNATURE-----

Wolfgang Schnerring

15 May 15 May

11:04 a.m.

* Tres Seaver [2010-05-14 16:10]:

...

Wolfgang Schnerring wrote:

...
* Jim Fulton [2010-05-12 21:51]:

...
- A syntax proposal. It has to address distinguishing file paths from urls from these new things. I suspect these should be expressed as URLs, e.g.: project://some.project/aconfig.cfg

I'd propose egg://some.dotted.name/config.cfg

I think egg:some.dotted.name#config.cfg would be clearer: the "hierarchical" URL form is misleading.

Agreed, that makes sense to me.

...

We could even adopt the convention that such config files needed to be registered as entry points under some given section, e.g.:

Using entry points sounds good. One could even define a default one (named... "default", maybe? ;), so you could just reference egg://some.dotted.name without a #filename.cfg modifier. Wolfgang

Chris Withers

17 May 17 May

12:58 p.m.

New subject: [buildout] extending a .cfg supplied by another package

Tres Seaver wrote:

...

...
...
project://some.project/aconfig.cfg

I'd propose egg://some.dotted.name/config.cfg

I think egg:some.dotted.name#config.cfg would be clearer: the "hierarchical" URL form is misleading.

How so? The .cfg could be a file in a folder in the egg, same as a template for bfg...

...

We could even adopt the convention that such config files needed to be registered as entry points under some given section, e.g.:

I don't see what purpose this would serve. I was ancticipating them being just like any other package_resource... Chris

Tres Seaver

5:37 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:

...

Tres Seaver wrote:

...
...
...
project://some.project/aconfig.cfg

I'd propose egg://some.dotted.name/config.cfg I think egg:some.dotted.name#config.cfg would be clearer: the "hierarchical" URL form is misleading.

How so? The .cfg could be a file in a folder in the egg, same as a template for bfg...

The hierarchical form URLs (containing ';//') imply an "authority" section (aka "nethost").

...

...
We could even adopt the convention that such config files needed to be registered as entry points under some given section, e.g.:

I don't see what purpose this would serve. I was ancticipating them being just like any other package_resource...

Unlike other hose config files are almost never normal "package data" -- they are practically always kept at the top level of the project checkout, along with other "packaging metdata" like setup.py, README.txt, etc. I just checked, and sure enough, *none* of those files are present in "installed" eggs, which makes the whole idea a good deal less practical / valuable: neither your path-based syntax nor my entry point is going to have the file available. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvxYqAACgkQ+gerLs4ltQ7A/QCfaLJ7BlXHEq+eYif15MB7kk0S tYAAoIJKiSKcxoSEyZvKGd1uX8DeSKmF =FDY5 -----END PGP SIGNATURE-----

Chris Withers

6:02 p.m.

New subject: [buildout] extending a .cfg supplied by another package

Tres Seaver wrote:

...

Unlike other hose config files are almost never normal "package data" -- they are practically always kept at the top level of the project checkout, along with other "packaging metdata" like setup.py, README.txt, etc. I just checked, and sure enough, *none* of those files are present in "installed" eggs, which makes the whole idea a good deal less practical / valuable: neither your path-based syntax nor my entry point is going to have the file available.

Indeed, this is an annoying bug I bumped into before: http://mail.python.org/pipermail/distutils-sig/2008-August/009900.html ...I remember it being an insane interaction of add and remvoe lists somewhere deep in distutils :-/ Still, if the .cfg was in a folder, then it should be possible to treat it as a normal package_resource, which would work fine with my pattern, where the root buildout.cfg is for that package only, and anything designed to be extended (generally a base.cfg and a versions.cfg) could be in a subfolder. When I get a chance (which may be never!) I'm going to have a go at doing my idea (enhanced by Jim's suggestion of urllib2 url handlers!) as a buildout extension. Thoughts welcome! cheers, Chris

Chris Withers

14 May 14 May

10:56 a.m.

Jim Fulton wrote:

...

On Wed, May 12, 2010 at 8:17 AM, Chris Withers wrote:

...
Hi All,

I'd like to be able to do:

[buildout] extends=some.package:aconfig.cfg

That's an interesting idea.

Yeah, I finally took a look at the package_resources part of setuptools ;-)

...

Low, in the short term. A good first step would be to work out a somewhat detailed proposal including:

- Descriptions of specific situations where this would solve a problem,

So, we have a bunch of packages that all depend on each other. We group all the boilerplate into one package, including things like base buildout .cfg files and the like. We currently have to specify http urls to tags on our svn server to extend these .cfg's, it would be better if we could just say "extend this configuration from that package"

...

with specific examples of how the proposed meachanism would be applied.

The tricky bit from my perspective is that some.package would need to be obtained before any parts had been run, but I guess the same is true of repice packages and buildout extension packages, so I'm guessing this problem has been solved already?

...

- A syntax proposal. It has to address distinguishing file paths from urls from these new things. I suspect these should be expressed as URLs, e.g.:

project://some.project/aconfig.cfg

I'll note that I suspect you can do this now using a buildout extension that installs a new URL handler in urtllib2.

That's an interesting idea! I'll certainly have a look... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Christian Zagrodnick

30 Jul 30 Jul

10:17 a.m.

On 2010-05-12 21:51:19 +0200, Jim Fulton said:

...

On Wed, May 12, 2010 at 8:17 AM, Chris Withers wrote:

...
Hi All,

I'd like to be able to do:

[buildout] extends=some.package:aconfig.cfg

That's an interesting idea.

...
...kinda like you can with templates in BFG, which I believe uses pkg_tools to do the hard work.

What're the chances of that happening?

Low, in the short term. A good first step would be to work out a somewhat detailed proposal including:

- Descriptions of specific situations where this would solve a problem, with specific examples of how the proposed meachanism would be applied.

- A syntax proposal. It has to address distinguishing file paths from urls from these new things. I suspect these should be expressed as URLs, e.g.:

project://some.project/aconfig.cfg

I propose egg://<requirement string>/<path in egg>. For example egg://some.package=0.6/foo/file.cfg would basically return pkg_resources.resource_string('some.package', 'foo/file.cfg').

...

I'll note that I suspect you can do this now using a buildout extension that installs a new URL handler in urtllib2.

That doesn't work for extends= because extends evaluated first to build the composed configuration. Extensions are loaded after that. I don't see a useful way to not integrating the egg://-handler to buildout directly. Are there chances this will integrated to buildout when we try that on a branch? Regards, -- Christian Zagrodnick · cz@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 4 · fax +49 345 1229889 1 Zope and Plone consulting and development

Christian Zagrodnick

2:44 p.m.

On 2010-07-30 10:17:09 +0200, Christian Zagrodnick said:

...

On 2010-05-12 21:51:19 +0200, Jim Fulton said:

...
On Wed, May 12, 2010 at 8:17 AM, Chris Withers wrote:

...
Hi All,

...
...
I'd like to be able to do:

...
...
[buildout] extends=some.package:aconfig.cfg

...
That's an interesting idea.

...
...
...kinda like you can with templates in BFG, which I believe uses pkg_tools to do the hard work.

...
...
What're the chances of that happening?

...
Low, in the short term. A good first step would be to work out a somewhat detailed proposal including:

...
- Descriptions of specific situations where this would solve a problem, with specific examples of how the proposed meachanism would be applied.

...
- A syntax proposal. It has to address distinguishing file paths from urls from these new things. I suspect these should be expressed as URLs, e.g.:

...
project://some.project/aconfig.cfg

I propose egg://<requirement string>/<path in egg>.

For example egg://some.package=0.6/foo/file.cfg would basically return pkg_resources.resource_string('some.package', 'foo/file.cfg').

...
I'll note that I suspect you can do this now using a buildout extension that installs a new URL handler in urtllib2.

That doesn't work for extends= because extends evaluated first to build the composed configuration. Extensions are loaded after that.

I don't see a useful way to not integrating the egg://-handler to buildout directly.

Are there chances this will integrated to buildout when we try that on a branch?

Actually I figured that an egg://-handler doesn't help with versions very much. I try something else and let you know when there is something interesting. -- Christian Zagrodnick · cz@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 4 · fax +49 345 1229889 1 Zope and Plone consulting and development

Jim Fulton

3:33 p.m.

On Fri, Jul 30, 2010 at 4:17 AM, Christian Zagrodnick wrote:

...

On 2010-05-12 21:51:19 +0200, Jim Fulton said:

...
On Wed, May 12, 2010 at 8:17 AM, Chris Withers ...

...

I don't see a useful way to not integrating the egg://-handler to buildout directly.

I have no confidence in my ability to interpret that double negative.

...

Are there chances this will integrated to buildout when we try that on a branch?

Not sure what you mean. If you get this working on a branch, I'm happy to review and, if agreeable, merge it. Note that I don't intend to do anything until after 1.5 lands. On Fri, Jul 30, 2010 at 8:44 AM, Christian Zagrodnick wrote: ...

...

Actually I figured that an egg://-handler doesn't help with versions very much. I try something else and let you know when there is something interesting.

I don't know what you're trying to say here. Jim -- Jim Fulton

5016

Age (days ago)

5095

Last active (days ago)

List overview

Download

11 comments

5 participants

participants (5)

Chris Withers
Christian Zagrodnick
Jim Fulton
Tres Seaver
Wolfgang Schnerring