Setuptools: omit namespace package directories?
Hi, when using namespace packages, the corresponding package directories and __init__.py files must physically exist in the source tree, even though they can't, by definition of a namespace package, contain anything other than subordinate directories and a fixed stanza of Python code, resp. This gets annoying, especially when using nested namespaces. For example, when writing a buildout recipe, the cleanest thing is to use several namespaces, like zc.recipe.egg does. However, you end up jumping through essentially empty directories all the time while working on the source. There's evidence that this problem is a real one: buildout recipes registered with PyPI tend to omit the "recipe" part at the expense of namespace hygiene and PEP 8 compliance, for example gocept.download or buildout_script. Since there's basically not information in those boilerplate directories and __init__.py files that couldn't be inferred from a keyword parameter to setup(), would it be a sensible feature request that setuptools do without the physical namespace directories in the future? -- Thomas
On Feb 8, 2007, at 11:05 AM, Thomas Lotze wrote:
Hi,
when using namespace packages, the corresponding package directories and __init__.py files must physically exist in the source tree, even though they can't, by definition of a namespace package, contain anything other than subordinate directories and a fixed stanza of Python code, resp.
This gets annoying, especially when using nested namespaces. For example, when writing a buildout recipe, the cleanest thing is to use several namespaces, like zc.recipe.egg does.
I don't agree that this is cleanest. I made a mistake by introducing the recipe namespace. New recipes I write won't use this. "Flat is better than nested." Also, for smaller projects, I'm avoiding intermediate src directories to avoid the annoying nesting. (I'm a little uneasy about this, but can't see a serious downside other than setup being importable in develop mode.)
However, you end up jumping through essentially empty directories all the time while working on the source.
Yup
There's evidence that this problem is a real one: buildout recipes registered with PyPI tend to omit the "recipe" part at the expense of namespace hygiene and PEP 8 compliance, for example gocept.download or buildout_script.
IMO, the only needed hygiene is the top-level namespace. (Maybe a huge organization would need sub namespaces, but we don't.)
Since there's basically not information in those boilerplate directories and __init__.py files that couldn't be inferred from a keyword parameter to setup(), would it be a sensible feature request that setuptools do without the physical namespace directories in the future?
I would love to see this. I haven't been able to think of a good way to do it reliably, especially considering the important use case of develop eggs, which want to run directly from a checkout. Again, for smaller project, the pain can be mitigated by skipping the src directory and avoiding nested namespaces. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
when writing a buildout recipe, the cleanest thing is to use several namespaces, like zc.recipe.egg does.
I don't agree that this is cleanest. I made a mistake by introducing the recipe namespace. New recipes I write won't use this. "Flat is better than nested."
To comment on buildout specifically, which I had only brought up as an example, though: While I agree with "flat is better than nested" most of the time, I don't when it comes to buildout recipes. It's the nature of buildout that some recipes deal with software that has a name of its own, or with general concepts. You wouldn't want to call a recipe for installing libfoo something like zc.foo or zc.libfoo as everybody would think it's your libfoo binding, and a name such as gocept.download is a bit too non-descriptive for my taste. So there should be a hint at buildout or recipes in the (dotted) name. Personally, I prefer a recipe namespace for recipes that deal with third-party software or generalities, and a recipe subpackage for one's own projects. Anything else feels rather non-elegant to me, but this might just be a matter of taste. What's your preferred naming scheme for new recipes?
Also, for smaller projects, I'm avoiding intermediate src directories to avoid the annoying nesting. (I'm a little uneasy about this, but can't see a serious downside other than setup being importable in develop mode.)
I do this all the time myself.
IMO, the only needed hygiene is the top-level namespace. (Maybe a huge organization would need sub namespaces, but we don't.)
I don't think it's a matter of running out of names when you have to name too many unrelated things, but one of finding concise and descriptive names for related or generic stuff. Descriptive and unique names in a flat namespace tend to grow long and in some cases, unreadable unless you violate PEP 8 and stick in an underscore or two. For example, I'm writing a bunch of buildout recipes for Apache stuff, and I'm not going to choose names such as tl.apache or tl.buildoutmodpython. Without introducing a namespace for my buildout recipes, buildout_apache and buildout_modpython is the least annoying naming scheme I can come up with. (BTW: Did I just miss something, or do recipes for these really not exist yet?)
Since there's basically not information in those boilerplate directories and __init__.py files that couldn't be inferred from a keyword parameter to setup(), would it be a sensible feature request that setuptools do without the physical namespace directories in the future?
I would love to see this. I haven't been able to think of a good way to do it reliably, especially considering the important use case of develop eggs, which want to run directly from a checkout.
I'd have to look closer at what eggs do to Python's importing mechanism, but naively I'd hope that if it's possible to find a subpackage in one of several eggs that all hold parts of its top-level package, it should be possible to find it in some other place. Installing a develop egg would then have to mean more than placing a link file somewhere, possibly a stub egg for the namespace package. Maybe this is day-dreaming, though - I'll have to study setuptools first. -- Thomas
On Feb 9, 2007, at 3:08 AM, Thomas Lotze wrote:
Jim Fulton wrote:
when writing a buildout recipe, the cleanest thing is to use several namespaces, like zc.recipe.egg does.
I don't agree that this is cleanest. I made a mistake by introducing the recipe namespace. New recipes I write won't use this. "Flat is better than nested."
To comment on buildout specifically, which I had only brought up as an example, though:
While I agree with "flat is better than nested" most of the time, I don't when it comes to buildout recipes. It's the nature of buildout that some recipes deal with software that has a name of its own, or with general concepts. You wouldn't want to call a recipe for installing libfoo something like zc.foo or zc.libfoo
Right, I'd call it zc.libfoorecipe.
as everybody would think it's your libfoo binding, and a name such as gocept.download is a bit too non-descriptive for my taste. So there should be a hint at buildout or recipes in the (dotted) name. Personally, I prefer a recipe namespace for recipes that deal with third-party software or generalities, and a recipe subpackage for one's own projects. Anything else feels rather non-elegant to me, but this might just be a matter of taste. What's your preferred naming scheme for new recipes?
namespace.foorecipe. Abstractly, namespace.recipe.foo is appealing except for the directory management issue. But the directory management issue is a significant one. ...
IMO, the only needed hygiene is the top-level namespace. (Maybe a huge organization would need sub namespaces, but we don't.)
I don't think it's a matter of running out of names when you have to name too many unrelated things,
Maybe not for you, but it is for me. In particular, I want to be able to name something without worrying about whether someone else has taken the name. The zc namespace protects me from that. I only need one level of namespace for that.
but one of finding concise and descriptive names for related or generic stuff. Descriptive and unique names in a flat namespace tend to grow long and in some cases, unreadable unless you violate PEP 8 and stick in an underscore or two.
For example, I'm writing a bunch of buildout recipes for Apache stuff, and I'm not going to choose names such as tl.apache or tl.buildoutmodpython. Without introducing a namespace for my buildout recipes, buildout_apache and buildout_modpython is the least annoying naming scheme I can come up with.
Why not just tl.apacherecipes, which could contain all of your apache recipes.
(BTW: Did I just miss something, or do recipes for these really not exist yet?)
I don't know, are you done yet? ;) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Abstractly, namespace.recipe.foo is appealing except for the directory management issue. But the directory management issue is a significant one.
And it is a technical one, so I think it should be fixed technically, in setuptools, instead of by bending our usage pattern around it.
Maybe not for you, but it is for me. In particular, I want to be able to name something without worrying about whether someone else has taken the name. The zc namespace protects me from that. I only need one level of namespace for that.
Ah, concerning the top level, it is of course an issue of namespace clutter for me as well. And one of modesty and responsibility: for example, I would neither want nor dare publish a PDF library as a top-level 'pdf' package until it has earned a reputation as _the_ standard Python PDF library. What I actually meant was that I'm personally not too afraid of polluting a namespace of my own but that organizing a few but related packages seems to be the bigger problem there.
Why not just tl.apacherecipes, which could contain all of your apache recipes.
I used to shy away from that name because of two issues that are probably very much up to personal taste and the degree one cares about this kind of detail: running the words right together looks ugly to me, and appending the "recipe" part to its end conflicts with the way hierarchical package names get more specific from left to right. But now that I see it actually spelled out, I think I might even like it better than my other suggestion in case I really decide to collect the recipes under a two-part name.
(BTW: Did I just miss something, or do recipes for these really not exist yet?)
I don't know, are you done yet? ;)
I wouldn't call it done. Setting up a server root (an instance in Zope speak) with a preinstalled Apache does work but could still do with some more configurability. I hope to add a recipe for building Apache today, and one for adding mod_python on the weekend. If I can stick to this plan, I'll do a first release next week. -- Thomas
participants (2)
-
Jim Fulton
-
Thomas Lotze