At 06:31 PM 11/11/2005 +0000, Richard Cooper wrote:
>From: Phillip J. Eby [mailto:pje@telecommunity.com]
> > What you probably want is a layout like this:
> >
> > app/
> > setup.py
> > My/
> > Project/
> > app
> > Plugin1/
> > setup.py
> > My/
> > Project/
> > plugin1/
> >
> > etc.
>
>My knee-jerk reaction is 'Ewww!' Is this what PEAK (or ant other large
>setuptools based project) does?
Yes, actually. More precisely, it's what PEAK *will* do, since I haven't
actually started breaking out individual projects yet.
>How do you handle My.Project.utils (for
>example) and the half dozen or so other packages at different levels of
>the hierarchy which are used in multiple "products"?
You make them individual projects. For example, I have a module,
peak.util.imports, that will be spun off as its own project. Its layout
will be:
setup.py
peak/
__init__.py
util/
__init__.py
imports.py
The __init__.py files will be empty, of course, since as namespace packages
they don't need any contents. Any project (including PEAK itself) that
needs to use this module will simply depend on this project as a whole.
My original plan for setuptools was to use an approach that could section
out parts of a large distribution like this. However, I quickly realized
that this was counterproductive to the Python "ecosystem"; it would be more
beneficial to have lots of small projects that can be used by others
(without buying into the entire PEAK system), than it would be for me to
keep using the same "all-in-one" source tree layout that I'm currently
using for PEAK.
I suspect that you may like Zope's "zpkg" system better for your purposes,
since it is designed to extract subsets of a larger distribution and
automatically generate setup scripts for them. I don't know if you can use
it to create eggs, but it might be possible to modify it to do so.
>My needs are a bit specialised, this is a commercial app so source
>distribution, easy_install, etc will never be on the cards. Distribution
>is strictly "frozen" products (py2exe and eggs) only.
easy_install is useful as an in-house tool for automatically building your
projects. For example, you could create a setup.py that lists all your
plugin project names as requirements, and a setup.cfg that lists their
subversion URLs, and you could run easy_install against that directory to
check out and build all the latest plugin eggs, dropping them in a
distribution directory. Your users don't ever have to see easy_install,
for it to be useful in your project. Of course, you may not be using
Subversion, etc., and this may be moot, but I'm not psychic and don't know
anything about what you're doing besides what you tell me. :)
> > If you are working on these projects at the same
> > time, you need only run "setup.py develop" in each one to set
> > up a sys.path structure that merges all the packages into one.
>
>We would probably end up with 20 or so "projects" if we did things that
>way as opposed to the single code tree we have now. Given that,
>"setup.py develop"*20 seems like more work than our current approach
>which is "add the root of the code tree to PYTHONPATH"
Does that mean you have 19 plugins? Perhaps I'm misunderstanding something
about your intentions. I assume that your projects would be one for the
app, and one for each plugin, so that you can create a py2exe for the app
and an egg for each plugin. If that's the case, then you would only have
20 projects if you had 19 plugins.
Also, if the plugins don't depend on each other, then to work on any one
plugin you only run "develop" twice: once for the app and once for the
plugin. Also, this may not be especially relevant to your project at the
moment, but "develop" also makes sure that your environment is synchronized
with any external requirements. For example, in a multi-developer scenario
where developer Aaron changes a dependency on the FooBar package from 1.1
to 1.2, then when developer Bob updates his source checkout and runs
"develop", it will find/fetch/build/install FooBar 1.2, thereby helping to
reduce the pain of adding or removing dependencies.
You could probably quite rightly say that this doesn't affect your project
because you don't have external dependencies, but a significant part of
*why* few projects have external dependencies is because they're a costly
pain -- if you're not using setuptools. I point this out because
setuptools is intended primarily to enable a new way of developing
projects, and enabling people to do things that were previously
unthinkable. If you aren't trying to do what was impossible before, then
certainly setuptools will not appear to help your current situation much. :)
> > Note, by the way, that you don't need namespace packages to
> > have plugins for an application. Plugins can be in any
> > package structure you like, as eggs' "entry points" system
> > can be used by plugins to advertise the functions they
> > provide, and the importing can be done automatically for you.
>
>Yeah I know. The entry point system is the main reason I started playing
>with setuptools. However I still wanted the plugins we produce to all
>live in the My.project.plugins package.
Note that giving each plugin its own top-level package would completely
eliminate any duplication of directory structure between plugins, if that's
your concern. But note that in any case, all that you'd be duplicating are
empty directories and zero-length __init__.py files.
>It looks like my problem is that I'm mingling separate products in the
>same code tree and setuptools/distutils doesn't like that. Fair enough.
>I don't think I'm quite ready to drink the "one tree per product"
>kool-aid just yet. Firstly, it seems weird to me to split up the source
>like that and secondly, my colleagues would probably kill me ;-)
Here's another possible layout, that will work nicely for eggs, but won't
work for "setup.py develop":
TheProject/
MyApp/
__init__.py
setup.py
*all app code here*
Plugin1/
__init__.py
setup.py
*plugin code here*
The idea here is that you have a MyApp top-level package, and a top-level
package for each plugin. Each setup script is *inside* the corresponding
package directory. Adding TheProject to PYTHONPATH makes everything
importable, just like you have now. But each project corresponds to a top
level package, and uses package_dir = {'MyApp':'.'}, or {'Plugin1':'.'}
etc. so that each project directory is also a package directory.
This compromise layout preserves the single-checkout+PYTHONPATH approach,
and should allow you to generate eggs without any special hacks, at the
cost of flattening your hierarchy a bit, and making it impossible to use
setuptools' "develop" or "test" commands (which need the package
directories to be somewhere *under* the directory containing setup.py).
Personally, I find that deeply nested foo.bar.baz hierarchies are a
Java-ism; Python doesn't need them and is usually better off without them;
"flat is better than nested", and all that. But I can also understand that
reorganizing your packages just to support eggs might be considered too
costly. Just make sure you factor in the cost of *maintaining* a hack to
do this, since the reorganization cost would be a one-time thing, but the
hack would be something you'd have to maintain forever.
>So what I will probably end up doing is hacking up setuptools to insert
>the __init__.py files I need. Which begs the question - Is this:
>
>a) A dirty, dirty hack I should never speak of again OR
>b) Not a bad idea and potentially useful to other people.
Yes. :)
It's both, really. Consider Chandler, for example, which is currently very
similar to your project. Actually, it's worse, because it doesn't even
have a setup script yet. In the next release cycle, we'll be moving to
eggs and we will have to do some source tree splitting, and there will be
some pain, and questioning, and maybe even some griping about it. *But*,
we spent a lot of time during the current release cycle doing quite a lot
of *flattening* of the source tree, turning deeply nested modules like
"osaf.contentmodel.contacts.Contacts" into "osaf.pim.contacts", and
consolidating APIs so that most code can now do:
from osaf import pim
aContact = pim.Contact()
instead of the former:
import osaf.pim.contacts.Contacts as Contacts
aContact = Contacts.Contact()
Flat really *is* better than nested. Much better. And as a bonus,
anything that we choose to split into separate eggs will not have as much
duplication. In the general case, the 'osaf' top-level package will be the
only thing that gets duplicated, and since it's a namespace package, the
__init__.py will be an empty file anyway.
So, I would encourage you to urge those you share the hack with, to
consider all of the issues I've brought out here, and use it only as a
crutch to allow a project to get past a temporary issue with its layout,
not as a cane to perpetually lean on. Putting on my software development
manager hat, I would tell you flatly that the time to manage the directory
structure duplication and use of "develop" is nothing compared to the time
saved by being able to intelligently manage external dependencies. And if
you don't have any external dependencies, it's only because they cost too
much before. Setuptools changes the cost equation, and therefore changes
what the ideal development patterns are. The reality may be that in your
particular environment you don't have the authority or reputation to lead
such a change, but we shouldn't let that stop other people from getting
that awareness.
Okay, I'll get off the soapbox now. :) Hack away, just make sure to point
people to this thread and especially this message.
One other idea, by the way... Have you tried adding:
py_modules = ['My.__init__', 'My.App.__init__', ...]
to the setup() arguments? It's just a guess on my part, but I think it
might actually work, without doing any hacking on distutils or
setuptools. Just a thought. Happy hacking, either way. :)
