[Distutils] EasyInstall: development environment

[Phillip J. Eby]

At 12:30 AM 6/16/2005 -0500, Ian Bicking wrote:
>Working from Phillip's last email, here's my proposed list of steps for
>creating a development environment; some of these could be automated,
>but lets presume the automation does the same thing this manual process
>does...
>
>Application name: MyApp
>
># We don't care where this directory is, except that we put all our
># in-development checkouts here...
>cd ~/devel-directory
>
>mkdir MyApp
># or an svn repository /MyApp/trunk, and check out to MyApp/
>cd MyApp
>mkdir MyApp.egg-info
>
>echo "Name: MyApp
>Version: 0.0" > MyApp.egg-info/PKG-INFO
>
>echo "SQLObject>=0.6.1
>pysqlite2
>Cheetah" > MyApp.egg-info/depends.txt
>
># But I'm going to be developing SQLObject, but not pysqlite and
># Cheetah...
>
>svn co http://svn.colorstudy.com/trunk/SQLObject ../SQLObject
>cd ../SQLObject
># Well, this isn't right, because SQLObject doesn't import
># setuptools... I guess I have to run package/easy_install somehow?
>python setup.py bdist_egg
>
>cd ..
>export PYTHONPATH=`pwd`
>
>cd MyApp
>mkdir app-packages
># What do I run here to get the dependencies that aren't already
># fulfilled?
>easy_install -d app-packages (?)
>
># But then I try to install pysqlite, but there's some weird conflict,
># and I wish I was just using the OS package... so what do I do to
># satisfy the dependency?  Preferably without writing to
># site-packages/, where pysqlite is installed?
>
># and where are my scripts at this point?  What should MyApp/setup.py
># look like?  What should MyApp/ look like?  Should it have a package
># underneath it?

Okay, I lost you somewhere here.  Here's my vision of your "project" directory:

MyApp/

     MyApp.egg-info/   # project egg-info

     SQLObject.egg/    # checkout of SQLObject's "project" directory

     pysqlite2-2.1-py2.4.egg/  # eggs installed with easy_install -d.
     Cheetah-1.9-py2.4.egg/

     some_sqlobject_script    # scripts installed by easy_install
     cheetah_compile_script
     pysqlite_shell_script.py

     myapp_script1.py   # scripts, packages, and modules from my project
     myapp_script2.py

     myapp_package1/
     myapp_somemodule.py

     setup.py    # setup stuff
     setup.cfg

So, a "project" directory always contains a .egg-info for the project, 
along with its setup.py and setup.cfg.  Thus, if you want to develop 
multiple libraries in a single development project, you can just check out 
those libraries' project directories, giving them names with .egg 
extensions.   (Note: right now the nested project's SQLObject.egg-info has 
to be renamed to EGG-INFO for metadata to work right, but I plan to fix 
this so that a simple checkout will suffice in future.)

There are still a few wrinkles in this layout, mostly relating to 
scripts.  If scripts all use require, and you use subproject scripts (like 
an SQLObject script in this scenario) by running the subproject's 
build_scripts+install_scripts targeting the main project's directory, then 
it all should work.  However, if the subproject isn't designed for use in 
eggs, it won't work because the egg won't be on sys.path.  So, there will 
really need to be a way to get easy_install to do a scripts-only install 
from a subproject directory.  (And it's going to have to use different stub 
scripts, because the subproject's scripts won't be under its .egg-info 
directory.)

The parent project's scripts don't have to require() the main project, 
since it's already on sys.path when they run.  But, if they don't, then the 
script's requirements won't be activated.  So that's a bit of a problem, 
although I suppose we could just say you have to use require('MyApp') or 
something similar in your scripts.

The other problem with this approach is that it assumes an overall project 
directory layout that doesn't necessarily match what people use 
today.  Zope and PEAK, for example, put their packages in a 'src' 
subdirectory under the "project directory", and that totally won't work 
with this approach unless there's a way to somehow configure the paths 
differently.  It makes me think that maybe there needs to be a ".egg-link" 
file format that just contains a path to the a project's source 
root.  Then, the layout might look like this instead:

MyApp/

     MyApp.egg-info/   # project egg-info

     SQLObject.project/    # checkout of SQLObject's "project" directory
     SQLObject.egg-link    # pointer to SQLObject's package root

     pysqlite2-2.1-py2.4.egg/  # eggs installed with easy_install -d.
     Cheetah-1.9-py2.4.egg/

     # ... etc.

This is more complex to set up, but it should work for any existing 
distutils projects, and on systems that don't have symlinks.

There are still enough wrinkles to make my head hurt, though.  For example, 
the parent project still has to have a single-directory layout, unless we 
complicate things even more...


MyApp/
      bin/  # all scripts, eggs, links get installed here


      SQLObject/  # subprojects can get checked out here, though
                  # and an .egg-link is put in bin

      setup.py   # setup stuff has to live in the root
      setup.cfg

      src/  # but you can put stuff under src or lib if you want
            # or in this directory if you prefer


Hm.  I suppose that actually isn't any more complicated.  However, whenever 
you add or remove scripts from the MyApp project, you'll have to run 
something to update your stub(s) in bin/.

I'm also not sure 'bin' is the best name for the directory; it might 
already be used by some projects.  We could call it 'run' or 'runtime' or 
something like that, though.

So with this layout, you just 'easy_install -dbin' your non-development 
dependencies, and it'll even create the bin directory for you.  You 
checkout subprojects, and then run some kind of setup command, e.g.:

    cd SQLObject
    setup.py dev_link -d../bin

To create the needed .egg-link script stubs in bin/.  In practice, you'd 
probably do some sort of easy_install or package.py command from the main 
directory, so that it can work with projects that don't use setuptools.

Hm.  Okay, that actually seems pretty reasonable, since it can now handle 
project-specific directory layouts with ease.  The only fixed parts are 
that you need to select a 'bin' location that doesn't clash with the 
location of the source of your project's scripts.

The downside is that '.egg-link' is purely a hypothetical concept at this 
point, and possibly introduces some new wrinkles into the egg installation 
process.  However, experimenters could pretty much get the same effect 
right now by hand-creating a .egg symlink to the subproject source root, 
and making an EGG-INFO symlink in the source root pointing to 
Subproject.egg-info.  That would work today, although a lot of the script 
pieces aren't fully ready for that configuration.

Your thoughts?