[Distutils] setuptools-0.4a2: Eggs, scripts, and __file__

[Phillip J. Eby]

At 12:16 AM 6/13/2005 -0400, Ryan Tomayko wrote:
> From "[Distutils] A plan for scripts (in EasyInstall)"
><http://mail.python.org/pipermail/distutils-sig/2005-June/004594.html>
>
>On Mon Jun 6 16:44:19 CEST 2005, Phillip J. Eby wrote:
> > The 'run_main' function would do several things:
><snip>
> >
> >   * Clear everything but __name__ from the __main__ namespace
><snip>
> >   * exec the script in __main__, using something like:
> >
> >      maindict['__file__'] = pseudo_filename code =
> >      compile(script_source, pseudo_filename, "exec") exec code in
> >      maindict, maindict
>
>It seems that bullet one is happening but not bullet two? I have
>scripts that are attempting to use __file__ but failing with:
>
>     NameError: name '__file__' is not defined

I'll fix this in the next release.


>The script looks like it would work properly if it was given a pseudo
>filename but this has me thinking about what the best way to detect
>development environments in scripts will look like in an eggified
>environment.

That's the wrong question to ask, IMO.  Think about how to make the script 
work exactly the same in all environments, instead.  :)


>My wrapper scripts generally look something like the
>following to detect whether the script is being run from a
>development location or a deployed location::
>
>     import sys
>     from os.path import dirname, abspath, join, exists
>     devel_dir = dirname(dirname(abspath(__file__)))
>     if exists(join(devel_dir, 'support')):
>          sys.path.insert(1, join(devel_dir, 'support'))
>          sys.path.insert(1, devel_dir)
>     from package.module import main
>     main(sys.argv)
>
>Assuming a directory layout of:
>
>     [devel-dir]/bin (scripts in here)
>     [devel-dir]/support (packages/modules)
>     [devel-dir]/[package-name]

Have you read this:

http://peak.telecommunity.com/DevCenter/PythonEggs#developing-with-eggs

The complexity you're incurring here is unnecessary; between require() and 
.pth files you should never need to mess with sys.path manually.


>At first I thought I should switch from using path operations on
>__file__ to using `pkg_resources.resource_isdir` and
>`resource_filename` but that doesn't make any sense - if the script
>is running from within a deployed egg, I'm not using it from a
>development environment

Not true; see the link above.  When you do development from your distutils 
package root, your development code *is* in an egg.  However, you still 
shouldn't be checking __file__ or fiddling with sys.path, and there is no 
need anyway.  Here's one idea of what your tree could look like:

devel_dir/
     some_dependency-1.2.egg
     otherpackage-3.9-whatever.egg

     package_you_are_working_on/
         __init__.py
         some_module.py

     somescript.py

     MyPackage.egg-info/   <- this makes devel_dir a "source egg" for MyPackage
     setup.py

     build/
     dist/
         MyPackage-1.1.egg  <- built egg; not actually used for development


You'll notice that everything is just thrown into the same directory; if 
you use this setup for development, everything will "just work".

So, you create a setup.py for your package, and you run setup.py bdist_egg; 
this will dump an egg in dist/, and create MyPackage.egg-info, marking 
devel_dir as a "development egg".

Install any other packages you need to the current directory using 
'easy_install -xd. package_you_need' (the -x excludes their scripts).  Now 
you're ready to party.  Make your scripts use 'require()' to ask for 
'MyPackage'; when you run them (whether you are in the devel_dir or not), 
they will find MyPackage.egg-info, find the dependencies declared, and add 
all the needed .eggs to sys.path automatically.

This is just *one* layout that works.  The dependency eggs don't have to be 
in devel_dir; they can be anywhere that will already be on sys.path, like 
site-packages or a directory named in a .pth file in site-packages.

Does this explain it better?  One side benefit of egg-based installation is 
that you can dump as many libraries in site-packages as you want and not 
worry about version conflicts, so it's definitely how I plan to do most 
development.  The directory where a script is located, however, takes 
precedence over site-packages, which means that even if you have the 
package you're developing already installed in site-packages, your 
development egg will take precedence if the script you're running is in 
that directory.


>  and the resource_*** functions don't make
>sense in __main__ context anyway. So my current thinking is that the
>existing idiom should remain and that a pseudo filename shouldn't
>pose any problems in the scenarios I'm dealing with:
>
>  * Deployed egg: don't tamper with sys.path
>  * Deployed site-packages: don't tamper with sys.path
>  * Development environment: insert development paths

You should be able to make that last one work the same way; i.e., without 
tampering with sys.path.  If you can't, please explain your situation 
further, because I want pkg_resources to be able to prevent all future 
sys.path munging by anything but EasyInstall itself, and by extensible 
applications that have to manage plugin directories.