organizing your scripts, with plenty of re-use

[Margie]

On Oct 5, 12:34 pm, Robert Kern <robert.k... at gmail.com> wrote:
> On 2009-10-05 13:48 PM, Buck wrote:
>
>
>
> > On Oct 5, 11:29 am, Robert Kern<robert.k... at gmail.com>  wrote:
> >> On 2009-10-05 12:42 PM, Buck wrote:
>
> >>>> With the package layout, you would just do:
>
> >>>>      from parrot.sleeping import sleeping_in_a_bed
> >>>>      from parrot.feeding.eating import eat_cracker
>
> >>>> This is really much more straightforward than you are making it out to be.
>
> >>> As in the OP, I need things to "Just Work" without installation
> >>> requirements.
> >>> The reason for this is that I'm in a large corporate environment
> >>> servicing many groups with their own custom environments.
>
> >> The more ad hoc hacks you use rather than the standard approaches, the harder it
> >> is going to be for you to support those custom environments.
>
> > I too would prefer a standard approach but there doesn't seem to be an
> > acceptable one.
>
> >> I do believe that you and Stef are exceptions. The vast majority of Python users
> >> seem to be able to grasp packages well enough.
>
> > You're failing to differentiate between python programmer and a
> > system's users. I understand packages well enough, but I need to
> > reduce the users' requirements down to simply running a command. I
> > don't see a way to do that as of now without a large amount of
> > boilerplate code in every script.
>
> I would like to see an example of such boilerplate. I do not understand why
> packages would require more than any other organization scheme.
>
> > I've considered installing the thing to the PYTHONPATH as most people
> > suggest, but this has two drawbacks:
> >    * Extremely hard to push thru my IT department. Possibly impossible.
> >    * Local checkouts of scripts use the main installation, rather than
> > the local, possibly revised package code. This necessitates the
> > boilerplate that installation to the PYTHONPATH was supposed to avoid.
> >    * We can work around the previous point by requiring a user-owned
> > dev installation of Python, but this raises the bar to entry past most
> > of my co-developers threshold. They are more comfortable with tcsh and
> > perl...
>
> Are you sure that you are not referring to site-packages/ when you say "PYTHONPATH"?
>
> PYTHONPATH an environment variable that the user can set. He can add whatever
> directories he wants to that environment variable. The user can make a directory
> for his checkouts:
>
>    $ mkdir ~/LocalToolCheckouts
>    $ cd ~/LocalToolCheckouts
>    $ cvs ...
>
> Now add that directory to the front of his PYTHONPATH:
>
>    $ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH
>
> Now everything works fine. Packages in ~/LocalToolCheckouts will get picked up
> before anything else. This is a simple no-installation way to use the normal
> Python package mechanism that works well if you don't actually need to build
> anything.
>
> > I think the issue here is that the current python-package system works
> > well enough for the core python devs but leaves normal python
> > developers without much options beyond "all scripts in one directory"
> > or "tons of boilerplate everywhere".
>
> The "vast majority" I am talking about *are* the normal Python developers.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
>   that is made terrible by our own mad attempt to interpret it as though it had
>   an underlying truth."
>    -- Umberto Eco

I think that Buck's issue with your above example is that he doesn't
want his users to have to type
   $ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH

For example, say that the developer does this:
   $ mkdir ~/LocalToolCheckouts
   $ cd ~/LocalToolCheckouts
   $ cvs ...

Which results in the following directory structure:

~/LocalToolCheckouts
    +-- scripts/
        +-- myscript.py
    +-- parrot/
        +-- __init__.py
        +-- feeding/
            +-- __init__.py
            +-- eating.py
            +-- drinking.py


I think he is looking for a way for users to be able to use scripts/
myscript.py (which imports parrot) without having to change their
PYTHON path with something like this:

   $ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH

I'm guessing that Buck has users that are running out of a cvs
repository.  Although many would say those users are now "developers",
they really are not.  They probably don't even know they are running
from a cvs repository.  They in fact may think of it as their own
personal installation, and all they know is that they have scripts
directory and that that scripts directory has some scripts they want
to run.

As Buck said, it can often be very difficult to get things properly
and quickly installed in a large corporate environment, and providing
the user with a way to check out a cvs repository can be very quick
and easy.  The problem is that once the user has access to that cvs
repository, it is difficult to tell them "hey, every time you run from
it, you need to execute this special command to set up your PYTHONPATH
environment variable."

I don't really see any good solution to this other than some
boilerplate code at the beginning of each script that overrides
PYTHON_PATH.  I think what I'd do in this situation is have a dot file
that indicates that you are running out of a "dev" (ie cvs repository)
area.  Then some boilerplate code like this at the beginning of your
scripts:

if os.path.exists(os.path.dirname(sys.argv[0]) + "/.dev"):
    sys.path.append(os.path.abspath(os.path.dirname(sys.argv[0])) +
"/..")

This basically says "if the .dev file exists, put the parent directory
of the script into sys.path."

When an installation is done, as part of the install the .dev file
should be removed.  In that case the libraries should be installed
into the standard site-packages location and the user running from an
install would automatically get their packages from the installation
area.

While having boilerplate code like this at the beginning of the script
is not great, I do think that it will save a lot of user questions/
confusion if the users are frequently running from one or more cvs
repositories.  For example, if requiring the user to explictly set
PYTHONPATH, if the user has two cvs repositories, with different
versions of the code, a user that is not very python-knowlegable will
frequently forget to update their PYTHONPATH and will end up running
new scripts with an old library.  Having this boilerplate code should
avoid a problem like that.  So I think the pros probably outweigh the
cons.


Margie