[Tutor] Best way to get root project directory in module search path

[Cameron Simpson]

On 27Aug2017 14:27, boB Stepp <robertvstepp at gmail.com> wrote:
>On Sun, Aug 27, 2017 at 2:13 AM, Cameron Simpson <cs at cskk.id.au> wrote:
>> On 26Aug2017 21:27, boB Stepp <robertvstepp at gmail.com> wrote:
>>> import sys
>>> sys.path.append('..')
[...]
>> The trouble with this specific approach is that '..' relies on your
>> _working_ directory being above ScriptMenuSystem i.e. the directory you
>> shell's in; '..' is not related to the path to the test.py file. What yu
>> want for this is the actual path to the code i.e. __file__.  Eg:
>>
>>  sys.path.append(dirname(dirname(__file__)))
>
>I tried to use this (on Python 2.4/2.6 -- one server has 2.4, the
>other 2.6) and got the following:
>
>Traceback (most recent call last):
>    File "... /test.py", line 9, in <module>
>        sys.path.append(dirname(dirname(__file__)))
>NameError:  name 'dirname' is not defined

from os.path import dirname

>Is this a version issue or do I have the syntax wrong?

Just a missing import. You'll find dirname in the Index section of the Python 
docs. Handy for locating something just mentioned.

>Anyway, I tried something a bit different that implemented this:
>
>pgm_dir = os.path.dirname(os.path.abspath(__file__))
>pgm_root = pgm_dir.replace('/ScriptMenuSystem', '')
>sys.path.append(pgm_root)

Calling os.path.dirname a second time does what your:

  pgm_dir.replace('/ScriptMenuSystem', '')

does, but reliably. Supposing you have the misfortune to have 
'/ScriptMenuSystem' higher up in the full path also? Supposing your package 
changes its name in the future, or is installed with another name? Fragility.

>And this gave me the results I wanted.  Funny that 'dirname' works
>with os.path but not with sys.path.

No no. "dirname" comes from the "os.path" module. That is all.

>BTW, I have not used os.walk yet, but would that be better than my bit
>of string manipulation above?  If yes, an example of walking up and
>getting the directory exactly one level up from __file__ would be
>appreciated.

Well:

  from os.path import abspath, dirname, join
  code_dir = abspath(join(dirname(__file), '..'))

or:

  from os.path import abspath, dirname, join
  code_dir = abspath(join(dirname(__file), '../..'))

depending how high you want to go.

Given that os.walk walks _down_ the tree from some directory, how do you think 
it would help you?

>> but that requires your module to know its own location within your library
>> i.e.  that it is .../dev_scripts/ScriptMenuSystem/test.py, and therefore
>> that you want .../dev_scripts. Fragile. (And __file__ doesn't work in some
>> more arcane ways of distributing modules, such as in zip files but let's not
>> go there, not least because I've never done that myself).
>
>I have been using the same project structure for some years now, so in
>that sense it is well-established and not fragile.

It is more that embedded specific hardwired knowledge in the code is an 
inherently fragile situation, if only because it must be remembered when naming 
changes occur. And they do. I have a quite large package I've been working on 
for a decade and am very seriously contemplating changing its name in the next 
month or so. Fortunately the installed base is small.

>I don't anticipate
>it changing.  There is actually no installation in a traditional
>sense.  Once I develop something worth using, it gets copied (by me as
>root) from my user area to an accessible location to those users that
>requested access.  I had been doing this copying to each user's area,
>but that has grown unwieldy as my couple of users have grown into
>around 15.

For simple packages/modules an install _is_ just a copy. So you're performing 
the install step.

>So I am moving to put everything in a centrally accessible
>location

Very sound. But to use it your users should have that location as part of their 
python path, or the code needs to be invoked by some wrapper which can insert 
the needed path. Both these things live _outside_ the package code.

>and will probably adopt Steve's suggestion to establish a
>group with permissions to use these programs.

Note that this is normally only needed for code that is privileged i.e. runs 
with permissions different to those of the user.

>So I think this idea (Or some variation thereof.) is what I need to
>do.  Or the good experts here show me I'm engaged in foolishness!

I think you're better off arranging your users' environments to include the 
shared library area, or providing a wrapper shell script which does that and 
then invokes the real code.

Eg:

  #!/bin/sh
  PYTHONPATH=/path/to/your/shared/lib:$PYTHONPATH
  export PYTHONPATH
  python -m your.module.name ${1+"$@"}

Such a script has the advantage that the $PYTHONPATH change applies only to the 
python running your module, not to your users' wider environment.

Cheers,
Cameron Simpson <cs at cskk.id.au>