[Tutor] file accessibility across directories?

Thu Feb 29 15:17:19 EST 2024

You are raising an interesting question, and I have not studied the
standard distribution structure of packages.  My intent was to separate
scripts used to generate a database from the src directory largely to
emphasize that these are stand-alone scripts which are to be executed in
"isolation" -- even if they are dependent upon code in imported from src.
If this is contrary to standard practices, I'm happy to rethink the matter.

Thank you for your comments.

On Thu, Feb 29, 2024 at 1:42 PM dn via Tutor <tutor at python.org> wrote:

> James,
>
> On 1/03/24 06:16, James Hartley wrote:
> > This has been a useful discussion (at least for me).  Thank you all for
> > your candor.
>
> This is a common concern/problem. However, you outlined it well. Plus,
> @Mat's expansion is a very good discussion of the interplay between
> Python (the interpreter) and one's directory structure. Using "python -m
> ..." is often a good way around. Thus, it's worth others reading (and
> learning)...
>
>
> >  From these comments, it sounds like playing with SYSPATH is a common
> task.
> > I have some database tables defined for SQLAlchemy which I have imported
> > into some Pytest tests.  This works fine, but to import the same table
> > definitions into scripts found in other directories suffers from the
> > problems prompting this thread.  It appears that I will have to modify
> > SYSPATH to make one (preferrable) definition work in all locations.
>
> Yes and No!
>
> If you are both customer and supplier, then you probably won't have a
> problem - either at coding-time or later in life.
>
> If however, you distribute the code to others, these aspects take-on a
> whole new (and 'exciting') dimension!
>
> BTW the sub-dir layout is 'standard' for some of the 'installers' used
> for Python distribution.
>
>
> A first question though: if something has been put into a 'utility'
> directory - which makes it sound as if it will be import-ed from
> different parts of the code (even different projects), then why is it
> being executed as a script? Once the utility routines have been
> exercised and tested (routines either located in the project/test
> sub-dir or in project/utils/tests) their (only) access should be by
> import (from project/src routines)?
>
>
> For further thought (as if there isn't enough swirling your brain - and
> bearing in-mind that I haven't used SQLAlchemy for a while) is that DRY
> is a concern for schema as much as it is for Python-code. There's no
> point in testing one and running another. Oops! Accordingly, a separate
> file (and/or potentially project sub-dir) which holds the SQL details
> and access methods, (hah! and here it comes...) accessible from both
> code and test modules.
>
>
>
> > Thanks again for your comments!
> >
> > On Thu, Feb 29, 2024 at 10:57 AM Mats Wichmann <mats at wichmann.us> wrote:
> >
> >> On 2/28/24 18:45, dn via Tutor wrote:
> >>> On 29/02/24 13:06, Mats Wichmann wrote:
> >>>> On 2/28/24 15:39, James Hartley wrote:
> >>>>> While in the project directory:
> >>>>> $ pwd
> >>>>> /home/me/project
> >>>>> $ cat src/foo.py
> >>>>> class Foo:
> >>>>>       pass
> >>>>> $ cat utils/baz.py
> >>>>> from src.foo import Foo
> >>>>> $ python utils/baz.py
> >>>>> Traceback (most recent call last):
> >>>>>     File "./baz.py", line 1 in <module>
> >>>>>       from src.foo import Foo
> >>>>> ModuleNotFoundError: No module named 'src'
> >>>>
> >>>>
> >>>> When you run a script, the directory of the script is added to
> >>>> sys.path. Not the directory you were in when you ran it.  You can
> >>>> print out sys.path in utils/baz.py to convince yourself of this.
> >>>> There's no src in that directory (utils).
> >>>
> >>> Whereas pytest works the other way around: run from the
> >>> project-directory because its (internal) 'process of discovery' will
> >>> find test* files by searching the entire directory-tree...
> >>> (hence the questions in earlier reply)
> >>
> >>
> >> Right... pytest takes a common situation and helps you out, in this case
> >> (a) by discovering tests and (b) fiddling paths so those tests can
> >> import cleanly. If you don't get that help by using (for example) a test
> >> framework, you have to take care of it yourself.
> >>
> >> This import fails because for an absolute import (src.foo - doesn't
> >> start with dots), the starting directory has to be in the search path
> >> (sys.path), but it isn't. You can fix that by adding to sys.path in the
> >> scripts in utils, or by using PYTHONPATH.  For example, this should
> work:
> >>
> >> PYTHONPATH=. python utils/baz.py
> >>
> >> You would think you could use relative paths. However, if you attempt a
> >> relative import like:
> >>
> >> from ..src.foo import Foo
> >>
> >> it doesn't work, because the way the script was invoked, Python doesn't
> >> think it's part of a package (invoked as a script, the name is
> >> "__main__", and contains no dots, thus you must not be in a package),
> >> and relative paths are package-hierarchy-relative, not
> >> filesystem-relative, so you'll get one of Python's famous error
> messages:
> >>
> >> ImportError: attempted relative import with no known parent package
> >>
> >> However, you could run baz as a module, like:
> >>
> >> python -m utils.baz
> >>
> >> because now the name of the running module is "utils.baz", so it looks
> >> like a package rooted at a level above "utils", and that directory will
> >> be in sys.path, so "from src/foo import Foo" should work.  It just seems
> >> a bit ugly to have to invoke a "top level script" as a module to get
> >> imports to work.
> >>
> >> Python imports are... challenging.
> >>
> >> (yes, I'm assuming answers to the question @dn asked, so this could be
> >> wrong if I made the wrong guess)
> >>
> >> _______________________________________________
> >> Tutor maillist  -  Tutor at python.org
> >> To unsubscribe or change subscription options:
> >> https://mail.python.org/mailman/listinfo/tutor
> >>
> > _______________________________________________
> > Tutor maillist  -  Tutor at python.org
> > To unsubscribe or change subscription options:
> > https://mail.python.org/mailman/listinfo/tutor
>
> --
> Regards,
> =dn
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>