Hello everyone,

Sorry for the late reply here, but I've been down with COVID after the conference so couldn't reply before. I'll try to explain and address some of the points raised above.

Ah good point. I am also curious how many folks really expect editable installs to be type checked, given tht type checking already happens on the original source.

The primary use case isn't necessarily the type checker. The primary use case here is to provide auto-complete (and type information) within IDEs. However, when we contacted the IDE maintainers their primary feedback was that their index is primarily built similarly to how type checkers do. Therefore, if we solve the problem for type checkers IDEs can reuse that.

> I'm not clear from your comment whether `.pth` support already exists in type checkers, or would need to be added

The support varies from type checker to type checker.

While PTH files (or more in general mechanisms that amend the sys.path) are a common way to achieve the editable effect it does have some downsides (as Paul described it in his detailed previous email). They have a few additional big downsides too:

only support file system paths (e.g. no support for loading code from a DB)
this actually impacts runtime: once you alter the sys.path you are no longer using import hooks, but instead fallback to a file path-based source loader.

The primary goal we want to achieve here though is to keep using import hooks at runtime while still providing static type/layout/shape information for static checkers. We established that static checkers (type, IDE index, etc) cannot evaluate runtime logic by design. Therefore they cannot directly use the dynamic import hooks, but instead, they'll need to operate on some separately provisioned files/paths that checkers can handle. Note, this is not a totally new concept for type checkers, e.g. mypy supports the MYPYPATH environment variable to extend the static checkers' discovery location, however, we don't have a standardized way across type (and other static) checkers. Furthermore, it's not today possible for each individual package to request indexing from additional paths/files. This is in my opinion the feature gap we need to address.

So here's my rough proposal I think could work. We should extend the binary distribution format (https://packaging.python.org/en/latest/specifications/binary-distribution-format/) to allow the addition of an additional STATIC_EXTRA.JSON file. If this is present IDE/type/static checkers are expected to handle them during indexing, on top of the current system. This JSON file would allow specifying the following operations as an ordered list:

Add an entire folder to be indexed (this would essentially have the same impact as a PTH file, but without impacting the interpreter at runtime).
Add a file to be indexed under a given module name (if the parent module is not already indexed the checker can refuse to load the file).

The way this would function:

for editable installation, this metadata file can be (re)generated as part of the editable wheel build and the path(s) will link back directly to the source directory,
for generated files during the build (e.g. generating python code from service schema files), these can be updated during the package build into a side folder and linked in from there,
for import hooks that load code from a database the user might explode the content of the database onto the disk during package installation and link it by using the `STATIC_EXTRA.json` file from there (while still loading from the DB directly during runtime),

For files directly available on the disk the static checker would be able to keep the index up to date without needing to actually move the file to the python interpreter site-packages. For files not available directly on disk (generated code or loaded from the database) the index would not reflect always reality and would require the user to periodically update the index files. But we'd get as close as we can without needing the static checkers to start up the Python interpreter. Let me know if anyone has any concerns with this approach, if not I'll try to summarize it as a PEP proposal. I believe this would make import hooks more usable in the language.

Thanks,

Bernat

On Wed, May 4, 2022 at 5:11 AM Paul Moore <p.f.moore@gmail.com> wrote:

On Wed, 4 May 2022 at 06:03, Jia Chen via Typing-sig
<typing-sig@python.org> wrote:

> I think at the end of the day, if `sys.path` cannot include everything, then what we want to see is what other (statically-determinable) paths we need to search for imports.
>
> But if I understand correctly, the point Eric brought up earlier was less about the technical difficulty of adding more places/mechanisms to look for imports -- as long as we agree on a standardized approach, that's always doable. His real concern was about the necessity and sustainability of such a proposal: why not just include "other paths we need to search for imports" into `sys.path` directly (e.g. with `pth` files)? Would it be nice if we could always maintain the assumption that `sys.path` is all-encompassing and all we need to resolve all imports, as opposed to introducing new tweaks every now and then, which forces type checkers to play this catch-up game every time the aforementioned assumption gets broken?

Unfortunately, that simply isn't remotely true, unless we restrict
what developers are allowed to do. (Assuming, as noted above, that
those developers expect to get typing information - an assumption that
I question, as I said). Import hooks are a feature of Python, and
while I'm fine with saying they are too dynamic for static type
checkers to support, ignoring their existence isn't reasonable. I'm
not saying there's an issue with type checkers here, apart from being
victims of their own success (they work so well in most cases that
users forget that they can only act on static information).

The canonical example of "why an editable install can't just add a
directory to sys.path" is the following:

Project directory
mylib.py
setup.py

The setup.py file is the setuptools configuration, and states that the
project consists of a single Python module, `mylib.py`. If we package
this file and install it normally, it puts the single file `mylib.py`
into site-packages, in a directory on `sys.path`.

If we want to do an "editable" install, the traditional approach adds
the project directory to sys.path via a `.pth` file that gets
installed to site-packages. Which does indeed make `import mylib`
work, and gets the code for mylib from the user's project directory.
But it also allows `import setup` to work (doing horrid things, as
`setup.py` is not intended to be imported). This is traditionally seen
as a problem, but one people just live with (for better or worse).

Part of the packaging community wants to provide "stricter" editable
installs, which make *only* `mylib` importable, but not `setup`. That
simply can't be done by saying "this directory gets scanned for
importable files". I wrote a helper library that handles this case by
using import hooks, and that's probably what triggered this
discussion, as build backends are using that library (or at least the
same approach). Personally, my attitude is "don't use that mechanism
if you care about type checker support, and type checkers can't handle
it". This discussion is mostly about whether that approach can be made
to be typechecker-friendly. But if you can't handle anything other
than adding the complete contents of a directory to the list of "what
is to be scanned", I think we're probably dead in the water (because
of the setup.py use case).

There are other possible mechanisms (such as creating a staging
directory with symlinks to just the expected files and exposing that
directory on sys.path) which may be something the packaging community
could explore. There are problems (symlinks aren't guaranteed
available everywhere) but those aren't typing issues. I don't know
whether anyone is looking at those options (I'm not, personally).

Paul
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
https://mail.python.org/mailman3/lists/typing-sig.python.org/
Member address: gaborjbernat@gmail.com