On Wed, 4 May 2022 at 06:03, Jia Chen via Typing-sig <typing-sig@python.org> wrote:
I think at the end of the day, if `sys.path` cannot include everything, then what we want to see is what other (statically-determinable) paths we need to search for imports.
But if I understand correctly, the point Eric brought up earlier was less about the technical difficulty of adding more places/mechanisms to look for imports -- as long as we agree on a standardized approach, that's always doable. His real concern was about the necessity and sustainability of such a proposal: why not just include "other paths we need to search for imports" into `sys.path` directly (e.g. with `pth` files)? Would it be nice if we could always maintain the assumption that `sys.path` is all-encompassing and all we need to resolve all imports, as opposed to introducing new tweaks every now and then, which forces type checkers to play this catch-up game every time the aforementioned assumption gets broken?
Unfortunately, that simply isn't remotely true, unless we restrict what developers are allowed to do. (Assuming, as noted above, that those developers expect to get typing information - an assumption that I question, as I said). Import hooks are a feature of Python, and while I'm fine with saying they are too dynamic for static type checkers to support, ignoring their existence isn't reasonable. I'm not saying there's an issue with type checkers here, apart from being victims of their own success (they work so well in most cases that users forget that they can only act on static information). The canonical example of "why an editable install can't just add a directory to sys.path" is the following: Project directory mylib.py setup.py The setup.py file is the setuptools configuration, and states that the project consists of a single Python module, `mylib.py`. If we package this file and install it normally, it puts the single file `mylib.py` into site-packages, in a directory on `sys.path`. If we want to do an "editable" install, the traditional approach adds the project directory to sys.path via a `.pth` file that gets installed to site-packages. Which does indeed make `import mylib` work, and gets the code for mylib from the user's project directory. But it also allows `import setup` to work (doing horrid things, as `setup.py` is not intended to be imported). This is traditionally seen as a problem, but one people just live with (for better or worse). Part of the packaging community wants to provide "stricter" editable installs, which make *only* `mylib` importable, but not `setup`. That simply can't be done by saying "this directory gets scanned for importable files". I wrote a helper library that handles this case by using import hooks, and that's probably what triggered this discussion, as build backends are using that library (or at least the same approach). Personally, my attitude is "don't use that mechanism if you care about type checker support, and type checkers can't handle it". This discussion is mostly about whether that approach can be made to be typechecker-friendly. But if you can't handle anything other than adding the complete contents of a directory to the list of "what is to be scanned", I think we're probably dead in the water (because of the setup.py use case). There are other possible mechanisms (such as creating a staging directory with symlinks to just the expected files and exposing that directory on sys.path) which may be something the packaging community could explore. There are problems (symlinks aren't guaranteed available everywhere) but those aren't typing issues. I don't know whether anyone is looking at those options (I'm not, personally). Paul