Hello, I think you severely underestimate how bad it is for code meant only for the static analysis tools to be part of the runtime interpreter. Your proposal effectively renders import hooks totally unusable as would be shadowed by whatever the pth file injects. Furthermore you also ignore on selective inclusion and exclusion of python files from within a folder (another problem PTH files cannot handle). Additionally I'm aware that many core developers would prefer removing support for PTH files, so that also discourages me to use them. Also pth files are not usable to inject a generated code as part of another package (PTH files can only inject paths at the root of purelib/platlib, not under a given package). For all those reasons I don't consider your proposal a viable path ahead. Thanks, Bernat On Sat, 7 May 2022, 17:14 Eric Traut, <eric@traut.com> wrote:
Bernat, sorry to hear you were down with Covid. I hope you are feeling better now.
As you point out, this issue affects more than static type checkers. It affects all tooling that relies on static analysis. That includes language servers (like pylance), smart editors (like PyCharm), code reformatters (like black and isort), scanners for detecting code vulnerabilities and IP (code provenance) issues, etc. In other words, it affects a wide variety of tools that are collectively used by the majority of Python developers in their day-to-day work.
Supporting scenarios where code is not located in the file system is difficult for any static analysis tool. I don’t think this will ever work well. Thankfully, this is used very rarely today. The few developers who are relying on loading code from alternative locations like DBs presumably have very specific needs and understand the tradeoffs of the approach. My recommendation is that we not optimize for this case because it greatly complicates things. And even the best of solutions will fall apart at the edges. Let’s explore solutions for code that lives in the file system, and once we’re happy with that solution, we can look for ways that might partially accommodate the harder case.
I think your proposal for “STATIC_EXTRA.json” (or something similar) could work technically, but it will be a long time before all tools implement it. As you point out, adoption of support for “.pth” files is still incomplete many years after they were introduced.
The import mechanism in Python is already extremely complex for static analysis tools to support. We need to handle all of the idiosyncrasies of traditional and namespace packages, stub libraries, native binaries, zip-based eggs, and the complex prioritization rules dictated by PEP 561. Each one of these by itself is not that complex, but some of them interact in extremely complex ways. This is a continual source of bugs and inconsistencies across tools and between tools and the runtime. Adding yet more complexity by introducing something like “STATIC_EXTRA.json” should be done only if we think that it provides sufficient new value. I’m unconvinced in this case. You’ve indicated that “.pth” files have some shortcomings that this new mechanism would address, but the shortcomings that you’ve described don’t seem very significant or problematic.
You mentioned that `STATIC_EXTRA.json` could be generated at install or package build time. The same thing is true of a “.pth” file. The way this would function:
- for editable installation, the “.pth” can be (re)generated - for generated files during the build, a “.pth” file can generated or updated to point to the directory where they are generated - for import hooks that load code from a DB, the content of the DB can be exploded to disk and a “.pth” file can point to them
I understand your point about the presence of “.pth” files changing the runtime behavior for import hooks that access code that lives outside of the file system. The small number of developers who want to store their code in DBs can decide whether the tradeoff is worth it.
Most static analysis tools already support “.pth” files. Jia mentioned that pyre doesn’t yet, but it should be straightforward for them to add this. My recommendation is that we rally behind “.pth” files. I think this approach is more pragmatic than introducing yet another mechanism that needs to be implemented by all of the tools in the ecosystem.
-Eric
-- Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gaborjbernat@gmail.com