Sadly, while the db/dynamic code stub files can be used to work around the problem; for files that live on the disk just not under the site-packages folder, they're less useful. 

What I mean that if I have a SRC folder on the disk and I'm using import hooks to expose them to the python interpreter as they would be installed into the purelib/platlib locations I cannot use the stub mechanism to expose these files to the static checkers. At best I could take all python files and copy them via a pyi files to a stub package. I'd like to get to a world where editable installations (even when implemented via import hooks) mostly work in IDEs. This will make significantly easier the life of these library developers. And while I agree users of the library are likely two order of magnitude bigger than maintainers, I think we can also agree that for a sustainable ecosystem the life of maintainers should also be easy as possible. Maintaining a library is already hard let's not make it harder by taking away the convenience of autocomplete in IDEs.

On Sat, 7 May 2022, 19:02 Guido van Rossum, <guido@python.org> wrote:
On Sat, May 7, 2022 at 17:30 Bernat Gabor <gaborjbernat@gmail.com> wrote:

TBH, if all we're trying to solve is editable installs, I am assuming that those would be used by a vanishingly small proportion of users (otherwise we're doing something else wrong with package installs), so I would personally be okay if an editable install wasn't visible to my IDE or type checker, or if those tools were to fall back to typeshed or a PEP 561 stub package.

Generally, they are two types of people when developing libraries:
  • who put everything in the current working directory and use the fact that the current working directory is added to the sys.path automatically - these people do not need editable installations as the lib appears to be installed without requiring installation,
  • who prefer to put the library into a src folder and use editable installations to manifest the content of the src folder to the import system (either via pth files, symlinks or import hooks).
In my experience, there's roughly a 60 to 40 divide in favour of the first. Especially more experienced/advanced users tend to prefer the src layout over the inline approach. Based on this I don't think it's accurate to say that it's used by a vanishingly small proportion of users. I personally use it for all my projects, and for why src layout can be better than the inline source see Paul Ganssle blog post here https://blog.ganssle.io/tag/tox.html explaining some of the problems (what's in your source directory is not necessarily what'll be put in your purelib/platlib during installation).


Ah, that’s not what I meant. I meant very few users are developing packages. E.g. there are millions of pandas users but few of them are modifying pandas. (I hope. :-)

Also, I do not think that what we want to solve is editable installations per se. What I'd like to solve here is supporting import hooks for static checkers. Editable installations are just a good example of where import hooks can be used.

Agreed. 

Sadly, while the db/dynamic code stub files can be used to work around the problem; for disks that live on the disk just not under the site-packages folder, they're less useful.

Not sure I follow.

—Guido

All the best,

Bernat

On Sat, May 7, 2022 at 6:11 PM Guido van Rossum <guido@python.org> wrote:
On Sat, May 7, 2022 at 4:14 PM Eric Traut <eric@traut.com> wrote:
Most static analysis tools already support “.pth” files. Jia mentioned that pyre doesn’t yet, but it should be straightforward for them to add this. My recommendation is that we rally behind “.pth” files. I think this approach is more pragmatic than introducing yet another mechanism that needs to be implemented by all of the tools in the ecosystem.

Hm. You're the first person I've met in a long time who regards PTH files as something we should do more of -- IIUC they are almost universally considered a necessary evil. Especially the fact that they can be used to do other stuff than add entries to sys.path. But also that -- a long sys.path tends to slow down module search, and it seems just wrong that adding N packages would append N sys.path entries: if we wanted each package to have its own entry we should have designed it as a mapping, not a search path.

TBH, if all we're trying to solve is editable installs, I am assuming that those would be used by a vanishingly small proportion of users (otherwise we're doing something else wrong with package installs), so I would personally be okay if an editable install wasn't visible to my IDE or type checker, or if those tools were to fall back to typeshed or a PEP 561 stub package.

Regarding your claim that almost nobody loads code from a database, apparently that's actually very popular in the banking sector (https://calpaterson.com/bank-python.html) -- and those people have a *lot* of Python code.

I believe that VS Code is using (or going to use, or may use?) a Virtual Filesystem to access user code. Would it really be so terrible if there were some corner of the VFS namespace to map to the Python module (spec) namespace? It could be served from a Python process that can give you the source of a module given its full name (without executing the module, of course -- the import loader machinery should be able to guarantee that already).

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
https://mail.python.org/mailman3/lists/typing-sig.python.org/
Member address: gaborjbernat@gmail.com
--
--Guido (mobile)