[Distutils] supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation

[Nick Coghlan]

On 7 September 2017 at 08:43, Paul Moore <p.f.moore at gmail.com> wrote:
> So pip install -e ., instead of installing a .pth file which included
> the current directory on sys.path, would install a set of .py files
> which loaded the actual code from the current directory? That sounds
> like a plausible approach, and it would certainly stop editable
> installs being quite as much of a special case as they currently are.

It sounds like a recipe for obscure bugs to me, since there's no
standard mechanism for one Python file to transparently pretend to be
a different one (other than symlinking them).

By contrast, manipulating sys.path to include additional directories
is entirely normal - it can just have surprising name shadowing
effects if the directory has other files and directories in it (this
is one reason for the popularity of "src/" directories that *only*
include the "for import" files).

So for systems with reliable user level symlink support, the most
robust approach will be along the lines of:

1. Generate an "_install/<name>" directory
2. Generate proper wheel metadata inside that directory
3. Symlink from that directory to any Python source files, package
directories and built extension modules based on the manifest in a
fully built wheel file
4. Install a *.pth file into the virtual environment that adds the
generated directory to sys.path (and refuse to run if no virtualenv is
active)

Doing things this way means that hot reloading in things like web
frameworks will still work properly, but you also won't inadvertently
be able to import random other files that happen to be lying around in
the source directory

Unfortunately, that approach *doesn't* work reliably on Windows, since
the symlinks in step 3 can't be generated with ordinary user
permissions. Instead, the installed *.pth needs to reference the given
source directory directly, which means you won't have the freedom to
rearrange things to match the layout in the wheel manifest.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia