[Distutils] Packages that have problems being installed from wheels

[Paul Moore]

On 28 August 2014 20:35, Donald Stufft <donald at stufft.io> wrote:
>> Both Ansible and Fabric want to put stuff in <virtualenv>/usr/share,
>> which doesn't work from a .whl file but does from the sdist.
>
> This should be possible I think, that’s just the “data” scheme. Maybe
> the tooling isn’t quite set up for it?

There is an issue here, though. I don't know if it's what Ansible and
Fabric try to do, but I have seen it in the past (MoinMoin, for
instance, used to do this). Packages want to install files to
arbitrary locations, maybe places like /etc or other FHS-mandated
locations on Unix. They can do this in setup.py by hacking locations
or using absolute paths. But the packaging standards only allow for
the various scheme locations.

For example, the Ansible setup.py does:

# always install in /usr/share/ansible if specified
# otherwise use the first module path listed
if '/usr/share/ansible' in module_paths:
    install_path = '/usr/share/ansible'
else:
    install_path = module_paths[0]
dirs=os.listdir("./library/")
data_files = []
for i in dirs:
    data_files.append((os.path.join(install_path, i),
glob('./library/' + i + '/*')))

That seems to be using absolute paths in data_files, which is
basically what I mean (a hack to force installation to an absolute
path). Joost is right that this won't install to
<virtualenv>/usr/share in a venv, though. That's just not how the data
scheme works.

I think we need to establish a "proper" solution for this situation.

My view is that Python packaging should not support installation of
files to anywhere other than subdirectories of the scheme locations.
Doing so simply won't work with wheels as things stand, and it'll
never be cross-platform (there is no /usr on Windows, for example).
But that does mean that we explicitly don't support hacks like the
above, so we break a certain proportion of packages.

For packages that need to install to absolute locations, I would
suggest that this be handled by a post-install script, that copies the
relevant files from the package data location to the final
destination. Such a script can check if it's running on an unsupported
platform, or in a virtualenv, and handle that case appropriately.

The files will not be removed on uninstall with this proposal. But the
distribution could include a pre-uninstall script for that (are
pre-uninstall scripts a thing?) or just accept that this is a
limitation of trying to use Python packaging tools in a non-standard
manner and advise users to clean up those files manually.

Comments? Is my proposal reasonable? Does anyone have an alternative
to suggest? Would Ansible, Fabric and any other projects using the
current hacks be willing to migrate to post-install scripts?

Paul.

PS Actually, installing Windows start menu shortcuts is logically the
same issue, and Windows projects have used post-install scripts to do
this for some time now - see pywin32 for an example.