On 28 August 2014 20:35, Donald Stufft wrote:

...

Both Ansible and Fabric want to put stuff in <virtualenv>/usr/share, which doesn't work from a .whl file but does from the sdist.

This should be possible I think, that’s just the “data” scheme. Maybe the tooling isn’t quite set up for it?

There is an issue here, though. I don't know if it's what Ansible and Fabric try to do, but I have seen it in the past (MoinMoin, for instance, used to do this). Packages want to install files to arbitrary locations, maybe places like /etc or other FHS-mandated locations on Unix. They can do this in setup.py by hacking locations or using absolute paths. But the packaging standards only allow for the various scheme locations. For example, the Ansible setup.py does: # always install in /usr/share/ansible if specified # otherwise use the first module path listed if '/usr/share/ansible' in module_paths: install_path = '/usr/share/ansible' else: install_path = module_paths[0] dirs=os.listdir("./library/") data_files = [] for i in dirs: data_files.append((os.path.join(install_path, i), glob('./library/' + i + '/*'))) That seems to be using absolute paths in data_files, which is basically what I mean (a hack to force installation to an absolute path). Joost is right that this won't install to <virtualenv>/usr/share in a venv, though. That's just not how the data scheme works. I think we need to establish a "proper" solution for this situation. My view is that Python packaging should not support installation of files to anywhere other than subdirectories of the scheme locations. Doing so simply won't work with wheels as things stand, and it'll never be cross-platform (there is no /usr on Windows, for example). But that does mean that we explicitly don't support hacks like the above, so we break a certain proportion of packages. For packages that need to install to absolute locations, I would suggest that this be handled by a post-install script, that copies the relevant files from the package data location to the final destination. Such a script can check if it's running on an unsupported platform, or in a virtualenv, and handle that case appropriately. The files will not be removed on uninstall with this proposal. But the distribution could include a pre-uninstall script for that (are pre-uninstall scripts a thing?) or just accept that this is a limitation of trying to use Python packaging tools in a non-standard manner and advise users to clean up those files manually. Comments? Is my proposal reasonable? Does anyone have an alternative to suggest? Would Ansible, Fabric and any other projects using the current hacks be willing to migrate to post-install scripts? Paul. PS Actually, installing Windows start menu shortcuts is logically the same issue, and Windows projects have used post-install scripts to do this for some time now - see pywin32 for an example.

On 28 August 2014 21:33, Joost Molenaar wrote:

Question; would it help to use pkg_resources, or being able to specify locations relative to the python interpreter prefix?

Generally, I'd recommend putting resource files in the package directory and reading them via pkgutil.get_data (from the stdlib). The advantage of that is that it supports non-filesystem importers (such as zipfiles). pkg_resources lets you be a bit more flexible but at the cost of a runtime dependency on setuptools (which I personally find unacceptable, but YMMV). For projects that don't mind failing when run from anything other than the filesystem, you can use package.__file__ to get the filesystem location of your package and work from there - but that's zipfile-hostile and won't work with tools like cx_Freeze/py2exe/bbfreeze. If you are willing to restrict your code to only working when installed to the actual filesystem, you could also use sysconfig.get_path(name) to get the filesystem location of the various distutils locations (name='data' is where data_files are located, for example) and use that to locate your files. I'm not entirely sure why this would be better than using pkgutil, though. The key thing is to *not* try to force your data files into a fixed location that you then hard-code in your application. Paul

On 2 Sep 2014 03:19, "Marcus Smith" wrote:

...

My view is that Python packaging should not support installation of files to anywhere other than subdirectories of the scheme [...]

For packages that need to install to absolute locations, I would

suggest that this be handled by a post-install scriptlocations

[...] Comments?

I'd prefer, and think it's reasonable for a new wheel spec support

absolute paths (minimally, so we don't break existing distributions)

Nick posted this a while back: https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirec...

Yep, while we may eventually end up having to relent on allowing arbitrary code execution when installing from a wheel file, I'd like to see how far we can get without it (installers are often run with elevated privileges, so arbitrary code execution = teh horribleness) For the "install to arbitrary locations" case, that should just be a matter of defining a new scheme directory that means "paths here are relative to the root of the installation target". There's no defined process for how we go about doing that, but one plausible option would be to write up a PEP for Python 3.5 that defines the new scheme directory in sysconfig (off the top of my head, I suggest "root"), where files placed there will end up on Windows, POSIX and in virtual environments, and how it impacts other things (like the fact wheels using it will be platform dependent). The PyPA toolset would then take care of ensuring the new scheme directory was also supported on earlier Python versions. Cheers, Nick.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig