Python 3.3 / packaging - how to make installation conditional on platform?
I've a project which needs to install different data files depending on what platform the project is being installed on - e.g. win32, linux2, darwin. Using packaging setup.cfg, how can I indicate different files for different platforms, as well as some common files for all platforms? It's easy enough to do with setup.py, of course. It doesn't seem possible to use project-specific Python files to do this (via setup_hooks, for example), as when pysetup3 runs, the project directory is not in sys.path, so you can't import anything from e.g. files adjacent to setup.cfg. Although my current need is for data files, there might be situations where different packages need installing for different platforms (e.g. having shim or adapter functionality). What's the official packaging solution for this? Regards, Vinay Sajip
Vinay Sajip
Hi,
It doesn't seem possible to use project-specific Python files to do this (via setup_hooks, for example), as when pysetup3 runs, the project directory is not in sys.path, so you can't import anything from e.g. files adjacent to setup.cfg.
This is not a design choice but a bug: http://bugs.python.org/issue11637 We definitely want to support project-local hooks module, as they will certainly be used a lot during the transition years.
Ignore my previous post- I believe environment markers are what I was looking for. Don't know how I missed that :-)
I’m not sure data files support environment markers. Anyway, please report any info missing from the docs, they’re far from complete. Thanks for trying out packaging! Cheers
Éric Araujo
This is not a design choice but a bug: http://bugs.python.org/issue11637 We definitely want to support project-local hooks module, as they will certainly be used a lot during the transition years.
Thanks for the pointer. I may need this to be fixed in my pythonv fork, so I will attempt a patch there.
I’m not sure data files support environment markers. Anyway, please report any info missing from the docs, they’re far from complete.
On further study, I see you're right - environment markers will work only with some of the data. This gap can certainly be handled by setup hooks, once #11637 is fixed, but there's also another problem which I've encountered: on Windows at least, installing into the conventional appdata paths is fine for simple applications, but insufficient for applications that need to interact better with the Windows ecosystem. For example, if you want to install Powershell modules, they live not in conventional appdata locations but rather in subdirectories below a user's Documents folder, which is obtained by making calls into the Windows Shell API, such as SHGetKnownFolderPath or SHGetSpecialFolderPath. I can't see any references to these in packaging or sysconfig, so how are these values being determined? I appreciate that might not be a packaging issue but rather a sysconfig one, but it's certainly of interest to packagers :-) The only references I found to the shell APIs is in PC\bdist_winst\install.c, which (I presume) wouldn't be invoked during a "pysetup3 install" operation. So, it seems to me like some way is needed for a project to be able to inject additional categories like {userdocs} and path mappings for them, for its resources. I appreciate that this is more an issue for Windows than Unix platforms, but is there such a mechanism? Incidentally, I noticed a repeated assignment in packaging/install.py:_run_install_from_dir(): func = install_methods[install_method] is called twice in succession, I couldn't see why. Regards, Vinay Sajip
Le 18/06/2011 23:23, Vinay Sajip a écrit :
Éric Araujo
writes: This is not a design choice but a bug: http://bugs.python.org/issue11637 We definitely want to support project-local hooks module, as they will certainly be used a lot during the transition years. Thanks for the pointer. I may need this to be fixed in my pythonv fork, so I will attempt a patch there.
Thanks, I’ll be here to review the patch. It should be easy, just remember to add or update tests.
I’m not sure data files support environment markers. Anyway, please report any info missing from the docs, they’re far from complete. On further study, I see you're right - environment markers will work only with some of the data. This gap can certainly be handled by setup hooks, once #11637 is fixed, but there's also another problem which I've encountered: on Windows at least, installing into the conventional appdata paths is fine for simple applications, but insufficient for applications that need to interact better with the Windows ecosystem. For example, if you want to install Powershell modules, they live not in conventional appdata locations but rather in subdirectories below a user's Documents folder, which is obtained by making calls into the Windows Shell API, such as SHGetKnownFolderPath or SHGetSpecialFolderPath. I can't see any references to these in packaging or sysconfig, so how are these values being determined?
Well, I don’t think they are. The paths used for Windows are a transposition of UNIX paths to C:\\PythonX.Y, with Scripts, Lib, etc. subdirs; they’re not very Windows-y. IOW, I think improving the installation schemes on Windows for 3.3 to feel more native would be a good thing. See also http://bugs.python.org/issue9878 : Avoid parsing pyconfig.h and Makefile by autogenerating extension module (which means that using C APIs is wholly in line with future direction for sysconfig).
I appreciate that might not be a packaging issue but rather a sysconfig one, but it's certainly of interest to packagers :-)
This is definitely on-topic: sysconfig was extracted from distutils, is maintained by Tarek and evolved by the packaging fellowship. Installation is part of packaging.
The only references I found to the shell APIs is in PC\bdist_winst\install.c, which (I presume) wouldn't be invoked during a "pysetup3 install" operation.
Yep, only during pysetup run bdist_wininst, IIUC.
So, it seems to me like some way is needed for a project to be able to inject additional categories like {userdocs} and path mappings for them, for its resources. I appreciate that this is more an issue for Windows than Unix platforms, but is there such a mechanism?
Categories are extensible, so it’s a way. See also the short description of the packaging configure command in my other message.
Incidentally, I noticed a repeated assignment in packaging/install.py:_run_install_from_dir(): func = install_methods[install_method] is called twice in succession, I couldn't see why.
Just a small mistake in ebff46b232ed.
Éric Araujo
Well, I don’t think they are. The paths used for Windows are a transposition of UNIX paths to C:\\PythonX.Y, with Scripts, Lib, etc. subdirs; they’re not very Windows-y. IOW, I think improving the installation schemes on Windows for 3.3 to feel more native would be a good thing.
See also http://bugs.python.org/issue9878 : Avoid parsing pyconfig.h and Makefile by autogenerating extension module (which means that using C APIs is wholly in line with future direction for sysconfig).
Okay.
Categories are extensible, so it’s a way. See also the short description of the packaging configure command in my other message.
Thinking about it further - I'm not sure it'll be sufficient, in a number of cases, at least under Windows but potentially on other platforms, too. Where software is self-contained, having a fixed set of categories for different types of assets is sensible and sufficient. Where software needs to interoperate with other software, for example, things can get a little more complicated. I've already given an example relating to Powershell. As another example: say I'm developing some software to interoperate with legacy systems such as Microsoft Office (including Outlook) or Sharepoint, which are prevalent in the corporate world. The chances are very high that I'll need to install things in what is, from packaging's perspective, an essentially arbitrary set of locations. It's seems that it's not really possible to codify these using the schemes which we have now - schemes which are perfectly useful in the context of e.g. harmonising how we work with Linux distro packagers. I don't know if this has been brought up before, or discussed and rejected - but I think that something along the lines of Debian's prerm/postrm, preinst/postinst should be available, defined in the same way as setup_hooks in setup.cfg, to be called at the appropriate time in the pysetup3 lifecycle. This would provide some measure of standardisation, allowing projects with simple needs to omit them entirely, but those with more complex needs to do things which we can't foresee. As a bare minimum, prerm and postinst: these can rely on the distribution being installed and so have access to packages inside it. Of course, this is the rationale behind the post-install script option for bdist_wininst. Regards, Vinay Sajip
Vinay Sajip
I think that something along the lines of Debian's prerm/postrm, preinst/postinst should be available, defined in the same way as setup_hooks setup.cfg, to be called at the appropriate time in the pysetup3 lifecycle.
Okay, found the command hooks documentation :-) Perhaps a mention in the main section on the setup.cfg specification would be useful - near where setup_hooks are mentioned. Regards, Vinay Sajip
Éric Araujo
Categories are extensible, so it’s a way. See also the short description of the packaging configure command in my other message.
Actually, I'd like to make a suggestion re. project-specific categories. It's easy to enable these, as follows: 1. In the install_data constructor, initialise self.categories to an empty dictionary. 2. In install_data.expand_categories, add a "local_vars.update(self.categories)" after the "local_vars = get_paths()" statement. Just this change is sufficient to allow sufficient control over custom installation locations. For projects that need custom categories, they just need to do the necessary setup in an install_data pre-hook: def pre_install_data(cmd): cmd.categories['customcategory'] = '/path/for/my/custom/category' I've implemented this approach in the pythonv branch, and it works well. Comments welcome. Regards, Vinay Sajip
participants (2)
-
Vinay Sajip -
Éric Araujo