Mailman 3 April 2015 - Distutils-SIG

Missing documentation for proper handling of PEP420 namespace packages in setup.py
by Christoph Schmitt 22 Apr '15

22 Apr '15

I am using the newest versions of setuptools (15.0) and pip (6.1.1) with Python 3.4. I wanted to satisfy the following two requirements at the same time, but had some trouble: A) creating and installing two source distribution tarballs which have the same top level namespace package (no __init__.py for namespace package) with setuptools and pip B) having other packages within the same namespace package outside of <python>/site-packages in the PYTHONPATH (not managed by pip/setuptools) Since Python >= 3.3 supports namespace packages out of the box (PEP420) this seemed to me like a straightforward task. However, it turned out to be not. Either, setuptools would not find my modules, pip would not install them properly or requirement B) was broken. The solution that worked for me, was to omit the namespace_packages=['whatever_namespace'] declaration in setup.py (to prevent the creation of *-nspkg.pth files by pip) and not to use find_packages, which does not comply with PEP420 (no packages found). This solution is somewhat counter-intuitive and I am not sure, whether it is an intended/valid configuration of a setup.py file. Definitely, it is not clear from the documentation (which has a section about namespace packages btw.). Since I read into https://bitbucket.org/pypa/setuptools/issue/97 I now know that convenient support PEP420 is not easy to achieve for setuptools (regarding find_packages). However it would have been very helpful, if the documentation explained how to handle PEP420-compliant namespace packages (without any __init__.py) in setup.py. At least I would have expected a hint that there are caveats regarding PEP420 with a link to issue 97. I also created a minimal example to reproduce the issue, which I can provide if anyone is interested. Kind regards, Christoph Schmitt

4 4

Python 3.x Adoption for PyPI and PyPI Download Numbers
by Donald Stufft 21 Apr '15

21 Apr '15

Just thought I'd share this since it shows how what people are using to download things from PyPI have changed over the past year. Of particular interest to most people will be the final graphs showing what percentage of downloads from PyPI are for Python 3.x or 2.x. As always it's good to keep in mind, "Lies, Damn Lies, and Statistics". I've tried not to bias the results too much, but some bias is unavoidable. Of particular note is that a lot of these numbers come from pip, and as of version 6.0 of pip, pip will cache downloads by default. This would mean that older versions of pip are more likely to "inflate" the downloads than newer versions since they don't cache by default. In addition if a project has a file which is used for both 2.x and 3.x and they do a ``pip install`` on the 2.x version first then it will show up as counted under 2.x but not 3.x due to caching (and of course the inverse is true, if they install on 3.x first it won't show up on 2.x). Here's the link: https://caremad.io/2015/04/a-year-of-pypi-downloads/ Anyways, I'll have access to the data set for another day or two before I shut down the (expensive) server that I have to use to crunch the numbers so if there's anything anyone else wants to see before I shut it down, speak up soon. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

5 9

elements of setuptools/wheel/FHS file categories proposal
by Daniel Holth 20 Apr '15

20 Apr '15

It might be helpful to discuss the fine grained install scheme proposal as several proposals: - Add more install paths (categories) to the set currently used by distutils/sysconfig https://docs.python.org/3/library/sysconfig.html#installation-paths . Define sensible defaults for each platform. - Allow the paths to be individually overridden for each installed package, in the same way that python setup.py install --install-purelib=/some-directory can override the categories we have now. - Record the { category : path } mapping used during installation. - Provide an API mapping (distribution name, category, relative path within category) to help applications using data that is only accessible via the module loader. https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoad… - Provide an API mapping the same to paths on the filesystem. - Make the recorded mapping available in a predictable location, so it can be perhaps understood by non-Python code. - Allow setup.py's setup() call to install files relative to each defined category. - Extend Python binary package formats to support the new categories, so the { category : path } mapping can be set at install time and not at build time.

4 18

Add additional file categories for distutils, setuptools, wheel
by Daniel Holth 19 Apr '15

19 Apr '15

I am working on a minor update to the wheel format to add more categories under which files can be installed. Constructive comments welcome. Distutills, setuptools, wheel currently have the best support for installing files relative to ('purelib', 'platlib', 'headers', 'scripts', 'data') with 'data' usually meaning '/' or the root of the virtualenv. In practice only exactly one of the 'purelib' or 'platlib' locations (which are usually mapped to the same directory on disk), and sometimes 'scripts' is used for any given package, and Python packages have no good way of loading any of their package files at run time if they are installed into any location not relative to sys.path. This works fairly well for Python libraries, but anyone packaging applications for a Linux distribution is required to follow the filesystem hierarchy standard or FHS. http://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_G…, http://www.debian.org/releases/etch/amd64/apcs02.html, http://www.pathname.com/fhs/ It would help Linux distribution package maintainers and Python application (not library) developers if wheel had better support for installing files into the FHS, but it would help everyone else who wanted to generate cross-platform packages if the FHS was not hardcoded as paths in the data/ category. To that end we should come up with some additional categories that map, or do not map, to the FHS based on the end user's platform. Distutils2 had the concept of resource categories: http://alexis.notmyidea.org/distutils2/setupcfg.html#resources """ Default categories are: * config * appdata * appdata.arch * appdata.persistent * appdata.disposable * help * icon * scripts * doc * info * man """ GNU has directory variables: https://www.gnu.org/prep/standards/html_node/Directory-Variables.html , for example prefix, exec_prefix, bindir, sbindir. Bento has a list based on the GNU paths, and allows new paths to be defined: prefix: install architecture-independent files eprefix: install architecture-dependent files bindir: user executables sbindir: system admin executables libexecdir: program executables sysconfdir: read-only single-machine data sharedstatedir: modifiable architecture-independent data localstatedir: modifiable single-machine data libdir: object code libraries includedir: C header files oldincludedir: C header files for non-gcc datarootdir: read-only arch.-independent data root datadir: read-only architecture-independent data infodir: info documentation localedir: locale-dependent data mandir: man documentation docdir: documentation root htmldir: html documentation dvidir: dvi documentation pdfdir: pdf documentation psdir: ps documentation I would like to add Bento's list to wheel and to setuptools. We would fix the disused setup.py setup(data_files = ) argument so that it could be used with $ substitution, or provide a utility function that would expand them in setup.py itself: data_files = { '$libdir/mylib' : [ 'some_library_file'], '$datadir/mydata' : ['some_data_file']} We would provide a default configuration file that mapped the categories to their installed locations. We would store the actual paths used at install time, so a package could look for files relative to the $datadir used when it was installed. Then it would be easier to distribute Python programs that need to install some files to paths that are not relative to the site-packages directory. - Daniel Holth

6 12

Idea: move accepted interoperability specifications to packaging.python.org
by Nick Coghlan 19 Apr '15

19 Apr '15

Daniel's started work on a new revision of the wheel specification, and it's crystallised a concern for me that's been building for a while: the Python Enhancement Proposal process is fundamentally a *change management* process and fundamentally ill-suited to acting as a *hosting service* for the resulting reference documentation. This is why we're seeing awkward splits like the one I have in PEP 440, where the specification itself is up top, with the rationale for changes below, and the large amounts of supporting material in PEP 426, where the specification is mixed in with a lot of background and rationale that isn't relevant if you just want the technical details of the latest version of the format. It also creates a problem where links to PEP based reference documents are fundamentally unstable - when we define a new version of the wheel format in a new PEP then folks are going to have to follow the daisy chain from PEP 427 through to the new PEP, rather than having a stable link that automatically presents the latest version of the format, perhaps with archived copies of the old version readily available. I think I see a way to resolve this, and I believe it should be fairly straightforward: we could create a "specifications" section on packaging.python.org, and as we next revise them, we start migrating the specs themselves out of the PEP system and into packaging.python.org. This would be akin to the change in the Python 3.3, where the specification of the way the import system worked finally moved from PEP 302 into the language reference. Under that model, the wheel 2.0 would be specifically focused on describing and justifying the *changes* between 1.0 and 2.0, but the final spec itself would be a standalone document living on packaging.python.org, and prominently linked to from both PEP 427 (which it would Supersede) and from the new PEP. This approach also gives a much nicer model for fixing typos in the specifications - those would just be ordinary GitHub PR's on the packaging.python.org repo, rather than needing to update the PEPs repo. Thoughts? Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 6

Secondary package indexes
by Ionel Cristian Mărieș 19 Apr '15

19 Apr '15

Hello, Probably this has been discussed in the past but I'm asking anyway cause I'm not sure what's it at now. Currently there's this problem with wheels, many package authors don't publish them for the platforms I'm using. I'm speaking about the wheels that need a compiler and/or other annoying dependencies. It would be really nice if one could configure pip to look into a secondary (external) indexes. This way someone could make his own index with windows wheels, another person could make an index with wheels for ubuntu 14.04 and people can use those to avoid the pain of compiling the packages. This is already popular, eg: http://www.lfd.uci.edu/~gohlke/pythonlibs/ but not readily usable with pip. Maybe PyPI could even provide hosting for these 3rd party wheels, this would make publishing very easy and lower the entry bar. I believe we'll never get package authors to publish wheels for platforms they don't care about. But other people might - we just need to make this easy and convenient. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

2 5

How to sign a exe created with bdist_wininst?
by Brian Cole 18 Apr '15

18 Apr '15

We've recently converted over to using bdist_wininst for creating our Windows .exe installers for our libraries. Unfortunately, whenever we use the Windows signtool utility to cryptographically sign our installer it appears to corrupt the .exe and it can't be run anymore. The error message thrown by Windows is "Setup program invalid or damaged". My best guess at this point is that bdist_wininst is creating a checksum of the file somehow and signtool is altering the file in such a way to invalidate that checksum. The commands we're using at this point is like this: python3.4.exe setup.py bdist_wininst --target-version 3.4 --bitmap OurLogo --title OurTitle-OurVersion cp DistUtilsSetupFileName.exe OurSetupFileName.exe call "C:\program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools\vsvars32.bat" signtool sign /n OurCompany /t http://timestamp.verisign.com/scripts/timstamp.dll /d OurProject /du OurWebsite OurSetupFileName.exe Anyone know of a way to cryptographically sign an .exe installer from bdist_wininst? Thanks, Brian

4 4

How to sign a exe created with bdist_wininst?
by Brian Cole 17 Apr '15

17 Apr '15

We've recently converted over to using bdist_wininst for creating our Windows .exe installers for our libraries. Unfortunately, whenever we use the Windows signtool utility to cryptographically sign our installer it appears to corrupt the .exe and it can't be run anymore. The error message thrown by Windows is "Setup program invalid or damaged". My best guess at this point is that bdist_wininst is creating a checksum of the file somehow and signtool is altering the file in such a way to invalidate that checksum. The commands we're using at this point is like this: python3.4.exe setup.py bdist_wininst --target-version 3.4 --bitmap OurLogo --title OurTitle-OurVersion cp DistUtilsSetupFileName.exe OurSetupFileName.exe call "C:\program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools\vsvars32.bat" signtool sign /n OurCompany /t http://timestamp.verisign.com/scripts/timstamp.dll /d OurProject /du OurWebsite OurSetupFileName.exe Anyone know of a way to cryptographically sign an .exe installer from bdist_wininst? Thanks, Brian

1 0

pip/warehouse feature idea: "help needed"
by Nick Coghlan 16 Apr '15

16 Apr '15

Guido mentioned in his PyCon keynote this morning that we don't currently have a great way for package authors to ask for help from their user base. It occurred to me that it could be useful to have a "Help needed" feature on PyPI (after the Warehouse migration) where package maintainers could register requests for assistance, such as: * looking for new maintainers * requests for help with Python 3 support * links to specific issues a maintainer would like help with * links to donation pages (including links to Patreon, Gratipay, etc) * links to crowdfunding campaigns for specific new features * links to CVs/LinkedIn if folks are looking for work Given a requirements.txt file, pip could then gain a "help-needed" command that pulled the "help needed" entries for the named projects. The general idea would be to provide a direct channel from project maintainers that may need help to their users who may be in a position to provide that help. It wouldn't need to be too complicated, just a Markdown field that maintainers could edit. In some cases, software is backed by folks that already have a sustainable support model. For these it could be nice if the Markdown field could be used to say "Help not needed", and give credit to the people or orgs supporting them. It's not something we can do anything about until after the Warehouse migration, but I figured I'd mention it while I was thinking about it :) Cheers, Nick.

9 21

name of the dependency problem
by Robin Becker 16 Apr '15

16 Apr '15

After again finding that pip doesn't have a correct dependency resolution solution a colleage and I discussed the nature of the problem. We examined the script capture of our install and it seems as though when presented with level 0 A A level 1 1.4<= C level 0 B B level 1 1.6<= C <1.7 pip manages to download version 1.8 of C(Django) using A's requirement, but never even warns us that the B requirement of C was violated. Surely even in the absence of a resolution pip could raise a warning at the end. Anyhow after some discussion I realize I don't even know the name of the problem that pip should try to solve, is there some tree / graph problem that corresponds? Searching on dependency seems to lead to topological sorts of one kind or another, but here we seem to have nodes with discrete values attached so in the above example we might have (assuming only singleton A & B) R --> A R --> B A --> C-1.4 A --> C-1.6 A --> C-1.6.11 A --> C-1.7 A --> C-1.8 B --> C-1.6 B --> C-1.6.11 so looking at C equivalent nodes seems to allow a solution set. Are there any real problem descriptions / solutions to this kind of problem? -- Robin Becker

10 17