Mailman 3 July 2014 - Distutils-SIG

Two fixes (at last)
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker April 29, 2018

April 29, 2018

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

windows permissions on bdist egg scripts, data?
by Erik Purins Aug. 25, 2014

Aug. 25, 2014

Hey there, I’m using python 2.7/setuptools 3.6 to build an egg of my code (checked into perforce) using my setup.py. This egg gets installed into a virtualenv. On osx/darwin this works fine. On windows, the bdist creates scripts and data whose permissions are read-only (attrib+r), and they cannot overwrite themselves in a subsequent build of that egg. Similarly, re-installing a new egg using virtualenv fails because the installed egg’s scripts and data are not writable. I thought it might be that when checked out from perforce, the permissions stayed read-only during the build of the egg, so I set all the files in my module to be read-write before building. Even when the original source files’ permissions are all set to read-write, the generated bdist egg script/data content end up becoming read-only. Couple of questions. 1. Since it works fine on osx, it seems like a bug in the windows implementation of setuptools. Is this known behavior, is there a reason for setting this on scripts/data on windows? 2. Is there a hook in the bdist egg build process I can use to set the attributes of the files correctly when the bdist egg is made? How/where should I define this in my setup.py? Kind regards, -e

2 2

Re: [Distutils] build a wheel with waf instead of setuptools
by Daniel Holth Aug. 1, 2014

Aug. 1, 2014

On Fri, Jul 25, 2014 at 5:51 AM, Daniel Holth <dholth(a)gmail.com> wrote: > Here's a little something I cooked up based on the waf (a build > system) playground/package example. It's a build script for wheel > (what else) that builds a .whl for wheel when you run "waf configure" > and then "waf package" with waf 1.8.0. I've tested it in Python 2.7. > > Waf is a build system that, unlike distutils, won't fall over > immediately when you try to extend it. One of its features is support > for building Python extensions, and it is itself written in Python. > > Right now for expedience instead of generating the METADATA or > entry_points.txt from setup() arguments it just copies them from files > at the root, and the command "waf dist" (for producing sdists) > includes too many files but that is easy to fix. In particular I liked > using ant_glob() a lot better than MANIFEST.in. The wscript does not > use MANIFEST.in. > > https://bitbucket.org/dholth/wheel/src/1bbbd010558afe215a01cfd238a26da42ad6… > > If there are any interested waf-wizards out there then we could take > the wheel building feature and pull it out of the individual wscript, > refine it a bit, and have another non-distutils way to publish Python > packages. > > Daniel Holth This kind of thing will require us to implement a flag that tells pip "setup.py cannot install; go through wheel" which is somewhere in the plans..

4 5

PEP draft on PyPI/pip package signing
by Giovanni Bajo July 29, 2014

July 29, 2014

Hello, on March 2013, on the now-closed catalog-sig mailing-list, I submitted a proposal for fixing several security problems in PyPI, pip and distutils[1]. Some of my proposals were obvious things like downloading packages through SSL, which was already in progress of being designed and implemented. Others, like GPG package signing, were discussed for several days/weeks, but ended up in discussion paralysis because of the upcoming TUF framework. 16 months later, we still don’t have a deployed solution for letting people install signed packages. I see that TUF is evolving, and there is now a GitHub project with documentation, but I am very worried about the implementation timeline. I was also pointed to PEP458, which I tried to read and found it very confusing; the PEP assumes that the reader must be familiar with the TUF academic paper (which I always found quite convoluted per-se), and goes with an analysis of integration of TUF with PyPI; to the best of my understanding, the PEP does not provide a clear answer to practical questions like: * what a maintainer is supposed to do to submit a new signed package * how can differ maintainers signal that they both maintain the same package * how the user interface of PyPI will change * what are the required security maintenance that will need to be regularly performed by the PyPI ops I’m not saying that the TUF team has no answers to these questions (in fact, I’m 100% sure of the opposite); I’m saying that the PEP doesn’t clearly provide such answers. I think the PEP is very complicated to read as it goes into integration details between the TUF architecture and PyPI, and thus it is very complicated to review and accept. I would love the PEP to be updated to provide an overview on the *practical* effects of the integration of TUF within PyPI/pip, that must be fully readable to somebody with zero previous knowledge of TUF. As suggested by Richard Jones during EuroPython, I isolated the package signing sections from my original document, evolved them a little bit, and rewritten them in PEP format: https://gist.github.com/rasky/bd91cf01f72bcc931000 To the best of my recollection, in the previous review round, there were no critical issues found in the design. It might well be that TUF provides more security in some of the described attack scenarios; on the other hand, my proposal: * is in line with the security of (e.g..) existing Linux distros * is very simple to review, analyze and discuss for anybody with even a basic understanding of security * is much simpler than TUF * is a clear step forward from the current situation * cover areas not covered by PEP458 (e.g.: increasing security of account management on PyPI) * can be executed in 2-3 months (to the alpha / pre-review stage), and I volunteer for the execution. I thus solicit a second round of review of my proposal; if you want me to upload to Google Docs for easier of commenting, I can do that as well. I would love to get the PEP to its final form and then ask for a pronouncement. I apologize in advance if I made technical mistakes in the PEP format/structure; it is my first PEP. [1] See here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTW… -- Giovanni Bajo :: rasky(a)develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it

9 26

Other ideas from today's packaging meetup at EuroPython
by Richard Jones July 27, 2014

July 27, 2014

Several great ideas came out of today's meetup. Some of those I'll leave to the proponents themselves to post about, but a couple of little nuggets for thought: 1. reject wheel uploads in the absence of an sdist in the index (the linux guys were really happy about that as a proposal ;) 2. add a system-wide configuration option to pip etc. so that there could be a system-wide override of the package index to use Richard

11 24

PEP 470 discussion, part 3
by Richard Jones July 25, 2014

July 25, 2014

I have been mulling over PEP 470 for some time without having the time to truly dedicate to addressing it. I believe I'm up to date with its contents and the (quite significant, and detailed) discussion around it. To summarise my understanding, PEP 470 proposes to remove the current link spidering (pypi-scrape, pypi-scrape-crawl) while retaining explicit hosting (pypi-explicit). I believe it retains the explicit links to external hosting provided by pypi-explicit. The reason given for this change is the current bad user experience around the --allow-external and --allow-unverified options to pip install. That is, that users currently attempt to install a non-pypi-explicit package and the result is an obscure error message. I believe the current PEP addresses the significant usability issues around this by swapping them for other usability issues. In fact, I believe it will make matters worse with potential confusion about which index hosts what, potential masking of release files or even, in the worst scenario, potential spoofing of release files by indexes out of the control of project owners. I would like us to consider instead working on the usability of the existing workflow, by rather than throwing an error, we start a dialog with the user: $ pip install PIL Downloading/unpacking PIL PIL is hosted externally to PyPI. Do you still wish to download it? [Y/n] y PIL has no checksum. Are you sure you wish to download it? [Y/n] y Downloading/unpacking PIL Downloading PIL-1.1.7.tar.gz (506kB): 506kB downloaded ... Obviously this would require scraping the site, but given that this interaction would only happen for a very small fraction of projects (those for which no download is located), the overall performance hit is negligible. The PEP currently states that this would be a "massive performance hit" for reasons I don't understand. The two prompts could be made automatic "y" responses for tools using the existing --allow-external and --allow-unverified flags. I also note that PEP 470 says "PEP 438 proposed a system of classifying file links as either internal, external, or unsafe", whereas PEP 438 has no mention of "unsafe". This leads "unsafe" to never actually be defined anywhere that I can see. Finally, the Rejected Proposals section of the PEP appears to have a couple of justifications for rejection which have nothing whatsoever to do with the Rationale ("PyPI is fronted by a globally distributed CDN...", "PyPI supports mirroring...") As Holger has already indicated, that second one is going to have a heck of a time dealing with PEP 470 changes at least in the devpi case. "PyPI has monitoring and an on-call rotation of sysadmins..." would be solved through improving the failure message reported to the user as discussed above. Richard

7 23

Packaging meetup @ EuroPython 2014
by Richard Jones July 23, 2014

July 23, 2014

I've set the time for the packaging meetup to 11:00 this Thursday. There is currently no meeting room available for such a gathering, so we'll just meet outside in the garden. Richard

2 2

setup.py sdist not including my README.md
by Joshua Richardson July 22, 2014

July 22, 2014

Sorry for the Noob question, but I've been banging my head on this all afternoon. Following some tutorial I found somewhere on PyPI, I created a setup.py which included a line like this: # Get the long description from the relevant file with open(path.join(here, 'README.md'), encoding='utf-8') as f: long_description = f.read() and then in the setup() function, I pass long_description=long_description Somehow this worked, when I installed the package on my development box (Mac with setuptools 5.1), so I prematurely published it to PyPI. Unfortunately, when I tried installing it to a fresh CentOS box (setuptools 0.6c5-2.el5), I got this error: Downloading/unpacking Dumper Downloading Dumper-1.0.1.tar.gz Running setup.py egg_info for package Dumper .... <stacktrace omitted, let me know if you need it> IOError: [Errno 2] No such file or directory: '/tmp/pip-build/Dumper/README.md' So, I checked the tarball and saw that README.md is not in there. I read the setuptools developer guide, and decided maybe I should try: include_package_data=True, I re-ran "setup.py sdist" and checked the tarball, and the README.md is still not in there. Ok, maybe because my source-control is git. So, I added a MANIFEST.in, containing one line: "README.md". Still no luck. So, I added: package_data={ '': ['README.md'], }, Still a no-go. Now I'm feeling stupid. SOS. Thanks for reading this far! --josh

6 7