Hi all --
at long last, I have fixed two problems that a couple people noticed a
* I folded in Amos Latteier's NT patches almost verbatim -- just
changed an `os.path.sep == "/"' to `os.name == "posix"' and added
some comments bitching about the inadequacy of the current library
installation model (I think this is Python's fault, but for now
Distutils is slavishly aping the situation in Python 1.5.x)
* I fixed the problem whereby running "setup.py install" without
doing anything else caused a crash (because 'build' hadn't yet
been run). Now, the 'install' command automatically runs 'build'
before doing anything; to make this bearable, I added a 'have_run'
dictionary to the Distribution class to keep track of which commands
have been run. So now not only are command classes singletons,
but their 'run' method can only be invoked once -- both restrictions
enforced by Distribution.
The code is checked into CVS, or you can download a snapshot at
Hope someone (Amos?) can try the new version under NT. Any takers for
BTW, all parties involved in the Great "Where Do We Install Stuff?"
Debate should take a good, hard look at the 'set_final_options()' method
of the Install class in distutils/install.py; this is where all the
policy decisions about where to install files are made. Currently it
apes the Python 1.5 situation as closely as I could figure it out.
Obviously, this is subject to change -- I just don't know to *what* it
Greg Ward - software developer gward(a)cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
I've been aware that the distutils sig has been simmerring away, but
until recently it has not been directly relevant to what I do.
I like the look of the proposed api, but have one question. Will this
support an installed system that has multiple versions of the same
package installed simultaneously? If not, then this would seem to be a
significant limitation, especially when dependencies between packages
Assuming it does, then how will this be achieved? I am presently
managing this with a messy arrangement of symlinks. A package is
installed with its version number in it's name, and a separate
directory is created for an application with links from the
unversioned package name to the versioned one. Then I just set the
pythonpath to this directory.
A sample of what the directory looks like is shown below.
I'm sure there is a better solution that this, and I'm not sure that
this would work under windows anyway (does windows have symlinks?).
So, has this SIG considered such versioning issues yet?
Tim Docker timd(a)macquarie.com.au
Quantative Applications Division
qad16:qad $ ls -l lib/python/
drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1
-r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so
drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1
dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8
drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1
dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1
dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7
dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1
qad16:qad $ ls -l lib/python/1.1
lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so
lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so
lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so
lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so
lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so
lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1
lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1
lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1
lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so
lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so
lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3
lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7
lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1
I’m using python 2.7/setuptools 3.6 to build an egg of my code (checked into perforce) using my setup.py. This egg gets installed into a virtualenv.
On osx/darwin this works fine.
On windows, the bdist creates scripts and data whose permissions are read-only (attrib+r), and they cannot overwrite themselves in a subsequent build of that egg. Similarly, re-installing a new egg using virtualenv fails because the installed egg’s scripts and data are not writable.
I thought it might be that when checked out from perforce, the permissions stayed read-only during the build of the egg, so I set all the files in my module to be read-write before building. Even when the original source files’ permissions are all set to read-write, the generated bdist egg script/data content end up becoming read-only.
Couple of questions.
1. Since it works fine on osx, it seems like a bug in the windows implementation of setuptools. Is this known behavior, is there a reason for setting this on scripts/data on windows?
2. Is there a hook in the bdist egg build process I can use to set the attributes of the files correctly when the bdist egg is made? How/where should I define this in my setup.py?
On Fri, Jul 25, 2014 at 5:51 AM, Daniel Holth <dholth(a)gmail.com> wrote:
> Here's a little something I cooked up based on the waf (a build
> system) playground/package example. It's a build script for wheel
> (what else) that builds a .whl for wheel when you run "waf configure"
> and then "waf package" with waf 1.8.0. I've tested it in Python 2.7.
> Waf is a build system that, unlike distutils, won't fall over
> immediately when you try to extend it. One of its features is support
> for building Python extensions, and it is itself written in Python.
> Right now for expedience instead of generating the METADATA or
> entry_points.txt from setup() arguments it just copies them from files
> at the root, and the command "waf dist" (for producing sdists)
> includes too many files but that is easy to fix. In particular I liked
> using ant_glob() a lot better than MANIFEST.in. The wscript does not
> use MANIFEST.in.
> If there are any interested waf-wizards out there then we could take
> the wheel building feature and pull it out of the individual wscript,
> refine it a bit, and have another non-distutils way to publish Python
> Daniel Holth
This kind of thing will require us to implement a flag that tells pip
"setup.py cannot install; go through wheel" which is somewhere in the
on March 2013, on the now-closed catalog-sig mailing-list, I submitted a proposal for fixing several security problems in PyPI, pip and distutils. Some of my proposals were obvious things like downloading packages through SSL, which was already in progress of being designed and implemented. Others, like GPG package signing, were discussed for several days/weeks, but ended up in discussion paralysis because of the upcoming TUF framework.
16 months later, we still don’t have a deployed solution for letting people install signed packages. I see that TUF is evolving, and there is now a GitHub project with documentation, but I am very worried about the implementation timeline.
I was also pointed to PEP458, which I tried to read and found it very confusing; the PEP assumes that the reader must be familiar with the TUF academic paper (which I always found quite convoluted per-se), and goes with an analysis of integration of TUF with PyPI; to the best of my understanding, the PEP does not provide a clear answer to practical questions like:
* what a maintainer is supposed to do to submit a new signed package
* how can differ maintainers signal that they both maintain the same package
* how the user interface of PyPI will change
* what are the required security maintenance that will need to be regularly performed by the PyPI ops
I’m not saying that the TUF team has no answers to these questions (in fact, I’m 100% sure of the opposite); I’m saying that the PEP doesn’t clearly provide such answers. I think the PEP is very complicated to read as it goes into integration details between the TUF architecture and PyPI, and thus it is very complicated to review and accept. I would love the PEP to be updated to provide an overview on the *practical* effects of the integration of TUF within PyPI/pip, that must be fully readable to somebody with zero previous knowledge of TUF.
As suggested by Richard Jones during EuroPython, I isolated the package signing sections from my original document, evolved them a little bit, and rewritten them in PEP format:
To the best of my recollection, in the previous review round, there were no critical issues found in the design. It might well be that TUF provides more security in some of the described attack scenarios; on the other hand, my proposal:
* is in line with the security of (e.g..) existing Linux distros
* is very simple to review, analyze and discuss for anybody with even a basic understanding of security
* is much simpler than TUF
* is a clear step forward from the current situation
* cover areas not covered by PEP458 (e.g.: increasing security of account management on PyPI)
* can be executed in 2-3 months (to the alpha / pre-review stage), and I volunteer for the execution.
I thus solicit a second round of review of my proposal; if you want me to upload to Google Docs for easier of commenting, I can do that as well. I would love to get the PEP to its final form and then ask for a pronouncement.
I apologize in advance if I made technical mistakes in the PEP format/structure; it is my first PEP.
 See here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTW…
Giovanni Bajo :: rasky(a)develer.com
Develer S.r.l. :: http://www.develer.com
My Blog: http://giovanni.bajo.it
Several great ideas came out of today's meetup. Some of those I'll leave to
the proponents themselves to post about, but a couple of little nuggets for
1. reject wheel uploads in the absence of an sdist in the index (the linux
guys were really happy about that as a proposal ;)
2. add a system-wide configuration option to pip etc. so that there could
be a system-wide override of the package index to use
I have been mulling over PEP 470 for some time without having the time to
truly dedicate to addressing it. I believe I'm up to date with its contents
and the (quite significant, and detailed) discussion around it.
To summarise my understanding, PEP 470 proposes to remove the current link
spidering (pypi-scrape, pypi-scrape-crawl) while retaining explicit hosting
(pypi-explicit). I believe it retains the explicit links to external
hosting provided by pypi-explicit.
The reason given for this change is the current bad user experience around
the --allow-external and --allow-unverified options to pip install. That
is, that users currently attempt to install a non-pypi-explicit package and
the result is an obscure error message.
I believe the current PEP addresses the significant usability issues around
this by swapping them for other usability issues. In fact, I believe it
will make matters worse with potential confusion about which index hosts
what, potential masking of release files or even, in the worst scenario,
potential spoofing of release files by indexes out of the control of
I would like us to consider instead working on the usability of the
existing workflow, by rather than throwing an error, we start a dialog with
$ pip install PIL
PIL is hosted externally to PyPI. Do you still wish to download it? [Y/n]
PIL has no checksum. Are you sure you wish to download it? [Y/n] y
Downloading PIL-1.1.7.tar.gz (506kB): 506kB downloaded
Obviously this would require scraping the site, but given that this
interaction would only happen for a very small fraction of projects (those
for which no download is located), the overall performance hit is
negligible. The PEP currently states that this would be a "massive
performance hit" for reasons I don't understand.
The two prompts could be made automatic "y" responses for tools using the
existing --allow-external and --allow-unverified flags.
I also note that PEP 470 says "PEP 438 proposed a system of classifying
file links as either internal, external, or unsafe", whereas PEP 438 has no
mention of "unsafe". This leads "unsafe" to never actually be defined
anywhere that I can see.
Finally, the Rejected Proposals section of the PEP appears to have a couple
of justifications for rejection which have nothing whatsoever to do with
the Rationale ("PyPI is fronted by a globally distributed CDN...", "PyPI
supports mirroring...") As Holger has already indicated, that second one is
going to have a heck of a time dealing with PEP 470 changes at least in the
"PyPI has monitoring and an on-call rotation of sysadmins..." would be
solved through improving the failure message reported to the user as
Sorry for the Noob question, but I've been banging my head on this all
Following some tutorial I found somewhere on PyPI, I created a setup.py
which included a line like this:
# Get the long description from the relevant file
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
and then in the setup() function, I pass
Somehow this worked, when I installed the package on my development box
(Mac with setuptools 5.1), so I prematurely published it to PyPI.
Unfortunately, when I tried installing it to a fresh CentOS box
(setuptools 0.6c5-2.el5), I got this error:
Running setup.py egg_info for package Dumper
.... <stacktrace omitted, let me know if you need it>
IOError: [Errno 2] No such file or directory:
So, I checked the tarball and saw that README.md is not in there. I read
the setuptools developer guide, and decided maybe I should try:
I re-ran "setup.py sdist" and checked the tarball, and the README.md is
still not in there. Ok, maybe because my source-control is git. So, I
added a MANIFEST.in, containing one line: "README.md". Still no luck. So,
Still a no-go. Now I'm feeling stupid. SOS.
Thanks for reading this far! --josh