Mailman 3 December 2015 - Distutils-SIG

PyPi not allowing duplicate filenames
by Chris Barker 20 Apr '20

20 Apr '20

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

13 24

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Surviving a Compromise of PyPI - PEP 458 and 480
by Vladimir Diaz 12 Feb '20

12 Feb '20

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

7 53

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Versioned trove classifiers for Django
by James Bennett 06 Apr '17

06 Apr '17

Following up on some IRC discussion with other folks: There is precedent (Plone) for PyPI trove classifiers corresponding to particular versions of a framework. So I'd like to get feedback on the idea of expanding that, particularly in the case of Django. The rationale here is that the ecosystem of Django-related packages is quite large, but -- as I know all too well from a project I'm working on literally at this moment -- it can be difficult to ensure that all of one's dependencies are compatible with the version of Django one happens to be using. Adding trove classifier support at the level of individual versions of Django would, I think, greatly simplify this: tools could easily analyze which packages are compatible with an end user's chosen version, there'd be far less manual guesswork, etc., and the rate of creation of new classifiers would be relatively low (we tend to have one X.Y release/year or thereabouts, and that's the level of granularity needed). Assuming there's consensus around the idea of doing this, what would be the correct procedure for getting such classifiers set up and maintained?

5 17

namespace_package
by KP 12 Sep '16

12 Sep '16

I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head. The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages. If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path. Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked? I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.

4 19

Missing IPv6 support on pypi.python.org
by Baptiste Jonglez 13 Jul '16

13 Jul '16

Hi, pypi.python.org is currently not reachable over IPv6. I know this issue was brought up before [1,2]. This is a real issue for us, because our backend servers are IPv6-only (clients never need to talk to backend servers, they go through IPv4-enabled HTTP frontends). So, deploying packages from pypi on the IPv6-only servers is currently a pain. What is the roadmap to add IPv6 support? It seems that Fastly has already deployed IPv6 [3]. Thanks, Baptiste [1] https://mail.python.org/pipermail/distutils-sig/2014-June/024465.html [2] https://bitbucket.org/pypa/pypi/issues/90/missing-ipv6-connectivity [3] http://bgp.he.net/AS54113#_prefixes6

4 6

Idea: Positive/negative extras in general and as replacement for features
by Ronny Pfannschmidt 09 Mar '16

09 Mar '16

Hello everyone, the talk about the sqlalchemy feature extra got me thinking what if i could specify extras that where installed by default, but users could opt out a concrete example i have in mind is the default set of pytest plugins i would like to be able to externalize legacy support details of pytest without needing people to suddenly depend on pytest[recommended] instead of pytest to have what they know function as is instead i would like to be able to do something like a dependency on pytest[-nose,-unittest,-cache] to subtract the items the extras declaration in setup.py would just include a * sign at the beginning an elaborate concrete example could be extras = { +cache: pytest-cache +lastfailed: pyytest[+cache], pytest-lastfailed +looponfail: pytest[+lastfailed], pytest-looponfail +unittest: pytest-unittest +nose: pytest[+unittest], pytest-nose } also a dependency declaration using the + sign in the extras should not imply the default extras of the package, while usage of the - should so depending on pytest[+unittest] would only imply the unittest support but depending on pytest[-nose] would include all positive extras except for nose please note in particular the dependencies on other postive extras, those are put in, so a negative for unittest can imply that nose can't be sensibly used as well if +nose did instead depend directly on pytest-unittest, then excluding it would be a unreasonably tricky resolving algorithm with potential for lots of mistakes instead spelling the direct dependency on positives/negatives can resolve inside of a package and still leave room for more outside of it this is in particular relevant if a package with extras is depended on twice in different packages, because in that case each of the dependence's requirements should add up to the combined set there is also room for fleshing out algorithms for interacting the positive/negative dependence sets, but i'll leave that for later as an addition to that later on there could be support for partial wheel wheels, so features could be materealiyed as wheel package with a special name and build tools could fall back to making them from a sdist as a example, there would be a sqlalchemy package as source wheel, a sqlalchemy*cext package as windows wheels, and pip would have to find a source distribution to compile the wheel package -- Ronny

5 4

Trouble using setuptools to build separate c++ extension
by Thomas Nyberg 04 Jan '16

04 Jan '16

Hello I'm having trouble understanding the right way to build a c++ module using setuptools. I've been reading the docs, but I'm confused where I should be putting my build options. Everything builds fine on its own. I have my sources in src/ and my headers in include/. My first problem is that I'm having trouble figuring out where to put my build flags. Here is the Makefile I'm currently using: -------- srcs=$(wildcard *.cpp) srcs+=$(wildcard src/*.cpp) objs=$(patsubst %.cpp,%.o,$(srcs)) cc=g++ ccflags=-std=c++11 -g -O3 -fPIC includes=-I. -I./include/ -I/usr/include/python2.7/ -I/usr/include/boost libflags=-L. -L/usr/lib/x86_64-linux-gnu ldflags= -shared -Wl,--export-dynamic patent_computer_cpp.so: $(objs) $(cc) $(libflags) $(ldflags) $(objs) -o patent_computer_cpp.so -lboost_python -lpython2.7 %.o:%.cpp $(cc) $(ccflags) $(includes) -c -o $@ $< -------- Unfortunately I can't post the sources, but they compile fine to produce the `patent_computer_cpp.so` file which can be imported as a module. Maybe I should also point out that I'm using boost-python (I don't think this is the issue though). I just can't figure out how to get setuptools.Extension to use these build flags. I've seen recommendations online saying that I should set CFLAGS as an environment variable and set OPT='' as an environment variable as well, but this just feels wrong given the simplicity of my setup. (Besides the shared object doesn't seem to compile correctly in this case.) I've tried using the extra_compile_args option in setup.py, but that fails. Is there a way to avoid setting environment variables like this or is this the accepted way to build this kind of software? Am I missing some obvious docs somewhere? Thanks for any help. Cheers, Thomas

2 1

Reinstall with pip
by Carlos Barera 27 Dec '15

27 Dec '15

Hi, I have an automation task that installs a python package from within the project dir using: pip install . What's the right way to reinstall the same package (same version)? --upgrade + --force-reinstall ? Is --no-deps recommended ? How about uninstalling and reinstalling ? Cheers, Carlos

2 2

PEP 470 - Repository Side Complete
by Donald Stufft 24 Dec '15

24 Dec '15

Just as an FYI, as the specified time has passed since the warnings specified in PEP 470, I went ahead and finished the implementation of PEP 470 on PyPI and Test PyPI. I hope to have a new version of pip out that has it’s side of that as well soon. Hope everyone has a Merry Christmas! ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

Building and installing packages on a (unix) system lacking network accesss
by KM 24 Dec '15

24 Dec '15

Greetings distutils-sig, I have a project with an autogenerated structure - that is, I ran a "helper" application which creates a directory structure and a setup.py for me. I am trying to build this package in a virtualenv on an isolated machine, necessiating the step of downloading all the prerequisite packages and making them available. I have done this and I can successfully install most of the prerequisites by passing "--no-index" and "-f <url>" to pip, making my full command-line: pip install -e . --no-index -f <url> This works up until a certain point, where pip (or something launched by pip) tries to download from the internet. I tried adding ''--global-option '--no-index -f <url>'" to my options, but this failed. I get the error: option --no-index -f <url> not recognized. This occurs right after "Running setup.py install for TurboGears2" in the pip output and after it has installed a number of other packages. [Using --global-option was a shot in the dark, after all]. The package it is looking for is pytz. It appears to not be finding the package when it looks for it the first time - i.e., in the build log, it doesn't announce that it has found the package, it just moves on to the next package. The system is a FreeBSD 10.2 system, vanilla except for some packages like bash, python2.7 and well openjdk which installs a bunch of X stuff, unfortunately. It runs as a VM under VirtualBox (latest version) on Windows 8 (kept "up-to-date" but on version 8 not 8.1). The pip in use is 7.0.3 under python 2.7 (because that is the latest version of pip available in the FreeBSD ports). I am hoping for some feedback or information that will allow me to install my application with all of its dependencies downloaded from a locally-available url or directory. Does anyone have any suggestions for me? KM

3 2

setup.py install using pip
by Erik Bray 23 Dec '15

23 Dec '15

Hi all, I've been on vacation for a bit in general, and on vacation from this mailing list even longer. I'm not entirely caught up yet on the latest developments so apologies if something like this is entirely moot by now. But I have seen some discussions here and in other lists related to using pip for all installations, and phasing out the old distutils `./setup.py install` (eg. [1]). This is not a new discussion, and there are many related discussions, for example, about changing setuptools not to default to egg installs anymore (see [2]). I'm definitely all for this in general. Nowadays whenever I install a package from source I run `pip install .` But of course there are a lot of existing tools, not to mention folk wisdom, assuming `./setup.py install`. We also don't want to change the long-existing behavior in setuptools. I have a modest proposal for a small addition to setuptools that might be helpful in a transition away from using setuptools+distutils for installation. This would be to add a `--pip` flag to setuptools' install command (or possibly straight in distutils too, but might as well start with setuptools). Therefore, running $ ./setup.py install --pip would be equivalent to running $ pip install . By extension, running $ ./setup.py install --pip --arg1 --arg2=foo would be equivalent to $ pip install --install-option="--arg1" --install-option="--arg2=foo" . and so on. By making `--pip` opt-in, it does not automatically break backward compatibility for users expecting `./setup.py install` to use easy_install. However, individual users may opt into it globally by adding [install] pip = True to their .pydistutils.cfg. Similarly, package authors who are confident that none of their users are ever going to care about egg installs (e.g. me) can add the same to their project's setup.cfg. Does something like this have any merit? I hacked together a prototype which follows the sig line. It's just a proof of concept, but seems to work in the most basic cases. I'd like to add it to my own projects too, but would appreciate some peer review. Thanks, Erik [1] https://mail.scipy.org/pipermail/numpy-discussion/2015-November/074142.html [2] https://bitbucket.org/pypa/setuptools/issues/371/setuptools-and-state-of-pe… $ cat pipinstall.py from distutils.errors import DistutilsArgError from setuptools.command.install import install as SetuptoolsInstall class PipInstall(SetuptoolsInstall): command_name = 'install' user_options = SetuptoolsInstall.user_options + [ ('pip', None, 'install using pip; ignored when also using ' '--single-version-externally-managed') ] boolean_options = SetuptoolsInstall.boolean_options + ['pip'] def initialize_options(self): SetuptoolsInstall.initialize_options(self) self.pip = False def finalize_options(self): SetuptoolsInstall.finalize_options(self) if self.single_version_externally_managed: self.pip = False if self.pip: try: import pip except ImportError: raise DistutilsArgError( 'pip must be installed in order to install with the ' '--pip option') def run(self): if self.pip: import pip opts = (['install', '--ignore-installed'] + ['--install-option="{0}"'.format(opt) for opt in self._get_command_line_opts()]) pip.main(opts + ['.']) else: SetuptoolsInstall.run(self) def _get_command_line_opts(self): # Generate a mapping from the attribute name associated with a # command-line option to the name of the command-line option (including # an = if the option takes an argument) attr_to_opt = dict((opt[0].rstrip('=').replace('-', '_'), opt[0]) for opt in self.user_options) opt_dict = self.distribution.get_option_dict(self.get_command_name()) opts = [] for attr, value in opt_dict.items(): if value[0] != 'command line' or attr == 'pip': # Only look at options passed in on the command line (ignoring # the pip option itself) continue opt = attr_to_opt[attr] if opt in self.boolean_options: opts.append('--' + opt) else: opts.append('--{0}{1}'.format(opt, value[1])) return opts @staticmethod def _called_from_setup(run_frame): # A hack to work around a setuptools hack return SetuptoolsInstall._called_from_setup(run_frame.f_back)

5 14

cached index url
by Carlos Barera 18 Dec '15

18 Dec '15

Hi, In the past I installed devpi locally on my laptop to serve as a local mirror of pypi and for testing of the distribution and install of packages. I stopped using it. But now, seems like the index url of my local devpi server is still cached somewhere. I thought that the index url should be found in pip.conf but I don't seem to have any pip.conf anywhere on my Mac. However, running python setup.py install fails as it's still trying to access my old devpi server for some reason. Questions: 1. I guess it must be cached some place else. Any idea where? 2. Why don't I have any pip.conf on my Mac Regards, Carlos

2 1

pypi simple index Inbox x
by Carlos Barera 17 Dec '15

17 Dec '15

Hi, I'm using install_requires in setup.py to specify a specific package my project is dependant on. When running python setup.py install, apparently the simple index is used as an older package is taken from pypi. While in https://pypi.python.org/pypi, there's a newer package. When installing directly using pip, the latest package is installed successfully. I noticed that the new package is only available as a wheel and older versions of setup tools won't install wheels for install_requires. However, upgrading setuptools didn't help. Several questions: 1. What's the difference between the pypi simple index and the general pypi index? 2. Why is setup.py defaulting to the simple index? 3. How can I make the setup.py triggered install use the main pypi index instead of simple Thanks!

2 1

workflow recommendations to update requirements.txt
by Chris Jerdonek 16 Dec '15

16 Dec '15

Hi, I have a development workflow question I was wondering if people on this list had a recommended solution for. Say you're working on a web application that you deploy using a requirements.txt file. And say you have a set of "abstract dependencies" that your application depends on. What are some convenient ways of storing your abstract dependencies in source control and periodically generating an updated requirements file from that information (e.g. when your dependencies come out with new versions)? The main idea that occurs to me is making a setup.py for the purposes of representing your abstract dependencies (e.g. using "install_requires," etc), creating a new virtualenv, running "pip install .", and then "pip freeze." One problem with this approach is that the pip freeze output includes an entry for the setup.py application itself, when the output should only include the _dependencies_ of the application and not the application itself. It also seems clunky to me to create a virtualenv and install dependencies only for the purposes of computing dependencies. Thanks for any help or suggestions. --Chris

4 4

Metadata 2.0: Warning if optional features are missing
by Paul Moore 16 Dec '15

16 Dec '15

This is more a thought for something that would be good to include in Metadata 2.0, or whatever ends up taking its place. I was installing some packages on a new PC, that doesn't have a compiler. As I did so, I noticed a dependency on sqlalchemy fly by, and I thought "oh, that's going to fail", as it's always needed a compiler. But it didn't. I presume sqlalchemy has an optional dependency on something like a speedup module. What would be good is a way for packages to declare that they provide optional features like this, and for pip (and user code) to be able to introspect that data. The n"extras" feature in Metadata 2.0 is sort of like that, but in an "on request" form (I can install foo[speedups] and that has additional requirements, but the install will fail if they are not present). What I'd like to be able to do: 1. pip install sqlalchemy works, but shows a warning "optional feature speedups not installed - no C compiler" 2. A command to list any installed packages with optional features that aren't installed: $ pip options -l sqlalchemy speedups [C compiler] 3. A command to reinstall the currently installed version with new options pip install --add-options sqlalchemy[speedups] (Note that a plain pip install doesn't do this, as it won't reinstall. And --upgrade or --ignore-installed will install newer versions). The use case is the one I described: installing on a system with no compiler (and no wheels provided by the project) where I want to do the install for now, but know I'm missing some stuff, and at a later date when I get a compiler installed I want to update things to add what I would have got if I'd had the compiler in the first place. Thoughts? Is this something that could be practical (obviously it needs projects to *add* the metadata, but they'd need the facility to be there before they could do so). Paul

5 8

Release/file cannot add file
by Kevin Glisson 15 Dec '15

15 Dec '15

I am having trouble adding files to my package. It seems as no matter what I try I am always getting the error "file exists". This is after version bumpsm the indexes lists no files at all. I have tried with both `setup.py upload` and `twine upload`. The rest of the request and meta data for the release gets update correctly. I believe this might have something to do with the fact that I took over the namespace from a different package. My package has never successfully been publish to pypi so I am not working about breaking anyone's builds. Package in question: https://pypi.python.org/pypi/lemur Cheers, Kevin

2 2

build system abstraction PEP, take #2
by Robert Collins 10 Dec '15

10 Dec '15

This is the update I promised...full prose, since enough has changed that a diff would be noise. :PEP: XX :Title: Build system abstraction for pip/conda etc :Version: $Revision$ :Last-Modified: $Date$ :Author: Robert Collins <rbtcollins(a)hp.com>, Nathaniel Smith <njs(a)pobox.com> :BDFL-Delegate: Donald Stufft <donald(a)stufft.io> :Discussions-To: distutils-sig <distutils-sig(a)python.org> :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 26-Oct-2015 :Post-History: XX :Requires: The new dependency-specification PEP. Abstract ======== This PEP specifies a programmatic interface for pip [#pip]_ and other distribution or installation tools to use when working with Python source trees (both the developer tree - e.g. the git tree - and source distributions). The programmatic interface allows decoupling of pip from its current hard dependency on setuptools [#setuptools]_ able for two key reasons: 1. It enables new build systems that may be much easier to use without requiring them to even appear to be setuptools. 2. It facilitates setuptools itself changing its user interface without breaking pip, giving looser coupling. The interface needed to permit pip to install build systems also enables pip to install build time requirements for packages which is an important step in getting pip to full feature parity with the installation components of easy-install. As PEP-426 [#pep426]_ is draft, we cannot utilise the metadata format it defined. However PEP-427 wheels are in wide use and fairly well specified, so we have adopted the METADATA format from that for specifying distribution dependencies. However something was needed for communicating bootstrap requirements and build requirements - but a thin JSON schema is sufficient when overlaid over the new dependency specification PEP. Motivation ========== There is significant pent-up frustration in the Python packaging ecosystem around the current lock-in between build system and pip. Breaking that lock-in is better for pip, for setuptools, and for other build systems like flit [#flit]_. Specification ============= Overview -------- Build tools will be located by reading a file ``pypa.json`` from the root directory of the source tree. That file describes how to get the build tool and the name of the command to run to invoke the tool. All tools will be expected to conform to a single command line interface modelled on pip's existing use of the setuptools setup.py interface. pypa.json --------- The file ``pypa.json`` acts as neutron configuration file for pip and other tools that want to build source trees to consult for configuration. The absence of a ``pypa.json`` file in a Python source tree implies a setuptools or setuptools compatible build system. The JSON has the following schema. Extra keys are ignored. schema The version of the schema. This PEP defines version "1". Defaults to "1" when absent. All tools reading the file must error on an unrecognised schema version. bootstrap_requires Optional list of dependency specifications [#dependencyspec] that must be installed before running the build tool. For instance, if using flit, then the requirements might be:: bootstrap_requires: ["flit"] build_command A mandatory Python format string [#strformat]_ describing the command to run. For instance, if using flit then the build command might be:: build_command: ["flit"] If using a command which is a runnable module fred:: {PYTHON} -m fred Process interface ----------------- The command to run is defined by a simple Python format string [#strformat]_. This permits build systems with dedicated scripts and those that are invoked using "python -m somemodule". Processes will be run with the current working directory set to the root of the source tree. When run, processes should not read from stdin - while pip currently runs build systems with stdin connected to it's own stdin, stdout and stderr are redirected and no communication with the user is possible. As usual with processes, a non-zero exit status indicates an error. Available variables ------------------- PYTHON The Python interpreter in use. This is important to enable calling things which are just Python entry points. ${PYTHON} -m foo Subcommands ----------- There are a number of separate subcommands that build systems must support. The examples below use a build_command of ``flit`` for illustrative purposes. build_requires Query build requirements. Build requirements are returned as a JSON document with one key ``build_requires`` consisting of a list of dependency specifications [#dependencyspec]_. Additional keys must be ignored. The build_requires command is the only command run without setting up a build environment. Example command:: flit build_requires metadata Query project metadata. The metadata and only the metadata should be output on stdout. pip would run metadata just once to determine what other packages need to be downloaded and installed. The metadata is output as a wheel METADATA file per PEP-427 [#pep427]_. Note that the metadata generated by the metadata command, and the metadata present in a generated wheel must be identical. Example command:: flit metadata wheel -d OUTPUT_DIR Command to run to build a wheel of the project. OUTPUT_DIR will point to an existing directory where the wheel should be output. Stdout and stderr have no semantic meaning. Only one file should be output - if more are output then pip would pick an arbitrary one to consume. Example command:: flit wheel -d /tmp/pip-build_1234 develop Command to do an in-place 'development' installation of the project. Stdout and stderr have no semantic meaning. Not all build systems will be able to perform develop installs. If a build system cannot do develop installs, then it should error when run. Note that doing so will cause use operations like ``pip install -e foo`` to fail. The build environment --------------------- Except for the build_requires command, all commands are run within a build environment. No specific implementation is required, but a build environment must achieve the following requirements. 1. All dependencies specified by the project's build_requires must be available for import from within ``$PYTHON``. 1. All command-line scripts provided by the build-required packages must be present in ``$PATH``. A corollary of this is that build systems cannot assume access to any Python package that is not declared as a build_requires or in the Python standard library. Hermetic builds --------------- This specification does not prescribe whether builds should be hermetic or not. Existing build tools like setuptools will use installed versions of build time requirements (e.g. setuptools_scm) and only install other versions on version conflicts or missing dependencies. However its likely that better consistency can be created by always isolation builds and using only the specified dependencies. However there are nuanced problems there - such as how can users force the avoidance of a bad version of a build requirement which meets some packages dependencies. Future PEPs may tackle this problem, but it is not currently in scope - it does not affect the metadata required to coordinate between build systems and things that need to do builds, and thus is not PEP material. Upgrades -------- 'pypa.json' is versioned to permit future changes without requiring compatibility. The sequence for upgrading either of schemas in a new PEP will be: 1. Issue new PEP defining an updated schema. If the schema is not entirely backward compatible then a new version number must be defined. 2. Consumers (e.g. pip) implement support for the new schema version. 3. Package authors opt into the new schema when they are happy to introduce a dependency on the version of 'pip' (and potentially other consumers) that introduced support for the new schema version. The *same* process will take place for the initial deployment of this PEP:- the propogation of the capability to use this PEP without a `setuptools shim`_ will be largely gated by the adoption rate of the first version of pip that supports it. Static metadata in sdists ------------------------- This PEP does not tackle the current inability to trust static metadata in sdists. That is a separate problem to identifying and consuming the build system that is in use in a source tree, whether it came from an sdist or not. Handling of compiler options ---------------------------- Handling of different compiler options is out of scope for this specification. pip currently handles compiler options by appending user supplied strings to the command line it runs when running setuptools. This approach is sufficient to work with the build system interface defined in this PEP, with the exception that globally specified options will stop working globally as different build systems evolve. That problem can be solved in pip (or conda or other installers) without affecting interoperability. In the long term, wheels should be able to express the difference between wheels built with one compiler or options vs another, and that is PEP material. Examples ======== An example 'pypa.json' for using flit:: {"bootstrap_requires": ["flit"], "build_command": ["flit"]} When 'pip' reads this it would prepare an environment with flit in it before trying to use flit. Because flit doesn't have setup-requires support today, `flit build_requires` would just output a constant string:: {"build_requires": []} `flit metadata` would interrogate `flit.ini` and marshal the metadata into a wheel METADATA file and output that on stdout. `flit wheel` would need to accept a `-d` parameter that tells it where to output the wheel (pip needs this). Backwards Compatibility ======================= Older pips will remain unable to handle alternative build systems. This is no worse than the status quo - and individual build system projects can decide whether to include a shim ``setup.py`` or not. All existing build systems that can product wheels and do develop installs should be able to run under this abstraction and will only need a specific adapter for them constructed and published on PyPI. In the absence of a ``pypa.json`` file, tools like pip should assume a setuptools build system and use setuptools commands directly. Network effects --------------- Projects that adopt build systems that are not setuptools compatible - that is that they have no setup.py, or the setup.py doesn't accept commands that existing tools try to use - will not be installable by those existing tools. Where those projects are used by other projects, this effect will cascade. In particular, because pip does not handle setup-requires today, any project (A) that adopts a setuptools-incompatible build system and is consumed as a setup-requirement by a second project (B) which has not itself transitioned to having a pypa.json will make B uninstallable by any version of pip. This is because setup.py in B will trigger easy-install when 'setup.py egg_info' is run by pip, and that will try and fail to install A. As such we recommend that tools which are currently used as setup-requires either ensure that they keep a `setuptools shim`_ or find their consumers and get them all to upgrade to the use of a `pypa.json` in advance of moving themselves. Pragmatically that is impossible, so the advice is to keep a setuptools shim indefinitely - both for projects like pbr, setuptools_scm and also projects like numpy. setuptools shim --------------- It would be possible to write a generic setuptools shim that looks like ``setup.py`` and under the hood uses ``pypa.json`` to drive the builds. This is not needed for pip to use the system, but would allow package authors to use the new features while still retaining compatibility with older pip versions. Rationale ========= This PEP started with a long mailing list thread on distutils-sig [#thread]_. Subsequent to that a online meeting was held to debug all the positions folk had. Minutes from that were posted to the list [#minutes]_. This specification is a translation of the consensus reached there into PEP form, along with some arbitrary choices on the minor remaining questions. The basic heuristic for the design has to been to focus on introducing an abstraction without requiring development not strictly tied to the abstraction. Where the gap is small to improvements, or the cost of using the existing interface is very high, then we've taken on having the improvement as a dependency, but otherwise defered such to future iterations. We chose wheel METADATA files rather than defining a new specification, because pip can already handle wheel .dist-info directories which encode all the necessary data in a METADATA file. PEP-426 can't be used as it's still draft, and defining a new metadata format, while we should do that, is a separate problem. Using a directory on disk would not add any value to the interface (pip has to do that today due to limitations in the setuptools CLI). The use of 'develop' as a command is because there is no PEP specifying the interoperability of things that do what 'setuptools develop' does - so we'll need to define that before pip can take on the responsibility for doing the 'develop' step. Once thats done we can issue a successor PEP to this one. The use of a command line API rather than a Python API is a little contentious. Fundamentally anything can be made to work, and the pip maintainers have spoken strongly in favour of retaining a process based interface - something that is mature and robust in pip today. The choice of JSON as a file format is a compromise between several constraints. Firstly there is no stdlib YAML interpreter, nor one for any of the other low-friction structured file formats. Secondly, INIParser is a poor format for a number of reasons, primarily that it has very minimal structure - but pip's maintainers are not fond of it. JSON is in the stdlib, has sufficient structure to permit embedding anything we want in future without requiring embedded DSL's. Donald suggested using ``setup.cfg`` and the existing setuptools command line rather than inventing something new. While that would permit interoperability with less visible changes, it requires nearly as much engineering on the pip side - looking for the new key in setup.cfg, implementing the non-installed environments to run the build in. And the desire from other build system authors not to confuse their users by delivering something that looks like but behaves quite differently to setuptools seems like a bigger issue than pip learning how to invoke a custom build tool. The metadata and wheel commands are required to have consistent metadata to avoid a race condition that could otherwise happen where pip reads the metadata, acts on it, and then the resulting wheel has incompatible requirements. That race is exploited today by packages using PEP-426 environment markers, to work with older pip versions that do not support environment markers. That exploit is not needed with this PEP, because either the setuptools shim is in use (with older pip versions), or an environment marker ready pip is in use. The setuptools shim can take care of exploiting the difference older pip versions require. References ========== .. [#pip] pip, the recommended installer for Python packages (http://pip.readthedocs.org/en/stable/) .. [#setuptools] setuptools, the defacto Python package build system (https://pythonhosted.org/setuptools/) .. [#flit] flit, a simple way to put packages in PyPI (http://flit.readthedocs.org/en/latest/) .. [#pypi] PyPI, the Python Package Index (https://pypi.python.org/) .. [#shellvars] Shellvars, an implementation of shell variable rules for Python. (https://github.com/testing-cabal/shellvars) .. [#pep426] PEP-426, Python distribution metadata. (https://www.python.org/dev/peps/pep-0426/) .. [#pep427] PEP-427, Python distribution metadata. (https://www.python.org/dev/peps/pep-0427/) .. [#thread] The kick-off thread. (https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html) .. [#minutes] The minutes. (https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html) .. [#strformat] The Python string formatting syntax. (https://docs.python.org/3.1/library/string.html#format-string-syntax) .. [#dependencyspec] Dependency specification language PEP. (https://github.com/pypa/interoperability-peps/pull/56) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

8 18

Partial installs for in-process distribution upgrades (was: Installing packages using pip)
by Erik Bray 10 Dec '15

10 Dec '15

Apologies for resending this--my original message got buried in unrelated commentary about how *nix filesystems work. Resending with new subject line... On Fri, Nov 13, 2015 at 3:09 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > On Nov 13, 2015 12:00 PM, "Alexander Walters" <tritium-list(a)sdamon.com> > wrote: >> >> import pip >> pip.install(PACKAGESPEC) >> >> something like that? > > This would be extremely handy if it could be made to work reliably... But > I'm skeptical about whether it can be made to work reliably. Consider all > the fun things that could happen once you start upgrading packages while > python is running, and might e.g. have half of an upgraded package already > loaded into memory. It's like the reloading problem but even more so. Sorry to resurrect an old thread, but I have an idea about how to do this somewhat safely, at least insofar as the running interpreter is concerned. It's still a terrible idea. Not such a terrible idea in principle, but as a practical matter in the context of Python it's probably a bad idea because it uses yet-another-.pth-hack. Consider a "partial install", wherein pip installs all files into a non-imported subdirectory of the target site-packages, along with a .pth file. This distribution is then considered "partially installed" in that the files are their (whether extracted from a wheel, or installed via distutils and the appropriate --root option or similar). For example, consider running >>> pip.install('requests') It would be up to the pip.install() command to determine whether or not the requests distribution was already installed. If it's not install it would proceed as normal. For now I'm assuming the user would still have to manually run `import requests` after this. Auto-import would be nice, but is a separate issue. Now, if requests were already installed and imported we don't want to clobber the existing requests running in the interpreter. pip would install install into the relevant site-packages: <...>/site-packages/requests-2.8.1.part/ requests/ requests-2.8.1.dist-info/ <...>/site-packages/requests-2.8.1.part.pth The .part/ directory contains the results of the partial installation (for example the contents of the wheel, for wheel installs). The .part.pth file is trickier, but could be something like this: $ cat requests-2.8.1.part.pth import inspect, shutil, sys, os, atexit;p = inspect.currentframe().f_locals['sitedir'];part = os.path.join(p, 'requests-2.8.1.part');files = os.path.isdir(part) and os.listdir(part);files and list(map(lambda s, d, f: (sys.modules['shutil'].rmtree(os.path.join(d, f), sys.modules['shutil'].move(os.path.join(s, f), os.path.join(d, f))), [part] * len(files), [p] * len(files), files));os.rmdir(part);pth = part + '.pth';os.path.isfile(pth) and atexit.register(os.unlink, os.path.abspath(pth)) This rifles through the contents of requests.2.8.1.part, deletes any existing directories in the parent site-packages of the same name, completes the install by moving the contents of the .part/ directory into the correct location and then deletes the .part/ directory. The .part.pth later deletes itself. By the time the user restarts the interpreter and runs `import requests` this will be completed. Obvious it would have to be communicated to the user that to upgrade an existing package they will have to restart the interpreter which is less than ideal, but relates to a deeper limitation of Python that they should get used to anyways. At least this would enable in-process installs/upgrades. There are of course all kinds of problems with this solution too. It should perhaps only work in a virtualenv and/or .local site-packages (or at least somewhere that the user will have write permissions on the next interpreter run), and probably other error handling too. The above .pth file could also be simplified by invoking a function in pip to complete any partial installs. Erik

3 2

Installing packages using pip
by Ines Barata 09 Dec '15

09 Dec '15

Hello, I want to install the following version of pygame in my windows10.pro:install pygame-1.9.2a0-cp33-none-win_amd64.whl, but the file is in *.whl* format which I dont have any program to open it I checked the information in https://docs.python.org/3/installing/ to understand how to use pip. But didn't make it... My result is the following, using *python3.5.0 IDLE*: >>> pip *install* pygame-1.9.2a0-cp34-none-win32.whl SyntaxError: invalid syntax >>> pip *pygame*-1.9.2a0-cp34-none-win32.whl SyntaxError: invalid syntax >>> python -m *pip* install pygame-1.9.2a0-cp33-none-win_amd64.whl SyntaxError: invalid syntax >>> python3.*5*.0 -m pip install pygame-1.9.2a0-cp33-none-win_amd64.whl SyntaxError: invalid syntax >>> Can you help me solve my problem?

15 30

distutils.command.build_clib: How to add additional compiler flags for cl.exe?
by Kim Walisch 03 Dec '15

03 Dec '15

Hi, For distutils.command.build_clib the commonly used code below does not work for adding additional compiler flags (it works using distutils.command.build_ext): extra_compile_args = '-fopenmp' On Unix-like system I found a workaround which allows to specify additional compiler flags for distutils.command.build_clib: cflags = distutils.sysconfig.get_config_var('CFLAGS') distutils.sysconfig._config_vars['CFLAGS'] = cflags + " -fopenmp" Unfortunately this does not work with Microsoft's C/C++ compiler cl.exe. Does anybody know how I can add additional compiler flags for cl.exe and distutils.command.build_clib? Thanks and best regards!

2 1