Mailman 3 July 2016 - Distutils-SIG

PyPi not allowing duplicate filenames
by Chris Barker 20 Apr '20

20 Apr '20

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

13 24

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Surviving a Compromise of PyPI - PEP 458 and 480
by Vladimir Diaz 13 Feb '20

13 Feb '20

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

7 53

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Versioned trove classifiers for Django
by James Bennett 06 Apr '17

06 Apr '17

Following up on some IRC discussion with other folks: There is precedent (Plone) for PyPI trove classifiers corresponding to particular versions of a framework. So I'd like to get feedback on the idea of expanding that, particularly in the case of Django. The rationale here is that the ecosystem of Django-related packages is quite large, but -- as I know all too well from a project I'm working on literally at this moment -- it can be difficult to ensure that all of one's dependencies are compatible with the version of Django one happens to be using. Adding trove classifier support at the level of individual versions of Django would, I think, greatly simplify this: tools could easily analyze which packages are compatible with an end user's chosen version, there'd be far less manual guesswork, etc., and the rate of creation of new classifiers would be relatively low (we tend to have one X.Y release/year or thereabouts, and that's the level of granularity needed). Assuming there's consensus around the idea of doing this, what would be the correct procedure for getting such classifiers set up and maintained?

5 17

namespace_package
by KP 12 Sep '16

12 Sep '16

I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head. The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages. If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path. Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked? I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.

4 19

cffi & Py_LIMITED_API
by Daniel Holth 21 Aug '16

21 Aug '16

The next version of cffi will contain small changes to generate code compliant with Python's Py_LIMITED_API: https://bitbucket.org/cffi/cffi/commits/8f867f5a869f (although cffi itself is not, the extensions it generates will be). If we also add an appropriate supported tag to pip ~= cp3.abi3.manylinux1 and provide a way to name the generated DLL's appropriately, it may become possible to reduce the burden of distributing cffi extensions, especially for Windows. One compiled artifact should work on Python 3.2 and above.

3 8

Re: [Distutils] Proposal: "Install and save"
by Wes Turner 01 Aug '16

01 Aug '16

On Jul 31, 2016 12:25 PM, "Jeremy Stanley" <fungi(a)yuggoth.org> wrote: > > On 2016-07-30 20:23:14 -0500 (-0500), Wes Turner wrote: > > pbr also supports "environment markers" > > which we would want to preserve when round-tripping (reading in, modifying, > > and writing out) requirements.txt files; > > though IDK if environment markers are part of any Python Packaging Spec? > > > > from http://docs.openstack.org/developer/pbr/#environment-markers : > > > > argparse; python_version=='2.6' > > I'm assuming you didn't follow the "conditional dependencies" link > from the description in that first paragraph, as it would have taken > you to the corresponding section of one: > > https://www.python.org/dev/peps/pep-0426/#environment-markers > > You should probably also see: > > https://www.python.org/dev/peps/pep-0496/ > https://www.python.org/dev/peps/pep-0508/#environment-markers > > Quickly skimming relevant changelogs, environment markers have been > supported by Setuptools since 0.7, and directly in requirements > lists since pip 6.0. So we still need InstallRequirement.to_requirement_str() as described above ? Is there an implementation which does not mangle hashes and/or environment markers when *writing out* requirements.txt; so that --save could be implemented? > -- > Jeremy Stanley > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig

1 1

Proposal: "Install and save"
by Chris Angelico 31 Jul '16

31 Jul '16

In teaching my students how to use third-party Python modules, I generally recommend starting every project with "python3 -m venv env", and then install dependencies into the virtual environment. They then need a requirements.txt to keep track of them. There are two workflows for that: $ pip install flask $ pip freeze >requirements.txt or: $ echo flask >>requirements.txt $ pip install -r requirements.txt Over in the JavaScript world, npm has a much tidier way to do things. I propose adding an equivalent to pip: $ pip install --save flask which will do the same as the second option above (or possibly write it out with a version number as per 'pip freeze' - bikeshed away). As well as cutting two commands down to one, it's heaps easier in the multiple installation case - "pip install --save flask sqlalchemy gunicorn" is much clearer than messing around with creating a multi-line file; and the 'pip freeze' option always ends up listing unnecessary dependencies. Thoughts? ChrisA

9 17

Switch to upload.pypi.org instead of upload.pypi.io
by Donald Stufft 31 Jul '16

31 Jul '16

Hey! The PSF was finally successful in purchasing pypi.org from the person who previously owned it. Previously discussions had dropped off and we assumed we weren’t going to be able to purchase it, so we moved forward with pypi.io instead. However, the person recently came back around and was willing to sell it this time for a reasonable price. We don’t have all of the pypi.io domains switched over to supporting or preferring pypi.org yet, but I just enabled upload.pypi.org. If you’re currently uploading to Warehouse using upload.pypi.io, nothing will break, but I suggest switching your URL to upload.pypi.org instead as it may not continue to work forever. Thanks! — Donald Stufft

4 4

Contributing money to package authors/maintainers via PyPI
by Nicholas Chammas 31 Jul '16

31 Jul '16

This may be a heretical idea, and it’s definitely not something anyone is likely to take on anytime soon, but I’d like to put it up for discussion and see what people think. I often find myself wanting to show gratitude to the authors or maintainers of a package by giving money. Most of the time, it is easier for me to give money than time. After all, contributing time in a meaningful way to a project can be quite difficult. I wonder how many others find themselves in a similar situation, wanting to chip a few bucks in as a token of gratitude for someone’s work, either one-time or on an ongoing basis. You can already do this today, of course, with services like PayPal, Gratipay <https://gratipay.com/>, and Salt <https://salt.bountysource.com/>. But the process is scattered and different for each Python package and the group of people behind it. What’s more, it seems wrong that these third-party services should capture some of the value generated by the Python community (e.g. by charging some transaction fee) without any of it going back to support that community (e.g. via the PSF). So that raises the question: How about if people could contribute money to the people behind projects they felt grateful for via PyPI? PyPI is already the community’s central package hub. Perhaps it could also be the community’s central hub for contributing money. The central goal would be to create a low-friction ecosystem of monetary contributions that benefits Python package authors/maintainers, as well as the larger Python ecosystem. At a high-level, the ecosystem would be opt-in, and contributions would go directly to their intended recipients, with tiny cuts of each transaction going to the payment processor and some Python community organization like the PSF. I know a more concrete proposal would have to address a lot of details (e.g. like how to split contributions across multiple maintainers), and perhaps there is no way to find the resources to build or maintain such a thing in the first place. But just for now I’d like to separate essence of idea from the practical concerns of implementing it. Assume the work is already done, and we have such a system of monetary contributions in place today. My questions are: - Would such a system benefit the Python community? - Would the cut of transactions that went to the PSF (or similar) be enough to cover the cost of maintaining the system and meaningfully impact other Python efforts? - Would such a system create perverse incentives? - Would it just be a dud that very few people would use? I’m trying to get a feel for whether the idea has any potential. Nick

11 16

bulk upload
by Robin Becker 26 Jul '16

26 Jul '16

I have started to make manylinux wheels for reportlab. Our work flow is split across multiple machines. In the end we create a total of 19 package files (10 manylinux, 8 windows + 1 source); these total 53Mb. 1) Is there a convenient way to upload a new version starting from the package files themselves? Normally we try to test the packages before they are uploaded which implies we cannot just use the distutils upload command. 2) I assume I cannot just keep on uploading new versions to pypi. Presumably I would have to delete a micro release before uploading a new one and only keep significant releases. 3) The manylinux builds are significantly larger than the windows ones because the manylinux build is not statically linking those bits of freetype which we use. Is there a way to detect that I'm building under manylinux? -- Robin Becker

5 7

Outdated packages on pypi
by Dima Tisnek 23 Jul '16

23 Jul '16

Hi all, Is anyone working on pruning old packages from pypi? I found something last updated in 2014, which, looking at the source appears half-done. Github link doesn't work any longer, no description, etc. I managed to find author's email address out of band, and he responded that he can't remember the password, yada yada. I wonder if some basic automation is possible here -- check if url's are reachable and if existing package satisfies basic requirements, failing that mark it as "possibly out of date" d.

12 21

Warehouse re: Celery < 4
by Wes Turner 22 Jul '16

22 Jul '16

>From @asksol "Time to pin your versions if you haven’t already. Celery 4 is out soon: https://t.co/XpZqbjt91t" http://docs.celeryproject.org/en/master/whatsnew-4.0.html - https://github.com/pypa/warehouse/blob/master/requirements/main.in > celery>=3.1 Should be: celery>3.1<4.0 I'm on my phone

2 1

Re: [Distutils] Outdated packages on pypi
by Donald Stufft 20 Jul '16

20 Jul '16

> On Jul 14, 2016, at 8:19 PM, Steve Dower <steve.dower(a)python.org> wrote: > > I'm still keen to find a way to redirect people to useful forks or alternative packages that doesn't require thousands of mentions at conferences for all time ala PIL. I’m not opposed to this but we’ll want to make sure we’re careful about how we do it. PIL is an easy example where the maintainer is gone and there is a community fork of it. But I struggle to come up with very many more examples of this where there is something that is: - Popular enough that enough people are tripping over it to make it worth it. - There is a clear successor (or successors). Off the top of my head I can only really think of PIL, and *maybe* suds. Unless there’s a lot of these maybe all we really need is a policy for when administrators can/will edit the page to direct people towards a different project or a way to add an admin message directing people to another project. — Donald Stufft

5 15

Re: [Distutils] PyPI index workaround
by Matt Bacchi 19 Jul '16

19 Jul '16

OT: I hope you're going to provide a setting to allow the user to disable this unnecessary and bandwith intensive 'feature'? -Matt From: "Михаил Голубев" <qsolo825(a)gmail.com> To: Donald Stufft <donald(a)stufft.io> Cc: Dmitry Trofimov <dmitry.trofimov(a)jetbrains.com>, distutils-sig(a)python.org Date: Wed, 13 Jul 2016 23:21:24 +0300 Subject: Re: [Distutils] PyPI index workaround Right, sorry, that initial question wasn't clear about that. We need the latest versions only for installed packages. Nonetheless, as you noted, it's still several dozens consecutive requests to "/simple/<package_name>" for each PyCharm session of every user. Can you handle that?

2 1

Add data-requires-python attribute to download links in simple repository API
by Thomas Kluyver 17 Jul '16

17 Jul '16

In a discussion about how to allow pip to select a package version compatible with the target Python version, Donald suggested adding a data-requires-python attribute to links in the simple repository API, which pip uses to find candidate downloads. <a href="..." data-requires-python=">=3">...</a> This would expose the Requires-Python metadata field already specified in PEP 345. There is a separate PR for setuptools to allow specifying this value. I have opened a PR to add this to PEP 503, which defines the simple repository API, and Donald asked that I post about it here for comments: https://github.com/python/peps/pull/56 Thanks, Thomas

3 2

Re: [Distutils] Outdated packages on pypi
by Wes Turner 15 Jul '16

15 Jul '16

There are types to describe this graph. Thing > CreativeWork > SoftwareApplication CreativeWork.comment r: [Comment] http://schema.org/SoftwareApplication http://schema.org/Comment ( #PEP426JSONLD because this is a graph of SoftwareApplication(s); now with TOML metadata ) There could be edge types as well. e.g. what is the relation between PIL/Pillow. - maintainerSuggests - communitySuggests - communitySaysUnmaintained - unaddressedVulns - etc Adding an embedded JS comments widget does/would add some additional maintenance burden (because user-generated content). Authors can specify an email address as structured data; and whatever they consider relevant in the long_description. On Jul 13, 2016 9:33 PM, "Steve Dower" <steve.dower(a)python.org> wrote: On 13Jul2016 1456, Glyph Lefkowitz wrote: > > On Jul 13, 2016, at 1:54 PM, Steve Dower <steve.dower(a)python.org >> <mailto:steve.dower@python.org>> wrote: >> >> Possibly such user-contributed content would be valuable anyway >> > > https://alternativeto.net but for PyPI? :) > Or just more general reviews/warnings/info. "Doesn't work with IronPython", "Works fine on 3.5 even though it doesn't say so", etc. Restrict it to 140 chars, signed in users, only allow linking to other PyPI packages, let the maintainer delete comments (or mark them as disputed) and I think you'd avoid abuse (or rants/detailed bug reports/etc.). Maybe automatically clear all comments on each new release as well. Doesn't have to be complicated and fancy - just enough that users can help each other when maintainers disappear. Cheers, Steve _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org https://mail.python.org/mailman/listinfo/distutils-sig

5 5

can someone explain why I have started to get these pip failures
by Robin Becker 14 Jul '16

14 Jul '16

> (myenv) rptlab@app0:~/myenv > $ python > Python 2.7.11+ (default, Apr 17 2016, 14:00:29) > [GCC 5.3.1 20160413] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> > (myenv) rptlab@app0:~/myenv > $ pip --version > pip 8.1.2 from /home/rptlab/myenv/local/lib/python2.7/site-packages (python 2.7) > (myenv) rptlab@app0:~/myenv > pip install -r requirements.txt > Collecting MySQL-python<1.3,>=1.2.5 (from -r requirements.txt (line 1)) > Downloading MySQL-python-1.2.5.zip (108kB) > 100% |################################| 112kB 2.1MB/s > Could not import setuptools which is required to install from a source distribution. > Traceback (most recent call last): > File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/pip/req/req_install.py", line 372, in setup_py > import setuptools # noqa > File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/__init__.py", line 11, in <module> > from setuptools.extern.six.moves import filterfalse, map > File "/home/rptlab/> (myenv) rptlab@app0:~/myenv/local/lib/python2.7/site-packages/setuptools/extern/__init__.py", line 1, in <module> > from pkg_resources.extern import VendorImporter > ImportError: No module named pkg_resources.extern > (myenv) rptlab@app0:~/myenv > $ pip install setuptools -U > Requirement already up-to-date: setuptools in ./lib/python2.7/site-packages This is on a unbuntu 16.04lts system > $ uname -a > Linux app0 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -- Robin Becker

5 7

PyPI index workaround
by Dmitry Trofimov 14 Jul '16

14 Jul '16

Hi, to have information about available packages, PyCharm IDE currently parses the PyPI index page (https://pypi.python.org/pypi?%3Aaction=index). As it is going to be deprecated soon, we are looking for a workaround. What we need is, making one request, to get the name and the version of all PyPI packages. Then we cache this information in the IDE ( https://github.com/JetBrains/intellij-community/blob/7e16c042a19767d5f548c8… ). What official API could you advise us to look at? Any hint is appreciated. Best regards, Dmitry

4 11

Deprecation of the endpoint "/pypi?%3Aaction=index"
by Михаил Голубев 14 Jul '16

14 Jul '16

Hi guys. I'd like to clarify Dmitry's question a bit. We have some issues with suggested "/simple" endpoint. Despite the need to scrap the web page, old endpoint allowed us to quickly find latest versions of the packages hosted on PyPI. We did a single request on IDE startup and showed outdated installed packages in the settings later. Index "/simple" however contains only package names and links to the dedicated pages with their artifacts (not for each of them, though). It means that now we have to make tons of individual requests to find the latest published version for each installed package. Isn't it going to load the service even worse? -- Best regards Mikhail Golubev

2 1

Missing IPv6 support on pypi.python.org
by Baptiste Jonglez 13 Jul '16

13 Jul '16

Hi, pypi.python.org is currently not reachable over IPv6. I know this issue was brought up before [1,2]. This is a real issue for us, because our backend servers are IPv6-only (clients never need to talk to backend servers, they go through IPv4-enabled HTTP frontends). So, deploying packages from pypi on the IPv6-only servers is currently a pain. What is the roadmap to add IPv6 support? It seems that Fastly has already deployed IPv6 [3]. Thanks, Baptiste [1] https://mail.python.org/pipermail/distutils-sig/2014-June/024465.html [2] https://bitbucket.org/pypa/pypi/issues/90/missing-ipv6-connectivity [3] http://bgp.he.net/AS54113#_prefixes6

4 6

package made up of only .so's?
by Chris Withers 07 Jul '16

07 Jul '16

Hi All, I need to build a package which is made up of a set of .so's. These .so's are compiled elsewhere, so I just need to write an appropriate setup.py and bdist_wheel to get what I want. But that first part is where I'm struggling. Each of these .so's is essentially a top level python module. How do I tell setuptools' setup() function to basically just roll these into a wheel? Any help gratefully received! Chris

5 8

Breaking up the stdlib (Was: [Python-Dev] release cadence)
by Petr Viktorin 05 Jul '16

05 Jul '16

On 07/04/2016 12:19 AM, Steve Dower wrote: > My thinking on this issue was that some/most packages from the stdlib > would move into site-packages. Certainly I'd expect asyncio to be in > this category, and probably typing. Even going as far as email and > urllib would potentially be beneficial (to those packages, is my thinking). > > Obviously not every single module can do this, but there are plenty that > aren't low-level dependencies for other modules that could. Depending on > particular versions of these then becomes a case of adding normal > package version constraints - we could even bundle version information > for non-updateable packages so that installs fail on incompatible Python > versions. > > The "Uber repository" could be a requirements.txt that pulls down wheels > for the selected stable versions of each package so that we still > distribute all the same code with the same stability, but users have > much more ability to patch their own stdlib after install. > > (FWIW, we use a system similar to this at Microsoft for building Visual > Studio, so I can vouch that it works on much more complicated software > than Python.) While we're on the subject, I'd like to offer another point for consideration: not all implementations of Python can provide the full stdlib, and not everyone wants the full stdlib. For MicroPython, most of Python's batteries are too heavy. Tkinter on Android is probably not useful enough for people to port it. Weakref can't be emulated nicely in Javascript. If packages had a way to opt-out of needing the whole standard library, and instead specify the stdlib subset they need, answering questions like "will this run on my phone?" and "what piece of the stdlib do we want to port next?" would be easier. Both Debian and Fedora package some parts of the stdlib separately (tkinter, venv, tests), and have opt-in subsets of the stdlib for minimal systems (python-minimal, system-python). Tools like pyinstaller run magic heuristics to determine what parts of stdlib can be left out. It would help these projects if the "not all of stdlib is installed" case was handled more systematically at the CPython or distutils level. As I said on the Language summit, this is just talk; I don't currently have the resources to drive this effort. But if anyone is thinking of splitting the stdlib, please keep these points in mind as well. I think that, at least, if "pip install -U asyncio" becomes possible, "pip uninstall --yes-i-know-what-im-doing asyncio" should be possible as well. > From: Paul Moore <mailto:p.f.moore@gmail.com> > Sent: ‎7/‎3/‎2016 14:23 > To: Brett Cannon <mailto:brett@python.org> > Cc: Guido van Rossum <mailto:guido@python.org>; Nick Coghlan > <mailto:ncoghlan@gmail.com>; Python-Dev <mailto:python-dev@python.org>; > Steve Dower <mailto:steve.dower@python.org> > Subject: Re: [Python-Dev] release cadence (was: Request for CPython > 3.5.3 release) > > On 3 July 2016 at 22:04, Brett Cannon <brett(a)python.org> wrote: >> This last bit is what I would advocate if we broke the stdlib out > unless an >> emergency patch release is warranted for a specific module (e.g. like >> asyncio that started this discussion). Obviously backporting is its own >> thing. > > It's also worth noting that pip has no mechanism for installing an > updated stdlib module, as everything goes into site-packages, and the > stdlib takes precedence over site-packages unless you get into > sys.path hacking abominations like setuptools uses (or at least used > to use, I don't know if it still does). So as things stand, > independent patch releases of stdlib modules would need to be manually > copied into place. > > Allowing users to override the stdlib opens up a different can of > worms - not necessarily one that we couldn't resolve, but IIRC, it was > always a deliberate policy that overriding the stdlib wasn't possible > (that's why backports have names like unittest2...) >

1 0

bdist_wheel alters namespace packages paths:
by Brian Allen Vanderburg II 04 Jul '16

04 Jul '16

In a manor similar to my previous messages about path manipulation from the .pth files, I've noticed that a wheel created by bdist_wheel also performs some path manipulation issues for namespace packages: First it seems that setup.py when creating the wheel does not install the __init__.py namespace file. When the nspkg.pth file gets loaded by site.py, it creates the dummy namespace module in sys.modules and sets the __path__ to contain where it was found. There seem to be couple issue with this: Every namespace package under the same namespace must use the same method to register it's path. That is, every namespace package under the same namespace will need an nspkg.pth file to extend the module path. This also prevents loading any namespace packages from the current directory, such as for testing purposes, since the dummy module is created by the nspkg.pth and seems to prevent the normal scanning of sys.path for namespace packages during import. Brian Allen Vanderburg II

1 0

Re: [Distutils] Module import order issues.
by Brian Allen Vanderburg II 04 Jul '16

04 Jul '16

I have devised the following hack which works for the time being. I add the following line to two new files called 000-fix-easyinstall.pth and zzz-fix-easyinstall.pth: import sys; sys.__egginsert = len(sys.path) This causes the next path re-arrangement by an easy-install.pth to be after all current paths. Why two: The .pth are sorted by site.py. So 000-fix-easyinstall.pth will get called first and set __egginsert after any current paths (such as the core paths and the user site-packages path). This way, when the user easy-install.pth is processed if there is one, any path re-arrangements will occur after them. Now the problem with just one is, if there are any other user .pth files that get processed after the user easy-install.pth, they will add paths but not update the egginsert value. So when the system easy-install.pth gets invoked, those paths will be moved above the user paths that were added after the user easy-install.pth. The solution to this is the second fix file zzz-fix-easyinstall.pth. Since it is processed last, i ensures that the egginsert is after any added paths. To see it in action: import sys for i in sys.path: print(i) Before: /home/allen/.local/lib/python3.4/site-packages/setuptools /usr/local/lib/python3.4/dist-packages/psutil-3.4.2-py3.4-linux-x86_64.egg /usr/local/lib/python3.4/dist-packages/docker_py-1.7.0_rc2-py3.4.egg /usr/local/lib/python3.4/dist-packages/raven-5.10.1-py3.4.egg /usr/local/lib/python3.4/dist-packages/Jinja2-2.8-py3.4.egg /usr/local/lib/python3.4/dist-packages/aiohttp-0.20.2-py3.4-linux-x86_64.egg /usr/local/lib/python3.4/dist-packages/jsonschema-2.5.1-py3.4.egg /usr/local/lib/python3.4/dist-packages/websocket_client-0.35.0-py3.4.egg /usr/local/lib/python3.4/dist-packages/requests-2.9.1-py3.4.egg /usr/local/lib/python3.4/dist-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg /usr/lib/python3/dist-packages /usr/local/lib/python3.4/dist-packages/paramiko-1.16.0-py3.4.egg /usr/local/lib/python3.4/dist-packages/configobj-5.0.6-py3.4.egg /usr/local/lib/python3.4/dist-packages/ecdsa-0.13-py3.4.egg /usr/local/lib/python3.4/dist-packages/pycrypto-2.6.1-py3.4-linux-x86_64.egg /usr/local/lib/python3.4/dist-packages/gns3_server-1.4.6-py3.4.egg /usr/local/lib/python3.4/dist-packages/gns3_gui-1.4.6-py3.4.egg /usr/local/lib/python3.4/dist-packages/gns3_net_converter-1.3.0-py3.4.egg /usr/lib/python3.4 /usr/lib/python3.4/plat-x86_64-linux-gnu /usr/lib/python3.4/lib-dynload /home/allen/.local/lib/python3.4/site-packages /usr/local/lib/python3.4/dist-packages After: /usr/lib/python3.4 /usr/lib/python3.4/plat-x86_64-linux-gnu /usr/lib/python3.4/lib-dynload /home/allen/.local/lib/python3.4/site-packages /home/allen/.local/lib/python3.4/site-packages/setuptools /usr/local/lib/python3.4/dist-packages/psutil-3.4.2-py3.4-linux-x86_64.egg /usr/local/lib/python3.4/dist-packages/docker_py-1.7.0_rc2-py3.4.egg /usr/local/lib/python3.4/dist-packages/raven-5.10.1-py3.4.egg /usr/local/lib/python3.4/dist-packages/Jinja2-2.8-py3.4.egg /usr/local/lib/python3.4/dist-packages/aiohttp-0.20.2-py3.4-linux-x86_64.egg /usr/local/lib/python3.4/dist-packages/jsonschema-2.5.1-py3.4.egg /usr/local/lib/python3.4/dist-packages/websocket_client-0.35.0-py3.4.egg /usr/local/lib/python3.4/dist-packages/requests-2.9.1-py3.4.egg /usr/local/lib/python3.4/dist-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg /usr/lib/python3/dist-packages /usr/local/lib/python3.4/dist-packages/paramiko-1.16.0-py3.4.egg /usr/local/lib/python3.4/dist-packages/configobj-5.0.6-py3.4.egg /usr/local/lib/python3.4/dist-packages/ecdsa-0.13-py3.4.egg /usr/local/lib/python3.4/dist-packages/pycrypto-2.6.1-py3.4-linux-x86_64.egg /usr/local/lib/python3.4/dist-packages/gns3_server-1.4.6-py3.4.egg /usr/local/lib/python3.4/dist-packages/gns3_gui-1.4.6-py3.4.egg /usr/local/lib/python3.4/dist-packages/gns3_net_converter-1.3.0-py3.4.egg /usr/local/lib/python3.4/dist-packages This keeps the core paths first, then the user paths, then the system paths Brian Allen Vanderburg II On 07/04/2016 04:59 AM, Piotr Dobrogost wrote: > On Sun, Jul 3, 2016 at 7:05 PM, Brian Allen Vanderburg II via > Distutils-SIG <distutils-sig(a)python.org> wrote: >> The issue I'm having is when I install a test/local version of a package >> in my user site-packages, but the same package is also installed in a >> system path and has a .pth entry in the system easy-install.pth. It >> seems that from a path perspective, items under the user path should >> automatically take precedence over the system path. However, because of >> the way the paths are rearranged in easy-install.pth, the system path >> always takes precedence since the easy-install.pth line moves all added >> items to the front of the path. > You might want to take a look at issue "pip list reports wrong version > of package installed both in system site and user site" at > https://github.com/pypa/pip/issues/1018. > One "solution" is not to install any packages in virtualenv with > setuptools (which hacks easy-install.pth file as you described). > Install only with pip instead. > > Regards, > Piotr Dobrogost > > > > > >> So while it seems like the path should be something like: >> >> cwd:core-paths:user-site-paths:user-pth-paths:system-dist-paths:system-pth-paths >> >> The path turns out to be: >> >> cwd:user-pth-paths:system-pth-paths:core-paths:user-site-paths:system-dist-paths >> >> Is there a reason why the .pth files even contain the final line that >> rearranges the paths for all the .eggs to be at the start of the system >> path? >> >> For the time being, my work-around is to manually add a .pth file to my >> user site-packages path for each item in site-packages as well, and add >> the lines needed so that when the system easy-install.pth is processed, >> it doesn't move all: >> >> import sys; sys.__plen = len(sys.path) >> <my custom entries for each item in user site packages> >> import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; >> p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = >> p+len(new) >> >> All of that is required, because without it, sys.__egginsert will still >> be unset, and when site.py processes the system path, items in >> easy-install.pth will be moved above any user .pth >> items as well. >> >> However, ideally it seems like this path should automatically have >> precedence over system packages, and it would if not for this line in >> the various easy-install.pth files: >> >> import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; >> p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = >> p+len(new) >> >> I guess I just find myself wondering what is the purpose of the above >> line in easy-install.pth files (for python3, but apparently not >> python2), and why not just keep the path in the order that site.py would >> add them. >> >> Thanks, >> >> Brian Allen Vanderburg II >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG(a)python.org >> https://mail.python.org/mailman/listinfo/distutils-sig

1 0

Module import order issues.
by Brian Allen Vanderburg II 03 Jul '16

03 Jul '16

The issue I'm having is when I install a test/local version of a package in my user site-packages, but the same package is also installed in a system path and has a .pth entry in the system easy-install.pth. It seems that from a path perspective, items under the user path should automatically take precedence over the system path. However, because of the way the paths are rearranged in easy-install.pth, the system path always takes precedence since the easy-install.pth line moves all added items to the front of the path. So while it seems like the path should be something like: cwd:core-paths:user-site-paths:user-pth-paths:system-dist-paths:system-pth-paths The path turns out to be: cwd:user-pth-paths:system-pth-paths:core-paths:user-site-paths:system-dist-paths Is there a reason why the .pth files even contain the final line that rearranges the paths for all the .eggs to be at the start of the system path? For the time being, my work-around is to manually add a .pth file to my user site-packages path for each item in site-packages as well, and add the lines needed so that when the system easy-install.pth is processed, it doesn't move all: import sys; sys.__plen = len(sys.path) <my custom entries for each item in user site packages> import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) All of that is required, because without it, sys.__egginsert will still be unset, and when site.py processes the system path, items in easy-install.pth will be moved above any user .pth items as well. However, ideally it seems like this path should automatically have precedence over system packages, and it would if not for this line in the various easy-install.pth files: import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) I guess I just find myself wondering what is the purpose of the above line in easy-install.pth files (for python3, but apparently not python2), and why not just keep the path in the order that site.py would add them. Thanks, Brian Allen Vanderburg II

1 0