Mailman 3 May 2017 - Distutils-SIG

PyPi not allowing duplicate filenames
by Chris Barker 20 Apr '20

20 Apr '20

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

13 24

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Surviving a Compromise of PyPI - PEP 458 and 480
by Vladimir Diaz 12 Feb '20

12 Feb '20

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

7 53

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Provisionally accepting PEP 517's declarative build system interface
by Nick Coghlan 04 Jun '17

04 Jun '17

Hi folks, I am hereby provisionally accepting PEP 517 as our CLI independent interface for build backends, allowing interested parties to move forward with implementing it as part of pip and other tools. As with other provisionally accepted PyPA interoperability specifications, it won't be considered completely settled until it has been implemented and released in pip and we have at least one iteration of end user feedback. Thanks to Nathaniel & Thomas for their work on the PEP, and to everyone that participated in the related discussions! Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

15 120

Proposal: moving PyPA projects to pydoctheme
by Jon Wayne Parrott 01 Jun '17

01 Jun '17

o/ Hello everyone, I've been working on the Packaging User Guide and various discussions have come up about the theme ( https://github.com/pypa/python-packaging-user-guide/issues/304) as well as the common brand for PyPA projects ( https://github.com/pypa/python-packaging-user-guide/issues/62). I'm proposing we switch PyPA projects (namely pypa.io, PyPUG, distlib, pip, setuptools, virtualenv, warehouse, and wheel) to match the upstream CPython docs for Python 3 (referred to as "pydoctheme"). Switching from the current readthedocs theme has a couple of advantages: * Higher contrast and sans-serif fonts means better readability and accessibility. * Consistency with Python re-enforces that these are "official"/"blessed" tools & documentation. * A central shared theme among these projects allows us to make consistent identity modifications across projects easily. This work has been started on PyPUG ( https://github.com/pypa/python-packaging-user-guide/pull/305#issuecomment-3…). I have staged a build of PyPUG using the new theme here ( http://temp.theadora.io/pypug-pydoctheme/index.html). Please take a look and comment on github with any concerns, and by all means tell me I'm crazy for trying to do this. :) If the primary maintainers of these projects all agree, I will create the theme package and submit PRs to all the projects to do this migration. You'll only need to approve. From what I understand those people are @dstufft, @pfmoore, @jaraco, @vsajip, @dholth, and @ncoglan, but please let me know if I missed anyone (I'm still new!) Thanks!

7 11

Installation problem
by Aneesh Kona 31 May '17

31 May '17

Hi, I'm getting an error while installing software, I did installed firstly but did in another in local disk, but I want to install again, now it shows 'fatal error,' can you please let me fix it, so that I can install again in C disk, plz hand me in this particular, Thanks. Venkata Aneesh Kona Graduate Assistant Department of Mining Engineering and Management South Dakota School of Mines & Technology @: venkataaneesh.kona(a)mines.sdsmt.edu ph: 6823042622

2 2

Introducing PyPIContents
by Luis Alejandro Martínez Faneyth 30 May '17

30 May '17

Hi everyone, I'm new to this list but I've been reading some threads in the archive. Around february, an idea about indexing modules from PyPI packages was brought up. I've been working on something similar for quite a while. PyPIContents is an index of PyPI packages that lists its modules and command line scripts in JSON format, like this: [ ... "1337": { "cmdline": [], "modules": [ "1337", "1337.1337" ], "version": "1.0.0" }, ... ] You can check it out here: https://github.com/LuisAlejandro/pypicontents And some use cases: https://github.com/LuisAlejandro/pypicontents#use-cases The actual index lives here, its around 60MB: https://raw.githubusercontent.com/LuisAlejandro/pypicontents/contents/pypi.… Is updated daily with the help of Travis: https://github.com/LuisAlejandro/pypicontents/blob/contents/.travis.yml Anyway, I hope is useful and I'll be around for any comments or questions. Cheers! Luis Alejandro Martínez Faneyth Blog: http://huntingbears.com.ve Github: http://github.com/LuisAlejandro Twitter: http://twitter.com/LuisAlejandro CODE IS POETRY

3 4

Last call for PEP 516 champions
by Nick Coghlan 29 May '17

29 May '17

Hi folks, The restarted discussion on PEP 517 meant I realised that we hadn't officially decided between its Python API based approach and PEP 516's approach of using the backend CLI as the standardised interface (akin to the current setup.py approach). My current intention is to reject PEP 516's CLI standardisation approach on the following grounds: - PEP 517 makes a convincing case for the benefits of the Python API based approach within the Python ecosystem - the difficulties encountered in evolving the setup.py CLI over time lend significant weight to the notion that a Python level API will be easier to update without breaking backwards compatibility - PEP 517 still advises front-ends to isolate back-end invocation behind a subprocess boundary due to all of the other practical benefits that brings, it just makes the specifics of that invocation an implementation detail of the front-end tool - third party tools that want an implementation language independent CLI abstraction over the Python ecosystem (including builds) can just use pip itself (or another standards-compliant frontend) This isn't particularly urgent, but I also don't see a lot of reason for an extended discussion, so if I don't hear a convincing counterargument in the meantime, I'll mark the PEP as rejected at the beginning of June :) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 1

PEP 517 - specifying build system in pyproject.toml
by Thomas Kluyver 28 May '17

28 May '17

Hi, I'd like to make another push for PEP 517, which would make it possible to build wheels from a source tree with other build tools, without needing setup.py. https://www.python.org/dev/peps/pep-0517/ Last time this was discussed we made a couple of minor changes to the PEP, but we didn't want to accept another packaging related PEP until PEP 518 was implemented in pip. I'm pleased to say that that implementation has just been merged: https://github.com/pypa/pip/pull/4144 . I have an implementation of a build backend according to the current PEP 517 spec in a pull request on flit: https://github.com/takluyver/flit/pull/77/files#diff-6df5630df9568322a453f0… Thanks, Thomas

9 59

setuptools 33.1.1 - 35 issue with source package install from pypi
by Matt Joyce 22 May '17

22 May '17

https://stackoverflow.com/questions/44120045/something-is-breaking-my-packa… has full info on what i am seeing, but it suffices to say: I use setuptools in my setup.py I iterate a requirements.txt file to satisfy dependencies. I publish as sdist to pypi my package used to work fine as it currently is on pypi. however as of late i've been seeing the python setup.py invocation ( from pip install python-symphony ) fail with a couple different failure scenarios. one is it attempting to install what looks like a binary wheel ( that doesn't exist on pypi or anywhere ) that ends up being a bunch of blank stubs. the other is a situation where the package looks 100% correct in site-packages, but none of the module components are registered when you try to use the imported module. pip install --isolated --no-cache-dir python-symphony seems to avoid the issue. but it also tends to do weird stuff like break bpython by messing up the pygments install. curious if anyone was familiar with some sort of bug that i am stumbling into here. -Matt

1 0

python 3.6.1
by Jost now 21 May '17

21 May '17

My Python does not recognise pip or pip3. I should be installed in the 64 bit executable I think. Did I forget something? (I first installed the 32 bit version by mistake but deinstalled it) I also installed VScode, and in that the DON Jayamanne python editor. Is that only an editor?, It does Diagnostics but does not seen to be able to execute python. I should also install a driver to my INVIDIA I was told, but told I could do that later, not sure how. Any advice please. PS: I always have the same problem with downloading programs, thet does not seem to be any guides that actually Works? Sendt fra Mail<https://go.microsoft.com/fwlink/?LinkId=550986> til Windows 10

2 1

installing pip or at least modules on mac
by Sébastien Turrel 21 May '17

21 May '17

Hello from France ! I am using python 3,5 since month and i am no problems to install modules on my window Os but i can't do it on my Mac. I got Sierra Os, maybe is that a problem ? i mean, i can't find pip on my mac, after search, can't find it, i spent hours on internet to find a way to install pip and evey times it's not working, maybe can you help me ? thanks a lot ! -- Sébastien TURREL

2 2

Finally making distlib handle spaces
by Radon Rosborough 20 May '17

20 May '17

Hi all, Since at least 2011, virtualenv has not supported spaces in paths. This has bitten many people, including myself, and caused numerous issues over the years [1] [2] [3] [4] [5] [6] [7]. However, as was discussed in [8], the issue lies not with virtualenv but with distlib, via pip. It would be possible for pip to use the existing distlib interface to hack around the problem, but I believe the current behavior of distlib is erroneous when it comes to spaces in paths. I therefore believe it would be more appropriate to fix the problem in distlib. Two separate patches [9] [10] that solve the problem in distlib were posted in January by Harald Nordgren. However, they were declined pending a discussion on distutils-sig [11]. As far as I can tell, no such discussion was ever started. However, the issue remains, and we have a clear solution proposal to consider, so I'd like to kick it off now. In the remainder of this email, I'll explain the problem and surrounding context in detail, and why I think the solution proposed by Harald (or some variation) is a good path forward for distlib. I look forward to hearing your thoughts on the matter. ========================== The behavior of virtualenv ========================== The following is written for: $ python --version Python 2.7.13 $ pip --version pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7) $ virtualenv --version 15.1.0 Creating a virtualenv is done as follows: $ virtualenv venv New python executable in /private/tmp/path with spaces/venv/bin/python2.7 Also creating executable in /private/tmp/path with spaces/venv/bin/python Installing setuptools, pip, wheel...done. This creates a directory structure looking as follows under the venv directory: ├── bin │ ├── activate │ ├── activate.csh │ ├── activate.fish │ ├── activate_this.py │ ├── easy_install │ ├── easy_install-2.7 │ ├── pip │ ├── pip2 │ ├── pip2.7 │ ├── python -> ./bin/python2.7 │ ├── python-config │ ├── python2 -> ./bin/python2.7 │ ├── python2.7 │ └── wheel ├── include │ └── ... ├── lib │ └── ... └── ... The idea is that one can call the pip and python executables inside the virtualenv, instead of the system ones. Like so: $ venv/bin/python --version Python 2.7.13 $ venv/bin/pip --version zsh: venv/bin/pip: bad interpreter: "/private/tmp/path: no such file or directory Unfortunately, as you can see, pip doesn't work at all! Why does this happen? While the python executable is a native binary, pip is actually just a Python script, which is specified to be run by the accompanying virtualenv python executable. Here are the contents: #!"/private/tmp/path with spaces/venv/bin/python2.7" # -*- coding: utf-8 -*- import re import sys from pip import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main()) The issue is that the python binary is specified using a shebang, which is known [12] [13] to be fragile, OS-dependent, and error-prone regarding paths that are long or that contain spaces or non-ASCII characters. In particular, the quoting of the shebang does not work on most operating systems, including macOS, which is what I ran this test on. ====================== virtualenv and distlib ====================== The issue is complicated by the fact that there are several different libraries at play. To perform the installation of pip and wheel into the virtualenv, virtualenv calls into pip [14]. The 'pip install' command then uses the subroutine library 'wheel.py', which generates the stub scripts using distlib's ScriptMaker [15]. It is actually distlib which generates the shebang, although this can be overridden by setting the 'executable' property of the ScriptMaker object [16] [17]. Any patch to fix the virtualenv problem would therefore need to be in either pip (as a consumer of the shebang-generation interface) or distlib (as the provider of that interface). The problem cannot be addressed by virtualenv without doing something like using pip/distlib to generate the scripts and then fixing them after the fact (this has been proposed [26] [29], but I consider it a hack). ================== Proposed solutions ================== There has been extended discussion about this issue, especially in [2]. Essentially, the solutions proposed fall into four categories: (1) Don't change anything; end users can work around the issue. For example, they can place their virtualenvs in a different directory than their project, or change their username to avoid having spaces or non-ASCII characters. (2) Don't fix the bug, but add a warning to virtualenv. If we absolutely can't fix the bug (which I strongly believe not to be the case), then this would be the next best thing to do. See [27] [30] [31]. (3) Attempt to escape the shebang, for example by using backslashes or quotes. The latter has already been implemented. Unfortunately, this does not work on most operating systems, since shebangs are interpreted by the kernel in a rather primitive way. (4) Patch either pip or distlib to use a different strategy for dispatching the python binary to be used. For example, make the pip script into a shell script with '#!/usr/bin/env sh' that then invokes the real pip script using the appropriate python binary [9]; this works because even POSIX sh supports escaping arbitrary characters in executable paths. Alternatively, use a clever hack to make the pip script executable as both a Python script and as a shell script [10]. This idea originated from [19]. I argue that approaches (1), (2), and (3) are inadequate, and that (4) or some variation thereof is the best path forward for distlib. Regarding (4), note that in addition to the second pull request [10], there is another, possibly cleaner implementation of the same idea at [18], by the same author. Since this issue has been discussed since 2011 without being fixed, I obviously need to justify my position. =================================== Justifications and counterarguments =================================== > There's an easy workaround for the end user, so there's no need to > change things on the pip/distlib side. The workaround may not necessarily be "easy". For example, there have been reports of this problem being unavoidable on macOS in certain circumstances, since the drive name ("Macintosh HD") has a space in it [1]. People working in corporate environments may not be able to simply move their virtualenvs to a different directory (possibly one outside their home directory, if their username has a space or non-ASCII character in it). These issues really do happen, and really do inconvenience people [20]. However, setting these cases aside, the workaround is indeed not too hard for most people. The question is: do we really want virtualenv to have this behavior? To me, it honestly seems embarrassing that such an important tool, and a cornerstone of the Python ecosystem, can't handle something as simple as spaces in directory names, even more than six years after the bug was first reported. If there were a bug in the development version of Git that prevented it from working at all when you had your repository in a path with spaces, nobody would even consider shipping a release! In 2017, people use Unicode and spaces in their filesystems, and they expect their tools to handle this. In my opinion, end-user workarounds should be reserved for those cases where it is absolutely impossible for the software to solve the problem itself without introducing even worse problems. This is simply not the case with the distlib/pip path bug. Patches that present elegant ways to solve the issue without breaking backward compatibility are available [9] [10]. There is no risk of breaking programs that rely on the previous behavior, since the previous behavior was a complete inability for virtualenv to function *at all*, even for 'pip --version'! > People who use long paths and/or paths with spaces and/or paths with > non-ASCII characters in them are rare. No, that's not true [1] [2] [3] [4] [5] [6] [7] [23] [24] [28]. Considering the tiny fraction of people running into bugs who actually report them, I'd say quite a few people run into this issue. It's pretty rare that I'm able to scrounge up this number of issues all leading to a single bug. (I've also run into it, on multiple occasions over several years, by the way.) > The issue with spaces is a limitation of the way in which the kernel > parses shebangs, and this can't be fixed. This is technically true [13], but irrelevant [21]. The implications of this fact are that virtualenv cannot solve this issue *if it continues to directly use shebangs for interpreter dispatch*. However, there are many other ways to dispatch interpreters [9] [10], none of which have the same limitations of the shebang. Thus the limitations of shebangs have no bearing on this issue, unless it turns out that it's completely impossible to dispatch scripts in any other way while maintaining compatibility with all supported platforms, which seems extremely unlikely to me. > We don't want to make the code more complicated unless there's a > good reason for it. I agree. In my opinion, the complete failure of virtualenv to operate in a common real-world use case is a good reason. However, beyond this, we can be even more conservative. In Harald's alternate patch [18], the polyglot script hack is only used in situations where the path is too long for a shebang (conditional on the OS) or it contains spaces. Thus, it's only possible for the patch to improve things, since it is only activated in cases where virtualenv was completely nonfunctional before. Look at software like Bash. It has implicit wordsplitting, which makes dealing with spaces in paths difficult. However, it is *extremely* well-understood that it is a best practice to disable wordsplitting by quoting arguments [22]. That is, modern software is expected to handle spaces and other special characters in paths. Failing to do so leads to hard-to-trace bugs and potential security vulnerabilities. > We don't want to change the default behavior of distlib. pip should > handle its shebang needs using the ScriptMaker.executable property, > which was provided for this purpose. Now we are getting somewhere. This is one of two solutions proposed which would actually fix the bug. However, I feel that it is the wrong way to go about it. In my opinion, distlib (and specifically ScriptMaker) are providing an interface. This interface specifies that when you use distlib to generate a script, and then run the script, it will invoke the executable that you told it to. This is not currently the case when there are spaces in the path leading to the executable. In my opinion, this is a *bug* in that distlib fails to fulfill its specification. Yes, other applications that use distlib could work around this bug, but it is fundamentally a bug in distlib, and should be fixed there. The ability to support spaces in paths should not be a special feature that must be hacked in (separately, resulting in code duplication and increased maintenance burden) to every application that wants to use distlib to generate executables. If we really want to go that way, though, Harald has made pull requests for both pip [25] and virtualenv [26]. ========== Next steps ========== I'd love to hear people's thoughts on the issues raised here. If we decide to go forward with patching distlib, I think the logical next step would be to look at Harald's patches and see if they need to modified to work correctly on all the systems that Python/pip support. As I mentioned earlier, the last comment on [9] was a request for a discussion to be started on distutils-sig, as this email intends. What do people think about [9], [10], [18]? Best regards, Radon Rosborough [1]: http://stackoverflow.com/q/15472430/3538165 [2]: https://github.com/pypa/virtualenv/issues/53 [3]: https://github.com/pypa/virtualenv/issues/994 [4]: https://github.com/pypa/pip/issues/923 [5]: https://bugs.python.org/issue20622 [6]: https://groups.google.com/forum/#!topic/virtualenvwrapper/vqW3fedTgKc [7]: https://bugs.launchpad.net/virtualenv/+bug/241581 [8]: https://github.com/pypa/virtualenv/issues/53#issuecomment-302005361 [9]: https://bitbucket.org/pypa/distlib/pull-requests/31 [10]: https://bitbucket.org/pypa/distlib/pull-requests/32 [11]: https://bitbucket.org/pypa/distlib/pull-requests/31/#comment-29795586 [12]: https://bitbucket.org/pypa/distlib/pull-requests/31/#comment-29734103 [13]: https://lists.gnu.org/archive/html/bug-bash/2008-05/msg00052.html [14]: https://github.com/pypa/virtualenv/blob/c1ef9e29bfda9f5b128476d0c6d865ffe68… [15]: https://github.com/pypa/pip/blob/7c4353e6834a38b36a34234db865e13a589c9e5f/p… [16]: https://bitbucket.org/pypa/distlib/src/216bd857431686493d37eb081614d91c1356… [17]: http://pythonhosted.org/distlib/tutorial.html#specifying-a-custom-executabl… [18]: https://bitbucket.org/pypa/distlib/pull-requests/33 [19]: https://hg.mozilla.org/mozilla-central/file/tip/mach [20]: https://bitbucket.org/pypa/distlib/pull-requests/31/#comment-29735496 [21]: https://github.com/pypa/virtualenv/issues/53#issuecomment-302019457 [22]: http://mywiki.wooledge.org/BashGuide/Practices#Quoting [23]: https://codedump.io/share/7xYRbcZCNihE/1/using-virtualenv-with-spaces-in-a-… [24]: https://github.com/pypa/virtualenv/issues/997 [25]: https://github.com/pypa/pip/pull/4237 [26]: https://github.com/pypa/virtualenv/pull/1004 [27]: https://github.com/pypa/virtualenv/issues/1039 [28]: https://github.com/pypa/virtualenv/issues/1014 [29]: https://github.com/pypa/virtualenv/pull/910 [30]: http://bugs.python.org/issue28446 [31]: https://github.com/pypa/virtualenv/issues/53#issuecomment-282466994

5 5

On vendoring binaries and headers with pypi packages
by Sylvain Corlay 16 May '17

16 May '17

Hi, When distributing python packages that depend on non-python components, the typical way of doing this with general-purpose package managers (linux distributions, conda) is to place binaries, headers, configuration and other artefacts for these components in their respective natural locations under the installation prefix (e.g. PREFIX/bin/, PREFIX/include/, PREFIX/etc/, PREFIX/share/). Now, if I have a python package that depends on such a thing (e.g. pyzmq vs zmq), the pypi wheel for pyzmq will vendor a binary for libzmq as package data. For the case of the header files, there is the distutils `install_headers` directive, but the target directory is (typically) under PREFIX/include/PythonX.Y, which is probably only a good choices for headers of a C extension linked with this specific python installation. Would it make sense, when vendoring packages packages that don't depend on the python version in a wheel to use `data_files` instead, and target respectively the `bin`, `include`, `etc` and `share` subdirectories of the prefix? Specifically, for the headers, one could do data_files = [ ('include/foobar', [list of the foobar header files])), ], Note: I was just using zmq as an example. Thanks, Sylvain

4 8

appveyor
by Robin Becker 15 May '17

15 May '17

I'm looking at building/testing reportlab with appveyor, but need some assistance building the C extensions in particular freetype. I looked at the code for pillow in https://github.com/python-pillow/Pillow/tree/master/winbuild along with https://github.com/python-pillow/Pillow/blob/master/appveyor.yml but I'm not sure I really understand it terribly well. In particular does appveyor support 3.5 & 3.6? The config.py compilers list seems to refer to versions 7.0 and 7.1, but that doesn't seem to be the visualc++ version at all. Looking at the appveyor.yml do people really reveal their ftp or s3 tokens? Or is it more reasonable to upload to a github acccount. -puzzled-ly yrs- Robin Becker

4 4

A little bit of fun with package dependencies
by Paul Moore 10 May '17

10 May '17

I was playing with pkg_resources today, and came up with the following little script. It's nothing special, but I thought it might be of interest. It uses pkg_resources and networkx to create a graph of dependencies between the packages installed in your Python environment. The output is a GraphML file, that can be loaded into a graph display tool (I used a tool called yEd). You could probably also output in other formats, like GraphViz. I don't know how interesting this will be to others, but I thought I'd share it - enjoy! Paul import networkx as nx from pkg_resources import Environment e = Environment() g = nx.DiGraph() for name in sorted(e): dists = e[name] for dist in dists: reqs = dist.requires() for r in reqs: g.add_edge(dist.project_name, r.name) for n in g.nodes(): g.node[n]['Label'] = n nx.write_graphml(g, 'deps.graphml')

1 0

Python modules -reg.,
by Kamalini 09 May '17

09 May '17

Dear sir, I am Nalini working as Assistant Professor in Velammal Institute of Technology,chennai. I am working in python 3.4.4. I would like to import excel for research purpose. Kindly guide me to install supporting packages. Thanks, S. Nalini 9894812725

2 1

ANN: distlib 0.2.5 released on PyPI
by Vinay Sajip 06 May '17

06 May '17

I've just released version 0.2.5 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: * Changed regular expressions to be compatible with 3.6 as regards escape sequences. * Closed some resource leaks related to XML-RPC proxies. * Changed wheel processing to look for metadata in metadata.json as well as pydist.json. * Updated requirement and marker parsing to be compatible with PEP 508. * Made downloadability a factor in scoring URLs for preferences. * Removed Python 2.6 from the support list. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.5 [2] https://goo.gl/M3kQzR [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

environment replication
by Robin Becker 04 May '17

04 May '17

I have a bunch of virtual environments which need replicating from one machine to another. In the past rsync was OK, but now with slightly different environments I have been using pip freeze to get a list of the installed requirements and pip install to carry out changes to the remote environment. Is there a way to resolve issues where pip reports a different package to the one installed eg the original requirements say git+http://github.com/django-cumulus/django-cumulus.git@master#egg=django-c… but pip freeze will only say django-cumulus===1.0.19-22-g506121d ie we lost the actual installation notation. Given that the git location is actually unstable ie the revision can change it seems quite hard to get reproducibility in an automatic fashion. g506121d is derived from the repo hash (https://github.com/django-cumulus/django-cumulus/commit/506121d8669e1cce884…), but I see no way to guess that from the version string unless there some standard decoding I can do of the revision. -- Robin Becker

2 2

package ownership transfer: kobo
by Ken Dreyer 02 May '17

02 May '17

Hi folks, Would someone please give me access to upload new versions of kobo to PyPI? I've filed the transfer request [1] a while back, and several users are asking for new versions. [2] Someone uploaded a very old version four years ago, and I'd like to make sure the PyPI copy is up-to-date. - Ken [1] https://sourceforge.net/p/pypi/support-requests/632/ [2] https://github.com/release-engineering/kobo/issues/24

1 0