Mailman 3 January 2014 - Distutils-SIG

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

error in windows 7 installation
by Manuel López 13 May '14

13 May '14

Hello Everybody: I am trying to install distribute-0.6.25 in a windows 7 machine. I have python 2.7.3 in 32 bits, although the machine is 64 bits. I installed the 32 bit python version because I want to install ipython and there is no 64 bit builds of the Windows installer for ipython. In any case, upon installing disuitils with python setup.py install I get the following error error: C:\manuel\prog_red\python\distribute-0.6.25\distribute-0.6.25\setuptools\cli-32.exe: No such file or directory Any help will be greatly appeciated. Thanks, Manuel -- Manuel López Mariscal Depto. de Oceanografía Física/CICESE malope(a)cicese.mx

2 1

503 errors from PyPI
by Paul Moore 22 Feb '14

22 Feb '14

Whenever I do a PyPI search at the moment, I get Error 503 backend read error backend read error Guru Meditation: XID: 477806858 ________________________________ Varnish cache server Paul

3 6

.. *explicitly* mark the package as pre-release
by anatoly techtonik 10 Feb '14

10 Feb '14

hi, please cc i want to release some versions to pypi, but hide them from installing by explicitly marking as pre-release. i want to preserve versions numbering as 0.1, 0.2 according to semver.org point 4. is that possible? -- anatoly t.

4 5

PEP 427
by Evgeny Sazhin 05 Feb '14

05 Feb '14

Hi, I saw that people from this list are responsible for Wheel related PEP. I’m comparatively new to the python packaging and need some help understanding the recommended way of dealing with python packages. I’m trying to create a development infrastructure that would allow for simple and unified ways of sharing, *deploying* and *reusing* the code within private entity. I can see that pip with virtual environments and requirements.txt is very similar to dependency management provided by maven or apache ivy. But there seems to be a disconnect between the egg carrying the possibility to be importable and executable, but in the same time considered to be deprecated format which is not fully supported by pip and virtualenv and wheel not having those basic questions answered... So, i would like to ask few questions about it : 1. I’m coming from java world, so it is bit hard to understand why new packaging format is not importable? What is the reason behind it? Wouldn’t it be way easier to deal with the dependencies and imports? 2. Why unzipping and installing the wheel is the way to go? There is no need to unzip java jar to import it into your code, neither to run your code with it? Why wheel can’t be used the same exact way? I would appreciate any insight about the development infrastructure the wheel designers have in mind. Thanks in advance, Eugene

7 23

PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)
by Brian Wickman 04 Feb '14

04 Feb '14

This is in response to Vinay's thread but since I wasn't subscribed to distutils-sig, I couldn't easily respond directly to it. Vinay's right, the technology here isn't revolutionary but what's notable is that we've been using it in production for almost 3 years at Twitter. It's also been open-sourced for a couple years at https://github.com/twitter/commons/tree/master/src/python/twitter/common/py… not widely announced (it is, after all, just a small subdirectory in a fairly large mono-repo, and was only recently published independently to PyPI as twitter.common.python.) PEX files are just executable zip files with hashbangs containing a carefully constructed __main__.py and a PEX-INFO, which is json-encoded dictionary describing how to scrub and bootstrap sys.path and the like. They work equally well unpacked into a standalone directory. In practice PEX files are simultaneously our replacement for virtualenv and also our way of distributing Python applications to production. Now we could use virtualenv to do this but it's hard to argue with a deployment process that is literally "cp". Furthermore, most of our machines don't have compiler toolchains or external network access, so hermetically sealing all dependencies once at build time (possibly for multiple platforms since all developers use Macs) has huge appeal. This is even more important at Twitter where it's common to run a dozen different Python applications on the same box at the same time, some using 2.6, some 2.7, some PyPy, but all with varying versions of underlying dependencies. Speaking to recent distutils-sig threads, we used to go way out of our way to never hit disk (going so far as building our own .egg packager and pure python recursive zipimport implementation so that we could import from eggs within zips, write ephemeral .so's to dlopen and unlink) but we've since moved away from that position for simplicity's sake. For practical reasons we've always needed "not zip-safe" PEX files where all code is written to disk prior to execution (ex: legacy Django applications that have __file__-relative business logic) so we decided to just throw away the magical zipimport stuff and embrace using disk as a pure cache. This seems more compatible philosophically with the direction wheels are going for example. Since there's been more movement in the PEP space recently, we've been evolving PEX in order to be as standards-compliant as possible, which is why I've been more visible recently re: talks, .whl support and the like. I'd also love to chat about more about PEX and how it relates to things like PEP 441 and/or other attempts like pyzzer. cheers, brian twitter.com/wickman

11 71

distlib.mount API design (was: wheels on sys.path clarification (reboot))
by Nick Coghlan 01 Feb '14

01 Feb '14

On 30 January 2014 21:57, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: >> My one technical issue is with going beyond zipimport >> behaviour to the point of extracting DLLs to the filesystem. >> I remain -1 on that feature, and I believe I have explained why I think >> there are issues (and why I think that any solution should be part of >> zipimport and not added on in library or user code). But I'm happy to go >> through the details again, if you like - or just to accept that I don't > > Yes please, let's get into some details. Of course I understand that you > might not want to use the feature, but I don't understand the -1 on the feature > per se - whether it is in distlib or in zipimport is a secondary consideration. I > agree that zipimport is the logical place for it, but ISTM the reason why it can't > go in there just yet is also the reason why one might have some reservations > about the feature: binary compatibility. I accept that this not yet a fully resolved > issue in general (cf. the parallel discussion about numpy), but if we can > isolate these issues, we can perhaps tackle them. But for me, that's the main > reason why this part of the distlib API is experimental. I actually think this is a useful thing to experiment with, I'm just not sure distlib is the best place for that experiment. With appropriately secure tempfile handling and the right sys.path (and module __path__) manipulation it's not obviously *impossible* to handle C extensions at arbitrary positions in the module namespace this way, just difficult. zipimport itself is a bad place to experiment though, since not only is it currently a complex ball of C code, but adding such a feature without clear evidence of robust support in a third party project would be irresponsible. In the case of distlib, the potential complexity of ensuring that such a scheme works consistently across multiple platforms and as part of various complex package layouts is enough to make me nervous about having it in the same library as the metadata 2.0 reference implementation :) Now, if you were to split that functionality out from distlib into a separate "wheeltab" project (or a name of your choice), I'd be substantially less nervous, because endorsing distlib as the metadata 2.0 reference implementation wouldn't carry any implications of endorsing a feature I consider "potentially interesting but rather challenging to implement in a robust manner". mount() would become something I could explore when I had some additional free time (hah!), rather than something I felt obliged to help get to a more robust state before releasing metadata 2.0. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 8

Letting the Python implementation produce the Python ABI tag
by Squeaky 31 Jan '14

31 Jan '14

Hi! This is my first message to this mailing list. I saw a discussion about construction of SOABI and I thought that it might be tiresome to keep code to produce that for every possible Python implementation in wheel generators. After all it's the Python implementation that knows better in which version and why they broke binary ABI. Let's say that PyPy 2.2 has this tag set to pp22 but they might have released PyPy 2.3 without breaking the ABI and prefer to advertise PyPy2.3 as pp22 compatible thus reducing the number of wheels a python package maintainer has to produce. The ABI is separate from releases. Also this would benefit if there were new Python implementations coming into ecosystem and they could have a function in their stdlib that advertises their abi level. This would allow these new implementations to work with wheels instantly. Does it make sense or it's an utopia? Regards, Squeaky

2 1

Extracting C extensions from zipfiles on sys.path (Was: wheels on sys.path clarification (reboot))
by Paul Moore 31 Jan '14

31 Jan '14

Changing the subject to clearly focus the discussion. On 30 January 2014 11:57, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: > If you have other reasons for your -1, I'd like to hear them. OK. Note that this is not, in my view, an issue with wheels, but rather about zipfiles on sys.path, and (deliberate) design limitations of the module loader and zipimport implementations.[1] First of all, it is not possible to load a DLL into a process' memory [2, 3] unless it is stored as a file in the filesystem. So any attempt to import a C extension from a zipfile must, by necessity, involve extracting that DLL to the filesystem. That's where I see the problems. None are deal-breaking issues, but they consist of a number of niggling issues that cumulatively chip away at the reliability of the concept until the end result has enough corner cases and risks to make it unacceptable (depending on your tolerance for risks - there's a definite judgement call involved). The issues I can see are: [4] 1. You need to choose a location to put the extracted file. On Windows in particular, there is no guaranteed-available filesystem location that can be used without risk. Some accounts have no home directory, some (locked down) users have no permissions anywhere but very specific places, even TEMP may not be usable if there's an aggressive housekeeping routine in place - but TEMP is probably the best choice of a bad lot. 2. There are race conditions to consider. If the extraction is not completely isolated per-process, what if 2 processes want to use different versions of the same DLL? How will these be distinguished? [5] So to avoid corner cases you have to assume only the one process uses a given extracted DLL. 3. Clean-up is an issue. How will the extracted files be removed? You can't unload the DLLs from Python, and you can't delete open files in Windows. So do you simply leave the files lying round? Or do you do some sort of atexit dance to run a separate process after the Python process terminates which will do the cleanup? What happens to that process when virus checkers hold the file open? Leaving the files around is probably the most robust answer, but it's not exactly friendly. As I've said elsewhere, these are fundamental issues with importing DLLs from zipfiles, and have no direct relationship to wheels. The only place where having a wheel rather than a general zipfile makes a difference is that a wheel *might* at some point contain metadata that allows the wheel to claim that it's "OK" to load its contents from a zipfile. But my points above are not something that the author of the C extension can address, so there's no way that I can see that an extension author can justifiably set that flag. So: as wheels don't give any additional reliability over any other zipfile, I don't see this (loading C extensions) as a wheel-related feature. Ideally, if these problems can be solved, the solution should be included in the core zipimport module so that all users can benefit. If there are still issues to iron out and experience to be gained, a 3rd party "enhanced zip importer" module would be a reasonable test-bed for the solution. A 3rd party solution could also be appropriate if the caveats and/or limitations were generally acceptable, but sufficient to prohibit stdlib inclusion. The wheel mount API could, if you wanted, look for the existence of that enhanced zipimport module and use it when appropriate, but baking the feature into wheel mount just limits your user base (and hence your audience for raising bug reports, etc) needlessly. I hope this explains my reasoning in sufficient detail. FINAL DISCLAIMER: I have no objection to this feature being provided per se, any more than I object to the existence of (say) Zope. Just because I'm not a member of the target audience doesn't mean that it's not a feature that some might benefit from. All I'm trying to do here is offer my input as someone who was involved in the initial implementation of zipimport, and who has kept an interested eye on how it has been used in the 11 years since its introduction - and in particular how people have tried to overcome the limitations we felt we had to impose when designing it. Ultimately, I would be overjoyed if someone could find a solution to this issue (in much the same way as I'm delighted by what Brett has done with importlib). Paul Footnotes: [1] Historical footnote - I was directly involved with the design of PEP 302 and the zipimport implementation, and we made a deliberate choice to only look at pure Python files, because the platform issues around C extensions were "too hard". [2] I'm talking from a Windows perspective here. I do not have sufficient low-level knowledge of Unix to comment on that case. I suspect that the issues are similar but I defer to the platform experts. [3] There is, I believe, code "out there" on the internet to map a DLL image into a process based purely in memory, but I think it's a fairly gross hack. I have a suspicion that someone - possibly Thomas Heller - experimented with it at one time, but never came up with a viable implementation. There's also TCL's tclkit technology, which *may* support binary extensions, and may be worth a look, but TCL has virtual filesystem support built in quite deep in the core, so how it works may not be applicable to Python. [4] I'm suggesting answers to the questions I'm raising here. The answers *may* be wrong - I've never tried to design a robust solution to this issue - but I believe the questions are the important point here. Please don't focus on why my suggested approach is wrong - I know it is! [5] To be fair, this is where the wheel metadata might help in distinguishing. But consider development and testing, where repeated test runs would not typically have different versions, but the user might well want to test whether running from zip still works. So wheel metadata helps, but isn't a complete solution. And compile data is probably just as good, so let's keep assuming we are looking at a general zipimport facility.

4 7

platform specific python only wheels
by Ronald Oussoren 31 Jan '14

31 Jan '14

Hi, Is there a way to create a wheel that contains only python code, but can only be installed on particular platforms? In particular, I’m looking for a way to create a wheel for py2app that can only be installed on OSX because py2app cannot “cross-compile” application bundles. Ronald

4 8

Using Wheel with zipimport
by Donald Stufft 30 Jan '14

30 Jan '14

I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed to be compatible with zipimport so that you could directly add them to sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that this was the case, and had thought otherwise, and have been telling people that although Wheels could technically be used that way that doing so was considered unsupported. I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. One of the major problems with Egg was that it confused the difference between a distribution format and a runtime installation format. I believe that if we have this as a supported option, that ultimately it will lead towards the same kind of problems that Egg had trying to fulfill two roles at once. Further more it won't even work accurately for all Wheels (as Nick's edit says) so we require that people wanting to do this figure out if their Wheel can or can not be added to the sys.path (which isn't as simple as looking at the tags because a platform specific Wheel may only contain optional C modules). If a single file importable item for a project is something that end users want, then an additional format should be developed that does not concern itself with the ability to unpack and install the project and instead solves the problems that come with trying to stick arbitrary things in a zipped file and import them. So yea, I think that the goals of an importable format and an distribution format are not completely overlapped, and if Wheel tries to be both, then both sides of that will suffer with a less useful and less suitable ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

12 66

wheels on sys.path clarification (reboot)
by Nick Coghlan 30 Jan '14

30 Jan '14

I have clearly done a bad job so far of explaining the clarification in PEP 427, so here's a new attempt that relies solely on the PEP text and the way the import system works, rather than the fact that the discussions around the PEP show that the import system compatibility was a deliberate feature that I took into account when accepting the PEP. Here is the key quote from PEP 427: """Although a specialized installer is recommended, a wheel file may be installed by simply unpacking into site-packages with the standard 'unzip' tool while preserving enough information to spread its contents out onto their final paths at any later time.""" That feature (which *is* explicitly documented in the accepted PEP), ensures that there will always be a subset of wheel files which can be used without needing a customised installer - just unzip them into a directory on sys.path and go. Not all wheels will work that way, and there are some downsides when you do it (e.g. launch scripts won't be created and bytecode files won't be compiled with elevated permissions if that is needed). However, it's explicitly covered in the PEP and is an apparently non-controversial design goal for the format. Once you have that "can be installed just by unzipping the archive" feature, then it is *an inherent property of how zipimport works* that there will also be an even smaller subset of wheels that will work correctly without even unpacking them first - you can just add the archive directly to sys.path and let zipimport do its thing. Aside from deliberately choosing a zipimport incompatible archive format (which we didn't do), you cannot design a format that meets the first requirement without inevitably also supporting the latter behaviour. However, this fact isn't obvious (as has become eminently clear to me since reading Armin Ronacher's recent wheel article), as not everyone is intimately with the subtleties of the import system. That is why I added the new text to the PEP - to make this capability clear without needing to make that connection independently. I do now plan to reword the new FAQ entry to make it clear that the import compatibility is mainly a side effect of the "no specialised installer needed" feature, but hopefully the above helps make it clear why this compatibility is already an inherent feature of the accepted PEP rather than something that can be discussed and accepted independently in a new version - it is pointing out a non-obvious consequence of an existing capability, not actually adding anything new. Regards, Nick.

9 20

wheels on sys.path clarification (reboot)
by Evgeny Sazhin 30 Jan '14

30 Jan '14

>Eh, I think both 1 and 3 are things that are possibly reasonable to happen and >they are both things that I've contemplated as things to bring forward in >using xz as an alternative compression format. Even if #1 would need a major >revision of Wheel to happen adding official "support" for zip import means that >the change would have to be weighed against also breaking that backwards >compatibility. Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path. I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i’m probably alone in the desert…l;)

6 21

PEX - Twitter's multi-platform executable archive format for Python
by Vinay Sajip 30 Jan '14

30 Jan '14

I searched this list for PEX and didn't find any hits, so I thought some people here might be interested in the PEX software from Twitter: https://www.youtube.com/watch?v=NmpnGhRwsu0 is a 15-minute presentation by Brian Wickman about it and other topics. It uses zipimport and is used to do packaging of Python software in Twitter across multiple platforms, so it could have some relevance to recent discussions we've been having about wheels, binary compatibility and the like. Brian has posted here (re. PEP 425 tags most recently) but not about PEX AFAICT. I don't yet know if there's anything particularly revolutionary about the tech (I haven't dug into it yet) but of course it's interesting because of Twitter's size and the usage in production. Regards, Vinay Sajip

1 0

pip on windows experience
by Thomas Heller 30 Jan '14

30 Jan '14

After having used and built bdist_wininst installers for years (well, I should say decades) I have recently started playing with pip. In theory, it is a grat solution. Respects virtual environments (although I haven't tested them yet), allows to freeze to a requirements file, and so on. It would allow to freeze and then commit a requirements file into our version control system, add a 'pip install -r requirements.txt' to our build scripts, and any user in out company as well as the build server would automatically have installed all the required stuff. (We have 8 developers working on our source code, plus a build server that does automatic builds). However, does it work in real life? There are no wheels for important packages like pywin32, numpy, lxml, at least not on PyPi. 'pip install' using the source packages fails to build for most but the simplest stuff; probably because distutils has been extended too much, or because header files or libraries are missing, or whatever... Is there a solution to this? I've seen that the wheel tool can convert bdist_wininst installers into wheels - does this work for the packages I mentioned above? Do we have to build or convert to wheel those packages, and setup a central place where we store them and make them available to our developers? BTW: It seems the pip 1.4.1 doesn't download the wheel built with the most recent setuptools/wheel version, instead it downloads and tries to build from the source package. Thanks, Thomas

11 38

problems with eggs?
by Daniel Holth 29 Jan '14

29 Jan '14

Here are some of the problems with eggs that I tried to solve: - Cannot be unzipped on top of each other due to the EGG-INFO directory. Wheels repeat the package name in the dist-info directory and are more like their installed layout, hopefully taking all the mystery out of the format. - The egg naming scheme doesn't let you distinguish between Python interpreter implementations (not an important issue at the time). - They include .pyc and so are always [at least somewhat] Python version specific. Wheels don't. - Eggs do not have categories of files. Wheel uses subdirectories of the .data/ directory, and are therefore a complete replacement for most other kinds of bdist_x - And of course wheels use .dist-info which is also something that is newer than eggs, and generally try to bring the useful packaging PEP work into reality. I remember dealing with the zipped eggs hassle, and I remember having to wait too long for binary packages to be uploaded to pypi as eggs when a new Python version was released.

3 2

wheels, metadata
by Thomas Heller 29 Jan '14

29 Jan '14

I'm reading up and done all those peps: 425 - compaibility tags 426 - metadata 2.0 427 - wheel binary format 1.0 440 - version identification and dependency specifications and trying to make sense from them. Well, they make sense to me, but to what do they apply? Example: I can now build, upload, and install a wheel for my distribution. I have imported setuptools in my setup.py script, and have run 'setup.py bdist_wheel'. The resulting file is named 'py2exe-0.9.0-py34-none-any.whl', which means: python tag 'py34', abi tag 'none', platform tag 'any' What this file really is: it is a library that only works on CPython 3.3 and 3.4, win32 and win_amd64. So I would have expected a filename like py2exe-0.9.0-py33.py34-none-win.whl or similar. Hoever, the only thing that I found which did change the filename is this section in the setup.cfg file (which is not appopriate for this distribution): [wheel] universal=1 So, the question is: Are the specifications in the PEPs above implemented somewhere or are they only 'specifications'? There are very detailed descriptions how to specify requirements in the metadata (in pep426), something like "supports_environments": ["python_version >= '3.3' and 'win' in sys_platform"] Where can I specify this? Does distutils support it? Do I have to manually edit some pdist.json file, or do I have to rename the created .whl? Thomas

5 9

PEP 440 (versioning) nearing completion (I hope)
by Nick Coghlan 28 Jan '14

28 Jan '14

I just pushed a new draft of PEP 440 live: http://www.python.org/dev/peps/pep-0440/ Relative to the version I posted just before Christmas, the main significant change is the inclusion of version epochs as part of the spec. In 99.9% of cases, those won't be needed and can be ignored. However, I'm swayed by the fact that at least Debian and Fedora found them useful enough to add to their versioning schemes, and they're a relatively elegant solution to the "How do I switch from date based versioning to semantic versioning without renaming my project?" problem. There are also two clarifications: - I changed the pseudo-syntax for the release segment from N[.N]+ to N(.N)* to make it clearer that single value version numbers are acceptable - I point out in the rationale section that "-" makes local version identifiers more readable in isolation, and the existing escaping rules in PEP 427 take care of converting them to hyphens to avoid ambiguity in wheel filenames. (Paul, you can consider this a belated reply to your post from last week...) With this update, I believe PEP 440 is functionally complete. However, there are three additional requirements that need to be met before I'm willing to declare it accepted: - a reference implementation (I'm hoping Vinay will be willing to step up to the plate with distlib again on that front) - rerunning the compatibility analysis against the PyPI of early 2014 - surviving distutils-sig review without too much grumbling or virtual tomato throwing ;) I know it's a rather complex spec, but robust and comprehensive versioning is a rather complex problem :) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

Explicitly constructing SOABI for Python 2.7
by Nick Coghlan 28 Jan '14

28 Jan '14

Currently, bdist_wheel doesn't set the ABI tag properly on Python 2.x - the SOABI symbol it expects to find isn't present. PEP 425 also disclaims dealing with the problem. However, PEP 3149 (which defined SOABI) explains how to define that based on 3 other syconfig flags: Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC >>> sysconfig.get_config_var("SOABI") >>> sysconfig.get_config_var("Py_DEBUG") 0 >>> sysconfig.get_config_var("Py_UNICODE_SIZE") 4 >>> sysconfig.get_config_var("WITH_PYMALLOC") 1 With those settings and the platform module (to get the current implementation), it is possible to figure out that the wheel ABI tag should be: cp27mu If Py_DEBUG was true, then it would be "cp27dmu" If WITH_PYMALLOC was false for some reason: "cp27u" And for a narrow Unicode build: "cp27m" That way, the ABI tagging would work properly in 2.x as well, rather than people getting the impression that wheels can't mark the ABI compatibility requirements properly. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

PyPI Migrated to New Infrastructure with some Breakage
by Donald Stufft 27 Jan '14

27 Jan '14

Today (Sat Jan 25, 2014) the Infrastructure team has migrated PyPI to new infrastructure. The old infrastructure was: - a single database server managed by OSUOSL - a pair of load balancers shared by all of the python.org services hosted on OSUOSL - a single backend VM that served as everything else for PyPI The VM that was acting as the backend server from PyPI was partially hand configured and partially setup to be managed by chef. Additionally it had an issue that caused it to kernel panic every so often which had been the cause of a number of downtimes in the last few months. Because it was primarily configured and administered by hand and because the way it was set up it was not feasible to have any sort of failover or spare. The new infrastructure is: - 2 Web VMs - 2 Database servers in Master/Slave Configuration - 2 PgPool Servers pooling connections to the database servers and load balancing reads across them. - 2 GlusterFS servers backed by Cloud Block Storage acting as the file storage for package and package docs - 1 metrics server to handle updating the download counts as they come in from Fastly All of the VMs are hosted on Rackspace’s Public Cloud and have their configuration completely controlled and managed using Salt. Going forward this will allow us to easily scale out as required or kill malfunctioning servers and spin up new ones easily. Additionally the setup has been setup so that where possible there is two servers performing the same role, ideally in an Active/Active configuration but at least in a Master/Slave configuration. This should allow PyPI to be far more stable moving forward and make downtimes much easier to recover from. The services are still fronted by Fasty’s CDN and in the new infrastructure we’ve removed our load balancer and have replaced it by having Fastly handle the load balancing for us. Additionally we’ve recently setup a static mirror of PyPI that is updated once every minute. This is hosted on Rackspace cloud as well but in a separate data center from the rest of PyPI. Fastly is configured to fall back to this static mirror in the case that neither of the two web heads are functioning. This should ensure that even in the event of a catastrophic failure of the PyPI service that the bulk of package installations should hopefully remain working. The bad news, (and the “Breakage” from the subject) is that while the new infrastructure was being planned out, built, and migrated to the “pypissh” package was forgotten. The pypissh package is an alternative way to upload packages to PyPI however it is very difficult, because of the way it works, to provide HA support for it as we’ve set up for everything else. We don’t have any numbers for how many people are actively using this package but looking at a roughly 2 week chunk of time in PyPI’s download history, the pypissh package was downloaded 7 times by a browser, and 7 times by pip. All other downloads were caused by the mirroring system. As of right now pypissh is non functional and due to the difficulty in HAing and monitoring the current setup and because it is apparently has a very small set of users we would like to effectively kill off this particular service. Additionally the benefits of pypissh have been reduced now that PyPI is available over a TLS connection with a well trusted certificate. My question to you is, is this something that distutils-sig is willing to have happen? If we are to re-enable pypissh we’ll need to write a new solution to doing it that can be properly HA’d and we’d prefer to put our efforts into improving things for a much larger set of people. So yea, PyPI should be loads more stable and more reliable now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

9 18

pip 1.5.2 and virtualenv 1.11.2 released
by Donald Stufft 25 Jan '14

25 Jan '14

I’ve just released pip 1.5.2 and virtualenv 1.11.2 1.5.2 (2014-01-26) ------------------ * Upgraded the vendored ``pkg_resources`` and ``_markerlib`` to setuptools 2.1. * Fixed an error that prevented accessing PyPI when pyopenssl, ndg-httpsclient, and pyasn1 are installed * Fixed an issue that caused trailing comments to be incorrectly included as part of the URL in a requirements file 1.11.2 (2014-01-26) ~~~~~~~~~~~~~~~~~~~ * Fixed easy_install installed virtualenvs by updated pip to 1.5.2 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

error message from testpypi.python.org
by Thomas Heller 23 Jan '14

23 Jan '14

I'm trying out to submit a wheel to testpypi.python.org and then install it from there with pip. It doesn't work; what does the following error message mean? c:\>py -3.4 -m pip install py2exe -r https://testpypi.python.org/pypi/ Exception: Traceback (most recent call last): File "C:\Python34-64\lib\site-packages\pip\basecommand.py", line 122, in main status = self.run(options, args) File "C:\Python34-64\lib\site-packages\pip\commands\install.py", line 258, in run for req in parse_requirements(filename, finder=finder, options=options, session=session): File "C:\Python34-64\lib\site-packages\pip\req.py", line 1572, in parse_requirements req = InstallRequirement.from_line(line, comes_from, prereleases=getattr(options, "pre", None)) File "C:\Python34-64\lib\site-packages\pip\req.py", line 172, in from_line return cls(req, comes_from, url=url, prereleases=prereleases) File "C:\Python34-64\lib\site-packages\pip\req.py", line 70, in __init__ req = pkg_resources.Requirement.parse(req) File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", line 2612, in parse reqs = list(parse_requirements(s)) File "C:\Python34-64\lib\site-packages\pip\_vendor\pkg_resources.py", line 2538, in parse_requirements raise ValueError("Missing distribution spec", line) ValueError: ('Missing distribution spec', '<?xml version="1.0" encoding="utf-8"?>') Storing debug log for failure in C:\Users\thomas\AppData\Roaming\pip\pip.log Thanks, Thomas

3 3

Re: [Distutils] setup.py issue - some files are included as intended, but one is not
by Dan Stromberg 22 Jan '14

22 Jan '14

Hi all. Yes, it does currently require make. I'm very comfortable with make, but I'm open to redoing the make rules in setup.py if that'll help get this problem solved. It'd probably be less puzzling to Python people anyway. I kept increasing the version number as I tried different ways of getting setup.py to do what I wanted. That's why the version numbers don't appear to match up. If there's a better way of testing setup.py's, please let me know! Thanks! >hello, I checked out your project from svn. >looks like it requires some make commands, before it can be packaged? >I'm getting tripped up on that. >also confused in that your src tree is at 1.30, but the tar you mention is >1.35, and then your install dir was 1.31. >maybe get your src tree up to date with where your working copy, and post >any make commands that need to be run. > >On Tue, Jan 14, 2014 at 1:32 PM, Ethan Furman <ethan at stoneleaf.us> wrote: > >> On 01/14/2014 01:26 PM, Dan Stromberg wrote: >> >>> On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg <drsalists at gmail.com> >>> wrote: >>> >>>> Hi folks. >>>> >>>> I have a setup.py problem that's driving me nuts. >>>> >>> >>> Anyone? I've received 0 responses. >>> >> >> I have no answer, but forwarding to Distutils (hopefully it's an >> appropriate topic ;) >> >> >> I have a treap.py file that tries to "import * from pyx_treap.so", and >>>> failing that, it'll "import * from py_treap.py" (sans extensions of >>>> course). Naturally, all 3 of these should be included - although in >>>> the case of pyx_treap.so, it probably should be a .c that's included. >>>> >>>> It is also intended to include nest.py, which has a simple form. >>>> >>>> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >>>> seem to get py_treap.py to be included for some reason. >>>> >>>> I get no errors during "python setup.py sdist upload", but upon >>>> "python $(which pip) install treap", I get: >>>> $ sudo /usr/bin/python $(which pip) install treap >>>> Downloading/unpacking treap >>>> Running setup.py egg_info for package treap >>>> >>>> file py_treap.py (for module py_treap) not found >>>> Installing collected packages: treap >>>> Running setup.py install for treap >>>> file py_treap.py (for module py_treap) not found >>>> file py_treap.py (for module py_treap) not found >>>> >>>> file py_treap.py (for module py_treap) not found >>>> file py_treap.py (for module py_treap) not found >>>> Successfully installed treap >>>> Cleaning up... >>>> >>>> And it's not successfully installed - py_treap.py is missing. The pyx >>>> code does its job, so the problem is masked, other than the messages >>>> above, and the absence of py_treap.py from >>>> /usr/local/lib/python2.7/dist-packages >>>> >>>> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >>>> least getting packaged up that much. It's not listed in >>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >>>> , but I can see it in >>>> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >>>> . >>>> >>>> My setup.py is at: >>>> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >>>> >>>> I've tried that setup.py and several variations on that theme, but >>>> none seem to include py_treap.py . >>>> >>>> Please make some suggestions? How can I get py_treap.py included in >>>> the pip install? >>>> >>>> Thanks! >>>> >>> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >

1 1

(no subject)
by Warren Postma 22 Jan '14

22 Jan '14

Today I ran across something that made me go "wat?". There is a package named wx in PyPI and pip happily installs it. It's completely useless, the module doc string is not in english, and it's definitely not wxPython. This has been noticed by others: https://bitbucket.org/pypa/pypi/issue/62/please-delete-or-rename-the-wx-pac… What is the proper channel for asking for the renaming or removal of a package? If this package is indeed useless, then it should be removed. If it is useful, but completely undocumented, and appears that import wx followed by dir(wx) shows absolutely nothing useful inside it. This is the entire content of the unfortunately named wx.py module: # -*- coding:utf-8 -*- """ 模拟 PHP 函数 file_get_content 我的Python测试模块 """ def file_get_contents(file): file = open(file) data = '' for line in file: data = data + line file.close() return data The last part translated from Translate.google.com is "My Python test module". How did this get into PyPI? It's pypi page url is: https://pypi.python.org/pypi/wx/1.0.0 Does anyone have to approve these things before they get into PyPI or can any post their hello world programs to PyPi? Can I write three lines, call it pyQt, and upload it? that particular four letter name is unclaimed, yet I bet most pip users would like it if it gave them an error rather than installing some package which is not PyQt, it is just called that. Warren

5 4

inconsistency between PEP425 and wheel tags
by Brian Wickman 21 Jan '14

21 Jan '14

For interpreter-specific tags PEP425 says: "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18, pp19." But in practice if you build with an alternate interpreter you just get the equivalent CPython version level, e.g. pycrypto-2.6.1-pp27-none-macosx_10_7_x86_64.whl for PyPy 2.2.1, when the PEP indicates it should be pp22. As an implementer, should I follow the spec or follow reality? If the latter, should PEP425 be amended? thanks! brian twitter.com/wickman

4 5

Local version identifiers and wheels
by Paul Moore 21 Jan '14

21 Jan '14

PEP 440 allows for local version identifiers, to mark local patches of a distribution. As this stands, local versions cannot be used in wheels, because the identifier is delimited with a dash, which clashes with the separator for the different parts in a wheel filename. What needs to be done to address this? Should we use a different delimiter in PEP 440? Or should we specify a translation for the dash to be used in wheel filenames? This is directly relevant at the moment, because there is a discussion on the distlib tracker of an API to allow locally modifying wheels, and I'm advocating that if this is done, the modified wheel should be forced to have a local version identifier, to avoid confusion with "official" versions. Any thoughts? Thanks, Paul

1 0

Pip 1.5.1 and Virtualenv 1.11.1 Released
by Donald Stufft 20 Jan '14

20 Jan '14

I’ve just released pip 1.5.1 and virtualenv 1.11.1 which fixes a number of issues with 1.5 and 1.11 respectively. It is recommend that all users upgrade to these releases. 1.5.1 (2014-01-20) ------------------ * pip now only requires setuptools (any setuptools, not a certain version) when installing distributions from src (i.e. not from wheel). (Pull #1434). * `get-pip.py` now installs setuptools, when it's not already installed (Pull #1475) * Don't decode downloaded files that have a ``Content-Encoding`` header. (Pull #1435) * Fix to correctly parse wheel filenames with single digit versions. (Pull #1445) * If `--allow-unverified` is used assume it also means `--allow-external`. (Pull #1457) 1.11.1 (2014-01-20) ~~~~~~~~~~~~~~~~~~~ * Fixed an issue where pip and setuptools were not getting installed when using the ``--system-site-packages`` flag. * Updated setuptools to fix an issue when installed with easy_install * Fixed an issue with Python 3.4 and sys.stdout encoding being set to ascii * Upgraded pip to v1.5.1 * Upgraded setuptools to v2.1 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

[ANN] pypiserver 1.1.5 - minimal private pypi server
by Ralf Schmitt 20 Jan '14

20 Jan '14

Hi, I've just uploaded pypiserver 1.1.5 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. https://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver I didn't announce the previous version, so here's the changelog for the last two versions: 1.1.5 (2014-01-20) ------------------ - only list devpi-server and proxypypi as alternatives - fix wheel file handling for certain wheels - serve wheel files as application/octet-stream - make pypiserver executable from wheel file - build universal wheel - remove scripts subdirectory - add --index-url cli parameter 1.1.4 (2014-01-03) ------------------ - make pypiserver compatible with pip 1.5 (https://github.com/schmir/pypiserver/pull/42) -- Cheers Ralf

1 0

Please do not remove dependency_links
by Hannes Schmidt 17 Jan '14

17 Jan '14

I read through the "Removing dependency_links" thread [1] and I beg you not follow through with the deprecation and removal of dependency_links and to rethink your approach. The mentioned thread indicates that research was done to gauge the popularity of the dependency_links in publicly hosted Python projects. That approach is fundamentally flawed: Publicly hosted projects are much more likely to also be available on PyPI than private, closed-source projects. Consequently, their dependencies are also more likely to be hosted on PyPI as well. Because of that, they are much less likely to rely on the dependency_links feature. Another misconception seem to be that dependency_links is predominantly used for installing patched or customized versions of dependencies hosted on PyPI. I'm pretty sure the predominant use case for dependency_links is with projects that are hosted privately, e.g. for an organization's internal use. I represent such an organization and removing dependency_links would impact us negatively. We host a set of internal projects and their dependencies on Bitbucket and we rely on dependency_links to install them directly from there. I understand the motivation for this change – security – but there must be smarter way to handle it. Could we fallback to dependency_links if a PyPI lookup isn't successful? Could we restrict dependency_links to links that share a prefix with the link from which the package is currently being installed? A combination of the two? [1]: https://mail.python.org/pipermail/distutils-sig/2013-October/022937.html -- Hannes Schmidt Software Application Developer Data Migration Engineer Cancer Genomics Hub University of California, Santa Cruz (206) 696-2316 (cell) hannes(a)ucsc.edu

6 17

pip 1.5.1rc1 and virtualenv 1.11.1rc2 released
by Donald Stufft 16 Jan '14

16 Jan '14

I’ve just released virtualenv 1.11.1rc2 and pip 1.5.1rc1, it fixes a number of issues with the latest releases of each one, but most importantly: * virtualenvs created with —system-site-packages not getting pip or setuptools installed into it * Errors about setuptools being too old * Bug fixes that affected certain third party servers hosting packages with Content-Encoding headers * Have —allow-unverified imply —allow-external * Bug fixes that affected cases when a DNS name failed to resolve * Errors with older versions of Python 2.6, most notably from OSX system Pythons In order to try this out in an isolated manner simply: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.11.1rc2.tar.gz $ echo "b23fd95ef06effa2fde9f06fdb86ec38 1.11.1rc2.tar.gz" | md5sum -c 1.11.1rc2.tar.gz: OK $ tar zxf 1.11.1rc2.tar.gz $ python virtualenv-1.11.1rc2/virtualenv.py myVE $ myVE/bin/pip install SomePackage Report any bugs to the pip or virtualenv issues trackers located at: https://github.com/pypa/pip/issues https://github.com/pypa/virtualenv/issues Please do try these out and let us know of any issues! Thanks, ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

distlib 0.1.7 released on PyPI
by Vinay Sajip 16 Jan '14

16 Jan '14

I've just released version 0.1.7 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Fixed issue #39: Changed to work in a situation where the user's home directory is unavailable. Fixed issue #40: Resources code manipulates paths differently for file system and zip resources. Script generation now uses correct logic to determine if running in a venv. Native strings are used in path computations to avoid errors on Windows with Unicode paths. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.7 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new

1 0

"Python Packaging User Guide" has moved
by Marcus Smith 16 Jan '14

16 Jan '14

Fyi, the "Python Packaging User Guide" has moved from bitbucket to github. The new project home is here: https://github.com/pypa/python-packaging-user-guide and the built site is still here: https://python-packaging-user-guide.readthedocs.org/en/latest/ For those who are interested in helping, a quick survey of the text will show many sections marked up with "FIXME" comments. Although general editing is helpful, I think we mostly need "packaging experts" at this point who are willing to invest the time in filling out and correcting the sections they feel strong on. As stated in the README, "The guide is part of a larger effort to improve all of the packaging and installation docs, including pip, setuptools, virtualenv, and wheel. Ultimately, users need more than a "guide" to feel confident about the current tools. They need complete, accurate and inter-consistent documentation across all the projects" As such, a lot of my effort will be going into improving and normalizing the pip/setuptools/virtualenv/wheel docs directly so the Packaging User Guide doesn't have to do all the work on top of those docs. I have issues open for each of the four projects: https://github.com/pypa/python-packaging-user-guide/issues My goal is to have something that's sufficient for a public announcement by PyCon. Marcus

10 30

Metadata 2.0 and single-component version numbers
by Paul Moore 15 Jan '14

15 Jan '14

Metadata 2.0 doesn't support versions that are a single number. This has recently been flagged as a pip issue (https://github.com/pypa/pip/pull/1445) because pip won't handle wheels with a single-component version. We can obviously accept the patch for pip as a backward-compatibility measure, but I'm not sure if that's a good idea. wheels, and much of the new pip functionality, is based around Metadata 2.0 and the new packaging PEPs, at least in principle, so I'd rather get a better view on why Metadata 2.0 doesn't accept these versions. Based on that, we can decide what to do in pip. As a data point, pywin32 uses a single-digit version and can't be built with the free MS compilers, so it would be an ideal candidate for distribution as a wheel. But I've seen nothing from the project indicating that they are even aware of wheels yet, and there may be good reasons why wheels won't work for them (registration of WIndows COM components, for example) So - should Metadata 2.0 allow single-number versions? Should we be advocating a version format change to projects like pywin32? If neither, what will be the impact (and do people have views on whether it's a serious enough issue to warrant pip special-casing this)? Thanks, Paul

4 5

Re: [Distutils] setup.py issue - some files are included as intended, but one is not
by Ethan Furman 14 Jan '14

14 Jan '14

On 01/14/2014 01:26 PM, Dan Stromberg wrote: > On Sat, Jan 11, 2014 at 2:04 PM, Dan Stromberg <drsalists(a)gmail.com> wrote: >> Hi folks. >> >> I have a setup.py problem that's driving me nuts. > > Anyone? I've received 0 responses. I have no answer, but forwarding to Distutils (hopefully it's an appropriate topic ;) >> I have a treap.py file that tries to "import * from pyx_treap.so", and >> failing that, it'll "import * from py_treap.py" (sans extensions of >> course). Naturally, all 3 of these should be included - although in >> the case of pyx_treap.so, it probably should be a .c that's included. >> >> It is also intended to include nest.py, which has a simple form. >> >> Well, treap.py, nest.py and pyx_treap.c are included fine, but I can't >> seem to get py_treap.py to be included for some reason. >> >> I get no errors during "python setup.py sdist upload", but upon >> "python $(which pip) install treap", I get: >> $ sudo /usr/bin/python $(which pip) install treap >> Downloading/unpacking treap >> Running setup.py egg_info for package treap >> >> file py_treap.py (for module py_treap) not found >> Installing collected packages: treap >> Running setup.py install for treap >> file py_treap.py (for module py_treap) not found >> file py_treap.py (for module py_treap) not found >> >> file py_treap.py (for module py_treap) not found >> file py_treap.py (for module py_treap) not found >> Successfully installed treap >> Cleaning up... >> >> And it's not successfully installed - py_treap.py is missing. The pyx >> code does its job, so the problem is masked, other than the messages >> above, and the absence of py_treap.py from >> /usr/local/lib/python2.7/dist-packages >> >> I can clearly see py_treap.py in ./dist/treap-1.35.tar.gz - it's at >> least getting packaged up that much. It's not listed in >> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/SOURCES.txt >> , but I can see it in >> /usr/local/lib/python2.7/dist-packages/treap-1.31.egg-info/top_level.txt >> . >> >> My setup.py is at: >> http://stromberg.dnsalias.org/svn/treap/trunk/setup.py >> >> I've tried that setup.py and several variations on that theme, but >> none seem to include py_treap.py . >> >> Please make some suggestions? How can I get py_treap.py included in >> the pip install? >> >> Thanks!

2 1

PyPI download issues from Rackspace Cloud
by Marius Gedminas 14 Jan '14

14 Jan '14

I recently spin up a Windows VM on Rackspace Cloud. I'm seeing a very weird problem: downloads from PyPI fail with checksum errors, nondeterministically. Sometimes it's a md5 hash mismatch error from pip[1]. Sometimes the error is a CRC error deep in the gzip module. It's not only pip -- I've seen this error from virtualenv[2], I've seen it from get-pip.py. [1] https://jenkins.gedmin.as/job/restview-on-windows/1/console https://jenkins.gedmin.as/job/restview-on-windows/2/console https://jenkins.gedmin.as/job/restview-on-windows/3/console https://jenkins.gedmin.as/job/check-manifest-on-windows/11/console [2] https://jenkins.gedmin.as/job/check-manifest-on-windows/7/console https://jenkins.gedmin.as/job/check-manifest-on-windows/6/console The error is sporadic and tends to go away (and come back) on retries (look at the restview-on-windows jobs: first pip install mock fails, then succeeds, then fails again on next retry). I haven't encountered any download troubles with Firefox. In fact that's how I got 'pip install tox' to work in the end -- pypi downloads of virtualenv/py libraries kept failing, so I downloaded the tarballs with Firefox and pip installed them from local files. I've ran curl https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.… | md5sum - like 10 times and got the same (correct) md5 hash each time. Meanwhile unset PIP_DOWNLOAD_CACHE rm -f virtualenv-1.10.1.tar.gz && pip install -d . virtualenv fails with a bad md5hash error 9 times out of 10. Curiously, it's the same bad hash every time: 20cf17baaf2126f99d6f652831f82d75. I used http://www.python.org/ftp/python/2.7.6/python-2.7.6.msi to install Python on this Windows 2012 server (64-bit OS, 32-bit Python). Any ideas? Marius Gedminas -- IBM motto: "We found five vowels hiding in a corner, and we used them _all_ for the 'eieio' instruction so that we wouldn't have to use them anywhere else" -- Linus Torvalds

3 5

A couple of questions on the PyPI API
by Paul Moore 12 Jan '14

12 Jan '14

I'm writing an application to maintain a mirror of PyPI package metadata, and I'm trying to understand the changelog data. In particular, I've not been able to find anything that lists the valid values for the "action" element of a change record. Looking at what's available if I just query the database, I get: pkg rename from XXX pkg, version remove pkg remove Owner XXX pkg remove Maintainer XXX pkg update hosting_mode pkg, version update list,of,fields pkg, version new release pkg add Owner XXX pkg add Maintainer XXX pkg, version add url XXX pkg, version add XXX file YYY But I don't think that can be all (for a start, there's no "add a new package" - unless the "new release" action can have a version of None, which I haven't seen) Is there a full list anywhere? I had a brief look at the Warehouse code, but short of tracking down all the places journal entries are stored, I couldn't see a way of finding out. Actually, given that the XMLRPC API only lets you get the releases of a package, the release data for a given release or a list of the url data for a given release, there's not really much need for me to worry beyond that level (so, for example, add url and add file both just mean "grab the url data and replace what's there", and "update" just means "replace the package data but don't bother refreshing the urls"). Does that sound reasonable? Thanks, Paul

1 0

Packaging today
by Steve Spicklemire 11 Jan '14

11 Jan '14

Hi Distutils folks, I'm kind of a lurker here, due primarily to the fact that I'm too swamped with various other things to materially contribute to the effort, but I've been lurking for some time hoping to learn enough to avoid troubling anyone with pesky questions. In that respect I've apparently failed, because here comes the question! Background: I'm helping out with a python package: vpython <http://vpython.org> and I'm also teaching an intro scientific computing class this spring. I'm mostly a Mac/Linux user, but my students are often windows users. I would love to permit my students to use enthought/canopy and/or continuum analytics (C.A.) along with vpython. At the moment we're creating binary releases of vpython for windows and mac and posting them on sourceforge <https://sourceforge.net/projects/vpythonwx/>. Bruce has been building the windows binary using VC (no setup.py) in a way that's compatible withpython.org python for windows. I've been building the mac version using a setup.py script I cobbled together that works on MacOSX and Linux. I've noticed that the anaconda system that C.A. installs uses MinGW on windows to build extensions. I'd love to figure out how to build vpython under this system so that my windows users could use them together transparently. (BTW is this the same 'anaconda' that has been discussed on this list recently, or is that something different?) I'm pretty sure I could work out how to build vpython with continuum analytics on the mac (which means building boost + wxPython using the C.A. python). Questions: Is there any way, *today*, to incorporate dependencies on external libraries (e.g., boost) in setup.py? (I've noticed the recent conversation about binary dependencies, but from the discussion it seemed to be mostly about the future...) Where should I go to get the 'latest' advise/documentation on distribution for a package that I want to distribute today (rather than pestering you folks)? thanks! -steve (P.S., I tried to send this earlier today, but it appears to have failed. Apologies if you've gotten this twice) Steve Spicklemire University of Indianapolis Dept. of Physics and Earth Space Sciences spicklemire(a)uindy.edu (317) 788-3313

4 12

Wheel "build tag"
by Bohuslav Kabrda 09 Jan '14

09 Jan '14

Hi all, the wheel package format, as defined in pep 427 specifies, that wheel name can contain an optional "build tag" [1]. This build tag is supposed to be a tie-breaker if two wheels have the same version. I decided that I would try to send a patch to upstream wheel project [2] to support adding this tag. I have a few questions and maybe someone here (Nick?) can help me with this: - Where is the "build tag" supposed to be stored in a built distribution? The pep doesn't seem to specify this. Should it be somewhere in METADATA or pydist.json? - How should the bdist_wheel command be told to use a "build tag"? By using some commandline option, e.g. "--build-tag 3"? My motivation for doing this is: For Fedora's Python 3.4 packaging, I'll have to unbundle the bundled setuptools and pip wheels and make ensurepip work differently. The best approach seems to be to make python3 require setuptools and pip (in other words, they'll always install along), so if user runs "python3 -m ensurepip", ensurepip won't need to do anything. This gets tricky if the same command is run in a virtualenv, since I don't have the bundled wheels. I've discussed this with Nick and we've agreed that the best solution would be to recreate the wheels from the system installed RPMs dynamically and then installing these wheels into virtualenv (I've already done some progress on this front with a small project that I call rewheel [3]). I'm now working on a downstream patch of ensurepip, that would use rewheel to do this. Now let me explain a bit about how we package setuptools/pip in Fedora. When there is a security issue or just a bug in a package, we don't upgrade it to latest upstream version in stable Fedora releases, since as a distribution we try to maintain API/ABI stability. We rather bump the "release" (which is sort of downstream version of the package). E.g. if there is a bug in package pip-%{version}-%{release}, we fix it and bump it to pip-%{version}-%{release+1}. But that means that if we fix a package and someone runs "python -m ensurepip --upgrade", he won't get the fix into virtualenv because the version is the same. That's why I'd like to use Fedora's release as the "build tag", so that this use case would work. I hope this all makes some sense :) Thanks! -- Regards, Bohuslav "Slavek" Kabrda. [1] http://www.python.org/dev/peps/pep-0427/#file-name-convention [2] https://bitbucket.org/dholth/wheel/ [3] https://github.com/bkabrda/rewheel

4 3

File names with spaces
by Maciej (Matchek) Bliziński 08 Jan '14

08 Jan '14

Hello setuptools developers, I'm attempting to package the newest setuptools version (1.1.6) on Solaris 9 and 10. One of the limitations of the Solaris package manager (the old one, pkgadd/pkgrm), is that it is unable to handle file names with spaces. Would you mind renaming "script template.py" to something like "script_template.py"? Maciej

2 3

Virtualenv 1.11.1rc1 Available for Testing
by Donald Stufft 07 Jan '14

07 Jan '14

virtualenv 1.11.1rc1 has been released, which solves a serious issue with the ``—system-site-packages`` flag. In order to test this copy of virtualenv: $ curl -L -O https://github.com/pypa/virtualenv/archive/1.11.1rc1.tar.gz $ echo "e517f28a31792f00699b369932a9b66a 1.11.1rc1.tar.gz" | md5sum -c 1.11.1rc1.tar.gz: OK $ tar zxf 1.11.1rc1.tar.gz $ python virtualenv-1.11.1rc1/virtualenv.py myVE $ myVE/bin/pip install SomePackage ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

errors trying to get setuptools 2.0.2
by Malik Rumi 06 Jan '14

06 Jan '14

Here is what I did and the response: c:\> c:\>ez_setup.py Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-2.0. 2.tar.gz Exception calling "DownloadFile" with "2" argument(s): "An exception occurred during a WebClient request." At line:1 char:1 + (new-object System.Net.WebClient).DownloadFile('https://pypi.python.org/packages ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~ ~~~ + CategoryInfo : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : WebException Traceback (most recent call last): File "C:\Python27\Scripts\ez_setup.py", line 361, in <module> sys.exit(main()) File "C:\Python27\Scripts\ez_setup.py", line 357, in main downloader_factory=options.downloader_factory) File "C:\Python27\Scripts\ez_setup.py", line 282, in download_setuptools downloader(url, saveto) File "C:\Python27\Scripts\ez_setup.py", line 169, in download_file_powershell _clean_check(cmd, target) File "C:\Python27\Scripts\ez_setup.py", line 152, in _clean_check subprocess.check_call(cmd) File "C:\Python27\lib\subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['powershell', '-Command', "(new-object System.Net.WebClient).DownloadFile ('https://pypi.python.org/packages/source/s/se tuptools/setuptools-2.0.2.tar.gz', 'c:\\\\setuptools-2.0.2.tar.gz')"]' returned non-zero exit status 1 c:\> The parts in red were in red in my terminal window. I am on windows 8.1 and I am brand new. Less than a week ago, I installed Distribute because someone on Django Google Groups recommended it to me, along with virtualenv and pip. Then today - well, last night since it was a few hours ago - I found out Distribute is deprecated. So I am trying to follow the uninstall instructions but this is what I got. Just for kicks I ran it again just now - same result. Please help. I took time off from work Thursday and Friday to get all this set up have been at this all weekend only to be stumped again. And no, I do not have Django up and running yet. Thank you.

2 1

Packaging today...
by Steve Spicklemire 06 Jan '14

06 Jan '14

Hi Distutils folks, I'm kind of a lurker here, due primarily to the fact that I'm too swamped with various other things to materially contribute to the effort, but I've been lurking for some time hoping to learn enough to avoid troubling anyone with pesky questions. In that respect I've apparently failed, because here comes the question! Background: I'm helping out with a python package: vpython <http://vpython.org> and I'm also teaching an intro scientific computing class this spring. I'm mostly a Mac/Linux user, but my students are often windows users. I would love to permit my students to use enthought/canopy and/or continuum analytics (C.A.) along with vpython. At the moment we're creating binary releases of vpython for windows and mac and posting them on sourceforge <https://sourceforge.net/projects/vpythonwx/>. Bruce has been building the windows binary using VC (no setup.py) in a way that's compatible with python.org python for windows. I've been building the mac version using a setup.py script I cobbled together that works on MacOSX and Linux. I've noticed that the anaconda system that C.A. installs uses MinGW on windows to build extensions. I'd love to figure out how to build vpython under this system so that my windows users could use them together transparently. (BTW is this the same 'anaconda' that has been discussed on this list recently, or is that something different?) I'm pretty sure I could work out how to build vpython with continuum analytics on the mac (which means building boost + wxPython using the C.A. python). Questions: Is there any way, *today*, to incorporate dependencies on external libraries (e.g., boost) in setup.py? (I've noticed the recent conversion about binary dependencies, but from the discussion it seemed to be mostly about the future...) Where should I go to get the 'latest' advise/documentation on distribution for a package that I want to distribute today (rather than pestering you folks)? thanks! -steve Steve Spicklemire University of Indianapolis Dept. of Physics and Earth Space Sciences spicklemire(a)uindy.edu (317) 788-3313

1 0

Buildout index URL override
by Christian Ullrich 05 Jan '14

05 Jan '14

Hello all, does buildout ever take its default index URL from the distutils configuration, i.e. [py]distutils.cfg? Given that it uses easy_install for most installation tasks, that could make sense, and I'm not sure whether it is actually supposed to happen already, but does not happen on my systems for some reason. Thanks for any info you can give me. -- Christian

2 1

Can pypi python 3 packages be sorted please?
by Mark Lawrence 03 Jan '14

03 Jan '14

So can they be sorted in the same way as the normal index? To my eye the unsorted list looks horrible. OTOH it's too much hassle or this is covered by the Warehouse project I won't lose any sleep over it. Also for pypi issues https://mail.python.org/mailman/listinfo/pydotorg-www points you to https://mail.python.org/mailman/listinfo/catalog-sig where it states it's retired and points you here. How do we get this changed? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

2 3

PyPI classifier for standard library modules/backports?
by Nick Coghlan 02 Jan '14

02 Jan '14

I started putting together a list of standard library modules that are also available on PyPI for the https://wiki.python.org/moin/Python2orPython3 wiki page. Dariusz Suchojad suggested it might be easier if there was a suitable classifier to mark such modules rather than trying to keep a list on the wiki up to date by hand. Perhaps something like "Progamming Language :: Python :: Standard Library"? Thoughts? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 6

Major update to the metadata 2.0 draft PEPs
by Nick Coghlan 01 Jan '14

01 Jan '14

I've just published the first draft of the metadata 2.0 spec that moves all of the fields that aren't part of the core metadata or potentially needed for dependency resolution out to a separate "standard metadata extensions" PEP. I think this makes PEP 426 itself substantially less overwhelming, and also provides convenient names to refer to the various other subsets of the metadata. The new PEP defining the standard extensions is PEP 459. Metadata 2.0: http://www.python.org/dev/peps/pep-0426/ Versioning: http://www.python.org/dev/peps/pep-0440/ Standard Extensions: http://www.python.org/dev/peps/pep-0459/ You can see the full deltas for PEP 426 and PEP 440 here: http://hg.python.org/peps/rev/d18629859154 Or slightly more incremental deltas in the BitBucket repo: https://bitbucket.org/pypa/pypi-metadata-formats/overview With this update, I believe PEP 440 is extremely close to acceptance, as I have tried to resolve a number of problems Donald noted while working on Warehouse, as well as the perennial problem of how to deal with versioning of packages where Linux distro vendors (or other system integrators) have applied additional patches. The PEP 440 changes: - added the "integrator suffix" (essentially an extra release segment, separated from the upstream version number by "-"). It is effectively ignored by most of the matching operators, but sorts as a higher version than the version with no suffix. - allowed date based versions for most purposes, but still reject them for the "~=" compatible version operator. The implied comparison operator for date based versions is ">=". - clarified that zeroes and leading zeroes are permitted and that they should be sorted like numbers - based on discussions with some of the Red Hat security folks, I tweaked the direct URL references to *always* require a hash in the URL, even when using HTTPS or SSH. PEP 426 and PEP 459 are further away from finalisation, but we're definitely getting closer to something I'm happy with. The PEP 426 changes: - many of the fields are gone, having moved to PEP 459 extensions - this also allowed the notion of "essential dependency metadata" (with it's own separate filename) to go away - the extension format changed to require a submapping - extensions are now versioned (with an implied 1.0 if not specified) - as with direct URLs in PEP 440, source URLs are now required to contain a hash PEP 459 overview: - standard extensions using the "python.*" namespace - the details, project, exports and commands extensions just lift several fields out of PEP 426 - the integrator extension is new, and matches the project extension structure. It allows redistributors like a Linux distro or a Python vendor to add their own info without overwriting the upstream project metadata - the new metadata_hooks extension replaces the install hooks design in earlier drafts of PEP 426. In this new scheme, Twisted, for example, would define an appropriate export group hook in order to be told about any new extensions that exported Twisted plugins and take appropriate action In this revised metadata model, distributions *will* trigger their own postinstall metadata hooks, but won't trigger their postuninstall hooks. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 5

migration path for entry_points?
by Marcus Smith 01 Jan '14

01 Jan '14

ok, so PEP459 has exports: http://www.python.org/dev/peps/pep-0459/#the-exports-extension and distlib seems to be implementing them. and "exports" seem to be entry points 2.0. so theoretically, when PEP426/PEP459 becomes live.... what does that mean for setuptools-based projects? (I understand that we might not have a clear idea now, but just trying to narrow down the possibilities) 1) the setuptools "entry_points" keyword (and pkg_resources api) gets re-implemented to understand PEP459 exports? or 2) "entry_points" keeps the old implementation, and possibly tools are written to handle the old and new metadata? or 3) "entry_points" are stranded, and consumers have to rewrite setup.py files to use some new keyword that understands PEP459 or 4) something altogether different... as for why I'm asking, pip itself is considering command extensions, so it's a direct practical matter for us. Marcus

4 6