Mailman 3 July 2014 - Distutils-SIG

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

windows permissions on bdist egg scripts, data?
by Erik Purins 25 Aug '14

25 Aug '14

Hey there, I’m using python 2.7/setuptools 3.6 to build an egg of my code (checked into perforce) using my setup.py. This egg gets installed into a virtualenv. On osx/darwin this works fine. On windows, the bdist creates scripts and data whose permissions are read-only (attrib+r), and they cannot overwrite themselves in a subsequent build of that egg. Similarly, re-installing a new egg using virtualenv fails because the installed egg’s scripts and data are not writable. I thought it might be that when checked out from perforce, the permissions stayed read-only during the build of the egg, so I set all the files in my module to be read-write before building. Even when the original source files’ permissions are all set to read-write, the generated bdist egg script/data content end up becoming read-only. Couple of questions. 1. Since it works fine on osx, it seems like a bug in the windows implementation of setuptools. Is this known behavior, is there a reason for setting this on scripts/data on windows? 2. Is there a hook in the bdist egg build process I can use to set the attributes of the files correctly when the bdist egg is made? How/where should I define this in my setup.py? Kind regards, -e

2 2

Re: [Distutils] build a wheel with waf instead of setuptools
by Daniel Holth 01 Aug '14

01 Aug '14

On Fri, Jul 25, 2014 at 5:51 AM, Daniel Holth <dholth(a)gmail.com> wrote: > Here's a little something I cooked up based on the waf (a build > system) playground/package example. It's a build script for wheel > (what else) that builds a .whl for wheel when you run "waf configure" > and then "waf package" with waf 1.8.0. I've tested it in Python 2.7. > > Waf is a build system that, unlike distutils, won't fall over > immediately when you try to extend it. One of its features is support > for building Python extensions, and it is itself written in Python. > > Right now for expedience instead of generating the METADATA or > entry_points.txt from setup() arguments it just copies them from files > at the root, and the command "waf dist" (for producing sdists) > includes too many files but that is easy to fix. In particular I liked > using ant_glob() a lot better than MANIFEST.in. The wscript does not > use MANIFEST.in. > > https://bitbucket.org/dholth/wheel/src/1bbbd010558afe215a01cfd238a26da42ad6… > > If there are any interested waf-wizards out there then we could take > the wheel building feature and pull it out of the individual wscript, > refine it a bit, and have another non-distutils way to publish Python > packages. > > Daniel Holth This kind of thing will require us to implement a flag that tells pip "setup.py cannot install; go through wheel" which is somewhere in the plans..

4 5

PEP draft on PyPI/pip package signing
by Giovanni Bajo 29 Jul '14

29 Jul '14

Hello, on March 2013, on the now-closed catalog-sig mailing-list, I submitted a proposal for fixing several security problems in PyPI, pip and distutils[1]. Some of my proposals were obvious things like downloading packages through SSL, which was already in progress of being designed and implemented. Others, like GPG package signing, were discussed for several days/weeks, but ended up in discussion paralysis because of the upcoming TUF framework. 16 months later, we still don’t have a deployed solution for letting people install signed packages. I see that TUF is evolving, and there is now a GitHub project with documentation, but I am very worried about the implementation timeline. I was also pointed to PEP458, which I tried to read and found it very confusing; the PEP assumes that the reader must be familiar with the TUF academic paper (which I always found quite convoluted per-se), and goes with an analysis of integration of TUF with PyPI; to the best of my understanding, the PEP does not provide a clear answer to practical questions like: * what a maintainer is supposed to do to submit a new signed package * how can differ maintainers signal that they both maintain the same package * how the user interface of PyPI will change * what are the required security maintenance that will need to be regularly performed by the PyPI ops I’m not saying that the TUF team has no answers to these questions (in fact, I’m 100% sure of the opposite); I’m saying that the PEP doesn’t clearly provide such answers. I think the PEP is very complicated to read as it goes into integration details between the TUF architecture and PyPI, and thus it is very complicated to review and accept. I would love the PEP to be updated to provide an overview on the *practical* effects of the integration of TUF within PyPI/pip, that must be fully readable to somebody with zero previous knowledge of TUF. As suggested by Richard Jones during EuroPython, I isolated the package signing sections from my original document, evolved them a little bit, and rewritten them in PEP format: https://gist.github.com/rasky/bd91cf01f72bcc931000 To the best of my recollection, in the previous review round, there were no critical issues found in the design. It might well be that TUF provides more security in some of the described attack scenarios; on the other hand, my proposal: * is in line with the security of (e.g..) existing Linux distros * is very simple to review, analyze and discuss for anybody with even a basic understanding of security * is much simpler than TUF * is a clear step forward from the current situation * cover areas not covered by PEP458 (e.g.: increasing security of account management on PyPI) * can be executed in 2-3 months (to the alpha / pre-review stage), and I volunteer for the execution. I thus solicit a second round of review of my proposal; if you want me to upload to Google Docs for easier of commenting, I can do that as well. I would love to get the PEP to its final form and then ask for a pronouncement. I apologize in advance if I made technical mistakes in the PEP format/structure; it is my first PEP. [1] See here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTW… -- Giovanni Bajo :: rasky(a)develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it

9 26

Other ideas from today's packaging meetup at EuroPython
by Richard Jones 27 Jul '14

27 Jul '14

Several great ideas came out of today's meetup. Some of those I'll leave to the proponents themselves to post about, but a couple of little nuggets for thought: 1. reject wheel uploads in the absence of an sdist in the index (the linux guys were really happy about that as a proposal ;) 2. add a system-wide configuration option to pip etc. so that there could be a system-wide override of the package index to use Richard

11 24

PEP 470 discussion, part 3
by Richard Jones 25 Jul '14

25 Jul '14

I have been mulling over PEP 470 for some time without having the time to truly dedicate to addressing it. I believe I'm up to date with its contents and the (quite significant, and detailed) discussion around it. To summarise my understanding, PEP 470 proposes to remove the current link spidering (pypi-scrape, pypi-scrape-crawl) while retaining explicit hosting (pypi-explicit). I believe it retains the explicit links to external hosting provided by pypi-explicit. The reason given for this change is the current bad user experience around the --allow-external and --allow-unverified options to pip install. That is, that users currently attempt to install a non-pypi-explicit package and the result is an obscure error message. I believe the current PEP addresses the significant usability issues around this by swapping them for other usability issues. In fact, I believe it will make matters worse with potential confusion about which index hosts what, potential masking of release files or even, in the worst scenario, potential spoofing of release files by indexes out of the control of project owners. I would like us to consider instead working on the usability of the existing workflow, by rather than throwing an error, we start a dialog with the user: $ pip install PIL Downloading/unpacking PIL PIL is hosted externally to PyPI. Do you still wish to download it? [Y/n] y PIL has no checksum. Are you sure you wish to download it? [Y/n] y Downloading/unpacking PIL Downloading PIL-1.1.7.tar.gz (506kB): 506kB downloaded ... Obviously this would require scraping the site, but given that this interaction would only happen for a very small fraction of projects (those for which no download is located), the overall performance hit is negligible. The PEP currently states that this would be a "massive performance hit" for reasons I don't understand. The two prompts could be made automatic "y" responses for tools using the existing --allow-external and --allow-unverified flags. I also note that PEP 470 says "PEP 438 proposed a system of classifying file links as either internal, external, or unsafe", whereas PEP 438 has no mention of "unsafe". This leads "unsafe" to never actually be defined anywhere that I can see. Finally, the Rejected Proposals section of the PEP appears to have a couple of justifications for rejection which have nothing whatsoever to do with the Rationale ("PyPI is fronted by a globally distributed CDN...", "PyPI supports mirroring...") As Holger has already indicated, that second one is going to have a heck of a time dealing with PEP 470 changes at least in the devpi case. "PyPI has monitoring and an on-call rotation of sysadmins..." would be solved through improving the failure message reported to the user as discussed above. Richard

7 23

Packaging meetup @ EuroPython 2014
by Richard Jones 23 Jul '14

23 Jul '14

I've set the time for the packaging meetup to 11:00 this Thursday. There is currently no meeting room available for such a gathering, so we'll just meet outside in the garden. Richard

2 2

setup.py sdist not including my README.md
by Joshua Richardson 22 Jul '14

22 Jul '14

Sorry for the Noob question, but I've been banging my head on this all afternoon. Following some tutorial I found somewhere on PyPI, I created a setup.py which included a line like this: # Get the long description from the relevant file with open(path.join(here, 'README.md'), encoding='utf-8') as f: long_description = f.read() and then in the setup() function, I pass long_description=long_description Somehow this worked, when I installed the package on my development box (Mac with setuptools 5.1), so I prematurely published it to PyPI. Unfortunately, when I tried installing it to a fresh CentOS box (setuptools 0.6c5-2.el5), I got this error: Downloading/unpacking Dumper Downloading Dumper-1.0.1.tar.gz Running setup.py egg_info for package Dumper .... <stacktrace omitted, let me know if you need it> IOError: [Errno 2] No such file or directory: '/tmp/pip-build/Dumper/README.md' So, I checked the tarball and saw that README.md is not in there. I read the setuptools developer guide, and decided maybe I should try: include_package_data=True, I re-ran "setup.py sdist" and checked the tarball, and the README.md is still not in there. Ok, maybe because my source-control is git. So, I added a MANIFEST.in, containing one line: "README.md". Still no luck. So, I added: package_data={ '': ['README.md'], }, Still a no-go. Now I'm feeling stupid. SOS. Thanks for reading this far! --josh

6 7

Installing setuptools on an un-networked computer
by Peter Ganong 22 Jul '14

22 Jul '14

Hi, I am doing research with confidential data on a Windows 7 computer not connected to the Internet. I can bring files in on a thumb drive. How can I install setuptools? To be clear, the issue is not the one answered in the help file here <https://pythonhosted.org/setuptools/easy_install.html#installing-on-un-netw…>, but is how to install setuptools itself on an unnetworked drive. Thanks, Peter -- Peter Ganong PhD Candidate in Economics at Harvard scholar.harvard.edu/ganong/

3 2

503 backend read error on PyPI
by Wichert Akkerman 18 Jul '14

18 Jul '14

Just seen on https://pypi.python.org/pypi from a Dutch IP: Error 503 backend read error backend read error Guru Mediation: Details: cache-iad2132-IAD 1405663962 2954825964 Wichert.

1 2

setuptools develop command garbles binary files specified as scripts setup() parameter on windows
by David Genest 17 Jul '14

17 Jul '14

Hi, In our environment, we are using the scripts parameter to package external binary files, which is really a practical use case: 1) it allows for bundling of executables in our package python distribution (with pertaining dlls etc) 2) the binary is available in an activated virtualenv For the install case, everything is ok, the binaries are correctly installed. But for the develop command, setuptools garbles the binary by copying the binary file opened in text mode and writing it in binary mode. This is because of the line ending translation. I implemented a naive fix, only to be surprised by the python3 unicode management, and caused a regression. (see https://bitbucket.org/pypa/setuptools/issue/210/on-windows-binary-files-spe…) The less naive approach to the problem was implemented in https://bitbucket.org/pypa/setuptools/pull-request/60/fix-regression-in-iss… but the pull request was rejected. But the problem still stands. How can I use the develop command and not have my binaries specified as scripts be garbled. Jason R. Coombs argued that this change would add undocumented special casing, but I am inclined to disagree because the use case works: * on *nix platforms (because the open function does not translate line endings (the "b" mode has no effect on this platform)) * it already works when we go through the install flow (on every platform) * there is a discrepancy between install and develop semantics, for the scripts parameter I would like to know what is the take in the community on this. Note that this is the first time I contribute a change, so I may not be up to speed with all the dos and don'ts. Thanks. David.

3 5

PyPI not rendering ReST long_description
by Wichert Akkerman 16 Jul '14

16 Jul '14

I just uploaded a new pyramid_sqlalchemy package to PyPI. Looking at the distribution page at https://pypi.python.org/pypi/pyramid_sqlalchemy PyPI is not rendering ReST. I can’t figure out why though: according to both “python setup.py check” and "python setup.py --long-description | rst2html-2.7.py > /dev/null” there are no ReST syntax errors in me description. Is there any way to see why PyPI is not rendering my ReST? Wichert.

4 10

devpi-2.0.0: web/search UI, replication, fixes
by holger krekel 16 Jul '14

16 Jul '14

devpi-2.0.0: web, search, replication for PyPI indexes ========================================================== The devpi system in version 2.0 brings tons of fixes and new features for the private github-style pypi caching server, most notably: - a new web interface featuring search of metadata and documentation as well as easier navigation and showing of test results per release file. - a new transactional storage system (based on sqlite) supporting real-time replication over http. - a new (experimental) hook system for extending server-side functionality. - ported to a solid web framework and wsgi-server: pyramid and waitress. Upgrade note: devpi-server-2.0 requires to ``--export`` your 1.2 server state and then using ``--import`` with the new version before you can serve your private packages through devpi-server-2.0.0. Also, please checkout the web plugin if you want to have a web interface:: http://doc.devpi.net/2.0/web.html Here is a Quickstart tutorial for efficient pypi-mirroring on your laptop:: http://doc.devpi.net/2.0/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/2.0/quickstart-releaseprocess.html If you want to host a devpi-server installation with nginx/supervisor and access it from clients from different hosts:: http://doc.devpi.net/2.0/quickstart-server.html More documentation here:: http://doc.devpi.net/2.0/ many many thanks to Florian Schulze who implemented the new ``devpi-web`` package and helped with many other improvements. have fun, Holger Krekel, merlinux GmbH 2.0.0 -------------- devpi-server: - major revamp of the internal core of devpi to support replication (both master and server code), a plugin architecture with the new devpi-web plugin providing a new web interface. Mostly done by Florian Schulze and Holger Krekel. - moved all html views except for files and the simple index to new devpi-web package. Thanks to Florian Schulze for the PR. - implement issue103: By default if you register a package in an index, no lookup on pypi is made for that package anymore. You have to add the package to the pypi_whitelist of the index to let pypi releases be mixed in. This is to prevent malicious uploads on pypi to overwrite private packages. - change json api to get rid of the different meaning of URLs with and without a trailing slash. "/{user}/" is now the same as "/user" and always lists indices. "/{user}/{index}" and "/{user}/{index}/ now always lists the index config and the contained per-stage projects (not inherited ones). - switch the wsgi app to use Pyramid and waitress for WSGI serving. - don't refresh releaselinks from the mirroring thread but rather rely on the next access to do it. - fix issue98: deleting a project config or a project version now accepts names which map to the canonical name of a project. - fix issue82 and fix issue81: root/pypi now provides the same attributes as normal indexes and results in a 409 MethodNotAllowed http code when trying to change the config. - fix issue91: make serverport available as well. Thanks David Bonner. - fix issue100: support large file uploads. As we switched away from bottle to pyramid, the body-size limit is gone. - fix issue99: make "devpi-server --start" etc work when devpi-server is not itself on PATH (by using sys.argv[0] for finding the binary) - fix issue84: uploading of wheels where the registered package name has an underscore works despite a wheel's metadata carrying hyphens instead. At submit-file time we now lookup the registered name and use that instead of assuming the one coming with the wheel is the correct one. - add refresh button on root/pypi project simple index pages which clears the internal cache to force a refetch from PyPI. - implement issue75: We use the custom X-Devpi-Auth header for authentication now, instead of overwriting the Authentication header. - added experimental support for using client certificates when running as a replica of a server running behind a proxy devpi-client: - Compatibility with devpi-server >= 2.0.0 - introduce "patchjson PATH JSONFILE" command which allows to send a request containing a json data structure to a specified path - fix issue85: "devpi list -v" now shows package names with latest versions. - implement issue75: We use the custom X-Devpi-Auth header for authentication now, instead of overwriting the Authentication header. - added experimental support for basic authentication by parsing user and password from the url given to the "devpi use" command. - issue74: added experimental support for client side certificates via "devpi use --client-cert" devpi-web: - initial release

1 1

PyPI changelog support, releases.json / common NEWS.rst format?
by mario＠include-once.org 15 Jul '14

15 Jul '14

I'm currently tinkering on a freshmeat substitute. And for automating release announcements been looking around for common package meta data schemes. PyPIs /pypi/pkgname/json (or the xmlrpc interface) looks quite interesting. It obviously mostly targets dependency management and systemic categorization. So from googling around this never came up: But would it be feasible to include a version changelog / release summary via du.register? Of course, I'm referring to a human-readable "This version adds and fixes..." changelog, not the (name, version, timestamp) journal tuple. The releases{} per-URL `comment_text` seems widely unused. Was that its purpose? I hope this isn't getting too off-topic, this is -just for comparison and context- what I'm intending to eventually map PyPI release streams onto: http://fossil.include-once.org/freshcode/wiki/releases.json There are probably other priorities for distutils / warehouse currently. So alternatively, is there a semi-standard for NEWS.txt or CHANGELOG etc. files within Python packages? Cheeseshop project homepages which also include a release notes list via `long_description` seem far and few between. I actually found just one: https://pypi.python.org/pypi/py-translate Which seems to share a reStructuredText source for documentation and pypi homepage: https://raw.githubusercontent.com/jjangsangy/py-translate/master/HISTORY.rst (I'd have presumed Markdown-style release notes to be favoured.) Anyway, is there an estimate on how many packages include release notes at all?

3 2

PyPI XMLRPC interface no longer works with Python 2.6
by M.-A. Lemburg 08 Jul '14

08 Jul '14

I just tried the documented interfaces for PyPI (https://wiki.python.org/moin/PyPIXmlRpc) with Python 2.6 and it fails with an error: > python pypirpc.py Traceback (most recent call last): File "pypirpc.py", line 7, in <module> pprint.pprint(client.package_releases('egenix-web-installer-test')) File "/usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py", line 1253, in request return self._parse_response(h.getfile(), sock) File "/usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py", line 1390, in _parse_response p.close() File "/usr/local/python-2.6-ucs2/lib/python2.6/xmlrpclib.py", line 604, in close self._parser.Parse("", 1) # end of data xml.parsers.expat.ExpatError: no element found: line 1, column 0 The call works with Python 2.7. It appears that xmlrpclib is not receiving any body data from the server. The returned data in 2.7 looks completely harmless: send: "POST /pypi HTTP/1.1\r\nHost: pypi.python.org\r\nAccept-Encoding: gzip\r\nUser-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: text/xml\r\nContent-Length: 185\r\n\r\n<?xml version='1.0'?>\n<methodCall>\n<methodName>package_releases</methodName>\n<params>\n<param>\n<value><string>egenix-web-installer-test</string></value>\n</param>\n</params>\n</methodCall>\n" reply: 'HTTP/1.1 200 OK\r\n' header: Server: nginx/1.6.0 header: Content-Type: text/xml header: charset: UTF-8 header: Strict-Transport-Security: max-age=31536000; includeSubDomains header: Transfer-Encoding: chunked header: Accept-Ranges: bytes header: Date: Tue, 08 Jul 2014 14:19:41 GMT header: Via: 1.1 varnish header: Connection: keep-alive header: X-Served-By: cache-fra1229-FRA header: X-Cache: MISS header: X-Cache-Hits: 0 header: X-Timer: S1404829181.210045,VS0,VE325 body: "<?xml version='1.0'?>\n<methodResponse>\n<params>\n<param>\n<value><array><data>\n<value><string>0.2.0</string></value>\n</data></array></value>\n</param>\n</params>\n</methodResponse>\n" Could this be a network error rather than a program one ? The code in 2.7 does a retry in case of a connection reset or abort, which code in 2.6 and earlier does not apply. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 08 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-07-21: EuroPython 2014, Berlin, Germany ... 13 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

2 7

pip, virtualenv, setuptools on Windows
by M.-A. Lemburg 07 Jul '14

07 Jul '14

I wonder what our story is for the pip, virtualenv, setuptools packaging setup on Windows. At the moment, the only way to get this setup seems to be following guides such as these: http://www.tylerbutler.com/2012/05/how-to-install-python-pip-and-virtualenv… But that's not really how Windows users expect things to work. They want to click on an installer, preferably a MSI one, since that integrates well with managed corporate Windows environments, and then expect everything to just work. Much in the same way they install Python itself. Is there anything planned in this area ? Hint: If someone were to send in a grant request to build such an installer for Windows, I'm sure the PSF board would love to fund work such work. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-07-21: EuroPython 2014, Berlin, Germany ... 14 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

6 8

wheel 0.24.0 released
by Daniel Holth 06 Jul '14

06 Jul '14

The much-anticipated 0.24.0 release of wheel has occurred. - The python tag used for pure-python packages is now .pyN (major version only). This change actually occurred in 0.23.0 when the --python-tag option was added, but was not explicitly mentioned in the changelog then. - wininst2wheel and egg2wheel removed. Use "wheel convert [archive]" instead. - Wheel now supports setuptools style conditional requirements via the extras_require={} syntax. Separate 'extra' names from conditions using the : character. Wheel's own setup.py does this. (The empty-string extra is the same as install_requires.) These conditional requirements should work the same whether the package is installed by wheel or by setup.py. So we support and convert setuptools' conditional requiremenst to METADATA. However, although pkg_resources.working_set.by_key['wheel']._dep_map is correct, pip install wheel[extraname] doesn't seem to work at all; this bug may be fixed in an unreleased pip? Daniel Holth

1 0

Specifying Python version as part of requirements
by Brett Cannon 06 Jul '14

06 Jul '14

I just checked PEP 440 and pip doesn't seem to have anything specific, so I thought I would ask if there is any way now or in the future to specify that a dependency is only needed for certain versions of Python? My current use case is I want to use unittest2, but pip errors out during installation under Python 3 of it since unittest2 imports itself and triggers a syntax error. Instead of having to hack around this by making it a conditional include in my setup.py I would like to declare that I only need it in Python 2.x in a requirements.txt file or something.

3 7

zc.buildout & Docker container images
by Nick Coghlan 04 Jul '14

04 Jul '14

Hey all (but especially the zc.buildout users), With zc.buildout being used to define repeatable application deployments, and Docker images acting as a way to snapshot entire execution environments for ease of deployment on any environment than can run a container (which is basically any modern Linux at this point, or will be once CentOS 7 is available), has any experimented with using zc.buildout from a Dockerfile to define an image? If yes, is it written up anywhere? If not, is anyone intrigued enough by the idea to considering trying it out and writing up the results as a blog post somewhere? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 9

Let's summit at EuroPython
by Richard Jones 03 Jul '14

03 Jul '14

Hi folks, I'd like to get interested folks together at EuroPython to get up to date, talk through current issues and generally catch up on all things Python Packaging. Specific things we'll be talking about: - current tools: where they're at and what their plans are - the state of PEPs 426, 440, 459, 470 - related to those: how linux distros are doing and feel about PEPs 426 and 459 - the state of the docs and user experience If you're interested in coming along (and I've not already contacted you) then please let me know and once we've sorted out the details of when and where I'll get back in touch with you. Please let me know what you'd like to have us discuss! Richard

3 3

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG July 2014