Mailman 3 December 2014 - Distutils-SIG

Can now compile simple extensions
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker April 29, 2018

April 29, 2018

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

force static linking
by gordon March 23, 2015

March 23, 2015

Hello, I am attempting to build statically linked distributions. I am using docker containers to ensure the deployment environment matches the build environment so there is no compatibility concern. Is there any way to force static linking so that wheels can be installed into a virtual env without requiring specific packages on the host? Thanks, Gordon

7 7

API CHANGE - Migrating from MD5 to SHA2, Take 2
by Donald Stufft March 8, 2015

March 8, 2015

Starting a new thread with more explicit details at Richard’s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256). Simple API ---------- Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the ~60 or so externally hosted files which are using #md5 links will be fetched (one time) verified, and have their #md5= hash replaced with a computed #sha256= hash. Impact: - pip: Will work with no issues, pip has supported sha256 since 1.2, and < 1.2 will install without a hash just fine. - setuptools: Will work with no issues, setuptools has supported sha256 since 0.9 and < 0.9 will install without a hash just fine. - distribute: Doesn't support sha256, will intall without a hash just fine. - buildout: Uses setuptools/distribute to do the downloads I believe. - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out if they do not exist. JSON / XMLRPC API ----------------- Keep the md5_sum field, add an additional sha256_sum, suggest that applications switch to using sha256 for verification. Impact: - bandersnatch: bandersnatch will continue to use the md5_sum field from the JSON (and previously XMLRPC) and should be updated to using sha256 in the future. Web UI ------ Simply replace any use of MD5 with SHA256, no clients are expected to access anything here so this should be perfectly fine. Other Clients ------------- - pep381client: Doesn't do anything special with the hash, will continue to work. - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I can't determine where it gets the md5sum from and what it will do if it doesn't exist. It does have some handling of md5 though. List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, which is generated by looking at what is downloading the files from PyPI. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

4 10

Role of setuptools and eggs in "modern" distributing...
by Chris Barker Jan. 8, 2015

Jan. 8, 2015

Hi folks, I'm trying to package up a complex system and would like to do it the correct, modern way. In particular, this involves a bunch of compiled extensions, as well as dependencies on both the scientific "stack" and common Web app packages. (can you tell I'm building a web service front-end for computational code?) This is actually a bit of a trick, because the web development community is generally doing a good job up supporting PyPi and pip, whereas the complications of the scientific software tools have folks relying more on tools like Enthought and Continuum. So far, we've been doing mostly pip and struggling with build our own for the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL, and....). But at the end of all this we'd like to be able to distribute and make it easy on end users to use our tools. I figure we we'll do one (or both) of: - providing a custom "wheel house" with our packages and the dependencies that are hard to come by - provide a binstar channel with conda packages of all the same stuff but a totally different set of "other" packages. At the moment, I'm working on making conda packages, which brings me to my questions. I'm a bit confused about the role of setuptools with pip. On the one hand, pip depends of setuptools. On the other hand, pip supposed doesn't "do" eggs, and prefers wheels. But I find that I keep getting eggs whether I want them or not. IN fact, as far as I can tell, the way to get pip to instal something from git repo is: git+https://url_to_the_repo.git#egg=name_of_package why isn't that "wheel=name_of_package" and will it work if setuptools was not used in the packages setup.py??? Frankly I've generally found setuptools and eggs to be overly heavy weight and annoying -- the only reason I use setuptools at all is that I REALLY like develop mode -- which I can now get with "pip --editable" or does that give me setuptools develop mode anyway, i.e. do I need to have used setuptools.setup for it to work? So question one: do I need to use setuptools.setup rather than plain old distutils.setup? Question 2: What about setuptools: "install_requires" I generally like the pip requirements.txt approach. It's up to the installation tool, not the packaging tool to mange requirements. But then again, it does make sense to declare the requirements in setup.py. But the issue at hand is that install_requires is doing some odd things with conda: conda, of course, is designed to manage dependencies itself, and those are declared in the conda build meta.yaml file. Note that conda dependencies can have nothign to do with python -- the whole point of conda -- a conda pacakge can depend on any other conda package, including C libs, etc. But the issue at hand is that conda build doesn't re-invent setup.py -- but rather you generally simple call "setup.py install" from your conda build script. Hence th issue at hand: Using setuptools.setup, and specifying "install_requires", then kicks in setuptools trying to resolve dependencies I don't want it to deal with. I read Donald's "setup.py vs requirements.txt", and I guess I get it, but it still seems quite redundant -- I see the point of separating out “abstract dependencies” and “concrete dependencies”. However, the nifty solution of only putting: --index-url https://pypi.python.org/simple/ in the requirements.txt doesn't work for complex cases. In practice, we find we need to specify version numbers of some dependencies, and have some go strait into a git repo, etc. So we need a messy requirements.txt file. And, in fact, I think that's where is belongs -- the goal of the requirements.txt file is so pip can "do the right thing" and go grab everything you need. But, in fact, it also is quite human readable, and serves quite well as the documentation for the "abstract dependencies" as well. So I think the way to go is to keep requirements in requirements.txt, and leave them out of the setup.py. Can I dump setuptools.setup, as well?? Sorry, this is a bit rambling and short of clear questions, but I'm trying to get a handle on what best practices really are these days... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

8 14

Last call for feedback: PEP 440 exclusive ordered comparison fix
by Nick Coghlan Jan. 2, 2015

Jan. 2, 2015

Donald is keen to get the updated versions of packaging/pip/setuptools out that fix the regression in handling exclusive ordered comparison, so this is a last call for feedback on those changes before we publish the updated version of the PEP to python.org (and then update the tools accordingly). Given the proximity to New Year's celebrations, I currently plan to publish the updated version some time on Saturday 3rd January (UTC+10). The actual definition of the change is in https://github.com/pypa/interoperability-peps/commit/225de0a41270189dbfd4a0… The key fix is making "> X.Y" accept "X.Y.Z" maintenance releases again, which brings PEP 440 into line with the behaviour in pkg_resources prior to the release of setuptools 8.0. The change log also notes that this update includes the revision to the normalised form of release candidates: https://github.com/pypa/interoperability-peps/commit/68ce85faaca1ee715e2a9f… There's at least one more clarification to the PEP text still to be made based on community feedback ( https://github.com/pypa/interoperability-peps/issues/6), but that doesn't require any changes to the tools, so I don't mind if it doesn't make it into this initial update. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

New User: depends not appearing in control file
by Geoff Clements Dec. 30, 2014

Dec. 30, 2014

Hi! I just started using stdeb but have run into a problem which I can't solve. Just for background, so you know at what level to aim replies, I'm a long-time programmer, over 30 years in fact. I make much of what I do available to others but I've only just got around to packaging, so I know absolutely nothing about how to package code (but I'm learning!). I'm using stdeb 0.8.2 installed via pypi because the Ubuntu repos are so hopelessly out of date. I'm building a package for python3 only and uploading to an Ubuntu ppa which I have done but with the following problems: Here's my stdeb.cfg file: [DEFAULT] Depends: libav-tools (>= 6:11), lsdvd (>= 0.17), mplayer2 (>= 2.0), python3- pyqt4 (>= 4.11), python3-pyudev (>= 0.16) XS-Python-Version: >= 3.4 MIME-Desktop-Files: share/ravenc.desktop Suite: utopic Section: universe/video Now here is the control file in the source package that is produced (minus the package description, slightly redacted): Source: ravenc Maintainer: Geoff Clements <e-mail address> Section: universe/video Priority: optional Build-Depends: python3-setuptools, python3-all, debhelper (>= 7.4.3) Standards-Version: 3.9.1 X-Python-Version: >= 3.4 Package: python3-ravenc Architecture: all Depends: ${misc:Depends}, ${python3:Depends} =========== The problem here is that the dependencies have not been carried across to the control file. In addition, when I try to build a binary package I get this warning: dpkg-gencontrol: warning: Depends field of package python3-ravenc: unknown substitution variable ${python3:Depends} This also appears in the build log if a submit the source packages to Ubuntu to include in my ppa. -- Geoff

2 1

Detecting rst errors when uploading to pypi
by John Anderson Dec. 30, 2014

Dec. 30, 2014

Hey, I'm trying to get my README to render properly on pypi but I haven't found a way to detect the errors pypi is experiencing. The package I'm trying to upload is here: https://pypi.python.org/pypi/pyramid_celery I've checked the README.rst with restview, rst2html, and collective.showdocs and they all render the RST just fine. It seems like I'm not the only one having this issue: https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-i… So what is the recommended route for debugging the problem? I would really like to fix it but without know the problem I would just to make random guesses. Thanks, John

5 6