Mailman 3 April 2013 - Distutils-SIG

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

error in windows 7 installation
by Manuel López 13 May '14

13 May '14

Hello Everybody: I am trying to install distribute-0.6.25 in a windows 7 machine. I have python 2.7.3 in 32 bits, although the machine is 64 bits. I installed the 32 bit python version because I want to install ipython and there is no 64 bit builds of the Windows installer for ipython. In any case, upon installing disuitils with python setup.py install I get the following error error: C:\manuel\prog_red\python\distribute-0.6.25\distribute-0.6.25\setuptools\cli-32.exe: No such file or directory Any help will be greatly appeciated. Thanks, Manuel -- Manuel López Mariscal Depto. de Oceanografía Física/CICESE malope(a)cicese.mx

2 1

Fixing Cheese Shop search
by David Wilson 14 May '13

14 May '13

Hi there, In a fit of madness caused by another 30 seconds-long PyPI search I decided to investigate the code, in the hopes of perhaps finding something simple that would alleviate the extremely long search time. I discovered what appears to be a function that makes 6 SQL queries for each term provided by the search user, in turn those queries expand to what appear to be %SUBSTRING% table scans across the releases table, which appears to contain upwards of half a gigabyte of strings. Now the root cause has been located, what to do about it? I looked at hacking on the code, but it seems webui.py is already too massive for its own good, and in any case PostgreSQL's options for efficient search are quite limited. It might be all-round good if the size of that module started to drop.. I wrote a crawler to pull a reasonable facsimile of the releases table on to my machine via the XML-RPC API, then arranged for Xapian to index only the newest releases for each package. The resulting full-text index weighs in at a very reasonable 334mb, and searches complete almost immediately, even on my lowly Intel Atom colocated server. I wrote a quick hack Flask app around it, which you can see here: http://5.39.91.176:5000/ The indexer takes as input a database produced by the crawler, which is smart enough to know how to use PyPI's exposed 'changelog' serial numbers. Basically it is quite trivial and efficient to run this setup in an incremental indexing mode. As you can see from the results, even my lowly colo is trouncing what is currently on PyPI, and so my thoughts tend toward making an arrangement like this more permanent. The crawler code weighs in at 150 lines, the indexer a meagre 113 lines, and the Flask example app is 74 lines. Implementing an exact replica of PyPI's existing scoring function is already partially done at indexing time, and the rest is quite easy to complete (mostly cutpasting code). Updating the Flask example to provide an XML-RPC API (or similar), then *initially augmenting* the old search facility seems like a good start, with a view to removing the old feature entirely. Integrating indexing directly would be pointless, the PyPI code really doesn't need anything more added to it until it gets at least reorganized a little. So for the cost of 334mb of disk, a cron job, and a lowly VPS with even just 700MB RAM, PyPI's search pains might be solved permanently. Naturally I'm writing this mail because it bothers me enough to volunteer help. :) Prototype code is here: https://bitbucket.org/dmw/pypi-search (note: relies on a pre-alpha quality DB library I'm hacking on) Thoughts?

2 3

HTTPS and certificate check update for distribute ?
by M.-A. Lemburg 05 May '13

05 May '13

The latest pip supports HTTPS URLs and certificate checks (according to the change log). Will there be a release of distribute that implements the same changes ? The current 0.6.36 still defaults to the HTTP PyPI address and doesn't do certificate checks. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 25 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2013-04-17: Released eGenix mx Base 3.2.6 ... http://egenix.com/go43 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

4 7

distil 0.1.1 released
by Vinay Sajip 03 May '13

03 May '13

I've released version 0.1.1 of distil, downloadable from [1]. It's based on distlib 0.1.2. The other changes are as follows: * Added "distil init" to support creating an initial version of package.json metadata, which can then be added to during development. * Added "distil link" to support "editable" installations, similar to pip install -e local_dir. * Take into account pre-confirmation (-y) during uninstallation when dists that are no longer needed are found. These are now removed automatically when -y is specified. * Fixed error in setting up SSL certificate verification, and adjusted PyPI URLs to be https:// where specified as http:// in metadata. Successful SSL verification is now logged. * Added --local-dists option to allow local wheels and sdists to be used for installation. * Fixed a bug in the handling of local archives (e.g. those returned through a configured DirectoryLocator). Local archives shouldn’t be deleted after unpacking. * Added --python-tags argument to distil package when building wheels to configure the tags for the built wheel. * Added --no-unpack option to distil download. * Fixed problem with rollback on error caused by not recording SHARED and RECORD correctly. * Fixed bug in writing entry points (EXPORTS) file. * Use of 2to3 now defaults to True when run under 3.x. * Fixed bug when run in venvs which caused e.g. dist-packages to be used instead of site-packages. * Improved error message when run with Python 2.5 (not supported, but this is now clear from the error message). Your feedback will be gratefully received! Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py

7 22

distlib 0.1.2 released on PyPI
by Vinay Sajip 30 Apr '13

30 Apr '13

I've just released version 0.1.2 of distlib on PyPI. The changes are as follows: compat.py: Added BaseConfigurator backport for 2.6. database.py: Return RECORD path from write_installed_files (or None if dry_run). Explicitly return None from write_shared_locations if dry run. metadata.py: Added missing condition in todict(). scripts.py: Add variants and clobber flag for generation of foo/fooX/foo-X.Y. Added .exe manifests for Windows. util.py: Regularised recording of written files. Added Configurator. version.py: Tidyups, most suggested by Donald Stufft: Made key functions private, removed _Common class, removed checking for huge version numbers, made UnsupportedVersionError a ValueError. wheel.py: Replaced absolute import with relative. Handle None return from write_shared_locations correctly. Fixed bug in Mounter for extension modules not in sub-packages. Made dylib-cache Python version-specific. docs: Numerous documentation updates, not detailed further here. tests: Numerous test refinements, not detailed further here. other: Corrected setup.py to ensure that sysconfig.cfg is included. The omission of sysconfig.cfg from the source distribution caused failures on 2.6. Please try it out! Regards, Vinay Sajip

1 0

[issue148] setup_requires dependency removal causes installation failure
by Ian Wienand 30 Apr '13

30 Apr '13

New submission from Ian Wienand: I noticed the following issue installing the keyring package [1] --- Downloading/unpacking keyring Downloading keyring-1.2.2.zip (79Kb): 79Kb downloaded Running setup.py egg_info for package keyring zip_safe flag not set; analyzing archive contents... Installed /tmp/easy_install-u0ESXQ/pytest-runner-1.2/hgtools-2.0.3-py2.6.egg Installed /root/build/keyring/pytest_runner-1.2-py2.6.egg Traceback (most recent call last): File "<string>", line 14, in <module> File "/root/build/keyring/setup.py", line 114, in <module> setup_mod.setup(**setup_params) File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "<string>", line 12, in replacement_run File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 254, in find_sources mm.run() File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 308, in run self.add_defaults() File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 335, in add_defaults rcfiles = list(walk_revctrl()) File "/usr/lib/python2.6/site-packages/setuptools/command/sdist.py", line 46, in walk_revctrl for item in ep.load()(dirname): File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) ImportError: No module named hgtools.plugins Complete output from command python setup.py egg_info: zip_safe flag not set; analyzing archive contents... --- The dependency chain works out to be keyring -> pytest-runner -> hgtools hgtools is a "setup_requires" dependency for pytest-runner. This means the following code is run: /usr/lib/python2.6/site-packages/setuptools/dist.py --- class Distribution(_Distribution): def __init__(): ... if attrs and 'setup_requires' in attrs: self.fetch_build_eggs(attrs.pop('setup_requires')) def fetch_build_eggs(self, requires): """Resolve pre-setup requirements""" from pkg_resources import working_set, parse_requirements for dist in working_set.resolve( parse_requirements(requires), installer=self.fetch_build_egg ): working_set.add(dist) --- That goes fine, hgtools gets installed into whatever /tmp directory, is added to sys.path and pytest-runner is happy. However, the temporary hgtools has registered some entry points with its distribution object in working_set. It then disappears but doesn't remove itself. Later on in sdist.py we have --- def walk_revctrl(dirname=''): """Find all files under revision control""" for ep in pkg_resources.iter_entry_points('setuptools.file_finders'): for item in ep.load()(dirname): yield item ---- pkg_resources.iter_entry_points walks all the distribution objects in working_set, finds hgtools' entry points for this "setuptools.file_finders" stuff, tries to call it and a backtrace ensues because that temporary directory with hgtools has now gone. This also explains why it works when run the second time around; pytest-runner is already there, so we don't need to look at its setup_requires and the problematic distribution doesn't get added. Somehow or other, hgtools needs to get removed from the working_set in dist.py before it is removed? [1] https://bugzilla.redhat.com/show_bug.cgi?id=924038 ---------- messages: 709 nosy: iwienand priority: bug status: unread title: setup_requires dependency removal causes installation failure _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue148> _______________________________________________

1 0

Are chained comparisons allowed in environment markers?
by PJ Eby 28 Apr '13

28 Apr '13

I notice that distlib appears to support chaining of comparisons, but the PEP does not. ISTM that there is no sane use case for chaining, given the nature of comparisons that are allowed. (That is, if you perform one equality or contains comparison between two values, then you know how the next comparison would turn out.) If chained comparisons are disallowed in the PEP, then distlib probably shouldn't be allowing them either, to prevent interop issues w/tools that only support what's in the PEP.

4 16