[New-bugs-announce] [issue23102] distutils: tip-toe around quirks owing to setuptools monkey-patching Extension

[Greg Turner]

New submission from Greg Turner:

Clinical Presentation:
Sometimes a developer consuming distutils will write what seems like a perfectly sensible setup.py, but something inscrutable happens: setup.py bombs out claiming "error: each element of 'ext_modules' option must be an Extension instance or 2-tuple".

Prognosis:
Once she manages to capture this error in pdb (takes some doing), intrepid developer may discover that, bizarrely, it IS an Extension instance, yet, somehow, (not isinstance(that_thing, Extension)) == True.  Sooner or later she'll likely throw up her hands and follow some far-seeing dude's advice on StackOverflow([1]), which will probably work.

If she undertakes a more thorough investigation, she may figure out the true etiology (see below), at which point, various minor tweaks will likely present themselves to her as obvious situation-specific solutions to the problem.

Etiology:
Developer likely employed code early in her module like:

  from distutils.extension import Extension

  .
  . (some other imports)
  .

  setup(..., ext_modules = [
      Extension(...), 
      ...,
  ], ...)

What happened was that setuptools got imported by (presumably) some third-party code, at which point, setuptools monkey-patched distutils.extension.Extension(*), as is setuptools' SOP.

However, in setup.py, a reference to the un-monkey-patched Extension was already saved off as __main__.Extension (along with, in all probability, other un-patched distutils things, as folks tend to consistently use one style or another of import).  So __main__ calls (an un-monkey-patched version of) distutils.core.setup, which ultimately iterates through the list of Extensions, checking isinstance(item, Extension), or so, which, as can now be seen, is not going to be true.

So, the error message is correct, it just fails to mention the possibility that there are multiple things called "Extension" floating around with identical repr's.

Epidemiological Outlook:
Seemingly this is a rare condition, but when a case develops, it can be costly, due to the likelihood of misdiagnosis and/or partial remission and relapse.

One possible vaccine has been developed and is enclosed.  It has not been subjected to clinical trial, nor peer-review (until now).  It is enclosed as a patch which applies to python 2.7-3.5 and seems to do the trick in the particular case that was buggin' me (wish I could say it will do the trick for any non-broken use-case, but I can't, as if I made such a claim, I'd clearly jinx it).

--
* Arguably, this is a bug or misfeature of setuptools, as here setuptools appears to too liberally assume that, if its modules were even casually imported, then it's a good time to monkey-patch distutils.  However, IME, fixing this putative bug, threatens to be a non-trivial undertaking and cause a bunch of regressions and contingent hassles.

Background URLS:

[1] http://stackoverflow.com/questions/21594925/error-each-element-of-ext-modules-option-must-be-an-extension-instance-or-2-t

https://bitbucket.org/pypa/setuptools/issue/309

https://bugs.gentoo.org/show_bug.cgi?id=532708

----------
components: Distutils
files: distutils_accomodate_extension_ducktypes.patch
keywords: patch
messages: 233034
nosy: dstufft, eric.araujo, gmt
priority: normal
severity: normal
status: open
title: distutils: tip-toe around quirks owing to setuptools monkey-patching Extension
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file37529/distutils_accomodate_extension_ducktypes.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23102>
_______________________________________