Mailman 3 September 2016 - Distutils-SIG

[proposal] module dependency specification and overriding for packages
by Ronny Pfannschmidt Sept. 11, 2016

Sept. 11, 2016

Hi, while trying to vendor packages that depend on six, i noticed a problematic implication it is impossible to do this without patching those modules, as such adding a fragile maintenance burden. in order to elevate that i propose creating a mechanism that allows to physically or virtually map sets of interdependent libraries (that are prepared for it into a sub-package of the package vendor-ing it this would elevate a few issues a) removing the need to patch b) making de-vendor-ing of exact versions trivial (just unpacked wheels and adding it to the sub-packages __path__) c) easing updates, as the vendor-ing turns into a process of fetching the wheels and unpacking them into the package folder i do have some implementation ideas, but i would like to hear the opinions of others before proposing an exact implementation -- Ronny

4 6

Improved Stats Features in Python
by Rui Sarmento Sept. 10, 2016

Sept. 10, 2016

Dear Sirs, I've implemented some stats functions related to Factor Analysis in the statistics area. Specifically, the KMO test and the Bartlett test also. At this time I do not seem to find any module performing these tests. Is there any chance I could add these functions to a package in Python. What is the procedure to perform such contribution. Thank you very much in advance for the suggestions. Best Regards, Rui

3 8

Pep 440 question/clarification Version specifiers no "Or" available
by Matthias Bussonnier Sept. 8, 2016

Sept. 8, 2016

Hi all, In the line of recent discussions[1] about releasing some packages only for some versions of Python. I was reading pep 440 and in particular the "Version specifier section". The following sentence caught my eye: > The comma (",") is equivalent to a logical and operator: a candidate version must match all given version clauses in order to match the specifier as a whole. Is there no way to have an "or" ? It seem to me that it might be useful for package wanting to express compatibility with `2.6`, `2.7` or `>3.3` for example, in the form `=~2.6 <or operator> =~3.3`. I completely agree that the use case is _limited_ and likely rare, I was just wondering if I'm missing something, if it was an oversight and if not if adding a section detailing that "or" was not considered, or that "or" was explicitly not included for X,Y,Z reason would be a good thing. Thanks, -- Matthias [1]:https://mail.python.org/pipermail/distutils-sig/2016-August/029604.html

4 5

Fwd: What is the official position on distutils?
by Sylvain Corlay Sept. 5, 2016

Sept. 5, 2016

Hi Nick (re-sending to the list) On Sun, Sep 4, 2016 at 5:16 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > It's important to note that one of our key goals in PyPA is to get to > a point where the API of a developer's publishing toolchain of choice > only depends on the *version of the publishing tools* they're using, > and not on the specific version of Python (as long as they're using a > supported version). Encouraging the expansion and increased use of the > distutils API runs counter to that goal in most cases (since it > encourages writing publication and distribution code that will only > work with newer versions of Python), but may sometimes make sense if > it's info that really is CPython centric, or if it's a feature > primarily aimed at tightly controlled environments that can > aggressively drop support for older versions. Today's state of things is that any project that is not a trivial pure-python module often has use of a combination of all three libraries: distutils, setuptools and pip in their setup.py and in a convoluted way. Newcomers to the toolchain look in similar projects for the right incantations that will fulfill their needs. We are pretty far from being in a state where good practices can even be defined. As a package author, I generally make sure that my need is not fulfilled by a proper usage of the standard library before adding a dependency to a third-party library, but packaging is an area where this does not work. Requiring pip only to know where distutils puts assets is pretty bad. Interestingly, on the subject of building extensions (where the has_flag method discussion applies), distutils.ccompiler is the only available tool and happens to not be monkey-patched by setuptools at the moment. Although setuptools re-defines buildext (and tries to get a version from cython first!) So it seems that there no real "right place" within setuptools to monkey-patch distutils' ccompiler. Sylvain

2 1

What is the official position on distutils?
by Brett Cannon Sept. 4, 2016

Sept. 4, 2016

The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus Python -- on changes to distutils, its future, etc. Is there an official position I'm not aware of? If not, could we get one so we know how to handle any more proposed changes to distutils? For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something doesn't service that explicit goal then it shouldn't go into distutils. Now that we are moving down the road towards making the build step configurable I'm fine with saying that distutils is not expected to work for people other than Python itself and others can use setuptools, enscons, or other projects while we continue to improve the situation to where the build system is just something you specify in pyproject.toml to generate your wheel. IOW distutils is only exposed publicly because Python doesn't hide anything, but making it useful for the general case of direct usage by people is a non-goal of the package.

11 22

Re: [Distutils] What is the official position on distutils?
by Sylvain Corlay Sept. 4, 2016

Sept. 4, 2016

Hi Steve, - I posted an on setuptools github https://github.com/pypa/ setuptools/issues/771 to ask the setuptools folks (and especially Jason) whether they would want such a feature. I still find it baffling that the preferred way to add distutils features would be to monkey-patch it via setuptools first and then to add it into distutils later, while my proposed approach was to add it to distutils and monkey patch for earlier versions of python. - Regarding the policy that you just nominated, does this apply to things that have been in pip for a long time? In the initial email that triggered this discussion, it was also question of adding the `pip.location.distutils_scheme` convenience function to distutils itself. Cf https://bugs.python.org/issue26955 and pip's implementation at https://github.com/pypa/pip/blob/281eb61b09d87765d7c2b92f6982b3fe76ccb0af/p… Indeed, the problem with certain distutils commands like `install_headers` is that the only way to retrieve the directory where things were installed is to instantiate a `distutils.dist.Distribution` object, which is not trivial to do. pip's distutils_scheme convenience function does it. This function is *about* distutils, and IMO should really be *in* distutils. Thanks, Sylvain On Sun, Sep 4, 2016 at 4:02 PM, Steve Dower <steve.dower(a)python.org> wrote: > "add it to setuptools first, and then add things to distutils where we > feel they're sufficiently stable to not need the benefit of the faster > setuptools update cycle" > > I nominate this as the official policy on API changes for distutils, with > regular maintenance mode policies applying to anything else. > > Top-posted from my Windows Phone > ------------------------------ > From: Nick Coghlan <ncoghlan(a)gmail.com> > Sent: ‎9/‎4/‎2016 2:19 > To: Sylvain Corlay <sylvain.corlay(a)gmail.com> > Cc: distutils sig <distutils-sig(a)python.org> > Subject: Re: [Distutils] What is the official position on distutils? > > On 4 September 2016 at 06:44, Sylvain Corlay <sylvain.corlay(a)gmail.com> > wrote: > > Hi Brett, > > > > On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon <brett(a)python.org> wrote: > >> > >> > >> If Jason is up for the responsibility that seems like a reasonable > >> approach to take. It also helps test out features in setuptools first > before > >> upstreaming it. > >> > > > > How do you see `has_flag` get into setuptools? By monkey-patching > distutils' > > ccompiler to have it aside of `has_function` when setuptools is imported? > > > > I find really weird the idea of adding this in a convoluted fashion > instead > > of allowing incremental improvement of distutils. > > The change to distutils would still be a plain patch to distutils, it > would just be accepted at the API design level in setuptools first. > > The problem you're running into right now isn't a technical one - it's > that there isn't anyone that currently feels like they have sufficient > design authority over the distutils API to accept your proposal, hence > Brett starting this thread to address that underlying recurring > question, rather than the specifics of your change. > > Jason *definitely* has that design authority over setuptools though, > and will be tasked with making any API additions available on older > versions of Python via setuptools regardless of what policy we adopt > for distutils maintenance, so if he's amenable to the idea, it makes > sense to me to invert the order we ask those questions: add it to > setuptools first, and then add things to distutils where we feel > they're sufficiently stable to not need the benefit of the faster > setuptools update cycle. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig >

2 1

setup_requires: the obvious option(?)
by Antony Lee Sept. 2, 2016

Sept. 2, 2016

Hi all, The `setup_requires` option to `setup()` is well-known to suffer from multiple issues. Most importantly, as it is a keyword argument to `setup()`, it appears too late for modules that may need to be imported for the build to occur (e.g., Cython, for which support must explicitly provided by setuptools itself rather than by letting Cython hook into it); additionally, there are various contorsions that people go to to avoid some `setup_requires` when not building the package (such as checking the value of `sys.argv`). `setup_requires` also uses `easy_install` rather than `pip`, but I do not see why this could not be fixed; let's focus on the first issue instead. If `setup_requires` appears too late to be useful, the obvious(?) option is to move it earlier: provide a function, say, `setuptools.setup_requires()`, that should be called *before* `setup()` itself, e.g.: from setuptools import setup, setup_requires setup_requires("numpy", needed_for=["build_ext"]) try: import numpy as np except ImportError: np = None setup(..., include_dirs=[np.get_include()] if np else []) When `setup.py` is invoked, either directly or by pip, upon the call to `setup_requires()`, if `sys.argv[0]` is in the `needed_for` kwarg, and at least one requirement is missing, `setup_requires()` calls asks pip to install the required packages (similarly to ` https://bitbucket.org/dholth/setup-requires`) in a temporary directory, and the whole Python process replaces itself (in the `os.execv()` sense) by a new call to `python setup.py` with this temporary directory prepended to the PYTHONPATH. In this new process, the arguments to `setup_requires()` are now available and we can proceed to the rest of `setup.py`. I feel like this idea is natural enough that someone must already have come up with it... but I may be missing something :-) Best, Antony

6 10

Accepting PEP 527 (reducing the variety of file types supported by PyPI)
by Nick Coghlan Sept. 2, 2016

Sept. 2, 2016

Having reviewed the comments on the PEP 527 thread and the latest draft of the PEP, I'm now accepting the PEP. This means that: * supported sdist types will be reduced to .tar.gz and .zip * projects will need to choose one or the other for future releases * supported binary formats will be reduced to bdist_wheel and bdist_egg The problem that Maurits van Rees pointed out with ez_setup.py currently still specifically looking for a zip file and processing it accordingly can be resolved by setuptools settling on zip as its sdist publication format (unless/until ez_setup.py is changed to handle tar.gz). Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1