From brett at python.org  Sat Mar  1 17:04:17 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 1 Mar 2014 11:04:17 -0500
Subject: [Distutils] Is there any sense to put setuptools as a
	requirement?
In-Reply-To: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>
References: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>
Message-ID: <CAP1=2W5nPcWOPQ2qmV6dmnU1rOFgNtSx1m54x1KoeTX8jDKFaA@mail.gmail.com>

On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost <
p at google-groups-2014.dobrogost.net> wrote:

> Hi!
>
> I've seen people putting 'setuptools' in 'install_requires' in
> setup.py starting with import from setuptools like this:
> from setuptools import setup, find_packages
>
> Does it make any sense?
> In what circumstances should 'setuptools' be listed in
> 'install_requires' or 'setup_requires'?
>

Someone can correct me if I'm wrong, but you can list setuptools in
setup.cfg as part of setup_requires and then tools that read setup.cfg
instead of/prior to executing setup.py can know they need setuptools to
properly execute setup.py.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140301/27dceaed/attachment.html>

From jim at zope.com  Sat Mar  1 22:47:12 2014
From: jim at zope.com (Jim Fulton)
Date: Sat, 1 Mar 2014 16:47:12 -0500
Subject: [Distutils] Is there any sense to put setuptools as a
	requirement?
In-Reply-To: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>
References: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>
Message-ID: <CAPDm-FiUCqPjt4oiGOPri44a7VX5gX+AOR_JdQ6xkB5bZ6khTg@mail.gmail.com>

On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost
<p at google-groups-2014.dobrogost.net> wrote:
> Hi!
>
> I've seen people putting 'setuptools' in 'install_requires' in
> setup.py starting with import from setuptools like this:
> from setuptools import setup, find_packages
>
> Does it make any sense?
> In what circumstances should 'setuptools' be listed in
> 'install_requires' or 'setup_requires'?

It makes sense when you use setuptools to implement namespace packages.

So, for example in zope packages, the __init__.py in the zope namespace
package typically has:

    import pkg_resources
    pkg_resources.declare_namespace(__name__)

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From pje at telecommunity.com  Sun Mar  2 03:49:21 2014
From: pje at telecommunity.com (PJ Eby)
Date: Sat, 1 Mar 2014 21:49:21 -0500
Subject: [Distutils] Is there any sense to put setuptools as a
	requirement?
In-Reply-To: <CAPDm-FiUCqPjt4oiGOPri44a7VX5gX+AOR_JdQ6xkB5bZ6khTg@mail.gmail.com>
References: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>
 <CAPDm-FiUCqPjt4oiGOPri44a7VX5gX+AOR_JdQ6xkB5bZ6khTg@mail.gmail.com>
Message-ID: <CALeMXf6ZK5K4hQbMdD5BvvrxcLSe6PHj_o-LE5S45Ln-adaQ_g@mail.gmail.com>

On Sat, Mar 1, 2014 at 4:47 PM, Jim Fulton <jim at zope.com> wrote:

> On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost
> <p at google-groups-2014.dobrogost.net> wrote:
> > Hi!
> >
> > I've seen people putting 'setuptools' in 'install_requires' in
> > setup.py starting with import from setuptools like this:
> > from setuptools import setup, find_packages
> >
> > Does it make any sense?
> > In what circumstances should 'setuptools' be listed in
> > 'install_requires' or 'setup_requires'?
>
> It makes sense when you use setuptools to implement namespace packages.
>
> So, for example in zope packages, the __init__.py in the zope namespace
> package typically has:
>
>     import pkg_resources
>     pkg_resources.declare_namespace(__name__)
>

It also makes sense if you use entry points, the plugin finder, or any
other pkg_resources features, for that matter.  ;-)  Or if you reuse
setuptools functionality (e.g. easy_install) to install plugins for your
app.

But in general, it only makes sense to depend on setuptools if your actual
project contents (aside from setup.py) are importing things from
pkg_resources or setuptools.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140301/838f0594/attachment-0001.html>

From qwcode at gmail.com  Sun Mar  2 04:28:05 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sat, 1 Mar 2014 19:28:05 -0800
Subject: [Distutils] Is there any sense to put setuptools as a
	requirement?
In-Reply-To: <CALeMXf6ZK5K4hQbMdD5BvvrxcLSe6PHj_o-LE5S45Ln-adaQ_g@mail.gmail.com>
References: <CAA6tFMthwY2mN2PhUVOe+VMeN+6smy2GL8fh0=n6JjSzOk+T=A@mail.gmail.com>
 <CAPDm-FiUCqPjt4oiGOPri44a7VX5gX+AOR_JdQ6xkB5bZ6khTg@mail.gmail.com>
 <CALeMXf6ZK5K4hQbMdD5BvvrxcLSe6PHj_o-LE5S45Ln-adaQ_g@mail.gmail.com>
Message-ID: <CAPYWazo=FXOreaQkAQPZPqL61AQL2XwzUY1Rz8fKv6QOQUVHRQ@mail.gmail.com>

with pip 1.5 able to install from wheel without setuptools, it's more
critical now to declare the dependency (in the cases PJ mentions), to
ensure setuptools gets installed.


On Sat, Mar 1, 2014 at 6:49 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Sat, Mar 1, 2014 at 4:47 PM, Jim Fulton <jim at zope.com> wrote:
>
>> On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost
>> <p at google-groups-2014.dobrogost.net> wrote:
>> > Hi!
>> >
>> > I've seen people putting 'setuptools' in 'install_requires' in
>> > setup.py starting with import from setuptools like this:
>> > from setuptools import setup, find_packages
>> >
>> > Does it make any sense?
>> > In what circumstances should 'setuptools' be listed in
>> > 'install_requires' or 'setup_requires'?
>>
>> It makes sense when you use setuptools to implement namespace packages.
>>
>> So, for example in zope packages, the __init__.py in the zope namespace
>> package typically has:
>>
>>     import pkg_resources
>>     pkg_resources.declare_namespace(__name__)
>>
>
> It also makes sense if you use entry points, the plugin finder, or any
> other pkg_resources features, for that matter.  ;-)  Or if you reuse
> setuptools functionality (e.g. easy_install) to install plugins for your
> app.
>
> But in general, it only makes sense to depend on setuptools if your actual
> project contents (aside from setup.py) are importing things from
> pkg_resources or setuptools.
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140301/c1309dd0/attachment.html>

From ncoghlan at gmail.com  Sun Mar  2 06:22:18 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 2 Mar 2014 15:22:18 +1000
Subject: [Distutils] Remaining tasks before metadata 2.0 can be accepted
 (was Re: The PEP 426 defined metadata version will be metadata 3.0)
Message-ID: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>

On 27 February 2014 10:46, Marcus Smith <qwcode at gmail.com> wrote:
> that would be good.  If you did, I would link to the tasks from the PUG
> future page.

OK, these are the things I consider blockers for an accepted metadata 2.0 spec:

https://bitbucket.org/pypa/pypi-metadata-formats/issues?priority=blocker&status=open&status=new

Finalising PEP 426/440/459 is on me. At this point, I think that
consists of *taking away* things that aren't yet settled (specifically
metadata hooks), so we can see how far this next iteration actually
gets us before trying to solve the remaining problems that need some
kind of trigger support.

A required preliminary task is to create a revision of PEP 425 that
expands its scope to also handle the parts of the file/directory
naming scheme that are common across sdist, wheel and the installation
database (with compatibility tags becoming a subsection), as well as
fixing the definition of the compatibility tags to better handle
Windows and Python 2.x binary extensions. (There's a separate
non-blocker issue for better Linux/POSIX support - building from
source is far more common there, and both conda and Linux distro
packages remain available as a near-term workaround for the lack of
upstream binary packages)

The other blockers are then sdist 2.0, wheel 1.1 and a second revision
of the installation database format.

There are more issues in that repo, but they're ones that I don't
consider *essential* as part of a usable metadata 2.0 spec - they're
about things like Linux binary support, distributing full applications
in addition to libraries and handling things that may require the
ability to run code at installation time. Once metadata 2.0 itself is
published, we can likely explore several of them as extensions before
committing to anything in the core PEPs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Mar  2 06:41:41 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 2 Mar 2014 15:41:41 +1000
Subject: [Distutils] Remaining tasks before metadata 2.0 can be accepted
 (was Re: The PEP 426 defined metadata version will be metadata 3.0)
In-Reply-To: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
References: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
Message-ID: <CADiSq7cgLa+JKqXdPV2ifLfyDSuM6ONrGSWV1b1Q+gh7nyUs6w@mail.gmail.com>

On 2 March 2014 15:22, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 February 2014 10:46, Marcus Smith <qwcode at gmail.com> wrote:
>> that would be good.  If you did, I would link to the tasks from the PUG
>> future page.
>
> OK, these are the things I consider blockers for an accepted metadata 2.0 spec:
>
> https://bitbucket.org/pypa/pypi-metadata-formats/issues?priority=blocker&status=open&status=new
>
> Finalising PEP 426/440/459 is on me. At this point, I think that
> consists of *taking away* things that aren't yet settled (specifically
> metadata hooks), so we can see how far this next iteration actually
> gets us before trying to solve the remaining problems that need some
> kind of trigger support.
>
> A required preliminary task is to create a revision of PEP 425 that
> expands its scope to also handle the parts of the file/directory
> naming scheme that are common across sdist, wheel and the installation
> database (with compatibility tags becoming a subsection), as well as
> fixing the definition of the compatibility tags to better handle
> Windows and Python 2.x binary extensions. (There's a separate
> non-blocker issue for better Linux/POSIX support - building from
> source is far more common there, and both conda and Linux distro
> packages remain available as a near-term workaround for the lack of
> upstream binary packages)
>
> The other blockers are then sdist 2.0, wheel 1.1 and a second revision
> of the installation database format.

Just remembered two more blockers - updating the JSON schema files to
account for the switch to making heavy use of schema extensions and
rerunning the PyPI compatibility analysis.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Mar  2 08:25:43 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 2 Mar 2014 17:25:43 +1000
Subject: [Distutils] Deferring metadata hooks
Message-ID: <CADiSq7fEPf6pQGQrUm-z_X5zX=ALaFaAC1fLETNGddpRZ1_jVg@mail.gmail.com>

I've just posted updated versions of PEP 426 and 459 that defer the
"metadata hooks" feature. The design and behaviour of that extension
is still way too speculative for me to approve in its current form,
but I also don't want to hold up the rest of the changes in metadata
2.0 while we thrash out the details of a hook system.

That said, I still don't want us to get into a situation where someone
later publishes a wheel file that expects metadata hook support and
older tools silently install it without running the hooks.

Accordingly, the revised PEP 426 adds a single simpler feature to the
extensions system: the idea of a "required extension".

If a project sets that flag for an extension (by including
"required_extension": true in the extension metadata), and an
installation tool doesn't understand it, then the tool is required to
either fail the installation attempt entirely or else fall back to
installing from source.

That way, project authors will be able to distinguish between "these
metadata hooks are just an optimisation, things will still work if you
don't run them" and "if you don't run these hooks, your installation
will be broken".

I think this approach may also encourage a design where projects do
something sensible *by default* (e.g. NumPy defaulting to SSE2) and
then use the (not yet defined) post-installation hooks to potentially
*change away* from the default to something more optimised for that
particular system (e.g. NumPy overwriting itself with an SSE3
version), while still *allowing* developers to refuse to let the
software install if the metadata hooks won't be run.

Regards,
Nick.

P.S. The draft PEP for metadata hooks is still available at
https://bitbucket.org/pypa/pypi-metadata-formats/src/default/metadata-hooks.rst

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From setuptools at bugs.python.org  Sun Mar  2 12:38:32 2014
From: setuptools at bugs.python.org (mrx23dot)
Date: Sun, 02 Mar 2014 11:38:32 +0000
Subject: [Distutils] [issue158] Python not recognized in registry,
	PATH not modified
Message-ID: <1393760312.62.0.323539663384.issue158@psf.upfronthosting.co.za>


New submission from mrx23dot:

setuptools Windows installer doesn't look for HKEY_LOCAL_MACHINE:
only looks in: HKEY_CURRENT_USER for Python:
[HKEY_LOCAL_MACHINE\Software\Python\PythonCore\2.7]
(problem on Win7 x64 Python 2.7 installed for All users)

Also it doesn't add Python\Scripts to PATH.
So cmd> installed_prg_name is not recognized

We have to add manually.

----------
messages: 745
nosy: mrx23dot
priority: critical
status: unread
title: Python not recognized in registry, PATH not modified

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue158>
_______________________________________________

From oscar.j.benjamin at gmail.com  Sun Mar  2 19:33:39 2014
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sun, 2 Mar 2014 18:33:39 +0000
Subject: [Distutils] Deferring metadata hooks
In-Reply-To: <CADiSq7fEPf6pQGQrUm-z_X5zX=ALaFaAC1fLETNGddpRZ1_jVg@mail.gmail.com>
References: <CADiSq7fEPf6pQGQrUm-z_X5zX=ALaFaAC1fLETNGddpRZ1_jVg@mail.gmail.com>
Message-ID: <CAHVvXxRgFnej4srtOy3TJJGWahce0ttGwGwu85pu_KZLSmoD_g@mail.gmail.com>

On 2 March 2014 07:25, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> I've just posted updated versions of PEP 426 and 459 that defer the
> "metadata hooks" feature. The design and behaviour of that extension
> is still way too speculative for me to approve in its current form,
> but I also don't want to hold up the rest of the changes in metadata
> 2.0 while we thrash out the details of a hook system.

The other idea that was discussed a few times but hasn't made it into
PEP 426 is the idea of compatibility tags. These are mentioned in the
(deferred) metabuild section:
http://legacy.python.org/dev/peps/pep-0426/#metabuild-system
but nowhere else in the PEP.

I certainly understand the desire to defer dealing with something as
complex as hooks but simple string compatibility tags are a much
simpler thing to include in the metadata and could be very useful. I'm
thinking of a situation where you can indicate things like ABI
compatibility for C/Fortran compiled code (e.g. libc, gfortran vs g77)
but there could easily be many other uses once wheel takes off.

> That said, I still don't want us to get into a situation where someone
> later publishes a wheel file that expects metadata hook support and
> older tools silently install it without running the hooks.
>
> Accordingly, the revised PEP 426 adds a single simpler feature to the
> extensions system: the idea of a "required extension".
>
> If a project sets that flag for an extension (by including
> "required_extension": true in the extension metadata), and an
> installation tool doesn't understand it, then the tool is required to
> either fail the installation attempt entirely or else fall back to
> installing from source.
>
> That way, project authors will be able to distinguish between "these
> metadata hooks are just an optimisation, things will still work if you
> don't run them" and "if you don't run these hooks, your installation
> will be broken".

Is there some need for metadata extensions to be optional by default?

> I think this approach may also encourage a design where projects do
> something sensible *by default* (e.g. NumPy defaulting to SSE2) and
> then use the (not yet defined) post-installation hooks to potentially
> *change away* from the default to something more optimised for that
> particular system (e.g. NumPy overwriting itself with an SSE3
> version), while still *allowing* developers to refuse to let the
> software install if the metadata hooks won't be run.

I'm not sure but there does seem to be some discussion and movement
toward the idea of numpy distributing openblas binaries (which would
solve the SSE problem). See the threads starting here for more:
http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069186.html
http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069106.html

(Note that shipping openblas binaries does not solve the ABI mismatch
problems that compatibility tags could address).


Oscar

From ncoghlan at gmail.com  Sun Mar  2 22:05:52 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 3 Mar 2014 07:05:52 +1000
Subject: [Distutils] Deferring metadata hooks
In-Reply-To: <CAHVvXxRgFnej4srtOy3TJJGWahce0ttGwGwu85pu_KZLSmoD_g@mail.gmail.com>
References: <CADiSq7fEPf6pQGQrUm-z_X5zX=ALaFaAC1fLETNGddpRZ1_jVg@mail.gmail.com>
 <CAHVvXxRgFnej4srtOy3TJJGWahce0ttGwGwu85pu_KZLSmoD_g@mail.gmail.com>
Message-ID: <CADiSq7fZy=iv3ZyC8jQshUcDrY1q2DLOmshA_xczOVdc9BUyRQ@mail.gmail.com>

On 3 Mar 2014 04:34, "Oscar Benjamin" <oscar.j.benjamin at gmail.com> wrote:
>
> On 2 March 2014 07:25, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > I've just posted updated versions of PEP 426 and 459 that defer the
> > "metadata hooks" feature. The design and behaviour of that extension
> > is still way too speculative for me to approve in its current form,
> > but I also don't want to hold up the rest of the changes in metadata
> > 2.0 while we thrash out the details of a hook system.
>
> The other idea that was discussed a few times but hasn't made it into
> PEP 426 is the idea of compatibility tags. These are mentioned in the
> (deferred) metabuild section:
> http://legacy.python.org/dev/peps/pep-0426/#metabuild-system
> but nowhere else in the PEP.
>
> I certainly understand the desire to defer dealing with something as
> complex as hooks but simple string compatibility tags are a much
> simpler thing to include in the metadata and could be very useful. I'm
> thinking of a situation where you can indicate things like ABI
> compatibility for C/Fortran compiled code (e.g. libc, gfortran vs g77)
> but there could easily be many other uses once wheel takes off.

There's an issue for a "python.expect" extension on the metadata tracker
(name TBD, "python.constraints" would answer work) . I'll list that as a
deferred feature as well, but similar to the metadata hooks, I'd prefer to
give people a chance to try out the baseline version before we lock in
additional extensions.

> > That said, I still don't want us to get into a situation where someone
> > later publishes a wheel file that expects metadata hook support and
> > older tools silently install it without running the hooks.
> >
> > Accordingly, the revised PEP 426 adds a single simpler feature to the
> > extensions system: the idea of a "required extension".
> >
> > If a project sets that flag for an extension (by including
> > "required_extension": true in the extension metadata), and an
> > installation tool doesn't understand it, then the tool is required to
> > either fail the installation attempt entirely or else fall back to
> > installing from source.
> >
> > That way, project authors will be able to distinguish between "these
> > metadata hooks are just an optimisation, things will still work if you
> > don't run them" and "if you don't run these hooks, your installation
> > will be broken".
>
> Is there some need for metadata extensions to be optional by default?

Yes, it's because I expect most extensions to be along the lines of the
standard ones, providing additional info for developers and tools *other
than* the installer. "If you can't process this extension, don't install
the software" should be relatively rare.

> > I think this approach may also encourage a design where projects do
> > something sensible *by default* (e.g. NumPy defaulting to SSE2) and
> > then use the (not yet defined) post-installation hooks to potentially
> > *change away* from the default to something more optimised for that
> > particular system (e.g. NumPy overwriting itself with an SSE3
> > version), while still *allowing* developers to refuse to let the
> > software install if the metadata hooks won't be run.
>
> I'm not sure but there does seem to be some discussion and movement
> toward the idea of numpy distributing openblas binaries (which would
> solve the SSE problem). See the threads starting here for more:
> http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069186.html
> http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069106.html
>
> (Note that shipping openblas binaries does not solve the ABI mismatch
> problems that compatibility tags could address).

So long as NumPy defines and publishes an extension with the relevant
details in its metadata, the metadata constraints extension would
eventually be able to automate consistency checks.

However, I'm starting to think you may be right and it will be worth having
that defined from the beginning, specifically to help ensure we keep the
NumPy dependent wheels on PyPI consistent with each other.

Cheers,
Nick.

>
>
> Oscar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140303/c2db12f4/attachment.html>

From oscar.j.benjamin at gmail.com  Sun Mar  2 22:36:09 2014
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sun, 2 Mar 2014 21:36:09 +0000
Subject: [Distutils] Deferring metadata hooks
In-Reply-To: <CADiSq7fZy=iv3ZyC8jQshUcDrY1q2DLOmshA_xczOVdc9BUyRQ@mail.gmail.com>
References: <CADiSq7fEPf6pQGQrUm-z_X5zX=ALaFaAC1fLETNGddpRZ1_jVg@mail.gmail.com>
 <CAHVvXxRgFnej4srtOy3TJJGWahce0ttGwGwu85pu_KZLSmoD_g@mail.gmail.com>
 <CADiSq7fZy=iv3ZyC8jQshUcDrY1q2DLOmshA_xczOVdc9BUyRQ@mail.gmail.com>
Message-ID: <CAHVvXxQUuiks2WCZfHbtYFisWZ9EUwhWdaOAacuNrsynTMCpDg@mail.gmail.com>

On 2 March 2014 21:05, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >
>> > I think this approach may also encourage a design where projects do
>> > something sensible *by default* (e.g. NumPy defaulting to SSE2) and
>> > then use the (not yet defined) post-installation hooks to potentially
>> > *change away* from the default to something more optimised for that
>> > particular system (e.g. NumPy overwriting itself with an SSE3
>> > version), while still *allowing* developers to refuse to let the
>> > software install if the metadata hooks won't be run.
>>
>> I'm not sure but there does seem to be some discussion and movement
>> toward the idea of numpy distributing openblas binaries (which would
>> solve the SSE problem). See the threads starting here for more:
>> http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069186.html
>> http://mail.scipy.org/pipermail/numpy-discussion/2014-February/069106.html
>>
>> (Note that shipping openblas binaries does not solve the ABI mismatch
>> problems that compatibility tags could address).
>
> So long as NumPy defines and publishes an extension with the relevant
> details in its metadata, the metadata constraints extension would eventually
> be able to automate consistency checks.
>
> However, I'm starting to think you may be right and it will be worth having
> that defined from the beginning, specifically to help ensure we keep the
> NumPy dependent wheels on PyPI consistent with each other.

I expect that those involved in distributing wheels for the scipy
stack would coordinate and quickly converge on a consistent set of
wheels for Windows/OSX on PyPI so I doubt that it would be an issue in
that sense.

Where it is an issue is for people who install different pieces from
different places i.e. mixing source builds, .exe installers, wheels
from PyPI, and perhaps even conda packages and wheels from other
places. If a mechanism was provided to prevent borken installs and
give helpful error messages I'm sure it would be taken advantage of.

If it were also possible to upload and select between multiple
variants of a distribution then that might lower the bar for numpy to
distribute e.g. openblas wheels. A user who was unhappy with openblas
could just as easily install an alternative. (Changing BLAS library is
a big deal for numpy).


Oscar

From qwcode at gmail.com  Mon Mar  3 07:12:50 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 2 Mar 2014 22:12:50 -0800
Subject: [Distutils] Remaining tasks before metadata 2.0 can be accepted
 (was Re: The PEP 426 defined metadata version will be metadata 3.0)
In-Reply-To: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
References: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
Message-ID: <CAPYWazqKsD4oe6VUVg1zeYpMLWuxWQ1JrYXLHZ+xMzp0dqU=eQ@mail.gmail.com>

>
>
> A required preliminary task is to create a revision of PEP 425 that
> expands its scope to also handle the parts of the file/directory
> naming scheme that are common across sdist, wheel and the installation
> database (with compatibility tags becoming a subsection)


can you elaborate a bit.
compatibility tags for built distributions seems like a pretty distinct
topic.
no grokking it combined with "naming scheme that are common across..."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140302/ec9cff49/attachment.html>

From qwcode at gmail.com  Mon Mar  3 07:25:02 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Sun, 2 Mar 2014 22:25:02 -0800
Subject: [Distutils] Remaining tasks before metadata 2.0 can be accepted
 (was Re: The PEP 426 defined metadata version will be metadata 3.0)
In-Reply-To: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
References: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
Message-ID: <CAPYWazpDiJJhzN67nVGp13iV2f8k3taSGf8YHXCS3kWqUj=gtA@mail.gmail.com>

On Sat, Mar 1, 2014 at 9:22 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 27 February 2014 10:46, Marcus Smith <qwcode at gmail.com> wrote:
> > that would be good.  If you did, I would link to the tasks from the PUG
> > future page.
>
> OK, these are the things I consider blockers for an accepted metadata 2.0
> spec:


>
> https://bitbucket.org/pypa/pypi-metadata-formats/issues?priority=blocker&status=open&status=new


ok, I added a number of links these issues on the PUG future page.
http://packaging.python.org/en/latest/future.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140302/35eb826c/attachment.html>

From ncoghlan at gmail.com  Mon Mar  3 07:28:41 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 3 Mar 2014 16:28:41 +1000
Subject: [Distutils] Remaining tasks before metadata 2.0 can be accepted
 (was Re: The PEP 426 defined metadata version will be metadata 3.0)
In-Reply-To: <CAPYWazqKsD4oe6VUVg1zeYpMLWuxWQ1JrYXLHZ+xMzp0dqU=eQ@mail.gmail.com>
References: <CADiSq7eJHcnCNCMoNzwwg63n5yAcOV_GKdJyRKH=MP4CgVVp=A@mail.gmail.com>
 <CAPYWazqKsD4oe6VUVg1zeYpMLWuxWQ1JrYXLHZ+xMzp0dqU=eQ@mail.gmail.com>
Message-ID: <CADiSq7dEqQhyThTNWzL21AC5j5_k3gdx2Ou8wOjmRDkMz55x8g@mail.gmail.com>

On 3 March 2014 16:12, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> A required preliminary task is to create a revision of PEP 425 that
>> expands its scope to also handle the parts of the file/directory
>> naming scheme that are common across sdist, wheel and the installation
>> database (with compatibility tags becoming a subsection)
>
>
> can you elaborate a bit.
> compatibility tags for built distributions seems like a pretty distinct
> topic.
> no grokking it combined with "naming scheme that are common across..."

They *could* be separate PEPs, but the main reason we care about the
compatibility tags is so we can include them in filenames. The design
of the compatibility tagging system is also constrained by the fact
that we want them to be suitable for embedding in the hyphen separated
filename format.

So I think it makes more sense to take the existing compatibility tags
PEP, and the filename definition parts of the wheel spec and combine
them into a single PEP naming scheme PEP that can be referenced from
other PEPs (wheel 1.1+, sdist 2.0+, installation DB 2+).

I see it as similar to the way PEP 440 has a broader scope than its
predecessor, as it covers both version numbers *and* version
specifiers, whereas its predecessor just covered covered the version
numbers.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From qwcode at gmail.com  Tue Mar  4 18:57:50 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Tue, 4 Mar 2014 09:57:50 -0800
Subject: [Distutils] The PEP 426 defined metadata version will be
	metadata 3.0
In-Reply-To: <CAPYWazpkqG89jxu8=fnOrwfk9_VW+KNuPwsOUWjgrOAUy7BnVw@mail.gmail.com>
References: <CADiSq7e1-1BvR8zqnBHVxjR6XJAxP6tWLigSzfcPgcVOmjKPDw@mail.gmail.com>
 <03EBD687-EE7C-4789-AC80-B13920E91C69@stufft.io>
 <CADiSq7dstMbiPM=M4cLn7dW1QNSxH8QAqDPzqv=wbUDHXW8_Vw@mail.gmail.com>
 <9C031EE3-1C31-4E57-B292-98AED58FAC2B@stufft.io>
 <CACac1F8kp-xcAmU=KPowrqOWmtHLyvu-8Ch43QgR73GzykG4_g@mail.gmail.com>
 <5C80EEFB-F177-4052-8196-E0E75B923236@stufft.io>
 <CACac1F80=Of33PZ2me1Akjy7qJgRA+K60cEx8PsVQHEvRVFmpA@mail.gmail.com>
 <CAG8k2+47Uc2_Kk28ipoWAN_V331F2QQTAUVqhfqGp8RdfY99_Q@mail.gmail.com>
 <CAPYWazr=70-We5g4K0C+YGrFCWvbD1NMSV6tHV0t6ah2cZz41g@mail.gmail.com>
 <CAG8k2+4H8a3+nO7nxU1mbwn9t0p=AGnWa=FOZ4i8moYJnGMVRg@mail.gmail.com>
 <CAPYWazrZdme6e5BOo1MvCn4CSRouGW=mYEDqqndUHTjUv8E6EQ@mail.gmail.com>
 <1393488960.8636.YahooMailNeo@web172404.mail.ir2.yahoo.com>
 <CADiSq7do2S+e6MSKJ5-iXeAn5oTuk-JhfhLeVtJff23D8awo4g@mail.gmail.com>
 <CAG8k2+6+konb0ViHeux3JL2PLnODAbGsnnoxeQzx51O=aRxndg@mail.gmail.com>
 <5D13088C-8315-428C-A16A-E7A349D4736D@stufft.io>
 <CAG8k2+6+AfMy2SWFsx2BCZ=YaDa4jYh42aC0Qnjx2mAKDpPtzA@mail.gmail.com>
 <C2A40700-3C14-43D5-B4EA-5DAE1F8C5871@stufft.io>
 <CAG8k2+62_T7TrQHvYC8duB2SPy0D0rNCDT6gkthpa=OSKoGEVQ@mail.gmail.com>
 <CAPYWazpkqG89jxu8=fnOrwfk9_VW+KNuPwsOUWjgrOAUy7BnVw@mail.gmail.com>
Message-ID: <CAPYWazp4Y1QtQ-+E4gNEUHXKLJ_h5C-f2DTNHNp8zDok1og6Rg@mail.gmail.com>

On Thu, Feb 27, 2014 at 9:41 PM, Marcus Smith <qwcode at gmail.com> wrote:

>
>
> On Thu, Feb 27, 2014 at 6:24 AM, Daniel Holth <dholth at gmail.com> wrote:
>
>> No PEP describes how METADATA supports setuptools' extras.
>>
>
> i.e. "Provides-Extras".   maybe we should at least add a footnote to
> PEP-345, or *something*, so this is not a mystery down the road where this
> came from, and how it ended being in "production" use for wheels
>
>

https://bitbucket.org/pypa/pypi-metadata-formats/issue/29/document-provides-extra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140304/4533ddc6/attachment.html>

From donald at stufft.io  Tue Mar  4 20:48:20 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 4 Mar 2014 14:48:20 -0500
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
Message-ID: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>

Hello! I?d like to propose PEP464, the removal of the PyPI Mirror Authenticity API which was originally described in PEP381.

The text of the PEP is below, or it can be viewed online at https://python.org/dev/peps/pep-0464/

PEP: 464
Title: Removal of the PyPI Mirror Authenticity API
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <donald at stufft.io>
BDFL-Delegate: Richard Jones <richard at python.org>
Discussions-To: distutils-sig at python.org
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 02-Mar-2014
Post-History: 03-Mar-2014
Replaces: 381


Abstract
========

This PEP proposes the deprecation and removal of the PyPI Mirror Authenticity
API, this includes the /serverkey URL and all of the URLs under /serversig.


Rationale
=========

The PyPI mirroring infrastructure (defined in PEP 381) provides a means to
mirror the content of PyPI used by the automatic installers, and as a component
of that, it provides a method for verifying the authenticity of the mirrored
content.

This PEP proposal the removal of this API due to:

* No known implementations that utilize this API are known, this includes
  `pip <http://www.pip-installer.org/en/latest/>`_ and
  `setuptools <http://pythonhosted.org//setuptools/>`_.
* Because this API uses DSA it is vulnerable to leaking the private key if
  there is *any* bias in the random nonce.
* This API solves one small corner of the trust problem, however the problem
  itself is much larger and it would be better to have a fully fledged system,
  such as `The Update Framework <https://python.org/dev/peps/pep-0458/>`_,
  instead.

Due to the issues it has and the lack of use it is the opinion of this PEP
that it does not provide any practical benefit to justify the additional
complexity.


Plan for Deprecation & Removal
==============================

Immediately upon the acceptance of this PEP the Mirror Authenticity API will
be considered deprecated and mirroring agents and installation tools should
stop accessing it.

Instead of actually removing it from the current code base (PyPI 1.0) the
current work to replace PyPI 1.0 with a new code base (PyPI 2.0) will simply
not implement this API. This would cause the API to be "removed" when the
switch from 1.0 to 2.0 occurs.

If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014 then
this PEP will be implemented in the PyPI 1.0 code base instead (by removing
the associated code).

No changes will be required in the installers, however PEP 381 compliant
mirroring clients, such as
`bandersnatch <https://pypi.python.org/pypi/bandersnatch/>`_ and
`pep381client <https://pypi.python.org/pypi/pep381client/>`_ will need to be
updated to no longer attempt to mirror the /serversig URLs.


Copyright
=========

This document has been placed in the public domain.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140304/683d5786/attachment.sig>

From donald at stufft.io  Wed Mar  5 01:41:38 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 4 Mar 2014 19:41:38 -0500
Subject: [Distutils] Warehouse XMLRPC
Message-ID: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>

Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
Warehouse today. If you have an application that depends on XMLRPC on PyPI and
you?d like to test it, please try against the url:

    https://pypi-preview.a.ssl.fastly.net/pypi

Let me know if you find any issues, or you can file a bug at:

    https://github.com/pypa/warehouse

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140304/78cdf47d/attachment.sig>

From donald at stufft.io  Wed Mar  5 13:01:49 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 5 Mar 2014 07:01:49 -0500
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
Message-ID: <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>

On Mar 5, 2014, at 5:48 AM, anatoly techtonik <techtonik at gmail.com> wrote:

> JSON-RPC is a better choice for recommended external API,
> because it doesn't rely on memory hungry and potentially unsafe
> XML libraries.
> 
> On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft <donald at stufft.io> wrote:
>> Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
>> Warehouse today. If you have an application that depends on XMLRPC on PyPI and
>> you?d like to test it, please try against the url:
>> 
>>    https://pypi-preview.a.ssl.fastly.net/pypi
>> 
>> Let me know if you find any issues, or you can file a bug at:
>> 
>>    https://github.com/pypa/warehouse
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>> 
> 
> 
> 
> -- 
> anatoly t.

XMLRPC is used in order to maintain compatibility with what is already there. Something
better will replace it eventually and XMLRPC will be deprecated.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140305/ab01c609/attachment.sig>

From techtonik at gmail.com  Wed Mar  5 11:48:22 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 5 Mar 2014 13:48:22 +0300
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
Message-ID: <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>

JSON-RPC is a better choice for recommended external API,
because it doesn't rely on memory hungry and potentially unsafe
XML libraries.

On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft <donald at stufft.io> wrote:
> Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
> Warehouse today. If you have an application that depends on XMLRPC on PyPI and
> you?d like to test it, please try against the url:
>
>     https://pypi-preview.a.ssl.fastly.net/pypi
>
> Let me know if you find any issues, or you can file a bug at:
>
>     https://github.com/pypa/warehouse
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
anatoly t.

From donald at stufft.io  Thu Mar  6 01:31:04 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 5 Mar 2014 19:31:04 -0500
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
Message-ID: <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>

Just a ping on this :) I?m assuming nobody actually cares because it?s an unused API
but since it was introduced through a PEP I wanted to remove it through a PEP.

On Mar 4, 2014, at 2:48 PM, Donald Stufft <donald at stufft.io> wrote:

> Hello! I?d like to propose PEP464, the removal of the PyPI Mirror Authenticity API which was originally described in PEP381.
> 
> The text of the PEP is below, or it can be viewed online at https://python.org/dev/peps/pep-0464/
> 
> PEP: 464
> Title: Removal of the PyPI Mirror Authenticity API
> Version: $Revision$
> Last-Modified: $Date$
> Author: Donald Stufft <donald at stufft.io>
> BDFL-Delegate: Richard Jones <richard at python.org>
> Discussions-To: distutils-sig at python.org
> Status: Draft
> Type: Process
> Content-Type: text/x-rst
> Created: 02-Mar-2014
> Post-History: 03-Mar-2014
> Replaces: 381
> 
> 
> Abstract
> ========
> 
> This PEP proposes the deprecation and removal of the PyPI Mirror Authenticity
> API, this includes the /serverkey URL and all of the URLs under /serversig.
> 
> 
> Rationale
> =========
> 
> The PyPI mirroring infrastructure (defined in PEP 381) provides a means to
> mirror the content of PyPI used by the automatic installers, and as a component
> of that, it provides a method for verifying the authenticity of the mirrored
> content.
> 
> This PEP proposal the removal of this API due to:
> 
> * No known implementations that utilize this API are known, this includes
>  `pip <http://www.pip-installer.org/en/latest/>`_ and
>  `setuptools <http://pythonhosted.org//setuptools/>`_.
> * Because this API uses DSA it is vulnerable to leaking the private key if
>  there is *any* bias in the random nonce.
> * This API solves one small corner of the trust problem, however the problem
>  itself is much larger and it would be better to have a fully fledged system,
>  such as `The Update Framework <https://python.org/dev/peps/pep-0458/>`_,
>  instead.
> 
> Due to the issues it has and the lack of use it is the opinion of this PEP
> that it does not provide any practical benefit to justify the additional
> complexity.
> 
> 
> Plan for Deprecation & Removal
> ==============================
> 
> Immediately upon the acceptance of this PEP the Mirror Authenticity API will
> be considered deprecated and mirroring agents and installation tools should
> stop accessing it.
> 
> Instead of actually removing it from the current code base (PyPI 1.0) the
> current work to replace PyPI 1.0 with a new code base (PyPI 2.0) will simply
> not implement this API. This would cause the API to be "removed" when the
> switch from 1.0 to 2.0 occurs.
> 
> If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014 then
> this PEP will be implemented in the PyPI 1.0 code base instead (by removing
> the associated code).
> 
> No changes will be required in the installers, however PEP 381 compliant
> mirroring clients, such as
> `bandersnatch <https://pypi.python.org/pypi/bandersnatch/>`_ and
> `pep381client <https://pypi.python.org/pypi/pep381client/>`_ will need to be
> updated to no longer attempt to mirror the /serversig URLs.
> 
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140305/a07a67fb/attachment.sig>

From donald at stufft.io  Thu Mar  6 01:43:53 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 5 Mar 2014 19:43:53 -0500
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
Message-ID: <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>

Also want to make sure the original authors of PEP381 and the mirroring clients
are aware of this PEP!

On Mar 5, 2014, at 7:31 PM, Donald Stufft <donald at stufft.io> wrote:

> Just a ping on this :) I?m assuming nobody actually cares because it?s an unused API
> but since it was introduced through a PEP I wanted to remove it through a PEP.
> 
> On Mar 4, 2014, at 2:48 PM, Donald Stufft <donald at stufft.io> wrote:
> 
>> Hello! I?d like to propose PEP464, the removal of the PyPI Mirror Authenticity API which was originally described in PEP381.
>> 
>> The text of the PEP is below, or it can be viewed online at https://python.org/dev/peps/pep-0464/
>> 
>> PEP: 464
>> Title: Removal of the PyPI Mirror Authenticity API
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Donald Stufft <donald at stufft.io>
>> BDFL-Delegate: Richard Jones <richard at python.org>
>> Discussions-To: distutils-sig at python.org
>> Status: Draft
>> Type: Process
>> Content-Type: text/x-rst
>> Created: 02-Mar-2014
>> Post-History: 03-Mar-2014
>> Replaces: 381
>> 
>> 
>> Abstract
>> ========
>> 
>> This PEP proposes the deprecation and removal of the PyPI Mirror Authenticity
>> API, this includes the /serverkey URL and all of the URLs under /serversig.
>> 
>> 
>> Rationale
>> =========
>> 
>> The PyPI mirroring infrastructure (defined in PEP 381) provides a means to
>> mirror the content of PyPI used by the automatic installers, and as a component
>> of that, it provides a method for verifying the authenticity of the mirrored
>> content.
>> 
>> This PEP proposal the removal of this API due to:
>> 
>> * No known implementations that utilize this API are known, this includes
>> `pip <http://www.pip-installer.org/en/latest/>`_ and
>> `setuptools <http://pythonhosted.org//setuptools/>`_.
>> * Because this API uses DSA it is vulnerable to leaking the private key if
>> there is *any* bias in the random nonce.
>> * This API solves one small corner of the trust problem, however the problem
>> itself is much larger and it would be better to have a fully fledged system,
>> such as `The Update Framework <https://python.org/dev/peps/pep-0458/>`_,
>> instead.
>> 
>> Due to the issues it has and the lack of use it is the opinion of this PEP
>> that it does not provide any practical benefit to justify the additional
>> complexity.
>> 
>> 
>> Plan for Deprecation & Removal
>> ==============================
>> 
>> Immediately upon the acceptance of this PEP the Mirror Authenticity API will
>> be considered deprecated and mirroring agents and installation tools should
>> stop accessing it.
>> 
>> Instead of actually removing it from the current code base (PyPI 1.0) the
>> current work to replace PyPI 1.0 with a new code base (PyPI 2.0) will simply
>> not implement this API. This would cause the API to be "removed" when the
>> switch from 1.0 to 2.0 occurs.
>> 
>> If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014 then
>> this PEP will be implemented in the PyPI 1.0 code base instead (by removing
>> the associated code).
>> 
>> No changes will be required in the installers, however PEP 381 compliant
>> mirroring clients, such as
>> `bandersnatch <https://pypi.python.org/pypi/bandersnatch/>`_ and
>> `pep381client <https://pypi.python.org/pypi/pep381client/>`_ will need to be
>> updated to no longer attempt to mirror the /serversig URLs.
>> 
>> 
>> Copyright
>> =========
>> 
>> This document has been placed in the public domain.
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140305/d11310c1/attachment-0001.sig>

From marius at pov.lt  Thu Mar  6 10:10:41 2014
From: marius at pov.lt (Marius Gedminas)
Date: Thu, 6 Mar 2014 11:10:41 +0200
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
Message-ID: <20140306091041.GA24120@fridge.pov.lt>

On Wed, Mar 05, 2014 at 07:01:49AM -0500, Donald Stufft wrote:
> On Mar 5, 2014, at 5:48 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> > JSON-RPC is a better choice for recommended external API,
> > because it doesn't rely on memory hungry and potentially unsafe
> > XML libraries.
> > 
> > On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft <donald at stufft.io> wrote:
> >> Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
> >> Warehouse today. If you have an application that depends on XMLRPC on PyPI and
> >> you?d like to test it, please try against the url:
> >> 
> >>    https://pypi-preview.a.ssl.fastly.net/pypi
> >> 
> >> Let me know if you find any issues, or you can file a bug at:
> >> 
> >>    https://github.com/pypa/warehouse
> 
> XMLRPC is used in order to maintain compatibility with what is already
> there. Something better will replace it eventually and XMLRPC will be
> deprecated.

This is a bit confusing to me now.

You're reacting as if anatoly suggested a new JSON interface to
replace the legacy XML-RPC one.

But the old PyPI codebase already had a JSON API[1].  I'm using it to keep
track of Python 3 support status of about 800 packages maintained by the
Zope Foundation: http://zope3.pov.lt/py3/

  [1] https://wiki.python.org/moin/PyPIJSON

I assume at some point Warehouse will add support for the JSON API and
you'll issue a call for testing?

Marius Gedminas
-- 
The typewriter was invented by Hungarian immigrant Qwert Yuiop, who left his
"signature" on the keyboard.
        -- "Kim"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/f792217d/attachment.sig>

From techtonik at gmail.com  Wed Mar  5 14:34:21 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 5 Mar 2014 16:34:21 +0300
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
Message-ID: <CAPkN8xJW4E+gyQk8xUf2ZXTYhmz-gHq6G+fq-8g4VkLgSM-jUA@mail.gmail.com>

On Wed, Mar 5, 2014 at 3:01 PM, Donald Stufft <donald at stufft.io> wrote:
>
> XMLRPC is used in order to maintain compatibility with what is already there. Something
> better will replace it eventually and XMLRPC will be deprecated.

Good to know that. Testing XMLRPC is a chance to port handlers to a better API,
so if "something better" already has some forms, or at least a ticket
- it will be nice
to get subscribed to it.

-- 
anatoly t.

From teh.ivo at gmail.com  Thu Mar  6 12:46:30 2014
From: teh.ivo at gmail.com (Matthew Iversen)
Date: Thu, 06 Mar 2014 22:46:30 +1100
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <20140306091041.GA24120@fridge.pov.lt>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
 <20140306091041.GA24120@fridge.pov.lt>
Message-ID: <53186016.9020203@notevencode.com>

On 06/03/14 20:10, Marius Gedminas wrote:
> On Wed, Mar 05, 2014 at 07:01:49AM -0500, Donald Stufft wrote:
>> On Mar 5, 2014, at 5:48 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>>> JSON-RPC is a better choice for recommended external API,
>>> because it doesn't rely on memory hungry and potentially unsafe
>>> XML libraries.
>>>
>>> On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft <donald at stufft.io> wrote:
>>>> Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
>>>> Warehouse today. If you have an application that depends on XMLRPC on PyPI and
>>>> you'd like to test it, please try against the url:
>>>>
>>>>    https://pypi-preview.a.ssl.fastly.net/pypi
>>>>
>>>> Let me know if you find any issues, or you can file a bug at:
>>>>
>>>>    https://github.com/pypa/warehouse
>> XMLRPC is used in order to maintain compatibility with what is already
>> there. Something better will replace it eventually and XMLRPC will be
>> deprecated.
> This is a bit confusing to me now.
>
> You're reacting as if anatoly suggested a new JSON interface to
> replace the legacy XML-RPC one.
>
> But the old PyPI codebase already had a JSON API[1].  I'm using it to keep
> track of Python 3 support status of about 800 packages maintained by the
> Zope Foundation: http://zope3.pov.lt/py3/
>
>   [1] https://wiki.python.org/moin/PyPIJSON
>
> I assume at some point Warehouse will add support for the JSON API and
> you'll issue a call for testing?
>
> Marius Gedminas
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
A JSON-RPC interface is slightly different from a "normal" 'RESTful
JSON' api.

PyPI currently offers some normal JSON REST resources, as well as an
XML-RPC interface.

AFAIK Anatoly was suggesting to replace XML-RPC, with JSON-RPC (not just
a JSON API, which as you mention already exists), however the former
needs to continue to exist because it is an existing feature that many
things use.

Also AFAIK, warehouse is currently in the business of gaining feature
parity with the current PyPI, after which it will be looked into if or
what new APIs will be useful to add to make better clients with.
Of course a prime candidate for that might be expanding the current
RESTful JSON api.

-- 
Matt Iversen
PGP: 0xc046e8a874522973 // 2F04 3DCC D6E6 D5AC D262  2E0B C046 E8A8 7452 2973

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/233bc16e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/233bc16e/attachment.sig>

From donald at stufft.io  Thu Mar  6 13:28:00 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 6 Mar 2014 07:28:00 -0500
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <20140306091041.GA24120@fridge.pov.lt>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
 <20140306091041.GA24120@fridge.pov.lt>
Message-ID: <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>


On Mar 6, 2014, at 4:10 AM, Marius Gedminas <marius at pov.lt> wrote:

> On Wed, Mar 05, 2014 at 07:01:49AM -0500, Donald Stufft wrote:
>> On Mar 5, 2014, at 5:48 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>>> JSON-RPC is a better choice for recommended external API,
>>> because it doesn't rely on memory hungry and potentially unsafe
>>> XML libraries.
>>> 
>>> On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft <donald at stufft.io> wrote:
>>>> Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
>>>> Warehouse today. If you have an application that depends on XMLRPC on PyPI and
>>>> you?d like to test it, please try against the url:
>>>> 
>>>>   https://pypi-preview.a.ssl.fastly.net/pypi
>>>> 
>>>> Let me know if you find any issues, or you can file a bug at:
>>>> 
>>>>   https://github.com/pypa/warehouse
>> 
>> XMLRPC is used in order to maintain compatibility with what is already
>> there. Something better will replace it eventually and XMLRPC will be
>> deprecated.
> 
> This is a bit confusing to me now.
> 
> You're reacting as if anatoly suggested a new JSON interface to
> replace the legacy XML-RPC one.

Well specifically he said JSON-RPC is better than XMLRPC which sounds like
he was disagreeing with Warehouse implementing the XMLRPC interface. If
that wasn?t the case then I misunderstood :) I don?t think an RPC interface
is really needed at all for PyPI.

> 
> But the old PyPI codebase already had a JSON API[1].  I'm using it to keep
> track of Python 3 support status of about 800 packages maintained by the
> Zope Foundation: http://zope3.pov.lt/py3/
> 
>  [1] https://wiki.python.org/moin/PyPIJSON
> 
> I assume at some point Warehouse will add support for the JSON API and
> you'll issue a call for testing?

Richard has actually already implemented it! But in testing it just now I see it
is failing on the server but isn?t locally for me so I?ll need to see why that is the
case.

> 
> Marius Gedminas
> -- 
> The typewriter was invented by Hungarian immigrant Qwert Yuiop, who left his
> "signature" on the keyboard.
>        -- "Kim"
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/2859b333/attachment.sig>

From techtonik at gmail.com  Thu Mar  6 16:21:16 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 6 Mar 2014 18:21:16 +0300
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
 <20140306091041.GA24120@fridge.pov.lt>
 <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>
Message-ID: <CAPkN8xKiMKt+o0V5ozi-gL_88+DhcXHJbD9V9Vh_84avgVp9WA@mail.gmail.com>

On Thu, Mar 6, 2014 at 3:28 PM, Donald Stufft <donald at stufft.io> wrote:
> On Mar 6, 2014, at 4:10 AM, Marius Gedminas <marius at pov.lt> wrote:
>> On Wed, Mar 05, 2014 at 07:01:49AM -0500, Donald Stufft wrote:
>>> On Mar 5, 2014, at 5:48 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>>>> JSON-RPC is a better choice for recommended external API,
>>>> because it doesn't rely on memory hungry and potentially unsafe
>>>> XML libraries.
>>>>
>>>> On Wed, Mar 5, 2014 at 3:41 AM, Donald Stufft <donald at stufft.io> wrote:
>>>>> Just a quick FYI that the last missing piece (search) for XMLRPC was merged in
>>>>> Warehouse today. If you have an application that depends on XMLRPC on PyPI and
>>>>> you?d like to test it, please try against the url:
>>>>>
>>>>>   https://pypi-preview.a.ssl.fastly.net/pypi
>>>>>
>>>>> Let me know if you find any issues, or you can file a bug at:
>>>>>
>>>>>   https://github.com/pypa/warehouse
>>>
>>> XMLRPC is used in order to maintain compatibility with what is already
>>> there. Something better will replace it eventually and XMLRPC will be
>>> deprecated.
>>
>> This is a bit confusing to me now.
>>
>> You're reacting as if anatoly suggested a new JSON interface to
>> replace the legacy XML-RPC one.
>
> Well specifically he said JSON-RPC is better than XMLRPC which sounds like
> he was disagreeing with Warehouse implementing the XMLRPC interface. If
> that wasn?t the case then I misunderstood :) I don?t think an RPC interface
> is really needed at all for PyPI.

Let me clarify to avoid further speculations.
1. I forgot that PyPI exposes JSON API
2. I expressed concern that XMLRPC is promoted for integration with PyPI and
    it should not, because it is ineffective

>> But the old PyPI codebase already had a JSON API[1].  I'm using it to keep
>> track of Python 3 support status of about 800 packages maintained by the
>> Zope Foundation: http://zope3.pov.lt/py3/
>>
>>  [1] https://wiki.python.org/moin/PyPIJSON
>>
>> I assume at some point Warehouse will add support for the JSON API and
>> you'll issue a call for testing?
>
> Richard has actually already implemented it! But in testing it just now I see it
> is failing on the server but isn?t locally for me so I?ll need to see why that is the
> case.

So yes. "Use JSON API, but test XMLRPC for compatibility" is the thing I
expected to hear. =)

-- 
anatoly t.

From mike_mp at zzzcomputing.com  Thu Mar  6 18:55:45 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Thu, 6 Mar 2014 12:55:45 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
Message-ID: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>

Hi Distutils !

I don?t follow this list and haven?t looked at it in a long time.   However, I?m learning via twitter that a brand new setuptools release that?s just gone out has just removed the ?Feature? mechanism.

Now as you?re all rolling your eyes and preparing to bang out frustrated replies how this was well announced and deprecated and warned about and wow really didn?t you know the term paper was due today, OK first off let me say I am sorry!  I am not a distutils/setuptools maintainer.    I am just someone that uses it, and as I include setuptools in my setup.py, I am also getting thousands of other people who download my product to use it as well.    And I don?t read the setuptools changelog!  Or the setuptools blog.  Or this list.  I assume you guys have it under control (and you certainly do!).  There seem to be other people like me (people who write very widely used software) who also seem surprised!   And that is surprising too (as I am usually the only person to be surprised by these things that should not be surprising).  So I hope you can hold back your frustration with my clueless RTFMness long enough to answer these questions:

1. where was this announced?   I?m wondering why there weren?t loud blaring blog posts and tweets all over the place stating that on March 6, dozens of major packages are going to all break.

2. what is the plan for unmaintained packages and old versions of existing packages on pypi that ?import Feature? and can no longer be installed?    Is it just the case that a large amount of pypi just won?t install anymore?

3. What, if any, is the recommended approach going forward to a Python package that wishes to specify a command-line flag during install.  Here?s multiple choice:

    a. you can use new setuptools API <some new way to add ?flags>

    b. you can roll it yourself in setup.py using <hack X>

    c. your setup.py should never accept any kind of flags as that interferes with <up-and-coming use case Q>

    d. other

If choice c., then here is another question.  My library includes optional C extensions.   On certain platforms, these C extensions don?t build (like on Pypy, or on Windows if you don?t have compilation tools installed).  In this regard it gracefully degrades.  But also, I want to be able to have a command line option to determine this as well.  Because!  Maybe I?m installing within a test suite where I need to test the entire library without the C extensions built.    Maybe some user has found a bug in the C extensions, and that user needs to temporarily install the tool without the extensions built.    Other cases for flags are, maybe your library can build with or without SSL support, something like that.

Keep in mind, I actually *won?t* be getting bug reports about this because my setup.py already gracefully degrades to distutils!  But still, I?d like to have my ?without-cextensions flag somehow.

Thanks for listening and again I apologize for not following the setuptools changelog on a regular basis!


- mike





From donald at stufft.io  Thu Mar  6 19:06:43 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 6 Mar 2014 13:06:43 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
Message-ID: <7FBA3058-31EB-4BD3-A252-BF91FE6144DB@stufft.io>


On Mar 6, 2014, at 12:55 PM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:

> Hi Distutils !
> 
> I don?t follow this list and haven?t looked at it in a long time.   However, I?m learning via twitter that a brand new setuptools release that?s just gone out has just removed the ?Feature? mechanism.
> 
> Now as you?re all rolling your eyes and preparing to bang out frustrated replies how this was well announced and deprecated and warned about and wow really didn?t you know the term paper was due today, OK first off let me say I am sorry!  I am not a distutils/setuptools maintainer.    I am just someone that uses it, and as I include setuptools in my setup.py, I am also getting thousands of other people who download my product to use it as well.    And I don?t read the setuptools changelog!  Or the setuptools blog.  Or this list.  I assume you guys have it under control (and you certainly do!).  There seem to be other people like me (people who write very widely used software) who also seem surprised!   And that is surprising too (as I am usually the only person to be surprised by these things that should not be surprising).  So I hope you can hold back your frustration with my clueless RTFMness long enough to answer these questions:
> 
> 1. where was this announced?   I?m wondering why there weren?t loud blaring blog posts and tweets all over the place stating that on March 6, dozens of major packages are going to all break.
> 
> 2. what is the plan for unmaintained packages and old versions of existing packages on pypi that ?import Feature? and can no longer be installed?    Is it just the case that a large amount of pypi just won?t install anymore?
> 
> 3. What, if any, is the recommended approach going forward to a Python package that wishes to specify a command-line flag during install.  Here?s multiple choice:
> 
>    a. you can use new setuptools API <some new way to add ?flags>
> 
>    b. you can roll it yourself in setup.py using <hack X>
> 
>    c. your setup.py should never accept any kind of flags as that interferes with <up-and-coming use case Q>
> 
>    d. other
> 
> If choice c., then here is another question.  My library includes optional C extensions.   On certain platforms, these C extensions don?t build (like on Pypy, or on Windows if you don?t have compilation tools installed).  In this regard it gracefully degrades.  But also, I want to be able to have a command line option to determine this as well.  Because!  Maybe I?m installing within a test suite where I need to test the entire library without the C extensions built.    Maybe some user has found a bug in the C extensions, and that user needs to temporarily install the tool without the extensions built.    Other cases for flags are, maybe your library can build with or without SSL support, something like that.
> 
> Keep in mind, I actually *won?t* be getting bug reports about this because my setup.py already gracefully degrades to distutils!  But still, I?d like to have my ?without-cextensions flag somehow.
> 
> Thanks for listening and again I apologize for not following the setuptools changelog on a regular basis!
> 
> 
> - mike
> 
> 
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


FWIW as the person who was mostly the responder on Twitter, I really didn?t mean to sound RTFM-y. I think that communicating things to users is *important*. I was mostly just responding to the fact it wasn?t completely out of the blue :)

As far as I?m aware here are the answers:

1. Setuptools changelog in Version 1.0
2. Use older versions of setuptools
3. No idea.

Please note that I?m not the setuptools maintainer and these are just what I?m aware of.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/557bbd24/attachment.sig>

From dholth at gmail.com  Thu Mar  6 19:07:52 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 6 Mar 2014 13:07:52 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
Message-ID: <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>

pje said:

The "Feature()" facility was never completely implemented or
supported, and even if it were, it should be deprecated now, as it
will not be compatible with the coming packaging systems based on PEP
426. If you need separate features, use separate distributions and
"extras" instead.

(The problem I discovered when I first implemented Feature is that
distutils' build system really isn't capable of having a sane
"configure" step, or saving that configuration. Essentially, distutils
is a little too stateless for that, but Features need it in order not
to act weird in various scenarios. "Extras" are basically the intended
replacement for this functionality, and Features were just kept around
to support legacy projects.)

See also

https://bitbucket.org/pypa/setuptools/issue/58

https://bitbucket.org/pypa/setuptools/issue/65

On Thu, Mar 6, 2014 at 12:55 PM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:
> Hi Distutils !
>
> I don't follow this list and haven't looked at it in a long time.   However, I'm learning via twitter that a brand new setuptools release that's just gone out has just removed the "Feature" mechanism.
>
> Now as you're all rolling your eyes and preparing to bang out frustrated replies how this was well announced and deprecated and warned about and wow really didn't you know the term paper was due today, OK first off let me say I am sorry!  I am not a distutils/setuptools maintainer.    I am just someone that uses it, and as I include setuptools in my setup.py, I am also getting thousands of other people who download my product to use it as well.    And I don't read the setuptools changelog!  Or the setuptools blog.  Or this list.  I assume you guys have it under control (and you certainly do!).  There seem to be other people like me (people who write very widely used software) who also seem surprised!   And that is surprising too (as I am usually the only person to be surprised by these things that should not be surprising).  So I hope you can hold back your frustration with my clueless RTFMness long enough to answer these questions:
>
> 1. where was this announced?   I'm wondering why there weren't loud blaring blog posts and tweets all over the place stating that on March 6, dozens of major packages are going to all break.
>
> 2. what is the plan for unmaintained packages and old versions of existing packages on pypi that "import Feature" and can no longer be installed?    Is it just the case that a large amount of pypi just won't install anymore?
>
> 3. What, if any, is the recommended approach going forward to a Python package that wishes to specify a command-line flag during install.  Here's multiple choice:
>
>     a. you can use new setuptools API <some new way to add --flags>
>
>     b. you can roll it yourself in setup.py using <hack X>
>
>     c. your setup.py should never accept any kind of flags as that interferes with <up-and-coming use case Q>
>
>     d. other
>
> If choice c., then here is another question.  My library includes optional C extensions.   On certain platforms, these C extensions don't build (like on Pypy, or on Windows if you don't have compilation tools installed).  In this regard it gracefully degrades.  But also, I want to be able to have a command line option to determine this as well.  Because!  Maybe I'm installing within a test suite where I need to test the entire library without the C extensions built.    Maybe some user has found a bug in the C extensions, and that user needs to temporarily install the tool without the extensions built.    Other cases for flags are, maybe your library can build with or without SSL support, something like that.
>
> Keep in mind, I actually *won't* be getting bug reports about this because my setup.py already gracefully degrades to distutils!  But still, I'd like to have my --without-cextensions flag somehow.
>
> Thanks for listening and again I apologize for not following the setuptools changelog on a regular basis!
>
>
> - mike
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From mike_mp at zzzcomputing.com  Thu Mar  6 19:14:47 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Thu, 6 Mar 2014 13:14:47 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
Message-ID: <BF356697-506C-4049-8E57-EF3ADA846F4B@zzzcomputing.com>


On Mar 6, 2014, at 12:55 PM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:

> changelog!  Or the setuptools blog.  Or this list.  I assume you guys have it under control (and you certainly do!).  There seem to be other people like me (people 

please note this correction to my last email:  ?I assume you guys have it under control? should read:  ?I assume you folks have it under control?.    I apologize for this mistake.



From mike_mp at zzzcomputing.com  Thu Mar  6 19:37:08 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Thu, 6 Mar 2014 13:37:08 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
Message-ID: <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>


On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:

> pje said:
> 
> The "Feature()" facility was never completely implemented or
> supported, and even if it were, it should be deprecated now, as it
> will not be compatible with the coming packaging systems based on PEP
> 426. If you need separate features, use separate distributions and
> "extras" instead.

wait, ok this is that thing.  ?separate distributions? means?.


SQLAlchemy-0.9.3-nocext.tar.gz
SQLAlchemy-0.9.3-cext.tar.gz

??

I think that?s what it means.    OK, not sure if I have to say this but that would be?OK very very unworkable.   A Python package distribution is a *source distribution*.  That suggests that one need to package up separate source distributions in order to specify an option, is? OK very very unworkable.   (note by ?very very unworkable? I mean ?list of hyperboles I had to backspace out because that?s not why I?m here?).

If a package has three flags, now there need to be  *eight source distros*?  Really?   On the plus side, I just used math.

how does one even maintain this in source control?  Do I have setup.py.nocext, setup.py.cext, maintain a copy of 99% the same code in each, then rename them when I do ?python setup.py sdist? (since sdist doesn?t take command line options either!!)  ?   The suggestion is completely inappropriate.

I don?t know the metadata format in pep426 well enough to comment (as I wanted to use it one day and found that it seemed to still be pretty much vapor), but I?ll reiterate: these are source distributions, not binaries or wheels or anything like that.  A source distro has to build, and builds need options.    A metadata format that purports to represent metadata about a source distribution and does not support flags is a broken metadata format.




From dholth at gmail.com  Thu Mar  6 20:21:34 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 6 Mar 2014 14:21:34 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
Message-ID: <CAG8k2+7D22TqL76AZQCQBxtPmRT8oDBTYguvNaAyC9x4RjaA8Q@mail.gmail.com>

It is certainly broken.

On Thu, Mar 6, 2014 at 1:37 PM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:
>
> On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> pje said:
>>
>> The "Feature()" facility was never completely implemented or
>> supported, and even if it were, it should be deprecated now, as it
>> will not be compatible with the coming packaging systems based on PEP
>> 426. If you need separate features, use separate distributions and
>> "extras" instead.
>
> wait, ok this is that thing.  "separate distributions" means....
>
>
> SQLAlchemy-0.9.3-nocext.tar.gz
> SQLAlchemy-0.9.3-cext.tar.gz
>
> ??
>
> I think that's what it means.    OK, not sure if I have to say this but that would be...OK very very unworkable.   A Python package distribution is a *source distribution*.  That suggests that one need to package up separate source distributions in order to specify an option, is... OK very very unworkable.   (note by "very very unworkable" I mean "list of hyperboles I had to backspace out because that's not why I'm here").
>
> If a package has three flags, now there need to be  *eight source distros*?  Really?   On the plus side, I just used math.
>
> how does one even maintain this in source control?  Do I have setup.py.nocext, setup.py.cext, maintain a copy of 99% the same code in each, then rename them when I do "python setup.py sdist" (since sdist doesn't take command line options either!!)  ?   The suggestion is completely inappropriate.
>
> I don't know the metadata format in pep426 well enough to comment (as I wanted to use it one day and found that it seemed to still be pretty much vapor), but I'll reiterate: these are source distributions, not binaries or wheels or anything like that.  A source distro has to build, and builds need options.    A metadata format that purports to represent metadata about a source distribution and does not support flags is a broken metadata format.
>
>
>

From vinay_sajip at yahoo.co.uk  Thu Mar  6 20:59:14 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 6 Mar 2014 19:59:14 +0000 (UTC)
Subject: [Distutils] tourist here, with a dumb RTFM question
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
Message-ID: <loom.20140306T205217-810@post.gmane.org>

Michael Bayer <mike_mp <at> zzzcomputing.com> writes:

> I don?t know the metadata format in pep426 well enough to comment (as I 
wanted to use it one day and found
> that it seemed to still be pretty much vapor), but I?ll reiterate: these 
are source distributions, not
> binaries or wheels or anything like that.  A source distro has to build, 
and builds need options.    A metadata
> format that purports to represent metadata about a source distribution and 
does not support flags is a
> broken metadata format.

PEP 426 only covers installation metadata - its scope does not cover source 
distributions (there will be another PEP covering that).

Apart from the most recent changes being not yet implemented and some open 
issues relating to hooks, PEP 426 seems to cover installation metadata 
reasonably well. However, sdist metadata is a different kettle of fish.

Regards,

Vinay Sajip



From mike_mp at zzzcomputing.com  Thu Mar  6 22:13:12 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Thu, 6 Mar 2014 16:13:12 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <loom.20140306T205217-810@post.gmane.org>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <loom.20140306T205217-810@post.gmane.org>
Message-ID: <4E94B134-831B-4F8A-B865-0C404347B26C@zzzcomputing.com>


On Mar 6, 2014, at 2:59 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> Michael Bayer <mike_mp <at> zzzcomputing.com> writes:
> 
>> I don?t know the metadata format in pep426 well enough to comment (as I 
> wanted to use it one day and found
>> that it seemed to still be pretty much vapor), but I?ll reiterate: these 
> are source distributions, not
>> binaries or wheels or anything like that.  A source distro has to build, 
> and builds need options.    A metadata
>> format that purports to represent metadata about a source distribution and 
> does not support flags is a
>> broken metadata format.
> 
> PEP 426 only covers installation metadata - its scope does not cover source 
> distributions (there will be another PEP covering that).
> 
> Apart from the most recent changes being not yet implemented and some open 
> issues relating to hooks, PEP 426 seems to cover installation metadata 
> reasonably well. However, sdist metadata is a different kettle of fish.

OK so why does PEP-426 compatibility imply removal of command line switches from setup.py files ?



From qwcode at gmail.com  Thu Mar  6 22:15:30 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Thu, 6 Mar 2014 13:15:30 -0800
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
Message-ID: <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>

On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com>wrote:

>
> On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
>
> > pje said:
> >
> > The "Feature()" facility was never completely implemented or
> > supported, and even if it were, it should be deprecated now, as it
> > will not be compatible with the coming packaging systems based on PEP
> > 426. If you need separate features, use separate distributions and
> > "extras" instead.
>
> wait, ok this is that thing.  "separate distributions" means....
>
>
> SQLAlchemy-0.9.3-nocext.tar.gz
> SQLAlchemy-0.9.3-cext.tar.gz
>

I'm new to understanding setuptools "Feature" myself, so don't crank too
much, if I get anything wrong.

afaik, it means something like:

SQLAlchemy-X.Y.tar.gz
SQLAlchemy-cext-X.Y.tar.gz

i.e. isolating the feature into a separate project.

and then instead of having the handy  "--with-cext" option,  it has to be
 "pip install SQLAlchemy[SQLAlchemy-cext]

PJ, Jason: can you clarify what the alternative is supposed to look like?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/a329ff05/attachment.html>

From vinay_sajip at yahoo.co.uk  Thu Mar  6 22:18:13 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 6 Mar 2014 21:18:13 +0000 (UTC)
Subject: [Distutils] tourist here, with a dumb RTFM question
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <7FBA3058-31EB-4BD3-A252-BF91FE6144DB@stufft.io>
Message-ID: <loom.20140306T221436-959@post.gmane.org>

Donald Stufft <donald <at> stufft.io> writes:

> 2. Use older versions of setuptools

That's not really an answer, unless downgrading works. For example, a 
recently created venv would contain the latest setuptools, and perhaps it 
would be required by other distributions in the venv. How then would e.g. 
SQLAlchemy specify that an older setuptools version should be used? Would pip 
downgrade the venv's setuptools to an older version? Even if it did, might 
that not break other distributions in the venv?

Regards,

Vinay Sajip


From donald at stufft.io  Thu Mar  6 22:24:01 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 6 Mar 2014 16:24:01 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
Message-ID: <8139901D-33A7-4676-B6EA-D3E97F91C322@stufft.io>



> On Mar 6, 2014, at 4:15 PM, Marcus Smith <qwcode at gmail.com> wrote:
> 
> 
> 
> 
>> On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:
>> 
>> On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
>> 
>> > pje said:
>> >
>> > The "Feature()" facility was never completely implemented or
>> > supported, and even if it were, it should be deprecated now, as it
>> > will not be compatible with the coming packaging systems based on PEP
>> > 426. If you need separate features, use separate distributions and
>> > "extras" instead.
>> 
>> wait, ok this is that thing.  ?separate distributions? means?.
>> 
>> 
>> SQLAlchemy-0.9.3-nocext.tar.gz
>> SQLAlchemy-0.9.3-cext.tar.gz
> 
> I'm new to understanding setuptools "Feature" myself, so don't crank too much, if I get anything wrong.
> 
> afaik, it means something like:
> 
> SQLAlchemy-X.Y.tar.gz
> SQLAlchemy-cext-X.Y.tar.gz
> 
> i.e. isolating the feature into a separate project.
> 
> and then instead of having the handy  "--with-cext" option,  it has to be  "pip install SQLAlchemy[SQLAlchemy-cext]
> 

This is my understanding as well except it inverses the default. SQLAlchemy wants the C exts installed unless they can't be if the user opts out. Extras would cause if to not be installed unless the user opts in. This is probably not a reasonable solution. 

One note if could just be SQLAlchemy[cext]

> PJ, Jason: can you clarify what the alternative is supposed to look like?
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/ff2384ad/attachment.html>

From vinay_sajip at yahoo.co.uk  Thu Mar  6 22:28:40 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Thu, 6 Mar 2014 21:28:40 +0000 (GMT)
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <4E94B134-831B-4F8A-B865-0C404347B26C@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <loom.20140306T205217-810@post.gmane.org>
 <4E94B134-831B-4F8A-B865-0C404347B26C@zzzcomputing.com>
Message-ID: <1394141320.79824.YahooMailNeo@web172406.mail.ir2.yahoo.com>



From:Michael Bayer <mike_mp at zzzcomputing.com>



> OK so why does PEP-426 compatibility imply removal of command line switches from setup.py files ?

As far as I know, it doesn't. My distil tools complies with a fairly recent version of PEP 426 and AFAIK has no problem building / installing SQLAlchemy with / without extensions, but it uses an extended set of metadata which is not specified in any PEP - for example,

http://www.red-dove.com/pypi/projects/S/SQLAlchemy/package-0.9.3.json


This is just FYI - it's experimental metadata [which has some redundancy for historical reasons, to be tidied up in the future] and is extracted from what is passed to setup.py - it includes the PEP 426 metadata as a subset ("index-metadata"), and the extra stuff is needed for building. This is used by the distil tool, but there's no support for any specific command-line flags to exclude the building extensions. (That's not because of PEP compliance - it's just that I haven't needed that level of control in the tool.)

Regards,

Vinay Sajip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/cc80a112/attachment.html>

From qwcode at gmail.com  Thu Mar  6 22:35:03 2014
From: qwcode at gmail.com (Marcus Smith)
Date: Thu, 6 Mar 2014 13:35:03 -0800
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <8139901D-33A7-4676-B6EA-D3E97F91C322@stufft.io>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
 <8139901D-33A7-4676-B6EA-D3E97F91C322@stufft.io>
Message-ID: <CAPYWazqtRs88Wxh7gD8zLnMLnRLM_0vwTpPAid4icbd6g0qU+w@mail.gmail.com>

>
>
> This is my understanding as well except it inverses the default.
> SQLAlchemy wants the C exts installed unless they can't be if the user opts
> out. Extras would cause if to not be installed unless the user opts in.
> This is probably not a reasonable solution.
>

ok, that's odd, so there is no extras replacement for the --without-cext
scenario.
so even more important to understand from PJ/Jason the rationale of how
Feature could be removed


>
> One note if could just be SQLAlchemy[cext]
>

oh yea,  you just pass a label
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/2ae83f1e/attachment.html>

From mike_mp at zzzcomputing.com  Thu Mar  6 23:11:04 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Thu, 6 Mar 2014 17:11:04 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CAPYWazqtRs88Wxh7gD8zLnMLnRLM_0vwTpPAid4icbd6g0qU+w@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
 <8139901D-33A7-4676-B6EA-D3E97F91C322@stufft.io>
 <CAPYWazqtRs88Wxh7gD8zLnMLnRLM_0vwTpPAid4icbd6g0qU+w@mail.gmail.com>
Message-ID: <B62E38EC-68F0-47E8-8897-2AD0F5C59FDF@zzzcomputing.com>


On Mar 6, 2014, at 4:35 PM, Marcus Smith <qwcode at gmail.com> wrote:

> 
> This is my understanding as well except it inverses the default. SQLAlchemy wants the C exts installed unless they can't be if the user opts out. Extras would cause if to not be installed unless the user opts in. This is probably not a reasonable solution. 
> 
> ok, that's odd, so there is no extras replacement for the --without-cext scenario.
> so even more important to understand from PJ/Jason the rationale of how Feature could be removed
>  
> 
> One note if could just be SQLAlchemy[cext]
> 
> oh yea,  you just pass a label

show me that


> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/d44f95c7/attachment-0001.html>

From benjaminrk at gmail.com  Fri Mar  7 01:27:46 2014
From: benjaminrk at gmail.com (MinRK)
Date: Thu, 6 Mar 2014 16:27:46 -0800
Subject: [Distutils] OS X and PEP 425 / wheels
Message-ID: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>

Hello,

I proposed a patch <https://github.com/pypa/pip/pull/1465> to pip, with
respect to treatment of the platform tag on OS X, and Chris Barker proposed
that I bring the discussion here.

The situation:

PEP 425 describes the platform tag as:

The platform tag is simply distutils.util.get*platform() with all hyphens -
and periods . replaced with underscore *.

but the PEP makes no mention of what should be done on OS X. On OS X,
get_platform() has the form:

macosx_10_6_intel

where 10_6 indicates that the OS X deployment target (of Python) is 10.6,
and intel indicates that it is a fat binary with both x86_64 and i386.
Other values for the arch include ?ppc?, ?ppc64?, and ?universal?, another
multi-arch tag indicating a fat binary with all four of 32/64bit intel and
ppc.

Where PEP 425 causes problems (or pip?s implementation thereof), is in
testing strict equality of this value, not taking into account the meaning
of its contents. I proposed two changes:

   1.

   Use >= comparison for the deployment target

   From the OS X platform
documentation<https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/cross_development/Configuring/configuring.html>
   :

   Choose a deployment target. This identifies the earliest OS version on
   which your software can run.

   so a binary with deployment target = 10.6 should be installable on OS X
   >= 10.6.

   This change restores the longstanding behavior of easy_install, where
   10.6 eggs are installable on >= 10.6.
    2.

   support multi-arch names (intel, universal) on their respective
   components
    - intel is valid on {x86_64, i386}
      - universal is valid on {intel, x86_64, i386, ppc64, ppc}

   easy_install, like pip, also does strict comparison here, so this would
   be new behavior.

My actual use case:

I have a wheel (pyzmq), which works on any intel-based Python targeting OS
X >= 10.6. To express this with pip-1.5, the filename has to be:

pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.macosx_10_7_intel.macosx_10_7_x86_64.macosx_10_7_i386.macosx_10_8_intel.macosx_10_8_x86_64.macosx_10_8_i386.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_9_i386.whl

and it has to grow every time there is a new OS release.

With both proposed changes, I could communicate the same information with
just one tag:

pyzmq-14.1.0-cp27-none-macosx_10_6_intel.whl

With just the first change, it would need three:

pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.whl

Any feedback from experts, especially if my understanding of deployment
targets or fat binaries is incorrect, would be much appreciated.

Thanks,

-MinRK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/94ded214/attachment.html>

From dholth at gmail.com  Fri Mar  7 02:26:02 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 6 Mar 2014 20:26:02 -0500
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
Message-ID: <CAG8k2+6sK3_vgLwL28xaAtd+FrqrLNo=eDFvOWWQWv5Z84yaBA@mail.gmail.com>

The pep425 design tries to avoid assigning any ordering to the tag
components. For the py version we just add py27, py26, py25 for example to
the list of accepted tags. Perhaps the same strategy works for osx. So the
osx 10.9 installer would generate old osx tags to add to its acceptable set.
On Mar 6, 2014 7:59 PM, "MinRK" <benjaminrk at gmail.com> wrote:

> Hello,
>
> I proposed a patch <https://github.com/pypa/pip/pull/1465> to pip, with
> respect to treatment of the platform tag on OS X, and Chris Barker proposed
> that I bring the discussion here.
>
> The situation:
>
> PEP 425 describes the platform tag as:
>
> The platform tag is simply distutils.util.get*platform() with all hyphens
> - and periods . replaced with underscore *.
>
> but the PEP makes no mention of what should be done on OS X. On OS X,
> get_platform() has the form:
>
> macosx_10_6_intel
>
> where 10_6 indicates that the OS X deployment target (of Python) is 10.6,
> and intel indicates that it is a fat binary with both x86_64 and i386.
> Other values for the arch include 'ppc', 'ppc64', and 'universal', another
> multi-arch tag indicating a fat binary with all four of 32/64bit intel and
> ppc.
>
> Where PEP 425 causes problems (or pip's implementation thereof), is in
> testing strict equality of this value, not taking into account the meaning
> of its contents. I proposed two changes:
>
>    1.
>
>    Use >= comparison for the deployment target
>
>    From the OS X platform documentation<https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/cross_development/Configuring/configuring.html>
>    :
>
>    Choose a deployment target. This identifies the earliest OS version on
>    which your software can run.
>
>    so a binary with deployment target = 10.6 should be installable on OS
>    X >= 10.6.
>
>    This change restores the longstanding behavior of easy_install, where
>    10.6 eggs are installable on >= 10.6.
>     2.
>
>    support multi-arch names (intel, universal) on their respective
>    components
>     - intel is valid on {x86_64, i386}
>       - universal is valid on {intel, x86_64, i386, ppc64, ppc}
>
>    easy_install, like pip, also does strict comparison here, so this
>    would be new behavior.
>
> My actual use case:
>
> I have a wheel (pyzmq), which works on any intel-based Python targeting OS
> X >= 10.6. To express this with pip-1.5, the filename has to be:
>
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.macosx_10_7_intel.macosx_10_7_x86_64.macosx_10_7_i386.macosx_10_8_intel.macosx_10_8_x86_64.macosx_10_8_i386.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_9_i386.whl
>
> and it has to grow every time there is a new OS release.
>
> With both proposed changes, I could communicate the same information with
> just one tag:
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.whl
>
> With just the first change, it would need three:
>
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.whl
>
> Any feedback from experts, especially if my understanding of deployment
> targets or fat binaries is incorrect, would be much appreciated.
>
> Thanks,
>
> -MinRK
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/136b17f4/attachment.html>

From nad at acm.org  Fri Mar  7 02:46:37 2014
From: nad at acm.org (Ned Deily)
Date: Thu, 06 Mar 2014 17:46:37 -0800
Subject: [Distutils] OS X and PEP 425 / wheels
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
Message-ID: <nad-A92CBC.17463706032014@news.gmane.org>

In article 
<CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw at mail.gmail.com>,
 MinRK <benjaminrk at gmail.com> wrote:
> I proposed a patch <https://github.com/pypa/pip/pull/1465> to pip, with
> respect to treatment of the platform tag on OS X, and Chris Barker proposed
> that I bring the discussion here.
> 
> The situation:
> 
> PEP 425 describes the platform tag as:
> 
> The platform tag is simply distutils.util.get*platform() with all hyphens -
> and periods . replaced with underscore *.
> 
> but the PEP makes no mention of what should be done on OS X. On OS X,
> get_platform() has the form:
> 
> macosx_10_6_intel
> 
> where 10_6 indicates that the OS X deployment target (of Python) is 10.6,
> and intel indicates that it is a fat binary with both x86_64 and i386.
> Other values for the arch include 'ppc', 'ppc64', and 'universal', another
> multi-arch tag indicating a fat binary with all four of 32/64bit intel and
> ppc.

As I commented in your pull request, the complete set of universal arch 
values is "fat", "intel", "fat3", "fat64", and "universal" besides the 
single arch values "ppc", "ppc64", "i386", and "x86_64".  It would be 
best if all of them were supported in pip.

> Where PEP 425 causes problems (or pip's implementation thereof), is in
> testing strict equality of this value, not taking into account the meaning
> of its contents. I proposed two changes:
> 
>    1.
> 
>    Use >= comparison for the deployment target
> 
>    From the OS X platform
> documentation<https://developer.apple.com/library/mac/documentation/DeveloperT
> ools/Conceptual/cross_development/Configuring/configuring.html>
>    :
> 
>    Choose a deployment target. This identifies the earliest OS version on
>    which your software can run.
> 
>    so a binary with deployment target = 10.6 should be installable on OS X
>    >= 10.6.
> 
>    This change restores the longstanding behavior of easy_install, where
>    10.6 eggs are installable on >= 10.6.
>     2.
> 
>    support multi-arch names (intel, universal) on their respective
>    components
>     - intel is valid on {x86_64, i386}
>       - universal is valid on {intel, x86_64, i386, ppc64, ppc}
> 
>    easy_install, like pip, also does strict comparison here, so this would
>    be new behavior.
[...]
> With both proposed changes, I could communicate the same information with
> just one tag:
> 
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.whl
>
> Any feedback from experts, especially if my understanding of deployment
> targets or fat binaries is incorrect, would be much appreciated.

Thanks for actually proposing an implementation of this.  I think both 
changes should be implemented, with the proviso that all universal arch 
values are supported (for example, the "fat" universal arch is still 
used in current 32-bit-only python.org Pythons.  As I also noted in the 
comments, it is not a perfect solution since it doesn't cover all 
possible cases (like Unicode ABI).  That will have to wait for Metadata 
2.0.  But it does provide important usability gains for OS X users right 
now and covers the most common (by far) use cases.

-- 
 Ned Deily,
 nad at acm.org


From chris.barker at noaa.gov  Fri Mar  7 02:29:24 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 6 Mar 2014 17:29:24 -0800
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
Message-ID: <CALGmxELvs4VmwzjY6_TLnkOje2gBd=E7k5EX-CLLmO2jDzvbyA@mail.gmail.com>

On Thu, Mar 6, 2014 at 4:27 PM, MinRK <benjaminrk at gmail.com> wrote:

> I proposed a patch <https://github.com/pypa/pip/pull/1465> to pip, with
> respect to treatment of the platform tag on OS X, and Chris Barker proposed
> that I bring the discussion here.
>

Note -- there is some more discusion on that patch...

>  The situation:
>
> PEP 425 describes the platform tag as:
>
> The platform tag is simply distutils.util.get*platform() with all hyphens
> - and periods . replaced with underscore *.
>
> but the PEP makes no mention of what should be done on OS X. On OS X,
> get_platform() has the form:
>
> macosx_10_6_intel
>
>
> .....

>
>    1.
>    2.
>
>    support multi-arch names (intel, universal) on their respective
>    components
>     - intel is valid on {x86_64, i386}
>       - universal is valid on {intel, x86_64, i386, ppc64, ppc}
>
>    easy_install, like pip, also does strict comparison here, so this
>    would be new behavior.
>
> yup -- and easy_install was actually quite broken for binary eggs for a
universal build -- so this would be great.

though a note: "universal:", as used by the python.org builds means
only  i386+ppc -- in theory once could do a quad build, but no on ever did.
Though ppc is pretty old now -- I don't think we need to worry about that
for anything future-looking anyway.

I have a wheel (pyzmq), which works on any intel-based Python targeting OS
> X >= 10.6. To express this with pip-1.5, the filename has to be:
>
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.macosx_10_7_intel.macosx_10_7_x86_64.macosx_10_7_i386.macosx_10_8_intel.macosx_10_8_x86_64.macosx_10_8_i386.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_9_i386.whl
>
> and it has to grow every time there is a new OS release.
>
clearly no ideal!

Any feedback from experts, especially if my understanding of deployment
> targets or fat binaries is incorrect, would be much appreciated.
>
I'm no expert, but his looks good to me.

As pointed out in the comments on the patch, there may be some issues with
C++  extensions and building on 10.9 -- we'll have to see how that turns
out....

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140306/601a4598/attachment.html>

From donald at stufft.io  Fri Mar  7 07:03:23 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 7 Mar 2014 01:03:23 -0500
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
 <20140306091041.GA24120@fridge.pov.lt>
 <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>
Message-ID: <0C5F8655-6340-4716-B8EF-6C25F7596FF7@stufft.io>


On Mar 6, 2014, at 7:28 AM, Donald Stufft <donald at stufft.io> wrote:

>> 
>> 
>> But the old PyPI codebase already had a JSON API[1].  I'm using it to keep
>> track of Python 3 support status of about 800 packages maintained by the
>> Zope Foundation: http://zope3.pov.lt/py3/
>> 
>> [1] https://wiki.python.org/moin/PyPIJSON
>> 
>> I assume at some point Warehouse will add support for the JSON API and
>> you'll issue a call for testing?
> 
> Richard has actually already implemented it! But in testing it just now I see it
> is failing on the server but isn?t locally for me so I?ll need to see why that is the
> case.

The JSON API is functional again and is available for testing!

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/b9cdf3e6/attachment.sig>

From p.f.moore at gmail.com  Fri Mar  7 11:18:51 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 7 Mar 2014 10:18:51 +0000
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CAG8k2+6sK3_vgLwL28xaAtd+FrqrLNo=eDFvOWWQWv5Z84yaBA@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
 <CAG8k2+6sK3_vgLwL28xaAtd+FrqrLNo=eDFvOWWQWv5Z84yaBA@mail.gmail.com>
Message-ID: <CACac1F_rL7gYGotQsPy-aqE51AbtmrYpvkDwHFRMZRJHTsU3Vg@mail.gmail.com>

On 7 March 2014 01:26, Daniel Holth <dholth at gmail.com> wrote:
> The pep425 design tries to avoid assigning any ordering to the tag
> components. For the py version we just add py27, py26, py25 for example to
> the list of accepted tags. Perhaps the same strategy works for osx. So the
> osx 10.9 installer would generate old osx tags to add to its acceptable set.

To clarify Daniel's point (because I struggled to understand this
aspect of the implementation when I first encountered it) the
responsibility is on the installer to explicitly enumerate all the
tags it is capable of using[1]. So in theory, that means that the pep
425 implementation used by pip when running on (for example) 10.9
x86_64 should claim it can accept a wheel
pyzmq-14.1.0-cp27-none-macosx_10_6_intel.whl.

Paul

[1] I'm not 100% sure that I agree with this design - I find it
confusing to say the least - but it is how things are
intended/expected to work.

From ncoghlan at gmail.com  Fri Mar  7 11:41:30 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 7 Mar 2014 20:41:30 +1000
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CACac1F_rL7gYGotQsPy-aqE51AbtmrYpvkDwHFRMZRJHTsU3Vg@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
 <CAG8k2+6sK3_vgLwL28xaAtd+FrqrLNo=eDFvOWWQWv5Z84yaBA@mail.gmail.com>
 <CACac1F_rL7gYGotQsPy-aqE51AbtmrYpvkDwHFRMZRJHTsU3Vg@mail.gmail.com>
Message-ID: <CADiSq7cu9VT6iem06eOxfBnY1yScHFTw3y1UNj3901UeV5rfuA@mail.gmail.com>



From benzolius at yahoo.com  Fri Mar  7 11:02:59 2014
From: benzolius at yahoo.com (Benedek Zoltan)
Date: Fri, 7 Mar 2014 02:02:59 -0800 (PST)
Subject: [Distutils] recommended setuptools version
Message-ID: <1394186579.18044.YahooMailNeo@web164703.mail.gq1.yahoo.com>

Hi,

I'm a bit confused with the setuptools versioning.
When I install it by ez_setup.py it installs the 3.0.2 version, but on pypi the latest version is 2.2.

Is there a difference between the two? Which is the recommended one?

Thanks
Zoltan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/b1865b39/attachment.html>

From marius at pov.lt  Fri Mar  7 13:45:24 2014
From: marius at pov.lt (Marius Gedminas)
Date: Fri, 7 Mar 2014 14:45:24 +0200
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <0C5F8655-6340-4716-B8EF-6C25F7596FF7@stufft.io>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
 <20140306091041.GA24120@fridge.pov.lt>
 <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>
 <0C5F8655-6340-4716-B8EF-6C25F7596FF7@stufft.io>
Message-ID: <20140307124524.GB11311@fridge.pov.lt>

On Fri, Mar 07, 2014 at 01:03:23AM -0500, Donald Stufft wrote:
> The JSON API is functional again and is available for testing!

Excellent!

I can observe two differences with the current PyPI code:

First, the Content-Type of the response differs:

  $ curl -sI https://pypi.python.org/pypi/setuptools/json | grep Content-Type
  Content-Type: application/json; charset="UTF-8"

  $ curl -sI https://pypi-preview.a.ssl.fastly.net/pypi/setuptools/json | grep Content-Type
  Content-Type: application/json

(I don't know which is correct, or whether the difference actually
matters.)

Second, the current PyPI codebase returns absolute URLs, while Warehouse
returns paths relative to the site root:

  $ curl -s https://pypi.python.org/pypi/setuptools/json | grep '"url":'
  "url": "https://pypi.python.org/packages/3.4/s/setuptools/setuptools-2.2-py2.py3-none-any.whl", 
  "url": "https://pypi.python.org/packages/source/s/setuptools/setuptools-2.2.tar.gz", 

  $ curl -s https://pypi-preview.a.ssl.fastly.net/pypi/setuptools/json | python -m json.tool | grep '"url":'
  "url": "/packages/3.4/s/setuptools/setuptools-2.2-py2.py3-none-any.whl"
  "url": "/packages/source/s/setuptools/setuptools-2.2.tar.gz"

I can adapt my script to cope.

Marius Gedminas
-- 
Key emulation:
[ ] Intuitive
[*] Emacs        (Seen in an MCEdit dialog) 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/a7c50482/attachment.sig>

From donald at stufft.io  Fri Mar  7 13:47:21 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 7 Mar 2014 07:47:21 -0500
Subject: [Distutils] Warehouse XMLRPC
In-Reply-To: <20140307124524.GB11311@fridge.pov.lt>
References: <73CB5E6A-D20D-465A-9994-A0C2F896DC68@stufft.io>
 <CAPkN8x+qp5DdTSsmfnusBBrLZ8p4yxQj47qE7Xz2cAYzm6nSUQ@mail.gmail.com>
 <295309C2-4476-40CC-BB3D-0EE1D452666D@stufft.io>
 <20140306091041.GA24120@fridge.pov.lt>
 <DD3C302F-043B-48ED-AEDA-ACF162922041@stufft.io>
 <0C5F8655-6340-4716-B8EF-6C25F7596FF7@stufft.io>
 <20140307124524.GB11311@fridge.pov.lt>
Message-ID: <A2905ACF-0F3A-453B-890D-8F3EF91109B1@stufft.io>


On Mar 7, 2014, at 7:45 AM, Marius Gedminas <marius at pov.lt> wrote:

> On Fri, Mar 07, 2014 at 01:03:23AM -0500, Donald Stufft wrote:
>> The JSON API is functional again and is available for testing!
> 
> Excellent!
> 
> I can observe two differences with the current PyPI code:
> 
> First, the Content-Type of the response differs:
> 
>  $ curl -sI https://pypi.python.org/pypi/setuptools/json | grep Content-Type
>  Content-Type: application/json; charset="UTF-8"
> 
>  $ curl -sI https://pypi-preview.a.ssl.fastly.net/pypi/setuptools/json | grep Content-Type
>  Content-Type: application/json
> 
> (I don't know which is correct, or whether the difference actually
> matters.)

Most things will accept the first, however the Warehouse style is correct. application/json
is always UTF8 and doesn?t accept a charset parameter.

> 
> Second, the current PyPI codebase returns absolute URLs, while Warehouse
> returns paths relative to the site root:
> 
>  $ curl -s https://pypi.python.org/pypi/setuptools/json | grep '"url":'
>  "url": "https://pypi.python.org/packages/3.4/s/setuptools/setuptools-2.2-py2.py3-none-any.whl", 
>  "url": "https://pypi.python.org/packages/source/s/setuptools/setuptools-2.2.tar.gz", 
> 
>  $ curl -s https://pypi-preview.a.ssl.fastly.net/pypi/setuptools/json | python -m json.tool | grep '"url":'
>  "url": "/packages/3.4/s/setuptools/setuptools-2.2-py2.py3-none-any.whl"
>  "url": "/packages/source/s/setuptools/setuptools-2.2.tar.gz"
> 
> I can adapt my script to cope.

Yea I already noted that they were relative, I think they should be absolute and I had
already filed it under https://github.com/pypa/warehouse/issues/233 .

> 
> Marius Gedminas
> -- 
> Key emulation:
> [ ] Intuitive
> [*] Emacs        (Seen in an MCEdit dialog) 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/d312248c/attachment.sig>

From squeaky_pl at gmx.com  Fri Mar  7 13:45:20 2014
From: squeaky_pl at gmx.com (Squeaky)
Date: Fri, 07 Mar 2014 13:45:20 +0100
Subject: [Distutils] recommended setuptools version
In-Reply-To: <1394186579.18044.YahooMailNeo@web164703.mail.gq1.yahoo.com>
References: <1394186579.18044.YahooMailNeo@web164703.mail.gq1.yahoo.com>
Message-ID: <5319BF60.5090907@gmx.com>

Hi,

You can see the detailed CHANGELOG here: 
https://bitbucket.org/pypa/setuptools/src/1041039c719012ca9e466cd43ba3fa80f0bd8f3b/CHANGES.txt?at=default

There were some deprecated things dropped. If you are not using them you 
are probably fine.

Squeaky

On 03/07/2014 11:02 AM, Benedek Zoltan wrote:
> Hi,
>
> I'm a bit confused with the setuptools versioning.
> When I install it by ez_setup.py it installs the 3.0.2 version, but on 
> pypi the latest version is 2.2.
>
> Is there a difference between the two? Which is the recommended one?
>
> Thanks
> Zoltan
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/038fefad/attachment.html>

From marius at pov.lt  Fri Mar  7 14:15:17 2014
From: marius at pov.lt (Marius Gedminas)
Date: Fri, 7 Mar 2014 15:15:17 +0200
Subject: [Distutils] recommended setuptools version
In-Reply-To: <1394186579.18044.YahooMailNeo@web164703.mail.gq1.yahoo.com>
References: <1394186579.18044.YahooMailNeo@web164703.mail.gq1.yahoo.com>
Message-ID: <20140307131517.GC11311@fridge.pov.lt>

On Fri, Mar 07, 2014 at 02:02:59AM -0800, Benedek Zoltan wrote:
> I'm a bit confused with the setuptools versioning.
> When I install it by ez_setup.py it installs the 3.0.2 version, but on pypi the latest version is 2.2.
> 
> Is there a difference between the two? Which is the recommended one?

3.0 was released last night, broke a bunch of popular software, and was
hurriedly pulled from PyPI.

The current version of ez_setup.py available at
https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
also installs 2.2.

I suggest you stick with 2.2 until the dust settles.

Marius Gedminas
-- 
2B OR NOT 2B == FF
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/0eb7bc6c/attachment-0001.sig>

From wickman at gmail.com  Fri Mar  7 18:50:39 2014
From: wickman at gmail.com (Brian Wickman)
Date: Fri, 7 Mar 2014 09:50:39 -0800
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
Message-ID: <CAAmsm4i5izhHJb3rj9oy8YYEeU2tAS1BW43vQ3wUYZ8hoMk=Ag@mail.gmail.com>

I've also run into similar issues.  What I do with PEX is fudge PEP425 tags
for OS X in order to be more correct:
https://github.com/wickman/commons/blob/wickman/pep425/src/python/twitter/common/python/pep425.py

The current version on master (not wheel-aware) does it slightly
differently for eggs, but it's exactly the same idea:
https://github.com/twitter/commons/blob/master/src/python/twitter/common/python/platforms.py

I'd love if some variation of this code could be added to setuptools or
whatever.

~brian



On Thu, Mar 6, 2014 at 4:27 PM, MinRK <benjaminrk at gmail.com> wrote:

> Hello,
>
> I proposed a patch <https://github.com/pypa/pip/pull/1465> to pip, with
> respect to treatment of the platform tag on OS X, and Chris Barker proposed
> that I bring the discussion here.
>
> The situation:
>
> PEP 425 describes the platform tag as:
>
> The platform tag is simply distutils.util.get*platform() with all hyphens
> - and periods . replaced with underscore *.
>
> but the PEP makes no mention of what should be done on OS X. On OS X,
> get_platform() has the form:
>
> macosx_10_6_intel
>
> where 10_6 indicates that the OS X deployment target (of Python) is 10.6,
> and intel indicates that it is a fat binary with both x86_64 and i386.
> Other values for the arch include 'ppc', 'ppc64', and 'universal', another
> multi-arch tag indicating a fat binary with all four of 32/64bit intel and
> ppc.
>
> Where PEP 425 causes problems (or pip's implementation thereof), is in
> testing strict equality of this value, not taking into account the meaning
> of its contents. I proposed two changes:
>
>    1.
>
>    Use >= comparison for the deployment target
>
>    From the OS X platform documentation<https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/cross_development/Configuring/configuring.html>
>    :
>
>    Choose a deployment target. This identifies the earliest OS version on
>    which your software can run.
>
>    so a binary with deployment target = 10.6 should be installable on OS
>    X >= 10.6.
>
>    This change restores the longstanding behavior of easy_install, where
>    10.6 eggs are installable on >= 10.6.
>     2.
>
>    support multi-arch names (intel, universal) on their respective
>    components
>     - intel is valid on {x86_64, i386}
>       - universal is valid on {intel, x86_64, i386, ppc64, ppc}
>
>    easy_install, like pip, also does strict comparison here, so this
>    would be new behavior.
>
> My actual use case:
>
> I have a wheel (pyzmq), which works on any intel-based Python targeting OS
> X >= 10.6. To express this with pip-1.5, the filename has to be:
>
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.macosx_10_7_intel.macosx_10_7_x86_64.macosx_10_7_i386.macosx_10_8_intel.macosx_10_8_x86_64.macosx_10_8_i386.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_9_i386.whl
>
> and it has to grow every time there is a new OS release.
>
> With both proposed changes, I could communicate the same information with
> just one tag:
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.whl
>
> With just the first change, it would need three:
>
>
> pyzmq-14.1.0-cp27-none-macosx_10_6_intel.macosx_10_6_x86_64.macosx_10_6_i386.whl
>
> Any feedback from experts, especially if my understanding of deployment
> targets or fat binaries is incorrect, would be much appreciated.
>
> Thanks,
>
> -MinRK
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/3f8ebdd9/attachment.html>

From chris.barker at noaa.gov  Fri Mar  7 20:53:25 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 7 Mar 2014 11:53:25 -0800
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CAAmsm4i5izhHJb3rj9oy8YYEeU2tAS1BW43vQ3wUYZ8hoMk=Ag@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
 <CAAmsm4i5izhHJb3rj9oy8YYEeU2tAS1BW43vQ3wUYZ8hoMk=Ag@mail.gmail.com>
Message-ID: <CALGmxEJabK6KAaVyTHoZGRxOvnvsmDB-BvraqVOaFvhzto5iaQ@mail.gmail.com>

On Fri, Mar 7, 2014 at 9:50 AM, Brian Wickman <wickman at gmail.com> wrote:

> I've also run into similar issues.  What I do with PEX is fudge PEP425
> tags for OS X in order to be more correct:
>
> https://github.com/wickman/commons/blob/wickman/pep425/src/python/twitter/common/python/pep425.py
>
>

> I'd love if some variation of this code could be added to setuptools or
> whatever.
>
>
+1 This looks good to me.

In theory, this logic should really go with python itself, not a
third-party lib. i.e., once built, a Python implementation will match
particular tags -- having a third party lib keep track of that seems to be
putting it in the wrong place.

In practice, there are only so many versions out there, and we're only
trying to support some of them, so it's probably fine for it to be in pip
(and it can be upgraded so much more easily there)

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/9ae6fed3/attachment.html>

From ncoghlan at gmail.com  Sat Mar  8 05:11:38 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 8 Mar 2014 14:11:38 +1000
Subject: [Distutils] OS X and PEP 425 / wheels
In-Reply-To: <CALGmxEJabK6KAaVyTHoZGRxOvnvsmDB-BvraqVOaFvhzto5iaQ@mail.gmail.com>
References: <CAHNn8BW+UfQRFqsuszq81WHM+xthEb-RP-D9Na0TktTgjPHCyw@mail.gmail.com>
 <CAAmsm4i5izhHJb3rj9oy8YYEeU2tAS1BW43vQ3wUYZ8hoMk=Ag@mail.gmail.com>
 <CALGmxEJabK6KAaVyTHoZGRxOvnvsmDB-BvraqVOaFvhzto5iaQ@mail.gmail.com>
Message-ID: <CADiSq7exLM8nh5aRcFyTP=kV_qNOn9GRsmZp+FNbFHGaptE=+A@mail.gmail.com>

On 8 March 2014 05:53, Chris Barker <chris.barker at noaa.gov> wrote:
> +1 This looks good to me.
>
> In theory, this logic should really go with python itself, not a third-party
> lib. i.e., once built, a Python implementation will match particular tags --
> having a third party lib keep track of that seems to be putting it in the
> wrong place.
>
> In practice, there are only so many versions out there, and we're only
> trying to support some of them, so it's probably fine for it to be in pip
> (and it can be upgraded so much more easily there)

The problem is that some platforms change faster than Python does
(particularly when people are running older Python versions in
production), so we've learned that coupling a lot of these details to
the standard library doesn't work. There's also the fact that we're
trying to retrofit capabilities to older versions of Python that are
no longer receiving new features (2.6, 2.7, 3.2, 3.3).

You see similar behaviour from the platform vendors themselves.
Microsoft Visual Studio and Red Hat's Developer Toolset, for example,
are both versioned independently from the underlying OS.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From wickman at gmail.com  Sat Mar  8 05:15:20 2014
From: wickman at gmail.com (Brian Wickman)
Date: Fri, 7 Mar 2014 20:15:20 -0800
Subject: [Distutils] bizarre behavior with wheels and namespace packages
Message-ID: <CAAmsm4inopsTtKOFyi8kwWg7duBT+A1J8H-a-GURNK6UPyFHfA@mail.gmail.com>

Not sure if this belongs on distutils-sig or import-sig, but I'm
experiencing slightly odd behavior with wheels and namespace packages.

Fetch a namespace package source distribution, e.g. twitter.common.python,
and build two copies of it, one with bdist_egg and one with bdist_wheel.

This leaves me with:
twitter.common.python-0.3.1-py2.6.egg
twitter.common.python-0.3.1-py26-none-any.whl

Explode them into the following directories (respectively):
egg_dist
whl_dist

Then attempt:

PYTHONPATH=egg_dist python -c 'import twitter.common.python'

(works)

PYTHONPATH=whl_dist python -c 'import twitter.common.python'

(drumroll)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named twitter.common.python

This is slightly unexpected.  Absent from the .whl distribution are the
__init__.pys with the proper declare_namespace declarations.  Even more
bizarrely:

PYTHONPATH=whl_dist python -c 'import pkg_resources'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File
"/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
line 2736, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File
"/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
line 698, in subscribe
    callback(dist)
  File
"/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
line 2736, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File
"/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
line 2274, in activate
    list(map(declare_namespace,
self._get_metadata('namespace_packages.txt')))
  File
"/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
line 1870, in declare_namespace
    path = sys.modules[parent].__path__
KeyError: 'twitter'

So even pkg_resources is surprised as well.  If I do the following:

PYTHONPATH=whl_dist python -c 'import site; site.addsitedir("whl_dist");
import twitter.common.python'

then it works, since the twitter.common.python-0.3.1-py2.6-nspkg.pth gets
processed properly and module.__path__s populated accordingly.  I would
expect pkg_resources to handle this fine but apparently it doesn't.

Any ideas?  Bug in pkg_resources, wheel or PEBKAC?

Thanks!
~brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140307/50a80b75/attachment-0001.html>

From jaraco at jaraco.com  Sat Mar  8 15:02:57 2014
From: jaraco at jaraco.com (Jason R. Coombs)
Date: Sat, 8 Mar 2014 14:02:57 +0000
Subject: [Distutils] recommended setuptools version
In-Reply-To: <20140307131517.GC11311@fridge.pov.lt>
References: <1394186579.18044.YahooMailNeo@web164703.mail.gq1.yahoo.com>
 <20140307131517.GC11311@fridge.pov.lt>
Message-ID: <49897a3980614325bc3ee9800c9b4f11@BLUPR06MB434.namprd06.prod.outlook.com>

To follow up, 3.1 has been released and the 3.0.x releases have been restored.

> -----Original Message-----
> From: Distutils-SIG [mailto:distutils-sig-
> bounces+jaraco=jaraco.com at python.org] On Behalf Of Marius Gedminas
> Sent: Friday, 07 March, 2014 08:15
> To: distutils-sig at python.org
> Subject: Re: [Distutils] recommended setuptools version
> 
> On Fri, Mar 07, 2014 at 02:02:59AM -0800, Benedek Zoltan wrote:
> > I'm a bit confused with the setuptools versioning.
> > When I install it by ez_setup.py it installs the 3.0.2 version, but on pypi the
> latest version is 2.2.
> >
> > Is there a difference between the two? Which is the recommended one?
> 
> 3.0 was released last night, broke a bunch of popular software, and was
> hurriedly pulled from PyPI.
> 
> The current version of ez_setup.py available at
> https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
> also installs 2.2.
> 
> I suggest you stick with 2.2 until the dust settles.
> 
> Marius Gedminas
> --
> 2B OR NOT 2B == FF

From jaraco at jaraco.com  Sat Mar  8 15:10:47 2014
From: jaraco at jaraco.com (Jason R. Coombs)
Date: Sat, 8 Mar 2014 14:10:47 +0000
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
Message-ID: <5d659acedc034e4ab57ae22571f166f5@BLUPR06MB434.namprd06.prod.outlook.com>

Thanks Mike for the report. I haven't read the rest of this thread yet, and I want to respond to your OP before I do. As a result, there will likely be some duplication.

You're absolutely right to be dismayed.

I had incorrectly assessed the potential impact of this change. I had expected the impact to be minor and isolated. I underestimated the number of systems that automatically upgrade Setuptools as well as the number of packages that could be using Features. I relied too heavily on the DeprecationWarning to have flagged the breakage (to package maintainers and their users). I had expected issues like you raised in this post to have been hashed out or at least raised long before I considered the release.

I hope my decision to backout the release serves as a strong signal that this type of breakage is unacceptable. I hope the tickets filed and discussion will lead to a better mechanism for communicating these types of changes in the future. In the future, I will leverage Twitter to announce in advance any backward-incompatible releases, giving additional visibility and oversight.

I do also want to ask that deployment teams consider the implications of always upgrading to the latest release of Setuptools. I'm encouraged that they do, and I do want to keep that as a viable deployment workflow. However, at the same time, there is some inherent risk with automatically upgrading to known-backward-breaking versions. The purpose of adopting semver for versioning is to allow the project to signal directly and unambiguously when a release is backward-breaking. I haven't yet decided when or where to raise this discussion, and it certainly doesn't excuse the challenges faced Thursday.

It's clear to me from your post that there are use-cases for Features that aren't covered by simply "using extras". Namely, extras doesn't support exclusion, only inclusion. I'll make mention of that in the deprecation ticket (https://bitbucket.org/pypa/setuptools/issue/65/) and follow up after understanding the problem better.

I'm the one that needs to apologize, not you. Thanks for your patience and understanding as I and the rest of the PyPA work toward the best means of communication (that doesn't require every package maintainer to follow distutils-sig).

Best regards,
Jason

> -----Original Message-----
> From: Distutils-SIG [mailto:distutils-sig-
> bounces+jaraco=jaraco.com at python.org] On Behalf Of Michael Bayer
> Sent: Thursday, 06 March, 2014 12:56
> To: distutils-sig at python.org
> Subject: [Distutils] tourist here, with a dumb RTFM question
> 
> Hi Distutils !
> 
> I don't follow this list and haven't looked at it in a long time.   However, I'm
> learning via twitter that a brand new setuptools release that's just gone out
> has just removed the "Feature" mechanism.
> 
> Now as you're all rolling your eyes and preparing to bang out frustrated
> replies how this was well announced and deprecated and warned about and
> wow really didn't you know the term paper was due today, OK first off let me
> say I am sorry!  I am not a distutils/setuptools maintainer.    I am just
> someone that uses it, and as I include setuptools in my setup.py, I am also
> getting thousands of other people who download my product to use it as
> well.    And I don't read the setuptools changelog!  Or the setuptools blog.  Or
> this list.  I assume you guys have it under control (and you certainly do!).
> There seem to be other people like me (people who write very widely used
> software) who also seem surprised!   And that is surprising too (as I am
> usually the only person to be surprised by these things that should not be
> surprising).  So I hope you can hold back your frustration with my clueless
> RTFMness long enough to answer these questions:
> 
> 1. where was this announced?   I'm wondering why there weren't loud
> blaring blog posts and tweets all over the place stating that on March 6,
> dozens of major packages are going to all break.
> 
> 2. what is the plan for unmaintained packages and old versions of existing
> packages on pypi that "import Feature" and can no longer be installed?    Is it
> just the case that a large amount of pypi just won't install anymore?
> 
> 3. What, if any, is the recommended approach going forward to a Python
> package that wishes to specify a command-line flag during install.  Here's
> multiple choice:
> 
>     a. you can use new setuptools API <some new way to add -flags>
> 
>     b. you can roll it yourself in setup.py using <hack X>
> 
>     c. your setup.py should never accept any kind of flags as that interferes
> with <up-and-coming use case Q>
> 
>     d. other
> 
> If choice c., then here is another question.  My library includes optional C
> extensions.   On certain platforms, these C extensions don't build (like on
> Pypy, or on Windows if you don't have compilation tools installed).  In this
> regard it gracefully degrades.  But also, I want to be able to have a command
> line option to determine this as well.  Because!  Maybe I'm installing within a
> test suite where I need to test the entire library without the C extensions
> built.    Maybe some user has found a bug in the C extensions, and that user
> needs to temporarily install the tool without the extensions built.    Other
> cases for flags are, maybe your library can build with or without SSL support,
> something like that.
> 
> Keep in mind, I actually *won't* be getting bug reports about this because
> my setup.py already gracefully degrades to distutils!  But still, I'd like to have
> my -without-cextensions flag somehow.
> 
> Thanks for listening and again I apologize for not following the setuptools
> changelog on a regular basis!
> 
> 
> - mike
> 
> 
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From jaraco at jaraco.com  Sat Mar  8 15:19:36 2014
From: jaraco at jaraco.com (Jason R. Coombs)
Date: Sat, 8 Mar 2014 14:19:36 +0000
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
Message-ID: <c01b3d929be84f3f965a486370bd0286@BLUPR06MB434.namprd06.prod.outlook.com>



From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Marcus Smith
Sent: Thursday, 06 March, 2014 16:16
To: DistUtils mailing list
Subject: Re: [Distutils] tourist here, with a dumb RTFM question



On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com<mailto:mike_mp at zzzcomputing.com>> wrote:

On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com<mailto:dholth at gmail.com>> wrote:

> pje said:
>
> The "Feature()" facility was never completely implemented or
> supported, and even if it were, it should be deprecated now, as it
> will not be compatible with the coming packaging systems based on PEP
> 426. If you need separate features, use separate distributions and
> "extras" instead.
wait, ok this is that thing.  "separate distributions" means....


SQLAlchemy-0.9.3-nocext.tar.gz
SQLAlchemy-0.9.3-cext.tar.gz

I'm new to understanding setuptools "Feature" myself, so don't crank too much, if I get anything wrong.

afaik, it means something like:

SQLAlchemy-X.Y.tar.gz
SQLAlchemy-cext-X.Y.tar.gz

i.e. isolating the feature into a separate project.

and then instead of having the handy  "--with-cext" option,  it has to be  "pip install SQLAlchemy[SQLAlchemy-cext]

PJ, Jason: can you clarify what the alternative is supposed to look like?

I'm new to understanding Features as well, but based on PJ's post, I would agree with Marcus here. This is how I interpreted the use of extras to supersede Features. I see a lot of advantages to this approach over build-time selection. Because it decouples the main project from the C-extension speedups, it gives the installer control over what behavior is present. It also gives deployment tools visibility over what capability is present (with features, it's harder to tell if the C-extension speedups are present; with extras, one can query pkg_resources).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140308/ff49922a/attachment-0001.html>

From mike_mp at zzzcomputing.com  Sat Mar  8 15:53:02 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Sat, 8 Mar 2014 09:53:02 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <c01b3d929be84f3f965a486370bd0286@BLUPR06MB434.namprd06.prod.outlook.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
 <c01b3d929be84f3f965a486370bd0286@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <E31506E4-A4C1-4A58-A1DB-1469B6A16FA5@zzzcomputing.com>


On Mar 8, 2014, at 9:19 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:

>  
>  
> From: Distutils-SIG [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Marcus Smith
> Sent: Thursday, 06 March, 2014 16:16
> To: DistUtils mailing list
> Subject: Re: [Distutils] tourist here, with a dumb RTFM question
>  
>  
>  
> 
> On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:
> 
> On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
> 
> > pje said:
> >
> > The "Feature()" facility was never completely implemented or
> > supported, and even if it were, it should be deprecated now, as it
> > will not be compatible with the coming packaging systems based on PEP
> > 426. If you need separate features, use separate distributions and
> > "extras" instead.
> 
> wait, ok this is that thing.  ?separate distributions? means?.
> 
> 
> SQLAlchemy-0.9.3-nocext.tar.gz
> SQLAlchemy-0.9.3-cext.tar.gz
>  
> I'm new to understanding setuptools "Feature" myself, so don't crank too much, if I get anything wrong.
>  
> afaik, it means something like:
>  
> SQLAlchemy-X.Y.tar.gz
> SQLAlchemy-cext-X.Y.tar.gz
>  
> i.e. isolating the feature into a separate project.
>  
> and then instead of having the handy  "--with-cext" option,  it has to be  "pip install SQLAlchemy[SQLAlchemy-cext]
>  
> PJ, Jason: can you clarify what the alternative is supposed to look like?
>  
> I?m new to understanding Features as well, but based on PJ?s post, I would agree with Marcus here. This is how I interpreted the use of extras to supersede Features. I see a lot of advantages to this approach over build-time selection. Because it decouples the main project from the C-extension speedups, it gives the installer control over what behavior is present. It also gives deployment tools visibility over what capability is present (with features, it?s harder to tell if the C-extension speedups are present; with extras, one can query pkg_resources).


If there?s an easy way to do this from setup.py it wouldn?t be so bad:

python setup.py ?without-cext sdist upload   

python setup.py sdist upload

but it still seems kind of strange to deliver the exact same source files with some flag somewhere different?   I?d assume that flag is in setup.cfg, which means, it?s a setup.py flag in any case!   not to mention its still needed as part of the ?sdist? and other commands to build dists.

I think the issue of ?you should have individual projects on pypi with different options to make life easier for builders? is orthogonal to ?setup.py should or should not support flags?.










>  
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140308/0b32f926/attachment.html>

From mike_mp at zzzcomputing.com  Sat Mar  8 15:53:39 2014
From: mike_mp at zzzcomputing.com (Michael Bayer)
Date: Sat, 8 Mar 2014 09:53:39 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <5d659acedc034e4ab57ae22571f166f5@BLUPR06MB434.namprd06.prod.outlook.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <5d659acedc034e4ab57ae22571f166f5@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <908F9C52-DB45-4F77-B4D8-CA9B729F6F53@zzzcomputing.com>

Thanks for this response, Jason!    Your thoughtfulness is appreciated.

- mike




On Mar 8, 2014, at 9:10 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:

> Thanks Mike for the report. I haven't read the rest of this thread yet, and I want to respond to your OP before I do. As a result, there will likely be some duplication.
> 
> You're absolutely right to be dismayed.
> 
> I had incorrectly assessed the potential impact of this change. I had expected the impact to be minor and isolated. I underestimated the number of systems that automatically upgrade Setuptools as well as the number of packages that could be using Features. I relied too heavily on the DeprecationWarning to have flagged the breakage (to package maintainers and their users). I had expected issues like you raised in this post to have been hashed out or at least raised long before I considered the release.
> 
> I hope my decision to backout the release serves as a strong signal that this type of breakage is unacceptable. I hope the tickets filed and discussion will lead to a better mechanism for communicating these types of changes in the future. In the future, I will leverage Twitter to announce in advance any backward-incompatible releases, giving additional visibility and oversight.
> 
> I do also want to ask that deployment teams consider the implications of always upgrading to the latest release of Setuptools. I'm encouraged that they do, and I do want to keep that as a viable deployment workflow. However, at the same time, there is some inherent risk with automatically upgrading to known-backward-breaking versions. The purpose of adopting semver for versioning is to allow the project to signal directly and unambiguously when a release is backward-breaking. I haven't yet decided when or where to raise this discussion, and it certainly doesn't excuse the challenges faced Thursday.
> 
> It's clear to me from your post that there are use-cases for Features that aren't covered by simply "using extras". Namely, extras doesn't support exclusion, only inclusion. I'll make mention of that in the deprecation ticket (https://bitbucket.org/pypa/setuptools/issue/65/) and follow up after understanding the problem better.
> 
> I'm the one that needs to apologize, not you. Thanks for your patience and understanding as I and the rest of the PyPA work toward the best means of communication (that doesn't require every package maintainer to follow distutils-sig).
> 
> Best regards,
> Jason
> 
>> -----Original Message-----
>> From: Distutils-SIG [mailto:distutils-sig-
>> bounces+jaraco=jaraco.com at python.org] On Behalf Of Michael Bayer
>> Sent: Thursday, 06 March, 2014 12:56
>> To: distutils-sig at python.org
>> Subject: [Distutils] tourist here, with a dumb RTFM question
>> 
>> Hi Distutils !
>> 
>> I don't follow this list and haven't looked at it in a long time.   However, I'm
>> learning via twitter that a brand new setuptools release that's just gone out
>> has just removed the "Feature" mechanism.
>> 
>> Now as you're all rolling your eyes and preparing to bang out frustrated
>> replies how this was well announced and deprecated and warned about and
>> wow really didn't you know the term paper was due today, OK first off let me
>> say I am sorry!  I am not a distutils/setuptools maintainer.    I am just
>> someone that uses it, and as I include setuptools in my setup.py, I am also
>> getting thousands of other people who download my product to use it as
>> well.    And I don't read the setuptools changelog!  Or the setuptools blog.  Or
>> this list.  I assume you guys have it under control (and you certainly do!).
>> There seem to be other people like me (people who write very widely used
>> software) who also seem surprised!   And that is surprising too (as I am
>> usually the only person to be surprised by these things that should not be
>> surprising).  So I hope you can hold back your frustration with my clueless
>> RTFMness long enough to answer these questions:
>> 
>> 1. where was this announced?   I'm wondering why there weren't loud
>> blaring blog posts and tweets all over the place stating that on March 6,
>> dozens of major packages are going to all break.
>> 
>> 2. what is the plan for unmaintained packages and old versions of existing
>> packages on pypi that "import Feature" and can no longer be installed?    Is it
>> just the case that a large amount of pypi just won't install anymore?
>> 
>> 3. What, if any, is the recommended approach going forward to a Python
>> package that wishes to specify a command-line flag during install.  Here's
>> multiple choice:
>> 
>>    a. you can use new setuptools API <some new way to add -flags>
>> 
>>    b. you can roll it yourself in setup.py using <hack X>
>> 
>>    c. your setup.py should never accept any kind of flags as that interferes
>> with <up-and-coming use case Q>
>> 
>>    d. other
>> 
>> If choice c., then here is another question.  My library includes optional C
>> extensions.   On certain platforms, these C extensions don't build (like on
>> Pypy, or on Windows if you don't have compilation tools installed).  In this
>> regard it gracefully degrades.  But also, I want to be able to have a command
>> line option to determine this as well.  Because!  Maybe I'm installing within a
>> test suite where I need to test the entire library without the C extensions
>> built.    Maybe some user has found a bug in the C extensions, and that user
>> needs to temporarily install the tool without the extensions built.    Other
>> cases for flags are, maybe your library can build with or without SSL support,
>> something like that.
>> 
>> Keep in mind, I actually *won't* be getting bug reports about this because
>> my setup.py already gracefully degrades to distutils!  But still, I'd like to have
>> my -without-cextensions flag somehow.
>> 
>> Thanks for listening and again I apologize for not following the setuptools
>> changelog on a regular basis!
>> 
>> 
>> - mike
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig


From ncoghlan at gmail.com  Sat Mar  8 16:20:49 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 9 Mar 2014 01:20:49 +1000
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <5d659acedc034e4ab57ae22571f166f5@BLUPR06MB434.namprd06.prod.outlook.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <5d659acedc034e4ab57ae22571f166f5@BLUPR06MB434.namprd06.prod.outlook.com>
Message-ID: <CADiSq7f6LMezOWGP0VVa0MfAHZ0m=b49KL5UdgF4oA7Mu76okw@mail.gmail.com>

On 9 Mar 2014 00:26, "Jason R. Coombs" <jaraco at jaraco.com> wrote:
>
> I had incorrectly assessed the potential impact of this change. I had
expected the impact to be minor and isolated. I underestimated the number
of systems that automatically upgrade Setuptools as well as the number of
packages that could be using Features. I relied too heavily on the
DeprecationWarning to have flagged the breakage (to package maintainers and
their users). I had expected issues like you raised in this post to have
been hashed out or at least raised long before I considered the release.

Ouch, I just realised this is likely a nasty consequence of the decision we
made in CPython quite some time ago to silence deprecation warnings by
default.

The reason we did that was because *end users* were getting the warnings,
not just developers of affected projects.

At the same time, unittest and other test runners were updated to enable
the warnings, under the theory that at least projects with reasonable test
coverage would still receive advance notice of the problem, providing a
counterbalance to the risks introduced by silencing them by default.

Perhaps once wheel files are more widespread, we can look at having pip
always enable deprecation warnings when building from source.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140309/2d2f916a/attachment.html>

From ncoghlan at gmail.com  Sat Mar  8 16:24:12 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 9 Mar 2014 01:24:12 +1000
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CADiSq7f6LMezOWGP0VVa0MfAHZ0m=b49KL5UdgF4oA7Mu76okw@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <5d659acedc034e4ab57ae22571f166f5@BLUPR06MB434.namprd06.prod.outlook.com>
 <CADiSq7f6LMezOWGP0VVa0MfAHZ0m=b49KL5UdgF4oA7Mu76okw@mail.gmail.com>
Message-ID: <CADiSq7ehv0Xb5WVnYZxABV6jvB8hqfFX3Quet-Egc-mpE=_waA@mail.gmail.com>

On 9 Mar 2014 01:20, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>
> On 9 Mar 2014 00:26, "Jason R. Coombs" <jaraco at jaraco.com> wrote:
> >
> > I had incorrectly assessed the potential impact of this change. I had
expected the impact to be minor and isolated. I underestimated the number
of systems that automatically upgrade Setuptools as well as the number of
packages that could be using Features. I relied too heavily on the
DeprecationWarning to have flagged the breakage (to package maintainers and
their users). I had expected issues like you raised in this post to have
been hashed out or at least raised long before I considered the release.
>
> Ouch, I just realised this is likely a nasty consequence of the decision
we made in CPython quite some time ago to silence deprecation warnings by
default.
>
> The reason we did that was because *end users* were getting the warnings,
not just developers of affected projects.
>
> At the same time, unittest and other test runners were updated to enable
the warnings, under the theory that at least projects with reasonable test
coverage would still receive advance notice of the problem, providing a
counterbalance to the risks introduced by silencing them by default.

Oops, missed a key paragraph here: the reason this is a problem for
setuptools deprecations in particular is that most developers are unlikely
to run their setup.py file under the control of a test runner that would
enable the warning.

>
> Perhaps once wheel files are more widespread, we can look at having pip
always enable deprecation warnings when building from source.
>
> Cheers,
> Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140309/9716714e/attachment.html>

From dholth at gmail.com  Sat Mar  8 16:49:14 2014
From: dholth at gmail.com (Daniel Holth)
Date: Sat, 8 Mar 2014 10:49:14 -0500
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <E31506E4-A4C1-4A58-A1DB-1469B6A16FA5@zzzcomputing.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
 <c01b3d929be84f3f965a486370bd0286@BLUPR06MB434.namprd06.prod.outlook.com>
 <E31506E4-A4C1-4A58-A1DB-1469B6A16FA5@zzzcomputing.com>
Message-ID: <CAG8k2+5489yQwj0DKOYPKjB8BxnhaSS5gwL=C-N4+oiQL_wWvw@mail.gmail.com>

Some packaging systems do not have the same 1:1 source : distribution
relationship that is so prominent in the distutils model. This should
probably change long term.

On Sat, Mar 8, 2014 at 9:53 AM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:
>
> On Mar 8, 2014, at 9:19 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:
>
>
>
> From: Distutils-SIG
> [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of
> Marcus Smith
> Sent: Thursday, 06 March, 2014 16:16
> To: DistUtils mailing list
> Subject: Re: [Distutils] tourist here, with a dumb RTFM question
>
>
>
>
>
> On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com>
> wrote:
>
>
> On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
>
>> pje said:
>>
>> The "Feature()" facility was never completely implemented or
>> supported, and even if it were, it should be deprecated now, as it
>> will not be compatible with the coming packaging systems based on PEP
>> 426. If you need separate features, use separate distributions and
>> "extras" instead.
>
> wait, ok this is that thing.  "separate distributions" means....
>
>
> SQLAlchemy-0.9.3-nocext.tar.gz
> SQLAlchemy-0.9.3-cext.tar.gz
>
>
> I'm new to understanding setuptools "Feature" myself, so don't crank too
> much, if I get anything wrong.
>
> afaik, it means something like:
>
> SQLAlchemy-X.Y.tar.gz
> SQLAlchemy-cext-X.Y.tar.gz
>
> i.e. isolating the feature into a separate project.
>
> and then instead of having the handy  "--with-cext" option,  it has to be
> "pip install SQLAlchemy[SQLAlchemy-cext]
>
> PJ, Jason: can you clarify what the alternative is supposed to look like?
>
> I'm new to understanding Features as well, but based on PJ's post, I would
> agree with Marcus here. This is how I interpreted the use of extras to
> supersede Features. I see a lot of advantages to this approach over
> build-time selection. Because it decouples the main project from the
> C-extension speedups, it gives the installer control over what behavior is
> present. It also gives deployment tools visibility over what capability is
> present (with features, it's harder to tell if the C-extension speedups are
> present; with extras, one can query pkg_resources).
>
>
>
> If there's an easy way to do this from setup.py it wouldn't be so bad:
>
> python setup.py --without-cext sdist upload
>
> python setup.py sdist upload
>
> but it still seems kind of strange to deliver the exact same source files
> with some flag somewhere different?   I'd assume that flag is in setup.cfg,
> which means, it's a setup.py flag in any case!   not to mention its still
> needed as part of the "sdist" and other commands to build dists.
>
> I think the issue of "you should have individual projects on pypi with
> different options to make life easier for builders" is orthogonal to
> "setup.py should or should not support flags".
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From wickman at gmail.com  Sat Mar  8 20:04:48 2014
From: wickman at gmail.com (Brian Wickman)
Date: Sat, 8 Mar 2014 11:04:48 -0800
Subject: [Distutils] bizarre behavior with wheels and namespace packages
In-Reply-To: <CAAmsm4inopsTtKOFyi8kwWg7duBT+A1J8H-a-GURNK6UPyFHfA@mail.gmail.com>
References: <CAAmsm4inopsTtKOFyi8kwWg7duBT+A1J8H-a-GURNK6UPyFHfA@mail.gmail.com>
Message-ID: <CAAmsm4iAjp-YNaFegRfmB_1o3SkgGGd3TvAG3-RWd7KSUoAy_g@mail.gmail.com>

More data:

This works fine with python 3

PYTHONPATH=whl_dist python3 -c 'import twitter.common.python'

but pkg_resource still barfs

PYTHONPATH=whl_dist python3 -c 'import pkg_resources'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1532, in
_find_and_load_unlocked
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2725, in <module>
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 681, in subscribe
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2725, in <lambda>
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 2259, in activate
  File
"/Users/wickman/Python/CPython-3.3.3/lib/python3.3/site-packages/setuptools-2.0.2-py3.3.egg/pkg_resources.py",
line 1843, in declare_namespace
KeyError: 'twitter'

I did find a related wheel issue:
https://bitbucket.org/pypa/wheel/issue/46/delete-namespace-__init__py-based-on

But this seems to describe the opposite behavior, i.e. that the installer
should be deleting __init__.pys instead of creating them.

In the meantime, my two options are building an installer that populates
__init__.pys, or instead just calling site.addsitedir each time I call
dist.activate().  I'm guessing the latter is preferable as it does not
unnecessarily introduce a pkg_resources dependency.

~brian


On Fri, Mar 7, 2014 at 8:15 PM, Brian Wickman <wickman at gmail.com> wrote:

> Not sure if this belongs on distutils-sig or import-sig, but I'm
> experiencing slightly odd behavior with wheels and namespace packages.
>
> Fetch a namespace package source distribution, e.g. twitter.common.python,
> and build two copies of it, one with bdist_egg and one with bdist_wheel.
>
> This leaves me with:
> twitter.common.python-0.3.1-py2.6.egg
> twitter.common.python-0.3.1-py26-none-any.whl
>
> Explode them into the following directories (respectively):
> egg_dist
> whl_dist
>
> Then attempt:
>
> PYTHONPATH=egg_dist python -c 'import twitter.common.python'
>
> (works)
>
> PYTHONPATH=whl_dist python -c 'import twitter.common.python'
>
> (drumroll)
>
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> ImportError: No module named twitter.common.python
>
> This is slightly unexpected.  Absent from the .whl distribution are the
> __init__.pys with the proper declare_namespace declarations.  Even more
> bizarrely:
>
> PYTHONPATH=whl_dist python -c 'import pkg_resources'
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 2736, in <module>
>     add_activation_listener(lambda dist: dist.activate())
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 698, in subscribe
>     callback(dist)
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 2736, in <lambda>
>     add_activation_listener(lambda dist: dist.activate())
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 2274, in activate
>     list(map(declare_namespace,
> self._get_metadata('namespace_packages.txt')))
>   File
> "/Users/wickman/Python/CPython-2.6.9/lib/python2.6/site-packages/pkg_resources.py",
> line 1870, in declare_namespace
>     path = sys.modules[parent].__path__
> KeyError: 'twitter'
>
> So even pkg_resources is surprised as well.  If I do the following:
>
> PYTHONPATH=whl_dist python -c 'import site; site.addsitedir("whl_dist");
> import twitter.common.python'
>
> then it works, since the twitter.common.python-0.3.1-py2.6-nspkg.pth gets
> processed properly and module.__path__s populated accordingly.  I would
> expect pkg_resources to handle this fine but apparently it doesn't.
>
> Any ideas?  Bug in pkg_resources, wheel or PEBKAC?
>
> Thanks!
> ~brian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140308/ca3d744c/attachment-0001.html>

From ncoghlan at gmail.com  Sun Mar  9 00:32:16 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 9 Mar 2014 09:32:16 +1000
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CAG8k2+5489yQwj0DKOYPKjB8BxnhaSS5gwL=C-N4+oiQL_wWvw@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
 <c01b3d929be84f3f965a486370bd0286@BLUPR06MB434.namprd06.prod.outlook.com>
 <E31506E4-A4C1-4A58-A1DB-1469B6A16FA5@zzzcomputing.com>
 <CAG8k2+5489yQwj0DKOYPKjB8BxnhaSS5gwL=C-N4+oiQL_wWvw@mail.gmail.com>
Message-ID: <CADiSq7dN0u6-5mW==eUq8ZCSQftqeoRKWSW4RzG7H7P6wpb3rw@mail.gmail.com>

On 9 Mar 2014 01:50, "Daniel Holth" <dholth at gmail.com> wrote:
>
> Some packaging systems do not have the same 1:1 source : distribution
> relationship that is so prominent in the distutils model. This should
> probably change long term.

Huh, I hadn't thought about that, and I really should have, since RPM is
one such system where you can get multiple binary RPMs from a single source
RPM. On the other hand, "installing" an SRPM just drops it into the
rpmbuild SRPM directory - it doesn't implicitly build anything.

So yes, I agree that once we get to fixing the build side of things, we may
need to figure out a way to handle multiple binary artifacts coming from a
single source artifact *on the same platform and Python version*.

Cheers,
Nick.
>
> On Sat, Mar 8, 2014 at 9:53 AM, Michael Bayer <mike_mp at zzzcomputing.com>
wrote:
> >
> > On Mar 8, 2014, at 9:19 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:
> >
> >
> >
> > From: Distutils-SIG
> > [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of
> > Marcus Smith
> > Sent: Thursday, 06 March, 2014 16:16
> > To: DistUtils mailing list
> > Subject: Re: [Distutils] tourist here, with a dumb RTFM question
> >
> >
> >
> >
> >
> > On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com
>
> > wrote:
> >
> >
> > On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
> >
> >> pje said:
> >>
> >> The "Feature()" facility was never completely implemented or
> >> supported, and even if it were, it should be deprecated now, as it
> >> will not be compatible with the coming packaging systems based on PEP
> >> 426. If you need separate features, use separate distributions and
> >> "extras" instead.
> >
> > wait, ok this is that thing.  "separate distributions" means....
> >
> >
> > SQLAlchemy-0.9.3-nocext.tar.gz
> > SQLAlchemy-0.9.3-cext.tar.gz
> >
> >
> > I'm new to understanding setuptools "Feature" myself, so don't crank too
> > much, if I get anything wrong.
> >
> > afaik, it means something like:
> >
> > SQLAlchemy-X.Y.tar.gz
> > SQLAlchemy-cext-X.Y.tar.gz
> >
> > i.e. isolating the feature into a separate project.
> >
> > and then instead of having the handy  "--with-cext" option,  it has to
be
> > "pip install SQLAlchemy[SQLAlchemy-cext]
> >
> > PJ, Jason: can you clarify what the alternative is supposed to look
like?
> >
> > I'm new to understanding Features as well, but based on PJ's post, I
would
> > agree with Marcus here. This is how I interpreted the use of extras to
> > supersede Features. I see a lot of advantages to this approach over
> > build-time selection. Because it decouples the main project from the
> > C-extension speedups, it gives the installer control over what behavior
is
> > present. It also gives deployment tools visibility over what capability
is
> > present (with features, it's harder to tell if the C-extension speedups
are
> > present; with extras, one can query pkg_resources).
> >
> >
> >
> > If there's an easy way to do this from setup.py it wouldn't be so bad:
> >
> > python setup.py --without-cext sdist upload
> >
> > python setup.py sdist upload
> >
> > but it still seems kind of strange to deliver the exact same source
files
> > with some flag somewhere different?   I'd assume that flag is in
setup.cfg,
> > which means, it's a setup.py flag in any case!   not to mention its
still
> > needed as part of the "sdist" and other commands to build dists.
> >
> > I think the issue of "you should have individual projects on pypi with
> > different options to make life easier for builders" is orthogonal to
> > "setup.py should or should not support flags".
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140309/3fc89369/attachment.html>

From ct at gocept.com  Sun Mar  9 17:44:10 2014
From: ct at gocept.com (Christian Theune)
Date: Sun, 9 Mar 2014 17:44:10 +0100
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
 <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
Message-ID: <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>


On 6. Mar2014, at 01:43, Donald Stufft <donald at stufft.io> wrote:

> Also want to make sure the original authors of PEP381 and the mirroring clients
> are aware of this PEP!

Happy with it from my side. Updating bandersnatch should be trivial.

Christian

-- 
Christian Theune ? gocept gmbh & co. kg
flyingcircus.io ? operations as a service
Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140309/a72e9f7f/attachment.sig>

From donald at stufft.io  Tue Mar 11 20:01:08 2014
From: donald at stufft.io (Donald Stufft)
Date: Tue, 11 Mar 2014 15:01:08 -0400
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
 <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
 <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>
Message-ID: <BBBB05ED-B769-4BF7-B784-285C3D195C6A@stufft.io>

I think maybe we?re ready for a decree on this? I didn?t expect many people
to actually care about it since it?s unused :)

On Mar 9, 2014, at 12:44 PM, Christian Theune <ct at gocept.com> wrote:

> 
> On 6. Mar2014, at 01:43, Donald Stufft <donald at stufft.io> wrote:
> 
>> Also want to make sure the original authors of PEP381 and the mirroring clients
>> are aware of this PEP!
> 
> Happy with it from my side. Updating bandersnatch should be trivial.
> 
> Christian
> 
> -- 
> Christian Theune ? gocept gmbh & co. kg
> flyingcircus.io ? operations as a service
> Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140311/f951936e/attachment.sig>

From ct at gocept.com  Tue Mar 11 20:53:07 2014
From: ct at gocept.com (Christian Theune)
Date: Tue, 11 Mar 2014 20:53:07 +0100
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <BBBB05ED-B769-4BF7-B784-285C3D195C6A@stufft.io>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
 <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
 <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>
 <BBBB05ED-B769-4BF7-B784-285C3D195C6A@stufft.io>
Message-ID: <43DFD689-361A-41D7-89AA-E0E89987C13C@gocept.com>


On 11. Mar2014, at 20:01, Donald Stufft <donald at stufft.io> wrote:

> I think maybe we?re ready for a decree on this? I didn?t expect many people
> to actually care about it since it?s unused :)

I think I qualify for being ?not many people?, so yeah. ;)

-- 
Christian Theune ? gocept gmbh & co. kg
flyingcircus.io ? operations as a service
Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140311/c53bb8b1/attachment.sig>

From pje at telecommunity.com  Tue Mar 18 18:04:57 2014
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 18 Mar 2014 13:04:57 -0400
Subject: [Distutils] Redirects for old distribute documentation?
Message-ID: <CALeMXf6SZoW-sHx3PvCa7WbGv8SWNnhLPggJWBLahoFqNsCD=A@mail.gmail.com>

I was browsing through some docs on a database API for Python today when I
noticed that they had a link to
http://pythonhosted.org/distribute/easy_install.html  -- which is defunct,
along with most everything else under pythonhosted.org/distribute/.

Perhaps it would be a good idea for the missing pages to redirect to some
active documentation instead?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140318/afa58391/attachment.html>

From vinay_sajip at yahoo.co.uk  Wed Mar 19 14:13:38 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 19 Mar 2014 13:13:38 +0000 (GMT)
Subject: [Distutils] [ANN]: distlib 0.1.8 released on PyPI
Message-ID: <1395234818.65024.YahooMailNeo@web172401.mail.ir2.yahoo.com>

I've released version 0.1.8 of distlib on PyPI [1]. For newcomers,
distlib is a library of packaging functionality which is intended
to be?usable as the basis for third-party packaging tools.

The main changes in this release are as follows:

* Fixed issue #45: Improved thread-safety in SimpleScrapingLocator.
*?Fixed issue #42: Handling of pre-release legacy version numbers
? now mirrors setuptools logic.
* Added exists, verify, update, is_compatible and is_mountable methods
? to?the Wheel class (the update method fixed issue #41).
* Added a search method to the PackageIndex class.
* Fixed a bug in the Metadata.add_requirements method.
*?Allowed versions with a single numeric component and a local version
? component (tracking changes to PEP 440).* Corrected spelling of environment variable used for the stub launcher
? on OS X.
*?Avoided using pydist.json in 1.0 wheels (bdist_wheel writes a non-
? conforming pydist.json).

* Improved computation of ABI tags on Python versions where SOABI is not
? available, and improved computation of compatibility tags on OS X to
? allow for multiple architectures and older OS X versions.


A more detailed change log is available at [2]. Please try it out, and
if you find any problems or have any suggestions for?improvements, please
give some feedback using the issue tracker! [3]

Regards,

Vinay Sajip

[1] https://pypi.python.org/pypi/distlib/0.1.8?
[2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib?
[3] https://bitbucket.org/pypa/distlib/issues/new


From dholth at gmail.com  Wed Mar 19 15:53:08 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 19 Mar 2014 10:53:08 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
Message-ID: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>

One reason people use distutils is because it can compile Python
extension modules. Here's an example from waf that compiles Python
extensions without using distutils. The problem of invoking the
compiler has been solved by many build systems and is not just a
unique and mysterious distutils feature.

https://code.google.com/p/waf/source/browse/#git%2Fdemos%2Fpython

From techtonik at gmail.com  Wed Mar 19 16:54:54 2014
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 19 Mar 2014 18:54:54 +0300
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
Message-ID: <CAPkN8xLmSuxUmYzP8UuHj7qSmwERdhk+i6uhpc517j_JKpOq+g@mail.gmail.com>

The idea is not new. Here are some examples for SCons:
https://stackoverflow.com/questions/2627731/using-scons-as-a-build-engine-for-distutils
The answers may need to be updated, so if anybody comes
up with convenient distutils tool semantics - it can be
integrated into next SCons release.

On Wed, Mar 19, 2014 at 5:53 PM, Daniel Holth <dholth at gmail.com> wrote:
> One reason people use distutils is because it can compile Python
> extension modules. Here's an example from waf that compiles Python
> extensions without using distutils. The problem of invoking the
> compiler has been solved by many build systems and is not just a
> unique and mysterious distutils feature.
>
> https://code.google.com/p/waf/source/browse/#git%2Fdemos%2Fpython
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig



-- 
anatoly t.

From vinay_sajip at yahoo.co.uk  Thu Mar 20 00:54:38 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Wed, 19 Mar 2014 23:54:38 +0000 (UTC)
Subject: [Distutils] waf can be used to compile Python extension modules
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
Message-ID: <loom.20140320T005101-661@post.gmane.org>

Daniel Holth <dholth <at> gmail.com> writes:

> extensions without using distutils. The problem of invoking the
> compiler has been solved by many build systems and is not just a
> unique and mysterious distutils feature.

Did someone say it was? Building extensions is something distil does too, and 
without using distutils or setuptools.

Regards,

Vinay Sajip


From ncoghlan at gmail.com  Thu Mar 20 05:50:50 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 20 Mar 2014 14:50:50 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <loom.20140320T005101-661@post.gmane.org>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
Message-ID: <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>

On 20 March 2014 09:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> Daniel Holth <dholth <at> gmail.com> writes:
>
>> extensions without using distutils. The problem of invoking the
>> compiler has been solved by many build systems and is not just a
>> unique and mysterious distutils feature.
>
> Did someone say it was? Building extensions is something distil does too, and
> without using distutils or setuptools.

Right, the problem is the lack of a standard interface for how the
packaging system is supposed to invoke them - that is, *implementation
independent* docs of what the various setup.py commands are *supposed*
to do.

The packaging system shouldn't have to care *how* setup.py is
implemented, but at the moment, the behaviour is underspecified to the
point where it's a matter of reverse engineering distutils and/or
implementing what seems necessary and waiting to see if people
complain.

Cheers,
Nick.

From bcannon at gmail.com  Thu Mar 20 14:16:53 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Thu, 20 Mar 2014 13:16:53 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
Message-ID: <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>

On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 20 March 2014 09:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> > Daniel Holth <dholth <at> gmail.com> writes:
> >
> >> extensions without using distutils. The problem of invoking the
> >> compiler has been solved by many build systems and is not just a
> >> unique and mysterious distutils feature.
> >
> > Did someone say it was? Building extensions is something distil does
> too, and
> > without using distutils or setuptools.
>
> Right, the problem is the lack of a standard interface for how the
> packaging system is supposed to invoke them - that is, *implementation
> independent* docs of what the various setup.py commands are *supposed*
> to do.
>
> The packaging system shouldn't have to care *how* setup.py is
> implemented, but at the moment, the behaviour is underspecified to the
> point where it's a matter of reverse engineering distutils and/or
> implementing what seems necessary and waiting to see if people
> complain.
>

What are the plans for the build step in the grand plan of Python
packaging? I think previously it has been suggested that once metadata is
done and distribution/installation is taken care of the
distutils/setuptools building part of all of this will be tackled. Is that
still accurate?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140320/6ed0dfbe/attachment.html>

From ncoghlan at gmail.com  Fri Mar 21 11:13:16 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Mar 2014 20:13:16 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
Message-ID: <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>

On 20 Mar 2014 23:16, "Brett Cannon" <bcannon at gmail.com> wrote:
>
>
>
> On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan <ncoghlan at gmail.com>
wrote:
>>
>> On 20 March 2014 09:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> > Daniel Holth <dholth <at> gmail.com> writes:
>> >
>> >> extensions without using distutils. The problem of invoking the
>> >> compiler has been solved by many build systems and is not just a
>> >> unique and mysterious distutils feature.
>> >
>> > Did someone say it was? Building extensions is something distil does
too, and
>> > without using distutils or setuptools.
>>
>> Right, the problem is the lack of a standard interface for how the
>> packaging system is supposed to invoke them - that is, *implementation
>> independent* docs of what the various setup.py commands are *supposed*
>> to do.
>>
>> The packaging system shouldn't have to care *how* setup.py is
>> implemented, but at the moment, the behaviour is underspecified to the
>> point where it's a matter of reverse engineering distutils and/or
>> implementing what seems necessary and waiting to see if people
>> complain.
>
>
> What are the plans for the build step in the grand plan of Python
packaging? I think previously it has been suggested that once metadata is
done and distribution/installation is taken care of the
distutils/setuptools building part of all of this will be tackled. Is that
still accurate?

That's the priority list for my personal focus, but I certainly wouldn't
complain if there was momentum behind addressing the build side in
parallel, rather than in series. Commenting on PEPs is usually easier than
writing them, and the starting point for the build side is mostly just
defining a suitable subset of setup.py commands, options, behaviour and
invocation context that will make up the "fallback legacy interface" a new
metabuild system would need regardless of any other details.

I just don't have the personal bandwidth to champion such an effort at this
point in time.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/072b1105/attachment.html>

From dholth at gmail.com  Fri Mar 21 14:46:08 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 09:46:08 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
Message-ID: <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>

Bento emulates setup.py: https://github.com/cournape/Bento/blob/master/setup.py

It mostly has to implement egg_info and install to be pip compatible.

In the future we'd like them to implement dist_info (generate a
.dist-info directory instead of an .egg-info directory) and wheel; an
sdist would signal that it doesn't implement setup.py install and pip
would always build and then install a wheel. This would be a
relatively simple thing to add to pip.

Once setup.py emulation is working then we could define a Python-level
plugin interface for build systems that might return metadata dicts,
and also provide a setup.py that talks to this plugin interface. A
later version of pip might talk directly to the plugin without
bothering with setup.py.

This strategy does not generally try to eliminate arbitrary code
execution during builds - builds are an inherently arbitrary-code
process. But once the build has happened most installs should work
without arbitrary code execution.

On Fri, Mar 21, 2014 at 6:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> On 20 Mar 2014 23:16, "Brett Cannon" <bcannon at gmail.com> wrote:
>>
>>
>>
>> On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan <ncoghlan at gmail.com>
>> wrote:
>>>
>>> On 20 March 2014 09:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>>> > Daniel Holth <dholth <at> gmail.com> writes:
>>> >
>>> >> extensions without using distutils. The problem of invoking the
>>> >> compiler has been solved by many build systems and is not just a
>>> >> unique and mysterious distutils feature.
>>> >
>>> > Did someone say it was? Building extensions is something distil does
>>> > too, and
>>> > without using distutils or setuptools.
>>>
>>> Right, the problem is the lack of a standard interface for how the
>>> packaging system is supposed to invoke them - that is, *implementation
>>> independent* docs of what the various setup.py commands are *supposed*
>>> to do.
>>>
>>> The packaging system shouldn't have to care *how* setup.py is
>>> implemented, but at the moment, the behaviour is underspecified to the
>>> point where it's a matter of reverse engineering distutils and/or
>>> implementing what seems necessary and waiting to see if people
>>> complain.
>>
>>
>> What are the plans for the build step in the grand plan of Python
>> packaging? I think previously it has been suggested that once metadata is
>> done and distribution/installation is taken care of the distutils/setuptools
>> building part of all of this will be tackled. Is that still accurate?
>
> That's the priority list for my personal focus, but I certainly wouldn't
> complain if there was momentum behind addressing the build side in parallel,
> rather than in series. Commenting on PEPs is usually easier than writing
> them, and the starting point for the build side is mostly just defining a
> suitable subset of setup.py commands, options, behaviour and invocation
> context that will make up the "fallback legacy interface" a new metabuild
> system would need regardless of any other details.
>
> I just don't have the personal bandwidth to champion such an effort at this
> point in time.
>
> Cheers,
> Nick.
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

From p.f.moore at gmail.com  Fri Mar 21 15:25:31 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Mar 2014 14:25:31 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
Message-ID: <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>

On 21 March 2014 13:46, Daniel Holth <dholth at gmail.com> wrote:
> It mostly has to implement egg_info and install to be pip compatible.

At a *very* brief scan, pip uses just the following setup.py
invocations (this definitely needs to be properly audited before
relying on it):

For installation from sdist
* setup.py egg_info [--egg-base XXX]
* setup.py install --record XXX [--single-version-externally-managed]
[--root XXX] [--compile|--no-compile] [--install-headers XXX]

For editable installs
* setup.py develop --no-deps

Not sure, used in RequirementSet
* setup.py clean

For pip wheel
* setup.py bdist_wheel -d XXX

Note that installation from wheel requires *no* setup.py interface.
Nearly all of this is for installation from sdist (which long-term is
expected to be deprecated - install via wheels is the intended "way of
the future", and PR #1572 is the first step on the way to that). While
theoretically you can build wheels any way you like, I expect "pip
wheel XXX" to be the canonical approach (it's how PR#1572 works) and
at the moment that means implementing setup.py bdist_wheel.

So, at a bare minimum, you could get away with just supporting
setup.py bdist_wheel (and not supporting editable installs or installs
direct from source). It might need PR #1572 to make this palatable, of
course.

Also, there's the fact (which I glossed over) that pip currently
doesn't run setup.py directly, but does some nasty hackery to inject
setuptools. Before working on standard build interfaces, we probably
need to refactor that out somehow. Maybe a simple flag (that could be
set in setup.cfg) that says "use setup.py without monkeypatching"
would be enough...

Paul

From dholth at gmail.com  Fri Mar 21 15:33:36 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 10:33:36 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
Message-ID: <CAG8k2+4gdBOLrSah3DdWg5wzm88T8t0d79BS_epiKCE-mTLQAw@mail.gmail.com>

On Fri, Mar 21, 2014 at 10:25 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 21 March 2014 13:46, Daniel Holth <dholth at gmail.com> wrote:
>> It mostly has to implement egg_info and install to be pip compatible.
>
> At a *very* brief scan, pip uses just the following setup.py
> invocations (this definitely needs to be properly audited before
> relying on it):
>
> For installation from sdist
> * setup.py egg_info [--egg-base XXX]
> * setup.py install --record XXX [--single-version-externally-managed]
> [--root XXX] [--compile|--no-compile] [--install-headers XXX]
>
> For editable installs
> * setup.py develop --no-deps
>
> Not sure, used in RequirementSet
> * setup.py clean
>
> For pip wheel
> * setup.py bdist_wheel -d XXX
>
> Note that installation from wheel requires *no* setup.py interface.
> Nearly all of this is for installation from sdist (which long-term is
> expected to be deprecated - install via wheels is the intended "way of
> the future", and PR #1572 is the first step on the way to that). While
> theoretically you can build wheels any way you like, I expect "pip
> wheel XXX" to be the canonical approach (it's how PR#1572 works) and
> at the moment that means implementing setup.py bdist_wheel.
>
> So, at a bare minimum, you could get away with just supporting
> setup.py bdist_wheel (and not supporting editable installs or installs
> direct from source). It might need PR #1572 to make this palatable, of
> course.
>
> Also, there's the fact (which I glossed over) that pip currently
> doesn't run setup.py directly, but does some nasty hackery to inject
> setuptools. Before working on standard build interfaces, we probably
> need to refactor that out somehow. Maybe a simple flag (that could be
> set in setup.cfg) that says "use setup.py without monkeypatching"
> would be enough...

No worries... if you are re-implementing setup.py, then setuptools is
monkey-patching code that you won't be executing.

From dholth at gmail.com  Fri Mar 21 18:56:09 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 13:56:09 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
Message-ID: <CAG8k2+5F4+g5JMU=UExmvhQfLCKdAnE=83hbXqkq=J3ZRT7sBw@mail.gmail.com>

On Fri, Mar 21, 2014 at 10:25 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 21 March 2014 13:46, Daniel Holth <dholth at gmail.com> wrote:
>> It mostly has to implement egg_info and install to be pip compatible.
>
> At a *very* brief scan, pip uses just the following setup.py
> invocations (this definitely needs to be properly audited before
> relying on it):
>
> For installation from sdist
> * setup.py egg_info [--egg-base XXX]
> * setup.py install --record XXX [--single-version-externally-managed]
> [--root XXX] [--compile|--no-compile] [--install-headers XXX]
>
> For editable installs
> * setup.py develop --no-deps
>
> Not sure, used in RequirementSet
> * setup.py clean
>
> For pip wheel
> * setup.py bdist_wheel -d XXX
>
> Note that installation from wheel requires *no* setup.py interface.
> Nearly all of this is for installation from sdist (which long-term is
> expected to be deprecated - install via wheels is the intended "way of
> the future", and PR #1572 is the first step on the way to that). While
> theoretically you can build wheels any way you like, I expect "pip
> wheel XXX" to be the canonical approach (it's how PR#1572 works) and
> at the moment that means implementing setup.py bdist_wheel.
>
> So, at a bare minimum, you could get away with just supporting
> setup.py bdist_wheel (and not supporting editable installs or installs
> direct from source). It might need PR #1572 to make this palatable, of
> course.

You do need to keep a mechanism to indicate & install build dependencies.

> Also, there's the fact (which I glossed over) that pip currently
> doesn't run setup.py directly, but does some nasty hackery to inject
> setuptools. Before working on standard build interfaces, we probably
> need to refactor that out somehow. Maybe a simple flag (that could be
> set in setup.cfg) that says "use setup.py without monkeypatching"
> would be enough...
>
> Paul

From p.f.moore at gmail.com  Fri Mar 21 19:37:03 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Mar 2014 18:37:03 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CAG8k2+5F4+g5JMU=UExmvhQfLCKdAnE=83hbXqkq=J3ZRT7sBw@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
 <CAG8k2+5F4+g5JMU=UExmvhQfLCKdAnE=83hbXqkq=J3ZRT7sBw@mail.gmail.com>
Message-ID: <CACac1F9TVQjHLpRhv2JigiaDr9LbyAMxo4Ts+eRdZ7z7-dyUvQ@mail.gmail.com>

On 21 March 2014 17:56, Daniel Holth <dholth at gmail.com> wrote:
> You do need to keep a mechanism to indicate & install build dependencies.

Do we have that at the moment? Metadata 2.0 covers that with
build_requires. At the moment setup_requires covers it, but I don't
know if/how that works (AIUI, setup_requires is processed by
setuptools, so it's somewhat outside of the normal pip mechanisms).

But yes, there needs to be some accessible data that tells pip
"install these packages before trying to run setup.py". Again, though,
that's only needed for "pip wheel" or for the
install-direct-from-source cases that may not even be expected to work
in the new metabuild system...

Paul

From dholth at gmail.com  Fri Mar 21 19:49:55 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 14:49:55 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F9TVQjHLpRhv2JigiaDr9LbyAMxo4Ts+eRdZ7z7-dyUvQ@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
 <CAG8k2+5F4+g5JMU=UExmvhQfLCKdAnE=83hbXqkq=J3ZRT7sBw@mail.gmail.com>
 <CACac1F9TVQjHLpRhv2JigiaDr9LbyAMxo4Ts+eRdZ7z7-dyUvQ@mail.gmail.com>
Message-ID: <CAG8k2+6Cgj9Gyi4uXt8n+yzGbKK9h_Rzfu3rDH3TG4qksWLc5g@mail.gmail.com>

On Fri, Mar 21, 2014 at 2:37 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 21 March 2014 17:56, Daniel Holth <dholth at gmail.com> wrote:
>> You do need to keep a mechanism to indicate & install build dependencies.
>
> Do we have that at the moment? Metadata 2.0 covers that with
> build_requires. At the moment setup_requires covers it, but I don't
> know if/how that works (AIUI, setup_requires is processed by
> setuptools, so it's somewhat outside of the normal pip mechanisms).
>
> But yes, there needs to be some accessible data that tells pip
> "install these packages before trying to run setup.py". Again, though,
> that's only needed for "pip wheel" or for the
> install-direct-from-source cases that may not even be expected to work
> in the new metabuild system...
>
> Paul

We will always need the Gentoo-model "compiled-on-demand" style of installation.

From tseaver at palladion.com  Fri Mar 21 19:50:10 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 21 Mar 2014 14:50:10 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F9TVQjHLpRhv2JigiaDr9LbyAMxo4Ts+eRdZ7z7-dyUvQ@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
 <CAG8k2+5F4+g5JMU=UExmvhQfLCKdAnE=83hbXqkq=J3ZRT7sBw@mail.gmail.com>
 <CACac1F9TVQjHLpRhv2JigiaDr9LbyAMxo4Ts+eRdZ7z7-dyUvQ@mail.gmail.com>
Message-ID: <lgi1kk$p0l$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/21/2014 02:37 PM, Paul Moore wrote:
> Again, though, that's only needed for "pip wheel" or for the 
> install-direct-from-source cases that may not even be expected to
> work in the new metabuild system...

Hmm, that is a clear case of baby-with-the-bathwater if I ever saw one.
Installing from an sdist has got to be orders-of-magnitude more common
than anything requiring a separate compilation / build step.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlMsieIACgkQ+gerLs4ltQ4cTACfbzcmVCMkAcXw8/4IWmNonAZV
hlYAn2iNppd+ICEz7L2qr4sxnIISqykY
=Dzaq
-----END PGP SIGNATURE-----


From donald at stufft.io  Fri Mar 21 19:59:58 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Mar 2014 14:59:58 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <lgi1kk$p0l$1@ger.gmane.org>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <CACac1F-HF3d-CaPWkGNZDAXn+R36NX8QB2=k0mfCBFhtUPX-hQ@mail.gmail.com>
 <CAG8k2+5F4+g5JMU=UExmvhQfLCKdAnE=83hbXqkq=J3ZRT7sBw@mail.gmail.com>
 <CACac1F9TVQjHLpRhv2JigiaDr9LbyAMxo4Ts+eRdZ7z7-dyUvQ@mail.gmail.com>
 <lgi1kk$p0l$1@ger.gmane.org>
Message-ID: <C5BBBC0F-610A-45EB-BE31-83DA569F5B24@stufft.io>


On Mar 21, 2014, at 2:50 PM, Tres Seaver <tseaver at palladion.com> wrote:

> Signed PGP part
> On 03/21/2014 02:37 PM, Paul Moore wrote:
> > Again, though, that's only needed for "pip wheel" or for the
> > install-direct-from-source cases that may not even be expected to
> > work in the new metabuild system...
> 
> Hmm, that is a clear case of baby-with-the-bathwater if I ever saw one.
> Installing from an sdist has got to be orders-of-magnitude more common
> than anything requiring a separate compilation / build step.
> 
> 
> Tres.
> --
> ===================================================================
> Tres Seaver          +1 540-429-0999          tseaver at palladion.com
> Palladion Software   "Excellence by Design"    http://palladion.com
> 
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Some confusion here :)

When pip switches to only install from Wheels it?ll still handle sdists,
it?ll just do them by invoking the metabuild system to create a Wheel
and then install that Wheel.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/4579774c/attachment.sig>

From theller at ctypes.org  Fri Mar 21 20:10:57 2014
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 21 Mar 2014 20:10:57 +0100
Subject: [Distutils] wheel questions
Message-ID: <lgi2ql$a76$1@ger.gmane.org>

1. question: Can wheels support a postinstall script?

Background: I'm in the process of setting up a build script for that
part of our company's software that is ported to Python 3.4.

I am using either wheels built by 'pip wheel' and cache them locally,
or, if this doesn't work (numpy and pywin32 for example), convert
bdist_wininst installers to wheels with 'wheel convert xxx.exe'.

Works fine for numpy and some other stuff, but not for pywin32. pywin32
contains a postinstall script that installs some dlls into the right
locations and also does other stuff.  Can a wheel run an included 
postinstall (and preremove) script?


2. question:  I am building wheels for py2exe.
If the wheel is built with python 3.3, the resulting wheel filename is
py2exe-0.9.0.3-py33-none-any.whl.
If the wheel is built with python 3.4, the filename is
py2exe-0.9.0.3-py34-none-any.whl.

Both are (kind-of) pure python distributions.  'pip install' can
install the former both for Python 3.3 and Python 3.4, the latter
only for Python 3.4.

The only reason can be the filename.  Does the -py34- part mean that
*at least* Python 3.4 is required, and does -py33- mean that *at least*
Python 3.3 is required?

What would -cp33- or -cp34- mean?

Sorry about these questions, but I'm lost what these filenames really 
mean (and how they are created).

Thanks,
Thomas


From dholth at gmail.com  Fri Mar 21 20:19:25 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 15:19:25 -0400
Subject: [Distutils] wheel questions
In-Reply-To: <lgi2ql$a76$1@ger.gmane.org>
References: <lgi2ql$a76$1@ger.gmane.org>
Message-ID: <CAG8k2+74YRZzT7DQ=am6JVP3PZrko9RK0+hA49KabTKK2EPp+g@mail.gmail.com>

1. No. Probably some day they will.

2. You are correct. We plan to switch the default to be py3-none-any
"runs on some Python 3 interpreter". cp33 means "only runs on CPython"
however we have decided this tag is probably less useful than
initially posited since there are other ways to say "has a C
extension".

On Fri, Mar 21, 2014 at 3:10 PM, Thomas Heller <theller at ctypes.org> wrote:
> 1. question: Can wheels support a postinstall script?
>
> Background: I'm in the process of setting up a build script for that
> part of our company's software that is ported to Python 3.4.
>
> I am using either wheels built by 'pip wheel' and cache them locally,
> or, if this doesn't work (numpy and pywin32 for example), convert
> bdist_wininst installers to wheels with 'wheel convert xxx.exe'.
>
> Works fine for numpy and some other stuff, but not for pywin32. pywin32
> contains a postinstall script that installs some dlls into the right
> locations and also does other stuff.  Can a wheel run an included
> postinstall (and preremove) script?
>
>
> 2. question:  I am building wheels for py2exe.
> If the wheel is built with python 3.3, the resulting wheel filename is
> py2exe-0.9.0.3-py33-none-any.whl.
> If the wheel is built with python 3.4, the filename is
> py2exe-0.9.0.3-py34-none-any.whl.
>
> Both are (kind-of) pure python distributions.  'pip install' can
> install the former both for Python 3.3 and Python 3.4, the latter
> only for Python 3.4.
>
> The only reason can be the filename.  Does the -py34- part mean that
> *at least* Python 3.4 is required, and does -py33- mean that *at least*
> Python 3.3 is required?
>
> What would -cp33- or -cp34- mean?
>
> Sorry about these questions, but I'm lost what these filenames really mean
> (and how they are created).
>
> Thanks,
> Thomas
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From theller at ctypes.org  Fri Mar 21 20:34:09 2014
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 21 Mar 2014 20:34:09 +0100
Subject: [Distutils] wheel questions
In-Reply-To: <CAG8k2+74YRZzT7DQ=am6JVP3PZrko9RK0+hA49KabTKK2EPp+g@mail.gmail.com>
References: <lgi2ql$a76$1@ger.gmane.org>
 <CAG8k2+74YRZzT7DQ=am6JVP3PZrko9RK0+hA49KabTKK2EPp+g@mail.gmail.com>
Message-ID: <lgi465$pc8$1@ger.gmane.org>

Am 21.03.2014 20:19, schrieb Daniel Holth:
> 2. You are correct. We plan to switch the default to be py3-none-any
> "runs on some Python 3 interpreter". cp33 means "only runs on CPython"
> however we have decided this tag is probably less useful than
> initially posited since there are other ways to say "has a C
> extension".

So, currently it is possible with 'pip install' to install a -cp33- 
wheel also on 3.4?
Is there a way to specify the actual filename of the wheel, or is this
always determined automatically?



From dholth at gmail.com  Fri Mar 21 20:45:24 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 15:45:24 -0400
Subject: [Distutils] wheel questions
In-Reply-To: <lgi465$pc8$1@ger.gmane.org>
References: <lgi2ql$a76$1@ger.gmane.org>
 <CAG8k2+74YRZzT7DQ=am6JVP3PZrko9RK0+hA49KabTKK2EPp+g@mail.gmail.com>
 <lgi465$pc8$1@ger.gmane.org>
Message-ID: <CAG8k2+7BWiwerAtabFywcHhHHeYUSVEMvhMDnR8711u8Dj3Tmg@mail.gmail.com>

On Fri, Mar 21, 2014 at 3:34 PM, Thomas Heller <theller at ctypes.org> wrote:
> Am 21.03.2014 20:19, schrieb Daniel Holth:
>
>> 2. You are correct. We plan to switch the default to be py3-none-any
>> "runs on some Python 3 interpreter". cp33 means "only runs on CPython"
>> however we have decided this tag is probably less useful than
>> initially posited since there are other ways to say "has a C
>> extension".
>
>
> So, currently it is possible with 'pip install' to install a -cp33- wheel
> also on 3.4?

Yes.

> Is there a way to specify the actual filename of the wheel, or is this
> always determined automatically?

No, but a patch is in the works. You can just use 'mv' (rename the file).

From donald at stufft.io  Fri Mar 21 21:51:04 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Mar 2014 16:51:04 -0400
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <43DFD689-361A-41D7-89AA-E0E89987C13C@gocept.com>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
 <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
 <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>
 <BBBB05ED-B769-4BF7-B784-285C3D195C6A@stufft.io>
 <43DFD689-361A-41D7-89AA-E0E89987C13C@gocept.com>
Message-ID: <6D3D5B53-2F37-45DC-A60B-5B36F7123581@stufft.io>

Ping on a decree/pronouncement for this? :]

On Mar 11, 2014, at 3:53 PM, Christian Theune <ct at gocept.com> wrote:

> 
> On 11. Mar2014, at 20:01, Donald Stufft <donald at stufft.io> wrote:
> 
>> I think maybe we?re ready for a decree on this? I didn?t expect many people
>> to actually care about it since it?s unused :)
> 
> I think I qualify for being ?not many people?, so yeah. ;)
> 
> -- 
> Christian Theune ? gocept gmbh & co. kg
> flyingcircus.io ? operations as a service
> Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/ca9663d0/attachment.sig>

From r1chardj0n3s at gmail.com  Fri Mar 21 22:25:39 2014
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Sat, 22 Mar 2014 08:25:39 +1100
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <6D3D5B53-2F37-45DC-A60B-5B36F7123581@stufft.io>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
 <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
 <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>
 <BBBB05ED-B769-4BF7-B784-285C3D195C6A@stufft.io>
 <43DFD689-361A-41D7-89AA-E0E89987C13C@gocept.com>
 <6D3D5B53-2F37-45DC-A60B-5B36F7123581@stufft.io>
Message-ID: <CAHrZfZAU4D08XKHdpAiSgm1NRhT+Bqgs03v494a7t3cx8f4Lpw@mail.gmail.com>

Let's do it. PEP accepted.


On 22 March 2014 07:51, Donald Stufft <donald at stufft.io> wrote:

> Ping on a decree/pronouncement for this? :]
>
> On Mar 11, 2014, at 3:53 PM, Christian Theune <ct at gocept.com> wrote:
>
> >
> > On 11. Mar2014, at 20:01, Donald Stufft <donald at stufft.io> wrote:
> >
> >> I think maybe we're ready for a decree on this? I didn't expect many
> people
> >> to actually care about it since it's unused :)
> >
> > I think I qualify for being "not many people", so yeah. ;)
> >
> > --
> > Christian Theune ? gocept gmbh & co. kg
> > flyingcircus.io ? operations as a service
> > Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7
> >
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140322/3cc72c68/attachment.html>

From donald at stufft.io  Fri Mar 21 22:26:02 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 21 Mar 2014 17:26:02 -0400
Subject: [Distutils] PEP464 - Removal of the PyPI Mirror Authenticity API
In-Reply-To: <CAHrZfZAU4D08XKHdpAiSgm1NRhT+Bqgs03v494a7t3cx8f4Lpw@mail.gmail.com>
References: <89CAACF8-C022-4CDE-8B07-92305BF74E6E@stufft.io>
 <0BCC24E4-1B4F-4C0E-8CA3-1F103BBB8C7B@stufft.io>
 <C9AB9DD3-CDC4-4DE4-8F48-A6A7B869E577@stufft.io>
 <AE073146-DC8A-4270-B17F-92A126961DAD@gocept.com>
 <BBBB05ED-B769-4BF7-B784-285C3D195C6A@stufft.io>
 <43DFD689-361A-41D7-89AA-E0E89987C13C@gocept.com>
 <6D3D5B53-2F37-45DC-A60B-5B36F7123581@stufft.io>
 <CAHrZfZAU4D08XKHdpAiSgm1NRhT+Bqgs03v494a7t3cx8f4Lpw@mail.gmail.com>
Message-ID: <954C34E1-4106-423E-85E4-C0199BF5F8D2@stufft.io>

Thank you sir!

On Mar 21, 2014, at 5:25 PM, Richard Jones <r1chardj0n3s at gmail.com> wrote:

> Let's do it. PEP accepted.
> 
> 
> On 22 March 2014 07:51, Donald Stufft <donald at stufft.io> wrote:
> Ping on a decree/pronouncement for this? :]
> 
> On Mar 11, 2014, at 3:53 PM, Christian Theune <ct at gocept.com> wrote:
> 
> >
> > On 11. Mar2014, at 20:01, Donald Stufft <donald at stufft.io> wrote:
> >
> >> I think maybe we?re ready for a decree on this? I didn?t expect many people
> >> to actually care about it since it?s unused :)
> >
> > I think I qualify for being ?not many people?, so yeah. ;)
> >
> > --
> > Christian Theune ? gocept gmbh & co. kg
> > flyingcircus.io ? operations as a service
> > Forsterstra?e 29 ? 06112 Halle (Saale) ? Tel +49 345 1229889-7
> >
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/2ab54ad0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/2ab54ad0/attachment.sig>

From cournape at gmail.com  Sat Mar 22 00:33:48 2014
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 21 Mar 2014 23:33:48 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
Message-ID: <CAGY4rcUQJcu+pfHvHFD-oRirKmtM634yxfMq+VT7g0KSu-i2ag@mail.gmail.com>

On Fri, Mar 21, 2014 at 1:46 PM, Daniel Holth <dholth at gmail.com> wrote:

> Bento emulates setup.py:
> https://github.com/cournape/Bento/blob/master/setup.py
>
> It mostly has to implement egg_info and install to be pip compatible.
>

Yes. While some things are quite not ideal in Bento, I think that part is a
good one: build a system that goes away completely from distutils, and then
emulate those interfaces (where it makes sense).

This allows compatibility in most cases, while avoiding re-introducing what
makes distutils horrible (commands and compilers).

Before attempting to solve the build problem itself, I would suggest to
have a small library extracted from distutils that does not impose any
'policy' on the build, but expose compilation information (flags, link
options, etc...). This library could be used from distutils, but also
(crucially) by 3rd party tools.

Regarding build, my own experience after having dealt with unholly
scons/waf/distutils/numpy.distutils integration projects:

  - solving the simple cases (simple C extensions) is easy, but moving to
more complex cases makes the problem much harder. Build is really a case
where solving 80 % of the usecases is solving 10 % of the problem. Solving
80 % of the usecases is ok as long as it does not prevent the remaining 20
% to do what they want.
  - neither scons or waf are really usable as libraries (scons much more so
than waf). Even fbuild, which I generally find more approachable, has a
'god' object that makes integration difficult.
  - having well documented, self-contained API to build final artefacts
(sdist, installs, eggs, wheels, bdist_wininst, bdist_msi, etc...) that help
integrating with 3rd party build tools is more critical than the build tool
itself.

David

>
> In the future we'd like them to implement dist_info (generate a
> .dist-info directory instead of an .egg-info directory) and wheel; an
> sdist would signal that it doesn't implement setup.py install and pip
> would always build and then install a wheel. This would be a
> relatively simple thing to add to pip.
>
> Once setup.py emulation is working then we could define a Python-level
> plugin interface for build systems that might return metadata dicts,
> and also provide a setup.py that talks to this plugin interface. A
> later version of pip might talk directly to the plugin without
> bothering with setup.py.
>
> This strategy does not generally try to eliminate arbitrary code
> execution during builds - builds are an inherently arbitrary-code
> process. But once the build has happened most installs should work
> without arbitrary code execution.
>
> On Fri, Mar 21, 2014 at 6:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > On 20 Mar 2014 23:16, "Brett Cannon" <bcannon at gmail.com> wrote:
> >>
> >>
> >>
> >> On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan <ncoghlan at gmail.com>
> >> wrote:
> >>>
> >>> On 20 March 2014 09:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> >>> > Daniel Holth <dholth <at> gmail.com> writes:
> >>> >
> >>> >> extensions without using distutils. The problem of invoking the
> >>> >> compiler has been solved by many build systems and is not just a
> >>> >> unique and mysterious distutils feature.
> >>> >
> >>> > Did someone say it was? Building extensions is something distil does
> >>> > too, and
> >>> > without using distutils or setuptools.
> >>>
> >>> Right, the problem is the lack of a standard interface for how the
> >>> packaging system is supposed to invoke them - that is, *implementation
> >>> independent* docs of what the various setup.py commands are *supposed*
> >>> to do.
> >>>
> >>> The packaging system shouldn't have to care *how* setup.py is
> >>> implemented, but at the moment, the behaviour is underspecified to the
> >>> point where it's a matter of reverse engineering distutils and/or
> >>> implementing what seems necessary and waiting to see if people
> >>> complain.
> >>
> >>
> >> What are the plans for the build step in the grand plan of Python
> >> packaging? I think previously it has been suggested that once metadata
> is
> >> done and distribution/installation is taken care of the
> distutils/setuptools
> >> building part of all of this will be tackled. Is that still accurate?
> >
> > That's the priority list for my personal focus, but I certainly wouldn't
> > complain if there was momentum behind addressing the build side in
> parallel,
> > rather than in series. Commenting on PEPs is usually easier than writing
> > them, and the starting point for the build side is mostly just defining a
> > suitable subset of setup.py commands, options, behaviour and invocation
> > context that will make up the "fallback legacy interface" a new metabuild
> > system would need regardless of any other details.
> >
> > I just don't have the personal bandwidth to champion such an effort at
> this
> > point in time.
> >
> > Cheers,
> > Nick.
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/cb78d803/attachment-0001.html>

From vinay_sajip at yahoo.co.uk  Sat Mar 22 00:37:07 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 21 Mar 2014 23:37:07 +0000 (GMT)
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
Message-ID: <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>



>?This strategy does not generally try to eliminate arbitrary code
>?execution during builds - builds are an inherently arbitrary-code
>?process. But once the build has happened most installs should work
> without arbitrary code execution.


I don't think builds should be a *completely* arbitrary-code process. I understand well that user-defined code should be accommodated, but IMO this should be within a declarative framework with well-defined hooks, otherwise it will ultimately lead to the same problems that setup.py has.

Regards,

Vinay Sajip


From ncoghlan at gmail.com  Sat Mar 22 00:44:45 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Mar 2014 09:44:45 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
Message-ID: <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>

On 22 March 2014 09:37, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> This strategy does not generally try to eliminate arbitrary code
>> execution during builds - builds are an inherently arbitrary-code
>> process. But once the build has happened most installs should work
>> without arbitrary code execution.
>
> I don't think builds should be a *completely* arbitrary-code process. I understand well that user-defined code should be accommodated, but IMO this should be within a declarative framework with well-defined hooks, otherwise it will ultimately lead to the same problems that setup.py has.

Agreed, but that can be a two step process:

1. Formally decouple the setup.py CLI from the distutils command system
2. Define a more declarative metabuild system, with the setup.py CLI
considered a legacy compatibility interface.

The reason I think we need to cover step 1 first is because without
doing that, I don't believe we're going to understand the existing
scope we need to cover for 2.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dholth at gmail.com  Sat Mar 22 01:39:53 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 21 Mar 2014 20:39:53 -0400
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
Message-ID: <CAG8k2+542wpUVrpG9Ai3N_B47DANLcR7RZNxmKU78yD2se6Ejw@mail.gmail.com>

I don't think there is a need to be that pessimistic. Most people will even
be able to keep setup.py as is. But when you really should be sidestepping
DistUtils instead of writing a 10k line extension there is a supported path.
On Mar 21, 2014 7:37 PM, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:

>
>
> > This strategy does not generally try to eliminate arbitrary code
> > execution during builds - builds are an inherently arbitrary-code
> > process. But once the build has happened most installs should work
> > without arbitrary code execution.
>
>
> I don't think builds should be a *completely* arbitrary-code process. I
> understand well that user-defined code should be accommodated, but IMO this
> should be within a declarative framework with well-defined hooks, otherwise
> it will ultimately lead to the same problems that setup.py has.
>
> Regards,
>
> Vinay Sajip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/a59eccd1/attachment.html>

From vinay_sajip at yahoo.co.uk  Sat Mar 22 23:37:03 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Sat, 22 Mar 2014 22:37:03 +0000 (GMT)
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>	<loom.20140320T005101-661@post.gmane.org>	<CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>	<CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>	<CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>	<CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>	<1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
Message-ID: <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>

> 1. Formally decouple the setup.py CLI from the distutils command system



Agreed - it looks like a compatible CLI needs to be part of the transition
to any new system (I assume that's what you meant).

Regards,

Vinay Sajip

From ncoghlan at gmail.com  Sun Mar 23 00:24:09 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 23 Mar 2014 09:24:09 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
Message-ID: <CADiSq7cM+qm0dKUr-6tA8v8=VoDTYFc1UQ0VYSyP8tVZ3yDFuw@mail.gmail.com>

On 23 March 2014 08:37, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> 1. Formally decouple the setup.py CLI from the distutils command system
>
> Agreed - it looks like a compatible CLI needs to be part of the transition
> to any new system (I assume that's what you meant).

Yeah - we have precedent there with distutils2, where they needed
d2to1 to allow it to be used with easy_install and pip, and I think
that model makes sense.

The main blockers at the moment are to get the setuptools CLI docs up
to date, and then to run through that and nominate what represents the
core functionality that can be expected from a reimplementation, and
what can be deemed setuptools specific.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From p.f.moore at gmail.com  Sun Mar 23 00:35:31 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 22 Mar 2014 23:35:31 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
Message-ID: <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>

On 22 March 2014 22:37, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> 1. Formally decouple the setup.py CLI from the distutils command system
>
> Agreed - it looks like a compatible CLI needs to be part of the transition
> to any new system (I assume that's what you meant).

Does this statement mean "document the exact setup.py invocations
which need to be supported by any tool" or does it mean "create a new
standard cli and update pip to use it and sort out a means of wrapping
existing setup.py scripts to expose it"? If the former, then I've
informally done that earlier in the thread (basically "setup.py
bdist_wheel -d XXX" is it[1], if we assume that by the time it's
official pip will have completed the move to installing from sdist by
using wheels as an intermediate step). Writing a wrapper setup.py that
simply invokes a different tool such as bento is trivial.

Paul

[1] The one exception is editable installations (setup.py develop).
That's supported by pip, implemented wholly in setuptools, and an
integral part of many workflows. I don't believe anyone has looked yet
at what might be needed to decouple that from the
distutils/setuptools/setup.py infrastructure.

From ncoghlan at gmail.com  Sun Mar 23 01:00:45 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 23 Mar 2014 10:00:45 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
 <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>
Message-ID: <CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>

On 23 Mar 2014 09:35, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 22 March 2014 22:37, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> >> 1. Formally decouple the setup.py CLI from the distutils command system
> >
> > Agreed - it looks like a compatible CLI needs to be part of the
transition
> > to any new system (I assume that's what you meant).
>
> Does this statement mean "document the exact setup.py invocations
> which need to be supported by any tool" or does it mean "create a new
> standard cli and update pip to use it and sort out a means of wrapping
> existing setup.py scripts to expose it"? If the former, then I've
> informally done that earlier in the thread (basically "setup.py
> bdist_wheel -d XXX" is it[1], if we assume that by the time it's
> official pip will have completed the move to installing from sdist by
> using wheels as an intermediate step). Writing a wrapper setup.py that
> simply invokes a different tool such as bento is trivial.

I mean actually writing it up and documenting all the *build options*
(including cross compilation support). As far as I am aware, that hasn't
been done to date (at least, not in an easily consumable form that people
can link to).

Cheers,
Nick.

>
> Paul
>
> [1] The one exception is editable installations (setup.py develop).
> That's supported by pip, implemented wholly in setuptools, and an
> integral part of many workflows. I don't believe anyone has looked yet
> at what might be needed to decouple that from the
> distutils/setuptools/setup.py infrastructure.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140323/eb152b4f/attachment.html>

From martin at v.loewis.de  Sun Mar 23 09:55:40 2014
From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 23 Mar 2014 09:55:40 +0100
Subject: [Distutils] setuptools tracker
Message-ID: <532EA18C.2040907@v.loewis.de>

We are still hosting a roundup installation for setuptools,
at http://bugs.python.org/setuptools/.

Is this still needed? If not: what should we do with it?

Regards,
Martin

From p.f.moore at gmail.com  Sun Mar 23 15:13:12 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 23 Mar 2014 14:13:12 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
 <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>
 <CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>
Message-ID: <CACac1F_0jEkq4QyW8S1=VUSy0vK7XgDO5Bjzr+efwT4=1FjCSQ@mail.gmail.com>

On 23 March 2014 00:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 23 Mar 2014 09:35, "Paul Moore" <p.f.moore at gmail.com> wrote:
>>
>> On 22 March 2014 22:37, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>> >> 1. Formally decouple the setup.py CLI from the distutils command system
>> >
>> > Agreed - it looks like a compatible CLI needs to be part of the
>> > transition
>> > to any new system (I assume that's what you meant).
>>
>> Does this statement mean "document the exact setup.py invocations
>> which need to be supported by any tool" or does it mean "create a new
>> standard cli and update pip to use it and sort out a means of wrapping
>> existing setup.py scripts to expose it"? If the former, then I've
>> informally done that earlier in the thread (basically "setup.py
>> bdist_wheel -d XXX" is it[1], if we assume that by the time it's
>> official pip will have completed the move to installing from sdist by
>> using wheels as an intermediate step). Writing a wrapper setup.py that
>> simply invokes a different tool such as bento is trivial.
>
> I mean actually writing it up and documenting all the *build options*
> (including cross compilation support). As far as I am aware, that hasn't
> been done to date (at least, not in an easily consumable form that people
> can link to).

Hmm. How about if, to start the ball rolling, I add a short section to
the pip documentation entitled "setup.py interface" that explains the
basic commands that pip uses. That can then be expanded on as
additional data is collected.

One thing though. I may be being dense here, but I'm not aware of any
way for pip to provide build options via "pip wheel". I certainly
don't know the pip wheel command you'd use to (for example)
cross-compile psutil for Windows on a Linux box. Or how you'd use pip
wheel to build pyYAML with libyaml support (you need to indicate where
the libyaml headers and library live). Obviously you can do this with
*distutils*, but that's different. We're not expecting bento, or
distil, or waf, or whatever, to support the full distutils command
line interface, surely?

The pip wheel command includes --build-options and --build-options
flags that translate to global and command-specific options on the
bdist_wheel command. Those are as follows:

Global options:
  --verbose (-v)  run verbosely (default)
  --quiet (-q)    run quietly (turns verbosity off)
  --dry-run (-n)  don't actually do anything
  --help (-h)     show detailed help message
  --no-user-cfg   ignore pydistutils.cfg in your home directory

Options for 'bdist_wheel' command:
  --bdist-dir (-b)  temporary directory for creating the distribution
  --plat-name (-p)  platform name to embed in generated filenames (default:
                    win-amd64)
  --keep-temp (-k)  keep the pseudo-installation tree around after creating
                    the distribution archive
  --dist-dir (-d)   directory to put final built distributions in
  --skip-build      skip rebuilding everything (for testing/debugging)
  --relative        build the archive using relative paths(default: false)
  --owner (-u)      Owner name used when creating a tar file [default: current
                    user]
  --group (-g)      Group name used when creating a tar file [default: current
                    group]
  --skip-scripts    skip building the setuptools console_scripts
  --universal       make a universal wheel (default: false)

Note that none of those control the build - the ones that do that are
options to the build_ext and build commands, but *distutils* doesn't
allow you to supply them in a bdist_wheel command. The same is true
for pip install, BTW. That sends the options to the setup.py install
command - again, not to setup.py build_ext.

I can imagine good ways of using the existing pip interface to drive
build tools (even distutils!) more effectively than we do at present.
But I don't think the current interface supports compilation arguments
(even if it looks like it does).

Paul.

From pje at telecommunity.com  Sun Mar 23 18:30:14 2014
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 23 Mar 2014 13:30:14 -0400
Subject: [Distutils] setuptools tracker
In-Reply-To: <532EA18C.2040907@v.loewis.de>
References: <532EA18C.2040907@v.loewis.de>
Message-ID: <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>

On Sun, Mar 23, 2014 at 4:55 AM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> We are still hosting a roundup installation for setuptools,
> at http://bugs.python.org/setuptools/.
>
> Is this still needed? If not: what should we do with it?
>
>
I think probably the remaining issues need to be moved to Bitbucket (unless
they're already addressed in later setuptools versions), and the tracker
closed.  At this point, I think it's safe to say that the 0.6 line isn't
getting any more changes; persons and organizations using older versions of
Python will have to take 0.6 as it is, or upgrade.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140323/b81bc345/attachment.html>

From martin at v.loewis.de  Sun Mar 23 19:00:10 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 23 Mar 2014 19:00:10 +0100
Subject: [Distutils] setuptools tracker
In-Reply-To: <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>
References: <532EA18C.2040907@v.loewis.de>
 <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>
Message-ID: <532F212A.6070008@v.loewis.de>

Am 23.03.14 18:30, schrieb PJ Eby:
> On Sun, Mar 23, 2014 at 4:55 AM, "Martin v. L?wis" <martin at v.loewis.de
> <mailto:martin at v.loewis.de>> wrote:
> 
>     We are still hosting a roundup installation for setuptools,
>     at http://bugs.python.org/setuptools/.
> 
>     Is this still needed? If not: what should we do with it?
> 
> 
> I think probably the remaining issues need to be moved to Bitbucket
> (unless they're already addressed in later setuptools versions), and the
> tracker closed.  At this point, I think it's safe to say that the 0.6
> line isn't getting any more changes; persons and organizations using
> older versions of Python will have to take 0.6 as it is, or upgrade.

Would you volunteer to move them? Alternatively, I could close them all
with an automatic message saying that they should re-report them if the
issue still exists.

Regards,
Martin



From ncoghlan at gmail.com  Mon Mar 24 00:13:04 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 24 Mar 2014 09:13:04 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F_0jEkq4QyW8S1=VUSy0vK7XgDO5Bjzr+efwT4=1FjCSQ@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
 <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>
 <CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>
 <CACac1F_0jEkq4QyW8S1=VUSy0vK7XgDO5Bjzr+efwT4=1FjCSQ@mail.gmail.com>
Message-ID: <CADiSq7e+yi3tmsupjC6A1y7HmbEfECQwTVncCZtxQB9iWyNjKA@mail.gmail.com>

On 24 Mar 2014 00:13, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 23 March 2014 00:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 23 Mar 2014 09:35, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >>
> >> On 22 March 2014 22:37, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> >> >> 1. Formally decouple the setup.py CLI from the distutils command
system
> >> >
> >> > Agreed - it looks like a compatible CLI needs to be part of the
> >> > transition
> >> > to any new system (I assume that's what you meant).
> >>
> >> Does this statement mean "document the exact setup.py invocations
> >> which need to be supported by any tool" or does it mean "create a new
> >> standard cli and update pip to use it and sort out a means of wrapping
> >> existing setup.py scripts to expose it"? If the former, then I've
> >> informally done that earlier in the thread (basically "setup.py
> >> bdist_wheel -d XXX" is it[1], if we assume that by the time it's
> >> official pip will have completed the move to installing from sdist by
> >> using wheels as an intermediate step). Writing a wrapper setup.py that
> >> simply invokes a different tool such as bento is trivial.
> >
> > I mean actually writing it up and documenting all the *build options*
> > (including cross compilation support). As far as I am aware, that hasn't
> > been done to date (at least, not in an easily consumable form that
people
> > can link to).
>
> Hmm. How about if, to start the ball rolling, I add a short section to
> the pip documentation entitled "setup.py interface" that explains the
> basic commands that pip uses. That can then be expanded on as
> additional data is collected.

Yes, explaining the interface that pip currently relies on would definitely
be a good starting point.

> One thing though. I may be being dense here, but I'm not aware of any
> way for pip to provide build options via "pip wheel". I certainly
> don't know the pip wheel command you'd use to (for example)
> cross-compile psutil for Windows on a Linux box. Or how you'd use pip
> wheel to build pyYAML with libyaml support (you need to indicate where
> the libyaml headers and library live). Obviously you can do this with
> *distutils*, but that's different. We're not expecting bento, or
> distil, or waf, or whatever, to support the full distutils command
> line interface, surely?

Now you begin to see the scope of the problem. It's definitely solvable,
but means asking a whole pile of "required, recommended or
distutils-specific?" questions about the existing distutils and setuptools
build system :)

"pip already relies on it" sets the minimum for the "required" category,
but there's more to a full build system abstraction than what pip currently
supports.

Cheers,
Nick.

>
> The pip wheel command includes --build-options and --build-options
> flags that translate to global and command-specific options on the
> bdist_wheel command. Those are as follows:
>
> Global options:
>   --verbose (-v)  run verbosely (default)
>   --quiet (-q)    run quietly (turns verbosity off)
>   --dry-run (-n)  don't actually do anything
>   --help (-h)     show detailed help message
>   --no-user-cfg   ignore pydistutils.cfg in your home directory
>
> Options for 'bdist_wheel' command:
>   --bdist-dir (-b)  temporary directory for creating the distribution
>   --plat-name (-p)  platform name to embed in generated filenames
(default:
>                     win-amd64)
>   --keep-temp (-k)  keep the pseudo-installation tree around after
creating
>                     the distribution archive
>   --dist-dir (-d)   directory to put final built distributions in
>   --skip-build      skip rebuilding everything (for testing/debugging)
>   --relative        build the archive using relative paths(default: false)
>   --owner (-u)      Owner name used when creating a tar file [default:
current
>                     user]
>   --group (-g)      Group name used when creating a tar file [default:
current
>                     group]
>   --skip-scripts    skip building the setuptools console_scripts
>   --universal       make a universal wheel (default: false)
>
> Note that none of those control the build - the ones that do that are
> options to the build_ext and build commands, but *distutils* doesn't
> allow you to supply them in a bdist_wheel command. The same is true
> for pip install, BTW. That sends the options to the setup.py install
> command - again, not to setup.py build_ext.
>
> I can imagine good ways of using the existing pip interface to drive
> build tools (even distutils!) more effectively than we do at present.
> But I don't think the current interface supports compilation arguments
> (even if it looks like it does).
>
> Paul.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140324/954cd317/attachment-0001.html>

From p.f.moore at gmail.com  Mon Mar 24 01:01:26 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 24 Mar 2014 00:01:26 +0000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CADiSq7e+yi3tmsupjC6A1y7HmbEfECQwTVncCZtxQB9iWyNjKA@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
 <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>
 <CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>
 <CACac1F_0jEkq4QyW8S1=VUSy0vK7XgDO5Bjzr+efwT4=1FjCSQ@mail.gmail.com>
 <CADiSq7e+yi3tmsupjC6A1y7HmbEfECQwTVncCZtxQB9iWyNjKA@mail.gmail.com>
Message-ID: <CACac1F9MBaNqdqC2AbpHjw6uE32FxYNK4WG2ZEgsFSow+3zfqw@mail.gmail.com>

On 23 March 2014 23:13, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Now you begin to see the scope of the problem. It's definitely solvable, but
> means asking a whole pile of "required, recommended or distutils-specific?"
> questions about the existing distutils and setuptools build system :)
>
> "pip already relies on it" sets the minimum for the "required" category, but
> there's more to a full build system abstraction than what pip currently
> supports.

OK, I see now. So the ultimate build system will include pip changes
to supply build-time options in an as-yet unspecified manner.

There's certainly no way I can do all of that myself, I don't have
remotely the level of experience with complex build requirements. But
I can probably take the first steps, and leave it to people with the
experience to add to it. No promises on timescales but I'll see what I
can do.

One thought. do we want to use a setup.py script as the interface,
with all its historical baggage, or would we be better using a new
script name as the "official" interface (with pip falling back to
equivalent setup.py invocations when that script isn't present, for
backward compatibility)?

Paul

From pje at telecommunity.com  Mon Mar 24 01:33:59 2014
From: pje at telecommunity.com (PJ Eby)
Date: Sun, 23 Mar 2014 20:33:59 -0400
Subject: [Distutils] setuptools tracker
In-Reply-To: <532F212A.6070008@v.loewis.de>
References: <532EA18C.2040907@v.loewis.de>
 <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>
 <532F212A.6070008@v.loewis.de>
Message-ID: <CALeMXf5=e0r+gLw4zhhHbFKW-m9Eo4=twgea7oD7riSHWa1BPQ@mail.gmail.com>

I think it would be a good idea to check with Jason and other PyPA
volunteers to see if there is anyone else who can handle the moves.  I'd
prefer we didn't lose the history, since my comments on those issues (and
the closed ones, too) often contain key information about use cases and
design decisions that may not be available elsewhere, even from my memory.
;-)  But, since I'm no longer in the lead on development, I think it would
be better for someone closer to the future of things to do the prioritizing
of what, if anything, to transfer as an issue or keep as documentation.


On Sun, Mar 23, 2014 at 2:00 PM, "Martin v. L?wis" <martin at v.loewis.de>wrote:

> Am 23.03.14 18:30, schrieb PJ Eby:
> > On Sun, Mar 23, 2014 at 4:55 AM, "Martin v. L?wis" <martin at v.loewis.de
> > <mailto:martin at v.loewis.de>> wrote:
> >
> >     We are still hosting a roundup installation for setuptools,
> >     at http://bugs.python.org/setuptools/.
> >
> >     Is this still needed? If not: what should we do with it?
> >
> >
> > I think probably the remaining issues need to be moved to Bitbucket
> > (unless they're already addressed in later setuptools versions), and the
> > tracker closed.  At this point, I think it's safe to say that the 0.6
> > line isn't getting any more changes; persons and organizations using
> > older versions of Python will have to take 0.6 as it is, or upgrade.
>
> Would you volunteer to move them? Alternatively, I could close them all
> with an automatic message saying that they should re-report them if the
> issue still exists.
>
> Regards,
> Martin
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140323/e08c20f6/attachment.html>

From ncoghlan at gmail.com  Mon Mar 24 09:59:16 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 24 Mar 2014 18:59:16 +1000
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CACac1F9MBaNqdqC2AbpHjw6uE32FxYNK4WG2ZEgsFSow+3zfqw@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>
 <loom.20140320T005101-661@post.gmane.org>
 <CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>
 <CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>
 <CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>
 <CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>
 <1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>
 <CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>
 <1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>
 <CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>
 <CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>
 <CACac1F_0jEkq4QyW8S1=VUSy0vK7XgDO5Bjzr+efwT4=1FjCSQ@mail.gmail.com>
 <CADiSq7e+yi3tmsupjC6A1y7HmbEfECQwTVncCZtxQB9iWyNjKA@mail.gmail.com>
 <CACac1F9MBaNqdqC2AbpHjw6uE32FxYNK4WG2ZEgsFSow+3zfqw@mail.gmail.com>
Message-ID: <CADiSq7cfjM_y8Y_boFUWDwfvub-jmEy3WVCQ+NSTFoZ0PkmQDA@mail.gmail.com>

On 24 Mar 2014 10:01, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 23 March 2014 23:13, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Now you begin to see the scope of the problem. It's definitely
solvable, but
> > means asking a whole pile of "required, recommended or
distutils-specific?"
> > questions about the existing distutils and setuptools build system :)
> >
> > "pip already relies on it" sets the minimum for the "required"
category, but
> > there's more to a full build system abstraction than what pip currently
> > supports.
>
> OK, I see now. So the ultimate build system will include pip changes
> to supply build-time options in an as-yet unspecified manner.
>
> There's certainly no way I can do all of that myself, I don't have
> remotely the level of experience with complex build requirements. But
> I can probably take the first steps, and leave it to people with the
> experience to add to it. No promises on timescales but I'll see what I
> can do.
>
> One thought. do we want to use a setup.py script as the interface,
> with all its historical baggage, or would we be better using a new
> script name as the "official" interface (with pip falling back to
> equivalent setup.py invocations when that script isn't present, for
> backward compatibility)?

Step 1 is "What does pip currently expect setup.py to support?"

Step 2 is "What other existing features of distutils/setuptools do we think
a reasonable replacement for setup.py should support?" (I don't believe
distutils2 reached the point of addressing this, but that should still be
checked)

Step 3 ("what, if anything, should replace the setup.py CLI as the build
system abstraction?") really depends on the outcome of steps 1 & 2 - this
is more a research/documentation consolidation project at this point than
it is a design project.

Cheers,
Nick.

>
> Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140324/8fe5717c/attachment.html>

From martin at v.loewis.de  Mon Mar 24 11:53:16 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 24 Mar 2014 11:53:16 +0100
Subject: [Distutils] setuptools tracker
In-Reply-To: <CALeMXf5=e0r+gLw4zhhHbFKW-m9Eo4=twgea7oD7riSHWa1BPQ@mail.gmail.com>
References: <532EA18C.2040907@v.loewis.de>
 <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>
 <532F212A.6070008@v.loewis.de>
 <CALeMXf5=e0r+gLw4zhhHbFKW-m9Eo4=twgea7oD7riSHWa1BPQ@mail.gmail.com>
Message-ID: <53300E9C.7010408@v.loewis.de>

Am 24.03.14 01:33, schrieb PJ Eby:
> I think it would be a good idea to check with Jason and other PyPA
> volunteers to see if there is anyone else who can handle the moves.  I'd
> prefer we didn't lose the history, since my comments on those issues
> (and the closed ones, too) often contain key information about use cases
> and design decisions that may not be available elsewhere, even from my
> memory.  ;-)  But, since I'm no longer in the lead on development, I
> think it would be better for someone closer to the future of things to
> do the prioritizing of what, if anything, to transfer as an issue or
> keep as documentation.

Yet alternatively, I could set the tracker to read-only, and keep it up
for any foreseeable future.

The reason I'm bringing this up is two-fold:
1. some people started using the tracker to distribute spam
2. some people apparently still think that the system is active,
   and continue reporting issues there.

Both problems would be resolved by setting the tracker to read-only;
shutting it down is actually not necessary (although it would slightly
reduce our maintenance efforts).

Regards,
Martin



From ncoghlan at gmail.com  Mon Mar 24 13:46:16 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 24 Mar 2014 22:46:16 +1000
Subject: [Distutils] setuptools tracker
In-Reply-To: <53300E9C.7010408@v.loewis.de>
References: <532EA18C.2040907@v.loewis.de>
 <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>
 <532F212A.6070008@v.loewis.de>
 <CALeMXf5=e0r+gLw4zhhHbFKW-m9Eo4=twgea7oD7riSHWa1BPQ@mail.gmail.com>
 <53300E9C.7010408@v.loewis.de>
Message-ID: <CADiSq7fOF-3LkRmhPkAZxcYjebnWwktRR8X1a1y+gWbdBOM8yw@mail.gmail.com>

On 24 March 2014 20:53, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Am 24.03.14 01:33, schrieb PJ Eby:
>> I think it would be a good idea to check with Jason and other PyPA
>> volunteers to see if there is anyone else who can handle the moves.  I'd
>> prefer we didn't lose the history, since my comments on those issues
>> (and the closed ones, too) often contain key information about use cases
>> and design decisions that may not be available elsewhere, even from my
>> memory.  ;-)  But, since I'm no longer in the lead on development, I
>> think it would be better for someone closer to the future of things to
>> do the prioritizing of what, if anything, to transfer as an issue or
>> keep as documentation.
>
> Yet alternatively, I could set the tracker to read-only, and keep it up
> for any foreseeable future.
>
> The reason I'm bringing this up is two-fold:
> 1. some people started using the tracker to distribute spam
> 2. some people apparently still think that the system is active,
>    and continue reporting issues there.
>
> Both problems would be resolved by setting the tracker to read-only;
> shutting it down is actually not necessary (although it would slightly
> reduce our maintenance efforts).

That sounds good to me.

I've also filed https://bitbucket.org/pypa/setuptools/issue/174/ to
decide on a longer term solution. If Jason decides to review/migrate
issues, it may be necessary to turn developer write access back on to
allow issues to be marked as closed once they have been dealt with
appropriately.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vinay_sajip at yahoo.co.uk  Mon Mar 24 14:11:33 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Mon, 24 Mar 2014 13:11:33 +0000 (GMT)
Subject: [Distutils] waf can be used to compile Python extension modules
In-Reply-To: <CADiSq7cfjM_y8Y_boFUWDwfvub-jmEy3WVCQ+NSTFoZ0PkmQDA@mail.gmail.com>
References: <CAG8k2+6RQ_yhuk_0pSQAg+eswJto1fE1BzyyVyj2m4cgLxxP+A@mail.gmail.com>	<loom.20140320T005101-661@post.gmane.org>	<CADiSq7cGEoEB=DE1ZDRFhq=Og1oa=sY1sBbB-XFm8Oo8d3-r5g@mail.gmail.com>	<CAP1=2W5n-yH_JjEXHQi762xVHsk7pLXs6HNKr2pFyrS+12-_AQ@mail.gmail.com>	<CADiSq7dT_qFsC58BQTbXBcQnQCrfW=iB8Ur1Tu32LOO3kq5Kig@mail.gmail.com>	<CAG8k2+6GVaU74SFrRDO4eXxoHUZ5dt6wJ0nN2RRPZUHyw1YqAA@mail.gmail.com>	<1395445027.52076.YahooMailNeo@web172406.mail.ir2.yahoo.com>	<CADiSq7eyWnCFDZb-RBJFMsQchBMAjp4n6WCU7C8eFB1zB3pSzw@mail.gmail.com>	<1395527823.18182.YahooMailNeo@web172405.mail.ir2.yahoo.com>	<CACac1F80bpVn9QTF=b2mv1JUpJXYy-ZLr6gv=ewenivF28nhvg@mail.gmail.com>	<CADiSq7ch4Y9_AxSV4RW1ZD0CFatQPk9JU=n+cNCWi3oV1jJrsA@mail.gmail.com>	<CACac1F_0jEkq4QyW8S1=VUSy0vK7XgDO5Bjzr+efwT4=1FjCSQ@mail.gmail.com>	<CADiSq7e+yi3tmsupjC6A1y7HmbEfECQwTVncCZtxQB9iWyNjKA@mail.gmail.com>	<CACac1F9MBaNqdqC2AbpHjw6uE32FxYNK4WG2ZEgsFSow+3zfqw@mail.gmail.com>
 <CADiSq7cfjM_y8Y_boFUWDwfvub-jmEy3WVCQ+NSTFoZ0PkmQDA@mail.gmail.com>
Message-ID: <1395666693.99707.YahooMailNeo@web172406.mail.ir2.yahoo.com>

> Step 3 ("what, if anything, should replace the setup.py CLI as the
> build system abstraction?") [...] research/documentation consolidation
> project at this point than it is a design project.


FYI, I've been experimenting with a design approach for a build system in
distil. Currently it only covers what distutils/setuptools already do - i.e.
build_py?functionality + build_ext functionality covering libraries,
extensions, setuptools Features,?Cython / Pyrex and Fortran / f2py. Not too
shabby, but clearly?that's not enough. Instead of compiler classes (the
abstractions used in?distutils / setuptools), distil's approach focuses on
a command line with variable?placeholders - this?allows just about anything
to be plugged in for custom?builds. There's still work to be done on it, but
the initial approach looks?promising.

Regards,

Vinay Sajip

From pje at telecommunity.com  Mon Mar 24 18:35:22 2014
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 24 Mar 2014 13:35:22 -0400
Subject: [Distutils] setuptools tracker
In-Reply-To: <CADiSq7fOF-3LkRmhPkAZxcYjebnWwktRR8X1a1y+gWbdBOM8yw@mail.gmail.com>
References: <532EA18C.2040907@v.loewis.de>
 <CALeMXf7zcXr+C3VjVQuSsngh_ZdEz8NMHQ0dtSpyXYwpWDbcVg@mail.gmail.com>
 <532F212A.6070008@v.loewis.de>
 <CALeMXf5=e0r+gLw4zhhHbFKW-m9Eo4=twgea7oD7riSHWa1BPQ@mail.gmail.com>
 <53300E9C.7010408@v.loewis.de>
 <CADiSq7fOF-3LkRmhPkAZxcYjebnWwktRR8X1a1y+gWbdBOM8yw@mail.gmail.com>
Message-ID: <CALeMXf4Yi52+QzBWrhgkH71Giw3ZtbvjwWJYJU1JB23zidSp5A@mail.gmail.com>

On Mon, Mar 24, 2014 at 8:46 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 24 March 2014 20:53, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > Both problems would be resolved by setting the tracker to read-only;
> > shutting it down is actually not necessary (although it would slightly
> > reduce our maintenance efforts).
>
> That sounds good to me.
>
> I've also filed https://bitbucket.org/pypa/setuptools/issue/174/ to
> decide on a longer term solution. If Jason decides to review/migrate
> issues, it may be necessary to turn developer write access back on to
> allow issues to be marked as closed once they have been dealt with
> appropriately.
>

Yep, looks like Jason came to the same conclusion(s) independently, but
also wants better banners on the old tracker to alert people to the move.
I guess we should move any further discussion to that ticket, since Jason's
response time is quicker there than here.  ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140324/bf31717f/attachment.html>

From barry at python.org  Mon Mar 24 22:48:07 2014
From: barry at python.org (Barry Warsaw)
Date: Mon, 24 Mar 2014 17:48:07 -0400
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
Message-ID: <20140324174807.33712c4d@limelight.wooz.org>

Apologies for cross-posting, but this intersects setuptools and the import
system, and I wanted to be sure it reached the right audience.

A colleague asked me why a seemingly innocent and common use case for
developing local versions of system installed packages wasn't working, and I
was quite perplexed.  As I dug into the problem, more questions than answers
came up.  I finally (think! I) figured out what is happening, but not so much
as to why, or what can/should be done about it.

This person had a local checkout of a package's source, where the package was
also installed into the system Python.  He wanted to be able to set
$PYTHONPATH so that the local package wins when he tries to import it.  E.g.:

% PYTHONPATH=`pwd`/src python3

but this didn't work because despite the setting of PYTHONPATH, the system
version of the package was always found first.  The package in question is
lazr.uri, although other packages with similar layouts will also suffer the
same problem, which prevents an easy local development of a newer version of
the package, aside from being a complete head-scratcher.

The lazr.uri package is intended to be a submodule of the lazr namespace
package.  As such, the lazr/__init__.py has the old style way of declaring a
namespace package:

try:
    import pkg_resources
    pkg_resources.declare_namespace(__name__)
except ImportError:
    import pkgutil
    __path__ = pkgutil.extend_path(__path__, __name__)

and its setup.py declares a namespace package:

setup(
    name='lazr.uri',
    version=__version__,
    namespace_packages=['lazr'],
    ...

One of the things that the Debian "helper" program does when it builds a
package for the archive is call `$python setup.py install_egg_info`.  It's
this command that breaks $PYTHONPATH overriding.

install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file,
in which it finds the string 'lazr', and it proceeds to write a
lazr-uri-1.0.3-py3.4-nspkg.pth file.  This causes other strange and unexpected
things to happen:

% python3
Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['lazr']  
<module 'lazr'>
>>> sys.modules['lazr'].__path__  
['/usr/lib/python3/dist-packages/lazr']

It's completely weird that sys.modules would contain a key for 'lazr' when
that package was never explicitly imported.  Even stranger, because a fake
module object is stuffed into sys.modules via the .pth file, tracing imports
with -v gives you no clue as to what's happening.  And while
sys.modules['lazr'] has an __path__, it has no other attributes.

I really don't understand what the purpose of the nspkg.pth file is,
especially for Python 3 namespace packages.

Here's what the nspkg.pth file contains:

import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)

The __path__ value is important here because even though you've never
explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the
existing lazr module object's __path__ takes over, and thus the system
lazr.uri package is found even though both lazr/ and lazr/uri/ should have
been found earlier on sys.path (yes, sys.path looks exactly as expected).

So the presence of the nspkg.pth file breaks $PYTHONPATH overriding.  That
seems bad. ;)

If you delete the nspkg.path file, then things work as expected, but even this
is a little misleading!

I think the Debian helper is running install_egg_info as a way to determine
what namespace packages are defined, so that it can actually *remove* the
parent's __init__.py file and use PEP 420 style namespace packages.  In fact,
in the Debian python3-lazr.uri binary package, you find no system
lazr/__init__.py file.  This is why removing the nspkg.pth file works.

So I thought, why not conditionally define setup(..., namespace_packages) only
for Python 2?  This doesn't work because the Debian helper will see that no
namespace packages are defined, and thus it will leave the original
lazr/__init__.py file in place.  This then breaks $PYTHONPATH overriding too
because of __path__ extension of the pre-PEP 420 code only *appends* the local
development path.  IOW, the system import path is the first element of a
2-element list on lazr.__path__.  While the local import path is the second
element, in this case too the local import fails.

It seems like what you want for Python 3 (and we're talking >= 3.2 here) is
for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and
let PEP 420 do it's thing.  In fact if you set things up this way, $PYTHONPATH
overriding works exactly as expected.

Because I don't know why install_egg_info is installing the nspkg.pth file, I
don't know which component needs to be changed:

 * Change setuptools install_egg_info command to not install an nspkg.pth file
   even for namespace_package declare packages, at least under Python 3.
   This behavior seems pretty nasty all by itself because it magically and
   untraceably installs stripped down module objects in sys.modules when
   Python first scans the import path.

 * Change the Debian helper to remove the nspkg.pth file, or not call
   install_egg_info *and* continue to remove <nspkg>/__init__.py in Python 3
   so as to take advantage of PEP 420.  It's nice to know that PEP 420
   actually represents something sane. :)

For added bonus, we have this additional oddity:

% PYTHONPATH=`pwd`/src python3
Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['lazr']
<module 'lazr'>
>>> sys.modules['lazr'].__path__
['/usr/lib/python3/dist-packages/lazr']
>>> import lazr.uri
>>> lazr.uri.__file__
'/usr/lib/python3/dist-packages/lazr/uri/__init__.py'
>>> sys.modules['lazr']
<module 'lazr' from '/home/barry/projects/ubuntu/lazruri/trusty/src/lazr/__init__.py'>
>>> sys.modules['lazr'].__path__
['/home/barry/projects/ubuntu/lazruri/trusty/src/lazr', '/usr/lib/python3/dist-packages/lazr']


Notice how importing lazr.uri *replaces* sys.modules['lazr'] with the local
development one, even though it still imports lazr.uri from the system path.
I'm not exactly sure how this happens, but I've traced that to
_LoaderBasics.exec_module()'s call of _call_with_frames_removed(), which
exec's lazr.uri's code object into that module's __dict__.   Nothing in
lazr/uri/__init__.py should be doing that, afaict from both visual inspection
of the code and disassembling the compiled code object.

Hopefully I've explained the situation correctly and lucidly.  Below I'll
describe how to set up a reproducible environment on a Debian machine.
Thoughts and comments are welcome!

Cheers,
-Barry

% sudo apt-get install python3-lazr.uri
% cd tmp
% bzr branch lp:lazr.uri trunk
% cd trunk
% PYTHONPATH=`pwd`/src python3
(Then try things at the Python prompt from above.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140324/9f398bce/attachment-0001.sig>

From donald at stufft.io  Mon Mar 24 22:53:52 2014
From: donald at stufft.io (Donald Stufft)
Date: Mon, 24 Mar 2014 17:53:52 -0400
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
In-Reply-To: <20140324174807.33712c4d@limelight.wooz.org>
References: <20140324174807.33712c4d@limelight.wooz.org>
Message-ID: <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>


See also https://github.com/pypa/pip/issues/3

Basically prior to PEP420 namespace packages were bad and using them
results in pain sooner or later :( I?m not sure if a good solution yet, perhaps
we can backport PEP420 to PyPI and have namespace packages depend
on that?

On Mar 24, 2014, at 5:48 PM, Barry Warsaw <barry at python.org> wrote:

> Apologies for cross-posting, but this intersects setuptools and the import
> system, and I wanted to be sure it reached the right audience.
> 
> A colleague asked me why a seemingly innocent and common use case for
> developing local versions of system installed packages wasn't working, and I
> was quite perplexed.  As I dug into the problem, more questions than answers
> came up.  I finally (think! I) figured out what is happening, but not so much
> as to why, or what can/should be done about it.
> 
> This person had a local checkout of a package's source, where the package was
> also installed into the system Python.  He wanted to be able to set
> $PYTHONPATH so that the local package wins when he tries to import it.  E.g.:
> 
> % PYTHONPATH=`pwd`/src python3
> 
> but this didn't work because despite the setting of PYTHONPATH, the system
> version of the package was always found first.  The package in question is
> lazr.uri, although other packages with similar layouts will also suffer the
> same problem, which prevents an easy local development of a newer version of
> the package, aside from being a complete head-scratcher.
> 
> The lazr.uri package is intended to be a submodule of the lazr namespace
> package.  As such, the lazr/__init__.py has the old style way of declaring a
> namespace package:
> 
> try:
>    import pkg_resources
>    pkg_resources.declare_namespace(__name__)
> except ImportError:
>    import pkgutil
>    __path__ = pkgutil.extend_path(__path__, __name__)
> 
> and its setup.py declares a namespace package:
> 
> setup(
>    name='lazr.uri',
>    version=__version__,
>    namespace_packages=['lazr'],
>    ...
> 
> One of the things that the Debian "helper" program does when it builds a
> package for the archive is call `$python setup.py install_egg_info`.  It's
> this command that breaks $PYTHONPATH overriding.
> 
> install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file,
> in which it finds the string 'lazr', and it proceeds to write a
> lazr-uri-1.0.3-py3.4-nspkg.pth file.  This causes other strange and unexpected
> things to happen:
> 
> % python3
> Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.modules['lazr']  
> <module 'lazr'>
>>>> sys.modules['lazr'].__path__  
> ['/usr/lib/python3/dist-packages/lazr']
> 
> It's completely weird that sys.modules would contain a key for 'lazr' when
> that package was never explicitly imported.  Even stranger, because a fake
> module object is stuffed into sys.modules via the .pth file, tracing imports
> with -v gives you no clue as to what's happening.  And while
> sys.modules['lazr'] has an __path__, it has no other attributes.
> 
> I really don't understand what the purpose of the nspkg.pth file is,
> especially for Python 3 namespace packages.
> 
> Here's what the nspkg.pth file contains:
> 
> import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
> 
> The __path__ value is important here because even though you've never
> explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the
> existing lazr module object's __path__ takes over, and thus the system
> lazr.uri package is found even though both lazr/ and lazr/uri/ should have
> been found earlier on sys.path (yes, sys.path looks exactly as expected).
> 
> So the presence of the nspkg.pth file breaks $PYTHONPATH overriding.  That
> seems bad. ;)
> 
> If you delete the nspkg.path file, then things work as expected, but even this
> is a little misleading!
> 
> I think the Debian helper is running install_egg_info as a way to determine
> what namespace packages are defined, so that it can actually *remove* the
> parent's __init__.py file and use PEP 420 style namespace packages.  In fact,
> in the Debian python3-lazr.uri binary package, you find no system
> lazr/__init__.py file.  This is why removing the nspkg.pth file works.
> 
> So I thought, why not conditionally define setup(..., namespace_packages) only
> for Python 2?  This doesn't work because the Debian helper will see that no
> namespace packages are defined, and thus it will leave the original
> lazr/__init__.py file in place.  This then breaks $PYTHONPATH overriding too
> because of __path__ extension of the pre-PEP 420 code only *appends* the local
> development path.  IOW, the system import path is the first element of a
> 2-element list on lazr.__path__.  While the local import path is the second
> element, in this case too the local import fails.
> 
> It seems like what you want for Python 3 (and we're talking >= 3.2 here) is
> for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and
> let PEP 420 do it's thing.  In fact if you set things up this way, $PYTHONPATH
> overriding works exactly as expected.
> 
> Because I don't know why install_egg_info is installing the nspkg.pth file, I
> don't know which component needs to be changed:
> 
> * Change setuptools install_egg_info command to not install an nspkg.pth file
>   even for namespace_package declare packages, at least under Python 3.
>   This behavior seems pretty nasty all by itself because it magically and
>   untraceably installs stripped down module objects in sys.modules when
>   Python first scans the import path.
> 
> * Change the Debian helper to remove the nspkg.pth file, or not call
>   install_egg_info *and* continue to remove <nspkg>/__init__.py in Python 3
>   so as to take advantage of PEP 420.  It's nice to know that PEP 420
>   actually represents something sane. :)
> 
> For added bonus, we have this additional oddity:
> 
> % PYTHONPATH=`pwd`/src python3
> Python 3.4.0 (default, Mar 22 2014, 22:51:25) 
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.modules['lazr']
> <module 'lazr'>
>>>> sys.modules['lazr'].__path__
> ['/usr/lib/python3/dist-packages/lazr']
>>>> import lazr.uri
>>>> lazr.uri.__file__
> '/usr/lib/python3/dist-packages/lazr/uri/__init__.py'
>>>> sys.modules['lazr']
> <module 'lazr' from '/home/barry/projects/ubuntu/lazruri/trusty/src/lazr/__init__.py'>
>>>> sys.modules['lazr'].__path__
> ['/home/barry/projects/ubuntu/lazruri/trusty/src/lazr', '/usr/lib/python3/dist-packages/lazr']
> 
> 
> Notice how importing lazr.uri *replaces* sys.modules['lazr'] with the local
> development one, even though it still imports lazr.uri from the system path.
> I'm not exactly sure how this happens, but I've traced that to
> _LoaderBasics.exec_module()'s call of _call_with_frames_removed(), which
> exec's lazr.uri's code object into that module's __dict__.   Nothing in
> lazr/uri/__init__.py should be doing that, afaict from both visual inspection
> of the code and disassembling the compiled code object.
> 
> Hopefully I've explained the situation correctly and lucidly.  Below I'll
> describe how to set up a reproducible environment on a Debian machine.
> Thoughts and comments are welcome!
> 
> Cheers,
> -Barry
> 
> % sudo apt-get install python3-lazr.uri
> % cd tmp
> % bzr branch lp:lazr.uri trunk
> % cd trunk
> % PYTHONPATH=`pwd`/src python3
> (Then try things at the Python prompt from above.)
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140324/25c7ced8/attachment.sig>

From barry at python.org  Mon Mar 24 23:09:17 2014
From: barry at python.org (Barry Warsaw)
Date: Mon, 24 Mar 2014 18:09:17 -0400
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
In-Reply-To: <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
Message-ID: <20140324180917.6d42a392@anarchist.wooz.org>

On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:

>See also https://github.com/pypa/pip/issues/3

Hah, yeah.  I didn't realize --single-version-externally-managed is implied by
--root, and in fact our Debian build scripts often (either explicitly or
implicitly) define --root, as is the case in lazr.uri.

>Basically prior to PEP420 namespace packages were bad and using them results
>in pain sooner or later :( I?m not sure if a good solution yet, perhaps we
>can backport PEP420 to PyPI and have namespace packages depend on that?

Or maybe do a version check before installing this file?  Python 3.3 and newer
will have PEP 420 support, and this file makes no sense in that case.

(Backporting PEP 420 support to Python 3.2 might be useful, though a pain
without importlib.  There's no sense in backporting to Python 3.1, IMHO.)

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140324/9c0300c3/attachment.sig>

From bcannon at gmail.com  Tue Mar 25 15:14:40 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Tue, 25 Mar 2014 14:14:40 +0000
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
Message-ID: <CAP1=2W5DoHhL5+1MA7vENxj6u_Y7Y5jq-5UzFRLTmeEbm0+nkQ@mail.gmail.com>

On Mon Mar 24 2014 at 6:09:48 PM, Barry Warsaw <barry at python.org> wrote:

> On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:
>
> >See also https://github.com/pypa/pip/issues/3
>
> Hah, yeah.  I didn't realize --single-version-externally-managed is
> implied by
> --root, and in fact our Debian build scripts often (either explicitly or
> implicitly) define --root, as is the case in lazr.uri.
>
> >Basically prior to PEP420 namespace packages were bad and using them
> results
> >in pain sooner or later :( I?m not sure if a good solution yet, perhaps we
> >can backport PEP420 to PyPI and have namespace packages depend on that?
>
> Or maybe do a version check before installing this file?  Python 3.3 and
> newer
> will have PEP 420 support, and this file makes no sense in that case.
>
> (Backporting PEP 420 support to Python 3.2 might be useful, though a pain
> without importlib.  There's no sense in backporting to Python 3.1, IMHO.)
>

Importlib is in Python 3.2. You would just need to do the right things to
set up the environment and set __import__ to avoid doubling the stat calls
on a failed import.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140325/1dbd9623/attachment.html>

From pje at telecommunity.com  Tue Mar 25 20:35:57 2014
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 25 Mar 2014 15:35:57 -0400
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
In-Reply-To: <20140324180917.6d42a392@anarchist.wooz.org>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
Message-ID: <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>

On Mon, Mar 24, 2014 at 6:09 PM, Barry Warsaw <barry at python.org> wrote:

> On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote:
>
> >See also https://github.com/pypa/pip/issues/3
>
> Hah, yeah.  I didn't realize --single-version-externally-managed is
> implied by
> --root, and in fact our Debian build scripts often (either explicitly or
> implicitly) define --root, as is the case in lazr.uri.
>
> >Basically prior to PEP420 namespace packages were bad and using them
> results
> >in pain sooner or later :( I'm not sure if a good solution yet, perhaps we
> >can backport PEP420 to PyPI and have namespace packages depend on that?
>
> Or maybe do a version check before installing this file?  Python 3.3 and
> newer
> will have PEP 420 support, and this file makes no sense in that case.
>

That won't *quite* work, because the .pth files are generated for other
types of direct-install distributions.

I think the correct fix would be to change the nspkg.pth magic to check for
PEP 420 support, but unfortunately it seems we may have to use version
checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or
similar object that can be directly checked for feature support.  :-(
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140325/9f30aed2/attachment.html>

From barry at python.org  Tue Mar 25 20:50:13 2014
From: barry at python.org (Barry Warsaw)
Date: Tue, 25 Mar 2014 15:50:13 -0400
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
In-Reply-To: <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
Message-ID: <20140325155013.4f6c1dd9@anarchist.wooz.org>

On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:

>I think the correct fix would be to change the nspkg.pth magic to check for
>PEP 420 support, but unfortunately it seems we may have to use version
>checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or
>similar object that can be directly checked for feature support.  :-(

There is.  It's *pronounced* sys.namespace_packages, but it's spelled
importlib._bootstrap._NamespaceLoader ;)

http://youtu.be/ehKGlT2EW1Q?t=44s

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140325/282b01a9/attachment.sig>

From pje at telecommunity.com  Tue Mar 25 21:30:56 2014
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 25 Mar 2014 16:30:56 -0400
Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides
In-Reply-To: <20140325155013.4f6c1dd9@anarchist.wooz.org>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
 <20140325155013.4f6c1dd9@anarchist.wooz.org>
Message-ID: <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>

On Tue, Mar 25, 2014 at 3:50 PM, Barry Warsaw <barry at python.org> wrote:

> On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:
>
> >I think the correct fix would be to change the nspkg.pth magic to check
> for
> >PEP 420 support, but unfortunately it seems we may have to use version
> >checking: on rereading PEP 420 I see there's no 'sys.namespace_packages'
> or
> >similar object that can be directly checked for feature support.  :-(
>
> There is.  It's *pronounced* sys.namespace_packages, but it's spelled
> importlib._bootstrap._NamespaceLoader ;)
>

Yeah, well that's not exactly a public attribute, so it's not necessarily a
great way to do it.  But if we did use that, presumably it'd add something
like:

    hnp =
hasattr(sys.modules.get('importlib._bootstrap',None),'_NamespaceLoader');

To the front of the magic, and then prefix all subsequent expression values
with 'not hnp and' in order to prevent them executing if PEP 420 support is
available. (Note: it's checking sys.modules since on any interpreter where
PEP 420 is natively available, the module should *already* be loaded by the
time site.py does its thing.)

And we should probably think about adding sys.namespace_packages or
something of the sort, or at least a proper flag for whether PEP 420
support is available on the platform.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140325/f965c567/attachment.html>

From erik.m.bray at gmail.com  Thu Mar 27 01:23:23 2014
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 26 Mar 2014 20:23:23 -0400
Subject: [Distutils] tourist here, with a dumb RTFM question
In-Reply-To: <CAG8k2+5489yQwj0DKOYPKjB8BxnhaSS5gwL=C-N4+oiQL_wWvw@mail.gmail.com>
References: <ECE63BA1-55A2-489D-8B6F-EA2D6B61EA9B@zzzcomputing.com>
 <CAG8k2+6L66wDXG_JKqZAvosuF-6iVeoGAL-uyTJrr+swjXA6zA@mail.gmail.com>
 <CC59C77E-E757-49D5-A63D-538DE04CFA32@zzzcomputing.com>
 <CAPYWazqUt8OkiwFE=1Bm4YDJkQGd68_SEpkHaDiA1Sx-xF9uXg@mail.gmail.com>
 <c01b3d929be84f3f965a486370bd0286@BLUPR06MB434.namprd06.prod.outlook.com>
 <E31506E4-A4C1-4A58-A1DB-1469B6A16FA5@zzzcomputing.com>
 <CAG8k2+5489yQwj0DKOYPKjB8BxnhaSS5gwL=C-N4+oiQL_wWvw@mail.gmail.com>
Message-ID: <CAOTD34ZO1B_5BHuj+pAt3kSpM+LsYF-oxmMZAdj6obTyvNw-Jw@mail.gmail.com>

On Sat, Mar 8, 2014 at 10:49 AM, Daniel Holth <dholth at gmail.com> wrote:
> Some packaging systems do not have the same 1:1 source : distribution
> relationship that is so prominent in the distutils model. This should
> probably change long term.

A common (I think?) example of this, which comes up in Astropy, is
pacakges containing Cython modules.  We don't keep the generated C
files in version control.  But we *do* make sure that when running
sdist that the Cython modules are collected and the C modules
generated and added to the manifest so that end users do not need
Cython to `pip install astropy`.  As long as their C compiler works it
can build the generated C modules without Cython.

Right now that's easy enough to do by extending the relevant distutils
commands.  Definitely a distinction to keep in mind in the future
though.

Erik

> On Sat, Mar 8, 2014 at 9:53 AM, Michael Bayer <mike_mp at zzzcomputing.com> wrote:
>>
>> On Mar 8, 2014, at 9:19 AM, Jason R. Coombs <jaraco at jaraco.com> wrote:
>>
>>
>>
>> From: Distutils-SIG
>> [mailto:distutils-sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of
>> Marcus Smith
>> Sent: Thursday, 06 March, 2014 16:16
>> To: DistUtils mailing list
>> Subject: Re: [Distutils] tourist here, with a dumb RTFM question
>>
>>
>>
>>
>>
>> On Thu, Mar 6, 2014 at 10:37 AM, Michael Bayer <mike_mp at zzzcomputing.com>
>> wrote:
>>
>>
>> On Mar 6, 2014, at 1:07 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>>> pje said:
>>>
>>> The "Feature()" facility was never completely implemented or
>>> supported, and even if it were, it should be deprecated now, as it
>>> will not be compatible with the coming packaging systems based on PEP
>>> 426. If you need separate features, use separate distributions and
>>> "extras" instead.
>>
>> wait, ok this is that thing.  "separate distributions" means....
>>
>>
>> SQLAlchemy-0.9.3-nocext.tar.gz
>> SQLAlchemy-0.9.3-cext.tar.gz
>>
>>
>> I'm new to understanding setuptools "Feature" myself, so don't crank too
>> much, if I get anything wrong.
>>
>> afaik, it means something like:
>>
>> SQLAlchemy-X.Y.tar.gz
>> SQLAlchemy-cext-X.Y.tar.gz
>>
>> i.e. isolating the feature into a separate project.
>>
>> and then instead of having the handy  "--with-cext" option,  it has to be
>> "pip install SQLAlchemy[SQLAlchemy-cext]
>>
>> PJ, Jason: can you clarify what the alternative is supposed to look like?
>>
>> I'm new to understanding Features as well, but based on PJ's post, I would
>> agree with Marcus here. This is how I interpreted the use of extras to
>> supersede Features. I see a lot of advantages to this approach over
>> build-time selection. Because it decouples the main project from the
>> C-extension speedups, it gives the installer control over what behavior is
>> present. It also gives deployment tools visibility over what capability is
>> present (with features, it's harder to tell if the C-extension speedups are
>> present; with extras, one can query pkg_resources).
>>
>>
>>
>> If there's an easy way to do this from setup.py it wouldn't be so bad:
>>
>> python setup.py --without-cext sdist upload
>>
>> python setup.py sdist upload
>>
>> but it still seems kind of strange to deliver the exact same source files
>> with some flag somewhere different?   I'd assume that flag is in setup.cfg,
>> which means, it's a setup.py flag in any case!   not to mention its still
>> needed as part of the "sdist" and other commands to build dists.
>>
>> I think the issue of "you should have individual projects on pypi with
>> different options to make life easier for builders" is orthogonal to
>> "setup.py should or should not support flags".
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From dholth at gmail.com  Thu Mar 27 04:29:31 2014
From: dholth at gmail.com (Daniel Holth)
Date: Wed, 26 Mar 2014 23:29:31 -0400
Subject: [Distutils] What is the syntax for passing conditional non-extra
	dependencies in setuptools?
Message-ID: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>

How do I specify a conditional (marker-guarded) non-extra dependency
in setuptools? The syntax for a conditional extra dependency is
currently:

    extras_require = {
        "ssl:sys_platform=='win32'": "wincertstore==0.2",
        "certs": "certifi==1.0.1",
    },

From pje at telecommunity.com  Thu Mar 27 19:35:55 2014
From: pje at telecommunity.com (PJ Eby)
Date: Thu, 27 Mar 2014 14:35:55 -0400
Subject: [Distutils] [Import-SIG] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <CALFfu7AWUb3xMQwaB04BKQkzfAOB9ceY6rXjp7HJdEYpNZ8SeA@mail.gmail.com>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
 <20140325155013.4f6c1dd9@anarchist.wooz.org>
 <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>
 <CALFfu7AWUb3xMQwaB04BKQkzfAOB9ceY6rXjp7HJdEYpNZ8SeA@mail.gmail.com>
Message-ID: <CALeMXf6C=uyKukqxc-XMyJdgidA65OYMcKMo5bKiFqe0qNeozg@mail.gmail.com>

On Wed, Mar 26, 2014 at 11:55 PM, Eric Snow <ericsnowcurrently at gmail.com>wrote:

> In 3.4 it's called _NamespaceLoader, but in 3.3 it's NamespaceLoader.
> <ducks>
>

Ouch. That is going to be a really *long* bit of code.  Not like it isn't
already, though.


By "available on the platform" do you mean "Python 3.3+ or has a
> backport installed"?  Otherwise you'd just check sys.version*.
>

I just hate doing version checks when a feature check ought to be
possible.  You never know when it's going to cause trouble.

But yes, certainly being able to handle the backport case would be nice.
When I wrote PEP 402, I was thinking in terms of transitioning the .pth
files to importing a compatibility module.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/80eb2dcf/attachment.html>

From pje at telecommunity.com  Thu Mar 27 19:39:46 2014
From: pje at telecommunity.com (PJ Eby)
Date: Thu, 27 Mar 2014 14:39:46 -0400
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
Message-ID: <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>

On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth <dholth at gmail.com> wrote:

> How do I specify a conditional (marker-guarded) non-extra dependency
> in setuptools? The syntax for a conditional extra dependency is
> currently:
>
>     extras_require = {
>         "ssl:sys_platform=='win32'": "wincertstore==0.2",
>         "certs": "certifi==1.0.1",
>     },
>

I only implemented support via extras, and the feature wasn't officially
supported (still isn't, I don't think) because the PEP specifying the
syntax wasn't fully baked yet.  I figured that if *only* setuptools itself
used it, then if the syntax changed only setuptools would break...  but fix
itself at the same time.

The same cannot be said for any other package, so use at your own risk.  Or
better yet, don't use it.  ;-)

(At least, not until it's a documented feature w/a PEP-approved syntax.)


_______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/c46a171e/attachment.html>

From dholth at gmail.com  Thu Mar 27 20:42:08 2014
From: dholth at gmail.com (Daniel Holth)
Date: Thu, 27 Mar 2014 15:42:08 -0400
Subject: [Distutils] Is build an inherently arbitrary-code process?
Message-ID: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>

I became convinced that build was an inherently arbitrary-code
process, and not something to be universally handled by a declarative
system, when I observed an autotools project under configuration. The
things spend ten minutes writing and compiling snippets of C code to
determine which features are and are not supported by the runtime.
This is *very* arbitrary code and probably one of the tamer things
people like to do during builds.

As usual I blame distutils, because the separation is not there. So
when you blame setup.py for having to run procedural code to generate
the *metadata* you might go to far and think you should also eliminate
a procedural build - simply because build and metadata are not
adequately separate in the distutils design. Declarative build systems
are a nice idea but they are not going to work for everyone.

From donald at stufft.io  Thu Mar 27 20:48:02 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 27 Mar 2014 15:48:02 -0400
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
Message-ID: <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>


On Mar 27, 2014, at 3:42 PM, Daniel Holth <dholth at gmail.com> wrote:

> I became convinced that build was an inherently arbitrary-code
> process, and not something to be universally handled by a declarative
> system, when I observed an autotools project under configuration. The
> things spend ten minutes writing and compiling snippets of C code to
> determine which features are and are not supported by the runtime.
> This is *very* arbitrary code and probably one of the tamer things
> people like to do during builds.
> 
> As usual I blame distutils, because the separation is not there. So
> when you blame setup.py for having to run procedural code to generate
> the *metadata* you might go to far and think you should also eliminate
> a procedural build - simply because build and metadata are not
> adequately separate in the distutils design. Declarative build systems
> are a nice idea but they are not going to work for everyone.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Well for the record auto tools does all of that because there?s near
zero ability to introspect the compiler to figure out what it supports
except by compiling little programs that depend on X feature and
then running them to see if they work or not.

That being said, yes building code is inherently a process that needs
to execute things and cannot be made completely static. I believe
the metadata itself can be static, and instead of an arbitrary build
script we can declare a build hook, but that?s an argument for a PEP!

I do believe a declarative build system can work for the 90% case
though and should probably be the ?default? option.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/d2bfcb9c/attachment.sig>

From p.f.moore at gmail.com  Thu Mar 27 21:19:24 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 27 Mar 2014 20:19:24 +0000
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
 <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>
Message-ID: <CACac1F-8FhjLOTzKzOPd1Rp=F2jH1tCd_NOiTcs-f56h2obDGw@mail.gmail.com>

On 27 March 2014 19:48, Donald Stufft <donald at stufft.io> wrote:
> I do believe a declarative build system can work for the 90% case
> though and should probably be the "default" option.

+1. Easy things should be easy, and hard things possible. Ultimately,
I want to say "this is my project, here are the Python files. And oh,
here's a couple of C files, nothing clever here, move along.

When I interface with an external library that may not be in the same
place on every system, I'd expect to say a bit more, but mainly just
"look in location X, but the user can override this with a command
line argument".

Going beyond that - autodetecting library locations, languages other
than C (for example Cython), detecting CPU capabilities, etc - I would
expect to be able to do it if I needed to, but I would not expect the
declarative framework to be catering for those cases, I'd assume I
needed to write code.

Paul

From donald at stufft.io  Thu Mar 27 21:30:32 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 27 Mar 2014 16:30:32 -0400
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <CACac1F-8FhjLOTzKzOPd1Rp=F2jH1tCd_NOiTcs-f56h2obDGw@mail.gmail.com>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
 <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>
 <CACac1F-8FhjLOTzKzOPd1Rp=F2jH1tCd_NOiTcs-f56h2obDGw@mail.gmail.com>
Message-ID: <B775FE6F-9A4F-4CEC-A58E-A430499701B0@stufft.io>


On Mar 27, 2014, at 4:19 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 27 March 2014 19:48, Donald Stufft <donald at stufft.io> wrote:
>> I do believe a declarative build system can work for the 90% case
>> though and should probably be the "default" option.
> 
> +1. Easy things should be easy, and hard things possible. Ultimately,
> I want to say "this is my project, here are the Python files. And oh,
> here's a couple of C files, nothing clever here, move along.
> 
> When I interface with an external library that may not be in the same
> place on every system, I'd expect to say a bit more, but mainly just
> "look in location X, but the user can override this with a command
> line argument?.

That?s the job of the linker, it has built in locations to look for libraries
and env vars/command line flags to override that. Someone packaging
a C-ext that needs a library to compile shouldn?t be specifying where
to find it anyways.

> 
> Going beyond that - autodetecting library locations, languages other
> than C (for example Cython), detecting CPU capabilities, etc - I would
> expect to be able to do it if I needed to, but I would not expect the
> declarative framework to be catering for those cases, I'd assume I
> needed to write code.
> 
> Paul


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/e26fb840/attachment.sig>

From p.f.moore at gmail.com  Thu Mar 27 21:49:53 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 27 Mar 2014 20:49:53 +0000
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <B775FE6F-9A4F-4CEC-A58E-A430499701B0@stufft.io>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
 <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>
 <CACac1F-8FhjLOTzKzOPd1Rp=F2jH1tCd_NOiTcs-f56h2obDGw@mail.gmail.com>
 <B775FE6F-9A4F-4CEC-A58E-A430499701B0@stufft.io>
Message-ID: <CACac1F_BuFbjxX=beq9-WgcMyYH7D-d-hHpWtaSLHV_j1y_gSg@mail.gmail.com>

On 27 March 2014 20:30, Donald Stufft <donald at stufft.io> wrote:
>> When I interface with an external library that may not be in the same
>> place on every system, I'd expect to say a bit more, but mainly just
>> "look in location X, but the user can override this with a command
>> line argument".
>
> That's the job of the linker, it has built in locations to look for libraries
> and env vars/command line flags to override that. Someone packaging
> a C-ext that needs a library to compile shouldn't be specifying where
> to find it anyways.

Ideally, yes. But not in practice (at least not on Windows). This is
getting off-topic, so I'll just give a quick example and leave it at
that.

I recently wanted to build PyYAML on 64-bit Windows, which uses
libyaml if that's available. First thing I had to do therefore was to
build libyaml. That's not too hard (considering). But the build leaves
include files in the project directory and a lib file somewhere in the
bowels of the project ...\x64\Release\Output\Lib\libyaml.lib, IIRC).
There is no install step, largely because there is nowhere standard to
install include and lib files *to*.

OK, so now I build PyYAML. I don't have the compiler, linker etc on my
PATH - the distutils code automatically detects that I have the right
version of the compiler installed and sets up the environment for me.
This is a *feature*, not a problem, as otherwise I would have to
remember which compiler to use for which Python version, and activate
it as an extra step. As a result of this fact, though, I can't set LIB
or INCLUDE environment variables to point to the libyaml project
directories (nor would I particularly want to, as this is purely a
one-off build). So I need to tell the build system (distutils) where
to find libyaml and yaml.h. There's no way setup.py can guess where
they might be in the general case, so I use the build_ext -L and -I
options to specify.

Consider "the linker can just find stuff" as the "look in location X"
scenario. And the above as the reason we might need build-system
support for adding compiler/linker flags or env vars at build time.

Paul

From cournape at gmail.com  Thu Mar 27 22:03:34 2014
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 27 Mar 2014 21:03:34 +0000
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
 <852675FB-4A2C-447F-958C-614F3F627522@stufft.io>
Message-ID: <CAGY4rcUqK77=c8Q7=Wc=6Us7LGn3kHL8LhaNWmXKD8_rJbUXQA@mail.gmail.com>

On Thu, Mar 27, 2014 at 7:48 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On Mar 27, 2014, at 3:42 PM, Daniel Holth <dholth at gmail.com> wrote:
>
> > I became convinced that build was an inherently arbitrary-code
> > process, and not something to be universally handled by a declarative
> > system, when I observed an autotools project under configuration. The
> > things spend ten minutes writing and compiling snippets of C code to
> > determine which features are and are not supported by the runtime.
> > This is *very* arbitrary code and probably one of the tamer things
> > people like to do during builds.
> >
> > As usual I blame distutils, because the separation is not there. So
> > when you blame setup.py for having to run procedural code to generate
> > the *metadata* you might go to far and think you should also eliminate
> > a procedural build - simply because build and metadata are not
> > adequately separate in the distutils design. Declarative build systems
> > are a nice idea but they are not going to work for everyone.
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
> Well for the record auto tools does all of that because there?s near
> zero ability to introspect the compiler to figure out what it supports
> except by compiling little programs that depend on X feature and
> then running them to see if they work or not.
>
> That being said, yes building code is inherently a process that needs
> to execute things and cannot be made completely static. I believe
> the metadata itself can be static, and instead of an arbitrary build
> script we can declare a build hook, but that?s an argument for a PEP!
>
> I do believe a declarative build system can work for the 90% case
> though and should probably be the ?default? option.
>

It is indeed possible for most python packages, and I tried to figure out
quite a few things to make that percentage as high as possible in Bento.

The really difficult part is not that, though, but making the system
pluggable to allow pretty much arbitrary extensibility. What makes
distutils very ill suited to that task is the whole subcommand model where
options are not known in advance (that was the #1 issue when integrating
distutils and scons).

For bento, I came up with a system of 'build manifest', where each command
produces a text-based 'protocol' that other commands might use (
https://github.com/pv/bento/blob/master/doc/source/hacking.rst). The
implementation is not great but I think the idea is still sound: this means
that in theory, you could build C extensions with something not involving
python at all (e.g. make), but as long as the build process output that
file, you could build installers from the system. IOW, the json file acts
as 'middleware'.

What any future solution should *not* try to do is inventing yet another
build system

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/29e65c6f/attachment.html>

From ncoghlan at gmail.com  Thu Mar 27 22:16:56 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 Mar 2014 07:16:56 +1000
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
Message-ID: <CADiSq7cHtBjr0rQM0VhFJ65hmr9iV9Fvo5jG6mgYT4tPqx=GvQ@mail.gmail.com>

On 28 Mar 2014 04:40, "PJ Eby" <pje at telecommunity.com> wrote:
>
> On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth <dholth at gmail.com> wrote:
>>
>> How do I specify a conditional (marker-guarded) non-extra dependency
>> in setuptools? The syntax for a conditional extra dependency is
>> currently:
>>
>>     extras_require = {
>>         "ssl:sys_platform=='win32'": "wincertstore==0.2",
>>         "certs": "certifi==1.0.1",
>>     },
>
>
> I only implemented support via extras, and the feature wasn't officially
supported (still isn't, I don't think) because the PEP specifying the
syntax wasn't fully baked yet.  I figured that if *only* setuptools itself
used it, then if the syntax changed only setuptools would break...  but fix
itself at the same time.
>
> The same cannot be said for any other package, so use at your own risk.
Or better yet, don't use it.  ;-)
>
> (At least, not until it's a documented feature w/a PEP-approved syntax.)

Various things got in the way of finalising at least PEP 440, but as far as
I am aware, it, 426 and 459 are basically done aside from updating the JSON
schema specs and redoing the PyPI compatibility analysis.

The other thing they need is for people to read through from scratch,
looking for inconsistencies and things that seem out of place, and
reporting those as issues at
https://bitbucket.org/pypa/pypi-metadata-formats

Cheers,
Nick.

>
>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/ab35e640/attachment-0001.html>

From donald at stufft.io  Thu Mar 27 22:18:16 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 27 Mar 2014 17:18:16 -0400
Subject: [Distutils] What is the syntax for passing conditional
	non-extra dependencies in setuptools?
In-Reply-To: <CADiSq7cHtBjr0rQM0VhFJ65hmr9iV9Fvo5jG6mgYT4tPqx=GvQ@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
 <CADiSq7cHtBjr0rQM0VhFJ65hmr9iV9Fvo5jG6mgYT4tPqx=GvQ@mail.gmail.com>
Message-ID: <E502AF52-13F4-47A0-9DA9-FFE47E405DDE@stufft.io>


On Mar 27, 2014, at 5:16 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 28 Mar 2014 04:40, "PJ Eby" <pje at telecommunity.com> wrote:
> >
> > On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth <dholth at gmail.com> wrote:
> >>
> >> How do I specify a conditional (marker-guarded) non-extra dependency
> >> in setuptools? The syntax for a conditional extra dependency is
> >> currently:
> >>
> >>     extras_require = {
> >>         "ssl:sys_platform=='win32'": "wincertstore==0.2",
> >>         "certs": "certifi==1.0.1",
> >>     },
> >
> >
> > I only implemented support via extras, and the feature wasn't officially supported (still isn't, I don't think) because the PEP specifying the syntax wasn't fully baked yet.  I figured that if *only* setuptools itself used it, then if the syntax changed only setuptools would break...  but fix itself at the same time.
> >
> > The same cannot be said for any other package, so use at your own risk.  Or better yet, don't use it.  ;-)
> >
> > (At least, not until it's a documented feature w/a PEP-approved syntax.)
> 
> Various things got in the way of finalising at least PEP 440, but as far as I am aware, it, 426 and 459 are basically done aside from updating the JSON schema specs and redoing the PyPI compatibility analysis.
> 
> The other thing they need is for people to read through from scratch, looking for inconsistencies and things that seem out of place, and reporting those as issues at https://bitbucket.org/pypa/pypi-metadata-formats
> 
> 

I can take a read through everything again this weekend and take a close look.
> Cheers,
> Nick.
> 
> >
> >
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/1b5384e4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140327/1b5384e4/attachment.sig>

From ncoghlan at gmail.com  Thu Mar 27 22:23:42 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 Mar 2014 07:23:42 +1000
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
Message-ID: <CADiSq7fE5k0treBWWsnO_q-Y43aQSajp0KxM=jKJgGnA1HbKTA@mail.gmail.com>

On 28 Mar 2014 05:42, "Daniel Holth" <dholth at gmail.com> wrote:
>
> I became convinced that build was an inherently arbitrary-code
> process, and not something to be universally handled by a declarative
> system,

It wasn't an accident that last years PyCon panel was subtitled "setup.py
*install* must die" :)

As others have suggested, declarative build will never be more than an 80%
solution, and then beyond that, it should be a question of plumbing to
invoke the project's own build system (and ideally that plumbing will be
per build system, not per project).

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/e0dd9eda/attachment.html>

From ncoghlan at gmail.com  Thu Mar 27 22:27:28 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 28 Mar 2014 07:27:28 +1000
Subject: [Distutils] What is the syntax for passing conditional
 non-extra dependencies in setuptools?
In-Reply-To: <CADiSq7cHtBjr0rQM0VhFJ65hmr9iV9Fvo5jG6mgYT4tPqx=GvQ@mail.gmail.com>
References: <CAG8k2+5h2dRfZBODubcz4g-cQdBkMm8kYDMhhEYaNSqbDeP4YA@mail.gmail.com>
 <CALeMXf7vzZ6tENVFk_CV7-GhhCXPTJG-DoZ9L4qWszh40zfnrA@mail.gmail.com>
 <CADiSq7cHtBjr0rQM0VhFJ65hmr9iV9Fvo5jG6mgYT4tPqx=GvQ@mail.gmail.com>
Message-ID: <CADiSq7cNpYjjW3dy-hDdS0w-wmYEZe__p_r6UVoMu1RPVQZkpQ@mail.gmail.com>

On 28 Mar 2014 07:16, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>
> On 28 Mar 2014 04:40, "PJ Eby" <pje at telecommunity.com> wrote:
> >
> > On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth <dholth at gmail.com> wrote:
> >>
> >> How do I specify a conditional (marker-guarded) non-extra dependency
> >> in setuptools? The syntax for a conditional extra dependency is
> >> currently:
> >>
> >>     extras_require = {
> >>         "ssl:sys_platform=='win32'": "wincertstore==0.2",
> >>         "certs": "certifi==1.0.1",
> >>     },
> >
> >
> > I only implemented support via extras, and the feature wasn't
officially supported (still isn't, I don't think) because the PEP
specifying the syntax wasn't fully baked yet.  I figured that if *only*
setuptools itself used it, then if the syntax changed only setuptools would
break...  but fix itself at the same time.
> >
> > The same cannot be said for any other package, so use at your own
risk.  Or better yet, don't use it.  ;-)
> >
> > (At least, not until it's a documented feature w/a PEP-approved syntax.)
>
> Various things got in the way of finalising at least PEP 440, but as far
as I am aware, it, 426 and 459 are basically done aside from updating the
JSON schema specs and redoing the PyPI compatibility analysis.

Actually, there are a some other existing issues I need to address as well
(I especially want to do the SPDX one):
https://bitbucket.org/pypa/pypi-metadata-formats/issues?status=new&status=open&component=Metadata%202.x

Cheers,
Nick.

>
> The other thing they need is for people to read through from scratch,
looking for inconsistencies and things that seem out of place, and
reporting those as issues at
https://bitbucket.org/pypa/pypi-metadata-formats
>
> Cheers,
> Nick.
>
> >
> >
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/cccded11/attachment.html>

From ericsnowcurrently at gmail.com  Thu Mar 27 04:55:18 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 26 Mar 2014 21:55:18 -0600
Subject: [Distutils] [Import-SIG] nspkg.pth files break $PYTHONPATH
	overrides
In-Reply-To: <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>
References: <20140324174807.33712c4d@limelight.wooz.org>
 <F93EBAE2-47F2-4CCC-B198-31AEE8874629@stufft.io>
 <20140324180917.6d42a392@anarchist.wooz.org>
 <CALeMXf4yznp2KYuhD7DakCkEj=KWSYm6L62sGKzTwSL-YcNMxg@mail.gmail.com>
 <20140325155013.4f6c1dd9@anarchist.wooz.org>
 <CALeMXf7hfUwp73BGmZPsogZRkU-t1xS4sEKMYnjANhLLV0zEhA@mail.gmail.com>
Message-ID: <CALFfu7AWUb3xMQwaB04BKQkzfAOB9ceY6rXjp7HJdEYpNZ8SeA@mail.gmail.com>

On Tue, Mar 25, 2014 at 2:30 PM, PJ Eby <pje at telecommunity.com> wrote:
> On Tue, Mar 25, 2014 at 3:50 PM, Barry Warsaw <barry at python.org> wrote:
>> There is.  It's *pronounced* sys.namespace_packages, but it's spelled
>> importlib._bootstrap._NamespaceLoader ;)
>
>
> Yeah, well that's not exactly a public attribute, so it's not necessarily a
> great way to do it.  But if we did use that, presumably it'd add something
> like:
>
>     hnp =
> hasattr(sys.modules.get('importlib._bootstrap',None),'_NamespaceLoader');

In 3.4 it's called _NamespaceLoader, but in 3.3 it's NamespaceLoader. <ducks>

>
> To the front of the magic, and then prefix all subsequent expression values
> with 'not hnp and' in order to prevent them executing if PEP 420 support is
> available. (Note: it's checking sys.modules since on any interpreter where
> PEP 420 is natively available, the module should *already* be loaded by the
> time site.py does its thing.)
>
> And we should probably think about adding sys.namespace_packages or
> something of the sort, or at least a proper flag for whether PEP 420 support
> is available on the platform.

By "available on the platform" do you mean "Python 3.3+ or has a
backport installed"?  Otherwise you'd just check sys.version*.

-eric

From p.f.moore at gmail.com  Fri Mar 28 15:56:30 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 28 Mar 2014 14:56:30 +0000
Subject: [Distutils] Versioned entry points
Message-ID: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>

One open question with pip's handling of script generation for wheels
is what to do with versioned entry points (virtualenv-2.7.exe etc).

At the moment, pip explicitly disables distlib's facility to generate
"versioned" script wrappers for entry points, and generates exactly
what the project metadata specifies[1]. The problem is that some
projects use sys.version in setup.py to explicitly specify versioned
commands. This does not work for wheels, where the build and install
environments are different.

There's an open issue for virtualenv, which has hit this problem, and
I suspect there are a number of other projects which will hit it if &
when they start publishing wheels (I think py.test and nose have the
same problem, for a start).

The first question is, who should be responsible for deciding whether
to have versioned commands anyway? Is that a project decision, or
should it be left to the user doing the install? Should the user be
allowed to override a project default?

If it's a user decision, pip probably needs an additional command line
argument to specify whether the user wants versioned commands. We may
also want pip to ignore any existing entry point specifications that
look like they are versioned. (And of course someone needs to tell the
projects that currently specify versioned entry points that they
should stop :-))

If it's a project decision, we need a means for the project to record
that data in metadata that pip can read. For Metadata 2.0, that means
something in the console script metadata extension [2]. For the short
term, we may need some sort of convention or addition for
entry_points.txt. But I'm not sure who would be affected by something
like that (beyond the obvious answer of setuptools and pip...)

Does anyone have any thoughts?

Paul

[1] There's a hack in there to generate versioned entry points for pip
and easy_install.
[2] What's the process for agreeing on extension schemas? Do they need
their own PEPs, or what? I was skimming the discussion at the point
when these things got pushed out to extensions, so I missed the
detail.

From p.f.moore at gmail.com  Fri Mar 28 16:01:49 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 28 Mar 2014 15:01:49 +0000
Subject: [Distutils] Versioned entry points
In-Reply-To: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
Message-ID: <CACac1F-j1au4t1EZ7kmfQC87mCqE9_FBt=MK_ugZijHkbsvceA@mail.gmail.com>

On 28 March 2014 14:56, Paul Moore <p.f.moore at gmail.com> wrote:
> [2] What's the process for agreeing on extension schemas? Do they need
> their own PEPs, or what? I was skimming the discussion at the point
> when these things got pushed out to extensions, so I missed the
> detail.

I just found PEP 459. So I guess the question is should the "commands"
extension include some sort of "versioned" flag in the
wrap_console/wrap_gui data?

Paul

From dholth at gmail.com  Fri Mar 28 16:32:32 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 28 Mar 2014 11:32:32 -0400
Subject: [Distutils] Versioned entry points
In-Reply-To: <CACac1F-j1au4t1EZ7kmfQC87mCqE9_FBt=MK_ugZijHkbsvceA@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <CACac1F-j1au4t1EZ7kmfQC87mCqE9_FBt=MK_ugZijHkbsvceA@mail.gmail.com>
Message-ID: <CAG8k2+4NFJvmfTj3KfgMhqa1EvzM_f9P0QeZ7xg=+cGYkPX2zg@mail.gmail.com>

I would like to see just a very simple $ interpolation language.
On Mar 28, 2014 11:03 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:

> On 28 March 2014 14:56, Paul Moore <p.f.moore at gmail.com> wrote:
> > [2] What's the process for agreeing on extension schemas? Do they need
> > their own PEPs, or what? I was skimming the discussion at the point
> > when these things got pushed out to extensions, so I missed the
> > detail.
>
> I just found PEP 459. So I guess the question is should the "commands"
> extension include some sort of "versioned" flag in the
> wrap_console/wrap_gui data?
>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/ed49a31b/attachment.html>

From p.f.moore at gmail.com  Fri Mar 28 16:51:46 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 28 Mar 2014 15:51:46 +0000
Subject: [Distutils] Versioned entry points
In-Reply-To: <CAG8k2+4NFJvmfTj3KfgMhqa1EvzM_f9P0QeZ7xg=+cGYkPX2zg@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <CACac1F-j1au4t1EZ7kmfQC87mCqE9_FBt=MK_ugZijHkbsvceA@mail.gmail.com>
 <CAG8k2+4NFJvmfTj3KfgMhqa1EvzM_f9P0QeZ7xg=+cGYkPX2zg@mail.gmail.com>
Message-ID: <CACac1F_UpyTE_17TO8yvUUH9xJAfpBOwKsKZHmjhHD2Kh3gSqw@mail.gmail.com>

On 28 March 2014 15:32, Daniel Holth <dholth at gmail.com> wrote:
> I would like to see just a very simple $ interpolation language.

So you think the project should decide, not the user? Or should the
user be allowed to override?
Paul

From donald at stufft.io  Fri Mar 28 16:54:47 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 28 Mar 2014 11:54:47 -0400
Subject: [Distutils] Versioned entry points
In-Reply-To: <CACac1F_UpyTE_17TO8yvUUH9xJAfpBOwKsKZHmjhHD2Kh3gSqw@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <CACac1F-j1au4t1EZ7kmfQC87mCqE9_FBt=MK_ugZijHkbsvceA@mail.gmail.com>
 <CAG8k2+4NFJvmfTj3KfgMhqa1EvzM_f9P0QeZ7xg=+cGYkPX2zg@mail.gmail.com>
 <CACac1F_UpyTE_17TO8yvUUH9xJAfpBOwKsKZHmjhHD2Kh3gSqw@mail.gmail.com>
Message-ID: <BE74A7DB-2BD4-4482-993A-92C51B742E2A@stufft.io>


On Mar 28, 2014, at 11:51 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 28 March 2014 15:32, Daniel Holth <dholth at gmail.com> wrote:
>> I would like to see just a very simple $ interpolation language.
> 
> So you think the project should decide, not the user? Or should the
> user be allowed to override?
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

For what it?s worth, if the person invoking the pip command picks that would
remove a need for a hack we have in place for ensurepip.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/12862cae/attachment-0001.sig>

From dholth at gmail.com  Fri Mar 28 17:53:26 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 28 Mar 2014 12:53:26 -0400
Subject: [Distutils] Versioned entry points
In-Reply-To: <CACac1F_UpyTE_17TO8yvUUH9xJAfpBOwKsKZHmjhHD2Kh3gSqw@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <CACac1F-j1au4t1EZ7kmfQC87mCqE9_FBt=MK_ugZijHkbsvceA@mail.gmail.com>
 <CAG8k2+4NFJvmfTj3KfgMhqa1EvzM_f9P0QeZ7xg=+cGYkPX2zg@mail.gmail.com>
 <CACac1F_UpyTE_17TO8yvUUH9xJAfpBOwKsKZHmjhHD2Kh3gSqw@mail.gmail.com>
Message-ID: <CAG8k2+5ip1shLn-cYphre-2jqbofQi1t5aNVqj3hX8FWhhQfHw@mail.gmail.com>

Definitely the project.
On Mar 28, 2014 11:51 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:

> On 28 March 2014 15:32, Daniel Holth <dholth at gmail.com> wrote:
> > I would like to see just a very simple $ interpolation language.
>
> So you think the project should decide, not the user? Or should the
> user be allowed to override?
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/df1d7364/attachment.html>

From vinay_sajip at yahoo.co.uk  Fri Mar 28 18:18:17 2014
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 28 Mar 2014 17:18:17 +0000 (GMT)
Subject: [Distutils] Versioned entry points
In-Reply-To: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
Message-ID: <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>

>?Does anyone have any thoughts?

Note that there is one situation where scripts for multiple Python versions reside in the same directory: per-user site-packages (PEP 370). If you install a package which has scripts and you don't write versioned scripts, a second installation of that package (with a different Python version) will cause the first script to be overwritten. Much of the time this won't matter, but there might be scenarios where it does.

It may also give a problem when uninstalling, or when verifying the installation (as the hash of an overwritten script may not match what's expected, as per the original installation).

To avoid overwriting, one would need to write versioned scripts *only* :-)

Regards,

Vinay Sajip

From tseaver at palladion.com  Fri Mar 28 19:24:01 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 28 Mar 2014 14:24:01 -0400
Subject: [Distutils] Versioned entry points
In-Reply-To: <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
Message-ID: <lh4enm$vum$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/28/2014 01:18 PM, Vinay Sajip wrote:
>> Does anyone have any thoughts?
> 
> Note that there is one situation where scripts for multiple Python 
> versions reside in the same directory: per-user site-packages (PEP
> 370). If you install a package which has scripts and you don't write
> versioned scripts, a second installation of that package (with a
> different Python version) will cause the first script to be
> overwritten. Much of the time this won't matter, but there might be
> scenarios where it does.
> 
> It may also give a problem when uninstalling, or when verifying the 
> installation (as the hash of an overwritten script may not match
> what's expected, as per the original installation).
> 
> To avoid overwriting, one would need to write versioned scripts
> *only* :-)

Heh, sounds like a call for Underdog^W:

  $ python setup.py altinstall


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlM1vkAACgkQ+gerLs4ltQ5CLACgodRwkXiS9irK/nP3dKPnlaXO
5GYAoKbJ4NiDFJdHNIWPozIpMzN6+z62
=q7Bh
-----END PGP SIGNATURE-----


From dholth at gmail.com  Fri Mar 28 20:06:59 2014
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 28 Mar 2014 15:06:59 -0400
Subject: [Distutils] Pycon
Message-ID: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>

Who is going to pycon? I will be there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/a9b41109/attachment.html>

From wickman at gmail.com  Fri Mar 28 20:22:33 2014
From: wickman at gmail.com (Brian Wickman)
Date: Fri, 28 Mar 2014 12:22:33 -0700
Subject: [Distutils] Pycon
In-Reply-To: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <CAAmsm4iVaRzxCBFT=StB3E9RexRwWKq_jv_j-K1SXAb_D6RUfg@mail.gmail.com>

+1.  Will be hanging out at the http://t.co/PyLadiesMixer and the Twitter
booth a bit should anyone want to chat python packaging.


On Fri, Mar 28, 2014 at 12:06 PM, Daniel Holth <dholth at gmail.com> wrote:

> Who is going to pycon? I will be there.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/38ae6d13/attachment.html>

From donald at stufft.io  Fri Mar 28 20:28:46 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 28 Mar 2014 15:28:46 -0400
Subject: [Distutils] Pycon
In-Reply-To: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <0B4148BC-0B62-46F5-B994-FE5A04C743E8@stufft.io>

No :(

On Mar 28, 2014, at 3:06 PM, Daniel Holth <dholth at gmail.com> wrote:

> Who is going to pycon? I will be there.
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/5002450f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/5002450f/attachment-0001.sig>

From tseaver at palladion.com  Fri Mar 28 20:48:47 2014
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 28 Mar 2014 15:48:47 -0400
Subject: [Distutils] Pycon
In-Reply-To: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <lh4jml$trd$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/28/2014 03:06 PM, Daniel Holth wrote:
> Who is going to pycon? I will be there.

I'll be there Tuesday evening through Saturday morning.  I'd love to buy
any fellowship^wPyPA members a beverage-of-choice.



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlM10h8ACgkQ+gerLs4ltQ48ewCgsyGo69m2iKAHAUtiC0/43G01
iTgAoKfZAWphPIoQj5Iih/qIJ3Z8hTqR
=ETBo
-----END PGP SIGNATURE-----


From ncoghlan at gmail.com  Fri Mar 28 22:06:35 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Mar 2014 07:06:35 +1000
Subject: [Distutils] Versioned entry points
In-Reply-To: <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
Message-ID: <CADiSq7fW9-ETb9NSA_g9d7qO8ih28JNU9qyG0Us8az7AG2UTTA@mail.gmail.com>

On 29 Mar 2014 06:20, "Vinay Sajip" <vinay_sajip at yahoo.co.uk> wrote:
>
> > Does anyone have any thoughts?
>
> Note that there is one situation where scripts for multiple Python
versions reside in the same directory: per-user site-packages (PEP 370). If
you install a package which has scripts and you don't write versioned
scripts, a second installation of that package (with a different Python
version) will cause the first script to be overwritten. Much of the time
this won't matter, but there might be scenarios where it does.
>
> It may also give a problem when uninstalling, or when verifying the
installation (as the hash of an overwritten script may not match what's
expected, as per the original installation).
>
> To avoid overwriting, one would need to write versioned scripts *only* :-)

Speaking of which, it's worth describing exactly what we had to do for
ensurepip. That has 3 configurations:

POSIX system default: pip3, pip3.4
Windows & venv default: pip, pip3, pip3.4
Alt install: pip3.4

There's currently an issue where "make altinstall && make install" is
slightly broken, as it won't add the missing "pip3" command.

So consider me in the school that suggests this be a standard installer
feature that can be applied to any entry point script, rather than
something that varies by project.

Cheers,
Nick.

>
> Regards,
>
> Vinay Sajip
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140329/ccbb35fc/attachment.html>

From p.f.moore at gmail.com  Fri Mar 28 22:23:00 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 28 Mar 2014 21:23:00 +0000
Subject: [Distutils] Versioned entry points
In-Reply-To: <CADiSq7fW9-ETb9NSA_g9d7qO8ih28JNU9qyG0Us8az7AG2UTTA@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
 <CADiSq7fW9-ETb9NSA_g9d7qO8ih28JNU9qyG0Us8az7AG2UTTA@mail.gmail.com>
Message-ID: <CACac1F9Rktot2fR2twRYos9kriyNsxkFq5DMUTx+S+_ka5G5Zg@mail.gmail.com>

On 28 March 2014 21:06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> So consider me in the school that suggests this be a standard installer
> feature that can be applied to any entry point script, rather than something
> that varies by project.

Doing that also implies (detecting and) deliberately ignoring any
versioned entry points created by setup.py. I'd suggest going for a
rule that if a project defines two entry points with the same
definitions, where one ends in the pattern -?\d(\.\d)? (in other
words, -X, -X.Y, X or X.Y) but they are otherwise the same, then
ignore the versioned one. If we don't do this, we'll likely end up
with abominations like virtualenv-2.7-3.4.exe.

I'm not 100% sure whether you're saying that the installer should
mandate a versioning policy, or if the user can specify. The
altinstall case suggests we need a user override option.

There's also the question of if I do "pip install virtualenv
--use-versioned X,X.Y" and then "pip install -U virtualenv" will the
installer remember my versioning choices for the upgrade? Doing so
needs some complex logic. Not doing so means that "pip install -U pip"
could change the executables ensurepip had chosen to create.

Paul

From p.f.moore at gmail.com  Fri Mar 28 22:23:51 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 28 Mar 2014 21:23:51 +0000
Subject: [Distutils] Pycon
In-Reply-To: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <CACac1F936PLqfCvGsaOTJ5WQbqGSyPQhbN-sO=1W6W132hj84g@mail.gmail.com>

On 28 March 2014 19:06, Daniel Holth <dholth at gmail.com> wrote:
> Who is going to pycon? I will be there.

Sadly, I won't.
Paul

From noah at coderanger.net  Fri Mar 28 22:26:00 2014
From: noah at coderanger.net (Noah Kantrowitz)
Date: Fri, 28 Mar 2014 14:26:00 -0700
Subject: [Distutils] Pycon
In-Reply-To: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <AAEAECE1-6EFD-4F57-9634-C8B15ABF5652@coderanger.net>


On Mar 28, 2014, at 12:06 PM, Daniel Holth <dholth at gmail.com> wrote:

> Who is going to pycon? I will be there.

Attending and presenting a talk that can tl;dr'd as a summary of the last 18 months of this list.

--Noah

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 163 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/06e73292/attachment.sig>

From chris.barker at noaa.gov  Fri Mar 28 23:17:37 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 28 Mar 2014 15:17:37 -0700
Subject: [Distutils] Is build an inherently arbitrary-code process?
In-Reply-To: <CADiSq7fE5k0treBWWsnO_q-Y43aQSajp0KxM=jKJgGnA1HbKTA@mail.gmail.com>
References: <CAG8k2+67pp9QBCgW0xFjLMjOMTd3MU8GLHpQR8Qx-VNewq3t9g@mail.gmail.com>
 <CADiSq7fE5k0treBWWsnO_q-Y43aQSajp0KxM=jKJgGnA1HbKTA@mail.gmail.com>
Message-ID: <CALGmxEJoeDzVonEbKVTWvuYGy_zzApw1_0rW-vzY-D3k0ZORtQ@mail.gmail.com>

On Thu, Mar 27, 2014 at 2:23 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 28 Mar 2014 05:42, "Daniel Holth" <dholth at gmail.com> wrote:
> > I became convinced that build was an inherently arbitrary-code
> > process, and not something to be universally handled by a declarative
> > system,
>
> It wasn't an accident that last years PyCon panel was subtitled "setup.py
> *install* must die" :)
>
> As others have suggested, declarative build will never be more than an 80%
> solution, and then beyond that, it should be a question of plumbing to
> invoke the project's own build system (and ideally that plumbing will be
> per build system, not per project).
>
Agreed -- I have been poking at this a bit, and trying to make gattai work
for me:

http://sourceforge.net/projects/gattai/

(It's a by and for python build system that essentially invokes the native
build systems : make, setup.py, nmake, etc... But that's not really the
point). the point is that I hit a wall pretty quickly with
its declarative approach (JSON files). I find I can't do what I need to do
with straight declaration (and I'm hacking gattai to support that)

However, for the most part, all I need to be able to do is run arbitrary
code to set up the declarations. The stuff that's been talked about:
finding the right libraries to link to, that sort of thing.

I actually think that aspect of the setup.py approach works pretty well,
even though it does sometimes get kind of messy.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/e67938c4/attachment-0001.html>

From ncoghlan at gmail.com  Fri Mar 28 23:51:35 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Mar 2014 08:51:35 +1000
Subject: [Distutils] Versioned entry points
In-Reply-To: <CACac1F9Rktot2fR2twRYos9kriyNsxkFq5DMUTx+S+_ka5G5Zg@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
 <CADiSq7fW9-ETb9NSA_g9d7qO8ih28JNU9qyG0Us8az7AG2UTTA@mail.gmail.com>
 <CACac1F9Rktot2fR2twRYos9kriyNsxkFq5DMUTx+S+_ka5G5Zg@mail.gmail.com>
Message-ID: <CADiSq7eeYWMxOOTCMc83fOt3cwAc9KWgr+WbHAykZgL_3TKj_A@mail.gmail.com>

On 29 March 2014 07:23, Paul Moore <p.f.moore at gmail.com> wrote:
> On 28 March 2014 21:06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> So consider me in the school that suggests this be a standard installer
>> feature that can be applied to any entry point script, rather than something
>> that varies by project.
>
> Doing that also implies (detecting and) deliberately ignoring any
> versioned entry points created by setup.py. I'd suggest going for a
> rule that if a project defines two entry points with the same
> definitions, where one ends in the pattern -?\d(\.\d)? (in other
> words, -X, -X.Y, X or X.Y) but they are otherwise the same, then
> ignore the versioned one. If we don't do this, we'll likely end up
> with abominations like virtualenv-2.7-3.4.exe.

We need to be careful with that - scripts may end with a version
number unrelated to the Python version. If we can get agreement on
what we want the *default* idiom to look like (more on that below),
then we can start considering the special cases where that idiom
breaks down (like those where the base script name already ends with a
number, perhaps by inserting a leading "-py" into the suffix in those
cases).

> I'm not 100% sure whether you're saying that the installer should
> mandate a versioning policy, or if the user can specify. The
> altinstall case suggests we need a user override option.

I'm biased because there are actually two places where this matters.
It's not just the user scripts directory (everywhere), but also the
system binary directories on Linux distros.

(Note: I haven't run the ideas below by Slavek or Toshio yet, so take
the finer details with a grain of salt - but the general principles
should be right. I don't believe Slavek is making it to PyCon, but
Toshio is, so we should be able to thrash out some more specifics
there)

>From a Linux distro perspective, these versioned scripts should
arguably have different shebang lines - ignoring the parts that would
remain the same, "pip" would correspond to "python", "pip3" to
"python3" and "pip3.4" to "python3.4". Otherwise you
could change the "python" symlink to point somewhere else, and then
have to figure out why "pip" appeared to succeed, but you couldn't
import things you just installed (because they were installed into the
wrong version).

For system wide installs, *which* ones to install should also be
possible to largely automate by following a convention based on the
final name element in sys.executable (and that approach would also fix
the problem of potential conflicts in the user scripts directory).

There's also the fact that for universal (or otherwise cross-version)
wheels, which scripts get installed should be governed by the version
of Python used for the *installation*, rather than the one used to do
the build.

If we put this under the control of the installer, then there is some
hope of eventually bringing order to the chaos and getting some kind
of consistency in naming schemes and alignment with the shebang lines.
If we leave it up to the individual projects? Not a chance (especially
since we *can't* get it right if the versioned scripts are embedded in
wheel files at build time).

> There's also the question of if I do "pip install virtualenv
> --use-versioned X,X.Y" and then "pip install -U virtualenv" will the
> installer remember my versioning choices for the upgrade? Doing so
> needs some complex logic. Not doing so means that "pip install -U pip"
> could change the executables ensurepip had chosen to create.

Yeah, I think driving it based on sys.executable is a better idea,
with a fallback of "-<interpretername>" as the suffix if the final
component of sys.executable doesn't follow one of the recognised
patterns (python, pythonX, pythonX.Y). The reason for using
sys.executable is because that is the thing that can tell us the
interpreter's role *with respect to the larger system*.

Under that model, on a current Fedora system, we would see the
following behaviour when run under the following interpreters:

python -> pip, pip2, pip2.7
python2 -> pip2, pip2.7
python2.7 -> pip2.7
python3 -> pip3, pip3.3
python3.3 -> pip3.3
pypy -> pip-pypy

That avoids collisions not only in /usr/bin (or /usr/local/bin if we
change the install directory for scripts on POSIX), but also in
~/.local/bin (and the per-user directory on Windows).

On Windows system wide installs, since the Scripts directory path
already includes the Python version, I'd just install all 3 in all
cases. Ditto for virtual environments (even when using an alternate
interpreter implementation).

That's essentially using ensurepip's behaviour as precedent for how we
*want* Python scripts to behave.

You may ask "why would you ever do that?", and from my perspective,
the answer is because Fedora is migrating over to using "pip" in the
RPM build process for Python packages, rather than calling setup.py
directly. This means we can more easily adapt to changes in upstream
conventions and automatically take advantage of the upcoming metadata
2.0 enhancements and related installation database changes. However,
it also means we're relying on being able to get pip to play nice with
Fedora's dual stack python2/python3 setup, even as we start migrating
more and more components over to Python 3

The one "user" (i.e. "specfile author") controlled behaviour we would
likely want is the ability to decide whether Python 2 or Python 3 was
considered the default for a given module - when pure applications
that just happen to be written in Python (rather than Python libraries
or Python specific tools) cut over to Python 3, they'll likely just
switch entirely, rather than going the dual stack Python 2/3 approach.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Fri Mar 28 23:55:46 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 28 Mar 2014 18:55:46 -0400
Subject: [Distutils] Versioned entry points
In-Reply-To: <CADiSq7eeYWMxOOTCMc83fOt3cwAc9KWgr+WbHAykZgL_3TKj_A@mail.gmail.com>
References: <CACac1F96P62UuYi5jriHmwta=-eKGQ9bmEc_mqMRQYsriF=NaA@mail.gmail.com>
 <1396027097.49978.YahooMailNeo@web172401.mail.ir2.yahoo.com>
 <CADiSq7fW9-ETb9NSA_g9d7qO8ih28JNU9qyG0Us8az7AG2UTTA@mail.gmail.com>
 <CACac1F9Rktot2fR2twRYos9kriyNsxkFq5DMUTx+S+_ka5G5Zg@mail.gmail.com>
 <CADiSq7eeYWMxOOTCMc83fOt3cwAc9KWgr+WbHAykZgL_3TKj_A@mail.gmail.com>
Message-ID: <95EFDB0C-452F-4DFE-ACC1-188FAAC859F9@stufft.io>


On Mar 28, 2014, at 6:51 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 29 March 2014 07:23, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 28 March 2014 21:06, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> So consider me in the school that suggests this be a standard installer
>>> feature that can be applied to any entry point script, rather than something
>>> that varies by project.
>> 
>> Doing that also implies (detecting and) deliberately ignoring any
>> versioned entry points created by setup.py. I'd suggest going for a
>> rule that if a project defines two entry points with the same
>> definitions, where one ends in the pattern -?\d(\.\d)? (in other
>> words, -X, -X.Y, X or X.Y) but they are otherwise the same, then
>> ignore the versioned one. If we don't do this, we'll likely end up
>> with abominations like virtualenv-2.7-3.4.exe.
> 
> We need to be careful with that - scripts may end with a version
> number unrelated to the Python version. If we can get agreement on
> what we want the *default* idiom to look like (more on that below),
> then we can start considering the special cases where that idiom
> breaks down (like those where the base script name already ends with a
> number, perhaps by inserting a leading "-py" into the suffix in those
> cases).
> 
>> I'm not 100% sure whether you're saying that the installer should
>> mandate a versioning policy, or if the user can specify. The
>> altinstall case suggests we need a user override option.
> 
> I'm biased because there are actually two places where this matters.
> It's not just the user scripts directory (everywhere), but also the
> system binary directories on Linux distros.
> 
> (Note: I haven't run the ideas below by Slavek or Toshio yet, so take
> the finer details with a grain of salt - but the general principles
> should be right. I don't believe Slavek is making it to PyCon, but
> Toshio is, so we should be able to thrash out some more specifics
> there)
> 
> From a Linux distro perspective, these versioned scripts should
> arguably have different shebang lines - ignoring the parts that would
> remain the same, "pip" would correspond to "python", "pip3" to
> "python3" and "pip3.4" to "python3.4". Otherwise you
> could change the "python" symlink to point somewhere else, and then
> have to figure out why "pip" appeared to succeed, but you couldn't
> import things you just installed (because they were installed into the
> wrong version).
> 
> For system wide installs, *which* ones to install should also be
> possible to largely automate by following a convention based on the
> final name element in sys.executable (and that approach would also fix
> the problem of potential conflicts in the user scripts directory).
> 
> There's also the fact that for universal (or otherwise cross-version)
> wheels, which scripts get installed should be governed by the version
> of Python used for the *installation*, rather than the one used to do
> the build.
> 
> If we put this under the control of the installer, then there is some
> hope of eventually bringing order to the chaos and getting some kind
> of consistency in naming schemes and alignment with the shebang lines.
> If we leave it up to the individual projects? Not a chance (especially
> since we *can't* get it right if the versioned scripts are embedded in
> wheel files at build time).
> 
>> There's also the question of if I do "pip install virtualenv
>> --use-versioned X,X.Y" and then "pip install -U virtualenv" will the
>> installer remember my versioning choices for the upgrade? Doing so
>> needs some complex logic. Not doing so means that "pip install -U pip"
>> could change the executables ensurepip had chosen to create.
> 
> Yeah, I think driving it based on sys.executable is a better idea,
> with a fallback of "-<interpretername>" as the suffix if the final
> component of sys.executable doesn't follow one of the recognised
> patterns (python, pythonX, pythonX.Y). The reason for using
> sys.executable is because that is the thing that can tell us the
> interpreter's role *with respect to the larger system*.
> 
> Under that model, on a current Fedora system, we would see the
> following behaviour when run under the following interpreters:
> 
> python -> pip, pip2, pip2.7
> python2 -> pip2, pip2.7
> python2.7 -> pip2.7
> python3 -> pip3, pip3.3
> python3.3 -> pip3.3
> pypy -> pip-pypy
> 
> That avoids collisions not only in /usr/bin (or /usr/local/bin if we
> change the install directory for scripts on POSIX), but also in
> ~/.local/bin (and the per-user directory on Windows).
> 
> On Windows system wide installs, since the Scripts directory path
> already includes the Python version, I'd just install all 3 in all
> cases. Ditto for virtual environments (even when using an alternate
> interpreter implementation).
> 
> That's essentially using ensurepip's behaviour as precedent for how we
> *want* Python scripts to behave.
> 
> You may ask "why would you ever do that?", and from my perspective,
> the answer is because Fedora is migrating over to using "pip" in the
> RPM build process for Python packages, rather than calling setup.py
> directly. This means we can more easily adapt to changes in upstream
> conventions and automatically take advantage of the upcoming metadata
> 2.0 enhancements and related installation database changes. However,
> it also means we're relying on being able to get pip to play nice with
> Fedora's dual stack python2/python3 setup, even as we start migrating
> more and more components over to Python 3
> 
> The one "user" (i.e. "specfile author") controlled behaviour we would
> likely want is the ability to decide whether Python 2 or Python 3 was
> considered the default for a given module - when pure applications
> that just happen to be written in Python (rather than Python libraries
> or Python specific tools) cut over to Python 3, they'll likely just
> switch entirely, rather than going the dual stack Python 2/3 approach.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

This all sounds reasonable to me. Ultimately the end user should be
in control of what gets put on their machine. They are the only one
who know which scheme makes sense.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/939cfb82/attachment.sig>

From ncoghlan at gmail.com  Sat Mar 29 00:10:39 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Mar 2014 09:10:39 +1000
Subject: [Distutils] Pycon
In-Reply-To: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>

On 29 March 2014 05:06, Daniel Holth <dholth at gmail.com> wrote:
> Who is going to pycon? I will be there.

As will I. ?ric Araujo was suggesting we do a Packaging Mini-Conf
again as an open space, which sounded like a good idea to me, I just
wasn't inclined to organise it myself this year :)

Open space page is at https://us.pycon.org/2014/community/openspaces/,
but I don't know if ?ric already submitted a request for a space.

Also not sure if it would be better to do it before or after Noah's
talk (the advantage of before is we might give him additional ideas,
after is that we can go watch his talk first as a "year in review"
kind of thing to remind us of how far we've come since this time last
year)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From richard at python.org  Sat Mar 29 02:20:20 2014
From: richard at python.org (Richard Jones)
Date: Sat, 29 Mar 2014 12:20:20 +1100
Subject: [Distutils] Pycon
In-Reply-To: <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
Message-ID: <CAHrZfZBo7t1ZVeCkTHTvuW=F79c48iCLO5BW59d6qCBtiyyqTw@mail.gmail.com>

I'll be there, doing my best to hold up the PyPI / Warehouse banner. And
work on some of the code during the sprints, assuming I don't get
distracted by writing another PEP like last time (but look where that
eventually led us :)

I am most likely going to go to the language summit as a lurker to see if
there's any PyPI-related stuff discussed, but I'm not sure of the actual
value of that.


     Richard


On 29 March 2014 10:10, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 29 March 2014 05:06, Daniel Holth <dholth at gmail.com> wrote:
> > Who is going to pycon? I will be there.
>
> As will I. ?ric Araujo was suggesting we do a Packaging Mini-Conf
> again as an open space, which sounded like a good idea to me, I just
> wasn't inclined to organise it myself this year :)
>
> Open space page is at https://us.pycon.org/2014/community/openspaces/,
> but I don't know if ?ric already submitted a request for a space.
>
> Also not sure if it would be better to do it before or after Noah's
> talk (the advantage of before is we might give him additional ideas,
> after is that we can go watch his talk first as a "year in review"
> kind of thing to remind us of how far we've come since this time last
> year)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140329/b54874bd/attachment-0001.html>

From jcappos at nyu.edu  Sat Mar 29 03:31:44 2014
From: jcappos at nyu.edu (Justin Cappos)
Date: Fri, 28 Mar 2014 22:31:44 -0400
Subject: [Distutils] Pycon
In-Reply-To: <CAHrZfZBo7t1ZVeCkTHTvuW=F79c48iCLO5BW59d6qCBtiyyqTw@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <CAHrZfZBo7t1ZVeCkTHTvuW=F79c48iCLO5BW59d6qCBtiyyqTw@mail.gmail.com>
Message-ID: <CAMVss_ozp9LyXyd3M+jgYCLhqcrc1YJfCY4PFQMveZPD2R4T+g@mail.gmail.com>

It sounds like a great crowd!   I'm sorry that no one from my group will be
there...   :(

We got caught up with other things and then registration filled up...

If anything related to things we've been working on (TUF, etc.) comes up,
feel free to ping us on Skype, etc.

Thanks,
Justin


On Fri, Mar 28, 2014 at 9:20 PM, Richard Jones <richard at python.org> wrote:

> I'll be there, doing my best to hold up the PyPI / Warehouse banner. And
> work on some of the code during the sprints, assuming I don't get
> distracted by writing another PEP like last time (but look where that
> eventually led us :)
>
> I am most likely going to go to the language summit as a lurker to see if
> there's any PyPI-related stuff discussed, but I'm not sure of the actual
> value of that.
>
>
>      Richard
>
>
> On 29 March 2014 10:10, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 29 March 2014 05:06, Daniel Holth <dholth at gmail.com> wrote:
>> > Who is going to pycon? I will be there.
>>
>> As will I. ?ric Araujo was suggesting we do a Packaging Mini-Conf
>> again as an open space, which sounded like a good idea to me, I just
>> wasn't inclined to organise it myself this year :)
>>
>> Open space page is at https://us.pycon.org/2014/community/openspaces/,
>> but I don't know if ?ric already submitted a request for a space.
>>
>> Also not sure if it would be better to do it before or after Noah's
>> talk (the advantage of before is we might give him additional ideas,
>> after is that we can go watch his talk first as a "year in review"
>> kind of thing to remind us of how far we've come since this time last
>> year)
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/1c27d723/attachment.html>

From lists at asheesh.org  Sat Mar 29 06:35:59 2014
From: lists at asheesh.org (Asheesh Laroia)
Date: Fri, 28 Mar 2014 22:35:59 -0700
Subject: [Distutils] Pycon
In-Reply-To: <CAMVss_ozp9LyXyd3M+jgYCLhqcrc1YJfCY4PFQMveZPD2R4T+g@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <CAHrZfZBo7t1ZVeCkTHTvuW=F79c48iCLO5BW59d6qCBtiyyqTw@mail.gmail.com>
 <CAMVss_ozp9LyXyd3M+jgYCLhqcrc1YJfCY4PFQMveZPD2R4T+g@mail.gmail.com>
Message-ID: <CAFyrOVQW1_Nexcy3qp0O5mrFUK44ucA+omX2sAKBiwBTeKdNgQ@mail.gmail.com>

I'm attempting to give a talk that is supposed to be an outsider's view of
packaging, and I'll be there! "Python packaging simplified, for end users,
app developers, and open source contributors" <
https://us.pycon.org/2014/schedule/presentation/219/>

Noah, maybe we should talk about the talk -- I hope to talk up wheels, for
example.

Relatedly, is it OK if I ask questions here as part of prepping for said
talk...?


On Fri, Mar 28, 2014 at 7:31 PM, Justin Cappos <jcappos at nyu.edu> wrote:

> It sounds like a great crowd!   I'm sorry that no one from my group will
> be there...   :(
>
> We got caught up with other things and then registration filled up...
>
> If anything related to things we've been working on (TUF, etc.) comes up,
> feel free to ping us on Skype, etc.
>
> Thanks,
> Justin
>
>
> On Fri, Mar 28, 2014 at 9:20 PM, Richard Jones <richard at python.org> wrote:
>
>> I'll be there, doing my best to hold up the PyPI / Warehouse banner. And
>> work on some of the code during the sprints, assuming I don't get
>> distracted by writing another PEP like last time (but look where that
>> eventually led us :)
>>
>> I am most likely going to go to the language summit as a lurker to see if
>> there's any PyPI-related stuff discussed, but I'm not sure of the actual
>> value of that.
>>
>>
>>      Richard
>>
>>
>> On 29 March 2014 10:10, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> On 29 March 2014 05:06, Daniel Holth <dholth at gmail.com> wrote:
>>> > Who is going to pycon? I will be there.
>>>
>>> As will I. ?ric Araujo was suggesting we do a Packaging Mini-Conf
>>> again as an open space, which sounded like a good idea to me, I just
>>> wasn't inclined to organise it myself this year :)
>>>
>>> Open space page is at https://us.pycon.org/2014/community/openspaces/,
>>> but I don't know if ?ric already submitted a request for a space.
>>>
>>> Also not sure if it would be better to do it before or after Noah's
>>> talk (the advantage of before is we might give him additional ideas,
>>> after is that we can go watch his talk first as a "year in review"
>>> kind of thing to remind us of how far we've come since this time last
>>> year)
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140328/1a8f8361/attachment.html>

From ncoghlan at gmail.com  Sat Mar 29 07:29:29 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 29 Mar 2014 16:29:29 +1000
Subject: [Distutils] Pycon
In-Reply-To: <CAFyrOVQW1_Nexcy3qp0O5mrFUK44ucA+omX2sAKBiwBTeKdNgQ@mail.gmail.com>
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
 <CADiSq7dP4hk8M2EGf3m2vsR77hSgd=8qYk8vVFO5LE+k=E6teQ@mail.gmail.com>
 <CAHrZfZBo7t1ZVeCkTHTvuW=F79c48iCLO5BW59d6qCBtiyyqTw@mail.gmail.com>
 <CAMVss_ozp9LyXyd3M+jgYCLhqcrc1YJfCY4PFQMveZPD2R4T+g@mail.gmail.com>
 <CAFyrOVQW1_Nexcy3qp0O5mrFUK44ucA+omX2sAKBiwBTeKdNgQ@mail.gmail.com>
Message-ID: <CADiSq7eHt2_c3hKcTziED8e4rvEgmU8EKR=cWqufOYAET1WbCA@mail.gmail.com>

On 29 March 2014 15:35, Asheesh Laroia <lists at asheesh.org> wrote:
> I'm attempting to give a talk that is supposed to be an outsider's view of
> packaging, and I'll be there! "Python packaging simplified, for end users,
> app developers, and open source contributors"
> <https://us.pycon.org/2014/schedule/presentation/219/>
>
> Noah, maybe we should talk about the talk -- I hope to talk up wheels, for
> example.
>
> Relatedly, is it OK if I ask questions here as part of prepping for said
> talk...?

Absolutely! Your feedback on the discoverability of the new 3.4+ docs
structure would also be greatly appreciated.

Core installation & distribution docs:

* http://docs.python.org/3/installing/index.html
* http://docs.python.org/3/distributing/index.html
* http://packaging.python.org/

(I also moved installing to the top of the right hand column in the
main docs index, and the embedding and extending guide down to the
middle)

Binary extension docs:

* http://docs.python.org/3.4/extending/index.html
* https://packaging.python.org/en/latest/extensions.html

It also occurred to me that we don't currently have a great "get
involved" setup for filling in the wide array of "FIXMEs" in the main
guide. I just added the guide to the top of the project summary page
at http://packaging.python.org/en/latest/projects.html, so it's a bit
easier to learn that the guide is itself a project people are able to
help with.

Perhaps that's something a sprint could be organised for after PyCon?
(I can see myself wandering between a lot of different sprint rooms at
the rate this is going...)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From bcannon at gmail.com  Sat Mar 29 14:37:04 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Sat, 29 Mar 2014 13:37:04 +0000
Subject: [Distutils]  Pycon
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <CAP1=2W5Of6+_ZAQdPGVXV_nUA-VUALrns18E9t1Hnb05KyFgHg@mail.gmail.com>

I'm arriving Tuesday night and not leaving until Wednesday morning of the
sprints.

On Fri Mar 28 2014 at 3:07:26 PM, Daniel Holth <dholth at gmail.com> wrote:

> Who is going to pycon? I will be there.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140329/21b6064b/attachment.html>

From donald at stufft.io  Sun Mar 30 00:31:43 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 29 Mar 2014 19:31:43 -0400
Subject: [Distutils] DOAP on PyPI
Message-ID: <57690961-8A0A-4F81-9E38-7EA2C2CEA69A@stufft.io>

Hey all!

I was looking at porting more things over to Warehouse (PyPI 2.0) and one of
the things left on the TODO list was the DOAP URLs. I'm not sure why they were
added originally but I'm not entirely sure they add any value to the service.

Looking at the past day's worth of traffic they got ~90k total hits, but almost
all of those 90k hits came from a single IP address using wget. Looking at the
traffic patterns for the accesses from that IP/client it appears to be someone
using wget as a simple mirroring client for PyPI and thus they are picking up
the DOAP URLs because they are linked from the the PyPI pages for those
projects.

Further more it appears interest in DOAP as a whole isn't very high, the
GitHub repository [1] has had 10 commits ever, 2 of which are importing the
history of the project 2 years ago, 2 of which are merge commits for two
different pull requests, and the remaining 6 come from the two pull requests.
The mailing list [2] has had no traffic since 2012.

So I would like to remove this functionality from PyPI by not implementing it
in Warehouse. It appears to be practically unused and the DOAP project itself
seems to have stalled or died. Is there anyone depending on DOAP? Would anyone
be upset if this went away?


[1] https://github.com/edumbill/doap
[2] http://lists.usefulinc.com/pipermail/doap-interest/

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140329/8fa85eea/attachment.sig>

From ncoghlan at gmail.com  Sun Mar 30 01:09:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 30 Mar 2014 10:09:53 +1000
Subject: [Distutils] DOAP on PyPI
In-Reply-To: <57690961-8A0A-4F81-9E38-7EA2C2CEA69A@stufft.io>
References: <57690961-8A0A-4F81-9E38-7EA2C2CEA69A@stufft.io>
Message-ID: <CADiSq7e4XqL0=Pna8fMezLyFR4phLazsBUQSEfm-GhsY6XSB1Q@mail.gmail.com>

On 30 March 2014 09:31, Donald Stufft <donald at stufft.io> wrote:
> Further more it appears interest in DOAP as a whole isn't very high, the
> GitHub repository [1] has had 10 commits ever, 2 of which are importing the
> history of the project 2 years ago, 2 of which are merge commits for two
> different pull requests, and the remaining 6 come from the two pull requests.
> The mailing list [2] has had no traffic since 2012.
>
> So I would like to remove this functionality from PyPI by not implementing it
> in Warehouse. It appears to be practically unused and the DOAP project itself
> seems to have stalled or died. Is there anyone depending on DOAP? Would anyone
> be upset if this went away?

I get the impression that SPDX killed it: https://spdx.org/

While there are still a couple of references to DOAP defined fields in
the SPDX spec, I suspect your overall impression that the DOAP format
itself is dead is correct.

So yeah, skipping implementing those URLs for the migration makes
sense to me. If anyone complains, we can always add them back later (I
doubt anyone will complain, though).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Mon Mar 31 19:25:25 2014
From: barry at python.org (Barry Warsaw)
Date: Mon, 31 Mar 2014 13:25:25 -0400
Subject: [Distutils] Pycon
References: <CAG8k2+6em=aukeG=WY7SWLU0O0z4Ay+oveb+L-rDM679-kAm7Q@mail.gmail.com>
Message-ID: <20140331132525.38a363ec@anarchist.wooz.org>

On Mar 28, 2014, at 03:06 PM, Daniel Holth wrote:

>Who is going to pycon? I will be there.

I'll be there, for the duration (language summit through sprints).  It would
be great to hav an OpenSpace or BoF for discussing the intersection of Python
packaging issues and distros.

-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140331/b7265a54/attachment.sig>