From sdouche at gmail.com  Fri Nov  1 00:14:14 2013
From: sdouche at gmail.com (Sebastien Douche)
Date: Fri, 1 Nov 2013 00:14:14 +0100
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
Message-ID: <CAAGHeXHRqhOHE7FAHz+H-fVD6NtOwmw4EF5q2XWxBqCG72GHvA@mail.gmail.com>

On Sat, Oct 19, 2013 at 3:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should
> be able to get back to updating the various metadata specs, with the aim of
> getting cross-platform wheel support in pip 1.6 :)

If I understand right, we must waiting Python 3.5 (~2016) to have
out-of-the-box wheel packages on Linux. Really?

-- 
Sebastien Douche <sdouche at gmail.com>
Twitter: @sdouche / G+: +sdouche

From donald at stufft.io  Fri Nov  1 00:17:53 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 31 Oct 2013 19:17:53 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CAAGHeXHRqhOHE7FAHz+H-fVD6NtOwmw4EF5q2XWxBqCG72GHvA@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
 <CAAGHeXHRqhOHE7FAHz+H-fVD6NtOwmw4EF5q2XWxBqCG72GHvA@mail.gmail.com>
Message-ID: <106B1A0A-B585-4CF2-964A-CA042CD383EA@stufft.io>


On Oct 31, 2013, at 7:14 PM, Sebastien Douche <sdouche at gmail.com> wrote:

> On Sat, Oct 19, 2013 at 3:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should
>> be able to get back to updating the various metadata specs, with the aim of
>> getting cross-platform wheel support in pip 1.6 :)
> 
> If I understand right, we must waiting Python 3.5 (~2016) to have
> out-of-the-box wheel packages on Linux. Really?

No, Python 3.4.x will ship whatever the latest version of pip is, so if 1.6 is out
when 3.4.1 ships it?ll ship with pip 1.6. You can also always pip install ?upgrade pip
before then.

There?s only so many hours in a day and so many people willing to get involved
some things have to be delayed.

> 
> -- 
> Sebastien Douche <sdouche at gmail.com>
> Twitter: @sdouche / G+: +sdouche
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/d8a9a1df/attachment-0001.sig>

From sdouche at gmail.com  Fri Nov  1 00:37:46 2013
From: sdouche at gmail.com (Sebastien Douche)
Date: Fri, 1 Nov 2013 00:37:46 +0100
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <106B1A0A-B585-4CF2-964A-CA042CD383EA@stufft.io>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
 <CAAGHeXHRqhOHE7FAHz+H-fVD6NtOwmw4EF5q2XWxBqCG72GHvA@mail.gmail.com>
 <106B1A0A-B585-4CF2-964A-CA042CD383EA@stufft.io>
Message-ID: <CAAGHeXHycKHEbpcXCOdp0cfEi5V-N_cbAL2ftinKuuhA8Ozy4A@mail.gmail.com>

On Fri, Nov 1, 2013 at 12:17 AM, Donald Stufft <donald at stufft.io> wrote:
> No, Python 3.4.x will ship whatever the latest version of pip is, so if 1.6 is out
> when 3.4.1 ships it?ll ship with pip 1.6.

Good to know. Thanks Donald.


-- 
Sebastien Douche <sdouche at gmail.com>
Twitter: @sdouche / G+: +sdouche

From chris.barker at noaa.gov  Fri Nov  1 00:32:49 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 31 Oct 2013 16:32:49 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7enQUVchS9e9DEgNbBWPG4CjwEABQdprU6e+c-2f0yunQ@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
 <CALGmxEJ7Wy-AAMFvzS5MjFwCGZKD3HHJBPXzfasekmf42SSH8w@mail.gmail.com>
 <EB992840-5763-496B-AEFB-5644701F85A9@stufft.io>
 <E0DC4524-DF0A-4D64-9D43-D1B1B9694185@mac.com>
 <CADiSq7cZLUchLYDXyXOze_E=HtCJrByOKMGj_QagDCVuye=JzA@mail.gmail.com>
 <CAG8k2+6oD77FgPr6J9FUQb3tujY5UCJm5soyaSdE5Rtx=gHE5w@mail.gmail.com>
 <9EF006C3-5FB4-463D-87F2-D8E1BFBC8CDB@mac.com>
 <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <l4ufp3$gbs$1@ger.gmane.org>
 <CADiSq7enQUVchS9e9DEgNbBWPG4CjwEABQdprU6e+c-2f0yunQ@mail.gmail.com>
Message-ID: <CALGmxELXDCLgWnHsZSjAcygFjt1X9OJfXLx+ZMCas9E3RUJp0w@mail.gmail.com>

On Thu, Oct 31, 2013 at 2:34 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> > For all platforms *except* Windows, wheels are essentially caches --
> > there is no real reason to distribute them via PyPI at all, because OSx
> > and Linux develpoers will have tools to build them from sdists.
>
That's not at all true -- it IS true of homebrew, etc users, but not the
least bit true of the genreral Mac user:

* Installing XCode is free, but not default, and less than trivial, and
even less than trivial to get right to build python extensions.

* Many packages require third party compiled libs -- even harder to do on
the Mac -- some are a downright pain in the &^%*&.

What if an OSX user wants to install numpy/scipy? How easy is it to do this
> from source (I really don't know)?


A serious pain in the %^&$ -- numpy is pretty easy, but scipy is a
nightmare, requiring Fortran, etc. The community has addressed with with
"scientific python distributions": Anaconda, Canopy, Python(x,y), etc. But
is sure would be nice to have binary wheels on PyPi

And the cache thing is really nice, actually.

> Given PEP 453, it's probably worth allowing wheels on Mac OS X in pip 1.5,
> then we can tackle the thornier general *nix problem in the pip 1.6 time
> frame (which should also improve external dependency handling on Windows
> and Mac OS X as well).
>
Sounds great!



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/80feb06b/attachment.html>

From noah at coderanger.net  Fri Nov  1 00:52:49 2013
From: noah at coderanger.net (Noah Kantrowitz)
Date: Thu, 31 Oct 2013 16:52:49 -0700
Subject: [Distutils] PyPI pull request #7
In-Reply-To: <CAPkN8xKUWoK6ckqwaZTNe1_+9koLafhp9mYvJu5khBHyBkKR0g@mail.gmail.com>
References: <CAPkN8xKWmSJUqH0QU36YbS4oHnv93B1nM8VhQ5+JXwDFR7HqMA@mail.gmail.com>
 <A95C5014-4DD9-4ED8-B77A-B28BB716EDAE@coderanger.net>
 <CAPkN8xKUWoK6ckqwaZTNe1_+9koLafhp9mYvJu5khBHyBkKR0g@mail.gmail.com>
Message-ID: <A9ED2A22-8716-4310-956E-B136D233AAE3@coderanger.net>


On Oct 31, 2013, at 4:32 AM, anatoly techtonik <techtonik at gmail.com> wrote:

> On Wed, Oct 30, 2013 at 11:11 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
>> Please stop submitting pull requests. Development on the existing codebase is halted except for critical fixes or security issues. You are making extra work for people on this list and it will not be tolerated. Please consider this your final warning.
> 
> I can't live as long as you are to see the new incantation of Python
> website (by PyCon 2013) or PyPI. I am willing to help, and this stuff
> you're saying is rather discouraging and like "no, go waste your time
> somewhere else, we are not giving any code reviews for free". I
> understand that my reputation precedes me, but can we keep this
> strictly technical?
> 
> What I am trying to do is to send small, incremental fixes. They don't
> affect security. I can commit it directly to avoid distracting
> overloaded PyPI (bus factor 2) team, and you can blame me for breaking
> things - ok, and ban if I break something - that's also ok. If learn
> previous PyPI and new PyPI, I can tell people more about it, and you
> can expect more pull requests - not from me, for new PyPI, once it is
> ready.
> 
> And if I am going to submit any new features, like reST validation on
> edit and Markdown support - the code will be more decoupled than
> existing one to be almost directly reused for the new site.
> 
> 
> Why I am skeptical that new site will replace old one soon? Just
> because I don't believe in rewrites by one man army. When you develop
> public resource, you need to rely on external feedback. You also need
> some designer guy in a team. You also need a backlog for
> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
> Donald and Richard will be working on it full time. So, instead of
> all-or-nothing scenario I can try to find some help with incremental
> approach.

Your opinion is noted, however my statement stands and as I said, your continued derailment and disruption will not be tolerated. Thank you for your input.

--Noah


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/e43910eb/attachment.sig>

From chris.barker at noaa.gov  Fri Nov  1 00:25:21 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Thu, 31 Oct 2013 16:25:21 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CAG8k2+6oD77FgPr6J9FUQb3tujY5UCJm5soyaSdE5Rtx=gHE5w@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
 <CALGmxEJ7Wy-AAMFvzS5MjFwCGZKD3HHJBPXzfasekmf42SSH8w@mail.gmail.com>
 <EB992840-5763-496B-AEFB-5644701F85A9@stufft.io>
 <E0DC4524-DF0A-4D64-9D43-D1B1B9694185@mac.com>
 <CADiSq7cZLUchLYDXyXOze_E=HtCJrByOKMGj_QagDCVuye=JzA@mail.gmail.com>
 <CAG8k2+6oD77FgPr6J9FUQb3tujY5UCJm5soyaSdE5Rtx=gHE5w@mail.gmail.com>
Message-ID: <CALGmxEJ=4RCZ3O8M2jG5LF_1VGO60q0DTMbTQojbBehZ+EgFiQ@mail.gmail.com>

On Thu, Oct 31, 2013 at 9:49 AM, Daniel Holth <dholth at gmail.com> wrote:

> I'm sure you could build your own broken Windows Python, but who
>  bothers?


As long as we are clear that we are talking about a social difference here,
not a technical one...

IMO it pretty much boils down to the fact that on Windows you
> are probably using the python.org version of Python and not linking
> with random shared libraries from C:\Program Files\, but on Linux you
> are probably using one of a few dozen distro x distro-release Pythons
> AND your extension probably dynamically links to some other things
> your specific distro provides AND maybe you are depending on some
> versioned symbols in glibc oh the horror.
>
> On OS X I would hope you are uploading only wheels built with
> python.org-Python but maybe you aren't, everyone has their
> preferences.
>

yes, they do -- but what is the target audience here? yes, a lot of folks
use macports, homebrew etc, fewer, but I'm sure some, build their own
Python from scratch -- but these are NOT the people that we want binary
wheels for -- they don't want them anyway.

The folks that we want to provide binary wheels for are NOT going to be
building their own esoteric python, really, they are not.

The MacPython community has a long standing tradition of building binaries
(if at all) that are compatible with the python.org builds (and secondarily
with the Apple-supplied Python) -- that is what I'd like to see supported
by PyPi -- just like Windows....

Sure, someone could upload some oddly-built binary wheel to PyPi -- then it
would not work for most users, and they would get complaints and hopefully
fix it -- just like uploading a package with any other kind of bug in it.

It is kind of a pain to build a truly portable binary package (when it
depends on third-party compiled libs), but there is a small
but committed group of folks doing that already -- let's make it easier to
get stuff out there.

> Will a C extension built with Homebrew Python work with the Python Mac
> > OS X installer from python.org? Probably, but given how painful ABI
> > mismatches with shared libraries can be to debug, "probably" doesn't
> > cut it until someone has the chance to thoroughly review the potential
> > for problems.
>

I disagree:

1) I don't care if homebrew built extensions work with other pythons -- you
want to build with homebrew, create a homebrew recipe. -- there should be a
policy about how binary packages posted on PyPi should be built.

2) We're never going to find out what the problems are until we give it a
try.

Fundamentally, I disagree with the premise here: "If we
can't guarantee that anything anyone uploads will work for everyone, we
shouldn't allow it" -- that's an unattainable goal.

If we do want a more fool-proof approach, then the name auto-generated by
wheel should include something that means "python.org-build" only if built
with the python.org build.

And I suppose we could try to put check code in there to make sure that
extensions aren't linked to outside libs. Actually, that would be a handy
utility to have, even if it didn't enforce anything. (and by the way, it's
rea;lly easy to build a binary for Windows that's linked to an external dll
also -- we expect package builders to be careful with that...)


I was told a wheel built on Ubuntu probably won?t work on Linux, so I shut
> off Linux Wheels, at the same time I asked about Windows and OSX wheels,
> the answers I got from people were they would generally just work on
> Windows, and nobody gave me a straight answer on OSX.
>

Sorry we weren't out there answering!

Linux is a different story -- not only are there a lot of variations out
there, but there also is no obvious "standard" one could point to that we'd
expect folks to build binary wheels for.

OS-X has (to much) variety  though it is less than Linux, and more to the
point, there is a standard Python out there -- the python.org builds. And
there is a tradition of building binaries for that standard. AFAIK, it is
pretty much the ONLY build of Python that package maintainers support with
binaries (if they support anything).

> If you build a Wheel with Homebrew Python will it work on the official
> OSX installers? What if I have a library installed from Homebrew?


probably not, but again, I don't care -- that's not what binary wheels on
Python would be for. And more to the point -- this is a policy question --
don't upload a binary wheel to pypi that depends on homebrew (or anything
else that Apple didn't provide)

Essentially trying to figure out how likely it is that with the existing
> tags a wheel is going to work if we select based on them.-Chris
>

One thing I'm not clear on -- if you do :

pip install something

will pip preferentially select a binary wheel (if enabled on pypi?) -- that
may be an issue as folks will surely try to pip install stuff with
homebrew, macports, etc. pythons (though the wheels are more likely to work
in that direction.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/4aaefbdc/attachment-0001.html>

From donald at stufft.io  Fri Nov  1 01:19:52 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 31 Oct 2013 20:19:52 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CALGmxEJ=4RCZ3O8M2jG5LF_1VGO60q0DTMbTQojbBehZ+EgFiQ@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
 <CALGmxEJ7Wy-AAMFvzS5MjFwCGZKD3HHJBPXzfasekmf42SSH8w@mail.gmail.com>
 <EB992840-5763-496B-AEFB-5644701F85A9@stufft.io>
 <E0DC4524-DF0A-4D64-9D43-D1B1B9694185@mac.com>
 <CADiSq7cZLUchLYDXyXOze_E=HtCJrByOKMGj_QagDCVuye=JzA@mail.gmail.com>
 <CAG8k2+6oD77FgPr6J9FUQb3tujY5UCJm5soyaSdE5Rtx=gHE5w@mail.gmail.com>
 <CALGmxEJ=4RCZ3O8M2jG5LF_1VGO60q0DTMbTQojbBehZ+EgFiQ@mail.gmail.com>
Message-ID: <C4F10E5F-EECC-4CAF-9213-094E4FDB8263@stufft.io>


On Oct 31, 2013, at 7:25 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Thu, Oct 31, 2013 at 9:49 AM, Daniel Holth <dholth at gmail.com> wrote:
> I'm sure you could build your own broken Windows Python, but who
> bothers?
> 
> As long as we are clear that we are talking about a social difference here, not a technical one...
> 
> IMO it pretty much boils down to the fact that on Windows you
> are probably using the python.org version of Python and not linking
> with random shared libraries from C:\Program Files\, but on Linux you
> are probably using one of a few dozen distro x distro-release Pythons
> AND your extension probably dynamically links to some other things
> your specific distro provides AND maybe you are depending on some
> versioned symbols in glibc oh the horror.
> 
> On OS X I would hope you are uploading only wheels built with
> python.org-Python but maybe you aren't, everyone has their
> preferences.
> 
> yes, they do -- but what is the target audience here? yes, a lot of folks use macports, homebrew etc, fewer, but I'm sure some, build their own Python from scratch -- but these are NOT the people that we want binary wheels for -- they don't want them anyway.
> 
> The folks that we want to provide binary wheels for are NOT going to be building their own esoteric python, really, they are not.
> 
> The MacPython community has a long standing tradition of building binaries (if at all) that are compatible with the python.org builds (and secondarily with the Apple-supplied Python) -- that is what I'd like to see supported by PyPi -- just like Windows....
> 
> Sure, someone could upload some oddly-built binary wheel to PyPi -- then it would not work for most users, and they would get complaints and hopefully fix it -- just like uploading a package with any other kind of bug in it.
> 
> It is kind of a pain to build a truly portable binary package (when it depends on third-party compiled libs), but there is a small but committed group of folks doing that already -- let's make it easier to get stuff out there.
> 
> > Will a C extension built with Homebrew Python work with the Python Mac
> > OS X installer from python.org? Probably, but given how painful ABI
> > mismatches with shared libraries can be to debug, "probably" doesn't
> > cut it until someone has the chance to thoroughly review the potential
> > for problems.
> 
> I disagree:
> 
> 1) I don't care if homebrew built extensions work with other pythons -- you want to build with homebrew, create a homebrew recipe. -- there should be a policy about how binary packages posted on PyPi should be built.

Well it was more of we didn?t know, so we played it safe. I don?t personally have a Python.org Installer Python (mine come from Homebrew) and I know others use the System Python. My knowledge of compiled stuff is pretty slim tbh :)

> 
> 2) We're never going to find out what the problems are until we give it a try.
> 
> Fundamentally, I disagree with the premise here: "If we can't guarantee that anything anyone uploads will work for everyone, we shouldn't allow it" -- that's an unattainable goal.

It was less about guarantee and more about ?Can we be reasonably sure this is going to work for most people?.

> 
> If we do want a more fool-proof approach, then the name auto-generated by wheel should include something that means "python.org-build" only if built with the python.org build.
> 
> And I suppose we could try to put check code in there to make sure that extensions aren't linked to outside libs. Actually, that would be a handy utility to have, even if it didn't enforce anything. (and by the way, it's rea;lly easy to build a binary for Windows that's linked to an external dll also -- we expect package builders to be careful with that...)
> 
> 
> I was told a wheel built on Ubuntu probably won?t work on Linux, so I shut off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX.
> 
> Sorry we weren't out there answering!

It?s not your fault! We realized this right before the pip release and sort of quickly added it after a quick check to see if anyone could tell us one way or another.

> 
> Linux is a different story -- not only are there a lot of variations out there, but there also is no obvious "standard" one could point to that we'd expect folks to build binary wheels for.
> 
> OS-X has (to much) variety  though it is less than Linux, and more to the point, there is a standard Python out there -- the python.org builds. And there is a tradition of building binaries for that standard. AFAIK, it is pretty much the ONLY build of Python that package maintainers support with binaries (if they support anything). 
> 
> > If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew?
> 
> probably not, but again, I don't care -- that's not what binary wheels on Python would be for. And more to the point -- this is a policy question -- don't upload a binary wheel to pypi that depends on homebrew (or anything else that Apple didn't provide)
> 
> Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them.-Chris
> 
> One thing I'm not clear on -- if you do :
> 
> pip install something
> 
> will pip preferentially select a binary wheel (if enabled on pypi?) -- that may be an issue as folks will surely try to pip install stuff with homebrew, macports, etc. pythons (though the wheels are more likely to work in that direction.

Right this second it would require opting in via ?use-wheel to get wheel installs from a simple ``pip install something`` but there?s discussion around making wheels enabled by default.

> 
> -Chris




-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/1040d91b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/1040d91b/attachment.sig>

From holger at merlinux.eu  Fri Nov  1 06:31:45 2013
From: holger at merlinux.eu (holger krekel)
Date: Fri, 1 Nov 2013 05:31:45 +0000
Subject: [Distutils] PyPI pull request #7
In-Reply-To: <A9ED2A22-8716-4310-956E-B136D233AAE3@coderanger.net>
References: <CAPkN8xKWmSJUqH0QU36YbS4oHnv93B1nM8VhQ5+JXwDFR7HqMA@mail.gmail.com>
 <A95C5014-4DD9-4ED8-B77A-B28BB716EDAE@coderanger.net>
 <CAPkN8xKUWoK6ckqwaZTNe1_+9koLafhp9mYvJu5khBHyBkKR0g@mail.gmail.com>
 <A9ED2A22-8716-4310-956E-B136D233AAE3@coderanger.net>
Message-ID: <20131101053145.GW3973@merlinux.eu>

On Thu, Oct 31, 2013 at 16:52 -0700, Noah Kantrowitz wrote:
> On Oct 31, 2013, at 4:32 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> 
> > On Wed, Oct 30, 2013 at 11:11 PM, Noah Kantrowitz <noah at coderanger.net> wrote:
> >> Please stop submitting pull requests. Development on the existing codebase is halted except for critical fixes or security issues. You are making extra work for people on this list and it will not be tolerated. Please consider this your final warning.
> > 
> > I can't live as long as you are to see the new incantation of Python
> > website (by PyCon 2013) or PyPI. I am willing to help, and this stuff
> > you're saying is rather discouraging and like "no, go waste your time
> > somewhere else, we are not giving any code reviews for free". I
> > understand that my reputation precedes me, but can we keep this
> > strictly technical?
> > 
> > What I am trying to do is to send small, incremental fixes. They don't
> > affect security. I can commit it directly to avoid distracting
> > overloaded PyPI (bus factor 2) team, and you can blame me for breaking
> > things - ok, and ban if I break something - that's also ok. If learn
> > previous PyPI and new PyPI, I can tell people more about it, and you
> > can expect more pull requests - not from me, for new PyPI, once it is
> > ready.
> > 
> > And if I am going to submit any new features, like reST validation on
> > edit and Markdown support - the code will be more decoupled than
> > existing one to be almost directly reused for the new site.
> > 
> > 
> > Why I am skeptical that new site will replace old one soon? Just
> > because I don't believe in rewrites by one man army. When you develop
> > public resource, you need to rely on external feedback. You also need
> > some designer guy in a team. You also need a backlog for
> > collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
> > Donald and Richard will be working on it full time. So, instead of
> > all-or-nothing scenario I can try to find some help with incremental
> > approach.
> 
> Your opinion is noted, however my statement stands and as I said, your continued derailment and disruption will not be tolerated. Thank you for your input.

Noah, I don't see Anatoly's postings here as derailment or disruption
and also see no reason to cultivate such a "won't be tolerated" tone
here.  Offering PRs is usually a total legit activity although generally
bitbucket/github is the primary place to handle them, rather than this
mailing list here. Differing oppinions on how things should evolve are also
daily open-source business.

However, Richard and Donald (the two pypi maintainers) have made it clear where
their priorities of spending their own time are.  Donald in particular
is heading the new pypi implementation and there are several areas of
possible collaboration there.  So it's obvious now that non-essential PRs on
the current PyPI code base have hardly a chance of getting eye time at the
moment.

best,
holger

From tismer at stackless.com  Fri Nov  1 11:39:30 2013
From: tismer at stackless.com (Christian Tismer)
Date: Fri, 01 Nov 2013 11:39:30 +0100
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
Message-ID: <527384E2.200@stackless.com>

On 19.10.13 03:22, Nick Coghlan wrote:
>
>
> On 19 Oct 2013 04:59, "Chris Barker" <chris.barker at noaa.gov 
> <mailto:chris.barker at noaa.gov>> wrote:
> >
> > Someone on another list indicated that pip installing binary wheels
> > from PyPi will ONLY work for Windows.
> >
> > Is that the case? I think it's desperately needed for OS-X as well.
> >
> > Linux is so diverse that I can't imagine it being useful, but OS-X has
> > only so many versions, and the python.org <http://python.org> OS-X 
> binaries are very clear
> > in their requirements -- it would be very useful if folks could easily
> > get binary wheels for OS-X
>
> We do plan to support it, but the pip devs uncovered a hole in the 
> current wheel spec that means it generates the same filename on *nix 
> systems for wheels that need to have different names for the download 
> side of things to work properly - hence the current Windows-only 
> restriction.
>
> Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we 
> should be able to get back to updating the various metadata specs, 
> with the aim of getting cross-platform wheel support in pip 1.6 :)
>

Ok, but then wheel should be explicit about that and not pretend to
work on OS X. I tried that a week ago on a PySide install on OS X,
which compiled very long, and crashed at the end.
If wheel does not support bdist_wheel on a platform, it should refuse
to install at all, IMHO.

cheers - chris

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/de6e4381/attachment.html>

From dholth at gmail.com  Fri Nov  1 13:34:52 2013
From: dholth at gmail.com (Daniel Holth)
Date: Fri, 1 Nov 2013 08:34:52 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <527384E2.200@stackless.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>
 <CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>
 <527384E2.200@stackless.com>
Message-ID: <CAG8k2+7vxu5UHiFeGfYB=3j7J2jvHHF53QcQbbKMPKqoF3M2iA@mail.gmail.com>

On Fri, Nov 1, 2013 at 6:39 AM, Christian Tismer <tismer at stackless.com> wrote:
> On 19.10.13 03:22, Nick Coghlan wrote:
>
>
> On 19 Oct 2013 04:59, "Chris Barker" <chris.barker at noaa.gov> wrote:
>>
>> Someone on another list indicated that pip installing binary wheels
>> from PyPi will ONLY work for Windows.
>>
>> Is that the case? I think it's desperately needed for OS-X as well.
>>
>> Linux is so diverse that I can't imagine it being useful, but OS-X has
>> only so many versions, and the python.org OS-X binaries are very clear
>> in their requirements -- it would be very useful if folks could easily
>> get binary wheels for OS-X
>
> We do plan to support it, but the pip devs uncovered a hole in the current
> wheel spec that means it generates the same filename on *nix systems for
> wheels that need to have different names for the download side of things to
> work properly - hence the current Windows-only restriction.
>
> Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should
> be able to get back to updating the various metadata specs, with the aim of
> getting cross-platform wheel support in pip 1.6 :)
>
>
> Ok, but then wheel should be explicit about that and not pretend to
> work on OS X. I tried that a week ago on a PySide install on OS X,
> which compiled very long, and crashed at the end.
> If wheel does not support bdist_wheel on a platform, it should refuse
> to install at all, IMHO.
>
> cheers - chris

A reminder that *uploading non-Windows binary wheels to pypi* is the
only thing that is restricted. That is because it was tried with eggs
and found to be frustrating.

bdist_wheel is supposed to work on all platforms.

From ronaldoussoren at mac.com  Fri Nov  1 14:41:19 2013
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 01 Nov 2013 13:41:19 +0000 (GMT)
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
Message-ID: <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>



On Oct 31, 2013, at 07:24 PM, Donald Stufft <donald at stufft.io> wrote:


On Oct 31, 2013, at 1:15 PM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:

Is it "just" a matter of researching if the various build options on OSX really lead to binaries with the same ABI, or is more work needed?

Basically it?s this:

I was told a wheel built on Ubuntu probably won?t work on Linux, so I shut off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX.
?
I can't promise anything, but I'll try to provide some documentation on what does and does not work on OSX.

From what I've seen the problem with Ubuntu wheels not working on other Linux's is due to changes in the glibc ABI (that is, code compiled with glibc X+1 might not work with glibc X), and that shouldn't be a problem on OSX.?



If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew? Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them.
?
Again, someone (and if no-one beats me to it, I) will have to document and test the various scenarios.?

The only real problem I expect is with extensions linking to shared libraries that aren't in the normal OSX install and aren't provided in the wheel. That is, a wheel for lxml that's linked with the Homebrew version of libxml2. That will only work on systems with Homebrew that have libxml2 installed.

Once we know what, if any, problems exist for the various platforms then we can come up with fixes for them. This just hasn?t been high priority for me because of PEP453 work.

To be honest the same problems likely exists on Windows, it?s just less likely and the benefits of prebuilt binaries greater.
?
I'm not sure about the?benefits of?prebuilt binaries.?Apparently building universal binaries on OSX is hard for a lot of users, although I've never had problems with that. But I've written lots of cross-platform C code when cross-platform Unix code was more than just supporting osx, linux and freebsd and am therefore not a good example of a Python users just wants to build an extension that works with his Python code :-)

Ronald

P.S. sorry if this message is somewhat garbled, I'm sending this from iCloud webmail and that doesn't always work properly with mailing lists


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/29128496/attachment.html>

From p.f.moore at gmail.com  Fri Nov  1 14:59:44 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 1 Nov 2013 13:59:44 +0000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
Message-ID: <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>

On 1 November 2013 13:41, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>
> If you build a Wheel with Homebrew Python will it work on the official OSX
> installers? What if I have a library installed from Homebrew? Essentially
> trying to figure out how likely it is that with the existing tags a wheel is
> going to work if we select based on them.
>
>
> Again, someone (and if no-one beats me to it, I) will have to document and
> test the various scenarios.

The key point here is the granularity of the PEP 425 tags used by wheel.

The risk is that a wheel created on another system might declare (via
its filename) that it is compatible with your system, and then not be,
causing segfaults or similar when used. On Windows, thanks to the
history of bdist_wininst, and the small number of people who build
their own *anything* on Windows, there is really only one ABI/C
Library/whatever to worry about and that is the python.org one.
(Actually, there are two - 32 and 64 bit).

If all builds on OSX are compatible at the ABI/CLib/architecture level
then there should be no problem. Equally, if incompatible builds
result in wheels with different compatibility tags, also no problem.
It's only if 2 incompatible environments use the same tags that there
could be an issue.

I don't believe that linking with external libraries should be a
problem here - if wheel X depends on library Y, then either it should
include it or it should simply document "Needs library Y installed".
It's not a *compatibility* issue as such. But maybe the Unix concept
of executables linking to libraries in hard-coded paths might be an
issue here, I don't know enough about how that works to say.

Hope this clarifies things a bit.
Paul

From chris.barker at noaa.gov  Fri Nov  1 16:48:27 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 1 Nov 2013 08:48:27 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
Message-ID: <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>

On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> The key point here is the granularity of the PEP 425 tags used by wheel.
>
> The risk is that a wheel created on another system might declare (via
> its filename) that it is compatible with your system, and then not be,
> causing segfaults or similar when used.


indeed, which is why it _might_ be a good idea to include an extra python
build flag or something: "python.org", "homebrew", "macports". However,
it's probably the case that those aren't really the issues that are going
to cause problems -- at least ones that aren't already handled by the OS-X
version flags --- i.e if a package is built for 10.6+, then it should have
the same system libs as a python built for 10.6+.

Practically speaking, the issues I've run into are:

* Packages built for a newer OS-X won't work on an older one -- but that
should be handled by the OS-X version handling that exists.

* "universal" binaries -- packages built for 32 bit aren't going to work
with a 64 bit python, and a universal python can be both 32 and 64 bit (and
PPC, but I think those days are gone...) -- but this _should_ be handled by
the platform flag: IIRC, "intel" means 32+64 bit Intel. Though I'm not sure
what homebrew or macports python report. But distutils generally does the
right thing with self-contained C code.

* External dependencies: This is the BIG ONE: it's the hardest to get
right, and the hardest to check for. Third party libs must:
  - Be built to match the python, including SDK and architecture (including
universal)
  - Be included somehow -- ideally statically linked, but I'm thinking that
they could be included as part of another dependent package (I think that's
how Anocanda does it). The trick with dynamic linking on OS-X is that that
standard way to install and link a lib has the path to the lib hard-coded
in -- so you can't move it without re-writing the headers. This can be done
on install, but I don't hink we want pip to have to deal with that! You
_can_ install and link libs with relative paths, which I think is what
Anaconda is doing, but I haven't figured out how yet, and it's certainly
not a no-brainer.

So I don't think there is any way to get around the fact that you need to
be careful to build a binary wheel that will work on the systems you
are targeting -- but this is no different than the situation we've had for
years with building binary installers for the Mac. But those dont work with
pip, or virtualenv, or...


> On Windows, thanks to the
> history of bdist_wininst, and the small number of people who build
> their own *anything* on Windows, there is really only one ABI/C
> Library/whatever to worry about and that is the python.org one.
> (Actually, there are two - 32 and 64 bit).
>

Well, technically, the situation is very similar -- it's hard to build a
Windows binary (at least if it has external dependencies), so that it will
just work.

Socially, the situation is different -- there are a (relatively) small
number of people building their own stuff. On the Mac, however, you have
homebrew and macports, and ??, so lots of people building their own stuff.
But those aren't the people we need to support with binaries!

Is anyone expecting a binary built for Windows to work with a cygwin python?
Is anyone expecting that they can build a binary on Windows with cygwin and
give it out? That's what we're talking about here with the Mac.

thanks to the history of bdist_wininst ..


The mac has a history of bdist_mpkg as well, not as widely used, and a bit
neglected lately, but it's there. And there is a history of folks providing
binary installers for the python.org mac. But it would be really nice if we
could go to wheels, and use pypi to distribute them.

It really is the same as Windows -- anyone putting a binary on PyPi has an
obligation to built is so it will work with the python.org python -- and
it's not inherently any harder to do that than on Windows -- the only
difference is that it may be easier to do it badly -- by simply running
bdist_wheel without thinking about it (i.e with homebrew, or macports and
whatever shared libs happen to be linked to).

But again, that's a social problem -- we need to have an understanding
about what is required to put a binary wheel up on pypi.

Also, where we _could_ have a way to identify python.org, vs homebrew, vs.
macports as different "platfroms", that's not going to help the hard
problem, which is making sure third party libs are built and included
properly.

>
> If all builds on OSX are compatible at the ABI/CLib/architecture level
> then there should be no problem. Equally, if incompatible builds
> result in wheels with different compatibility tags, also no problem.
> It's only if 2 incompatible environments use the same tags that there
> could be an issue.
>

yeah -- but the third party libs are the bigger issue anyway...


> I don't believe that linking with external libraries should be a
> problem here - if wheel X depends on library Y, then either it should
> include it or it should simply document "Needs library Y installed".
> It's not a *compatibility* issue as such.


no -- but it's the hard problem -- the POINT of providing binaries is for
people that don't know how to built it themselves to have something they
can use. There is no point at all in providing a binary python package that
depends on a particular freetype, or libpng, or whatever -- if you can
build those, you can build the package. (or have homebrew, etc. do it for
you).

But maybe the Unix concept
> of executables linking to libraries in hard-coded paths might be an
> issue here, I don't know enough about how that works to say.
>

well, that can be gotten around, but it's not really different -- if you
have a binary linked to a non-system shared lib, that shared lib needs to
be properly installed. Properly may be differently defined than on other
systems, but it still needs to be done.

NOTE: somewhere in this thread a saw anote about binary eggs not working
well with OS-X. AFAIK, the issue there is that setuptools did not
understand "universal" packages -- i.e. if you happened to be running ppc,
then it would look for a ppc egg, and not know a "universal" egg was what
it needed. IN fact, it would often install it correctly, but then not know
it had, and try to build it from source anyway.

But I think pip and wheel have got this right now -- but we won't know
'till we try!

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/02aaa145/attachment-0001.html>

From p.f.moore at gmail.com  Fri Nov  1 22:20:19 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 1 Nov 2013 21:20:19 +0000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
Message-ID: <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>

On 1 November 2013 15:48, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> The key point here is the granularity of the PEP 425 tags used by wheel.
>>
>> The risk is that a wheel created on another system might declare (via
>> its filename) that it is compatible with your system, and then not be,
>> causing segfaults or similar when used.
>
> indeed, which is why it _might_ be a good idea to include an extra python
> build flag or something: "python.org", "homebrew", "macports". However, it's
> probably the case that those aren't really the issues that are going to
> cause problems -- at least ones that aren't already handled by the OS-X
> version flags --- i.e if a package is built for 10.6+, then it should have
> the same system libs as a python built for 10.6+.

The first thing to address is what wheel currently *does*. Basically,
a (binary, let's ignore pure Python here) wheel is tagged as follows:

Python version - cpXY
Platform - distutils.util.get_platform(), which is "darwin" for OSX,
if I'm reading the code correctly.
ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists

The only one here which may vary based on build is ABI. So the
question is, do different Python builds on OSX have different SOABI
values?

If not, then does a simple extension with no external dependencies
work when built with one Python and used with a different one?
   If it does, then there's probably no OSX issue[1].
   If it doesn't, then there *is* an issue, and some code needs to
change for wheels to be tagged correctly. Probably by changing the
SOABI values, or by special coding in the ABI tag generation in
bdist_wheel and wherever else it needs to be determined.

If there are different SOABI values, then the above question about a
simple extension should be asked for any SOABI value used by more than
one Python build (if any).

The issue is different on Windows (whether for technical or social
reasons is debatable but irrelevant) because there is *no* ABI tag
used. The platform tag catches cygwin and 32-bit vs 64-bit
differences, and the Python version catches the C runtime
compatibility (because any version of Python is only supported with
one particular CRT - that's embedded in distutils in a way that means
that you'd need to change the code to alter it, which is why it's
arguably a technical issue rather than a social one).

The point marked [1] above is where I ignore the question of "external
libraries" :-)

I'm honestly not sure what is meant by "external libraries" here. Does
it mean which version of the C runtime is used? If so, then that
should probably affect the ABI, but I don't know enough to know if
that's right or not. If the libc version isn't part of the ABI, then I
think this is the issue that caused people to back off on wheels for
non-Windows platforms. I'd like to see a clear statement on whether
building an extension with one libc version and running it with a
different one is supported (I suspect "no") and whether it is
likely/possible (on Windows, "not possible for all practical
purposes", on Linux I suspect "happens all the time" and I have no
idea for OSX).

If by "external libraries" you mean something other than libc (for
example, something like BLAS libraries for scipy) then my view is that
this is not something taht wheel compatibility tags is intended to
solve. The project should document what external libraries need to be
present and any limitations on how they were built that may apply. If
the user disregards these, that's their problem. If projects want to
distribute one wheel for BLAS-compiled-with-homebrew, and one for
BLAS-compiled-with-macports, then I'd put the onus on the project to
come up with a solution - I think it's a specialised enough case that
the wheel spec shouldn't be worrying about it yet, if ever.

I hope this is of some use - maybe it'll help the people with access
to OSX to have a clearer idea of what needs checking.

Paul

From tismer at stackless.com  Fri Nov  1 22:28:34 2013
From: tismer at stackless.com (Christian Tismer)
Date: Fri, 01 Nov 2013 22:28:34 +0100
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CAG8k2+7vxu5UHiFeGfYB=3j7J2jvHHF53QcQbbKMPKqoF3M2iA@mail.gmail.com>
References: <CALGmxEKPEs0gAun3_SWVXPXjRo17NyTRbkrh0pN2tSkX5CfNJQ@mail.gmail.com>	<CADiSq7cfFyE4gaa5DvphpJYq0iv2rzJqgQ2uygQGF3RS_ep34w@mail.gmail.com>	<527384E2.200@stackless.com>
 <CAG8k2+7vxu5UHiFeGfYB=3j7J2jvHHF53QcQbbKMPKqoF3M2iA@mail.gmail.com>
Message-ID: <52741D02.90304@stackless.com>

On 01.11.13 13:34, Daniel Holth wrote:
> On Fri, Nov 1, 2013 at 6:39 AM, Christian Tismer <tismer at stackless.com> wrote:
>> On 19.10.13 03:22, Nick Coghlan wrote:
>>
>>
>> On 19 Oct 2013 04:59, "Chris Barker" <chris.barker at noaa.gov> wrote:
>>> Someone on another list indicated that pip installing binary wheels
>>> from PyPi will ONLY work for Windows.
>>>
>>> Is that the case? I think it's desperately needed for OS-X as well.
>>>
>>> Linux is so diverse that I can't imagine it being useful, but OS-X has
>>> only so many versions, and the python.org OS-X binaries are very clear
>>> in their requirements -- it would be very useful if folks could easily
>>> get binary wheels for OS-X
>> We do plan to support it, but the pip devs uncovered a hole in the current
>> wheel spec that means it generates the same filename on *nix systems for
>> wheels that need to have different names for the download side of things to
>> work properly - hence the current Windows-only restriction.
>>
>> Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should
>> be able to get back to updating the various metadata specs, with the aim of
>> getting cross-platform wheel support in pip 1.6 :)
>>
>>
>> Ok, but then wheel should be explicit about that and not pretend to
>> work on OS X. I tried that a week ago on a PySide install on OS X,
>> which compiled very long, and crashed at the end.
>> If wheel does not support bdist_wheel on a platform, it should refuse
>> to install at all, IMHO.
>>
>> cheers - chris
> A reminder that *uploading non-Windows binary wheels to pypi* is the
> only thing that is restricted. That is because it was tried with eggs
> and found to be frustrating.
>
> bdist_wheel is supposed to work on all platforms.

Ah, sorry, then I have hit a real bug and should report that ;-)

-- 
Christian Tismer             :^)   <mailto:tismer at stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/


From greg.ewing at canterbury.ac.nz  Fri Nov  1 23:09:47 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Nov 2013 11:09:47 +1300
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
Message-ID: <527426AB.2000006@canterbury.ac.nz>

Chris Barker wrote:
> anyone putting a [MacOSX] binary on PyPi has 
> an obligation to built is so it will work with the python.org 
> <http://python.org> python -- and it's not inherently any harder to do 
> that than on Windows

Is there some way that a wheel could be checked to make
sure it doesn't depend on any libraries in non-standard
locations, etc?

If not, could enough information be added to the metadata
to make it possible?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Nov  1 23:35:46 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Nov 2013 11:35:46 +1300
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
Message-ID: <52742CC2.3000401@canterbury.ac.nz>

Paul Moore wrote:
> Python version - cpXY
> Platform - distutils.util.get_platform(), which is "darwin" for OSX,
> if I'm reading the code correctly.
> ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists

Does the MAXOSX_DEPLOYMENT_TARGET figure into this
anywhere? Just knowing that the platform is "darwin"
isn't really enough.

-- 
Greg

From donald at stufft.io  Fri Nov  1 23:36:24 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 18:36:24 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <52742CC2.3000401@canterbury.ac.nz>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
Message-ID: <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>


On Nov 1, 2013, at 6:35 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Paul Moore wrote:
>> Python version - cpXY
>> Platform - distutils.util.get_platform(), which is "darwin" for OSX,
>> if I'm reading the code correctly.
>> ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists
> 
> Does the MAXOSX_DEPLOYMENT_TARGET figure into this
> anywhere? Just knowing that the platform is "darwin"
> isn't really enough.
> 
> -- 
> Greg
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))"
macosx_10_8_x86_64

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/c71c5c46/attachment.sig>

From p.f.moore at gmail.com  Fri Nov  1 23:49:37 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 1 Nov 2013 22:49:37 +0000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
Message-ID: <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>

On 1 November 2013 22:36, Donald Stufft <donald at stufft.io> wrote:

> python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))"
> macosx_10_8_x86_64

OK, cool. That means that binary wheels will be specific to the OSX
version and the word size. Even more specific than Windows.

So that just leaves SOABI. Do homebrew, macports etc actually use
different libc versions, or is there a systemwide libc that everything
uses which only changes when the OSX version changes?

Paul

From donald at stufft.io  Fri Nov  1 23:51:11 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 18:51:11 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
Message-ID: <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>


On Nov 1, 2013, at 6:49 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 1 November 2013 22:36, Donald Stufft <donald at stufft.io> wrote:
> 
>> python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))"
>> macosx_10_8_x86_64
> 
> OK, cool. That means that binary wheels will be specific to the OSX
> version and the word size. Even more specific than Windows.
> 
> So that just leaves SOABI. Do homebrew, macports etc actually use
> different libc versions, or is there a systemwide libc that everything
> uses which only changes when the OSX version changes?
> 
> Paul

How can I get the SOABI?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/653bdc9b/attachment.sig>

From p.f.moore at gmail.com  Fri Nov  1 23:57:34 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 1 Nov 2013 22:57:34 +0000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
 <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
Message-ID: <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>

On 1 November 2013 22:51, Donald Stufft <donald at stufft.io> wrote:
> How can I get the SOABI?

Sorry, didn't I include that before?

sysconfig.get_config_var('SOABI')

should do it.
Paul

From donald at stufft.io  Fri Nov  1 23:58:47 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 18:58:47 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
 <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
 <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>
Message-ID: <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>


On Nov 1, 2013, at 6:57 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 1 November 2013 22:51, Donald Stufft <donald at stufft.io> wrote:
>> How can I get the SOABI?
> 
> Sorry, didn't I include that before?
> 
> sysconfig.get_config_var('SOABI')
> 
> should do it.
> Paul

Hmm,

python -c "import sysconfig; print(sysconfig.get_config_var('SOABI?))"
None

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/3ce3818a/attachment.sig>

From donald at stufft.io  Sat Nov  2 00:02:20 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 19:02:20 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
 <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
 <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>
 <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>
Message-ID: <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2@stufft.io>

Looks like wheels are not being created with an ABI tag at all.

Here?s a Wheel I made to test things:

https://testpypi.python.org/pypi/PyNaCl

On Nov 1, 2013, at 6:58 PM, Donald Stufft <donald at stufft.io> wrote:

> 
> On Nov 1, 2013, at 6:57 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> On 1 November 2013 22:51, Donald Stufft <donald at stufft.io> wrote:
>>> How can I get the SOABI?
>> 
>> Sorry, didn't I include that before?
>> 
>> sysconfig.get_config_var('SOABI')
>> 
>> should do it.
>> Paul
> 
> Hmm,
> 
> python -c "import sysconfig; print(sysconfig.get_config_var('SOABI?))"
> None
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/3a29bd67/attachment.sig>

From greg.ewing at canterbury.ac.nz  Sat Nov  2 00:10:14 2013
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 02 Nov 2013 12:10:14 +1300
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
Message-ID: <527434D6.9020603@canterbury.ac.nz>

Donald Stufft wrote:
> python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))"
> macosx_10_8_x86_64

Hmm, this just appears to reflect the version of MacOSX
that the Python running distutils was built on, or is
running on (not sure which).

This is not quite the same thing as MACOSX_DEPLOYMENT_TARGET,
which is the minimum version of MacOSX that is needed to
run a piece of code.

Distutils seems to assume they're the same, but if you're
building a binary wheel for distribution, it makes sense
to set MACOSX_DEPLOYMENT_TARGET as low as possible.

Will there be a mechanism to get the actual MacOSX version
needed into the metadata, rather than the one you happen
to be building on?

-- 
Greg

From p.f.moore at gmail.com  Sat Nov  2 00:11:46 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 1 Nov 2013 23:11:46 +0000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
 <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
 <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>
 <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>
Message-ID: <CACac1F8WvZVTQhpO2zYJoiqUc1PNVt2HsmnwQG6Dp-f=Xzcmmg@mail.gmail.com>

On 1 November 2013 22:58, Donald Stufft <donald at stufft.io> wrote:
> python -c "import sysconfig; print(sysconfig.get_config_var('SOABI?))"
> None

That implies no explicit ABI tag - the platform and Python version
tags are deemed sufficient to ensure interoperability.

Paul

From p.f.moore at gmail.com  Sat Nov  2 00:15:30 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 1 Nov 2013 23:15:30 +0000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <527434D6.9020603@canterbury.ac.nz>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
Message-ID: <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>

On 1 November 2013 23:10, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Will there be a mechanism to get the actual MacOSX version
> needed into the metadata, rather than the one you happen
> to be building on?

There can be anything - the question here is really whether anything
is *needed*, or is what we have sufficient.

If it's sufficient, we can lift the restriction on uploading OSX
wheels and we're done.
If it's not, we need to keep the restriction until the wheel code is
updated to provide sufficiently flexible tags. We can take as long as
we need to define and implement those. (When I say "we" here, as a
Windows user I really mean "you", of course :-))

Paul

From nad at acm.org  Sat Nov  2 01:37:36 2013
From: nad at acm.org (Ned Deily)
Date: Fri, 01 Nov 2013 17:37:36 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
Message-ID: <nad-0B97CF.17373601112013@news.gmane.org>

In article <527434D6.9020603 at canterbury.ac.nz>,
 Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Donald Stufft wrote:
> > python -c "import distutils; 
> > print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))"
> > macosx_10_8_x86_64
> 
> Hmm, this just appears to reflect the version of MacOSX
> that the Python running distutils was built on, or is
> running on (not sure which).
>
> This is not quite the same thing as MACOSX_DEPLOYMENT_TARGET,
> which is the minimum version of MacOSX that is needed to
> run a piece of code.
> 
> Distutils seems to assume they're the same, but if you're
> building a binary wheel for distribution, it makes sense
> to set MACOSX_DEPLOYMENT_TARGET as low as possible.
> 
> Will there be a mechanism to get the actual MacOSX version
> needed into the metadata, rather than the one you happen
> to be building on?

Assuming that the wheel build is using distutils.util.get_platform(), that 
number *is* the value of MACOSX_DEPLOYMENT_TARGET that was used when Python 
was built unless it is overridden during execution of Python by setting a 
MACOSX_DEPLOYMENT_TARGET env variable.

For example, with the python.org 64-bit installer used on OS X 10.8:

$ /usr/local/bin/python2.7 -c 'import 
distutils.util;print(distutils.util.get_platform())'
macosx-10.6-intel

It works on 10.6, 10.7, 10.8, and 10.9.

With the Apple supplied system Python 2.7 on OS X 10.8:
$ /usr/bin/python2.7 -c 'import 
distutils.util;print(distutils.util.get_platform())'
macosx-10.8-intel

-- 
 Ned Deily,
 nad at acm.org


From nad at acm.org  Sat Nov  2 01:39:45 2013
From: nad at acm.org (Ned Deily)
Date: Fri, 01 Nov 2013 17:39:45 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
 <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
 <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>
 <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>
 <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2@stufft.io>
Message-ID: <nad-BC14D9.17394501112013@news.gmane.org>

In article <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2 at stufft.io>,
 Donald Stufft <donald at stufft.io> wrote:

> Looks like wheels are not being created with an ABI tag at all.

IIRC, distutils.util.get_platform() was not modified to reflect the SOABI when 
that feature was introduced in Python 3.2 (?).  Perhaps it should have been or 
still can be for 3.4.

-- 
 Ned Deily,
 nad at acm.org


From donald at stufft.io  Sat Nov  2 01:40:51 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 20:40:51 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <nad-BC14D9.17394501112013@news.gmane.org>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <CACac1F8zkj=EzVHFucuTFMNyqXyn_BZ=BJ=EXdbBugqMt7ENPA@mail.gmail.com>
 <3BA87057-A9C9-4E92-96F4-00350D67644D@stufft.io>
 <CACac1F9KFPAM0TXvp37n=WoeBpGwuCHCF=BoYYPHRS9ufg4zYw@mail.gmail.com>
 <7C960F23-6D68-4D7F-BA2D-7C018D945250@stufft.io>
 <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2@stufft.io>
 <nad-BC14D9.17394501112013@news.gmane.org>
Message-ID: <0170EC58-45A2-4AC6-8C7B-30381DC8934C@stufft.io>


On Nov 1, 2013, at 8:39 PM, Ned Deily <nad at acm.org> wrote:

> In article <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2 at stufft.io>,
> Donald Stufft <donald at stufft.io> wrote:
> 
>> Looks like wheels are not being created with an ABI tag at all.
> 
> IIRC, distutils.util.get_platform() was not modified to reflect the SOABI when 
> that feature was introduced in Python 3.2 (?).  Perhaps it should have been or 
> still can be for 3.4.

Oh that is a 3.x feature? That would explain why my 2.7 built Wheel doesn?t have it then :)


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/d798555f/attachment.sig>

From ncoghlan at gmail.com  Sat Nov  2 01:45:28 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 2 Nov 2013 10:45:28 +1000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
Message-ID: <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>

On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 1 November 2013 23:10, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > Will there be a mechanism to get the actual MacOSX version
> > needed into the metadata, rather than the one you happen
> > to be building on?
>
> There can be anything - the question here is really whether anything
> is *needed*, or is what we have sufficient.
>
> If it's sufficient, we can lift the restriction on uploading OSX
> wheels and we're done.
> If it's not, we need to keep the restriction until the wheel code is
> updated to provide sufficiently flexible tags. We can take as long as
> we need to define and implement those. (When I say "we" here, as a
> Windows user I really mean "you", of course :-))

I spoke to Donald about this on IRC yesterday, and I take the following
view:

* the key relevant points about users on Windows and Mac OS X are that most
(perhaps only many on Mac OS X) tutorials and introductory courses will
direct them to the binary installers on python.org, and such users are
highly unlikely to have a C compiler installed, so their current "out of
the box" user experience with pip is that it doesn't work for even the
simplest of C extensions.

* by contrast, in other *nix environments (including cygwin on Windows and
homebrew etc on Mac OS X), using the system/environment Python is far more
common, and a C compiler is far more likely to be available

* accordingly, the following defaults make sense for pip 1.5:
- allow wheel files from the index server for Windows and Mac OS X
- allow local wheel files everywhere

* the following options should also be available:
- force use of wheel files from the index server (useful for private index
servers)
- prevent use of wheel files from the index server (useful to force local
builds on Windows and Mac OS X)
- prevent use of wheel files (useful to force a local rebuild, overwriting
the wheel cache)

It would be good to enable the use of index server wheel files everywhere
by default, but we need to add some kind of "environment" or "variant"
marker to the wheel naming scheme before we can do that (i.e. wheel 1.1,
which isn't planned until the 1.6 time frame).

There are also problems with how the abi and platform tags are set and
interpreted that need to be resolved before we can expand wheel sharing
through PyPI beyond the environments defined by the python.org binary
installers.

Cheers,
Nick.

>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131102/bf3d75d9/attachment-0001.html>

From donald at stufft.io  Sat Nov  2 02:10:07 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 21:10:07 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
Message-ID: <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>


On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >
> > On 1 November 2013 23:10, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > > Will there be a mechanism to get the actual MacOSX version
> > > needed into the metadata, rather than the one you happen
> > > to be building on?
> >
> > There can be anything - the question here is really whether anything
> > is *needed*, or is what we have sufficient.
> >
> > If it's sufficient, we can lift the restriction on uploading OSX
> > wheels and we're done.
> > If it's not, we need to keep the restriction until the wheel code is
> > updated to provide sufficiently flexible tags. We can take as long as
> > we need to define and implement those. (When I say "we" here, as a
> > Windows user I really mean "you", of course :-))
> 
> I spoke to Donald about this on IRC yesterday, and I take the following view:
> 
> * the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions.
> 
> * by contrast, in other *nix environments (including cygwin on Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
> 
> * accordingly, the following defaults make sense for pip 1.5:
> - allow wheel files from the index server for Windows and Mac OS X
> - allow local wheel files everywhere
> 
> 

Just to be clear about things, pip already supports any wheel from any source *except* for pypi.python.org.
> * the following options should also be available:
> - force use of wheel files from the index server (useful for private index servers)
> 
At the exclusion of sdists? Not sure I see a point?
> - prevent use of wheel files from the index server (useful to force local builds on Windows and Mac OS X)
> 
I don?t think this needs to be a special option, the solution for the below should work fine here.
> - prevent use of wheel files (useful to force a local rebuild, overwriting the wheel cache)
> 
I do think we?ll need a ?no-use-wheel
> It would be good to enable the use of index server wheel files everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame).
> 
> There are also problems with how the abi and platform tags are set and interpreted that need to be resolved before we can expand wheel sharing through PyPI beyond the environments defined by the python.org binary installers.
> 
> Cheers,
> Nick.
> 
> >
> > Paul
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/a7328cfa/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/a7328cfa/attachment.sig>

From ncoghlan at gmail.com  Sat Nov  2 02:56:45 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 2 Nov 2013 11:56:45 +1000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
Message-ID: <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>

On 2 Nov 2013 11:10, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>>
>> On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore at gmail.com> wrote:
>> >
>> > On 1 November 2013 23:10, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:
>> > > Will there be a mechanism to get the actual MacOSX version
>> > > needed into the metadata, rather than the one you happen
>> > > to be building on?
>> >
>> > There can be anything - the question here is really whether anything
>> > is *needed*, or is what we have sufficient.
>> >
>> > If it's sufficient, we can lift the restriction on uploading OSX
>> > wheels and we're done.
>> > If it's not, we need to keep the restriction until the wheel code is
>> > updated to provide sufficiently flexible tags. We can take as long as
>> > we need to define and implement those. (When I say "we" here, as a
>> > Windows user I really mean "you", of course :-))
>>
>> I spoke to Donald about this on IRC yesterday, and I take the following
view:
>>
>> * the key relevant points about users on Windows and Mac OS X are that
most (perhaps only many on Mac OS X) tutorials and introductory courses
will direct them to the binary installers on python.org, and such users are
highly unlikely to have a C compiler installed, so their current "out of
the box" user experience with pip is that it doesn't work for even the
simplest of C extensions.
>>
>> * by contrast, in other *nix environments (including cygwin on Windows
and homebrew etc on Mac OS X), using the system/environment Python is far
more common, and a C compiler is far more likely to be available
>>
>> * accordingly, the following defaults make sense for pip 1.5:
>> - allow wheel files from the index server for Windows and Mac OS X
>> - allow local wheel files everywhere
>>
>>
>
> Just to be clear about things, pip already supports any wheel from any
source *except* for pypi.python.org.
>
>> * the following options should also be available:
>> - force use of wheel files from the index server (useful for private
index servers)
>
> At the exclusion of sdists? Not sure I see a point?

I didn't realise wheel files were already permitted by default for private
index servers. With that clarification, I agree there's no need for this
option.

>>
>> - prevent use of wheel files from the index server (useful to force
local builds on Windows and Mac OS X)
>
> I don?t think this needs to be a special option, the solution for the
below should work fine here.

If someone adds a new dependency, they should be able to easily say "build
anything that I don't already have a local wheel file for from source". By
contrast, I'd expect "--no-use-wheel" to rebuild everything (even stuff
with already cached wheels).

Cheers,
Nick.

>>
>> - prevent use of wheel files (useful to force a local rebuild,
overwriting the wheel cache)
>
> I do think we?ll need a ?no-use-wheel
>
>> It would be good to enable the use of index server wheel files
everywhere by default, but we need to add some kind of "environment" or
"variant" marker to the wheel naming scheme before we can do that (i.e.
wheel 1.1, which isn't planned until the 1.6 time frame).
>>
>> There are also problems with how the abi and platform tags are set and
interpreted that need to be resolved before we can expand wheel sharing
through PyPI beyond the environments defined by the python.org binary
installers.
>>
>> Cheers,
>> Nick.
>>
>> >
>> > Paul
>> > _______________________________________________
>> > Distutils-SIG maillist  -  Distutils-SIG at python.org
>> > https://mail.python.org/mailman/listinfo/distutils-sig
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131102/ce2fa5b1/attachment-0001.html>

From donald at stufft.io  Sat Nov  2 02:58:17 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 1 Nov 2013 21:58:17 -0400
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
 <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
Message-ID: <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>


On Nov 1, 2013, at 9:56 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 2 Nov 2013 11:10, "Donald Stufft" <donald at stufft.io> wrote:
> >
> >
> > On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> >>
> >> On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >> >
> >> > On 1 November 2013 23:10, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> >> > > Will there be a mechanism to get the actual MacOSX version
> >> > > needed into the metadata, rather than the one you happen
> >> > > to be building on?
> >> >
> >> > There can be anything - the question here is really whether anything
> >> > is *needed*, or is what we have sufficient.
> >> >
> >> > If it's sufficient, we can lift the restriction on uploading OSX
> >> > wheels and we're done.
> >> > If it's not, we need to keep the restriction until the wheel code is
> >> > updated to provide sufficiently flexible tags. We can take as long as
> >> > we need to define and implement those. (When I say "we" here, as a
> >> > Windows user I really mean "you", of course :-))
> >>
> >> I spoke to Donald about this on IRC yesterday, and I take the following view:
> >>
> >> * the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions.
> >>
> >> * by contrast, in other *nix environments (including cygwin on Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
> >>
> >> * accordingly, the following defaults make sense for pip 1.5:
> >> - allow wheel files from the index server for Windows and Mac OS X
> >> - allow local wheel files everywhere
> >>
> >>
> >
> > Just to be clear about things, pip already supports any wheel from any source *except* for pypi.python.org.
> >
> >> * the following options should also be available:
> >> - force use of wheel files from the index server (useful for private index servers)
> >
> > At the exclusion of sdists? Not sure I see a point?
> 
> I didn't realise wheel files were already permitted by default for private index servers. With that clarification, I agree there's no need for this option.
> 
> >>
> >> - prevent use of wheel files from the index server (useful to force local builds on Windows and Mac OS X)
> >
> > I don?t think this needs to be a special option, the solution for the below should work fine here.
> 
> If someone adds a new dependency, they should be able to easily say "build anything that I don't already have a local wheel file for from source". By contrast, I'd expect "--no-use-wheel" to rebuild everything (even stuff with already cached wheels).
> 
The ?local wheel? path isn?t the greatest right now, but to do this

# Assumes you?ve already populated the wheelhouse with pip wheel
pip install ?find-links ~/.pip/wheelhouse whatever

Wheels take precedence over sdists.
> Cheers,
> Nick.
> 
> >>
> >> - prevent use of wheel files (useful to force a local rebuild, overwriting the wheel cache)
> >
> > I do think we?ll need a ?no-use-wheel
> >
> >> It would be good to enable the use of index server wheel files everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame).
> >>
> >> There are also problems with how the abi and platform tags are set and interpreted that need to be resolved before we can expand wheel sharing through PyPI beyond the environments defined by the python.org binary installers.
> >>
> >> Cheers,
> >> Nick.
> >>
> >> >
> >> > Paul
> >> > _______________________________________________
> >> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> > https://mail.python.org/mailman/listinfo/distutils-sig
> >>
> >> _______________________________________________
> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
> >> https://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
> >
> > -----------------
> > Donald Stufft
> > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> >


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/31b1293a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/31b1293a/attachment.sig>

From chris.barker at noaa.gov  Sat Nov  2 03:04:32 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 1 Nov 2013 19:04:32 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
Message-ID: <CALGmxE+LP9z-K5omR9EO7acMYkHBOt2ygrzgUWqk1BehXuvPtw@mail.gmail.com>

On Fri, Nov 1, 2013 at 5:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> * the key relevant points about users on Windows and Mac OS X are that
> most (perhaps only many on Mac OS X) tutorials and introductory courses
> will direct them to the binary installers on python.org, and such users
> are highly unlikely to have a C compiler installed, so their current "out
> of the box" user experience with pip is that it doesn't work for even the
> simplest of C extensions.


Thank you for being so articulate about that -- I"ve been unsuccesfully
trying to say this this whole thread ....

Note also that it's not just what tutorials say, it's what the _should_
say. WE really wouldn't want to say to new users:

"""
Want to learn to program in Python? First, install a compiler, which, by
the way is a multi-GB download from Apple, that you have to register as a
developer to get.....
"""
Though I"ll also add that binaries for the python.org builds also support
users that may have  compiler, but not the expertise to build third-party
libs, and build re-distributable binaries for older OS versions, etc.


> * by contrast, in other *nix environments (including cygwin on Windows and
> homebrew etc on Mac OS X), using the system/environment Python is far more
> common, and a C compiler is far more likely to be available
>
indeed, required, for homebrew and macports (and cygwin?)


> * accordingly, the following defaults make sense for pip 1.5:
> - allow wheel files from the index server for Windows and Mac OS X
> - allow local wheel files everywhere
>
> sounds good. -- and have a stated policy )or at least recommendation) that
binary wheels for OS-X be built for the python.org pythons.

* the following options should also be available:
> - force use of wheel files from the index server (useful for private index
> servers)
> - prevent use of wheel files from the index server (useful to force local
> builds on Windows and Mac OS X)
> - prevent use of wheel files (useful to force a local rebuild, overwriting
> the wheel cache)
>
sounds good.

One question: should pip be able to install a incompatible binary wheel
directly without even a warning? It does now, but I don't think it should.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/bf6e8f06/attachment-0001.html>

From ncoghlan at gmail.com  Sat Nov  2 03:57:34 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 2 Nov 2013 12:57:34 +1000
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
 <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
 <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>
Message-ID: <CADiSq7cWFZqoO_JNaBaQ_zaxPitOHq95hM6kkt1z--UwnSZsmg@mail.gmail.com>

On 2 Nov 2013 11:59, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Nov 1, 2013, at 9:56 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>>
>> On 2 Nov 2013 11:10, "Donald Stufft" <donald at stufft.io> wrote:
>> >
>> >
>> > On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >
>> >>
>> >> On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore at gmail.com> wrote:
>> >> >
>> >> > On 1 November 2013 23:10, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:
>> >> > > Will there be a mechanism to get the actual MacOSX version
>> >> > > needed into the metadata, rather than the one you happen
>> >> > > to be building on?
>> >> >
>> >> > There can be anything - the question here is really whether anything
>> >> > is *needed*, or is what we have sufficient.
>> >> >
>> >> > If it's sufficient, we can lift the restriction on uploading OSX
>> >> > wheels and we're done.
>> >> > If it's not, we need to keep the restriction until the wheel code is
>> >> > updated to provide sufficiently flexible tags. We can take as long
as
>> >> > we need to define and implement those. (When I say "we" here, as a
>> >> > Windows user I really mean "you", of course :-))
>> >>
>> >> I spoke to Donald about this on IRC yesterday, and I take the
following view:
>> >>
>> >> * the key relevant points about users on Windows and Mac OS X are
that most (perhaps only many on Mac OS X) tutorials and introductory
courses will direct them to the binary installers on python.org, and such
users are highly unlikely to have a C compiler installed, so their current
"out of the box" user experience with pip is that it doesn't work for even
the simplest of C extensions.
>> >>
>> >> * by contrast, in other *nix environments (including cygwin on
Windows and homebrew etc on Mac OS X), using the system/environment Python
is far more common, and a C compiler is far more likely to be available
>> >>
>> >> * accordingly, the following defaults make sense for pip 1.5:
>> >> - allow wheel files from the index server for Windows and Mac OS X
>> >> - allow local wheel files everywhere
>> >>
>> >>
>> >
>> > Just to be clear about things, pip already supports any wheel from any
source *except* for pypi.python.org.
>> >
>> >> * the following options should also be available:
>> >> - force use of wheel files from the index server (useful for private
index servers)
>> >
>> > At the exclusion of sdists? Not sure I see a point?
>>
>> I didn't realise wheel files were already permitted by default for
private index servers. With that clarification, I agree there's no need for
this option.
>>
>> >>
>> >> - prevent use of wheel files from the index server (useful to force
local builds on Windows and Mac OS X)
>> >
>> > I don?t think this needs to be a special option, the solution for the
below should work fine here.
>>
>> If someone adds a new dependency, they should be able to easily say
"build anything that I don't already have a local wheel file for from
source". By contrast, I'd expect "--no-use-wheel" to rebuild everything
(even stuff with already cached wheels).
>
> The ?local wheel? path isn?t the greatest right now, but to do this
>
> # Assumes you?ve already populated the wheelhouse with pip wheel
> pip install ?find-links ~/.pip/wheelhouse whatever
>
> Wheels take precedence over sdists.

Cool - sounds like allowing wheels by default on both Windows and Mac OS X
and offering a --no-use-wheel option are the only changes needed then.

Cheers,
Nick.

>>
>> Cheers,
>> Nick.
>>
>> >>
>> >> - prevent use of wheel files (useful to force a local rebuild,
overwriting the wheel cache)
>> >
>> > I do think we?ll need a ?no-use-wheel
>> >
>> >> It would be good to enable the use of index server wheel files
everywhere by default, but we need to add some kind of "environment" or
"variant" marker to the wheel naming scheme before we can do that (i.e.
wheel 1.1, which isn't planned until the 1.6 time frame).
>> >>
>> >> There are also problems with how the abi and platform tags are set
and interpreted that need to be resolved before we can expand wheel sharing
through PyPI beyond the environments defined by the python.org binary
installers.
>> >>
>> >> Cheers,
>> >> Nick.
>> >>
>> >> >
>> >> > Paul
>> >> > _______________________________________________
>> >> > Distutils-SIG maillist  -  Distutils-SIG at python.org
>> >> > https://mail.python.org/mailman/listinfo/distutils-sig
>> >>
>> >> _______________________________________________
>> >> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> >> https://mail.python.org/mailman/listinfo/distutils-sig
>> >
>> >
>> >
>> > -----------------
>> > Donald Stufft
>> > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9
3372 DCFA
>> >
>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131102/fcfa414b/attachment.html>

From qwcode at gmail.com  Sat Nov  2 06:38:57 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 1 Nov 2013 22:38:57 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
 <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
 <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>
Message-ID: <CAPYWazp1xUN=tVUYS3w51BW_3_EUoCKunXPpZUNtiwP8h+4-1Q@mail.gmail.com>

>
>
> Wheels take precedence over sdists.
>

also, for those that don't know, pip also prioritizes wheels over other
wheels (using the same PEP425 module that the wheel project uses, which has
a sort order).
In short, plat-specific over python-specific, over general wheels.
not a very likely scenario, but the logic is there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/30cd3288/attachment.html>

From qwcode at gmail.com  Sat Nov  2 06:48:44 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 1 Nov 2013 22:48:44 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
 <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
Message-ID: <CAPYWazquQzAjOuiMxUP84ip0dMO3+0QYDz+LMxWdyeKJa3x+zw@mail.gmail.com>

> If someone adds a new dependency, they should be able to easily say "build
> anything that I don't already have a local wheel file for from source".
>

"pip wheel -r requirements.txt" will blindly rebuild wheels for all your
dependencies, regardless of it being in your wheel dir already.
it's been on the list for about 9 months now to make this better  :  )
https://github.com/pypa/pip/issues/855
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/bb4785d1/attachment.html>

From qwcode at gmail.com  Sat Nov  2 07:09:33 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 1 Nov 2013 23:09:33 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <7F729985-1B56-4339-AF40-EFA9E7B0601C@stufft.io>
 <CADiSq7dBAwXtJWgZt8jggKBab-Uv4XdWMKYho3C7CUk1xFtAAw@mail.gmail.com>
 <753DB864-B11B-4420-8088-79781D2862FD@stufft.io>
Message-ID: <CAPYWazouc4pfADDodKn6so69QLR5ETPoncHdO8veSdGOa_t5yg@mail.gmail.com>

>
>
> The ?local wheel? path isn?t the greatest right now, but to do this
>
> # Assumes you?ve already populated the wheelhouse with pip wheel
> pip install ?find-links ~/.pip/wheelhouse whatever
>
>
fwiw, I found one of the discussions we had ~year ago about "wheel
caching".  https://github.com/pypa/pip/pull/684
at the time, we weren't even sure if wheel was going to get merged, so
something like "pip wheel" (that was outside of the main install pipeline)
seemed like the first step.
but now that things are moving along,  an automatic wheel cache sounds
awesome.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/47a68da7/attachment-0001.html>

From qwcode at gmail.com  Sat Nov  2 07:25:23 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 1 Nov 2013 23:25:23 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CALGmxE+LP9z-K5omR9EO7acMYkHBOt2ygrzgUWqk1BehXuvPtw@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <CALGmxE+LP9z-K5omR9EO7acMYkHBOt2ygrzgUWqk1BehXuvPtw@mail.gmail.com>
Message-ID: <CAPYWazqE6Khu_20UyZesztjAsSJkLJwSQXvvZJSNsrnVfo6H=Q@mail.gmail.com>

>
>
> One question: should pip be able to install a incompatible binary wheel
> directly without even a warning? It does now, but I don't think it should.
>

pip does confirm the wheel file is compatible with your platform/python
(based on the file tags), when downloading from indexes and links.
BUT, just noticed, when installing directly from a specific file, it seems
it's not checking, so I'll look into that and log an issue.
Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131101/0ff1b504/attachment.html>

From aclark at aclark.net  Sat Nov  2 19:20:47 2013
From: aclark at aclark.net (Alex Clark)
Date: Sat, 2 Nov 2013 18:20:47 +0000 (UTC)
Subject: [Distutils] PyPI pull request #7
References: <CAPkN8xKWmSJUqH0QU36YbS4oHnv93B1nM8VhQ5+JXwDFR7HqMA@mail.gmail.com>
 <A95C5014-4DD9-4ED8-B77A-B28BB716EDAE@coderanger.net>
 <CAPkN8xKUWoK6ckqwaZTNe1_+9koLafhp9mYvJu5khBHyBkKR0g@mail.gmail.com>
 <A9ED2A22-8716-4310-956E-B136D233AAE3@coderanger.net>
 <20131101053145.GW3973@merlinux.eu>
Message-ID: <loom.20131102T190642-586@post.gmane.org>

holger krekel <holger <at> merlinux.eu> writes:


> > 
> > Your opinion is noted, however my statement stands and as I said, your
continued derailment and
> disruption will not be tolerated. Thank you for your input.
> 
> Noah, I don't see Anatoly's postings here as derailment or disruption
> and also see no reason to cultivate such a "won't be tolerated" tone
> here. 


+1. I think the only "derailment" here is the way Anatoly has been treated
for his contributions. Let's let Anatoly continue to submit and discuss his
PRs and let Donald do whatever he wants to do whenever he wants to do it.


From chris.barker at noaa.gov  Sat Nov  2 19:33:23 2013
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Sat, 2 Nov 2013 11:33:23 -0700
Subject: [Distutils] Plans for binary wheels, and PyPi and OS-X
In-Reply-To: <CAPYWazqE6Khu_20UyZesztjAsSJkLJwSQXvvZJSNsrnVfo6H=Q@mail.gmail.com>
References: <049ABB88-78A4-43A9-B3D3-54110CE3D53B@stufft.io>
 <88b68921-23ba-4bf7-a699-5105bb18c5ad@me.com>
 <CACac1F9VU9VUVE8TBH3qNr1YX9x4K6z6AiZ1W-r=-9w=in=bfw@mail.gmail.com>
 <CALGmxELBs64isAjFyE_Ga4Zo5o+wMuNWw+gey-GL5ks3naRHrA@mail.gmail.com>
 <CACac1F8YFgVc-uz_YUddsSjpvUGuC_2y6FaszBX+EzWN5eK=DQ@mail.gmail.com>
 <52742CC2.3000401@canterbury.ac.nz>
 <8AD3B12E-1E64-4605-B72E-5A18084C571A@stufft.io>
 <527434D6.9020603@canterbury.ac.nz>
 <CACac1F8vFVeJfUjjKs1pnhYRjAX8C9F=kbvamx9yW5HF5+aVJg@mail.gmail.com>
 <CADiSq7fbMTOukucaj=AKYEfPXe3tSuETufqTg8Hskp6KBkCbjQ@mail.gmail.com>
 <CALGmxE+LP9z-K5omR9EO7acMYkHBOt2ygrzgUWqk1BehXuvPtw@mail.gmail.com>
 <CAPYWazqE6Khu_20UyZesztjAsSJkLJwSQXvvZJSNsrnVfo6H=Q@mail.gmail.com>
Message-ID: <-355713740276822637@unknownmsgid>

On Nov 1, 2013, at 11:25 PM, Marcus Smith <qwcode at gmail.com> wrote:
>
> pip does confirm the wheel file is compatible with your platform/python (based on the file tags), when downloading from indexes and links.
> BUT, just noticed, when installing directly from a specific file, it seems it's not checking, so I'll look into that and log an issue

Great, thanks!

From ncoghlan at gmail.com  Sun Nov  3 02:11:50 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Nov 2013 11:11:50 +1000
Subject: [Distutils] PyPI pull request #7
In-Reply-To: <loom.20131102T190642-586@post.gmane.org>
References: <CAPkN8xKWmSJUqH0QU36YbS4oHnv93B1nM8VhQ5+JXwDFR7HqMA@mail.gmail.com>
 <A95C5014-4DD9-4ED8-B77A-B28BB716EDAE@coderanger.net>
 <CAPkN8xKUWoK6ckqwaZTNe1_+9koLafhp9mYvJu5khBHyBkKR0g@mail.gmail.com>
 <A9ED2A22-8716-4310-956E-B136D233AAE3@coderanger.net>
 <20131101053145.GW3973@merlinux.eu>
 <loom.20131102T190642-586@post.gmane.org>
Message-ID: <CADiSq7eO4HS4xi+t=qf_hymsmdbZHQoExyJA6x7eErvyoAL2mg@mail.gmail.com>

On 3 Nov 2013 04:26, "Alex Clark" <aclark at aclark.net> wrote:
>
> holger krekel <holger <at> merlinux.eu> writes:
>
>
> > >
> > > Your opinion is noted, however my statement stands and as I said, your
> continued derailment and
> > disruption will not be tolerated. Thank you for your input.
> >
> > Noah, I don't see Anatoly's postings here as derailment or disruption
> > and also see no reason to cultivate such a "won't be tolerated" tone
> > here.
>
>
> +1. I think the only "derailment" here is the way Anatoly has been treated
> for his contributions. Let's let Anatoly continue to submit and discuss
his
> PRs and let Donald do whatever he wants to do whenever he wants to do it.

For the public record: there's additional context here related to Anatoly's
interaction with the core CPython development team over the span of years.
Noah's response isn't related just to these pull requests, but is informed
by that past history.

(I'm not going to go into any more detail here, but if anyone is really
interested in the specifics, the relevant discussion is in the public
archives for the python-committers list)

Regards,
Nick.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131103/d813bd64/attachment.html>

From r1chardj0n3s at gmail.com  Thu Nov  7 23:43:44 2013
From: r1chardj0n3s at gmail.com (Richard Jones)
Date: Fri, 8 Nov 2013 09:43:44 +1100
Subject: [Distutils] HTTPS upload to PyPI
Message-ID: <CAHrZfZAf4F-Yncc22bxatwMEKCSNgfgj43NRPKce_KyO8E_U+Q@mail.gmail.com>

Because (distutils history) it's currently not possible to "python setup.py
upload" using the secure HTTPS URL of PyPI. It is possible using twine*
which I'd like to promote at:

  https://pypi.python.org/pypi    (in the "Package Authors" box)
  http://docs.python.org/3/distutils/packageindex.html

I say this with the caveat that I've never actually _used_ twine myself.


    Richard

* https://pypi.python.org/pypi/twine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131108/78da772d/attachment.html>

From ncoghlan at gmail.com  Fri Nov  8 00:44:34 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Nov 2013 09:44:34 +1000
Subject: [Distutils] HTTPS upload to PyPI
In-Reply-To: <CAHrZfZAf4F-Yncc22bxatwMEKCSNgfgj43NRPKce_KyO8E_U+Q@mail.gmail.com>
References: <CAHrZfZAf4F-Yncc22bxatwMEKCSNgfgj43NRPKce_KyO8E_U+Q@mail.gmail.com>
Message-ID: <CADiSq7eDFAaXMuFLqmyNXdMH8G_Datd2NVo26tECWdds4S_6aQ@mail.gmail.com>

On 8 Nov 2013 08:44, "Richard Jones" <r1chardj0n3s at gmail.com> wrote:
>
> Because (distutils history) it's currently not possible to "python
setup.py upload" using the secure HTTPS URL of PyPI. It is possible using
twine* which I'd like to promote at:
>
>   https://pypi.python.org/pypi    (in the "Package Authors" box)
>   http://docs.python.org/3/distutils/packageindex.html
>
> I say this with the caveat that I've never actually _used_ twine myself.

I'd hold off on changing the CPython docs until after they mention pip (as
a docs change, that's currently towards the bottom of my "before beta 1"
todo list)

However, the distribution section of the packaging guide would be a good
place to mention it.

Cheers,
Nick.

>
>
>     Richard
>
> * https://pypi.python.org/pypi/twine
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131108/e49eda51/attachment.html>

From vinay_sajip at yahoo.co.uk  Fri Nov  8 12:50:02 2013
From: vinay_sajip at yahoo.co.uk (Vinay Sajip)
Date: Fri, 8 Nov 2013 11:50:02 +0000 (GMT)
Subject: [Distutils] HTTPS upload to PyPI
In-Reply-To: <CAHrZfZAf4F-Yncc22bxatwMEKCSNgfgj43NRPKce_KyO8E_U+Q@mail.gmail.com>
Message-ID: <1383911402.56959.YahooMailBasic@web171403.mail.ir2.yahoo.com>

Note that distil also uses SSL and verifies the host certificate when uploading, just as it does for downloads (N.B. distil has been recently updated since recent distlib releases, so the latest version [1] should be used).

Regards,

Vinay Sajip

[1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py

--------------------------------------------
On Thu, 7/11/13, Richard Jones <r1chardj0n3s at gmail.com> wrote:

 Subject: [Distutils] HTTPS upload to PyPI
 To: "disutils-sig" <distutils-sig at python.org>
 Date: Thursday, 7 November, 2013, 22:43
 
 Because (distutils history)
 it's currently not possible to "python setup.py
 upload" using the secure HTTPS URL of PyPI. It is
 possible using twine* which I'd like to promote
 at:
 
 ? https://pypi.python.org/pypi
 ? ?(in the "Package Authors" box)
 ? http://docs.python.org/3/distutils/packageindex.html
 
 
 I say this with the caveat that I've never
 actually _used_ twine myself.
 
 ? ? Richard
 *?https://pypi.python.org/pypi/twine
 
 
 -----Inline Attachment Follows-----
 
 _______________________________________________
 Distutils-SIG maillist? -? Distutils-SIG at python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
 

From techtonik at gmail.com  Fri Nov  8 13:21:42 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 8 Nov 2013 15:21:42 +0300
Subject: [Distutils] PyPI v2 (Was:  PyPI pull request #7)
Message-ID: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>

On Thu, Oct 31, 2013 at 2:38 PM, Donald Stufft <donald at stufft.io> wrote:
>
> On Oct 31, 2013, at 7:32 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>
>> Why I am skeptical that new site will replace old one soon? Just
>> because I don't believe in rewrites by one man army. When you develop
>> public resource, you need to rely on external feedback.
>
> Warehouse is live at https://preview-pypi.python.org/ (No landing page yet)

Not very impressive - I don't enjoy huge box for one word query -
guess Google is small for a reason. =)

Updated https://wiki.python.org/moin/CheeseShopDev

It still needs to fill up few historical details for historic preservation:
1. How many PyPI versions was there?
2. When the first one was deployed? (web.archive.org 1st reference is
dated 2007)
3. Where it was hosted, who were authors?
4. Other interesting historic facts

I like the initiative - there is something to learn from (if you're
already proficient to read advanced web code). Still I am not yet
ready to help with something I can't use. I guess I need to wait until
beta, or until there is some two-week spring-planning process, where I
can see if there is something worthy to add.

Still I couldn't resist to ask about opinion on things that don't
match my idealistic universe.


First some background questions:
1. Everything for core development is in HG. Why Git now? Why
Mercurial suxx (three major personal annoyances will do)? Why
BitBucket suxx?
2. Why Apache 2.0? Is it because everybody is using that? Why not try
CC0/CC-BY/MIT/ISC - code under these licenses is easier to copy/paste,
and is a better investment for your time as a coder studying the stuff
and filling up the space of your brain.

3. Why the raw SQL? It cripples the ability to scale, to host package
indexes on GAE or OpenStack. And it is right at the core. Why not to
"port NDB" to SQL or use HyperDB from Roundup?
4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff)

>> You also need
>> some designer guy in a team.
>
> Have one who?ll have time in a few weeks to go over everything and make it really great :)

That's cool. Will it be an iterative process or just one shot? Do you
care about public feedback? I mean of course you do, but is there a
public channel to allow users set status if they liked the previous
version more, and what should be changed for them to change their
opinion?

>> You also need a backlog for
>> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
>> Donald and Richard will be working on it full time.
>
> Possibly! I?m unsure of how long it will take, it?s primarily Richard and I but we?ve a
> few domain experts in particular pieces who have offered to help out as well when
> we?re ready for their pieces.

PostgreSQL is killing all the fun. It takes a few hours to get
acquainted with its way of doing things - groups as users, per table
grants and insufficient DB permissions. There should be an option to
run on SQlite for development, like in Trac, Roundup etc.

>> So, instead of
>> all-or-nothing scenario I can try to find some help with incremental
>> approach.
>
> Mostly the problem with improving the current base is every change is particularly
> dangerous. The code base is large, it?s untested, and it?s not very nice. It?s extremely
> easy to break things by accident with seemingly unrelated change. Richard and I
> have both done this multiple times.

This smells like a bad spaghetti. Do you think it will be maintainable
if it is already so fragile on the early stages of development?

> So every PR we accept has a danger of breaking
> things. This incurs a high cognitive overload for accepting a PR because it means we
> have to spin up a copy of the site and manually go through and test any feature we
> think *might* be affected which typically catches most things but not always. It?s a time
> and labor intensive process which none of us enjoy and which we?ve decided not to
> prioritize unless the pay offs are large.

If you're not doing this only for yourself and your own pleasure, then
dedicating more time to improve the process and making it more
inclusive will definitely pay off for the rest of pydotorg projects
and active part of Python community, who is still here.


Do you have a diagram of the system what are you trying to build, or
is it just an experiment? It may worth to put IDE aside and try to
de-couple some things first on the whiteboard. I am afraid that I will
never be ready to help with reinventing Django. At least not without
some research and art coverage.


Most of this work about best practices, collaboration experiments and
people communication should be made by PSF. When I donate to PSF I
hope it thinks about and develops ways to encourage collaboration,
help people grow in Python, share experience and learn from each
other. Sprints, pydotorg projects, public discussions and code reviews
- they should all work together. There are good examples of open
source collaboration in Qt, Chromium and Android projects. The process
can be streamlined and fun, but nobody can even meet each other to
work on commonly interested things. Only random people who once a year
can afford time and budget to visit PyCon. I dislike "motivational"
blog posts about initiatives, without real visible research and
actions with weekly retrospectives. For that I'd propose PSF to change
to post-factum writing style from motivational, especially when the
future mechanism of the good intent is unknown or unreliable. And the
first thing it needs to do - is to start helping people inside
pydotorg communicate with outside, removing the barriers and telling
what help is needed, who is needed and where.
-- 
anatoly t.

From donald at stufft.io  Fri Nov  8 13:41:54 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 8 Nov 2013 07:41:54 -0500
Subject: [Distutils] PyPI v2 (Was:  PyPI pull request #7)
In-Reply-To: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>
References: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>
Message-ID: <8100BD88-9CDC-4FFB-89A9-0554F89B91CE@stufft.io>


On Nov 8, 2013, at 7:21 AM, anatoly techtonik <techtonik at gmail.com> wrote:

> First some background questions:
> 1. Everything for core development is in HG. Why Git now? Why
> Mercurial suxx (three major personal annoyances will do)? Why
> BitBucket suxx?

I?m most familiar with git, so I used git when I started it.

> 2. Why Apache 2.0? Is it because everybody is using that? Why not try
> CC0/CC-BY/MIT/ISC - code under these licenses is easier to copy/paste,
> and is a better investment for your time as a coder studying the stuff
> and filling up the space of your brain.

Apache 2.0 is a good license, it is similar to MIT/ISC except it has a
patent clause.

> 
> 3. Why the raw SQL? It cripples the ability to scale, to host package
> indexes on GAE or OpenStack. And it is right at the core. Why not to
> "port NDB" to SQL or use HyperDB from Roundup?

It was just recently switched to raw sql from using the SQL Alchemy
SQL expression layer. The reason for the switch was we were not
using the portability features and it was more complicated to understand
the expression layer than just plain SQL.

> 4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff)

I?ll be adding a section to the documentation on this eventually, but
basically.

Django?s ORM wasn?t powerful enough to represent the existing Schema. When
I had to throw out the ORM I found it difficult to integrate other pieces without
instating more global state. Ultimately I felt that a Fraken-Django that used Django
on the surface but with pieces replaced was going to require as much or more
of  a learning curve, even for Django developers, that it didn?t make sense to
do it.

I then took a look at Flask, but Flask?s reliance on thread locals everywhere is
not something I?m a fan of.

At that point I was a bit frustrated and fed up and turned to using just Werkzeug
as a library. This enabled a rapid increase in pace (I got more down over the
initial weekend then I had in a month previously).

> 
>>> You also need
>>> some designer guy in a team.
>> 
>> Have one who?ll have time in a few weeks to go over everything and make it really great :)
> 
> That's cool. Will it be an iterative process or just one shot? Do you
> care about public feedback? I mean of course you do, but is there a
> public channel to allow users set status if they liked the previous
> version more, and what should be changed for them to change their
> opinion?

Design by committee doesn?t work. People are free to give feedback through
either this list, the IRC channel, github issues, or to me privately but ultimately
I trust the designer to do a good job, to listen to any feedback if we get any,
and to execute well.

Feedback is good, but it?ll be tempered to prevent http://theoatmeal.com/comics/design_hell

> 
>>> You also need a backlog for
>>> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
>>> Donald and Richard will be working on it full time.
>> 
>> Possibly! I?m unsure of how long it will take, it?s primarily Richard and I but we?ve a
>> few domain experts in particular pieces who have offered to help out as well when
>> we?re ready for their pieces.
> 
> PostgreSQL is killing all the fun. It takes a few hours to get
> acquainted with its way of doing things - groups as users, per table
> grants and insufficient DB permissions. There should be an option to
> run on SQlite for development, like in Trac, Roundup etc.

Warehouse only runs on PostgreSQL. It?s not that hard to have a local copy of
PostgreSQL and It?ll enable using the advanced feature set of PostgreSQL
instead of being limited to the lowest common denominator.

> 
>>> So, instead of
>>> all-or-nothing scenario I can try to find some help with incremental
>>> approach.
>> 
>> Mostly the problem with improving the current base is every change is particularly
>> dangerous. The code base is large, it?s untested, and it?s not very nice. It?s extremely
>> easy to break things by accident with seemingly unrelated change. Richard and I
>> have both done this multiple times.
> 
> This smells like a bad spaghetti. Do you think it will be maintainable
> if it is already so fragile on the early stages of development?

This is talking about the current code base, not about Warehouse. Warehouse has
and requires 100% unit test coverage and will be gaining a functional test suite
on top of that to test high level user stories.

> 
>> So every PR we accept has a danger of breaking
>> things. This incurs a high cognitive overload for accepting a PR because it means we
>> have to spin up a copy of the site and manually go through and test any feature we
>> think *might* be affected which typically catches most things but not always. It?s a time
>> and labor intensive process which none of us enjoy and which we?ve decided not to
>> prioritize unless the pay offs are large.
> 
> If you're not doing this only for yourself and your own pleasure, then
> dedicating more time to improve the process and making it more
> inclusive will definitely pay off for the rest of pydotorg projects
> and active part of Python community, who is still here.

This was again about the existing code base and not Warehouse. Warehouse
is (hopefully) easy to iterate on, has what I believe is a well laid out
code base, and has an extensive unit test suite.

> 
> 
> Do you have a diagram of the system what are you trying to build, or
> is it just an experiment? It may worth to put IDE aside and try to
> de-couple some things first on the whiteboard. I am afraid that I will
> never be ready to help with reinventing Django. At least not without
> some research and art coverage.

It?s not a reinvention of Django. The ?framework? parts of Warehouse
are very small and are mostly glue that holds together pieces like
Werkzeug and SQLAlchemy.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131108/a3f45de2/attachment-0001.sig>

From domma at procoders.net  Mon Nov 11 16:26:06 2013
From: domma at procoders.net (Achim Domma)
Date: Mon, 11 Nov 2013 16:26:06 +0100
Subject: [Distutils] Isolation, virtualenvs and buildout
Message-ID: <8E1E2532-CEBC-47F7-B1A7-DE59494D74D9@procoders.net>

Hi,

I'm used to virtualenvs and I think I understood the general idea of build out. I need to build a pice of software out of several independent Python packages and I think buildout is the tool to use. But I don't get how buildout and virtualenv are related to each other. Asking Google, I found some hints that buildout is supposed to provide the same kind of isolation as virtualenvs - but only in the "old" 1.x version. The "new"(?) 2.0 version will change that. So I'm totally confused and ask for help.

My use case is simple: I have some python modules on our private pypi server and would like to deploy a pice of software composed out of them (including all dependencies).

Any starting guidance would be very appreciated!

cheers,
Achim

From jim at zope.com  Mon Nov 11 16:46:27 2013
From: jim at zope.com (Jim Fulton)
Date: Mon, 11 Nov 2013 10:46:27 -0500
Subject: [Distutils] Isolation, virtualenvs and buildout
In-Reply-To: <8E1E2532-CEBC-47F7-B1A7-DE59494D74D9@procoders.net>
References: <8E1E2532-CEBC-47F7-B1A7-DE59494D74D9@procoders.net>
Message-ID: <CAPDm-Fipfw1t8BTyB+J-ATusecJKYPmp4sgMM1GYq2HQOc1xRg@mail.gmail.com>

On Mon, Nov 11, 2013 at 10:26 AM, Achim Domma <domma at procoders.net> wrote:
> Hi,
>
> I'm used to virtualenvs and I think I understood the general idea of build out. I need to build a pice of software out of several independent Python packages and I think buildout is the tool to use. But I don't get how buildout and virtualenv are related to each other. Asking Google, I found some hints that buildout is supposed to provide the same kind of isolation as virtualenvs - but only in the "old" 1.x version. The "new"(?) 2.0 version will change that. So I'm totally confused and ask for help.

Buildout1 tried to provide isolation, but it was incomplete and too
hard to maintain.

virtualenv works crazy hard to provide isolation. (/me applauds virtualenv.)

Many people run buildout inside virtualenvs.  Personally, I always keep a
unadulterated Python around that I run buildout with.  Buildouts main
contribution
is that it doesn't install things into site packages and won't mess up
your clean Python
or your virtualenv.   You could use virtualenv to make yourself a
clean Python and
run any number of buildouts with that virtualenv.

> My use case is simple: I have some python modules on our private pypi server and would like to deploy a pice of software composed out of them (including all dependencies).

That's similar to our use cases.

> Any starting guidance would be very appreciated!

FWIW, here's a script I use to make RPMs out of buildouts:

https://gist.github.com/jimfulton/6629791

It builds on zc.sourcerelease, which builds self-contained
source tar balls.

We (ZC) deploy *software* using RPM and use other buildout based tools
to deploy application configuration.

I suspect in the future we'll to docker or maybe something simpler
than RPM, which doesn't really match our use cases very well and tends
to cause us no end of pain.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From p.f.moore at gmail.com  Mon Nov 11 17:29:32 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 11 Nov 2013 16:29:32 +0000
Subject: [Distutils] setuptools and use_2to3
Message-ID: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>

I'm trying to convert a setup.py to use setuptools  (specifically the
docutils build, FWIW). The existing setup.py has some custom command
stuff to run 2to3 on build.

As I understand it, setuptools should do this automatically by using
use_2to3=True.

So, I've changed the code to just do:

setup(
    ...
    packages=find_packages(),
    use_2to3=True,
    ...
)

However, when I do python setup.py bdist_wheel, I find that
docutils/__init__.py within the wheel still contains (for example)

    class ApplicationError(StandardError):
        ...

So the StandardError -> Exception fixer isn't being run.

Am I doing something wrong here? The setuptools documentation isn't
very clear on what I should do, or what to expect.

Thanks,
Paul

From marius at pov.lt  Mon Nov 11 18:26:22 2013
From: marius at pov.lt (Marius Gedminas)
Date: Mon, 11 Nov 2013 19:26:22 +0200
Subject: [Distutils] setuptools and use_2to3
In-Reply-To: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
References: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
Message-ID: <20131111172622.GA8229@fridge.pov.lt>

On Mon, Nov 11, 2013 at 04:29:32PM +0000, Paul Moore wrote:
> I'm trying to convert a setup.py to use setuptools  (specifically the
> docutils build, FWIW). The existing setup.py has some custom command
> stuff to run 2to3 on build.

Aside: I believe there's consensus now that 2to3 is not the best way to
support Python 2.x and 3.x.  It's not that hard to have a single source
tree in a portable subset of Python 2.x and 3.x.  See python3porting.com
and https://pypi.python.org/pypi/six.

> As I understand it, setuptools should do this automatically by using
> use_2to3=True.
> 
> So, I've changed the code to just do:
> 
> setup(
>     ...
>     packages=find_packages(),
>     use_2to3=True,
>     ...
> )
> 
> However, when I do python setup.py bdist_wheel, I find that
> docutils/__init__.py within the wheel still contains (for example)
> 
>     class ApplicationError(StandardError):
>         ...
> 
> So the StandardError -> Exception fixer isn't being run.
> 
> Am I doing something wrong here? The setuptools documentation isn't
> very clear on what I should do, or what to expect.

I wouldn't be surprised to learn that bdist_wheel doesn't support
use_2to3, but I don't actually *know* that for a fact, since I never
used 2to3.

Marius Gedminas
-- 
What goes up, must come down. Ask any system administrator.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131111/d2570b01/attachment.sig>

From p.f.moore at gmail.com  Mon Nov 11 18:55:25 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 11 Nov 2013 17:55:25 +0000
Subject: [Distutils] setuptools and use_2to3
In-Reply-To: <20131111172622.GA8229@fridge.pov.lt>
References: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
 <20131111172622.GA8229@fridge.pov.lt>
Message-ID: <CACac1F_GqxyPrg2CSg3bcYNdSE5qY57umWY0GEs5LaFbRruXfQ@mail.gmail.com>

On 11 November 2013 17:26, Marius Gedminas <marius at pov.lt> wrote:
> On Mon, Nov 11, 2013 at 04:29:32PM +0000, Paul Moore wrote:
>> I'm trying to convert a setup.py to use setuptools  (specifically the
>> docutils build, FWIW). The existing setup.py has some custom command
>> stuff to run 2to3 on build.
>
> Aside: I believe there's consensus now that 2to3 is not the best way to
> support Python 2.x and 3.x.  It's not that hard to have a single source
> tree in a portable subset of Python 2.x and 3.x.  See python3porting.com
> and https://pypi.python.org/pypi/six.

Yes, I realise that. I'm not a fan of 2to3 at all myself, but it's
what docutils uses and I'm not trying to do a full docutils port to
shared source, just a quick local fork of setup.py to allow me to make
some simple changes (using entry points rather than custom scripts, in
particular).

I only started looking at the 2to3 stuff because docutils' setup.py
does some complex distutils customisation that I thought might give
setuptools some problems. And as all it was doing was implementing
things that are built into setuptools, it seemed worth looking at as a
simplification.

> I wouldn't be surprised to learn that bdist_wheel doesn't support
> use_2to3, but I don't actually *know* that for a fact, since I never
> used 2to3.

It's possible I guess. But I thought bdist_wheel used the normal build
command, so stuff like 2to3 support should be unchanged.

Thanks for the comments anyway,
Paul

From aclark at aclark.net  Mon Nov 11 19:39:53 2013
From: aclark at aclark.net (Alex Clark)
Date: Mon, 11 Nov 2013 18:39:53 +0000 (UTC)
Subject: [Distutils] Isolation, virtualenvs and buildout
References: <8E1E2532-CEBC-47F7-B1A7-DE59494D74D9@procoders.net>
Message-ID: <loom.20131111T193547-162@post.gmane.org>

Achim Domma <domma <at> procoders.net> writes:

> 
> Any starting guidance would be very appreciated!


As of 2.0, Buildout no longer provides isolation so you should do something
like:

$ bin/virtualenv .
$ bin/pip install zc.buildout
$ bin/buildout init

Then start writing your buildout.cfg.




From regebro at gmail.com  Mon Nov 11 21:40:19 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 11 Nov 2013 21:40:19 +0100
Subject: [Distutils] setuptools and use_2to3
In-Reply-To: <20131111172622.GA8229@fridge.pov.lt>
References: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
 <20131111172622.GA8229@fridge.pov.lt>
Message-ID: <CAL0kPAU7j1A4etbQq0s=mVy+cLSF5B2t=kkmCLxLr7+SjhQ7rA@mail.gmail.com>

On Mon, Nov 11, 2013 at 6:26 PM, Marius Gedminas <marius at pov.lt> wrote:
> Aside: I believe there's consensus now that 2to3 is not the best way to
> support Python 2.x and 3.x.  It's not that hard to have a single source
> tree in a portable subset of Python 2.x and 3.x.  See python3porting.com
> and https://pypi.python.org/pypi/six.

Well, that depends. Mainly on what Python versions you need to support.

From regebro at gmail.com  Mon Nov 11 21:45:53 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 11 Nov 2013 21:45:53 +0100
Subject: [Distutils] setuptools and use_2to3
In-Reply-To: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
References: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
Message-ID: <CAL0kPAXkxZ=CSD3e+gdLzsDHaT0jOOhYJz-iHpUND1VcT++KwQ@mail.gmail.com>

On Mon, Nov 11, 2013 at 5:29 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> I'm trying to convert a setup.py to use setuptools  (specifically the
> docutils build, FWIW). The existing setup.py has some custom command
> stuff to run 2to3 on build.
>
> As I understand it, setuptools should do this automatically by using
> use_2to3=True.
>
> So, I've changed the code to just do:
>
> setup(
>     ...
>     packages=find_packages(),
>     use_2to3=True,
>     ...
> )
>
> However, when I do python setup.py bdist_wheel, I find that
> docutils/__init__.py within the wheel still contains (for example)
>
>     class ApplicationError(StandardError):
>         ...
>
> So the StandardError -> Exception fixer isn't being run.
>
> Am I doing something wrong here? The setuptools documentation isn't
> very clear on what I should do, or what to expect.

If 2to3 is being run you'll get quite a lot of fairly obvious output
to that effect. So it sounds like it's not.

A quick look at bdist_egg seems to indicate that it never runs the
build_py step, which is what runs 2to3. So this may be a bug.

//Lennart

From p.f.moore at gmail.com  Mon Nov 11 23:26:22 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 11 Nov 2013 22:26:22 +0000
Subject: [Distutils] setuptools and use_2to3
In-Reply-To: <CAL0kPAXkxZ=CSD3e+gdLzsDHaT0jOOhYJz-iHpUND1VcT++KwQ@mail.gmail.com>
References: <CACac1F-R1DzJ7R1e3mNveQ2jhM0RfC=j5no0-PX2O0YUFevqnA@mail.gmail.com>
 <CAL0kPAXkxZ=CSD3e+gdLzsDHaT0jOOhYJz-iHpUND1VcT++KwQ@mail.gmail.com>
Message-ID: <CACac1F-h6E4isktXdYehQW5ChOdA4NRzRJ-WVBquFK=ee7Z1GA@mail.gmail.com>

On 11 November 2013 20:45, Lennart Regebro <regebro at gmail.com> wrote:
> If 2to3 is being run you'll get quite a lot of fairly obvious output
> to that effect. So it sounds like it's not.
>
> A quick look at bdist_egg seems to indicate that it never runs the
> build_py step, which is what runs 2to3. So this may be a bug.

Good thought. python setup.py build does run 2to3, and as you say the
output is not something you can miss. So setup.py bdist_wheel isn't
running build, which is likely a bug (but also not hard to work
around).

That's probably good enough for now - this whole exercise is a local
hack, so just getting it to work is likely good enough. I'm not enough
of a fan of the whole 2to3 process to be bothered spending time
producing a reproducible test case, etc, for a good-quality bug
report. I might just report it as it stands and leave it at that,
though, so at least it doesn't get forgotten.

Thanks.
Paul

From contact at stephane-klein.info  Tue Nov 12 14:12:05 2013
From: contact at stephane-klein.info (=?ISO-8859-1?Q?St=E9phane_Klein?=)
Date: Tue, 12 Nov 2013 14:12:05 +0100
Subject: [Distutils] Update nose2-cov owner request,
 because current maintenair is away since 2013-03
Message-ID: <l5t9er$9mr$1@ger.gmane.org>

Hi,

I would like get write access on "nose2-cov" package on pypi
:https://pypi.python.org/pypi/nose2-cov/1.0a4

My username is : "harobed"

Why this request : because I've did this pull request
https://bitbucket.org/memedough/nose2-cov/pull-request/2/improve-readme-append-how-to-enable-nose2/diff

I've no answer from author since 2013-03-24.

I've send bitbucket direct message, I send directly one email at
memedough at gmail.com (2013-08-30), no answer :(

So, I would like get write access to this package on pypi.

My apologies if I post my request in bad mailing list, if so what is the good
place to post this request ?

Best regards,
Stephane
-- 
St?phane Klein <contact at stephane-klein.info>
blog: http://stephane-klein.info
Twitter: http://twitter.com/klein_stephane
cv: http://cv.stephane-klein.info


From cfinch at ieee.org  Tue Nov 12 16:35:18 2013
From: cfinch at ieee.org (Craig Finch)
Date: Tue, 12 Nov 2013 10:35:18 -0500
Subject: [Distutils] How do I use pypissh to upload a package to pypi?
Message-ID: <CAF_Mnr9-tEkFNQHrSvV89p8JmKA2FzzUM=k-yvBHCy_bscwV5g@mail.gmail.com>

I am trying to distribute a package on pypi, but I have not been able to
figure out how to use pypissh to establish a secure connection with the
command-line tools.

I successfully uploaded my package using the traditional username/password
method.  It can be found at:
https://pypi.python.org/pypi/Pizza.py/0.1.0

I then installed pypissh (on Linux) using the command
pip install --user pypissh
and verified that Python can load the module.  I generated an RSA key pair
and added the public key into my pypi account using the web interface.  I
added my private key to my keychain using ssh-add.  Now, when I try to run

python setup.py register

I get the following:

running register
running egg_info
writing requirements to Pizza.py.egg-info/requires.txt
writing Pizza.py.egg-info/PKG-INFO
writing top-level names to Pizza.py.egg-info/top_level.txt
writing dependency_links to Pizza.py.egg-info/dependency_links.txt
reading manifest file 'Pizza.py.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'Pizza.py.egg-info/SOURCES.txt'
running check
Registering Pizza.py to http://submit at ssh.pypi.python.org/pypi
Permission denied (publickey).
Traceback (most recent call last):
  File "setup.py", line 18, in <module>
    data_files=[('pizza/test/files', glob("pizza/test/files/*"))]
  File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/site-packages/setuptools/command/register.py",
line 9, in run
    _register.run(self)
  File "/usr/lib64/python2.7/distutils/command/register.py", line 57, in run
    self.send_metadata()
  File "/usr/lib64/python2.7/distutils/command/register.py", line 168, in
send_metadata
    auth)
  File "/usr/lib64/python2.7/distutils/command/register.py", line 300, in
post_to_server
    result = opener.open(req)
  File "/home/cfinch/.local/lib64/python2.7/site-packages/pypissh.py", line
66, in open
    return OpenerDirector.open(self, req, data=data)
  File "/usr/lib64/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/home/cfinch/.local/lib64/python2.7/site-packages/pypissh.py", line
49, in httpssh_open
    return self.do_open(HTTPSSHConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 373, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

It seems that my key is being rejected.  How do I go about debugging this?
Did I miss a step in the configuration of pypyssh?

   Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131112/469da3da/attachment.html>

From techtonik at gmail.com  Thu Nov 14 23:00:04 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 15 Nov 2013 01:00:04 +0300
Subject: [Distutils] PyPI v2 (Was:  PyPI pull request #7)
In-Reply-To: <8100BD88-9CDC-4FFB-89A9-0554F89B91CE@stufft.io>
References: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>
 <8100BD88-9CDC-4FFB-89A9-0554F89B91CE@stufft.io>
Message-ID: <CAPkN8x+a3=zF6Yw=Be5ysdWW_k05ogdCBae7Lam0+LMeGys+NA@mail.gmail.com>

First things first, thanks for the detailed response.

On Fri, Nov 8, 2013 at 3:41 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> First some background questions:
>> 1. Everything for core development is in HG. Why Git now? Why
>> Mercurial suxx (three major personal annoyances will do)? Why
>> BitBucket suxx?
>
> I?m most familiar with git, so I used git when I started it.

Seems fair.

>> 2. Why Apache 2.0? Is it because everybody is using that? Why not try
>> CC0/CC-BY/MIT/ISC - code under these licenses is easier to copy/paste,
>> and is a better investment for your time as a coder studying the stuff
>> and filling up the space of your brain.
>
> Apache 2.0 is a good license, it is similar to MIT/ISC except it has a
> patent clause.

Now everybody is also afraid of patents. Are there any other *short* licenses
with patent grant? Why patents started to work by default? I thought you
should explicitly mention if you work is patented or pending for them to work.

Regarding licenses, I'd still prefer to learn codebase that I can freely
copy/paste to my own projects without additional obligations like some that
imposed by Apache 2.0. I can understand crediting, but everything else just
keeps people away.

If not sure if CC-BY applies to software projects, but I'd choose something
as simple as this not only for code, but also for pydotorg content.

>> 3. Why the raw SQL? It cripples the ability to scale, to host package
>> indexes on GAE or OpenStack. And it is right at the core. Why not to
>> "port NDB" to SQL or use HyperDB from Roundup?
>
> It was just recently switched to raw sql from using the SQL Alchemy
> SQL expression layer. The reason for the switch was we were not
> using the portability features and it was more complicated to understand
> the expression layer than just plain SQL.

Ok. SQL Alchemy is too big and complicated. Why NoSQL didn't fit?
MongoDB is all over the net nowadays.

>> 4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff)
>
> I?ll be adding a section to the documentation on this eventually, but
> basically.
>
> Django?s ORM wasn?t powerful enough to represent the existing Schema. When
> I had to throw out the ORM I found it difficult to integrate other pieces without
> instating more global state. Ultimately I felt that a Fraken-Django that used Django
> on the surface but with pieces replaced was going to require as much or more
> of  a learning curve, even for Django developers, that it didn?t make sense to
> do it.

Maybe a good partitioning of application with big diagram could bring the best
from both world? Bare SQL and getting to the basics seems to radical to me.

> At that point I was a bit frustrated and fed up and turned to using just Werkzeug
> as a library. This enabled a rapid increase in pace (I got more down over the
> initial weekend then I had in a month previously).

Isn't it just a WSGI library? What about reusable components that are already
written? I wouldn't risk rewriting OAuth support from scratch.

> Feedback is good, but it?ll be tempered to prevent http://theoatmeal.com/comics/design_hell

That's a nice argument. =)

>>>> You also need a backlog for
>>>> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
>>>> Donald and Richard will be working on it full time.
>>>
>>> Possibly! I?m unsure of how long it will take, it?s primarily Richard and I but we?ve a
>>> few domain experts in particular pieces who have offered to help out as well when
>>> we?re ready for their pieces.
>>
>> PostgreSQL is killing all the fun. It takes a few hours to get
>> acquainted with its way of doing things - groups as users, per table
>> grants and insufficient DB permissions. There should be an option to
>> run on SQlite for development, like in Trac, Roundup etc.
>
> Warehouse only runs on PostgreSQL. It?s not that hard to have a local copy of
> PostgreSQL and It?ll enable using the advanced feature set of PostgreSQL
> instead of being limited to the lowest common denominator.

What feature sets you need? Does they really worth killing the fun of working,
hack-a-toning and for local development?

>>>> So, instead of
>>>> all-or-nothing scenario I can try to find some help with incremental
>>>> approach.
>>>
>>> Mostly the problem with improving the current base is every change is particularly
>>> dangerous. The code base is large, it?s untested, and it?s not very nice. It?s extremely
>>> easy to break things by accident with seemingly unrelated change. Richard and I
>>> have both done this multiple times.
>>
>> This smells like a bad spaghetti. Do you think it will be maintainable
>> if it is already so fragile on the early stages of development?
>
> This is talking about the current code base, not about Warehouse. Warehouse has
> and requires 100% unit test coverage and will be gaining a functional test suite
> on top of that to test high level user stories.

At least somebody who is not getting mad using "user story" buzzwords.
A pleasure. =)
Sorry I missed the point that you broke old PyPI code, now the new
one. Still, 100% of
test coverage makes it hard to make changes, sometimes more hard than necessary.
Perhaps I should take a look at this.

BTW, why didn't you just copy crate.io?

> Warehouse
> is (hopefully) easy to iterate on, has what I believe is a well laid out
> code base, and has an extensive unit test suite.

Ok. I should take a closer look.

>> Do you have a diagram of the system what are you trying to build, or
>> is it just an experiment? It may worth to put IDE aside and try to
>> de-couple some things first on the whiteboard. I am afraid that I will
>> never be ready to help with reinventing Django. At least not without
>> some research and art coverage.
>
> It?s not a reinvention of Django. The ?framework? parts of Warehouse
> are very small and are mostly glue that holds together pieces like
> Werkzeug and SQLAlchemy.

Still, is there a diagram of Warehouse architecture? Blueprint,
CAD model of a building?

From amk at amk.ca  Fri Nov 15 12:47:51 2013
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 15 Nov 2013 06:47:51 -0500
Subject: [Distutils] Reviews wanted for two 3.4 regression fixes
Message-ID: <20131115114751.GA15894@datlandrewk.home>

Some bugfixes that are in Python 2.7 disappeared from Python 3.x
because a bunch of Distutils work was reverted.  Jason Coombs and I
spent last Sunday figuring out what got rolled back; see
http://bugs.python.org/issue19544 for details, and for links to the 5
bugfixes that were lost.

Two of the five have already been fixed on the 3.4 'default' branch.

Two of the issues have patches against 3.4 that need review.  Please review
these patches!

    http://bugs.python.org/issue1180
    http://bugs.python.org/issue6516

3.4beta1 is due in roughly two weeks, so I'd like to get at least one
review of these patches before I apply them.

The fifth issue, <http://bugs.python.org/issue6466>, is a code cleanup
to remove some code duplication.  Jason and I were going to leave this
out, because it's mostly a code-tidiness issue and not worth much risk
(no one seems to have run into the discrepancy between 2.7 and 3.4 in
practice).  OTOH, the changes *are* a difference; perhaps making 2.7
and 3.4 match is worth the risk.  If someone wants to apply that issue
in 3.4 as well, or argue for doing so, please do!  Otherwise, I don't
intend to work on #6466.

--amk

From kiorky at cryptelium.net  Fri Nov 15 12:57:02 2013
From: kiorky at cryptelium.net (kiorky)
Date: Fri, 15 Nov 2013 12:57:02 +0100
Subject: [Distutils] is pypi in bad state ?
Message-ID: <52860C0E.60703@cryptelium.net>

Is pypi down only for me ?

$ virtualenv --no-site-packages foo
$ . foo/bin/activate
$ easy_install -N plone.multilingualbehavior==1.0
Searching for plone.multilingualbehavior==1.0
Reading http://pypi.python.org/simple/plone.multilingualbehavior/
Best match: plone.multilingualbehavior 1.0
Downloading 
https://pypi.python.org/packages/2.7/p/plone.multilingualbehavior/plone.multilingualbehavior-1.0-py2.7.egg#md5=8baedb0b8eed289eabd1ddfa6ec371e6
error: Can't download 
https://pypi.python.org/packages/2.7/p/plone.multilingualbehavior/plone.multilingualbehavior-1.0-py2.7.egg: 
503 Backend is unhealthy

-- 
Cordialement,
kiorky


From robert.kern at gmail.com  Fri Nov 15 13:09:12 2013
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 15 Nov 2013 12:09:12 +0000
Subject: [Distutils] is pypi in bad state ?
In-Reply-To: <52860C0E.60703@cryptelium.net>
References: <52860C0E.60703@cryptelium.net>
Message-ID: <l652sn$hp7$1@ger.gmane.org>

On 2013-11-15 11:57, kiorky wrote:
> Is pypi down only for me ?

No, I see the same error.

> $ virtualenv --no-site-packages foo
> $ . foo/bin/activate
> $ easy_install -N plone.multilingualbehavior==1.0
> Searching for plone.multilingualbehavior==1.0
> Reading http://pypi.python.org/simple/plone.multilingualbehavior/
> Best match: plone.multilingualbehavior 1.0
> Downloading
> https://pypi.python.org/packages/2.7/p/plone.multilingualbehavior/plone.multilingualbehavior-1.0-py2.7.egg#md5=8baedb0b8eed289eabd1ddfa6ec371e6
>
> error: Can't download
> https://pypi.python.org/packages/2.7/p/plone.multilingualbehavior/plone.multilingualbehavior-1.0-py2.7.egg:
> 503 Backend is unhealthy
>


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From donald at stufft.io  Fri Nov 15 13:10:01 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 15 Nov 2013 07:10:01 -0500
Subject: [Distutils] is pypi in bad state ?
In-Reply-To: <l652sn$hp7$1@ger.gmane.org>
References: <52860C0E.60703@cryptelium.net> <l652sn$hp7$1@ger.gmane.org>
Message-ID: <639E6934-3C9D-4445-9737-770314230B56@stufft.io>

http://status.python.org/

On Nov 15, 2013, at 7:09 AM, Robert Kern <robert.kern at gmail.com> wrote:

> On 2013-11-15 11:57, kiorky wrote:
>> Is pypi down only for me ?
> 
> No, I see the same error.
> 
>> $ virtualenv --no-site-packages foo
>> $ . foo/bin/activate
>> $ easy_install -N plone.multilingualbehavior==1.0
>> Searching for plone.multilingualbehavior==1.0
>> Reading http://pypi.python.org/simple/plone.multilingualbehavior/
>> Best match: plone.multilingualbehavior 1.0
>> Downloading
>> https://pypi.python.org/packages/2.7/p/plone.multilingualbehavior/plone.multilingualbehavior-1.0-py2.7.egg#md5=8baedb0b8eed289eabd1ddfa6ec371e6
>> 
>> error: Can't download
>> https://pypi.python.org/packages/2.7/p/plone.multilingualbehavior/plone.multilingualbehavior-1.0-py2.7.egg:
>> 503 Backend is unhealthy
>> 
> 
> 
> -- 
> Robert Kern
> 
> "I have come to believe that the whole world is an enigma, a harmless enigma
> that is made terrible by our own mad attempt to interpret it as though it had
> an underlying truth."
>  -- Umberto Eco
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/1acdb813/attachment.sig>

From qwcode at gmail.com  Fri Nov 15 22:16:14 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 15 Nov 2013 13:16:14 -0800
Subject: [Distutils] "pip uninstall" and pip's lack of support for
	installing eggs
Message-ID: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>

Under the covers, pip uses "setup.py install --record" (which requires also
using --single-version-externally-managed) in order to create an install
log, that get's used for uninstalls.

And due to using `--single-version-externally-managed`, the install is done
"old-style",  i.e., not installed as an egg, and not knowing how to install
*from* an egg.  the one caveat being that setuptools (not pip) does throw
in an .egg-info directory with the project metadata.

So is "pip uninstall" the primary historical reason for  the lack of egg
install support? or is there more to the story?

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/16f9db92/attachment.html>

From qwcode at gmail.com  Fri Nov 15 23:14:14 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 15 Nov 2013 14:14:14 -0800
Subject: [Distutils] "pip uninstall" and pip's lack of support for
	installing eggs
In-Reply-To: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
References: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
Message-ID: <CAPYWazrBjch+jFYctjDDPJbY0pBrfK8U0mpqwQvn0SD=aZWG+g@mail.gmail.com>

ok, looking at this more, "pip uninstall" wasn't added until v0.6, so
probably it just used `--record` for it's log as a side-effect of the
previous decision to use  `--single-version-externally-managed`.
and the decision to use `--single-version-externally-managed`, was probably
just based on pip needing to override the dependency resolution, and have
it's signature "requirements file" feature that supported overriding
dependencies.


On Fri, Nov 15, 2013 at 1:16 PM, Marcus Smith <qwcode at gmail.com> wrote:

> Under the covers, pip uses "setup.py install --record" (which requires
> also using --single-version-externally-managed) in order to create an
> install log, that get's used for uninstalls.
>
> And due to using `--single-version-externally-managed`, the install is
> done "old-style",  i.e., not installed as an egg, and not knowing how to
> install *from* an egg.  the one caveat being that setuptools (not pip) does
> throw in an .egg-info directory with the project metadata.
>
> So is "pip uninstall" the primary historical reason for  the lack of egg
> install support? or is there more to the story?
>
> Marcus
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/2c044ae0/attachment-0001.html>

From carl at oddbird.net  Fri Nov 15 22:37:15 2013
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 15 Nov 2013 14:37:15 -0700
Subject: [Distutils] "pip uninstall" and pip's lack of support for
 installing eggs
In-Reply-To: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
References: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
Message-ID: <5286940B.7040406@oddbird.net>

On 11/15/2013 02:16 PM, Marcus Smith wrote:
> Under the covers, pip uses "setup.py install --record" (which requires
> also using --single-version-externally-managed) in order to create an
> install log, that get's used for uninstalls. 
> 
> And due to using `--single-version-externally-managed`, the install is
> done "old-style",  i.e., not installed as an egg, and not knowing how to
> install *from* an egg.  the one caveat being that setuptools (not pip)
> does throw in an .egg-info directory with the project metadata.
> 
> So is "pip uninstall" the primary historical reason for  the lack of egg
> install support? or is there more to the story?

No, "pip uninstall" was a relatively late addition to pip; pip's choices
around installation and distribution formats long predate the addition
of the uninstall feature.

Actually if uninstall had been the primary motivating factor, pip
probably would install egg-style, as one of egg's benefits is that it
makes file-removal trivial (everything's in one zipfile/directory).

Ian Bicking is the only one who can really answer "original motivation"
questions about pip, but my understanding is that lack of support for
installing from eggs was intentional because binary eggs were kind of
broken for the platforms Ian cared about (for the same reasons
platform-specific wheels are still troublesome today on Linux/OSX).

And the flat installation style (--single-version-externally-managed)
was again, as far as I know, simply due to a preference for having a
single site-packages directory rather than the pth file stuff that
setuptools does to make eggs importable.

(For one thing, for a long time setuptools' pth stuff sort of broke
virtualenv; because setuptools forces installed eggs to the top of
sys.path, it meant that you couldn't override something installed as an
egg globally with something installed as a non-egg inside a virtualenv.
I fixed that a long time ago with an explicit workaround in virtualenv.)

Carl

From qwcode at gmail.com  Fri Nov 15 23:21:07 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 15 Nov 2013 14:21:07 -0800
Subject: [Distutils] "pip uninstall" and pip's lack of support for
 installing eggs
In-Reply-To: <5286940B.7040406@oddbird.net>
References: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
 <5286940B.7040406@oddbird.net>
Message-ID: <CAPYWazrqWXBSTdUA0miGrq2R9NcPPM+6pk0uMOS5dF_3H0Y5rw@mail.gmail.com>

> questions about pip, but my understanding is that lack of support for
> installing from eggs was intentional because binary eggs were kind of
> broken for the platforms Ian cared about (for the same reasons
> platform-specific wheels are still troublesome today on Linux/OSX).
>


thanks carl, we crossed in mid-air.
you didn't mention it helping the "requirements file" feature of dependency
overrides?
Am I wrong on that?



> (For one thing, for a long time setuptools' pth stuff sort of broke
> virtualenv; because setuptools forces installed eggs to the top of
> sys.path, it meant that you couldn't override something installed as an
> egg globally with something installed as a non-egg inside a virtualenv.
> I fixed that a long time ago with an explicit workaround in virtualenv.)
>

there's a similar situation now with easy_installed packages overriding pip
--user installs

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/49b3a12b/attachment.html>

From carl at oddbird.net  Fri Nov 15 23:39:14 2013
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 15 Nov 2013 15:39:14 -0700
Subject: [Distutils] "pip uninstall" and pip's lack of support for
 installing eggs
In-Reply-To: <CAPYWazrBjch+jFYctjDDPJbY0pBrfK8U0mpqwQvn0SD=aZWG+g@mail.gmail.com>
References: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
 <CAPYWazrBjch+jFYctjDDPJbY0pBrfK8U0mpqwQvn0SD=aZWG+g@mail.gmail.com>
Message-ID: <5286A292.7050502@oddbird.net>

On 11/15/2013 03:14 PM, Marcus Smith wrote:
> ok, looking at this more, "pip uninstall" wasn't added until v0.6, so
> probably it just used `--record` for it's log as a side-effect of the
> previous decision to use  `--single-version-externally-managed`.

Pip's use of --record to record a list of installed files actually
predates uninstall too (though of course it was useful/necessary in
implementing uninstall).

> and the decision to use `--single-version-externally-managed`, was
> probably just based on pip needing to override the dependency
> resolution, and have it's signature "requirements file" feature that
> supported overriding dependencies.

Uh, maybe? What does --single-version-externally-managed have to do with
dependencies? Does it prevent the installation of dependencies? I didn't
realize that.

In that case, does pip's --egg flag (which removes
--single-version-externally-managed from the install command, but
doesn't replace it with anything else) break requirements files?

Carl

From donald at stufft.io  Fri Nov 15 23:41:34 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 15 Nov 2013 17:41:34 -0500
Subject: [Distutils] "pip uninstall" and pip's lack of support for
	installing eggs
In-Reply-To: <5286A292.7050502@oddbird.net>
References: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
 <CAPYWazrBjch+jFYctjDDPJbY0pBrfK8U0mpqwQvn0SD=aZWG+g@mail.gmail.com>
 <5286A292.7050502@oddbird.net>
Message-ID: <DA248238-EA82-4F95-8AC8-32176C2C71DF@stufft.io>

Yes as I understand it. 

> On Nov 15, 2013, at 5:39 PM, Carl Meyer <carl at oddbird.net> wrote:
> 
> In that case, does pip's --egg flag (which removes
> --single-version-externally-managed from the install command, but
> doesn't replace it with anything else) break requirements files?

From qwcode at gmail.com  Sat Nov 16 00:47:54 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Fri, 15 Nov 2013 15:47:54 -0800
Subject: [Distutils] "pip uninstall" and pip's lack of support for
 installing eggs
In-Reply-To: <5286A292.7050502@oddbird.net>
References: <CAPYWazroKRwCTqEk4=mNE988n7YLFi7VKM2qSbHs00aiiTCCBw@mail.gmail.com>
 <CAPYWazrBjch+jFYctjDDPJbY0pBrfK8U0mpqwQvn0SD=aZWG+g@mail.gmail.com>
 <5286A292.7050502@oddbird.net>
Message-ID: <CAPYWazoeWV44QQr8TAJriKQ-g0ZZ5zrR2_iD+1bo+Z50DPJvHA@mail.gmail.com>

>
>
> Uh, maybe? What does --single-version-externally-managed have to do with
> dependencies? Does it prevent the installation of dependencies? I didn't
> realize that.
>

honestly, I have to re-remember how it works every month or so,  but yea,
that's what's going on in my understanding.


>
> In that case, does pip's --egg flag (which removes
> --single-version-externally-managed from the install command, but
> doesn't replace it with anything else) break requirements files?
>

yes, it breaks it  :  (
just logged an issue  https://github.com/pypa/pip/issues/1320
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/d2dabebb/attachment.html>

From donald at stufft.io  Sat Nov 16 04:12:26 2013
From: donald at stufft.io (Donald Stufft)
Date: Fri, 15 Nov 2013 22:12:26 -0500
Subject: [Distutils] Warehouse and XMLRPC
Message-ID: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>

The bulk of XMLRPC has been implemented in Warehouse. It would be great if
people who are using the XMLRPC could try it out using the https://preview-pypi.python.org/pypi
url.

Currently the search API does not work.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/d3427a4b/attachment-0001.sig>

From donald at stufft.io  Sat Nov 16 06:00:59 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 16 Nov 2013 00:00:59 -0500
Subject: [Distutils] PyPI v2 (Was:  PyPI pull request #7)
In-Reply-To: <CAPkN8x+a3=zF6Yw=Be5ysdWW_k05ogdCBae7Lam0+LMeGys+NA@mail.gmail.com>
References: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>
 <8100BD88-9CDC-4FFB-89A9-0554F89B91CE@stufft.io>
 <CAPkN8x+a3=zF6Yw=Be5ysdWW_k05ogdCBae7Lam0+LMeGys+NA@mail.gmail.com>
Message-ID: <54B6FCEB-F7BE-4E45-B18B-0A9D12458154@stufft.io>


On Nov 14, 2013, at 5:00 PM, anatoly techtonik <techtonik at gmail.com> wrote:

> First things first, thanks for the detailed response.
> 
> On Fri, Nov 8, 2013 at 3:41 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>>> First some background questions:
>>> 1. Everything for core development is in HG. Why Git now? Why
>>> Mercurial suxx (three major personal annoyances will do)? Why
>>> BitBucket suxx?
>> 
>> I?m most familiar with git, so I used git when I started it.
> 
> Seems fair.
> 
>>> 2. Why Apache 2.0? Is it because everybody is using that? Why not try
>>> CC0/CC-BY/MIT/ISC - code under these licenses is easier to copy/paste,
>>> and is a better investment for your time as a coder studying the stuff
>>> and filling up the space of your brain.
>> 
>> Apache 2.0 is a good license, it is similar to MIT/ISC except it has a
>> patent clause.
> 
> Now everybody is also afraid of patents. Are there any other *short* licenses
> with patent grant? Why patents started to work by default? I thought you
> should explicitly mention if you work is patented or pending for them to work.
> 
> Regarding licenses, I'd still prefer to learn codebase that I can freely
> copy/paste to my own projects without additional obligations like some that
> imposed by Apache 2.0. I can understand crediting, but everything else just
> keeps people away.
> 
> If not sure if CC-BY applies to software projects, but I'd choose something
> as simple as this not only for code, but also for pydotorg content.

If people don?t like the requirements of Apache License 2.0 that?s fine they
don?t need to contribute. Perhaps there might be a contributor or two lost
to that but I?m not too worried about it.

License is a bike shed issue and as it?s mine and Richard?s bike shed
it?ll be painted our color.

> 
>>> 3. Why the raw SQL? It cripples the ability to scale, to host package
>>> indexes on GAE or OpenStack. And it is right at the core. Why not to
>>> "port NDB" to SQL or use HyperDB from Roundup?
>> 
>> It was just recently switched to raw sql from using the SQL Alchemy
>> SQL expression layer. The reason for the switch was we were not
>> using the portability features and it was more complicated to understand
>> the expression layer than just plain SQL.
> 
> Ok. SQL Alchemy is too big and complicated. Why NoSQL didn't fit?
> MongoDB is all over the net nowadays.

NoSQL is a buzzword that doesn?t mean a whole lot other than ?Not a
relational database?. The various ?NoSQL? solutions each fit well in
different scenarios however this is not one of them. The database
is well within the limits of PostgreSQL and referential integrity, indexes,
and the like are useful constructs.

> 
>>> 4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff)
>> 
>> I?ll be adding a section to the documentation on this eventually, but
>> basically.
>> 
>> Django?s ORM wasn?t powerful enough to represent the existing Schema. When
>> I had to throw out the ORM I found it difficult to integrate other pieces without
>> instating more global state. Ultimately I felt that a Fraken-Django that used Django
>> on the surface but with pieces replaced was going to require as much or more
>> of  a learning curve, even for Django developers, that it didn?t make sense to
>> do it.
> 
> Maybe a good partitioning of application with big diagram could bring the best
> from both world? Bare SQL and getting to the basics seems to radical to me.

I?m not real keen on needing to do big diagrams. I don?t see any issue with writing
SQL here, it?s a perfectly fine language for querying a database.

> 
>> At that point I was a bit frustrated and fed up and turned to using just Werkzeug
>> as a library. This enabled a rapid increase in pace (I got more down over the
>> initial weekend then I had in a month previously).
> 
> Isn't it just a WSGI library? What about reusable components that are already
> written? I wouldn't risk rewriting OAuth support from scratch.

Werkzeug is just a WSGI library yes. This means that any reusable pieces we use
cannot be tied to a single framework. This might mean we need to make our own
reusable pieces and release them. I consider this a benefit to the greater ecosystem
as there is very little reason an OAuth implementation (for instance) needs to be
tied to a single framework.

> 
>> Feedback is good, but it?ll be tempered to prevent http://theoatmeal.com/comics/design_hell
> 
> That's a nice argument. =)
> 
>>>>> You also need a backlog for
>>>>> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if
>>>>> Donald and Richard will be working on it full time.
>>>> 
>>>> Possibly! I?m unsure of how long it will take, it?s primarily Richard and I but we?ve a
>>>> few domain experts in particular pieces who have offered to help out as well when
>>>> we?re ready for their pieces.
>>> 
>>> PostgreSQL is killing all the fun. It takes a few hours to get
>>> acquainted with its way of doing things - groups as users, per table
>>> grants and insufficient DB permissions. There should be an option to
>>> run on SQlite for development, like in Trac, Roundup etc.
>> 
>> Warehouse only runs on PostgreSQL. It?s not that hard to have a local copy of
>> PostgreSQL and It?ll enable using the advanced feature set of PostgreSQL
>> instead of being limited to the lowest common denominator.
> 
> What feature sets you need? Does they really worth killing the fun of working,
> hack-a-toning and for local development?

As I said, running a local copy of PostgreSQLis very easy. There are installers for
Windows and OSX, and pretty much every *nix distribution will have it packaged.

Off the top of my head I?m planning on using Array support, HStore, and functional/partial
indexes.

> 
>>>>> So, instead of
>>>>> all-or-nothing scenario I can try to find some help with incremental
>>>>> approach.
>>>> 
>>>> Mostly the problem with improving the current base is every change is particularly
>>>> dangerous. The code base is large, it?s untested, and it?s not very nice. It?s extremely
>>>> easy to break things by accident with seemingly unrelated change. Richard and I
>>>> have both done this multiple times.
>>> 
>>> This smells like a bad spaghetti. Do you think it will be maintainable
>>> if it is already so fragile on the early stages of development?
>> 
>> This is talking about the current code base, not about Warehouse. Warehouse has
>> and requires 100% unit test coverage and will be gaining a functional test suite
>> on top of that to test high level user stories.
> 
> At least somebody who is not getting mad using "user story" buzzwords.
> A pleasure. =)
> Sorry I missed the point that you broke old PyPI code, now the new
> one. Still, 100% of
> test coverage makes it hard to make changes, sometimes more hard than necessary.
> Perhaps I should take a look at this.

It raises the bar for adding new code, but decreases the bar for maintaing the existing
code. Given the fact that the code will be maintained more than new features will
be written I consider this a positive.

> 
> BTW, why didn't you just copy crate.io?

Crate?s code base suffers from a lot of bad assumptions that are baked into it when
I was still learning how packaging really worked. On top of that most of the pieces
that we need to replace PyPI don?t exist in Crate since Crate just mirrored PyPI. As it
is Warehouse almost is at feature parity with Crate (but not PyPI).

> 
>> Warehouse
>> is (hopefully) easy to iterate on, has what I believe is a well laid out
>> code base, and has an extensive unit test suite.
> 
> Ok. I should take a closer look.
> 
>>> Do you have a diagram of the system what are you trying to build, or
>>> is it just an experiment? It may worth to put IDE aside and try to
>>> de-couple some things first on the whiteboard. I am afraid that I will
>>> never be ready to help with reinventing Django. At least not without
>>> some research and art coverage.
>> 
>> It?s not a reinvention of Django. The ?framework? parts of Warehouse
>> are very small and are mostly glue that holds together pieces like
>> Werkzeug and SQLAlchemy.
> 
> Still, is there a diagram of Warehouse architecture? Blueprint,
> CAD model of a building?

I have a todo list item to write up architecture documentation that explains
the layout and where the different pieces are at.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/cc3150d2/attachment.sig>

From maphew at gmail.com  Sat Nov 16 08:12:42 2013
From: maphew at gmail.com (Matt Wilkie)
Date: Fri, 15 Nov 2013 23:12:42 -0800
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
Message-ID: <CANGB9XQdLy68FU16F2ombWBpgh5nF0M8P8uFp9JW4RT-0C11yg@mail.gmail.com>

On Fri, Nov 15, 2013 at 7:12 PM, Donald Stufft <donald at stufft.io> wrote:

> https://preview-pypi.python.org/pypi



yields "Internal Server Error" at the moment.

-matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131115/1464d4bd/attachment.html>

From richard at python.org  Sat Nov 16 09:29:22 2013
From: richard at python.org (Richard Jones)
Date: Sat, 16 Nov 2013 19:29:22 +1100
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <CANGB9XQdLy68FU16F2ombWBpgh5nF0M8P8uFp9JW4RT-0C11yg@mail.gmail.com>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
 <CANGB9XQdLy68FU16F2ombWBpgh5nF0M8P8uFp9JW4RT-0C11yg@mail.gmail.com>
Message-ID: <CAHrZfZAchjE-gNBzz8P=cCFuXp3TWTyNtxDnh=A0_kBQaj493Q@mail.gmail.com>

Hi Matt,

What method you were calling? What arguments did you pass? What xml-rpc
library you were using?

Actual code would be ideal.


Thanks,

     Richard


On 16 November 2013 18:12, Matt Wilkie <maphew at gmail.com> wrote:

>
> On Fri, Nov 15, 2013 at 7:12 PM, Donald Stufft <donald at stufft.io> wrote:
>
>> https://preview-pypi.python.org/pypi
>
>
>
> yields "Internal Server Error" at the moment.
>
> -matt
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/43dfcef1/attachment.html>

From maphew at gmail.com  Sat Nov 16 09:41:25 2013
From: maphew at gmail.com (Matt Wilkie)
Date: Sat, 16 Nov 2013 00:41:25 -0800
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <CAHrZfZAchjE-gNBzz8P=cCFuXp3TWTyNtxDnh=A0_kBQaj493Q@mail.gmail.com>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
 <CANGB9XQdLy68FU16F2ombWBpgh5nF0M8P8uFp9JW4RT-0C11yg@mail.gmail.com>
 <CAHrZfZAchjE-gNBzz8P=cCFuXp3TWTyNtxDnh=A0_kBQaj493Q@mail.gmail.com>
Message-ID: <CANGB9XTtx=rq4UJA4w9tePX+LdCCZk9YP=ofM9uxR9tfONW-GQ@mail.gmail.com>

sorry, I should have been more explicit: I just clicked on the link in my
browser (FF), no rpc call was made. I expected a "not for browsers" message
or similar rather than an error.

-matt


On Sat, Nov 16, 2013 at 12:29 AM, Richard Jones <richard at python.org> wrote:

> Hi Matt,
>
> What method you were calling? What arguments did you pass? What xml-rpc
> library you were using?
>
> Actual code would be ideal.
>
>
> Thanks,
>
>      Richard
>
>
> On 16 November 2013 18:12, Matt Wilkie <maphew at gmail.com> wrote:
>
>>
>> On Fri, Nov 15, 2013 at 7:12 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>> https://preview-pypi.python.org/pypi
>>
>>
>>
>> yields "Internal Server Error" at the moment.
>>
>> -matt
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/c7c8cd98/attachment-0001.html>

From richard at python.org  Sat Nov 16 09:55:18 2013
From: richard at python.org (Richard Jones)
Date: Sat, 16 Nov 2013 19:55:18 +1100
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <CANGB9XTtx=rq4UJA4w9tePX+LdCCZk9YP=ofM9uxR9tfONW-GQ@mail.gmail.com>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
 <CANGB9XQdLy68FU16F2ombWBpgh5nF0M8P8uFp9JW4RT-0C11yg@mail.gmail.com>
 <CAHrZfZAchjE-gNBzz8P=cCFuXp3TWTyNtxDnh=A0_kBQaj493Q@mail.gmail.com>
 <CANGB9XTtx=rq4UJA4w9tePX+LdCCZk9YP=ofM9uxR9tfONW-GQ@mail.gmail.com>
Message-ID: <CAHrZfZBs2tT8gQVBfzT5F8ztcfcQ9f89za_O0QVmsd3maGwzbQ@mail.gmail.com>

OK, that explains the error we saw. The only thing implemented at /pypi is
the XML-RPC interface. We haven't had time to implement the rest of the
legacy interface yet.


On 16 November 2013 19:41, Matt Wilkie <maphew at gmail.com> wrote:

> sorry, I should have been more explicit: I just clicked on the link in my
> browser (FF), no rpc call was made. I expected a "not for browsers" message
> or similar rather than an error.
>
> -matt
>
>
> On Sat, Nov 16, 2013 at 12:29 AM, Richard Jones <richard at python.org>wrote:
>
>> Hi Matt,
>>
>> What method you were calling? What arguments did you pass? What xml-rpc
>> library you were using?
>>
>> Actual code would be ideal.
>>
>>
>> Thanks,
>>
>>      Richard
>>
>>
>> On 16 November 2013 18:12, Matt Wilkie <maphew at gmail.com> wrote:
>>
>>>
>>> On Fri, Nov 15, 2013 at 7:12 PM, Donald Stufft <donald at stufft.io> wrote:
>>>
>>>> https://preview-pypi.python.org/pypi
>>>
>>>
>>>
>>> yields "Internal Server Error" at the moment.
>>>
>>> -matt
>>>
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/e1c4e620/attachment.html>

From theller at ctypes.org  Sat Nov 16 18:59:17 2013
From: theller at ctypes.org (Thomas Heller)
Date: Sat, 16 Nov 2013 18:59:17 +0100
Subject: [Distutils] Distribution format for Python3 libraries
Message-ID: <l68bku$kh5$1@ger.gmane.org>

What is the preferred format to distribute Python-3 libraries (for 
Windows) nowadays: wininst, egg, ...?

It has been a long time that I have released something.

Thanks,
Thomas


From aclark at aclark.net  Sat Nov 16 20:52:16 2013
From: aclark at aclark.net (Alex Clark)
Date: Sat, 16 Nov 2013 19:52:16 +0000 (UTC)
Subject: [Distutils] Distribution format for Python3 libraries
References: <l68bku$kh5$1@ger.gmane.org>
Message-ID: <loom.20131116T205130-107@post.gmane.org>

Thomas Heller <theller <at> ctypes.org> writes:

> 
> What is the preferred format to distribute Python-3 libraries (for 
> Windows) nowadays: wininst, egg, ...?
> 
> It has been a long time that I have released something.


I don't know about preferred, but for Pillow we create all of them:
https://pypi.python.org/pypi?name=Pillow&version=2.2.1&:action=files




From p.f.moore at gmail.com  Sat Nov 16 21:15:48 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 16 Nov 2013 20:15:48 +0000
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <l68bku$kh5$1@ger.gmane.org>
References: <l68bku$kh5$1@ger.gmane.org>
Message-ID: <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>

On 16 November 2013 17:59, Thomas Heller <theller at ctypes.org> wrote:
> What is the preferred format to distribute Python-3 libraries (for Windows)
> nowadays: wininst, egg, ...?

Wheel is the format of the future.

pip install will only use wheels (recent versions of pip)
wininst doesn't support virtualenv
egg is only supported by easy_install, which is not recommended these
days (use pip instead)
MSI is rarely used, and can't easily be converted to other formats

Personally, I'd recommend wheel and wininst.

Paul

From donald at stufft.io  Sat Nov 16 22:59:23 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 16 Nov 2013 16:59:23 -0500
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
Message-ID: <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>

And sdist!

On Nov 16, 2013, at 3:15 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 16 November 2013 17:59, Thomas Heller <theller at ctypes.org> wrote:
>> What is the preferred format to distribute Python-3 libraries (for Windows)
>> nowadays: wininst, egg, ...?
> 
> Wheel is the format of the future.
> 
> pip install will only use wheels (recent versions of pip)
> wininst doesn't support virtualenv
> egg is only supported by easy_install, which is not recommended these
> days (use pip instead)
> MSI is rarely used, and can't easily be converted to other formats
> 
> Personally, I'd recommend wheel and wininst.
> 
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/b93206b1/attachment.sig>

From p.f.moore at gmail.com  Sat Nov 16 23:06:03 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 16 Nov 2013 22:06:03 +0000
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
Message-ID: <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>

On 16 November 2013 21:59, Donald Stufft <donald at stufft.io> wrote:
> And sdist!

Sorry - I assumed compiled C extensions. You'll always provide sdist,
and unless you're distributing a compiled extension, that's all you'll
need. (Wheel distributions even for pure Python is the intention in
the future, but they aren't needed currently).

Paul

From donald at stufft.io  Sat Nov 16 23:07:06 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 16 Nov 2013 17:07:06 -0500
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
Message-ID: <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>

Wheels are way faster for pure python though, so they should totally be
uploaded :D

On Nov 16, 2013, at 5:06 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 16 November 2013 21:59, Donald Stufft <donald at stufft.io> wrote:
>> And sdist!
> 
> Sorry - I assumed compiled C extensions. You'll always provide sdist,
> and unless you're distributing a compiled extension, that's all you'll
> need. (Wheel distributions even for pure Python is the intention in
> the future, but they aren't needed currently).
> 
> Paul


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/a1ed7d41/attachment.sig>

From ncoghlan at gmail.com  Sun Nov 17 02:58:05 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Nov 2013 11:58:05 +1000
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
Message-ID: <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>

On 17 Nov 2013 08:07, "Donald Stufft" <donald at stufft.io> wrote:
>
> Wheels are way faster for pure python though, so they should totally be
> uploaded :D

In chatting to someone proposing a patch for IPython to start publishing
wheels, I noted that wheels don't yet work even for distributions with
essential logic in their setup.py file, even if it's a pure Python
distribution.

The specific case affecting IPython was the need to define platform
dependent dependencies. Metadata 2.0 will handle that, but isn't going to
be available until pip 1.6 at the earliest. (That does suggest "Does this
let IPython start publishing wheels?" would make a good acceptance
criterion for PEP 426 et al. It can join the setuptools metadata
replacement criteria, the Linux distro integration criteria and the Twisted
plugin model handling criteria).

Cheers,
Nick.

>
> On Nov 16, 2013, at 5:06 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> > On 16 November 2013 21:59, Donald Stufft <donald at stufft.io> wrote:
> >> And sdist!
> >
> > Sorry - I assumed compiled C extensions. You'll always provide sdist,
> > and unless you're distributing a compiled extension, that's all you'll
> > need. (Wheel distributions even for pure Python is the intention in
> > the future, but they aren't needed currently).
> >
> > Paul
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131117/bf9a9381/attachment-0001.html>

From donald at stufft.io  Sun Nov 17 05:05:10 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 16 Nov 2013 23:05:10 -0500
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
Message-ID: <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>


On Nov 16, 2013, at 8:58 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 17 Nov 2013 08:07, "Donald Stufft" <donald at stufft.io> wrote:
> >
> > Wheels are way faster for pure python though, so they should totally be
> > uploaded :D
> 
> In chatting to someone proposing a patch for IPython to start publishing wheels, I noted that wheels don't yet work even for distributions with essential logic in their setup.py file, even if it's a pure Python distribution.
> 
> The specific case affecting IPython was the need to define platform dependent dependencies. Metadata 2.0 will handle that, but isn't going to be available until pip 1.6 at the earliest. (That does suggest "Does this let IPython start publishing wheels?" would make a good acceptance criterion for PEP 426 et al. It can join the setuptools metadata replacement criteria, the Linux distro integration criteria and the Twisted plugin model handling criteria).
> 

That?s not true, Wheel let?s you use environment markers.

https://github.com/dstufft/twine/blob/master/setup.cfg#L9-L13

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/b01cf294/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/b01cf294/attachment.sig>

From tk47 at students.poly.edu  Sun Nov 17 05:22:27 2013
From: tk47 at students.poly.edu (Trishank Karthik Kuppusamy)
Date: Sat, 16 Nov 2013 23:22:27 -0500
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
Message-ID: <52884483.5080803@students.poly.edu>

Hello everyone,

Donald, Justin and I have co-authored a PEP that recommends a
comprehensive security solution to allow PyPI to secure its users
against a wide array of compromises.

The gist of the PEP is that the changes to PyPI are essentially
invisible to users and developers unless an attack is underway.

The key design ideas are as follows:

* The main PyPI server will continue running as it is now, exposing
HTTPS and legacy XML-RPC operations.

* The next-generation PyPI server (Warehouse) will be exposing new API
as well as TUF metadata to clients.

* Developers do not have to opt-in to secure their projects with their
own TUF metadata. In that case, PyPI will sign these "unclaimed"
projects on their behalf. However, unclaimed projects will not be secure
against a PyPI compromise.

* To protect against a PyPI compromise, developers may choose to
register their public keys with Warehouse and upload their own signed
TUF metadata about their projects.

* Therefore, developers do not have to concern themselves with key
management in case they leave their projects as "unclaimed". When they
do claim their projects, they simply have to register their keys once
with Warehouse. After that, they may delegate signing for distributions
as they wish without depending on Warehouse.

* Clients will be instructed to first search for a project in the more
secure claimed metadata (protected by offline keys) before looking for
it in the less secure unclaimed metadata (protected by online keys).

* Whether or not a project is claimed or unclaimed, all projects will be
available through continuous delivery.

* Consistent snapshots allow clients and mirrors to safely read metadata
and data despite the addition of new files to PyPI.

* It is efficient to securely install or update a project despite
hundreds of thousands of files.

The official PEP is here:

http://www.python.org/dev/peps/pep-0458/

Whereas latest revisions to the PEP are here:

https://github.com/theupdateframework/pep-on-pypi-with-tuf

We welcome your feedback and suggestions.

Thanks,
The PEP 458 team

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 901 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131116/c80cee95/attachment.sig>

From marius at pov.lt  Sun Nov 17 11:05:35 2013
From: marius at pov.lt (Marius Gedminas)
Date: Sun, 17 Nov 2013 12:05:35 +0200
Subject: [Distutils] PyPI v2 (Was:  PyPI pull request #7)
In-Reply-To: <54B6FCEB-F7BE-4E45-B18B-0A9D12458154@stufft.io>
References: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>
 <8100BD88-9CDC-4FFB-89A9-0554F89B91CE@stufft.io>
 <CAPkN8x+a3=zF6Yw=Be5ysdWW_k05ogdCBae7Lam0+LMeGys+NA@mail.gmail.com>
 <54B6FCEB-F7BE-4E45-B18B-0A9D12458154@stufft.io>
Message-ID: <20131117100534.GA9459@fridge.pov.lt>

On Sat, Nov 16, 2013 at 12:00:59AM -0500, Donald Stufft wrote:
> As I said, running a local copy of PostgreSQLis very easy. There are
> installers for Windows and OSX, and pretty much every *nix
> distribution will have it packaged.

My ex-coworker Ignas came up with a Makefile snippet that sets up a
local PostgreSQL sandbox inside the working tree:
https://gist.github.com/Ignas/1676353

It's incredibly handy for Jenkins: you can clone jobs as you like
without having to manually assign database names to avoid conflicts from
jobs running simultaneously.  All you have to do is use the Port
Allocator Plugin to set unique PGPORTs for the jobs.

I also like that I can git clone a project and run 'make test' without
having to worry about setting up PostgreSQL databases.

It probably only works on Debian/Ubuntu, and it requires that you have
PostgreSQL installed, but it doesn't need a system-wide PostgreSQL to be
running.

Marius Gedminas
-- 
Those who can't write, write manuals.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131117/d4ec56d2/attachment.sig>

From theller at ctypes.org  Sun Nov 17 12:12:56 2013
From: theller at ctypes.org (Thomas Heller)
Date: Sun, 17 Nov 2013 12:12:56 +0100
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
Message-ID: <l6a870$qni$1@ger.gmane.org>

Am 16.11.2013 21:15, schrieb Paul Moore:
> On 16 November 2013 17:59, Thomas Heller <theller at ctypes.org> wrote:
>> What is the preferred format to distribute Python-3 libraries (for Windows)
>> nowadays: wininst, egg, ...?
>
> Wheel is the format of the future.
>
> pip install will only use wheels (recent versions of pip)
> wininst doesn't support virtualenv
> egg is only supported by easy_install, which is not recommended these
> days (use pip instead)
> MSI is rarely used, and can't easily be converted to other formats
>
> Personally, I'd recommend wheel and wininst.

Ok, I'll probably follow this recommandation (but maybe I'll also
include egg).

Another question:

Is there a way to mark the distribution so that it will work for
Python 3.3 and 3.4, but not 2.x, 3.0, 3.1, or 3.2? There are existing
installers for 2.x (containing compiled extensions) already registered
on pypi.

All this is for py2exe.  Maybe I should name the python3 version 
'py3exe' instead ;-)

Thomas



From ncoghlan at gmail.com  Sun Nov 17 14:44:15 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Nov 2013 23:44:15 +1000
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
 <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
Message-ID: <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>

On 17 November 2013 14:05, Donald Stufft <donald at stufft.io> wrote:
>
> On Nov 16, 2013, at 8:58 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The specific case affecting IPython was the need to define platform
> dependent dependencies. Metadata 2.0 will handle that, but isn't going to
be
> available until pip 1.6 at the earliest. (That does suggest "Does this let
> IPython start publishing wheels?" would make a good acceptance criterion
for
> PEP 426 et al. It can join the setuptools metadata replacement criteria,
the
> Linux distro integration criteria and the Twisted plugin model handling
> criteria).
>
> That?s not true, Wheel let?s you use environment markers.
>
> https://github.com/dstufft/twine/blob/master/setup.cfg#L9-L13

More accurately, it appears setuptools *does* support PEP 345 style
environment markers, it just isn't documented at
http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies*sigh*

I filed a setuptools issue about the missing docs, but also suggested the
IPython folks wait for pip 1.5 to get the install-time script handling and
other wheel related fixes.

Cheers,
Nick.

-- 
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131117/8fe28ebe/attachment.html>

From p.f.moore at gmail.com  Sun Nov 17 17:02:48 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sun, 17 Nov 2013 16:02:48 +0000
Subject: [Distutils] venv and _ensurepip
Message-ID: <CACac1F-i-c1EhLO_TyVB7YofkbsF0R1rcaCpCnjS6xHfqbH-Yw@mail.gmail.com>

I've not seen any documentation and/or code (other than the PEP) yet
for the _ensurepip changes that will be going into Python 3.4, but can
I check what the intention is for people using the extensibility API
in venv?

Specifically, if I want to install something as well as (or other
than) pip/setuptools when I create a venv, will venv.EnvBuilder
support that? Specifically, I'd need to be able to say whether or not
I want _ensurepip to run in the venv. Ideally, I'd also like it if the
temporary pip instance that _ensurepip uses was exposed somehow, so
that I could use it to (for example) install extra packages in my venv
build script. Obviously I could just run pip again after the venv is
created but (1) only if I install pip (I might want a builder that
uses pip to install another package, but I don't want pip in the final
venv) and (2) that's an extra process run, which has a non-trivial
runtime cost on Windows at least.

Obviously I'll be looking at testing this type of thing once the beta
comes out, but I wondered if the scripting API for venv was something
that had been thought about in advance.

Paul

From ncoghlan at gmail.com  Mon Nov 18 00:53:59 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Nov 2013 09:53:59 +1000
Subject: [Distutils] venv and _ensurepip
In-Reply-To: <CACac1F-i-c1EhLO_TyVB7YofkbsF0R1rcaCpCnjS6xHfqbH-Yw@mail.gmail.com>
References: <CACac1F-i-c1EhLO_TyVB7YofkbsF0R1rcaCpCnjS6xHfqbH-Yw@mail.gmail.com>
Message-ID: <CADiSq7chy44STRZyx7KSU8GUzkG2ZiSUg=SFp_J6ft9L_DvPBA@mail.gmail.com>

It's covered in the PEP (including the venv module API changes):
http://www.python.org/dev/peps/pep-0453/#changes-to-virtual-environments

Associated (more concise) issue: http://bugs.python.org/issue19552

I aim to implement that later this week, but wouldn't complain is someone
beat me to a draft patch ;)

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131118/3749baba/attachment.html>

From p.f.moore at gmail.com  Mon Nov 18 08:59:33 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 18 Nov 2013 07:59:33 +0000
Subject: [Distutils] venv and _ensurepip
In-Reply-To: <CADiSq7chy44STRZyx7KSU8GUzkG2ZiSUg=SFp_J6ft9L_DvPBA@mail.gmail.com>
References: <CACac1F-i-c1EhLO_TyVB7YofkbsF0R1rcaCpCnjS6xHfqbH-Yw@mail.gmail.com>
 <CADiSq7chy44STRZyx7KSU8GUzkG2ZiSUg=SFp_J6ft9L_DvPBA@mail.gmail.com>
Message-ID: <CACac1F9k=OGH4yGR8XxepKZr=1nsCv14W183sHZ1VKDx0mmBsw@mail.gmail.com>

On 17 November 2013 23:53, Nick Coghlan <ncoghlan at gmail.com> wrote:
> It's covered in the PEP (including the venv module API changes):
> http://www.python.org/dev/peps/pep-0453/#changes-to-virtual-environments

Thanks (and sorry I missed that). And thanks for your other answer as well.

From ncoghlan at gmail.com  Mon Nov 18 00:58:15 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Nov 2013 09:58:15 +1000
Subject: [Distutils] venv and _ensurepip
In-Reply-To: <CADiSq7chy44STRZyx7KSU8GUzkG2ZiSUg=SFp_J6ft9L_DvPBA@mail.gmail.com>
References: <CACac1F-i-c1EhLO_TyVB7YofkbsF0R1rcaCpCnjS6xHfqbH-Yw@mail.gmail.com>
 <CADiSq7chy44STRZyx7KSU8GUzkG2ZiSUg=SFp_J6ft9L_DvPBA@mail.gmail.com>
Message-ID: <CADiSq7d8YV_8L-PQONmoYUrc1cNKR07pjyGErnAWnWVRf4U1Yw@mail.gmail.com>

To answer your other question, no, you cannot use ensurepip as if it was an
uninstalled pip, and this is a deliberate limitation to control the
complexity and compatibility requirements. If you want something like that,
install pip globally and run it with root set to the base of the venv
(that's similar to what venv itself will likely do internally when invoking
ensurepip).

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131118/109bdddb/attachment.html>

From holger at merlinux.eu  Mon Nov 18 14:57:41 2013
From: holger at merlinux.eu (holger krekel)
Date: Mon, 18 Nov 2013 13:57:41 +0000
Subject: [Distutils] devpi-{server,
	client}-1.2.1: bug fixes + refinement to test
Message-ID: <20131118135741.GE11363@merlinux.eu>

devpi-1.2.1: bug fixes and improved test command
==========================================================

The devpi-{server,client}-1.2.1 releases bring important
bug fixes and refinements.  See Changelog below.

For getting started with deploying your own pypi server
on a laptop or on  company server and for using the 
"devpi" workflow tool (optional), see:

    http://doc.devpi.net

If you want to upgrade an existing installation, you should 
be able to execute::

    $ pip install -U devpi
    $ devpi-server --upgrade-state [--serverdir YOUR_SERVER_DIR]

Have fun,

holger krekel

Changelog 1.2.1
----------------------

devpi-server:

- fix an import issue for doc files which were wrongly tied to a newer
  version of a base index. now version "auto" detection for storing
  doc files only works within a stage.  Thanks Laurent Brack for bringing
  it up and providing the repo.

- fix issue66: api endpoints now also respect --outside-url setting
  so that you can serve devpi from a subpath.  Thanks for Fabian
  Snovna for reporting and analysis.

- fix issue63: skip egg links that go to a directory (this requires
  doing a SVN checkout which devpi-server does not do).  Thanks
  Ken Jung for analyzing the problem.

- fix issue68: don't derive metadata from filename but instead
  look it up in metadata or submitted form.

- fix cache-invalidation when normalized_project_name != real_name
  (e.g. for Django but also many others). addresses issue59.

- add newline to simple list output for better human readability of the
  page (thanks Brandon Maister)

- make xmlrpc calls to pypi's changelog API use "requests" sessions 
  so that http proxies are respected there as well (fixes issue58).
  thanks to riehlm for identifying the problem and testing the fix.

- internally refactor and consolidate mocking against requests library

- --upgrade-state will upgrade now between major.minor/major.minor+1 changes.

devpi-client:

- fix "python -m devpi" invocation. Thanks Sebastian Ralph.

- fix issue66: "devpi use user/index" can now switch between URLs
  if user/index is mounted on a subpath.

- fix issue71: allow pip/setuptools like requirements specs
  with the test subcommand, e.g.  "devpi test 'pkg>=1.0'".
  Thanks Sebastian Rahlf for the PR.



From barry at python.org  Mon Nov 18 17:54:49 2013
From: barry at python.org (Barry Warsaw)
Date: Mon, 18 Nov 2013 11:54:49 -0500
Subject: [Distutils] Warehouse and XMLRPC
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
Message-ID: <20131118115449.27529521@anarchist>

On Nov 15, 2013, at 10:12 PM, Donald Stufft wrote:

>The bulk of XMLRPC has been implemented in Warehouse. It would be great if
>people who are using the XMLRPC could try it out using the
>https://preview-pypi.python.org/pypi url.
>
>Currently the search API does not work.

Are you planning on putting a REST API on Warehouse?

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131118/f9f38c28/attachment.sig>

From donald at stufft.io  Mon Nov 18 18:19:36 2013
From: donald at stufft.io (Donald Stufft)
Date: Mon, 18 Nov 2013 12:19:36 -0500
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <20131118115449.27529521@anarchist>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
 <20131118115449.27529521@anarchist>
Message-ID: <A6B9C0DF-1F87-43D4-8587-5CC3F230F92E@stufft.io>


On Nov 18, 2013, at 11:54 AM, Barry Warsaw <barry at python.org> wrote:

> Are you planning on putting a REST API on Warehouse?

Eventually, but the primary goal right now is feature parity with PyPI legacy.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131118/df933924/attachment.sig>

From chris.barker at noaa.gov  Mon Nov 18 19:17:30 2013
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 18 Nov 2013 10:17:30 -0800
Subject: [Distutils] pip feedback to user...
Message-ID: <CALGmxEL3EAUWPiKiLnicbFkLawzKoa4fyZWaZ7fp-o=u1u_uOQ@mail.gmail.com>

Is this the right place to discuss UX issues for pip? If not, point to the
right place, if so, read on:

I think pip's usability could be much improved with a little tweaking to
the messages it prints to the console when it does its thing. For instance,
when I do a:

pip install some_package

I get the message:

Downloading/unpacking some_package

When in fact, pip is not (yet) doing that -- what it is doing is looking
for the package on pypi (or ???) Which is fine if the package is found, but
if it's not, you then get some confusing messages, like:

Could not find any downloads that satisfy the requirement some-package
Cleaning up...

not to bad -- it does say it wasn't found, but it seems it could be a bit
odd to a new user -- "but I thought it was being downloaded?"

I was just trying to upgrade a packe that I was told could now be pip
installed:

$ pip install --upgrade tracpy
Could not find any downloads that satisfy the requirement tracpy in
/Users/chris.barker/PythonStuff/TracPy/tracpy
Downloading/unpacking tracpy
Cleaning up...
No distributions at all found for tracpy in
/Users/chris.barker/PythonStuff/TracPy/tracpy
Storing complete log in /Users/chris.barker/.pip/pip.log

It turns out that they have not (yet?) registered it with pypi (and I have
a n older copy in /Users/chris.barker/PythonStuff/TracPy/tracpy). But it
does seem weird and confusing that it first says it can't find any
downloads, then tells me its downloading, then cleaning up, then says it
couldn't find anything...


Anyway, small stuff, but these little things make a difference.

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131118/23998d23/attachment.html>

From qwcode at gmail.com  Mon Nov 18 19:56:58 2013
From: qwcode at gmail.com (Marcus Smith)
Date: Mon, 18 Nov 2013 10:56:58 -0800
Subject: [Distutils] pip feedback to user...
In-Reply-To: <CALGmxEL3EAUWPiKiLnicbFkLawzKoa4fyZWaZ7fp-o=u1u_uOQ@mail.gmail.com>
References: <CALGmxEL3EAUWPiKiLnicbFkLawzKoa4fyZWaZ7fp-o=u1u_uOQ@mail.gmail.com>
Message-ID: <CAPYWazpyeXk_fSx8OyzJzzSGvHQsAy1HjEAHjhoOZOFy31nGbw@mail.gmail.com>

I agree with you Chris.  It think it's confusing too.
There is an existing issue for this. https://github.com/pypa/pip/issues/376
Marcus


On Mon, Nov 18, 2013 at 10:17 AM, Chris Barker <chris.barker at noaa.gov>wrote:

> Is this the right place to discuss UX issues for pip? If not, point to the
> right place, if so, read on:
>
> I think pip's usability could be much improved with a little tweaking to
> the messages it prints to the console when it does its thing. For instance,
> when I do a:
>
> pip install some_package
>
> I get the message:
>
> Downloading/unpacking some_package
>
> When in fact, pip is not (yet) doing that -- what it is doing is looking
> for the package on pypi (or ???) Which is fine if the package is found, but
> if it's not, you then get some confusing messages, like:
>
> Could not find any downloads that satisfy the requirement some-package
> Cleaning up...
>
> not to bad -- it does say it wasn't found, but it seems it could be a bit
> odd to a new user -- "but I thought it was being downloaded?"
>
> I was just trying to upgrade a packe that I was told could now be pip
> installed:
>
> $ pip install --upgrade tracpy
> Could not find any downloads that satisfy the requirement tracpy in
> /Users/chris.barker/PythonStuff/TracPy/tracpy
> Downloading/unpacking tracpy
> Cleaning up...
> No distributions at all found for tracpy in
> /Users/chris.barker/PythonStuff/TracPy/tracpy
> Storing complete log in /Users/chris.barker/.pip/pip.log
>
> It turns out that they have not (yet?) registered it with pypi (and I have
> a n older copy in /Users/chris.barker/PythonStuff/TracPy/tracpy). But it
> does seem weird and confusing that it first says it can't find any
> downloads, then tells me its downloading, then cleaning up, then says it
> couldn't find anything...
>
>
> Anyway, small stuff, but these little things make a difference.
>
> -Chris
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131118/d305614d/attachment-0001.html>

From ncoghlan at gmail.com  Mon Nov 18 23:43:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Nov 2013 08:43:53 +1000
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <A6B9C0DF-1F87-43D4-8587-5CC3F230F92E@stufft.io>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
 <20131118115449.27529521@anarchist>
 <A6B9C0DF-1F87-43D4-8587-5CC3F230F92E@stufft.io>
Message-ID: <CADiSq7fvgFY_hsOhYyNjY4t+Uo4fBgmBJyt=xivFY6fbhD=nDQ@mail.gmail.com>

On 19 Nov 2013 03:20, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Nov 18, 2013, at 11:54 AM, Barry Warsaw <barry at python.org> wrote:
>
> > Are you planning on putting a REST API on Warehouse?
>
> Eventually, but the primary goal right now is feature parity with PyPI
legacy.

Some details of the REST API will also depend on the final shape of PEP
426, so that's the other blocker on designing that side of things.

Cheers,
Nick.

>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131119/e520e923/attachment.html>

From pje at telecommunity.com  Tue Nov 19 06:09:43 2013
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 19 Nov 2013 00:09:43 -0500
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
 <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
 <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>
Message-ID: <CALeMXf7fMGvckb-W04HTNV47R4NgdRhnddxaFi=LQzXm7Tk5qg@mail.gmail.com>

On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> More accurately, it appears setuptools *does* support PEP 345 style
> environment markers, it just isn't documented at
> http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies
> *sigh*

That's because it's technically an experimental feature I hacked into
0.6c12 development shortly before the distribute merge in order to
handle setuptools' own requirements for SSL, so that setuptools could
ship just one platform-independent egg and still have
platform-specific dependencies.  ;-)

I think Jason et al may have since upgraded it to a supported feature,
but last I looked I think there may have still been issues with Jython
support for the feature.

So, the lack of documentation may still be a feature rather than a bug ATM.  ;-)

From marius at pov.lt  Tue Nov 19 08:44:52 2013
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 19 Nov 2013 09:44:52 +0200
Subject: [Distutils] Warehouse and XMLRPC
In-Reply-To: <A6B9C0DF-1F87-43D4-8587-5CC3F230F92E@stufft.io>
References: <3EAC4D69-2775-47F1-B096-B13EE3DD2CA2@stufft.io>
 <20131118115449.27529521@anarchist>
 <A6B9C0DF-1F87-43D4-8587-5CC3F230F92E@stufft.io>
Message-ID: <20131119074452.GA746@fridge.pov.lt>

On Mon, Nov 18, 2013 at 12:19:36PM -0500, Donald Stufft wrote:
> 
> On Nov 18, 2013, at 11:54 AM, Barry Warsaw <barry at python.org> wrote:
> 
> > Are you planning on putting a REST API on Warehouse?
> 
> Eventually, but the primary goal right now is feature parity with PyPI legacy.

JSON API is part of the PyPI legacy: http://wiki.python.org/moin/PyPiJson

I use it to build http://zope3.pov.lt/py3/ with
https://github.com/mgedmin/ztk-py3-status/blob/master/get_pypi_status.py

(I tried pointing that script to preview-pypi.python.org, but all the
package/json pages returned "{}", which, I guess, simply means the JSON
API is not implemented yet.)

I don't know whether "REST API" is the same thing, or if we're talking
about two different things here.

Marius Gedminas
-- 
Voodoo Programming:  Things programmers do that they know shouldn't work but
they try anyway, and which sometimes actually work, such as recompiling
everything.
-- Karl Lehenbauer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131119/5858bc63/attachment.sig>

From techtonik at gmail.com  Tue Nov 19 09:18:31 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 19 Nov 2013 11:18:31 +0300
Subject: [Distutils] sdist README.rst without MANIFEST
Message-ID: <CAPkN8xJz-Bpvn6RoQGCoSTLktrpNP6q9b+T+9bK_Yqc34gapXg@mail.gmail.com>

I use this in setup.py:
    ..
    long_description = open('README.rst', 'rb').read(),
    ..

Which fails during installation, because README.rst is not shipped.
The usual way to include is through MANIFEST.in

    http://docs.python.org/2/distutils/sourcedist.html

Is it possible to include README.rst with source distribution, but
without creating yet another file? Is it possible to specify includes
in setup.py?
--
anatoly t.

From regebro at gmail.com  Tue Nov 19 10:15:29 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 19 Nov 2013 10:15:29 +0100
Subject: [Distutils] sdist README.rst without MANIFEST
In-Reply-To: <CAPkN8xJz-Bpvn6RoQGCoSTLktrpNP6q9b+T+9bK_Yqc34gapXg@mail.gmail.com>
References: <CAPkN8xJz-Bpvn6RoQGCoSTLktrpNP6q9b+T+9bK_Yqc34gapXg@mail.gmail.com>
Message-ID: <CAL0kPAWdZTuxHD8OwDSDArg6q2digyBHEFkSjPb-24PtnJD5Yw@mail.gmail.com>

For plain distutils, I don't think so. Setuptools will include it if
it's included in the version control, but it seems that consensus is
moving towards always having a MANIFEST.in.

check-manifest is a useful tool here:

https://pypi.python.org/pypi/check-manifest/0.17

It integrates nicely with zest.releaser as well.
https://pypi.python.org/pypi/zest.releaser


On Tue, Nov 19, 2013 at 9:18 AM, anatoly techtonik <techtonik at gmail.com> wrote:
> I use this in setup.py:
>     ..
>     long_description = open('README.rst', 'rb').read(),
>     ..
>
> Which fails during installation, because README.rst is not shipped.
> The usual way to include is through MANIFEST.in
>
>     http://docs.python.org/2/distutils/sourcedist.html
>
> Is it possible to include README.rst with source distribution, but
> without creating yet another file? Is it possible to specify includes
> in setup.py?
> --
> anatoly t.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

From ncoghlan at gmail.com  Tue Nov 19 12:22:13 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Nov 2013 21:22:13 +1000
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CALeMXf7fMGvckb-W04HTNV47R4NgdRhnddxaFi=LQzXm7Tk5qg@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
 <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
 <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>
 <CALeMXf7fMGvckb-W04HTNV47R4NgdRhnddxaFi=LQzXm7Tk5qg@mail.gmail.com>
Message-ID: <CADiSq7dS2Bamec+emTZR=0jO-zfYf5TKQ02aFrohWA6nTgnGwA@mail.gmail.com>

On 19 November 2013 15:09, PJ Eby <pje at telecommunity.com> wrote:
> On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> More accurately, it appears setuptools *does* support PEP 345 style
>> environment markers, it just isn't documented at
>> http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies
>> *sigh*
>
> That's because it's technically an experimental feature I hacked into
> 0.6c12 development shortly before the distribute merge in order to
> handle setuptools' own requirements for SSL, so that setuptools could
> ship just one platform-independent egg and still have
> platform-specific dependencies.  ;-)
>
> I think Jason et al may have since upgraded it to a supported feature,
> but last I looked I think there may have still been issues with Jython
> support for the feature.
>
> So, the lack of documentation may still be a feature rather than a bug ATM.  ;-)

A fair point :)

Regardless, it highlights the fact I need to ensure PEP 426 preserves
the legacy spellings for the markers that have dots in their names in
metadata 1.2 (https://bitbucket.org/pypa/pypi-metadata-formats/issue/12).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From techtonik at gmail.com  Tue Nov 19 16:15:06 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 19 Nov 2013 18:15:06 +0300
Subject: [Distutils] sdist README.rst without MANIFEST
In-Reply-To: <CAL0kPAWdZTuxHD8OwDSDArg6q2digyBHEFkSjPb-24PtnJD5Yw@mail.gmail.com>
References: <CAPkN8xJz-Bpvn6RoQGCoSTLktrpNP6q9b+T+9bK_Yqc34gapXg@mail.gmail.com>
 <CAL0kPAWdZTuxHD8OwDSDArg6q2digyBHEFkSjPb-24PtnJD5Yw@mail.gmail.com>
Message-ID: <CAPkN8xKu4ZAkrm5OrFqepQ4G9YCMAuponyO7Cwx+7ezbcsKjLg@mail.gmail.com>

On Tue, Nov 19, 2013 at 12:15 PM, Lennart Regebro <regebro at gmail.com> wrote:
> For plain distutils, I don't think so. Setuptools will include it if
> it's included in the version control, but it seems that consensus is
> moving towards always having a MANIFEST.in.

Was there a discussion of moving MANIFEST.in into setup.py?

From regebro at gmail.com  Tue Nov 19 16:17:02 2013
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 19 Nov 2013 16:17:02 +0100
Subject: [Distutils] sdist README.rst without MANIFEST
In-Reply-To: <CAPkN8xKu4ZAkrm5OrFqepQ4G9YCMAuponyO7Cwx+7ezbcsKjLg@mail.gmail.com>
References: <CAPkN8xJz-Bpvn6RoQGCoSTLktrpNP6q9b+T+9bK_Yqc34gapXg@mail.gmail.com>
 <CAL0kPAWdZTuxHD8OwDSDArg6q2digyBHEFkSjPb-24PtnJD5Yw@mail.gmail.com>
 <CAPkN8xKu4ZAkrm5OrFqepQ4G9YCMAuponyO7Cwx+7ezbcsKjLg@mail.gmail.com>
Message-ID: <CAL0kPAXUkPX1zYEWtSMhoMQrFS6hWUNHRJu5Bi9H+U9XW-Tt5w@mail.gmail.com>

Possibly, but not that I remember. Moving it into setup.cfg, perhaps.

On Tue, Nov 19, 2013 at 4:15 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Tue, Nov 19, 2013 at 12:15 PM, Lennart Regebro <regebro at gmail.com> wrote:
>> For plain distutils, I don't think so. Setuptools will include it if
>> it's included in the version control, but it seems that consensus is
>> moving towards always having a MANIFEST.in.
>
> Was there a discussion of moving MANIFEST.in into setup.py?

From pje at telecommunity.com  Tue Nov 19 18:33:32 2013
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 19 Nov 2013 12:33:32 -0500
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CADiSq7dS2Bamec+emTZR=0jO-zfYf5TKQ02aFrohWA6nTgnGwA@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
 <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
 <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>
 <CALeMXf7fMGvckb-W04HTNV47R4NgdRhnddxaFi=LQzXm7Tk5qg@mail.gmail.com>
 <CADiSq7dS2Bamec+emTZR=0jO-zfYf5TKQ02aFrohWA6nTgnGwA@mail.gmail.com>
Message-ID: <CALeMXf7iM3WFFnW4-b8ynrOLWjBD+s70MGkfzEEK21qiZ_kV0A@mail.gmail.com>

On Tue, Nov 19, 2013 at 6:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 19 November 2013 15:09, PJ Eby <pje at telecommunity.com> wrote:
>> On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> More accurately, it appears setuptools *does* support PEP 345 style
>>> environment markers, it just isn't documented at
>>> http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies
>>> *sigh*
>>
>> That's because it's technically an experimental feature I hacked into
>> 0.6c12 development shortly before the distribute merge in order to
>> handle setuptools' own requirements for SSL, so that setuptools could
>> ship just one platform-independent egg and still have
>> platform-specific dependencies.  ;-)
>>
>> I think Jason et al may have since upgraded it to a supported feature,
>> but last I looked I think there may have still been issues with Jython
>> support for the feature.
>>
>> So, the lack of documentation may still be a feature rather than a bug ATM.  ;-)
>
> A fair point :)
>
> Regardless, it highlights the fact I need to ensure PEP 426 preserves
> the legacy spellings for the markers that have dots in their names in
> metadata 1.2 (https://bitbucket.org/pypa/pypi-metadata-formats/issue/12).

Eh?  I implemented PEP 426 (draft-at-the-time) markers in the feature
I mentioned above.

Hm.  Actually, looking over it now, I see the real reason it's
experimental: it was implemented based on consensus-in-progress for
PEP 426, so not a finalized PEP.  It also sort-of handles PEP 345.
And the Jython fallback, IIRC, was maybe only doing PEP 345.  (And
setuptools itself only needed the Python version and platform name.)

So it's actually a bit of a mess -- which is why it wasn't documented,
in order to avoid people forcing the new PEP's hand with another de
facto standard.  I just figured it would get us most of the way there
and when the PEP dust settled, the tests and code could be tweaked a
bit to match, then documented with reference to the PEP.

From kura at kura.io  Tue Nov 19 19:56:07 2013
From: kura at kura.io (Kura)
Date: Tue, 19 Nov 2013 18:56:07 +0000
Subject: [Distutils] How setuptools reads requirements
Message-ID: <ad55a65a-06be-4a1f-b7a0-c5c4fca9179c@getmailbird.com>

Hey guys

I'm trying to dig in to how setuptools/distutils reads the [install_]requires keyword and how requirements.txt is handled.

I've tried running setup.py --requires but get an empty response back, this also fails when using >=x.x,<=x.y and causes a Python stack trace.

Was wondering if someone on here could shed some light on the topic for me.

It's for a Python dependency management/monitoring system I am writing so hopefully would benefit the overall community.


--
Kura

t: @kuramanga [https://twitter.com/kuramanga]
w:?https://kura.io [https://kura.io]
k:?https://kura.io/static/files/kura.asc [https://kura.io/static/files/kura.asc]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131119/bc28be9b/attachment.html>

From ncoghlan at gmail.com  Tue Nov 19 23:24:30 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 20 Nov 2013 08:24:30 +1000
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CALeMXf7iM3WFFnW4-b8ynrOLWjBD+s70MGkfzEEK21qiZ_kV0A@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
 <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
 <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>
 <CALeMXf7fMGvckb-W04HTNV47R4NgdRhnddxaFi=LQzXm7Tk5qg@mail.gmail.com>
 <CADiSq7dS2Bamec+emTZR=0jO-zfYf5TKQ02aFrohWA6nTgnGwA@mail.gmail.com>
 <CALeMXf7iM3WFFnW4-b8ynrOLWjBD+s70MGkfzEEK21qiZ_kV0A@mail.gmail.com>
Message-ID: <CADiSq7dU8yS4z1LkT1J-dC21nUie6u=VxNbF75yafU7o0whz2Q@mail.gmail.com>

On 20 Nov 2013 03:33, "PJ Eby" <pje at telecommunity.com> wrote:
>
> On Tue, Nov 19, 2013 at 6:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 19 November 2013 15:09, PJ Eby <pje at telecommunity.com> wrote:
> >> On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan <ncoghlan at gmail.com>
wrote:
> >>> More accurately, it appears setuptools *does* support PEP 345 style
> >>> environment markers, it just isn't documented at
> >>>
http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies
> >>> *sigh*
> >>
> >> That's because it's technically an experimental feature I hacked into
> >> 0.6c12 development shortly before the distribute merge in order to
> >> handle setuptools' own requirements for SSL, so that setuptools could
> >> ship just one platform-independent egg and still have
> >> platform-specific dependencies.  ;-)
> >>
> >> I think Jason et al may have since upgraded it to a supported feature,
> >> but last I looked I think there may have still been issues with Jython
> >> support for the feature.
> >>
> >> So, the lack of documentation may still be a feature rather than a bug
ATM.  ;-)
> >
> > A fair point :)
> >
> > Regardless, it highlights the fact I need to ensure PEP 426 preserves
> > the legacy spellings for the markers that have dots in their names in
> > metadata 1.2 (https://bitbucket.org/pypa/pypi-metadata-formats/issue/12
).
>
> Eh?  I implemented PEP 426 (draft-at-the-time) markers in the feature
> I mentioned above.
>
> Hm.  Actually, looking over it now, I see the real reason it's
> experimental: it was implemented based on consensus-in-progress for
> PEP 426, so not a finalized PEP.  It also sort-of handles PEP 345.
> And the Jython fallback, IIRC, was maybe only doing PEP 345.  (And
> setuptools itself only needed the Python version and platform name.)
>
> So it's actually a bit of a mess -- which is why it wasn't documented,
> in order to avoid people forcing the new PEP's hand with another de
> facto standard.  I just figured it would get us most of the way there
> and when the PEP dust settled, the tests and code could be tweaked a
> bit to match, then documented with reference to the PEP.

Ah, that definitely makes sense - and then I went and postponed further
work on metadata 2.0 until after the Python 3.4 feature freeze once I
realised how far we could get with metadata 1.1 + the setuptools metadata
files.

So it turns out my *first* answer to the IPython folks was the most
accurate: technically, we need PEP 426 finalised before the static metadata
can handle their requirements.

Perhaps it would be worth breaking the environment markers out as their own
PEP?

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131120/cb9a9d2e/attachment-0001.html>

From pje at telecommunity.com  Wed Nov 20 00:21:51 2013
From: pje at telecommunity.com (PJ Eby)
Date: Tue, 19 Nov 2013 18:21:51 -0500
Subject: [Distutils] Distribution format for Python3 libraries
In-Reply-To: <CADiSq7dU8yS4z1LkT1J-dC21nUie6u=VxNbF75yafU7o0whz2Q@mail.gmail.com>
References: <l68bku$kh5$1@ger.gmane.org>
 <CACac1F9__HnRSL6X6+oVeQoDA3_1o0+s7Mx2RsA+M7HEQFTzKQ@mail.gmail.com>
 <241CC67B-27D4-4F3A-912B-657A70971CD5@stufft.io>
 <CACac1F9Q6+zKepnkNSLqEgf9DCtEskVSMA_o8L1dzoKycxXNZQ@mail.gmail.com>
 <EF7B3DE4-0DC3-4B8D-878B-9806144AA445@stufft.io>
 <CADiSq7f+oi4RchzCWYu4PYHNLzbHc4LP_z_9b=Dgi77a67k6hw@mail.gmail.com>
 <AADB2E6E-9882-4B18-98B3-B212E2DAE2FF@stufft.io>
 <CADiSq7cdBCPEzemeEQigEThFmkyD6LsACKn4LSmS=nZ0TQ1vPw@mail.gmail.com>
 <CALeMXf7fMGvckb-W04HTNV47R4NgdRhnddxaFi=LQzXm7Tk5qg@mail.gmail.com>
 <CADiSq7dS2Bamec+emTZR=0jO-zfYf5TKQ02aFrohWA6nTgnGwA@mail.gmail.com>
 <CALeMXf7iM3WFFnW4-b8ynrOLWjBD+s70MGkfzEEK21qiZ_kV0A@mail.gmail.com>
 <CADiSq7dU8yS4z1LkT1J-dC21nUie6u=VxNbF75yafU7o0whz2Q@mail.gmail.com>
Message-ID: <CALeMXf7LBiq_K8HgcRZS43wVL66UizLKCG-qav1S0WEi24895A@mail.gmail.com>

On Tue, Nov 19, 2013 at 5:24 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Perhaps it would be worth breaking the environment markers out as their own
> PEP?

As I said back then, I think *everything* in PEP 426 is worth breaking
out into its own PEP.  ;-)

Less jokingly, I think that the scope of PEP 426 includes many things
that have different (if overlapping) audiences, and splitting it into
smaller pieces will help those audiences concentrate on the bits
relevant to them.

The audience for markers is a subset of the audience for requirement
specs, which is a subset of the audience for version syntax.  Lots of
other subtopics are of interest mainly to build tool and distro
makers, so there's little point in making everybody read everything.
(Also, a lot of these smaller things are (hopefully) easier to reach
consensus on, at which point the PEPs are settled and allow moving
discussion on to the more open issues.)

So, yes, good idea to break out environment markers.  ;-)

From jcappos at nyu.edu  Thu Nov 21 20:56:34 2013
From: jcappos at nyu.edu (Justin Cappos)
Date: Thu, 21 Nov 2013 14:56:34 -0500
Subject: [Distutils] [tuf] PEP 458: Surviving a Compromise of PyPI:
	Round 1
In-Reply-To: <52884483.5080803@students.poly.edu>
References: <52884483.5080803@students.poly.edu>
Message-ID: <CAMVss_rCyF0owciTOSC=GqkzXjthwKELko5+_fKcpfU=A9X4tQ@mail.gmail.com>

I wanted to let you guys know that the RubyGEMS guys are doing a
hack-a-thon this week to integrate TUF into gems.   It sounds like there
will be some press coverage of this effort and TUF in general.   Our media
department is likely to make a big deal out of this so it may get blown up
beyond just the tech media.

Is there something we can do to help to move integration along with
Warehouse?   For example, we have just published some developer tools that
should integrate smoothly into Warehouse.   Would you like us to submit a
pull request with proposed changes?

We'd love to be able to announce protecting both languages at the same time.

Justin
P.S.   No worries if there are other priorities right now.   We're content
to help with integration on whatever timeframe you prefer.


On Sat, Nov 16, 2013 at 11:22 PM, Trishank Karthik Kuppusamy <
tk47 at students.poly.edu> wrote:

> Hello everyone,
>
> Donald, Justin and I have co-authored a PEP that recommends a
> comprehensive security solution to allow PyPI to secure its users
> against a wide array of compromises.
>
> The gist of the PEP is that the changes to PyPI are essentially
> invisible to users and developers unless an attack is underway.
>
> The key design ideas are as follows:
>
> * The main PyPI server will continue running as it is now, exposing
> HTTPS and legacy XML-RPC operations.
>
> * The next-generation PyPI server (Warehouse) will be exposing new API
> as well as TUF metadata to clients.
>
> * Developers do not have to opt-in to secure their projects with their
> own TUF metadata. In that case, PyPI will sign these "unclaimed"
> projects on their behalf. However, unclaimed projects will not be secure
> against a PyPI compromise.
>
> * To protect against a PyPI compromise, developers may choose to
> register their public keys with Warehouse and upload their own signed
> TUF metadata about their projects.
>
> * Therefore, developers do not have to concern themselves with key
> management in case they leave their projects as "unclaimed". When they
> do claim their projects, they simply have to register their keys once
> with Warehouse. After that, they may delegate signing for distributions
> as they wish without depending on Warehouse.
>
> * Clients will be instructed to first search for a project in the more
> secure claimed metadata (protected by offline keys) before looking for
> it in the less secure unclaimed metadata (protected by online keys).
>
> * Whether or not a project is claimed or unclaimed, all projects will be
> available through continuous delivery.
>
> * Consistent snapshots allow clients and mirrors to safely read metadata
> and data despite the addition of new files to PyPI.
>
> * It is efficient to securely install or update a project despite
> hundreds of thousands of files.
>
> The official PEP is here:
>
> http://www.python.org/dev/peps/pep-0458/
>
> Whereas latest revisions to the PEP are here:
>
> https://github.com/theupdateframework/pep-on-pypi-with-tuf
>
> We welcome your feedback and suggestions.
>
> Thanks,
> The PEP 458 team
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131121/ccbd487c/attachment.html>

From donald at stufft.io  Fri Nov 22 04:06:40 2013
From: donald at stufft.io (Donald Stufft)
Date: Thu, 21 Nov 2013 22:06:40 -0500
Subject: [Distutils] [tuf] PEP 458: Surviving a Compromise of PyPI:
	Round 1
In-Reply-To: <CAMVss_rCyF0owciTOSC=GqkzXjthwKELko5+_fKcpfU=A9X4tQ@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAMVss_rCyF0owciTOSC=GqkzXjthwKELko5+_fKcpfU=A9X4tQ@mail.gmail.com>
Message-ID: <C6BBB02D-63D8-434C-8F69-A00626D60EB4@stufft.io>

Right now Warehouse is primarily focused on feature parity, and I haven?t had time
to re-read the PEP to see what it looks like at this time :(

On Nov 21, 2013, at 2:56 PM, Justin Cappos <jcappos at nyu.edu> wrote:

> I wanted to let you guys know that the RubyGEMS guys are doing a hack-a-thon this week to integrate TUF into gems.   It sounds like there will be some press coverage of this effort and TUF in general.   Our media department is likely to make a big deal out of this so it may get blown up beyond just the tech media.
> 
> Is there something we can do to help to move integration along with Warehouse?   For example, we have just published some developer tools that should integrate smoothly into Warehouse.   Would you like us to submit a pull request with proposed changes?
> 
> We'd love to be able to announce protecting both languages at the same time.
> 
> Justin
> P.S.   No worries if there are other priorities right now.   We're content to help with integration on whatever timeframe you prefer.
> 
> 
> On Sat, Nov 16, 2013 at 11:22 PM, Trishank Karthik Kuppusamy <tk47 at students.poly.edu> wrote:
> Hello everyone,
> 
> Donald, Justin and I have co-authored a PEP that recommends a
> comprehensive security solution to allow PyPI to secure its users
> against a wide array of compromises.
> 
> The gist of the PEP is that the changes to PyPI are essentially
> invisible to users and developers unless an attack is underway.
> 
> The key design ideas are as follows:
> 
> * The main PyPI server will continue running as it is now, exposing
> HTTPS and legacy XML-RPC operations.
> 
> * The next-generation PyPI server (Warehouse) will be exposing new API
> as well as TUF metadata to clients.
> 
> * Developers do not have to opt-in to secure their projects with their
> own TUF metadata. In that case, PyPI will sign these "unclaimed"
> projects on their behalf. However, unclaimed projects will not be secure
> against a PyPI compromise.
> 
> * To protect against a PyPI compromise, developers may choose to
> register their public keys with Warehouse and upload their own signed
> TUF metadata about their projects.
> 
> * Therefore, developers do not have to concern themselves with key
> management in case they leave their projects as "unclaimed". When they
> do claim their projects, they simply have to register their keys once
> with Warehouse. After that, they may delegate signing for distributions
> as they wish without depending on Warehouse.
> 
> * Clients will be instructed to first search for a project in the more
> secure claimed metadata (protected by offline keys) before looking for
> it in the less secure unclaimed metadata (protected by online keys).
> 
> * Whether or not a project is claimed or unclaimed, all projects will be
> available through continuous delivery.
> 
> * Consistent snapshots allow clients and mirrors to safely read metadata
> and data despite the addition of new files to PyPI.
> 
> * It is efficient to securely install or update a project despite
> hundreds of thousands of files.
> 
> The official PEP is here:
> 
> http://www.python.org/dev/peps/pep-0458/
> 
> Whereas latest revisions to the PEP are here:
> 
> https://github.com/theupdateframework/pep-on-pypi-with-tuf
> 
> We welcome your feedback and suggestions.
> 
> Thanks,
> The PEP 458 team
> 
> 


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131121/1aae7a56/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131121/1aae7a56/attachment.sig>

From bascule at gmail.com  Fri Nov 22 04:05:06 2013
From: bascule at gmail.com (Tony Arcieri)
Date: Thu, 21 Nov 2013 19:05:06 -0800
Subject: [Distutils] [tuf] PEP 458: Surviving a Compromise of PyPI:
	Round 1
In-Reply-To: <CAMVss_rCyF0owciTOSC=GqkzXjthwKELko5+_fKcpfU=A9X4tQ@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAMVss_rCyF0owciTOSC=GqkzXjthwKELko5+_fKcpfU=A9X4tQ@mail.gmail.com>
Message-ID: <CAHOTMVL+RPsB=Y6xoSG73g4+xbPhFMCrRh5hqxcQ1J4jChH6GQ@mail.gmail.com>

Hi there! We'll probably publicize our work on our blog The Corner:

http://corner.squareup.com/


On Thu, Nov 21, 2013 at 11:56 AM, Justin Cappos <jcappos at nyu.edu> wrote:

> I wanted to let you guys know that the RubyGEMS guys are doing a
> hack-a-thon this week to integrate TUF into gems.   It sounds like there
> will be some press coverage of this effort and TUF in general.   Our media
> department is likely to make a big deal out of this so it may get blown up
> beyond just the tech media.
>
> Is there something we can do to help to move integration along with
> Warehouse?   For example, we have just published some developer tools that
> should integrate smoothly into Warehouse.   Would you like us to submit a
> pull request with proposed changes?
>
> We'd love to be able to announce protecting both languages at the same
> time.
>
> Justin
> P.S.   No worries if there are other priorities right now.   We're content
> to help with integration on whatever timeframe you prefer.
>
>
> On Sat, Nov 16, 2013 at 11:22 PM, Trishank Karthik Kuppusamy <
> tk47 at students.poly.edu> wrote:
>
>> Hello everyone,
>>
>> Donald, Justin and I have co-authored a PEP that recommends a
>> comprehensive security solution to allow PyPI to secure its users
>> against a wide array of compromises.
>>
>> The gist of the PEP is that the changes to PyPI are essentially
>> invisible to users and developers unless an attack is underway.
>>
>> The key design ideas are as follows:
>>
>> * The main PyPI server will continue running as it is now, exposing
>> HTTPS and legacy XML-RPC operations.
>>
>> * The next-generation PyPI server (Warehouse) will be exposing new API
>> as well as TUF metadata to clients.
>>
>> * Developers do not have to opt-in to secure their projects with their
>> own TUF metadata. In that case, PyPI will sign these "unclaimed"
>> projects on their behalf. However, unclaimed projects will not be secure
>> against a PyPI compromise.
>>
>> * To protect against a PyPI compromise, developers may choose to
>> register their public keys with Warehouse and upload their own signed
>> TUF metadata about their projects.
>>
>> * Therefore, developers do not have to concern themselves with key
>> management in case they leave their projects as "unclaimed". When they
>> do claim their projects, they simply have to register their keys once
>> with Warehouse. After that, they may delegate signing for distributions
>> as they wish without depending on Warehouse.
>>
>> * Clients will be instructed to first search for a project in the more
>> secure claimed metadata (protected by offline keys) before looking for
>> it in the less secure unclaimed metadata (protected by online keys).
>>
>> * Whether or not a project is claimed or unclaimed, all projects will be
>> available through continuous delivery.
>>
>> * Consistent snapshots allow clients and mirrors to safely read metadata
>> and data despite the addition of new files to PyPI.
>>
>> * It is efficient to securely install or update a project despite
>> hundreds of thousands of files.
>>
>> The official PEP is here:
>>
>> http://www.python.org/dev/peps/pep-0458/
>>
>> Whereas latest revisions to the PEP are here:
>>
>> https://github.com/theupdateframework/pep-on-pypi-with-tuf
>>
>> We welcome your feedback and suggestions.
>>
>> Thanks,
>> The PEP 458 team
>>
>>
>


-- 
Tony Arcieri
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131121/d13f0c7d/attachment-0001.html>

From jcappos at nyu.edu  Fri Nov 22 04:39:52 2013
From: jcappos at nyu.edu (Justin Cappos)
Date: Thu, 21 Nov 2013 22:39:52 -0500
Subject: [Distutils] [tuf] PEP 458: Surviving a Compromise of PyPI:
	Round 1
In-Reply-To: <C6BBB02D-63D8-434C-8F69-A00626D60EB4@stufft.io>
References: <52884483.5080803@students.poly.edu>
 <CAMVss_rCyF0owciTOSC=GqkzXjthwKELko5+_fKcpfU=A9X4tQ@mail.gmail.com>
 <C6BBB02D-63D8-434C-8F69-A00626D60EB4@stufft.io>
Message-ID: <CAMVss_pBha6=DpS-Y+xkmR5HOoY78rPi29VTsM_Jc_ULgxt53Q@mail.gmail.com>

Okay, no worries.   We just wanted to let everyone know where things are at.

Justin


On Thu, Nov 21, 2013 at 10:06 PM, Donald Stufft <donald at stufft.io> wrote:

> Right now Warehouse is primarily focused on feature parity, and I haven?t
> had time
> to re-read the PEP to see what it looks like at this time :(
>
> On Nov 21, 2013, at 2:56 PM, Justin Cappos <jcappos at nyu.edu> wrote:
>
> I wanted to let you guys know that the RubyGEMS guys are doing a
> hack-a-thon this week to integrate TUF into gems.   It sounds like there
> will be some press coverage of this effort and TUF in general.   Our media
> department is likely to make a big deal out of this so it may get blown up
> beyond just the tech media.
>
> Is there something we can do to help to move integration along with
> Warehouse?   For example, we have just published some developer tools that
> should integrate smoothly into Warehouse.   Would you like us to submit a
> pull request with proposed changes?
>
> We'd love to be able to announce protecting both languages at the same
> time.
>
> Justin
> P.S.   No worries if there are other priorities right now.   We're content
> to help with integration on whatever timeframe you prefer.
>
>
> On Sat, Nov 16, 2013 at 11:22 PM, Trishank Karthik Kuppusamy <
> tk47 at students.poly.edu> wrote:
>
>> Hello everyone,
>>
>> Donald, Justin and I have co-authored a PEP that recommends a
>> comprehensive security solution to allow PyPI to secure its users
>> against a wide array of compromises.
>>
>> The gist of the PEP is that the changes to PyPI are essentially
>> invisible to users and developers unless an attack is underway.
>>
>> The key design ideas are as follows:
>>
>> * The main PyPI server will continue running as it is now, exposing
>> HTTPS and legacy XML-RPC operations.
>>
>> * The next-generation PyPI server (Warehouse) will be exposing new API
>> as well as TUF metadata to clients.
>>
>> * Developers do not have to opt-in to secure their projects with their
>> own TUF metadata. In that case, PyPI will sign these "unclaimed"
>> projects on their behalf. However, unclaimed projects will not be secure
>> against a PyPI compromise.
>>
>> * To protect against a PyPI compromise, developers may choose to
>> register their public keys with Warehouse and upload their own signed
>> TUF metadata about their projects.
>>
>> * Therefore, developers do not have to concern themselves with key
>> management in case they leave their projects as "unclaimed". When they
>> do claim their projects, they simply have to register their keys once
>> with Warehouse. After that, they may delegate signing for distributions
>> as they wish without depending on Warehouse.
>>
>> * Clients will be instructed to first search for a project in the more
>> secure claimed metadata (protected by offline keys) before looking for
>> it in the less secure unclaimed metadata (protected by online keys).
>>
>> * Whether or not a project is claimed or unclaimed, all projects will be
>> available through continuous delivery.
>>
>> * Consistent snapshots allow clients and mirrors to safely read metadata
>> and data despite the addition of new files to PyPI.
>>
>> * It is efficient to securely install or update a project despite
>> hundreds of thousands of files.
>>
>> The official PEP is here:
>>
>> http://www.python.org/dev/peps/pep-0458/
>>
>> Whereas latest revisions to the PEP are here:
>>
>> https://github.com/theupdateframework/pep-on-pypi-with-tuf
>>
>> We welcome your feedback and suggestions.
>>
>> Thanks,
>> The PEP 458 team
>>
>>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131121/c36cf332/attachment.html>

From techtonik at gmail.com  Fri Nov 22 08:22:19 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Fri, 22 Nov 2013 10:22:19 +0300
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
In-Reply-To: <52884483.5080803@students.poly.edu>
References: <52884483.5080803@students.poly.edu>
Message-ID: <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>

On Sun, Nov 17, 2013 at 7:22 AM, Trishank Karthik Kuppusamy
<tk47 at students.poly.edu> wrote:
> Hello everyone,
>
> Donald, Justin and I have co-authored a PEP that recommends a
> comprehensive security solution to allow PyPI to secure its users
> against a wide array of compromises.
>
> The gist of the PEP is that the changes to PyPI are essentially
> invisible to users and developers unless an attack is underway.
>
> The key design ideas are as follows:
> ...

These are not design - these are implementation details. What's the idea
about that metadata? I don't get it. I already spent 15 minutes reading
here and there and still can't see any short concept description. Only
vague "end-to-end" "security best practices" buzzwords.


> * Developers do not have to opt-in to secure their projects with their
> own TUF metadata. In that case, PyPI will sign these "unclaimed"
> projects on their behalf. However, unclaimed projects will not be secure
> against a PyPI compromise.

So PyPI will sign malicious packages from compromised developer
account who still uses Python 2 for his enterprise application and was not
careful enough to pay attention to http://bugs.python.org/issue12226
and uploaded his package through unsecure PyCon WiFi network.

Until bugs like this exists, any complication and will lead to "safe
security" feeling, which is much worse than being aware of poor state.

> * To protect against a PyPI compromise, developers may choose to
> register their public keys with Warehouse and upload their own signed
> TUF metadata about their projects.

If developer account is hacked, there is no protection. The real protection
is only known hashsize of the known package of specific version.
Protection against change of package version contents, not against new
malicious release / upload. If PyPI is a central repository, and hashsizes
are not distributed, it is easy to hack PyPI, alter hashsizes, serve
malicious package to few target recipients and return everything back. If
hashsizes are distributed, you can't go unnoticed.

Is it that TUF provides? If yes, what is the distribution mechanism?

> * Therefore, developers do not have to concern themselves with key
> management in case they leave their projects as "unclaimed". When they
> do claim their projects, they simply have to register their keys once
> with Warehouse. After that, they may delegate signing for distributions
> as they wish without depending on Warehouse.

Use case explanation needed. I'd split in two:
1. developers who don't care don't need to care
2. developers who care need to "claim project" by registering keys, and
    then delegate signing without depending on WH to who?

"unclaimed" project. What's this? What is the process of "claiming a
project"? Is there a better terminology? This reads like picking abandoned
project or project without authorship.


> * Clients will be instructed to first search for a project in the more
> secure claimed metadata (protected by offline keys) before looking for
> it in the less secure unclaimed metadata (protected by online keys).

What are the main points that makes offline signatures more secure
once more?

> The official PEP is here:
>
> http://www.python.org/dev/peps/pep-0458/

Too wordy. Complicated. Picture is good, but didn't help. The document
should be readable to ordinary corporate Joe, who knows that he got PGP,
but doesn't know what root keys are signed for. And it should be concise
enough for people with knowledge. I don't know if it is possible to do both.

> We welcome your feedback and suggestions.

From ncoghlan at gmail.com  Fri Nov 22 10:08:33 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Nov 2013 19:08:33 +1000
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
In-Reply-To: <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
Message-ID: <CADiSq7dygf_wo0vecXJqpAQ_GeCuMvRAyjt6Rj+Rc9WazQmE2A@mail.gmail.com>

Anatoly, the target audience for this doc is the Warehouse developers and
the PSF infrastructure team, not end users.

The end user docs will come later.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131122/54631a1a/attachment.html>

From stefan at bytereef.org  Fri Nov 22 16:15:35 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Fri, 22 Nov 2013 16:15:35 +0100
Subject: [Distutils] Download count for external packages on PyPI
Message-ID: <20131122151535.GA14864@sleipnir.bytereef.org>


[I'm not subscribed, could you please cc me?]

Hi,

On the danger of sounding like a broken record:  PyPI displays
incorrect download counts (0) for external packages.

Would it be possible to skip the download count entirely for
external packages?


Stefan Krah




From ncoghlan at gmail.com  Sat Nov 23 00:21:45 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Nov 2013 09:21:45 +1000
Subject: [Distutils] Download count for external packages on PyPI
In-Reply-To: <20131122151535.GA14864@sleipnir.bytereef.org>
References: <20131122151535.GA14864@sleipnir.bytereef.org>
Message-ID: <CADiSq7dHZtiO+xNdunDP=7u50Ff5BbJ1pU6CGGN_OCUDp804Ww@mail.gmail.com>

On 23 Nov 2013 02:32, "Stefan Krah" <stefan at bytereef.org> wrote:
>
>
> [I'm not subscribed, could you please cc me?]
>
> Hi,
>
> On the danger of sounding like a broken record:  PyPI displays
> incorrect download counts (0) for external packages.
>
> Would it be possible to skip the download count entirely for
> external packages?

UI changes for the existing PyPI web service are low priority at the
moment, as development efforts are focused on building out the replacement
service that has full automated test coverage (and is hence significantly
less fragile).

Regards,
Nick.

>
>
> Stefan Krah
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131123/903384d1/attachment.html>

From marius at pov.lt  Sat Nov 23 11:05:36 2013
From: marius at pov.lt (Marius Gedminas)
Date: Sat, 23 Nov 2013 12:05:36 +0200
Subject: [Distutils] PyPI download issues from Rackspace Cloud
Message-ID: <20131123100536.GA30416@fridge.pov.lt>

I recently spin up a Windows VM on Rackspace Cloud.  I'm seeing a very
weird problem: downloads from PyPI fail with checksum errors,
nondeterministically.

Sometimes it's a md5 hash mismatch error from pip[1].  Sometimes the
error is a CRC error deep in the gzip module.  It's not only pip -- I've
seen this error from virtualenv[2], I've seen it from get-pip.py.

[1] https://jenkins.gedmin.as/job/restview-on-windows/1/console
    https://jenkins.gedmin.as/job/restview-on-windows/2/console
    https://jenkins.gedmin.as/job/restview-on-windows/3/console
    https://jenkins.gedmin.as/job/check-manifest-on-windows/11/console

[2] https://jenkins.gedmin.as/job/check-manifest-on-windows/7/console
    https://jenkins.gedmin.as/job/check-manifest-on-windows/6/console

The error is sporadic and tends to go away (and come back) on retries
(look at the restview-on-windows jobs: first pip install mock fails,
then succeeds, then fails again on next retry).

I haven't encountered any download troubles with Firefox.  In fact
that's how I got 'pip install tox' to work in the end -- pypi downloads
of virtualenv/py libraries kept failing, so I downloaded the tarballs
with Firefox and pip installed them from local files.

I've ran

  curl https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.gz | md5sum -

like 10 times and got the same (correct) md5 hash each time.  Meanwhile

  unset PIP_DOWNLOAD_CACHE
  rm -f virtualenv-1.10.1.tar.gz && pip install -d . virtualenv

fails with a bad md5hash error 9 times out of 10.  Curiously, it's the
same bad hash every time: 20cf17baaf2126f99d6f652831f82d75.

I used http://www.python.org/ftp/python/2.7.6/python-2.7.6.msi to
install Python on this Windows 2012 server (64-bit OS, 32-bit Python).

Any ideas?

Marius Gedminas
-- 
IBM motto: "We found five vowels hiding in a corner, and we used
them _all_ for the 'eieio' instruction so that we wouldn't have to use
them anywhere else"
		-- Linus Torvalds
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131123/e309be87/attachment.sig>

From donald at stufft.io  Sat Nov 23 14:59:35 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 23 Nov 2013 08:59:35 -0500
Subject: [Distutils] PyPI download issues from Rackspace Cloud
In-Reply-To: <20131123100536.GA30416@fridge.pov.lt>
References: <20131123100536.GA30416@fridge.pov.lt>
Message-ID: <D79878C3-1B51-4296-83A1-2DEB67CDD525@stufft.io>

Can you try with 1.5rc1? We switched to requests in that version and
perhaps it side steps the issue?

On Nov 23, 2013, at 5:05 AM, Marius Gedminas <marius at pov.lt> wrote:

> I recently spin up a Windows VM on Rackspace Cloud.  I'm seeing a very
> weird problem: downloads from PyPI fail with checksum errors,
> nondeterministically.
> 
> Sometimes it's a md5 hash mismatch error from pip[1].  Sometimes the
> error is a CRC error deep in the gzip module.  It's not only pip -- I've
> seen this error from virtualenv[2], I've seen it from get-pip.py.
> 
> [1] https://jenkins.gedmin.as/job/restview-on-windows/1/console
>    https://jenkins.gedmin.as/job/restview-on-windows/2/console
>    https://jenkins.gedmin.as/job/restview-on-windows/3/console
>    https://jenkins.gedmin.as/job/check-manifest-on-windows/11/console
> 
> [2] https://jenkins.gedmin.as/job/check-manifest-on-windows/7/console
>    https://jenkins.gedmin.as/job/check-manifest-on-windows/6/console
> 
> The error is sporadic and tends to go away (and come back) on retries
> (look at the restview-on-windows jobs: first pip install mock fails,
> then succeeds, then fails again on next retry).
> 
> I haven't encountered any download troubles with Firefox.  In fact
> that's how I got 'pip install tox' to work in the end -- pypi downloads
> of virtualenv/py libraries kept failing, so I downloaded the tarballs
> with Firefox and pip installed them from local files.
> 
> I've ran
> 
>  curl https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.gz | md5sum -
> 
> like 10 times and got the same (correct) md5 hash each time.  Meanwhile
> 
>  unset PIP_DOWNLOAD_CACHE
>  rm -f virtualenv-1.10.1.tar.gz && pip install -d . virtualenv
> 
> fails with a bad md5hash error 9 times out of 10.  Curiously, it's the
> same bad hash every time: 20cf17baaf2126f99d6f652831f82d75.
> 
> I used http://www.python.org/ftp/python/2.7.6/python-2.7.6.msi to
> install Python on this Windows 2012 server (64-bit OS, 32-bit Python).
> 
> Any ideas?
> 
> Marius Gedminas
> -- 
> IBM motto: "We found five vowels hiding in a corner, and we used
> them _all_ for the 'eieio' instruction so that we wouldn't have to use
> them anywhere else"
> 		-- Linus Torvalds
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131123/71ed8ea7/attachment.sig>

From jcappos at nyu.edu  Fri Nov 22 18:06:53 2013
From: jcappos at nyu.edu (Justin Cappos)
Date: Fri, 22 Nov 2013 12:06:53 -0500
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
In-Reply-To: <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
Message-ID: <CAMVss_rhuq0EaMtHbFQiyyfbCc-6Az6hEfAVm_tNAr58V72_1Q@mail.gmail.com>

>
> These are not design - these are implementation details. What's the idea
> about that metadata? I don't get it. I already spent 15 minutes reading
> here and there and still can't see any short concept description. Only
> vague "end-to-end" "security best practices" buzzwords.
>
>
I'm confused by your comments.   Is the document too low level
(implementation details) or high level (vague)?

While the PEP is self-contained, if you want to look at supplemental
materials, the TUF spec has more low level information.   The Ruby folks
have reimplemented TUF based upon it so it seems pretty clear to them.
Conversely, if you want more high level details, the TUF paper is a good
source.   It was published at a top security conference so must have been
readable to (at least) the reviewers.

 > * Developers do not have to opt-in to secure their projects with their
> > own TUF metadata. In that case, PyPI will sign these "unclaimed"
> > projects on their behalf. However, unclaimed projects will not be secure
> > against a PyPI compromise.
>
> So PyPI will sign malicious packages from compromised developer
> account who still uses Python 2 for his enterprise application and was not
> careful enough to pay attention to http://bugs.python.org/issue12226
> and uploaded his package through unsecure PyCon WiFi network.
>

We are trying to protect all users in the case that PyPI or its
infrastructure is compromised.   Ultimately with any software distribution
scheme I am aware of, you trust the author of your software not to be
malicious (or claim the user needs to do something impractical like read
all the code before installing).


> > * To protect against a PyPI compromise, developers may choose to
> > register their public keys with Warehouse and upload their own signed
> > TUF metadata about their projects.
>
> If developer account is hacked, there is no protection. The real protection
> is only known hashsize of the known package of specific version.
> Protection against change of package version contents, not against new
> malicious release / upload. If PyPI is a central repository, and hashsizes
> are not distributed, it is easy to hack PyPI, alter hashsizes, serve
> malicious package to few target recipients and return everything back. If
> hashsizes are distributed, you can't go unnoticed.
>

Look at the document for details, but it isn't possible to do as you say
with the changes detailed in the PEP.   The metadata is laid out so that
even if you compromise every key on the repository, you cannot make
metadata that will have a client trust a project like Django, which manages
its own keys.   This is because the root, targets, claimed and Django keys
are all offline.

> * Therefore, developers do not have to concern themselves with key
> > management in case they leave their projects as "unclaimed". When they
> > do claim their projects, they simply have to register their keys once
> > with Warehouse. After that, they may delegate signing for distributions
> > as they wish without depending on Warehouse.
>
> Use case explanation needed. I'd split in two:
> 1. developers who don't care don't need to care
> 2. developers who care need to "claim project" by registering keys, and
>     then delegate signing without depending on WH to who?
>
> "unclaimed" project. What's this? What is the process of "claiming a
> project"? Is there a better terminology? This reads like picking abandoned
> project or project without authorship.
>

Yes, it is essentially a project where the owner hasn't uploaded a public
key to signal they will manage their own project.   So it seems like you
got the gist of this from the name.

> * Clients will be instructed to first search for a project in the more
> > secure claimed metadata (protected by offline keys) before looking for
> > it in the less secure unclaimed metadata (protected by online keys).
>
> What are the main points that makes offline signatures more secure
> once more?
>

It is not stored on the main server.   The strength of offline / air gapped
keys is well known and used widely in practice.   See the TUF CCS paper for
more details / references.


> > The official PEP is here:
> >
> > http://www.python.org/dev/peps/pep-0458/
>
> Too wordy. Complicated. Picture is good, but didn't help. The document
> should be readable to ordinary corporate Joe, who knows that he got PGP,
> but doesn't know what root keys are signed for. And it should be concise
> enough for people with knowledge. I don't know if it is possible to do
> both.
>

As Nick pointed out, this is meant for PyPI devels, not ordinary Joe.
Ordinary Joe is not going to notice TUF at all unless an attack is underway.

Details for how ordinary Joe publishes a package will be forthcoming, but
will require a very minimal amount of additional effort.

Justin
P.S.   I feel there is quite a bit of confusion.   Feel free to respond to
me directly if you'd prefer to discuss this more without spamming DistUtils
or theupdateframework.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131122/b9be6a87/attachment.html>

From p.f.moore at gmail.com  Sat Nov 23 15:58:23 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 23 Nov 2013 14:58:23 +0000
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
In-Reply-To: <CAMVss_rhuq0EaMtHbFQiyyfbCc-6Az6hEfAVm_tNAr58V72_1Q@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
 <CAMVss_rhuq0EaMtHbFQiyyfbCc-6Az6hEfAVm_tNAr58V72_1Q@mail.gmail.com>
Message-ID: <CACac1F-UHpzqK-ySk+=n6M1T-Bo8e7kAamxKCGqDGE0ZXR6uZw@mail.gmail.com>

On 22 November 2013 17:06, Justin Cappos <jcappos at nyu.edu> wrote:
>> "unclaimed" project. What's this? What is the process of "claiming a
>> project"? Is there a better terminology? This reads like picking abandoned
>> project or project without authorship.
>
>
> Yes, it is essentially a project where the owner hasn't uploaded a public
> key to signal they will manage their own project.   So it seems like you got
> the gist of this from the name.

Personally, I'm not too keen on the term "unclaimed". If I upload, own
and manage a project but don't want to bother with the hassle of
generating and managing signing keys, I don't think that means my
project should be described by the (frankly, somewhat detrimental)
term "unclaimed". "Unsigned" is accurate and specific - "unclaimed"
sounds like I don't care about my project.

Paul

From ncoghlan at gmail.com  Sat Nov 23 16:11:29 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Nov 2013 01:11:29 +1000
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
In-Reply-To: <CACac1F-UHpzqK-ySk+=n6M1T-Bo8e7kAamxKCGqDGE0ZXR6uZw@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
 <CAMVss_rhuq0EaMtHbFQiyyfbCc-6Az6hEfAVm_tNAr58V72_1Q@mail.gmail.com>
 <CACac1F-UHpzqK-ySk+=n6M1T-Bo8e7kAamxKCGqDGE0ZXR6uZw@mail.gmail.com>
Message-ID: <CADiSq7d9Pkz7hCA0GEonOtbzAHjuqpTOkeDiS=gcAqB=FiWFsw@mail.gmail.com>

On 24 Nov 2013 00:58, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 22 November 2013 17:06, Justin Cappos <jcappos at nyu.edu> wrote:
> >> "unclaimed" project. What's this? What is the process of "claiming a
> >> project"? Is there a better terminology? This reads like picking
abandoned
> >> project or project without authorship.
> >
> >
> > Yes, it is essentially a project where the owner hasn't uploaded a
public
> > key to signal they will manage their own project.   So it seems like
you got
> > the gist of this from the name.
>
> Personally, I'm not too keen on the term "unclaimed". If I upload, own
> and manage a project but don't want to bother with the hassle of
> generating and managing signing keys, I don't think that means my
> project should be described by the (frankly, somewhat detrimental)
> term "unclaimed". "Unsigned" is accurate and specific - "unclaimed"
> sounds like I don't care about my project.

That sounds like an incentive for people to use offline keys to me - in
this scheme, that's a feature, not a bug.

Leaving PyPI packages unclaimed is unequivocally *bad*. The PEP only allows
it to ensure it isn't introducing new barriers to entry for software
distribution through PyPI.

We *don't* want people to have to trust the integrity of PyPI - the volume
of damage that can be done by a PyPI compromise is too high when it allows
malicious replacement of most packages. Getting developers to create and
register their own keys has problems of its own, but many manage to do it
effectively for ssh, and that's a closer model for the PEP than the GPG web
of trust.

Cheers,
Nick.

>
> Paul
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131124/8165c568/attachment.html>

From donald at stufft.io  Sat Nov 23 16:23:14 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 23 Nov 2013 10:23:14 -0500
Subject: [Distutils] PEP 458: Surviving a Compromise of PyPI: Round 1
In-Reply-To: <CACac1F-UHpzqK-ySk+=n6M1T-Bo8e7kAamxKCGqDGE0ZXR6uZw@mail.gmail.com>
References: <52884483.5080803@students.poly.edu>
 <CAPkN8x+McjMmMJuwpjGOAFPL5gHufuqkhT1Va777uXiBwadpTw@mail.gmail.com>
 <CAMVss_rhuq0EaMtHbFQiyyfbCc-6Az6hEfAVm_tNAr58V72_1Q@mail.gmail.com>
 <CACac1F-UHpzqK-ySk+=n6M1T-Bo8e7kAamxKCGqDGE0ZXR6uZw@mail.gmail.com>
Message-ID: <2C7A2AD4-D797-4E59-8B81-DEE2A2324C7F@stufft.io>

They are signed. Just not by an author key. 

> On Nov 23, 2013, at 9:58 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> "Unsigned" is accurate and specific - "unclaimed"
> sounds like I don't care about my project.

From techtonik at gmail.com  Sat Nov 23 17:57:45 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 23 Nov 2013 19:57:45 +0300
Subject: [Distutils] PyPI v2 (Was:  PyPI pull request #7)
In-Reply-To: <54B6FCEB-F7BE-4E45-B18B-0A9D12458154@stufft.io>
References: <CAPkN8xJJCcTobGc2mHEkExaELTJPR+Y1Bt187dJvx8So6DQkbA@mail.gmail.com>
 <8100BD88-9CDC-4FFB-89A9-0554F89B91CE@stufft.io>
 <CAPkN8x+a3=zF6Yw=Be5ysdWW_k05ogdCBae7Lam0+LMeGys+NA@mail.gmail.com>
 <54B6FCEB-F7BE-4E45-B18B-0A9D12458154@stufft.io>
Message-ID: <CAPkN8xL71r3zQMp3o6ZkZefWVmC5mh4vxQrK9ohGjKatbHc1zw@mail.gmail.com>

On Sat, Nov 16, 2013 at 8:00 AM, Donald Stufft <donald at stufft.io> wrote:
>
> If people don?t like the requirements of Apache License 2.0 that?s fine they
> don?t need to contribute. Perhaps there might be a contributor or two lost
> to that but I?m not too worried about it.
>
> License is a bike shed issue and as it?s mine and Richard?s bike shed
> it?ll be painted our color.

Just sad to see lawyers are winning battle with simplicity, which makes
more jobs for them and less progress for us in the same time span.

>>>> 3. Why the raw SQL? It cripples the ability to scale, to host package
>>>> indexes on GAE or OpenStack. And it is right at the core. Why not to
>>>> "port NDB" to SQL or use HyperDB from Roundup?
>>>
>>> It was just recently switched to raw sql from using the SQL Alchemy
>>> SQL expression layer. The reason for the switch was we were not
>>> using the portability features and it was more complicated to understand
>>> the expression layer than just plain SQL.
>>
>> Ok. SQL Alchemy is too big and complicated. Why NoSQL didn't fit?
>> MongoDB is all over the net nowadays.
>
> NoSQL is a buzzword that doesn?t mean a whole lot other than ?Not a
> relational database?. The various ?NoSQL? solutions each fit well in
> different scenarios however this is not one of them. The database
> is well within the limits of PostgreSQL and referential integrity, indexes,
> and the like are useful constructs.

Right. That's why it is extremely interesting to learn by example from people
experienced in both. The question "why" that is not answered "because we
know how to work with it", but "because {{this}} and {{that}} use cases are
not solved on MongoDB level, more complicated, slow and not reliable".

>>>> 4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff)
>>>
>>> I?ll be adding a section to the documentation on this eventually, but
>>> basically.
>>>
>>> Django?s ORM wasn?t powerful enough to represent the existing Schema. When
>>> I had to throw out the ORM I found it difficult to integrate other pieces without
>>> instating more global state. Ultimately I felt that a Fraken-Django that used Django
>>> on the surface but with pieces replaced was going to require as much or more
>>> of  a learning curve, even for Django developers, that it didn?t make sense to
>>> do it.
>>
>> Maybe a good partitioning of application with big diagram could bring the best
>> from both world? Bare SQL and getting to the basics seems to radical to me.
>
> I?m not real keen on needing to do big diagrams. I don?t see any issue with writing
> SQL here, it?s a perfectly fine language for querying a database.

You're writing SQL, but nobody except you will understand why did you wrote it
(SQL) here and why SQL is the best choice here. Because there is no high level
overview that gives the details of the problem (queries that needs to be run) in
this particular part of application. Partitioning and blueprints are required to
understand what the code does - they don't specify that this is SQL or NoSQL -
these are implementation details, but blueprint helps to understand what exactly
SQL does and if it could be replaced.

>>> At that point I was a bit frustrated and fed up and turned to using just Werkzeug
>>> as a library. This enabled a rapid increase in pace (I got more down over the
>>> initial weekend then I had in a month previously).
>>
>> Isn't it just a WSGI library? What about reusable components that are already
>> written? I wouldn't risk rewriting OAuth support from scratch.
>
> Werkzeug is just a WSGI library yes. This means that any reusable pieces we use
> cannot be tied to a single framework. This might mean we need to make our own
> reusable pieces and release them. I consider this a benefit to the greater ecosystem
> as there is very little reason an OAuth implementation (for instance) needs to be
> tied to a single framework.

"reusable pieces" on WSGI level? Or on application level? I see
"reusable pieces" in
WSGI context strictly as middleware. In that case you make implicit API which is
highly dependent on middleware order, which makes code more convoluted and not
intuitive. Well, it may be that it is the best solution from
available, but I am not aware
of too much problems with isolating architecture inside application
itself without going
out to the level of global WSGI variables.

If application structure becomes dependent on middleware stack in
addition to API
versions, URL routing scheme and component specific API calls, it
needs a diagram
of all that zoo for every affected state, or some good description of this API.

>>>> PostgreSQL is killing all the fun. It takes a few hours to get
>>>> acquainted with its way of doing things - groups as users, per table
>>>> grants and insufficient DB permissions. There should be an option to
>>>> run on SQlite for development, like in Trac, Roundup etc.
>>>
>>> Warehouse only runs on PostgreSQL. It?s not that hard to have a local copy of
>>> PostgreSQL and It?ll enable using the advanced feature set of PostgreSQL
>>> instead of being limited to the lowest common denominator.
>>
>> What feature sets you need? Does they really worth killing the fun of working,
>> hack-a-toning and for local development?
>
> As I said, running a local copy of PostgreSQLis very easy. There are installers for
> Windows and OSX, and pretty much every *nix distribution will have it packaged.

That's not "easy" as in "clone and run" as it is now with Roundup,
Spyder and many
other projects _accessible_ for hacking and fixing typos and minor bugs.

> Off the top of my head I?m planning on using Array support, HStore, and functional/partial
> indexes.

Do you have examples for people not aware with these details, what
will they be good for?
Is that complication really necessary?

>> 100% of
>> test coverage makes it hard to make changes, sometimes more hard than necessary.
>> Perhaps I should take a look at this.
>
> It raises the bar for adding new code, but decreases the bar for maintaing the existing
> code. Given the fact that the code will be maintained more than new features will
> be written I consider this a positive.

Ok, I accept it.

>> BTW, why didn't you just copy crate.io?
>
> Crate?s code base suffers from a lot of bad assumptions that are baked into it when
> I was still learning how packaging really worked. On top of that most of the pieces
> that we need to replace PyPI don?t exist in Crate since Crate just mirrored PyPI. As it
> is Warehouse almost is at feature parity with Crate (but not PyPI).

I may be asking too much, but I am really interested to learn from these bad
assumptions. There is a good chance that I and other people who will be willing
to contribute still have them.

>>> It?s not a reinvention of Django. The ?framework? parts of Warehouse
>>> are very small and are mostly glue that holds together pieces like
>>> Werkzeug and SQLAlchemy.
>>
>> Still, is there a diagram of Warehouse architecture? Blueprint,
>> CAD model of a building?
>
> I have a todo list item to write up architecture documentation that explains
> the layout and where the different pieces are at.

Will be much appreciated. Thank you.

From stefan at bytereef.org  Sun Nov 24 18:07:50 2013
From: stefan at bytereef.org (Stefan Krah)
Date: Sun, 24 Nov 2013 18:07:50 +0100
Subject: [Distutils] Download count for external packages on PyPI
In-Reply-To: <CADiSq7dHZtiO+xNdunDP=7u50Ff5BbJ1pU6CGGN_OCUDp804Ww@mail.gmail.com>
References: <20131122151535.GA14864@sleipnir.bytereef.org>
 <CADiSq7dHZtiO+xNdunDP=7u50Ff5BbJ1pU6CGGN_OCUDp804Ww@mail.gmail.com>
Message-ID: <20131124170750.GA27565@sleipnir.bytereef.org>

Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Would it be possible to skip the download count entirely for
> > external packages?
> 
> UI changes for the existing PyPI web service are low priority at the moment, as
> development efforts are focused on building out the replacement service that
> has full automated test coverage (and is hence significantly less fragile).

I see.  I thought that the newly displayed download count for external packages
(IMHO it did not show up a year ago or so?) was part of the effort to redesign
PyPI, so I assumed it was a regression.


Stefan Krah



From ncoghlan at gmail.com  Sun Nov 24 23:34:11 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Nov 2013 08:34:11 +1000
Subject: [Distutils] Download count for external packages on PyPI
In-Reply-To: <20131124170750.GA27565@sleipnir.bytereef.org>
References: <20131122151535.GA14864@sleipnir.bytereef.org>
 <CADiSq7dHZtiO+xNdunDP=7u50Ff5BbJ1pU6CGGN_OCUDp804Ww@mail.gmail.com>
 <20131124170750.GA27565@sleipnir.bytereef.org>
Message-ID: <CADiSq7ca7wOm8hi7nqh+cwgk3tVkBsHjKpAtqJYJT8EL+PggZA@mail.gmail.com>

On 25 Nov 2013 08:09, "Stefan Krah" <stefan at bytereef.org> wrote:
>
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > > Would it be possible to skip the download count entirely for
> > > external packages?
> >
> > UI changes for the existing PyPI web service are low priority at the
moment, as
> > development efforts are focused on building out the replacement service
that
> > has full automated test coverage (and is hence significantly less
fragile).
>
> I see.  I thought that the newly displayed download count for external
packages
> (IMHO it did not show up a year ago or so?) was part of the effort to
redesign
> PyPI, so I assumed it was a regression.

It probably *is* a regression, as it was likely a result of the switch to
reporting downloads based on information from the Fastly CDN (originally
download counts were disabled entirely after the CDN was introduced, but
that change caused a large number of immediate complaints, and Fastly
offered the additional API access needed to fix it).

A patch to switch it to showing "No download information available for
externally hosted packages" when appropriate might be acceptable, but that
would be up to Donald and Richard.

Cheers,
Nick.

>
>
> Stefan Krah
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131125/8d969190/attachment.html>

From marius at pov.lt  Mon Nov 25 07:48:05 2013
From: marius at pov.lt (Marius Gedminas)
Date: Mon, 25 Nov 2013 08:48:05 +0200
Subject: [Distutils] PyPI download issues from Rackspace Cloud
In-Reply-To: <D79878C3-1B51-4296-83A1-2DEB67CDD525@stufft.io>
References: <20131123100536.GA30416@fridge.pov.lt>
 <D79878C3-1B51-4296-83A1-2DEB67CDD525@stufft.io>
Message-ID: <20131125064805.GA19537@fridge.pov.lt>

On Sat, Nov 23, 2013 at 08:59:35AM -0500, Donald Stufft wrote:
> Can you try with 1.5rc1?

This was trickier than I thought, because pip appears to be incapable of
upgrading itself on Windows:

  $ git clone https://github.com/pypa/pip
  $ virtualenv env
  $ env/scripts/pip install -U ./pip
  WindowsError: [Error 5] Access is denied: 'c:\\users\\mg\\...\\scripts\\pip.exe'

but the advice in https://github.com/pypa/pip/issues/1299 helped:

  $ env/scripts/python -m pip -U ./pip

> We switched to requests in that version and perhaps it side steps the issue?

Looks like.  10 out of 10 successful downloads with pip git master using:

  $ unset PIP_DOWNLOAD_CACHE
  $ rm -f virtualenv-1.10.1.tar.gz && env/scripts/pip install -d .  virtualenv

As a control, I tested again with pip 1.4.1, and 2 out of 4 downloads
failed with a bad hash error.

Interesting.

Marius Gedminas
-- 
Those parts of the system that you can hit with a hammer (not advised)
are called hardware; those program instructions that you can only curse
at are called software.
                -- Levitating Trains and Kamikaze Genes: Technological
                   Literacy for the 1990's.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131125/2f4df33a/attachment.sig>

From marius at pov.lt  Mon Nov 25 09:38:09 2013
From: marius at pov.lt (Marius Gedminas)
Date: Mon, 25 Nov 2013 10:38:09 +0200
Subject: [Distutils] PyPI download issues from Rackspace Cloud
In-Reply-To: <20131125064805.GA19537@fridge.pov.lt>
References: <20131123100536.GA30416@fridge.pov.lt>
 <D79878C3-1B51-4296-83A1-2DEB67CDD525@stufft.io>
 <20131125064805.GA19537@fridge.pov.lt>
Message-ID: <20131125083809.GA13083@fridge.pov.lt>

Digging deeper:

- SSLSocket.read() returns a premature EOF, truncating the downloaded
  file, which makes the md5 hash to be different.

- if I fish out the SSLSocket object and set suppress_ragged_eofs = False,
  then I get

    ssl.SSLError: [Errno 8] _ssl.c:1426: EOF occurred in violation of protocol

- SSLSocket.read(8192) returns chunks of 8000 bytes except the very
  first one (7669 bytes) and, in cases when the download does _not_
  fail, the last one (5634 bytes).  When the download fails, it's
  missing one or two last chunks (it varies).

- SSLSocket.read(16384) does the same; SSLSocket.read(4096) returns
  pairs of chunks of 4096 and 3904 bytes, hinting that the underlying
  SSL communication works in multiples of 8000.

My experimental code: https://gist.github.com/mgedmin/7637559

I now have two Wireshark pcap files of two SSL conversations: one
failed, one successful.  It's a bit beyond my skills to analyze them.
I think the server did send everything (I see a TLSv1 Application Data
record of size 5654, which looks like the final chunk), but Wireshark
marks it as [TCP Out-Of-Order], and then there are some
desperate-looking [TCP Retransmission] packets at the end.

(Incidentally, the captured download was truncated at 1303669
bytes -- i.e. it was missing the last three TLS application data chunks
of 8000, 8000 and 5634 bytes.)

Given that Firefox/curl seem to be able to download PyPI packages over
HTTPS without problems, I'd be inclined to blame it on a bug in the
CPython's ssl module, maybe.  But doesn't Requests use it too?

Confused,
Marius Gedminas
-- 
MCSE == Marginal Computer Software Enthusiast
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131125/5ca7ee95/attachment.sig>

From ncoghlan at gmail.com  Thu Nov 28 14:18:53 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 28 Nov 2013 23:18:53 +1000
Subject: [Distutils] Restructuring metadata 2.0 to leverage the extension
	model
Message-ID: <CADiSq7ej+w=cRpSCrKDvV6K5mzkECVFAdo0KwWfEXan0QcTa-A@mail.gmail.com>

It's (very) early days at this point, but I've taken the first steps
towards breaking up the metadata 2.0 monolith into multiple relatively
independent metadata extensions:

https://bitbucket.org/pypa/pypi-metadata-formats/commits/2a4692ee5d1181fdbb8008318c2a1aea7f14a0f8

Finalising metadata 2.0 is still in the "post CPython 3.4, leading up
to pip 1.6" time frame, but with 3.4 hitting feature freeze last
weekend with a mostly integrated pip, I figured it was time to share
at least something about my current plans for this, even if it's
another few months before I update it again.

The general aim is that the stuff that was previously identified as
the "essential dependency metadata" will be all that remains directly
defined in PEP 426, with everything else moved out to metadata
extensions.

A couple of other notable transitions started in this commit:

- install_hooks are out, metadata_hooks are in (as an extension). The
general idea is that you can register an interest in export groups and
other extensions, and installation tools will invoke your hooks at the
appropriate time. Still a lot of details to be worked out, but I
already like it much better than the previous install_hook model.

- started adding a formal notion of an "integrator suffix" to the
versioning PEP. This is a matter of recognising the fact that
redistributors will often apply additional patches to their
redistributed version, while still aiming to preserve the API and most
of the behaviour of the upstream release.

One key advantage of this model is that I'm proposing for the
extensions to be independently versioned, so we'd only have to update
the main metadata PEP in order to change the essential dependency
resolution metadata.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Nov 28 23:45:16 2013
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Nov 2013 08:45:16 +1000
Subject: [Distutils] Restructuring metadata 2.0 to leverage the
	extension model
In-Reply-To: <CADiSq7ej+w=cRpSCrKDvV6K5mzkECVFAdo0KwWfEXan0QcTa-A@mail.gmail.com>
References: <CADiSq7ej+w=cRpSCrKDvV6K5mzkECVFAdo0KwWfEXan0QcTa-A@mail.gmail.com>
Message-ID: <CADiSq7dUKo-mT27_doc6cXoAuBJMcMNr303MmsC3L6HSLOxS7Q@mail.gmail.com>

On 28 Nov 2013 23:18, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> It's (very) early days at this point, but I've taken the first steps
> towards breaking up the metadata 2.0 monolith into multiple relatively
> independent metadata extensions:
>
>
https://bitbucket.org/pypa/pypi-metadata-formats/commits/2a4692ee5d1181fdbb8008318c2a1aea7f14a0f8
>
> Finalising metadata 2.0 is still in the "post CPython 3.4, leading up
> to pip 1.6" time frame, but with 3.4 hitting feature freeze last
> weekend with a mostly integrated pip, I figured it was time to share
> at least something about my current plans for this, even if it's
> another few months before I update it again.
>
> The general aim is that the stuff that was previously identified as
> the "essential dependency metadata" will be all that remains directly
> defined in PEP 426, with everything else moved out to metadata
> extensions.
>
> A couple of other notable transitions started in this commit:
>
> - install_hooks are out, metadata_hooks are in (as an extension). The
> general idea is that you can register an interest in export groups and
> other extensions, and installation tools will invoke your hooks at the
> appropriate time. Still a lot of details to be worked out, but I
> already like it much better than the previous install_hook model.
>
> - started adding a formal notion of an "integrator suffix" to the
> versioning PEP. This is a matter of recognising the fact that
> redistributors will often apply additional patches to their
> redistributed version, while still aiming to preserve the API and most
> of the behaviour of the upstream release.
>
> One key advantage of this model is that I'm proposing for the
> extensions to be independently versioned, so we'd only have to update
> the main metadata PEP in order to change the essential dependency
> resolution metadata.

Something that *isn't* in this model yet is a requirement resolution hook
to allow system package managers to intercept (and potentially resolve)
package installation and removal requests in the system Python, while
leaving virtual environments alone.

That's definitely just a wish list item though - even if we can't figure
out a nicer way to handle that, we're still no worse off than we are now.

Cheers,
Nick.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131129/d1c1aaad/attachment.html>

From techtonik at gmail.com  Sat Nov 30 12:05:48 2013
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 30 Nov 2013 14:05:48 +0300
Subject: [Distutils] .. *explicitly* mark the package as pre-release
Message-ID: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>

hi, please cc

i want to release some versions to pypi, but hide them from installing
by explicitly marking as pre-release. i want to preserve versions
numbering as 0.1, 0.2 according to semver.org point 4. is that
possible?
--
anatoly t.

From p.f.moore at gmail.com  Sat Nov 30 12:55:09 2013
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 30 Nov 2013 11:55:09 +0000
Subject: [Distutils] .. *explicitly* mark the package as pre-release
In-Reply-To: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
References: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
Message-ID: <CACac1F9HCGQKGTCBHw+BVcfjkhvEXFsoztqemmef=Zad7aybUg@mail.gmail.com>

On 30 November 2013 11:05, anatoly techtonik <techtonik at gmail.com> wrote:
> hi, please cc
>
> i want to release some versions to pypi, but hide them from installing
> by explicitly marking as pre-release. i want to preserve versions
> numbering as 0.1, 0.2 according to semver.org point 4. is that
> possible?

No, I don't believe so.
Paul

From richard at python.org  Sat Nov 30 22:59:40 2013
From: richard at python.org (Richard Jones)
Date: Sun, 1 Dec 2013 08:59:40 +1100
Subject: [Distutils] .. *explicitly* mark the package as pre-release
In-Reply-To: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
References: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
Message-ID: <CAHrZfZD8nuKK6WawpVivNHkoJTXQZ+9ONnOwXvZC1sbmhR65nA@mail.gmail.com>

There is no way to mark a release as "pre-release" but you can hide it from
view which would prevent an installer discovering any files related to the
release. Log into PyPI and use the "releases" page for the project.



On 30 November 2013 22:05, anatoly techtonik <techtonik at gmail.com> wrote:

> hi, please cc
>
> i want to release some versions to pypi, but hide them from installing
> by explicitly marking as pre-release. i want to preserve versions
> numbering as 0.1, 0.2 according to semver.org point 4. is that
> possible?
> --
> anatoly t.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131201/e921be28/attachment.html>

From donald at stufft.io  Sat Nov 30 23:10:48 2013
From: donald at stufft.io (Donald Stufft)
Date: Sat, 30 Nov 2013 17:10:48 -0500
Subject: [Distutils] .. *explicitly* mark the package as pre-release
In-Reply-To: <CAHrZfZD8nuKK6WawpVivNHkoJTXQZ+9ONnOwXvZC1sbmhR65nA@mail.gmail.com>
References: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
 <CAHrZfZD8nuKK6WawpVivNHkoJTXQZ+9ONnOwXvZC1sbmhR65nA@mail.gmail.com>
Message-ID: <4640B83A-03C1-4DA9-8F81-42665C2799E6@stufft.io>

Hiding a release doesn?t hide it from the simple index, so installers will still find it.

On Nov 30, 2013, at 4:59 PM, Richard Jones <richard at python.org> wrote:

> There is no way to mark a release as "pre-release" but you can hide it from view which would prevent an installer discovering any files related to the release. Log into PyPI and use the "releases" page for the project.
> 
> 
> 
> On 30 November 2013 22:05, anatoly techtonik <techtonik at gmail.com> wrote:
> hi, please cc
> 
> i want to release some versions to pypi, but hide them from installing
> by explicitly marking as pre-release. i want to preserve versions
> numbering as 0.1, 0.2 according to semver.org point 4. is that
> possible?
> --
> anatoly t.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131130/efa21b1e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131130/efa21b1e/attachment.sig>

From richard at python.org  Sat Nov 30 23:34:04 2013
From: richard at python.org (Richard Jones)
Date: Sun, 1 Dec 2013 09:34:04 +1100
Subject: [Distutils] .. *explicitly* mark the package as pre-release
In-Reply-To: <4640B83A-03C1-4DA9-8F81-42665C2799E6@stufft.io>
References: <CAPkN8xKZwpgVyyhtRkopmPn3npV9SkXEv+e30U-p=hKTKBBAGA@mail.gmail.com>
 <CAHrZfZD8nuKK6WawpVivNHkoJTXQZ+9ONnOwXvZC1sbmhR65nA@mail.gmail.com>
 <4640B83A-03C1-4DA9-8F81-42665C2799E6@stufft.io>
Message-ID: <CAHrZfZDC=41R_92R3VbppmvciTANG-r-yunRifDtPwj3c8_ugQ@mail.gmail.com>

I was working from (clearly hazy) memory. Sorry for the red herring.


On 1 December 2013 09:10, Donald Stufft <donald at stufft.io> wrote:

> Hiding a release doesn?t hide it from the simple index, so installers will
> still find it.
>
> On Nov 30, 2013, at 4:59 PM, Richard Jones <richard at python.org> wrote:
>
> There is no way to mark a release as "pre-release" but you can hide it
> from view which would prevent an installer discovering any files related to
> the release. Log into PyPI and use the "releases" page for the project.
>
>
>
> On 30 November 2013 22:05, anatoly techtonik <techtonik at gmail.com> wrote:
>
>> hi, please cc
>>
>> i want to release some versions to pypi, but hide them from installing
>> by explicitly marking as pre-release. i want to preserve versions
>> numbering as 0.1, 0.2 according to semver.org point 4. is that
>> possible?
>> --
>> anatoly t.
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131201/3ee1764b/attachment-0001.html>