[Distutils] Getting more momentum for pip

Thu Mar 5 20:58:53 CET 2015

> On Mar 5, 2015, at 2:53 PM, Donald Stufft <donald at stufft.io> wrote:
> 
>> 
>> On Mar 5, 2015, at 2:34 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> 
>> On 5 March 2015 at 18:58, Donald Stufft <donald at stufft.io> wrote:
>>> Yet another issue is that pip’s test suite is not particularly very good.
>>> We’re missing a lot of coverage and we don’t have *any* CI running on
>>> platforms other than Ubuntu. This means that merging things is somewhat
>>> “dangerous” because it’s easy to break things without noticing unless you
>>> pull down the change and manually test things you can try. Even then that’s
>>> not good enough unless you can test it on other platforms as well. I’m sure
>>> Paul can fill in the blank on how often the test suite simply doesn’t run on
>>> Windows because of some POSIX assumption snuck in somewhere.
>> 
>> The test suite is pretty much broken on Windows, from what I recall. I
>> intended at one point to try to get it running cleanly on Windows, but
>> it was soul-destroyingly hard work, and I never got very far with it.
>> Given that there's no good Windows CI service (Appveyor is great but
>> it's very slow even on simple projects, and I think it has limits on
>> how long test suites can run so I'm not even sure we could run pip's
>> tests on it) I fear that any work done getting the test suite working
>> on Windows would pretty quickly regress...
>> 
>> If there was one thing on the infrastructure and support side of
>> things that would help enormously, it would be someone setting up CI
>> services for more platforms - Windows in particular, but things like
>> the ancient RHEL systems that people keep having issues on would also
>> be good. And resource willing to get the test suite working on those
>> platforms. (I'd be happy to help someone work on fixing the test suite
>> on Windows, but I really don't have the time to do it all myself).
> 
> Yea, I wouldn’t personally put effort into fixing the test suite on Windows
> without something in place to ensure it doesn’t break again.
> 
> I know the folks behind Travis CI. I know they were looking at adding Windows
> support and said that if someone can write a go app that boots up a Windows
> Azure instance and then uses winrm to run a command on it, even echo, that
> would get them a big step of the way towards being able to support it.
> 
> OSX and lots of other POSIX based systems are also not covered of course,
> it would be great to setup something that other people can contribute
> build machines to. We can spin up anything on a Rackspace cloud (and probably
> other clouds too) but some things people might care about (AIX?) we can’t do.
> I think it wouldn’t be unreasonable to say that for things we can’t run a
> builder ourselves for, that people who care about that platform needs to
> provide us with a suitable instance.
> 
> None of that matters much though without something that allows us to run tests
> on more platforms than whatever Travis provides us. Ideally this would support
> PR based testing (which means it needs some sort of VM or isolation support to
> do it securely) but if some platforms can’t be easily virtualized like that then
> a post merge trigger is acceptable too.
> 
>> 
>>> Other things that would help are:
>>> 
>>> * People doing in-depth reviews of the current PRs that are there and
>>> suggesting changes or pointing out issues, etc.
>> 
>> Very much so. Anyone can add review comments to PRs. We could (and
>> probably should) document some quality guidelines for PRs (must
>> include a test, must include docs if there's a user-visible impact,
>> must be cross-platform, must work on Python 2 and 3). Having more
>> people who can test PRs on a wider range of platforms (Windows!!!)
>> would be great too - a simple comment "checked and confirmed on
>> Windows" is a great help.
> 
> Yea, I don’t have a Windows machine so I’m often times just guessing if
> it works on Windows, or pinging you for it. For major things I can spin
> up a Windows VM but that takes a good 30+ minutes to do which again goes
> back to that our current setup has a lot of time wasters for pip core.
> 
>> 
>>> * People triaging issues (unfortunately this one isn’t super easy with
>>> GitHub Issues since you have to be a committer to change these things).
>> 
>> Hmm, that's a problem - but yes, even if they can only add comments,
>> saying "Please close as unreproducible", "Duplicate of XXX", "Please
>> add label YYY" would be helpful. The committers could trawl such
>> comments occasionally and action them.
> 
> I personally get emails for every issue, closing duplicates or adding
> labels and such is something that takes 15 seconds to do if someone leaves
> a comment like that. Another option is to move our issue tracker off of
> Github into something else that supports non-committers being able to
> manage the issue tracker. Empowering non core to do more things is another
> thing that would be useful and requires someone to take the time to figure
> out how we can best do that (switch away from GH issues? To What?) and then
> actually do the work to make it happen (create salt states to deploy, create
> scripts to migrate etc).

Oh, and another thing that would empower non core to do things is either figuring
out a way to allow non core to restart Travis CI builds (whether this is something
we run that interacts with the Travis CI API, or helping get Travis CI to support
authors restarting builds on their own PRs or whatever) or coming up with a
proposal for something we can use instead of Travis that solves some of these issues.

Yet another improvement would be to figure out how we can make the test suite not
randomly fail as much. Likely this is going to involve finding the places where we’re
reaching out to places on the internet (PyPI, Github, etc) and instead mocking out
that interaction or making a test server that provides whatever we’re using that
external service for and then spinning up a copy of that test server whenever we run
the tests and run it against that instead.

> 
>> 
>>> * People going through and reviewing old issues and PRs to try and figure
>>> out if the situations that caused them to be opened originally still apply
>>> or if that problem has been fixed or if the code has changed significantly
>>> enough that it’s likely to not longer exist.
>> 
>> Oh yes, please. And in particular, the old issues with repeated "+1"
>> or "me, too" comments, ping all the people who said "me too" and ask
>> them if they can provide a patch. And weeding out issues that only
>> apply when using ancient versions of setuptools, that sort of thing.
> 
> I try to do this periodically but I mostly only do the ones that I can tell from
> a glance that are safe to close.
> 
>> 
>>> These sorts of things would make it *much* easier to merge new things
>>> because there would be less risk and less things involved in actually going
>>> through and figuring out if any particular merge is a good idea or not. I
>>> also think that people willing to put in the work to do things like this
>>> would be good candidates for becoming core developers themselves, which
>>> would also help by increasing the number of people we have able to review
>>> and commit.
>> 
>> +1
>> 
>> Paul
> 
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150305/9018de80/attachment.sig>