[pytest-dev] plugin status page

[holger krekel]

On Thu, Oct 24, 2013 at 05:08 -0200, Bruno Oliveira wrote:
> Hi Holger,
> 
> On Thu, Oct 24, 2013 at 4:52 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > Hi Bruno,
> >
> > On Wed, Oct 23, 2013 at 21:51 -0200, Bruno Oliveira wrote:
> > > Hi Holger, all,
> > >
> > > <snip>
> > > I have updated the script to create a tox.ini file that just uses
> > "py.test
> > > --help" as a test command for those packages that don't have one. Now the
> > > script tests all pytest plugins in pypi, please take a look at the
> > summary
> > > at the bottom of the page:
> > >
> > > https://www.travis-ci.org/nicoddemus/pytest-plugs/jobs/12959702
> >
> > looks good!  I guess having a py27 and py33 row would be great
> > to also determine py3 compatibility/installability.
> >
> 
> Those are already in place as different build jobs, please take a look:
> 
> https://www.travis-ci.org/nicoddemus/pytest-plugs

Ah, see it now.

> Currently we have py27 and py33 against pytest 2.3.5 and 2.4.2. Adding more
> python/pytest versions is easy, we just have to add them to the .travis
> file. :)
> 
> https://github.com/nicoddemus/pytest-plugs/blob/master/.travis.yml
 
OK.  I think you can leave "--use-mirrors" away, btw.

> >
> > > I guess it would be useful to identify which plugins are using just
> > > "py.test --help" as smoke test and which ones have real tests, since the
> > > former doesn't guarantee that the plugin actually works on that
> > > python/pytest combination.
> > >
> > > I looked into devpi to simplify the download/unpack/test process and
> > > stumbled in a minor roadblock: is there anyway to provide a tox.ini file
> > > for those plugins that don't have one? This is the only issue I see right
> > > now with using devpi for this particular application.
> >
> > Now, but i am going to add a "devpi test -c path-to-ini" option today
> > and plan to release devpi-1.2 real soon now.  I'll get back here when it's
> > implemented.
> >
> 
> Perfect, that would greatly simplify things. Let me know if there is
> anything I can do to help.

I just implemented it:

https://bitbucket.org/hpk42/devpi/commits/3a8ff0503599358deeefedc581e88c03c4b94ae3

However, it's maybe not exactly what we need.  "devpi test PKG" works by
looking up the newest PKG release on the index, downloading and unpacking 
it and then running tox.  Passing "devpi test -c tox.ini" will always
prefer the external tox.ini, though.  We could maybe add a second option
"--fallback-ini" option that only is used if there is no tox.ini in the
downloaded package.

Note that to play with the devpi-{client,server} current trunk you can do:

    pip install --pre -i https://devpi.net/hpk/dev/+simple/ -U devpi

and then run your own devpi-server similar to:

    http://doc.devpi.net/latest/quickstart-releaseprocess.html

And especially try "devpi list PKG" after "devpi test PKG" to see
test results.

> > So, what do you think of the idea of using devpi.net to host test results
> > > and add an endpoint to the API to provide test status images? Or should I
> > > create another app to do this?
> >
> > What do you mean with "test status images"?
> 
> In general i am open to extending devpi-server to provide more test-status
> > info -- just didn't have a chance to think about it in detail yet.
> > I guess we could take the plugin testing bits as a starting point.
> >
> 
> I mean like those badge images served by travis that give the current build
> status, for example:
> 
> https://secure.travis-ci.org/hpk42/pytest.png
> 
> I think we can follow the same approach, generating a badge that gives the
> full compatibility status, like a single "py 2.6 2.7 | pytest 2.3.5 |
> compatible" image, or even multiple images.
> 
> But I guess this is too specific to pytest...

Hum, some badge makes sense.  Not sure which, though :)

holger

> 
> Cheers,
> Bruno.
> 
> 
> >
> > best,
> > holger
> >
> > > Cheers,
> > > Bruno.
> > >
> > >
> > > > I found the reason for that (http vs https) and now the job runs
> > against
> > > > Python 2.6, 2.7, 3.2 and 3.2.
> > > >
> > > > Thanks!  Got back but caught a flue ...
> > > >>
> > > >
> > > > It happens, hope you get better soon. :)
> > > >
> > > > Cheers,
> > > > Bruno.
> > > >
> > > >
> > > > On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger at merlinux.eu>
> > wrote:
> > > >
> > > >> On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote:
> > > >> > > > >
> > > >> > > > > - i'd collapse NAME and VERSION columns to save space, i.e.
> > > >> > > > >   "pytest-bdd-0.6.1".
> > > >> > > > >
> > > >> > > > > - what about adding download numbers?
> > > >> > > > >
> > > >> > > > > - as to code organisation: you can leave it as is for now or
> > > >> (maybe
> > > >> > > > >   better) put all code and generated files into a dedicated
> > > >> directory
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> > Done, opened a PR for discussion and reviewing. :)
> > > >>
> > > >> only got to it now, looks good so far!
> > > >>
> > > >> > > > > - we should try to collect repository locations. maybe parsing
> > > >> > > > >   for github/bitbucket urls would yield most of them
> > > >> automatically?
> > > >> > > > >   Maybe we need to add some manually.
> > > >> > > > >
> > > >> > > >
> > > >> > > > May I ask why we would need the repository locations? I mean, to
> > > >> work on
> > > >> > > > the compatibility feature we can work directly with packages in
> > > >> pypi or
> > > >> > > > devpi...
> > > >> > >
> > > >> > > purely for the person reading the page and wanting to look at the
> > code
> > > >> > > or do a PR.
> > > >> > >
> > > >> >
> > > >> > Oh I see! Well, parsing the urls seems problematic, because we also
> > > >> have to
> > > >> > take in account the username ("hpk42/pytest" for example). I looked
> > at
> > > >> the
> > > >> > GitHub and BitBucket's rest APIs to see if we could use them for
> > > >> searching,
> > > >> > but it seems only GitHub at the moment has a search API (and in beta
> > > >> stage
> > > >> > at that).
> > > >> >
> > > >> > I think we will have to maintain the repository list manually.
> > Package
> > > >> > authors can help as well by simply opening PRs adding repositories
> > for
> > > >> > their plugins I guess.
> > > >>
> > > >> What about generating a "plugin details " page for each plugin so we
> > don't
> > > >> have to think too hard about what to put in the index page?
> > > >>
> > > >> > > A very interesting bit will be the "2.4.2 compat" and "dev" compat
> > > >> > > > <snip>
> > > >> > >
> > > >> >
> > > >> > Some late night quick hacking and I have come up with this proof of
> > > >> concept:
> > > >> >
> > > >> > https://www.travis-ci.org/nicoddemus/pytest-plugs
> > > >> >
> > > >> > As it is right now it serves only as a prove of concept, of course.
> > The
> > > >> > idea is using travis to handle running things for us, driving it
> > using a
> > > >> > script that downloads and run tests using tox. We can then collect
> > json
> > > >> > information from that and POST those results back to devpi. Also we
> > can
> > > >> use
> > > >> > its build matrix to test against different pytest versions, for
> > example.
> > > >>
> > > >> Hum, interesting.  Is this mainly to make use of travis-ci's build
> > hosts?
> > > >> What is the advantage over just invoking "devpi test PLUGIN-NAME"
> > from the
> > > >> script to automate the download/unpack/test/post-results steps?
> > > >>
> > > >> > I guess we can implement in devpi a concept similar to travis build
> > > >> status
> > > >> > imagens, which we could then use in the plugins_index page.
> > > >> >
> > > >> > Against python 2.6 the script is failing, I have yet to investigate.
> > > >> >
> > > >> > great, i am travelling to Pycon DE tomorrow morning but should be
> > online
> > > >> > >  from time to time.
> > > >> > >
> > > >> >
> > > >> > Nice tripe and good PyCon!
> > > >>
> > > >> Thanks!  Got back but caught a flue ...
> > > >>
> > > >> cheers,
> > > >> holger
> > > >>
> > > >
> > > >
> >