[pytest-dev] plugin status page

[holger krekel]

Hi Bruno,

On Tue, Nov 12, 2013 at 22:51 -0200, Bruno Oliveira wrote:
> Hi Holger, all,
> 
> On Tue, Nov 12, 2013 at 7:25 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> > > I'm thinking we can use those URLs to embed status images for the latest
> >  > pytest version into the plugin index page, similar to the summary table
> > at
> > > the start of the pytest-plugs heroku app.
> > >
> > > As I noted earlier, the tests reported for pytest-2.3.5 are actually for
> > > 2.4.2 since I didn't overwrite the environment in the tox.ini files. I
> > will
> > > try the approach of replacing the "pytest" dependency manually, unless
> > you
> > > have another suggestion.
> >
> > Are the tests run via creating a ``tox.ini`` yourself, overriding any
> > tox.ini that might be contained with the package?
> >
> 
> I'm creating a tox.ini file if there is none, or if there is already a
> tox.ini inplace I try to use a simple regex to replace "pytest" occurrences
> to "pytest==$PYTEST_VERSION" in an attempt to enforce the pytest version
> that should be used for running the tests. This has worked for most
> plugins, but some of them setup the dependencies in a slightly different
> manner than what my simple regex-replace trick could handle:
> 
> * Some packages declare a specific pytest a version as dependency, like
> "pytest==2.4.0" or "pytest>=2.0".
> * One of the packages installed the dependencies from a requirements file,
> using "-r requirements-tests.txt" instead of listing the dependencies on
> the tox.ini file itself.
> 
> I could of course make my regex more complex in an attempt to cover all the
> cases, but I feel this would mean a lot of duplicated work that is already
> done by tox itself. Any suggestions?

I think we should allow instructing tox to override 
a version for a dependency.  Something like 

    tox --force-dep-version=pytest==2.4.2

This would make tox replace any "pytest" dependency that is specified
with tox itself.  It would not work for implicit dependencies 
specified within a setup.py's install_requires setting.  I guess
we should extend the result-json format to include a list of all
packages and dependencies for each environment so that we can do
a double-check after tests were run.

> In any case, it would be useful to everyone (plugin authors, users,
> > pytest core devs) to see what is actually breaking (or working), i.e.
> > the actual test failure/success log.  Is that available somewhere already?
> >
> 
> Hmm good point. It is available from the travis-ci output log, but that is
> not very friendly. I will change the travis-ci jobs to also post the
> results from "tox --result-json" which contains all the details from that
> test session, and make that available from the webpage as well.
> 
> So I'm thinking this should be the next steps:
> 
> 1. Update plugins index page to include the status images;
> 2. Post test results to the pytest-plugs page and make it available from
> there;
> 3. Find a nice way to ensure we are running "tox" with the pytest version
> we want;

makes sense.

> About 3), I just had an idea... what if we install a devpi server that
> would serve only the pytest version we want (say, 2.3.5) and force tox to
> use that devpi server by using the "-i" option? Would that be possible?
> ("Shadowed Release Files" seems to be what would be required for this to
> work).

See my suggestion above.  I think it would be more effort to do it
indirectly via devpi-server here.

cheers,
holger

> Cheers,
> Bruno.